I'm writing an small program to put tweets from the Twitter public stream into an HBase database. The program uses two threads, one to collect the tweets and one to process them.
The first thread uses twitter4j StatusListener to get the tweets and puts them in an ArrayBlockingQueue with an capacity of 100.
The second thread takes an status from the queue, filters the needed data and moves them to the database.
The processing takes more time, than the collecting of the status.
The producer looks like this:
public void onStatus(Status status) {
try {
this.queue.put(status);
} catch(Exception ex) {
ex.printStackTrace();
}
}
The consumer uses take and calls an function to process the new status:
public void run() {
try {
while(true) {
// Get new status to process
this.status = this.queue.take();
this.analyse();
}
} catch(Exception ex) {
ex.printStackTrace();
}
}
In the main function the two threads were created and started:
ArrayBlockingQueue<Status> queue_public = new ArrayBlockingQueue<Status>(100);
Thread ta_public = new Thread(new TweetAnalyser(cl.getOptionValue("config"), queue_public));
Thread st_public = new Thread(new RunPublicStream(cl.getOptionValue("config"), queue_public));
ta_public.start();
st_public.start();
The program runs for awhile without any problem, but then stops suddenly. At this time the queue is full and it seems, that the consumer is not able to take a new status from it. I tried several variations of the producer/consumer pattern without success. No exception is thrown.
I don't know were to look for the failure. I hope someone could give me an hint or an solution.
If working with blocking queues double check for blocking commands (put and take for ArrayBlockingQueue) in the code and typos if working with multiple lists.
Related
I'm building an interactive LED table with a 14x14 matrix consisting of addressable LED strips for an university assignment. Those are being controlled by 2 arduinos that get the data about which LED should have which RGB value from a Pi running a server that runs several games which should be playable on the LED table. To control the games I send respective int codes from an android app to the server running on the Raspi.
The serial communication is realized by using jSerialComm. The problem I'm facing is, that I don't want to permanently send data over the serial port but only at the moment, when a new array that specifies the matrix is available.
Therefore I don't want to be busy waiting and permanently checking if the matrix got updated not do I want to check for a update with
while(!matrixUpdated) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {}
}
So what I've been trying was running a while(true) in which I call wait(), so the thread stops until I wake the thread up by calling notify when an updated matrix is available.
My run() method in the serial thread looks like this at the moment:
#Override
public void run() {
arduino1.setComPortTimeouts(SerialPort.TIMEOUT_SCANNER, 0, 0);
arduino2.setComPortTimeouts(SerialPort.TIMEOUT_SCANNER, 0, 0);
try {
Thread.sleep(100);
} catch (Exception e) {}
PrintWriter outToArduino1 = new PrintWriter(arduino1.getOutputStream());
PrintWriter outToArduino2 = new PrintWriter(arduino2.getOutputStream());
while(true) {
try {
wait();
} catch (InterruptedException e) {}
System.out.println("Matrix received");
outToArduino1.print(matrix);
outToArduino2.print(matrix);
}
}
I wake the thread up by this method which is nested in the same class:
public void setMatrix(int[][][] pixelIdentifier) {
matrix = pixelIdentifier;
notify();
}
I also tried notifyAll() which didn't change the outcome.
In one of the games (Tic Tac Toe) I call this method after every game turn to update and send the matrix to the arduinos:
private void promptToMatrix() {
synchronized (GameCenter.serialConnection) {
GameCenter.serialConnection.setMatrix(matrix);
}
}
I previously called it without using the synchronized block but as I've been reading through many articles on that topic on StackOverflow I have read that one should use synchronized for this. Further I have also read that using wait() and notify() is not recommended, however as the assignment needs to get done quite quickly I don't know if any other approach makes sense as I don't want to restructure my whole application as I run up to 5 threads when a game is being played (due to threads for communication and so on).
If there is a possibility to solve this using wait() and notify() I would be really grateful to hear how that would be done, as I have not been able to really comprehend how working properly with the synchronized block is being done and so on.
However if such a solution is not possible or would also end in restructuring the whole application I'm also open to different suggestions. Pointing out that using wait() and notify() is not recommended however doesn't help me, as I've already read that often enough, I'm aware of that but prefer to use it in that case if possible!!!
EDIT:
The application executes like this:
Main Thread
|--> SerialCommunication Thread --> waiting for updated data
|--> NetworkController Thread
|--> Client Thread --> interacting with the game thread
|--> Game Thread --> sending updated data to the waiting SerialCommunication Thread
Really appreciate any help and thanks in advance for your time!
You are dealing with asynchronous update possibly running on different threads, the best match in my opinion is using RxJava.
You could use a Subject to receive matrix event and then subscribe to it to update the leds.
You can write something like this (don't take it too literally).
public static void main(String[] args) {
int[][] initialValue = new int[32][32];
BehaviorSubject<int[][]> matrixSubject = BehaviorSubject.createDefault(initialValue);
SerialPort arduino1 = initSerial("COM1");
SerialPort arduino2 = initSerial("COM2");;
PrintWriter outToArduino1 = new PrintWriter(arduino1.getOutputStream());
PrintWriter outToArduino2 = new PrintWriter(arduino2.getOutputStream());
Observable<String> serializedMatrix = matrixSubject.map(Sample::toChars);
serializedMatrix.observeOn(Schedulers.io()).subscribe(mat -> {
// Will run on a newly created thread
outToArduino1.println(mat);
});
serializedMatrix.observeOn(Schedulers.io()).subscribe(mat -> {
// Will run on a newly created thread
outToArduino2.println(mat);
});
// Wait forever
while(true) {
try {
// get your matrix somehow ...
// then publish it on your subject
// your subscribers will receive the data and use it.
matrixSubject.onNext(matrix);
Thread.sleep(100);
} catch (InterruptedException e) {
// SWALLOW error
}
}
}
public static String toChars(int[][] data) {
// Serialize data
return null;
}
There are may operators that you could use to make it do what you need, also you can use different schedulers to choose from different thread policies.
You can also transform your input in the subject you publish, an observable or a subject can be created directly from your input.
I am consuming from a certain source (say Kafka) and periodically dumping the collected messages (to, say, S3). My class definition is as follows:
public class ConsumeAndDump {
private List<String> messages;
public ConsumeAndDump(){
messages = new ArrayList<>();
// initialize required resources
}
public void consume(){
// this runs continuously and keeps consuming from the source.
while(true){
final String message = ...// consume from Kafka
messages.add(message);
}
}
public void dump(){
while(true){
final String allMessages = String.join("\n", messages);
messages.clear(); // shown here simply, but i am synchronising this to avoid race conditions
// dump to destination (file, or S3, or whatever)
TimeUnit.SECONDS.sleep(60); // sleep for a minute
}
}
public void run() {
// This is where I don't know how to proceed.
// How do I start consume() and dump() as separate threads?
// Is it even possible in Java?
// start consume() as thread
// start dump() as thread
// wait for those to finish
}
}
I want to have two threads - consume and dump. consume should run continuously whereas dump wakes up periodically, dumps the messages, clears the buffer and then goes back to sleep again.
I am having trouble starting consume() and dump() as threads. Honestly, I don't know how to do that. Can we even run member methods as threads? Or do I have to make separate Runnable classes for consume and dump? If so, how would I share messages between those?
First of all, you can't really use ArrayList for this. ArrayList is not thread-safe. Check out BlockingQueue for example. You will have to deal with things like back pressure. Don't use an unbounded queue.
Starting a thread is pretty simple, you can use lambdas for it.
public void run() {
new Thread(this::consume).start();
new Thread(this::produce).start();
}
Should work, but gives you little to no control over when those processes should end.
So my goal is to measure the performance of a Streaming Engine. It's basically a library to which i can send data-packages. The idea to measure this is to generate data, put it into a Queue and let the Streaming Engine grab the data and process it.
I thought of implementing it like this: The Data Generator runs in a thread and generates data packages in an endless loop with a certain Thread.sleep(X) at the end. When doing the tests the idea is to minimize tis Thread.sleep(X) to see if this has an impact on the Streaming Engine's performance. The Data Generator writes the created packages into a queue, that is, a ConcurrentLinkedQueue, which at the same time is a Singleton.
In another thread I instantiate the Streaming Engine which continuously removes the packages from the queue by doing queue.remove(). This is done in an endlees loop without any sleeping, because it should just be done as fast as possible.
In a first try to implement this I ran into a problem. It seems as if the Data Generator is not able to put the packages into the Queue as it should be. It is doing that too slow. My suspicion is that the endless loop of the Streaming Engine thread is eating up all the resources and therefore slows down everything else.
I would be happy about how to approach this issue or other design patterns, which could solve this issue elegantly.
the requirements are: 2 threads which run in parallel basically. one is putting data into a queue. the other one is reading/removing from the queue. and i want to measure the size of the queue regularly in order to know if the engine which is reading/removing from the queue is fast enough to process the generated packages.
You can use a BlockingQueue, for example ArrayBlockingQueue, you can initialize these to a certain size, so the number of items queued will never exceed a certain number, as per this example:
// create queue, max size 100
final ArrayBlockingQueue<String> strings = new ArrayBlockingQueue<>(100);
final String stop = "STOP";
// start producing
Runnable producer = new Runnable() {
#Override
public void run() {
try {
for(int i = 0; i < 1000; i++) {
strings.put(Integer.toHexString(i));
}
strings.put(stop);
} catch(InterruptedException ignore) {
}
}
};
Thread producerThread = new Thread(producer);
producerThread.start();
// start monitoring
Runnable monitor = new Runnable() {
#Override
public void run() {
try {
while (true){
System.out.println("Queue size: " + strings.size());
Thread.sleep(5);
}
} catch(InterruptedException ignore) {
}
}
};
Thread monitorThread = new Thread(monitor);
monitorThread.start();
// start consuming
Runnable consumer = new Runnable() {
#Override
public void run() {
// infinite look, will interrupt thread when complete
try {
while(true) {
String value = strings.take();
if(value.equals(stop)){
return;
}
System.out.println(value);
}
} catch(InterruptedException ignore) {
}
}
};
Thread consumerThread = new Thread(consumer);
consumerThread.start();
// wait for producer to finish
producerThread.join();
consumerThread.join();
// interrupt consumer and monitor
monitorThread.interrupt();
You could also have third thread monitoring the size of the queue, to give you an idea of which thread is outpacing the other.
Also, you can used the timed put method and the timed or untimed offer methods, which will give you more control of what to do if the queue if full or empty. In the above example execution will stop until there is space for the next element or if there are no further elements in the queue.
I have an application that is redirecting my System.out text to a Jtextarea. This works fine but when I call one of the methods in my application is creates multiple threads and uses a latch counter to wait for them to finish. That method then calls latch.await() so that it does not finish running its code until the other threads are finished. The problem is that once the latch.await() code is called my JtextArea stops posting text until all the threads have finished. Any ideas around this? Eclipse console is able to keep posting while the latch.await() is running so it has to be possible.
Example:
From the GUI:
btnStart.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent arg0) {
System.out.println("You pressed Start");
MyGoogleSearch startParsing = new MyGoogleSearch();
try {
startParsing.startParser(othoSelection); ...
MyGoogleSearch:
Enumeration e = hm.elements();
//Read in src/Ontology/Ontology.txt
//Put each line into the searchQuery ArrayQueue
while ((strLine = br.readLine()) != null)
{
searchQuery.put(strLine);
}
System.out.println("Finsihed loading");
//Create 32 threads (More threads allows you to pull data from Bing faster. Any more than 32 and Bing errors out (you start to pull data
//too fast)
for(int i = 0; i < 32; i++)
{
System.out.println("Starting thread: " + i);
new NewThread();
}
//Wait for all of the threads to finish
latch.await();
e = hm.keys();
//Write the URL's from the hashmap to a file
while (e.hasMoreElements())
{
out.write(e.nextElement() + "\n");
}
//close input/output stream
in.close();
out.close();
System.out.println("Done");
and the Thread does some stuff then
MyGoogleSearch.latch.countDown();
This works fine but when I call one of the methods in my application is creates multiple threads and uses a latch counter to wait for them to finish.
You could get around that by calling the method in a separate thread. However, I suspect the method is waiting for all the threads to complete because it wants to aggregate some results and then return the aggregated result (or something similar). If that's the case, then there are several ways to deal with it, but probably the one that will make the most sense for a graphical application is to have the thread invoke a callback with any results which were obtained from the method.
If you post some sample code, then we can provide you with more specific answers and examples of how to do it.
Update:
I'm having a hard time reading your code, but I presume that 'startParser' is the call that's blocking. Additionally, it doesn't appear that the UI needs to wait for the results so I would recommend you do the simplest thing possible:
MyGoogleSearch startParsing = new MyGoogleSearch();
Thread t = new Thread(new Runnable(){
public void run(){
startParsing.startParser(othoSelection);
}
}
// don't wait for this thread to finish
t.start();
I am trying to program a game in which I have a Table class and each person sitting at the table is a separate thread. The game involves the people passing tokens around and then stopping when the party chime sounds.
how do i program the run() method so that once I start the person threads, they do not die and are alive until the end of the game
One solution that I tried was having a while (true) {} loop in the run() method but that increases my CPU utilization to around 60-70 percent. Is there a better method?
While yes, you need a loop (while is only one way, but it is simplest) you also need to put something inside the loop that waits for things to happen and responds to them. You're aiming to have something like this pseudocode:
loop {
event = WaitForEvent();
RespondToEvent(event);
} until done;
OK, that's the view from 40,000 feet (where everything looks like ants!) but it's still the core of what you want. Oh, and you also need something to fire off the first event that starts the game, obviously.
So, the key then becomes the definition of WaitForEvent(). The classic there is to use a queue to hold the events, and to make blocking reads from the queue so that things wait until something else puts an event in the queue. This is really a Concurrency-101 data-structure, but an ArrayBlockingQueue is already defined correctly and so is what I'd use in my first implementation. You'll probably want to hide its use inside a subclass of Thread, perhaps like this:
public abstract class EventHandlingThread<Event> extends Thread {
private ArrayBlockingQueue<Event> queue = new ArrayBlockingQueue<Event>();
private boolean done;
protected abstract void respondToEvent(Event event);
public final void postEvent(Event event) throws InterruptedException {
queue.put(event);
}
protected final void done() {
done = true;
}
public final void run() {
try {
while (!done) {
respondToEvent(queue.take());
}
} catch (InterruptedException e) {
// Maybe log this, maybe not...
} catch (RuntimeException e) {
// Probably should log this!
}
}
}
Subclass that for each of your tasks and you should be able to get going nicely. The postEvent() method is called by other threads to send messages in, and you call done() on yourself when you've decided enough is enough. You should also make sure that you've always got some event that can be sent in which terminates things so that you can quit the game…
I would look into Locks and Conditions. This way you can write code that waits for a certain condition to happen. This won't take a lot of CPU power and is even much more efficient and better performing than sleeping .
To make a thread run for an infinite time:
final Object obj = new Object();
try {
Thread th = new Thread(new Runnable() {
public void run() {
synchronized(obj) {
try {
System.out.println("Waiting");
obj.wait();
System.out.println("Done waiting");
}catch(Exception ex) {
ex.printStackTrace();
}
}
}
});
th.start();
System.out.println("Waiting to join.");
// Dont notify; but wait for joining. This will ensure that main thread is running always.
th.join();
System.out.println("End of the Program");
} catch(Exception ex) {
ex.printStackTrace();
}
You may add Thread.sleep() with appropriate time to minimize useless loop iterations.
Another solution is using synchronization. While threads are not required to do anything, they enter into a sleeping state on a monitor using the wait() method, and then when the turn comes, required thread is woken up by the notify() method.
Actor model seems suitable for this scenario. Each person sitting on the table and the table itself can be modelled as actors and the event of passing the tokens and starting and stopping of the game can be modelled as messages to be passed between the actors.
As a bonus, by modelling the scenario as actors you get rid of explicit manipulation of threads, synchronization and locking.
On JVM I will prefer using Scala for modelling actors. For Java you can use libraries like Kilim. See this post for a comparison of Actor model related libraries in Java.
One Way is to use while loop but keep a check i.e
while(true){
if(condition!=true){
Thread.sleep(time);
}else{
break;
}
}
This way if your condition says game is not over it will keep person thread at sleep and memory consumption will be very low.
You should test for a condition in the while loop:
while (!gameOver)
{
do_intersting_stuff();
}
Heavy CPU load is typical for busy wait. Is your loop actually just checking a flag over and over, like
while (!gameOver)
{
if (actionNeeded)
{
do_something();
}
}
you might change to another notification system to sleep and wake up, as this just burns CPU time for nothing.