I have an application that is redirecting my System.out text to a Jtextarea. This works fine but when I call one of the methods in my application is creates multiple threads and uses a latch counter to wait for them to finish. That method then calls latch.await() so that it does not finish running its code until the other threads are finished. The problem is that once the latch.await() code is called my JtextArea stops posting text until all the threads have finished. Any ideas around this? Eclipse console is able to keep posting while the latch.await() is running so it has to be possible.
Example:
From the GUI:
btnStart.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent arg0) {
System.out.println("You pressed Start");
MyGoogleSearch startParsing = new MyGoogleSearch();
try {
startParsing.startParser(othoSelection); ...
MyGoogleSearch:
Enumeration e = hm.elements();
//Read in src/Ontology/Ontology.txt
//Put each line into the searchQuery ArrayQueue
while ((strLine = br.readLine()) != null)
{
searchQuery.put(strLine);
}
System.out.println("Finsihed loading");
//Create 32 threads (More threads allows you to pull data from Bing faster. Any more than 32 and Bing errors out (you start to pull data
//too fast)
for(int i = 0; i < 32; i++)
{
System.out.println("Starting thread: " + i);
new NewThread();
}
//Wait for all of the threads to finish
latch.await();
e = hm.keys();
//Write the URL's from the hashmap to a file
while (e.hasMoreElements())
{
out.write(e.nextElement() + "\n");
}
//close input/output stream
in.close();
out.close();
System.out.println("Done");
and the Thread does some stuff then
MyGoogleSearch.latch.countDown();
This works fine but when I call one of the methods in my application is creates multiple threads and uses a latch counter to wait for them to finish.
You could get around that by calling the method in a separate thread. However, I suspect the method is waiting for all the threads to complete because it wants to aggregate some results and then return the aggregated result (or something similar). If that's the case, then there are several ways to deal with it, but probably the one that will make the most sense for a graphical application is to have the thread invoke a callback with any results which were obtained from the method.
If you post some sample code, then we can provide you with more specific answers and examples of how to do it.
Update:
I'm having a hard time reading your code, but I presume that 'startParser' is the call that's blocking. Additionally, it doesn't appear that the UI needs to wait for the results so I would recommend you do the simplest thing possible:
MyGoogleSearch startParsing = new MyGoogleSearch();
Thread t = new Thread(new Runnable(){
public void run(){
startParsing.startParser(othoSelection);
}
}
// don't wait for this thread to finish
t.start();
Related
I'm building an interactive LED table with a 14x14 matrix consisting of addressable LED strips for an university assignment. Those are being controlled by 2 arduinos that get the data about which LED should have which RGB value from a Pi running a server that runs several games which should be playable on the LED table. To control the games I send respective int codes from an android app to the server running on the Raspi.
The serial communication is realized by using jSerialComm. The problem I'm facing is, that I don't want to permanently send data over the serial port but only at the moment, when a new array that specifies the matrix is available.
Therefore I don't want to be busy waiting and permanently checking if the matrix got updated not do I want to check for a update with
while(!matrixUpdated) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {}
}
So what I've been trying was running a while(true) in which I call wait(), so the thread stops until I wake the thread up by calling notify when an updated matrix is available.
My run() method in the serial thread looks like this at the moment:
#Override
public void run() {
arduino1.setComPortTimeouts(SerialPort.TIMEOUT_SCANNER, 0, 0);
arduino2.setComPortTimeouts(SerialPort.TIMEOUT_SCANNER, 0, 0);
try {
Thread.sleep(100);
} catch (Exception e) {}
PrintWriter outToArduino1 = new PrintWriter(arduino1.getOutputStream());
PrintWriter outToArduino2 = new PrintWriter(arduino2.getOutputStream());
while(true) {
try {
wait();
} catch (InterruptedException e) {}
System.out.println("Matrix received");
outToArduino1.print(matrix);
outToArduino2.print(matrix);
}
}
I wake the thread up by this method which is nested in the same class:
public void setMatrix(int[][][] pixelIdentifier) {
matrix = pixelIdentifier;
notify();
}
I also tried notifyAll() which didn't change the outcome.
In one of the games (Tic Tac Toe) I call this method after every game turn to update and send the matrix to the arduinos:
private void promptToMatrix() {
synchronized (GameCenter.serialConnection) {
GameCenter.serialConnection.setMatrix(matrix);
}
}
I previously called it without using the synchronized block but as I've been reading through many articles on that topic on StackOverflow I have read that one should use synchronized for this. Further I have also read that using wait() and notify() is not recommended, however as the assignment needs to get done quite quickly I don't know if any other approach makes sense as I don't want to restructure my whole application as I run up to 5 threads when a game is being played (due to threads for communication and so on).
If there is a possibility to solve this using wait() and notify() I would be really grateful to hear how that would be done, as I have not been able to really comprehend how working properly with the synchronized block is being done and so on.
However if such a solution is not possible or would also end in restructuring the whole application I'm also open to different suggestions. Pointing out that using wait() and notify() is not recommended however doesn't help me, as I've already read that often enough, I'm aware of that but prefer to use it in that case if possible!!!
EDIT:
The application executes like this:
Main Thread
|--> SerialCommunication Thread --> waiting for updated data
|--> NetworkController Thread
|--> Client Thread --> interacting with the game thread
|--> Game Thread --> sending updated data to the waiting SerialCommunication Thread
Really appreciate any help and thanks in advance for your time!
You are dealing with asynchronous update possibly running on different threads, the best match in my opinion is using RxJava.
You could use a Subject to receive matrix event and then subscribe to it to update the leds.
You can write something like this (don't take it too literally).
public static void main(String[] args) {
int[][] initialValue = new int[32][32];
BehaviorSubject<int[][]> matrixSubject = BehaviorSubject.createDefault(initialValue);
SerialPort arduino1 = initSerial("COM1");
SerialPort arduino2 = initSerial("COM2");;
PrintWriter outToArduino1 = new PrintWriter(arduino1.getOutputStream());
PrintWriter outToArduino2 = new PrintWriter(arduino2.getOutputStream());
Observable<String> serializedMatrix = matrixSubject.map(Sample::toChars);
serializedMatrix.observeOn(Schedulers.io()).subscribe(mat -> {
// Will run on a newly created thread
outToArduino1.println(mat);
});
serializedMatrix.observeOn(Schedulers.io()).subscribe(mat -> {
// Will run on a newly created thread
outToArduino2.println(mat);
});
// Wait forever
while(true) {
try {
// get your matrix somehow ...
// then publish it on your subject
// your subscribers will receive the data and use it.
matrixSubject.onNext(matrix);
Thread.sleep(100);
} catch (InterruptedException e) {
// SWALLOW error
}
}
}
public static String toChars(int[][] data) {
// Serialize data
return null;
}
There are may operators that you could use to make it do what you need, also you can use different schedulers to choose from different thread policies.
You can also transform your input in the subject you publish, an observable or a subject can be created directly from your input.
I have a piece of java code which does the following -
Opens a file with data in format {A,B,C} and each file has approx. 5000000 lines.
For each line in file, call a service that gives a column D and append it to {A,B,C} as {A,B,C,D}.
Write this entry into a chunkedwriter that eventually groups together 10000 lines to write back chunk to a remote location
Right now the code is taking 32 hours to execute. This process would again get repeated across another file which hypothetically takes another 32 hours but we need these processes to run daily.
Step 2 is further complicated by the fact that sometimes the service does not have D but is designed to fetch D from its super data store so it throws a transient exception asking you to wait. We have retries to handle this so an entry could technically be retried 5 times with a max delay of 60000 millis. So we could be looking at 5000000 * 5 in worst case.
The combination of {A,B,C} are unique and thus result D can't be cached and reused and a fresh request has to be made to get D every time.
I've tried adding threads like this:
temporaryFile = File.createTempFile(key, ".tmp");
Files.copy(stream, temporaryFile.toPath(),
StandardCopyOption.REPLACE_EXISTING);
reader = new BufferedReader(new InputStreamReader(new
FileInputStream(temporaryFile), StandardCharsets.UTF_8));
String entry;
while ((entry = reader.readLine()) != null) {
final String finalEntry = entry;
service.execute(() -> {
try {
processEntry(finalEntry);
} catch (Exception e) {
log.error("something");
});
count++;
}
Here processEntry method abstracts the implementation details explained above and threads are defined as
ExecutorService service = Executors.newFixedThreadPool(10);
The problem I'm having is the first set of threads spin up but the process doesn't wait until all threads finish their work and all 5000000 lines are complete. So the task that used to wait for completion for 32 hours now ends in <1min which messes up our system's state. Are there any alternative ways to do this? How can I make process wait on all threads completing?
Think about using ExecutorCompletionService if you want to take tasks as they complete you need an ExecutorCompletionService. This acts as a BlockingQueue that will allow you to poll for tasks as and when they finish.
Another solution is to wait the executor termination then you shut it down using:
ExecutorService service = Executors.newFixedThreadPool(10);
service .shutdown();
while (!service .isTerminated()) {}
One alternative is to use a latch to wait for all the tasks to complete before you shutdown the executor on the main thread.
Initialize a CountdownLatch with 1.
After you exit the loop that submits the tasks, you call latch.await();
In the task you start you have to have a callback on the starting class to let it know when a task has finished.
Note that in the starting class the callback function has to be synchronized.
In the starting class you use this callback to take the count of completed tasks.
Also inside the callback, when all tasks have completed, you call latch.countdown() for the main thread to continue, lets say, shutting down the executor and exiting.
This shows the main concept, it can be implemented with more detail and more control on the completed tasks if necessary.
It would be something like this:
public class StartingClass {
CountDownLatch latch = new CountDownLatch(1);
ExecutorService service = Executors.newFixedThreadPool(10);
BufferedReader reader;
Path stream;
int count = 0;
int completed = 0;
public void runTheProcess() {
File temporaryFile = File.createTempFile(key, ".tmp");
Files.copy(stream, temporaryFile.toPath(),
StandardCopyOption.REPLACE_EXISTING);
reader = new BufferedReader(new InputStreamReader(new
FileInputStream(temporaryFile), StandardCharsets.UTF_8));
String entry;
while ((entry = reader.readLine()) != null) {
final String finalEntry = entry;
service.execute(new Task(this,finalEntry));
count++;
}
latch.await();
service.shutdown();
}
public synchronized void processEntry(String entry) {
}
public synchronized void taskCompleted() {
completed++;
if(completed == count) {
latch.countDown();;
}
}
//This can be put in a different file.
public static class Task implements Runnable {
StartingClass startingClass;
String finalEntry;
public Task(StartingClass startingClass, String finalEntry) {
this.startingClass = startingClass;
this.finalEntry = finalEntry;
}
#Override
public void run() {
try {
startingClass.processEntry(finalEntry);
startingClass.taskCompleted();
} catch (Exception e) {
//log.error("something");
};
}
}
}
Note that you need to close the file. Also the sutting down of the executor could be written to wait a few seconds before forcing a shutdown.
The problem I'm having is the first set of threads spin up but the process doesn't wait until all threads finish their work and all 5000000 lines are complete.
When you are running jobs using an ExecutorService they are added into the service and are run in the background. To wait for them to complete you need to wait for the service to terminate:
ExecutorService service = Executors.newFixedThreadPool(10);
// submit jobs to the service here
// after the last job has been submitted, we immediately shutdown the service
service.shutdown();
// then we can wait for it to terminate as the jobs run in the background
service.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
Also, if there is a crap-ton of lines in these files, I would recommend that you use a bounded queue for the jobs so that you don't blow out memory effectively caching all of the lines in the file. This only works if the files stay around and don't go away.
// this is the same as a newFixedThreadPool(10) but with a queue of 100
ExecutorService service = new ThreadPoolExecutor(10, 10,
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>(100));
// set a rejected execution handler so we block the caller once the queue is full
threadPool.setRejectedExecutionHandler(new RejectedExecutionHandler() {
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
try {
executor.getQueue().put(r);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
return;
}
}
});
Write this entry into a chunkedwriter that eventually groups together 10000 lines to write back chunk to a remote location
As each A,B,C job finishes, if it needs to be processed in a second step then I would also recommend looking into a ExecutorCompletionService which allows you to chain various different thread pools together so as lines finish they will immediately start working on the 2nd phase of the processing.
If instead this chunkedWriter is just a single thread then I'd recommend sharing a BlockingQueue<Result> and having the executor threads put to the queue once the lines are done and the chunkedWriter taking from the queue and doing the chunking and writing of the results. In this situation, indicating to the writer thread that it is done needs to be handled carefully -- maybe with some sort of END_RESULT constant put to the queue by the main thread waiting for the service to terminate.
I have this Transmitter class, which contains one BufferedReader and one PrintWriter. The idea is, on the main class, to use Transmitter.receive() and Transmitter.transmit() to the main socket. The problem is:
public void receive() throws Exception {
// Reads from the socket
Thread listener = new Thread(new Runnable() {
public void run() {
String res;
try {
while((res = input.readLine()) != null) {
System.out.println("message received: " + res);
outputMessage = (res);
if (res.equals("\n")) {
break;
}
}
} catch (IOException e) {
e.printStackTrace();
}
};
});
listener.start();
listener.join();
}
The thread changes the 'outputMessage' value, which I can get using an auxiliary method. The problem is that, without join, my client gets the outputMessage but I want to use it several times on my main class, like this:
trans1.receive();
while(trans1.getOutput() == null);
System.out.println("message: " + trans1.getOutput());
But with join this system.out never executes because trans1.receive() is stuck... any thoughts?
Edit 1: here is the transmitter class https://titanpad.com/puYBvlVery
You might send \n; that doesn't mean that you will see it in your Java code.
As it says in the Javadoc for BufferedReader.readLine() (emphasis mine):
(Returns) A String containing the contents of the line, not including any line-termination characters
so "\n" will never be returned.
Doing this:
{
Thread listener = new Thread(new Runnable() {
public void run() {
doSomeWork();
};
});
listener.start();
listener.join();
}
will create a new thread and then wait for it to do its work and finish. Therefore it's more or less the same as just directly doing:
doSomeWork();
The new thread doesn't serve any real purpose here.
Also, the extra thread introduces synchronization problems because in your code you don't make sure your variables are synchronized.
Thirdly, your thread keeps reading lines from the input in a loop until there's nothing more to read and unless the other side closes the stream, it will block on the readLine() call. What you will see in with getOutput() will be a random line that just happens to be there at the moment you look, the next time you look it might be the same line, or some completely different line; some lines will be read and forgotten immediatelly without you ever noticing it from the main thread.
You can just call input.readLine() directly in your main thread when you actually need to get a new line message from the input, you don't need an extra reader thread. You could store the read messages into a Queue as yshavit suggests, if that's desirable, e.g. for performance reasons it might be better to read the messages as soon as they are available and have them ready in memory. But if you only need to read messages one by one then you can simply call input.readLine() only when you actually need it.
I'm writing an small program to put tweets from the Twitter public stream into an HBase database. The program uses two threads, one to collect the tweets and one to process them.
The first thread uses twitter4j StatusListener to get the tweets and puts them in an ArrayBlockingQueue with an capacity of 100.
The second thread takes an status from the queue, filters the needed data and moves them to the database.
The processing takes more time, than the collecting of the status.
The producer looks like this:
public void onStatus(Status status) {
try {
this.queue.put(status);
} catch(Exception ex) {
ex.printStackTrace();
}
}
The consumer uses take and calls an function to process the new status:
public void run() {
try {
while(true) {
// Get new status to process
this.status = this.queue.take();
this.analyse();
}
} catch(Exception ex) {
ex.printStackTrace();
}
}
In the main function the two threads were created and started:
ArrayBlockingQueue<Status> queue_public = new ArrayBlockingQueue<Status>(100);
Thread ta_public = new Thread(new TweetAnalyser(cl.getOptionValue("config"), queue_public));
Thread st_public = new Thread(new RunPublicStream(cl.getOptionValue("config"), queue_public));
ta_public.start();
st_public.start();
The program runs for awhile without any problem, but then stops suddenly. At this time the queue is full and it seems, that the consumer is not able to take a new status from it. I tried several variations of the producer/consumer pattern without success. No exception is thrown.
I don't know were to look for the failure. I hope someone could give me an hint or an solution.
If working with blocking queues double check for blocking commands (put and take for ArrayBlockingQueue) in the code and typos if working with multiple lists.
So my goal is to measure the performance of a Streaming Engine. It's basically a library to which i can send data-packages. The idea to measure this is to generate data, put it into a Queue and let the Streaming Engine grab the data and process it.
I thought of implementing it like this: The Data Generator runs in a thread and generates data packages in an endless loop with a certain Thread.sleep(X) at the end. When doing the tests the idea is to minimize tis Thread.sleep(X) to see if this has an impact on the Streaming Engine's performance. The Data Generator writes the created packages into a queue, that is, a ConcurrentLinkedQueue, which at the same time is a Singleton.
In another thread I instantiate the Streaming Engine which continuously removes the packages from the queue by doing queue.remove(). This is done in an endlees loop without any sleeping, because it should just be done as fast as possible.
In a first try to implement this I ran into a problem. It seems as if the Data Generator is not able to put the packages into the Queue as it should be. It is doing that too slow. My suspicion is that the endless loop of the Streaming Engine thread is eating up all the resources and therefore slows down everything else.
I would be happy about how to approach this issue or other design patterns, which could solve this issue elegantly.
the requirements are: 2 threads which run in parallel basically. one is putting data into a queue. the other one is reading/removing from the queue. and i want to measure the size of the queue regularly in order to know if the engine which is reading/removing from the queue is fast enough to process the generated packages.
You can use a BlockingQueue, for example ArrayBlockingQueue, you can initialize these to a certain size, so the number of items queued will never exceed a certain number, as per this example:
// create queue, max size 100
final ArrayBlockingQueue<String> strings = new ArrayBlockingQueue<>(100);
final String stop = "STOP";
// start producing
Runnable producer = new Runnable() {
#Override
public void run() {
try {
for(int i = 0; i < 1000; i++) {
strings.put(Integer.toHexString(i));
}
strings.put(stop);
} catch(InterruptedException ignore) {
}
}
};
Thread producerThread = new Thread(producer);
producerThread.start();
// start monitoring
Runnable monitor = new Runnable() {
#Override
public void run() {
try {
while (true){
System.out.println("Queue size: " + strings.size());
Thread.sleep(5);
}
} catch(InterruptedException ignore) {
}
}
};
Thread monitorThread = new Thread(monitor);
monitorThread.start();
// start consuming
Runnable consumer = new Runnable() {
#Override
public void run() {
// infinite look, will interrupt thread when complete
try {
while(true) {
String value = strings.take();
if(value.equals(stop)){
return;
}
System.out.println(value);
}
} catch(InterruptedException ignore) {
}
}
};
Thread consumerThread = new Thread(consumer);
consumerThread.start();
// wait for producer to finish
producerThread.join();
consumerThread.join();
// interrupt consumer and monitor
monitorThread.interrupt();
You could also have third thread monitoring the size of the queue, to give you an idea of which thread is outpacing the other.
Also, you can used the timed put method and the timed or untimed offer methods, which will give you more control of what to do if the queue if full or empty. In the above example execution will stop until there is space for the next element or if there are no further elements in the queue.