I am using this API to parse wikipedia dump available at
http://code.google.com/p/wikixmlj/
I am using my API as like this
Class wiki
{
public void parseWiki()
{
PageCallbackHandler handler = new WikiPageCallbackHandler()
WikiXMLSAXParser.parseWikipediaDump(filepath, handler )
}
}
Class WikiPageCallbackHandler implements PageCallbackHandler
{
//Override
//This is the callback
void process(WikiPage page)
{
String Text = page.GetText();
//Write this text into a file
.....
}
}
The issue is before all callbacks are done and I finish writing it into a file, the application is terminating and I am not getting all callbacks.
I want to know why this is happening and is there any way to hold my main thread, so that I get all my callbacks. Also, How would I know if I have got all my callbacks?
Thanks
Have you try using 'Semaphore' at Oracle Docs.
Here's an example use of Semaphore:
class wiki
{
public void parseWiki(){
// initialize Semaphore object
Semaphore semaphore = new Semaphore(1);
//pass it to callback handler so it can release semaphore latter
PageCallbackHandler handler = new WikiPageCallbackHandler(semaphore);
//This is the async operation, right?
WikiXMLSAXParser.parseWikipediaDump(filepath, handler );
//wait until a permit is available (when semaphore.release() is called)
semaphore.acquire(); //this throw InterruptedException, please handle it else where
}
}
class WikiPageCallbackHandler implements PageCallbackHandler
{
private Semaphore semaphore;
public WikiPageCallbackHandler(Semaphore semaphore){
this.semaphore = semaphore;
}
//Override
//This is the callback
void process(WikiPage page)
{
try{
String Text = page.GetText();
//Write this text into a file
.....
}finally{
semaphore.release();
}
}
}
Related
On this project, a Manager performs event queuing, and to return the result of the event a callback is used (the callback does not extend Runnable). The manager runs on a separate thread, dispatching the events. Once the events terminate, this same thread calls the callbacks. This means that the next event will not be dispatched before the callback of the previous event terminates. In order to avoid this, I though about having the manager create a new thread for each callback, and executing the callbacks there. How good is this solution in terms of design practices, and is there a better way to achieve this?
A simple Callback code:
import java.util.concurrent.*;
import java.util.*;
public class CallBackDemo{
public CallBackDemo(){
System.out.println("creating service");
ExecutorService service = Executors.newFixedThreadPool(10);
try{
for ( int i=0; i<10; i++){
Callback callback = new Callback(i+1);
MyCallable myCallable = new MyCallable((long)i+1,callback);
Future<Long> future = service.submit(myCallable);
//System.out.println("future status:"+future.get()+":"+future.isDone());
}
}catch(Exception err){
err.printStackTrace();
}
service.shutdown();
}
public static void main(String args[]){
CallBackDemo demo = new CallBackDemo();
}
}
class MyCallable implements Callable<Long>{
Long id = 0L;
Callback callback;
public MyCallable(Long val,Callback obj){
this.id = val;
this.callback = obj;
}
public Long call(){
//Add your business logic
System.out.println("Callable:"+id+":"+Thread.currentThread().getName());
callback.callbackMethod();
return id;
}
}
class Callback {
private int i;
public Callback(int i){
this.i = i;
}
public void callbackMethod(){
System.out.println("Call back:"+i);
// Add your business logic
}
}
output:
creating service
Callable:1:pool-1-thread-1
Call back:1
Callable:2:pool-1-thread-2
Call back:2
Callable:8:pool-1-thread-8
Call back:8
Callable:3:pool-1-thread-3
Call back:3
Callable:10:pool-1-thread-10
Callable:4:pool-1-thread-4
Call back:10
Callable:7:pool-1-thread-7
Call back:7
Callable:6:pool-1-thread-6
Call back:6
Callable:9:pool-1-thread-9
Callable:5:pool-1-thread-5
Call back:9
Call back:4
Call back:5
Summary:
Replace Manager with ExecutorService of your preferred choice.
Either your can pass Callaback object to Callable/Runnable object Or you can create Callback object inside Callable/Runnable. In my example, I have explicitly passed Callback object to Callable.
Before returning the result, Callable object invokes Callback method. If you want to block on proceeding further unless you get response from current event, just uncomment below line.
System.out.println("future status:"+future.get()+":"+future.isDone());
I think you are going to avoid it and hence keep above line commented. You don't have to create new thread for Callback method invocation. If you want to process Callback event asynchronously, you can create one more ExecutorService and submit the event.
I would have the thread which executes the task, also execute the call back. Instead of creating a Thread each time, I suggest you use an ExecutorService.
public static <T> void submit(ExecutorService service,
Callable<T> callable,
Consumer<T> callback) {
service.submit(() -> {
try {
callback.accept(callable.call());
} catch (Throwable t) {
// log the Throwable
}
});
}
I am having hard time converting old fashioned wait notify with spurious waits to java.util.concurrent API
First Problem: What to use, Future or CountdownLatch or CyclicBarrier according to this question
Second Question: How to use it? Because in all the examples I have looked at are converting a single async method to sync which is not a problem
Thirdly: What is the best option in my case out of Future task's get method, CountDownLatch or CyclicBarrier, since I dont have multiple threads, but only 2.
My async code
Main class:
public static void main(String[] args) throws InterruptedException {
Request req = new Request(1);
Thread tReq = new Thread(req);
tReq.start();
synchronized(req){
req.wait();
}
LogProperties.log.info("Response is: " + req.responseString);
}
Request Class:
public class Request implements Runnable {
private int requestID;
public boolean isComplete;
public String responseString;
public Request(int id) {
this.requestID = id;
}
#Override
public void run() {
FutureTest.hmTest.put(requestID, this);
try {
//simulate a request
Thread.sleep(10000);
} catch (InterruptedException ex) {
}
Response response = new Response(requestID);
Thread tResponse = new Thread(response);
tResponse.start();
}
}
Response Class:
public class Response implements Runnable {
int id;
public Response(int responseId) {
this.id = responseId;
}
#Override
public void run() {
Request req = (Request) FutureTest.hmTest.get(id);
req.isComplete = true;
req.responseString = "Request for id [" + id + "] has been completed";
synchronized(req){
req.notify();
}
}
}
My Problem with using future callable and CyclicBarrier is that Im not returning a variable, I want to wait on a object, which is of type Request in this case, so what is the best solution
One of the most versatile means for threads to communicate is a BlockingQueue.
In your case, you have one thread that creates a "response" (i.e., a producer), and you have another thread that is waiting for the "response" (a consumer). One way to implement that is for the producer to put() the response into a BlockingQueue, and have the consumer take() the response out of the queue.
The take() operation will implicitly wait for the response to become available before it returns.
I think Pipes can be ideal for this, it can achieve synchronous communication easily.
Check out this link with pipes for producer consumer problem- http://www.informit.com/articles/article.aspx?p=26326&seqNum=10
I'm currently unit testing my asynchronous methods using thread locking, usually I inject a CountDownLatch into my asynchronous component and let the main thread wait for it to reach 0. However, this approach just looks plain ugly, and it doesn't scale well, consider what happens when I write 100+ tests for a component and they all sequentially have to wait for a worker thread to do some fake asynchronous job.
So is there another approach? Consider the following example for a simple search mechanism:
Searcher.java
public class Searcher {
private SearcherListener listener;
public void search(String input) {
// Dispatch request to queue and notify listener when finished
}
}
SearcherListener.java
public interface SearcherListener {
public void searchFinished(String[] results);
}
How would you unit test the search method without using multiple threads and blocking one to wait for another? I've drawn inspiration from How to use Junit to test asynchronous processes but the top answer provides no concrete solution to how this would work.
Another approach:
Just dont start the thread. thats all.
Asume you have a SearcherService which uses your Searcher class.
Then don't start the async SearcherService, instead just call searcher.search(), which blocks until search is finished.
Searcher s = new Searcher();
s.search(); // blocks and returns when finished
// now somehow check the result
Writing unit test for async never looks nice.
It's necessary that the testMyAsyncMethod() (main thread) blocks until you are ready to check the correct behaviour. This is necessary because the test case terminates at the end of the method. So there is no way around, the question is only how you block.
A straightforward approach that does not influence much the productive code is to
use a while loop: asume AsyncManager is the class under test:
ArrayList resultTarget = new ArrayList();
AsyncManager fixture = new AsyncManager(resultTarget);
fixture.startWork();
// now wait for result, and avoid endless waiting
int numIter = 10;
// correct testcase expects two events in resultTarget
int expected = 2;
while (numIter > 0 && resulTarget.size() < expected) {
Thread.sleep(100);
numIter--;
}
assertEquals(expected, resulTarget.size());
productive code would use apropriate target in the constructor of AsyncManager or uses another constructor. For test purpose we can pass our test target.
You will write this only for inherent async tasks like your own message queue.
for other code, only unitest the core part of the class that performs the calculation task, (a special algorithm, etc) you dont need to let it run in a thread.
However for your search listener the shown principle with loop and wait is appropriate.
public class SearchTest extends UnitTest implements SearchListener {
public void searchFinished() {
this.isSearchFinished = true;
}
public void testSearch1() {
// Todo setup your search listener, and register this class to receive
Searcher searcher = new Searcher();
searcher.setListener(this);
// Todo setup thread
searcherThread.search();
asserTrue(checkSearchResult("myExpectedResult1"));
}
private boolean checkSearchResult(String expected) {
boolean isOk = false;
int numIter = 10;
while (numIter > 0 && !this.isSearchFinished) {
Thread.sleep(100);
numIter--;
}
// todo somehow check that search was correct
isOk = .....
return isOk;
}
}
Create a synchronous version of the class that listens for its own results and uses an internal latch that search() waits on and searchFinished() clears. Like this:
public static class SynchronousSearcher implements SearcherListener {
private CountDownLatch latch = new CountDownLatch(1);
private String[] results;
private class WaitingSearcher extends Searcher {
#Override
public void search(String input) {
super.search(input);
try {
latch.await();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
}
public String[] search(String input) {
WaitingSearcher searcher = new WaitingSearcher();
searcher.listener = this;
searcher.search(input);
return results;
}
#Override
public void searchFinished(String[] results) {
this.results = results;
latch.countDown();
}
}
Then to use it, simply:
String[] results = new SynchronousSearcher().search("foo");
There are no threads, no wait loops and the method returns in the minimal possible time. It also doesn't matter if the search returns instantly - before the call to await() - because await() will immediately return if the latch is already at zero.
I have a callback function in which i receive a string.
This string is to be passed to a separate thread for processing since the processing takes time.
Also, since multiple callbacks can come simultaneously, I would like to have a synchronized lock till i pass the string into the new thread. But I do not wish to have the new thread (where processing is going on) to be locked also.
Could someone please help me figure out the design for this?
I have written the following code but in this I think in this no callbacks can be received till the whole processing of the separate thread is also done, thereby defeating the whole purpose of this new thread.
String sLine;
onClick(String line){
synchronized (lock) {
sLine = line;
new Thread(new Runnable() {
#Override
public void run() {
doProcessing(Sline);
}).start();
}
}
Also, since multiple callbacks can come simultaneously, I would like to have a synchronized lock till i pass the string into the new thread.
A: I don't think you need to put a lock here. This string is not accessed by multi-thread.
But I do not wish to have the new thread (where processing is going on) to be locked also.
A: As I see nothing was locked here :) I think it could be better if you do something like that:
Create an class Runner implement Runnable, this class will do processing
Everytime you got callback, use ThreadPoolExecutor to execute this Runner. This help you reuse Thread instance.
Note that: These line code doesn't need to synchronized, put synchronized inside processing method if you need.
// Implement class Runner
public class Runner implements Runnable {
private String mLine;
public Runner(String line) {
mLine = line;
}
#Override
public void run() {
process();
}
public void process() {
// Do processing with mLine
// Put synchronized if you need, it bases on your context
}
}
// Initialize thread pool
private ThreadPoolExecutor executor = new ThreadPoolExecutor(10, 100, 1000, TimeUnit.SECONDS, new LinkedBlockingDeque<Runnable>());
// Execute runner when receiving callback
onClick(String s) {
Runner runner = new Runner(s);
executor.execute(runner);
}
Try changing like below
String sLine;
onClick(final String line){
sLine = line;
new Thread(new Runnable() {
#Override
public void run() {
doProcessing(line);
}).start();
}
I'd like to have my thread (the main/EDT) wait until changes to a file occur and then wait. DefaultFileMonitor extends Runnable and hence runs in a thread of its own. Here is a SSCE:
import java.io.File;
import org.apache.commons.vfs.*;
import org.apache.commons.vfs.impl.DefaultFileMonitor;
public class FileChangeListener implements FileListener {
DefaultFileMonitor fm;
public final static File logFile = new File("t.txt");
public void startListening() throws FileSystemException {
final FileSystemManager fsManager = VFS.getManager();
final FileObject listendir = fsManager.toFileObject(logFile);
fm = new DefaultFileMonitor(this);
fm.addFile(listendir);
fm.start();
}
#Override
public void fileCreated(FileChangeEvent fce) throws Exception {
fileChanged(fce);
}
#Override
public void fileDeleted(FileChangeEvent fce) throws Exception {
//hmm..why deleted?
}
#Override
public void fileChanged(FileChangeEvent fce) throws Exception {
System.out.println("fileChanged executed");
}
}
The main:
import java.io.PrintWriter;
public class App {
public static void main(String[] args) {
FileChangeListener fcl = new FileChangeListener();
try {
fcl.startListening();
final PrintWriter printWriter = new PrintWriter(FileChangeListener.logFile);
printWriter.println("Hello Threads!");
printWriter.close();
//EXECUTE THE FOLLOWING ONLY AFTER fileChanged
System.out.println("Mission complete.");
} catch (Exception ex) {
}
}
}
Append the following to App.main(..) after printWriter.close():
synchronized (fcl) {
fcl.wait();
}
//EXECUTE THE FOLLOWING ONLY AFTER fileChanged
System.out.println("Mission complete.");
and append the following to FileChangeListener.fileChanged(..) after System.out.println("fileChanged executed"):
synchronized (this) {
this.notifyAll();
}
You could communicate between teh two using "Conditions" : http://download.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/Condition.html
Basically, create a new "shared" Condition (say fileChanged). Now, whenever the file changes (in fileChanged() trigger this condition (fileChanged.signal()). In your main code, wait for this condition to occur (fileChanged.await()).
Hope you get the idea.
For making the condition accessible to multiple code unit, here is what I can think (decreasing order of preference) :
Assuming you are going to need as many conditions as many files you listen to, create a factory method getCondition(String file path/name/attribute) which will return the Condition object based on the file (its path or name or other attributes). Use this factory method to get the condition in all cases. The factory should internally create new Condition() instances for each new file to be listened to AND must throw away older instances as the processing of the files is complete (so probably you should add a destroy/deleteCondition(String file) method as well.)
Store the condition as a public field in the listener class (kind of hack if you have the listener instance available).
Store the condition as a public static field in the listener class (kind of hack if you have only one listener instance throughout).
Why? FileChangeListener is a callback: it is executed when the event occurs. In this specific case you've just closed the file so you alreayd know that the mission is complete on that file, so just proceed to the next step. I don't see why you need a FileChangeListener at all here.