I'm trying to run two threads inside JUnit. The following code will be invoked by several JUnit test.
I want to stop both threads when result is not null. What should I do with it? The problem is multiple JUnit test share the same String result object and somehow this code gets blocked by previous test. The problem is when another test invoke this method, the result would assign to null, and previous test would block in while(true) loop.
static String result = null;
public static synchronized String remoteClogTailDir(final int maxRetry, String hostName,
final String identifier, final String remoteClogDirPaths, final String whichKeyValue) {
result = null;
final String[] hosts = hostName.split(",");
if(hosts != null && hosts.length == 2){
Thread t1 = null;
Thread t2 = null;
t1 = new Thread(new Runnable(){
#Override
public void run(){
String resultOfThread = null;
resultOfThread = remoteClogTailDir(maxRetry, hosts[0].trim(), identifier, null,
remoteClogDirPaths, false, whichKeyValue);
if(result == null && resultOfThread != null){
result = resultOfThread;
}
}
});
t2 = new Thread(new Runnable(){
#Override
public void run(){
String resultOfThread = null;
resultOfThread = remoteClogTailDir(maxRetry, hosts[1].trim(), identifier, null,
remoteClogDirPaths, false, whichKeyValue);
if(result == null && resultOfThread != null){
result = resultOfThread;
}
}
});
t1.start();
t2.start();
while(true){
if(result != null){
t1.interrupt();
t2.interrupt();
return result;
}
}
}else{
return remoteClogTailDir(maxRetry, hostName, identifier, null,
remoteClogDirPaths, false, whichKeyValue);
}
}
If I understand correctly, you want to execute several search in parallel, and take the first search which complete. You shouldn't use static properties for that.
You could use a ExecutorCompletionService for such tasks:
Executor executor = Executors.newCachedThreadPool();
CompletionService<String> ecs = new ExecutorCompletionService<String>(executor);
List<Future<String>> futures = new ArrayList<Future<String>>();
try {
futures.add(ecs.submit(search1));
futures.add(ecs.submit(search2));
for (int i = 0; i < futures.size(); ++i) {
String result = ecs.take().get();
if (result != null) {
return result;
}
}
} finally {
for (Future<String> f : futures) {
f.cancel(true);
}
}
executor.shutdownNow();
with search1 or search2 a simple Callable :
Callable<String> search1 = new Callable<String() {
public String call() {
return remoteClogTailDir(...)
}
}
Related
In my opinion, run method in Thread is invoked by jvm, is there a concurrency problem? when I read FutureTask's source code, I found it use CAS to set current thread. Why can't use:
runner = Thread.currentThread()
public void run() {
if (state != NEW ||
!UNSAFE.compareAndSwapObject(this, runnerOffset,
null, Thread.currentThread()))
return;
try {
Callable<V> c = callable;
if (c != null && state == NEW) {
V result;
boolean ran;
try {
result = c.call();
ran = true;
} catch (Throwable ex) {
result = null;
ran = false;
setException(ex);
}
if (ran)
set(result);
}
}
Also, why don't use if (state != NEW && !UNSAFE.compareAndSwapObject(this, runnerOffset,null, Thread.currentThread())) so that run method can only be execute once,then set(result) can replace to
protected void set(V v) {
outcome = v;
UNSAFE.putOrderedInt(this, stateOffset, NORMAL); // final state
finishCompletion();
}
}
not
protected void set(V v) {
if (UNSAFE.compareAndSwapInt(this, stateOffset, NEW, COMPLETING)) {
outcome = v;
UNSAFE.putOrderedInt(this, stateOffset, NORMAL); // final state
finishCompletion();
}
}
Is there necessary to use CAS？
when I use Future<Integer> futureTask1 = executor.submit(callable),submit method will RunnableFuture<T> ftask = newTaskFor(task). If I use
FutureTask futureTask = new FutureTask(new Callable() {
#Override
public Object call() throws Exception {
return null;
}
})
new Thread(futureTask);
new Thread(futureTask);
this is useless. So in diffrent threads there are diffrent RunnableFuture Object，therefore, there is no need for guarantee concurrent calls to run(), Could someone tell me what I miss,thanks
Developers who are new to the workplace are trying hard to learn multithreading knowledge, thank you for your answers
in java.util.concurrent.FutureTask#run
public void run() {
if (state != NEW ||
!UNSAFE.compareAndSwapObject(this, runnerOffset,
null, Thread.currentThread()))
return;
try {
Callable<V> c = callable;
if (c != null && state == NEW) {
V result;
boolean ran;
try {
result = c.call();
ran = true;
} catch (Throwable ex) {
result = null;
ran = false;
setException(ex);
}
if (ran)
set(result);
}
} finally {
// runner must be non-null until state is settled to
// prevent concurrent calls to run()
runner = null;
// state must be re-read after nulling runner to prevent
// leaked interrupts
int s = state;
if (s >= INTERRUPTING)
handlePossibleCancellationInterrupt(s);
} }
what's mean of finally block comment ?
run() method cannot be concurrent called, because of the CAS .
why must wait the INTERRUPTING -> INTERRUPTED state transform finished ?
thank you !
I am trying to write a Multithreaded Web Crawler in Java using Jsoup.I have a Java Class "Master" which creates 6 threads(5 for crawling and 1 for maintenance of queues) ,and 3 queues namely "to_do","to_do_next"(to be done in next iteration) and "done"(final links).
I am using sunchronized locks on shared queues.The idea is as soon as all the 5 threads find the "to_do" queue empty they notify a maintenance thread which does some work and notify these threads back.But the problem is the program is getting blocked sometimes (so i assume there is some race condition I am not able to take care of)....also upon checking I found that not all threads are getting notified by maintenace thread.so is it possible that some notify signals might be lost??
Code for Master class
private Queue<String> to_do = new LinkedList<String>();
private Queue<String> done= new LinkedList<String>();
private Queue<String> to_do_next = new LinkedList<String>();
private int level = 1;
private Object lock1 = new Object();
private Object lock2 = new Object();
private Object lock3 = new Object();
private static Thread maintenance;
public static Master mref;
public static Object wait1 = new Object();
public static Object wait2 = new Object();
public static Object wait3 = new Object();
public static int flag = 5;
public static int missedSignals = -1;
public boolean checkToDoEmpty(){
return to_do.isEmpty();
}
public int getLevel() {
return level;
}
public void incLevel() {
this.level++;
}
public static void interrupt() {
maintenance.interrupt();
}
public void transfer() {
to_do = to_do_next;
}
public String accessToDo() {
synchronized(lock1){
String tmp = to_do.peek();
if(tmp != null)
tmp = to_do.remove();
return tmp;
}
}
public void addToDoNext(String url){
synchronized(lock2){
to_do_next.add(url);
}
}
public void addDone(String string) {
synchronized(lock3){
done.add(string);
}
}
public static void main(String[] args){
Master m = new Master();
mref = m;
URL startUrl = null;
try {
startUrl = new URL("http://cse.iitkgp.ac.in");
}catch (MalformedURLException e1) {
e1.printStackTrace();
}
Thread t1 = new Thread(new Worker(1));
Thread t2 = new Thread(new Worker(2));
Thread t3 = new Thread(new Worker(3));
Thread t4 = new Thread(new Worker(4));
Thread t5 = new Thread(new Worker(5));
maintenance = new Thread(new MaintenanceThread());
m.to_do.add(startUrl.toString());
maintenance.start();
t1.start();
t2.start();
t3.start();
t4.start();
t5.start();
try {
t1.join();
t2.join();
t3.join();
t4.join();
t5.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
/*for(String s:m.done)
System.out.println(s);
for(String s:m.to_do)
System.out.println(s);*/
}
Code for Worker threads
public void run() {
while(Master.mref.getLevel() != 3){
if(!Master.mref.checkToDoEmpty()){
String url = Master.mref.accessToDo();
if(url != null && url.contains("iitkgp") && url.contains("http://")){
try {
Document doc = Jsoup.connect(url).get();
org.jsoup.select.Elements links = doc.select("a[href]");
for(org.jsoup.nodes.Element l: links){
Master.mref.addToDoNext(l.attr("abs:href").toString());
}
Master.mref.addDone(url);
} catch (IOException e) {
System.out.println(url);
e.printStackTrace();
}
continue;
}
}
//System.out.println("thread " + id + " about to notify on wait1");
synchronized(Master.wait1){
Master.wait1.notify();
Master.missedSignals++;
}
synchronized(Master.wait2){
try {
Master.wait2.wait();
System.out.println("thread " + id + " coming out of wait2");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
System.out.println("Terminating " + id + " thread");
Master.flag--;
if(Master.flag == 0)
Master.interrupt();
}
Code for Maintenace thread
while(Master.flag != 0){
try {
synchronized(Master.wait1){
if(Master.missedSignals != -1){
count += Master.missedSignals;
Master.missedSignals = -1;
}
while(count != 5){
Master.wait1.wait();
if(Master.missedSignals != -1)
count += Master.missedSignals;
Master.missedSignals = -1;
count++;
}
count = 0;
}
//System.out.println("in between");
Master.mref.incLevel();
Master.mref.transfer();
synchronized(Master.wait2){
Master.wait2.notifyAll();
}
} catch (InterruptedException e) {
break;
}
}
System.out.println("Mainta thread gone");
Your design is way too complicated
i suggest using for your to_do queue the following: LinkedBlockingQueue
This is a blocking queue, which means that your threads will ask for an object from the queue and only when one will appear they will get the object, till then they will stay blocking.
Just use the following methods to put and take objects in the queue: put() & take()
Please look at the following two links for more explanations on this special queue:
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/LinkedBlockingQueue.html
http://tutorials.jenkov.com/java-util-concurrent/linkedblockingqueue.html
Now, your only concern is killing the threads when they are finished with their work, for that I suggest the following:
boolean someThreadStillAlive = true;
while (someThreadStillAlive) {
someThreadStillAlive = false;
Thread.sleep(200);
for (Thread t : fetchAndParseThreads) {
someThreadStillAlive = someThreadStillAlive || t.isAlive();
}
}
This will occur in your main code block, where it will loop & sleep till all threads are finished.
Ohh, instead of take(), you can use poll(int timeout...) where it will wait for the timeout to finish and if no new object is inserted into the queue it will kill the thread.
All of the above, were used successfully in my own crawler.
Im working on a project and is confused at why the thread wont start when I call .start()
int count = 0;
while (count < urls.length) {
try {
Thread thread = new Thread(new read(urls[count]));
thread.start();
} catch (Exception e) {
}
count++;
}
but if I add
public void start() {
run();
}
in the read class and change the code to
int count = 0;
while (count < urls.length) {
try {
read thread = new read(urls[count]);
thread.start();
} catch (Exception e) {
}
count++;
}
it works fine.
EDIT : here is my read class code
its reading data from url and calling some other methods i have in the class to store data etc.
public class read implements Runnable {
URL url;
public read(String str) throws IOException {
url = new URL(str);
}
public void run() {
try {
URLConnection connect = url.openConnection();
BufferedReader reader = new BufferedReader(new InputStreamReader(connect.getInputStream()));
String input;
String[] temp;
int x = 0;
while (x < 10) {
reader.readLine();
x++;
}
while ((input = reader.readLine()) != null) {
temp = input.split(" ");
temp[2].replaceAll("<br>", "");
String name = temp[0];
int flightNum = Integer.parseInt(temp[1]);
String des = temp[2];
if (Airport.containsKey(flightNum) != true) {
addFlight(flightNum, des);
addPassengerReservation(flightNum, name);
}
else {
addPassengerReservation(flightNum, name);
}
}
reader.close();
}catch (Exception e) {}
}
}
You should make a class implementing Runnable (ClassA), and override the
public void run()
method. From your "main" program, you should call:
Thread th = new Thread(new ClassA());
th.start();
You should never override the start method, or call th.run(). Calling the start method will do some "behind the scenes" work, then call your Runnable Object's run() method for you.
Thread.start() code registers the Thread with scheduler and the scheduler calls the run() method.You need to override the run() method.No need to call run() implicitly.
I have some problem with CompletionService.
My task: to parse parallely for about 300 html pages, i need to wait for all the results only for 5 seconds, then - return the result to main code.
I`ve decided to use CompletionService + Callable for this.
The question is how to stop all threads, that were caused by CompletionService and return the result from that pages, that were successfully parsed? In this code removed printlines, but i can say that 5 seconds is enough(there are good results, but program wait when all threads will be completed). My code performed about 2 minutes.
My calling code:
Collection<Callable<HCard>> solvers = new ArrayList<Callable<HCard>>();
for (final String currentUrl : allUrls) {
solvers.add(new Callable<HCard>() {
public HCard call() throws ParserException {
HCard hCard = HCardParser.parseOne(currentUrl);
if (hCard != null) {
return hCard;
} else {
return null;
}
}
});
}
ExecutorService execService = Executors.newCachedThreadPool();
Helper helper = new Helper();
List<HCard> result = helper.solve(execService, solvers);
//then i do smth with result list
My called code:
public class Helper {
List<HCard> solve(Executor e, Collection<Callable<HCard>> solvers) throws InterruptedException {
CompletionService<HCard> cs = new ExecutorCompletionService<HCard>(e);
int n = solvers.size();
Future<HCard> future = null;
HCard hCard = null;
ArrayList<HCard> result = new ArrayList<HCard>();
for (Callable<HCard> s : solvers) {
cs.submit(s);
}
for (int i = 0; i < n; ++i) {
try {
future = cs.take();
hCard = future.get();
if (hCard != null) {
result.add(hCard);
}
} catch (ExecutionException e1) {
future.cancel(true);
}
}
return result;
}
I attempted to use:
awaitTermination(5000, TimeUnit.MILLISECONDS)
future.cancel(true)
execService.shutdownNow()
future.get(5000, TimeUnit.MILLISECONDS);
TimeOutException: i can`t get TimeOutException.
Please, help me on context of my code.
Thanks in advance!
You need to ensure that the tasks that you submit respond properly to interruption, i.e., they check Thread.isInterrupted() or are otherwise deemed "interruptible".
I'm not sure that you need a completion service for this.
ExecutorService service = ...
// Submit all your tasks
for (Task t : tasks) {
service.submit(t);
}
service.shutdown();
// Wait for termination
boolean success = service.awaitTermination(5, TimeUnit.SECONDS);
if (!success) {
// awaitTermination timed out, interrupt everyone
service.shutdownNow();
}
At this point, there is not much you can do if your Task objects don't respond to interruption
The problem is you always get every single result, so the code will always run to completion. I'd do it with a CountDownLatch as per the code below.
Also, don't use Executors.newCachedThreadPool - chances are this will spawn a lot of threads (up to 300 if your tasks take any amount of time, as the executor won't let the number of idle threads drop to zero).
Classes are all inline to make it easier - paste the whole code block into a class called LotsOfTasks and run it.
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.List;
import java.util.Random;
import java.util.concurrent.Callable;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
public class LotsOfTasks {
private static final int SIZE = 300;
public static void main(String[] args) throws InterruptedException {
String[] allUrls = generateUrls(SIZE);
Collection<Callable<HCard>> solvers = new ArrayList<Callable<HCard>>();
for (final String currentUrl : allUrls) {
solvers.add(new Callable<HCard>() {
public HCard call() {
HCard hCard = HCardParser.parseOne(currentUrl);
if (hCard != null) {
return hCard;
} else {
return null;
}
}
});
}
ExecutorService execService = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors()); // One thread per cpu, ideal for compute-bound
Helper helper = new Helper();
System.out.println("Starting..");
long start = System.nanoTime();
List<HCard> result = helper.solve(execService, solvers, 5);
long stop = System.nanoTime();
for (HCard hCard : result) {
System.out.println("hCard = " + hCard);
}
System.out.println("Took: " + TimeUnit.SECONDS.convert((stop - start), TimeUnit.NANOSECONDS) + " seconds");
}
private static String[] generateUrls(final int size) {
String[] urls = new String[size];
for (int i = 0; i < size; i++) {
urls[i] = "" + i;
}
return urls;
}
private static class HCardParser {
private static final Random random = new Random();
public static HCard parseOne(String currentUrl) {
try {
Thread.sleep(random.nextInt(1000)); // Wait for a random time up to 1 seconds per task (simulate some activity)
} catch (InterruptedException e) {
// ignore
}
return new HCard(currentUrl);
}
}
private static class HCard {
private final String currentUrl;
public HCard(String currentUrl) {
this.currentUrl = currentUrl;
}
#Override
public String toString() {
return "HCard[" + currentUrl + "]";
}
}
private static class Helper {
List<HCard> solve(ExecutorService e, Collection<Callable<HCard>> solvers, int timeoutSeconds) throws InterruptedException {
final CountDownLatch latch = new CountDownLatch(solvers.size());
final ConcurrentLinkedQueue<HCard> executionResults = new ConcurrentLinkedQueue<HCard>();
for (final Callable<HCard> s : solvers) {
e.submit(new Callable<HCard>() {
public HCard call() throws Exception {
try {
executionResults.add(s.call());
} finally {
latch.countDown();
}
return null;
}
});
}
latch.await(timeoutSeconds, TimeUnit.SECONDS);
final List<Runnable> unfinishedTasks = e.shutdownNow();
System.out.println("There were " + unfinishedTasks.size() + " urls not processed");
return Arrays.asList(executionResults.toArray(new HCard[executionResults.size()]));
}
}
}
Typical output on my system looks something like this:
Starting..
There were 279 urls not processed
hCard = HCard[0]
hCard = HCard[1]
hCard = HCard[2]
hCard = HCard[3]
hCard = HCard[5]
hCard = HCard[4]
hCard = HCard[6]
hCard = HCard[8]
hCard = HCard[7]
hCard = HCard[10]
hCard = HCard[11]
hCard = HCard[9]
hCard = HCard[12]
hCard = HCard[14]
hCard = HCard[15]
hCard = HCard[13]
hCard = HCard[16]
hCard = HCard[18]
hCard = HCard[17]
hCard = HCard[20]
hCard = HCard[19]
Took: 5 seconds
I never used CompletionService, but I'm sure there is a poll(timeunit,unit) call to do a limited amount wait. Then check for null. Measure the time waited and stop waiting after 5 seconds. Approximately:
public class Helper {
List<HCard> solve(Executor e, Collection<Callable<HCard>> solvers)
throws InterruptedException {
CompletionService<HCard> cs = new ExecutorCompletionService<HCard>(e);
int n = solvers.size();
Future<HCard> future = null;
HCard hCard = null;
ArrayList<HCard> result = new ArrayList<HCard>();
for (Callable<HCard> s : solvers) {
cs.submit(s);
}
long timeleft = 5000;
for (int i = 0; i < n; ++i) {
if (timeleft <= 0) {
break;
}
try {
long t = System.currentTimeMillis();
future = cs.poll(timeleft, TimeUnit.MILLISECONDS);
timeleft -= System.currentTimeMillis() - t;
if (future != null) {
hCard = future.get();
if (hCard != null) {
result.add(hCard);
}
} else {
break;
}
} catch (ExecutionException e1) {
future.cancel(true);
}
}
return result;
}
Not tested though.