I call the below testMethod, after putting it into a Callable(with other few Callable tasks), from an ExecutorService. I suspect that, the map.put() suffers OutOfMemoryError, as I'm trying to put some 20 million entries.
But, I'm not able to see the error trace in console. Just the thread stops still. I tried to catch the Error ( I know.. we shouldnt, but for debug I caught). But, the error is not caught. Directly enters finally and stops executing.. and the thread stands still.
private HashMap<String, Integer> testMethod(
String file ) {
try {
in = new FileInputStream(new File(file));
br = new BufferedReader(new InputStreamReader(in), 102400);
for (String line; (line= br.readLine()) != null;) {
map.put(line.substring(1,17),
Integer.parseInt(line.substring(18,20)));
}
System.out.println("Loop End"); // Not executed
} catch(Error e){
e.printStackTrace(); //Not executed
}finally {
System.out.println(map.size()); //Executed
br.close();
in.close();
}
return map;
}
Wt could be the mistake, I'm doing?
EDIT: This is how I execute the Thread.
Callable<Void> callable1 = new Callable<Void>() {
#Override
public Void call() throws Exception {
testMethod(inputFile);
return null;
}
};
Callable<Void> callable2 = new Callable<Void>() {
#Override
public Void call() throws Exception {
testMethod1();
return null;
}
};
List<Callable<Void>> taskList = new ArrayList<Callable<Void>>();
taskList.add(callable1);
taskList.add(callable2);
// create a pool executor with 3 threads
ExecutorService executor = Executors.newFixedThreadPool(3);
List<Future<Void>> future = executor.invokeAll(taskList);
//executor.invokeAll(taskList);
latch.await();
future.get(0);future.get(1); //Added this as per SubOptimal'sComment
But, this future.get() didn't show OOME in console.
You should not throw away the future after submitting the Callable.
Future future = pool.submit(callable);
future.get(); // this would show you the OOME
example based on the informations of the requestor to demonstrate
public static void main(String[] args) throws InterruptedException, ExecutionException {
Callable<Void> callableOOME = new Callable<Void>() {
#Override
public Void call() throws Exception {
System.out.println("callableOOME");
HashMap<String, Integer> map = new HashMap<>();
// some code to force an OOME
try {
for (int i = 0; i < 10_000_000; i++) {
map.put(Integer.toString(i), i);
}
} catch (Error e) {
e.printStackTrace();
} finally {
System.out.println("callableOOME: map size " + map.size());
}
return null;
}
};
Callable<Void> callableNormal = new Callable<Void>() {
#Override
public Void call() throws Exception {
System.out.println("callableNormal");
// some code to have a short "processing time"
try {
TimeUnit.SECONDS.sleep(5);
} catch (InterruptedException ex) {
System.err.println(ex.getMessage());
}
return null;
}
};
List<Callable<Void>> taskList = new ArrayList<>();
taskList.add(callableOOME);
taskList.add(callableNormal);
ExecutorService executor = Executors.newFixedThreadPool(3);
List<Future<Void>> future = executor.invokeAll(taskList);
System.out.println("get future 0: ");
future.get(0).get();
System.out.println("get future 1: ");
future.get(1).get();
}
Try catching Throwable as it could be an Exception like IOException or NullPointerException, Throwable captures everything except System.exit();
Another possibility is that the thread doesn't die, instead it becomes increasingly slower and slower due to almost running out of memory but never giving up. You should be able to see this with a stack dump or using jvisualvm while it is running.
BTW Unless all you strings are exactly 16 characters long, you might like to call trim() on the to remove any padding in the String. This could make them shorter and use less memory.
I assume you are using a recent version of Java 7 or 8. If you are using Java 6 or older, it will use more memory as .substring() doesn't create a new underlying char[] to save CPU, but in this case wastes memory.
Related
Below is my code:
public class Controller {
public Button button_submitWork;
#FXML
public void handleSubmitWork(ActionEvent event) {
final ExecutorService executorService = Executors.newFixedThreadPool(1, r -> {
Thread t = Executors.defaultThreadFactory().newThread(r);
t.setDaemon(true);
return t;
});//set thread daemon, let all threads terminate when the program is closed.
Callable<String> callable = new Callable<String>() {
#Override
public String call() throws Exception {
System.out.println("Executor Service thread");
StringBuilder stringBuilder_output = new StringBuilder();
for (int k = 0; k < 5; k++) {
stringBuilder_output.append(k);
}
//Thread.sleep(1000);
return stringBuilder_output.toString() + "\n";
}
};
Future<String> future = executorService.submit(callable);//Weird line.
//This line must be placed inside the "watchThread" to get the result, but why???
Thread watchThread = new Thread(new Runnable() {
#Override
public void run() {
//<----------Moving to here solve the problem!
System.out.println("Watch thread");
while (!Thread.currentThread().isInterrupted() && !future.isDone()) {
try {
String result = future.get();
System.out.println(result);
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
} finally {
executorService.shutdownNow();
}
}
}
});
watchThread.setDaemon(true);
watchThread.start();
System.out.println("FX thread");
}
}
The question is that the System.out.println(result); inside "watchThread" is never been called. The console output looks like this:
Executor Service thread
FX thread
Watch thread
But when I move the Future<String> future = executorService.submit(callable); to the inside of run method of "watchThread", the output change to:
FX thread
Watch thread
Executor Service thread
01234
which is I expected.
I also discovered that if the call() method has a longer task, say a Thread.sleep(1000), the output change to the result I expected.
So why is that?
The thread you submit to executorService finishes before this line:
while (!Thread.currentThread().isInterrupted() && !future.isDone()) { is called so future.isDone returns true and the while loop is not executed.
If you add Thread.sleep(1000) then it still runs and future.isDone returns false and the while loop executes. The same thing happens when you move Future<String> future = executorService.submit(callable); inside watchThread.
I am trying to call a method multiple times every 60 seconds until a success response from the method which actually calls a rest end point on a different service. As of now I am using do while loop and using
Thread.sleep(60000);
to make the main thread wait 60 seconds which I feel is not the ideal way due to concurrency issues.
I came across the CountDownLatch method using
CountDownLatch latch = new CountDownLatch(1);
boolean processingCompleteWithin60Second = latch.await(60, TimeUnit.SECONDS);
#Override
public void run(){
String processStat = null;
try {
status = getStat(processStatId);
if("SUCCEEDED".equals(processStat))
{
latch.countDown();
}
} catch (Exception e) {
e.printStackTrace();
}
}
I have the run method in a different class which implements runnable. Not able to get this working. Any idea what is wrong?
You could use a CompletableFuture instead of CountDownLatch to return the result:
CompletableFuture<String> future = new CompletableFuture<>();
invokeYourLogicInAnotherThread(future);
String result = future.get(); // this blocks
And in another thread (possibly in a loop):
#Override
public void run() {
String processStat = null;
try {
status = getStat(processStatId);
if("SUCCEEDED".equals(processStat))
{
future.complete(processStat);
}
} catch (Exception e) {
future.completeExceptionally(e);
}
}
future.get() will block until something is submitted via complete() method and return the submitted value, or it will throw the exception supplied via completeExceptionally() wrapped in an ExecutionException.
There is also get() version with timeout limit:
String result = future.get(60, TimeUnit.SECONDS);
Finally got it to work using Executor Framework.
final int[] value = new int[1];
pollExecutor.scheduleWithFixedDelay(new Runnable() {
Map<String, String> statMap = null;
#Override
public void run() {
try {
statMap = coldService.doPoll(id);
} catch (Exception e) {
}
if (statMap != null) {
for (Map.Entry<String, String> entry : statMap
.entrySet()) {
if ("failed".equals(entry.getValue())) {
value[0] = 2;
pollExecutor.shutdown();
}
}
}
}
}, 0, 5, TimeUnit.MINUTES);
try {
pollExecutor.awaitTermination(40, TimeUnit.MINUTES);
} catch (InterruptedException e) {
}
I'm a tapestry-hibernate user and I'm experiencing an issue where my session remains closed once I exceed my Executors.newFixedThreadPool(1);
I have the following code which will work perfectly for the first thread while the remaining threads experience a closed session. If I increase the thread pool to 10, all the threads will run without issue. As soon as I exceed the fixedThreadPool, I get the session closed exception. I do not know how to open it since it's managed by tapestry-hibernate. If I use newCachedThreadPool, everything works perfectly. Does anybody know what might be happening here?
public void setupRender() {
ExecutorService executorService = Executors.newFixedThreadPool(1);
final ConcurrentHashMap<String, Computer> map = new ConcurrentHashMap<>();
final String key = "myKey";
final Date date = new Date();
List<Future> futures = new ArrayList<>();
for (int i = 0; i < 10; i++) {
final int thread = i;
Future future = executorService.submit(new Callable() {
#Override
public String call() {
try {
Computer computer = new Computer("Test Computer thread");
computer = getComputer(map, key, key, computer);
Monitor monitor = new Monitor();
monitor.setComputer(computer);
session.save(monitor);
session.flush();
System.out.println("thread " + thread);
try {
sessionManager.commit();
} catch (HibernateException ex) {
sessionManager.abort();
} finally {
session.close();
}
} catch (Exception ex) {
System.out.println("ex " + ex);
}
System.out.println( new Date().getTime() - date.getTime());
return "completed";
}
});
futures.add(future);
}
for(Future future : futures) {
try {
System.out.println(future.get());
} catch (InterruptedException | ExecutionException ex) {
Logger.getLogger(MultiThreadDemo.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
public synchronized Computer getComputer(ConcurrentHashMap<String, Computer> map, String key, String thread, Computer computer) {
if (map.putIfAbsent(key, computer) == null) {
session.save(computer);
} else {
computer = map.get(key);
}
return computer;
}
I've told you this before.... you MUST either use ParallelExecutor OR call PerThreadManager.cleanup(). You need to understand that tapestry-hibernate has PerThread scoped services that MUST be cleaned up if you are using them outside of a normal request/response (or ParallelExecutor).
I also don't think you should be calling session.close(). You should mimmic CommitAfterWorker.
It would probably look something like:
#Inject PerThreadManager perThreadManager;
#Inject HibernateSessionManager sessionManager; // this is a proxy to a per-thread value
#Inject Session session; // this is a proxy to a per-thread value
public void someMethod() {
ExecutorService executorService = ...;
executorService.submit(new Callable() {
public String call() {
try {
Monitor monitor = ...
session.save(monitor);
session.flush(); // optional
sessionManager.commit();
} catch (Exception ex) {
sessionManager.abort();
} finally {
// this allows Session and HibernateSessionManager to
// clean up after themselves
perThreadManager.cleanup();
}
return ...
}
});
}
If you choose to use the ParallelExecutor (and Invokable) instead of Executors.newFixedThreadPool(1) you can remove the references to PerThreadManager since it automatically cleans up the thread.
I have to send a set of files to several computers through a certain port. The fact is that, each time that the method that sends the files is called, the destination data (address and port) is calculated. Therefore, using a loop that creates a thread for each method call, and surround the method call with a try-catch statement for a BindException to process the situation of the program trying to use a port which is already in use (different destination addresses may receive the message through the same port) telling the thread to wait some seconds and then restart to retry, and keep trying until the exception is not thrown (the shipping is successfully performed).
I didn't know why (although I could guess it when I first saw it), Netbeans warned me about that sleeping a Thread object inside a loop is not the best choice. Then I googled a bit for further information and found this link to another stackoverflow post, which looked so interesting (I had never heard of the ThreadPoolExecutor class). I've been reading both that link and the API in order to try to improve my program, but I'm not yet pretty sure about how am I supposed to apply that in my program. Could anybody give a helping hand on this please?
EDIT: The important code:
for (Iterator<String> it = ConnectionsPanel.list.getSelectedValuesList().iterator(); it.hasNext();) {
final String x = it.next();
new Thread() {
#Override
public void run() {
ConnectionsPanel.singleAddVideos(x);
}
}.start();
}
private static void singleAddVideos(String connName) {
String newVideosInfo = "";
for (Iterator<Video> it = ConnectionsPanel.videosToSend.iterator(); it.hasNext();) {
newVideosInfo = newVideosInfo.concat(it.next().toString());
}
try {
MassiveDesktopClient.sendMessage("hi", connName);
if (MassiveDesktopClient.receiveMessage(connName).matches("hello")) {
MassiveDesktopClient.sendMessage(newVideosInfo, connName);
}
} catch (BindException ex) {
MassiveDesktopClient.println("Attempted to use a port which is already being used. Waiting and retrying...", new Exception().getStackTrace()[0].getLineNumber());
try {
Thread.sleep(MassiveDesktopClient.PORT_BUSY_DELAY_SECONDS * 1000);
} catch (InterruptedException ex1) {
JOptionPane.showMessageDialog(null, ex1.toString(), "Error", JOptionPane.ERROR_MESSAGE);
}
ConnectionsPanel.singleAddVideos(connName);
return;
}
for (Iterator<Video> it = ConnectionsPanel.videosToSend.iterator(); it.hasNext();) {
try {
MassiveDesktopClient.sendFile(it.next().getAttribute("name"), connName);
} catch (BindException ex) {
MassiveDesktopClient.println("Attempted to use a port which is already being used. Waiting and retrying...", new Exception().getStackTrace()[0].getLineNumber());
try {
Thread.sleep(MassiveDesktopClient.PORT_BUSY_DELAY_SECONDS * 1000);
} catch (InterruptedException ex1) {
JOptionPane.showMessageDialog(null, ex1.toString(), "Error", JOptionPane.ERROR_MESSAGE);
}
ConnectionsPanel.singleAddVideos(connName);
return;
}
}
}
Your question is not very clear - I understand that you want to rerun your task until it succeeds (no BindException). To do that, you could:
try to run your code without catching the exception
capture the exception from the future
reschedule the task a bit later if it fails
A simplified code would be as below - add error messages and refine as needed:
public static void main(String[] args) throws Exception {
ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(corePoolSize);
final String x = "video";
Callable<Void> yourTask = new Callable<Void>() {
#Override
public Void call() throws BindException {
ConnectionsPanel.singleAddVideos(x);
return null;
}
};
Future f = scheduler.submit(yourTask);
boolean added = false; //it will retry until success
//you might use an int instead to retry
//n times only and avoid the risk of infinite loop
while (!added) {
try {
f.get();
added = true; //added set to true if no exception caught
} catch (ExecutionException e) {
if (e.getCause() instanceof BindException) {
scheduler.schedule(yourTask, 3, TimeUnit.SECONDS); //reschedule in 3 seconds
} else {
//another exception was thrown => handle it
}
}
}
}
public static class ConnectionsPanel {
private static void singleAddVideos(String connName) throws BindException {
String newVideosInfo = "";
for (Iterator<Video> it = ConnectionsPanel.videosToSend.iterator(); it.hasNext();) {
newVideosInfo = newVideosInfo.concat(it.next().toString());
}
MassiveDesktopClient.sendMessage("hi", connName);
if (MassiveDesktopClient.receiveMessage(connName).matches("hello")) {
MassiveDesktopClient.sendMessage(newVideosInfo, connName);
}
for (Iterator<Video> it = ConnectionsPanel.videosToSend.iterator(); it.hasNext();) {
MassiveDesktopClient.sendFile(it.next().getAttribute("name"), connName);
}
}
}
Sorry I have to open a new thread to describe this problem.
This morning I asked this question, there're some replies but my problem is still not solved.
This time I will attach some runnable code(simplified but with the same problem) for you to reproduce the problem:
public class ThreadPoolTest {
public static void main(String[] args) throws Exception {
final ExecutorService taskExecutor = Executors.newFixedThreadPool(5);
Future<Void> futures[] = new Future[5];
for (int i = 0; i < futures.length; ++i)
futures[i] = startTask(taskExecutor);
for (int i = 0; i < futures.length; ++i)
System.out.println("futures[i].cancel(true): " + futures[i].cancel(true));
System.out.println("Cancel DONE.");
taskExecutor.shutdown();
}
private static Future<Void> startTask(final ExecutorService taskExecutor) {
Future<Void> f = taskExecutor.submit(new Callable<Void>() {
public Void call() throws Exception {
try {
downloadFile(new URI("http://stackoverflow.com"));
while(true) {
System.out.println(Thread.currentThread().getName() + ": " + Thread.currentThread().isInterrupted());
if(Thread.currentThread().isInterrupted())
break;
}
} catch (Exception ex) {
ex.printStackTrace();
}
return null;
}
});
return f;
}
private static void downloadFile (final URI uri) throws Exception {
// if(true) return;
Socket socket = new Socket (uri.getHost(), uri.getPort() == -1 ? 80 : uri.getPort());
return;
}
}
The code above will most likely be trapped in an infinite loop(you may want to run the code multiple times to witness what I saw), as you can see in the main method I have called futures[i].cancel(true) for all tasks, I don't know why this is happening, this has been torturing me for more than a day.
Your help will be greatly appreciated.
I've played with your code, and noticed that the thread's interrupt status is sometimes true before the socket creation, and false after.
I have tried interrupting a thread and calling the Socket constructor, and the thread always stays interrupted after. I also tried removing the shutdown of the threadpool, and the problem continued to happen.
Then I have tried using 5 different URIs, rather than always the same one. And the problem never happened.
So I wrote this simple program, showing that the thread pool is not the culprit, but the socket is:
public static void main(String[] args) throws Exception {
final URI uri = new URI("http://stackoverflow.com");
for (int i = 0; i < 5; i++) {
Runnable r = new Runnable() {
#Override
public void run() {
Thread.currentThread().interrupt();
System.out.println(Thread.currentThread().isInterrupted());
try {
Socket socket = new Socket (uri.getHost(), uri.getPort() == -1 ? 80 : uri.getPort());
}
catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println(Thread.currentThread().isInterrupted());
}
};
new Thread(r).start();
}
}
And indeed, when 5 threads create a socket to the same host and port, 4 of them have their interrupt status cleared.
Then I tried to synchronize the socket creation (on a single lock, but I guess you might use one lock per host/port) :
synchronized(lock) {
try {
Socket socket = new Socket (uri.getHost(), uri.getPort() == -1 ? 80 : uri.getPort());
}
catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
and TADA... the problem disappeared. I would open a bug at Oracle to signal the problem.
I ran your code, and it didn't stop, as you said.
Didn't have much time to investigate why it behaves so, but I found out that declaring the executor service's threads as daemons made the problem go away :
private static ExecutorService TaskExecutor = Executors.newFixedThreadPool(5, new ThreadFactory() {
public Thread newThread(Runnable r) {
Thread t = new Thread(r);
t.setDaemon(true);
return t;
}
});
I'll come back if I find a better explanation.
I think the problem that task are not started when you try to cancel them. I added Thread.sleep(100) like this:
for (int i = 0; i < futures.length; ++i)
futures[i] = startTask(taskExecutor);
Thread.sleep(100);
for (int i = 0; i < futures.length; ++i)
System.out.println("futures[i].cancel(true): " + futures[i].cancel(true));
and everything was cancelled ok.