I have a dedicated server running CentOS 5.9, Apache-Tomcat 5.5.36. I have written a JAVA web applications which runs every minute to collect the data from multiple sensors. I am using ScheduledExecutorService to execute the threads. (one thread for each sensor every minute and there can be more than hundred sensors) The flow of the thread is
Collect sensor information from the database.
Sends the command to the instrument to collect data.
Update the database with the data values.
There is another application that checks the database every minute and send the alerts to the users (if necessary). I have monitored the application using jvisualVM, I cant find any memory leak. for every thread. The applications work fine but after some time(24 Hour - 48 Hours) the applications stop working. I cant find out what the problem could be, is it server configuration problem, too many threads or what?
Does anyone have any idea what might be going wrong or is there anyone who has done think kind of work? Please help, Thanks
UPDATE : including code
public class Scheduler {
private final ScheduledExecutorService scheduler =
Executors.newScheduledThreadPool(1);
public void startProcess(int start) {
final Runnable uploader = new Runnable() {
#SuppressWarnings("rawtypes")
public void run()
{
//Select data from the database
ArrayList dataList = getData();
for(int i=0;i<dataList.size();i++)
{
String args = dataList.get(i).toString();
ExecutorThread comThread = new ExecutorThread(args...);
comThread.start();
}
}
};
scheduler.scheduleAtFixedRate(uploader, 0, 60 , TimeUnit.SECONDS);
}
}
public class ExecutorThread extends Thread {
private variables...
public CommunicationThread(args..)
{
//Initialise private variable
}
public void run()
{
//Collect data from sensor
//Update Database
}
}
Can't say much without a code, but you need to be sure that your thread always exits properly - doesn't hang in memory on any exception, closes connection to database, etc.
Also, for monitoring your application, you can take a thread dump every some period of time to see how many threads the application generates.
Another suggestion is configure Tomcat to take a heap dump on OutOfMemoryError. If that's an issue, you'll be able to analyze what is filling up the memory
Take heed of this innocuous line from the ScheduledExecutorService.schedule... Javadoc
If any execution of the task encounters an exception, subsequent executions are suppressed.
This means that if you are running into an Exception at some point and not handling it, the Exception will propagate into the ScheduledExecutorService and it will kill your task.
To avoid this problem you need to make sure the entire Runnable is wrapped in a try...catch and Exceptions are guaranteed to never be unhandled.
You can also extend the ScheduledExecutorService (also mentioned in the javadoc) to handle uncaught exceptions :-
final ScheduledExecutorService ses = new ScheduledThreadPoolExecutor(10){
#Override
protected void afterExecute(Runnable r, Throwable t) {
super.afterExecute(r, t);
if (t == null && r instanceof Future<?>) {
try {
Object result = ((Future<?>) r).get();
} catch (CancellationException ce) {
t = ce;
} catch (ExecutionException ee) {
t = ee.getCause();
} catch (InterruptedException ie) {
Thread.currentThread().interrupt(); // ignore/reset
}
}
if (t != null) {
System.out.println(t);
}
}
};
Here the afterExecute method simply System.out.printlns the Throwable but it could do other things. Alert users, restart tasks etc...
Related
For instance consider the below scenario.
App1: I have a multiple-threaded java app, which enters a lot of files in DB.
App2: when i access the DB using some other app, its slow in fetching results.
So when both apps work simultaneously, it takes great time for DB fetching results on the front-end app2.
Here, i want to pause all transactions(threads) on App1 for some 'x min' time. Considering a trigger has already been installed when app 2 is being used. So when App2 is idle, App1 will resume as if nothing happened. Please list some or one best approach to achieve this
Map<Thread, StackTraceElement[]> threads = Thread.getAllStackTraces();
for (Map.Entry<Thread, StackTraceElement[]> entry : threads.entrySet()) {
entry.getKey().sleep();
}
This didn't worked well.
Just to try:
private List<PausableThread> threads = new ArrayList<PausableThread>();
private void pauseAllThreads()
{
for(PausableThread thread : this.threads)
{
thread.pause();
}
}
And your Thread class will be something like this:
public class MyThread extends Thread implements PausableThread
{
private boolean isPaused = false;
#Override
public void pause()
{
this.isPaused = true;
}
#Override
public void run()
{
while(!Thread.currentThread().isInterrupted())
{
// Do your work...
// Check if paused
if(this.isPaused)
{
try
{
Thread.sleep(10 * 1000);
}
catch (InterruptedException e)
{
e.printStackTrace();
}
}
}
}
}
And the PausableThread interface:
public interface PausableThread
{
void pause();
}
Posting a solution answer, for my scenario.
I created a global flag and used it as a switch.
SO now, before DB interaction i just added a condition [in various functions where threads were performing variety of jobs, this solved the instance issue i was worried about]
if(isFlagChecked){thread.sleep(someDefinedTime);}
wait here if flag is true
continue with business logic...[db transacts here]
So, my issue was solved with just this, although it wouldn't pause thread running in intermediate state, which is kind of a good thing - one less trouble.
Parallel, in my trigger function - i checked for the elapsed time and changed the flag to false after desired time has passed. Check code skeleton below.
#async
void pause() // triggered by parallel running app when required
{
isFlagChecked=true;
resumeTime=new Date(timeInMillis + (someDefinedTime)) // resume time to switch flag condition
while (true) {
if (new Date().compareTo(resumeTime) > 0)
isFlagChecked=false;
}
}
Tried and tested, all running well, the performance improved significantly [least for my scenario].
I have an application that receives alerts from other applications, usually once a minute or so but I need to be able to handle higher volume per minute. The interface I am using, and the Alert framework in general, requires that alerts may be processed asynchronously and can be stopped if they are being processed asynchronously. The stop method specifically is documented as stopping a thread. I wrote the code below to create an AlertRunner thread and then stop the thread. However, is this a proper way to handle terminating a thread? And will this code be able to scale easily (not to a ridiculous volume, but maybe an alert a second or multiple alerts at the same time)?
private AlertRunner alertRunner;
#Override
public void receive(Alert a) {
assert a != null;
alertRunner = new alertRunner(a.getName());
a.start();
}
#Override
public void stop(boolean synchronous) {
if(!synchronous) {
if(alertRunner != null) {
Thread.currentThread().interrupt();
}
}
}
class AlertRunner extends Thread {
private final String alertName;
public AlertRunner(String alertName) {
this.alertName = alertName;
}
#Override
public void run() {
try {
TimeUnit.SECONDS.sleep(5);
log.info("New alert received: " + alertName);
} catch (InterruptedException e) {
log.error("Thread interrupted: " + e.getMessage());
}
}
}
This code will not scale easily because Thread is quite 'heavy' object. It's expensive to create and it's expensive to start. It's much better to use ExecutorService for your task. It will contain a limited number of threads that are ready to process your requests:
int threadPoolSize = 5;
ExecutorService executor = Executors.newFixedThreadPool(threadPoolSize);
public void receive(Alert a) {
assert a != null;
executor.submit(() -> {
// Do your work here
});
}
Here executor.submit() will handle your request in a separate thread. If all threads are busy now, the request will wait in a queue, preventing resource exhausting. It also returns an instance of Future that you can use to wait for the completion of the handling, setting the timeout, receiving the result, for cancelling execution and many other useful things.
I have the following piece of code:
public class Test {
List<Future> future = new ArrayList<Future>();
public static void main(String args[]) throws Exception {
Adapter b1 = new Adapter();
final ExecutorService threadPool = Executors.newCachedThreadPool();
for(//iterate for number of files) {
while(data exists in file) {
//Call a function to process and update values in db
future.add(threadPool.submit(new Xyz(b1)));
//read next set of data in file;
}
}
try {
for(Future f: future) {
f.get();
}
}
catch(Exception e) {
throw e;
}
}
}
class Xyz implements Runnable {
private Adapter a1;
public Xyz(Adapter al) {
this.a1=a1;
}
#Override
public void run() {
try {
a1.abc();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}
When the number of files is 1 (for loop runs for 1 time), the code runs fine.
But, when the number of files increases, the code never returns back from future.get() method.
just out of curiosity.. do i need to shutdown the executor somewhere ??
Yes, and this is likely the problem. Each Future.get() will block until the corresponding task is complete, then once all the tasks are complete your main thread will exit. But your java process will not exit because the thread pool threads are still active in the background. You should shut down the executor once you have finished with it, most likely as the last thing in your main method.
I also note that you're submitting many tasks that wrap the same Adapter instance and all call its abc() method - check that there's nothing in there that will deadlock when called simultaneously in more than one thread.
Your Callable::call / Runable::run does not return. Otherwise the corresponding future would not block.
Additional executor.shutdown or future.cancel will thow an InterruptedException to stop the thread processing the object you submitted but it is up to you if to catch it or not. Your are responsible for making the jobs you submitted stop.
When you submit thousands Callables/Runnables to a CachedExecutor that it might spawn so many threads that your machine gets so slow that you think it takes forever. But you would have noticed that.
When dealing with an undefined number of parallelizable tasks i suggest to use a FixedThreadPool with not much more threads that there are cpu cores.
Edit: Therefore when you set a breakpoints at a1.abc(); and step forward you will probably find out that it never returns.
I have a method in java which calls webservice. Now if the webservice is taking more than 1 sec. of time. I should be able to kill the method and continue in the program flow.
Google Guava's SimpleTimeLimiter will do what you need, specifically its callWithTimeout() method.
I would recommend you to use the Executors framework and call Future.get(long, TimeUnit). As for killing that call, you can call Future.cancel(true). Of course, you'll have to submit a Callable that contains the call.
In case you don't want to add a new library, you can easily accomplish this by delegating the Web Service invocation to another Thread, and join on it using a timeout of 1 sec. Below is a complete program which does these tasks. There are 2 key points:
Use Asynchronous Thread to do the call and join on it with TimeOut. You can additionally set it as Daemon.
Convey to the Asynchronous thread when the value from operation doesn't need to be consumed, so that it doesn't make unnecessary assignments.
Code:
public class Main {
String returnVar;
private static final long TIME_OUT=1000;//in mills
private void makeCall() {
WebServiceStubDummy ws = new WebServiceStubDummy();
Boolean timeElapsed = false;
Thread t = new Thread(new AsyncWSCall(ws,timeElapsed));
t.start();
try {
t.join(TIME_OUT);
} catch (InterruptedException e) {}
synchronized (ws) {
timeElapsed=true;
}
System.out.println(returnVar);
}
private class AsyncWSCall implements Runnable{
WebServiceStubDummy ws;
Boolean timeElapsed;
public AsyncWSCall(WebServiceStubDummy ws, Boolean timeElapsed){
this.ws=ws;
this.timeElapsed=timeElapsed;
}
#Override
public void run() {
String myStr = ws.dummyMethod();
//synchronize for shared variable timeElapsed
synchronized (ws) {
if(!timeElapsed){
//if time elapsed don't assign
returnVar=myStr;
}
}
}
}
class WebServiceStubDummy{
public String dummyMethod(){
try {
//Dummy Call: if changed to 2000 value will not be consumed
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
return "From Dummy Metho";
}
}
/**
* #param args
*/
public static void main(String[] args) {
Main m=new Main();
m.makeCall();
}}
You can additionally fine tune your code as per how you wish to pass the WebService object and how you want to assign the result of WS operation
Use a timeout of 1 sec. on your webservice request.
The SO-thread "Killing thread after some specified time limit in java" could help. It makes use of ExecutorService
Time Limitation on method execution in java could also be used. Keep in mind though that Thread.stop is deprecated:
Why is Thread.stop deprecated?
Because it is inherently unsafe. Stopping a thread causes it to unlock
all the monitors that it has locked. (The monitors are unlocked as the
ThreadDeath exception propagates up the stack.) If any of the objects
previously protected by these monitors were in an inconsistent state,
other threads may now view these objects in an inconsistent state.
Such objects are said to be damaged. When threads operate on damaged
objects, arbitrary behavior can result. This behavior may be subtle
and difficult to detect, or it may be pronounced. Unlike other
unchecked exceptions, ThreadDeath kills threads silently; thus, the
user has no warning that his program may be corrupted. The corruption
can manifest itself at any time after the actual damage occurs, even
hours or days in the future.
source: Java Thread Primitive Deprecation
I'm working on a Java project where I need to have multiple tasks running asynchronously. I'm led to believe Executor is the best way for me to do this, so I'm familiarizing myself with it. (Yay getting paid to learn!) However, it's not clear to me what the best way is to accomplish what I'm trying to do.
For the sake of argument, let's say I have two tasks running. Neither is expected to terminate, and both should run for the duration of the application's life. I'm trying to write a main wrapper class such that:
If either task throws an exception, the wrapper will catch it and restart the task.
If either task runs to completion, the wrapper will notice and restart the task.
Now, it should be noted that the implementation for both tasks will wrap the code in run() in an infinite loop that will never run to completion, with a try/catch block that should handle all runtime exceptions without disrupting the loop. I'm trying to add another layer of certainty; if either I or somebody who follows me does something stupid that defeats these safeguards and halts the task, the application needs to react appropriately.
Is there a best practice for approaching this problem that folks more experienced than me would recommend?
FWIW, I've whipped-up this test class:
public class ExecTest {
private static ExecutorService executor = null;
private static Future results1 = null;
private static Future results2 = null;
public static void main(String[] args) {
executor = Executors.newFixedThreadPool(2);
while(true) {
try {
checkTasks();
Thread.sleep(1000);
}
catch (Exception e) {
System.err.println("Caught exception: " + e.getMessage());
}
}
}
private static void checkTasks() throws Exception{
if (results1 == null || results1.isDone() || results1.isCancelled()) {
results1 = executor.submit(new Test1());
}
if (results2 == null || results2.isDone() || results2.isCancelled()) {
results2 = executor.submit(new Test2());
}
}
}
class Test1 implements Runnable {
public void run() {
while(true) {
System.out.println("I'm test class 1");
try {Thread.sleep(1000);} catch (Exception e) {}
}
}
}
class Test2 implements Runnable {
public void run() {
while(true) {
System.out.println("I'm test class 2");
try {Thread.sleep(1000);} catch (Exception e) {}
}
}
}
It's behaving the way I want, but I don't know if there are any gotchas, inefficiencies, or downright wrong-headedness waiting to surprise me. (In fact, given that I'm new to this, I'd be shocked if there wasn't something wrong/inadvisable about it.)
Any insight is welcomed.
I faced a similar situation in my previous project, and after my code blew in the face of an angry customer, my buddies and I added two big safe-guards:
In the infinite loop, catch Errors too, not just exceptions. Sometimes unexcepted things happen and Java throws an Error at you, not an Exception.
Use a back-off switch, so if something goes wrong and is non-recoverable, you don't escalate the situation by eagerly starting another loop. Instead, you need to wait until the situation goes back to normal and then start again.
For example, we had a situation where the database went down and during the loop an SQLException was thrown. The unfortunate result was that the code went through the loop again, only to hit the same exception again, and so forth. The logs showed that we hit the same SQLException about 300 times in a second!! ... this happened intermittently several times with occassional JVM pauses of 5 seconds or so, during which the application was not responsive, until eventually an Error was thrown and the thread died!
So we implemented a back-off strategy, approximately shown in the code below, that if the exception is not recoverable (or is excepted to recover within a matter of minutes), then we wait for a longer time before resuming operations.
class Test1 implements Runnable {
public void run() {
boolean backoff = false;
while(true) {
if (backoff) {
Thread.sleep (TIME_FOR_LONGER_BREAK);
backoff = false;
}
System.out.println("I'm test class 1");
try {
// do important stuff here, use database and other critical resources
}
catch (SqlException se) {
// code to delay the next loop
backoff = true;
}
catch (Exception e) {
}
catch (Throwable t) {
}
}
}
}
If you implement your tasks this way then I don't see a point in having a third "watch-dog" thread with the checkTasks() method. Furthermore, for the same reasons I outlined above, I'd be cautious to just start the task again with the executor. First you need to understand why the task failed and whether the environment is in a stable condition that running the task again would be useful.
Aside to eyeballing it, I generally run Java code against static analysis tools like PMD and FindBugs to look for deeper issues.
Specifically for this code FindBugs didn't like that results1 and results2 are not volatile in the lazy init, and that the run() methods might ignore the Exception because they aren't explicitly being handled.
In general I am a bit leery of the use of Thread.sleep for concurrency testing, preferring timers or terminating states/conditions. Callable might be useful in returning something in the event of a disruption that throws an exception if unable to compute a result.
For some best practices and more food for thought, check out Concurrency in Practice.
how about this
Runnable task = () -> {
try{
// do the task steps here
} catch (Exception e){
Thread.sleep (TIME_FOR_LONGER_BREAK);
}
};
ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor();
executor.scheduleAtFixedRate(task,0, 0,TimeUnit.SECONDS);
have you tried Quartz framework ?