I have created a service using procrun which launches certain jars through reflection. When the service is started it starts a thread and rest of the execution happens in that thread. Then each of the plugin loads its own threads and does the execution in there.
During service stop, I have called the stop method of the plugins. Those methods have returned and whatever thread I have created has been terminated for the plugins. But even after that the following threads are still running.
INFO: Thread No:0 = Timer-0
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
INFO: Thread No:1 = WebSocketWorker-14
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
INFO: Thread No:2 = WebSocketWorker-15
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
INFO: Thread No:3 = WebSocketWorker-16
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
INFO: Thread No:4 = WebSocketWorker-17
Jan 13, 2016 10:49:58 AM com.kube.desktop.KubeSdkMain stop
INFO: Thread No:5 = WebsocketSelector18
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
INFO: Thread No:6 = AWT-EventQueue-0
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
INFO: Thread No:7 = DestroyJavaVM
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
INFO: Thread No:8 = Thread-11
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
The following is how I printed those threads.
ThreadGroup currentGroup = Thread.currentThread().getThreadGroup();
int noThreads = currentGroup.activeCount();
Thread[] lstThreads = new Thread[noThreads];
currentGroup.enumerate(lstThreads);
for (int i = 0; i < noThreads; i++)
LOGGER.log(Level.INFO, "Thread No:" + i + " = " + lstThreads[i].getName());
Because of these threads, when I stop the service, it takes forever and then times out. But when I call System.exit(0) the service stops quickly. What should I do to get rid of these threads? When I launch the jars through reflection, are there separate threads created for each plugin? If so could these be them? Please advice.
It looks like the plugins are themself launching threads ("INFO: Thread No:1 = WebSocketWorker-14" -> sockets usually should be put in seperate threads) which will not be shut down if you kill the initiating thread. You'll have to enforce your plugins to kill all threads they started when they get shut down to make sure that they will not leave stuff behind. "some plugins don't do a good job cleaning up whatever they created. it's just sloppy programming." - bayou.io is describing it really good there.
Calling System.exit() will just kill the process meaning it will kill all threads created by the process as well.
The other way would be to manually iterate over all running threads, check if it's the main thread, and if not proceed to kill it. You can get all running threads in an iterable set using
Set<Thread> threadSet = Thread.getAllStackTraces().keySet();
And you can get your currently running Thread using
Thread currentThread = Thread.currentThread();
Still this is the way you would not want to do it, it's more of a way to clean up if plugins decide to leave stuff behind rather than just doing it that way. The plugins themselfes should take care of shutting down the threads when they get disabled but if they don't do that you can use above way to manually clean it up.
Related
I wanted to use custom ThreadPool for parallelStream. Reason being I wanted to use MDCContext in the task. This is the code I wrote to use the custom ThreadPool:
final ExecutorService mdcPool = MDCExecutors.newCachedThreadPool();
mdcPool.submit(() -> ruleset.getOperationList().parallelStream().forEach(operation -> {
log.info("Sample log line");
});
When the MDC context was not getting copied to the task, I looked at the logs. These are the logs I found. The first log is executed in "(pool-16-thread-1)" but other tasks are getting executed on "ForkJoinPool.commonPool-worker". The first log also has MdcContextID. But as I am using custom ThreadPool for submitting the task, all tasks should be executing in custom ThreadPool.
16 Oct 2018 12:46:58,298 [INFO] 8fcfa6ee-d141-11e8-b84a-7da6cd73aa0b (pool-16-thread-1) com.amazon.rss.activity.business.VariablesEvaluator: Sample log line
16 Oct 2018 12:46:58,298 [INFO] (ForkJoinPool.commonPool-worker-11) com.amazon.rss.activity.business.VariablesEvaluator: Sample log line
16 Oct 2018 12:46:58,298 [INFO] (ForkJoinPool.commonPool-worker-4) com.amazon.rss.activity.business.VariablesEvaluator: Sample log line
16 Oct 2018 12:46:58,298 [INFO] (ForkJoinPool.commonPool-worker-13) com.amazon.rss.activity.business.VariablesEvaluator: Sample log line
16 Oct 2018 12:46:58,298 [INFO] (ForkJoinPool.commonPool-worker-9) com.amazon.rss.activity.business.VariablesEvaluator: Sample log line
16 Oct 2018 12:46:58,299 [INFO] (ForkJoinPool.commonPool-worker-2) com.amazon.rss.activity.business.VariablesEvaluator: Sample log line
16 Oct 2018 12:46:58,299 [INFO] (ForkJoinPool.commonPool-worker-15) com.amazon.rss.activity.business.VariablesEvaluator: Sample log line
Is this supposed to happen or am I missing something?
There is no support for running a parallel stream in a custom thread pool. It happens to be executed in a different Fork/Join pool when the operation is initiated in a worker thread of a different Fork/Join pool, but that does not seem to be a planned feature, as the Stream implementation code will still use artifacts of the common pool internally in some operations then.
In your case, it seems that the ExecutorService returned by MDCExecutors.newCachedThreadPool() is not a Fork/Join pool, so it does not exhibit this undocumented behavior at all.
There is a feature request, JDK-8032512, regarding more thread control. It’s open and, as far as I can see, without much activity.
I'm getting the exception when I try to connect to the orient db using Java. Below is the exception I'm getting.
Jun 07, 2016 12:43:40 PM com.orientechnologies.common.log.OLogManager log
INFO: OrientDB auto-config DISKCACHE=891MB (heap=891MB direct=891MB os=4,006MB), assuming maximum direct memory size equals to maximum JVM heap size
Jun 07, 2016 12:43:40 PM com.orientechnologies.common.log.OLogManager log
WARNING: MaxDirectMemorySize JVM option is not set or has invalid value, that may cause out of memory errors. Please set the -XX:MaxDirectMemorySize=4006m option when you start the JVM.
Jun 07, 2016 12:43:40 PM com.orientechnologies.common.log.OLogManager log
WARNING: MaxDirectMemorySize JVM option is not set or has invalid value, that may cause out of memory errors. Please set the -XX:MaxDirectMemorySize=4006m option when you start the JVM.
Exception in thread "main" com.orientechnologies.orient.core.exception.OFileLockedByAnotherProcessException: File 'F:\Program Files\orientdb-community-2.2.0\databases\mydbo\database.ocf' is locked by another process, maybe the database is in use by another process. Use the remote mode with a OrientDB server to allow multiple access to the same database at com.orientechnologies.orient.core.storage.fs.OFileClassic.lock(OFileClassic.java:756)
at com.orientechnologies.orient.core.storage.fs.OFileClassic.openChannel(OFileClassic.java:813)
at com.orientechnologies.orient.core.storage.fs.OFileClassic.open(OFileClassic.java:603)
at com.orientechnologies.orient.core.storage.impl.local.OSingleFileSegment.open(OSingleFileSegment.java:51)
at com.orientechnologies.orient.core.storage.impl.local.OStorageConfigurationSegment.load(OStorageConfigurationSegment.java:80)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:186)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:231)
at orient.insert.Insert.main(Insert.java:12)
this is the code that i tried.
ODatabaseDocumentTx db = new ODatabaseDocumentTx("plocal:F:/Program Files/orientdb-community-2.2.0/databases/mydbo").open("admin", "admin");
ODocument doc = new ODocument("Person");
doc.field( "name", "Luke" );
doc.field( "surname", "Skywalker" );
doc.field( "city", new ODocument("City").field("name","Rome").field("country", "Italy") );
doc.save();
db.close();
I can't figure out the error I'm having.
You have a server running and you try to open the database from another process in plocal.
Could you please verify that you have no active OrientDB instances while accessing it in plocal (console or external processes) and that you open one plocal connection at a time?
By default, the servlet container loads only a single instance of a
servlet. Requests serviced by the servlet are run in separate threads,
but share the same instance. This enables applications to be
responsive and scalable, since it requires fewer resources and uses
them efficiently.
Almost everyone knows that this is the default threading model.
This question/discussion is a follow-up on this particular stackoverflow thread.
But the output happens to be completely different in my case,
May 21, 2016 1:57:59 PM com.stackoverflow.question8011138.MyServlet doGet
INFO: [doGet] Test before // 1st thread
May 21, 2016 1:58:26 PM com.stackoverflow.question8011138.MyServlet doGet
INFO: [doGet] Test before // 2nd thread
May 21, 2016 1:58:59 PM com.stackoverflow.question8011138.MyServlet doGet
INFO: [doGet] Test after // 1st thread
May 21, 2016 1:59:26 PM com.stackoverflow.question8011138.MyServlet doGet
INFO: [doGet] Test after // 2nd thread
Even though I hit the url mapped to this particlular servlet in 2 browser windows of the same browser, the difference b/w these 2 hitting ~1 second, but I see the below output
May 21, 2016 1:58:26 PM com.stackoverflow.question8011138.MyServlet doGet
INFO: [doGet] Test before // 2nd thread
after seconds more than that previously mentioned ~1 second,
i.e to this,
May 21, 2016 1:57:59 PM com.stackoverflow.question8011138.MyServlet doGet
INFO: [doGet] Test before // 1st thread
Why?
Clearly, from the output above, the second request is not served after the first has been served completely, but in b/w, contrary to what the OP to that question suggested.
What exactly is going on?
Why 2 different outputs to the same scenario?
The result/ output clearly shows that the server is not synchronizing the 2 requests.
I can't find a proper way to handle spring-batch exception in asynchronous context.
When I set a ThreadPoolTaskManager to my JobLauncher, the real job/step exception is not logged anymore. Instead the log will be something like:
org.springframework.batch.core.JobInterruptedException: Job interrupted by step execution
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:165)
at ...
I tried to resolve this adding a JobExecutionListener like this:
#Override
public void afterJob(JobExecution jobExecution) {
List<Throwable> jobExceptions = jobExecution.getFailureExceptions();
if (CollectionUtils.isNotEmpty(jobExceptions)) {
Throwable lastJobException = jobExceptions.get(jobExceptions.size() - 1);
LOGGER.error("Spring-Batch error at job level", lastJobException);
String lastJobExceptionMessage = ExceptionUtils.getRootCauseMessage(lastJobException);
// storing message in ExecutionContext for the batch-admin webapp
String message = "";
if (jobExecution.getExecutionContext().get(Consts.JOB_EXECUTION_MESSAGE_KEY) != null) {
message = jobExecution.getExecutionContext().getString(Consts.JOB_EXECUTION_MESSAGE_KEY);
}
message += "\n" + lastJobExceptionMessage;
jobExecution.getExecutionContext().put(Consts.JOB_EXECUTION_MESSAGE_KEY, message);
}
}
But I still end with a JobInterruptedException.
Is there a way to retrieve the initial cause of the interruption (might be a error in the reader/processor/writer code?
I don't think your diagnosis is correct. That exception is thrown with that error message only in SimpleStepHandler:
if (currentStepExecution.getStatus() == BatchStatus.STOPPING
|| currentStepExecution.getStatus() == BatchStatus.STOPPED) {
// Ensure that the job gets the message that it is stopping
execution.setStatus(BatchStatus.STOPPING);
throw new JobInterruptedException("Job interrupted by step execution");
}
and only if the step itself didn't throw JobInterruptedException. The most obvious case where this can happen is if the job was stopped. See this example, whose output ends with
INFO: Executing step: [step1]
Feb 24, 2016 1:25:02 PM org.springframework.batch.core.repository.support.SimpleJobRepository checkForInterruption
INFO: Parent JobExecution is stopped, so passing message on to StepExecution
Feb 24, 2016 1:25:02 PM org.springframework.batch.core.step.ThreadStepInterruptionPolicy isInterrupted
INFO: Step interrupted through StepExecution
Feb 24, 2016 1:25:02 PM org.springframework.batch.core.step.AbstractStep execute
INFO: Encountered interruption executing step step1 in job myJob : Job interrupted status detected.
Feb 24, 2016 1:25:02 PM org.springframework.batch.core.repository.support.SimpleJobRepository checkForInterruption
INFO: Parent JobExecution is stopped, so passing message on to StepExecution
Feb 24, 2016 1:25:02 PM org.springframework.batch.core.job.AbstractJob execute
INFO: Encountered interruption executing job: Job interrupted by step execution
Feb 24, 2016 1:25:02 PM org.springframework.batch.core.launch.support.SimpleJobLauncher$1 run
INFO: Job: [SimpleJob: [name=myJob]] completed with the following parameters: [{}] and the following status: [STOPPED]
Status is: STOPPED
This other example shows that throwing an exception when using a thread pool changes nothing. The final output is
INFO: Executing step: [step1]
Feb 24, 2016 1:28:44 PM org.springframework.batch.core.step.AbstractStep execute
SEVERE: Encountered an error executing step step1 in job myJob
java.lang.RuntimeException: My exception
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
(...)
Feb 24, 2016 1:28:44 PM org.springframework.batch.core.launch.support.SimpleJobLauncher$1 run
INFO: Job: [SimpleJob: [name=myJob]] completed with the following parameters: [{}] and the following status: [FAILED]
Status is: FAILED, job execution id 0
#1 step1 FAILED
Step step1
java.lang.RuntimeException: My exception
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
(...)
So the answer was so simple than I felt really stupid when I understood it :
#Artefacto was right. The job was stopped. By the end of the process. Because it reached the end of the main() method.
When I switch to an asynchronous mode with my ThreadPoolTaskManager, I forgot to add one very important line to my main method:
// Wait for the end of the JobExecution
main.endOfJobLatch.await();
Hope this answer will help someone else...
I am using Restlet to implement a web service. From the PHP Client making many consecutive calls to the server, but after a small number of calls complete successfully, further calls hang the server, which shows the message:
INFO: Worker service state: Full
Jun 22, 2015 2:38:31 PM org.restlet.engine.http.connector.BaseHelper$1 rejectedExecution
INFO: Worker service tasks: 0 queued, 10 active, 16 completed, 26 scheduled.
Jun 22, 2015 2:38:31 PM org.restlet.engine.http.connector.BaseHelper$1 rejectedExecution
INFO: Worker service thread pool: 1 core size, 10 largest size, 10 maximum size, 10 current size
Jun 22, 2015 2:38:31 PM org.restlet.engine.http.connector.Controller run
INFO: Stop accepting new connections and transactions. Consider increasing the maximum number of threads.
Please find below the restlet server component code base:
final Component serverComponent = new Component();
serverComponent.getServers().add(Protocol.HTTP, 8182);
final Series<Parameter> parameters = serverComponent.getContext().getParameters();
parameters.add("maxThreads", "150");
parameters.add("minThreads", "10");
parameters.add("lowThreads", "145");
parameters.add("maxQueued", "20");
parameters.add("maxTotalConnections", "10000");
serverComponent.getDefaultHost().attach("/v1", new Appv1());
try {
serverComponent.start();
} catch (final Exception e) {
LOGGERS.error("Exception while starting the application: " + e.getMessage());
}
Can anybody know what changes I have missed in the setup ?
I suggest you to use the jetty extension of the framework, instead of the internal connector.
If you want to do so, just add the org.restlet.ext.jetty.jar and org.eclipse.jetty.jar to your classpath.