I have around ~20 scheduled methods for automation. They all run fine but they all stop after a few hours.
My logs don't show any errors/server crash, it's just as if spring boot decided to not do any #scheduled methods anymore.
My first intuition was that there was probably infinite loop in the body of a method, however, all my methods have loggers at the start and at the end. i.e. if there we an infinite loop, my final log wouldn't be saying [foo finished successfully].
I even created a tester that just prints every 5 minutes, and that function also stopped, with all the other ones, after a few hours.
My second intuition was to check the file size, since maybe the file size was too big and the logger just stopped logging into the file, and somehow this made the automation stop (scraping the barrel at this point), but since the automation only ran for a few hours, the file size is only ~1200kb, so this was not the issue.
Basically, I don't think there's an infinite loop somewhere because of the way my loggers are set up, I'm not getting any error messages in my logs and I don't know how to debug this.
I tried to include as much useful information, if something is not clear/missing, please let me know.
Other than that, any ideas on how to debug or what could be causing this?
The problem I was encountering was quite unique, posting answer in case it helps someone in the future.
Loggers stopped writing to file even though they weren't meant to. (Another issue I'll fix eventually) Which meant that even if the last log said [foo finished successfully], it was necessarily the last real log of the application.
Since Scheduling on Spring is single thread by default, there was a poorly optimized method call that would take ~12h to complete and this would make it seem as the automation stopped since the logs weren't updating and automation elsewhere wasn't happening.
I would never let this ~12h method call finish, had I waited the 12h I would've realized that the automation wasn't stopped, it simply had a bottleneck method. I would always restart the automation before this method could finish and this made it look as if the automation had indeed stopped for an unknown reason.
How I found out : In my case, my application runs in a container and once it runs I just CTRL + Z.
I had a feeling that the loggers weren't working properly and so once the logger file stopped updating, so I decided to check the live logging of the application instead by typing fg and realized that even though the log file wasn't updating, the server was still running fine.
Related
I've created a simple web application on the base of spring-boot. It gets request, do some block operations (like writing to the console numbers from 1 to 10_000) and returns simple response.
When I start to test it with JVisual VM I've noticed that threads work in a strange way.
In the beginning, with low load, they don't work at the same time, they work in turn:
If I try to high-load application than Tomcat creates new Threads (It's OK as I understand) and after time threads work at the same time:
I've used Netling app to test application. It just simple request on localhost, nothing special.
Could you please explain why it works in this way? Or provide some links with description.
Thanks in advance!
I've understood the root of the issue.
On the JVisualVM I saw, that all Threads was blocked by each other, but I didn't know why. Now it's clear - I've use system.out.println() as a block operation, but Threads can't use it at the same time, the console is locked.
I've change console output to file output (use thread name as a filename to prevent lock) and start the application again. Now it's work as expected.
Stupid mistake, but I hope it will be useful for someone :)
I have a browser game built on a java web server using jsp.
I added a new module that uses some http session object and keep data in it. However, after it runs 3-4 hours, it suddenly stops working and freezes. When I check the error log, I dont see any exception thrown.
The server has 50-60 online in a moment.
I monitored the server using visualVM and here is the result after 4 hours until it stops :
I set the max memory as 1024Mb. As you can see its problem is not about the memory.
The thing that I notice is when the server stops, the thread amount increased.
According to the screenshot, should I doubt the httpsession object ? Why does the server stop responding ??
It looks like a system limitation or a deadlock.
Your thread graph looks like problematic : the number of living thread is important and never decreases. A web application should be stateless. The living tread count should rises when the requests arrive but also drops when the requests are finished.
I have not the impression it is the case in your application.
MGorgon is right.
You should also check "Deadlock detection" in jconsole.
If you use a JDK 6+ version, you could use ThreadMXBean. It has a findDeadlockedThreads() methods and other interesting methods to address your need.
Anyway, if it is not a deadlock, to get more information about the cause of the problem, I advise you to look in the system log whatever you OS is. You would have maybe interesting things.
Usually, when I hit the stop button from within Eclipse on the embedded Tomcat instance, the current console is swapped with a new, empty, console window. The former console window is still available though, and if I switch back to it I can see the Tomcat shutdown sequence (with all the messages, warning and such).
But it happens to me, at times, that when I hit the stop button, the server is in fact stopped, but the shutdown process isn't visible — i.e. nothing gets written to the console. The console stays on the last application message.
The Tomcat instance is actually stopped almost instantaneously.
Why is this happening? Am I doing something wrong that can happen in production too, or is this just a quirk in the Eclipse Tomcat plugin?
I never saw such behaviour until a couple of days ago, when I started hunting down never ending threads. I'm putting togheter a simple cleaning strategy — I basically trace the creation of each component that makes use of threads in a singleton, and I call any interrupt(), shutdown() or cleanup() method those components expose in a ServletContextListener.
I just discovered that when the shutdown process is correctly called, on the debug pane it says:
<terminated, exit value: 0>\path\to\java_home\bin\javaw.exe
when, instead, the server is suddenly halted, I see:
<terminated, exit value: 1>\path\to\java_home\bin\javaw.exe
I have scenario where I am scheduling task at fixed duration repetitively.Fixed delay is generated by calling start method of another class that implements runnable interface using Thread.sleep(long ms ) method.
But when I test this application in my local pc it is working.But when I run this application in ibm blade server(64 bit) having OS(Windows server 2008 R2) it do not work as desired. It do not come out of sleep method.
Kindly suggest the solution?
Thank You in Advance.
There is not much information in your question to see what the problem is. Thread.sleep should either return or throw an exception. Maybe something different is happening. For example, an exception has occurred, caught and forgotten, or you have a deadlock somewhere. Anyway, different versions of Java sometimes have subtle differences causing bugs. You will have to investigate the problem yourself.
Try debugging the application. When it hangs, press Pause and examine all threads to find the hanging one.
If you can't install a debugger on the server, add System.out.println in every reasonable place of the code; reading the output in the console, you will probably be able to track down the issue.
If you can't launch the application with console, create a text file and write the messages to it. Don't forget to flush it every time.
I'm seeing strange behavior and I don't know how to gain any further insight into and am hoping someone can help.
Background: I have a query that takes a long time to return results so instead of making the user wait for the data directly upon request I execute this query via a Timer object at regular intervals and store the results in a static variable. Therefore, when the user requests the data I always just pull from the static variable, therefore making the response virtually instant. So far so good.
Issue: The behavior I'm seeing, however, is that if I make a request for the data just as the background (Timer) request has begun to query the data, my user's request waits for the data to come back before responding -- forcing the user to wait. It's as if tomcat is behaving synchronously with the threads (I know it's not -- it just looks that way).
This is in a Production environment and, for the most part, everything works great but for users there are times when the site just hangs for them and they feel it's unreliable (well, in a sense it is).
What I've done: Being that the requests for the data were in a static method I thought "A ha! The threads are syncronized which is causing the delay!" so i pulled all of my static methods out, removed the syncronization and forced each call to instantiate it's own object to retrieve the data (to keep it thread safe). There isn't any syncronization on a semaphore to the static variable either.
I've also installed javamelody to try and gain some insight into what's going on but nothing new thus far. I have noticed a lot (majority) of threads are in "WAITING" state but they also have 0ms for User and CPU time so don't think that is pointing to anything(?).
Running Tomcat 5.5 (no apache layer), struts 2, Java 1.5
If anyone has any idea why a simple request to a static variable hangs for longer background processes I would really appreciate it! Or if you know how I can gain insight that would be great too.
Thanks!
One possible explanation is that the threads are actually blocking at the database level due to database locking (or something) caused by the long-running query.
The way to figure out what is going on is to find out exactly where the blocked threads are blocking. A thread dump can be produced by sending a SIGQUIT (or equivalent) to the JVM, and included stack traces for all Java thread stacks. Alternatively, you can get the same information (and more) by attaching a debugger, etcetera. Either way, the class name and line number of the top frame of each stack should allow you to look at the source code and figure out (at least) what kind of locking or blocking is going on.
For those who would like to know I eventually found VisualVM (http://visualvm.java.net/download.html). It's perfect. I run Tomcat from eclipse like I normally do and it appears within the VisualVM client. Right-mouse click the tomcat icon, choose Thread Dump and, boom, I've got it all.
Thanks, all, for the help and pointers towards the right direction!