I've created a simple web application on the base of spring-boot. It gets request, do some block operations (like writing to the console numbers from 1 to 10_000) and returns simple response.
When I start to test it with JVisual VM I've noticed that threads work in a strange way.
In the beginning, with low load, they don't work at the same time, they work in turn:
If I try to high-load application than Tomcat creates new Threads (It's OK as I understand) and after time threads work at the same time:
I've used Netling app to test application. It just simple request on localhost, nothing special.
Could you please explain why it works in this way? Or provide some links with description.
Thanks in advance!
I've understood the root of the issue.
On the JVisualVM I saw, that all Threads was blocked by each other, but I didn't know why. Now it's clear - I've use system.out.println() as a block operation, but Threads can't use it at the same time, the console is locked.
I've change console output to file output (use thread name as a filename to prevent lock) and start the application again. Now it's work as expected.
Stupid mistake, but I hope it will be useful for someone :)
Related
I have around ~20 scheduled methods for automation. They all run fine but they all stop after a few hours.
My logs don't show any errors/server crash, it's just as if spring boot decided to not do any #scheduled methods anymore.
My first intuition was that there was probably infinite loop in the body of a method, however, all my methods have loggers at the start and at the end. i.e. if there we an infinite loop, my final log wouldn't be saying [foo finished successfully].
I even created a tester that just prints every 5 minutes, and that function also stopped, with all the other ones, after a few hours.
My second intuition was to check the file size, since maybe the file size was too big and the logger just stopped logging into the file, and somehow this made the automation stop (scraping the barrel at this point), but since the automation only ran for a few hours, the file size is only ~1200kb, so this was not the issue.
Basically, I don't think there's an infinite loop somewhere because of the way my loggers are set up, I'm not getting any error messages in my logs and I don't know how to debug this.
I tried to include as much useful information, if something is not clear/missing, please let me know.
Other than that, any ideas on how to debug or what could be causing this?
The problem I was encountering was quite unique, posting answer in case it helps someone in the future.
Loggers stopped writing to file even though they weren't meant to. (Another issue I'll fix eventually) Which meant that even if the last log said [foo finished successfully], it was necessarily the last real log of the application.
Since Scheduling on Spring is single thread by default, there was a poorly optimized method call that would take ~12h to complete and this would make it seem as the automation stopped since the logs weren't updating and automation elsewhere wasn't happening.
I would never let this ~12h method call finish, had I waited the 12h I would've realized that the automation wasn't stopped, it simply had a bottleneck method. I would always restart the automation before this method could finish and this made it look as if the automation had indeed stopped for an unknown reason.
How I found out : In my case, my application runs in a container and once it runs I just CTRL + Z.
I had a feeling that the loggers weren't working properly and so once the logger file stopped updating, so I decided to check the live logging of the application instead by typing fg and realized that even though the log file wasn't updating, the server was still running fine.
I am writing a HTTP THREAD-POOLED WEB SERVER code. I made my code at my best. It works fine also.
But there is one problem, whenever I want to shut down my server, I have to use CTRL + C. But that is a bad way to shut down.
I think a lot, to add shutdown feature in my code. But I am unable to do that.
Please help me to add this feature, give any suggestion to do that, I will definitely code that.
EDIT NO. 1
One method that I think is to make one thread that is only listening STDIN input given by keyboard. Whenever it gets "SHUTDOWN". It calls ThreadPol.shutdown(). This way I can achieve this goal.
Is this a right method ? If yes, please help me to implement this.
You can do
ThreadPoolExecuto.shutdownNow()
Attempts to stop all actively executing tasks, halts the processing of waiting tasks, and returns a list of the tasks that were awaiting execution.
I have scenario where I am scheduling task at fixed duration repetitively.Fixed delay is generated by calling start method of another class that implements runnable interface using Thread.sleep(long ms ) method.
But when I test this application in my local pc it is working.But when I run this application in ibm blade server(64 bit) having OS(Windows server 2008 R2) it do not work as desired. It do not come out of sleep method.
Kindly suggest the solution?
Thank You in Advance.
There is not much information in your question to see what the problem is. Thread.sleep should either return or throw an exception. Maybe something different is happening. For example, an exception has occurred, caught and forgotten, or you have a deadlock somewhere. Anyway, different versions of Java sometimes have subtle differences causing bugs. You will have to investigate the problem yourself.
Try debugging the application. When it hangs, press Pause and examine all threads to find the hanging one.
If you can't install a debugger on the server, add System.out.println in every reasonable place of the code; reading the output in the console, you will probably be able to track down the issue.
If you can't launch the application with console, create a text file and write the messages to it. Don't forget to flush it every time.
I'm seeing strange behavior and I don't know how to gain any further insight into and am hoping someone can help.
Background: I have a query that takes a long time to return results so instead of making the user wait for the data directly upon request I execute this query via a Timer object at regular intervals and store the results in a static variable. Therefore, when the user requests the data I always just pull from the static variable, therefore making the response virtually instant. So far so good.
Issue: The behavior I'm seeing, however, is that if I make a request for the data just as the background (Timer) request has begun to query the data, my user's request waits for the data to come back before responding -- forcing the user to wait. It's as if tomcat is behaving synchronously with the threads (I know it's not -- it just looks that way).
This is in a Production environment and, for the most part, everything works great but for users there are times when the site just hangs for them and they feel it's unreliable (well, in a sense it is).
What I've done: Being that the requests for the data were in a static method I thought "A ha! The threads are syncronized which is causing the delay!" so i pulled all of my static methods out, removed the syncronization and forced each call to instantiate it's own object to retrieve the data (to keep it thread safe). There isn't any syncronization on a semaphore to the static variable either.
I've also installed javamelody to try and gain some insight into what's going on but nothing new thus far. I have noticed a lot (majority) of threads are in "WAITING" state but they also have 0ms for User and CPU time so don't think that is pointing to anything(?).
Running Tomcat 5.5 (no apache layer), struts 2, Java 1.5
If anyone has any idea why a simple request to a static variable hangs for longer background processes I would really appreciate it! Or if you know how I can gain insight that would be great too.
Thanks!
One possible explanation is that the threads are actually blocking at the database level due to database locking (or something) caused by the long-running query.
The way to figure out what is going on is to find out exactly where the blocked threads are blocking. A thread dump can be produced by sending a SIGQUIT (or equivalent) to the JVM, and included stack traces for all Java thread stacks. Alternatively, you can get the same information (and more) by attaching a debugger, etcetera. Either way, the class name and line number of the top frame of each stack should allow you to look at the source code and figure out (at least) what kind of locking or blocking is going on.
For those who would like to know I eventually found VisualVM (http://visualvm.java.net/download.html). It's perfect. I run Tomcat from eclipse like I normally do and it appears within the VisualVM client. Right-mouse click the tomcat icon, choose Thread Dump and, boom, I've got it all.
Thanks, all, for the help and pointers towards the right direction!
We have a Java App that connects via RMI to another Java app.
There are multiple instances of this app running at the same time, and after a few days an instance just stops processing... the CPU is in 0 and I have an extra thread listening to an specific port that helps to shutdown the App.
I can connect to the specific port but the app doesn't do anything.
We're using Log4j to keep a log and nothing is written, so there aren't any exceptions thrown.
We also use c3p0 for the DB connections.
Anyone have ideas?
Thanks,
I would suggest starting with a thread dump of the affected application.
You need to see what is going on on a thread by thread basis. It could be that you have a long running thread, or other process which is blocking other work from being done.
Since you are running linux, you can get your thread dump with the following command
kill -3 <pid>
If you need help reading the output, please post it in your original question.
If nothing is shown from the thread dump, other alternatives can be looked at.
Hum... I would suggest using JMetter to stress the Application and take note of anything weird that might be happening (such as Memory Leaks, Deadlocks and such). Also review the code for any Exceptions that might interrupt the program (or System.exit() calls). Finally, if other people have access to the computer, makes sense to check if the process wasn't killed manually somehow.