I am kind of stuck on an issue. I have a Spring + Hibernate app, which for the last few days has been behaving really strange.
Normally, even on Debug mode, it bootstrapped for around 15 seconds.
As of a couple of days, without any significant errors or problems being shown, it started running twice, if not three times as slowly.
I thought maybe the problem was in Tomcat, but even when I ran the series of unit tests which I've written, they went amazingly slow (we are talking 8+ sec on a test + 20sec for the initial context bootstrapping). I use a local PostgreSQL database for the tests which is normally not so bad (around one second on a test)
I am stuck. I know that the last thing I did was to add support for #Transactional on my #Controller classes. Could this be the reason? I doubt, because when I deployed the transactional modifications to the stable server and restarted it, it ran just as fast as before.
I am completely stuck
You are aware that based on your description we can only guess. Obviously if you are using continuous-integration, build times of subsequent builds will reveal which code change could be the reason. If you identify the code, diagnosing will be much simpler.
Chances are it wasn't really caused by code change but by the environment change. E.g.:
Spring fetches some external resource on bootstrap (XML schemas, DTDs, etc.)
there is some extra network overhead (router/switch reconfiguration, firewall)
Database is fragmented or somehow slow. Try vacuuming
Investigating:
Capture few stack traces during bootstrap. No need for full-blown profiling, I bet there is a single blocking/waiting operation slowing things down
Enable every logging statement you can. Every. Investigate which pieces of your application are slow.
My guess is that your DB is the root cause for this.
Probably you have some query that is taking longer than it used to due to data size growth, schema changes, or the like.
Here are some tips:
Try to check the startup logs to see what seems to be slower.
Can you try to start an older version of the app, and see if it's faster?
Can you see which tests are slower now?
Maybe profiling will help you to gain more information. There is a nice tool which comes shipped with JDK called jvisualvm (you can find it within the bin folder of your jdk installation). It's pretty much self explanory. You can connect to you applciation and start sampeling.
Related
We have a costumer that have 3 stores with different databases. In every store has a wildfly running some webservices which communicate between them. Each json request with 10/30 rows spends 1 seconds in average. Every wildfly uses 1,5 gb of RAM. I know that memory is always a problem in Java, but can I be more economic using some microframework like Javalin or microservices rather than a java ee app server? And node.js would be a option for better performance?
Before you start looking into a different architecture, which would probably make for a major rewrite, find out where all that time is going. Set up profiling on the WildFly servers. Start by doing that on one, then have some calls coming in. Check how much time is spent in various parts of the stack. Is one call to the web service handled rather slowly? Then see where that time goes. It might be the database access. Is one such call handled pretty quickly on the server itself once it comes in? Then your best bet is your losing time on the network layer.
Check the network traffic. You can use Wireshark or a similar tracing tool for this. See how much time actually passes between a request coming in and the response going out. Is that slow but the processing on Wildfly itself seems fast enough? Maybe there's some overhead going on (like security). Is the time between request and response very fast? You're definitely looking at the network as the culprit.
Eventually you may need to have profiling and network tracing active on all three servers simultaneously to see what's going on, or for each combination of two servers. It may turn out only one of them is the bottleneck. And if you have servers A, B and C, from the sound of it your setup might cause a call from A to B to also require a call from B to C before some result can be returned to A. If that is the case, it's little wonder you may see some serious latency.
But measure and find the root of the problem before you start deciding to change the entire framework and a different programming language. Otherwise you may put a lot of time into something for no improvement at all. If the architecture is fundamentally flawed you need to think of a different approach. If this is still in the prototyping phase that would be substantially easier.
Well, first you may prune your WildFly installation or try Quarkus :)
I know this issue has been beat to death and I have read up on all the various solutions such as adjusting the lazy fetching. The issue is that this application has been running for a couple years now and just recently started having this issue.
The strange part is this application works fine on my local dev machine as well as our testing server. Once I deploy this into our production site I get the error. So I don't believe its an application problem, it has to be something with the server, maybe some sort of tomcat configuration that might somehow be different?
The setups are the same between all 3 servers with versions of tomcat/mysql/java. Does anyone have ideas on what I can look at?
For what its working im running hibernate 3.5.3, Spring 3.02, Struts 2.2.1, Tomcat 6.0.24, Mysql 5.1.67 on a RHEL 6.3 Server
Thanks
I would explore 2 parallel paths to figuring out the answer.
One is, try to troubleshoot the problem itself. Crank up your log levels to DEBUG and see if that illuminates anything. Are you able to successfully fetch a simple object, and only complex objects (those that have a sub-object) cause a problem? etc. It may have to do with the data source / connection pool, which often has a different setup in production.
The 2nd approach is, figure out what has changed. If this problem just started happening, then something must have changed. Have you upgraded a library, changed versions of Tomcat, changed the OS, etc.
Related to this 2nd approach is figuring out what is different between your development/test environment and production. They can't be "the same" in every way if the problem occurs reliably in production but not the others. If you have a version of "production" that you can tinker with, without screwing with active users, then you can experiment with changes and see if you can isolate the specific thing that causes the problem to appear.
I'm trying to eliminate a slow database being the cause of some performance issues for a distributed application I'm supporting. I've done local profiling of various facets of the application and performance monitoring of the server itself, leading me to suspect that the database is at least partially responsible for the poor performance.
Currently I'm using JBoss for the back-end (using a Hibernate / JDBC layer to connect to the database), but I only have source access to some of the code.
I've found Elvyx, but this project seems to have been abandoned in 2008. Is there a newer JDBC profiler available - what's the current 'de facto' standard for profiling a database in a distributed app?
Alternatively, can anyone suggest a better / alternative approach?
Try using YourKit, it supports a reasonable degree of JDBC profiling:
You can view executed SQL after you capture a CPU snapshot.
You can also enable JDBC probes and view multiple things live, such as timings, stack traces, threads, SQL statements and many more, see attached screen shot from my colleague's computer (looks like you'll need to open this image in another browser tab/window to see it full size):
Don't really want this to sound as an ad for YourKit, but get yourself a trial license and give it a go.
I too would recommend AppDynamics. The 'Lite' version was more than adequate for my purposes.
Our Java EE application runs within JBoss and we knew we had performance bottlenecks in certain areas. By quickly and easily (and I do mean easily) installing and spinning up AppDynamics, then running some load through the application, we were able to see straight away exactly where our performance hits were located. On the clean and concise dashboard we were able to drill right down the stack to see which class needed some improvement.
Highly recommended. Definitely check it out. I heat the 'Pro' version is even better.
If you are trying to hunt down (or at least confirm) issues related to a slow database, IMO using the profiling tools provided by the database would be a good starting point.
We had done something along these lines previously by profiling JDBC calls (noting the timings) and comparing them against the time required to execute the same query "on" the database itself. This gave a pretty good idea of how much time was exactly spent by the JDBC in making the db call and getting back the result.
dynaTrace supports SQL call introspection and measures how long each sql call took. The field is called application performance management in general.
Is there a particular Use case which you feel is slow or Is it in general that you feel DB responsiveness is slow.
In case its a use case, I would suggest go for tools like AppDynamics or GlassBox.
In case its in general, Starting from DB is a better approach.
I am assuming you would have already done the analysis at distributed applications side regarding the connection metrics and at DB server OS side regarding the socket opens and IO permissible.
Arcturus Applicare does support JDBC profiling on JBoss (and other Java app servers). You can view all SQLs with min, max, avg stats aggregated across all servers on you env or on individual servers.
With full profiling enabled, you will be able to see the execution trace for each and every request/transaction processed by the server including SQLs with execution parameters. Making it pretty easy to detect expensive SQLs and where exactly they are been executed.
I have an open source Java application that uses Hibernate and HSQLDB for persistence. In all my toy tests, things run fast and everything is good. I have a client who has been running the software for several months continuously and their database has grown significantly over that time, and performance has dropped gradually. It finally occurred to me that the database could be the problem. As far as I can tell from log statements, all of the computation in the server happens quickly, so this is consistent with the hypothesis that the DB might be at fault.
I know how to do normal profiling of a program to figure out where hot spots are and what is taking up significant amounts of time. But all the profilers I know of monitor execution time within the program and don't give you any help about calls to external resources. What tools do people use to profile programs that are using external db calls to find out where to optimize performance?
A little blind searching around has already found a few hot spots--I noticed a call where I was enumerating all the objects of a particular class in order to find out whether there were any. A one line change to the criterion [.setMaxResults(1)] changed that call from a half-second to virtually instantaneous. I also see places where I ask the same question from the db many times within a single transaction. I haven't figured out how to cache the answer yet, but what I really want is a tool to help me look for these kinds of things more systematically.
Unfortunately, as far as I know, there is no tool for that.
But there are some things you might want to check:
Are you using eager loading instead of lazy loading? By the description of your problem, it really looks like you are not using lazy loading...
Have you turned on and properly configured your second-level caching? Including the Query Cache? Hibernate caching mechanism is extremely powerful and flexible.
Have you consider using Hibernate Search? Depending on your query, Hibernate Search Full Text index on top of Apache Lucene can speed up you queries (since it indexing system is so powerful)
How much data are you storing in HSQLDB? I don't think it performs well when managing large sets of data, since it's just storing everything in files...
There was once a tool called IronGrid/IronEye/IronTrackSql that did exactly what you are looking for. Unfortunately, they went out of business. They did open source their product at the last minute, but I have not been able to find source or a binary for quite some time.
I have been using YourKit for profiling lately, partly because you can have it profile SQL time to find your most called statements and longest running statements. It is not as detailed as IronGrid was, but it does give you valuable information. In my latest database/hibernate tuning session, the problem turned out to be hibernate and how and when it was doing eager vs. lazy loading, and adding some judicious overrides of the default when selecting large numbers of items.
Lots to report on here. I have some results, and am still looking for good answers.
I've found a couple of tools that help:
VisualVM (with BTrace, or the built in Trace) claims to help with tracing, but I haven't been able to find any tool that shows timing on method calls.
YourKit is reputed to be useful; I've asked for an open source license.
The most useful thing I found is Hibernate's built in statistics. If you set
hibernate.generate_statistics true in your properties, you can send sessionFactory.getStatistics(), and see detailed statistics on what objects have been stored and retrieved and what affects the caches are having. I found one of the answers I wanted in the qeuryStatistics, which reports for each compiled query, the cache hits and misses, the number of times the query has run, how many rows were returned, and the average, max and min execution times. These timings made it abundantly clear where the time was going.
I then did some reading on caching. Razenha's suggestion was right on. [I'll mark his answer as right for now.] I added hibernate.cache.use_query_cache true to my properties, and added query.setCacheable(true); to most of my queries. I also added <cache usage="read-write"/> to a few of my .hbm.xml files. Now most of my statistics are showing a vast predominance of cache hits, and the performance is vastly better.
I'd still like some tools to help me trace execution timing so I can attack the worst problems rather than the most obvious, but this is a big help. Maybe one of the tracing tools above will turn out to help.
In Terracotta 3.1, you can monitor all of those statistics in real-time using the Terracotta Developer Console. You can see historical graphs for cache statistics, and see the hibernate statistics or cache statistics cluster-wide or on an per-node basis.
Terracotta is open source. More details and download is at Terracotta for Hibernate.
I am calling a vendor's Java API, and on some servers it appears that the JVM goes into a low priority polling loop after logging into the API (CPU at 100% usage). The same app on other servers does not exhibit this behavior. This happens on WebSphere and Tomcat. The environment is tricky to set up so it is difficult to try to do something like profiling within Eclipse.
Is there a way to profile (or some other method of inspecting) an existing Java app running in Tomcat to find out what methods are being executed while it's in this spinwait kind of state? The app is only executing one method when it gets in this state (vendor's method). Vendor can't replicate the behavior (of course).
Update:
Using JConsole I was able to determine who was running and what they were doing. It took me a few hours to then figure out why it was doing it. The problem ended up being that the vendor's API jar that was being used did not match exactly to the the database configuration that it was using. It was defaulting to having tracing and performance monitoring enabled on the servers that had the slight mis-match in configuration. I used a different jar and all is well.
So thanks, Joshua, for your answer. JConsole was extremely easy to setup and use to monitor an existing application.
#Cringe - I did some experimenting with some of the options you suggested. I had some problems with getting JProfiler set up, it looks good (but pricey). Going forward I went ahead and added the Eclipse Profiler plugin and I'll be looking over the different open source profilers to compare functionality.
If you are using Java 5 or later, you can connect to your application using jconsole to view all running threads. jstack also will do a stack dump. I think this should still work even inside a container like Tomcat.
Both of these tools are included with JDK5 and later (I assume the process needs to be at least Java 5, though I could be wrong)
Update:
It's also worth noting that starting with JDK 1.6 update 7 there is now a bundled profiler called VisualVM which can be launched with 'jvisualvm'. It looks like it is a java.net project, so additional info may be available at that page. I haven't used this yet but it looks useful for more serious analysis.
Hope that helps
Facing the same problem I used YourKit profiler. It's loader doesn't activate unless you actually connect to it (though it does open a port to listen for connections). The profiler itself has a nice "get amount of time spent in each method" while working in it's less obtrusive mode.
Another way is to detect CPU load (via JNI, so you'd need an external library for this) in a "watchdog" thread with highest priority and start logging all threads when the CPU is high enough for a long enough time. You might find this article enlightining.
If it's for professional purpose and you have some money to spend, try to get your hands on JProfiler. If you just want to get some insights, try out the Eclipse Profiler Plugin. I used it several times, but I don't know the current state.
A new(?) project from the eclipse project itself is available too: http://www.eclipse.org/tptp/ (See this article). Never used it, so I can't tell if it is worth the effort.
There's also a very good list of open source profilers available at http://www.manageability.org/blog/stuff/open-source-profilers-for-java
If JConsole can't be used you can
press CTRL+BREAK under Windows
send kill -3 <process id> under Linux
to get a full Thread Dump. This doesn't affect performance and can always be run in production.
JRockit Mission Control Latency Analyzer.
The Latency Analyzer that comes with JRockit shows you what the JVM is "doing" when it's not doing anything. In the latest version you can see latencies for:
Java wait/blocked/sleep/parked.
File I/O
Network I/O
Memory allocation
GC pauses
JVM latencies, e.g code generation and class loading
Thread suspension
The tool will give you the stack trace when the latency occurred. You can view the latency data in many different ways (aggregated traces, as a histogram, in a thread graph etc.). The tool also allows you to see transitions between threads, for instance when one thread notifies another.
latency analyzer http://blogs.oracle.com/hirt/WindowsLiveWriter/The.0LatencyAnalyserMigratedfromtheoldBE_7246/latency_graph_2.png
The overhead is negligible and unlike many other tools it can be used in a production environment.
This blog post gives you a brief introduction and the program can be downloaded here.
It's free to use for development!
Use a profiler. Yes they cost money, and using them can occasionally be a bit awkward, but they do provide you with a great deal more real evidence rather than guesswork.
Human beings are universally bad at guessing where performance bottlenecks are. It just seems to be something our brains aren't build to do very well. It may seem obvious, you may have great ideas about what the problem is, but the real world often turns out to be doing something different. And optimising the wrong part of code means, at best, lots of work for minimal benefit. More often it makes things slower, and sometimes it breaks things entirely. So before you make any changes for the sake of optimisation, you should always have real evidence from a profiler or other accurate tool.
As mentioned, both JProfiler and YourKit are both fairly good and not prohibitively expensive. Last time I looked, they both had free demos too.
For completeness sake: even though my company more or less standardizes on Eclipse we use Netbeans (6 and up) with its included, free profiler on a daily basis. It works better than the Eclipse TPTP plugin (last checked 3 months ago) and for us it removes any need for a commercial profiler such as JProfiler, which is excellent, but fast becoming unnecessary.
VisualVM should be the profiler from netbeans as standalone. I tried the TPTP for eclipse but visualVm seems as a much nicer option!