I have recently downloaded a trial version of YourKit and after playing with it for a while it looks great, but I am concerned about running it in the production environment. Previous profilers I have used have put unacceptable overhead on the servers.
Does anyone know the typical overhead of the YourKit software? Or has anyone had any problems running YourKit in a production environment?
I am running YourKit for Java. The server I am profiling is RedHat running JBoss 4.
For everbody wondering: Initially we already have had really strange performance issues, which we couldn't place for quite a while.
So we installed yourkit on our production servers (Tomcat) and disabled the telemetry (disableexceptiontelemetry,disablestacktelemetry) as advised.
Then we started tuning the Garbage Collection but this did not seem to make any difference.
Still from time to time randomly one of the servers would start to show real bad performance. Sometimes it did recovered by itself, more often a restart was the only solution.
After a lot of debugging & log-reading we found some very long periods of reference checking concerning the JNI Weak References in the gc log.
The probes of yourkit seemed to mess with that somehow.
After disabling the probes as well (builtinprobes=none) everything got back to normal.
The tuning of the gc config had solved our initial performance problems already, but leaving the yourkit probes active created a new issue, which was so similar, that we couldn't tell them apart.
See here for more details:
Java G1 GC Processing Reference objects works slow
I have used Yourkit on JBOSS and Mule servers in production. What I felt is that when the load increases , it is throwing out of memory error. Then we stopped using it in production and we use it only for local testing.
And we use Jconsole in production server to monitor server resources like CPU , memory,Threads.
It is really helpful.
I have used yourkit in production but on a tomcat server. It works pretty well and we didn't noticed any major performance overheads.
We had many instances of Tomcat server running behind a load balancer. So we put yourkit on one of the servers and things work out pretty well.
Related
I have an issue that some of our microservices have a memory allocation spike that triggers an OOMKiller Kubernetes event.
The issue doesn't happen that often but enought to be noticed.
The service is not doing anything crazy at the moment when it's happening, just normal load.
Also, interesting fact that memory doesn't gradually increase, on the contrary - it's like a sudden explosion.
I also did some deep analysis on what apps are doing at the time of mem explosion - all seem normal to be normal which is why I don't think it's some problem or memory leak in the code. Plus same behaviour happens across different services that are not connected with each other in almost any way. Also, i most of the cases after restarting service proceeding with the work where it was left off. Which is why I'm leaning towards the idea that it's some kind of Framework or Software problem.
More on the setup:
Spring Boot (2.4.x - 2.5.x). Different stacks(Web and WebFlux) and dependencies (Spring Cloud, Spring Kafka, ElasticSearch, Postgres, etc) are used accross microservices.
Java 11 (latest OpenJDK image)
The apps are running on one of the newest versions of Kubernetes cluster(AKS).
Before we had no resource limits implemented for microservices. So in case of an incident, we had an infinite memory allocation until it eats all of the memory of K8S node which forces everything to get down & restart.
Currently we have resource limits for pods implemented to avoid that behaviour.
Questions:
I'm wondering if it's a JVM/Spring issue / memory leak. Maybe someone had a similar experience?
Are there any tools / approaches you can recommend to take a snapshot of the pod before it's getting killed to have an ability to analyze that later?
We were able to catch metrics at the time when it's happening. Turned out, that memory limits were misconfigured in helm charts and default values were taken by JVM. This leaded to situtation when JVM had a lot (around 3GB) memory allocated but only used some (150MB). So our K8s node was experiencing some memory starvation at the time when it was running lots of in the mode like that mode.
And yes, Instana turned out to be quite helpful tool to debug that. See metrics listed below:
We have an application running on Weblogic 8.1.3, using the bundled 1.4.2 JDK, and it's leaking memory moderately rapidly.
I've done some reading around about how to fix memory leaks, but most of it seems to assume that the JDK being used is 5 or higher. Are there any tools available for earlier versions?
Other than that, there's very little information that we've found: the leak only seems to occur on the full production environment, rather than the test environments.
We have two machines running weblogic, clustered for load balancing
The leak occurs on one of the clustered servers at a time (?!), but never both
The leak sometimes, but not always, switches from server to server when Weblogic is restarted.
So I figure that there must be an object created at server startup that can be created on one (but not both) servers that is behind the leak. Does this seem a reasonable place to start looking?
JProfiler supports profiling Java 1.4 in its current version (7.0)
You could have a look at this screen cast on how to search for memory leaks with JProfiler.
Disclaimer: My company develops JProfiler
Have you tryed to run jvisualvm and look into the memory used (heap dump)?
-> http://download.oracle.com/javase/6/docs/technotes/tools/share/jvisualvm.html
I'm new into Java profiling, and I'd like to ask about it.
Does it make sense to profile an application on the server, where I only have the console?
Is there any console profiler which make sense?
Or should I profile the application only on localhost?
VisualVM is able to connect to a remote host. The profiling works the same way as local. It's part of the JDK since JDK 6 update 7.
When you do profiling you should generally try to reproduce the production environment as closely as possible. Differences in hardware (# of cores, memory, etc) and software (OS, JVM version) can make your profiling results as unique as the runtime environment.
For example, what looks like a CPU bottleneck worth optimizing on your local machine might disappear entirely or turn into a disk bottleneck on your production server depending on the differences in the CPU.
All modern profilers will allow to attach to a remotely running JVM so you don't need to worry about only having console access.
What profiler you decide to use will depend on your needs and preferences. Certain profilers will show you "hotspots" where your code is spending the majority of the time and these are often good candidates for optimization.
I prefer to use JProfiler for its extensive features and good performance. I previously used YourKit but switched to JProfiler for its memory and thread profiling features.
Does it make sense to profile an
application on the server, where I
only have the console?
Fortunately, that does not matter, since Java has always (well, for a long time, anyway) supported remote profiling, i.e. the Profiler can run on a different machine than the JVM being profiled and get its data via the network.
All Java profilers that I've ever seen support this, including visualvm, which comes with recent JDKs (in the bin directory).
There's a "quick and dirty" but effective way to find performance problems in Java.
This is a language-agnostic explanation of why it works.
Note that the JDK comes with a built-in profiler, HPROF. HPROF is a bit simplistic, but will find many problems. It is activated simply by invoking the JVM with parameter -agentlib:hprof; it will then automatically run as long as your JVM is running. It collects data until the JVM terminates, then dumps it into a file on the server, which you can analyze.
See e.g. http://java.sun.com/developer/technicalArticles/Programming/HPROF.html for a good introduction. A nice graphical analyzer for HPROF's results is PerfAnal: http://java.sun.com/developer/technicalArticles/Programming/perfanal/
Has anyone used the new Java 1.6 JDK tool, VisualVM, to profile a production application and how does the application perform while being profiled?
The documentation say that it is designed for both Production and Development use, but based on previous profiling experience, with other profiling tools, I am hesitant.
While i haven't personally used VisualVM, I saw this blog post just today that might have some useful information for you. He talks about profiling a production app using it.
I tried it on a dev box and found that when I turned off profiling it would shut Tomcat down unexpectedly. I'd be very cautious about rolling this out to production- can you simulate load in a staging environment instead? It's not as good as the real thing, but it probably won't get you fired if it goes wrong...
I've used VisualVM before to profile something running locally. A big win was that I just start it up, and it can connect to the running JVM. It's easier to use than other profiling tools I've used before and didn't seem to have as much overhead.
I think it does sampling. The overhead on a CPU intensive application didn't seem significant. I didn't measure anything (I was interested in how my app performed, not how the tool performed), but it definitely didn't have the factor of 10 slowdown I'm used to seeing from profiling.
For just monitoring your application, running VisualVM remotely should not slow it down much. If the system is not on the edge of collapsing, I still haven't seen any problems. It's basically just reading out information from the coarse grained built-in instrumentation of the JVM. If you start profiling, however, you'll have the same issues as with other profilers. Basically because they all work almost they same way, often using the support in the JVM.
Many people have problems with running VisualVM remotely, due to firewall issues, but you can even run Visual VM remotely over ssh, with some system properties set.
It is possible to remote connect to your server from a different computer using VisualVM. You just need to right click on the "Remote" node and say "Add Remote Host."
This would at least eliminate the VisualVM overhead (if there is any) from impacting performance while it is running.
This may not eliminate all performance concerns, especially in Production environments, but it will help a little.
I've used the Net Beans profiler which uses the same underpinnings as Visual VM.
I was working with an older version of Weblogic, which meant using the 1.5 JVM, so I couldn't do a dynamic attach. The application I was profiling had several thousand classes and my workstation was pretty much unusable while the profiler instrumented them all. Once instrumentation was complete, the system was sluggish but not completely unusable. The amount of slowdown really depends on what you need to capture. The basic CPU metrics are pretty light weight. Profiling memory allocation slows things down a lot.
I would not use it on a production system. Aside from the potential for slowdown, I eventually ran out of PermGen space because the profiler reinstruments and reloads classes when you change settings. (This may be fixed in the 1.6 agent, I don't know)
I've been using VisualVM a lot since before it was included in the JDK. It has a negligable impact on the performance of the system. I've never noticed it cause a problem with performance on the system, but then again, our Java server had enough headroom at the time to support a little extra load. If your server is running at a level that is completely tacked out and can't handle the VisualVM running, then I would say its more likely that you need to buy another server . Any production server should have some memory headroom , otherwise what you have is a disaster just waiting to happen.
I have used VVM(VavaVoom?) quite extensively, works like a charm in the light mode, i.e. no profiling, just getting the basic data from the VM. But once you start profiling and there are many classes, then there is considerable slowdown. I wouldn't profile in a production environment even if you have 128 core board with 2 tera of memory purely because the reloading and re-defining of the classes is tricky, the server classloaders are another thing, also vary from one server implementation to another, interfering with them in production is not a very good idea.
I am calling a vendor's Java API, and on some servers it appears that the JVM goes into a low priority polling loop after logging into the API (CPU at 100% usage). The same app on other servers does not exhibit this behavior. This happens on WebSphere and Tomcat. The environment is tricky to set up so it is difficult to try to do something like profiling within Eclipse.
Is there a way to profile (or some other method of inspecting) an existing Java app running in Tomcat to find out what methods are being executed while it's in this spinwait kind of state? The app is only executing one method when it gets in this state (vendor's method). Vendor can't replicate the behavior (of course).
Update:
Using JConsole I was able to determine who was running and what they were doing. It took me a few hours to then figure out why it was doing it. The problem ended up being that the vendor's API jar that was being used did not match exactly to the the database configuration that it was using. It was defaulting to having tracing and performance monitoring enabled on the servers that had the slight mis-match in configuration. I used a different jar and all is well.
So thanks, Joshua, for your answer. JConsole was extremely easy to setup and use to monitor an existing application.
#Cringe - I did some experimenting with some of the options you suggested. I had some problems with getting JProfiler set up, it looks good (but pricey). Going forward I went ahead and added the Eclipse Profiler plugin and I'll be looking over the different open source profilers to compare functionality.
If you are using Java 5 or later, you can connect to your application using jconsole to view all running threads. jstack also will do a stack dump. I think this should still work even inside a container like Tomcat.
Both of these tools are included with JDK5 and later (I assume the process needs to be at least Java 5, though I could be wrong)
Update:
It's also worth noting that starting with JDK 1.6 update 7 there is now a bundled profiler called VisualVM which can be launched with 'jvisualvm'. It looks like it is a java.net project, so additional info may be available at that page. I haven't used this yet but it looks useful for more serious analysis.
Hope that helps
Facing the same problem I used YourKit profiler. It's loader doesn't activate unless you actually connect to it (though it does open a port to listen for connections). The profiler itself has a nice "get amount of time spent in each method" while working in it's less obtrusive mode.
Another way is to detect CPU load (via JNI, so you'd need an external library for this) in a "watchdog" thread with highest priority and start logging all threads when the CPU is high enough for a long enough time. You might find this article enlightining.
If it's for professional purpose and you have some money to spend, try to get your hands on JProfiler. If you just want to get some insights, try out the Eclipse Profiler Plugin. I used it several times, but I don't know the current state.
A new(?) project from the eclipse project itself is available too: http://www.eclipse.org/tptp/ (See this article). Never used it, so I can't tell if it is worth the effort.
There's also a very good list of open source profilers available at http://www.manageability.org/blog/stuff/open-source-profilers-for-java
If JConsole can't be used you can
press CTRL+BREAK under Windows
send kill -3 <process id> under Linux
to get a full Thread Dump. This doesn't affect performance and can always be run in production.
JRockit Mission Control Latency Analyzer.
The Latency Analyzer that comes with JRockit shows you what the JVM is "doing" when it's not doing anything. In the latest version you can see latencies for:
Java wait/blocked/sleep/parked.
File I/O
Network I/O
Memory allocation
GC pauses
JVM latencies, e.g code generation and class loading
Thread suspension
The tool will give you the stack trace when the latency occurred. You can view the latency data in many different ways (aggregated traces, as a histogram, in a thread graph etc.). The tool also allows you to see transitions between threads, for instance when one thread notifies another.
latency analyzer http://blogs.oracle.com/hirt/WindowsLiveWriter/The.0LatencyAnalyserMigratedfromtheoldBE_7246/latency_graph_2.png
The overhead is negligible and unlike many other tools it can be used in a production environment.
This blog post gives you a brief introduction and the program can be downloaded here.
It's free to use for development!
Use a profiler. Yes they cost money, and using them can occasionally be a bit awkward, but they do provide you with a great deal more real evidence rather than guesswork.
Human beings are universally bad at guessing where performance bottlenecks are. It just seems to be something our brains aren't build to do very well. It may seem obvious, you may have great ideas about what the problem is, but the real world often turns out to be doing something different. And optimising the wrong part of code means, at best, lots of work for minimal benefit. More often it makes things slower, and sometimes it breaks things entirely. So before you make any changes for the sake of optimisation, you should always have real evidence from a profiler or other accurate tool.
As mentioned, both JProfiler and YourKit are both fairly good and not prohibitively expensive. Last time I looked, they both had free demos too.
For completeness sake: even though my company more or less standardizes on Eclipse we use Netbeans (6 and up) with its included, free profiler on a daily basis. It works better than the Eclipse TPTP plugin (last checked 3 months ago) and for us it removes any need for a commercial profiler such as JProfiler, which is excellent, but fast becoming unnecessary.
VisualVM should be the profiler from netbeans as standalone. I tried the TPTP for eclipse but visualVm seems as a much nicer option!