We Keep Coding

Why Java application is not working when VM size is increased? [duplicate]

I am looking for a Java code profiler which I can use to profile my application (its a service which runs in backend) on production (so means low over head, and it must not slow down my application). Primarily I want calling tree profiling, that is if a() calls b() and then b() calls c(), then how much time a() b() and c() took, both inclusively and exclusively.
Have seen jvisualvm and jprofiler, but this is not what I am looking for, because I cannot tie my production application to them as it will cause a major performance issue.
Also, I did go through metrics (https://github.com/dropwizard/metrics), but it does not give me a functionality to profile the calling tree.
Callgrind (http://valgrind.org/docs/manual/cl-manual.html) type library is what I need, as it gives calling tree profiling functionality and advanced options like avoiding calling cycles (recursion). But I am not sure that Callgrind can be used on production as it dumps data when program terminates.
Can anyone suggest good calling tree profiler for java that can be used on production without compromising the performance?

Take a look at Java Mission Control in conjunction with Flight Recorder. Starting with the release of Oracle JDK 7 Update 40 (7u40), Java Mission Control is bundled with the HotSpot JVM, so it is highly integrated and purports to have small effects on run-time performance. I have only just started looking at it, and I do see some call tree functionality.

In general you don't (or I won't recommend) profilers that instrument your application. Instrumenting always means an uncontrollable production overhead.
What you can use is a sampling profiler. A sampling profiler makes a snapshot of the stack traces at a controllable interval. What you don't get is call counts, but, after some runtime, you get a good overview where you have hotspots. Since you can adjust the sample interval of the profiler you can influence the overhead of it.
A usable sampling profiler is shipped with the JDK, see the hprof page in the java 7 documentation.
In former times there existed some graphical analysis tools for the hprof cpu traces (not the heap traces). Now they are gone. However, you can already work with the generated text file.
I took a quick look on the Java Mission Control stuff mentioned above. I think it is quite mighty and will satisfy a lot of needs, in the white paper they say it has only 2% overhead. However, it is not totally what I personally need or want. For my applications it is better to have a "light" profiling enabled all the time.

Intel Amplifier XE http://software.intel.com/en-us/intel-vtune-amplifier-xe has got low overhead if any noticeable. It uses stack sampling technology to minimize the impact and it can attach and detach to running non-stop processes in production. You even do not need to have sources during profiling, you can dive into sources later after offline performance results browsing.

I don't know of any tool which can do profiling without an impact on performance.
You could add logging to the methods that you're interested in. Make sure you include the time stamp in the log; then you can do the timing. You should also configure the logging framework to log asynchronously to reduce the performance loss.
A load time weaver like AspectJ is able to add these calls at runtime, which would allow you to easily select the methods you want to monitor without changing the source code all the time.
Using an around aspect, you can even add timing logging, so you don't have to parse the logs and try to find matching log entries. See this blog post for details.
Have a look at perfspy (tutorial), it might already do out of the box what you need.
Related:
How to use AOP with AspectJ for logging?
http://mathewjhall.wordpress.com/2011/03/31/tracing-java-method-execution-with-aspectj/
http://www.jroller.com/holy/entry/injecting_timing_aspect_into_junit

Java application profiling

I am looking for a Java code profiler which I can use to profile my application (its a service which runs in backend) on production (so means low over head, and it must not slow down my application). Primarily I want calling tree profiling, that is if a() calls b() and then b() calls c(), then how much time a() b() and c() took, both inclusively and exclusively.
Have seen jvisualvm and jprofiler, but this is not what I am looking for, because I cannot tie my production application to them as it will cause a major performance issue.
Also, I did go through metrics (https://github.com/dropwizard/metrics), but it does not give me a functionality to profile the calling tree.
Callgrind (http://valgrind.org/docs/manual/cl-manual.html) type library is what I need, as it gives calling tree profiling functionality and advanced options like avoiding calling cycles (recursion). But I am not sure that Callgrind can be used on production as it dumps data when program terminates.
Can anyone suggest good calling tree profiler for java that can be used on production without compromising the performance?

Take a look at Java Mission Control in conjunction with Flight Recorder. Starting with the release of Oracle JDK 7 Update 40 (7u40), Java Mission Control is bundled with the HotSpot JVM, so it is highly integrated and purports to have small effects on run-time performance. I have only just started looking at it, and I do see some call tree functionality.

In general you don't (or I won't recommend) profilers that instrument your application. Instrumenting always means an uncontrollable production overhead.
What you can use is a sampling profiler. A sampling profiler makes a snapshot of the stack traces at a controllable interval. What you don't get is call counts, but, after some runtime, you get a good overview where you have hotspots. Since you can adjust the sample interval of the profiler you can influence the overhead of it.
A usable sampling profiler is shipped with the JDK, see the hprof page in the java 7 documentation.
In former times there existed some graphical analysis tools for the hprof cpu traces (not the heap traces). Now they are gone. However, you can already work with the generated text file.
I took a quick look on the Java Mission Control stuff mentioned above. I think it is quite mighty and will satisfy a lot of needs, in the white paper they say it has only 2% overhead. However, it is not totally what I personally need or want. For my applications it is better to have a "light" profiling enabled all the time.

Intel Amplifier XE http://software.intel.com/en-us/intel-vtune-amplifier-xe has got low overhead if any noticeable. It uses stack sampling technology to minimize the impact and it can attach and detach to running non-stop processes in production. You even do not need to have sources during profiling, you can dive into sources later after offline performance results browsing.

I don't know of any tool which can do profiling without an impact on performance.
You could add logging to the methods that you're interested in. Make sure you include the time stamp in the log; then you can do the timing. You should also configure the logging framework to log asynchronously to reduce the performance loss.
A load time weaver like AspectJ is able to add these calls at runtime, which would allow you to easily select the methods you want to monitor without changing the source code all the time.
Using an around aspect, you can even add timing logging, so you don't have to parse the logs and try to find matching log entries. See this blog post for details.
Have a look at perfspy (tutorial), it might already do out of the box what you need.
Related:
How to use AOP with AspectJ for logging?
http://mathewjhall.wordpress.com/2011/03/31/tracing-java-method-execution-with-aspectj/
http://www.jroller.com/holy/entry/injecting_timing_aspect_into_junit

Any good test examples for testing profiler?

In order to learn more about testing, we're going to use a profiler on a larger project (to actually get some values and measurements) and since we don't have any large project ourselves, we're forced to use something else. Any good suggestions? Maybe testing JUnit perhaps? (not "With" JUnit)?
Edit:
Not looking for any specific data, just... something... The problem is that all of this is so new so it gets kinda confusing. The point is to get slightly accustomed to testing tools such as a profiler. In other words, there shouldn't be too necessary to know much about the actual program since the program don't really matter and the data gained isn't too significant either and is mostly supposed to merely demonstrate that you can actually get stuff out of testing. So it's a bit confusing how I should proceed since I am not used to big actual programs.
Can I just download normal java files and just run/profile them with NetBeans (or similar) without having to do or care about a bunch of stuff?

Well, I've got my standard scenario. It's in C++, but it shouldn't take more than a day or two to recode it in Java.
Caveat: The scenario is not about measuring, per se, but about performance tuning, which is not at all the same thing.
It makes the point that serious code often contains multiple performance problems, and if you're really trying to make it go fast, profilers are not necessarily the best tools.

It depends on what type of data you want to profile. But the best way to get a "larger project" if you don't have one, is to find some open source project on the web that fit with what you want.
Edit: I never profile with NetBeans, so I can't tell you for this tool, but if you don't care about the tool, you can start trying with VisualVM (included with the JDK), it's a tool for monitoring the JVM. It's very usefull, and if you already run java application (like NetBeans) you'll not need to download extra applications.
Description of the tool taken on their website: VisualVM monitors application CPU usage, GC activity, heap and permanent generation memory, number of loaded classes and running threads.
VisualVM website
If you really want to profile with some source code, a little java application with a main will do the job, but again it depends on what data/amout of data you want to profile. Maybe you can find some "test applications" written in java on the web.

What versions of java are slow for gc logging?

I've been told by my company's support team that some versions of java have a significant performance impact when we turn on -verbose:gc. However I can't figure out if this is the case or not.
Was this logging slow(ish) at some point, and when did it stop?
The reason I ask is that there's some hesitation about applying this to a production environment to investigate potential memory leaks (and whether we can stop doing periodic restarts of the system...).
Specifically I'm talking about Java 1.4.2 which I think introduced the argument, and what service pack it applies up to.

I know you asked about the impact of verbose:gc (Amir is correct), but based on the comments I see you are investigating a memory leak.
Is it possible for you to get a histogram of your environment? verbose GC will only show you that there is a memory leak, not where the memory is sitting.
you mention java 1.4.2, is that your current version? If you are using 1.5 or higher you can use
jmap -histo <pid> > file.txt
This will give you a breakdown of all the objects in memory. You will freeze your JVM for a time dependent on the amount of memory in the system. (2GB can freeze for a minute or so on even good hardware) test this on a development system first. I know you don't want to impact your production environment but this is a necessary evil to find the source of the problem. Do a capture right before the periodic restart to lesson your impact.

I suggest that you do the following:
Write some benchmark that is likely to stress the garbage collection. (Create large linked data structures with weak references, etc, etc).
Install a copy of the same version of the JVM as you are using in production on some test box.
Run the benchmark with various GC logging settings, including the settings that you want to run in production, measuring the performance impact on the benchmark.
If you do this right, it will give you some solid evidence about what the likely performance impact will be for your production server.

If you have a Java application that is consuming CPU when it isn't doing anything, how do you determine what it is doing?

I am calling a vendor's Java API, and on some servers it appears that the JVM goes into a low priority polling loop after logging into the API (CPU at 100% usage). The same app on other servers does not exhibit this behavior. This happens on WebSphere and Tomcat. The environment is tricky to set up so it is difficult to try to do something like profiling within Eclipse.
Is there a way to profile (or some other method of inspecting) an existing Java app running in Tomcat to find out what methods are being executed while it's in this spinwait kind of state? The app is only executing one method when it gets in this state (vendor's method). Vendor can't replicate the behavior (of course).
Update:
Using JConsole I was able to determine who was running and what they were doing. It took me a few hours to then figure out why it was doing it. The problem ended up being that the vendor's API jar that was being used did not match exactly to the the database configuration that it was using. It was defaulting to having tracing and performance monitoring enabled on the servers that had the slight mis-match in configuration. I used a different jar and all is well.
So thanks, Joshua, for your answer. JConsole was extremely easy to setup and use to monitor an existing application.
#Cringe - I did some experimenting with some of the options you suggested. I had some problems with getting JProfiler set up, it looks good (but pricey). Going forward I went ahead and added the Eclipse Profiler plugin and I'll be looking over the different open source profilers to compare functionality.

If you are using Java 5 or later, you can connect to your application using jconsole to view all running threads. jstack also will do a stack dump. I think this should still work even inside a container like Tomcat.
Both of these tools are included with JDK5 and later (I assume the process needs to be at least Java 5, though I could be wrong)
Update:
It's also worth noting that starting with JDK 1.6 update 7 there is now a bundled profiler called VisualVM which can be launched with 'jvisualvm'. It looks like it is a java.net project, so additional info may be available at that page. I haven't used this yet but it looks useful for more serious analysis.
Hope that helps

Facing the same problem I used YourKit profiler. It's loader doesn't activate unless you actually connect to it (though it does open a port to listen for connections). The profiler itself has a nice "get amount of time spent in each method" while working in it's less obtrusive mode.
Another way is to detect CPU load (via JNI, so you'd need an external library for this) in a "watchdog" thread with highest priority and start logging all threads when the CPU is high enough for a long enough time. You might find this article enlightining.

If it's for professional purpose and you have some money to spend, try to get your hands on JProfiler. If you just want to get some insights, try out the Eclipse Profiler Plugin. I used it several times, but I don't know the current state.
A new(?) project from the eclipse project itself is available too: http://www.eclipse.org/tptp/ (See this article). Never used it, so I can't tell if it is worth the effort.
There's also a very good list of open source profilers available at http://www.manageability.org/blog/stuff/open-source-profilers-for-java

If JConsole can't be used you can
press CTRL+BREAK under Windows
send kill -3 <process id> under Linux
to get a full Thread Dump. This doesn't affect performance and can always be run in production.

JRockit Mission Control Latency Analyzer.
The Latency Analyzer that comes with JRockit shows you what the JVM is "doing" when it's not doing anything. In the latest version you can see latencies for:
Java wait/blocked/sleep/parked.
File I/O
Network I/O
Memory allocation
GC pauses
JVM latencies, e.g code generation and class loading
Thread suspension
The tool will give you the stack trace when the latency occurred. You can view the latency data in many different ways (aggregated traces, as a histogram, in a thread graph etc.). The tool also allows you to see transitions between threads, for instance when one thread notifies another.
latency analyzer http://blogs.oracle.com/hirt/WindowsLiveWriter/The.0LatencyAnalyserMigratedfromtheoldBE_7246/latency_graph_2.png
The overhead is negligible and unlike many other tools it can be used in a production environment.
This blog post gives you a brief introduction and the program can be downloaded here.
It's free to use for development!

Use a profiler. Yes they cost money, and using them can occasionally be a bit awkward, but they do provide you with a great deal more real evidence rather than guesswork.
Human beings are universally bad at guessing where performance bottlenecks are. It just seems to be something our brains aren't build to do very well. It may seem obvious, you may have great ideas about what the problem is, but the real world often turns out to be doing something different. And optimising the wrong part of code means, at best, lots of work for minimal benefit. More often it makes things slower, and sometimes it breaks things entirely. So before you make any changes for the sake of optimisation, you should always have real evidence from a profiler or other accurate tool.
As mentioned, both JProfiler and YourKit are both fairly good and not prohibitively expensive. Last time I looked, they both had free demos too.

For completeness sake: even though my company more or less standardizes on Eclipse we use Netbeans (6 and up) with its included, free profiler on a daily basis. It works better than the Eclipse TPTP plugin (last checked 3 months ago) and for us it removes any need for a commercial profiler such as JProfiler, which is excellent, but fast becoming unnecessary.

VisualVM should be the profiler from netbeans as standalone. I tried the TPTP for eclipse but visualVm seems as a much nicer option!