We Keep Coding

Profiling tomcat application with VisualVM

I am trying to run visualvm under the username tomcat6 because apparently visualvm can only find applications running under its username. So by default it is only finding applications running under my username. I have been able to connect visualvm with tomcat6 through jmx but that lacks the fine granularity of instrumented profiling.
I tried the following to run visualvm under the username tomcat6 but got the following error that I don't understand.
$ sudo -u tomcat6 jvisualvm
No protocol specified
Exception in thread "main" java.awt.AWTError: Can't connect to X11 window server using ':0' as the value of the DISPLAY variable.
at sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)
at sun.awt.X11GraphicsEnvironment.access$200(X11GraphicsEnvironment.java:65)
at sun.awt.X11GraphicsEnvironment$1.run(X11GraphicsEnvironment.java:115)
...

If the computer running your application is remote -- like a server -- then you can't run GUI applications without some work. It's probably going to be easier to enable remote access to VisualVM.
You can use two techniques to attach to a remote JVM: using jstatd or using JMX. I'm not sure what you think you are losing by using JMX, but evidently jstatd doesn't give you access to profiling tools, CPU monitor, etc.).
You need to configure your JVM and Tomcat to allow for remote access. That requires 3 steps:
Enable remote JMX. Turns out, there's a guide for that.
Fix the "wandering port" used for RMI. There's a guide for that, too.
(Optional) Arrange for secure remote-access to the server. The easiest way to do that would be to use ssh -Lport:localhost:port with a series of -L arguments to forward multiple ports from your workstation to your server. Map all the ports you had to configure in steps #1 and #2. If you don't do this, you'll need to have non-firewalled access to all the aforementioned ports.
Restart your JVM and connect with JVisualVM.
Update 2022-06-01
Note that the "wandering port" problem has been fixed at the JVM level, so there is no need for application (i.e. Tomcat) support for that. Item #1 for Tomcat 8.5 and later contains updated instructions making item #2 unnecessary with a recent JVM.

Unfortunately only sampling is available in remote mode so JMX will lack instrumentation tools.
Actually your approach to running visualvm under tomcat6 user is correct. You should take a look at this question on how to run X11 applications under sudo.
The easiest way to pass DISPLAY and XAUTHORITY environment variables is to use sudo -E command to preserve current user environment.
Also if you can't see your process under tomcat6 user you should check if CATALINA_TMPDIR is pointing to /tmp. Otherwise you should pass it to visualvm
jvisualvm -J-Djava.io.tmpdir="${CATALINA_TMPDIR}"
Actually there is a lot of alternatives like yourkit or jprofiler shipped with java agents which allows remote instrumentation profiling.

The easiest is to open a remote JXM port on Tomcat in order to be able to remotely (from your desktop computer) connect to your remote Tomcat (on your server) with jvisualvm.
You need to pass the following system properties to your JVM :
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=<whatever_port_you_want>
-Dcom.sun.management.jmxremote.ssl=false
Then open jvisualvm on your local computer (JVM version needs to be the same or newer), File -> Add Remote Host -> Enter the name on the Host. It will create an entry for this host. Right lick on this Entry -> Add JMX connection -> Enter the port -> OK
jvisualvm will then be able to access remotely to your application.
You can also secure the connection if needed by using the following system properties (you need to create the files and locate them where you want :
-Dcom.sun.management.jmxremote.password.file=jmxremote.password
-Dcom.sun.management.jmxremote.access.file=jmxremote.access
These properties needs to be added to the CATALINA_OPTS environment variable. Fr exemple :
export CATALINA_OPTS = "$CATALINA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8888 "
More info on JMX lies here : https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html

I tried to do something similar but I was not allowed to install JVisualVM on the server. Having JVisualVM connect to the remote machine never seemed to work correctly. I suspect firewall rules were blocking part of the the network connections.
The only way I found to remotely profile the server was via an ssh tunnel.
Set the JMX port in CATALINA_OPTS on the server
CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote.port=13333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false; export CATALINA_OPTS
On your desktop machine open an ssh connection to the server
ssh -D 61444 username#tomcat.server.address
Add a flag to JVisualVM so that it will proxy its network connection
"C:\Program Files\Java\jdk1.7.0_79\bin\jvisualvm.exe" -J-Dnetbeans.system_socks_proxy=localhost:61444 -J-Djava.net.useSystemProxies=true
Have JVisualVM connect to the jmxport and the network traffic is tunneled via ssh.
Good luck.

Analyze stand alone Java Application locally for threads

Respected Experts,
I have a stand alone java application and want to monitor the threads created by it. I am planning to use a tool like JConsole or JVisualVM. However, I am not able to connect these tools locally to my Java Application.
I am using Windows machine. JConsole and Java program are running locally. I have tried to run Java application with following JMV arguments with no success:
java -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false LinkedListTest
When I try to connect using JConsole, my process id is greyed and following message appears:
Note: The management agent is not enabled on this process
I think I should be able to connect JConsole to a stand alone java application. Any thoughts what I am missing here
Thanks and Regards

Thanks for the inputs. I was able to solve the problem and the details are as follows:
As #Holger mentioned, both JConsole and JVisualVM can connect to the local Java Application running without any JMX arguments. The problem that I was facing was machine specific. I restarted the machine and deleted the following directory:
%TMP%\hsperfdata_User.Name
(I was not able to delete this directory without doing a restart)
Restarted JConsole/JVisualVM and was able to connect to local java processes using the process id.
In fact, the use of JVisualVM pointed me towards this fix. On start of JVisualVM, I got an error message stating something like local processes/applications can't be monitored. The message had a link to Troubleshooting guide. I am reproducing the relevant snippet:
Local Applications Cannot Be Monitored (Error Dialog On Startup)
Description: An error dialog saying that local applications cannot be monitored is shown >immediately after VisualVM startup. Locally running Java applications are displayed as Application> (pid ###).
Resolution: This can happen on Windows systems if the username contains capitalized letters. In >this case, username is UserName but the jvmstat directory created by JDK is >%TMP%\hsperfdata_username. To workaround the problem, exit all Java applications, delete the >%TMP%\hsperfdata_username directory and create new %TMP%\hsperfdata_UserName directory.
However, on my machine directory had the following format:
%TMP%\hsperfdata_User.Name
So, my recommendations are:
-check the name of the above mentioned directory for presence of camel case user name
-If not, follow the steps from the Trouble Shooting guide
-If the problem persists, delete the directory (may require machine reboot, as in my case)
-Restart JConsole/JVisualVM
Hopefully, the problem would be resolved.
Thanks and Regards

Visual GC in Java VisualVM via jstadt for remote Tomcat

I'm trying to get the memory layout of a remote Tomcat visualized with Visual GC in Java VisualVM.
On the Linux server which hosts the Tomcat jstatd is running (including tools.policy and the proper [-J]-Djava.rmi.server.hostname=[SERVER_IP], which is to be found in the jvm options of the Tomcat as well).
The connection from VisualVM to the server/jstatd works properly:
But all i can see in the Visual GC tab of the Jstatd connection are the memory details of the Jstatd JVM itself, not the data of the Tomcat.
Edit: And in the Visual GC tab of the Tomcat connection (prod apl01 jmx) I'm being told: "Not supported for this JVM"
Testing this with a system we run inhouse I get proper data here.
The remote Tomcat runs with it's own user "tomcat".
Must jstatd be started by this user as well?
Or is there another issue I might be missing?
Any working alternative like another visual tool that provides a visualization similar to the one below would be absolutely acceptable.
We're using Yourkit as well, but I couldn't find anything in there that resembles Visual GC easy to understand style.

Having done some more research and tests on our locale development environment i realized that we haven't had the right permissions on the production environment.
So i asked the admin to add jstatd to the sudoers list and voilà it works as expected:
sudo /usr/java/latest/bin/jstatd -J-Djava.security.policy=/home/empulse/tools.policy -J-Djava.rmi.server.hostname=[SERVER_IP]

On remote server create policy,
vim /tmp/tools.policy
Add Permission,
grant codebase "file:${java.home}/../lib/tools.jar" {
permission java.security.AllPermission;
};
And start jstatsd,
jstatd -J-Djava.security.policy=/tmp/tools.policy -J-Djava.rmi.server.hostname=<public_ip>
and you ready are with Visual GC inside VisualVM on local machine.

Monitor Java application with VisualVM

I have a few Java programs running on my EC2 instance. I want to profile them using VisualVM. they are not web applications that run on Jetty or Tomcat. I did go through the stuff mentioned here, but I dont know how to set up my VisualVM after I generate the jar files with those commands. Can some help me out?
Thanks

You normally attach VisualVM to the PID of the process you want to profile. If that's Jetty or Tomcat or some other Java EE app server, that means the PID of the app server. If not, it's the PID of the JVM that's running your app.
If you've already got a JVM installed on your EC2 instance, I'd recommend looking in the JVM /bin folder to see if jvisualvm.exe is already there. If it is, fire it up in a separate command shell and attach it the PID of your application.

How to remotely profile a web application with JProfiler?

I am running a web application that is deployed on remote machine server and I have the IP address and URL of this application. When I hit the URL with a browser, the application displays.
I now have to profile that web application, as I need to find out why it is running so slow.
I tried using JProfiler, have not been successful. Could someone please advise how to configure JProfiler for remote profiling?

The easiest way to profile a remote JVM on a system without a GUI is this:
Extract the JProfiler archive (not the installer) somewhere on the remote machine
In the installation directory call bin/jpenable and select the JVM you want to profile
On your local machine start JProfiler and create a new session of type "Attach to profiled JVM (local or remote)"
In the session configuration, enter the host name and the port as given by jpenable
Start the session and profile
Tip: To find a bottleneck, use sampling not instrumentation.
Update for JProfiler 10.0+
Since JProfiler 10.0, there is a remote attach feature that does not require any of the above steps, you just need SSH credentials to the remote machine.
SSH connections are made directly by JProfiler, you don't have to set up the SSH tunnel yourself. It's also possible to configure multi-hop tunnels.
JProfiler will automatically download the required agent package, upload it to the remote machine and use its command line tools to gather the information that you see in the attach dialog. The agent package is cached, so this is only done once.
Because you have to authenticate as the same user that has started the JVM that you want to profile, it is possible to switch the user for the remote attach. For example, you can sudo to the root user to attach to a service that was started as root.
All the JVMs started by the selected user are shown and you can either start a full profiling session or just take a low-overhead HPROF heap dump and open it in JProfiler.