We Keep Coding

Profiling tomcat application with VisualVM

I am trying to run visualvm under the username tomcat6 because apparently visualvm can only find applications running under its username. So by default it is only finding applications running under my username. I have been able to connect visualvm with tomcat6 through jmx but that lacks the fine granularity of instrumented profiling.
I tried the following to run visualvm under the username tomcat6 but got the following error that I don't understand.
$ sudo -u tomcat6 jvisualvm
No protocol specified
Exception in thread "main" java.awt.AWTError: Can't connect to X11 window server using ':0' as the value of the DISPLAY variable.
at sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)
at sun.awt.X11GraphicsEnvironment.access$200(X11GraphicsEnvironment.java:65)
at sun.awt.X11GraphicsEnvironment$1.run(X11GraphicsEnvironment.java:115)
...

If the computer running your application is remote -- like a server -- then you can't run GUI applications without some work. It's probably going to be easier to enable remote access to VisualVM.
You can use two techniques to attach to a remote JVM: using jstatd or using JMX. I'm not sure what you think you are losing by using JMX, but evidently jstatd doesn't give you access to profiling tools, CPU monitor, etc.).
You need to configure your JVM and Tomcat to allow for remote access. That requires 3 steps:
Enable remote JMX. Turns out, there's a guide for that.
Fix the "wandering port" used for RMI. There's a guide for that, too.
(Optional) Arrange for secure remote-access to the server. The easiest way to do that would be to use ssh -Lport:localhost:port with a series of -L arguments to forward multiple ports from your workstation to your server. Map all the ports you had to configure in steps #1 and #2. If you don't do this, you'll need to have non-firewalled access to all the aforementioned ports.
Restart your JVM and connect with JVisualVM.
Update 2022-06-01
Note that the "wandering port" problem has been fixed at the JVM level, so there is no need for application (i.e. Tomcat) support for that. Item #1 for Tomcat 8.5 and later contains updated instructions making item #2 unnecessary with a recent JVM.

Unfortunately only sampling is available in remote mode so JMX will lack instrumentation tools.
Actually your approach to running visualvm under tomcat6 user is correct. You should take a look at this question on how to run X11 applications under sudo.
The easiest way to pass DISPLAY and XAUTHORITY environment variables is to use sudo -E command to preserve current user environment.
Also if you can't see your process under tomcat6 user you should check if CATALINA_TMPDIR is pointing to /tmp. Otherwise you should pass it to visualvm
jvisualvm -J-Djava.io.tmpdir="${CATALINA_TMPDIR}"
Actually there is a lot of alternatives like yourkit or jprofiler shipped with java agents which allows remote instrumentation profiling.

The easiest is to open a remote JXM port on Tomcat in order to be able to remotely (from your desktop computer) connect to your remote Tomcat (on your server) with jvisualvm.
You need to pass the following system properties to your JVM :
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=<whatever_port_you_want>
-Dcom.sun.management.jmxremote.ssl=false
Then open jvisualvm on your local computer (JVM version needs to be the same or newer), File -> Add Remote Host -> Enter the name on the Host. It will create an entry for this host. Right lick on this Entry -> Add JMX connection -> Enter the port -> OK
jvisualvm will then be able to access remotely to your application.
You can also secure the connection if needed by using the following system properties (you need to create the files and locate them where you want :
-Dcom.sun.management.jmxremote.password.file=jmxremote.password
-Dcom.sun.management.jmxremote.access.file=jmxremote.access
These properties needs to be added to the CATALINA_OPTS environment variable. Fr exemple :
export CATALINA_OPTS = "$CATALINA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8888 "
More info on JMX lies here : https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html

I tried to do something similar but I was not allowed to install JVisualVM on the server. Having JVisualVM connect to the remote machine never seemed to work correctly. I suspect firewall rules were blocking part of the the network connections.
The only way I found to remotely profile the server was via an ssh tunnel.
Set the JMX port in CATALINA_OPTS on the server
CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote.port=13333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false; export CATALINA_OPTS
On your desktop machine open an ssh connection to the server
ssh -D 61444 username#tomcat.server.address
Add a flag to JVisualVM so that it will proxy its network connection
"C:\Program Files\Java\jdk1.7.0_79\bin\jvisualvm.exe" -J-Dnetbeans.system_socks_proxy=localhost:61444 -J-Djava.net.useSystemProxies=true
Have JVisualVM connect to the jmxport and the network traffic is tunneled via ssh.
Good luck.

Monitoring JMX via the command line in a script

I would like to monitor JMX applictions via the command line between 2 centos boxes, ideally as part of a python (jython?) script so it can be automated
I've looked at applications like jmxterm, jmxbox and nagios-jmx but the problem I have is it doesn't seem to work! Each jar just sits there trying to connect and I just close it because nothing seems to happen
I've setup a vanilla tomcat5 which is listening to jmx via the following runtime options
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=7009
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=true
-Djava.rmi.server.hostname=IP_ADDRESS
-Dcom.sun.management.jmxremote.password.file=/etc/tomcat5/jmxremote.password
-Dcom.sun.management.jmxremote.access.file=/etc/tomcat5/jmxremote.access
-Dcom.sun.management.jmxremote.local.only=false
I've checked to ensure tomcat is listening on the ports I have set and I have also checked I can telnet between the 2 boxes and all firewalls are off and have managed to connect via jconsole (from a windows machine) which outputs data and graphs so it looks like everything is working.
Can anyone show me how to monitor tomcat JMX via the command line?

For anyone interested in how I achieved this, I used command line jmx and a custom python script using popen to query the beans http://crawler.archive.org/cmdline-jmxclient/
p.s I also got caught out by the jmx random port selector problem!

Viewing MBeans through jstatd

I am trying to monitor all Java processes running on a server via jstatd. I've got it set up enough that I can connect with VisualVM and see all running processes. Most displays work fine, however certain things (especially CPU usage and MBeans) do not display. Instead, it says:
MBeans Browser
Data not available because JMX connection to the JMX agent could not be established.
I assumed that the problem was that the application must "announce" through the jstatd RMI registry rather than a local one, so I tried out the following (per these suggestions) but it still won't display. The code I tried is as follows:
public class JmxRmiConnectorTest {
public static void main(String[] args) throws Exception {
Registry rmiRegistry = LocateRegistry.createRegistry(9994);
String svc =
"service:jmx:rmi://localhost:9994/jndi/rmi://localhost:1099/connector";
MBeanServer mbeanServer = ManagementFactory.getPlatformMBeanServer();
JMXServiceURL url = new JMXServiceURL(svc);
RMIConnectorServer rmiServer = new RMIConnectorServer(url, null, mbeanServer);
rmiServer.start();
Thread.sleep(100000);
rmiServer.stop();
}
}
How can I get my MBeans and CPU usage to show up in VisualVM when seen through jstatd?

jstatd has nothing to do with JMX. Jstatd is a proxy for Jvmstat. To get MBeans and CPU usage you need to also enable JMX. See JMX Remote Monitoring and Management for more details. Once you have JMX enabled, VisualVM will automatically detect (via jvmstat) that it can also use JMX and it will display data from both jvmstat and JMX in one place.

In remote, launch java like this to enabl jmxremote.
java -Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9191 \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname=`hostname` \
-jar app.jar
In locale,use jps like thisjps YOUHOSTNAME:9191.

Just for those who need a quick working method.
JMX can be start/stop dynamically, that's without restart of JVM.
In my case, after start jmx dynamically and restart jvisualvm, JMX and cpu usage works fine.
jcmd PID ManagementAgent.start jmxremote.port=9999 jmxremote.ssl=false jmxremote.authenticate=false
or
jcmd PID ManagementAgent.stop

Why can't I attach jvisualvm to an instance of tomcat started from inside osx?

I'm starting tomcat from inside netbeans. I'd like to monitor the heap usage on that instance of tomcat so I fire up jVisualVM. However the process isn't listed. Any ideas?

Are you using Java version 6u24? Then you may fall victim to this bug which will be fixed in 6u25 (in a nutshell, jVisualVM can't find your process' hsperfdata).
Otherwise, you have to enable monitoring via JMX:
Pass the following JVM parameters to Tomcat:
-Dcom.sun.management.jmxremote.port=8888 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
Then add the remote process to jVisualVM via File -> Add JMX Connection. You can connect to the process using port 8888
jVisualVM has some other gotchas, e.g. the user starting jVisualVM needs to be the same as the user owning the process you want to monitor (although, I don't think this is your issue). See more here.

Can visualvm connect automatically via JMX to a remote process?

I have a Java process running on a remote machine, and the process sets up some mbeans. I also have jstatd running on that machine as the same user as the Java process. (The mbeans can be set up programmatically or using -Dcom.sun.management.jmxremote... etc, this doesn't appear to make a difference).
VisualVM is able to make a jstatd connection to the process, which it discovers automatically, but this means I don't get access to mbeans or, for example, the CPU history chart. Alternatively I can create an explicit JMX connection, which gives me the usual range of useful tools, but I want for the application to be assigned a random JMX port when it starts, this config can't be static.
Is there any way to get VisualVM to auto-connect to my process via JMX? This would require it to auto-discover the JMX ports, but I would have thought jstatd could do that. Does anyone know of any plugins for visualvm to automate this?

Unfortunately there is no way to assign random JMX port to the remote application. You can start your remote application with
-Dcom.sun.management.jmxremote.port=<fixed port>
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
and VisualVM will be able to read this configuration via Jvmstat (provide by jstatd) and open JMX connection to your remote application automatically. So you need to assign fixed port(s) to your remote application(s). Once you have it, everything will work fine and VisualVM will automatically connect to your application via JMX (in fact it will combine data from both Jvmstat and JMX).