Monitoring JMX via the command line in a script - java

I would like to monitor JMX applictions via the command line between 2 centos boxes, ideally as part of a python (jython?) script so it can be automated
I've looked at applications like jmxterm, jmxbox and nagios-jmx but the problem I have is it doesn't seem to work! Each jar just sits there trying to connect and I just close it because nothing seems to happen
I've setup a vanilla tomcat5 which is listening to jmx via the following runtime options
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=7009
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=true
-Djava.rmi.server.hostname=IP_ADDRESS
-Dcom.sun.management.jmxremote.password.file=/etc/tomcat5/jmxremote.password
-Dcom.sun.management.jmxremote.access.file=/etc/tomcat5/jmxremote.access
-Dcom.sun.management.jmxremote.local.only=false
I've checked to ensure tomcat is listening on the ports I have set and I have also checked I can telnet between the 2 boxes and all firewalls are off and have managed to connect via jconsole (from a windows machine) which outputs data and graphs so it looks like everything is working.
Can anyone show me how to monitor tomcat JMX via the command line?

For anyone interested in how I achieved this, I used command line jmx and a custom python script using popen to query the beans http://crawler.archive.org/cmdline-jmxclient/
p.s I also got caught out by the jmx random port selector problem!

Related

Profiling tomcat application with VisualVM

I am trying to run visualvm under the username tomcat6 because apparently visualvm can only find applications running under its username. So by default it is only finding applications running under my username. I have been able to connect visualvm with tomcat6 through jmx but that lacks the fine granularity of instrumented profiling.
I tried the following to run visualvm under the username tomcat6 but got the following error that I don't understand.
$ sudo -u tomcat6 jvisualvm
No protocol specified
Exception in thread "main" java.awt.AWTError: Can't connect to X11 window server using ':0' as the value of the DISPLAY variable.
at sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)
at sun.awt.X11GraphicsEnvironment.access$200(X11GraphicsEnvironment.java:65)
at sun.awt.X11GraphicsEnvironment$1.run(X11GraphicsEnvironment.java:115)
...
If the computer running your application is remote -- like a server -- then you can't run GUI applications without some work. It's probably going to be easier to enable remote access to VisualVM.
You can use two techniques to attach to a remote JVM: using jstatd or using JMX. I'm not sure what you think you are losing by using JMX, but evidently jstatd doesn't give you access to profiling tools, CPU monitor, etc.).
You need to configure your JVM and Tomcat to allow for remote access. That requires 3 steps:
Enable remote JMX. Turns out, there's a guide for that.
Fix the "wandering port" used for RMI. There's a guide for that, too.
(Optional) Arrange for secure remote-access to the server. The easiest way to do that would be to use ssh -Lport:localhost:port with a series of -L arguments to forward multiple ports from your workstation to your server. Map all the ports you had to configure in steps #1 and #2. If you don't do this, you'll need to have non-firewalled access to all the aforementioned ports.
Restart your JVM and connect with JVisualVM.
Update 2022-06-01
Note that the "wandering port" problem has been fixed at the JVM level, so there is no need for application (i.e. Tomcat) support for that. Item #1 for Tomcat 8.5 and later contains updated instructions making item #2 unnecessary with a recent JVM.
Unfortunately only sampling is available in remote mode so JMX will lack instrumentation tools.
Actually your approach to running visualvm under tomcat6 user is correct. You should take a look at this question on how to run X11 applications under sudo.
The easiest way to pass DISPLAY and XAUTHORITY environment variables is to use sudo -E command to preserve current user environment.
Also if you can't see your process under tomcat6 user you should check if CATALINA_TMPDIR is pointing to /tmp. Otherwise you should pass it to visualvm
jvisualvm -J-Djava.io.tmpdir="${CATALINA_TMPDIR}"
Actually there is a lot of alternatives like yourkit or jprofiler shipped with java agents which allows remote instrumentation profiling.
The easiest is to open a remote JXM port on Tomcat in order to be able to remotely (from your desktop computer) connect to your remote Tomcat (on your server) with jvisualvm.
You need to pass the following system properties to your JVM :
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=<whatever_port_you_want>
-Dcom.sun.management.jmxremote.ssl=false
Then open jvisualvm on your local computer (JVM version needs to be the same or newer), File -> Add Remote Host -> Enter the name on the Host. It will create an entry for this host. Right lick on this Entry -> Add JMX connection -> Enter the port -> OK
jvisualvm will then be able to access remotely to your application.
You can also secure the connection if needed by using the following system properties (you need to create the files and locate them where you want :
-Dcom.sun.management.jmxremote.password.file=jmxremote.password
-Dcom.sun.management.jmxremote.access.file=jmxremote.access
These properties needs to be added to the CATALINA_OPTS environment variable. Fr exemple :
export CATALINA_OPTS = "$CATALINA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8888 "
More info on JMX lies here : https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html
I tried to do something similar but I was not allowed to install JVisualVM on the server. Having JVisualVM connect to the remote machine never seemed to work correctly. I suspect firewall rules were blocking part of the the network connections.
The only way I found to remotely profile the server was via an ssh tunnel.
Set the JMX port in CATALINA_OPTS on the server
CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote.port=13333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false; export CATALINA_OPTS
On your desktop machine open an ssh connection to the server
ssh -D 61444 username#tomcat.server.address
Add a flag to JVisualVM so that it will proxy its network connection
"C:\Program Files\Java\jdk1.7.0_79\bin\jvisualvm.exe" -J-Dnetbeans.system_socks_proxy=localhost:61444 -J-Djava.net.useSystemProxies=true
Have JVisualVM connect to the jmxport and the network traffic is tunneled via ssh.
Good luck.

Deploying a Play Framework application on port 80 beside Apache

I'm looking for a way to deploy my Play-Framework-1.0 application on the port 80.
So first I made the zip file with 'dist' command, then I unzipped it.
When I run the command to lauch the application (play-java-1.0-SNAPSHOT/bin/play-java -Dhttp.port=80 -Dhttp.adresse=127.0.0.1), I get this error :
[error] p.c.s.NettyServer - Failed to listen for HTTP on /0.0.0.0:80!
Oops, cannot start the server.
play.core.server.ServerListenException: Failed to listen for HTTP on /0.0.0.0:80!
at play.core.server.NettyServer.play$core$server$NettyServer$$bindChannel(NettyServer.scala:215)
at play.core.server.NettyServer$$anonfun$1.apply(NettyServer.scala:203)
at play.core.server.NettyServer$$anonfun$1.apply(NettyServer.scala:203)
at scala.Option.map(Option.scala:146)
at play.core.server.NettyServer.<init>(NettyServer.scala:203)
at play.core.server.NettyServerProvider.createServer(NettyServer.scala:266)
at play.core.server.NettyServerProvider.createServer(NettyServer.scala:265)
at play.core.server.ServerProvider$class.createServer(ServerProvider.scala:25)
at play.core.server.NettyServerProvider.createServer(NettyServer.scala:265)
at play.core.server.ProdServerStart$.start(ProdServerStart.scala:53)
at play.core.server.ProdServerStart$.main(ProdServerStart.scala:22)
at play.core.server.ProdServerStart.main(ProdServerStart.scala)
Moreover, in the real server, Apache has been installed. So I wonder, whether that will be a problem.
Thanks!
Optionally, also remember that on most systems, running processes on ports lower than 8000 is disabled in default, in such case you need to allow it, i.e. on Unix servers, just using sudo command(prefix).
If you are using a Linux server, you can try 'fuser 80/tcp' to see whether another process is already running on that port (80). If so (there's showing a process-id, when you enter the command), you cannot use the same port for 2 processes.
Either, you have to start the Play-app in a different port or you can kill the already running process by 'sudo fuser -k 80/tcp' and start the Play-app on the same port (80).
It's not possible to have two processes running on the same host listening on the same port.
However, you could run you Play application on different port, e.g. 8080 and set up Apache as a reverse proxy (Nginx would do too, but you mentioned that you already have Apache running on the server) to forward requests to your Play application.
Example guide how to do that:
https://www.digitalocean.com/community/tutorials/how-to-use-apache-http-server-as-reverse-proxy-using-mod_proxy-extension

Tomcat debugging not possible through eclipse

I've made a script called debug.sh and placed it under the bin directory (start it with ./debug.sh) to start Tomcat 8 in debugging mode:
set JPDA_ADDRESS=8000
set JPDA_TRANSPORT=dt_socket
set JAVA_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n
bash catalina.sh jpda start
But if it started, there is now message which says, that Tomcat is listening on port 8000. Also if I type
netstat -nat
there is no application listening on port 8000.
What exact configuration do I have to set, to remote debug my Tomcat 8 server which is running on a specific IP or do I have just a little problem in my script?
This answer has been updated following the comments, I did not understand the problem in the first place.
I guess you have followed that doc: this is about developing Tomcat itself.
I am not sure you are using the proper way to configure the port (I don't know your specific configuration details). In a standard environment, the ports are configured in the server.xml (note that several different ports are used by Tomcat for the different services).
To remotely monitor your server, you should use a JMX client. As far as I know, Eclipse doesn't include one (or at least not one documented) - you could code one as this is a Java specification (JSR262). You have one in a standard Java environement (JConsole). By default, JMX is not enabled on Tomcat. If you need to enable it, the fine way is to follow the doc.
Tomcat JMX monitoring and JConsole are both available in most versions of Tomcat and the Java runtime.
The following exchange seems to be about your problem.

How to test JBoss server is started from external java application

how can i test from external java application that my server jboss is running ?
I've a JBoss (4.2.3) server and I want know from a stand-alone java application if that server i started or not.
Thanks!
EDIT
I don't have access to the jboss machine and the jmx console is disabled for safety reasons.
You can check inspecting the running processes if you have access to the machine where jboss is running.
If you don't have access to the machine, then you'll have to try to connect to it, checking if it's listening to the http port or if you can reach it via JMX, but then you can't be sure if it's really not running or if some firewall rule is blocking your request.
One of the possible solution if you are running on linux is to execute a shell command like
ps -ef | grep jboss >> somelog.txt
execute it using Runtime class using exec() method in Runtime and check the output of that command from your java program
Surely there might be some other better alternative , but this is just a simple thought

Couldnt connect to JMX remote with jconsole

I developing under Spring3.1 standalone env.
I am trying to connect my application remotely via jconsole.
It's working locally but when I deploy my application into the linux machine it gets time out.
I am using Daemon in order to run my environment.
this is what I add in the run.sh script:
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=6969 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
com.mypackage.daemon.FixDaemon
and inside applicationContext.xml:
<context:mbean-server />
<context:mbean-export />
now on the linux machine after doing netstat thats what we see:
[root# logs]# netstat -an | grep 6969
tcp 0 0 :::6969 :::* LISTEN
so it's seems like it does listening.
but when I add my ip:6969 inside the jconsole interface I get connection failed popup.
any idea what am I doing wrong?
thanks,
ray.
First try to add also this option to your application:
-Djava.rmi.server.hostname=<ip>
Also keep in mind jconsole is using RMI for the communication. This means jconsole first connects to ip:6969. Then server generates a random port X which is passed back to the jconsole. Jconsole then opens another connection to ip:X. Since X is random, there is no way you can open this specific port in the firewall. You have either to open all ports or use a socks proxy which is another subject.
Try connecting to that port using telnet from your machine. If this does not succeed it usually is because of a firewall dropping packets. You will have to talk to your network administrator to open up that port.
Note: You will have to open up two ports. One for binding the RMIRegistry and another one to export the RMI objects. RMI usually exports objects on random high ports. But this will not work in a firewall-ed environment hence you would have to configure the port on which it is exported. This is done by using a RMI URL.
If you are running this on Linux then do a hostname -i, if it returns 127.0.0.1 then fix /etc/hosts. The FAQ entry for JConsole has more information on this.
Another option I would strongly suggest is to look at Jolokia which does not involve changing the firewall configuration but still provides the JMX goodies over HTTP.

Categories