How to use VisualVM and JMX? - java

I have tried every recipe in the book, but things just wont work today...
I am trying to use VisualVM to profile my Java app running in a remote server, so I googled and googled and googled for ways to do this and I end up with this solution:
java -Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9199 \
-Dcom.sun.management.jmxremote.local.only=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-jar bin/felix.jar
Running this will get me JMX to run I guess, so I can see him running on 9199:
java 21947 root 9u IPv6 1811434 0t0 TCP *:7192 (LISTEN)
java 26376 root 14u IPv6 1844518 0t0 TCP *:9199 (LISTEN)
Moreover, I can test that the port is visible on the web:
Now, when I try to open the connection in VisualVM in my machine (which can also ping the jmx server) this happens:
I must be so dumb, that everyone in this world was able to put this thing running but me. Damn...
EDIT: I installed wireshark to know what's going on under the hood, so I saw this. As soon as I add a remote IP in VisualVM wireshark starts detecting this repetitive activity:
However, if I try to add a new JMX connection and ask him to connect, wireshark will not grab even one single packet of that connection attempt.
This seems to me that VisualVM is not even trying to connect, despite giving the error message "Cannot connect"...!! What the hell is going on?

Start your service using these options:
-Djava.rmi.server.hostname=193.163.XXX.XXX
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9199
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
Then you can use jvisualvm or jconsole with just the "193.163.XXX.XXX:9199" address.

Follow the below steps.
1) Go to remote machine open X11 forwarding,
2) Installed XMING in windows
3) Start visual VM in Remote machine the GUI will come in your windows xming.
4) Instead of IP use localhost and use same port number.
If above steps not working it means something is wrong with configuration.
If above steps working fine then go to iptables and ensure port 9199 open for outer world.
as well start the program with java -Djava.rmi.server.hostname=YOUR_IP
For more details refer this link

Please use the following JVM options :
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=<PORT> -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=<IP>
In the VisualVM use the following to connect:
service:jmx:rmi:///jndi/rmi:/<IP>:<PORT>/jmxrmi
Hopefully this will help.

Related

Unable to open debugger port through IntelliJ

I've got a server running on DigitalOcean and a JAR file that I want to debug. I first start the JAR on the remote server using
java -jar Server.jar -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
but on the console I see no output like "listening on port 5005...".
When I press debug in IntelliJ it says
unable to open debugger port (198.xxx.xxx.xx:5005):
java.net.ConnectException "Connection refused"
This is my IntelliJ configuration:
I also tried using -Xdebug but it still didn't work.
If I set suspend=y it should wait until a debugger is connected, but instead, it starts without problems.
The command to start the remote Java process in debug mode looks correct. If you don't see "Listening to Port blah" when you start the server JAR, then it might mean that the debug args are not being picked up. Another way to quickly check this would be to test with a telnet localhost 5005 on the machine where the server JAR is being executed. The telnet will fail if that port is not being used.
I suggest that you try the following, since the order of the parameters might be significant (I'll add some official evidence for this later):
java "agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" -jar Server.jar
this command worked for me:
export JAVA_OPTS='-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=0.0.0.0:5005'
by default idea remote dialog suggest:
'agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005'
change it to:
'agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=0.0.0.0:5005'
and issues port 5005.
This command worked for me:
java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 -jar Server.jar
Thanks to suifengpiao14 but I'm going to describe the problem a bit more in detail.
I checked multiple things, and at the end found the reason: actually as like as a restful service we want to be accessible from out of the server we are running it that we should set 0.0.0.0 as the nameserver, here we should do a similar one.
I checked the difference between the server from which I can remotely debug and the one which I can't. using netstat command:
for the server which I was ok with:
tcp 0 0 0.0.0.0:5005 0.0.0.0:* LISTEN 8323/java
for the server which I had problem with:
tcp 0 0 127.0.0.1:5005 0.0.0.0:* LISTEN 8323/java
So, using below jvm options should be helpful:
'agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005'
'agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=0.0.0.0:5005'
In my case it was because under settings -> build, execution, deployment -> debugger I had a built in server running on the same port as which I was trying to attach my debugger to for some reason.
For people like me who sometimes forget to read...
Copy and paste the arguments to the command line when JVM is started
says the Run/Debug Configuration in IntelliJ, directly under:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
Means:
Copy this line and go to your docker configuration. Add an environment variable (modify options dropdown). Paste it there with JAVA_OPTS= prepended.
Now when you did every correctly, you will have
Listening for transport dt_socket at address: 5005
and
Command line argument: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
This is who I solved it...
I've reproduced similar issue:
Unable to open debugger port (localhost:5005): java.net.ConnectException "Connection refused (Connection refused)"
I had it while running debugger w/ command line arguments for remote JVM using Run/Debug configurations:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
by attaching IDE with exposed debug agent server.
Since I've used Kubernetes cluster, the reason was:
once the pod has been created and was running as expected, I needed to proceed to set up port forwarding with the kubectl port-forward command, with the port to expose locally for traffic from the application before running debugger.
Based on kubectl port-forward syntax:
kubectl port-forward <resource-type/resource-name> [local_port]:<pod_port>
In format like:
kubectl port-forward <pod-name> -n <namespace> 5005:5005
or in a shorter form:
kubectl port-forward <pod-name> 5005:5005
The client listens on port 5000 locally and forwards to 5000 in the
pod.
Accordingly entrypoint.sh file of the service was configured w/ the following command for Kubernetes cluster:
java -Djava.security.egd.=file:/dev/./urandom -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -Xmx400m -jar /service.jar
As result, running port-forward before debugger solved the issue.
One of the reason is that the port is not enabled. You can try by enabling the port using the below command
firewall-cmd --zone=public --permanent --add-port=8001/tcp
Once the port is up, restart tomcat & try connecting again.
Hope it helps.
This might help someone That port in JVM debug is not your web app port

Monitoing of remote JVM using Jconsole

I'm trying to monitor remote jvm using Jconsole.
jdk1.7.0_75 is installed and configured the below parameter in jre/lib/management/management.properties file on remote machine.
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8002
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=x.x.x.x
Getting Connection failed: connection refused.
checked the port number 8002 is free and disabled the firewall, Kindly provide the solution.
You are setting the right properties, but mixing two different approaches here. To enable JMX on your application you either need to:
start your application with these command line parameters you used above (java -Dcom.sun.management.jmxremote.port=8002 -cp somedependency.jar Appplication)
add similar entries to your management.properties BUT WITHOUT the "-D" prefixes. So entries like: com.sun.management.jmxremote.port=8002

Jconsole Remote Executable Jar File

All,
I have a remote server that I recently enabled VNC for using vnc4server and Chicken for mac as the client.
The purpose for doing so was to enable running Java's Jconsole to monitor an executable jar file that is running my server logic.
However, after logging into my server using VNC, I keep getting an error when I try to use Jconsole on vnc.
It states connection failed do you want to try again. Now I am logged in as the same user that started the process.
Is there something I am missing when using jconsole in VNC? Also can I monitor my executable jar file remotely using Jconsole on my local machine?
These are the options I am including to run the jar file: java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9005 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.remote.ssl=false -Djava.rmi.server.hostname=ipaddress -jar path
Thanks
These JVM options fixed things. Fix found here: You need to pass to the VM: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false
https://forums.oracle.com/thread/1177644
This does not seem like an VNC issue- either the ports are not open, they are being blocked b a firewall, or there is some kind of permission/authentication issue with the app itself related to monitoring it.
In order to eliminate VNC as the cause (and use localhost in a local connection on jconsole), do "ssh -X REMOTHOST -n jconsole" and see. This will also eliminate the overhead of running the full X server and VNC.
Also on linux you can find out what process holds a port open by doing:
netstat -ap | grep PORT_NUMBER on the remote host you want to run on.
Colin

JVM remote profiling with JVisualVM

I am trying to profile my jvm applications on a remote host.
I am using Jvisualvm
I setup jstatd as recomended in this (link removed, leads to phishing website).
I am able to "telnet remotehost 1099" successfully. All the tcp/ip ports are firewall open on the remote.
I added my remote host in jvisualvm. Also, made sure the jstatd port is the 1099 with a refresh of 3 secs.
I started my application with
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8011
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
But, my jvm applications dont show up in my jvisualvm. Is there anything else I need to configure or check ?
jvisualvm 1.3.5
local: Mac OSX Oracle jdk 1.6.37
remote: centos 6 oracle jdk 1.6.24
1) You should start your application with JMX enabled
Example for remote connection
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9990
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
2) You should have JMX classes to be able to do something with them. I suppose you are aware of this
You need to start jstatd on the remote machine. For this purpose create a jstatd.all.policy file in the home directory of your remote machine and add the following lines:
grant codebase "file:${java.home}/../lib/tools.jar" {
permission java.security.AllPermission;
};
Then on the command line of your remote machine you will type
jstatd -J-Djava.security.policy=jstatd.all.policy -J-Djava.rmi.server.hostname={Your IP address} '
Once jstatd service start on the remote machine you basically add the remote connection IP address connection on the jvisualVM UI using add remote host.
The Oracle documents for JvisualVM can be referred at https://docs.oracle.com/javase/8/docs/technotes/guides/visualvm/applications_remote.html but it is really confusing to understand jstatd steps.

Connecting remote tomcat JMX instance using jConsole

I am trying to connect to a remote tomcat JMX instance using jConsole. But can't connect successfully. Any Idea?
I included the following option in remote tomcat catalina.sh:
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9004 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false"
I had a similar, if not the same, problem. I could connect to the JMX server if I started jconsole locally on the machine.
It appears the RMI server was not listening on the correct ip. So, as was suggested in this related question, I added the following:
-Djava.rmi.server.hostname=<host ip>
to JAVA_OPTS as well, and then it worked.
I've collected information spread over the net, found with hints from other members.
Most pain caused by JMX is (imo) the fact that JMX opens a second dynamically allocated network port. A firewall (like iptables) will block this.
Solution for tomcat on linux :
use tomcat 6.0.24 or newer
download catalina-jmx-remote.jar from apache tomcat extras (use browse on tomcat download page)
copy it in the $CTALINA_HOME\lib
This allows you to set both ports used by JMX
edit Server section in your server.xml
<Server port="8005" ..>
...
<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="9840" rmiServerPortPlatform="9841"/>
set some environment variables (e.g. in setenv.sh)
CATALINA_OPTS="
-Djava.rmi.server.hostname=IP-TO-LISTEN
-Dcom.sun.management.jmxremote.password.file=$CATALINA_BASE/conf/jmxremote.password
-Dcom.sun.management.jmxremote.access.file=$CATALINA_BASE/conf/jmxremote.access
-Dcom.sun.management.jmxremote.ssl=false"
this activates access control for JMX
jmxremote.access will look like
monitorRole readonly
controlRole readwrite
end jmxremote.password will be
monitorRole tomcat
controlRole tomcat
(just simple spaces)
restart tomcat.
Now configure firewall on the server (e.g. iptables)
/etc/sysconfig/iptables
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 9840 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 9841 -j ACCEPT
and /etc/sysconfig/ip6tables
-A RH-Firewall-1-INPUT -m tcp -p tcp --dport 9840 -j ACCEPT
-A RH-Firewall-1-INPUT -m tcp -p tcp --dport 9841 -j ACCEPT
restart iptables
Done!
Now use VisualVM or JConsole on your workstation to establish a connection to rmiRegistryPortPlatform, 9840 in our sample.
If there are no more firewalls between workstation and server it should work.
Tried with Java 8
1. Add this to your java tomcat startup script:
-Dcom.sun.management.jmxremote.port=1616
-Dcom.sun.management.jmxremote.rmi.port=1616
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.local.only=false
-Djava.rmi.server.hostname=localhost
for example add into bin/setenv.sh this:
export CATALINA_OPTS="$CATALINA_OPTS \
-Dcom.sun.management.jmxremote.port=1616 \
-Dcom.sun.management.jmxremote.rmi.port=1616 \
-Dcom.sun.management.jmxremote.local.only=true \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false "
2. Execute this on your computer.
Windows users:
putty.exe -ssh user#remote-host -L 1616:remote-host:1616
Linux and Mac Users:
ssh user#remote-host -L 1616:remote-host:1616
3. Start jconsole on your computer
jconsole localhost:1616
4. Have fun!
P.S.: during step 2, using ssh and -L you specify that the port 1616 on the local (client) host is to be forwarded to the remote side.
P.S.2.: you can specify same port for JMX and RMI conversations
what string are you using as the JMX connection url. I don't mean to point out the obvious but JConsole has a terrible interface and to me requires an overly complex url before it will connect to a remote jmx app. Mine looks like this:
service:jmx:rmi:///jndi/rmi://(hostname):(jmxport)/jmxrmi
Enable JMX in Tomcat8, successfully tested in my POC
1/ Download the catalina-jmx-remote.jar from apache website and place in $CATALINA_HOME/lib.
2/ Take server.xml / setenv.sh backup. Make the changes to server.xml like below-
<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002" />
3/ Make the changes to $CATALINA_BASE/bin/setenv.sh like -
[...]
JVM_OPTS="[...]
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=true
-Djava.rmi.server.hostname=<eth:0_IP>| <`hostname -i`>
-Dcom.sun.management.jmxremote.password.file=/apps/data/apache-tomcat-8_8080/conf/jmxremote.password
-Dcom.sun.management.jmxremote.access.file=/apps/data/apache-tomcat-8_8080/conf/jmxremote.access
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote=true "
4/ Create these two files as -
$touch $CATALINA_BASE/conf/jmxremote.password containing:
admin letmein
$touch $CATALINA_BASE/conf/jmxremote.access containing:
admin readwrite
$ chmod 600 jmxremote.password
5/ Restart tomcat and test on jconsole tool :)
$echo|telnet 10.105.14.90 10001
What exactly do you mean when you say "But can't connect successfully."? Is there an error message? Try turning on logging in jconsole and see if that helps debug it.
To turn on jconsole logging, edit a file named logging.properties in the directory you will be running jconsole in, add:
handlers= java.util.logging.ConsoleHandler
.level=INFO
java.util.logging.FileHandler.pattern = %h/java%u.log
java.util.logging.FileHandler.limit = 50000
java.util.logging.FileHandler.count = 1
java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter
java.util.logging.ConsoleHandler.level = FINEST
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
javax.management.level=FINEST
javax.management.remote.level=FINEST
Then, start jconsole with:
jconsole -J-Djava.util.logging.config.file=logging.properties
if you are working on linux, modify the catalina.sh file adding:
CATALINA_OPTS="-Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=<HOST_IP> -Dcom.sun.management.jmxremote.port=<HOST_PORT> -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
export CATALINA_OPTS
or modify the /etc/profile file as root and rerun the file (source /etc/profile)
if you are working on windows and you are starting tomcat from the command line, use
the environment variable CATALINA_OPTS
if you are working on windows and you are starting tomcat as a service, you'll need to use the monitor service utility to configure the service initialization parameters (neither setenv.bat, catalina.bat or env-vars will work). for that you'll need the service name that appears listed in services.msc (for example jasperreportsTomcat). After, you'll need to open a console as administrator and execute (for example):
tomcat6w.exe //MS//jasperreportsTomcat
with this command will appear a tray icon where you can open a panel. In the "Java" tab now you can modify the jmx options. Be careful to not add trailing whitespaces and use the "[enter]" symbol to separate each option line by line.
-Dcom.sun.management.jmxremote
-Djava.rmi.server.hostname=192.168.61.101
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
Hope it helps
Check if your server is behind the firewall. JMX is base on RMI, which open two port when it start. One is the register port, default is 1099, and can be specified by the com.sun.management.jmxremote.port option. The other is for data communication, and is random, which is what cause problem. A good news is that, from JDK6, this random port can be specified by the com.sun.management.jmxremote.rmi.port option.
add the line in you {tomcat_dir}/bin/setenv.sh:
export CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8991 -Dcom.sun.management.jmxremote.rmi.port=8991 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
I got something for all of you, in order to complete the investigation of this whole thing. There is a trick, it happens that profiler tool connnects with the jvm using a port, but the jvm continues the conversation using another random port. If the jvm is running inside a remote machine (for example : a tomcat web-app server), and the remote machine has protection against outgoing and incoming connections, you must set the java system property com.sun.management.jmxremote.rmi.port to the same value of the property named com.sun.management.jmxremote.port
Source : https://serverfault.com/questions/308662/how-do-i-fix-a-failed-to-retrieve-rmiserver-stub-jmx-error
And also check this out : http://blog.cantremember.com/debugging-with-jconsole-jmx-ssh-tunnels/
Hope to contribute guys!
And good luck!
Well, I had this problem in a Linux box (virtual machine) and I fixed it using -Djava.rmi.server.hostname property but there's a thing I can't understand. My machine has 5 tomcat servers, all of them has jmx enabled in consecutive ports (8008,8018,8028...) and only one of them had this issue connecting JMX. No firewall, no -Djava.rmi.server.hostname property in any tomcat....
So the thing is that I understand the problem but I can't understand why 4 of my tomcats worked and 1 of them not.
P.D: My english is very poor, I know. My Apologies.
PROTIP: You need to fix (as in having a known number) the RMI Registry and JMX/RMI Server ports. You do this by putting jar-file in the lib-dir and configuring a special listener. (And ofcourse the usual flags for activating JMX
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=8999 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Djava.rmi.server.hostname=<HOSTNAME> \
See: JMX Remote Lifecycle Listener at http://tomcat.apache.org/tomcat-6.0-doc/config/listeners.html
Changing the /etc/hosts on linux, where I replaced the localhost address associated to my account to the machine ip, solved this problem for me.

Categories