I developing under Spring3.1 standalone env.
I am trying to connect my application remotely via jconsole.
It's working locally but when I deploy my application into the linux machine it gets time out.
I am using Daemon in order to run my environment.
this is what I add in the run.sh script:
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=6969 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
com.mypackage.daemon.FixDaemon
and inside applicationContext.xml:
<context:mbean-server />
<context:mbean-export />
now on the linux machine after doing netstat thats what we see:
[root# logs]# netstat -an | grep 6969
tcp 0 0 :::6969 :::* LISTEN
so it's seems like it does listening.
but when I add my ip:6969 inside the jconsole interface I get connection failed popup.
any idea what am I doing wrong?
thanks,
ray.
First try to add also this option to your application:
-Djava.rmi.server.hostname=<ip>
Also keep in mind jconsole is using RMI for the communication. This means jconsole first connects to ip:6969. Then server generates a random port X which is passed back to the jconsole. Jconsole then opens another connection to ip:X. Since X is random, there is no way you can open this specific port in the firewall. You have either to open all ports or use a socks proxy which is another subject.
Try connecting to that port using telnet from your machine. If this does not succeed it usually is because of a firewall dropping packets. You will have to talk to your network administrator to open up that port.
Note: You will have to open up two ports. One for binding the RMIRegistry and another one to export the RMI objects. RMI usually exports objects on random high ports. But this will not work in a firewall-ed environment hence you would have to configure the port on which it is exported. This is done by using a RMI URL.
If you are running this on Linux then do a hostname -i, if it returns 127.0.0.1 then fix /etc/hosts. The FAQ entry for JConsole has more information on this.
Another option I would strongly suggest is to look at Jolokia which does not involve changing the firewall configuration but still provides the JMX goodies over HTTP.
Related
I am trying to get some CPU sampling working on a remote Java-in-Docker process.
I've already looked at the related questions here, and tried everything, to no avail, so I'm posting my setup here.
I have a Java process (openjdk-8) running in a Docker container on a Google Compute Engine (GCE) instance. The GCE instance and container are both running Debian-9. I want to attach VisualVM or JConsole to my Java process.
I am able to run my docker container locally and connect with both visualvm and jconsole using localhost:9010.
I start the container in the VM startup script with:
docker run -d -p 9010:9010 <my container>
The Dockerfile also has:
EXPOSE 9010
The Java process, started by the Dockerfile CMD, has the following relevant args:
"-Dcom.sun.management.jmxremote", \
"-Dcom.sun.management.jmxremote.port=9010", \
"-Dcom.sun.management.jmxremote.rmi.port=9010", \
"-Dcom.sun.management.jmxremote.local.only=false", \
"-Dcom.sun.management.jmxremote.authenticate=false", \
"-Dcom.sun.management.jmxremote.ssl=false", \
I have opened up port 9010 in my gcloud firewall using:
gcloud compute firewall-rules create jmx-port --allow=tcp:9010,udp:9010
I have verified with netcat that the port is open and I can make a TCP connection to it.
I have other ports open from the same Docker container, with clients connecting successfully to those ports. They were exposed and mapped to the host ports the same way (-p port:port) and opened in the firewall the same way.
I am passing the external IP address of the GCE instance. For instance, if I do:
gcloud compute instances list
and it tells me:
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
my-server-b23j us-central1-d n1-standard-1 10.240.0.2 108.357.213.99 RUNNING
Then I will use the argument:
108.357.213.99:9010
as the remote jmx connection host:port pair.
VisualVM and JConsole both tell me they can't connect to the remote JMX service. In both cases, I decline the secure connection, and then they say:
Cannot connect to 108.357.213.99:9010 using
service:jmx:rmi:////jndi/rmi://108.357.213.99:9010/jmxrmi
In desperation, I added a firewall rule that enables TCP/UDP connections on all ports 0-65535, but it did not make a difference -- they still could not connect.
I've read that JMX-RMI opens up anonymous ports, and that you can (at least partly?) disable this behavior by specifying both:
"-Dcom.sun.management.jmxremote.port=9010", \
"-Dcom.sun.management.jmxremote.rmi.port=9010", \
However, it doesn't do the trick in my case.
I've read here that you need to specify the rmi server hostname:
-Djava.rmi.server.hostname='192.168.99.100'
but my server IP is ephemeral -- it is assigned by Google Compute Engine when I create the instance, and so I can't hardwire it into the Dockerfile with the rest of the Java args.
Am I going to have to get a static IP address to make this work?
One possible solution would be to ssh into your GCE box and port-forward port 9010. This can be done from the local console with:
gcloud compute ssh name-of-your-gce-engine -- -L 9010:localhost:9010
Then in jconsole or jvisualvm you connect to localhost:9010. Using localhost here means that jconsole/jvisualvm will connect to your local machine, this connect is tunneled by ssh into your GCE engine and there to the host and port defined in the -L argument, which is localhost:9010, but from the GCE-engine's view. Meaning that you will end up at your application.
You still have to set the rmi server name before starting your program, but you must use
-Djava.rmi.server.hostname='localhost'
so that RMI will tell jconsole/jvisualvm to use localhost and this will then resolve to your local tunneled endpoint. And of course you still need these:
"-Dcom.sun.management.jmxremote", \
"-Dcom.sun.management.jmxremote.port=9010", \
"-Dcom.sun.management.jmxremote.rmi.port=9010", \
"-Dcom.sun.management.jmxremote.local.only=false", \
"-Dcom.sun.management.jmxremote.authenticate=false", \
"-Dcom.sun.management.jmxremote.ssl=false", \
I am trying to run visualvm under the username tomcat6 because apparently visualvm can only find applications running under its username. So by default it is only finding applications running under my username. I have been able to connect visualvm with tomcat6 through jmx but that lacks the fine granularity of instrumented profiling.
I tried the following to run visualvm under the username tomcat6 but got the following error that I don't understand.
$ sudo -u tomcat6 jvisualvm
No protocol specified
Exception in thread "main" java.awt.AWTError: Can't connect to X11 window server using ':0' as the value of the DISPLAY variable.
at sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)
at sun.awt.X11GraphicsEnvironment.access$200(X11GraphicsEnvironment.java:65)
at sun.awt.X11GraphicsEnvironment$1.run(X11GraphicsEnvironment.java:115)
...
If the computer running your application is remote -- like a server -- then you can't run GUI applications without some work. It's probably going to be easier to enable remote access to VisualVM.
You can use two techniques to attach to a remote JVM: using jstatd or using JMX. I'm not sure what you think you are losing by using JMX, but evidently jstatd doesn't give you access to profiling tools, CPU monitor, etc.).
You need to configure your JVM and Tomcat to allow for remote access. That requires 3 steps:
Enable remote JMX. Turns out, there's a guide for that.
Fix the "wandering port" used for RMI. There's a guide for that, too.
(Optional) Arrange for secure remote-access to the server. The easiest way to do that would be to use ssh -Lport:localhost:port with a series of -L arguments to forward multiple ports from your workstation to your server. Map all the ports you had to configure in steps #1 and #2. If you don't do this, you'll need to have non-firewalled access to all the aforementioned ports.
Restart your JVM and connect with JVisualVM.
Update 2022-06-01
Note that the "wandering port" problem has been fixed at the JVM level, so there is no need for application (i.e. Tomcat) support for that. Item #1 for Tomcat 8.5 and later contains updated instructions making item #2 unnecessary with a recent JVM.
Unfortunately only sampling is available in remote mode so JMX will lack instrumentation tools.
Actually your approach to running visualvm under tomcat6 user is correct. You should take a look at this question on how to run X11 applications under sudo.
The easiest way to pass DISPLAY and XAUTHORITY environment variables is to use sudo -E command to preserve current user environment.
Also if you can't see your process under tomcat6 user you should check if CATALINA_TMPDIR is pointing to /tmp. Otherwise you should pass it to visualvm
jvisualvm -J-Djava.io.tmpdir="${CATALINA_TMPDIR}"
Actually there is a lot of alternatives like yourkit or jprofiler shipped with java agents which allows remote instrumentation profiling.
The easiest is to open a remote JXM port on Tomcat in order to be able to remotely (from your desktop computer) connect to your remote Tomcat (on your server) with jvisualvm.
You need to pass the following system properties to your JVM :
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=<whatever_port_you_want>
-Dcom.sun.management.jmxremote.ssl=false
Then open jvisualvm on your local computer (JVM version needs to be the same or newer), File -> Add Remote Host -> Enter the name on the Host. It will create an entry for this host. Right lick on this Entry -> Add JMX connection -> Enter the port -> OK
jvisualvm will then be able to access remotely to your application.
You can also secure the connection if needed by using the following system properties (you need to create the files and locate them where you want :
-Dcom.sun.management.jmxremote.password.file=jmxremote.password
-Dcom.sun.management.jmxremote.access.file=jmxremote.access
These properties needs to be added to the CATALINA_OPTS environment variable. Fr exemple :
export CATALINA_OPTS = "$CATALINA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8888 "
More info on JMX lies here : https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html
I tried to do something similar but I was not allowed to install JVisualVM on the server. Having JVisualVM connect to the remote machine never seemed to work correctly. I suspect firewall rules were blocking part of the the network connections.
The only way I found to remotely profile the server was via an ssh tunnel.
Set the JMX port in CATALINA_OPTS on the server
CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote.port=13333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false; export CATALINA_OPTS
On your desktop machine open an ssh connection to the server
ssh -D 61444 username#tomcat.server.address
Add a flag to JVisualVM so that it will proxy its network connection
"C:\Program Files\Java\jdk1.7.0_79\bin\jvisualvm.exe" -J-Dnetbeans.system_socks_proxy=localhost:61444 -J-Djava.net.useSystemProxies=true
Have JVisualVM connect to the jmxport and the network traffic is tunneled via ssh.
Good luck.
I'm attempting to connect to a remote JMX service using VisualVM 1.3.8 with JRE 1.8.0. Making exactly the same connection with JConsole works perfectly, however the attempt to use the same parameters with VVM just hangs at the 'Adding services' stage. I've looked in the VVM log but there is literally nothing logged.
I also read here at the VVM remote JMX doc:
Note: To retrieve and display information on applications running on the remote host, the jstatd utility needs to be running on the remote host.
That puzzles me since I can make the remote JMX connection using JConsole and as far as I can tell, jstatd is not running on the remote.
Does anyone know if jstatd a requirement only for VVM as a remote JMX client? As the JMX connection can be made with JConsole between the same client and server endpoints, then there's no problem with network/firewall etc.
Other than that I'm really puzzled where else to look for clues?
I had the same trouble when I switched from Java 7 to 8, while using SOCKS where I was updating proxy settings through VisualVM -> Tools -> Plugins -> Settings -> Proxy Settings. However, I have been successful with the following:
Run your JVM with the following options:
-Dcom.sun.management.jmxremote.port=<JMX_PORT>
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
Note that you could obviously do something more secure, both w.r.t SSL and authentication.
Setup a SOCKS proxy from your localhost to the remote server:
ssh -D<SOCKS_PORT> -vvv -N <REMOTE_HOST>
Run either of these commands on your localhost:
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=<SOCKS_PORT> service:jmx:rmi:///jndi/rmi://<REMOTE_HOST:JMX_PORT>/jmxrmi
jvisualvm -J-DsocksProxyHost=localhost -J-DsocksProxyPort=<SOCKS_PORT> --openjmx <REMOTE_HOST>
I am working in Linux environment where my java process is running on Machine C.
in order to connect to Machine C from my machine A i need to connect through B for security reason and there is no option to connect directly to machine C.
is there any way to debug the java process that running on machine C.?
Thanks in advance
You can set up a tunnel through machine B. You can use ssh tunneling by issuing this command from machineA:
ssh root#machineB -L 5005:machineC:5005
You will need to configure your IDE to debug remotely, pointed at machineA port 5005, and configure the remote JVM on machineC for remote debugging on port 5005, something like this:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
Assuming your IDE is running on the same machine as the ssh client, that is, machineA, you can point your IDE to localhost:5005 and the ssh command will attempt to tunnel all connections you make to localhost:5005 through to machineC:5005.
Sorry for this question which must have been asked many times, but I can't succeed in resolving my problem.
I've read a lot of blogs, sites, forums, .... and didn't find any solution in my case.
Case :
I need to connect VisualVM on my box to distant servers (tomcats, weblogics) for performance / threads / memory monitoring.
Those servers are installed on (physical or virtual) machines which are protected by a firewall.
Large intervals of ports are open in the firewall and can be used, but not all ports.
Tests
I've tried direct connections through JMX in VisualVM, using following JVM options on the server-side at server startup :
-Djava.rmi.server.hostname=[hostname]
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=[port]
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
I've precised the hostname because from my network the hostname and the IP address of the server are not the same than those from the network of the remote server.
No success, VisualVM always seems searching for an unknown server.
tried starting jstatd on the server-side on a port accessible (-p option) from my box (telnet on this port works), but when launching visualVM on this host with the jstatd port, it still seems waiting for something unreachable.... Same behavior with jps connecting to this remote host.
tried using the same tools on a server with less network protection, and it works. So I have seen the connections between my box and the server and they are done on ports different from what I've specified to jstatd.
I understand that this port is needed for first communication (kind of handshake) and real communications are done on other ports, but not predictible (ex: 60305, 55197, ...).
Not sure I understand very well how RMI works.
Please, help me, I'm going crazy !
Unfortunately JMX tries to open ports other than the one you configure. Just yesterday I succeeded connecting to tomcat behind firewall via JMX. The two tricky parts are:
put a file called jmxremote.access in CATALINA_HOME/conf, which contains the following lines:
monitorRole readonly
controlRole readwrite
in server.xml set the ports that will be used by jmx, via a special tomcat listener (catalina-jmx-remote.jar required in /lib):
<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
rmiRegistryPortPlatform="9009" rmiServerPortPlatform="9010" />
Then open these two ports on the firewall. It works. But that's just for tomcat.
Another option is to use ssh tunnelling. In short - you connect via SSH and configure it to forward some local port (where the jmx client is running) to some ports on the other side of the tunnel.
References:
Connecting to JMX on Tomcat 6 through a firewall
Connecting MySQL server on Amazon EC2 instance using ssh tunnel
Here are the steps to do this:
Launch an ejstatd in your remote host this way (in ejstatd folder): mvn exec:java -Djava.rmi.server.hostname=[remote_host_name] -Dexec.args="-pr 1099 -ph 1100 -pv 1101" (used for "jstatd" type connection)
Launch your Java application with those additional Java parameters: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=1102 -Dcom.sun.management.jmxremote.rmi.port=1102 -Djava.rmi.server.hostname=[remote_host_name] (used for "JMX" type connection) (java.rmi.server.hostname is required here only because the IP and hostname from your network is not the same as the server point of view)
Open those 4 ports on your remote host and make them available to your local machine: 1099, 1100, 1101 and 1102
Launch JVisualVM
Right-click on "Remote" > "Add Remote Host..." and enter your remote host name in "Host name" (if you don't use the port 1099, you can change this in the "Advanced Settings")
Right-click on the remote host you've just created > "Add JMX Connection..." and enter "[remote_host_name]:1102" in "Connection" input, and check "Do not require SSL connection"
Your Java process will appear twice: one from the "jstatd" connection type, and one from the "JMX" connection type.
Disclaimer: I'm the author of the open source ejstatd tool.
On your [hostname], open up [port] and tcp port range 40000-60000 for your IP only.
This did the trick for me fairly well.