I am trying to get some CPU sampling working on a remote Java-in-Docker process.
I've already looked at the related questions here, and tried everything, to no avail, so I'm posting my setup here.
I have a Java process (openjdk-8) running in a Docker container on a Google Compute Engine (GCE) instance. The GCE instance and container are both running Debian-9. I want to attach VisualVM or JConsole to my Java process.
I am able to run my docker container locally and connect with both visualvm and jconsole using localhost:9010.
I start the container in the VM startup script with:
docker run -d -p 9010:9010 <my container>
The Dockerfile also has:
EXPOSE 9010
The Java process, started by the Dockerfile CMD, has the following relevant args:
"-Dcom.sun.management.jmxremote", \
"-Dcom.sun.management.jmxremote.port=9010", \
"-Dcom.sun.management.jmxremote.rmi.port=9010", \
"-Dcom.sun.management.jmxremote.local.only=false", \
"-Dcom.sun.management.jmxremote.authenticate=false", \
"-Dcom.sun.management.jmxremote.ssl=false", \
I have opened up port 9010 in my gcloud firewall using:
gcloud compute firewall-rules create jmx-port --allow=tcp:9010,udp:9010
I have verified with netcat that the port is open and I can make a TCP connection to it.
I have other ports open from the same Docker container, with clients connecting successfully to those ports. They were exposed and mapped to the host ports the same way (-p port:port) and opened in the firewall the same way.
I am passing the external IP address of the GCE instance. For instance, if I do:
gcloud compute instances list
and it tells me:
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
my-server-b23j us-central1-d n1-standard-1 10.240.0.2 108.357.213.99 RUNNING
Then I will use the argument:
108.357.213.99:9010
as the remote jmx connection host:port pair.
VisualVM and JConsole both tell me they can't connect to the remote JMX service. In both cases, I decline the secure connection, and then they say:
Cannot connect to 108.357.213.99:9010 using
service:jmx:rmi:////jndi/rmi://108.357.213.99:9010/jmxrmi
In desperation, I added a firewall rule that enables TCP/UDP connections on all ports 0-65535, but it did not make a difference -- they still could not connect.
I've read that JMX-RMI opens up anonymous ports, and that you can (at least partly?) disable this behavior by specifying both:
"-Dcom.sun.management.jmxremote.port=9010", \
"-Dcom.sun.management.jmxremote.rmi.port=9010", \
However, it doesn't do the trick in my case.
I've read here that you need to specify the rmi server hostname:
-Djava.rmi.server.hostname='192.168.99.100'
but my server IP is ephemeral -- it is assigned by Google Compute Engine when I create the instance, and so I can't hardwire it into the Dockerfile with the rest of the Java args.
Am I going to have to get a static IP address to make this work?
One possible solution would be to ssh into your GCE box and port-forward port 9010. This can be done from the local console with:
gcloud compute ssh name-of-your-gce-engine -- -L 9010:localhost:9010
Then in jconsole or jvisualvm you connect to localhost:9010. Using localhost here means that jconsole/jvisualvm will connect to your local machine, this connect is tunneled by ssh into your GCE engine and there to the host and port defined in the -L argument, which is localhost:9010, but from the GCE-engine's view. Meaning that you will end up at your application.
You still have to set the rmi server name before starting your program, but you must use
-Djava.rmi.server.hostname='localhost'
so that RMI will tell jconsole/jvisualvm to use localhost and this will then resolve to your local tunneled endpoint. And of course you still need these:
"-Dcom.sun.management.jmxremote", \
"-Dcom.sun.management.jmxremote.port=9010", \
"-Dcom.sun.management.jmxremote.rmi.port=9010", \
"-Dcom.sun.management.jmxremote.local.only=false", \
"-Dcom.sun.management.jmxremote.authenticate=false", \
"-Dcom.sun.management.jmxremote.ssl=false", \
Related
So I'm fairly new to Docker, only been using it for a day or two and recently pulled this https://hub.docker.com/r/ncbi/blast
Which is an image that has a set of commands to get information from a database. I have the Desktop version of Docker for windows and manage to get the image to pop up just fine. It shows up in my images,
I run the image via the desktop app. Give it the name "Test", see it's running using the "docker ps". I get it's IP just fine but see there are no ports.
I've heard that it might be connected to my local host which is why no ports show up but still where would I go to find my host port then and or just change the port? A java program/application I'm making is supposed to connect to this using IP and Port but I can't find the port. Any help would be appreciated, thank you again.
You have to include -p flag in your docker run command to expose ports for your application.
-d options runs docker container in the background.
For e.g. In Docker Run Command
docker run -p 8080:8080 -d ncbi/blast:latest
I have a container running a Java web service with jetty. I would like to connect to the Docker host process, ie tcp://dockerhost:2376, to build and publish images to the local registry on demand.
I can connect to the host from the container but for some annoying reason, I can't seem to connect to the docker host process... Could this be down to some internal firewall rules? The strange thing is I can connect to other docker services, ie on other machines, but NOT the one running the container.
EDIT - just to be clear
my docker daemon runs on 192.168.22.150, port 2376
I can do curl -X GET http://192.168.22.150:2376 from any machine (VM) on that net and I get the usual message
I can do curl -X GET http://192.168.22.150 from my docker container and that works fine (I have a process listening on 80)
I can do curl -X GET http://192.168.22.XXX:2376 on another machine running a docker daemon from my container
If I do curl -X GET http://192.168.22.150:2376 from my container, it hangs
Feels like it's a firewall issue?
I was down to a firewall issue... the IP address of the container wasn't allowed through for the 2376 port, I added it and it worked. Sorry for the trouble and thanks for the help.
Assuming I have a server in my local network with ip 192.168.100.10.
There is docker container running in it with java application.
Now i want to connect to this java application with VisualVM from my computer which has ip address 192.168.100.20. I thought I had everything configured properly but it still does not work.
I have passed these JVM options:
-Dcom.sun.management.jmxremote"
-Dcom.sun.management.jmxremote.port=9010"
-Dcom.sun.management.jmxremote.authenticate=false"
-Dcom.sun.management.jmxremote.ssl=false"
-Dcom.sun.management.jmxremote.local.only=false"
-Dcom.sun.management.jmxremote.rmi.port=9010"
-Djava.rmi.server.hostname=192.168.100.10"
Then I have exposed port 9010 in Dockerfile:
EXPOSE 9010
Then added this port to docker-compose:
ports:
- "9010:9010"
I am trying to connect to remote host with JConsole or VisualVM from my local machine. In "Remote Process" input in JConsole I put "192.168.100.10:9010" but connection fails with error:
"The connection to 192.168.100.10:9010 did not succeed. Would you like to try again?"
What am I doing wrong?
The solution above is sufficient and working. I've been using env variable to set port number which was not working properly.
Now we try to use H2o to construct training cluster. It is easy to use by running java -jar ./h2o.jar and we can setup the cluster with simple flatfile.txt which contain multiple ip and ports.
But we found that it is impossible to setup the h2o cluster within docker containers. Although we can start multiple containers to run java -jar ./h2o.jar and add the prepared flatfile.txt, the h2o process will try to bind local(container's eth0) ip which is different from the one in flatfile.txt. We can java -jar ./h2o.jar -ip $ip to set the one which is in flatfile.txt but h2o instance is not able to run without this "external" ip.
If you use use "docker run --network=host ..." it will work.
See my response to a similar issue here. I describe how it is possible to start an H2O cluster using a flatfile and docker swarm. Basically, you have to run a script in each service before starting H2O, to identify the correct IP addresses for the cluster. This is because docker assigns two IPs to each service. The flatfile needs to use the $HOSTNAME IP for each cluster member, which is difficult to determine in advance.
I developing under Spring3.1 standalone env.
I am trying to connect my application remotely via jconsole.
It's working locally but when I deploy my application into the linux machine it gets time out.
I am using Daemon in order to run my environment.
this is what I add in the run.sh script:
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=6969 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
com.mypackage.daemon.FixDaemon
and inside applicationContext.xml:
<context:mbean-server />
<context:mbean-export />
now on the linux machine after doing netstat thats what we see:
[root# logs]# netstat -an | grep 6969
tcp 0 0 :::6969 :::* LISTEN
so it's seems like it does listening.
but when I add my ip:6969 inside the jconsole interface I get connection failed popup.
any idea what am I doing wrong?
thanks,
ray.
First try to add also this option to your application:
-Djava.rmi.server.hostname=<ip>
Also keep in mind jconsole is using RMI for the communication. This means jconsole first connects to ip:6969. Then server generates a random port X which is passed back to the jconsole. Jconsole then opens another connection to ip:X. Since X is random, there is no way you can open this specific port in the firewall. You have either to open all ports or use a socks proxy which is another subject.
Try connecting to that port using telnet from your machine. If this does not succeed it usually is because of a firewall dropping packets. You will have to talk to your network administrator to open up that port.
Note: You will have to open up two ports. One for binding the RMIRegistry and another one to export the RMI objects. RMI usually exports objects on random high ports. But this will not work in a firewall-ed environment hence you would have to configure the port on which it is exported. This is done by using a RMI URL.
If you are running this on Linux then do a hostname -i, if it returns 127.0.0.1 then fix /etc/hosts. The FAQ entry for JConsole has more information on this.
Another option I would strongly suggest is to look at Jolokia which does not involve changing the firewall configuration but still provides the JMX goodies over HTTP.