I am running a java application on Mesos and Marathon. How do i monitor the heap stats of the java process? I tried jstat which complains "pid not found", but the process is running and i can check it by 'ps auwx'. Is this because the java process is running inside a cgroup? If that is the case how do we monitor mesos based applications.
The best option to monitor application is to publish it's metrics from itself. There are plenty of ways to add monitoring to your application starting from proprietarty such as NewRelic or DataDog to opensource Prometheus.
If you need to get acces via JMX you can expose this port with following configuration:
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.local.only=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.port=$PORT1 \
-Djava.rmi.server.hostname=$MESOS_HOSTNAME
Above configuration assume you have two ports and second is dedicated for JMX
Related
I am new to Visual VM, how to connect to Spring Boot application remotely in Visual VM?
Multiple instances are running as a java -jar app.jar with random ports. How to connect from Visual VM, I have root access to the remote system. Do I need to enable any security configuration?
In the local system, applications shows up automatically but when I entered remote system IP address it's prompting Add JMX connection and Add jstatd connection.
We need to specify the remote IP ADDRESS and expose the listening PORT while running the jar.
Syntax:
java
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=<PORT>
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=<IP_ADDRESS>
-Dcom.sun.management.jmxremote.rmi.port=<PORT>
-jar app-1.0.0-SNAPSHOT.jar
Example: Listening port is 6001 and available for remote ip address 192.168.0.23
java -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=6001 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=192.168.0.23 -Dcom.sun.management.jmxremote.rmi.port=6001 -jar app-1.0.0-SNAPSHOT.jar
For More details about set up Visual VM https://github.com/M-Thirumal/installation_guide/blob/master/visualVm/visualvm_remote_set_up.md
I encountered this dilemma with Spring Boot as well. But I was not using a jar. I passed the jmx arguments to maven to profile the application without having to build a jar first.
mvn spring-boot:run -Dspring-boot.run.jvmArguments="-Xdebug -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.rmi.port=9010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=localhost"
For more details, I documented my steps here.
I have a Spring Boot Java app running on Ubuntu 14.x using Oracle Java 1.8.0 that I want to debug remotely with IntelliJ. I have tried to get it to listen on a port for debug purposes but with no success. Note, the ports I tried are all well above the port 1024, to make sure it's not a permission problem. I am not root but I do have sudo access to the box.
I tried adding this to the java command line:
-agentlib:jdwp=transport=dt_socket,address=localhost:9009,server=y,suspend=y
A technique I got from this document:
http://javahowto.blogspot.com/2010/09/java-agentlibjdwp-for-attaching.html
However when I run this command:
sudo netstat -an | grep LISTEN
I don't see port 9009. Also, the app does not wait for debugger attachment as indicated by the "suspend=y" parameter, because I see the app initialization messages stream by as normal as the app starts up. Why isn't this working?
Here is the shell script that launches the app. Note, this shell script is launched by supervisord. I point this out in case that might be causing any trouble:
# !/bin/bash
# Shell script to launch Spring Boot app
# Kill subprocess when parent bash process is terminated by supervisor or when CTRL+C is received
trap 'kill -TERM $PID' TERM INT
java \
-Dnetworkaddress.cache.ttl=5 \
-Dnetworkaddress.cache.negative.ttl=5 \
\
-jar spbootapp.jar \
-agentlib:jdwp=transport=dt_socket,address=localhost:9009,server=y,suspend=y
--spring.application.name=spbootapp-awsdev \
--spring.profiles.active=cluster \
--spring.cloud.config.enabled=false \
--endpoints.configprops.enabled=false \
--endpoints.health.sensitive=false \
&
The debug parameters -agentlib:jdwp=transport=dt_socket,address=localhost:9009,server=y,suspend=y need to go before the -jar in the command.
I've been able to connect VisualVM to my Java process, running in a Docker container, doing something as
docker run \
--rm \
--entrypoint=java \
-p 9010:9010 \
my-user/my-image \
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9010 \
-Dcom.sun.management.jmxremote.rmi.port=9010 \
-Dcom.sun.management.jmxremote.local.only=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname='192.168.99.100' \
-jar /my-app-1.0-SNAPSHOT.jar
I have a couple problems with this solution, though:
I don't like having to specify the hostname, as I would like to use the same scripts for running my process in my dev machine and in my different servers
This works if I only want to monitor one JVM. But I'm using a Docker Compose file to my Java app as a service that can be scaled to multiple containers, so I can expose port 9010 but it will be linked to different port numbers in the host
I've been playing with SSH tunnels, but with no luck. The JVM doesn't like being accessed from port 32878, i.e., even if it's mapped to 9010 in the container.
Any ideas, please?
I use docker with a wide cluster of web applications and rest services.
To access them directly into my local browser (i'm windows 10 user) i have installed proxy container. That proxy have one exposed port "1080:1080".
Then i go to my browser with proxy plugin (FoxyProxy). And i can access any port any address. Use that and you don't need to do any other port mapping on other containers, the proxy handles all traffic.
that is the command i use in the container
service sshd start && sshpass -p 'password' ssh -o StrictHostKeyChecking=no -N -D 0.0.0.0:1080 localhost
I am trying to create docker containers for ZooKeeper and configure them in cluster mode (full code is here and here).
Containers are based on Alpine Linux (alpine:3.2 on Docker Hub), but the problem that I'm going to describe happens also with the official Java container (java:7).
I use the following commands to start the cluster:
docker run -d -h zk1 --name zk1 dockmob/zookeeper -s zk1,zk2,zk3
# wait some time ...
docker run -d -h zk2 --name zk2 dockmob/zookeeper -s zk1,zk2,zk3
docker run -d -h zk3 --name zk3 dockmob/zookeeper -s zk1,zk2,zk3
(They are available on docker hub, you can try them).
If I wait some time before starting the second and third containers, then the host names zk2 and zk3 are put in /etc/hosts too late (by docker), and Java is unable to find them: I get java.net.UnknownHostException in the logs of zk1 for both zk2 and zk3.
I found on the web that I need to disable JVM DNS cache in order to refresh the host names, so I introduced the following command in the Dockerfile in order to update the java.security settings:
RUN grep '^networkaddress.cache.ttl=' /usr/lib/jvm/java-1.7-openjdk/jre/lib/security/java.security || echo 'networkaddress.cache.ttl=10' >> /usr/lib/jvm/java-1.7-openjdk/jre/lib/security/java.security
It sets the DNS TTL property (networkaddress.cache.ttl) to 10 seconds.
The variable networkaddress.cache.negative.ttl is already set to its default value (10).
The behavior does not change. I get lots of java.net.UnknownHostException repeatedly.
What can be the cause of the problem?
In my case the java application was failing with java.net.UnknownHostException when running in docker. The reason was that I used --network=none docker flag (getting ip/hostname via dhcp and pipework). In this case, docker does not add automatically to /etc/hosts entry like
127.0.0.1 15e326aecf84
And getCanonicalHostName() Java function threw this exception.
Possible solutions:
add hostname entry to /etc/hosts file via docker run parameter --hostname=your-hostname.com
switch to docker-managed network configuration
I managed to get rid of the DNS issues by switching to Oracle JRE 8 and using the following hack in the Dockerfile:
RUN echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf
I created a working Java 8 docker container container on Docker Hub (the code is on github).
I am trying to use VisualVM in my system to monitor a Tomcat instance running over EC2. I tried steps provided in multiple blogs about how to configure it, but still when I try to run tomcat it gives me following error.
./catalina.sh: 5: /home/gvr/apache-tomcat-8.0.18/bin/setenv.sh: -Dcom.sun.management.jmxremote: not found
I added following statement in server.xml
<listener classname="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
rmiregistryportplatform="10001"
rmiserverportplatform="10002"
uselocalports="true" />
And my setenv.sh is as follows
CATALINA_OPTS="-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=ec2-xx-xxx-xx-xx.ap-southeast-1.compute.amazonaws.com"
$CATALINA_OPTS
Besides this I have added, catalina-jmx-remote.jar in tomcat's lib directory
Could anyone please provide me some hint, what is possibly going wrong. I tried everything I have found related to configuring VisualVM
I am running Tomcat 8.0.18, java 8 over ubuntu
I believe there are two issues
formatting in setenv.sh, you need \ to split across lines
last line $CATALINA_OPTS which tries to execute the arguments, hence -Dcom.sun.management.jmxremote not found...
Suggested fix
CATALINA_OPTS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Djava.rmi.server.hostname=ec2-xx-xxx-xx-xx.ap-southeast-1.compute.amazonaws.com"
echo $CATALINA_OPTS
I also tried using the JmxRemoteLifecycleListener and it didn't work for me.
Here it is how I did it:
1)I choose 1005 as jmx port and my setenve.sh is like:
CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=10005 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=localhost"
2) redirect using ssh the jmx port and the rmi
3) Run visualvm using the following uri: service:jmx:rmi:///jndi/rmi://localhost:10005/jmxrmi
If you need more information have a look to this post:
http://ignaciosuay.com/how-to-connect-a-java-profiler-like-visualvm-or-jconsole-to-a-remote-tomcat-running-on-amazon-ec2/