We Keep Coding

How to obtain threaddump of a java application on Cloud foundry?

I tried a couple of options
a. cf java thread-dump myapp -i0 (Only 1 instance is there so 0)
Response is nothing.
logs of myapp show
2019-12-13T14:52:41.15+0800 [SSH/0] OUT Successful remote access by 10.x.x.x:35764
2019-12-13T14:52:41.84+0800 [SSH/0] OUT Remote access ended for 10.x.x.x:35764
b. I did cf ssh myapp
/home/vcap/app/.java-buildpack/open_jdk_jre/bin/ has no jstack and jmap command doesn't work either
/home/vcap/app/.java-buildpack/open_jdk_jre/bin/ has the following components
java orbd servertool java-buildpack-memory-calculator-3.13.0_RELEASE pack200 tnameserv
jjs policytool unpack200
jvmkill-1.16.0_RELEASE rmid
keytool rmiregistry
Question is-> How to get the java threaddump?

I believe the easiest option is to run cf logs to watch your app logs. Then in a second terminal, cf ssh to the app and run ps aux and look for the process id of the Java process. Then run kill -3 <pid>. Your Java process will catch this signal and issue a thread dump. It will go to STDOUT, which conveniently goes to your app's log stream so it should show up in the first terminal.
The crummy part about this is that cf logs will inject some additional text at the beginning of every line, which makes it hard to load the thread dump into tools. You can usually strip that off with cut or awk, or you can redirect cf logs to a file, open with a text editor and find/replace the leading text.
Besides that you have a couple other options:
If you're using Spring, enable Spring Boot Actuators. You can then use the /actuator/threaddump endpoint to generate and download a thread dump (note the format for this is JSON, not the standard format)
Use an APM tool or profiler like NewRelic, AppDynamics, Dynatrace or YourKit. These are well supported by the Java buildpack and most will just work.
Use JMX. It's easy enough to enable with the Java buildpack, just set the env variable JBP_CONFIG_JMX to '{enabled: true}' and restage your app. Once enabled, you can open a tunnel with cf ssh -N -T -L 5000:localhost:5000 <APP_NAME>. Then open jvisualvm, make a new "local" JMX connection to "localhost:5000" and connect. Click the "Threads" tab and click the "Thread Dump" button. It might be a little slow as it's going across the tunnel, but give it a few seconds to finish and you should have a thread dump.
What won't work on Cloud Foundry are jstack and jcmd. The Java buildpack, at the time of writing, installs a JRE and these tools are no longer shipped with the OpenJDK JRE and for legal reasons can't be bundled with it. If you were inclined to do so, you could scp the tools and required shared libraries from a JDK up to your app container and run them, but it's a lot of work and not fun. I'm not super familiar with the cf java plugin, but I suspect it tries to use these tools, which used to be available, and no longer are there.

You should be able to get ThreadDumps with kill -3 <PID> and open that in heap analyzer such as eclipse memory analyzer (MAT) and select Thread Stacks

Tomcat thread dump

Is there away to take a thread dump from Tomcat? I would like to monitor the threads running in Tomcat at a certain point in time.
Note: I was doing that on Web logic but i don't know how it can be done on Tomcat.

If you use Linux, you can send a kill -3 [pid of your tomcat] and it will dump all current threads in the catalina.out.

There is a simple way to monitor tomcat threads and do a dump.
Start tomcat with the folowing java options :
-Dcom.sun.management.jmxremote.port=<some free port>
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
After the tomcat is restarted you can simply start jconsole (part of jdk) or visualvm and create a new jmx connection to the port you have chosen above.
In both tools you also have an option to take a dump ;)
!!!But do that only in closed/secured environment - since there is no authentication set.!!!

add java melody to your server
https://code.google.com/p/javamelody/
it give you a lot of info including threads in runtime

The following steps should help you get the current Java threads running and analyze them:
Download IBM JCA which is a nice thread dump analyzer from IBM Thread & Dump Monitor Analyzer
Execute the following to get the Tomcat threads dump: root#localhost:~# sudo -u $TOMCAT_USER $JAVA_HOME/bin/jstack -J-d64 -l $(ps aux | grep '[c]atalina' | awk '{print $2}') > ~/threads.log
Open the threads dump with IBM Thread & Dump Monitor Analyzer
Note: Replace $JAVA_HOME with your path to your current JDK, $TOMCAT_USER with the user running Tomcat; if you pass the wrong user or even using root you will get an error like this:
3047: well-known file is not secure

Please try netbeans profiler to profile any web applications.
https://profiler.netbeans.org/
http://wiki.netbeans.org/FaqProfilerAttachRemoteServer

Jconsole Remote Executable Jar File

All,
I have a remote server that I recently enabled VNC for using vnc4server and Chicken for mac as the client.
The purpose for doing so was to enable running Java's Jconsole to monitor an executable jar file that is running my server logic.
However, after logging into my server using VNC, I keep getting an error when I try to use Jconsole on vnc.
It states connection failed do you want to try again. Now I am logged in as the same user that started the process.
Is there something I am missing when using jconsole in VNC? Also can I monitor my executable jar file remotely using Jconsole on my local machine?
These are the options I am including to run the jar file: java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9005 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.remote.ssl=false -Djava.rmi.server.hostname=ipaddress -jar path
Thanks
These JVM options fixed things. Fix found here: You need to pass to the VM: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false
https://forums.oracle.com/thread/1177644

This does not seem like an VNC issue- either the ports are not open, they are being blocked b a firewall, or there is some kind of permission/authentication issue with the app itself related to monitoring it.
In order to eliminate VNC as the cause (and use localhost in a local connection on jconsole), do "ssh -X REMOTHOST -n jconsole" and see. This will also eliminate the overhead of running the full X server and VNC.
Also on linux you can find out what process holds a port open by doing:
netstat -ap | grep PORT_NUMBER on the remote host you want to run on.
Colin

Error Shutting down JBOSS

I am running JBOSS on Solaris10 and trying to shutdown and restart the JBOSS. When I restart the JBOSS I am getting the following error.
LifecycleException: Protocol handler initialization failed: java.net.BindException: Address already in use:33409
I got this problem before and used to kill the process id related to the port 33409 using the lsof command. Unfortunately the lsof command is not working in my solaris box. Is there any other command I can use to identify the process id related to the port number 33409 and kill that process id, so that I can restart the JBOSS without any port conflicts.
Thanks in Advance

Use <JAVA_HOME>/bin/jps -lvm to see all java processes with their main class and all its arguments and JVM arguments.
jps -lvm | grep 'org.jboss.Main' -- this is how I usually find PIDs of any running jboss processes.

You can identify the correct process with something like this:
$ ps -feA | grep "jboss"
It takes a bit of guesswork, but you´ll get the hang of it. Make sure you identify the correct process before killing it, though.

What is the default properties that eclipse uses to launch a java process in its debugging mode?

I would like to connect to java process started by eclipse using a command line debugger but not sure what default properties of the eclipse launched java process are ? I wouldn't mind using attaching using sockets but not sure how much slower that would be ?

how about
-Xdebug -Xnoagent -agentlib:jdwp=transport=dt_socket,suspend=n,address=localhost:12345
then you can attach a java debugger to port 12345

I investigated on the differences of Java application startup in Eclipse between "Run" and "Debug". I used Sysinternals' Process Explorer to see the command line of the java process that Eclipse forks when starting a Java application with either option. (I'm on a Windows 7 system)
With "Run HelloWorld"
"C:\Program Files (x86)\Java\jdk1.7.0_07\jre\bin\javaw.exe" -Dfile.encoding=Cp1252 -classpath C:\workspace-juno\HelloWorld\bin com.example.HelloWorld
.......
With "Debug HelloWorld"
"C:\Program Files (x86)\Java\jdk1.7.0_07\jre\bin\javaw.exe" -agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:2404 -Dfile.encoding=Cp1252 -classpath C:\workspace-juno\HelloWorld\bin com.example.HelloWorld
(The address port changed in succeeding invocations, e.g. it became address=localhost:2564 at the next invocation.)
So java (or javaw) actually gets started with different options depending on whether "Run" or "Debug" was used: the "Debug" start-up adds an additional agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:NNNN option. This is a standard way of putting a JVM into debuggable mode. The HelloWorld program will first wait for its debugger to successfully connect to it before continuing (option suspend=y). The HelloWorld JVM will connect itself automatically to some debugger running on address localhost:NNNN (... option address=localhost:NNNN and implicit default option server=n). (... Oracle provides authoritative information on the agentlib start-up options)
Eclipse itself will act as the debugger application providing a port NNNN. The HelloWorld JVM will connect to this port, via its own port of number NNNN+1 (one can see a process's used ports with Process Explorer on that process's TCP/IP tab).