We Keep Coding

Java process running as a jar start using high CPU with no visible progress

Have written a java process in spring-boot, process uses fork-join pool.
Process run fine sometimes and completed successfully, sometimes it hangs indefinitely start using very high cpu still making no progress on task.
Thread dump show runnable state for only 3 types of work which are as follows:
1. java.lang.Thread.State: RUNNABLE
at java.util.WeakHashMap.put(WeakHashMap.java:453)
at java.util.Collections$SetFromMap.add(Collections.java:5461)
at org.apache.cxf.jaxrs.client.spec.ClientImpl$WebTargetImpl.initTargetClientIfNeeded(ClientImpl.java:358)
at org.apache.cxf.jaxrs.client.spec.ClientImpl$WebTargetImpl.request(ClientImpl.java:260)
at org.apache.cxf.jaxrs.client.spec.ClientImpl$WebTargetImpl.request(ClientImpl.java:366)
2. "C2 CompilerThread2" #30 daemon prio=9 os_prio=0 tid=0x00007ff9b0979000 nid=0x250be waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
3."GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007ff9b0034800 nid=0x24f94 runnable
Further trying to debug issue, I have few inputs ::
I am using org.apache.cxf.jaxrs.client, When analysed found initTargetClientIfNeeded method of ClientImpl is using 2 property thread.safe.client and thread.safe.client.state.cleanup.period. Do i need to define these 2 properties to make it thread safe ?

Thread hung at sun.nio.ch.FileDispatcherImpl.size0(Native Method)

In my java application, I need to copy the content of a directory from one to another. But sometimes (very rare) the copyDirectory stuck forever and the code does not execute after that. Which result in high CPU.
I checked the jstack of my application multiple times and found that the same thread is in the runnable state for long. Below is the stack trace of the thread.
"pool-2-thread-3" #17 prio=5 os_prio=0 tid=0x00007fab5585c000 nid=0xa81 runnable [0x00007fab0af6f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.size0(Native Method)
at sun.nio.ch.FileDispatcherImpl.size(FileDispatcherImpl.java:84)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:310)
- locked <0x00000000c5f59728> (a java.lang.Object)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:705)
at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1147)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1428)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1389)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1261)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1230)
I tried copying same manually with shell command but they copied successfully. Also, there is one more thread which is in running state for long with the following stack trace.
"pool-2-thread-52" #81581 prio=5 os_prio=0 tid=0x00007fab55951800 nid=0x5db runnable [0x00007faafb2f0000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileChannelImpl.position0(Native Method)
at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:288)
- locked <0x00000000c5ffde60> (a java.lang.Object)
at sun.nio.ch.FileChannelImpl.transferFromFileChannel(FileChannelImpl.java:651)
- locked <0x00000000c5ffde60> (a java.lang.Object)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:708)
at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1147)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1428)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1389)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1261)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1230)
I am not getting any clue why the thread stuck in that native call. Is there any environmental issue or related to the machine?

I just got the clue from the IO-385
Since I was using apache commons-io version 2.4. Which is having a bug where FileUtils.doCopyFile can potentially lead to infinite loop.
for(long count = 0L; pos < size; pos += output.transferFrom(input, pos, count))
{
count = size - pos > 31457280L ? 31457280L : size - pos;
}

Fork Join pool hangs

The case is that an application hangs infinitely from time to time.
Seems that the bug sits in the following snippet:
ForkJoinPool pool = new ForkJoinPool(1); // parallelism = 1
List<String> entries = ...;
pool.submit(() -> {
entries.stream().parallel().forEach(entry -> {
// An I/O op.
...
});
}).get();
Thread pool-4-thread-1 that executes the code freezes on get():
"pool-4-thread-1" #35 prio=5 os_prio=0 tid=0x00002b42e4013800 nid=0xb7d1 in Object.wait() [0x00002b427b72f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.concurrent.ForkJoinTask.externalInterruptibleAwaitDone(ForkJoinTask.java:367)
- locked <0x00000000e08b68b8> (a java.util.concurrent.ForkJoinTask$AdaptedRunnableAction)
at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1001)
...other app methods
One can assume that the task passed to submit() executes too long.
But surprisingly there is no ForkJoinPool-N-worker-N occurrences in the thread dump, so looks like the pool doesn't perform any computations!
How is that possible? If no tasks are executed by the pool, why pool-4-thread-1 thread waits inside get()?
P.S. I know that it's not recommended to execute I/O-related tasks in ForkJoinPool, but still interested in the root of the problem.
Update. When parallelism is set to value greater than 1, no problems are detected.

Set parallelism = N where N > 1 solved the problem.
Strange thing but seems that there is some bug in ForkJoinPool similar to what is stated here.

Explain a thread dump information of Java Hotspot virtual machine

Learning and getting a hands on java thread dump I created a thread dump of my IDEA intellij running process.
The problem is that I don't understand what it means.
Can you explain what this information means and how to read it ?
No need to list every single detail just the main fields and what each means and what it indicates as information.
Here is a small snippet
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode):
"JobScheduler FJ pool 0/4" #165 daemon prio=6 os_prio=0 tid=0x00007f9cb0001800 nid=0x1d2a waiting on condition [0x00007f9ca7e1e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e74c3c90> (a jsr166e.ForkJoinPool)
at jsr166e.ForkJoinPool.awaitWork(ForkJoinPool.java:1756)
at jsr166e.ForkJoinPool.scan(ForkJoinPool.java:1694)
at jsr166e.ForkJoinPool.runWorker(ForkJoinPool.java:1642)
at jsr166e.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:109)
and here is the whole file

<Name> <nature> <vm priority> <os priority><vm id> <os id> <status [adress]>
<thread state>
<current execution point>
- detail : wait until timeout or thread 0x00000000e74c3c90 notification
<stack>
As far as I understand you have a job scheduler worker thread which has been released to its thread pool (and is waiting for a new job)
See also How to analyze a java thread dump?

OSX: JavaVM, AWT/Swing and possibly a deadlock

I'm really new to java programming therefore I apologise in advance if this sounds like a stupid question.
I'm trying to build a simple application written in plain C, which must create a JavaVM and then create a new window by loading java code based on AWT/Swing.
Following this technical note I'v learned that in Mac OSX only, JavaVM must be invoked from a thread different from the main thread in order to be able to create a GUI based on AWT.
Therefore, in the main function of my C application I created a new thread that performs everything, from the creation of the javaVM to the creation of the GUI.
Since the application isn't in reality so simple, I will post a simplified version.
main function:
int main(int argc, char** argv)
{
// Run-time loading of JavaVM framework
void *result;
result = dlopen("/System/Library/Frameworks/JavaVM.framework/JavaVM", RTLD_LAZY);
if (!result) {
printf("can't open library JavaVM: %s\n", dlerror());
}
else {
printf("library JavaVM loaded\n");
}
/* Start the thread that runs the VM. */
pthread_t vmthread;
// create a new pthread copying the stack size of the primordial pthread
struct rlimit limit;
size_t stack_size = 0;
int rc = getrlimit(RLIMIT_STACK, &limit);
if (rc == 0) {
if (limit.rlim_cur != 0LL) {
stack_size = (size_t)limit.rlim_cur;
}
}
pthread_attr_t thread_attr;
pthread_attr_init(&thread_attr);
pthread_attr_setscope(&thread_attr, PTHREAD_SCOPE_SYSTEM);
pthread_attr_setdetachstate(&thread_attr, PTHREAD_CREATE_DETACHED);
if (stack_size > 0) {
pthread_attr_setstacksize(&thread_attr, stack_size);
}
/* Start the thread that we will start the JVM on. */
pthread_create(&vmthread, &thread_attr, startJava, (void *)&thread_data_struct);
pthread_attr_destroy(&thread_attr);
pthread_exit(NULL);
return 0;
}
Thread function:
void *startJava(void *jvm_lib)
{
JavaVMInitArgs args;
const char* classpath = getenv("CLASSPATH");
// determine classpath
char* classpath_opt = str_printf("-Djava.class.path=%s", classpath);
JavaVMOption* option = malloc(sizeof(JavaVMOption) * 2);
option[0].optionString = classpath_opt;
option[1].optionString = str_printf("-verbose:jni");
args.version = JNI_VERSION_1_6;
args.nOptions = 2;
args.options = option;
args.ignoreUnrecognized = JNI_FALSE; // don't ignore unrecognized options
fptr_JNI_CreateJavaVM JNI_CreateJavaVM_fp = (fptr_JNI_CreateJavaVM)dl_dlsym(jvm_lib,
"JNI_CreateJavaVM");
int result = JNI_CreateJavaVM_fp(&jvm, (void**) &env, &args);
free(option);
free(classpath_opt);
// launch java code
jclass init_class = (*env)->FindClass(env, "org/classes/Loader");
jmethodID load_id = (*env)->GetStaticMethodID(env, init_class, "Load",
"(Ljava/lang/String;Lorg/classes/stuff;J)V");
(*env)->CallStaticVoidMethod(env, init_class, load_id);
}
Java code: (UPDATED)
package org.classes;
import java.awt.AWTException;
import java.awt.Component;
import java.awt.Frame;
import java.awt.image.BufferedImage;
import java.awt.EventQueue;
public class Loader {
public static void Load(String baseDir, Stuff stuff, long nativePointer)
{
EventQueue.invokeLater(new Runnable() {
public void run() {
System.loadLibrary("drawingHelperLibrary");
...
...
...
// start test window
Frame frame = new Frame();
frame.setSize(640,480);
frame.setLocation(50, 50);
frame.setVisible(true);
}
});
}
}
All of the above code executes successfully except for the creation of the window which causes a deadlock or something similar since terminal remains busy without any CPU usage and both threads remain alive.
If I comment out the lines concerning the creation of the window, the application execute successfully and quit.
This is the output from jstack:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.4-b02-402 mixed mode):
"Attach Listener" daemon prio=9 tid=1040b1800 nid=0x11b888000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Low Memory Detector" daemon prio=5 tid=103806000 nid=0x10b137000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=9 tid=103805800 nid=0x10b034000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=9 tid=103804800 nid=0x10af31000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=9 tid=103804000 nid=0x10ae2e000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=103803000 nid=0x10ad2b000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=10409b800 nid=0x10ac28000 in Object.wait() [10ac27000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7f3001300> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <7f3001300> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=10409b000 nid=0x10ab25000 in Object.wait() [10ab24000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7f30011d8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <7f30011d8> (a java.lang.ref.Reference$Lock)
"main" prio=5 tid=104000800 nid=0x10048d000 runnable [10048a000]
java.lang.Thread.State: RUNNABLE
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1827)
- locked <7f30010a8> (a java.util.Vector)
- locked <7f3001100> (a java.util.Vector)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1724)
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
- locked <7f3004e90> (a java.lang.Runtime)
at java.lang.System.loadLibrary(System.java:1045)
at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:50)
at java.security.AccessController.doPrivileged(Native Method)
at sun.awt.NativeLibLoader.loadLibraries(NativeLibLoader.java:38)
at sun.awt.DebugHelper.<clinit>(DebugHelper.java:29)
at java.awt.Component.<clinit>(Component.java:566)
at org.classes.Loader.Load(Loader.java:69)
"VM Thread" prio=9 tid=104096000 nid=0x10aa22000 runnable
"Gang worker#0 (Parallel GC Threads)" prio=9 tid=104002000 nid=0x103504000 runnable
"Gang worker#1 (Parallel GC Threads)" prio=9 tid=104002800 nid=0x103607000 runnable
"Concurrent Mark-Sweep GC Thread" prio=9 tid=10404d000 nid=0x10a6f0000 runnable
"VM Periodic Task Thread" prio=10 tid=103817800 nid=0x10b23a000 waiting on condition
"Exception Catcher Thread" prio=10 tid=104001800 nid=0x103401000 runnable
JNI global references: 913
I really don't know what more can I do. Maybe it's a stupid mistake but I'm not skilled enough with this Java-C mix since it's the first time that I'm looking at it.
UPDATE: I have updated the java code (thanks to trashgod) but it still doesn't work.
Am I missing something?

I was able to resolve this issue by looking at how the Eclipse project creates its launchers. You need to spawn a separate thread for the JVM as you have done, but the main method needs to start the CFRunLoop.
There might be some additional details for your particular implementation, but something similar to this is currently working in our case:
...
#include <CoreServices/CoreServices.h>
static void dummyCallback(void * info) {}
...
...
if (stack_size > 0) {
pthread_attr_setstacksize(&thread_attr, stack_size);
}
CFRunLoopRef loopRef = CFRunLoopGetCurrent();
/* Start the thread that we will start the JVM on. */
pthread_create(&vmthread, &thread_attr, startJava, (void *)&thread_data_struct);
pthread_attr_destroy(&thread_attr);
CFRunLoopSourceContext sourceContext = {
.version = 0, .info = NULL, .retain = NULL, .release = NULL,
.copyDescription = NULL, .equal = NULL, .hash = NULL,
.schedule = NULL, .cancel = NULL, .perform = &dummyCallback };
CFRunLoopSourceRef sourceRef = CFRunLoopSourceCreate(NULL, 0, &sourceContext);
CFRunLoopAddSource(loopRef, sourceRef, kCFRunLoopCommonModes);
CFRunLoopRun();
CFRelease(sourceRef);
...
You can look through the Eclipse implementation here:
http://git.eclipse.org/c/equinox/rt.equinox.framework.git

Following this example, you don't need a separate thread on the C side unless you're using Cocoa. You do need to construct your Java GUI on the event dispatch thread using invokeLater().

I have the same issue, if I load my native library before AWT, then it hangs. The solution is to load AWT native library BEFORE loading my native library.
ColorModel.getRGBdefault(); //static code in ColorModel loads AWT native library
System.loadLibrary("MyLibrary"); //then load your native code

This doesn't actually solve the original poster's problem, but I found his/her post while trying to solve a similar problem. In my case, I need to start a c++ program, and have it call an imaging library written in Java. That library uses some awt classes, so I was seeing the deadlock issue, even though I wasn't creating a UI in the Java code.
Also, I want to compile the same c++ code on different platforms, so was avoiding using Cocoa.
Because I don't need to create a Java UI, it worked for me to add "-Djava.awt.headless=true" as an option to the jvm when it's started from the c++ code.
I wanted to post this, in case someone else in a similar situation stumbles upon this post looking for answers.