We Keep Coding

Strange bug with wait statement in synchronized block

An excerpt from Worker.java file:
public class Worker extends Thread{
public void run(){
// Worker Thread periodically does its job.
Master.getInstance().decrementNumOfWorkingWorkers();
// This is the reporting part of the thread.
// Aimed to wait other threads finish their job.
synchronized (Master.getInstance().allFinished) {
while ( Master.getInstance().getNumOfWorkingWorkers() > 0) {
try {
Master.getInstance().allFinished.wait();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Main.printSync("Worker Thread-" + getPId() + " worked on");
}
}
}
This is from Master.Java:
import java.util.LinkedList;
import java.util.Timer;
import java.util.TimerTask;
public class Master extends Timer {
AllFinished allFinished;
int day;
public TimerTask task;
LinkedList<Worker> Workers;
private static Master instance = null;
int numOfWorkingWorkers = 0;
public class AllFinished
{
}
public class PeriodicIncrement extends TimerTask {
// Complete this class
public void run() {
Main.printSync("Day " + day + ":");
Main.printSync("Queue: " + TaskQueue.getInstance().ConvertToString());
day++;
for (int i = 0; i < Workers.size(); i++) {
synchronized (Workers.get(i)) {
Workers.get(i).notify();
}
}
if (0 == numOfWorkingWorkers) {
synchronized (allFinished) {
allFinished.notifyAll();
}
cancel(); // Terminate the timer thread
}
}
}
private Master(LinkedList<Worker> Workers) {
super();
this.task = new PeriodicIncrement();
day = 0;
allFinished = new AllFinished();
this.Workers = Workers;
numOfWorkingWorkers = this.Workers.size();
this.schedule(task, 100, 100);
}
}
For a test with 4 worker threads, everything was fine before I added the excerpted part in Worker.java. Then, to report each worker's action after all workers are done, I added that part. Algorithm is very simple. When a worker finishes its job, it checks if there is any job in a TaskQueue and ProductOwner. If there are not any, it breaks its loop and then decrement by 1 active worker threads counter in Master and then calls wait on AllFinished field of Master. run() method of PeriodicIncrement checks this counter and if it is 0 (meaning all workers finished their job), it calls notifyAll() on AllFinished .
Problem is, one sometimes two threads are entering that excerpted code block in Worker.java but remainings never enter so active worker threads counter never decremented to 0 and my program never finishes.
If I just comment out the excerpted part in Worker.java, except finishing and reporting randomly, everything is fine. What I mean is the excerpted part seems to be problematic.
Could you help me to find out?

That was a fun one to debug, after so much time not playing with low-level concurrency primitives. The trick to root-cause this was to use the jstack tool provided by the JDK.
╭───courtino ~
╰➤ sudo jstack -l 63978
2020-03-29 21:26:01
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode):
"DestroyJavaVM" #18 prio=5 os_prio=31 tid=0x00007ffa91491000 nid=0x1803 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Timer-0" #17 prio=5 os_prio=31 tid=0x00007ffa91f36800 nid=0x5903 waiting for monitor entry [0x0000700009aa4000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.amazon.adnumsmissionmanagerservice.homework.ScrumMaster$PeriodicIncrement.run(ScrumMaster.java:42)
- waiting to lock <0x000000076bc70db8> (a com.amazon.adnumsmissionmanagerservice.homework.Programmer)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
"Programmer-4" #15 prio=5 os_prio=31 tid=0x00007ffa92cc6800 nid=0x5603 in Object.wait() [0x000070000989e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.completeTasksUntilNoneAvailable(Programmer.java:230)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.work(Programmer.java:165)
- locked <0x000000076bc71a58> (a com.amazon.adnumsmissionmanagerservice.homework.Programmer)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.run(Programmer.java:241)
"Programmer-3" #14 prio=5 os_prio=31 tid=0x00007ffa923af800 nid=0x5503 in Object.wait() [0x000070000979b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.work(Programmer.java:176)
- locked <0x000000076bcc8ac0> (a com.amazon.adnumsmissionmanagerservice.homework.ScrumMaster$AllFinished)
- locked <0x000000076bc70db8> (a com.amazon.adnumsmissionmanagerservice.homework.Programmer)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.run(Programmer.java:241)
"Programmer-2" #13 prio=5 os_prio=31 tid=0x00007ffa92c25000 nid=0x3f03 in Object.wait() [0x0000700009698000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.work(Programmer.java:176)
- locked <0x000000076bcc8ac0> (a com.amazon.adnumsmissionmanagerservice.homework.ScrumMaster$AllFinished)
- locked <0x000000076bc5fca0> (a com.amazon.adnumsmissionmanagerservice.homework.Programmer)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.run(Programmer.java:241)
"Programmer-1" #12 prio=5 os_prio=31 tid=0x00007ffa91fab800 nid=0x4203 in Object.wait() [0x0000700009595000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.completeTasksUntilNoneAvailable(Programmer.java:230)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.work(Programmer.java:165)
- locked <0x000000076bc43c80> (a com.amazon.adnumsmissionmanagerservice.homework.Programmer)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.run(Programmer.java:241)
"Service Thread" #11 daemon prio=9 os_prio=31 tid=0x00007ffa91f30800 nid=0x4403 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #10 daemon prio=9 os_prio=31 tid=0x00007ffa9227c000 nid=0x3c03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #9 daemon prio=9 os_prio=31 tid=0x00007ffa9227b000 nid=0x4603 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #8 daemon prio=9 os_prio=31 tid=0x00007ffa92272800 nid=0x4803 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"JDWP Command Reader" #7 daemon prio=10 os_prio=31 tid=0x00007ffa9200f000 nid=0x3a03 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"JDWP Event Helper Thread" #6 daemon prio=10 os_prio=31 tid=0x00007ffa91019800 nid=0x4a03 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"JDWP Transport Listener: dt_socket" #5 daemon prio=10 os_prio=31 tid=0x00007ffa9181a000 nid=0x4b07 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007ffa9180d800 nid=0x3603 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007ffa91002000 nid=0x3003 in Object.wait() [0x0000700008b77000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076ab08ed8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x000000076ab08ed8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007ffa92006800 nid=0x2e03 in Object.wait() [0x0000700008a74000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076ab06c00> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x000000076ab06c00> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"VM Thread" os_prio=31 tid=0x00007ffa90843000 nid=0x2d03 runnable
"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007ffa91001800 nid=0x2307 runnable
"GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007ffa91801800 nid=0x2a03 runnable
"GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007ffa91802000 nid=0x5303 runnable
"GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007ffa91802800 nid=0x5203 runnable
"VM Periodic Task Thread" os_prio=31 tid=0x00007ffa91357800 nid=0x3d03 waiting on condition
JNI global references: 2236
A few observations:
Thread Timer-0 (that's your periodic task) is in BLOCKED state, waiting for lock 0x000000076bc70db8, which is an instance of Programmer
there are 4 programmers:
2 of them are still doing some work and are holding one lock of type Programmer (they're actually holding a lock on themselves)
the other 2 programmers are done and are holding two locks: one on themselves, and one on AllFinished. Programmer-3 is an example of such thread.
Since the periodic task tries to acquire a lock on Programmer-3 before notifying it, it has to wait for Programmer-3 to release the lock on itself, which it can't do because it's waiting for all tasks to complete. Deadlock!
The reason why your programmers are holding a lock on themselves is this:
public synchronized void work()
This is going to put the entire work method into a synchronized block which monitor belongs to this. Since the Programmer class is stateless and it mostly does work that doesn't interact with other threads, you can actually synchronize a much smaller part of the work method. So you have two changes to make:
remove synchronized from the signature of work
synchronize the call to wait inside the work method:
synchronized (this) {
wait();
}
One lesson that it can give is that, when using synchronized block, you always want to synchronize as little code as possible. Anything that doesn't need to be synchronized should be outside of the block, in order to maximize parallelism (everything that happens in the block is sequential), potentially for requiring the lock less often (there might be conditions that allow you to skip the lock acquisition if you put it at the lowest level, so that will reduce the overhead of synchronization), and in some cases like this one, avoiding deadlocks.

What are threads (i.e. lightweight processes) named `{java}` created for?

I wrote a Java program that sleeps for a while:
package com.mycompany.app;
import java.lang.System;
import java.util.concurrent.TimeUnit;
public class Main {
public static void main(String[] args) {
System.out.println("the current process's pid is " + ProcessHandle.current().pid());
try {
TimeUnit.SECONDS.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Hello World!"); // Prints the string to the console.
}
}
I run the program with:
$ java -cp target com.mycompany.app.Main
the current process's pid is 10172
I inspect the processes that Ubuntu creates to run it:
$ pstree -pau -l -G -s 10172
systemd,1 splash
└─lxterminal,3194,t
└─bash,12150
└─java,10172 -cp target com.mycompany.app.Main
├─{java},10173
├─{java},10174
├─{java},10175
├─{java},10176
├─{java},10177
├─{java},10178
├─{java},10179
├─{java},10180
├─{java},10181
├─{java},10182
├─{java},10183
├─{java},10184
├─{java},10185
├─{java},10186
├─{java},10187
├─{java},10188
├─{java},10189
└─{java},10190
What are those threads (i.e. lightweight processes) named {java} created for?
Is it possible to find out what programs they run from shell using some commands?
Which processes (and LWPs) are running JVM?
Which processes (and LWPs) are running my Java program?

All of these threads belong to JVM.
Run jstack <pid> to get the thread list.
"main" #1 prio=5 os_prio=0 cpu=150.00ms elapsed=8.04s tid=0x00007f9f90011000 nid=0x107 waiting on condition [0x00007f9f99f9f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base#11.0.1/Native Method)
at java.lang.Thread.sleep(java.base#11.0.1/Thread.java:339)
at java.util.concurrent.TimeUnit.sleep(java.base#11.0.1/TimeUnit.java:446)
at com.mycompany.app.Main.main(Main.java:10)
"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=0.00ms elapsed=7.95s tid=0x00007f9f901f9000 nid=0x10e waiting on condition [0x00007f9f6c10f000]
java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList(java.base#11.0.1/Native Method)
at java.lang.ref.Reference.processPendingReferences(java.base#11.0.1/Reference.java:241)
at java.lang.ref.Reference$ReferenceHandler.run(java.base#11.0.1/Reference.java:213)
"Finalizer" #3 daemon prio=8 os_prio=0 cpu=0.00ms elapsed=7.95s tid=0x00007f9f901fd800 nid=0x10f in Object.wait() [0x00007f9f65fef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base#11.0.1/Native Method)
- waiting on <0x0000000712108f80> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base#11.0.1/ReferenceQueue.java:155)
- waiting to re-lock in wait() <0x0000000712108f80> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base#11.0.1/ReferenceQueue.java:176)
at java.lang.ref.Finalizer$FinalizerThread.run(java.base#11.0.1/Finalizer.java:170)
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.00ms elapsed=7.93s tid=0x00007f9f90210000 nid=0x110 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 cpu=40.00ms elapsed=7.93s tid=0x00007f9f90212000 nid=0x111 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C1 CompilerThread0" #7 daemon prio=9 os_prio=0 cpu=40.00ms elapsed=7.93s tid=0x00007f9f90214000 nid=0x112 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"Sweeper thread" #8 daemon prio=9 os_prio=0 cpu=10.00ms elapsed=7.93s tid=0x00007f9f90216000 nid=0x113 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #9 daemon prio=9 os_prio=0 cpu=0.00ms elapsed=7.90s tid=0x00007f9f902d3800 nid=0x114 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Common-Cleaner" #10 daemon prio=8 os_prio=0 cpu=0.00ms elapsed=7.89s tid=0x00007f9f902df800 nid=0x116 in Object.wait() [0x00007f9f656ef000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base#11.0.1/Native Method)
- waiting on <0x0000000712002df0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base#11.0.1/ReferenceQueue.java:155)
- waiting to re-lock in wait() <0x0000000712002df0> (a java.lang.ref.ReferenceQueue$Lock)
at jdk.internal.ref.CleanerImpl.run(java.base#11.0.1/CleanerImpl.java:148)
at java.lang.Thread.run(java.base#11.0.1/Thread.java:834)
at jdk.internal.misc.InnocuousThread.run(java.base#11.0.1/InnocuousThread.java:134)
"Attach Listener" #11 daemon prio=9 os_prio=0 cpu=0.00ms elapsed=0.21s tid=0x00007f9f44001000 nid=0x126 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"VM Thread" os_prio=0 cpu=0.00ms elapsed=7.95s tid=0x00007f9f901f1000 nid=0x10d runnable
"GC Thread#0" os_prio=0 cpu=0.00ms elapsed=8.01s tid=0x00007f9f90038800 nid=0x108 runnable
"G1 Main Marker" os_prio=0 cpu=0.00ms elapsed=8.00s tid=0x00007f9f90097800 nid=0x109 runnable
"G1 Conc#0" os_prio=0 cpu=0.00ms elapsed=8.00s tid=0x00007f9f90099800 nid=0x10a runnable
"G1 Refine#0" os_prio=0 cpu=0.00ms elapsed=8.00s tid=0x00007f9f9018d000 nid=0x10b runnable
"G1 Young RemSet Sampling" os_prio=0 cpu=0.00ms elapsed=8.00s tid=0x00007f9f9018f000 nid=0x10c runnable
"VM Periodic Task Thread" os_prio=0 cpu=0.00ms elapsed=7.90s tid=0x00007f9f902d6000 nid=0x115 waiting on condition
Here nid is the hexadecimal ID of a thread in the OS - you may match it to the output of pstree.
The first thread named main is the thread executing your code.
Reference Handler thread is responsible for adding Weak, Soft and Phantom references discovered by Garbage Collector into their registered ReferenceQueues.
Finalizer thread runs finalize method of the objects ready to be finalized.
Signal Dispatcher waits for specific OS signals and handles them. In particular, it makes thread dump on SIGQUIT, and also initiates VM shutdown process on SIGTERM, SIGINT and SIGHUP.
CompilerThreads perform JIT compilation of the bytecode.
Sweeper thread cleans up obsolete compiled methods.
Service Thread runs several background JVM tasks: detects low memory condition, cleans up StringTable and SymbolTable, sends deferred JVMTI events and GC notifications and so on.
Common-Cleaner runs cleaning actions of java.lang.ref.Cleaner instances.
Attach Listener thread supports Dynamic Attach mechanism. It listens for incoming Dynamic Attach connections and executes VM commands. For example, it is used by jstack, jmap and jcmd utilities.
VM Thread runs internal VM operations that require a safepoint. The examples of such operations are deoptimization, class redifinition, biased lock revocation, thread dump, heap inspection etc.
G1 threads are involved in Garbage Collection.
VM Periodic Task Thread is used to simulate timer interrupts.

Flyway regularly hangs (MariaDB connector, RDS)

I've been seeing frequent hangs on deployment, at the migration step. Java/Scala application packaged in WAR for Tomcat. Database is RDS Aurora using MariaDB connector (https://downloads.mariadb.org/connector-java/).
Probably has nothing to do with Flyway but is a generic problem getting a connection.
Migration is run from shell in container:
java -cp `echo WEB-INF/lib/*|tr ' ' :` foo.Migrate
Migration code looks like:
def main(args: Array[String]): Unit = {
Environment.dbFlywayPassword.foreach { pass =>
val flyway = new Flyway
flyway.setDataSource(Environment.jdbcUrl, "flyway", pass)
flyway.migrate
}
}
Connection string:
jdbc:mysql:aurora://%RDS_HOST%/xxx?serverSslCert=/rds-ca-2015-root.pem&useSSL=true&connectTimeout=10000
I've tried increasing logging level in Flyway, but nothing is logged after this line:
15:57:35.115 [main] INFO o.f.c.internal.util.VersionPrinter - Flyway 4.2.0 by Boxfuse
So I got a thread dump, that looks like this:
15:57:35.115 [main] INFO o.f.c.internal.util.VersionPrinter - Flyway 4.2.0 by Boxfuse
2017-06-08 15:57:56
Full thread dump OpenJDK 64-Bit Server VM (25.121-b13 mixed mode):
"MariaDb-failover-1" #8 daemon prio=5 os_prio=0 tid=0x00005555f80ae000 nid=0x14 waiting on condition [0x00007fc330b8f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f5c59b10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00005555f70bf000 nid=0x12 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00005555f7063000 nid=0x11 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00005555f7060800 nid=0x10 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00005555f705e800 nid=0xf waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00005555f702f000 nid=0xe in Object.wait() [0x00007fc331616000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f5a30c58> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000f5a30c58> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00005555f702c000 nid=0xd in Object.wait() [0x00007fc331717000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f5a30e10> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000f5a30e10> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00005555f6f85000 nid=0xb runnable [0x00007fc34341d000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
- locked <0x00000000f0a66090> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000000f0a81eb0> (a sun.security.ssl.AppInputStream)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000000f06b5118> (a java.io.BufferedInputStream)
at org.mariadb.jdbc.internal.io.input.StandardPacketInputStream.getPacketArray(StandardPacketInputStream.java:125)
at org.mariadb.jdbc.internal.io.input.StandardPacketInputStream.getPacket(StandardPacketInputStream.java:95)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1002)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:982)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.readRequestSessionVariables(AbstractConnectProtocol.java:498)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.readPipelineAdditionalData(AbstractConnectProtocol.java:544)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:410)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:357)
at org.mariadb.jdbc.internal.protocol.AuroraProtocol.loop(AuroraProtocol.java:149)
at org.mariadb.jdbc.internal.failover.impl.AuroraListener.reconnectFailedConnection(AuroraListener.java:179)
at org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:154)
at org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:94)
at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:464)
at org.mariadb.jdbc.Driver.connect(Driver.java:103)
at org.flywaydb.core.internal.util.jdbc.DriverDataSource.getConnectionFromDriver(DriverDataSource.java:416)
at org.flywaydb.core.internal.util.jdbc.DriverDataSource.getConnection(DriverDataSource.java:381)
at org.flywaydb.core.internal.util.jdbc.JdbcUtils.openConnection(JdbcUtils.java:51)
at org.flywaydb.core.Flyway.execute(Flyway.java:1418)
at org.flywaydb.core.Flyway.migrate(Flyway.java:971)
at tgam.service.data.Migrate$.$anonfun$main$1(Migrate.scala:11)
at foo.Migrate$.$anonfun$main$1$adapted(Migrate.scala:8)
at foo.Migrate$$$Lambda$4/458209687.apply(Unknown Source)
at scala.Option.foreach(Option.scala:257)
at foo.Migrate$.main(Migrate.scala:8)
at foo.Migrate.main(Migrate.scala)
"VM Thread" os_prio=0 tid=0x00005555f7022000 nid=0xc runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00005555f70da800 nid=0x13 waiting on condition
JNI global references: 232
Heap
def new generation total 4928K, used 1611K [0x00000000f0600000, 0x00000000f0b50000, 0x00000000f5950000)
eden space 4416K, 24% used [0x00000000f0600000, 0x00000000f0712c10, 0x00000000f0a50000)
from space 512K, 100% used [0x00000000f0a50000, 0x00000000f0ad0000, 0x00000000f0ad0000)
to space 512K, 0% used [0x00000000f0ad0000, 0x00000000f0ad0000, 0x00000000f0b50000)
tenured generation total 10944K, used 4187K [0x00000000f5950000, 0x00000000f6400000, 0x0000000100000000)
the space 10944K, 38% used [0x00000000f5950000, 0x00000000f5d66e78, 0x00000000f5d67000, 0x00000000f6400000)
Metaspace used 13061K, capacity 13306K, committed 13568K, reserved 1060864K
class space used 1381K, capacity 1449K, committed 1536K, reserved 1048576K
Looks like an I/O hang in org.mariadb.jdbc.Driver.connect, but I do have a connectTimeout set (10 seconds). This timeout doesn't seem to be effective (would I need a socketTimeout per https://github.com/brettwooldridge/HikariCP/issues/754 ?)
This has been happening for a while. The same thing happened when I was using Tomcat's contextInitialized hook to do migrations. I decided to refactor out into a separate invocation before starting Tomcat, which looks like a better idea in general, but it hasn't affected this behaviour.
What will typically happen is that the code will hang, after 2-3 minutes ECS will timeout, and trigger a redeploy. After some number of these retries (e.g. up to 10), Flyway will run successfully and the service will start.

OK, seems this is a known bug in RDS Aurora - and it's alluded to in the MariaDB Connector doc (surely it should be a runtime warning, though!)
https://mariadb.com/kb/en/mariadb/about-mariadb-connector-j/#infrequently-used
usePipelineAuth Not compatible with aurora During connection,
different queries are executed. When option is active those queries
are send using pipeline (all queries are send, then only all results
are reads), permitting faster connection creation. Default: true.
Since 1.6.0
Also kudos to wlad_ in Freenode #maria who pointed me in the right direction.

Using BufferedReader.lines() breaks in parallel

I'm writing some code that will be reading log lines and doing some processing in the background on that data. This processing would probably benifit from parallelization, such as what is offered by the Stream.parallel methods, and I was attempting to use this. This is the code I started out with that works perfectly.
public static void main(String[] args) {
try {
final Socket socket = new Socket(ADDRESS, PORT);
final BufferedReader reader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
socket.getOutputStream().write(QUERY);
reader.lines().forEach(System.out::println);
} catch (IOException e) {
e.printStackTrace();
}
}
This code connects and prints out all of my data. I would very much like to restructure this code as follows:
public static void main(String[] args) {
try (Socket socket = new Socket(ADDRESS, PORT);
BufferedReader reader = new BufferedReader(new InputStreamReader(socket.getInputStream()))) {
socket.getOutputStream().write(QUERY);
reader.lines().forEach(System.out::println);
} catch (IOException e) {
e.printStackTrace();
}
}
But sadly this doesn't work. Even worse, going back to the original code, this doesn't even work:
public static void main(String[] args) {
try {
final Socket socket = new Socket(ADDRESS, PORT);
final BufferedReader reader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
socket.getOutputStream().write(QUERY);
reader.lines().parallel().forEach(System.out::println);
} catch (IOException e) {
e.printStackTrace();
}
}
All that was added here was a .parallel call and this completely doesn't work. It just sits there and nothing is printed out.
I can live perfectly well and good without the 2nd version using the modified try(A a = new A()) {} as that doesn't look too good in this case. What I can't live without is figuring out why this .parallel call breaks everything.
I'm assuming the modified try statement is closing the streams as soon as I fall out of them (right after we start the forEach) so they are being killed and GC'd before operation. I can't for the life of me figure out what the hell is going on with the .parallel call.
As requested here is the output of jstack running on the .parellel() version of this code.
Full thread dump OpenJDK 64-Bit Server VM (25.112-b15 mixed mode):
"Attach Listener" #9 daemon prio=9 os_prio=0 tid=0x00007fd4f4001000 nid=0x4907 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd5280be000 nid=0x48d2 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd5280bb000 nid=0x48d1 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd5280b9800 nid=0x48d0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd5280b6800 nid=0x48cf waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd5280b5000 nid=0x48ce runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd528082000 nid=0x48cd in Object.wait() [0x00007fd515c6d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ec008e98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000ec008e98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd52807d800 nid=0x48cc in Object.wait() [0x00007fd515d6e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ec006b40> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000ec006b40> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00007fd528008000 nid=0x48c2 runnable [0x00007fd52fd9f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
- locked <0x00000000ec086790> (a java.net.SocksSocketImpl)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at com.gravypod.Test.main(Test.java:48)
"VM Thread" os_prio=0 tid=0x00007fd528075800 nid=0x48ca runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd52801d800 nid=0x48c4 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd52801f000 nid=0x48c5 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd528021000 nid=0x48c6 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd528022800 nid=0x48c7 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007fd5280c0800 nid=0x48d3 waiting on condition
JNI global references: 18
The Test.java:48 line is the Socket socket = new Socket line. This is the result of the fully-working non-parallel code (just using .lines()).
Full thread dump OpenJDK 64-Bit Server VM (25.112-b15 mixed mode):
"Attach Listener" #9 daemon prio=9 os_prio=0 tid=0x00007f9048001000 nid=0x4982 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f90800be800 nid=0x496f runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f90800bb000 nid=0x496e waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f90800b9800 nid=0x496d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f90800b6800 nid=0x496c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f90800b5000 nid=0x496b runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f9080082000 nid=0x496a in Object.wait() [0x00007f907018d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ec008e98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000ec008e98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f908007d800 nid=0x4969 in Object.wait() [0x00007f907028e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ec006b40> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000ec006b40> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00007f9080008000 nid=0x4961 runnable [0x00007f90884c3000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
- locked <0x00000000ec08e890> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
- locked <0x00000000ec08e890> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at java.io.BufferedReader$1.hasNext(BufferedReader.java:571)
at java.util.Iterator.forEachRemaining(Iterator.java:115)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at com.gravypod.Test.main(Test.java:51)
"VM Thread" os_prio=0 tid=0x00007f9080075800 nid=0x4968 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f908001d800 nid=0x4963 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f908001f000 nid=0x4964 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f9080021000 nid=0x4965 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f9080022800 nid=0x4966 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f90800c1000 nid=0x4970 waiting on condition
JNI global references: 319
The line Test.java:51 is the reader.lines().forEach line.

I imagine that parallel() or the forEach() on a parallel stream waits to read all the input before parallelising the task. Because the server never closes the connection, it will wait forever.
Your task is not really parallelizable. Data comes sequentially over the wire, so reading it in parallel cannot work.

It seems, your application isn’t hanging, technically, but just waiting for a lot of input, before performing observable work. This is a combination of two implementation details. When you are starting a parallel stream operation, it will first try to split the workload until every CPU core has something to do, before actually starting to process elements. This combines with the Reader#lines() parallelizes badly due to nonconfigurable batch size issue.
Simply said, when a Stream has an unknown size, the implementation will try to buffer batches of sizes that are multiples of 1024, growing on each split. This great answer shows, how the splitting will happen for a stream of an unknown size with multiple cores, showing that multiples of 1024 elements will get buffered in the process. This can take a very long time, before the consumer passed to forEach is ever invoked.
Note that processing an infinite source via the non-short-circuiting forEach is outside the scope of the Stream API anyway. Assuming a timely side effect is an assumption about the processing order of the Stream, but there is no guaranty about it.
This answer guides you to a work-around. You can use something like
try(Socket socket = new Socket(ADDRESS, PORT);
BufferedReader reader = new BufferedReader(
new InputStreamReader(socket.getInputStream()))) {
socket.getOutputStream().write(QUERY);
Stream.generate(() -> {
try { return reader.readLine(); }
catch (IOException ex) { throw new UncheckedIOException(ex); }
}).parallel().forEach(System.out::println);
} catch(IOException|UncheckedIOException e) {
e.printStackTrace();
}
But, as said, this isn’t the intended use case of the Stream API…

My program will not stop after run() method in main thread finishes

I am using Java. The main thread sends data, while a worker thread listens to responses. I also have Timer in case timeout occurs. In main(), I am calling run(), which can finish, according to the output. Here is what it looks like:
class Send {
Worker w;
run() {
// w was initialized in constructor
w.start();
....
w.join();
}
main(args) {
Send s = new Send();
s.run();
}
private class Worker extend Thread {
public void run() {
....
}
}
}
In s.run(), every time I need to cancel the Timer or restart the Timer, I would do
timer.cancel();
timer.purge();
timer = new Timer();
timer.schdule(...);
The TimerTask is simply calling a static method in Send to handle the timeout.
So what did I do wrong to cause my program hanging after the main thread finishes?
Thank you.
EDIT: the out put of kill -3 process-id:
Full thread dump OpenJDK 64-Bit Server VM (19.0-b09 mixed mode):
"DestroyJavaVM" prio=10 tid=0x00007f9f20035000 nid=0x73f1 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Timer-10" prio=10 tid=0x0000000001a7d800 nid=0x740f in Object.wait() [0x00007f9f1e39c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000075833fe78> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:505)
- locked <0x000000075833fe78> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:484)
"Low Memory Detector" daemon prio=10 tid=0x00007f9f20004800 nid=0x7402 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x0000000001a70000 nid=0x7401 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x00007f9f20001000 nid=0x7400 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x0000000001a6e800 nid=0x73ff waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x0000000001a49000 nid=0x73fe in Object.wait() [0x00007f9f1f3f2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007580b1310> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
- locked <0x00000007580b1310> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
"Reference Handler" daemon prio=10 tid=0x0000000001a47000 nid=0x73fd in Object.wait() [0x00007f9f1f4f3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007580b11e8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000007580b11e8> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x0000000001a40000 nid=0x73fc runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00000000019d7000 nid=0x73f2 runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00000000019d9000 nid=0x73f3 runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x00000000019da800 nid=0x73f4 runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x00000000019dc800 nid=0x73f5 runnable
"GC task thread#4 (ParallelGC)" prio=10 tid=0x00000000019de800 nid=0x73f6 runnable
"GC task thread#5 (ParallelGC)" prio=10 tid=0x00000000019e0000 nid=0x73f7 runnable
"GC task thread#6 (ParallelGC)" prio=10 tid=0x00000000019e2000 nid=0x73f8 runnable
"GC task thread#7 (ParallelGC)" prio=10 tid=0x00000000019e4000 nid=0x73f9 runnable
"GC task thread#8 (ParallelGC)" prio=10 tid=0x00000000019e5800 nid=0x73fa runnable
"GC task thread#9 (ParallelGC)" prio=10 tid=0x00000000019e7800 nid=0x73fb runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f9f20007800 nid=0x7403 waiting on condition
JNI global references: 886
Heap
PSYoungGen total 150528K, used 7745K [0x00000007580b0000, 0x00000007628a0000, 0x0000000800000000)
eden space 129088K, 6% used [0x00000007580b0000,0x00000007588405b8,0x000000075fec0000)
from space 21440K, 0% used [0x00000007613b0000,0x00000007613b0000,0x00000007628a0000)
to space 21440K, 0% used [0x000000075fec0000,0x000000075fec0000,0x00000007613b0000)
PSOldGen total 343936K, used 0K [0x0000000608200000, 0x000000061d1e0000, 0x00000007580b0000)
object space 343936K, 0% used [0x0000000608200000,0x0000000608200000,0x000000061d1e0000)
PSPermGen total 21248K, used 3130K [0x00000005fdc00000, 0x00000005ff0c0000, 0x0000000608200000)
object space 21248K, 14% used [0x00000005fdc00000,0x00000005fdf0ea30,0x00000005ff0c0000)

A Java program will exit after the last thread finishes.
To prevent that, mark the other threads as daemon threads.

Get a thread dump by running kill -3 <process id>. This will tell you which threads are hanging around & preventing the java process from exiting. Feel free to post (some of) the thread dump & folks can help you figure out what to do next.

It looks like you are creating several Timer threads. Check that all of them have exited properly, that is probably the problem.
If you check the Timer JavaDoc - http://download.oracle.com/javase/6/docs/api/java/util/Timer.html - you will notice the following note:
By default, the task execution thread does not run as a daemon thread
You can use a debugger (both Eclipse and NetBeans have excellent ones) to see which threads are still alive.