We Keep Coding

Java thread pooled complex constraints data processing

I have Java EE (desktop) application that had to process data files generated by multiple sources (up to a 200 different sources). Each source periodically generates data file with unique name which also contains that source's unique ID.
I need to create a thread pool with 15 threads which will process and remove files with these constraints:
Multiple threads can't process files from the same source simultaneously.
Multiple files from the same source should be processed in order of it's creation timestamp.
No synchronization with the file generator sources is possible so it means that the next file(s) may be generated by source while it's previous file is processed or scheduled for processing.
Processing should be multi threaded because of performance reasons (single threaded processing is not enough so I'm planning to use 10-15 threads).
A file processing operation may be time consuming 3-15 seconds.
Any suggestions on architecture of such complex synchronization of the threads in pool are welcome.
P.S. Due to the limitation on simultaneous processing the design I've used in more simple situations earlier i.e. using ArrayBlockingQueue does not fit this case.

General idea:
You have a task-queue per source.
And you have a central queue which is effectively a queue of task-queues which is shared between all worker threads.
For each source you create a task-queue. And you stick these task-queue's in a hashtable based on the unique id. This way you get the guarantee that tasks from the same source are processed in order (requirement 2).
If a tasks is received, you look up (or create) the task-queue in the hashtable and you add the task to the taskqueue. If it was the first task added to the queue, you also add it to the central queue.
Then there are a bunch of worker-threads that take task-queues from this central queue and then take a single task from this task-queue they just took and process that task. Once they are done with the task, they need to decide if the task-queue needs to be reinserted back into the central-queue or not.
There are a few parts were things could easily go wrong:
You don't want to end up with a task-queue being inserted into the central-queue multiple times. That would violate your first requirement.
You don't want the task-queue not being reinserted into the central-queue even though a task is available.
So you need to take care of the appropriate synchronization and it might be a bit more complex than you would initially think. But seen the fact that the tasks are long running, I would start out with a regular mutex (could be per task-queue) e.g. synchronized or a lock and don't worry about making it non blocking.

So this is the skeleton/tester of the class I've created to solve my problem. See #pveentjer answer for some details on whats going on here.
package org.zur.test;
import java.io.File;
import java.util.Comparator;
import java.util.HashMap;
import java.util.PriorityQueue;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.Semaphore;
import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.TimeUnit;
public class PmFileProcessor {
private static final int THREADS_COUNT = 15;
ArrayBlockingQueue<File> scheduledFiles = new ArrayBlockingQueue<>(10000, false);
HashMap<String, PmFileProcessingJob> allJobs = new HashMap<>();
ScheduledExecutorService jobsExecutorService = Executors.newScheduledThreadPool(THREADS_COUNT);
public PmFileProcessor() {
super();
SourceJobsManager fc = new SourceJobsManager();
fc.setDaemon(true);
fc.start();
}
public void scheduleFile(File f) {
try {
scheduledFiles.add(f);
} catch (Exception e) {
// TODO: handle exception
}
}
/**
* Assigns files to file source processing job.
*
* #author
* <ul>
* <li>Zur13</li>
* </ul>
*
*/
public class SourceJobsManager extends Thread {
#Override
public void run() {
// assigns scheduled files to per-source jobs and schedules job for additional execution
while ( true ) {
try {
File f = scheduledFiles.take();
PmFileProcessingJob job = getSourceJob(f);
job.scheduleSourceFile(f);
jobsExecutorService.execute(job); // schedules job execution
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
// TODO: check disk space periodically
}
}
/**
* Finds existing job for file source or creates a new one.
*
* #param f
* #return
*/
private PmFileProcessingJob getSourceJob(File f) {
// TODO: test code
String fname = f.getName();
String[] parts = fname.split("_");
String uid = parts[0];
PmFileProcessingJob res = allJobs.get(uid);
if ( res == null ) {
res = new PmFileProcessingJob(uid);
allJobs.put(uid, res);
}
return res;
}
}
/**
* Process first file from scheduledSourceFiles queue (i.e. each job execution processes a single file or
* reschedules itself for later execution if another thread already processes the file from the same source).
*
* #author
* <ul>
* <li>Zur13</li>
* </ul>
*
*/
public class PmFileProcessingJob implements Runnable {
public final String fileSourceUidString;
PriorityQueue<File> scheduledSourceFiles = new PriorityQueue<>(1000, new Comparator<File>() {
#Override
public int compare(File o1, File o2) {
// TODO Auto-generated method stub
return 0;
}
});
Semaphore onePassSemaphore = new Semaphore(1);
public PmFileProcessingJob(String fileSourceUid) {
super();
this.fileSourceUidString = fileSourceUid;
}
/**
* Schedules file from for processing by this job.
*/
public void scheduleSourceFile(File f) {
scheduledSourceFiles.add(f);
}
#Override
public void run() {
File f = null;
if ( scheduledSourceFiles.size() > 0 ) { // fail fast optimization 1
if ( onePassSemaphore.tryAcquire() ) { // fail fast optimization 2
try {
f = scheduledSourceFiles.poll();
if ( f != null ) {
// TODO: process the file
try {
System.err.println(f.getName() + "\t" + Thread.currentThread().getId());
Thread.sleep(1000);
return;
} catch (Exception e) {
// TODO: handle exception
return; // prevents reschedule loop for failing files
}
} else {
// scheduledSourceFiles queue is empty
return;
}
} finally {
onePassSemaphore.release();
}
}
if ( f == null && scheduledSourceFiles.size() > 0 ) {
// this thread did not process the scheduled file because another thread holds the critical section
// pass
// this thread should reschedule this Job to release this thread and try to process this job later
// with another thread
// reschedule the job with 4 seconds delay to prevent excess CPU usage
// System.err.println("RESCHEDULE");
jobsExecutorService.schedule(this, 3, TimeUnit.SECONDS);
}
}
}
#Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = prime * result + ((this.fileSourceUidString == null) ? 0 : this.fileSourceUidString.hashCode());
return result;
}
#Override
public boolean equals(Object obj) {
if ( this == obj )
return true;
if ( !(obj instanceof PmFileProcessingJob) )
return false;
PmFileProcessingJob other = (PmFileProcessingJob) obj;
if ( this.fileSourceUidString == null ) {
if ( other.fileSourceUidString != null )
return false;
} else if ( !this.fileSourceUidString.equals(other.fileSourceUidString) )
return false;
return true;
}
}
public static void main(String[] args) {
PmFileProcessor fp = new PmFileProcessor();
fp.unitTest();
}
private void unitTest() {
// TODO Auto-generated method stub
int filesCount = 1000;
for (int i = 0; i < filesCount; i++) {
int sourceUid = ThreadLocalRandom.current().nextInt(1, 30);
File f = new File(sourceUid + "_" + i);
scheduleFile(f);
}
Thread.yield();
try {
Thread.sleep(999000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}

How do I stop and skip a command after 3.5 seconds?

I want to stop and skip a command while it's waiting for input after 3.5 seconds. I have tried to use System.currentTimeMillis() by subtracting from the start time, however the code I made does not skip the input.
food is an arrayList from the table class.
public void timer() {
startTime = System.currentTimeMillis();
while(false||(System.currentTimeMillis()-startTime)<3500)
{
correct = input(); //What I want to skip after 3.5 seconds
}
record();
}
Here is the input() method:
public boolean input()
{
Scanner console = new Scanner (System.in);
//I want to skip everything after this after 3.5 seconds.
int num = console.nextInt();
num--;
System.out.println("You selected " + table.food.get(num).toString());
table.food.remove(num);
if (num==choice)
{
return true;
}
return false;
}

One of the problems you are facing is that any of the Scanner's next methods can not be interrupted when reading from a console. Therefore you have to read the input in a different way, for example by using a InputStreamReader.
After that you can submit a specific task to a ExecutorService that handels the execution of the "input reading" seperately from the main Thread. You will get a Future on which you can define a timeout.
Note that this operation is still blocking (on both threads).
This solution is somewhat based on this article.
import java.io.*;
import java.util.concurrent.*;
public class Test {
static class ReadInput implements Callable<Integer> {
public Integer call() throws IOException {
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
try {
while (br.ready() == false) {
Thread.sleep(250);
}
String input = br.readLine();
return Integer.parseInt(input);
} catch (InterruptedException e) {
return null;
}
}
}
public static void main(String[] args) {
Integer input = null;
ExecutorService ex = Executors.newSingleThreadExecutor();
try {
Future<Integer> future = ex.submit(new ReadInput());
input = future.get(3500, TimeUnit.MILLISECONDS);
} catch (ExecutionException | InterruptedException | TimeoutException e) {
// handle exceptions that need to be handeled
} finally {
ex.shutdownNow();
}
System.out.println("done: " + input);
}
}
Note that timeout in the ReadInput should be lower than the timeout in the main Thread.

Scheduled process stops working after several executions

I have a program running on a linux machine. That program has a class, that schedules a thread for execution every minute. The thread supposed to perform several calculations, create a text file and then synchronize the file to a different machine using rsync.
The class looks like this:
public class Generator {
private ScheduledThreadPoolExecutor SchEventPool = new ScheduledThreadPoolExecutor(5);
public void generate() {
LocalDateTime ldt = LocalDateTime.now();
int delay = 60 - ldt.getSecond();
SchEventPool.scheduleWithFixedDelay(new HelperClass(), delay, 60, SECONDS);
}
private class HelperClass implements Runnable{
#Override
public void run() {
// some calculations
// .
// .
// .
FileBuild();
}
private void FileBuild(){
try{
String filepath = System.getProperty("user.dir") + "/SomeDir/somefile.txt";
File f = new File;
BufferedWriter writer = new BufferedWriter(new FileWriter(f));
for (int i = 0; i < 100; i++){
writer.write("write something into the file" + "\n");
}
writer.close();
} catch (IOException ex) {ex.printStackTrace();}
// synchronize the file
synchronize();
}
public void synchronize(){
try {
String[] cmd = new String[]{"rsync", "--remove-source-files", "-avre", "/usr/bin/sudo /usr/bin/ssh -i /home/user/.ssh/id_rsa", "/home/user/project/SomeDir, "user#server.ip:/home/user/project/AnotherDir/"};
Process p = new ProcessBuilder().command(cmd).start();
} catch (IOException ex) {ex.printStackTrace();}
}
}
When i run the main program and invoke the generate() method, the process runs successfully for about 30 minutes, sometimes more sometimes less, and then the execution of the scheduled thread stops. Without any errors or any warnings.
I should mention that the file that is being synchronized to the second machine is about 1500 KB, very small file.
I have also monitored the processors load during running and business as usual, server is not overloaded.
What could be the issue here?
Why would the scheduled thread stops being executed?
Any help would be appreciated. Thank you.

Force stop Java Files.copy() running on external thread

The answer here seemed to be a valid solution before Java 8:
How to cancel Files.copy() in Java?
But now it doesn't work, because ExtendedCopyOption.INTERRUPTIBLE is private.
Basically, I need to download a file from some given URL and save it to my local file-system using Files.copy().
Currently, I am using a JavaFX Service because I need to show the progress in a ProgressBar.
However, I don't know how to block the thread running Files.copy() if the operation takes too long.
Using Thread.stop() is at least not wanted. Even Thread.interrupt() fails.
I also want the operation to terminate gracefully if the internet connection becomes unavailable.
To test the case when no internet connection is available, I'm removing my ethernet cable and putting it back after 3 seconds.
Unfortunately, Files.copy() returns only when I put back the ethernet cable, while I would like it to fail immediately.
As I can see, internally Files.copy() is running a loop, which prevents the thread from exiting.
Tester(Downloading OBS Studio exe):
/**
* #author GOXR3PLUS
*
*/
public class TestDownloader extends Application {
/**
* #param args
*/
public static void main(String[] args) {
launch(args);
}
#Override
public void start(Stage primaryStage) throws Exception {
// Block From exiting
Platform.setImplicitExit(false);
// Try to download the File from URL
new DownloadService().startDownload(
"https://github.com/jp9000/obs-studio/releases/download/17.0.2/OBS-Studio-17.0.2-Small-Installer.exe",
System.getProperty("user.home") + File.separator + "Desktop" + File.separator + "OBS-Studio-17.0.2-Small-Installer.exe");
}
}
DownloadService:
Using #sillyfly comment with FileChannel and removing File.copy seems to work only with calling Thread.interrupt() but it is not exiting when the internet is not available..
import java.io.File;
import java.net.URL;
import java.net.URLConnection;
import java.nio.channels.Channels;
import java.nio.channels.FileChannel;
import java.nio.file.StandardOpenOption;
import java.util.logging.Level;
import java.util.logging.Logger;
import javafx.concurrent.Service;
import javafx.concurrent.Task;
/**
* JavaFX Service which is Capable of Downloading Files from the Internet to the
* LocalHost
*
* #author GOXR3PLUS
*
*/
public class DownloadService extends Service<Boolean> {
// -----
private long totalBytes;
private boolean succeeded = false;
private volatile boolean stopThread;
// CopyThread
private Thread copyThread = null;
// ----
private String urlString;
private String destination;
/**
* The logger of the class
*/
private static final Logger LOGGER = Logger.getLogger(DownloadService.class.getName());
/**
* Constructor
*/
public DownloadService() {
setOnFailed(f -> System.out.println("Failed with value: " + super.getValue()+" , Copy Thread is Alive? "+copyThread.isAlive()));
setOnSucceeded(s -> System.out.println("Succeeded with value: " + super.getValue()+" , Copy Thread is Alive? "+copyThread.isAlive()));
setOnCancelled(c -> System.out.println("Succeeded with value: " + super.getValue()+" , Copy Thread is Alive? "+copyThread.isAlive()));
}
/**
* Start the Download Service
*
* #param urlString
* The source File URL
* #param destination
* The destination File
*/
public void startDownload(String urlString, String destination) {
if (!super.isRunning()) {
this.urlString = urlString;
this.destination = destination;
totalBytes = 0;
restart();
}
}
#Override
protected Task<Boolean> createTask() {
return new Task<Boolean>() {
#Override
protected Boolean call() throws Exception {
// Succeeded boolean
succeeded = true;
// URL and LocalFile
URL urlFile = new URL(java.net.URLDecoder.decode(urlString, "UTF-8"));
File destinationFile = new File(destination);
try {
// Open the connection and get totalBytes
URLConnection connection = urlFile.openConnection();
totalBytes = Long.parseLong(connection.getHeaderField("Content-Length"));
// --------------------- Copy the File to External Thread-----------
copyThread = new Thread(() -> {
// Start File Copy
try (FileChannel zip = FileChannel.open(destinationFile.toPath(), StandardOpenOption.CREATE,
StandardOpenOption.TRUNCATE_EXISTING, StandardOpenOption.WRITE)) {
zip.transferFrom(Channels.newChannel(connection.getInputStream()), 0, Long.MAX_VALUE);
// Files.copy(dl.openStream(), fl.toPath(),StandardCopyOption.REPLACE_EXISTING)
} catch (Exception ex) {
stopThread = true;
LOGGER.log(Level.WARNING, "DownloadService failed", ex);
}
System.out.println("Copy Thread exited...");
});
// Set to Daemon
copyThread.setDaemon(true);
// Start the Thread
copyThread.start();
// -------------------- End of Copy the File to External Thread-------
// ---------------------------Check the %100 Progress--------------------
long outPutFileLength;
long previousLength = 0;
int failCounter = 0;
// While Loop
while ((outPutFileLength = destinationFile.length()) < totalBytes && !stopThread) {
// Check the previous length
if (previousLength != outPutFileLength) {
previousLength = outPutFileLength;
failCounter = 0;
} else
++failCounter;
// 2 Seconds passed without response
if (failCounter == 40 || stopThread)
break;
// Update Progress
super.updateProgress((outPutFileLength * 100) / totalBytes, 100);
System.out.println("Current Bytes:" + outPutFileLength + " ,|, TotalBytes:" + totalBytes
+ " ,|, Current Progress: " + (outPutFileLength * 100) / totalBytes + " %");
// Sleep
try {
Thread.sleep(50);
} catch (InterruptedException ex) {
LOGGER.log(Level.WARNING, "", ex);
}
}
// 2 Seconds passed without response
if (failCounter == 40)
succeeded = false;
// --------------------------End of Check the %100 Progress--------------------
} catch (Exception ex) {
succeeded = false;
// Stop the External Thread which is updating the %100
// progress
stopThread = true;
LOGGER.log(Level.WARNING, "DownloadService failed", ex);
}
//----------------------Finally------------------------------
System.out.println("Trying to interrupt[shoot with an assault rifle] the copy Thread");
// ---FORCE STOP COPY FILES
if (copyThread != null && copyThread.isAlive()) {
copyThread.interrupt();
System.out.println("Done an interrupt to the copy Thread");
// Run a Looping checking if the copyThread has stopped...
while (copyThread.isAlive()) {
System.out.println("Copy Thread is still Alive,refusing to die.");
Thread.sleep(50);
}
}
System.out.println("Download Service exited:[Value=" + succeeded + "] Copy Thread is Alive? "
+ (copyThread == null ? "" : copyThread.isAlive()));
//---------------------- End of Finally------------------------------
return succeeded;
}
};
}
}
Interesting questions:
1-> What does java.lang.Thread.interrupt() do?

I strongly encourage you to use a FileChannel.
It has the transferFrom() method which returns immediately when the thread running it is interrupted.
(The Javadoc here says that it should raise a ClosedByInterruptException, but it doesn't.)
try (FileChannel channel = FileChannel.open(Paths.get(...), StandardOpenOption.CREATE,
StandardOpenOption.WRITE)) {
channel.transferFrom(Channels.newChannel(new URL(...).openStream()), 0, Long.MAX_VALUE);
}
It also has the potential to perform much better than its java.io alternative.
(However, it turns out that the implementation of Files.copy() may elect to delegate to this method instead of actually performing the copy by itself.)
Here's an example of a reusable JavaFX Service that lets you fetch a resource from the internet and save it to your local file-system, with automatic graceful termination if the operation takes too long.
The service task (spawned by createTask()) is the user of the file-channel API.
A separate ScheduledExecutorService is used to handle the time constraint.
Always stick to the good practices for extending Service.
If you choose to use such an high-level method, you won't be able to track down the progress of the task.
If the connection becomes unavailable, transferFrom() should eventually return without throwing an exception.
To start the service (may be done from any thread):
DownloadService downloadService = new DownloadService();
downloadService.setRemoteResourceLocation(new URL("http://speedtest.ftp.otenet.gr/files/test1Gb.db"));
downloadService.setPathToLocalResource(Paths.get("C:", "test1Gb.db"));
downloadService.start();
and then to cancel it (otherwise it will be automatically cancelled after the time expires):
downloadService.cancel();
Note that the same service can be reused, just be sure to reset it before starting again:
downloadService.reset();
Here is the DownloadService class:
public class DownloadService extends Service<Void> {
private static final long TIME_BUDGET = 2; // In seconds
private final ScheduledExecutorService watchdogService =
Executors.newSingleThreadScheduledExecutor(new ThreadFactory() {
private final ThreadFactory delegate = Executors.defaultThreadFactory();
#Override
public Thread newThread(Runnable r) {
Thread thread = delegate.newThread(r);
thread.setDaemon(true);
return thread;
}
});
private Future<?> watchdogThread;
private final ObjectProperty<URL> remoteResourceLocation = new SimpleObjectProperty<>();
private final ObjectProperty<Path> pathToLocalResource = new SimpleObjectProperty<>();
public final URL getRemoteResourceLocation() {
return remoteResourceLocation.get();
}
public final void setRemoteResourceLocation(URL remoteResourceLocation) {
this.remoteResourceLocation.set(remoteResourceLocation);
}
public ObjectProperty<URL> remoteResourceLocationProperty() {
return remoteResourceLocation;
}
public final Path getPathToLocalResource() {
return pathToLocalResource.get();
}
public final void setPathToLocalResource(Path pathToLocalResource) {
this.pathToLocalResource.set(pathToLocalResource);
}
public ObjectProperty<Path> pathToLocalResourceProperty() {
return pathToLocalResource;
}
#Override
protected Task<Void> createTask() {
final Path pathToLocalResource = getPathToLocalResource();
final URL remoteResourceLocation = getRemoteResourceLocation();
if (pathToLocalResource == null) {
throw new IllegalStateException("pathToLocalResource property value is null");
}
if (remoteResourceLocation == null) {
throw new IllegalStateException("remoteResourceLocation property value is null");
}
return new Task<Void>() {
#Override
protected Void call() throws IOException {
try (FileChannel channel = FileChannel.open(pathToLocalResource, StandardOpenOption.CREATE,
StandardOpenOption.WRITE)) {
channel.transferFrom(Channels.newChannel(remoteResourceLocation.openStream()), 0, Long.MAX_VALUE);
}
return null;
}
};
}
#Override
protected void running() {
watchdogThread = watchdogService.schedule(() -> {
Platform.runLater(() -> cancel());
}, TIME_BUDGET, TimeUnit.SECONDS);
}
#Override
protected void succeeded() {
watchdogThread.cancel(false);
}
#Override
protected void cancelled() {
watchdogThread.cancel(false);
}
#Override
protected void failed() {
watchdogThread.cancel(false);
}
}

There is one important aspect not covered by the other answers/comments; and that is a wrong assumption of yours:
What I want is it to fail immediately when no internet connection is there.
It is not that easy. The TCP stack/state machine is actually a pretty complicated thing; and depending on your context (OS type; TCP stack implementation, kernel parameters, ...), there can be situations where a network partition takes place and a sender doesn't notice for 15 or more minutes. Listen here for more details on that.
In other words: "just pulling the plug" is no way equal to "immediately breaking" your existing TCP connection. And just for the record: you don't need to plug cables manually to simulate network outages. In a reasonable test setup, tools like iptables aka firewalls can do that for you.

You seem to need an Asynchronous/Cancellable HTTP GET which can be tough.
The problem is that if read stalls waiting for more data (cable is pulled) it won't quit until either the socket dies or new data comes in.
There are a few path you could follow, tinkering with socket factories to set a good timeout, using http client with timeouts and others.
I would have a look at Apache Http Components which has non blocking HTTP based on java NIO Sockets.

How to make a Java thread hang

I am trying to create a solution to treat hung threads due to memory leaks, locked resources in our applications. One of the main problems I am having is trying to simulate a hung thread to deal with it. Any sugestions?
This is what I tried, but it just doesn't seem to do the job. Any thoughts?
class KillerThread extends Thread{
public KillerThread() {
super();
}
public KillerThread(String threadName) {
super(threadName);
}
public void run (){
System.out.println("Start of KillerThread " + this.getName() );
if ( System.currentTimeMillis() % 2L == 0 ){
try {
sleep(Long.MAX_VALUE);
} catch (InterruptedException e) {
e.printStackTrace();
}
} else {
for(;;);
}
}
}

Joining on one's own thread works well for me:
Thread.currentThread().join();

try running sleep in a while loop like:
while(true) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}

running a thread then tell it to sleep in an unstoppable loop, is a good idea,.
but how if you are trying to make it waiting another thread,.? make more than one thread and make them wait one each other, a deadlock condition, is that a hung to,.?

I know what you need exactly, you are testing something through stopping the executor thread. Try something like this:
private void testKillingThread() {
Object kill = new Object();
try {
synchronized (kill) {
kill.wait();
}
} catch (Exception e) {
// Auto-generated catch block
}
}

Simply enough, just create a private member
private Object lock = new Object();
then use it to wait for a notification (that will never happen, unless you use reflection...)
while (true) {
try {
synchronized (lock) {
lock.wait();
}
} cath (InterruptedException e) {
/* ignore interruption */
}
}
and you thread will hang there, uninterruptable.

Here's a quick fix I'm using for testing. Just have the thread you want to lock up call new Hanger().hang().
Remove the logging if you're not interested in seeing it. You can add throws InterruptedException (although, in fact, it never does) to the hang method so you can just replace a Thread.sleep() with a new Hanger().hang() without otherwise modifying your code.
public class Hanger {
private final static Logger log = Logger.getLogger(Hanger.class);
private long started = 0;
private final int beat = 100; // ms
/**
* Hangs a thread for the indicated time
* #param millis the amount of time to hang the thread, in milliseconds
*/
public void hang(int millis) {
started = System.currentTimeMillis();
log.debug("Hanging this thread for " + millis + " ms");
while (hung() < millis) {
try {
Thread.sleep(beat);
} catch (InterruptedException e) {
log.debug("Still hanging, will release in " + (millis - hung()) + " ms.");
}
}
log.debug("Releasing thread again after " + hung() + " ms");
}
private int hung() {
return (int)(System.currentTimeMillis() - started);
}
}