We Keep Coding

Java Thread listening to socket, WHILE loop not waiting

I am creating a socket server for a school project. The client is written in C# so the ObjectInputStream object I have been using so far is not working.
I am now trying to use InputStreamReader. In this example, the while (true) loop is running at full throttle all the time. Using the ObjectInputStream it'd wait for an object to arrive.
How can I do this in a proper way?
private InputStreamReader inputStream;
#Override
public void run() {
while (true) {
try {
// ObjectInputStream.read() waits for an object to arrive,
// so the execution is paused here
int data;
String string = new String();
while ( (data = inputStream.read() ) != -1) {
char thisChar = (char) data;
string = string + thisChar;
}
} catch (IOException ex) {
Logger.getLogger(Client.class.getName()).log(Level.SEVERE, null, ex);
break;
}
}
}

Get rid of the outer loop. It's pointless. The inner loop will run until end of stream. After that, further iteration is futile. You're spinning at EOS.

Exec'ing multiple processes from Java: outputs mixed up?

I have a servlet which creates a new Action object inside doGet(), and this object uses exec() to run external processes. There may be several requests to the servlet at the same time, so I might have several Action objects where each one is running an external process at the same time. Occasionally when this happens I find that the output from one process gets mixed up with the output from one of the others.
Each Action creates a unique temporary directory and runs the process with this as its current directory. Separate threads then read the output streams from the process into a string buffer. The code that the Action object executes looks basically like this:
Process proc = null;
ReaderThread stdout = null;
ReaderThread stderr = null;
StringBuffer buff = new StringBuffer();
int exit = -1;
try {
//
// Run the process
//
proc = Runtime.getRuntime().exec(command,null,directory);
//
// Read the output from the process
//
stdout = new ReaderThread(proc.getInputStream(),buff);
stderr = new ReaderThread(proc.getErrorStream(),buff);
stdout.start();
stderr.start();
//
// Get the exit code
//
exit = proc.waitFor();
//
// Wait for all the output to be read
//
stdout.join();
stderr.join();
}
catch (InterruptedException e) {
if (proc != null) {
proc.destroy();
}
}
catch (Exception e) {
buff.append(e.getClass() + ": " + e.getMessage());
if (proc != null) {
proc.destroy();
}
}
So each request uses a separate Action object to run a process, and this has its own StringBuffer "buff" that the output of the process is accumulated into by the two ReaderThreads. But what I find is that, when two requests are running two processes at the same time, the output of one will sometimes end up in the StringBuffer of the thread that is running the other one, and one of the two servlet requests will see output intended for the other one. It basically behaves as if Runtime.exec() provides a single global pipe to which the output streams of all the processes are connected.
The ReaderThread looks like this:
public class ReaderThread extends Thread {
private BufferedReader reader;
private StringBuffer buffer;
public ReaderThread (InputStream stream, StringBuffer buffer) {
this.reader = new BufferedReader(new InputStreamReader(stream));
this.buffer = buffer;
}
#Override
public void run () {
try {
String line;
while ((line = reader.readLine()) != null) {
synchronized (buffer) {
buffer.append(line + "\n");
}
}
}
catch (IOException e) {
synchronized (buffer) {
buffer.append(e.getMessage() + "\n");
}
}
}
}
Can anyone suggest what I can do to fix this?

Use a ThreadLocal variable to store the output of each thread.
When and how should I use a ThreadLocal variable?

Here's the explanation, partly as a cautionary tale:
the output was being added to an ArrayList in the XML node that
accessed it
the XML nodes were created by cloning prototypes
the ArrayList was initialised in the declaration, not in the initialise()
method of the class, so every instance ended up referring to the same
object.
Duh.
Another two days of my life down the drain!
Happy new year...

Reading Input and Error Streams Concurrently using BufferedReaders Hangs

First off let me apologize to the SO community for coming to you with something that ought to be so trivial. But I've been at this all day and I'm at the end of my rope.
There is a section of my program that needs pull text from an input stream and an error stream from a process that is launched using Runtime.getrunTime().exec() and pass it through to standard input and output in an orderly manner. I have a function that near as I can tell should work. But it seems to be getting caught in a catch-22 where it's waiting for the stream to report ready - but the stream has finished and is not reporting. I'm baffled. I can't think of another way to do this that fits my constraints and I'm rather skeptical that such a catch-22 can exist.
Here is my code:
private void forwardStreamtoStd(InputStream in, InputStream err)
throws IOException {
int c = -1;
BufferedReader inReader = new BufferedReader(
new InputStreamReader(in, "US-ASCII"));
BufferedReader errReader = new BufferedReader(
new InputStreamReader(err, "US-ASCII"));
boolean inFinished = false, errFinished = false;
try {
System.out.println("Begin stream read loop...");
while (!inFinished && !errFinished) {
if (!inFinished) {
while (inReader.ready()) {
if ((c = inReader.read()) == -1) {
inFinished = true;
}
else {
System.out.print((char) c);
}
}
}
if (!errFinished) {
while (errReader.ready()) {
if ((c = errReader.read()) == -1) {
errFinished = true;
}
else {
System.err.print((char) c);
}
}
}
}
System.out.println("End stream read loop.");
}
catch (IOException e) {
throw e;
}
finally {
errReader.close();
inReader.close();
}
}
The problem seems to be that the reading loops are waiting for the streams to report ready, and as a result aren't seeing the -1 returned by read telling them that it's time to quit. I'm trying to avoid having either stream blocking, so that I can pull from both in turn when they are prepared. However, how can I catch the process's end of stream? Am I missing something? Shouldn't read report that it's read when it has an end of stream -1? The processes are finishing, and so their streams should be dying. What am I doing wrong here?

There are two more possibilities:
Use the ProcessBuilder and invoke redirectErrorStream(true) to join the two streams and you need to read one stream. I have an example here.
In JDK7, you could call the inheritIO() to automatically forward everything
Edit On the second guess, it seems the ready() call is misleading your program. Try this:
private void forwardStreamtoStd(InputStream in, InputStream err)
throws IOException {
int c = -1;
BufferedReader inReader = new BufferedReader(
new InputStreamReader(in, "US-ASCII"));
BufferedReader errReader = new BufferedReader(
new InputStreamReader(err, "US-ASCII"));
boolean inFinished = false, errFinished = false;
try {
System.out.println("Begin stream read loop...");
if (!inFinished) {
while ((c = inReader.read()) != -1) {
System.out.print((char) c);
}
inFinished = true;
}
if (!errFinished) {
while ((c = errReader.read()) != -1) {
System.err.print((char) c);
}
errFinished = true;
}
System.out.println("End stream read loop.");
}
catch (IOException e) {
throw e;
}
finally {
errReader.close();
inReader.close();
}
}
Or better yet, leave off the BufferedReader if you don't plan any extra transformation:
private void createReader(final InputStream in, final OutputStream out) {
new Thread() {
public void run() {
try {
int c = 0;
while ((c = in.read()) != -1) {
out.write(c);
}
} catch (IOException ex) {
ex.printStackTrace();
} finally {
in.close();
}
}
}.start();
}
private void forwardStreamtoStd(InputStream in, InputStream err)
throws IOException {
createReader(in, System.out);
createReader(err, System.err);
}

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4090471
The solution I've always used is to create a separate thread to read one of the streams, join on the thread when the main thread finishes reading, and then waitFor the process.

It's essential to consume the 2 streams concurrently, to prevent blocking. See this article for more info, and in particular note the StreamGobbler mechanism that captures stdout/err in separate threads.

If I remember correctly, the spawned process will never close the stream - so you would need to have the readers in their own threads, sleep on the main thread until the process is done, and then close the readers.

Implementing the "system" command in Java

I have need for a "system" function call, the same as those in Python, Perl, PHP, Ruby, &c. It will be a component of a JavaScript standard library called Narwhal, when it's run on the Rhino JavaScript engine, which is in turn run on Java.
The trouble is that Java's standard library appears to have abstracted away the ability to spawn a subprocess that shares the parent process's stdio. This means that you can't defer interactivity to the subprocess.
My first crack at this was to implement Python's subprocess.popen. This uses three "pumper" threads to actively copy the parent process's stdio independently (to prevent deadlock). Unfortunately this is giving us two problems. First, the input does not close automatically when the sub-process voluntarily exits. Second, the streams to the child process do not buffer and flush properly.
I'm looking for solutions that would make our require("os").system() command work as one would expect.
The project is at http://narwhaljs.org
Relevant code:
http://github.com/tlrobinson/narwhal/blob/d147c160f11fdfb7f3c0763acf352b2b0e2713f7/lib/os.js#L10
http://github.com/tlrobinson/narwhal/blob/d147c160f11fdfb7f3c0763acf352b2b0e2713f7/engines/rhino/lib/os-engine.js#L37

Not sure if this is what you're looking for, but you can invoke the C system function through the JNA library:
public class System {
public interface C extends Library {
C INSTANCE = (C) Native.loadLibrary(
(Platform.isWindows() ? "msvcrt" : "c"), C.class);
public int system(String format);
}
public static void main(String[] args) {
C.INSTANCE.system("vi");
}
}
Cursory testing worked on Windows, anyhow.

If I'm understanding you properly, you want something like that:
import java.util.*;
import java.io.*;
class StreamGobbler extends Thread
{
InputStream is;
String type;
OutputStream os;
StreamGobbler(InputStream is, String type)
{
this(is, type, null);
}
StreamGobbler(InputStream is, String type, OutputStream redirect)
{
this.is = is;
this.type = type;
this.os = redirect;
}
public void run()
{
try
{
PrintWriter pw = null;
if (os != null)
pw = new PrintWriter(os);
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line=null;
while ( (line = br.readLine()) != null)
{
if (pw != null)
pw.println(line);
System.out.println(type + ">" + line);
}
if (pw != null)
pw.flush();
} catch (IOException ioe)
{
ioe.printStackTrace();
}
}
}
public class GoodWinRedirect
{
public static void main(String args[])
{
if (args.length < 1)
{
System.out.println("USAGE java GoodWinRedirect <outputfile>");
System.exit(1);
}
try
{
FileOutputStream fos = new FileOutputStream(args[0]);
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec("java jecho 'Hello World'");
// any error message?
StreamGobbler errorGobbler = new
StreamGobbler(proc.getErrorStream(), "ERROR");
// any output?
StreamGobbler outputGobbler = new
StreamGobbler(proc.getInputStream(), "OUTPUT", fos);
// kick them off
errorGobbler.start();
outputGobbler.start();
// any error???
int exitVal = proc.waitFor();
System.out.println("ExitValue: " + exitVal);
fos.flush();
fos.close();
} catch (Throwable t)
{
t.printStackTrace();
}
}
}
I found this piece of code in: JavaWorld sometime ago when I was looking for a similar solution to wrap system calls to some exe files.
My code has evolved a bit since then, but I think is a good example.

You need to monitor the process exit-status separately, either by polling the exit code or by having a separate thread waiting in the Process.waitFor() method.
On the matter of buffering and flushing the streams, I don't thing there is an easy solution. There are several Java-classes which do buffering in different forms (BufferedInputStream, etc). Maybe one of them can help?

It's really important to consume the process standard output and error concurrently. See Carlos Tasada's example code elsewhere here.
If you don't do this your code may (or may not) work depending on the output from your spawned process. As and when that output changes (if your spawned process encounters an error, say) then without the concurrent consumption your process will deadlock. Most of the issues I see on SO related to Process.exec() are blocking related.

Run external program from Java, read output, allow interruption

I want to launch a process from Java, read its output, and get its return code. But while it's executing, I want to be able to cancel it. I start out by launching the process:
ProcessBuilder pb = new ProcessBuilder(args);
pb.redirectErrorStream(true);
Process proc = pb.start();
If I call proc.waitFor(), I can't do anything until the process exits. So I'm assuming I need to something like this:
while (true) {
see if process has exited
capture some output from the process
decide if I want to cancel it, and if so, cancel it
sleep for a while
}
Is this right? Can someone give me an example of how to do this in Java?

Here's an example of what I think you want to do:
ProcessBuilder pb = new ProcessBuilder(args);
pb.redirectErrorStream(true);
Process proc = pb.start();
InputStream is = proc.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line;
int exit = -1;
while ((line = br.readLine()) != null) {
// Outputs your process execution
System.out.println(line);
try {
exit = proc.exitValue();
if (exit == 0) {
// Process finished
}
} catch (IllegalThreadStateException t) {
// The process has not yet finished.
// Should we stop it?
if (processMustStop())
// processMustStop can return true
// after time out, for example.
proc.destroy();
}
}
You can improve it :-) I don't have a real environment to test it now, but you can find some more information here.

I recommend checking out Apache Commons Exec to avoid recreating the wheel. It has some nice features like choosing between synchronous vs. asynchronous execution, as well as a standard solution to spawning a watchdog process that can help in timing out the execution in case it gets stuck.

A helper class like this would do the trick:
public class ProcessWatcher implements Runnable {
private Process p;
private volatile boolean finished = false;
public ProcessWatcher(Process p) {
this.p = p;
new Thread(this).start();
}
public boolean isFinished() {
return finished;
}
public void run() {
try {
p.waitFor();
} catch (Exception e) {}
finished = true;
}
}
You would then implement your loop exactly as you describe:
Process p = Runtime.getRuntime().exec("whatever command");
ProcessWatcher pw = new ProcessWatcher(p);
InputStream output = p.getInputStream();
while(!pw.isFinished()) {
processOutput(output);
if(shouldCancel()) p.destroy();
Thread.sleep(500);
}
Depending upon what conditions would make you want to destroy the process, you might want to do that in a separate thread. Otherwise, you may block while waiting for more program output to process and never really get the option to destroy it.
EDIT: McDowell is 100% right in his comment below, so I've made the finished variable volatile.

How about this (see how it works in jcabi-heroku-maven-plugin):
/**
* Wait for the process to stop, logging its output in parallel.
* #param process The process to wait for
* #return Stdout produced by the process
* #throws InterruptedException If interrupted in between
*/
private String waitFor(final Process process) throws InterruptedException {
final BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream())
);
final CountDownLatch done = new CountDownLatch(1);
final StringBuffer stdout = new StringBuffer();
new Thread(
new VerboseRunnable(
new Callable<Void>() {
#Override
public Void call() throws Exception {
while (true) {
final String line = reader.readLine();
if (line == null) {
break;
}
System.out.println(">> " + line);
stdout.append(line);
}
done.countDown();
return null;
}
},
false
)
).start();
try {
process.waitFor();
} finally {
done.await();
IOUtils.closeQuietly(reader);
}
return stdout.toString();
}
ps. Now this implementation is available as com.jcabi.log.VerboseProcess class from jcabi-log artifact.

What would make you decide to kill the process -- an asynchronous event (such as input from the user), or a synchronous event (e.g., the process has done what you wanted it to do)? I'm guessing it's the former -- input from the user makes you decide to cancel the subprocess.
Also, how much output do you expect the subprocess to produce? If it's a lot, then the subprocess may block if you don't read from its output stream quickly enough.
Your situation may vary, but it seems that you're likely going to need at least two different threads -- one to decide whether to cancel the process, and one that handles the output from the subprocess.
Have a look here for a bit more detail: http://java.sun.com/developer/JDCTechTips/2005/tt0727.html#2