We Keep Coding

Is there a better way to implement multi-threading REPL?

I'm trying to implement a REPL application.
There are two threads, which are REPL and a loop function. The function will print messages from time to time, and I need the REPL to get the command of input then dispatch it.
This is my implement method (I use JLine):
Thread replThread = new Thread(() -> {
try {
Terminal terminal = TerminalBuilder.builder().system(true).build();
LineReader reader = LineReaderBuilder.builder().terminal(terminal).build();
for (;;) {
String line;
line = reader.readLine("> ");
System.out.println("input command: " + line);
}
} catch (IOException e) {
e.printStackTrace();
}
});
Thread loopThread = new Thread(() -> {
for (;;) {
try {
System.out.println("running");
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
replThread.start();
loopThread.start();
This is the running screenshot
It shows that the prompt is overlapped with the prompt, and two threads' output conflicts with each other. I need the prompt (such as > ) to be always in the last line on the console.
So is there a better way or library can implement this requirement?

How to run java.lang.Process one by one?

I need help in running maven command one by one through Java.
In my code I am trying to run maven build command and print the log to Swing TextArea window.
But the problem is Process.waitFor() is not working properly. As a result multiple processes
are running simultaneously and logs are not getting printed properly.
In Internet I found that waitFor() sometimes doesnt work. Instead I can use
Process.isAlive(). But I am not sure in my code how would I use that method.
Please help
Timer timer = new Timer(300, new ActionListener() {
#Override
public void actionPerformed(ActionEvent e) {
String line;
for (int i = 0; i < process.length; i++) {
BufferedReader buffer=new BufferedReader(new InputStreamReader(process[i].getInputStream()));
try {
if ((line = buffer.readLine()) != null) {
logTextArea[i].append(line + "\n");
} else {
((Timer) e.getSource()).stop();
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
});
try {
for ( int count = 0 ; count < destList.getModel().getSize(); count++){
String projectPath = findDir(new File(codebasePath), destList.getModel().getElementAt(count).toString());
process[count] = Runtime.getRuntime().exec(System.getenv("M2_HOME")+"//bin//mvn.bat clean install -DskipTests -f "+projectPath + "\\pom.xml");
logTextArea[count] = new JTextArea(25, 60);
logFrame[count] = new JFrame("Show Log");
logFrame[count].add(new JScrollPane(logTextArea[count]), BorderLayout.CENTER);
logFrame[count].setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
logFrame[count].pack();
logFrame[count].setLocationByPlatform(true);
logFrame[count].setVisible(true);
timer.start();
process[count].waitFor()
}
} catch (IOException e) {
e.printStackTrace();
}

I am a bit puzzled where exactly you struggle, so maybe I am overlooking something, but you'd generally use it like so:
process[count].waitFor();
while (process[count].isAlive()) {
Thread.sleep(100);
}
You might want to get rid of waitFor completely and add some additional checks to the loop, e.g. a timeout or a signal to cancel the process, in case the process hangs, so your application can terminate properly:
while (process[count].isAlive()) {
if (someTimeoutCondition || someCancelCondition) {
// throw exception or do whatever else you want to do in this situation to gracefully exit
}
Thread.sleep(100);
}

Java exec stdin not flushing

For a school project I am trying to create a terminal in Java. The terminal works in the following way:
User types a command
Program grabs command and replaces <command> with the command in the string
/bin/bash -c "cd current/directory/; <command>; echo kjsfdjkadhlga; pwd
Program starts the process created via a ProcessBuilder object
Program spawns a thread that reads from stdout and stderr
Program continues looking for user input, and if the command is done running, then whatever the user entered is run as a command, otherwise it is fed to the currently running command as input.
As output is generated, program looks through the output for the kjsfdjkadhlga string so it knows when the user's command is done being run, and then grabs the remaining output and stores it as the current path that the user is at.
How this works/reasons for everything:
In order to avoid myself having to implement my own input parser to handle things like multiple commands on a line, IO redirection, and whatnot to work with the ProcessBuilder, I just essentially convert the command to a bash script and let bash execute it.
Since every process executes only a single command (or whatever it was given at the time of creation, which is a single user command in this case) then terminates, no process specific information is stored, such as the current working directory. To transfer that information, I call pwd after the user's command and then in the process of the next command, but before the user's command is run, I cd to that directory, effectively allowing the value of $PWD to persist between processes.
The Problem:
It all works well, except for when user interaction is required. If the user just types cat, it is supposed to wait for a line of user input, then print it, then wait for a line of user input, then print it, and repeat forever (I don't handle Crtl+C yet...). However, what actually happens is that the terminal waits for a line of user input, then prints it, then terminates without waiting for more input.
What I have tried:
Currently, I provide input to the command being run with:
BufferedWriter stdin = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
stdin.write(input);
stdin.newLine();
stdin.close();
If instead of calling close(), I call flush(), then cat ends up waiting for user input and not doing anything until I terminate my Terminal program, at which point it then prints everything the user had input.
It appears that the flush() function doesn't actually do anything. A Stack Overflow question mentioned using the raw OutputStream and calling write() instead of using a BufferedWriter. However, that has the same effect. In the OutputStream documentation for flush(), it states that "The flush method of OutputStream does nothing."
I have also tried using a BufferedOutputStream, but the documentation says that its flush function simply forces the buffered data to be written to the underlying OutputStream, which doesn't change the fact that the OutputStream is not flushing its stream.
This question seems to be the most promising, but I couldn't get it to work when implementing it. It may be because I am on Mac OS instead of Windows.
Does anybody know how to do this if keeping stdin open long enough to submit multiple lines of input is possible, or if I am going about it wrong?
Code
main()
Terminal terminal = new Terminal();
Scanner in = new Scanner(System.in);
while (in.hasNextLine())
{
String line = in.nextLine();
terminal.sendInput(line, terminal);
}
terminal.sendInput() called by main
// ProcessReaderDelegate implements functions called when receiving output on stdout, stderr, and when the process terminates.
public int sendInput(String text, ProcessReaderDelegate delegate)
{
if (processes.size() > 0)
{
processes.get(0).sendInput(text); // Is a ProcessReader object
return 1;
}
run(text, delegate); // runs the given text as the <command> text described above
return 2;
}
ProcessReader's sendInput() called by terminal.sendInput()
public boolean sendInput(String input)
{
try
{
// stdin and process are a instance fields
// tried this and doesn't seem to work (with either flush or close)
stdin = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
stdin.write(input);
stdin.newLine();
stdin.close();
// tried this and doesn't seem to work (with either flush or close)
//BufferedOutputStream os = new BufferedOutputStream(process.getOutputStream());
//os.write(input.getBytes());
//os.write("\n".getBytes());
//os.flush();
//os.close();
return true;
}
catch (IOException e)
{
System.out.println("ERROR: this should never happen: " + e.getMessage());
return false;
}
}
terminal.run() called by terminal.sendInput()
public void run(String command, ProcessReaderDelegate delegate)
{
// don't do anything with empty command since it screws up the command concatentaion later
if (command.equals(""))
{
delegate.receivedOutput(null, prompt);
return;
}
try
{
// create the command
List<String> list = new ArrayList<String>();
list.add(shellPath);
list.add(UNIX_BASED ? "-c" : "Command : ");
String cmd = (UNIX_BASED ? getUnixCommand(command) : getWindowsCommand(command));
list.add(cmd);
//System.out.println("command='" + list.get(0) + " " + list.get(1) + " " + list.get(2) + "'");
// create the process and run it
ProcessBuilder builder = new ProcessBuilder(list);
Process p = builder.start();
ProcessReader stdout = new ProcessReader(p, delegate, this);
new Thread(stdout).start();
processes.add(stdout);
}
catch (IOException e)
{
System.out.println(e.getMessage());
}
}
ProcessReader.run() executed in thread and reads stdout and stderr
public void run()
{
try
{
boolean hitend = false;
String buffer = "";
while (true)
{
int c;
String text;
// ======================================================
// read from stdout
// read the next character
c = stdout.read();
// build the string
while (c != -1) // while data available in the stream
{
buffer += (char)c;
c = stdout.read();
}
// send the string to the delegate
if ((!hitend) && (buffer.length() > 0))
{
// END_STRING is the "kjsfdjkadhlga" echoed after the command executes
int index = buffer.indexOf(END_STRING);
if (index >= 0)
{
hitend = true;
text = buffer.substring(0, index);
buffer = buffer.substring(index + END_STRING.length());
if (outputDelegate != null)
{
outputDelegate.receivedOutput(process, text);
}
}
else
{
for (int i = END_STRING.length() - 1; i >= 0; i--)
{
index = buffer.indexOf(END_STRING.substring(0, i));
if (i == 0)
{
index = buffer.length();
}
if (index >= 0)
{
text = buffer.substring(0, index);
buffer = buffer.substring(index + i);
if (outputDelegate != null)
{
outputDelegate.receivedOutput(process, text);
}
}
}
}
}
// ======================================================
// read from stderr
// read the next character
c = stderr.read();
text = ""; // slow method; make faster with array
// build the string
while (c != -1) // while data available in the stream
{
text += (char)c;
c = stderr.read();
}
// send the string to the delegate
if ((text.length() > 0) && (outputDelegate != null))
{
outputDelegate.receivedError(process, text);
}
// ======================================================
// check if the process is done (and hence no more output)
boolean done = false;
try
{
int value = process.exitValue();
done = true; // if got to this point, then process is done
// read the ending environment variables
Map<String, String> env = new HashMap<String, String>();
String[] words = buffer.split(" ");
env.put(ENV_WORKING_DIR, words[0]);
if (envDelegate != null)
{
envDelegate.processTerminatedWithEnvironment(process, env);
}
// end the process
outputDelegate.processEnded(process);
stdout.close();
stderr.close();
break;
}
catch (Exception e) {System.out.println(e.getMessage());} // no exit value --> process not done
if (done) // just on the off chance that closing the streams crashes everything
{
break;
}
}
}
catch (IOException e)
{
System.out.println("ERROR: ProcessReader: " + e.getMessage());
}
}

Reading Input and Error Streams Concurrently using BufferedReaders Hangs

First off let me apologize to the SO community for coming to you with something that ought to be so trivial. But I've been at this all day and I'm at the end of my rope.
There is a section of my program that needs pull text from an input stream and an error stream from a process that is launched using Runtime.getrunTime().exec() and pass it through to standard input and output in an orderly manner. I have a function that near as I can tell should work. But it seems to be getting caught in a catch-22 where it's waiting for the stream to report ready - but the stream has finished and is not reporting. I'm baffled. I can't think of another way to do this that fits my constraints and I'm rather skeptical that such a catch-22 can exist.
Here is my code:
private void forwardStreamtoStd(InputStream in, InputStream err)
throws IOException {
int c = -1;
BufferedReader inReader = new BufferedReader(
new InputStreamReader(in, "US-ASCII"));
BufferedReader errReader = new BufferedReader(
new InputStreamReader(err, "US-ASCII"));
boolean inFinished = false, errFinished = false;
try {
System.out.println("Begin stream read loop...");
while (!inFinished && !errFinished) {
if (!inFinished) {
while (inReader.ready()) {
if ((c = inReader.read()) == -1) {
inFinished = true;
}
else {
System.out.print((char) c);
}
}
}
if (!errFinished) {
while (errReader.ready()) {
if ((c = errReader.read()) == -1) {
errFinished = true;
}
else {
System.err.print((char) c);
}
}
}
}
System.out.println("End stream read loop.");
}
catch (IOException e) {
throw e;
}
finally {
errReader.close();
inReader.close();
}
}
The problem seems to be that the reading loops are waiting for the streams to report ready, and as a result aren't seeing the -1 returned by read telling them that it's time to quit. I'm trying to avoid having either stream blocking, so that I can pull from both in turn when they are prepared. However, how can I catch the process's end of stream? Am I missing something? Shouldn't read report that it's read when it has an end of stream -1? The processes are finishing, and so their streams should be dying. What am I doing wrong here?

There are two more possibilities:
Use the ProcessBuilder and invoke redirectErrorStream(true) to join the two streams and you need to read one stream. I have an example here.
In JDK7, you could call the inheritIO() to automatically forward everything
Edit On the second guess, it seems the ready() call is misleading your program. Try this:
private void forwardStreamtoStd(InputStream in, InputStream err)
throws IOException {
int c = -1;
BufferedReader inReader = new BufferedReader(
new InputStreamReader(in, "US-ASCII"));
BufferedReader errReader = new BufferedReader(
new InputStreamReader(err, "US-ASCII"));
boolean inFinished = false, errFinished = false;
try {
System.out.println("Begin stream read loop...");
if (!inFinished) {
while ((c = inReader.read()) != -1) {
System.out.print((char) c);
}
inFinished = true;
}
if (!errFinished) {
while ((c = errReader.read()) != -1) {
System.err.print((char) c);
}
errFinished = true;
}
System.out.println("End stream read loop.");
}
catch (IOException e) {
throw e;
}
finally {
errReader.close();
inReader.close();
}
}
Or better yet, leave off the BufferedReader if you don't plan any extra transformation:
private void createReader(final InputStream in, final OutputStream out) {
new Thread() {
public void run() {
try {
int c = 0;
while ((c = in.read()) != -1) {
out.write(c);
}
} catch (IOException ex) {
ex.printStackTrace();
} finally {
in.close();
}
}
}.start();
}
private void forwardStreamtoStd(InputStream in, InputStream err)
throws IOException {
createReader(in, System.out);
createReader(err, System.err);
}

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4090471
The solution I've always used is to create a separate thread to read one of the streams, join on the thread when the main thread finishes reading, and then waitFor the process.

It's essential to consume the 2 streams concurrently, to prevent blocking. See this article for more info, and in particular note the StreamGobbler mechanism that captures stdout/err in separate threads.

If I remember correctly, the spawned process will never close the stream - so you would need to have the readers in their own threads, sleep on the main thread until the process is done, and then close the readers.

How do I handle multiple streams in Java?

I'm trying to run a process and do stuff with its input, output and error streams. The obvious way to do this is to use something like select(), but the only thing I can find in Java that does that is Selector.select(), which takes a Channel. It doesn't appear to be possible to get a Channel from an InputStream or OutputStream (FileStream has a getChannel() method but that doesn't help here)
So, instead I wrote some code to poll all the streams:
while( !out_eof || !err_eof )
{
while( out_str.available() )
{
if( (bytes = out_str.read(buf)) != -1 )
{
// Do something with output stream
}
else
out_eof = true;
}
while( err_str.available() )
{
if( (bytes = err_str.read(buf)) != -1 )
{
// Do something with error stream
}
else
err_eof = true;
}
sleep(100);
}
which works, except that it never terminates. When one of the streams reaches end of file, available() returns zero so read() isn't called and we never get the -1 return that would indicate EOF.
One solution would be a non-blocking way to detect EOF. I can't see one in the docs anywhere. Alternatively is there a better way of doing what I want to do?
I see this question here:
link text
and although it doesn't exactly do what I want, I can probably use that idea, of spawning separate threads for each stream, for the particular problem I have now. But surely that isn't the only way to do it? Surely there must be a way to read from multiple streams without using a thread for each?

As you said, the solution outlined in this Answer is the traditional way of reading both stdout and stderr from a Process. A thread-per-stream is the way to go, even though it is slightly annoying.

You will indeed have to go the route of spawning a Thread for each stream you want to monitor. If your use case allows for combining both stdout and stderr of the process in question you need only one thread, otherwise two are needed.
It took me quite some time to get it right in one of our projects where I have to launch an external process, take its output and do something with it while at the same time looking for errors and process termination and also being able to terminate it when the java app's user cancels the operation.
I created a rather simple class to encapsulate the watching part whose run() method looks something like this:
public void run() {
BufferedReader tStreamReader = null;
try {
while (externalCommand == null && !shouldHalt) {
logger.warning("ExtProcMonitor("
+ (watchStdErr ? "err" : "out")
+ ") Sleeping until external command is found");
Thread.sleep(500);
}
if (externalCommand == null) {
return;
}
tStreamReader =
new BufferedReader(new InputStreamReader(watchStdErr ? externalCommand.getErrorStream()
: externalCommand.getInputStream()));
String tLine;
while ((tLine = tStreamReader.readLine()) != null) {
logger.severe(tLine);
if (filter != null) {
if (filter.matches(tLine)) {
informFilterListeners(tLine);
return;
}
}
}
} catch (IOException e) {
logger.logExceptionMessage(e, "IOException stderr");
} catch (InterruptedException e) {
logger.logExceptionMessage(e, "InterruptedException waiting for external process");
} finally {
if (tStreamReader != null) {
try {
tStreamReader.close();
} catch (IOException e) {
// ignore
}
}
}
}
On the calling side it looks like this:
Thread tExtMonitorThread = new Thread(new Runnable() {
public void run() {
try {
while (externalCommand == null) {
getLogger().warning("Monitor: Sleeping until external command is found");
Thread.sleep(500);
if (isStopRequested()) {
getLogger()
.warning("Terminating external process on user request");
if (externalCommand != null) {
externalCommand.destroy();
}
return;
}
}
int tReturnCode = externalCommand.waitFor();
getLogger().warning("External command exited with code " + tReturnCode);
} catch (InterruptedException e) {
getLogger().logExceptionMessage(e, "Interrupted while waiting for external command to exit");
}
}
}, "ExtCommandWaiter");
ExternalProcessOutputHandlerThread tExtErrThread =
new ExternalProcessOutputHandlerThread("ExtCommandStdErr", getLogger(), true);
ExternalProcessOutputHandlerThread tExtOutThread =
new ExternalProcessOutputHandlerThread("ExtCommandStdOut", getLogger(), true);
tExtMonitorThread.start();
tExtOutThread.start();
tExtErrThread.start();
tExtErrThread.setFilter(new FilterFunctor() {
public boolean matches(Object o) {
String tLine = (String)o;
return tLine.indexOf("Error") > -1;
}
});
FilterListener tListener = new FilterListener() {
private boolean abortFlag = false;
public boolean shouldAbort() {
return abortFlag;
}
public void matched(String aLine) {
abortFlag = abortFlag || (aLine.indexOf("Error") > -1);
}
};
tExtErrThread.addFilterListener(tListener);
externalCommand = new ProcessBuilder(aCommand).start();
tExtErrThread.setProcess(externalCommand);
try {
tExtMonitorThread.join();
tExtErrThread.join();
tExtOutThread.join();
} catch (InterruptedException e) {
// when this happens try to bring the external process down
getLogger().severe("Aborted because auf InterruptedException.");
getLogger().severe("Killing external command...");
externalCommand.destroy();
getLogger().severe("External command killed.");
externalCommand = null;
return -42;
}
int tRetVal = tListener.shouldAbort() ? -44 : externalCommand.exitValue();
externalCommand = null;
try {
getLogger().warning("command exit code: " + tRetVal);
} catch (IllegalThreadStateException ex) {
getLogger().warning("command exit code: unknown");
}
return tRetVal;
Unfortunately I don't have to for a self-contained runnable example, but maybe this helps.
If I had to do it again I would have another look at using the Thread.interrupt() method instead of a self-made stop flag (mind to declare it volatile!), but I leave that for another time. :)