I'm trying to run a process and do stuff with its input, output and error streams. The obvious way to do this is to use something like select(), but the only thing I can find in Java that does that is Selector.select(), which takes a Channel. It doesn't appear to be possible to get a Channel from an InputStream or OutputStream (FileStream has a getChannel() method but that doesn't help here)
So, instead I wrote some code to poll all the streams:
while( !out_eof || !err_eof )
{
while( out_str.available() )
{
if( (bytes = out_str.read(buf)) != -1 )
{
// Do something with output stream
}
else
out_eof = true;
}
while( err_str.available() )
{
if( (bytes = err_str.read(buf)) != -1 )
{
// Do something with error stream
}
else
err_eof = true;
}
sleep(100);
}
which works, except that it never terminates. When one of the streams reaches end of file, available() returns zero so read() isn't called and we never get the -1 return that would indicate EOF.
One solution would be a non-blocking way to detect EOF. I can't see one in the docs anywhere. Alternatively is there a better way of doing what I want to do?
I see this question here:
link text
and although it doesn't exactly do what I want, I can probably use that idea, of spawning separate threads for each stream, for the particular problem I have now. But surely that isn't the only way to do it? Surely there must be a way to read from multiple streams without using a thread for each?
As you said, the solution outlined in this Answer is the traditional way of reading both stdout and stderr from a Process. A thread-per-stream is the way to go, even though it is slightly annoying.
You will indeed have to go the route of spawning a Thread for each stream you want to monitor. If your use case allows for combining both stdout and stderr of the process in question you need only one thread, otherwise two are needed.
It took me quite some time to get it right in one of our projects where I have to launch an external process, take its output and do something with it while at the same time looking for errors and process termination and also being able to terminate it when the java app's user cancels the operation.
I created a rather simple class to encapsulate the watching part whose run() method looks something like this:
public void run() {
BufferedReader tStreamReader = null;
try {
while (externalCommand == null && !shouldHalt) {
logger.warning("ExtProcMonitor("
+ (watchStdErr ? "err" : "out")
+ ") Sleeping until external command is found");
Thread.sleep(500);
}
if (externalCommand == null) {
return;
}
tStreamReader =
new BufferedReader(new InputStreamReader(watchStdErr ? externalCommand.getErrorStream()
: externalCommand.getInputStream()));
String tLine;
while ((tLine = tStreamReader.readLine()) != null) {
logger.severe(tLine);
if (filter != null) {
if (filter.matches(tLine)) {
informFilterListeners(tLine);
return;
}
}
}
} catch (IOException e) {
logger.logExceptionMessage(e, "IOException stderr");
} catch (InterruptedException e) {
logger.logExceptionMessage(e, "InterruptedException waiting for external process");
} finally {
if (tStreamReader != null) {
try {
tStreamReader.close();
} catch (IOException e) {
// ignore
}
}
}
}
On the calling side it looks like this:
Thread tExtMonitorThread = new Thread(new Runnable() {
public void run() {
try {
while (externalCommand == null) {
getLogger().warning("Monitor: Sleeping until external command is found");
Thread.sleep(500);
if (isStopRequested()) {
getLogger()
.warning("Terminating external process on user request");
if (externalCommand != null) {
externalCommand.destroy();
}
return;
}
}
int tReturnCode = externalCommand.waitFor();
getLogger().warning("External command exited with code " + tReturnCode);
} catch (InterruptedException e) {
getLogger().logExceptionMessage(e, "Interrupted while waiting for external command to exit");
}
}
}, "ExtCommandWaiter");
ExternalProcessOutputHandlerThread tExtErrThread =
new ExternalProcessOutputHandlerThread("ExtCommandStdErr", getLogger(), true);
ExternalProcessOutputHandlerThread tExtOutThread =
new ExternalProcessOutputHandlerThread("ExtCommandStdOut", getLogger(), true);
tExtMonitorThread.start();
tExtOutThread.start();
tExtErrThread.start();
tExtErrThread.setFilter(new FilterFunctor() {
public boolean matches(Object o) {
String tLine = (String)o;
return tLine.indexOf("Error") > -1;
}
});
FilterListener tListener = new FilterListener() {
private boolean abortFlag = false;
public boolean shouldAbort() {
return abortFlag;
}
public void matched(String aLine) {
abortFlag = abortFlag || (aLine.indexOf("Error") > -1);
}
};
tExtErrThread.addFilterListener(tListener);
externalCommand = new ProcessBuilder(aCommand).start();
tExtErrThread.setProcess(externalCommand);
try {
tExtMonitorThread.join();
tExtErrThread.join();
tExtOutThread.join();
} catch (InterruptedException e) {
// when this happens try to bring the external process down
getLogger().severe("Aborted because auf InterruptedException.");
getLogger().severe("Killing external command...");
externalCommand.destroy();
getLogger().severe("External command killed.");
externalCommand = null;
return -42;
}
int tRetVal = tListener.shouldAbort() ? -44 : externalCommand.exitValue();
externalCommand = null;
try {
getLogger().warning("command exit code: " + tRetVal);
} catch (IllegalThreadStateException ex) {
getLogger().warning("command exit code: unknown");
}
return tRetVal;
Unfortunately I don't have to for a self-contained runnable example, but maybe this helps.
If I had to do it again I would have another look at using the Thread.interrupt() method instead of a self-made stop flag (mind to declare it volatile!), but I leave that for another time. :)
Related
I need help in running maven command one by one through Java.
In my code I am trying to run maven build command and print the log to Swing TextArea window.
But the problem is Process.waitFor() is not working properly. As a result multiple processes
are running simultaneously and logs are not getting printed properly.
In Internet I found that waitFor() sometimes doesnt work. Instead I can use
Process.isAlive(). But I am not sure in my code how would I use that method.
Please help
Timer timer = new Timer(300, new ActionListener() {
#Override
public void actionPerformed(ActionEvent e) {
String line;
for (int i = 0; i < process.length; i++) {
BufferedReader buffer=new BufferedReader(new InputStreamReader(process[i].getInputStream()));
try {
if ((line = buffer.readLine()) != null) {
logTextArea[i].append(line + "\n");
} else {
((Timer) e.getSource()).stop();
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
});
try {
for ( int count = 0 ; count < destList.getModel().getSize(); count++){
String projectPath = findDir(new File(codebasePath), destList.getModel().getElementAt(count).toString());
process[count] = Runtime.getRuntime().exec(System.getenv("M2_HOME")+"//bin//mvn.bat clean install -DskipTests -f "+projectPath + "\\pom.xml");
logTextArea[count] = new JTextArea(25, 60);
logFrame[count] = new JFrame("Show Log");
logFrame[count].add(new JScrollPane(logTextArea[count]), BorderLayout.CENTER);
logFrame[count].setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
logFrame[count].pack();
logFrame[count].setLocationByPlatform(true);
logFrame[count].setVisible(true);
timer.start();
process[count].waitFor()
}
} catch (IOException e) {
e.printStackTrace();
}
I am a bit puzzled where exactly you struggle, so maybe I am overlooking something, but you'd generally use it like so:
process[count].waitFor();
while (process[count].isAlive()) {
Thread.sleep(100);
}
You might want to get rid of waitFor completely and add some additional checks to the loop, e.g. a timeout or a signal to cancel the process, in case the process hangs, so your application can terminate properly:
while (process[count].isAlive()) {
if (someTimeoutCondition || someCancelCondition) {
// throw exception or do whatever else you want to do in this situation to gracefully exit
}
Thread.sleep(100);
}
So ... I have a bit of software that's supposed to communicate with a memcached server (using no external libraries).
For testing purposes, let's settle on a simple get hello\r\n command.
I start memcached with the -vv option, this is what the command produces via telnet:
<15 new auto-negotiating client connection
15: Client using the ascii protocol
<15 get hello
>15 END
Now here is what the same command issued from my software produces:
<15 new auto-negotiating client connection
I'm connecting as following:
private void reconnect(){
InetSocketAddress remote;
int nofServers = m.servers.size();
for(int i = 0; i < R; ++i){
boolean success = false;
while(!success) {
try {
SocketChannel oldConnection = connections.get(i);
if (oldConnection != null) oldConnection.close();
remote = m.servers.get((myServerIndex + i) % nofServers).address();
SocketChannel chan = SocketChannel.open(remote);
chan.configureBlocking(false);
chan.register(this.selector, SelectionKey.OP_READ);
connections.set(i, chan);
success = true;
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
}
After that, the software falls into simple enough a NIO loop:
#Override
public void run() {
MyRequest curr = null;
this.canHandleNewRequest = true;
while (true) {
if (canHandleNewRequest) {
curr = myQueue.poll();
}
if (canHandleNewRequest && curr != null) {
canHandleNewRequest = false;
for (int i = 0; i < R; ++i) {
connections.get(i).keyFor(this.selector).interestOps(SelectionKey.OP_WRITE);
}
}
try {
selector.select();
Iterator<SelectionKey> it = selector.selectedKeys().iterator();
while (it.hasNext()) {
SelectionKey k = it.next();
it.remove();
if (!k.isValid()) continue;
if (k.isConnectable()) finishConnection(k);
else if (k.isReadable()) this.read(k, curr);
else if (k.isWritable()) this.write(k, curr);
}
} catch (IOException ex) {
ex.printStackTrace();
reconnect();
}
if(curr != null && /*have sent to all servers I need to*/){
curr = null;
this.canHandleNewRequest = true;
}
}
}
where
private void write(SelectionKey k, MyRequest currentRequest){
try {
SocketChannel chan = (SocketChannel) k.channel();
ByteBuffer out = currentRequest.getSendBuffer(); //DO NOT FLIP (to resume sending from last sent position)
assert(chan != null);
assert(out != null);
//System.out.println(new String(out.array()));
chan.write(out); //TODO: make this work!
if(!out.hasRemaining()) { //do not enable read until whole command has been sent
currentRequest.partiallySent();
k.interestOps(SelectionKey.OP_READ);
}
}catch(IOException ex){
ex.printStackTrace();
}
//TODO: create response structure
}
I even tried to substitute the write method for a dummy command provider:
else if(k.isWritable()){
SocketChannel chan = (SocketChannel)k.channel();
ByteBuffer msg = ByteBuffer.wrap("get hello\r\n".getBytes());
msg.flip();
while(msg.hasRemaining()) {
System.out.println("writing ...");
chan.write(msg);
}
k.interestOps(SelectionKey.OP_READ);
}
but this only gets stuck in the "writing" loop (never terminates).
You should think that at least the server should react to that command but it doesn't.
So ... how do I get this working?
The second line from the log providing the command via telnet produces,
15: Client using the ascii protocol
makes me think there might be something I need to send to the server prior to engaging in actual memcached commands... except I seem to miss it in the protocol.
Help would be appreciated.
EDIT
This seems to be the issue: flipping a buffer in the getSendBuffer method and then returning it is not the same as returning it unflipped and then flipping it in the write method.
I find this rather strange. Can this be or is this merely masking a different error?
With NIO you should always check whether all of the buffer has been written, which is not being done doing in the first write block. Having said that, unless there's a lot of data being written, the whole buffer is usually written in a single call to write. So, it's unlikely to be the root problem here.
In the alternative writing code block the hasRemaining condition is negated, it should be:
while(msg.hasRemaining()) {
System.out.println("writing ...");
chan.write(msg);
}
Could you include what's being sent first? Is the first command terminated with \r\n?
I have a Java program with the following code. I execute an external C program (on my Windows 7 machine) which prints a single character, P or R, every second, to the standard error stream. However, the result of this output, (the execution of pressed() and released(), only shows up when the process is terminated (while I need it live). So if I terminate the process after a while it works, but doesn't if the program is left open (as it is now and as it needs to be). What might be wrong?
try {
final Process p = Runtime.getRuntime().exec("remotek.exe");
Runtime.getRuntime().addShutdownHook(new Thread() {
#Override
public void run() {
p.destroy();
}
});
Reader r = new InputStreamReader(p.getErrorStream());
while (true) {
int c = r.read();
if (c == 'P') {
pressed();
}
if (c == 'R') {
released();
}
}
} catch (IOException e) {
e.printStackTrace();
}
I have the following code that runs a process and I want to know if there was an exception while running it. It hangs for no reason
Runtime runtime = Runtime.getRuntime();
proc = runtime.exec(command.toString());
ProcessHandler errorStream = new ProcessHandler(proc.getErrorStream(),"ERROR", rdyFilePath);
ExecutorService pool = Executors.newSingleThreadExecutor();
Future future = pool.submit(errorStream);
pool.shutdown();
try {
if(future.get() == null) {
log.info("Done completing error thread");
}
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
Here is the class for process handler
public class ProcessHandler implements Callable<Integer> {
private static Logger log = Logger.getLogger(ProcessHandler.class.getName());
InputStream inpStr;
String strType;
String rdyFile;
public ProcessHandler(InputStream inpStr, String strType, String rdyFile) {
this.inpStr = inpStr;
this.strType = strType;
this.rdyFile = rdyFile;
}
public Integer call() throws FileMetadataException {
StringBuffer sb = new StringBuffer();
try {
InputStreamReader inpStrd = new InputStreamReader(inpStr);
BufferedReader buffRd = new BufferedReader(inpStrd);
String line = null;
while((line = buffRd.readLine()) != null) {
if("ERROR".equalsIgnoreCase(strType)) {
sb.append(line + "\n");
log.info(strType + "->" + line);
}
}
if(sb != null) {
log.info("Error Stream length : " + sb.length());
throw new RuntimeException();
}
buffRd.close();
} catch(IOException e) {
log.error("IOException in ProcessHandler Thread" + e.fillInStackTrace());
System.err.println(e);
throw new FileMetadataException();
} finally {
if(sb != null) {
if(sb.toString().length() > 0 ) {
log.error("Error string buffer length " + sb.length());
// do not create rdy file
} else {
log.error("Error string buffer length : " + sb.length());
File f = new File(rdyFile);
try {
f.createNewFile();
} catch(IOException e) {
log.error("IOException while creating rdy file");
}
}
// create rdy file.
}
}
return sb.length();
}
}
}
I have the following code that runs a process and I want to know if there was an exception while running it. It hangs for no reason
If there was an exception your future.get() should have thrown with an ExecutionException -- it would not "hang". Are you sure that your exception is not being printed but somehow lost in the logs or console output?
In tracing your code, I see no way for your program to not finish after it finishes reading the stream. Maybe the process whose error-stream you are reading from is still running and the InputStream has not been closed yet? Maybe there is so much output that you are filling up core with the StringBuffer (which should be changed to a StringBuilder btw).
Can you attach to your application with jconsole to see if the thread is still running and if it is, what is it doing?
if(future.get() == null) {
log.info("Done completing error thread");
}
So this will only log output if you return null from your call() method. That will never happen since the only return is return sb.length();. So either you will get an exception from your call() or your result will be a non-null Integer.
From the Process javadoc:
Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
Per this warning, your future.get() method is hanging because you are only consuming the error stream from the process object. On all platforms I have ever utilized Process objects, I have observed that you need to consume both the Error and Standard out streams.
This is a good tool that uses a multithreaded approach to ensure that streams of a process are consumed.
In this case, since it appears that you don't care about the standard output of the process you might add something like this, borrowing the StreamHelper class:
StreamHelper inStreamHelper = new StreamHelper(proc.getInputStream());
inStreamHelper.start();
First off let me apologize to the SO community for coming to you with something that ought to be so trivial. But I've been at this all day and I'm at the end of my rope.
There is a section of my program that needs pull text from an input stream and an error stream from a process that is launched using Runtime.getrunTime().exec() and pass it through to standard input and output in an orderly manner. I have a function that near as I can tell should work. But it seems to be getting caught in a catch-22 where it's waiting for the stream to report ready - but the stream has finished and is not reporting. I'm baffled. I can't think of another way to do this that fits my constraints and I'm rather skeptical that such a catch-22 can exist.
Here is my code:
private void forwardStreamtoStd(InputStream in, InputStream err)
throws IOException {
int c = -1;
BufferedReader inReader = new BufferedReader(
new InputStreamReader(in, "US-ASCII"));
BufferedReader errReader = new BufferedReader(
new InputStreamReader(err, "US-ASCII"));
boolean inFinished = false, errFinished = false;
try {
System.out.println("Begin stream read loop...");
while (!inFinished && !errFinished) {
if (!inFinished) {
while (inReader.ready()) {
if ((c = inReader.read()) == -1) {
inFinished = true;
}
else {
System.out.print((char) c);
}
}
}
if (!errFinished) {
while (errReader.ready()) {
if ((c = errReader.read()) == -1) {
errFinished = true;
}
else {
System.err.print((char) c);
}
}
}
}
System.out.println("End stream read loop.");
}
catch (IOException e) {
throw e;
}
finally {
errReader.close();
inReader.close();
}
}
The problem seems to be that the reading loops are waiting for the streams to report ready, and as a result aren't seeing the -1 returned by read telling them that it's time to quit. I'm trying to avoid having either stream blocking, so that I can pull from both in turn when they are prepared. However, how can I catch the process's end of stream? Am I missing something? Shouldn't read report that it's read when it has an end of stream -1? The processes are finishing, and so their streams should be dying. What am I doing wrong here?
There are two more possibilities:
Use the ProcessBuilder and invoke redirectErrorStream(true) to join the two streams and you need to read one stream. I have an example here.
In JDK7, you could call the inheritIO() to automatically forward everything
Edit On the second guess, it seems the ready() call is misleading your program. Try this:
private void forwardStreamtoStd(InputStream in, InputStream err)
throws IOException {
int c = -1;
BufferedReader inReader = new BufferedReader(
new InputStreamReader(in, "US-ASCII"));
BufferedReader errReader = new BufferedReader(
new InputStreamReader(err, "US-ASCII"));
boolean inFinished = false, errFinished = false;
try {
System.out.println("Begin stream read loop...");
if (!inFinished) {
while ((c = inReader.read()) != -1) {
System.out.print((char) c);
}
inFinished = true;
}
if (!errFinished) {
while ((c = errReader.read()) != -1) {
System.err.print((char) c);
}
errFinished = true;
}
System.out.println("End stream read loop.");
}
catch (IOException e) {
throw e;
}
finally {
errReader.close();
inReader.close();
}
}
Or better yet, leave off the BufferedReader if you don't plan any extra transformation:
private void createReader(final InputStream in, final OutputStream out) {
new Thread() {
public void run() {
try {
int c = 0;
while ((c = in.read()) != -1) {
out.write(c);
}
} catch (IOException ex) {
ex.printStackTrace();
} finally {
in.close();
}
}
}.start();
}
private void forwardStreamtoStd(InputStream in, InputStream err)
throws IOException {
createReader(in, System.out);
createReader(err, System.err);
}
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4090471
The solution I've always used is to create a separate thread to read one of the streams, join on the thread when the main thread finishes reading, and then waitFor the process.
It's essential to consume the 2 streams concurrently, to prevent blocking. See this article for more info, and in particular note the StreamGobbler mechanism that captures stdout/err in separate threads.
If I remember correctly, the spawned process will never close the stream - so you would need to have the readers in their own threads, sleep on the main thread until the process is done, and then close the readers.