We Keep Coding

Empty string parsing ntpq command result

I'm parsing the result of executing this composite command
ntpq -c peers | awk ' $0 ~ /^*/ {print $9}'
in order to obtain the offset of the active ntp server.
This is the java code used and executed periodically
public Double getClockOffset() {
Double localClockOffset = null;
try {
String[] cmd = {"/bin/sh",
"-c",
"ntpq -c peers | awk \' $0 ~ /^\\*/ {print $9}\'"};
Process p = Runtime.getRuntime().exec(cmd);
p.waitFor();
BufferedReader buf = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = buf.readLine();
if (!StringUtils.isEmpty(line)) {
localClockOffset = Double.parseDouble(line.trim());
} else {
// Log "NTP -> Empty line - No active servers - Unsynchronized"
}
} catch (Exception e) {
// Log exception
}
return localClockOffset;
}
ntpq result example
> remote refid st t when poll reach delay offset jitter
> ==============================================================================
> *server001s1 .LOCL. 1 u 33 64 377 0.111 -0.017 0.011
> +server002s1 10.30.10.6 2 u 42 64 377 0.106 -0.006 0.027
> +server003s1 10.30.10.6 2 u 13 64 377 0.120 -0.009 0.016
Notice that awk searchs the first line beginnig with '*' and extracts its ninth column. In the example: -0.017
The problem is that sometimes I'm obtaining the no-active-servers log message - intended to appear when there is no server with '*'- while the execution of the command through the console returns a number.
I know that I'm not closing the BufferedReader in that code but is that the reason of this behaviour? A new instance is being created (and left open until garbage collecting) in each method invocation but I think that it shouldn't be the cause of this problem.

Runtime.exec() simply invokes the ProcessBuilder inside it, like that:
public Process More ...exec(String[] cmdarray, String[] envp, File dir)
throws IOException {
return new ProcessBuilder(cmdarray)
.environment(envp)
.directory(dir)
.start();
}
see OpenJDK Runtime.java
So there is nothing wrong with using it instead of the ProcessBuilder as is.
The problem is that you invoke:
p.waitFor();
before you obtained the InputStream.
Which means that the process will be already terminated, by the time you obtain the InputStream, and the output stream data might be or might not be available to you, depending on the OS buffering implementation nuances and precise timing of the operations.
So, if you move the waitFor() to the bottom, your code should start working more reliably.
Under Linux however you should normally be able to read the remaining data from the PIPE buffer, even after the writing process has ended.
And the UNIXProcess implementation in OpenJDK, actually makes an explicit use of that, and tries to drain the remaining data, once the process has exited, so that file descriptor can be reclaimed:
/** Called by the process reaper thread when the process exits. */
synchronized void processExited() {
synchronized (closeLock) {
try {
InputStream in = this.in;
// this stream is closed if and only if: in == null
if (in != null) {
byte[] stragglers = drainInputStream(in);
in.close();
this.in = (stragglers == null) ?
ProcessBuilder.NullInputStream.INSTANCE :
new ByteArrayInputStream(stragglers);
}
} catch (IOException ignored) {}
}
}
And this seems to work reliable enough, at least in my tests, so it would be nice to know which specific version of Linux|Unix and JRE your are running.
Have you also considered the possibility of an application-level problem ?
I.e. ntpq is not really guaranteed to always return a * row.
So, it would be nice to remove the awk part from your pipe, to see if there will be some output at all the times.
Another thing to note is that if one of your shell pipeline steps fails (e.g. the ntpq itself), you will also get an empty output, so you will have to track the STDERR as well (e.g. by merging it with STDOUT via the ProcessBuilder).
Sidenote
Doing waitFor before you start consuming the data, is a bad idea in any case, as if your external process will produce enough output to fill the pipe buffer, it will just hang waiting for someone to read it, which will never happen, as your Java process will be locked in waitFor at the same time.

As pointed by Andrew Thompson, you shall try ProcessBuilder instead.
String[] cmd = {"/bin/sh",
"-c",
"ntpq -c peers | awk \' $0 ~ /^\\*/ {print $9}\'"};
ProcessBuilder pb = new ProcessBuilder(cmd);
pb.redirectErrorStream(true);
Process proc = pb.start();
BufferedReader buf = new BufferedReader(new
InputStreamReader(proc.getInputStream()));
String line = null;
while ((line = buf.readLine()) != null) {
localClockOffset = Double.parseDouble(line.trim());
break;
}
proc.destroy();
Ref ProcessBuilder

Finally we have found the real problem.
I'm not gonna change the accepted anwser, I think that it's useful too but maybe someone can learn from our experience.
My java program is launched with a shell script. When we execute the script manually, ntpq command is found and invoked successfully. The problem arises when the software is fully deployed. In the final environment we've got a cron scheduled demon that keeps our program alive but PATH established by cron is different from the PATH that our profile has got assigned.
PATH used by cron:
.:/usr/bin:/bin
PATH that we got login for launching the script manually:
/usr/sbin:/usr/bin:/bin:/sbin:/usr/lib:/usr/lib64:/local/users/nor:
/usr/local/bin:/usr/local/lib:.
Usually ntpq is in
/usr/sbin/ntpq
After we found the key of our problem, I search StackOverflow and got this relevant question where the problem is better explained and solved.
How to get CRON to call in the correct PATHs

ffmpeg won't start until java exits

I am trying to make a java program that automatically converts wtv files in an input folder to mpg files in output folder. The twist is that I make it run periodically, so it acts as a synchronizer.
The following code works for converting the .wtv to a .dvr-ms, which is required by ffmpeg since it cannot convert .wtv files directly.
Process p = Runtime.getRuntime().exec("C:\\Windows\\ehome\\WTVConverter C:\\Users\\Andrew\\Desktop\\test\\input\\input.wtv C:\\Users\\Andrew\\Desktop\\test\\output\\input.dvr-ms");
p.waitFor();
WTVConverter has no problems running from a java application. ffmpeg is a different story. Once the above line runs, I then run this...
Process p = Runtime.getRuntime().exec("ffmpeg\\bin\\ffmpeg -y -i \"C:\\Users\\Andrew\\Desktop\\test\\output\\input.dvr-ms'" -vcodec copy -acodec copy -f dvd \"C:\Users\Andrew\Desktop\test\output\input.mpg\"");
p.waitFor();
Suddenly, there is a problem... The application ffmpeg shows up in the task manager, but it's cpu usage is 0, and no mpeg files is being generated. If I force the java application to close, though, suddenly it starts working! Huh?
What reason would there be for a command line application to wait for its calling application to quit before it executes? I'm not incredibly command line savvy, so I don't really know how to diagnose this problem.

Bah, this always happens. I post a question, and THEN I figure it out on my own. Turns out, ffmpeg expects you to read in its text output before it loads each frame. If a calling program does not do this, it simply waits. If there is no calling program, I assume that it just outputs it nowhere. What I did is ran the program as usual, but also read in text from the process's input stream like so...
Process p = Runtime.getRuntime.exec(".....");
final Scanner in = new Scanner(p.getInputStream());
new Thread()
{
#Override
public void run()
{
System.out.println(in.nextLine());
}
}.start();
Lesson learned, I guess.

Setting nice value of Java program running on linux

I want my Java program to lower it's priority some so it doesn't overwhelm the system. My initial thought was to use Thread.currentThread().setPriority(5) but that appears to be merely its priority within the JVM.
Then I thought maybe I'd cludge it and invoke a system command, but Thread.getId() is also merely the JVM's id, so I don't even know what process id to pass to renice.
Is there any way for a Java program to do something like this?

Since we must do it in a platform dependent way, I run a shell process from java and it renices its parent. The parrent happens to be our java process.
import java.io.*;
public class Pid
{
public static void main(String sArgs[])
throws java.io.IOException, InterruptedException
{
Process p = Runtime.getRuntime().exec(
new String[] {
"sh",
"-c",
"renice 8 `ps h -o ppid $$`"
// or: "renice 8 `cat /proc/$$/stat|awk '{print $4}'`"
}
);
// we're done here, the remaining code is for debugging purposes only
p.waitFor();
BufferedReader bre = new BufferedReader(new InputStreamReader(
p.getErrorStream()));
System.out.println(bre.readLine());
BufferedReader bro = new BufferedReader(new InputStreamReader(
p.getInputStream()));
System.out.println(bro.readLine());
Thread.sleep(10000);
}
}
BTW: are you Brad Mace from jEdit? Nice to meet you.

If your program is the only running java program, then you can run
renice +5 `pgrep java`

In addition to renice - you may also use ionice comand. For example :
ionice -c 3 -n 7 -p PID

Also look at https://github.com/jnr/jnr-posix/.
This POSIX library should allow you to get at some of the Linux Posix Nice functions like...
https://github.com/jnr/jnr-posix/blob/master/src/main/java/jnr/posix/LibC.java for the OS level setPriority(), i.e. setpriority(2)
jnr-posix is also in Maven.

Use:
nice --adjustment=5 java whatever
to run your java program and assign the priority in just one step.

My suggestion is to invoke your java application from a bash script or start/stop service script then find the process id after startup and renice it.

cmd java wait for process

I'm writing an application with a Java GUI which calls some FORTRAN code. I want to return a file (solution.ps) which is updated and compiled based on changes in the FORTRAN code, which are created earlier in my ActionPerformed method. However the code I have at present just returns the old version of the file rather than waiting for the updated results of the cmd compilation. Is there a way to make the cmd wait for the process to run before completing the next step? (It works fine running directly from cmd)
I've searched but can't find anything except process.waitFor() which won't seem to pause the execution at the right point. Tried Thread.waitFor() too.
I'm thinking this could be useful for anyone who wants to send user inputs to another program and return a compiled result which uses these inputs.
Anyway here is the code, thanks in advance for any help and I hope I made the problem clear.
String[] command ={"cmd",};
try {
Process p = Runtime.getRuntime().exec(command);
new Thread(new SyncPipe(p.getErrorStream(), System.err)).start();
new Thread(new SyncPipe(p.getInputStream(), System.out)).start();
PrintWriter stdin = new PrintWriter(p.getOutputStream());
stdin.println("cd c:\\g77");
stdin.println("g77setup.bat");
stdin.println("cd c:\\users\\laurence\\workspace\\areaplanner");
stdin.println("g77 -O4 genpack.f -o genpack");
stdin.println("genpack");
stdin.println("5");
/*
* The following line sets the time to run the FORTRAN code for
* - need to wait for this to complete before calling mpost
*/
stdin.println("30");
stdin.println("mpost solution.mp");
stdin.println("latex solution.tex");
stdin.println("dvips solution.dvi -o solution.ps");
stdin.close();
} catch(IOException e4){}

You are only runnng the windows shell command. To fix, suggest writing the batch file first and wait for it to finish:
String command = "cmd /c mybatchfile.bat";
Process p = Runtime.getRuntime().exec(command);
p.waitFor();
To get another section to kick off before the the first set of commands have completed, you will have to write another batch file and repeat the above. Make sure you have both process then in separate threads.

Try using waitFor so as to make the current thread wait for the process to finish its job.
Process p = Runtime.getRuntime().exec(command);
p.waitFor()
The command in your code is incomplete. And also it is advisable to use a ProcessBuilder.start() instead of Process.

Java ProcessBuilder: Resultant Process Hangs

I've been trying to use Java's ProcessBuilder to launch an application in Linux that should run "long-term". The way this program runs is to launch a command (in this case, I am launching a media playback application), allow it to run, and check to ensure that it hasn't crashed. For instance, check to see if the PID is still active, and then relaunch the process, if it has died.
The problem I'm getting right now is that the PID remains alive in the system, but the GUI for the application hangs. I tried shifting the ProcessBuilder(cmd).start() into a separate thread, but that doesn't seem to be solving anything, as I hoped it would have.
Basically the result is that, to the user, the program APPEARS to have crashed, but killing the Java process that drives the ProcessBuilder.start() Process actually allows the created Process to resume its normal behavior. This means that something in the Java application is interfering with the spawned Process, but I have absolutely no idea what, at this point. (Hence why I tried separating it into another thread, which didn't seem to resolve anything)
If anyone has any input/thoughts, please let me know, as I can't for the life of me think of how to solve this problem.
Edit: I have no concern over the I/O stream created from the Process, and have thus taken no steps to deal with that--could this cause a hang in the Process itself?

If the process writes to stderr or stdout, and you're not reading it - it will just "hang" , blocking when writing to stdout/err. Either redirect stdout/err to /dev/null using a shell or merge stdout/err with redirectErrorStream(true) and spawn another thread that reads from stdout of the process

You want the trick?
Don't start your process from ProcessBuilder.start(). Don't try to mess with stream redirection/consumption from Java (especially if you give no s**t about it ; )
Use ProcessBuilder.start() to start a little shell script that gobbles all the input/output streams.
Something like that:
#!/bin/bash
nohup $1 >/dev/null 2>error.log &
That is: if you don't care about stdout and still want to log stderr (do you?) to a file (error.log here).
If you don't even care about stderr, just redirect it to stdout:
#!/bin/bash
nohup $1 >/dev/null 2>1 &
And you call that tiny script from Java, giving it as an argument the name of the process you want to run.
If a process running on Linux that is redirecting both stdout and stderr to /dev/null still produce anything then you've got a broken, non-compliant, Linux install ;)
In other word: the above Just Works [TM] and get rid of the problematic "you need to consume the streams in this and that order bla bla bla Java-specific non-sense".

The thread running the process may block if it does not handle the output. This can be done by spawning a new thread that reads the output of the process.
final ProcessBuilder builder = new ProcessBuilder("script")
.redirectErrorStream(true)
.directory(workDirectory);
final Process process = builder.start();
final StringWriter writer = new StringWriter();
new Thread(new Runnable() {
public void run() {
IOUtils.copy(process.getInputStream(), writer);
}
}).start();
final int exitValue = process.waitFor();
final String processOutput = writer.toString();

Just stumbled on this after I had a similar issue. Agreeing with nos, you need to handle the output. I had something like this:
ProcessBuilder myProc2 = new ProcessBuilder(command);
final Process process = myProc2.start();
and it was working great. The spawned process even did output some output but not much. When I started to output a lot more, it appeared my process wasn't even getting launched anymore. I updated to this:
ProcessBuilder myProc2 = new ProcessBuilder(command);
myProc2.redirectErrorStream(true);
final Process process = myProc2.start();
InputStream myIS = process.getInputStream();
String tempOut = convertStreamToStr(myIS);
and it started working again. (Refer to this link for convertStreamToStr() code)

Edit: I have no concern over the I/O stream created from the Process, and have thus taken no steps to deal with that--could this cause a hang in the Process itself?
If you don't read the output streams created by the process then it is possible that the application will block once the application's buffers are full. I've never seen this happen on Linux (although I'm not saying that it doesn't) but I have seen this exact problem on Windows. I think this is likely related.

JDK7 will have builtin support for subprocess I/O redirection:
http://download.oracle.com/javase/7/docs/api/java/lang/ProcessBuilder.html
In the meantime, if you really want to discard stdout/stderr, it seems best (on Linux) to invoke ProcessBuilder on a command that looks like:
["/bin/bash", "-c", "exec YOUR_COMMAND_HERE >/dev/null 2>&1"]

Another solution is to start the process with Redirect.PIPE and close the InputStream like this:
ProcessBuilder builder = new ProcessBuilder(cmd);
builder.redirectOutput(Redirect.PIPE);
builder.redirectErrorStream(true); // redirect the SysErr to SysOut
Process proc = builder.start();
proc.getInputStream().close(); // this will close the pipe and the output will "flow"
proc.waitFor(); //wait
I tested this in Windows and Linux, and works!

In case you need to capture stdout and stderr and monitor the process then using Apache Commons Exec helped me a lot.

I believe the problem is the buffering pipe from Linux itself.
Try to use stdbuf with your executable
new ProcessBuilder().command("/usr/bin/stdbuf","-o0","*executable*","*arguments*");**
The -o0 says not to buffer the output.
The same goes to -i0 and -e0 if you want to unbuffer the input and error pipe.

you need to read the output before waiting to finish the cycle. You will not be notified If the output doesn't fill the buffer. If it does, it will wait until you read the output.
Suppose you have some errors or responses regarding your command which you are not reading. This would cause the application to stop and waitFor to wait forever. A simple way around is to re-direct the errors to the regular output.
I was spent 2 days on this issue.
public static void exeCuteCommand(String command) {
try {
boolean isWindows = System.getProperty("os.name").toLowerCase().startsWith("windows");
ProcessBuilder builder = new ProcessBuilder();
if (isWindows) {
builder.command("cmd.exe", "/c", command);
} else {
builder.command("sh", "-c", command);
}
Process process = builder.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = reader.readLine()) != null)
System.out.println("Cmd Response: " + line);
process.waitFor();
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}