I have a Spring Boot REST application with 2 endpoints: first starts the Process (db shell) with a command like 'mysql -e root'. Second one accepts command (query) and writes it to OutputStream (which is BufferedOutputStream from Process implementation).
Starting the Process (MySQL shell):
ProcessBuilder builder = new ProcessBuilder(commands);
builder.redirectErrorStream(true);
process = builder.start();
out = process.getInputStream();
in = process.getOutputStream();
Executing a command (e.g. 'select * from db.some_table;\n'):
byte[] commandBytes = command.getBytes(Charset.defaultCharset());
in.write(commandBytes, 0, commandBytes.length);
in.flush();
After running a command (query) I want to return its result (or at least, output it to the console) with:
int no = out.available();
if (no > 0) {
int n = out.read(buffer, 0, Math.min(no, buffer.length));
System.out.println(new String(buffer, 0, n));
}
The problem is that out.available() is always 0.
If I call close() on an output stream, out.available() returns all the input stream length and I can read from it. But that is not what I want.
Can I somehow force BufferedInputStream to make result available to be read without closing the stream?
I see that internally BufferedInputStream uses FileInputStream and FileChannel, but I haven't found a way to capture the result when output stream is not closed.
I think what's happening is that the mysql client detects that standard input is not a terminal, and runs in batch mode rather than in interactive mode. This isn't caused by the behaviour of BufferedReader: it's blocking indefinitely on read, and reporting 0 bytes available because there genuinely isn't anything to read from the output of the subprocess.
In batch mode, the client expects to read a list of commands from standard input, and only executes them once the end of file is reached. In other words, the subprocess will not produce any output on the InputStream you see in your parent process until your parent process closes the OutputStream of the subprocess.
It appears that there's no way to force mysql to run in interactive mode (according to this question: "How to force mysql.exe to run in "interactive" mode?", and the documentation of command line options).
The mysqlsh client can be forced into interactive mode, but it is worth considering whether this is really the best solution for your use case. Other alternatives include:
Embracing batch mode and executing all of the commands together, if the form of each command does not depend on the results of previous ones
Sequentially invoking the subprocess multiple times in batch mode, if subsequent commands do depend on the results of previous ones
Performing the queries using JDBC (as g00se recommended in the comments)
Related
Hello stackoverflow community,
I try to run a command on the console and read the input stream of this process:
String command[] = {"ffmpeg"}; //minimal command example
Process proc = processBuilder.command(command).start();
byte[] bytes = IOUtils.toByteArray(proc.getInputStream());
proc.waitFor();
The command is a very long ffmpeg command, which prints bytes to the output. Everything works just fine, but it is very slow using java. When I run this command on my regular command line tool, it takes round about 20ms, everything is printed and done.
For the Java process is takes more than 2s. I also tried to redirect all I/O streams to the std, but the performance is the same, since I thought my reading is too slow. Also, reading the stream in additional threads or using other stream readers etc. did not change anything. The only thing which has an effect is adding cmd (currently working with Windows 10) to the command:
String command[] = {"cmd","/C","ffmpeg"};
resulting in a execution time of round about 400ms. I did not know before that this makes a difference, but it really does.
Since this is a Spring Boot web application and the command is used to output images from a video, 400ms is still a lot. The issue here is, that the frontend/browser requests a bunch of images (lets say 36). Apparently the simultaneous requests of a browser to one host is limited (chrome 6 requests), see here. Therefore it takes at best 6 x 400ms to deliver the content.
So is there a way to improve the performance to the java process or maybe keep it open and fire commands to avoid overhead?
Here is the pseudo code to your question in the comments above:
ByteArrayOutputStream out = new ByteArrayOutputStream();
ProcessBuilder pb = new ProcessBuilder(command);
// Consuming STDOUT and STDERR in same thread can lead to the process freezing if it writes large amounts.
pb.redirectErrorStream(true);
Process process = pb.start();
try (var infoStream = process.getInputStream()) {
infoStream.transferTo(out);
}
status = process.waitFor();
if (status > 0)
// errorhandling
#transferTo(out) Consumes the output stream inside the processbuilder thread. That's it! It runs inside the thread of process.
I have a process created as follows:
Process p = Runtime.getRuntime().exec(new String[]{"su"});
In my program, I only want to create this process once. I am developing a root file explorer application for Android, and whenever this process is created, the Android device will prompt the user to grant root permissions. This is a very slow operation, and as this is a file browser, it will need root permissions often. So, I have decided to create this process once and write commands to its OutputStream in the following manner (stdin is this OutputStream):
stdin.writeBytes(command + "\n");
Before I can read the output of the command, I need my program to wait until the command written by writeBytes has terminated. I have tried p.waitFor(), but this causes the program to hang.
Here is how I read bytes from the InputStream:
int read;
String out = "";
stdout = p.getInputStream();
byte[] buffer = new byte[262144];
while (true) {
read = stdout.read(buffer);
out += new String(buffer, 0, read);
if (read < BUFF_LEN) {
//we have read everything
break;
}
}
Note that although the read(buffer) method blocks until input data is available, it does not block in this case because it thinks it has reached the end of the InputStream.
I have tried to include only relevant portions of my code in this post, but if you would like to take a look at the entire source code of the class where this is contained, see here: http://pastebin.com/t6JdWmQr.
How can I make sure the command has finished running before reading the process' InputStream?
I also encounter similar problem, and I found the answer here:
Wait until a command in su finishes
If you don't need any read stream in this shell process, simply add shell read stream may completed the shell process.
Or in XDA also have better way:
[HowTo]Execute Root Commands and read output
I'm trying to use the WMIC command 'DESKTOP' in an application through a method which is implemented from an interface. Before using WMIC commands in cmd, it's necessary to first input 'WMIC' and press enter, as I'm sure you all know. I can do this just fine, however after the command line enters WMIC mode, I need to subsequently enter the DESKTOP command, or any other WMIC command (see list : http://ss64.com/nt/wmic.html). Here's the overridden method code I'm currently using to enter WMIC mode :
#Override
public void Desktop(){
try {
Runtime rt = Runtime.getRuntime();
String cmd = "cmd /c WMIC";
Process pr = rt.exec(cmd);
}
Perhaps I'm going about this the wrong way? Any help would be much appreciated.
You need to get the output stream of the process you started and write to it.
OutputStream os = pr.getOutputStream();
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(os));
bw.write("DESKTOP");
bw.newLine();
bw.flush();
If you need to see the output/errors from the process you started you need to use getInputStream() and getErrorStream() methods to get the output/error streams and read from it.
Check the javadoc of Process for more details.
http://docs.oracle.com/javase/7/docs/api/java/lang/Process.html
Pay special attention to this part -
By default, the created subprocess does not have its own terminal or
console. All its standard I/O (i.e. stdin, stdout, stderr) operations
will be redirected to the parent process, where they can be accessed
via the streams obtained using the methods getOutputStream(),
getInputStream(), and getErrorStream(). The parent process uses these
streams to feed input to and get output from the subprocess. Because
some native platforms only provide limited buffer size for standard
input and output streams, failure to promptly write the input stream
or read the output stream of the subprocess may cause the subprocess
to block, or even deadlock.
It's a fairly silly answer.
WMIC is normally used in non interactive mode.
wmic desktop get
However you are a programmer. Therefore you are supposed to program.
Here's vbs script that does it. The help is filled with sample scripts in C, VB, VBS, and JScript.
Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!\\.\root\cimv2")
Set colItems = objWMIService.ExecQuery("Select * From Win32_Desktop")
For Each objItem in colItems
msgbox objItem.Name & " " & objItem.Wallpaper
Next
I tested this code(below) on several different linux boxes(4+) and it worked fine. However, on one linux box I ran into an issue with readline() hanging for the error inputStream(errorStream). This stream should be empty so I suspected that box was not writing out a line terminator to the errorStream for the error. I changed my code to use read() instead of readline()...but read() also hung.
I tried retrieving the input inputStream first, and that worked and there was no hangs with readline()/read() for the error inputstream. I could not do this since I needed to obtain possible errors first. Appearing to be a deadlock, I was able to resolve this by having each inputstream read from it's own thread. Why did I only see this issue on one box? Is there a kernel setting or some other setting specific to this box that could have caused this?
ProcessBuilder processBuilder = new ProcessBuilder()
try
{
Process processA = null;
synchronized (processBuilder)
{
processBuilder.command("/bin/sh","-c"," . /Home/SomeScript.ksh");
processA = processBuilder.start();
}
inputStream = processA.getInputStream();
reader = new BufferedReader(new InputStreamReader(inputStream));
errorStream = processA.getErrorStream();
errorReader = new BufferedReader(new InputStreamReader(errorStream));
String driverError;
while ((driverError = errorReader.readLine()) != null)
{
//some code
}
Why did I only see this issue on one box?
Most likely because of something in the script that is being run ... and its interactions with its environment (e.g. files, environment variables, etc)
Is there a kernel setting or some other setting specific to this box that could have caused this?
It is possible but unlikely that it is a kernel setting. It might be "something else". Indeed, it has to be "something" outside of the Java application that is to blame, at least in part.
I suggest you do the following temporarily (at least):
ProcessBuilder processBuilder = new ProcessBuilder();
processBuilder.command("/bin/sh","-c"," . /Home/SomeScript.ksh");
processBuilder.redirectErrorStream(true);
processA = processBuilder.start();
inputStream = processA.getInputStream();
reader = new BufferedReader(new InputStreamReader(inputStream));
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
System.out.println("Return code is " + processA.exitValue());
That way you can see what all of the output is.
There should not be a problem if the external process fails to put a newline at the end of the last line. The Java process will see an EOF on the input stream, and the BufferedReader will return what characters it has ... and return null on the next call.
Another possibility is that the external process is blocking because it is trying to read from its standard input.
UPDATE
The redirectErrorStream also resolved the issue, but I need the error stream separate.
OK so if it did (reliably) solve the problem then that (most likely) means that you have to read the external processes stdout and stderr streams in parallel. The simple way to do is to create 2 threads to read and buffer the two streams separately. For example: Capturing stdout when calling Runtime.exec
(Your problem is due to the fact that pipes have a finite buffering capacity. The external problem is most likely alternating between writing stuff to stdout and stderr. If it tries to write to one of the pipes when that pipe is "full", it will block. But if your application is reading all of the other pipe (to EOF) before it reads the blocked pipe, then everything will deadlock. The fact that the external process is stuck in PIPE_W state is more evidence for this explanation.
One possible reason that you are seeing different behaviour on different systems is that the amount of buffering in a pipe is system dependent. But it could also be due to differences in what the external process is doing; e.g. its inputs.)
You are running OS specific commands in a script, any one could be holding the error output. You can avoid this by discarding the errors, but that is unlikely to be a good idea.
I would check the version of the OS are the same and whether you have any significant differences in the command you run in the script. If this doesn't help, take out commands from the script until it starts working. I assume an empty script doesn't do this.
Everything works fine on the command line, but when I translate what I want into Java, the receiving process never gets anything on stdin.
Here's what I have:
private void deployWarFile(File warFile, String instanceId) throws IOException, InterruptedException {
Runtime runtime = Runtime.getRuntime();
// FIXME(nyap): Use Jsch.
Process deployWarFile = runtime.exec(new String[]{
"ssh",
"gateway",
"/path/to/count-the-bytes"});
OutputStream deployWarFileStdin = deployWarFile.getOutputStream();
InputStream deployWarFileStdout = new BufferedInputStream(deployWarFile.getInputStream());
InputStream warFileInputStream = new FileInputStream(warFile);
IOUtils.copy(warFileInputStream, deployWarFileStdin);
IOUtils.copy(deployWarFileStdout, System.out);
warFileInputStream.close();
deployWarFileStdout.close();
deployWarFileStdin.close();
int status = deployWarFile.waitFor();
System.out.println("************ Deployed with status " + status + " file handles. ************");
}
The script 'count-the-bytes' is simply:
#!/bin/bash
echo "************ counting stdin bytes ************"
wc -c
echo "************ counted stdin bytes ************"
The output indicates that the function hangs at the 'wc -c' line -- it never gets to the 'counted stdin bytes' line.
What's going on? Would using Jsch help?
You might try closing the output stream before you expect wc -c to return.
IOUtils.copy(warFileInputStream, deployWarFileStdin);
deployWarFileStdin.close();
IOUtils.copy(deployWarFileStdout, System.out);
warFileInputStream.close();
deployWarFileStdout.close();
Would using Jsch help?
Using JSch would only help if you would be using the setInputStream() and setOutputStream() methods of the channel instead of the IOUtils.copy method, since they manage the copying on a separate thread.
ChannelExec deployWarFile = (ChannelExec)session.openChannel("exec");
deployWarFile.setCommand("/path/to/count-the-bytes");
deployWarFile.setOutputStream(System.out);
deployWarFile.setInputStream(new BufferedInputStream(new FileInputStream(warFile)));
deployWarFile.connect();
(Here you somehow have to wait until the other side closes the channel.)
If you simply replaced the Runtime.exec with opening an ChannelExec (and starting it after getting the streams), the problem would be completely the same, and could be solved by the same solution mentioned by antlersoft, i.e. closing the input before reading the output:
ChannelExec deployWarFile = (ChannelExec)session.openChannel("exec");
deployWarFile.setCommand("/path/to/count-the-bytes");
OutputStream deployWarFileStdin = deployWarFile.getOutputStream();
InputStream deployWarFileStdout = new BufferedInputStream(deployWarFile.getInputStream());
InputStream warFileInputStream = new FileInputStream(warFile);
deployWarFile.connect();
IOUtils.copy(warFileInputStream, deployWarFileStdin);
deployWarFileStdin.close();
warFileInputStream.close();
IOUtils.copy(deployWarFileStdout, System.out);
deployWarFileStdout.close();
(Of course, if you have longer output, you will want to do input and output in parallel, or simply use the first method.)
You probably get an error, but the process hangs because you are not reading the error stream.
Taken from the Process JavaDoc
All its standard io (i.e. stdin, stdout, stderr) operations will be redirected to the parent process through three streams (Process.getOutputStream(), Process.getInputStream(), Process.getErrorStream()). The parent process uses these streams to feed input to and get output from the subprocess. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
So you need to read all of them. Using the ProcessBuilder is probably easier