I am trying to make my Java program interact with Linux bash but something goes wrong. I have a simple executable prog that reads the one integer from stdin and outputs its square. Executing
echo 5 | ./prog
from bash itself prints correct answer 25 in stdout but running
import java.io.*;
public class Main {
public static void main(String[] args) throws InterruptedException, IOException {
Runtime run = Runtime.getRuntime();
Process proc = run.exec("echo 5 | ./prog");
proc.waitFor();
BufferedReader br = new BufferedReader(new InputStreamReader(proc.getInputStream()));
while(br.ready())
System.out.println(br.readLine());
}
}
unexpectedly gives 5 | ./prog. What is the solution?
Java exec cannot run shell commands like that. If you want to run a shell command, you need to explicitly invoke the shell; e.g.
Process proc = run.exec(new String[]{"/bin/sh", "-c", "echo 5 | ./prog"});
For more details on what Java does with this, read the javadocs for exec(String) and exec(String[]). Note that these are "convenience methods", and you need to follow the chain of links to the underlying methods for a complete understanding of what the javadoc is saying.
If you want even more detail on how Java handles this, there is the source code ...
If you want to understand in depth why Java doesn't handle the shell syntax itself, you probably need to take a deep dive into the architecture and philosophy of UNIX / Linux systems, and the separation of concerns between application, operating system and command shell. Note that there are a myriad different shells, each with (potentially) different rules for quoting, argument splitting, redirection, pipes, etcetera. Most of the popular shells are similar, but that's just the way things panned out.
Explanation of the solution:
The reason for splitting the command by hand is that exec(String) won't split the command into a single command and arguments correctly. It can't. This is an example where there are two commands in a pipeline.
The reason for using "sh" is ... well ... you need a shell to parse and process a shell command line. Java's exec command does not support shell syntax ... as the javadoc explains.
The purpose of the "-c" option is explained by "man sh". Basically, sh -c "a b c" means "use 'sh' to run the command line 'a b c'".
FWIW, it is technically possible to construct and run a pipeline solely in Java (i.e. without relying on an external shell), but it is generally not worth the effort. You've already introduced a platform dependency by running external commands, so an extra dependency in the form of a specific shell command and syntax doesn't make things significantly worse.
Related
I'm having the next error when execute a cmd command using Java. I'm working in a mac laptop. This is my code:
private static String exportContainerFromImage(String container) {
//docker export mysql_dummy > ~/Documents/mysql_dummy.tar
String errorMessage = "";
String[] cmdArgs =
{"docker export mysql_dummy > ~/Documents/mysql_dummy.tar", "bash"};
Process process = Runtime.getRuntime().exec(cmdArgs);
}
But I'm getting the error, error=2, No such file or directory, if I execute the command directly on the terminal it runs successfully, I tried also changing the directory to ~\\Documents\\mysql_dummy.tar and got the same result.
But if I run the command with the arguments:
{"docker create -ti --name mysql_dummy mysql", "bash"};
It runs properly
Any ideas?
You're conflating 'shell magic' with 'an OS'. Also, you seem to be wildly confused about what the array form of cmdArgs does, because you've tagged a bash in there at the end. That array is supposed to contain the executable's full path at arr[0], and all arguments at arr[1] and up. docker create -ti ... is clearly not a filename, and bash is clearly not an argument.
Shell magic?
If you type:
docker create -ti --name mysql_dummy mysql
on the command line, bash (or cmd.exe if on windows, or whatever shell you are using) reads it and does a whole bunch of replacement magic and parsing on this. It's the shell that does this, not the OS, and java's processbuilder stuff is not a shell and therefore isn't going to do all that. What you're attempting to do? Run that entire line as if it's a single file name that is executable which it clearly isn't.
This is all shell magic - all things that you CANNOT do with exec. Fortunately, java is a programming language, so you can do all these things by, well, programming it.
Parsing out params by splitting on whitespace.
quoting to avoid that splitting, but then removing the quotes.
Treating ~ as a ref to a homedir.
Replacing * and ? in filename paths.
Variable substitution
Setting up redirects with > somefile.txt or 2> /dev/null or < file.in or whatnot.
You must do those things.
In addition, exec cannot be used to this, period. As usual, the only non-problematic way to run processes is to always use ProcessBuilder, no exceptions. Consider runtime.exec a known-broken method you must never call.
ProcessBuilder lets you redirect the output.
String[] cmdArgs = {
"/bin/docker" // note, FULL PATH!
"export",
"mysql_dummy"
};
File out = new File(System.getProperty("user.home"), "Documents/mysql_dummy.tar");
ProcessBuilder pb = new ProcessBuilder(cmdArgs);
pb.redirectOutput(new File(out));
pb.start();
That does what you want, presumably.
The alternative is to make a script (script.sh or script.bat) and then start bash or cmd.exe and ask it to run that script.
String[] args = { "/bin/bash", "-c", "/fully/qualified/path/to/the/script.sh" }
and then exec that. Now you can pile *.txt, > foobar.txt, ~/homediref, and all the other shellisms in that script as much as you like.
Overall #rzwiterloot's answer is good, but there are some alternatives it leaves out.
First, what I would consider the best solution to this problem: the -o option to docker export.
"docker", "export", "mysql_dummy", "-o", "Documents/mysql_dummy.tar"
Ignoring ~/ here, this set of command and arguments will achieve the same thing as ... > Documents/mysql_dummy.tar but doesn't rely on the shell for the redirection; docker export is perfectly capable of handling that operation itself.
Second, if you wanted to run a shell command from the program, you could. I would not recommend this. But in certain circumstances it might make sense.
The alternative is to make a script
You don't have to put the command in a separate file. Actually this is one inaccuracy in #rzwiterloot's answer; -c allows you to pass command(s) to bash, not the path to a file containing commands.
"bash", "-c", "docker export mysql_dummy > ~/Documents/mysql_dummy.tar"
However, I'd recommend you avoid invoking shells from any program you write. They're quirky and esoteric and there's almost always a simpler way to achieve what you want, such as docker export's -o optiopn, in this case.
public class Test_Python
{
public static void main( String[] args ) throws IOException
{
String command = "cmd /k start cmd.exe /k \"cd C:\\Workspace\\supply\\environment\\ && setup.bat && python -V ";
Runtime.getRuntime().exec(command);
}
}
When I execute my code, the "python -V" does not work, it is not executed in the cmd window.
But if I delete the "setup.bat" from my command, the "python -V" is executed (I can read the version in the cmd window)
Also, if I manually start a cmd, type setup.bat and then python -V it works.
My batch file is used to set my working environment:
SET basedir=%~dp0
echo %basedir%
cmd /k "cd %basedir%\..\scripts && set PYTHONPATH=%basedir%\..\lib"
Is there any way to know why it fails?
Unlike python Java may need some help. As I can see you are running on Windows.
You invoke the Runtime.exec() method. The method returns a Process instance, and in it's documentation you can read
By default, the created process does not have its own terminal or console. All its standard I/O (i.e. stdin, stdout, stderr) operations will be redirected to the parent process, where they can be accessed via the streams obtained using the methods getOutputStream(), getInputStream(), and getErrorStream(). The parent process uses these streams to feed input to and get output from the process. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the process may cause the process to block, or even deadlock.
So it is likely your process is started by the OS but gets blocked due to I/O restrictions. Get around that by reading the STDOUT and STDERR streams until your process finishes. One good programming model is visible at https://www.baeldung.com/run-shell-command-in-java
Now that we know the OS is not blocking the process, the issue may be inside the command itself. Note you concatenated several commands using the && operator. I do not have a windows system but searched a bit online:
https://www.ionos.com/digitalguide/server/know-how/windows-cmd-commands/
CommandA && CommandB (the second command is only run if the first was successful)
To check how the first part of your command exits run it separately in Java and do not forget to print the process.exitValue() method.
But looking at the whole picture, all you do with the first part is to change the working directory and set some environment variable. What stops you from running the ProcessBuilder methods? The example in the documentation directly sets the environment and directory.
My question consists of several things I don't understand about the use of "cmd.exe", "/c" when executing sub-processes from Java in Windows. Basically, I couldn't find a good explanation about when and why they're needed.
My specific problems: I have a small framework for sub-processes execution. One use is a Java application which "manages" several other JVMs created by ProcessBuilders. One of the key requirements is that when a sub-process is stuck, or the hosting application is terminating, it must be able to kill the sub-processes.
The problem is, on one hand, doing this:
new ProcessBuilder("java", "...").start();
Causes this:
Could not find or load main class ...
As if the system variables or directory are different (which they're not). On the other hand, wrapping it in a cmd.exe like this:
new ProcessBuilder("cmd.exe", "/c", "java", "...").start();
WORKS, but creates another cmd.exe process, which has a side effect: the child JVM is now a sub-sub-process, and process.destroy(); doesn't kill it (a known bug in the Windows JRE as I found).
This specific problem was handled on a different level, since all those applications are ours and we know their PIDs. But it's an example how cmd.exe makes everything work differently (or prevent the JVM from working at all). So I'd like to know what exactly happens there.
Here the framework itself comes into the picture as well. It's also going to be used by our testing platform. I'd like to provide an API which allows wrapping the command with a cmd.exe /c by a parameter. But, what exactly is the meaning of that parameter? How do the users decide if they want a cmd.exe wrapping?
AND a bonus I'd appreciate: is any of this relevant in other OS? Does it have some kind of an equivalent, say, in Linux?
Came across it again and found the problem, and some additional insights.
#HarryJohnston - good point, according to the javadoc the sub-process inherits the environment from both ProcessBuilder and Runtime.getRuntime.exec(...) APIs so it's not the issue (and java.exe is in fact found so PATH is obviously available).
But it seems certain command line features are lost. Among others it's this kind of %VARIABLE% usage in command line - they're not interpreted unless the command starts with cmd.exe /c.
In this case of class not found (it could also cause NoClassDefFoundError), I had several variables used in the classpath (perhaps I should have posted the entire command, but it's kinda long). When I replaced them with the literary paths, everything worked.
Basically:
java -cp %classpath% Main - bad
cmd.exe /c java -cp %classpath% Main - good
java -cp D:\proj\bin Main - good
So, on the general question of how's the behavior different with cmd.exe:
Variable references like this: %VARIABLE% are only interpreted by cmd.exe.
With cmd.exe you don't really need to split the command into String array of parameters, you can use one long String (array of "cmd.exe", "/c", "the rest...").
The sub-process will be cmd.exe and not the actual executable, so process.destroy() won't kill it. This might be a bug fixed in later versions (I used Java 7 and 8, Windows 7 and Server 2012).
File associations only work with cmd.exe, so you can't "execute" a file that is not a program or script ("CreateProcess error=193, %1 is not a valid Win32 application"). Calling executables defined in PATH does work.
start and call are only available under cmd.exe.
Command level errors behave differently as well: With cmd.exe the error will just be in the output stream, whereas without it the API throws an exception, which might also have a slightly different description.
For example:
nothing will cause java.io.IOException: CreateProcess error=2, The system cannot find the file specified
cmd.exe /c nothing will return output: 'nothing' is not recognized as an internal or external command, operable program or batch file, and a return value of 1 (the only indication something went wrong).
On the deeper question of why, and is this the full list of differences, I still don't really know. Looks like the JVM has its way of running commands, which you can "wrap" with cmd.exe for additional features and protection.
Another interesting fact is when you run a Batch, it's actually run under a cmd.exe (with all its features). From my tests it seems process.destroy() can only kill it if it's not waiting for anything external (if stuck on pause for example), with or without additional cmd.exe /c wrapping, but not if it runs a sub-process - which is consistent with the cmd.exe limitation mentioned in point 3.
I want to run the following shell command from a java application:
java -jar saxon9he.jar -warnings:fatal a.xml a.xsl param1=123 param2=abc
Currently, I am simply executing this as a shell command using
ProcessBuilder pb = new ProcessBuilder(commandLineParts);
[...]
Process process = pb.start();
What is the correct way to do this in java?
This is the correct way of executing a command in Java. Just to clear possible confusion: ProcessBuilder doesn't execute the program using a shell. That's the reason why you have to provide it with a list of arguments and not a single string (that would be parsed by a shell).
There are two possibilities:
either you want to run the Java program in a new JVM and then using
the ProcessBuilder is the way to go
or you don't mind if it is executed in the same JVM and then you can call the main method yourself as Sean suggests (possibly in a different thread)
Another option, depending on the type of the application, would be to perform some acrobatics with an application server to start the app in it.
If you use the ProcessBuilder just be careful about handling its input and output streams - if you don't handle them your application can hang: Java ProcessBuilder: Input/Output Stream This has been improved in Java 7.
I have a Java program that I'd like to daemonize on a linux system. In other words, I want to start running it in a shell and have it continue running after I've logged out. I also want to be able to stop the program cleanly.
I found this article which uses a combination of shell scripting and Java code to do the trick. It looks good, but I'd like something simpler, if possible.
What's your preferred method to daemonize a Java program on a Linux system?
Apache Commons Daemon will run your Java program as Linux daemon or WinNT Service.
If you can't rely on Java Service Wrapper cited elsewhere (for instance, if you are running on Ubuntu, which has no packaged version of SW) you probably want to do it the old fashioned way: have your program write its PID in /var/run/$progname.pid, and write a standard SysV init script (use for instance the one for ntpd as an example, it's simple) around it. Preferably, make it LSB-compliant, too.
Essentially, the start function tests if the program is already running (by testing if /var/run/$progname.pid exists, and the contents of that file is the PID of a running process), and if not run
logfile=/var/log/$progname.log
pidfile=/var/run/$progname.pid
nohup java -Dpidfile=$pidfile $jopts $mainClass </dev/null > $logfile 2>&1
The stop function checks on /var/run/$progname.pid, tests if that file is the PID of a running process, verifies that it is a Java VM (so as not to kill a process that simply reused the PID from a dead instance of my Java daemon) and then kills that process.
When called, my main() method will start by writing its PID in the file defined in System.getProperty("pidfile").
One major hurdle, though: in Java, there is no simple and standard way to get the PID of the process the JVM runs in.
Here is what I have come up with:
private static String getPid() {
File proc_self = new File("/proc/self");
if(proc_self.exists()) try {
return proc_self.getCanonicalFile().getName();
}
catch(Exception e) {
/// Continue on fall-back
}
File bash = new File("/bin/bash");
if(bash.exists()) {
ProcessBuilder pb = new ProcessBuilder("/bin/bash","-c","echo $PPID");
try {
Process p = pb.start();
BufferedReader rd = new BufferedReader(new InputStreamReader(p.getInputStream()));
return rd.readLine();
}
catch(IOException e) {
return String.valueOf(Thread.currentThread().getId());
}
}
// This is a cop-out to return something when we don't have BASH
return String.valueOf(Thread.currentThread().getId());
}
I frequently find myself writing scripts or command lines which essentially look like this, if I want to:
Run a program that is immune to sighups
That is completely disconnected from the shell which spawns it, and
Produces a log file from stderr and stdout the contents of which are displayed as well, but
Allows me to stop viewing the log in progress and do other stuff without disrupting the running process
Enjoy.
nohup java com.me.MyProgram </dev/null 2>&1 | tee logfile.log &
I prefer the nohup command. The blog post says there are better ways, but I don't think they're enough better.
You could try Java Service Wrapper, the community edition is free and meets your needs.
My preferred way on Ubuntu is to use the libslack 'daemon' utility. This is what Jenkins uses on Ubuntu (which is where I got the idea.) I've used it for my Jetty-based server applications and it works well.
When you stop the daemon process it will signal the JVM to shutdown. You can execute shutdown/cleanup code at this point by registering a shutdown hook with Runtime.addShutdownHook().
That depends. If it's just a one-time thing, I want to daemonize it and then go home, but usually I wait for the results, I might do:
nohup java com.me.MyProgram &
at the command line. To kill it cleanly, you have a lot of options. You might have a listener for SIGKILL, or listen on a port and shutdown when a connection is made, periodically check a file. Difference approaches have different weaknesses. If it's for use in production, I'd give it more thought, and probably throw a script into /etc/init.d that nohups it, and have a more sophisticated shutdown, such as what tomcat has.
DaemonTools :- A cleaner way to manage services at UNIX https://cr.yp.to/daemontools.html
Install daemon tools from the url https://cr.yp.to/daemontools/install.html
follow the instruction mentioned there,for any issues please try instructions https://gist.github.com/rizkyabdilah/8516303
Create a file at /etc/init/svscan.conf and add the below lines.(only required for cent-os-6.7)
start on runlevel [12345]
stop on runlevel [^12345]
respawn
exec /command/svscanboot
Create a new script named run inside /service/vm/ folder and add the below lines.
#!/bin/bash
echo starting VM
exec java -jar
/root/learning-/daemon-java/vm.jar
Note:
replace the Jar with your own Jar file. or any java class file.
Reboot the system
svstat /service/vm should be up and running now !.
svc -d /service/vm should bring vm down now !.
svc -u /service/vm should bring vm up now !.
This question is about daemonizing an arbitrary program (not java-specific) so some of the answers may apply to your case:
Take a look here:
http://jnicookbook.owsiak.org/recipe-no-022/
for a sample code that is based on JNI. In this case you daemonize the code that was started as Java and main loop is executed in C. But it is also possible to put main, daemon's, service loop inside Java.
https://github.com/mkowsiak/jnicookbook/tree/master/recipes/recipeNo029
Have fun with JNI!
nohup java -jar {{your-jar.jar}} > /dev/null &
This may do the trick.