We Keep Coding

ProcessBuilder's inputstream empty depending of OS

I made this simple piece of code to test ProcessBuilder:
#SpringBootApplication
public class TerminalDemoApplication {
public static void main(String[] args) {
SpringApplication.run(TerminalDemoApplication.class, args);
try {
System.out.println("hello");
Process process = new ProcessBuilder("python", "--version").start();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
int exitCode = process.waitFor();
System.out.println("\nExited with error code : " + exitCode);
} catch (IOException | InterruptedException e) {
e.printStackTrace();
}
}
}
It works in Windows (returns python version of my system) but the same code in my macbook returns end of line, so basically empty. ¿This needs further configuration according to the OS? ¿why is this happening?

What error code are you getting?
There are (at least) two explanations; that error code would indicate which one it is.
You're not running python, or running 'the wrong' python
This would mean you are getting an error code of some sort, or an exception.
The likely reason for this is a path issue.
Running python, just like that - as in, no path information at all, is nominally neccessarily broken: That's just not how your OS works, it has no idea what to do with this path.
It's a bashism (as in, the shell does it, not the OS) to interpret such a command as 'oh, actually, go through each listed entry in the $PATH environment variable, and stick that path in front of this name, see if you find an executable there. If you do, run that and stop).
Java mostly doesn't engage in any bashisms. But, in a few bizarre places, it does - it tries to do basic space splitting when you use the single-string version of new ProcessBuilder), which is a shellism, and it does attempt to do basic PATH lookup, but that's about where it ends. It won't do * unpacking, which on windows is an OS-level thing but on posix systems is a shellism.
I strongly, strongly advise you to avoid java's basic shellisms. It's unreliable and highly OS-specific.
So: Always pass arguments explicitly (good, you're doing that), always use ProcessBuilder (good, you're doing that), never use relative paths (that's where you're going wrong).
It's going to the error stream instead
processes on OSes are generally hooked up to 3 pipes, not 2. There's the 'standard in', the 'standard out' and the 'standard err'. Your own java process exposes these as System.out, in, and err.
In linux in particular, it is common to redirect standard out of some process to a file or another process.
This means that standard err naturally has the property that it tends to emit to the console, even if you are redirecting things. In other words, the terms 'standard out' and 'standard err' are really stupid names on posix. The much better naming would be 'standard process output' and 'standard process messages'.
Asking python to print its version is in a bit of a limbo scenario. The string "Python v3.0.1" or whatnot is certainly not an error, but it's a bit dubious if one should consider this as 'the output of the process'. It's likely that the authors of the python tool consider it more 'some information I should print to you, even if you are redirecting things.
Thus, my guess is that this version is heading out to standard err instead.
You can solve this in two ways: Either read from standard err as well, or, use process builder's features: You can ask it to bundle up standard out and standard err into a single stream (.redirectErrorStream(true)).
I would expect the exit code to be 0 if this explanation is the correct one.

Java jar execution stuck when launched in a process by another java program

This is a very unusual problem I've come across and I'm hoping someone might have some insight on it. I'm on macOS Mojave (10.14.6), using Amazon's JRE Corretto-11.0.9.12.1 (build 11.0.9.1+12-LTS)
I have a program I've written that is something of a scripting engine that we use to do our routine bulk data processing. It takes an xml "script" file which contains the processing directions to execute and a list of input files as arguments. I'll refer to this as the "engine" from this point on. This engine is the backbone of a large portion of our automation. It shows a progress bar while processing to let users know that it is working.
There are 2 programs that use this engine:
One is a thin UI written in Swing, which we use to manually process data; it generates an xml file from the user input and passes it along with the input files and launches the engine in a separate process; the UI itself doesn't process any data.
The other watches a folder on our file server and processes incoming data from our clients daily when a folder is created inside of it so we can rip the data into our database. I'll call this the "importer".
Recently, a problem has come up where the engine becomes stuck while processing. Older versions of the engine did not have this problem, and I'm trying to figure out what exactly changed that caused this to happen, but while I've been trying to do this, our UI and importer programs have been using and older version of the engine. There are new features that we need to use in the new version of the engine, but we can't use it until this problem is solved.
The programs that uses the engine launch it in a process then waits for the result before continuing:
// example command generated from external program
String commandString = "java -jar engine.jar script.xml input_file1.txt input_file2.txt input_file3.txt";
String[] command = {"bash", "-c", commandString};
// I can grab the command from here for debugging
System.out.println(Arrays.toString(command));
ProcessBuilder pb = new ProcessBuilder(command);
// wait for the process to complete before continuing
Process p = pb.start();
p.waitFor();
int result = p.exitValue();
try (BufferedReader e = new BufferedReader(new InputStreamReader(proc.getErrorStream()));
BufferedReader i = new BufferedReader(new InputStreamReader(proc.getInputStream()))) {
String line;
while ((line = e.readLine()) != null) {
System.out.println(line);
}
while ((line = i.readLine()) != null) {
System.out.println(line);
}
}
p.destroy();
// do other stuff
When launched in this way, there is a specific operation that causes the engine to hang. But if I take the command and launch it directly from the command line, the engine runs just fine! This is making it difficult to pin down where exactly the problem is; is it in the engine, or in the other programs? I've spent a couple of days looking for answers and come up with nothing. It's even more frustrating that this problem has appeared seemingly out of nowhere when it was working perfectly before, using the exact code above, for a quite a long time.
The operation where the engine hangs sorts files into folders based on their file names. When I watch my activity monitor while it runs, it's not taxing my resources at all, and disk space isn't an issue. It isn't a file permission issue, as the engine is creating files and folders all the time and in every step leading up to the step where it hangs. And as I said, if I run the command directly from the command line, it creates the folders and sorts the files without issue, to my extreme confusion.
The importer and UI run locally on a station, but the engine jar file lives on our file server, so that it is accessible to every station without individually downloading it everywhere each time there is an update. I thought at first that the issue might lie in the fact that it is being accessed over the network, but the problem occurs even when I use a local copy of the engine on my dev machine, so I have ruled that out. I've also ruled out that it's the JRE, even though we switched to it recently, since the older version of the engine still perform as expected.

There might of course be any reason why your 'engine' program may hang ;-) but certainly it will hang you don't read the its output, and in the right way:
The parent process needs to read the standard output and standard error streams of the child process, given that the child process does generate any substantial amount of output on any of these two channels. This must be done in two separate background threads. If the parent does not read the child's output, then the child process will block as soon as the (small) buffer between the processes is filled up.
The threads should be started as soon as the child process is started, and before the parent calls process.waitFor().
The simplest way to do this is the following:
Process process = processBuilder.start();
InputStream stdout = process.getInputStream();
InputStream stderr = process.getErrorStream();
Thread stdoutThread = new Thread() {
#Override
public void run() {
// read stdout here, e.g.
try {
int c;
while (-1 != (c = stdout.read())) {
// do whatever with c
}
} catch (IOException ioex) {
// ...
}
}
};
Thread stderrThread = new Thread() {
// ... same as for stdout
};
stdoutThread.start();
stderrThread.start();
}
Only after both threads have been started you may wait for the child process and join the threads:
int exitValue = process.waitFor();
stdoutThread.join();
stderrThread.join();
There might be more sophisticated ways to work with background threads using the Concurrency Framework introduced in Java 5, but this basic code gives the idea.

How to listen to (take a copy / split ) System.out messages

I have an integration test (ClientIT) which uses the logging output from a test helper Class (ClientBasic) to determine whether the test passes or fails. I have re-directed the System.out/System.err inside the ClientBasic class to provide a link back to ClientIT using an OutputStream as follows,
System.setOut(new PrintStream(ClientBasic.out));
System.setErr(new PrintStream(ClientBasic.err));
where,
static OutputStream out;
static OutputStream err;
In ClientIT I call Client basic as follows,
ClientBasic.process(clientArgs.getArguments(), out, err);
This works fine, exactly how I wanted except when I run it using the Maven Failsafe plugins as part of the Package/Verify goal I get the message,
[WARNING] Corrupted STDOUT by directly writing to native stream in
forked JVM 1. See FAQ web page and the dump file
/path-to-project/target/failsafe-reports/2019-07-17T13-33-44_769-jvmRun1.dumpstream
which has the effect of really corrupting the display output of any logging info when the ClientBasic runs as part of my integration test - run locally from the command line or in Jenkins (using mvn ...) - strangely it still works fine when I run it from inside the IDE - maybe its not calling the failsafe plug-in directly ?
Anyway the above effect is documented on the Maven site as Corrupted STDOUT
So to try and get around this problem what I would like to do is to simply hook onto the System.out/err stream and create a copy - i.e. a bit like a splitter, rather than re-directing the System.out stream.
Does anyone now if this is possible and if so how to do this ?

As the error does not immediately blame setOut/setErr, it should be possible to make ones own splitter extending PrintStream.
public class Splitter extends PrintStream {
public Splitter(PrintStream sysPS, PrintWriter myOut) { ... }
... override methods by code generation in IDE.
}
PrintStream myOut = ...;
Splitter outSplitter = new Splitter(System.out, myOut);
Splitter errSplitter = new Splitter(System.err, myOut);
System.setOut(outSplitter);
System.setErr(errSplitter);
...
// At end:
System.setOut(outSplitter.getSysPS());
System.setErr(errSplitter.getSysPS());
IDE code generation and a couple of smart regex replaces will give you a correct class.
Alternatively there probably exists such a splitter as a Logger.

how to implement expect "interact" command using java

I want to implement the expect "interact" command using java. In expect, it's possible to open an ssh session, authenticate and, then, use the "interact" command to give the control back to the user. Is that possible with java? I've tried with expectJ, expect4J and expectForJava but there's little documentation and almost no examples of how to do this. TIA.
Update: for "interact" command reference, please check this out: http://wiki.tcl.tk/3914
"Interact is an Expect command which gives control of the current
process to the user, so that keystrokes are sent to the current
process, and the stdout and stderr of the current process are
returned."

In case anyone is interested, I have added basic interactive loop support to ExpectIt, my own open source Expect for Java implementation (sorry for self-promotion), since version 0.8.
Here is an example of interacting with the system input stream in Java 8:
try (final Expect expect = new ExpectBuilder()
.withInputs(System.in)
.build()) {
expect.interact()
.when(contains("abc")).then(r -> System.out.println("A"))
.when(contains("xyz")).then(r -> System.err.println("B"))
.until(contains("exit"));
System.out.println("DONE!");
}
System.in.close();

These libraries might suit your needs better:
SSHJ
https://github.com/shikhar/sshj
JSCH
http://www.jcraft.com/jsch/

linux shell and java

I want to send a command to linux shell and get it's response with java.How can i do this?

Have a look at ProcessBuilder - example here.

You should look at the Runtime class, and its exec() family of methods.
It's probably best to explicitly specify that you want to run the command through a shell, i.e. create a command line like "bash -c 'my command'".

Execute a process like this
Runtime.getRuntime().exec("ls");
...then you could get the process input stream and read it with a Reader to get the response

See the Runtime class and the exec() method.
Note that you need to consume the process's stdout/sterr concurrently, otheriwse you'll get peculiar blocking behaviour. See this answer for more information.

I wrote a little class to do this in a very similar question a couple of weeks ago:
java shell for executing/coordinating processes?
The class basically let's you do:
ShellExecutor excutor = new ShellExecutor("/bin/bash", "-s");
try {
System.out.println(excutor.execute("ls / | sort -r"));
} catch (IOException e) {
e.printStackTrace();
}