This is a very unusual problem I've come across and I'm hoping someone might have some insight on it. I'm on macOS Mojave (10.14.6), using Amazon's JRE Corretto-11.0.9.12.1 (build 11.0.9.1+12-LTS)
I have a program I've written that is something of a scripting engine that we use to do our routine bulk data processing. It takes an xml "script" file which contains the processing directions to execute and a list of input files as arguments. I'll refer to this as the "engine" from this point on. This engine is the backbone of a large portion of our automation. It shows a progress bar while processing to let users know that it is working.
There are 2 programs that use this engine:
One is a thin UI written in Swing, which we use to manually process data; it generates an xml file from the user input and passes it along with the input files and launches the engine in a separate process; the UI itself doesn't process any data.
The other watches a folder on our file server and processes incoming data from our clients daily when a folder is created inside of it so we can rip the data into our database. I'll call this the "importer".
Recently, a problem has come up where the engine becomes stuck while processing. Older versions of the engine did not have this problem, and I'm trying to figure out what exactly changed that caused this to happen, but while I've been trying to do this, our UI and importer programs have been using and older version of the engine. There are new features that we need to use in the new version of the engine, but we can't use it until this problem is solved.
The programs that uses the engine launch it in a process then waits for the result before continuing:
// example command generated from external program
String commandString = "java -jar engine.jar script.xml input_file1.txt input_file2.txt input_file3.txt";
String[] command = {"bash", "-c", commandString};
// I can grab the command from here for debugging
System.out.println(Arrays.toString(command));
ProcessBuilder pb = new ProcessBuilder(command);
// wait for the process to complete before continuing
Process p = pb.start();
p.waitFor();
int result = p.exitValue();
try (BufferedReader e = new BufferedReader(new InputStreamReader(proc.getErrorStream()));
BufferedReader i = new BufferedReader(new InputStreamReader(proc.getInputStream()))) {
String line;
while ((line = e.readLine()) != null) {
System.out.println(line);
}
while ((line = i.readLine()) != null) {
System.out.println(line);
}
}
p.destroy();
// do other stuff
When launched in this way, there is a specific operation that causes the engine to hang. But if I take the command and launch it directly from the command line, the engine runs just fine! This is making it difficult to pin down where exactly the problem is; is it in the engine, or in the other programs? I've spent a couple of days looking for answers and come up with nothing. It's even more frustrating that this problem has appeared seemingly out of nowhere when it was working perfectly before, using the exact code above, for a quite a long time.
The operation where the engine hangs sorts files into folders based on their file names. When I watch my activity monitor while it runs, it's not taxing my resources at all, and disk space isn't an issue. It isn't a file permission issue, as the engine is creating files and folders all the time and in every step leading up to the step where it hangs. And as I said, if I run the command directly from the command line, it creates the folders and sorts the files without issue, to my extreme confusion.
The importer and UI run locally on a station, but the engine jar file lives on our file server, so that it is accessible to every station without individually downloading it everywhere each time there is an update. I thought at first that the issue might lie in the fact that it is being accessed over the network, but the problem occurs even when I use a local copy of the engine on my dev machine, so I have ruled that out. I've also ruled out that it's the JRE, even though we switched to it recently, since the older version of the engine still perform as expected.
There might of course be any reason why your 'engine' program may hang ;-) but certainly it will hang you don't read the its output, and in the right way:
The parent process needs to read the standard output and standard error streams of the child process, given that the child process does generate any substantial amount of output on any of these two channels. This must be done in two separate background threads. If the parent does not read the child's output, then the child process will block as soon as the (small) buffer between the processes is filled up.
The threads should be started as soon as the child process is started, and before the parent calls process.waitFor().
The simplest way to do this is the following:
Process process = processBuilder.start();
InputStream stdout = process.getInputStream();
InputStream stderr = process.getErrorStream();
Thread stdoutThread = new Thread() {
#Override
public void run() {
// read stdout here, e.g.
try {
int c;
while (-1 != (c = stdout.read())) {
// do whatever with c
}
} catch (IOException ioex) {
// ...
}
}
};
Thread stderrThread = new Thread() {
// ... same as for stdout
};
stdoutThread.start();
stderrThread.start();
}
Only after both threads have been started you may wait for the child process and join the threads:
int exitValue = process.waitFor();
stdoutThread.join();
stderrThread.join();
There might be more sophisticated ways to work with background threads using the Concurrency Framework introduced in Java 5, but this basic code gives the idea.
I am trying to do hp fortify security scan for my java application. I have few issues and i have fixed it. But i am unable to find the fix for the below issue.
Command Injection
String hostname = execReadToString("hostname").split("\\.")[0];
public static String execReadToString(String execCommand) throws IOException {
try (Scanner s = new Scanner(Runtime.getRuntime().exec(execCommand).getInputStream()).useDelimiter("\\A")) {
return s.hasNext() ? s.next() : "";
}
The method execReadToString() calls exec() to execute a command. This call might allow an attacker to inject malicious commands.
So i have tried with process builder also.
private static void gethostname(String cmd1) throws IOException {
if(Pattern.matches("[A-Za-z]+", cmd1)) {
ProcessBuilder pb = new ProcessBuilder(cmd1);
Process p = pb.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(
p.getInputStream()));
String readline;
while ((readline = reader.readLine()) != null) {
System.out.println(readline);
}
}
}
Even this is giving me an security issue This start() call might allow an attacker to inject malicious commands.
What will be the ideal fix for this issue?
Thanks in advance
Usually this is because you're using user input to frame the command string, wherein user can inject malicious code to manipulate what command is being run ultimately (even if you add validation there will be ways to circumvent that).
In your case you seem to be hardcoding the command so this shouldn't be a problem, however, see the OWASP page on hardcoded command invocation (emphasis mine):
Unlike the previous examples, the command in this example is
hardcoded, so an attacker cannot control the argument passed to
system(). However, since the program does not specify an absolute path
for make, and does not scrub any environment variables prior to
invoking the command, the attacker can modify their $PATH variable to
point to a malicious binary named make and execute the CGI script from
a shell prompt. And since the program has been installed setuid root,
the attacker's version of make now runs with root privileges.
The environment plays a powerful role in the execution of system
commands within programs. Functions like system() and exec() use the
environment of the program that calls them, and therefore attackers
have a potential opportunity to influence the behavior of these calls.
Resolution:
Use native Java APIs / libraries to achieve what you want, instead of running a command - this is probably the best option. Use commands only when unavoidable, eg: 3rd party tools which do not have a Java client library. This approach has the added advantage of being more portable and in most cases, more efficient too. This library might help your scenario.
If you have to run a command, ensure you do not use user-supplied or external data even indirectly to construct it.
Or if you're hardcoding the command to run from the code, use absolute path to the command and do not use environment variables as part of it. For hostname (assuming you use the built-in command) this is usually /usr/bin/hostname but you can find the command path for your environment using which hostname.
This is my first post here, although I've been reading stuff here for quite some time :)
I'm currently doing my bachelor thesis in computer science where we're doing an Indoors-positioning thing with iBeacons and Bluetooth. Where the device "being found" is currently a Raspberry Pi due to complications with smartphones.
I've got a bit of a problem with the btmon command on my Raspberry Pi. The full structure of our program is as follows:
Scan for bluetooth devices (iBeacons)
Save the output result <---- Here's the problem
Filter the desired data
Send to server node
As this is a part of a program we're building, the commands are executed through Java code that executes the terminal command, and returns the output as a String. The method looks like this:
private String executeCommand(String command) {
StringBuffer output = new StringBuffer();
Process p;
try {
p = Runtime.getRuntime().exec(command);
p.waitFor();
BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = "";
while ((line = reader.readLine()) != null) {
output.append(line + "\r\n");
}
} catch (Exception e) {
e.printStackTrace();
}
return output.toString();
}
I've got everything working when using another command (such as ifconfig) so the method works for ordinary commands, but I can't get it to work with btmon.
The full command I'm trying to run is this:
sudo btmon & sudo hcitool lescan
It seems to me as if btmon is running as a different process/runtime which would explain why the above Java code doesn't catch its output. I've been trying to "get" the btmon runtime but have failed to do so, and I've been trying to use ProcessBuilder with inheritIO() to change it's output source but couldn't get that to work either...
I've been googling this for a couple of days by now, and I've bumped into a couple of Python scripts that I can't get to work either (probably because I've never used Python before ^^).
So, in short: How do I save the output of sudo btmon & sudo hcitool lescan? Preferably to a String through Java code, but saving it as a file would also work!
Thanks in advance! :)
I got the same problem and spent many days to solve this issue. The key point was Thread and not using &. Here is what I discovered:
& literally runs in the background. Change sudo btmon & to sudo btmon
Use Thread.
Thread 1 - start btmon (You can get result this Thread)
Thread 2 - start hcitool lescan
(btmon should first - because "lescan" occur hci events so btmon can get them) Summary - do not use & and use independent thread.
What's the easiest way to execute a Python script from Java, and receive the output of that script? I've looked for different libraries like Jepp or Jython, but most appear out of date. Another problem with the libraries is that I need to be able to easily include a library with the source code (though I don't need to source for the library itself) if I use a library.
Because of this, would the easiest/most effective way be to simply do something like call the script with runtime.exec, and then somehow capture printed output? Or, even though it would be very painful for me, I could also just have the Python script output to a temporary text file, then read the file in Java.
Note: the actual communication between Java and Python is not a requirement of the problem I am trying to solve. This is, however, the only way I can think of to easily perform what needs to be done.
Not sure if I understand your question correctly, but provided that you can call the Python executable from the console and just want to capture its output in Java, you can use the exec() method in the Java Runtime class.
Process p = Runtime.getRuntime().exec("python yourapp.py");
You can read up on how to actually read the output from this resource:
http://www.devdaily.com/java/edu/pj/pj010016
import java.io.*;
public class JavaRunCommand {
public static void main(String args[]) {
String s = null;
try {
// run the Unix "ps -ef" command
// using the Runtime exec method:
Process p = Runtime.getRuntime().exec("ps -ef");
BufferedReader stdInput = new BufferedReader(new
InputStreamReader(p.getInputStream()));
BufferedReader stdError = new BufferedReader(new
InputStreamReader(p.getErrorStream()));
// read the output from the command
System.out.println("Here is the standard output of the command:\n");
while ((s = stdInput.readLine()) != null) {
System.out.println(s);
}
// read any errors from the attempted command
System.out.println("Here is the standard error of the command (if any):\n");
while ((s = stdError.readLine()) != null) {
System.out.println(s);
}
System.exit(0);
}
catch (IOException e) {
System.out.println("exception happened - here's what I know: ");
e.printStackTrace();
System.exit(-1);
}
}
}
There is also an Apache library (the Apache exec project) that can help you with this. You can read more about it here:
http://www.devdaily.com/java/java-exec-processbuilder-process-1
http://commons.apache.org/exec/
You can include the Jython library in your Java Project. You can download the source code from the Jython project itself.
Jython does offers support for JSR-223 which basically lets you run a Python script from Java.
You can use a ScriptContext to configure where you want to send your output of the execution.
For instance, let's suppose you have the following Python script in a file named numbers.py:
for i in range(1,10):
print(i)
So, you can run it from Java as follows:
public static void main(String[] args) throws ScriptException, IOException {
StringWriter writer = new StringWriter(); //ouput will be stored here
ScriptEngineManager manager = new ScriptEngineManager();
ScriptContext context = new SimpleScriptContext();
context.setWriter(writer); //configures output redirection
ScriptEngine engine = manager.getEngineByName("python");
engine.eval(new FileReader("numbers.py"), context);
System.out.println(writer.toString());
}
And the output will be:
1
2
3
4
5
6
7
8
9
As long as your Python script is compatible with Python 2.5 you will not have any problems running this with Jython.
I met the same problem before, also read the answers here, but doesn't found any satisfy solution can balance the compatibility, performance and well format output, the Jython can't work with extend C packages and slower than CPython. So finally I decided to invent the wheel myself, it took my 5 nights, I hope it can help you too: jpserve(https://github.com/johnhuang-cn/jpserve).
JPserve provides a simple way to call Python and exchange the result by well format JSON, few performance loss. The following is the sample code.
At first, start jpserve on Python side
>>> from jpserve.jpserve import JPServe
>>> serve = JPServe(("localhost", 8888))
>>> serve.start()
INFO:JPServe:JPServe starting...
INFO:JPServe:JPServe listening in localhost 8888
Then call Python from JAVA side:
PyServeContext.init("localhost", 8888);
PyExecutor executor = PyServeContext.getExecutor();
script = "a = 2\n"
+ "b = 3\n"
+ "_result_ = a * b";
PyResult rs = executor.exec(script);
System.out.println("Result: " + rs.getResult());
---
Result: 6
Jep is anther option. It embeds CPython in Java through JNI.
import jep.Jep;
//...
try(Jep jep = new Jep(false)) {
jep.eval("s = 'hello world'");
jep.eval("print(s)");
jep.eval("a = 1 + 2");
Long a = (Long) jep.getValue("a");
}
I've looked for different libraries like Jepp or Jython, but most seem to be very out of date.
Jython is not "a library"; it's an implementation of the Python language on top of the Java Virtual Machine. It is definitely not out of date; the most recent release was Feb. 24 of this year. It implements Python 2.5, which means you will be missing a couple of more recent features, but it is honestly not much different from 2.7.
Note: the actual communication between Java and Python is not a requirement of the aforementioned assignment, so this isn't doing my homework for me. This is, however, the only way I can think of to easily perform what needs to be done.
This seems extremely unlikely for a school assignment. Please tell us more about what you're really trying to do. Usually, school assignments specify exactly what languages you'll be using for what, and I've never heard of one that involved more than one language at all. If it did, they'd tell you if you needed to set up this kind of communication, and how they intended you to do it.
Jython approach
Java is supposed to be platform independent, and to call a native application (like python) isn't very platform independent.
There is a version of Python (Jython) which is written in Java, which allow us to embed Python in our Java programs. As usually, when you are going to use external libraries, one hurdle is to compile and to run it correctly, therefore we go through the process of building and running a simple Java program with Jython.
We start by getting hold of jython jar file:
https://www.jython.org/download.html
I copied jython-2.5.3.jar to the directory where my Java program was going to be. Then I typed in the following program, which do the same as the previous two; take two numbers, sends them to python, which adds them, then python returns it back to our Java program, where the number is outputted to the screen:
import org.python.util.PythonInterpreter;
import org.python.core.*;
class test3{
public static void main(String a[]){
PythonInterpreter python = new PythonInterpreter();
int number1 = 10;
int number2 = 32;
python.set("number1", new PyInteger(number1));
python.set("number2", new PyInteger(number2));
python.exec("number3 = number1+number2");
PyObject number3 = python.get("number3");
System.out.println("val : "+number3.toString());
}
}
I call this file "test3.java", save it, and do the following to compile it:
javac -classpath jython-2.5.3.jar test3.java
The next step is to try to run it, which I do the following way:
java -classpath jython-2.5.3.jar:. test3
Now, this allows us to use Python from Java, in a platform independent manner. It is kind of slow. Still, it's kind of cool, that it is a Python interpreter written in Java.
ProcessBuilder is very easy to use.
ProcessBuilder pb = new ProcessBuilder("python","Your python file",""+Command line arguments if any);
Process p = pb.start();
This should call python. Refer to the process approach here for full example!
https://bytes.com/topic/python/insights/949995-three-ways-run-python-programs-java
You can try using groovy. It runs on the JVM and it comes with great support for running external processes and extracting the output:
http://groovy.codehaus.org/Executing+External+Processes+From+Groovy
You can see in this code taken from the same link how groovy makes it easy to get the status of the process:
println "return code: ${ proc.exitValue()}"
println "stderr: ${proc.err.text}"
println "stdout: ${proc.in.text}" // *out* from the external program is *in* for groovy
First I would recommend to use ProcessBuilder ( since 1.5 )
Simple usage is described here
https://stackoverflow.com/a/14483787
For more complex example refer to
http://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html
I've encountered problem when launching Python script from Java, script
was producing too much output to standard out and everything went bad.
The best way to achieve would be to use Apache Commons Exec as I use it for production without problems even for Java 8 environment because of the fact that it lets you execute any external process (including python, bash etc) in synchronous and asynchronous way by using watchdogs.
CommandLine cmdLine = new CommandLine("python");
cmdLine.addArgument("/my/python/script/script.py");
DefaultExecuteResultHandler resultHandler = new DefaultExecuteResultHandler();
ExecuteWatchdog watchdog = new ExecuteWatchdog(60*1000);
Executor executor = new DefaultExecutor();
executor.setExitValue(1);
executor.setWatchdog(watchdog);
executor.execute(cmdLine, resultHandler);
// some time later the result handler callback was invoked so we
// can safely request the exit value
resultHandler.waitFor();
Complete source code for a small but complete POC is shared here that addresses another concern in this post;
https://github.com/raohammad/externalprocessfromjava.git
I am trying to create a GUI using java swing. From there I have to run linux system commands. I tried using exec(). But the exec() function is unable to parse the string if it contains single quotes. The code which I have used is as follows-
Process p = Runtime.getRuntime().exec("cpabe-enc pub_key message.txt '( it_department or ( marketing and manager ) )'")
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
But I am getting error when I run the program as--syntax error at "'(".
The same command runs when I write
Process p = Runtime.getRuntime().exec("cpabe-enc pub_key message.txt default")
Please help. Thanks in advance for your help.
Split up the parameters into an array instead, one string for each argument, and use the exec-method that takes as String[] instead, that generally works better for arguments.
Somethign along the lines of:
Runtime.getRuntime().exec(new String[] {"cpabe-enc", "pub_key", "message.txt", "( it_department or ( marketing and manager ) )"});
or whatever what your exact parameters are.
Its because the runtime does not interpret the '(...)' as a single parameter like you intend.
Try using ProcessBuilder instead:
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/ProcessBuilder.html
I recently got this kind of problem solved. I was using javaFX to call shell scripts on button click .. which is very much similar to your swing application scenario...
Here are the links hope it might help you...
How to code in java to run unix shell script which use rSync internally in windows environment using cygwin?
Getting error in calling shell script in windows environment using java code and cygwin...!
Happy coding... :)