We Keep Coding

ProcessBuilder/Runtime.exec() with Weka Command Line Demonstrating Peculiar Behavior

Below is basically an MCVE of my full problem, which is much messier. What you need to know is that the following line runs when directly put in terminal:
java -classpath /path/to/weka.jar weka.filters.MultiFilter \
-F "weka.filters.unsupervised.attribute.ClusterMembership -I first" \
-i /path/to/in.arff
This is relatively straightforward. Basically, all I am doing is trying to cluster the data from in.arff using all of the default settings for the ClusterMembership filter, but I want to ignore the first attribute. I have the MultiFilter there because in my actual project, there are other filters, so I need this to stay. Like previously mentioned, this works fine. However, when I try to run the same line with ProcessBuilder, I get a "quote parse error", and it seems like the whole structure of nesting quotes breaks down. One way of demonstrating this is trying to get the following to work:
List<String> args = new ArrayList<String>();
args.add("java");
args.add("-cp");
args.add("/path/to/weka.jar");
args.add("weka.filters.MultiFilter");
args.add("-F");
args.add("\"weka.filters.unsupervised.attribute.ClusterMembership");
args.add("-I");
args.add("first\"");
args.add("-i");
args.add("/path/to/in.arff");
ProcessBuilder pb = new ProcessBuiler(args);
// ... Run the process below
At first glance, you might think this is identical to the above line (that's certainly what my naive self thought). In fact, if I just print args out with spaces in between each one, the resulting strings are identical and run perfectly if directly copy and pasted to the terminal. However, for whatever reason, the program won't work as I got the message (from Weka) Quote parse error. I tried googling and found this question about how ProcessBuilder adds extra quotes to the command line (this led me to try numerous combinations of escape sequences, all of which did not work), and read this article about how ProcessBuilder/Runtime.exec() work (I tried both ProcessBuilder and Runtime.exec(), and ultimately the same problem persisted), but couldn't find anything relevant to what I needed. Weka already had bad documentation, and then their Wikispace page went down a couple weeks ago due to Wikispaces shutting down, so I have found very little info on the Weka side.
My question then is this: Is there a way to get something like the second example I put above to run such that I can group arguments together for much larger commands? I understand it may require some funky escape sequences (or maybe not?), or perhaps something else I have not considered. Any help here is much appreciated.
Edit: I updated the question to hopefully give more insight into what my problem is.

You don't need to group arguments together. It doesn't even work, as you've already noted. Take a look what happens when I call my Java programm like this:
java -jar Test.jar -i -s "-t 500"
This is my "program":
public class Test {
public static void main(String[] args) {
for( String arg : args ) {
System.out.println(arg);
}
}
}
And this is the output:
-i
-s
-t 500
The quotes are not included in the arguments, they are used to group the arguments. So when you pass the arguments to the ProcessBuilder like you did, it is essentially like you'd written them with quotes on the command line and they are treated as a single argument, which confuses the parser.
The quotes are only necessary when you have nested components, e.g. FilteredClassifier. Maybe my answer on another Weka question can help you with those nested components. (I recently changed the links to their wiki to point to the Google cache until they established a new wiki.)
Since you didn't specify what case exactly caused you to think about grouping, you could try to get a working command line for Weka and then use that one as input for a program like mine. You can then see how you would need to pass them to a ProcessBuilder.
For your example I'd guess the following will work:
List<String> args = new ArrayList<String>();
args.add("java");
args.add("-cp");
args.add("/path/to/weka.jar");
args.add("weka.filters.MultiFilter");
args.add("-F");
args.add("weka.filters.unsupervised.attribute.ClusterMembership -I first");
args.add("-i");
args.add("/path/to/in.arff");
ProcessBuilder pb = new ProcessBuiler(args);
Additional details
What happens inside Weka is basically the following: The options from the arguments are first processed by weka.filters.Filter, then all non-general filter options are processed by weka.filters.MultiFilter, which contains the following code in setOptions(...):
filters = new Vector<Filter>();
while ((tmpStr = Utils.getOption("F", options)).length() != 0) {
options2 = Utils.splitOptions(tmpStr);
filter = options2[0];
options2[0] = "";
filters.add((Filter) Utils.forName(Filter.class, filter, options2));
}
Here, tmpStr is the value for the -F option and will be processed by Utils.splitOption(tmpStr) (source code). There, all the quoting and unquoting magic happens, so that the next component will receive an options array that looks just like it would look if it was a first-level component.

Fortify issue - Command Injection

I am trying to do hp fortify security scan for my java application. I have few issues and i have fixed it. But i am unable to find the fix for the below issue.
Command Injection
String hostname = execReadToString("hostname").split("\\.")[0];
public static String execReadToString(String execCommand) throws IOException {
try (Scanner s = new Scanner(Runtime.getRuntime().exec(execCommand).getInputStream()).useDelimiter("\\A")) {
return s.hasNext() ? s.next() : "";
}
The method execReadToString() calls exec() to execute a command. This call might allow an attacker to inject malicious commands.
So i have tried with process builder also.
private static void gethostname(String cmd1) throws IOException {
if(Pattern.matches("[A-Za-z]+", cmd1)) {
ProcessBuilder pb = new ProcessBuilder(cmd1);
Process p = pb.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(
p.getInputStream()));
String readline;
while ((readline = reader.readLine()) != null) {
System.out.println(readline);
}
}
}
Even this is giving me an security issue This start() call might allow an attacker to inject malicious commands.
What will be the ideal fix for this issue?
Thanks in advance

Usually this is because you're using user input to frame the command string, wherein user can inject malicious code to manipulate what command is being run ultimately (even if you add validation there will be ways to circumvent that).
In your case you seem to be hardcoding the command so this shouldn't be a problem, however, see the OWASP page on hardcoded command invocation (emphasis mine):
Unlike the previous examples, the command in this example is
hardcoded, so an attacker cannot control the argument passed to
system(). However, since the program does not specify an absolute path
for make, and does not scrub any environment variables prior to
invoking the command, the attacker can modify their $PATH variable to
point to a malicious binary named make and execute the CGI script from
a shell prompt. And since the program has been installed setuid root,
the attacker's version of make now runs with root privileges.
The environment plays a powerful role in the execution of system
commands within programs. Functions like system() and exec() use the
environment of the program that calls them, and therefore attackers
have a potential opportunity to influence the behavior of these calls.
Resolution:
Use native Java APIs / libraries to achieve what you want, instead of running a command - this is probably the best option. Use commands only when unavoidable, eg: 3rd party tools which do not have a Java client library. This approach has the added advantage of being more portable and in most cases, more efficient too. This library might help your scenario.
If you have to run a command, ensure you do not use user-supplied or external data even indirectly to construct it.
Or if you're hardcoding the command to run from the code, use absolute path to the command and do not use environment variables as part of it. For hostname (assuming you use the built-in command) this is usually /usr/bin/hostname but you can find the command path for your environment using which hostname.

How to execute a linux terminal command from LUAJ?

I want to simply execute a linux terminal command like ls from LuaJ and the result that it will return or anything that returns i want to receive it and will show the names in the Java Gui. I searched but found this but not one with LuaJ.
Is there any function to execute the terminal command from LuaJ ??

There are multiple ways to do this, for one, you can implement it yourself in Java then link it to LuaJ.
LuaFunction command = new OneArgFunction()
{
public LuaValue call(LuaValue cmd)
{
Process p = Runtime.getRuntime().exec("/bin/sh", "-c", cmd.checkstring());
BufferedReader br = new BufferedReader(new InputStreamReader(p.getInputStream()));
int returnCode = p.waitFor();
return LuaValue.valueOf(returnCode);
}
}
globals.set("command", command);
Then in Lua:
local code = command("ls");
The problem with actually getting the output of a command is that you can't just have a fixall solution. For all the system knows you could be calling a program which runs for 2 hours generating constant output, which could be an issue, not to mention if the program requires input. If you know you're only going to use certain functions you can make a dirty version of above function to capture the output from the stream and return it all instead of the exit code, just don't use it on other processes that don't return quickly. The other alternative is to create a class that wraps the input and output streams from the process and return a coerced version of that class, and manage the input and output from lua.
Lua does have a function that's part of the OsLib called execute(), if execute doesn't exist in your current environment then in Java call:
globals.load(new OsLib());
Before loading the lua code. the os.execute() function returns the status code, and doesn't return the streams, so no way to get the output there. To get around this you can modify the command to pipe the output to a temp file and open it with the io library (new IoLib() if doesn't exist in current environment).
The other option is to use io.openProcess, which also executes the command and returns a file to read the output from.
Resources:
http://luaj.org/luaj/3.0/api/org/luaj/vm2/lib/OsLib.html
http://luaj.org/luaj/3.0/api/org/luaj/vm2/lib/IoLib.html

Running a python code using process builder Java

I need some help.
I am trying to run a python script called mantime.py from a directory. I've tried to google it and found several ways to do it. Yet, I still got 2 as the exit value, which I expect it 0 (terminate normally). Here is my code:
public int performedManTime() throws IOException, InterruptedException{
ProcessBuilder pb = new ProcessBuilder("/usr/bin/python","/Users/ab/Downloads/ManTIME/mantime.py","-ppp","test",inputDir.getAbsolutePath(),"i2b2");
Map<String,String>env = pb.environment();
env.put("MANTIME_CRF_TRAIN", "/usr/local/Cellar/crf++/0.58/bin/crf_learn");
env.put("MANTIME_CRF_TEST", "/usr/local/Cellar/crf++/0.58/bin/crf_test");
env.put("MANTIME_CORENLP_FOLDER","/Users/ab/Downloads/ManTIME/externals/stanford-corenlp-full-2014-08-27");
Process process = pb.start();
process.waitFor();
System.out.println("Exit Value: "+process.exitValue());
return process.exitValue();
}
-ppp, test, input.dir and i2b2 are the arguments for the mantime.py
I tried to set up the environment as we can see above. Does anyone knows what are the problems? Any comment or suggestion would be really appreciated. Thank you
EDIT: I suspect since the python is on different directory with my tool (/usr/local/python). After I put the code bellow, somehow it works.
ProcessBuilder("/usr/bin/python","/Users/ab/Downloads/ManTIME/mantime.py","-ppp","test",inputDir.getAbsolutePath(),"i2b2");
pb.directory(/myToolsDir)

I suspect since the python is on different directory with my tool (/usr/local/python). After I put the code bellow, somehow it works. Thank you guys
ProcessBuilder("/usr/bin/python","/Users/ab/Downloads/ManTIME/mantime.py","-ppp","test",inputDir.getAbsolutePath(),"i2b2");
pb.directory(/myToolsDir)

How to call a bash script from GWT-RPC server (Java)

I want to call a bash script from GWT server
I coded up my first application with GWT/RPC, and I need to call a bash script on the server side (from MyOwnServiceImpl extends RemoteServiceServlet implements MyOwnService).
ProcessBuilder doesn't work
To do that, I confess that I am using java.lang.ProcessBuilder, which is apparently "not supported by GAE" (I just ignored the warning). As it is running on the server side, it seemed to me that it should work anyways. I feel that I am missing something.
Something seems to be preventing the call from being executed, even though the required packages are correctly imported, the binaries are found, the execution doesn't crash. But the call is just not successful (for example even mkdir is not executed on the server).
Related posts weren't much help...
How to execute a Unix shell script via GWT? (does not give a complete answer, and I could not simply comment on the answer)
GWT + ProcessBuilder (mentions precisely the solution I implemented which is not working for me, see above)
Any thoughts on this would be much appreciated, thanks!

In a GWT application without GAE you can use Runtime.getRuntime().exec("some command");
If you want to read out the result of the command, you can use:
Process p = Runtime.getRuntime().exec("A command");
BufferedReader in = new BufferedReader(new InputStreamReader(p.getErrorStream()));
StringBuilder builder = new StringBuilder();
String line = null;
while ((line = in.readLine()) != null) {
builder.append(line + "\n");
}
String result = builder.toString();
If the command above should not work, i guess you have to remove GAE from your project to run a bash script.

you can call bash script with GWT,if your servet side on your computer.
the limit is GAE for secury. no way to cross this limit.