I followed these tutorials How to use cURL in Java?, https://www.baeldung.com/java-curl to learn how to run curl from Java but it is not working.
The curl command (to delete page) is running fine from the terminal and gives a response in XML, but nada when I try in java, no error, no XML, the page also remains intact.
This is what I tried:
public class TryCurl {
static String command = "curl -u username:password -X POST -F cmd=\"deletePage\" -F path=\"/content/demo-task/section\" https://author1.dev.demo.adobecqms.net/bin/wcmcommand";
public static void main(String[] args) throws Exception {
Process process = Runtime.getRuntime().exec(command);
process.getInputStream();
process.destroy();
}
}
What am I doing wrong here?
Why are you using CURL for this; that's very 'fragile' (likely to fail in the future, it requires a lot of things to be in place, such as: curl to even be installed on the hardware you run this on, curl to be in the path, the password to not have spaces or other special characters in it, and more. Java is perfectly capable of making a POST request to a server.
That's not how a single process command works. Runtime.exec is not a shell - things like: Untangle quotes into arguments are shellisms, and process doesn't do any of it. Use ProcessBuilder and pass each argument separately, no quotes.
You can't just get the inputstream and discard, you need to actually read the bytes off of it, or curl will just sit there and wait for somebody to grab the data it is spitting out to standard out. This gets complicated, which brings us back to #1 which is probably a lot easier here.
you immediately run process.destroy() which will, rather obviously, destroy that process, hence the name. curl will just quit, because you asked it to, before it so much as finishes sending the POST.
execing is a lot more complicated than it sounds.
Related
I'm making an app which, using root, takes a logcat so it can find a certain error on another app. The easiest approach for me is saving the logcat to a file directly from the command. So all I have to do is run su and then logcat | grep --line-buffered "search string" > /path/to/save/logcat.log. When I run this on a terminal emulator (like this or even this), it saves the output to a file just exactly how I want it to do so. But when I run the exact same command from my app, it gets me a blank file. I've tried many different ways to output the logcat but they all get me an empty file. Interestingly, when I take a normal logcat using the app (without grep, using ">" to output), the file is being saved as it should and it contains the string I want to grep. What am I doing wrong?
Here is the code I use:
try {
Process p = Runtime.getRuntime().exec("su");
DataOutputStream dos = new DataOutputStream(p.getOutputStream());
dos.writeBytes("logcat | grep --line-buffered \"search string\" > /storage/emulated/0/logcat.log\n");
dos.flush();
} catch (IOException e) {
e.printStackTrace();
}
I'm listing my comment as an answer since that evidently helped solve your issue but there is more going on here which I'm not capable of fully addressing:
Try redirecting stderr as well to see if there is any error which can then be captured - I think that would be &> (that's bash) - and >outfile 2>&1 for more general syntax.
So
dos.writeBytes("logcat | grep --line-buffered \"search string\" &> /storage/emulated/0/logcat.log\n");
Or
dos.writeBytes("logcat | grep --line-buffered \"search string\" > /storage/emulated/0/logcat.log 2>&1\n");
The original intention of the comment was to get you more information as to what was really going on - as it turns out, it helped you get the result you were looking for.
I believe there are a few factors at work here which may contribute to why adding stderr helped:
stderr (which the comment suggested adding) is non-buffered - I think (but can't prove) the non-buffering is what is helping you even though the output being captured is stdout.
stdout processing is sensitive to TTY (terminal emulation) vs non-TTY (your program) and has different buffering approaches (line buffering in TTY otherwise fully buffered). I realize your grep option should overcome this. This difference in TTY-nonTTY may explain the source of your problem.
The code posted is sending a command (logcat...) to the process created and continuing. So for example if the logcat were to output lots of data, in theory your posted code would continue and leave scope - what happens to the process created when p is out scope - not sure.
Anyways, glad you were able to proceed.
Below is basically an MCVE of my full problem, which is much messier. What you need to know is that the following line runs when directly put in terminal:
java -classpath /path/to/weka.jar weka.filters.MultiFilter \
-F "weka.filters.unsupervised.attribute.ClusterMembership -I first" \
-i /path/to/in.arff
This is relatively straightforward. Basically, all I am doing is trying to cluster the data from in.arff using all of the default settings for the ClusterMembership filter, but I want to ignore the first attribute. I have the MultiFilter there because in my actual project, there are other filters, so I need this to stay. Like previously mentioned, this works fine. However, when I try to run the same line with ProcessBuilder, I get a "quote parse error", and it seems like the whole structure of nesting quotes breaks down. One way of demonstrating this is trying to get the following to work:
List<String> args = new ArrayList<String>();
args.add("java");
args.add("-cp");
args.add("/path/to/weka.jar");
args.add("weka.filters.MultiFilter");
args.add("-F");
args.add("\"weka.filters.unsupervised.attribute.ClusterMembership");
args.add("-I");
args.add("first\"");
args.add("-i");
args.add("/path/to/in.arff");
ProcessBuilder pb = new ProcessBuiler(args);
// ... Run the process below
At first glance, you might think this is identical to the above line (that's certainly what my naive self thought). In fact, if I just print args out with spaces in between each one, the resulting strings are identical and run perfectly if directly copy and pasted to the terminal. However, for whatever reason, the program won't work as I got the message (from Weka) Quote parse error. I tried googling and found this question about how ProcessBuilder adds extra quotes to the command line (this led me to try numerous combinations of escape sequences, all of which did not work), and read this article about how ProcessBuilder/Runtime.exec() work (I tried both ProcessBuilder and Runtime.exec(), and ultimately the same problem persisted), but couldn't find anything relevant to what I needed. Weka already had bad documentation, and then their Wikispace page went down a couple weeks ago due to Wikispaces shutting down, so I have found very little info on the Weka side.
My question then is this: Is there a way to get something like the second example I put above to run such that I can group arguments together for much larger commands? I understand it may require some funky escape sequences (or maybe not?), or perhaps something else I have not considered. Any help here is much appreciated.
Edit: I updated the question to hopefully give more insight into what my problem is.
You don't need to group arguments together. It doesn't even work, as you've already noted. Take a look what happens when I call my Java programm like this:
java -jar Test.jar -i -s "-t 500"
This is my "program":
public class Test {
public static void main(String[] args) {
for( String arg : args ) {
System.out.println(arg);
}
}
}
And this is the output:
-i
-s
-t 500
The quotes are not included in the arguments, they are used to group the arguments. So when you pass the arguments to the ProcessBuilder like you did, it is essentially like you'd written them with quotes on the command line and they are treated as a single argument, which confuses the parser.
The quotes are only necessary when you have nested components, e.g. FilteredClassifier. Maybe my answer on another Weka question can help you with those nested components. (I recently changed the links to their wiki to point to the Google cache until they established a new wiki.)
Since you didn't specify what case exactly caused you to think about grouping, you could try to get a working command line for Weka and then use that one as input for a program like mine. You can then see how you would need to pass them to a ProcessBuilder.
For your example I'd guess the following will work:
List<String> args = new ArrayList<String>();
args.add("java");
args.add("-cp");
args.add("/path/to/weka.jar");
args.add("weka.filters.MultiFilter");
args.add("-F");
args.add("weka.filters.unsupervised.attribute.ClusterMembership -I first");
args.add("-i");
args.add("/path/to/in.arff");
ProcessBuilder pb = new ProcessBuiler(args);
Additional details
What happens inside Weka is basically the following: The options from the arguments are first processed by weka.filters.Filter, then all non-general filter options are processed by weka.filters.MultiFilter, which contains the following code in setOptions(...):
filters = new Vector<Filter>();
while ((tmpStr = Utils.getOption("F", options)).length() != 0) {
options2 = Utils.splitOptions(tmpStr);
filter = options2[0];
options2[0] = "";
filters.add((Filter) Utils.forName(Filter.class, filter, options2));
}
Here, tmpStr is the value for the -F option and will be processed by Utils.splitOption(tmpStr) (source code). There, all the quoting and unquoting magic happens, so that the next component will receive an options array that looks just like it would look if it was a first-level component.
I am trying to use py4j to open up a gateway that I can use to pass objects from java into python. When I try to open a gateway with the py4j function launch_gateway it does not seem to properly connect to my Java class. However, when I launch my java class in the command line and then connect to it in python using JavaGateway everything works as expected. I would like to be able to use the built in method as I am sure that I am not accounting for things that have already been considered in the design of py4j, but I'm just not sure what I'm doing wrong.
Let's say I wanted to create a gateway to the class sandbox.demo.solver.UtilityReporterEntryPoint.class. In the command line I can do this by executing the following:
java -cp /Users/grr/anaconda/share/py4j/py4j0.10.4.jar: sandbox.demo.solver.UtilityReporterEntryPoint py4j.GatewayServer
This launches as expected and I can use the methods in my class from within python after connecting to the gateway. So far so good.
My understanding of the py4j documentation would lead me to believe I should do the following to launch the gateway in python:
port = launch_gateway(classpath='sandbox.demo.solver.UtilityReporterEntryPoint')
params = GatewayParameters(port=port)
gateway= JavaGateway(gateway_parameters=params)
I get no errors when executing these three lines, but when I try to access my java class methods with gateway.entry_point.someMethod() it fails with the following error:
Py4JError: An error occurred while calling t.getReport. Trace:
py4j.Py4JException: Target Object ID does not exist for this gateway :t
at py4j.Gateway.invoke(Gateway.java:277)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)
Obviously something is not getting called correctly within launch_gateway or I am feeding it the wrong information.
In the py4j source code for launch_gateway you can see that given the inputs you provide and those constructed by the function, a command is constructed that eventually gets called by subprocess.Popen. So given the input passed to launch_gateway above the command passed into Popen would be:
command = ['java', '-classpath', '/Users/grr/anaconda/share/py4j/py4j0.10.4.jar:sandbox.demo.solver.UtilityReporterEntryPoint', 'py4j.GatewayServer', '0']
Passing this command to Popen returns the listening port as expected. However, connecting to this listening port still does not allow access to my class methods.
Finally, passing the command as a single string to Popen without the final argument ('0'), properly launches a gateway which again operates as expected. Having taken a glance at the Java source code for py4j.GatewayServer.class this makes no sense as the main method seems to indicate that the class should exit with status 1 if the length of arguments is 0.
At this point I'm kind of at a loss. I can hack my way into a workable solution, but as I said I'm sure that ignores important aspects of the gateway behavior and I don't like hacky solutions. I'd love to tag #Barthelemy in this one, but hopefully he reads this. Thanks in advance for any help.
EDIT
For now I have been able to work around this issue with the following steps.
Package entire project including all external dependencies into a single jar file magABM-all.jar, with 'Main-Class' set to UtilityReporterEntryPoint.
Include if...else block regarding presence of --die-on-exit exactly like it is in GatewayServer.java
Use subprocess.Popen to call the command to run the project jar.
UtilityReporterEntryPoint.java
public static void main(String[] args) throws IOException {
GatewayServer server = new GatewayServer(new UtilityReporterEntryPoint());
System.out.println("Gateway Server Started");
server.start();
if (args[0].equals("--die-on-exit")) {
try {
BufferedReader stdin = new BufferedReader(new InputStreamReader(System.in, Charset.forName("UTF-8")));
stdin.readLine();
System.exit(0);
} catch (java.io.IOException e) {
System.exit(1);
}
}
}
app.py
def setup_gateway()
"""Launch a py4j gateway using UtilityReporterEntryPoint."""
process = subprocess.Popen('java -jar magABM-all.jar --die-on-exit', shell=True)
time.sleep(0.5)
gateway = JavaGateway()
return gateway
In this way I can still use gateway.shutdown if necessary and if the python process that starts the py4j gateway dies or is closed the gateway will be closed.
N.B I would by no means consider this a final solution as py4j was written by much smarter individuals with a clear purpose in mind and I am sure that there is a way to manage this exact workflow within the confines of py4j. This is just a stopgap solution.
There are a few issues:
The classpath parameter in launch_gateway should be a directory or a jar file, not a class name. For example, if you want to include additional Java libraries, you would add them to the classpath parameter.
The error you receive when you call gateway.entry_point.someMethod() means that you have no entry point. When you call launch_gateway, the JVM is started with GatewayServer.main, which launches a GatewayServer with no entry point: GatewayServer server = new GatewayServer(null, port). It is not possible currently to use launch_gateway and specify an entry point.
When you start the JVM with java -cp /Users/grr/anaconda/share/py4j/py4j0.10.4.jar: sandbox.demo.solver.UtilityReporterEntryPoint py4j.GatewayServer I believe the JVM uses UtilityReporterEntryPoint as the main class. Although you did not provide the code, I assume that this class has a main method and that it launches a GatewayServer with an instance of UtilityReporterEntryPoint as the entry point. Note that there is a whitespace between the colon and the class name so UtilityReporterEntryPoint is seen as the main class and not as being part of the classpath.
I am using OpenSSL in my c++ app, The problem is if I use exec("Open ssl command")
Then it will execute that particular command , but actually this command is repsonsive,I mean it further asks you "Are you sure you want to do this Y/N?"
I don't know how to cater this scenario.How can I use java or C++ to run a command line which is responsive,Any help would be appreciated.
Thanks
Easy enough in Java. Just:
Get the Process handle.
Read the Process' input stream for prompts written to stdout.
Respond to prompts by writing to the Process' output stream.
Here's a quick Groovy sample because it's even easier than Java:
def cmd = ... // the command you want to run
def process = cmd.execute()
def processStdout = new Scanner(process.inputStream)
def processStdin = process.outputStream
def outputLine = processStdout.nextLine()
if (outputLine == 'some prompt written to stdout') {
processStdin << 'your response\n'
}
If you can't follow the Groovy, I can expand it to Java.
Note that this sample doesn't handle the potentially important tasks of ensuring the stdout and stderr of the nested process are fully consumed to prevent blocking, nor does it handle ensuring the process exits cleanly.
Update: Here's that same thing in Java:
import java.io.OutputStream;
import java.util.Scanner;
public class SubprocessIO {
public static void main(String[] args) throws Exception {
String[] cmd = { ... your command as a series of strings ... };
Process process = Runtime.getRuntime().exec(cmd);
Scanner processStdout = new Scanner(process.getInputStream());
OutputStream processStdin = process.getOutputStream();
String outputLine = processStdout.nextLine();
if (outputLine.equals("some prompt written to stdout")) {
processStdin.write("your response\n".getBytes());
processStdin.flush();
}
}
}
I forgot to make a note on the first go-round that the \n in the response is crucial, assuming the app is expecting you to enter something and then press Enter. Also, you're probably better off using the line.separator system property
Basically you just need to make sure you enter all required information on the commandline, and use -batch to avoid further questions, for example:
openssl ca -days 3650 -out client.crt -in client.csr -config \path\to\configs -batch -passin pass:PASSWORD -key password
If this does not work for any specific openssl command, please specify it in your question which command you need to execute.
For openssl the answer by Wimmel is the right approach. Depending on your exact use case, you may want to prepare or construct a configuration file that contains recurring parameters and specify the varying parameters (and a pointer to the config file) on the command line. The -batch option, that is available at least with the common openssl commands for managing certificates will ensure that no interactivity occurs - if you have specified insufficient parameters the commands will fail.
For running the command and evaluating its results, you still need the corresponding functionality. In Java you use the ProcessBuilder and Process classes. In C++ there is no standard way to do this (the system() function is too limited for most uses), so you need to use platform-specific C functions (e.g. CreateProcess, posix_spawn or fork/exec) or find a suitable C++ library.
But for directly answering interactive questions programmatically these interfaces may be insufficient. Interactive dialog may be quite complicated. Typically this is not as simple as treating all input and output as a simple character stream. Details depend on the platform and program, but you may need something like expect (see http://en.wikipedia.org/wiki/Expect) to deal with this.
Update: of course the approach to invoke an external CLI for all of this is not necessarily the best and introduces a whole new set of extraneous side-issues. You may be better off using a suitable cryptographic API (for example BouncyCastle http://www.bouncycastle.org/)
I must be missing something here, but how do I call something like "cd /root/some/dir/" with Ganymed SSH API?
I created a Connection object
In the first session created, I called "cd /root/some/dir"
In the second session created, I called "ls ." or "./myApp"
That didnt work, because ganymed probably starts each session with its own directory
So do I need to perform both commands on the same session? something like:
session.getStdin().write("cd /root/somedir \n".getBytes());
session.getStdin().write("ls . ".getBytes());
Is that the correct way?? if so, why do we need Session.execCommand?
After doing some research, the only good solution I managed to find is calling the "cd" command within the same code as the "ls" command, like this
session.execCommand("cd /root/somedir ; ls .");
The semicolon will separate the two commands as in any bash code.
In this way, you can query the session's result [session.getExitStatus()] of both the cd and ls commands, which is much better then writing the two commands to session.getStdIn() (after writing to stdin, you kinda loose all the ability to check for exit status...)
Hope this will help the rest
Eyal
According to the Ganymed FAQ (http://www.ganymed.ethz.ch/ssh2/FAQ.html), you are not allowed to send more than one command per Session object you generate. This is how SSH-2 apparently wants you to handle it. Your two options are to either combine the two commands like
session.execCommand("cd /root/somedir ; ls .");
However this wont always work and it get very ugly if you have more than a couple commands. The other way to do this is to open an interactive shell session and write the commands to standard in. This could look something like this:
Session sess = conn.openSession();
sess.requestDumbPTY();
sess.startShell();
OutputStream os = sess.getStdin();
os.write("cd /root/somedir\n".getBytes());
os.write("ls -1\n".getBytes());
os.write("exit\n".getBytes());
InputStream stdout = new StreamGobbler(sess.getStdout());
BufferedReader br = new BufferedReader(new InputStreamReader(stdout));
//TODO
Note the use of the final exit command. Since this is being treated like a terminal window, if you do not exit from the program any loop you have reading the output of the server will never terminate because the server will be expecting more input
OK, I took a quick look on the Ganymed javadoc and although I did not try it myself I assume that you should use method execCommand() of session instead of writing into the STDIN. I am pretty sure that session is connected to remote shell and therefore handles the shell state including current directory, environment variables etc.
So, just do the following:
session.execCommand("cd /root/somedir \n".getBytes());
session.execCommand("ls . ".getBytes());
I hope this will work for you. Good luck.