I am trying to run a java command line operation from Ruby through a series of system commands. The first command works fine, I never receive any success from the second (maui java command). Is there something that I am doing wrong?
Dir.chdir('/var/lib/maui') do
system 'cat %s/*.txt > %s/topics.txt' % [text_path, identifier_path]
system 'java maui.main.MauiTopicExtractor -l %s -m go_model' % [identifier_path]
end
I have also tried calling a bash script that processes the java commands:
output = %x[/var/lib/maui/process.sh #{identifier_path}]
However, the same result occurs-- the files are concatenated, but never reach the Maui.
When I capture the stdoutput from the maui command, I get the following (even though I have all the environment vars setup as needed and works when I run the command outside of ruby):
Exception in thread "main" java.lang.NoClassDefFoundError: maui/main/MauiTopicExtractor
Caused by: java.lang.ClassNotFoundException: maui.main.MauiTopicExtractor
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: maui.main.MauiTopicExtractor. Program will exit
Any help appreciated.
Based on your comment about class not found I think you need to invoke like this:
system 'java -cp /path/to/maui.jar maui.main.MauiTopicExtractor -l %s -m go_model' % [identifier_path]
if it works from the command line doing simply:
java maui.main.MauiTopicExtractor -l /path/to/file -m go_model
then can you post the output of (from the shell):
echo $CLASSPATH
and from ruby:
system "echo $CLASSPATH"
Related
I'm trying to run some command on a Windows agent with cmd using the groovy String.execute() in a Jenkins pipeline.
I know that there exists the bat script which I can use and I intentionally don't want to use it in this specific case since it bloats the logs and the Blue Ocean plugin has a limit of number of steps it can shows for any stage.
Basically what I have is basically some cleanup function that I call very often which does a lot of checks and runs multiple commands (at the moment with sh and bat in addition to isUnix() and similar). The result is usually a very bloated logs of steps like sh and bat which makes the logs larger and more difficult to analyze.
I decided to use some native way of running shell commands "silently" and only print its stdout and stderr in case the command fails.
I have written such a function that executes the command
# NonCPS
def exec(cmd) {
def isUnix = Jenkins.instance.getNodes().find { it.getNodeName() == env.NODE_NAME }.toComputer().isUnix()
cmd = (isUnix ? ["/bin/bash", "-c"] : ["cmd.exe", "/c"]) + [cmd]
def proc = cmd.execute()
proc.waitFor()
def result = [out: proc.in.text.trim(), err: proc.err.text.trim(), exitCode: proc.exitValue()]
return result
}
The above function works perfectly for linux agents but not on Windows. I always get the following error:
java.io.IOException: error=2, No such file or directory
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
at java.base/java.lang.Runtime.exec(Runtime.java:591)
at java.base/java.lang.Runtime.exec(Runtime.java:415)
at java.base/java.lang.Runtime.exec(Runtime.java:312)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:47)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
Caused: java.io.IOException: Cannot run program "cmd.exe": error=2, No such file or directory
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
at java.base/java.lang.Runtime.exec(Runtime.java:591)
at java.base/java.lang.Runtime.exec(Runtime.java:415)
at java.base/java.lang.Runtime.exec(Runtime.java:312)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Or mainly the following line Cannot run program "cmd.exe": error=2, No such file or directory.
I tried replacing "cmd.exe" with the full path "C:\Windows\System32\cmd.exe" or "C:\Windows\SysWOW64\cmd.exe", replace cmd.exe with ["start", "cmd.exe", "/C"], using Runtime.getRuntime().exec() and many others and I cannot make it work.
If I do "git version".execute(), that works fine but I cannot run some more complex commands like "echo hi && git version".execute() which will print "hi && git version" instead of "hi" and the real git version. I thought maybe the PATH environment variable is not set properly so I ran the following command "set".execute() which resulted in the same error above ("set" no such file or directory).
I have been trying various combinations of this for the past 2 days and that is not going anywhere.
Any help is much appreciated. Thanks in advance.
Update:
Found the reason why it is failing on Windows but not on Linux nodes with Jenkins and hit another "harder" wall here.
The problem is that String.exucute() method behaves internally like Runtime.getRuntime().exec() and this uses the runtime environment from the master node even if encapsulated within the node() directive. This basically means that when it was running fine on linux, it was actually running all the time on the master node which is also linux. The reason why it cannot run cmd or start or any similar is because those commands (obviously) do not exist on linux.
I have noticed an issue that crops up repeatedly: using an env var for the classpath on the java command line within a shell script does not work.
First off, let us see what does work: both using hard-coded classpath in a script as follows: (note: the "classpath is" statement is printed within the java program itself)
steve#mithril:/shared$ java -classpath .:/shared/mysql-connector-java-5.1.25-bin.jar DbPing com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/mysql user password
classpath is .:/shared/mysql-connector-java-5.1.25-bin.jar
Attempting connection to com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/mysql password
Connecting to user using URL=jdbc:mysql://localhost:3306/mysql
Successfully connected.
and also using env var directly within the shell:
steve#mithril:/shared$ export CP=.:/shared/mysql-connector-java-5.1.25-bin.jar
steve#mithril:/shared$ java -classpath $CP DbPing com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/mysql user password
classpath is .:/shared/mysql-connector-java-5.1.25-bin.jar
Attempting connection to com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/mysql password
Connecting to user using URL=jdbc:mysql://localhost:3306/mysql
Successfully connected.
What does *not * work: running the same commands as shown above within a shell script:
steve#mithril:/shared$ cat dbping.mysql
CP=.:/shared/mysql-connector-java-5.1.25-bin.jar
echo $CP
java -classpath $CP DbPing com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/mysql user password
#java -classpath .:/shared/mysql-connector-java-5.1.25-bin.jar DbPing com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/mysql user password
steve#mithril:/shared$ ./dbping.mysql
.:/shared/mysql-connector-java-5.1.25-bin.jar
classpath is .:/shared/mysql-connector-java-5.1.25-bin.jar
Attempting connection to com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/mysql password
Could not load db driver com.mysql.jdbc.Driver
java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at DbPing.getConnection(DbPing.java:34)
at DbPing.main(DbPing.java:22)
Exception in thread "main" java.sql.SQLException: com.mysql.jdbc.Driver
at DbPing.getConnection(DbPing.java:41)
at DbPing.main(DbPing.java:22)
Follow-up: The script had windows style newlines in it. Apparently the \r clobbered the internal environment variable. found this using
od -cx
. I am going to give credit to stephen c anyways, since his prodding got me on the right track to finding the solution
The symptoms you describe are rather puzzling. I can't see the problem (the script looks correct), but I've got an idea on how to start tracking it down.
Get rid of the commented out lines from the script.
Add a #!/bin/sh line to the start of the script to make sure that you are actually using the right shell to execute it. (It is always a good idea to do this ... even if you think that you will get the right shell by default. That might change, depending on the platform.)
To figure out what the shell is doing, add set -vx as after the #!/bin/sh line.
The "-v" says echo each script line read, and the "-x" says echo the actual command lines that are executed. This will tell you exactly what commands are being run ... so that you can figure out what the command arguments really are.
I have a simple BASH script that wraps a java program with the intention of restarting it if that application crashes:
STOP=0
while [ "$STOP" -eq 0 ]
do
echo "Starting"
exec java com.site.app.Worker
echo "Crashed"
sleep 3
done
However if the Java process exits it also quits the bash script so the process is never started again.
E.g. (pointing at a fake class):
$ ./RestartApp.ksh
Starting
Exception in thread "main" java.lang.NoClassDefFoundError: com/site/app/Worker
Caused by: java.lang.ClassNotFoundException: com.site.app.Worker
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: com.site.app.Worker. Program will exit.
$
Is there a way I can catch the errors (but still display them) to allow the script to continue running?
Remove the exec. That's completely replacing the current process (your shell) with the Java VM.
Just remove that and it should work fine.
As Mat said, what exec does is to replace the current shell process by the Java process. It it fails, there is no-one waiting for it to relaunch it. exec can be a very useful and professional tool to use, but it is rather advanced.
An example of a right use for it would be a script that sets variables or priorities in the current shell, and then exec's the process you are wrapping.
The variable "STOP" does not seem to be used. I would simply go for:
while ! java com.site.app.Worker
do
echo Failed: Sleeping and restarting >&2
sleep 3
done
I am trying to run an application that reads and writes to the amazon dynamo DB. I downloaded the Eclipse toolkit and AWS SDK and if I run the application from my local PC it works perfectly. Next, I exported it to a jar file and uploaded it to my EC2 instance. However, when I run it there I get an error.
/home/apps/java/database/bin$ java -jar myJar.jar
Exception in thread "main" java.lang.NoClassDefFoundError: com/amazonaws/auth/AW SCredentials
Caused by: java.lang.ClassNotFoundException: com.amazonaws.auth.AWSCredentials
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: DynamoDB. Program will exit.
I assume it has to do with the classpath, but in /home/ubuntu/.bashrc I have set it as such:
CLASSPATH="./:/home/apps/java/database/bin/*:/home/apps/java/database/bin/aws-java-sdk-1.3.12.jar"
export CLASSPATH
/home/apps/java/database/bin contains all the .jar files that are in the AWS SDK:
aspectjrt.jar
aspectjweaver.jar
aws-java-sdk-1.3.12.jar
aws-java-sdk-1.3.12-javadoc.jar
aws-java-sdk-1.3.12-sources.jar
aws-java-sdk-flow-build-tools-1.3.12.jar
commons-codec-1.3.jar
commons-logging-1.1.1.jar
freemarker-2.3.18.jar
httpclient-4.1.1.jar
httpcore-4.1.jar
jackson-core-asl-1.8.7.jar
jackson-mapper-asl-1.8.7.jar
mail-1.4.3.jar
myJar.jar
spring-beans-3.0.7.jar
spring-context-3.0.7.jar
spring-core-3.0.7.jar
stax-1.2.0.jar
stax-api-1.0.1.jar
What am I missing?? I have been looking at this for a day and a half. Thank you in advance!!
The classpath entry of "/directory/*" may be messing things up. Classpaths are separated with colons, but asterisk expansion gives spaces. Try this little shell script to start it.
#!/bin/sh
JAVA_OPTS="-Xms256M -Xmx4G"
CP=`find /home/apps/java/database/bin/*jar -exec echo -n "{}:" \;`
java -cp ${CP%?} -jar yourjar.jar
A couple of notes:
the JAVA_OPTS is only there as a reminder that you may need more memory than the default.
the crazy syntax for CP on the final line strips the last character, since the "find" line is leaving a colon on the end.
You may want to include your jar and launch the correct class if it isn't an executable jar.
Hope this helps!
This list of libs works for me
aws-java-sdk-1.11.285-javadoc.jar
aws-java-sdk-1.11.285-sources.jar
aws-java-sdk-1.11.285.jar
aws-java-sdk.jar
aspectjrt-1.8.2.jar
aspectjweaver.jar
aws-swf-build-tools-1.1.jar
commons-codec-1.9.jar
commons-logging-1.1.3.jar
freemarker-2.3.9.jar
httpclient-4.5.2.jar
httpcore-4.4.4.jar
ion-java-1.0.2.jar
jackson-annotations-2.6.0.jarÅ
jackson-core-2.6.7.jar
jackson-databind-2.6.7.1.jar
jackson-dataformat-cbor-2.6.7.jar
javax.mail-api-1.4.6.jar
jmespath-java-1.11.285.jar
joda-time-2.8.1.jar
netty-buffer-4.1.17.Final.jar
netty-codec-4.1.17.Final.jar
netty-codec-http-4.1.17.Final.jar
netty-common-4.1.17.Final.jar
netty-handler-4.1.17.Final.jar
netty-resolver-4.1.17.Final.jar
netty-transport-4.1.17.Final.jar
spring-beans-3.0.7.RELEASE.jar
spring-context-3.0.7.RELEASE.jar
spring-core-3.0.7.RELEASE.jar
spring-test-3.0.7.RELEASE.jar
I found a couple other questions with my problem, but the solutions did not work. I have a web page on my local server that you can input text for a whole java class. When submit is clicked I want the output to be displayed. Using shell_exec I can compile the java file. I get nothing when I try to run it, though.
shell_exec("javac /folder/Test.java"); // works
echo shell_exec("/usr/bin/java folder.Test"); // nothing is returned
I decided to use the full path to java when I read that it solved the problem for someone else. When I run:
/usr/bin/java folder.Test
from the command line it works.
The Test.java file looks like this:
package folder;
public class Test{
public static void main(String[] args){
System.out.println("testing");
}
}
EDIT: I could not get w0rldart's to work. I can actually see something when I try Mathieu's. I tried adding the -classpath option (I am not sure if I am using it correctly)
echo shell_exec("java -classpath ./folder/ Test 2>&1");
I get the same error as without -classpath:
Exception in thread "main" java.lang.NoClassDefFoundError: Test Caused by:
java.lang.ClassNotFoundException: Test at
java.net.URLClassLoader$1.run(URLClassLoader.java:217) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:205) at
java.lang.ClassLoader.loadClass(ClassLoader.java:321) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) at
java.lang.ClassLoader.loadClass(ClassLoader.java:266) Could not find the main class:
Test. Program will exit.
IT WORKS:
Just had to remove the period from
echo shell_exec("java -classpath ./folder/ Test 2>&1"); //error
echo shell_exec("java -classpath /folder/ Test 2>&1"); //works
I'm very uncertain this will make a difference. But just to be sure, have you tried adding 2>&1 after your command?
echo shell_exec("/usr/bin/java folder.Test 2>&1");
I don't think system.out.println is using stderr to display its output, but it might be worth trying anyway...
If you want to use Test class from folder package then Test.class needs to be saved in a folder named folder. Then you need to have folder's parent directory on the CLASSPATH.
So if your configuration is the following: /rootdir/foo/folder/Test.class then you can start Test like this: java -classpath /rootdir/foo folder.Test.
I think this behavior is the same if you use java from php's shell_exec.