Running stanford NER classifier in server mode - java

To make the NER classification faster I am trying to execute it in server mode listerning on port xxxx, so that it can give faster result when request is send.
Here is the original execution command without server that I am using.
java -mx1500m -cp $1/stanford-ner.jar edu.stanford.nlp.ie.crf.CRFClassifier -loadClassifier $1/classifiers/ner-eng-ie.crf-3-all2008-distsim.ser.gz -textFile $2
(this command is in .sh file and executed by python script. $1 is input file name)
This documentation explain how it can be run in server mode - Link
Here how the server get started:
java -mx400m -cp stanford-ner.jar edu.stanford.nlp.ie.NERServer -loadClassifier classifiers/ner-eng-ie.crf-3-all2008.ser.gz 1234
Now server is in listerning mode on port 1234.
How can I make call using input text file for this server?
I followed this tut : Link and executed this command:
java -cp stanford-ner.jar edu.stanford.nlp.ie.NERServer -port 1234 -client
But it just print this message:
Usage: NERServer [-loadFile file|-loadJarFile resource] portNumber
I am working on linux system.

To run the NER in server mode, you have to use the following command line :
java -mx400m -cp stanford-ner.jar edu.stanford.nlp.ie.NERServer -loadClassifier classifiers/ner-eng-ie.crf-3-all2008.ser.gz -port 1234
(-port is missing in the README file, it works well for me)

Related

Exception in thread "main" java.io.StreamCorruptedException: invalid stream header: 0BDAACED

I am trying to execute jenkins cli command from master.
ssh user#192.168.1.2 -C "/usr/bin/java -jar /home/user/slave.jar"
Getting following error:
<===[JENKINS REMOTING CAPACITY]===>Exception in thread "main" java.io.StreamCorruptedException: invalid stream header: 0BDAACED
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:808)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
at hudson.remoting.ChannelBuilder.makeTransport(ChannelBuilder.java:430)
at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:389)
at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:310)
at hudson.remoting.Launcher.main(Launcher.java:528)
at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:468)
at hudson.remoting.Launcher.run(Launcher.java:242)
at hudson.remoting.Launcher.main(Launcher.java:195)
ERROR: Unexpected error in launching an agent. This is probably a bug in Jenkins
hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected EOF
at hudson.remoting.Request.abort(Request.java:303)
at hudson.remoting.Channel.terminate(Channel.java:847)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:92)
at ......remote call to ubuntu-slave(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.Request.call(Request.java:172)
at hudson.remoting.Channel.call(Channel.java:780)
at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:508)
at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:381)
at hudson.slaves.CommandLauncher.launch(CommandLauncher.java:131)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:253)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unexpected EOF
at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
ERROR: Connection terminated
java.io.IOException: Unexpected EOF
at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
ERROR: Process terminated with exit code 1
Please tell me what could be the problem?
The short answer
Jenkins' slave.jar communicates with the Jenkins server through the slave's stdin/stdout. You rely on ssh to transfer stdin/stdout to/from the slave.
Something that you are not aware of tampers with the stdin before it reaches the slave, and the resulting communication protocol violation causes the exception that you see.
The longer explanation and a fix
I suppose that you have the ssh command from your question, which starts the slave, in a shell script on the jenkins server, and that this shell script also has other commands in it that precede this command. Because this is how it was in my case, inspired by the jenkins remoting documetation that suggests you could do things like copying the correct slave.jar to the slave.
So we are both using the "Launch agent via execution of command on the server" option to start the slave agent. In my case I do this to be able to use an ssh jump host to reach the slave, but this is not of relevance to this answer.
I will now give examples of non-working and working shell scripts to start a slave.jar on a remote node, and then come up with reasoning of what might cause the observed behaviour. I use /bin/bash as the shell, feel free to use others.
1) These versions both work if you already have slave.jar on the slave
#!/bin/bash
ssh user#host "java -jar /home/user/slave.jar"
.
#!/bin/bash
exec ssh user#host "java -jar /home/user/slave.jar"
Both versions have only a single command in them. Nothing else is in here that tampers with stdin or stdout. In the first version, the shell lingers around and forwards stdin/stdout to/from the ssh command. In the second version, the shell is replaced by the ssh process and directly inherits its stdin/stdout. Both versions work ok, and the additional shell process in the first version should not matter on any system.
2) This version copies slave.jar from some location on the server to the slave before executing it on the slave
#!/bin/bash
scp -q /some/location/slave.jar user#host:.
exec ssh user#host "java -jar /home/user/slave.jar"
Of course, this only works if you have /some/location/slave.jar on the server.
3) This version tries to do some additional cleanup on the slave before starting the slave agent.
#!/bin/bash
ssh user#host "rm -rf /home/user/tmp/jenkins"
exec ssh user#host "java -jar /home/user/slave.jar"
The /home/user/tmp/jenkins location on the slave is just an example.
This version fails after a 4 minute timeout. With the exact error message from the question. The failure is not caused by anything important missing from /home/user/tmp/jenkins, as you will see in the next example:
4) Different ways to make example 3 work
#!/bin/bash
ssh user#host "rm -rf /home/user/tmp/jenkins" </dev/null
exec ssh user#host "java -jar /home/user/slave.jar"
.
#!/bin/bash
exec ssh user#host "rm -rf /home/user/tmp/jenkins && java -jar /home/user/slave.jar"
In the first fix, we make sure stdin to communicate with the slave is not forwarded to the remote rm command. rm does not read its stdin, but apparently ssh does not know that and buffers some bytes from its own stdin and forwards them to the remote command just in case it is needed? This is fixed by forwarding /dev/null as stdin to the rm command instead of the stdin destined for the slave communication.
In the second fix, only one ssh command is used, the rm command again does not read anything from stdin, and the slave.jar receives the untampered stream.

Bash - Script to deploy and run jar files

im working on distributed systems and my program is complete, i have tested it and it runs fine over 10 machines, but every time i want to test the program i have to:
- Copy the file for each machine
- ssh to each machine and type "java -jar file"
To avoid that painfull process I made this
for i in {1..11}
do
if [ $i -ne 6 ];
then
sshpass -p "qwerty" scp myJar.jar user#l040101-ws$i.XXXX.XX.XX:someFolder;
sshpass -p "qwerty" ssh user#l040101-ws$i.XXXX.XX.XX "java -jar someFolder/myJar.jar &" &
fi
done
And for some reason it doesnt work like it should, the scp command executes as it should, but the other one doesnt.
The program should produce a folder with 2 logs inside and if i do it manually it does, so i guess is not permission problem, but not with the script.
The weired thing is if i run top, i can see the java processes running in each machine.
BTW: those 2 & is so it the script doesnt get stuck after running each jar
I recommend using SSH keys rather than expressing the password in a command (which gets logged and is visible to other users, not to mention its presence in your script). The github ssh key generation docs are pretty good for this (to add, append to the server's ~/.ssh/authorized_keys file).
Once you have generated a key on the client and added its pubkey to the server, you should be able to run:
for i in {1..11}
do
if [ $i -ne 6 ] # skip server six
then
cat myJar.jar |ssh user#l040101-ws$i.XXXX.XX.XX \
"cd someFolder; cat > myJar.jar; java -jar myJar.jar" &
fi
done
Note the single ampersand there, which is outside the quotes (so it is run on your client, not your server). You can't send an SSH session to the background because the parent would be killed.
I wrangled this into one line in order to minimize the number of connections (the first cat command dumps the file into standard output while the second cat command writes the standard input (the contents of myJar.jar) to the target location). I wasn't sure if I could just pipe it straight to java (cat myJar.jar |ssh user#host "cd someFolder; java -jar -"), so I left that alone.
I'm assuming you don't have to run the .jar from the parent of someFolder. It seems simpler to actually be in the target directory. If there's a chance the target directory does not exist, add mkdir -p someFolder; before the cd command. The -p will ensure nothing happens if the directory already exists. If you do have to run it from the parent, remove the cd command and replace "myJar.jar" with "someFolder/myJar.jar"

ArrayIndexOutOfBounds Exception on executing linux sh file

I have a program in java which takes 0'th aargument as file location like
File f = new File(args[0]);
so when i execute it using a windows batch(.bat) file it works correctly .
but when i execute the same using a linux shell file(.sh) in linux i get ArrayIndexOutOfBoundsException.
WINDOWS BATCH FILE :
#echo off
for /f %%i in ("%0") do set scriptpath=%%~dpi
set cp=%scriptpath%/../lib/*.jar;
java -classpath %cp% com.synchronizer.main.MYSynchronizer %scriptpath% "%1" "%2"
LINUX SH FILE:
export JAVA_HOME=/usr/local/java
PATH=/usr/local/java/bin:${PATH}
THE_CLASSPATH=
for i in `ls ../lib/*.jar`
do
THE_CLASSPATH=${THE_CLASSPATH}:${i}
done
java -cp ".:${THE_CLASSPATH}" \
com.synchronizer.main.MYSynchronizer
please help!
It looks like a problem in script (no arguments are passed to the Java program).
You can consider to debug the script like this: debugging scripts
Hope this helps
Your shell script is not passing any parameters:
java -cp ".:${THE_CLASSPATH}" com.synchronizer.main.MYSynchronizer
Try:
java -cp ".:${THE_CLASSPATH}" com.synchronizer.main.MYSynchronizer "$1" "$2"
As stated above, your Linux shell script is not sending any arguments to the Java program that you are trying to start.
And, adding to that, you are not showing us how you run the Linux shell script. If no argument is given on the command line when you start the shell script, no arguments can be passed to your Java application from the shell script.
If you want to see the actual command that is going to be run by your shell script, you can always put "echo" in front of a line and see what all variables are expanded to. This is a simple way to debug shell scripts.

PHP exec java on IIS 7

I am trying to exec a java jar file using php exec();
The command is
"C:\Program Files\Java\jdk1.5.0_15\bin\java.exe" -jar "C:\batik\batik-rasterizer.jar" -m image/png -d "C:/path/to/file/filename.png" -w 800 "C:/path/to/file/filename.svg"
When I run this command on the server using CMD.exe or Power Shell works fine but when I run it with PHP using exec() or system() or shell_exec() it will not execute returning a blank page
OK I got it figured out,
First the java jdk shouldn't be installed in the programs folder for some reason iis_iusrs doesn't have the permission to execute any files there.
Secondly the code should look like this
exec("C:\\Java\\jre6\\bin\\java.exe -jar C:\\batik\\batik-rasterizer.jar -m image/png -d "C:\\path\\to\\file\\filename.png" -w 800 "C:\\path\\to\\file\\filename.svg");

How to change/assign process name of java .jar

I'm running Minecraft under Linux, which involves running an executable .jar file. This means it shows up as "java" under ps, rather than "minecraft". I would like to assign it the process name "minecraft".
Looking around, I found the following tip for assigning a process name via bash:
how to change the name of a Java application process?
exec -a goodname java ...
I usually run with:
java -cp ~/Games/Minecraft/Minecraft.jar net.minecraft.LauncherFrame
So tried make a bash script:
#!/bin/bash
exec -a minecraft java -cp ~/Games/Minecraft/Minecraft.jar net.minecraft.LauncherFrame
But when I run this, it still shows up as "java" under the ps command.
What am I doing wrong?
It works for me. I haven't tested with java, but I tested with sleep:
victor#vz:~$ exec -a minecraft sleep 1m &
[1] 3858
victor#vz:~$ ps x | grep mine
3858 pts/2 S 0:00 minecraft 1m
3860 pts/2 S+ 0:00 grep --color=auto mine
victor#vz:~$
However, this seems to be merely a cosmetic change as far as I can tell by the documentation:
victor#vz:~$ help exec exec: exec
[-cl] [-a name] [command [arguments
...]] [redirection ...]
Replace the shell with the given command.
Execute COMMAND, replacing this shell with the specified program.
ARGUMENTS become the arguments to COMMAND. If COMMAND is not specified,
any redirections take effect in the current shell.
Options:
-a name pass NAME as the zeroth argument to COMMAND
In reference to OP's comment to this answer: I just tested it on a remote machine with java as well:
victorz#exa:~$ javac test.java # spits out an Administrator.class file among others
victorz#exa:~$ exec -a minecraft java Administrator &
[1] 13142
victorz#exa:~$ ps x | grep mine
13142 pts/1 Sl 0:00 minecraft Administrator
13161 pts/1 S+ 0:00 grep --color=auto mine
victorz#exa:~$
Maybe you are not using the x switch to ps? I get no match unless I use the x switch.

Categories