We Keep Coding

How to access Java dependency on command line?

I have a moderately old, small Java application which has an option to read and export PDF files using the Apache PDFBox library (hereunder, "pdfbox-app.jar"). All the files, including this resource, are stored in a single flat folder.
This works fine when called from a JAR file:
D:\Prog\!GitHub\Arena>java -jar Athena.jar NPCGenerator -p
OED NPC Generator
-----------------
Writing Gwenllian-ElfFtr1Wiz1.pdf
It similarly works fine when run from my IDE (jGrasp).
But it fails when called from the command line, outside of its JAR:
D:\Prog\!GitHub\Arena>java NPCGenerator -p
OED NPC Generator
-----------------
Writing Eoin-HalflingFtr1.pdf
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/pdfbox/pdmodel/PDDoc
ument
at CharacterPDF.writePDF(CharacterPDF.java:49)
at NPCGenerator.printToPDF(NPCGenerator.java:294)
at NPCGenerator.makeAllNPCs(NPCGenerator.java:270)
at NPCGenerator.main(NPCGenerator.java:308)
Caused by: java.lang.ClassNotFoundException: org.apache.pdfbox.pdmodel.PDDocument
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.j
ava:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoader
s.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
... 4 more
What should I be doing to run this on the command line outside of its own JAR?

You should put the pdfbox jar, and any other dependencies, on the classpath:
java -classpath .;pdfbox-app.jar NPCGenerator -p
Without that, Java doesn't know where to look for org/apache/pdfbox classes. It looks for .class files relative to the default classpath (which is just ., the current directory), but does not look inside jars.

Apache Storm - Could not find or load main class org.apache.storm.starter.ExclamationTopology

I'm really new to maven and storm so I'm trying to follow the instructions in https://github.com/apache/storm/tree/master/examples/storm-starter. My current path is /home/luc/theTest/storm/examples/storm-starter. Inside the target folder there is a storm-starter-2.0.0-SNAPSHOT.jar file. I'm getting stuck when running
storm jar target/storm-starter-*.jar org.apache.storm.starter.ExclamationTopology -local
I get this error
ionTopology -local
Running: /usr/lib/jvm/java-8-openjdk-amd64/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/home/luc/stormTest/apache-storm-1.1.1 -Dstorm.log.dir=/home/luc/stormTest/apache-storm-1.1.1/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/luc/stormTest/apache-storm-1.1.1/lib/servlet-api-2.5.jar:/home/luc/stormTest/apache-storm-1.1.1/lib/slf4j-api-1.7.21.jar:/home/luc/stormTest/apache-storm-1.1.1/lib/objenesis-2.1.jar:/home/luc/stormTest/apache-storm-1.1.1/lib/kryo-3.0.3.jar:/home/luc/stormTest/apache-storm-1.1.1/lib/log4j-core-2.8.2.jar:/home/luc/stormTest/apache-storm-1.1.1/lib/log4j-over-slf4j-1.6.6.jar:/home/luc/stormTest/apache-storm-1.1.1/lib/storm-core-1.1.1.jar:/home/luc/stormTest/apache-storm-1.1.1/lib/log4j-slf4j-impl-2.8.2.jar:/home/luc/stormTest/apache-storm-1.1.1/lib/minlog-1.3.0.jar:/home/luc/stormTest/apache-storm-1.1.1/lib/log4j-api-2.8.2.jar:/home/luc/stormTest/apache-storm-1.1.1/lib/clojure-1.7.0.jar:/home/luc/stormTest/apache-storm-1.1.1/lib/ring-cors-0.1.5.jar:/home/luc/stormTest/apache-storm-1.1.1/lib/asm-5.0.3.jar:/home/luc/stormTest/apache-storm-1.1.1/lib/reflectasm-1.10.1.jar:/home/luc/stormTest/apache-storm-1.1.1/lib/disruptor-3.3.2.jar:/home/luc/stormTest/apache-storm-1.1.1/lib/storm-rename-hack-1.1.1.jar:target/storm-starter-2.0.0-SNAPSHOT.jar:/home/luc/stormTest/apache-storm-1.1.1/conf:/home/luc/stormTest/apache-storm-1.1.1/bin -Dstorm.jar=target/storm-starter-2.0.0-SNAPSHOT.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} org.apache.storm.starter.ExclamationTopology -local
Error: Could not find or load main class org.apache.storm.starter.ExclamationTopology
Am I doing something wrong? I'm also a bit confused on whether I have to run the nimbus and supervisor first. I tried with and without them and neither worked anyways. Been searching the web but nothing works. Not sure what else to try.

This is usually caused by inconsistent versions of storm-client and storm-starter. Try following these steps to get the example working.
download the latest release from http://storm.apache.org/downloads.html
in this example, we will use version 1.1.1
extract this to a folder, lets call it ${STORM_HOME}
cd into ${STORM_HOME}/examples/storm-starter
execute mvn package -DskipTests=true
this should build the storm-starter jar in target folder
${STORM_HOME}/examples/storm-starter/target/storm-starter-1.1.1.jar
run the example from ${STORM_HOME} directory:
./bin/storm jar examples/storm-starter/target/storm-starter-1.1.1.jar org.apache.storm.starter.ExclamationTopology
don't add the -local flag since it seems like the ExclamationTopology is only deployed in a LocalCluster if no args are passed. You can check the source code here: ${STORM_HOME}/examples/storm-starter/src/jvm/org/apache/storm/starter/ExclamationTopology.java

Can't create a Hadoop sequence file on a local file system

I found this example of how to write to a local file system, but it throws this exception:
Exception in thread "main" java.io.IOException: (null) entry in command string: null chmod 0644 C:\temp\test.seq
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1168)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
Running this on a Windows 10 box. I even tried using the msys git bash shell thinking maybe that would help the JVM simulate a chmod operation. Didn't change anything. Any suggestions on how to do this on Windows?

I too faced this error and it was resolved after following the steps. (Note : I am using Spark 2.0.2 and Hadoop 2.7)
Verify whether you are getting "java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.". You check it by running "spark-shell" command.
I got the above mentioned error. It occurred because I didn't add "HADOOP_HOME" in environment var. After adding the "HADOOP_HOME", in my case same as "SPARK_HOME", the issue was resolved.

Running a Hadoop program using only jars on Windows requires a few steps beyond just referencing the jars.
Credit to Professor Lu at University of Helsinki for posting a Hadoop on Windows guide for his students.
Here is a rundown of steps I had to take using Windows 10 and Hadoop 2.7.3:
Download and extract Hadoop binaries to somewhere like C:\hadoop-2.7.3.
Download patch files from https://github.com/srccodes/hadoop-common-2.2.0-bin/archive/master.zip and extract them to your %HADOOP_HOME%\bin directory.
Set a HADOOP_HOME environment variable. For example, C:\hadoop-2.7.3.
Download the Hadoop source code, copy hadoop-common-project\hadoop-common\src\main\java\org\apache\hadoop\io\nativeio\NativeIO.java to your project, and modify line 609 from
return access0(path, desiredAccess.accessRight());
to
return true;

One of the solutions is as follows.
In the Project Structure (Intelij), under SDK's ensure there is no other version of Hadoop referenced. In my case - I was running Spark earlier and it was referring Hadoop JAR's and this was causing access issues. Once I removed them and ran the MR job it ran fine.

cobertura-instrument.sh fails to instrument jar file with java.lang.NoClassDefFoundError: net.sourceforge.cobertura.instrument.InstrumentMain

I'm trying to instrument jar file (from Spacewalk project) so I can measure code coverage of mine testing, but it is failing:
# /opt/cobertura-2.1.1/cobertura-instrument.sh --datafile /tmp/out /usr/share/rhn/lib/rhn.jar
Exception in thread "main" java.lang.NoClassDefFoundError: net.sourceforge.cobertura.instrument.InstrumentMain
Caused by: java.lang.ClassNotFoundException: net.sourceforge.cobertura.instrument.InstrumentMain
at java.net.URLClassLoader.findClass(URLClassLoader.java:432)
at java.lang.ClassLoader.loadClass(ClassLoader.java:676)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:358)
at java.lang.ClassLoader.loadClass(ClassLoader.java:642)
Could not find the main class: net.sourceforge.cobertura.instrument.InstrumentMain. Program will exit.
I have tried to provide one random class (in the ideal state I want to instrument all of them) from that jar as well, but with same result:
# jar tf rhn.jar | tail
org/cobbler/CobblerConnection.class
[...]
# /opt/cobertura-2.1.1/cobertura-instrument.sh --datafile /tmp/out /usr/share/rhn/lib/rhn.jar org.cobbler.CobblerConnection
I'm pretty sure I'm just missing something what it is trying to tell me.
I'm using cobertura-2.1.1 downloaded from SourceForge and extracted into /opt, running on Red Hat Enterprise Linux 6.

OK, this was simple:
# dos2unix /opt/cobertura-2.1.1/cobertura-instrument.sh
also it is missing bash shebang (#!/bin/bash), so you might need to add it to the beginning of the file (I do not know why it worked for me even without that).

Amazon Web Services Java classpath

I am trying to run an application that reads and writes to the amazon dynamo DB. I downloaded the Eclipse toolkit and AWS SDK and if I run the application from my local PC it works perfectly. Next, I exported it to a jar file and uploaded it to my EC2 instance. However, when I run it there I get an error.
/home/apps/java/database/bin$ java -jar myJar.jar
Exception in thread "main" java.lang.NoClassDefFoundError: com/amazonaws/auth/AW SCredentials
Caused by: java.lang.ClassNotFoundException: com.amazonaws.auth.AWSCredentials
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: DynamoDB. Program will exit.
I assume it has to do with the classpath, but in /home/ubuntu/.bashrc I have set it as such:
CLASSPATH="./:/home/apps/java/database/bin/*:/home/apps/java/database/bin/aws-java-sdk-1.3.12.jar"
export CLASSPATH
/home/apps/java/database/bin contains all the .jar files that are in the AWS SDK:
aspectjrt.jar
aspectjweaver.jar
aws-java-sdk-1.3.12.jar
aws-java-sdk-1.3.12-javadoc.jar
aws-java-sdk-1.3.12-sources.jar
aws-java-sdk-flow-build-tools-1.3.12.jar
commons-codec-1.3.jar
commons-logging-1.1.1.jar
freemarker-2.3.18.jar
httpclient-4.1.1.jar
httpcore-4.1.jar
jackson-core-asl-1.8.7.jar
jackson-mapper-asl-1.8.7.jar
mail-1.4.3.jar
myJar.jar
spring-beans-3.0.7.jar
spring-context-3.0.7.jar
spring-core-3.0.7.jar
stax-1.2.0.jar
stax-api-1.0.1.jar
What am I missing?? I have been looking at this for a day and a half. Thank you in advance!!

The classpath entry of "/directory/*" may be messing things up. Classpaths are separated with colons, but asterisk expansion gives spaces. Try this little shell script to start it.
#!/bin/sh
JAVA_OPTS="-Xms256M -Xmx4G"
CP=`find /home/apps/java/database/bin/*jar -exec echo -n "{}:" \;`
java -cp ${CP%?} -jar yourjar.jar
A couple of notes:
the JAVA_OPTS is only there as a reminder that you may need more memory than the default.
the crazy syntax for CP on the final line strips the last character, since the "find" line is leaving a colon on the end.
You may want to include your jar and launch the correct class if it isn't an executable jar.
Hope this helps!

This list of libs works for me
aws-java-sdk-1.11.285-javadoc.jar
aws-java-sdk-1.11.285-sources.jar
aws-java-sdk-1.11.285.jar
aws-java-sdk.jar
aspectjrt-1.8.2.jar
aspectjweaver.jar
aws-swf-build-tools-1.1.jar
commons-codec-1.9.jar
commons-logging-1.1.3.jar
freemarker-2.3.9.jar
httpclient-4.5.2.jar
httpcore-4.4.4.jar
ion-java-1.0.2.jar
jackson-annotations-2.6.0.jarÅ
jackson-core-2.6.7.jar
jackson-databind-2.6.7.1.jar
jackson-dataformat-cbor-2.6.7.jar
javax.mail-api-1.4.6.jar
jmespath-java-1.11.285.jar
joda-time-2.8.1.jar
netty-buffer-4.1.17.Final.jar
netty-codec-4.1.17.Final.jar
netty-codec-http-4.1.17.Final.jar
netty-common-4.1.17.Final.jar
netty-handler-4.1.17.Final.jar
netty-resolver-4.1.17.Final.jar
netty-transport-4.1.17.Final.jar
spring-beans-3.0.7.RELEASE.jar
spring-context-3.0.7.RELEASE.jar
spring-core-3.0.7.RELEASE.jar
spring-test-3.0.7.RELEASE.jar