We Keep Coding

How to run a Java/Maven application with external libraries in Heroku?

I've been building a Java application, which runs as expected when executing it locally. Now I'm trying the next step, to move it into an external platform. I chose Heroku because of their free plans for a starter, and also because it lets me have a small PostGreSQL database, which I use along with the application.
Due to the way Heroku works, I was forced to integrate Java with Maven, which I know little to none about. But I was still able to work my way there until I was able to compile and deploy the application (source code is in GitHub), albeit with errors.
For once, I'm using a couple external libraries:
jsoup-1.13.1.jar – Jsoup, for easier parsing of HTML files
mail.jar – Java mail, for emailing reports
Whenever the application is deployed in Heroku, there's an Exception in thread "main" java.lang.NoClassDefFoundError: org/jsoup/Jsoup, ultimately triggered by the Jsoup.connect invocation. From what I understand, Maven only add the references to the program but does not include the actual libraries there for runtime, which I think may be the cause of the issue.
I've tried modifying my pom.xml file by adding these dependencies manually:
<dependencies>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.2.18</version>
</dependency>
<dependency>
<groupId>com.sun.mail</groupId>
<artifactId>javax.mail</artifactId>
<version>1.6.2</version>
</dependency>
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.13.1</version>
</dependency>
</dependencies>
And also these plugins in the build section:
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<finalName>checker</finalName>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.4</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
</execution>
</executions>
</plugin>
What am I doing wrong? I would be very grateful for hints to help me get my application running, at least resolving the reason why Jsoup isn't working at all in Heroku. Thanks in advance!

neo4j procedure depending on signed jar (mapdb)

I have created a neo4j user-defined procedure. It also compiles and works in neo4j.
Recently though, I added a dependency to my procedure plugin that prevents neo4j from starting when I try to run the newly built jar. Concretely, I receive following exception at the bottom of the stack trace:
Caused by: java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
The only thing I changed is to add MapDB to my dependencies. So I suspect that it depends on some signed artifact. As it turns out neo4j plugins are "uber-jars" (i.e., shaded jars) which doesn't work very well with signed dependencies.
I figured I could try to exclude MapDB from the shading by changing the scope to provided and additionally adding the mapdb jar to plugins folder of neo4j. So the plugins folder of neo4j now includes both mapdb.jar and myprocedure.jar.
This doesn't seem to work though: neo4j starts, but when calling my procedure I receive a ClassNotFound Exception.
Any ideas on how I can solve this dilemma? I really depend on something like MapDB as my graph is very large and keeping everything in my procedure in-memory regularly leads to memory exceptions.
Many thanks in advance!
The important part of my pom.xml should it help (I started off with the procedure template so it still looks quite similar):
<dependencies>
<!-- other dependencies that don't make a difference ... -->
<dependency>
<groupId>org.mapdb</groupId>
<artifactId>mapdb</artifactId>
<version>3.0.7</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j</artifactId>
<version>${neo4j.version}</version>
<scope>provided</scope>
</dependency>
<!-- Test Dependencies -->
<dependency>
<groupId>org.neo4j.test</groupId>
<artifactId>neo4j-harness</artifactId>
<version>${neo4j.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.4.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<!-- Neo4j Procedures require Java 8 -->
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<!-- This generates a jar-file with our procedure code,
plus any dependencies marked as `compile` scope.
This should then be deployed in the `plugins` directory
of each Neo4j instance in your deployment.
After a restart, the procedure is available for calling. -->
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

So, one solution that worked was to use the maven-assembly-plugin instead of the maven-shade-plugin. The assembly plugin creates a standalone jar with all dependencies without re-packaging all dependencies.
Of course, I had to remove the <scope>provided</scope> for mapdb so it is also included in the assembled jar.
The rest is just replacing the shade plugin with following snippet (thanks to this question here):
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
Afterwards I received another exception which I thought was due to a wrong signature:
Caused by: java.lang.VerifyError ...
Turns out this was just because I had another procedure in neo4j that uses the same dependencies as mapdb. Quick fix was to remove that procedure library.
Of course, there are also other solutions, including removing the signature, or re-signing it. However, I explicitly did not want to remove any signatures.

I'm not aware that neo4j is checking any jar signatures by default. I suspect the repackaging of mapdb artifact being the culprit.
To validate that, remove the shade or assembly plugin in and build a jar that solely contains your code. Copy that jar and the mapdb.jar into the plugin folder and observe logs during a restart.

Maven: How do I get .so libraries bundled in package

I have a third party library coming with .jar and .so file.
I configured pom.xml as following:
<dependency>
<groupId>com.abc.def</groupId>
<artifactId>sdc</artifactId>
<version>1.1</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.abc.def</groupId>
<artifactId>sdc</artifactId>
<version>3</version>
<scope>compile</scope>
<type>so</type>
</dependency>
With this configure, I successfully tested through Intellij and apk file under target contains structure like lib/armeabi/sdc.so
However, after I do mvn clean package, the apk file generated did not contain sdc.so file, and after installing apk file on android device, lib folder is empty.
Searching through the internet, and did not find answer.
Btw, I do add <nativeLibrariesDirectory>${project.basedir}/libs</nativeLibrariesDirectory> into pluginManagement as mentioned Native libraries (.so files) are not added to an android project, but does not help.
Updates:
If someone is facing same problem as I am that SO file did not copy over, please try manually copy the so file over as following:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.1</version>
<executions>
<execution>
<id>copy</id>
<phase>prepare-package</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>com.abc.def</groupId>
<artifactId>libdef</artifactId>
<version>2.1.3</version>
<type>so</type>
<destFileName>libdef.so</destFileName>
</artifactItem>
</artifactItems>
<outputDirectory>${project.build.directory}/libs/armeabi</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

I suggest you to use maven-android-plugin version 2.8.3 or more.
The scope of your native lib must be runtime (not sure it is required, but anyway it's a fact)
The artifactId must start with lib (i.e. it must be libsdc)
<dependency>
<groupId>com.abc.def</groupId>
<artifactId>libsdc</artifactId>
<version>3</version>
<scope>runtime</scope>
<type>so</type>
</dependency>
The artifactId will be the library name so load it with this line of code
System.loadLibrary("sdc");
reference
Note: I don't know if sdc if the real artifactId, but if it's the case you must consider re-publishing it with the lib prefix.

I would suggest looking into the Assembly plugin: http://maven.apache.org/plugins/maven-assembly-plugin/ I have not used it in any Android project yet, but I do use it in my regular java server projects which require non-maven items be in expected locations.

NoClassDefFoundError on Maven dependency

My first use of Maven and I'm stuck with dependencies.
I created a Maven project with Eclipse and added dependencies, and it was working without problems.
But when I try to run it via command line:
$ mvn package # successfully completes
$ java -cp target/bil138_4-0.0.1-SNAPSHOT.jar tr.edu.hacettepe.cs.b21127113.bil138_4.App # NoClassDefFoundError for dependencies
It downloads dependencies, successfully builds, but when I try to run it, I get NoClassDefFoundError:
Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/jackson/JsonParseException
at tr.edu.hacettepe.cs.b21127113.bil138_4.db.DatabaseManager.<init>(DatabaseManager.java:16)
at tr.edu.hacettepe.cs.b21127113.bil138_4.db.DatabaseManager.<init>(DatabaseManager.java:22)
at tr.edu.hacettepe.cs.b21127113.bil138_4.App.main(App.java:10)
Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.JsonParseException
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
... 3 more
My pom.xml is like this:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>tr.edu.hacettepe.cs.b21127113</groupId>
<artifactId>bil138_4</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>bil138_4</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>1.9.6</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.6</version>
</dependency>
</dependencies>
</dependencyManagement>
</project>
Can anyone help me?

By default, Maven doesn't bundle dependencies in the JAR file it builds, and you're not providing them on the classpath when you're trying to execute your JAR file at the command-line. This is why the Java VM can't find the library class files when trying to execute your code.
You could manually specify the libraries on the classpath with the -cp parameter, but that quickly becomes tiresome.
A better solution is to "shade" the library code into your output JAR file. There is a Maven plugin called the maven-shade-plugin to do this. You need to register it in your POM, and it will automatically build an "uber-JAR" containing your classes and the classes for your library code too when you run mvn package.
To simply bundle all required libraries, add the following to your POM:
<project>
...
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.4.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
...
</project>
Once this is done, you can rerun the commands you used above:
$ mvn package
$ java -cp target/bil138_4-0.0.1-SNAPSHOT.jar tr.edu.hacettepe.cs.b21127113.bil138_4.App
If you want to do further configuration of the shade plugin in terms of what JARs should be included, specifying a Main-Class for an executable JAR file, and so on, see the "Examples" section on the maven-shade-plugin site.

when I try to run it, I get NoClassDefFoundError
Run it how? You're probably trying to run it with eclipse without having correctly imported your maven classpath. See the m2eclipse plugin for integrating maven with eclipse for that.
To verify that your maven config is correct, you could run your app with the exec plugin using:
mvn exec:java -D exec.mainClass=<your main class>
Update: First, regarding your error when running exec:java, your main class is tr.edu.hacettepe.cs.b21127113.bil138_4.App. When talking about class names, they're (almost) always dot-separated. The simple class name is just the last part: App in your case. The fully-qualified name is the full package plus the simple class name, and that's what you give to maven or java when you want to run something. What you were trying to use was a file system path to a source file. That's an entirely different beast. A class name generally translates directly to a class file that's found in the class path, as compared to a source file in the file system. In your specific case, the class file in question would probably be at target/classes/tr/edu/hacettepe/cs/b21127113/bil138_4/App.class because maven compiles to target/classes, and java traditionally creates a directory for each level of packaging.
Your original problem is simply that you haven't put the Jackson jars on your class path. When you run a java program from the command line, you have to set the class path to let it know where it can load classes from. You've added your own jar, but not the other required ones. Your comment makes me think you don't understand how to manually build a class path. In short, the class path can have two things: directories containing class files and jars containing class files. Directories containing jars won't work. For more details on building a class path, see "Setting the class path" and the java and javac tool documentation.
Your class path would need to be at least, and without the line feeds:
target/bil138_4-0.0.1-SNAPSHOT.jar:
/home/utdemir/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.6/jackson-core-asl-1.9.6.jar:
/home/utdemir/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.6/jackson-mapper-asl-1.9.6.jar
Note that the separator on Windows is a semicolon (;).
I apologize for not noticing it sooner. The problem was sitting there in your original post, but I missed it.

You have to make classpath in pom file for your dependency. Therefore you have to copy all the dependencies into one place.
Check my blog.
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.1</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<overWriteReleases>false</overWriteReleases>
<overWriteSnapshots>false</overWriteSnapshots>
<overWriteIfNewer>true</overWriteIfNewer>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>$fullqualified path to your main Class</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>

This is due to Morphia jar not being part of your output war/jar. Eclipse or local build includes them as part of classpath, but remote builds or auto/scheduled build don't consider them part of classpath.
You can include dependent jars using plugin.
Add below snippet into your pom's plugins section
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>

For some reason, the lib is present while compiling, but missing while running.
My situation is, two versions of one lib conflict.
For example, A depends on B and C, while B depends on D:1.0, C depends on D:1.1, maven may
just import D:1.0. If A uses one class which is in D:1.1 but not in D:1.0, a NoClassDefFoundError will be throwed.
If you are in this situation too, you need to resolve the dependency conflict.

I was able to work around it by running mvn install:install-file with -Dpackaging=class. Then adding entry to POM as described here:

Choosing to Project -> Clean should resolve this

Maven exec plugin- how to include "system" classpath?

I have a project that uses "system" scope to specify a jar file included in my project's WEB-INF/lib dir. This artifact is not in any of the maven repositories, so I must include it as part of my project. I do so with the following:
<dependency>
<groupId>com.example</groupId>
<artifactId>MySpecialLib</artifactId>
<version>1.2</version>
<scope>system</scope>
<systemPath>${basedir}/src/main/webapp/WEB-INF/lib/MySpecialLib-1.2.jar</systemPath>
</dependency>
This has worked great for most things.
But now I'm trying to run some code on the command line (outside of my webapp, via a main() method I have added) and mvn exec:java can't resolve code in MySpecialLib because it's not included in the "runtime" classpath.
How can I either:
add MySpecialLib to the runtime classpath
or
tell mvn exec:java to also use the system classpath ?
I've tried mvn exec:java -Dexec.classpathScope=system, but that leaves off everything that's on runtime.

Use 'compile' scope to run maven exec plugin - mvn exec:java -Dexec.classpathScope=compile. This will include system-scoped dependencies.

As E.G. pointed out, the solution is to use the compile scope when running exec.
On each invocation:
mvn exec:java -Dexec.classpathScope=compile
or directly in the exec-plugin-configuration:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
...
<configuration>
<classpathScope>compile</classpathScope>
</configuration>
</plugin>

Interesting to know that classpathScope=system drops runtime dependencies. I found that by including it as a plugin in the pom.xml works as an alternative. Could you please try and let me know if it works for you too?
So I added a system level dependency to commons-collection as an example like you have for your artifact:-
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.0</version>
<scope>system</scope>
<systemPath>C:\\<some_path>\\commons-collections-3.0.jar</systemPath>
</dependency>
Then in the <build> tag I have the exec-maven-plugin plugin to be executed in the install phase:-
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.1</version>
<executions>
<execution>
<phase>install</phase>
<goals>
<goal>java</goal>
</goals>
<configuration>
<mainClass>com.stackoverflow.test.App</mainClass>
</configuration>
</execution>
</executions>
</plugin>
Then I ran mvn install. I also made sure com.stackoverflow.test.App class has some code that invokes a class from commons-collections-3.0.
Hope this helps.

The right answer is to use the maven-install-plugin and Put The Jar Into Your Local Repo. Or, better yet, run nexus or artifactory and use the deploy plugin to put the jar into there. System classpath is just a world of hurt.