We Keep Coding

neo4j procedure depending on signed jar (mapdb)

I have created a neo4j user-defined procedure. It also compiles and works in neo4j.
Recently though, I added a dependency to my procedure plugin that prevents neo4j from starting when I try to run the newly built jar. Concretely, I receive following exception at the bottom of the stack trace:
Caused by: java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
The only thing I changed is to add MapDB to my dependencies. So I suspect that it depends on some signed artifact. As it turns out neo4j plugins are "uber-jars" (i.e., shaded jars) which doesn't work very well with signed dependencies.
I figured I could try to exclude MapDB from the shading by changing the scope to provided and additionally adding the mapdb jar to plugins folder of neo4j. So the plugins folder of neo4j now includes both mapdb.jar and myprocedure.jar.
This doesn't seem to work though: neo4j starts, but when calling my procedure I receive a ClassNotFound Exception.
Any ideas on how I can solve this dilemma? I really depend on something like MapDB as my graph is very large and keeping everything in my procedure in-memory regularly leads to memory exceptions.
Many thanks in advance!
The important part of my pom.xml should it help (I started off with the procedure template so it still looks quite similar):
<dependencies>
<!-- other dependencies that don't make a difference ... -->
<dependency>
<groupId>org.mapdb</groupId>
<artifactId>mapdb</artifactId>
<version>3.0.7</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j</artifactId>
<version>${neo4j.version}</version>
<scope>provided</scope>
</dependency>
<!-- Test Dependencies -->
<dependency>
<groupId>org.neo4j.test</groupId>
<artifactId>neo4j-harness</artifactId>
<version>${neo4j.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>1.4.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<!-- Neo4j Procedures require Java 8 -->
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<!-- This generates a jar-file with our procedure code,
plus any dependencies marked as `compile` scope.
This should then be deployed in the `plugins` directory
of each Neo4j instance in your deployment.
After a restart, the procedure is available for calling. -->
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

So, one solution that worked was to use the maven-assembly-plugin instead of the maven-shade-plugin. The assembly plugin creates a standalone jar with all dependencies without re-packaging all dependencies.
Of course, I had to remove the <scope>provided</scope> for mapdb so it is also included in the assembled jar.
The rest is just replacing the shade plugin with following snippet (thanks to this question here):
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
Afterwards I received another exception which I thought was due to a wrong signature:
Caused by: java.lang.VerifyError ...
Turns out this was just because I had another procedure in neo4j that uses the same dependencies as mapdb. Quick fix was to remove that procedure library.
Of course, there are also other solutions, including removing the signature, or re-signing it. However, I explicitly did not want to remove any signatures.

I'm not aware that neo4j is checking any jar signatures by default. I suspect the repackaging of mapdb artifact being the culprit.
To validate that, remove the shade or assembly plugin in and build a jar that solely contains your code. Copy that jar and the mapdb.jar into the plugin folder and observe logs during a restart.

Documents4j remoteServer conversion server setup

I'm using Documents4j to convert an rtf file to pdf file. I don't have MS word or anything on my computer, so it seems that I will need to use a remote converter. Information here: http://documents4j.com/#/.
My project is setup with spring-boot. I went through and set up shading for maven using maven-shade-plugin based on some issues on GitHub. However, I cannot run the command that Documents4j suggests to get the server running:
java -jar documents4j-server-standalone-shaded.jar http://localhost:9998
I get:
Error: Unable to access jarfile documents4j-server-standalone-shaded.jar.
My pom.xml file brings in the shade plugin.
Here is the plugin in my pom.xml:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
</configuration>
</execution>
</executions>
</plugin>
Here are the dependencies I am using in regards to Documents4j: (perhaps something important is missing?)
<dependency>
<groupId>com.documents4j</groupId>
<artifactId>documents4j-api</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>com.documents4j</groupId>
<artifactId>documents4j-client</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>com.documents4j</groupId>
<artifactId>documents4j-server-standalone</artifactId>
<version>1.0.3</version>
</dependency>
Based on what I read on the Documents4j page, it seems that the command should just work, so I assume that the jar file isn't being created.
I can't seem to find that jarfile anywhere... so I have been unable to run the java -jar command with a path instead of just a name.
Also, I was unsure about what "configuration" I may need in the plugin. Perhaps there's a trick to getting the maven shading to work? Maybe I'm misunderstanding what the Documents4j page is saying? Maybe it works differently for Macs? Maybe Documents4j isn't a good choice?
I greatly appreciate all assistance.

Note that it is only possible to run on a Windows Server that supports .NET and has Office installed.
https://github.com/documents4j/documents4j/issues/53

java.lang.NoSuchMethodError Bouncy Castle and Apache Spark

I actually found a company (where I work) specific work-around this issue, but the problem is so bizarre, I wanted to see if anyone else has ran into this issue and how they solved it.
First I added two bouncy castle dependencies to my spark job:
<!-- https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on -->
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcprov-jdk15on</artifactId>
<version>1.52</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.bouncycastle/bcpg-jdk15on -->
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcpg-jdk15on</artifactId>
<version>1.52</version>
</dependency>
I also have this plugin to create a jar with all dependencies:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4.1</version>
<configuration>
<!-- get all project dependencies -->
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<!-- MainClass in mainfest make a executable jar -->
<archive>
<manifest>
<mainClass>com.theclasspath.jobs.TestingJob</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<!-- bind to the packaging phase -->
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
So all the dependencies are getting installed. Now this will compile and create the jar. When I run spark-submit with the jar with dependencies, I get a no such method error for some methods in the org.bouncycastle.util package. The main one being:
java.lang.NoSuchMethodError: org.bouncycastle.util.Strings.newList()Lorg/bouncycastle/util/StringList;
when trying to just run Strings.newList(); in the main class. So I ran mvn dependency:tree to see if there were multiple versions of bouncy castle, and I only saw the dependencies I added to the pom.
Here is the really weird part: if I build the jar and run it with java -jar jar-name-jar-with-dependencies.jar (meaning not through spark-submit) the dependency gets added just fine, and I do not see any errors. So with that in mind, what would be the differences between the spark submit jar and just running the jar directly?
Why would one work and the other fail? I have tried running it on different machines, such as an emr job, and I still see the same error. (and to avoid this getting asked, I am using the jar-with-dependencies jar for the spark job)
One final note, I have seen this post Bouncy castle no such method error, and it doesn't apply, since I can get the jar to run when I just run the jar by itself without spark-submit.

Running spock unit tests with Maven

On a previous project I used the Spock testing framework to unit test my Java code. I found this really productive so I am trying to add Spock tests to my current project which uses Maven as its build tool (The previous project used Gradle). While I can get Maven to compile my Spock tests (using groovy-eclipse-compiler), I am unable to get Maven to run the tests.
I've made a simple example to demonstrate my problem with 2 files:
pom.xml
src/test/java/ASpec.groovy
Contents of pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>my.group</groupId>
<artifactId>my-artifact</artifactId>
<version>0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
<version>2.0.8</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.spockframework</groupId>
<artifactId>spock-core</artifactId>
<version>0.7-groovy-2.0</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<compilerId>groovy-eclipse-compiler</compilerId>
</configuration>
<dependencies>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-compiler</artifactId>
<version>2.8.0-01</version>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-batch</artifactId>
<version>2.1.8-01</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</project>
Contents of ASpec.groovy:
import spock.lang.Specification
class ASpec extends Specification {
def "Test A"(){
// Always fail
expect: false
}
}
When I execute mvn clean test (or mvn clean install) I would expect my single unit test to be run and fail. While it is compiled, Maven does not run it. Does any one know how to run a Spock unit test from Maven (or if it is possible?)
(I have not put my test in a package to keep the example simple. Also I have put my groovy code in src/test/java to avoid configuring the example to pick up source files from an additional directory, again to keep the example as simple as possible.)

This answer is purely supplemental to #PeterNiederwieser's answer. In it he mentions that you can configure the name pattern used by Surefire. Here is an example of what worked for me:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18</version>
<configuration>
<includes>
<!-- By default only files ending in 'Test' will be included, so also include support for Spock style naming convention -->
<!-- Oddly enough for Groovy files, *Spec.groovy does not work, but *Spec.java does -->
<include>**/*Test.java</include>
<include>**/*Spec.java</include>
</includes>
</configuration>
</plugin>
Source
As I mention in the comments, I'm not sure why **/*Spec.groovy didn't work, but I'm happy to be able to use the normal Spock convention here.

Maven Surefire finds test classes by their name. Either change the class name to ATest, or reconfigure the name pattern used by Surefire. The POM for the spock-example project demonstrates how to do the latter.

I had the same requirement to add Spock to my existing java web app.
I tried Peters but it did not work for me.
gmavenplus-plugin somehow (no idea) replaced my guava dependency with a very old google lib and my Spring application broke complaining about a non-existent method.
After literally maybe 2 or 3 dozen attempts, I finally was able to integrate my Spock Unit tests, and Integration tests and more importantly to isolate the compilation of the Spock groovy classes from my existing Java/Junit Spring/Hibernate application.
Of course if I had gradle it would have solved the issue...but this is a legacy project and therefore I had not the choice.
Below are the plugins I added.
Please note the Spock unit tests end with Spec.
The Spock integration tests end with IT (but most probably should be SpecIT).
I put my Spock tests under src/test/groovy.
<plugins>
<plugin>
<groupId>org.codehaus.gmavenplus</groupId>
<artifactId>gmavenplus-plugin</artifactId>
<version>1.4</version>
<executions>
<execution>
<!-- Without joint compilation - no dependencies between Java and Groovy (inheritance)-->
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<sources>
<source>
<directory>${project.basedir}/src/main/java/groovy</directory>
<includes>
<include>**/*.groovy</include>
</includes>
</source>
</sources>
<testSources>
<testSource>
<directory>${project.basedir}/src/test/groovy</directory>
<includes>
<include>**/*.groovy</include>
</includes>
</testSource>
</testSources>
</configuration>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<testSourceDirectory>src/test/groovy</testSourceDirectory>
<testSourceDirectory>src/test/java</testSourceDirectory>
<includes>
<include>**/*Spec.java</include>
<!-- Yes, .java extension -->
<include>**/*Test.java</include>
<!-- Just in case having "normal" JUnit tests -->
</includes>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.20</version>
<configuration>
<useFile>false</useFile>
<includes>
<include>**/*IT.java</include>
</includes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
And here are my dependencies:
<!--Spock -->
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy</artifactId>
<version>2.4.7</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.spockframework</groupId>
<artifactId>spock-core</artifactId>
<version>1.1-groovy-2.4</version>
</dependency>
<dependency>
<groupId>org.spockframework</groupId>
<artifactId>spock-spring</artifactId>
<version>1.1-groovy-2.4</version>
</dependency>
<dependency>
<groupId>org.codehaus.groovy.modules.http-builder</groupId>
<artifactId>http-builder</artifactId>
<version>0.7.1</version>
</dependency>
<!--Spock mocking dependencies -->
<dependency>
<groupId>cglib</groupId>
<artifactId>cglib-nodep</artifactId>
<version>3.2.5</version>
</dependency>
<dependency>
<groupId>org.objenesis</groupId>
<artifactId>objenesis</artifactId>
<version>2.6</version>
</dependency>
And just to let you know, my original POM had absolutely no explicit plugins AT ALL. So I had a very simply POM for my project. So, it should work for you.
It is a Java 1.7 project.
...and finally, just to give you some confidence that this is not a rubbish post, I did multiple tests in order to ensure the above worked:
Just build the WAR without the tests and deploy and smoke test it locally
mvn clean install -DskipTests -Dmaven.test.skip=true
Do a test compile and see if the Groovy Unit tests get compiled as well
mvn -X clean test-compile
Do a clean install without the Integration test (i made sure it was failing for this test) and see if the Groovy unit tests are run
mvn clean install -DskipITs
Just run the integration test
mvn failsafe:integration-test
I would have liked to include screenshots of the above as proof but it would have had to be censored...So, I sincerely hope this helps you, as I was going mental trying to get this working...Maven is a huge subject area. Good luck :=)

A problem I encountered is incompatible dependencies.
The spock-core dependency of course pulls in a dependency on the version of groovy it needs. But it is possible that other dependencies or plugins (I think thegmaven-plus plugin in my case) pull in a different, incompatible, version of groovy.
I fixed the problem by explicitly giving a dependency on the groovy package I wanted to use.

Maven: How do I get .so libraries bundled in package

I have a third party library coming with .jar and .so file.
I configured pom.xml as following:
<dependency>
<groupId>com.abc.def</groupId>
<artifactId>sdc</artifactId>
<version>1.1</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.abc.def</groupId>
<artifactId>sdc</artifactId>
<version>3</version>
<scope>compile</scope>
<type>so</type>
</dependency>
With this configure, I successfully tested through Intellij and apk file under target contains structure like lib/armeabi/sdc.so
However, after I do mvn clean package, the apk file generated did not contain sdc.so file, and after installing apk file on android device, lib folder is empty.
Searching through the internet, and did not find answer.
Btw, I do add <nativeLibrariesDirectory>${project.basedir}/libs</nativeLibrariesDirectory> into pluginManagement as mentioned Native libraries (.so files) are not added to an android project, but does not help.
Updates:
If someone is facing same problem as I am that SO file did not copy over, please try manually copy the so file over as following:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.1</version>
<executions>
<execution>
<id>copy</id>
<phase>prepare-package</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>com.abc.def</groupId>
<artifactId>libdef</artifactId>
<version>2.1.3</version>
<type>so</type>
<destFileName>libdef.so</destFileName>
</artifactItem>
</artifactItems>
<outputDirectory>${project.build.directory}/libs/armeabi</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

I suggest you to use maven-android-plugin version 2.8.3 or more.
The scope of your native lib must be runtime (not sure it is required, but anyway it's a fact)
The artifactId must start with lib (i.e. it must be libsdc)
<dependency>
<groupId>com.abc.def</groupId>
<artifactId>libsdc</artifactId>
<version>3</version>
<scope>runtime</scope>
<type>so</type>
</dependency>
The artifactId will be the library name so load it with this line of code
System.loadLibrary("sdc");
reference
Note: I don't know if sdc if the real artifactId, but if it's the case you must consider re-publishing it with the lib prefix.

I would suggest looking into the Assembly plugin: http://maven.apache.org/plugins/maven-assembly-plugin/ I have not used it in any Android project yet, but I do use it in my regular java server projects which require non-maven items be in expected locations.