We Keep Coding

Maven copy-dependencies + shade - classpath management

To package a maven project with its dependencies, among many solutions, one may use maven-dependency-plugin with its goal copy-dependencies to get the dependencies in a folder besides, or one may use maven-shade-plugin to get all the code in a single jar.
I actually do both: I choose to have external dependencies (e.g. apache commons) as external libs, and my own dependencies (I have a multi-module maven parent project) shaded into a unique jar.
And it works, except for the classpath. I copy-dependencies with option excludeGroupIds to exclude my own maven group id. I shade with option to include only my own maven group id. Before that, I jar with option to add classpath to the manifest. All set, it works. But my classpath also contains my own dependencies that were actually shaded in the final jar.
It is no big deal, because the result works even with this erroneous classpath. But I wonder if there is a simple means to have the correct classpath, in order not to expose my internal structure to my users.
Here is a basic example demonstrating the problem:
<groupId>com.foo.bar</groupId>
<artifactId>com.foo.bar.launcher</artifactId>
<dependencies>
<dependency>
<groupId>com.foo.bar</groupId>
<artifactId>com.foo.bar.utils</artifactId>
<version>0.0.1</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.8.1</version>
</dependency>
</dependencies>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<excludeGroupIds>com.foo.bar</excludeGroupIds>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<includes>
<include>com.foo.bar:*</include>
</includes>
</artifactSet>
</configuration>
</execution>
</executions>
</plugin>
The resulting manifest contains this:
Class-Path: lib/com.foo.bar.utils-0.0.1.jar lib/commons-lang3-3.8.1.jar while the com.foo.bar.utils one does not exist.

If you look into the following mvnrepository link maven shade plugin depends upon maven dependency tree. As per the above pom.xml maven dependency plugin, you have excluded com.foo.bar dependency. You can omit the maven-dependency-plugin to create fat jar. It is not mandatory to use in case of shade plugin.
You can use the following command to check and copy all the dependencies used in the project.
mvn dependency:copy-dependencies
https://mvnrepository.com/artifact/org.apache.maven.plugins/maven-shade-plugin/3.2.1

Getting my dependencies into my JAR

I can package a JAVA project I've written. One which uses the Gson library for JSON features. I'm very new to JAVA so I could be making a dumb mistake but here's what I've assumed:
In the source code I have:
import com.google.gson.Gson;
and then use this import like so:
Gson gson = new Gson();
String json = gson.toJson(result);
In my Maven pom.xml I've included the following in the dependency section:
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.2</version>
<scope>compile</scope>
</dependency>
</dependencies>
As I say, it does package to a JAR file with no errors (using Maven for packaging) but my JAR file is being used on AWS as a serverless function and so I believe what I need to do is include the Gson dependency as part of my JAR file (could be wrong). This seems to be backed up by the errors I get on AWS:
Having done some google searches it looked like maybe Maven's "shade" plugin might be the ticket. I added it into my pom.xml like so:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<artifactSet>
<includes>
<include>com/google/gson/**</include>
</includes>
</artifactSet>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
But looking in the generated JAR file I see no difference. What do I need to do to get this to work?
The full POM file can be found here: pom.xml

I'm unfamiliar with the shade plugin others have referenced. The way it sounds to me, you need an artifact that's an executable jar: a jar, with its dependencies.
Here's how I do that, using Maven:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.company.groupId</groupId>
<artifactId>artifact-id</artifactId>
<version>1.0.0</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.7.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<archive>
<manifest>
<mainClass>com.company.App</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<id>attach-javadoc</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>3.0.1</version>
<executions>
<execution>
<id>attach-source</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.2</version>
<scope>compile</scope>
</dependency>
</dependencies>
</project>
What you want to pay attention to is the build plugin, maven-assembly-plugin. This tells Maven how to assemble/package the results of the build. In its configuration, you define the main class that contains the runnable application, which usually is going to be where your public static void main(String[] args) declaration is. You also define a descriptor-ref, which is a String that will be appended to the jar's name. So, you'd end up with artifactId-1.0.0-jar-with-dependencies.jar, using my POM as an example.
To further explain what's going on, without the changes I recommend, your POM is telling Maven to just build your code. As part of that, you declare dependencies, of which you have two right now: aws-lambda-java-core and gson. When you don't provide a scope, it defaults to compile scope. This tells Maven to grab that dependency when the program is compiled, so that the program can use that dependency's code. But, when packaging the build artifact of your program, Maven, by default, will not include those dependencies in the final jar; it expects that when you run the jar, you'll have those dependencies on your classpath.
By adding the assembly build plugin, you're changing those instructions to Maven. With that plugin, you're telling Maven that when it builds the program, it needs to assemble it in such a way that all declared dependencies are included (read: assembled) with the program, and to do that during the package phase of the build; you'll see these dependencies in the lib folder of the build artifact. And then, like I mentioned earlier, the descriptorRef is descriptive info that will be appended onto the build artifact's name.
As an aside, and not truly relevant to your question, I'd recommend looking into FasterXML for JSON handling and manipulation. So much more powerful, so much easier, and it's widely supported and used, which means it has a great community behind it.

If you have dependencies that will be provided from runtime container, you should set scope provided to these dependencies.
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.0.0</version>
<scope>provided</scope>
</dependency>
And remove <artifactSet> section from plugin configuration, execute mvn package then jar with required dependencies will be created.

Maven not including manifest attributes for LWJGL install

I am trying to setup a LWJGL project using Maven. I am using the example "getting started" source code from the official website.
This includes a few lines accessing LWJGL's manifest attributes, such as a simple version check:
System.out.println("Hello LWJGL " + Version.getVersion() + "!");
This runs without any problems in the Eclipse environment (of course after having built the project with Maven), but when running clean install and then running **-jar-with-dependencies.jar through cmd, the following exception get's thrown:
java.lang.NullPointerException
at org.lwjgl.system.APIUtil.apiGetManifestValue(APIUtil.java:97)
at org.lwjgl.Version.getVersion(Version.java:33)
at HelloWorld.run(HelloWorld.java:43)
at HelloWorld.main(HelloWorld.java:130)
This is because the Manifest object created by APIUtil does not include any attributes - but only in the built version by Maven.
Why is this? Is my pom.xml buggy, or is LWJGL 3.0.0 just not ready for this?
This is my pom.xml:
<properties>
<mainClass>HelloWorld</mainClass>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<finalName>${project.artifactId}-${project.version}.jar</finalName>
</properties>
<dependencies>
<dependency>
<groupId>org.lwjgl</groupId>
<artifactId>lwjgl</artifactId>
<version>3.0.0</version>
</dependency>
<dependency>
<groupId>org.lwjgl</groupId>
<artifactId>lwjgl-platform</artifactId>
<version>3.0.0</version>
<classifier>natives-windows</classifier>
</dependency>
<dependency>
<groupId>org.lwjgl</groupId>
<artifactId>lwjgl-platform</artifactId>
<version>3.0.0</version>
<classifier>natives-linux</classifier>
</dependency>
<dependency>
<groupId>org.lwjgl</groupId>
<artifactId>lwjgl-platform</artifactId>
<version>3.0.0</version>
<classifier>natives-osx</classifier>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>${mainClass}</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>

This errors happens because LWJGL 3.0.0 is looking inside the Manifest a property called "Implementation-Version", but when you made the uber-jar, this property was not set.
This is not really an issue with how you made the uber-jar: the Manifest that was created by maven-assembly-plugin looks like:
Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Built-By: Me
Created-By: Apache Maven 3.3.9
Build-Jdk: 1.8.0_102
Main-Class: HelloWorld
You can see it inside META-INF/MANIFEST.MF of the jar-with-dependencies. This file does not have a "Implementation-Version" property. This is normal: when this executable JAR was created, all the MANIFEST of all dependencies were (rightfully) ignored, only to generate one containing the "Main-Class", just so that the JAR is executable.
The uber-jar cannot contain what is inside each of the dependencies manifest. For example, "Implementation-Version" is a property that is present in the manifest of multiple libraries, so which one should it keep? (There can be only one Manifest at the end, in the uber-jar). So the issue comes up because we're making an executable JAR, which can only have 1 Manifest so it cannot aggregate all the properties inside each of the dependencies manifest.
There are 2 possible solutions:
Ignore it. After all, this is not really an error.
Don't make an executable jar by embedding all the dependencies inside a single JAR, but create a ZIP assembly with each dependencies inside a lib folder: this way, each Manifest will be kept. This is done by telling the maven-jar-plugin to add a Manifest entry for the main class with the addition of the classpath and creating a custom assembly descriptor.
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
<configuration>
<archive>
<manifest>
<mainClass>${mainClass}</mainClass>
<addClasspath>true</addClasspath>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptors>
<descriptor>/path/to/assembly.xml</descriptor>
</descriptors>
</configuration>
</plugin>
where /path/to/assembly.xml is the path to the assembly descriptor, relative to the location of the POM, being:
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd">
<id>dist</id>
<formats>
<format>zip</format>
</formats>
<dependencySets>
<dependencySet>
<outputDirectory>lib</outputDirectory>
<useProjectArtifact>true</useProjectArtifact>
</dependencySet>
</dependencySets>
</assembly>
With such a configuration, running mvn clean install will create a ZIP file artifactId-version-dist.zip. Unpacking it and running (replacing <finalName> with the finalName of your JAR)
java -jar lib\<finalName>.jar
will print the version without any issues.

CouchDB4j/ mvn dependencies are missing

I am having trouble setting up a connection to my local CouchDB programmatically.
I am using couchDb4j- and things seem to look good, until I run and try to connect to the DB.
My console is throwing the following error:
Exception in thread "AWT-EventQueue-0" java.lang.NoClassDefFoundError: org/apache/http/params/HttpParams
[...]
Caused by: java.lang.ClassNotFoundException: org.apache.http.params.HttpParams
Since my small application is not finding a class, I've checked my dependencies- everything should be fine. I have:
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.0-beta3</version>
</dependency>
<dependency>
<groupId>commons-httpclient</groupId>
<artifactId>commons-httpclient</artifactId>
<version>3.1</version>
</dependency>
Which should include all necessary http specific .jar (especially the first one should include the httpParams binaries; source: http://mvnrepository.com/artifact/org.apache.httpcomponents/httpcore/4.0-beta3 ).
To prevent from including wrong binaries cached on my system, I am running the following:
mvn clean && mvn package
I've also deleted my .m2 folder.
Googling around gave me the hint that my classpath may be wrong leading to missing dependencies in runtime. But I have set up my classpath in my pom; see here:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>Packagename.Main</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
Hopefully anyone can give me one idea so that I can fix this issue :)
Best regards.

Add these dependencies inside plugin declaration like this:
<plugin>
<groupId>...</groupId>
<artifactId>...</artifactId>
<version>...</version>
<dependencies>
...
</dependencies>
</plugin>
Plugins have their own classpath

Found the solution.
The problem were missing dependencies during execution.
To make sure, that all necessary files are binded during runtime, I had to include the following plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>1.6</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
which, if I got it right, binds on an "uber" jar all dependencies...
Source: https://maven.apache.org/plugins/maven-shade-plugin/

How is JSPC called?

Given:
Upon running mvn clean install JSP compiler jspc reports issues
Issues are confirmed real. When deploying code as is (without fixing anything) JSP is broken at runtime
Upon fixing the issues and deploying the application, issues go away
Problem: Fix is not recognized by JSPC
How is JSPC called?
<plugin>
<groupId>org.codehaus.mojo.jspc</groupId>
<artifactId>jspc-maven-plugin</artifactId>
<configuration>
<includeInProject>false</includeInProject>
<sources>
<directory>${basedir}/myapp/src/main/webapp/</directory>
<includes>
<include>**/*.jsp</include>
</includes>
</sources>
<source>1.6</source>
<target>1.6</target>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.codehaus.mojo.jspc</groupId>
<artifactId>jspc-compiler-tomcat6</artifactId>
<version>2.0-alpha-3</version>
</dependency>
</dependencies>
</plugin>
What errors are reported?
[ERROR] MyClass cannot be resolved to a type
More details about the problem:
Similar question about this here
It seems like a classpath problem, but where in pom can it be set please?

Try adding
<workingDirectory>${basedir}/myapp/target/classes</workingDirectory>
inside <configuration> tags