We Keep Coding

How to reduce jar size while using maven-assembly-plugin in pom.xml for building spark application

I am using the below plugin: maven-assembly-plugin in my pom.xml file with many other separate dependencies mentioned in the same file as below to build a jar file for my spark application using IntelliJ:
pom.xml
Maven-assembly-plugin
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.3</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id> <!-- this is used for inheritance merges -->
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
pom.xml
other dependencies:
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-emr</artifactId>
<version>1.11.125</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.11.106</version>
</dependency>
to name a few..............!
What's happening is, this maven-assembly-plugin is bundling in all the dependency jars together with the main jar file and the size of the final jar file goes beyond 150+ Mb.
Is there a way in which I can mention only a particular dependency from my pom.xml to be included in the final jar file and excluding all the other dependencies??
With only this dependency included in the jar, hopefully, the final jar size should come down below 1 Mb.
The specific dependency which I want to include in my final jar is the below one:
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20200518</version>
</dependency>

Optional jar inclusion in fat jar in maven plugin

How to create a fat jar with specific dependencies.
I have spark project which need 2 external jar which I wanted to add in application jar. when I am creating executable jar then no dependency is included in jar and when I create fat jar all the dependencies are getting added including spark etc.. I wanted to add only those 2 jars in my jar. below is the pom file I created using maven assembly plugin.
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<!-- Below dependencies need to be added in Application jar -->
<dependency>
<groupId>netacuity</groupId>
<artifactId>common-netacuity-db</artifactId>
<version>3.1.2</version>
</dependency>
<dependency>
<groupId>netacuity</groupId>
<artifactId>common</artifactId>
<version>2.1.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4.1</version>
<configuration>
<!-- get all project dependencies -->
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<!-- MainClass in mainfest make a executable jar -->
<archive>
<manifest>
<mainClass>com....App</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<!-- bind to the packaging phase -->
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>

You can use the scope for this. By default scope is compile and so all the jars will be included when you package it.
To include a jar you can either provide scope as compile and keep the default
<dependency>
<groupId>netacuity</groupId>
<artifactId>common-netacuity-db</artifactId>
<version>3.1.2</version>
<scope>compile</scope>
</dependency>
To exclude the jar, you can change the scope to provided. These jars should be available during runtime.
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.2.0</version>
<scope>provided</scope>
</dependency>

Getting my dependencies into my JAR

I can package a JAVA project I've written. One which uses the Gson library for JSON features. I'm very new to JAVA so I could be making a dumb mistake but here's what I've assumed:
In the source code I have:
import com.google.gson.Gson;
and then use this import like so:
Gson gson = new Gson();
String json = gson.toJson(result);
In my Maven pom.xml I've included the following in the dependency section:
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.2</version>
<scope>compile</scope>
</dependency>
</dependencies>
As I say, it does package to a JAR file with no errors (using Maven for packaging) but my JAR file is being used on AWS as a serverless function and so I believe what I need to do is include the Gson dependency as part of my JAR file (could be wrong). This seems to be backed up by the errors I get on AWS:
Having done some google searches it looked like maybe Maven's "shade" plugin might be the ticket. I added it into my pom.xml like so:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<artifactSet>
<includes>
<include>com/google/gson/**</include>
</includes>
</artifactSet>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
But looking in the generated JAR file I see no difference. What do I need to do to get this to work?
The full POM file can be found here: pom.xml

I'm unfamiliar with the shade plugin others have referenced. The way it sounds to me, you need an artifact that's an executable jar: a jar, with its dependencies.
Here's how I do that, using Maven:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.company.groupId</groupId>
<artifactId>artifact-id</artifactId>
<version>1.0.0</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.7.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<archive>
<manifest>
<mainClass>com.company.App</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<id>attach-javadoc</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>3.0.1</version>
<executions>
<execution>
<id>attach-source</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.2</version>
<scope>compile</scope>
</dependency>
</dependencies>
</project>
What you want to pay attention to is the build plugin, maven-assembly-plugin. This tells Maven how to assemble/package the results of the build. In its configuration, you define the main class that contains the runnable application, which usually is going to be where your public static void main(String[] args) declaration is. You also define a descriptor-ref, which is a String that will be appended to the jar's name. So, you'd end up with artifactId-1.0.0-jar-with-dependencies.jar, using my POM as an example.
To further explain what's going on, without the changes I recommend, your POM is telling Maven to just build your code. As part of that, you declare dependencies, of which you have two right now: aws-lambda-java-core and gson. When you don't provide a scope, it defaults to compile scope. This tells Maven to grab that dependency when the program is compiled, so that the program can use that dependency's code. But, when packaging the build artifact of your program, Maven, by default, will not include those dependencies in the final jar; it expects that when you run the jar, you'll have those dependencies on your classpath.
By adding the assembly build plugin, you're changing those instructions to Maven. With that plugin, you're telling Maven that when it builds the program, it needs to assemble it in such a way that all declared dependencies are included (read: assembled) with the program, and to do that during the package phase of the build; you'll see these dependencies in the lib folder of the build artifact. And then, like I mentioned earlier, the descriptorRef is descriptive info that will be appended onto the build artifact's name.
As an aside, and not truly relevant to your question, I'd recommend looking into FasterXML for JSON handling and manipulation. So much more powerful, so much easier, and it's widely supported and used, which means it has a great community behind it.

If you have dependencies that will be provided from runtime container, you should set scope provided to these dependencies.
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.0.0</version>
<scope>provided</scope>
</dependency>
And remove <artifactSet> section from plugin configuration, execute mvn package then jar with required dependencies will be created.

Maven not including manifest attributes for LWJGL install

I am trying to setup a LWJGL project using Maven. I am using the example "getting started" source code from the official website.
This includes a few lines accessing LWJGL's manifest attributes, such as a simple version check:
System.out.println("Hello LWJGL " + Version.getVersion() + "!");
This runs without any problems in the Eclipse environment (of course after having built the project with Maven), but when running clean install and then running **-jar-with-dependencies.jar through cmd, the following exception get's thrown:
java.lang.NullPointerException
at org.lwjgl.system.APIUtil.apiGetManifestValue(APIUtil.java:97)
at org.lwjgl.Version.getVersion(Version.java:33)
at HelloWorld.run(HelloWorld.java:43)
at HelloWorld.main(HelloWorld.java:130)
This is because the Manifest object created by APIUtil does not include any attributes - but only in the built version by Maven.
Why is this? Is my pom.xml buggy, or is LWJGL 3.0.0 just not ready for this?
This is my pom.xml:
<properties>
<mainClass>HelloWorld</mainClass>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<finalName>${project.artifactId}-${project.version}.jar</finalName>
</properties>
<dependencies>
<dependency>
<groupId>org.lwjgl</groupId>
<artifactId>lwjgl</artifactId>
<version>3.0.0</version>
</dependency>
<dependency>
<groupId>org.lwjgl</groupId>
<artifactId>lwjgl-platform</artifactId>
<version>3.0.0</version>
<classifier>natives-windows</classifier>
</dependency>
<dependency>
<groupId>org.lwjgl</groupId>
<artifactId>lwjgl-platform</artifactId>
<version>3.0.0</version>
<classifier>natives-linux</classifier>
</dependency>
<dependency>
<groupId>org.lwjgl</groupId>
<artifactId>lwjgl-platform</artifactId>
<version>3.0.0</version>
<classifier>natives-osx</classifier>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>${mainClass}</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>

This errors happens because LWJGL 3.0.0 is looking inside the Manifest a property called "Implementation-Version", but when you made the uber-jar, this property was not set.
This is not really an issue with how you made the uber-jar: the Manifest that was created by maven-assembly-plugin looks like:
Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Built-By: Me
Created-By: Apache Maven 3.3.9
Build-Jdk: 1.8.0_102
Main-Class: HelloWorld
You can see it inside META-INF/MANIFEST.MF of the jar-with-dependencies. This file does not have a "Implementation-Version" property. This is normal: when this executable JAR was created, all the MANIFEST of all dependencies were (rightfully) ignored, only to generate one containing the "Main-Class", just so that the JAR is executable.
The uber-jar cannot contain what is inside each of the dependencies manifest. For example, "Implementation-Version" is a property that is present in the manifest of multiple libraries, so which one should it keep? (There can be only one Manifest at the end, in the uber-jar). So the issue comes up because we're making an executable JAR, which can only have 1 Manifest so it cannot aggregate all the properties inside each of the dependencies manifest.
There are 2 possible solutions:
Ignore it. After all, this is not really an error.
Don't make an executable jar by embedding all the dependencies inside a single JAR, but create a ZIP assembly with each dependencies inside a lib folder: this way, each Manifest will be kept. This is done by telling the maven-jar-plugin to add a Manifest entry for the main class with the addition of the classpath and creating a custom assembly descriptor.
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
<configuration>
<archive>
<manifest>
<mainClass>${mainClass}</mainClass>
<addClasspath>true</addClasspath>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptors>
<descriptor>/path/to/assembly.xml</descriptor>
</descriptors>
</configuration>
</plugin>
where /path/to/assembly.xml is the path to the assembly descriptor, relative to the location of the POM, being:
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd">
<id>dist</id>
<formats>
<format>zip</format>
</formats>
<dependencySets>
<dependencySet>
<outputDirectory>lib</outputDirectory>
<useProjectArtifact>true</useProjectArtifact>
</dependencySet>
</dependencySets>
</assembly>
With such a configuration, running mvn clean install will create a ZIP file artifactId-version-dist.zip. Unpacking it and running (replacing <finalName> with the finalName of your JAR)
java -jar lib\<finalName>.jar
will print the version without any issues.

How to create multiplatform executable jar with native libraries for sikulix api using maven pom.xml?

I have problem creating executable single jar dependent on sikulixapi.
Relevant parts from pom.xml
<dependencies>
<dependency>
<groupId>com.sikulix</groupId>
<artifactId>sikulixapi</artifactId>
<version>1.1.0</version>
</dependency>
</dependencies>
I am using maven-assembly-plugin.
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>com.myProject.myClass</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
I am able to produce working jar on windows. jar contains sikulixlibs directory in jar root and everything works OK on windows platform.
But I would like to produce jar working multiplatform. jar created on windows contains only windows subdirectory in sikulixlibs and due to this fact there are missing native binary code when calling in linux.
[error] RunTimeINIT: *** terminating: libs to export not found on above classpath: /sikulixlibs/linux/libs64
How to include all linux/windows/OSX native libraries into produced jar?

I have edited dependecies part od pom.xml, add additional maven artifacts, one for every supported OS, and make maven producing jar including all native libraries.
<dependencies>
<dependency>
<groupId>com.sikulix</groupId>
<artifactId>sikulixapi</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>com.sikulix</groupId>
<artifactId>sikulixlibslux</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>com.sikulix</groupId>
<artifactId>sikulixlibsmac</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>com.sikulix</groupId>
<artifactId>sikulixlibswin</artifactId>
<version>1.1.0</version>
</dependency>
</dependencies>