We Keep Coding

How to reduce jar size while using maven-assembly-plugin in pom.xml for building spark application

I am using the below plugin: maven-assembly-plugin in my pom.xml file with many other separate dependencies mentioned in the same file as below to build a jar file for my spark application using IntelliJ:
pom.xml
Maven-assembly-plugin
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.3</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id> <!-- this is used for inheritance merges -->
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
pom.xml
other dependencies:
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-emr</artifactId>
<version>1.11.125</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.11.106</version>
</dependency>
to name a few..............!
What's happening is, this maven-assembly-plugin is bundling in all the dependency jars together with the main jar file and the size of the final jar file goes beyond 150+ Mb.
Is there a way in which I can mention only a particular dependency from my pom.xml to be included in the final jar file and excluding all the other dependencies??
With only this dependency included in the jar, hopefully, the final jar size should come down below 1 Mb.
The specific dependency which I want to include in my final jar is the below one:
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20200518</version>
</dependency>

How to include libraries/dependencies when creating a jar file?

I created a Confluence plugin(A Java application) which has Maven on it and includes some dependencies in the pom.xml as follows: (It needs to use the Google Client Library)
<dependencies>
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-calendar</artifactId>
<version>v3-rev254-1.22.0</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client</artifactId>
<version>1.22.0</version>
<scope>compile</scope>
</dependency>
..... Skip .....
</dependencies>
I also downloaded the Google Client Library and created a "libs" folder at the "src/main/resources/" path in this maven project to store them, and added them as jars in Eclipse as follows:
However, after executed "atlas-debug" to invoke a Confluence instance or "atlas-package" commands, the final exported jar file usually does not include the dependencies/libraries (I found this according to the failed jar file size, it is much smaller than the successful one).
How to make the library files really be included into the exported jar file every time I executed "atlas-debug" or "atlas-package" commands?

You can use the maven-assembly-plugin plugin that will package all your dependency in the jar. You can configure it in the plugins section under the build section in your pom.xml:
<build>
...
<plugins>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2.2</version>
<executions>
<execution>
<id>assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
</archive>
</configuration>
</execution>
</executions>
</plugin>
...
</plugins>
...
</build>
Remember that the dependency configured with <scope>provided</scope> won't be included in the jar.

dbcp jar added automatically into project

I'm working on a maven based web project. In one module that generates .war file i have some dependencies in POM file, some jars are added to WEB-INF/lib folder. i don't have added jar: commons-dbcp-1.3.jar into POM nor in lib folder, but when i build my project using maven, commons-dbcp-1.3.jar added .war file also i can view it to lib folder in target directory.
Can anyone help me to explain how this jar is being added to war or lib folder in target directory.
I have also checked the "build path" and this jar is not added as an external jar.
I'm using Eclipse(Indigo).
Here is the POM file.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.company</groupId>
<artifactId>rpt</artifactId>
<version>4.0.2</version>
<packaging>war</packaging>
<dependencies>
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-web-api</artifactId>
<version>6.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.struts</groupId>
<artifactId>struts-core</artifactId>
<version>1.3.5</version>
</dependency>
<dependency>
<groupId>com.lowagie</groupId>
<artifactId>itext</artifactId>
<version>2.1.4</version>
</dependency>
<dependency>
<groupId>com.itextpdf</groupId>
<artifactId>itextpdf</artifactId>
<version>5.4.2</version>
</dependency>
<dependency>
<groupId>struts</groupId>
<artifactId>struts</artifactId>
<version>1.1</version>
<optional>false</optional>
</dependency>
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20131018</version>
</dependency>
<dependency>
<groupId>com.mind</groupId>
<artifactId>mind-common-framework</artifactId>
<version>1.3.1</version>
<optional>false</optional>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.16</version>
<optional>false</optional>
</dependency>
</dependencies>
<organization>
<name>OrgName</name>
</organization>
<build>
<defaultGoal>package</defaultGoal>
<finalName>${project.artifactId}</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>2.1.2</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
<configuration>
<finalName>${project.artifactId}-${project.version}-r${prefix.revision}</finalName>
<attach>false</attach>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.1.1</version>
<configuration>
<manifest>
<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
</manifest>
<archive>
<manifestEntries>
<Implementation-Build>${project.version}</Implementation-Build>
<Build-Time>${maven.build.timestamp}</Build-Time>
</manifestEntries>
</archive>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<targetJdk>1.7</targetJdk>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<maven.build.timestamp.format>yyyy/MM/dd HH:mm</maven.build.timestamp.format>
</properties>
</project>

This jar can possibly be one of the dependencies for running your project and Maven automatically downloads and packages the required dependencies(jars) for the project at the time of compilation and packaging.
DBCP commons jar is provided by Apache for maintaining the pool of connection objects . It is basically used in the applications involving database connections where many users can simultaneously try to access the DB(eg: Web Base Applications involving DB operations) to improve the performance of the application and the system.
If you don't wish to bundle this jar with your package and such a capability will be provided by the web container where your application will be deployed(connection pooling), you can set the scope of this dependency to be 'provided'.

The scope you should use for this is "provided". This indicates to Maven that the dependency will be provided at run time by its container or the JDK, for example.
Dependencies with this scope will not be passed on transitively, nor will they be bundled in an package such as a WAR, or included in the runtime classpath.
https://maven.apache.org/general.html#scope-provided

How to create multiplatform executable jar with native libraries for sikulix api using maven pom.xml?

I have problem creating executable single jar dependent on sikulixapi.
Relevant parts from pom.xml
<dependencies>
<dependency>
<groupId>com.sikulix</groupId>
<artifactId>sikulixapi</artifactId>
<version>1.1.0</version>
</dependency>
</dependencies>
I am using maven-assembly-plugin.
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>com.myProject.myClass</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
I am able to produce working jar on windows. jar contains sikulixlibs directory in jar root and everything works OK on windows platform.
But I would like to produce jar working multiplatform. jar created on windows contains only windows subdirectory in sikulixlibs and due to this fact there are missing native binary code when calling in linux.
[error] RunTimeINIT: *** terminating: libs to export not found on above classpath: /sikulixlibs/linux/libs64
How to include all linux/windows/OSX native libraries into produced jar?

I have edited dependecies part od pom.xml, add additional maven artifacts, one for every supported OS, and make maven producing jar including all native libraries.
<dependencies>
<dependency>
<groupId>com.sikulix</groupId>
<artifactId>sikulixapi</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>com.sikulix</groupId>
<artifactId>sikulixlibslux</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>com.sikulix</groupId>
<artifactId>sikulixlibsmac</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>com.sikulix</groupId>
<artifactId>sikulixlibswin</artifactId>
<version>1.1.0</version>
</dependency>
</dependencies>

Maven build has missing package linkages

I am having problems using Maven. I have an Apache Flink project and wanted to run it on my server. Locally it runs fine but on the server it aborts with the error:
java.lang.NoClassDefFoundError: org/apache/flink/examples/java/ml/util/LinearRegressionData
In my Java project I imported the class with
import org.apache.flink.examples.java.ml.util.LinearRegressionData;
And I used the correct class at the import:
After the build I had a look into the Jar file. The following classes were included:
The /util/ folder is completely missing. My dependency-section in the pom file looks like the following:
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>0.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-core</artifactId>
<version>0.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients</artifactId>
<version>0.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table</artifactId>
<version>0.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-runtime</artifactId>
<version>0.9.0</version>
</dependency>
</dependencies>
When I have seen the package organization in the repository located at https://github.com/apache/flink/tree/release-0.9, I thought it would be possible to add the following lines to flink:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-examples</artifactId>
<version>0.9.0</version>
</dependency>
But these dependencies cannot be resolved. Since Maven does not throw an error when performing a clean install, I think this is a dependency issue. I thought Maven would include all used imports automatically. How can I make this runnable on my server?

You should include ML example as follows:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java-examples</artifactId>
<version>0.9.0</version>
</dependency>
flink-examples is a parent pom module; not a jar module. How do you build your jar file? Using maven-jar-plugin? A regular mvn package or mvn install does not packages dependencies.
Using maven-jar-plugin requires to specify what you need to include using <include>. See here: https://maven.apache.org/plugins/maven-jar-plugin/examples/include-exclude.html
In your case it should be something like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifestEntries>
<Main-Class>org.apache.flink.examples.java.ml.LinearRegression</Main-Class>
</manifestEntries>
</archive>
<includes>
<include>org/apache/flink/examples/java/ml/*.class</include>
<include>org/apache/flink/examples/java/ml/util/*.class</include>
</includes>
</configuration>
</plugin>
You can also compare with https://github.com/apache/flink/blob/master/flink-examples/flink-java-examples/pom.xml
You also need to "pull" and unpack your dependencies into your project such that they can be re-packages using maven-dependency-plugin
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.9</version><!--$NO-MVN-MAN-VER$-->
<executions>
<execution>
<id>unpack</id>
<phase>prepare-package</phase>
<goals>
<goal>unpack</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java-examples</artifactId>
<version>0.9.0</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>${project.build.directory}/classes</outputDirectory>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>