We Keep Coding

maven shade plugin override a file

I am using maven shade plugin to create a fat jar. I want to override 1 properties file that is coming from a jar i don't control. I want to update that properties file before creating the shaded jar. I tried using transformers configuration but when i am trying to override I am getting error
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (default) on project dsp-santa-breakbox: Error creating shaded jar: duplicate entry: assets/components/hystrixCommand/hystrixCommand.js -> [Help 1]
Here is my shade plugin configuration
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.4</version>
<executions>
<execution>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer">
<resource>assets/components/hystrixCommand/hystrixCommand.js</resource>
<file>src/main/resources/hystrixCommand.js</file>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
Is there a way to achieve this?

i had to filter the file from other artifact,
Here is how the final configuration looked like
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.4</version>
<executions>
<execution>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>com.yammer.breakerbox:breakerbox-service</artifact>
<excludes>
<exclude>assets/components/hystrixCommand/hystrixCommand.js</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer">
<resource>assets/components/hystrixCommand/hystrixCommand.js</resource>
<file>src/main/resources/hystrixCommand.js</file>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>

Override file in META-INF

In my project, I have a dependency whose Jar-file contains the file META-INF/org/languagetool/language-module.properties. For a particular reason, I want to replace this file entirely with my own version. I googled and found out that I can add files to META-INF by simply creating the file as such:
src/resources/META-INF/org/languagetool/language-module.properties
This works, as long as the file I wish to add and the file that already exists have different names. But if they have the same name, the maven-assembly-plugin uses the file from the dependency instead of using my file. How can I fix that?
Configuration of the maven-assembly-plugin:
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.3.0</version>
<configuration>
<archive>
<manifest>
<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
<addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
<mainClass>com.github.vatbub.autoHotkeyNounReplacer.CreateAutoHotkeyScriptKt</mainClass>
</manifest>
<manifestEntries>
<Implementation-Build>20210702123553</Implementation-Build>
<Custom-Implementation-Build>20210702123553</Custom-Implementation-Build>
</manifestEntries>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>

I found the answer myself: The maven shade plugin allows much finer control over how the assembled jar is created. I hence replaced the maven-assembly-plugin with the maven-shade-plugin with the following configuration:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
<configuration>
<filters>
<filter>
<artifact>org.languagetool:*</artifact>
<excludes>
<exclude>META-INF/org/languagetool/language-module.properties</exclude>
</excludes>
</filter>
</filters>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>jar-with-dependencies</shadedClassifierName>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Main-Class>${mainClass}</Main-Class>
<!-- custom manifest entries -->
</manifestEntries>
</transformer>
</transformers>
</configuration>
</plugin>

Maven exec works, but java -jar does not

I have simple maven project with only one dependency. I can install it and run it command-line via Exec Maven Plugin:
mvn exec:java -D"exec.mainClass"="com.MyClass"
After packaging maven generates a .jar file in my directory. With a help of Maven JAR Plugin I made its manifest to know my main method class. It looks like:
...
Created-By: Apache Maven 3.3.1
Build-Jdk: 1.8.0_66
Main-Class: com.MyClass
Now I want to run this .jar file like regular java executable using java command, but after doing the following:
java -jar myFile.jar
it gives an error java.lang.NoClassDefFoundError concerning my only dependency.
How can I make maven to add all dependencies into my executable jar file?

One way to achieve this is using Apache Maven Shade Plugin:
This plugin provides the capability to package the artifact in an
uber-jar, including its dependencies and to shade - i.e. rename - the
packages of some of the dependencies.
This plugin has some advantages for large project with many dependencies. This is explained here: Difference between maven plugins ( assembly-plugins , jar-plugins , shaded-plugins)
In my project I use it with this configuration:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.2</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.xxx.tools.imagedump.ImageDumpLauncher</mainClass>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.handlers</resource>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.schemas</resource>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>

You could use the Apache Maven Assembly Plugin, in order to create a jar with all its dependencies, so your pom.xml should be like the following:
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>mypackage.myclass</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
I hope it helps you, bye.

Uber jar not reading external properties files

I am creating an uber jar i.e. jar with dependencies for my project. I have a bunch of properties files that the project uses. I want to be able to change these properties files before running my project so i want them to be outside of the jar. Here is the relevant sections of my pom
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2</version>
<configuration>
<artifactSet>
<excludes>
<exclude>**/*.properties</exclude>
<exclude>**/*.json</exclude>
</excludes>
</artifactSet>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>path.to.main.Main</mainClass>
</manifest>
<manifestEntries>
<Class-Path>.</Class-Path>
<Class-Path>conf/</Class-Path>
</manifestEntries>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<version>2.4</version>
<executions>
<execution>
<id>copy-resources</id>
<phase>install</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>${basedir}/target/conf</outputDirectory>
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.properties</include>
<include>**/*.json</include>
</includes>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
so essentially, I want to create a folder ${basedir}/target/conf and copy all the .properties and .json files to it. Also, here is how I am reading the files
InputStream in = this.getClass().getClassLoader().getResourceAsStream("filename.properties");
I am facing a couple of problems
When i do mvn clean install, i still see the all the .properties and .json files in the classes folder. Shouldn't they have been excluded?
The conf folder is created with all of the files, but when I run the jar adn try to change the properties, the changes are not picked up. How can i ensure that the conf folder is being added to the classpath?
I want to be able to load the .properties and .json files from the src/main/resources folder while i am developing so i dont want to put them in a separate folder. Is this possible?

I was facing the same issue where Uber jar is not reading the external configuration file.
I tried below configuration and it worked like charm. Refer below configuration it may help someone having the issue with uber jar not reading extenarl files.
I am not sure if this is the best way but haven't found any soultion online :)
I have included the resources using IncludeResourceTransformer.
Using filter removed the properties file from uber jar.
In classpath /conf reading the properties from external folder.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions> <!-- Run shade goal on package phase -->
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
add Main-Class to manifest file
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Main-Class>JobName</Main-Class>
<Class-Path>conf/</Class-Path>
</manifestEntries>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer">
<resource>src/main/resources/config.properties</resource>
<file>${project.basedir}/src/main/resources/config.properties</file>
</transformer>
</transformers>
<finalName>FinalJarName</finalName>
<filters>
<filter>
<artifact>groupId:artifactId</artifact>
<excludes>
<exclude>**/*.properties</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
good luck.

Run map reduce job with 3rd party dependency in Jar

I am trying to test if I can jar 3rd party dependencies into my jar, SSH the jar over to a remote machine, and then run a map reduce job.
Process:
With my project, I run mvn clean package and that produces the files my-appy-1.0-SNAPSHOT.jar and original-my-appy-1.0-SNAPSHOT.jar. I scp the first file over to my remote machine and run the command:
hadoop jar hadoop my-appy-1.0-SNAPSHOT.jar /user/bli1/wordcount/input /user/bli1/wordcount/output
I also tried:
hadoop my-appy-1.0-SNAPSHOT.jar WordCount /user/bli1/wordcount/input /user/bli1/wordcount/output
I'm not sure why I am getting this error:
Error: Could not find or load main class my-appy-1.0-SNAPSHOT.jar
pom.xml:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.2</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<artifactSet>
<excludes>
<exclude>org.hamcrest:*</exclude>
<exclude>org.mockito:*</exclude>
<exclude>org.objenesis:*</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/LICENSE</exclude>
<exclude>META-INF/license</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Main-Class>com.mycompany.app.WordCount</Main-Class>
<Build-Number>1</Build-Number>
</manifestEntries>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

Open your *.jar file and check it's MANIFEST.mf file. It should contain the "Main-Class" row with your class with "public static main()" method. If there is no "Main-Class" add it yourself
This link will help you do this: https://docs.oracle.com/javase/tutorial/deployment/jar/appman.html

you need to give full path to main class including package name.
hadoop my-appy-1.0-SNAPSHOT.jar com.mycompany.app.WordCount /user/bli1/wordcount/input /user/bli1/wordcount/output

The command you run through jar file is as below:
hadoop jar jar_filename.jar package_name.class_name HDFS_inputfile_name HDFS_output_directory
Your command should be
hadoop jar my-appy-1.0-SNAPSHOT.jar com.mycompany.app.WordCount /user/bli1/wordcount/input /user/bli1/wordcount/output