We Keep Coding

Maven Shade - add local JAR file [duplicate]

I already found an answer here on Stack Overflow how to include a 3rd party JAR in a project without installing it to a "local repository":
Can I add jars to maven 2 build classpath without installing them?
But, when I use the Maven Shade Plugin to create a JAR that includes all the dependencies of the project as well, the 3rd party JAR is not included automatically.
How can I make the Maven Shade Plugin add such a 3rd party JAR in to the shaded JAR?
As per the answer gotten, I made it work. What I did was, added this snippet to the beginning of my pom.xml:
<repositories>
<repository>
<id>repo</id>
<url>file://${basedir}/repo</url>
</repository>
</repositories>
Then added a dependency for my project, also to pom.xml:
<dependencies>
<dependency>
<groupId>dummy</groupId>
<artifactId>dummy</artifactId>
<version>0.0.0</version>
<scope>compile</scope>
</dependency>
</dependencies>
And then ran a command line to add a package to 'repo':
mvn org.apache.maven.plugins:maven-install-plugin:2.3.1:install-file
-Dfile=<my-jar>.jar -DgroupId=dummy -DartifactId=dummy
-Dversion=0.0.0 -Dpackaging=jar -DlocalRepositoryPath=`pwd`/repo/
(Not sure if the repo path needs to be a full path, but didn't want to take chances.)
The contents of the repo subdirectory is now:
repo/dummy/dummy/0.0.0/dummy-0.0.0.jar
repo/dummy/dummy/0.0.0/dummy-0.0.0.pom
repo/dummy/dummy/maven-metadata-local.xml
Now I can check this in to version control, and have no local or remote dependencies.

But, when I use the Maven Shade Plugin to create a JAR that includes all the dependencies of the project as well, the 3rd party JAR is not included automatically.
Yes, because the system scoped dependencies are assumed to be always present (this is exactly what the system scope is about) so they won't be included. People actually don't understand what system scope dependencies are, they just keep abusing them (yes, this is abuse), and then get side effects and wonder why (as Brian pointed out in his answer).
I already wrote many, many, really many times about this here on SO and in 99% of the cases, system scoped dependencies should be avoided. And I'll repeat what the Dependency Scopes mini guide says one more time:
system: This dependency is required in some phase of your project's lifecycle, but is system-specific. Use of this scope is discouraged: This is considered an "advanced" kind of feature and should only be used when you truly understand all the ramifications of its use, which can be extremely hard if not actually impossible to quantify. This scope by definition renders your build non-portable. It may be necessary in certain edge cases. The system scope includes the <systemPath> element which points to the physical location of this dependency on the local machine. It is thus used to refer to some artifact expected to be present on the given local machine an not in a repository; and whose path may vary machine-to-machine. The systemPath element can refer to environment variables in its path: ${JAVA_HOME} for instance.
So, instead of using the system scope, either:
Add your libraries to your local repository via install:install-file. This is a quick and dirty way to get things working, it might be an option if you're alone but it makes your build non portable.
Install and run an "enterprise repository" like Nexus, Archiva, or Artifactory and add your libraries via deploy:deploy-file. This is the ideal scenario.
Setup a file based repository as described in this previous answer and put your libraries in there. This is the best compromise if you don't have a corporate repository but need to work as a team and don't want to sacrifice portability.
Please, stop using the system scope.

The Maven addjars plugin solves this problem - see
http://code.google.com/p/addjars-maven-plugin/wiki/UsagePage

Used <resources> to include my lib with all jars. i.e:
<build>
<resources>
<resource>
<directory>${project.basedir}</directory>
<includes>
<include>lib/*.jar</include>
</includes>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

If you only need a quick and dirty solution, you can add the content of the extracted jar file to your src/main/resource directory.

Maven find dependency having a transitive dependency in a specific version in a remote repository

Edit: "Reverse Dependency analysis" would have been the keyword for what i searched - unfortunately I was not able to proper describe my question (maybe it has something to do with this not beeing a standard thing - i acctually dont even have a real usecase for this).
Say i have a typical convergence issue and the dependency:tree tells me something like this:
[INFO] com.my.group:myProject:jar:1.0.1
[INFO] +- org.not.my.group:a-direct-dependency:jar:1.1:compile
[INFO] | \- org.not.my.group:transitive-dependency-A:jar:1.14.0:compile
[INFO] \- org.not.my.group:another-direct-dependency:jar:1.1:compile
[INFO] \- (org.not.my.group:transitive-dependency-A:jar:1.18.0:compile - omitted for conflict with 1.14.0)
is there a way to analyze a central repository (with nexus webgui available as well) to find out if there is a (newer) version of org.not.my.group:a-direct-dependency:jar that relies on the transitive dependency org.not.my.group:transitive-dependency-A:jar but in version 1.18.0 instead of 1.14.0?
Or generally asked: Can i find out who is depending on a artifact via the central repository (or any remote repository) similar as i would find it out using this locally?:
mvn dependency:tree -Dincludes=org.not.my.group:transitive-dependency-A:jar:1.18.0 -Dverbose
For further clarification: In the above (fictive) szenario i would solve the convergence issue using three "levels" of solutions whereas the first will whenever possible be the choice and the latest is "a dirty correction".
1 - Allign versions: If my dependencies - own or third party - direct or transitive - do rely on the same artifact X but define a different version i preferably try to find a newer release of X or a dependency in the tree to X that has a common version to the rest of the dependencies. This way i assume "a proper upgrade" of X because they may needed to apply code changes.
2 - exclude dependency: If i am not able to find such a artifact i will successivly try to exclude the lower versions in the hope the dependency that has this exclusion can handle the newer version. This requires intense testing since i dont have the guarantee of a newer release that properly points to the version - i basicially tell maven "this dependency will work with another dependency of version as well even thus i have no idear of the internal design of this dependency" - even if compilation works i may still experience runtime issues.
3 - use dependencyManagement: Since using the dependencyManagement can lead to "mask" (hide/outpass) certain convergence issues for the enforcer plugin i acctually dont consider this a solution (for my setup) anymore .... say i experienced a convergence issue and solved it using the dependencyManagement - later - one of the transitive dependencies changes and lead to a similar convergence issue that the enforcer plugin is not able to detect anymore.
Side-Note: I wish my English would be better so it would be easier for me to describe such specific topics ... and in the end easier for you guys to understand me. Thanks for the input i already received :)

First as already mentioned define those dependencies which are used directly instead of relying on the transitive dependencies.
Furthermore i would use maven-enforcer-plugin to prevent such situations.
<project>
...
<build>
<plugins>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>1.4</version>
<executions>
<execution>
<id>enforce</id>
<configuration>
<rules>
<dependencyConvergence/>
</rules>
</configuration>
<goals>
<goal>enforce</goal>
</goals>
</execution>
</executions>
</plugin>
...
</plugins>
</build>
...
</project>

As often my English skills were not good enoth - "Reverse dependency analysis" would have been the term to make myself clear - There are a few Plugins to archive this such as described here http://tech.finn.no/2013/01/31/i-wish-i-knew-my-consumers-maven-reverse-dependency/

Parent pom and microservices

We have several projects that are microservices, every project is independent (running on separate spring boot server, exposing rest services, using separate DB schema...)
We use maven to manage the dependencies.
Is it a good idea to have a parent pom declaring each microservices as modules? And so helping to manage the common dependencies (like the lib servlet-api witch is used in every project, to remove it of all of them and declare it in only the parent pom)

The 'problem' with a multi-module parent pom is that, without complicated profiles, it locks the modules in the same release cycle (assuming you're using the Release Plugin, which you should be).
The way I work with Maven is to have a parent pom that declares:
common dependencies (logging APIs, JUnit, etc).
common plugins.
all dependencies in the dependencyManagement section.
all plugins in the pluginManagement section.
Each module delcares the parent pom as its parent but the parent knows nothing about the modules.
The benefit of this comes from the last to two bullets above, the 'management' sections. Anything contained in a 'management' section needs to be redeclared in a module that wants to use a particular dependency or plugin.
For example the parent might look like this:
<project>
<groupId>com.example</groupId>
<artifactId>parent</artifactId>
<version>1.0.00-SNAPSHOT</version>
...
<dependencies>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.7</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
</dependencies>
<dependencyManagement>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.6</version>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>2.1</version>
</dependency>
</dependencyManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugins>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<configuration>
<appendAssemblyId>false</appendAssemblyId>
<descriptors>
<descriptor>src/main/assembly/assembly.xml</descriptor>
</descriptors>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
</project>
And the module might look like this:
<project>
<parent>
<groupId>com.example</groupId>
<artifactId>parent</artifactId>
<version>1.0.00-SNAPSHOT</version>
</parent>
<groupId>com.example</groupId>
<artifactId>module</artifactId>
<version>1.0.00-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
</dependency>
</dependencies>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
</plugin>
</plugins>
</project>
The module would:
have dependencies on org.slf4j:slf4j-api:1.7.7:compile, junit:junit:4.11:test and commons-lang:commons-lang:2.6:compile.
has the plugin org.apache.maven.plugins:maven-assembly-plugin:2.4

I would avoid dependencies in the parent pom. It's awkward if one of your (independent) microservices would want some other things. It's weird to have the parent know of each microservice.
You can stick with dependencyManagement though to suggest default versions/scopes if you want. A parent pom is, non the less, very convenient to define plugins, repositories and the like.
Instead, I would group a common set of dependencies into a specific artifact(s), that may be only a single pom with dependencies. Then you can depend on, say "com.example/common-db-dependencies/1.2" to include a standard set of database dependencies, like hibernate, apache derby and JPA specs. (or whatever you're using). A service does not use JPA/SQL could avoid that dependency all together.
Don't entangle yourself though. It's easy to overwork dependency structures if you're trying to cover each case. So, only try to standardize things that really get used by a majority of the services.

I would definitely use a parent project.
I've been working for years with both the structures...Microservices and not, modular or not, Ant, Maven and Gradle..
We need to understand that using a parent pom does not mean talk about microservices not coupled and independent:
they can be still independent and not coupled using parent pom,
they can be still built release and updated in isolation even if you are using a parent pom.
I heard saying "a microservice may need to use different versions for a dependency", well you can, just override the dependency in the specific microservice pom.
We need to focus on "What are here the benefit and what are the cons":
Control and standardization: I can manage the common dependencies (with the dependencies management) in a single point, it makes easier to roll out dependencies changes across all the modules, yes we may need different third parties version, but same time we need to avoid losing control over all the dependencies, so exceptions may be allowed but they needs to be balanced with the "standardization"
Group management: I can still release just a single module, but I can also manage multi modules releases in a easier way, without having to release module by module, but simply the modules that are under development, in this case I still have a single entry point and all the common dependencies can be overviews withing the parent
And much more:
common third parties and platform dependencies management
common third parties and platform standardization
Full control of the dependencies ecosystem withing the whole application (structured in micro services)
common plugins management and standardization
reduce duplication and redundant logic.
Configurations management and standardization
Easier maintenance, change in one place instead of potentially 30 places!!
easier to test and roll out common change.
What about the cons?
I don't see any for the moment, as exceptions can be managed through overriding common behaviour in the specific microservices pom, I can still manage anything in isolation (build in isolation, release in isolation, deploy in isolation..)
There is nothing coupled
Not sure yet what we mean with "it locks the modules in the same release cycle" It does not, unless you are using external SNAPSHOT, I can release a microservice in isolation re-using the same parent version.
for example I can have module 1 declaring Parent 1.0 and be released in isolation without having to run the release process from the parent, I can run it directly on the submodule, but I need to not declare any external SNAPSHOT within the submodule project (you would have same issues with or without parent)

Here there is one issue with dependency and dependency management. Say one of your micro service wants to upgrade to newer version of common for some reason...you cant do that as you have parent. I do understand temptation of reducing duplication of redundant things like plugin configuration. In micro service we need to think more about independence of each service.
Some config like say your repository or release configuration etc can be common.

Most books on microservice architecture recommend autonomy as a principle. Using a parent pom violates that principle.
First of all with a parent pom you can no
longer adopt a polyglot approach and write your microservices in different languages.
You'll also be forced to use the dependencies prescribed by the parent, especially if the enforcer plugin is employed.
The microservices will no longer be independently deployable.
There is also the risk that your work on any one microservice may break others if that work involves altering the parent.

A major drawback of using a parent pom approach with microservices is it will make the release management for microservices a slightly tricky affair. Few related pointers -
The parent pom should not be frequently changed, should be managed as a separate project in a separate repo.
Every change to the parent pom should increment the parent pom version. Once the changes are finalized, the parent pom repo should also be tagged. (treating is as a separate library with independent releases)
Moreover the child pom of all the microservices being touched should ideally be updated to point to the latest parent pom version (affecting the autonomy of microservices to some extent). This may also lead to forceful ask of upgrading the microservice to use newer versions of the libraries, which may not always be a feasible option.
Even if the only change in a microservice is to point to the new parent pom version, it would call for a new (mostly minor) version release of the service.
Suggestions -
You can use the maven enforcer plugin to check for duplicate dependency versions specified between parent and child poms.
The parent pom will not be a good option for extensive dependencies and dependency-management, but can certainly be used for things like repositories, distribution management, and plugin management which shall generally not have clashes between microservices.

Maven: Different library versions in one JVM

I have a project whose dependency tree is huge i.e. it packs in modules from several teams.
Now there are some commonly used dependencies which are common across several modules.
A simplified example can be:
TopModule.jar
ChildModule.jar
CommonModule-v1.jar
CommonModule-v2.jar
When I build my project, I specify the latest version of common dependencies, but its very hard to ask the same from every other team.
So, frequently, the TopModule is built using different versions of CommonModule (v1 and v2 in the above example).
My question is:
If the final jar file contains both CommonModule-v1.jar and CommonModule-v2.jar, how does it affect the runtime?
Can the runtime erroneously load versions v2 where v1 is required and vice versa?

Maven will only use one version of each artifact in the end -- it doesn't do any fancy classloader isolation tricks. You can see which version it'll use with mvn dependency:resolve.
If you need to use specific versions within dependencies, you can use the shade plugin. It'll do renaming trickery so that dependencies get their own versions of libraries.

To fight with this problem globally use this DependencyConvergence Rule
This rule requires that dependency version numbers converge. If a
project has two dependencies, A and B, both depending on the same
artifact, C, this rule will fail the build if A depends on a different
version of C then the version of C depended on by B.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>1.3.1</version>
<executions>
<execution>
<id>enforce</id>
<configuration>
<rules>
<DependencyConvergence/>
</rules>
</configuration>
<goals>
<goal>enforce</goal>
</goals>
</execution>
</executions>
</plugin>
After this all teams work together with consistent versions of dependencies.

This depends on the way the modules are named in maven. Usually, maven tries to resolve the conflicting libs and takes the highest version into the tree. But if the libraries are different artifacts in terms of artifactId, then maven will not see that they are from the same breed and thus will not resolve the ambiguity.
Usually you resolve this by a common parent.pom, where you define the versions of commonly used libraries throughout the project. If you have no control over the other projects (not part of your build, only dependencies), you may be lucky to have your final project working fine. If the library breaks compatibility in the newer version, you will not be able to use it.
So, does your final project contain both versions of the library or not, did you check it? The dependency tree may show both versions, but if maven will use only the latest version of a dependency in the hierarchy.

Classloader will load the first JAR which appears on your classpath. In more details - it will search for the first class on your class path, so in each case all these searches would fall into i.e. CommonModule-v2.jar. So the answer is yes - it can erroneously load versions v2 where v1 if it appears earlier on your classpath.

If your pom.xml is only an aggregator of already packaged modules then this apply. If it is not the case and your project actually compile and packages all of those modules as a submodules then maven will choose one. If it compiles every project on its own then it will be packaged using that dependency. But if all of them end up in the same class loader then it won't work fine.
At runtime it can cause errors, think of method not found and the like. Your byte code classes were compiled and linked with the correct dependencies but since the class loader finds two candidates it just load one at runtime.
What you can do is set a parent pom defining a <dependencyManagement> and then ask all teams to use it as a parent and don't declare <version> but inherit it from the parent pom.xml.
References http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management

On top of what #yshavit said, ideally you'd exclude the earlier version of the CommonModule so that only v2 is in the classpath. This is only possible if the CommonModule v2 api is backwards compatible with CommonModule v1.
Here's an example of how you exclude:
<dependency>
<groupId>ChildModuleGroupid</groupId>
<artifactId>ChildModuleArtifactid</artifactId>
<version>1.0</version>
<exclusions>
<exclusion>
<groupId>CommonModuleGroupId</groupId>
<artifactId>CommonModuleArtifactId</artifactId>
</exclusion>
</exclusions>
</dependency>
You'd put that in the TopModule pom.xml.

Maven dependency resolution (conflicted)

Let's say I have four projects:
Project A (has a dependency on B and D)
Project B (has a dependency on D)
Project C (has a dependency on D)
Project D
In this scenario if I run project A, Maven will correctly resolve the dependency to D. If I understand this correctly Maven always takes the dependency with the shortest path. Since D is a direct dependency of A it will be used rather then, the D which is specified within B.
But now assume this structure:
Project A (has a dependency on B and C)
Project B (has a dependency on D)
Project C (has a dependency on D)
Project D
In this case the paths to resolving D have the same depth. What happens is that Maven will have a conflict. I know that it is possible to tell Maven that he should exclude dependencies. But my question is how to address such kind of problems. I mean in a real world application you have a lot of dependencies and possibly a lot of conflicts as well.
Is the best practice solution really to exclude stuff or are there other possible solutions to this? I find it very hard to deal with when i suddenly get a ClassNotFound Exception because some versions have changed, which caused Maven to take a different dependency. Of course, knowing this fact makes it a little bit easier to guess that the problem is a dependency conflict.
I'm using maven 2.1-SNAPSHOT.

The maven way of resolving situations like this is to include a <dependencyManagement> section in your project's root pom, where you specify which version of which library will be used.
EDIT:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>foo</groupId>
<artifactId>bar</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</dependencyManagement>
Now no matter which version of library foo:bar is requested by a dependency, version 1.2.3 will always be used for this project and all sub-projects.
Reference:
Dependency Management

Maven can handle both situations without any conflict. Conflicts will exist when two versions of a transitive dependency are required. The ClassNotFoundException you describe results from the app (or a dependency) attempting to use a class not available in the version of the conflicted dependency that actually gets used.
There are multiple ways to fix the problem.
Update the versions of the libraries you are using that depend on the conflicted dependency, so that they all depend on the same version version of that dependency
Declare the conflicted dependency as a direct dependency of your project with the version you want to be included (in the example, the one with the missing class included in it)
Specify which version of the conflicted dependency that transitive dependencies should use, via the <dependencyManagement> section of the POM
Explicitly exclude the unwanted versions of the conflicted dependency from being included with the dependencies that rely on them using an <exclusion>

This is fundamentally not a maven issue, but a java issue.
If Project B and Project C needs two incompatible versions of project D, then you can't use them both in Project A.
The Maven way of resolving conflicts like these is unfortunately, as you already know, to choose which ones to exclude.
Using mvn dependency:analyze and mvn dependency:tree helps in finding what conflicts you have.

You can enforce consistent dependencies in whole project with rule Dependency Convergence.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>1.3.1</version>
<executions>
<execution>
<id>enforce</id>
<configuration>
<rules>
<DependencyConvergence/>
</rules>
</configuration>
<goals>
<goal>enforce</goal>
</goals>
</execution>
</executions>
</plugin>

One possible strategy is to specify for main project, what version of D to use (the newest one f.g.). However, if library D is not backward-compatible, you have a problem as stated by kukudas - it's impossible to use both libaries in your project.
In such situation there may be necessary to use either B or C in older version, so that both would depend on compatibile versions of D.