We Keep Coding

How to pin transitive dependency versions across multi-pom Maven project

I'm working on a large Java codebase that's split into multiple modules, each with a separate pom.xml, all parented by a top-level pom.xml.
I'm currently in the process of bringing in a couple of library dependencies. The transitive set of dependencies is large, and as luck would have it, there are conflicting dependency versions for different modules.
Here's a simplification of my situation:
project/pom.xml
/module-a/pom.xml # references library-a, depends on library-c:v1
/module-b/pom.xml # references library-b, depends on library-c:v2
/module-c/pom.xml # references module-a and module-b
Now the unit tests for module-a will exercise library-a in the presence of library-c:v1, while module-b will exercise library-b in the presence of library-c:v2.
The trouble is that module-a and module-b need to live together on the same classpath when module-c is deployed, but whatever version of library-c is chosen when module-c is packaged, at least one combination of libraries hasn't been unit tested!
I'd like to pin the version of library-c at the parent pom level somehow, rather than repeating myself in every module that transitively depends on library-c; ideally it would be added in such a way indicating that it's a transitive dependency that is allowed to go away should library-a and library-b no longer rely on it.
I'd like a guarantee that there is exactly one version selected for
every transitive dependency across the entire project rooted from this parent pom, and I'd like the build to blow up if this isn't true. I wrote a tool to parse the output of mvn dependency:tree (turning the leaves of the tree into a forest of paths from leaf to root, then finding all different versions of leaf with the dependency path) so I can see the problem, but without explicitly resolving the transitive dependencies for every conflict and bloating out poms with redundant declarations, this doesn't seem fruitful. It's what I'll do if I have no alternative, naturally.
What's the best way to handle this transitive dependency conflict problem with Maven?
How severe is this problem? Quite apart from getting unconvincing test coverage, in practice I see JVM-killing NoSuchMethodError at runtime from the wrong versions getting deployed. I'd prefer to see these at test time at the very least.

Looks like there are two aspects to this:
You need to insist on a single version of a dependency, whether it is declared explicitly or acquired transitively
You can use <dependencyManagement/> here. For example in the top-level pom.xml you can pin the version of library-c:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>your.group.id</groupId>
<artifactId>library-c</artifactId>
<version>2</version>
<dependency>
<dependencies>
<dependencyManagement>
And then in library-a, library-b you would declare the dependency on library-c as follows:
<dependencies>
<dependency>
<groupId>your.group.id</groupId>
<artifactId>library-c</artifactId>
<dependency>
<dependencies>
By declaring this dependency in the parent's dependencyManagement you are insisting on both of the child modules using the version declared in the parent.
You want to protect yourself from unhappy dependency additions occurring in future
You can use the Maven Enforcer plugin here, specifically the dependencyConvergence rule. For example:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>3.0.0-M1</version>
<executions>
<execution>
<id>enforce</id>
<configuration>
<rules>
<dependencyConvergence/>
</rules>
</configuration>
<goals>
<goal>enforce</goal>
</goals>
</execution>
</executions>
</plugin>
The enforcer can be configured to either fail or warn if it discovers a non convergent dependency.

Why is Maven including multiple versions of the same dependency?

I have a Maven java web app (.WAR) project that includes several libraries, including the Wicket libraries (but I don't think the problem is wicket itself, but rather with maven).
Here's the problem: even tho I only include Wicket 6.20.0, the resulting .WAR contains two copies of the Wicket libraries: 6.20.0 and 6.18.0, as you can see in this screenshot:
Thinking of some conflicting imports I printed the dependency tree using the:
mvn dependency:tree
commnad... but there is no mention of Wicket 6.18.0 in the dependency tree! I also double-checked using Eclipse's "dependency hierarchy" view and I can confirm there's no trace of that import.
I even did a search for string "6.18.0" across the entire workspace with Eclipse, but it's nowhere to be found!
How can I find out what is causing the inclusion of that duplicate version of the library?

Maven doesn't work in this way.
The resolution of more than one dependency with the same artifactId and groupId but with a different version will result to a single dependency (the version used is no determinist).
The presence of two artifacts with the same artifactId and groupId but with two distinct versions in a same lib folder of the WAR is probably related to one of these :
you don't execute mvn clean package but only mvn package.
your use a bugged version of the Maven war plugin. Try to update it to check that.
you have a Maven plugin that copies Wicket jars 6.18.0 in the WEB-INF/lib folder of the target folder during the build of the component.
the maven WAR project you are building has as dependency an artifact of type WAR. In this case, the dependencies of the WAR dependency are so overlaid in the WAR project that you are building.
An interesting Maven issue about duplicated JAR because of WAR dependencies :
JARs with different versions can be in WEB-INF/lib with war as dependencies
Your answer and your comment indicate that actually you have a WAR dependency in your build.
Unfortunately, there is not really a good and long term effective solution to bypass this limitation.
As said in my comment, using the packagingExcludes property of the maven war plugin is a valid workaround for the actual issue :
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.4</version>
<configuration>
<!-- ... -->
<packagingExcludes>WEB-INF/lib/wicket-*-6.18.0.jar</packagingExcludes>
</configuration>
</plugin>
But beware, using that will do your build less robust through the time.
The day where you update the version of the WAR dependency and that in its new version, it pulls again a different version of wicket, you have still a risk to have duplicate jars with two distinct versions in your built WAR.
Using the overlay feature by specifying the overlay element of the maven-war-plugin is generally better as it focuses on the overlay applied for the war dependency. It fixes the problem early.
As a result, you could define to exclude any wicket JARs from the WAR dependency :
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<version>2.4</version>
<artifactId>maven-war-plugin</artifactId>
<configuration>
<overlays>
<overlay>
<groupId>com.whatever.youlike</groupId>
<artifactId>myArtifact</artifactId>
<excludes>
<exclude>WEB-INF/lib/wicket-*.jar</exclude>
</excludes>
</overlay>
</overlays>
</configuration>
</plugin>
This way is better but this is still a workaround.
The day where the dependency WAR is updated and that it pulls new dependencies (other than Wicket) that are declared in your actual build but with different versions, you may finish with the same kind of issue.
I think that declaring a dependency on a WAR artifact should be done only as we don't have choice.
As poms and projects refactoring are possible, introducing a common JAR dependency which the two WARs depend on and that contains only common sources and resources for the two WARs makes really things simpler.

Well, I figured it out while poking around.
I had a dependency of type "war" in the project:
<dependency>
<groupId>com.whatever.youlike</groupId>
<artifactId>myArtifact</artifactId>
<version>1.0.7-SNAPSHOT</version>
<type>war</type>
</dependency>
Apparently (I wasn't aware of this, my fault here) these type of dependencies will include themselves in the classpath by copying all libs to the main WAR /libs folder, but these will NOT show app in the dependency tree / dependency hierarchy.
I solved by configuring an explicit exclusion in the WAR plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.4</version>
<configuration>
<!-- ... -->
<packagingExcludes>WEB-INF/lib/wicket-*-6.18.0.jar</packagingExcludes>
</configuration>
</plugin>

Use clean install and the double dependency will probably be gone.

Because other libs can use same libs but different version or you tried different version and didn't make mvn clean

The command mvn dependency:tree is telling you the correct information - what you are looking at here is an eclipse / build issue.
Clear out all the target and build areas in your project. If need be, check it out from source control to a new folder.
Alternatively you can build your project in IntelliJ IDEA, and see if you get the correct dependencies (most likely you will).

Circular dependencies in Maven and horrible structure of project

I have a problem with maven. I have a huge multi-module project with limited possibilities to change the structure of it.
So, let's say I have 3 modules: A,B,C.
A is a parent of B and C.
C uses classes from B, so we have a B as a dependency in C's POM file.
B doesn't need to have a C as a dependency to be successfully compiled.
For now, there aren't any problems.
Unfortunately, B uses C during the runtime (spring, ioc, ...), so someone added C as B's dependency, so we have a horrible cycle in Maven. Build finishes with failure (something like "cycle detected" in log).
I would like to keep it this way (provide somehow C dependency in B module) as I need to compile and deliver B with all needed JAR archives (including JARs from C).
Can I somehow build C and copy its JARs to B's target directory after the full compilation of B? Is there a plugin or tool which can be used by maven to do this?
If this post is not clear, I will try to describe it in more details.
Thanks in advance ;)

It sounds like there are two issues:
How to tell Maven about a runtime-only dependency between artifacts.
How to get Maven to leverage the dependency to pull together all the relevant dependencies.
Maven has a feature of dependency management that allows you to inform maven of such nuances using the "scope" element of the dependency element. In this case, I think you want
<scope>runtime</scope>
See the documentation. In particular:
runtime - This scope indicates that the dependency is not required for compilation, but is for execution. It is in the runtime and test classpaths, but not the compile classpath.
Proper utilization of the runtime scope should resolve the circular dependency issue.
Regarding the second issue, you have not supplied enough information to provide a definitive answer. However, you almost certainly want to use a maven plugin in order to leverage maven's information about the dependencies. For example, if you want to produce a single "fat" jar that contains everything, you want to look at maven-shade-plugin. Another option is maven-assembly-plugin, which is extremely flexible and can include all dependencies in the assembly. There are other plugins that excel at handling various other common circumstances. You may want to formulate a separate question if you have problems with how to use a specific plugin.

Here is how I would do it. From C's pom file:
<profile>
<id>compile</id>
<dependencies>
<dependency>
use B here but do not use in the main dependencies section
</dependency>
+ other dependencies
</dependencies>
</build>
</profile>
For example you can compile C module using mvn compile -Pcompile
From B's pom file:
<profile>
<id>run</id>
<dependencies>
<dependency>
use C here but do not use in the main dependencies section
</dependency>
+ other dependencies
</dependencies>
</build>
</profile>
you run B module using mvn yourcommandforrunning -Prun
In this way you can escape the cyclic dependencies issue.

Maven: Different library versions in one JVM

I have a project whose dependency tree is huge i.e. it packs in modules from several teams.
Now there are some commonly used dependencies which are common across several modules.
A simplified example can be:
TopModule.jar
ChildModule.jar
CommonModule-v1.jar
CommonModule-v2.jar
When I build my project, I specify the latest version of common dependencies, but its very hard to ask the same from every other team.
So, frequently, the TopModule is built using different versions of CommonModule (v1 and v2 in the above example).
My question is:
If the final jar file contains both CommonModule-v1.jar and CommonModule-v2.jar, how does it affect the runtime?
Can the runtime erroneously load versions v2 where v1 is required and vice versa?

Maven will only use one version of each artifact in the end -- it doesn't do any fancy classloader isolation tricks. You can see which version it'll use with mvn dependency:resolve.
If you need to use specific versions within dependencies, you can use the shade plugin. It'll do renaming trickery so that dependencies get their own versions of libraries.

To fight with this problem globally use this DependencyConvergence Rule
This rule requires that dependency version numbers converge. If a
project has two dependencies, A and B, both depending on the same
artifact, C, this rule will fail the build if A depends on a different
version of C then the version of C depended on by B.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>1.3.1</version>
<executions>
<execution>
<id>enforce</id>
<configuration>
<rules>
<DependencyConvergence/>
</rules>
</configuration>
<goals>
<goal>enforce</goal>
</goals>
</execution>
</executions>
</plugin>
After this all teams work together with consistent versions of dependencies.

This depends on the way the modules are named in maven. Usually, maven tries to resolve the conflicting libs and takes the highest version into the tree. But if the libraries are different artifacts in terms of artifactId, then maven will not see that they are from the same breed and thus will not resolve the ambiguity.
Usually you resolve this by a common parent.pom, where you define the versions of commonly used libraries throughout the project. If you have no control over the other projects (not part of your build, only dependencies), you may be lucky to have your final project working fine. If the library breaks compatibility in the newer version, you will not be able to use it.
So, does your final project contain both versions of the library or not, did you check it? The dependency tree may show both versions, but if maven will use only the latest version of a dependency in the hierarchy.

Classloader will load the first JAR which appears on your classpath. In more details - it will search for the first class on your class path, so in each case all these searches would fall into i.e. CommonModule-v2.jar. So the answer is yes - it can erroneously load versions v2 where v1 if it appears earlier on your classpath.

If your pom.xml is only an aggregator of already packaged modules then this apply. If it is not the case and your project actually compile and packages all of those modules as a submodules then maven will choose one. If it compiles every project on its own then it will be packaged using that dependency. But if all of them end up in the same class loader then it won't work fine.
At runtime it can cause errors, think of method not found and the like. Your byte code classes were compiled and linked with the correct dependencies but since the class loader finds two candidates it just load one at runtime.
What you can do is set a parent pom defining a <dependencyManagement> and then ask all teams to use it as a parent and don't declare <version> but inherit it from the parent pom.xml.
References http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management

On top of what #yshavit said, ideally you'd exclude the earlier version of the CommonModule so that only v2 is in the classpath. This is only possible if the CommonModule v2 api is backwards compatible with CommonModule v1.
Here's an example of how you exclude:
<dependency>
<groupId>ChildModuleGroupid</groupId>
<artifactId>ChildModuleArtifactid</artifactId>
<version>1.0</version>
<exclusions>
<exclusion>
<groupId>CommonModuleGroupId</groupId>
<artifactId>CommonModuleArtifactId</artifactId>
</exclusion>
</exclusions>
</dependency>
You'd put that in the TopModule pom.xml.

How to handle sub projects dependencies in Maven

My project is made of 5 sub projects. One is a War, and the other 4 are jars. Basically the war project needs all 4 jar projects, and their dependencies.
I can strip down the dependencies to have something like war->A->B->C->D. Every sub project add their share of external dependencies (spring, struts, hibernate) so that in the end the war gets everything needed to run.
This looks pretty well organised and square, but then I ask myself if this is very practical to make changes.
Imagine I have to change one line of code in project D, without changing anything to its Maven dependencies. I would have to re-release project D obviously, but then I have to re-release projects C, B, A, and the war just to reflect this change in their pom files. This can be long and annoying, especially if you have to quickly release a new version to fix something in production.
I could make the war depend on all 4 projects, so then I just have to change project D version number in the war pom file. But then I have project A depending indirectly on project D 1.0 and the war specifying project D 1.1. I think that the war direct dependency would win in that case wouldn't it ?
This would make the new war release quicker, but it would also mess my sub projects dependencies, as they would be outdated.
What would be an acceptable way to handle this situation ?

There is no simple answer to your problem.
If you indeed do have a chain of transitive dependencies (A->B->C->D), then releasing each modules up the chain independently is not a bad option. Although it is tedious, there is a good chance your nested dependencies are simple lib jars and will not see changes too often. Hopefully you will not be forced to go through that process frequently. Pretend it would be the same situation as if log4j was updated and all of your modules needed to be updated as well.
Another thing to consider is your WAR's dependencies. Yes, Maven will pull dependencies in automatically for you but it is often a good practice to declare your known dependencies explicitly so you can specify a version number yourself for each module. This would mean A depends on D and the others directly. Unfortunately, if you have conflicting version numbers, as you've described, then you are looking for trouble on your classpath. If you really need to do this though, maven does allow you exclude transitive dependencies explicitly:
<project>
...
<dependencies>
<dependency>
<groupId>my.project</groupId>
<artifactId>module-B</artifactId>
<version>1.0</version>
<exclusions>
<exclusion>
<groupId>my.project</groupId>
<artifactId>module-C</artifactId>
</exclusion>
<exclusion>
<groupId>my.project</groupId>
<artifactId>module-D</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>my.project</groupId>
<artifactId>module-C</artifactId>
<version>1.0</version>
<exclusions>
<exclusion>
<groupId>my.project</groupId>
<artifactId>module-D</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>my.project</groupId>
<artifactId>module-D</artifactId>
<version>1.0</version>
</dependency>
</dependencies>
...
</project>
Here is the documentation describing these optional dependencies and exclusions.
Do you actually need to release B, C, and D independently? If not, consider using an Aggregator pom.xml file at the root of your modules. This will allow you to use SNAPSHOT versions throughout your modules and then release the bunch at once. This is the way our team manages our multi-module project. Using SNAPSHOT dependencies ensures you use the version that was JUST built when those artifacts are needed.

Do you actually release any of projects A to D independently, without the WAR? If not, I don't see any problems with your current setup. You should absolutely use the same version of any module throughout the project. Otherwise you open the door to classloader hell - believe me, you don't want to get there :-(
To make releases easier, the maven-release-plugin may help you.

The best answer these days is now use gradle which is best of ant and maven. I never really liked maven but gradle took alot of the common concepts but made it more like ant in that it is flexible so that there is no an easy answer to your question in gradle ;).