I have a project whose dependency tree is huge i.e. it packs in modules from several teams.
Now there are some commonly used dependencies which are common across several modules.
A simplified example can be:
TopModule.jar
ChildModule.jar
CommonModule-v1.jar
CommonModule-v2.jar
When I build my project, I specify the latest version of common dependencies, but its very hard to ask the same from every other team.
So, frequently, the TopModule is built using different versions of CommonModule (v1 and v2 in the above example).
My question is:
If the final jar file contains both CommonModule-v1.jar and CommonModule-v2.jar, how does it affect the runtime?
Can the runtime erroneously load versions v2 where v1 is required and vice versa?
Maven will only use one version of each artifact in the end -- it doesn't do any fancy classloader isolation tricks. You can see which version it'll use with mvn dependency:resolve.
If you need to use specific versions within dependencies, you can use the shade plugin. It'll do renaming trickery so that dependencies get their own versions of libraries.
To fight with this problem globally use this DependencyConvergence Rule
This rule requires that dependency version numbers converge. If a
project has two dependencies, A and B, both depending on the same
artifact, C, this rule will fail the build if A depends on a different
version of C then the version of C depended on by B.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>1.3.1</version>
<executions>
<execution>
<id>enforce</id>
<configuration>
<rules>
<DependencyConvergence/>
</rules>
</configuration>
<goals>
<goal>enforce</goal>
</goals>
</execution>
</executions>
</plugin>
After this all teams work together with consistent versions of dependencies.
This depends on the way the modules are named in maven. Usually, maven tries to resolve the conflicting libs and takes the highest version into the tree. But if the libraries are different artifacts in terms of artifactId, then maven will not see that they are from the same breed and thus will not resolve the ambiguity.
Usually you resolve this by a common parent.pom, where you define the versions of commonly used libraries throughout the project. If you have no control over the other projects (not part of your build, only dependencies), you may be lucky to have your final project working fine. If the library breaks compatibility in the newer version, you will not be able to use it.
So, does your final project contain both versions of the library or not, did you check it? The dependency tree may show both versions, but if maven will use only the latest version of a dependency in the hierarchy.
Classloader will load the first JAR which appears on your classpath. In more details - it will search for the first class on your class path, so in each case all these searches would fall into i.e. CommonModule-v2.jar. So the answer is yes - it can erroneously load versions v2 where v1 if it appears earlier on your classpath.
If your pom.xml is only an aggregator of already packaged modules then this apply. If it is not the case and your project actually compile and packages all of those modules as a submodules then maven will choose one. If it compiles every project on its own then it will be packaged using that dependency. But if all of them end up in the same class loader then it won't work fine.
At runtime it can cause errors, think of method not found and the like. Your byte code classes were compiled and linked with the correct dependencies but since the class loader finds two candidates it just load one at runtime.
What you can do is set a parent pom defining a <dependencyManagement> and then ask all teams to use it as a parent and don't declare <version> but inherit it from the parent pom.xml.
References http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management
On top of what #yshavit said, ideally you'd exclude the earlier version of the CommonModule so that only v2 is in the classpath. This is only possible if the CommonModule v2 api is backwards compatible with CommonModule v1.
Here's an example of how you exclude:
<dependency>
<groupId>ChildModuleGroupid</groupId>
<artifactId>ChildModuleArtifactid</artifactId>
<version>1.0</version>
<exclusions>
<exclusion>
<groupId>CommonModuleGroupId</groupId>
<artifactId>CommonModuleArtifactId</artifactId>
</exclusion>
</exclusions>
</dependency>
You'd put that in the TopModule pom.xml.
Related
I'm working on a large Java codebase that's split into multiple modules, each with a separate pom.xml, all parented by a top-level pom.xml.
I'm currently in the process of bringing in a couple of library dependencies. The transitive set of dependencies is large, and as luck would have it, there are conflicting dependency versions for different modules.
Here's a simplification of my situation:
project/pom.xml
/module-a/pom.xml # references library-a, depends on library-c:v1
/module-b/pom.xml # references library-b, depends on library-c:v2
/module-c/pom.xml # references module-a and module-b
Now the unit tests for module-a will exercise library-a in the presence of library-c:v1, while module-b will exercise library-b in the presence of library-c:v2.
The trouble is that module-a and module-b need to live together on the same classpath when module-c is deployed, but whatever version of library-c is chosen when module-c is packaged, at least one combination of libraries hasn't been unit tested!
I'd like to pin the version of library-c at the parent pom level somehow, rather than repeating myself in every module that transitively depends on library-c; ideally it would be added in such a way indicating that it's a transitive dependency that is allowed to go away should library-a and library-b no longer rely on it.
I'd like a guarantee that there is exactly one version selected for
every transitive dependency across the entire project rooted from this parent pom, and I'd like the build to blow up if this isn't true. I wrote a tool to parse the output of mvn dependency:tree (turning the leaves of the tree into a forest of paths from leaf to root, then finding all different versions of leaf with the dependency path) so I can see the problem, but without explicitly resolving the transitive dependencies for every conflict and bloating out poms with redundant declarations, this doesn't seem fruitful. It's what I'll do if I have no alternative, naturally.
What's the best way to handle this transitive dependency conflict problem with Maven?
How severe is this problem? Quite apart from getting unconvincing test coverage, in practice I see JVM-killing NoSuchMethodError at runtime from the wrong versions getting deployed. I'd prefer to see these at test time at the very least.
Looks like there are two aspects to this:
You need to insist on a single version of a dependency, whether it is declared explicitly or acquired transitively
You can use <dependencyManagement/> here. For example in the top-level pom.xml you can pin the version of library-c:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>your.group.id</groupId>
<artifactId>library-c</artifactId>
<version>2</version>
<dependency>
<dependencies>
<dependencyManagement>
And then in library-a, library-b you would declare the dependency on library-c as follows:
<dependencies>
<dependency>
<groupId>your.group.id</groupId>
<artifactId>library-c</artifactId>
<dependency>
<dependencies>
By declaring this dependency in the parent's dependencyManagement you are insisting on both of the child modules using the version declared in the parent.
You want to protect yourself from unhappy dependency additions occurring in future
You can use the Maven Enforcer plugin here, specifically the dependencyConvergence rule. For example:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>3.0.0-M1</version>
<executions>
<execution>
<id>enforce</id>
<configuration>
<rules>
<dependencyConvergence/>
</rules>
</configuration>
<goals>
<goal>enforce</goal>
</goals>
</execution>
</executions>
</plugin>
The enforcer can be configured to either fail or warn if it discovers a non convergent dependency.
I have a Maven java web app (.WAR) project that includes several libraries, including the Wicket libraries (but I don't think the problem is wicket itself, but rather with maven).
Here's the problem: even tho I only include Wicket 6.20.0, the resulting .WAR contains two copies of the Wicket libraries: 6.20.0 and 6.18.0, as you can see in this screenshot:
Thinking of some conflicting imports I printed the dependency tree using the:
mvn dependency:tree
commnad... but there is no mention of Wicket 6.18.0 in the dependency tree! I also double-checked using Eclipse's "dependency hierarchy" view and I can confirm there's no trace of that import.
I even did a search for string "6.18.0" across the entire workspace with Eclipse, but it's nowhere to be found!
How can I find out what is causing the inclusion of that duplicate version of the library?
Maven doesn't work in this way.
The resolution of more than one dependency with the same artifactId and groupId but with a different version will result to a single dependency (the version used is no determinist).
The presence of two artifacts with the same artifactId and groupId but with two distinct versions in a same lib folder of the WAR is probably related to one of these :
you don't execute mvn clean package but only mvn package.
your use a bugged version of the Maven war plugin. Try to update it to check that.
you have a Maven plugin that copies Wicket jars 6.18.0 in the WEB-INF/lib folder of the target folder during the build of the component.
the maven WAR project you are building has as dependency an artifact of type WAR. In this case, the dependencies of the WAR dependency are so overlaid in the WAR project that you are building.
An interesting Maven issue about duplicated JAR because of WAR dependencies :
JARs with different versions can be in WEB-INF/lib with war as dependencies
Your answer and your comment indicate that actually you have a WAR dependency in your build.
Unfortunately, there is not really a good and long term effective solution to bypass this limitation.
As said in my comment, using the packagingExcludes property of the maven war plugin is a valid workaround for the actual issue :
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.4</version>
<configuration>
<!-- ... -->
<packagingExcludes>WEB-INF/lib/wicket-*-6.18.0.jar</packagingExcludes>
</configuration>
</plugin>
But beware, using that will do your build less robust through the time.
The day where you update the version of the WAR dependency and that in its new version, it pulls again a different version of wicket, you have still a risk to have duplicate jars with two distinct versions in your built WAR.
Using the overlay feature by specifying the overlay element of the maven-war-plugin is generally better as it focuses on the overlay applied for the war dependency. It fixes the problem early.
As a result, you could define to exclude any wicket JARs from the WAR dependency :
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<version>2.4</version>
<artifactId>maven-war-plugin</artifactId>
<configuration>
<overlays>
<overlay>
<groupId>com.whatever.youlike</groupId>
<artifactId>myArtifact</artifactId>
<excludes>
<exclude>WEB-INF/lib/wicket-*.jar</exclude>
</excludes>
</overlay>
</overlays>
</configuration>
</plugin>
This way is better but this is still a workaround.
The day where the dependency WAR is updated and that it pulls new dependencies (other than Wicket) that are declared in your actual build but with different versions, you may finish with the same kind of issue.
I think that declaring a dependency on a WAR artifact should be done only as we don't have choice.
As poms and projects refactoring are possible, introducing a common JAR dependency which the two WARs depend on and that contains only common sources and resources for the two WARs makes really things simpler.
Well, I figured it out while poking around.
I had a dependency of type "war" in the project:
<dependency>
<groupId>com.whatever.youlike</groupId>
<artifactId>myArtifact</artifactId>
<version>1.0.7-SNAPSHOT</version>
<type>war</type>
</dependency>
Apparently (I wasn't aware of this, my fault here) these type of dependencies will include themselves in the classpath by copying all libs to the main WAR /libs folder, but these will NOT show app in the dependency tree / dependency hierarchy.
I solved by configuring an explicit exclusion in the WAR plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.4</version>
<configuration>
<!-- ... -->
<packagingExcludes>WEB-INF/lib/wicket-*-6.18.0.jar</packagingExcludes>
</configuration>
</plugin>
Use clean install and the double dependency will probably be gone.
Because other libs can use same libs but different version or you tried different version and didn't make mvn clean
The command mvn dependency:tree is telling you the correct information - what you are looking at here is an eclipse / build issue.
Clear out all the target and build areas in your project. If need be, check it out from source control to a new folder.
Alternatively you can build your project in IntelliJ IDEA, and see if you get the correct dependencies (most likely you will).
I'm using two jars A and B. B is a library and A has classes that uses some old classes from library B. Now this is causing me a problem when I include both jars in my project classpath as there are the same names of two classes but one of them is older than the other and behave differently.
One solution to this problem I found is by first importing library B into Eclipse and then I click OK and the project builds. Then I add the jar A. This way all my existing code will use the newer versions of B and the classes of A will be untouched.
However now I want to use Maven for my projects but I'm unable to know how to make this trick again using Maven. Please help.
Maybe you can solve your problem by renaming the package of the class that you don't want.
You can do it by using Maven Shade Plugin
This plugin allows to rename package names at compilation.
Usage :
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<relocations>
<relocation>
<pattern>com.example.package.name.YourClass</pattern>
<shadedPattern>com.example.rename.package.name.YourClass</shadedPattern>
</relocation>
</relocations>
<promoteTransitiveDependencies>true</promoteTransitiveDependencies>
</configuration>
</execution>
</executions>
</plugin>
This isn't necessarily a Maven problem. The default classloader will search for classes according to the order of the jars on the classpath. When you are adding the jars to Eclipse you are doing so in a way that their order will ensure the correct classes are loaded - specifically B appears on the classpath before A and therefore, when the same class is in both jars, it will be loaded from B.
Since version 2.0.9 of Maven, the classpath is built according to the order of the dependencies in the POM. So, providing dependency B is declared before the dependency A, you should get the same behaviour are with Eclipse.
Needless to say, relying on classpath order in this way is rather fragile and personally I'd look to clean-up the jars if that's possible.
The problem is very real (unless for me).
If you have detected what libraries are the conflicted, you can use exclusions to prevent import libraries that you dont want.
If you dont know what are the conflicted libraries, in eclipse using the default maven plugin you can open the pom file and select the tab "Dependency Hierarchy" in the right column you can see all your resolved dependencies for your project, and in the left what library import each dependency.
I hope it can help you.
I'm currently in the midst of converting a large multi-module project (~100 sub-modules) to use Maven. Currently we use Ant + Ivy.
So far no major issues have cropped up and I'm comfortable that Maven is still a good fit. However, I wonder if there is a better way to handle native dependencies.
So far I have come to the following conclusions.
It's best to install each native dependency into the maven repo either as a standalone library or an archived package containing multiple dependencies.
Rather than get lost in declaring each and every dependency with the Maven dependency plugin, I opted to give each a classifier (e.g. natives-win32) and use the following in the parent POM:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.4</version>
<executions>
<execution>
<id>copy</id>
<phase>compile</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<includeScope>runtime</includeScope>
<includeClassifiers>natives-win32</includeClassifiers>
<outputDirectory>${project.build.directory}/natives</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
So far this seems to be a simple all-round solution that doesn't require too much messing about to add new native dependencies. It also offers me a simple all-round solution for managing natives. The only thing I must do is ensure that my /natives/ directory is defined on java.library.path.
One thing that bothers me (a little) about this approach is that all my native dependencies get copied around to each sub-module that expresses a transitive dependency on them, whilst my happy jar libraries are added to the classpath referenced to where they sit in my local repository (no copy required).
Is there no way to be smarter about this and have my natives referenced from withing their repository location (assuming I don't have them archived, i.e. dll). That would save a bunch of unnecessary copying about.
Are there any other potential gotchas' that I should be concerned about with the above approach?
Your snippet shows a goal attached to a build phase, not a dependency. Is the 'copy dependencies' goal in a super pom and inherited by all modules? There's no way to move it only to the modules which are going to be run/packaged as an app?
It could be, that I didn't got it. But why don't you deploy all your native libs into the repository at first. If the native libs are stable and change seldom, That could be done in a seperate reactor.
And afterwards you reference those native dependencies simply via GAV as any other dependency. Also the problem af unnecessary copying is solved by that.
I ended up using the maven natives plugin and dealing with the fact that I have redundant copies of the native libraries around the place. The reason for this was primarily due to the simplicity that the plugin offers and the fact that it also has a related eclipse plugin that sets up natives in developers eclipse environment without intervention.
Let's say I have four projects:
Project A (has a dependency on B and D)
Project B (has a dependency on D)
Project C (has a dependency on D)
Project D
In this scenario if I run project A, Maven will correctly resolve the dependency to D. If I understand this correctly Maven always takes the dependency with the shortest path. Since D is a direct dependency of A it will be used rather then, the D which is specified within B.
But now assume this structure:
Project A (has a dependency on B and C)
Project B (has a dependency on D)
Project C (has a dependency on D)
Project D
In this case the paths to resolving D have the same depth. What happens is that Maven will have a conflict. I know that it is possible to tell Maven that he should exclude dependencies. But my question is how to address such kind of problems. I mean in a real world application you have a lot of dependencies and possibly a lot of conflicts as well.
Is the best practice solution really to exclude stuff or are there other possible solutions to this? I find it very hard to deal with when i suddenly get a ClassNotFound Exception because some versions have changed, which caused Maven to take a different dependency. Of course, knowing this fact makes it a little bit easier to guess that the problem is a dependency conflict.
I'm using maven 2.1-SNAPSHOT.
The maven way of resolving situations like this is to include a <dependencyManagement> section in your project's root pom, where you specify which version of which library will be used.
EDIT:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>foo</groupId>
<artifactId>bar</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</dependencyManagement>
Now no matter which version of library foo:bar is requested by a dependency, version 1.2.3 will always be used for this project and all sub-projects.
Reference:
Dependency Management
Maven can handle both situations without any conflict. Conflicts will exist when two versions of a transitive dependency are required. The ClassNotFoundException you describe results from the app (or a dependency) attempting to use a class not available in the version of the conflicted dependency that actually gets used.
There are multiple ways to fix the problem.
Update the versions of the libraries you are using that depend on the conflicted dependency, so that they all depend on the same version version of that dependency
Declare the conflicted dependency as a direct dependency of your project with the version you want to be included (in the example, the one with the missing class included in it)
Specify which version of the conflicted dependency that transitive dependencies should use, via the <dependencyManagement> section of the POM
Explicitly exclude the unwanted versions of the conflicted dependency from being included with the dependencies that rely on them using an <exclusion>
This is fundamentally not a maven issue, but a java issue.
If Project B and Project C needs two incompatible versions of project D, then you can't use them both in Project A.
The Maven way of resolving conflicts like these is unfortunately, as you already know, to choose which ones to exclude.
Using mvn dependency:analyze and mvn dependency:tree helps in finding what conflicts you have.
You can enforce consistent dependencies in whole project with rule Dependency Convergence.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>1.3.1</version>
<executions>
<execution>
<id>enforce</id>
<configuration>
<rules>
<DependencyConvergence/>
</rules>
</configuration>
<goals>
<goal>enforce</goal>
</goals>
</execution>
</executions>
</plugin>
One possible strategy is to specify for main project, what version of D to use (the newest one f.g.). However, if library D is not backward-compatible, you have a problem as stated by kukudas - it's impossible to use both libaries in your project.
In such situation there may be necessary to use either B or C in older version, so that both would depend on compatibile versions of D.