We Keep Coding

How can I have a multi-module Maven project without specifying the version all over the place?

I am currently learning Maven for a specific project. In this project, I need to have multiple modules (in the sense of a collection of classes and resources) that depend on each-other. Specifically, I need a module to store the classes of a public-facing API (the API is a Java API, as the desktop program can load addons at runtime), another for a set of common classes that are shared between the client and server but that are not part of the API, and of course the client and server themselves need a module each.
Using the system that I used before Maven (IntelliJ Idea's build system), I would simply create different modules and setup dependencies using IntelliJ's GUI. This works well because I can then put the whole IntelliJ Idea project in a git repository, and keep everything nicely tracked together with no fuss and no problems.
When researching how to do this in Maven, however, I ran into some problems. Maven's documentation has a section that seems to explain how to accomplish something similar, but its technique seems to have two problems. When Each sub-project, with its own pom.xml, needs to specify the version of the other sub-projects that it depends on¹. This implies that I need to make many changes all over the project whenever the version changes. This will (hopefully) be quite frequent. As it is perfectly plausible that my project will grow to have hundreds of modules, this is obviously impractical.
Additionally, based on my other research, it seems like a Maven repository is involved in the documented technique. Specifically, it seems like when, for example, the API module is built to be included in the client and server modules it will first be placed in a local repository and then the client and server modules will retrieve it from there. This sounds problematic as the API module will also be published in a public repository (maybe Maven central, I haven't really thought about this too much -- but it will be public in a repo), and this repository step seems like it could end up building the client and server with a published jar rather than the local one, which is problematic for many reasons. (eg. if a developer is making local changes to the api jar, building half the project without those changes is a problem).
Is there some better way to combine multiple modules to mitigate these problems (versions all over the place, and getting wrong jars from repos)? Is it better to modify the technique in the linked documentation instead? Am I misunderstanding something and the problems don't exist?
I have checked DuckDuckGo and all relevant questions that I can find on StackOverflow, and none address either of the two aforementioned problems.
¹:
<dependencies>
<dependency>
<groupId>com.mycompany.app</groupId>
<artifactId>my-app</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
</dependencies>

In your multi-module project, you usually use just one version for all modules. This means that you can define dependencies between modules with <version>${project.version}</version> where the property ${project.version}$ is resolved during the build. Maven builds all the modules in the correct order.

Regarding your first question: you can create a parent pom over your whole project. In this parent pom you can add a dependencyManagement section where you can list your own modules with their version. In your modules you then only need to specify the dependeny without the version.
Here an example:
in parent pom:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>yourgroup</groupId>
<artifactId>yourmoduleA</artifactId>
<version>3.0.1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>yourgroup</groupId>
<artifactId>yourmoduleB</artifactId>
<version>3.0.1-SNAPSHOT</version>
</dependency>
</dependencies>
</dependencyManagement>
and if your module A uses module B, you can add in pom of module A
<dependencies>
<dependency>
<groupId>yourgroup</groupId>
<artifactId>yourmoduleB</artifactId>
</dependency>
</dependencies>

Force Maven use only first level dependencies

I have a Maven Java project. I don't want my project dependencies to be satisfied by chance through a chain of subdependencies when compiling the project. It is OK for me when building the final war when maven must check all used dependencies and add necessary libs to the war, but when compiling the code I want to be sure that only direct dependencies are used. Why?
Let's say I have two dependencies:
<dependency>
<groupId>com.package</groupId>
<artifactId>module-1</artifactId>
</dependency>
<dependency>
<groupId>com.package</groupId>
<artifactId>module-2</artifactId>
</dependency>
For our project module-1 and module-2 serve completely different purposes, but somewhere in the dependency tree of module-2, module-1 is used. I delete module-1 dependency, but maven continue to build my project without compilation errors, because it resolves module-1 from module-2 sub-dependencies. This change goes unnoticed.
After sometime we decide to remove module-2, because we don't need it. Strange enough but we can not any more compile classes which were using imports from module-1 and which are not connected to module-2 logic.
This is a simple case, but in big project this can make quite a dependency mess.

You can use the Maven dependency plugin goal "dependency:analyze" to give you a report of all used dependencies which are not declared on the current module (included transitively). That way Maven will still use transitive dependencies (no way around that I guess), but you can force yourself via the plugin to make sure these are also declared. It will also warn you of unnecessary dependencies. Mind, the plugin analyzes the compiled classes. At times, you may need to configure the plugin, because occasionally it may not detect that a dependency is required at compile time but not at runtime, e.g. because a constant was inlined.

If you really need to do this then you can setup exclusions in the pom.
e.g. here's an example of an exclusion in one of my poms where I don't want it to automatically get commons-logging because I'm using a different logging provider.
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>${org.springframework-version}</version>
<exclusions>
<!-- Exclude Commons Logging in favor of SLF4j -->
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
You could do something like this (untested)
<dependency>
<groupId>com.package</groupId>
<artifactId>module-2</artifactId>
<exclusions>
<exclusion>
<groupId>com.package</groupId>
<artifactId>module-1</artifactId>
</exclusion>
</exclusions>
</dependency>
I wouldn't necessarily recommend this though. It makes sense in the case of my logging exclusion because I'm using slf4j instead of commons logging. I've seen other examples where this is used to exclude spring 2 if the project as a whole is using spring 3.
It's a bit difficult to tell from your example because it's so vague. In general you should keep your dependencies to a minimum. If module-2 depends on module-1 then it implies that your application won't compile or run without module-1. If in fact it can live happily without it then it's not really dependent.
As a side note it's a bit alarming that you don't have a version number against the dependencies. You'll probably find maven warns you about this. It's good practice to always include a version number. If you're dependent on a module which is currently in development then you should use the .SNAPSHOT suffix on the version to get the latest build for that version.

There seems to be no way to tell maven not to resolve dependency transitively: How to exclude all transitive dependencies of a Maven dependency. One of the reason's I think, is that the user can soon run into runtime troubles, when he finds that some of the artifacts are not being resolved at runtime or there are artifact versions problems. However, if you check the link out, you can make each of the deps 'standalone' with a wildcard exclusion pattern.
One other option is to use <optional> dependency for each of your module-X sub-dependencies. This will make sure the project compiles and non of your module-X would be resolved transitively. Like:
<dependency>
<groupId>com.package</groupId>
<artifactId>module-1</artifactId>
<optional>true</optional>
</dependency>
Still, analyzing the dependency tree might be the most safe and predictable choice.

It does sound a bit strange what you plan to do. In a way you sabotage the dependency management you want to use.
If your module-2 depends on module-1 and has a dependency to it, then any module that depends on module-2 only need to define that one.
You may be able to restrict the depth of the resolution using exclusions: Exclude all transitive dependencies of a single dependency
Newer versions of maven allow wildcards in those.
But: you will need to re-add the ones you actually need, this is by repeating the dependencies you have an other modules. This duplicates the work.
If there are artifacts that cause weirdness it may be possible to define a scope: http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html so it is not propagated to dependant modules as well.

Solving Maven dependency convergence issues

I use the maven-enforcer-plugin to check for dependency convergence issues. A typical output would be:
[WARNING] Rule 1: org.apache.maven.plugins.enforcer.DependencyConvergence failed
with message:
Failed while enforcing releasability the error(s) are [
Dependency convergence error for junit:junit:3.8.1 paths to dependency are:
+-foo:bar:1.0-SNAPSHOT
+-ca.juliusdavies:not-yet-commons-ssl:0.3.9
+-commons-httpclient:commons-httpclient:3.0
+-junit:junit:3.8.1
and
+-foo:bar:1.0-SNAPSHOT
+-junit:junit:4.11
]
Seeing this message, I would normally "solve" it by excluding the transitive dependency, e.g.
<dependency>
<groupId>ca.juliusdavies</groupId>
<artifactId>not-yet-commons-ssl</artifactId>
<version>0.3.9</version>
<exclusions>
<!-- This artifact links to another artifact which stupidly includes
junit in compile scope -->
<exclusion>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</exclusion>
</exclusions>
</dependency>
I'd like to understand whether this is truly a fix and the risks involved in excluding libraries in this fashion. As I see it:
The "fix" is normally safe, provided I'm choosing to use the newer version. This relies on the library authors maintaining backwards compatibility.
There is typically no impact on the Maven build (since the nearer definition wins), however by excluding the dependency I'm telling Maven that I know about this problem and thus appeasing the maven-enforcer-plugin.
Are my thoughts correct and is there an alternative way of handling this issue? I'm interested in answers that focus on the general case - I realise the junit example above is a little strange.

We all agree that JUnit should never be set to another scope than test. Generally speaking I don't think either that there is another solution than excluding the unwanted dependency, so we all agree that your are right to do it.
A SIMPLE CASE :
As Andreas Krueger says, there may be a risk with versions (I actually encountered it). Let say that the project's dependencies are the following:
+-foo:bar:1.0-SNAPSHOT
+-group1:projectA:2.0
+-group2:projectB:3.8.1
+-group2:projectB:4.11
Note that it is only a mere simplification of your case. Seeing this dependency tree, you would exclude the dependency projectB given by projectA :
<dependency>
<groupId>group1</groupId>
<artifactId>projectA</artifactId>
<version>2.0</version>
<exclusions>
<exclusion>
<groupId>group2</groupId>
<artifactId>projectB</artifactId>
</exclusion>
</exclusions>
</dependency>
After packaging the project with maven, the remaining dependency would be group2-someProjectB-4.11.jar, version 4.11 and not 3.8.1. Everything would be fine and the project would run without encountering any problem at all.
Then, a while after, let say that you decide to upgrade to the next version of project A, version 3.0 which adds new great features :
<dependency>
<groupId>group1</groupId>
<artifactId>projectA</artifactId>
<version>3.0</version>
<exclusions>
<exclusion>
<groupId>group2</groupId>
<artifactId>projectB</artifactId>
</exclusion>
</exclusions>
</dependency>
The problem is that you are not aware yet that projectA version 3.0 also have upgraded its dependency projectB to version 5.0 :
+-foo:bar:1.0-SNAPSHOT
+-group1:projectA:3.0
+-group2:projectB:5.0
+-group2:projectB:4.11
In that case, the exclusion you would have made a while ago excludes projectB version 5.0.
However, projectA version 3.0 needs the improvements from project B version 5.0. Because of the exclusion, after packaging the project with maven, the remaining dependency would be group2-someProjectB-4.11.jar, version 4.11 and not 5.0. At the moment you use any of projectA's new features, the program wouldn't run correctly.
WHAT WAS THE SOLUTION ?
I encountered this problem in a Java-EE project.
A team developped database services. They packaged it as projectA. Each time they updated the services, they also updated a file listing all their current dependencies and the current versions.
ProjectA was a dependency for the Java-EE project I was working on. Each time the service-team updated ProjectA, I also checked the versions' updates.
In fact, there is no harm in excluding a dependency. But each time you update a dependency where an exclusion has been set, You have to check :
if this exclusion still makes sense.
if you need to upgrade the version of the excluded dependency in your own project.
I guess maven exclusions are like kitchen knifes. It's sharp, cuts vegetables with no effort, but requires care when handling it...

If JUnit as an artifact is coming through as a dependency in compile scope, it is a bug of one of your libraries, here: ca.juliusdavies.
JUnit should always be included in test scope. Thus, it is not packed into the produced .jar, .war or .ear file, on successful build.
Generally speaking, there is no harm in excluding already included dependencies, as when library 1 and library 2 share one common dependency.
The only problem, of course, that can occur, is when library 1 and library 2 include different versions of the same dependent artifact. This can cause run-time errors, when the features of the library have changed.
Fortunately, this is not often the case, unless the difference in the version numbers is great. In general, it is advisable to include the latest dependency version and exlude the older one. This is most of the time viable.
If not, check wheter there are updates to the first-level dependencies of your project.

Create dependency groups in Maven for reuse - including 'provided' dependencies

I am new to Maven and am setting up my first maven project. I am also creating some maven assets in the form of some poms that can be inherited from or used as dependencies in any future projects as well. I want to group dependencies together and to be able to selectively add them to a project as needed.
I read this article on pom best practices. I like the idea of grouping related dependencies together into poms and then adding the pom as a dependency to a project as needed. This approach works great for compile scoped dependencies. However it fails for provided scoped ones since as transitive dependencies, they get omitted.
Here's an example of what I mean: Lets say I group together web dependencies for my projects into a web-deps pom.xml. These include compile scoped spring framework dependencies and also a provided scoped javaee one:
<modelVersion>4.0.0</modelVersion>
<groupId>com.xyz</groupId>
<artifactId>mvn-web-deps</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>pom</packaging>
<dependencies>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-web</artifactId>
<version>${org.springframework.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>${org.springframework.version}</version>
</dependency>
<dependency>
<groupId>javaee</groupId>
<artifactId>javaee-api</artifactId>
<version>${javaee.version}</version>
<scope>provided</scope>
</dependency>
</dependencies>
I then add this pom as a dependency in another project:
<modelVersion>4.0.0</modelVersion>
<groupId>com.xyz</groupId>
<artifactId>project-a</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependency>
<groupId>com.xyz</groupId>
<artifactId>mvn-web-deps</artifactId>
<version>0.0.1-SNAPSHOT</version>
<type>pom</type>
</dependency>
The dependencies in mvn-web-deps now become transitive. Since the dependency reference above is compile scoped, the provided transitive dependency gets omitted.
I want to avoid adding them to the dependency section of a parent since there can only be one parent and a project may need only some of these dependency groups, not all. I can perhaps add them to the dependencyManagement section, but then I will have to redeclare each dependency (sans the version) in each child project.
What is the correct/better way of grouping dependencies while avoiding the issues like above?

The short answer to your question is that you should only include 'provided' dependencies locally where the code requires it to compile, but not in parent pom.xml or other structures. Indicating that you have a 'provided' dependency in global pom.xml is non-sense for maven, because it does not need it to compile in such pom.xml.
Here is the long answer:
When I started using Maven, I had the same idea of trying to group artifacts and dependencies into pom.xml modules hoping they would be useful in the future. Now, that I have a bit more experience, I got to understand that it is a complete waste of time. For me, this was form of over-engineering.
I have learned to split my big projects into separate modules, each in their own subversion repository. I am including dependencies as necessary for each local module in their pom.xml. I release versioned tags of each module as I am coding and as necessary (i.e., when tested and stable).
I build my big projects by creating a separate maven project with its own pom.xml and import my modules as dependencies. From time to time, I update the module's version in the dependency when I have made a release. Then, I let maven do the job of pulling whatever it has to pull, transitively of not, when compiling/releasing the big project.
Maven allows all sorts of complex constructions and hierarchy between pom.xmls, but IMHO this feature creates unnecessary mess and complexities. So far it has not proved to be a real benefit for me. At the beginning, I was hoping that compiling one pom.xml would compile the rest properly in a cascading way. I did get some result, but what a mess to maintain in all the global pom.xml.
Releasing my module's artifacts separately and building my project on these releases has saved me so much time that I can only recommend it. In total, I have less pom.xml to maintain and they are also less complex. For the same final result...
So, if your only reason for building global/structural pom.xml is a hope to save time, I recommend abandoning this idea... Separate code in separate projects, release and THEN compile globally.

I concluded that Maven was not designed for this kind of use-case. I ended up having a parent pom.xml with all the libraries I use added to its <dependencyManagement> section. Any new projects/modules that I create have their pom.xml inherit from the parent pom.xml and add each dependency they need to their own <dependencies> section, minus the version. This scheme allows me to manage the versions for the libraries that I use and the respository declarations they need at a single place. Another advantage (over trying to create dependency bundles somehow) is that this gives more fine-grained control over the libraries added to child poms - only the dependencies that are actually needed are added.

Provided-scope dependencies are indeed inherited from parent POM, but NOT from POM defined as dependencies and I consider that a Maven weakness.
Given that Maven has also difficulties in adding modules as dependencies across module hierarchies, I can't say Maven is a sophisticated tool to manage multi-module projects. Maven expects a strict single-rooted hierarchy that is only suitable for the simplest projects.

dependency management with maven

I have lately become a big fan of Maven for controlling the build cycle for my application. However I've encountered some rough edges with Maven's dependency management. I'm wondering if these are limitations of the tool and paradigm, necessary evils of dependancy management, or if I"m just using the tool incorrectly.
First is the matter of transitive dependencies. As I understand it, if you provide a dependency, Maven will in turn find any dependencies of that dependency. That is great, but for many of my dependencies, this has not worked. For example, including Hibernate in my project:
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>3.3.2.GA</version>
</dependency>
Results in a missing dependency of slf4j. I need to manually add this dependency which I assumed would be Maven's job. The same goes for Spring. If I add Spring-MVC as a dependency, shouldn't all of the basic servlet dependencies be added for me (because Spring-MVC would need this stuff)? I'm referring to the servlet, jsp, jstl libraries.
Second is the management of repositories. Maven comes shipped with a default main repository, but I've found that in many cases this repository is not up to date. For example, ifyou want spring3, you have to manually add the springsource repository, and if you want hibernate 3.5+ you have to add the jboss repository. It seems to defeat the point of automatic dependency management when you have to hunt down the correct repositories yourself. This hunting soon gets complicated. For example to add Spring3, you may want the spring release repo, the spring externals repo and the spring milestone repo.
Closely related to number 2 is ensuring you have the correct version of an artifact. I have been burned several times by including the wrong versions of dependent artifacts for a given artifact. For example the wrong version of the servlet/jsp/jstl apis for spring3, or the wrong version of persistence / annotation apis for hibernate. The repositories are filled with many versions, some with confusing names like productx-3.ga, productx-3-rc1, productx-3-SNAPSHOT, productx-3-cr, product-3-beta, etc. Some of these are obvious (rc= release candidate), but it can be confusing trying to determine the order of these versions.
Finally, the issue of the type a dependency. I probably just don't understand this well enough, but many repo artifacts are of type "pom" not "jar". Several times i have added a dependency jar to my project only to find out at build time that the repo jar does not actually exist (example is org.hibernate ejb3-persistence in the jboss repo).
With some experimenting, I can usually get a build to work, but is dependency management in general this complicated? I still prefer this approach to manually adding jar files to my project, but I would be interested to learn how to improve my maven dependency management skills.

Can't answer all parts of the question, but about some of them:
Some transitive dependecies are marked optional, so people who don't need these features wouldn't download them, but people, who need, have to set them explicitly in their poms.
Maven Central repository contains only releases. Therefore, it doesn't contain Hibernate 3.5 (which is beta) as well as it hadn't contained Spring 3 until it was released (by the way, you don't need to specify special Spring repository for Spring 3 any more - release is already in Maven Central)
slf4j is a very special kind of dependency - its runtime behavior depends on which implementation of slf4j you use. Therefore, to control its behavior you have to specify slf4j implementation explicitly
About management skills: to get useful information for maintaining your pom.xml you can use mvn dependency:tree (especially with -Dverbose=true) and mvn dependency:analyze. It can be also useful to check pom file of your dependency to look for optional dependencies.

First is the matter of transitive dependencies. As I understand it, if you provide a dependency, Maven will in turn find any dependencies of that dependency. That is great, but for many of my dependencies, this has not worked. (...)
As already pointed out, some dependencies may be marked as optional (and are not pulled transitively). The idea is that some dependencies are only used for certain features and will not be needed if that feature isn't used. If a user wants to use functionality related to an optional dependency, they will have to redeclare that optional dependency in their own project. So this just works as designed :)
Second is the management of repositories. Maven comes shipped with a default main repository, but I've found that in many cases this repository is not up to date. (...)
Even if the idea behind the concept of a central repo is noble, you can't objectively expect it to contain all jars in the world. One of the most obvious reason is that uploading artifacts to the Central Repository just takes time and resources are not infinite. And because companies like RedHat JBoss or SpringSource or Sun or even me need flexibility, reactivity (in one word, control), it's not surprising that they use their own repository. And, actually, I'm pretty happy that they expose them. But indeed, projects need to document where to find their artifacts if they are not available in central. Just in case, you may find this How to find dependencies on public Maven repositories? helpful. In a corporate environment, the best way to handle this would be to setup a centralized (corporate) proxying repository. See this page for such solutions.
Closely related to number 2 is ensuring you have the correct version of an artifact. (...)
Sorry but you need a bit to know what you're are doing. A project can't guess for you what JSTL version you are going to use. Then, regarding the various versions of artifacts, the naming convention used by projects has nothing to do with maven, this is a project/vendor choice (except for SNAPSHOT that maven handles specially). FWIW, common used schemes include: M1 = Milestone 1, RC1 = Release Candidate 1, GA = General Availability (final release), CR = Customer Release (often a bug-fix release). You may also see alpha, beta. This really depends on the project lifecycle and convention (nothing really unusual here though).
Finally, the issue of the type a dependency. I probably just don't understand this well enough, but many repo artifacts are of type "pom" not "jar". (...)
I think that you're indeed lacking of experience. You seem to be fighting with dependencies while things just go smoothly for me :) Maybe using a repository search engine will help.

You can also exclude some transitive dependencies, like the following example:
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.15</version>
<exclusions>
<exclusion>
<groupId>javax.mail</groupId>
<artifactId>mail</artifactId>
</exclusion>
<exclusion>
<groupId>javax.jms</groupId>
<artifactId>jms</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jdmk</groupId>
<artifactId>jmxtools</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jmx</groupId>
<artifactId>jmxri</artifactId>
</exclusion>
</exclusions>
</dependency>
I don't remember the details, but apparently some other dependencies were being discovered when we wanted to use log4j, and we had no interest (or need) in them in our case, so my cow orker just said "no thanks to these" and it works.

Transitive dependencies are only there if they have been encoded explicitly in your direct dependencies' POM's. Sometimes you actually do not want them, either because there are alternative implementations or you're using only part of a library and you don't want to drag in dependencies for things you're actually not using.
You are using a repository manager such as Nexus, aren't you? I advise you to setup one even if you code alone.
I find that Maven actually helps with this problem. I would hate having to perform this trial-and-error from ftp sites and download pages.
Yes, I hate that too :-)
Read the book! I actually learned most of what I know from this other book, but given it comes from the same people, I expect them to be rather interchangeable.