We Keep Coding

How can I have a multi-module Maven project without specifying the version all over the place?

I am currently learning Maven for a specific project. In this project, I need to have multiple modules (in the sense of a collection of classes and resources) that depend on each-other. Specifically, I need a module to store the classes of a public-facing API (the API is a Java API, as the desktop program can load addons at runtime), another for a set of common classes that are shared between the client and server but that are not part of the API, and of course the client and server themselves need a module each.
Using the system that I used before Maven (IntelliJ Idea's build system), I would simply create different modules and setup dependencies using IntelliJ's GUI. This works well because I can then put the whole IntelliJ Idea project in a git repository, and keep everything nicely tracked together with no fuss and no problems.
When researching how to do this in Maven, however, I ran into some problems. Maven's documentation has a section that seems to explain how to accomplish something similar, but its technique seems to have two problems. When Each sub-project, with its own pom.xml, needs to specify the version of the other sub-projects that it depends on¹. This implies that I need to make many changes all over the project whenever the version changes. This will (hopefully) be quite frequent. As it is perfectly plausible that my project will grow to have hundreds of modules, this is obviously impractical.
Additionally, based on my other research, it seems like a Maven repository is involved in the documented technique. Specifically, it seems like when, for example, the API module is built to be included in the client and server modules it will first be placed in a local repository and then the client and server modules will retrieve it from there. This sounds problematic as the API module will also be published in a public repository (maybe Maven central, I haven't really thought about this too much -- but it will be public in a repo), and this repository step seems like it could end up building the client and server with a published jar rather than the local one, which is problematic for many reasons. (eg. if a developer is making local changes to the api jar, building half the project without those changes is a problem).
Is there some better way to combine multiple modules to mitigate these problems (versions all over the place, and getting wrong jars from repos)? Is it better to modify the technique in the linked documentation instead? Am I misunderstanding something and the problems don't exist?
I have checked DuckDuckGo and all relevant questions that I can find on StackOverflow, and none address either of the two aforementioned problems.
¹:
<dependencies>
<dependency>
<groupId>com.mycompany.app</groupId>
<artifactId>my-app</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
</dependencies>

In your multi-module project, you usually use just one version for all modules. This means that you can define dependencies between modules with <version>${project.version}</version> where the property ${project.version}$ is resolved during the build. Maven builds all the modules in the correct order.

Regarding your first question: you can create a parent pom over your whole project. In this parent pom you can add a dependencyManagement section where you can list your own modules with their version. In your modules you then only need to specify the dependeny without the version.
Here an example:
in parent pom:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>yourgroup</groupId>
<artifactId>yourmoduleA</artifactId>
<version>3.0.1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>yourgroup</groupId>
<artifactId>yourmoduleB</artifactId>
<version>3.0.1-SNAPSHOT</version>
</dependency>
</dependencies>
</dependencyManagement>
and if your module A uses module B, you can add in pom of module A
<dependencies>
<dependency>
<groupId>yourgroup</groupId>
<artifactId>yourmoduleB</artifactId>
</dependency>
</dependencies>

Understanding Maven dependencies and assembly

I am not very much experienced with Maven and it's compilation and packaging logic gets me confused.
I have some dependencies declares as :
<dependency>
<groupId>com.dependency_group</groupId>
<artifactId>dependency_1</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>com.dependency_group</groupId>
<artifactId>dependency_2</artifactId>
<version>1.0.0</version>
<scope>provided</scope>
</dependency>
So as far as I understand, dependency_1 will be added to the classpath of my program as something that comes along with my jar, and dependency_2 on the other hand, will be added to the classpath as something that the system runtime will provide upon deployment.
Then I run the package goal of Maven and none of my dependencies are packed with my code (I am using the shade plugin, but even without it nothing changes).
I expected that when some dependency is set as compile scope, it will be exported with my compiled code, since AFAICS, there's no point in setting the classpath saying a dependency will come along with my code, and Maven just don't package that dependency with it. It looks to me as if Maven is not obeying it's contract.
So:
1 - What's the logic behind this?
2 - Do I have to always use the Assembly plugin?
3 - Are there cases where people will define a dependency as compile and will not want it packaged within a jar?

Let me shed some light on the main point here. There are fundamentally two kinds of java artifacts:
Applications, i.e. ears, wars, executable jars
Libraries, i.e. jars that are meant to be used as dependencies for other artifacts.
For Applications your reasoning makes perfectly sense. Wars and Ears automatically package all their compile dependencies and you need no assembly plugin for that. For libraries, you do not pack the dependencies into the library. Maven handles transitive dependency resolution and would be confused if you put a fat jar on the classpath.
The thing is that packaging jar can be both a libary or an application. If you want a standalone application, you need to tell Maven to package everything, e.g. by using the assembly plugin or shade plugin.

You use compile scope when you want some dependencies to come along with your code. For example you want Jackson to be a part of your application if you are using it for json serialization.
You use provided scope, if you want dependency to be on the classpath during the compilation but wont be included within your application. It must be provided by running environment. For example you want Lombok as it is compile only library, or you want to have Servlet Api dependency as provided when you are writing servlet application because such app will be ran on servlet container thus there is no need to pack it within your application (it will be available in container runtime)
Do I have to always use the Assembly plugin
Nobody forces you to do so.

Maven - installed to local repo, not getting transitive dependencies

I have a maven project that I'd like to share with several of my other projects. It has some custom code, and then a few dependencies on things like log4j, jasypt, etc.
I build it and install it to my local maven repo. I can see it's successfully put there. Looking at .m2/repository/derp/Foo/1.0 it has a .pom with all its dependencies defined.
I can also define it as a dependency in my higher level projects, and it compiles.
<dependency>
<groupId>my.group</groupId>
<artifactId>Foo</artifactId>
<version>1.0</version>
</dependency>
I can see the Jar in my 'Maven Dependencies' in eclipse, and expanding that jar I can see it has the correct pom with dependencies in META-INF/maven/derp/Foo/pom.xml.
But my top level project above Foo isn't getting the dependencies that Foo needs. I get runtime exceptions, and I can see none of the transitive dependency jars are in my maven dependencies in eclipse.
What do I need to do to make sure Maven knows to look at the Pom for Foo in my local repo, so it will get all the transitive dependencies it needs?
----- edit -----
Answer to comment below, they are defined like this, with the dependencies tag at the top level under the project tag.
<dependencies>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.2</version>
</dependency>
etc...
These are the jars that maven correctly finds when I am just building and running this Foo project, but maven doesn't include these jars if a project depends on Foo, (Bar), and I find out when I try to run Bar.

What does "top level project above Foo isn't getting the dependencies" mean?
Anything "above" Foo in the build reactor should depend directly on Foo as you have stated. The <dependency/> specified will resolve to Foo's dependencies (within the scope that Foo specifies).
Without understanding the specifics of your project, it's improbable that we can to help you any further.
Some possible common situations:
You expect to be able to get access to test scoped dependencies in some non-test phase of execution. Just not true.
You expect that specifying a dependency on an artifact causes the java runtime to load those dependencies for you automagically. That's also not true. You'll want to invoke the exec:java goal on the maven exec plugin and specify your desired resolution scope within the <configuration/>, possibly also for that <execution/>.
You've mistaken <dependencyManagement> for <dependencies>. This happens way more than I would have expected.

Introducing dependency breaks existing dependency?

I'm working on a project that builds and deploys fine. I'm trying to add some code that uses JWebUnit, and use the following Maven code to bring it in:
<dependency>
<groupId>net.sourceforge.jwebunit</groupId>
<artifactId>jwebunit-htmlunit-plugin</artifactId>
<version>3.2</version>
<scope>test</scope>
</dependency>
Maven seems to resolve this fine and it's bringing everything in (I'm using Intellij, and it now appears under 'Dependencies' in the 'Maven Projects' tab, and also under 'External Libraries' in the Project tab).
However, when I bring this dependency in, the IDE is not able to find it (e.g. if I use import net.sourceforge.jwebunit.junit.WebTester, it can't find it).
But an even bigger issue is it actually breaks some existing code -- I have some JUnit tests that use org.apache.commons.httpclient.HttpClient, and now on Maven's install goal I get a
NoClassDefFoundError - Could not initialize class for that class.
If I remove the JWebUnit dependency, the Maven install goal exits successfully.
I'm used to seeing errors about dependency version convergence when bringing new dependencies, and I feel like chasing this 'no class def found' error could be a red herring, but I'm not sure of the general types of issues in Maven that could be causing it.
EDIT: the dependency code for pulling in HttpClient is:
<dependency>
<groupId>commons-httpclient</groupId>
<artifactId>commons-httpclient</artifactId>
<version>3.1</version>
</dependency>

jwebunit-htmlunit-plugin includes transitive dependencies which seem like they're conflicting with some of your dependencies (likely because they are different versions).
Maven puts classpath priority on artifacts declared earlier. Try moving jwebunit to the end of your dependencies section, or at least after where you pull in the httpclient classes. Alternatively, you can manually exclude certain transitive dependencies from being pulled in, but this can be tedious.
As for your IDE not allowing imports on the library, remember that you have this declared in the test scope. Production classes cannot see test dependencies.

Create dependency groups in Maven for reuse - including 'provided' dependencies

I am new to Maven and am setting up my first maven project. I am also creating some maven assets in the form of some poms that can be inherited from or used as dependencies in any future projects as well. I want to group dependencies together and to be able to selectively add them to a project as needed.
I read this article on pom best practices. I like the idea of grouping related dependencies together into poms and then adding the pom as a dependency to a project as needed. This approach works great for compile scoped dependencies. However it fails for provided scoped ones since as transitive dependencies, they get omitted.
Here's an example of what I mean: Lets say I group together web dependencies for my projects into a web-deps pom.xml. These include compile scoped spring framework dependencies and also a provided scoped javaee one:
<modelVersion>4.0.0</modelVersion>
<groupId>com.xyz</groupId>
<artifactId>mvn-web-deps</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>pom</packaging>
<dependencies>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-web</artifactId>
<version>${org.springframework.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>${org.springframework.version}</version>
</dependency>
<dependency>
<groupId>javaee</groupId>
<artifactId>javaee-api</artifactId>
<version>${javaee.version}</version>
<scope>provided</scope>
</dependency>
</dependencies>
I then add this pom as a dependency in another project:
<modelVersion>4.0.0</modelVersion>
<groupId>com.xyz</groupId>
<artifactId>project-a</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependency>
<groupId>com.xyz</groupId>
<artifactId>mvn-web-deps</artifactId>
<version>0.0.1-SNAPSHOT</version>
<type>pom</type>
</dependency>
The dependencies in mvn-web-deps now become transitive. Since the dependency reference above is compile scoped, the provided transitive dependency gets omitted.
I want to avoid adding them to the dependency section of a parent since there can only be one parent and a project may need only some of these dependency groups, not all. I can perhaps add them to the dependencyManagement section, but then I will have to redeclare each dependency (sans the version) in each child project.
What is the correct/better way of grouping dependencies while avoiding the issues like above?

The short answer to your question is that you should only include 'provided' dependencies locally where the code requires it to compile, but not in parent pom.xml or other structures. Indicating that you have a 'provided' dependency in global pom.xml is non-sense for maven, because it does not need it to compile in such pom.xml.
Here is the long answer:
When I started using Maven, I had the same idea of trying to group artifacts and dependencies into pom.xml modules hoping they would be useful in the future. Now, that I have a bit more experience, I got to understand that it is a complete waste of time. For me, this was form of over-engineering.
I have learned to split my big projects into separate modules, each in their own subversion repository. I am including dependencies as necessary for each local module in their pom.xml. I release versioned tags of each module as I am coding and as necessary (i.e., when tested and stable).
I build my big projects by creating a separate maven project with its own pom.xml and import my modules as dependencies. From time to time, I update the module's version in the dependency when I have made a release. Then, I let maven do the job of pulling whatever it has to pull, transitively of not, when compiling/releasing the big project.
Maven allows all sorts of complex constructions and hierarchy between pom.xmls, but IMHO this feature creates unnecessary mess and complexities. So far it has not proved to be a real benefit for me. At the beginning, I was hoping that compiling one pom.xml would compile the rest properly in a cascading way. I did get some result, but what a mess to maintain in all the global pom.xml.
Releasing my module's artifacts separately and building my project on these releases has saved me so much time that I can only recommend it. In total, I have less pom.xml to maintain and they are also less complex. For the same final result...
So, if your only reason for building global/structural pom.xml is a hope to save time, I recommend abandoning this idea... Separate code in separate projects, release and THEN compile globally.

I concluded that Maven was not designed for this kind of use-case. I ended up having a parent pom.xml with all the libraries I use added to its <dependencyManagement> section. Any new projects/modules that I create have their pom.xml inherit from the parent pom.xml and add each dependency they need to their own <dependencies> section, minus the version. This scheme allows me to manage the versions for the libraries that I use and the respository declarations they need at a single place. Another advantage (over trying to create dependency bundles somehow) is that this gives more fine-grained control over the libraries added to child poms - only the dependencies that are actually needed are added.

Provided-scope dependencies are indeed inherited from parent POM, but NOT from POM defined as dependencies and I consider that a Maven weakness.
Given that Maven has also difficulties in adding modules as dependencies across module hierarchies, I can't say Maven is a sophisticated tool to manage multi-module projects. Maven expects a strict single-rooted hierarchy that is only suitable for the simplest projects.