We Keep Coding

Maven - installed to local repo, not getting transitive dependencies

I have a maven project that I'd like to share with several of my other projects. It has some custom code, and then a few dependencies on things like log4j, jasypt, etc.
I build it and install it to my local maven repo. I can see it's successfully put there. Looking at .m2/repository/derp/Foo/1.0 it has a .pom with all its dependencies defined.
I can also define it as a dependency in my higher level projects, and it compiles.
<dependency>
<groupId>my.group</groupId>
<artifactId>Foo</artifactId>
<version>1.0</version>
</dependency>
I can see the Jar in my 'Maven Dependencies' in eclipse, and expanding that jar I can see it has the correct pom with dependencies in META-INF/maven/derp/Foo/pom.xml.
But my top level project above Foo isn't getting the dependencies that Foo needs. I get runtime exceptions, and I can see none of the transitive dependency jars are in my maven dependencies in eclipse.
What do I need to do to make sure Maven knows to look at the Pom for Foo in my local repo, so it will get all the transitive dependencies it needs?
----- edit -----
Answer to comment below, they are defined like this, with the dependencies tag at the top level under the project tag.
<dependencies>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.2</version>
</dependency>
etc...
These are the jars that maven correctly finds when I am just building and running this Foo project, but maven doesn't include these jars if a project depends on Foo, (Bar), and I find out when I try to run Bar.

What does "top level project above Foo isn't getting the dependencies" mean?
Anything "above" Foo in the build reactor should depend directly on Foo as you have stated. The <dependency/> specified will resolve to Foo's dependencies (within the scope that Foo specifies).
Without understanding the specifics of your project, it's improbable that we can to help you any further.
Some possible common situations:
You expect to be able to get access to test scoped dependencies in some non-test phase of execution. Just not true.
You expect that specifying a dependency on an artifact causes the java runtime to load those dependencies for you automagically. That's also not true. You'll want to invoke the exec:java goal on the maven exec plugin and specify your desired resolution scope within the <configuration/>, possibly also for that <execution/>.
You've mistaken <dependencyManagement> for <dependencies>. This happens way more than I would have expected.

How can I use dependencies of a project that's been marked as provided?

I have the following scenario, simplified:
projectX ---> projectA ---> projectB
Where ---> means "depends on".
ProjectB is really simple. It doesn't declare any dependenci. In fact, the only relevant part is this:
<packaging>jar</packaging>
In pom.xml of projectA I have declared the dependency to projectB:
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>com.mycompany</groupId>
<artifactId>projectB</artifactId>
<version>1.0.0</version>
<scope>provided</scope>
</dependency>
</dependencies>
And in pom.xml of projectX I have:
<packaging>war</packaging>
<dependencies>
<dependency>
<groupId>com.mycompany</groupId>
<artifactId>projectA</artifactId>
<version>1.0.0</version>
<scope>compile</scope>
</dependency>
</dependencies>
The problem is that projectX needs to use components (classes and such) that are defined in projectB. If I change the scope in projectA to use compile for projectB, everything will work, but then projectB will be included when generating the war in projectX, and I need to have this library out of the generated war because I'm providing projectB in other part of the project.
In the real scenario, I have several dependencies like projectB that affect projectA, so to reduce the size of the generated war, I would like to set them as provided, but then projectX cannot use the components defined in any of those libraries. Example of components: Spring, Hibernate, etc.
Question: Is there a way to achieve this in a clean way without re-declaring dependencies in lots of places?

The problem is that projectX needs to use components (classes and such) that are defined in projectB.
Then ProjetB should be a dependency of ProjectX indeed, in provided scope, again. Re-declaring the dependency would change how Maven would handle it as transitive dependency (that is, you can override its behavior by saying: I want this dependency in my project with this scope).
but then projectB will be included when generating the war in projectX, and I need to have this library out of the generated war
This will not happen by re-declaring it in provided scope. Alternatively, you could change the scope of ProjectB in ProjectA to compile and then configure the maven-war-plugin to exclude them, using inclusions/exclusions.
Before doing so though, you should double-check why semantically (or requirements-wise) ProjectB was set as provided in ProjectA and what would be its impact on other consumer projects.
Update
Both approaches above may suit your needs. Alternatively, as suggested in comments further options could be more sustainable and clear in the long run towards a better governance and maintenance (centralization):
Use a common parent pom where you could declare the list of shared dependencies, that is, dependencies that will be used by all children projects, defined (and maintained) once.
Use the dependencyManagement section in ProjectX to change how Maven dependencies mediation would handle transitive dependencies on ProjectA, without changing ProjectA or ProjectB (similar to option 1 at the top of this answer, with the difference that it will only be applied whenever the dependency comes into scope and ignored otherwise). To centralize this management, again, would be better to do so in a common parent (option above).

Maven EAR compiling

I have this dependency (and many others like this) :
<dependency>
<groupId>jShrink</groupId>
<artifactId>jShrink</artifactId>
<version>3.0.2</version>
<scope>system</scope>
<systemPath>${project.basedir}/../kladr-ear/lib/jShrink-3.0.2.jar</systemPath>
</dependency>
So how to add this JAR to EAR/LIB folder with MAVEN while package?

Looking at the available Maven scopes, I would expect you to mark this as
<scope>compile</scope>
and upload the jar file to your repository (local or remote). Maven should give you the appropriate command line to perform that upload.
EDIT: As noted below, this scope is the default, and as such you could omit it.

Dependencies going into the EAR should generally be based on he dependencies needed for EJB modules. If you have an EJB module with library X on compile scope then that will trigger Maven to want to package the library in the ear that the EJB is part of. No need to manage any scopes at all in the ear pom.
If you don't have any EJBs and only WARs, I would not package libraries in the EAR but simply deploy them as part of the WAR (so they end up in WEB-INF/lib). Keep the web dependencies nice and contained per module, even if that means you get duplicates.
Also compile is the default scope, so you don't ever need to manually declare it unless you're doing it to override another scope set for the dependency.

Maven dependency order

In my project there is a class implementing an interface. The interface comes from a dependency. I have another dependency that itself has a dependency on a jar that also contains the same interface, except a version with more methods; the two jars containing the same package-interface DON'T have the same groupId or artifactId.
Compilation is failing because the compiler complains that the class in my project is not implementing all the methods. I realized it is because the compiler is taking the interface reference from the wrong jar. My question is, why is maven using the interface from the transitive dependency instead of the one from the jar that I explicitly mention in the project POM? I can see the jar that is used appears earlier in the definition (so i imagine in the classpath as well), but i thought in these cases maven resolved it by using the colliding class/interface from the dependency with the shortest path
Here is part of the dependency tree. Note that this is grepped but it can still be seen that javax.servlet:servlet-api (the one actually used) is deeper in the tree compared to tomcat:servlet (the one that should be used)
[builder#ca-rd-build11 proj]$ mvn dependency:tree | grep servlet
[INFO] | +- javax.servlet:servlet-api:jar:2.4:compile
[INFO] +- tomcat:servlet:jar:4.0.6:compile
I'm using maven 3.0.4

why is maven using the interface from the transitive dependency
instead of the one from the jar that I explicitly mention in the
project POM?
Because, to Maven, the two have nothing to do with each other. Maven doesn't know about class or package names, Maven just knows about groupId and artifactId. Since those are not the same, maven doesn't have any reason to omit the transitive dependency.
And if I am correct, Maven places the dependencies on the classpath in the order they are defined, i.e. the transitive dependencies of dependency a appear before dependency b.

When you declare your dependency on the other jar file you can tell it to exclude the transitive dependency on the conflicting jar file:
<dependency>
<groupId>group</groupId>
<artifactId>artifact</artifactId>
<version>1.0.0</version>
<exclusions>
<exclusion>
<groupId>othergroup</groupId>
<artifactId>ArtifactToExclude</artifactId>
</exclusion>
</exclusions>
</dependency>

For future visitors (since the title is generic):
In Maven you have two categories of dependencies:
(external) dependencies: what you define directly in your pom.xml files inside <dependencies>
transitive dependencies: the dependencies your pom.xml dependencies require (automatically included, together with their dependencies, and so on)
As per the official Maven documentation the following mediation is applied when multiple versions are encountered as dependencies:
Maven picks the "nearest definition". That is, it uses the version of the closest dependency to your project in the tree of dependencies. You can always guarantee a version by declaring it explicitly in your project's POM. Note that if two dependency versions are at the same depth in the dependency tree, the first declaration wins.
Example: if dependencies for A, B, and C are defined as A -> B -> C -> D 2.0 and A -> E -> D 1.0, then D 1.0 will be used when building A because the path from A to D through E is shorter. You could explicitly add a dependency to D 2.0 in A to force the use of D 2.0.
In the above example, A is the actual project, B and E are external dependencies defined in its pom.xml file, and C and D are transitive dependencies.

Create dependency groups in Maven for reuse - including 'provided' dependencies

I am new to Maven and am setting up my first maven project. I am also creating some maven assets in the form of some poms that can be inherited from or used as dependencies in any future projects as well. I want to group dependencies together and to be able to selectively add them to a project as needed.
I read this article on pom best practices. I like the idea of grouping related dependencies together into poms and then adding the pom as a dependency to a project as needed. This approach works great for compile scoped dependencies. However it fails for provided scoped ones since as transitive dependencies, they get omitted.
Here's an example of what I mean: Lets say I group together web dependencies for my projects into a web-deps pom.xml. These include compile scoped spring framework dependencies and also a provided scoped javaee one:
<modelVersion>4.0.0</modelVersion>
<groupId>com.xyz</groupId>
<artifactId>mvn-web-deps</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>pom</packaging>
<dependencies>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-web</artifactId>
<version>${org.springframework.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>${org.springframework.version}</version>
</dependency>
<dependency>
<groupId>javaee</groupId>
<artifactId>javaee-api</artifactId>
<version>${javaee.version}</version>
<scope>provided</scope>
</dependency>
</dependencies>
I then add this pom as a dependency in another project:
<modelVersion>4.0.0</modelVersion>
<groupId>com.xyz</groupId>
<artifactId>project-a</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependency>
<groupId>com.xyz</groupId>
<artifactId>mvn-web-deps</artifactId>
<version>0.0.1-SNAPSHOT</version>
<type>pom</type>
</dependency>
The dependencies in mvn-web-deps now become transitive. Since the dependency reference above is compile scoped, the provided transitive dependency gets omitted.
I want to avoid adding them to the dependency section of a parent since there can only be one parent and a project may need only some of these dependency groups, not all. I can perhaps add them to the dependencyManagement section, but then I will have to redeclare each dependency (sans the version) in each child project.
What is the correct/better way of grouping dependencies while avoiding the issues like above?

The short answer to your question is that you should only include 'provided' dependencies locally where the code requires it to compile, but not in parent pom.xml or other structures. Indicating that you have a 'provided' dependency in global pom.xml is non-sense for maven, because it does not need it to compile in such pom.xml.
Here is the long answer:
When I started using Maven, I had the same idea of trying to group artifacts and dependencies into pom.xml modules hoping they would be useful in the future. Now, that I have a bit more experience, I got to understand that it is a complete waste of time. For me, this was form of over-engineering.
I have learned to split my big projects into separate modules, each in their own subversion repository. I am including dependencies as necessary for each local module in their pom.xml. I release versioned tags of each module as I am coding and as necessary (i.e., when tested and stable).
I build my big projects by creating a separate maven project with its own pom.xml and import my modules as dependencies. From time to time, I update the module's version in the dependency when I have made a release. Then, I let maven do the job of pulling whatever it has to pull, transitively of not, when compiling/releasing the big project.
Maven allows all sorts of complex constructions and hierarchy between pom.xmls, but IMHO this feature creates unnecessary mess and complexities. So far it has not proved to be a real benefit for me. At the beginning, I was hoping that compiling one pom.xml would compile the rest properly in a cascading way. I did get some result, but what a mess to maintain in all the global pom.xml.
Releasing my module's artifacts separately and building my project on these releases has saved me so much time that I can only recommend it. In total, I have less pom.xml to maintain and they are also less complex. For the same final result...
So, if your only reason for building global/structural pom.xml is a hope to save time, I recommend abandoning this idea... Separate code in separate projects, release and THEN compile globally.

I concluded that Maven was not designed for this kind of use-case. I ended up having a parent pom.xml with all the libraries I use added to its <dependencyManagement> section. Any new projects/modules that I create have their pom.xml inherit from the parent pom.xml and add each dependency they need to their own <dependencies> section, minus the version. This scheme allows me to manage the versions for the libraries that I use and the respository declarations they need at a single place. Another advantage (over trying to create dependency bundles somehow) is that this gives more fine-grained control over the libraries added to child poms - only the dependencies that are actually needed are added.

Provided-scope dependencies are indeed inherited from parent POM, but NOT from POM defined as dependencies and I consider that a Maven weakness.
Given that Maven has also difficulties in adding modules as dependencies across module hierarchies, I can't say Maven is a sophisticated tool to manage multi-module projects. Maven expects a strict single-rooted hierarchy that is only suitable for the simplest projects.