We Keep Coding

How the class loader works in Java with a shaded jar

My experience with Java class loading is limited. With tools like Maven I have a rusty understanding of how they resolve dependency versions. But I've hit a problem that's making me question how Java loads classes.
My scenario, I have a dependency on version 30.1.1-jre of com.google.guava. I also have a shaded jar which has a dependency on Guava 18.0.
In my application I end up seeing the following exception
java.lang.IncompatibleClassChangeError: Class com.google.common.base.Suppliers$MemoizingSupplier does not implement the requested interface java.util.function.Supplier
which I cannot reproduce locally. Based on https://github.com/crabhi/celery-java/issues/9 it sounds like this error is produced when an older version of Guava is on the classpath.
Checking the classes in the war I see
WEB-INF/lib/java-driver-shaded-guava-25.1-jre-graal-sub-1.jar.d/com/datastax/oss/driver/shaded/guava/common/base/Suppliers$MemoizingSupplier.class
WEB-INF/lib/nautilus-es2-library-2.3.4.jar.d/com/google/common/base/Suppliers$MemoizingSupplier.class
WEB-INF/lib/guava-30.1.1-jre.jar.d/com/google/common/base/Suppliers$MemoizingSupplier.class
This makes me think the shaded jars are causing a problem.
Is that possible? Are there any articles explaining how classes are loaded when shaded jars enter the picture?

You have two copies of com.google.common.base.Suppliers.MemoizingSupplier on the classpath. Only one can be loaded, and in your case, it's the older version.
You shouldn't have multiple classes with the same name available to one classloader. What is nautilus-es2-library-2.3.4.jar and why does it bundle Guava rather than expressing it as a transitive dependency?

Maven repackage transitive dependencies

The app consists of modules and each module is particular maven project with one parent. These modules are used as jars in main app.
Each module uses third party SDK via maven.
These SDKs have many dependencies and these dependencies can be with the same group and artifact but with different versions. It is hard to manage them and we have problems with 'NoSuchMethodError's. As for me the simplest way to fix problem is to get sources of problematic dependencies and source of SDK, change the package (add some prefix) and imports and put it inside sdk jar.
But this approach requires: one more git repository for forked SDK's jars and private maven repository to publish them and a lot of time.
Is there maven plugin that can help with this problem, for example plugin that can build jar (our module jar) with all dependencies included inside it but can change package of all dependencies?

You can try to use the Maven shade plugin which allows you to relocate classes
https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
but I have never tried this myself.
In most cases it is better to try to align the dependencies so that you use the same versions in most places and also to reduce dependencies to a minimum.

Congratulations, you are confronted with a classpath problem called "jar hell". The only safely working solution i know is the use of different classloaders for each version of the same class (if it's not backward compatible).
A class is identified by its qualified name (package and class name) and by the classloader it has been loaded. If the application uses the default system classloader and there is the same class in different versions in the classpath only one version will be loaded by the classloader. In the oracle JDK/JRE it is the one which occurred first in the classpath. If you are lucky you can bring the classpath (order and excluding dependencies) in a special order in which your application will run, but i wouldn't recommend it since it relies heavy on the JDK/JRE implementation.
OSGI is a technology which might be helpful for this since it provides a module based framework for using different classloaders.

Java 9 modules and uber-jar

When developing Java libraries we're currently using the Apache Maven Shade Plugin to hide internal dependencies on other libraries (jars) by renaming their package names.
Is it possible to hide these internal library-dependencies by using the Java 9 module system and not exporting the name of the internally used libraries?
I.e:
Both module A and B include, but does not export, class org.codehaus.jackson.map.ObjectMapper (included using e.g. Maven Shade plugin) with different versions for the class
Module A uses module B
Will each module still use its implementation org.codehaus.jackson.map.ObjectMapper?
I believe it should by so, but I have found no documentation explicitely confirming this, nor any texts / examples recommending this approach for this quite usual versioning issue.

This issue is described as http://openjdk.java.net/projects/jigsaw/spec/issues/#MultiModuleExecutableJARs and there's no support for it yet. In case all dependencies are modules, it would make sense to use jlink to solve this. But as long as there is at least one non-module, there's no solution available yet. This is something that needs to be solved within the JDK/JRE.
It is still a valid case, so I would suggest to ask this question the at jigsaw-dev mailinglist and refer to #MultiModuleExecutableJARs

Issues excluding transitive dependency of project reference from eclipse class path

I have several gradle projects in my eclipse workspace. For the sake of simplicity I'm only really interested in 2 of them, let's just use A and B for this.
So the problem I'm having is that Project A has an included dependency on JBoss, which pulls in javax validation-api 1.0.0.GA, and Project B has a dependency on javax validation-api 1.1.0.Final. Since Gradle itself resolves the conflict by using the newer library first, B is happy when built by gradle. But Eclipse itself includes errors which are very distracting while editing.
The correct version of the validation-api jar ends up in B's class path but the problem is that the Gradle IDE plugin changes the project(':A') dependency to a project reference, and Eclipse seems to give the project reference precedence over the external jar. So the old jar is preferred by extension.
I tried adding { exclude module: 'validation-api' } in B's build.gradle for the dependency on A which works according to the output of 'gradle dependencies', however since Eclipse just gets as far as making it a project reference, it won't exclude the jar and the problem remains.
Also per this question I tried adding { transitive = false } and the same thing happens. I don't think even the hack posed there would work for me since the .classpath contains a single reference to the Gradle container so there's nothing to remove.
I've managed to get around this by explicitly including a reference to the correct version of the jar from my gradle cache and then moving it above the Gradle Classpath Container so that eclipse sees that version first.
My question is: Is there a better/more generic way to do this? Preferably one that I can commit to source control without breaking other people's builds or requiring them to manually modify paths or properties somewhere? There is another project with what appears to be a similar issue so something I can fix in the build.gradle file would be awesome.
Worst case scenario, I could probably switch to IntelliJ if that behaves itself better than the Eclipse-Gradle integration?

These kind of transitive dependency issues are long-standing problem with Gradle Eclipse integration (both in STS tooling and also commandline generated .classpath metadata from Gradle's Eclipse plugin. The problem is the way that Eclipse computes transitive classpaths.
Only recently we found a reasonable solution to this problem. Actually there are now two solutions, one better than the other but depending on your situation you might want to use either of them.
The first solution is a bug fix that changes the classpath order of project dependencies so that they are no longer 'preferred' over jar dependencies PR-74. To get this fix you may need to install gradle tooling from a snapshot update site because the fix went in after 3.6.3.
This solution doesn't fix the real problem (you still have the 'wrong' stuff on the classpath) but just makes it less likely to cause real problem in your projects.
The second solution is to enable use of the 'Custom Tooling API model' PR-55 introduced in STS 3.6.3. This is a bit experimental and only works for recent version of Gradle, at least 1.12 but probably better to use 2.x. It also only works for projects that have 'Dependency management' enabled (if not enabled you are using the .classpath generated by Gradle's eclipse plugin which has the same 'broken' classpath issues as the STS tooling).
The 'custom tooling model' is really the better solution in principle as it fixes the way gradle classpath get mapped to eclipse projects so that project dependencies are no longer exported and each project gets its own classpath considering dependencies conflict resolution.
To enable this go to "Window >> Preferences >> Gradle" and enable checkbox "Use Custom Tooling Model".

org.apache.http.entity.ContentType is not in Apache anymore?

I can't import org.apache.http.entity.ContentType for some reason. I have added Apache HTTP 4.3.3:
client
core
commons-codec
httpclient-cache
commons-logging
httpmime
fluent-hc
originally I didn't import all of these, but since the problem persists I've done it all now, and I have cleaned my project and rebuilt it
import org.apache.http.entity.mime.MultipartEntityBuilder; works
and
import org.apache.http.HttpEntity; also works
but
import org.apache.http.entity.ContentType; is not found at all
I am using Android Studio (IntelliJ) so I'm not sure build path answers are helpful here
I also tried importing via gradle link, but I encountered a different error and need more control over the jar files myself

org.apache.http.entity.ContentType is in the org.apache.httpcomponents:httpcore:4.3.x module. But so is org.apache.http.HttpEntity. The fact you are finding one and not the other most likely means they are not being pulled in from the v4.3.x JAR. The Apache HttpCommponents project moved a lot of classes around recently. So it may be finding HttpEntity in an older/different version of httpcore that is getting pulled in somehow; a version that does not have the ContentType class. You'll want to verify where it is finding HttpEntity. There are a few ways to see from where a dependency is getting pulled in from.
Option 1
One of the easiest ways is to put your cursor on the class (either in the import statement or a variable declaration) and open the quick documentation (Ctrl+Q or ⌃J). At the top of the documentation dialog, it will show the dependency where the class is found:
Option 2
This option will show you if you have multiple instances of a class on your class path. Open the Goto Class dialog (Ctrl+N or ⌘O) and enter the class name (you can fully qualify or not. You can even paritally enter the name and use camel case search). In the list of classes that are found, to the right will be where the class is found.
TO the right is the dependency from which the class is coming from. If it shows more than once, it means that it is being pulled in multiple times in different jars. This can happen (when using a build tool) if another dependency is pulling in a different version of httpcore as a transitive dependency. So you will need to resolve that. Notice in my screen shot I have two different versions of the org.springframework.http.HttpEntity class. One from Spring 3.2.10 and one from 4.0.6. In this particular case it is because I have a Spring 3.2.x based module and a separate Spring 4.0.x module in my project. So I'm OK with that as they do not clash since those are independent modules. But in most cases, that would be cause for concern.
Option 3
You can also expand and look at the External Libraries node in the Project Tool window to see if a dependency is being pulled in twice.
It's unclear how your project is configured. You mentioned gradle, but it seemed more like an after thought. If you are using maven or gradle, and the proper httpcore dependency is declared in your pom or build file, make sure you perform a reimport on the appropriate build tool window so that it synchronizes properly and that module is added to the project's dependency. Then use the above information to see if you have multiple versions of the httpcore module being pulled in.

I had the same issue. You have to make sure you are using jdk 1.7 and not 1.6.

Download httpcore-4.3.2 jar and add to lib folder.