We Keep Coding

How do big companies tackle with the package dependencies conflict problem?

Just as shown in the picture, one app (Java) referenced two third-party package jars (packageA and packageB), and they referenced packageC-0.1 and packageC-0.2 respectively. It would work well if packageC-0.2 was compatible with packageC-0.1. However sometimes packageA used something that could not be supported in packageC-0.2 and Maven can only use the latest version of a jar. This issue is also known as "Jar Hell".
It would be difficult in practice to rewrite package A or force its developers to update packageC to 0.2.
How do you tackle with these problems? This often happens in large-scale companies.
I have to declare that this problem is mostly occurred in BIG companies due to the fact that big company has a lot of departments and it would be very expensive to let the whole company update one dependency each time certain developers use new features of new version of some dependency jars. And this is not big deal in small companies.
Any response will be highly appreciated.
Let me throw away a brick in order to get a gem first.
Alibaba is one of the largest E-Commerces in the world. And we tackle with these problems by creating an isolation container named Pandora. Its principle is simple: packaging those middle-wares together and load them with different ClassLoaders so that they can work well together even they referenced same packages with different versions. But this need a runtime environment provided by Pandora which is running as a tomcat process. I have to admit that this is a heavy plan. Pandora is developed based on a fact that JVM identifies one class by class-loader plus classname.
If you know someone maybe know the answers, share the link with him/her.

We are a large company and we have this problem a lot. We have large dependency trees that over several developer groups. What we do:
We manage versions by BOMs (lists of Maven dependencyManagement) of "recommended versions" that are published by the maintainers of the jars. This way, we make sure that recent versions of the artifacts are used.
We try to reduce the large dependency trees by separating the functionality that is used inside a developer group from the one that they offer to other groups.
But I admit that we are still trying to find better strategies. Let me also mention that using "microservices" is a strategy against this problem, but in many cases it is not a valid strategy for us (mainly because we could not have global transactions on databases any more).

This is a common problem in the java world.
Your best options are to regularly maintain and update dependencies of both packageA and packageB.
If you have control over those applications - make time to do it. If you don't have control, demand that the vendor or author make regular updates.
If both packageA and packageB are used internally, you can use the following practise: have all internal projects in your company refer to a parent in the maven pom.xml that defines "up to date" versions of commonly used third party libraries.
For example:
<framework.jersey>2.27</framework.jersey>
<framework.spring>4.3.18.RELEASE</framework.spring>
<framework.spring.security>4.2.7.RELEASE</framework.spring.security>
Therefore, if your project "A" uses spring, if they use the latest version of your company's "parent" pom, they should both use 4.3.18.RELEASE.
When a new version of spring is released and desirable, you update your company's parent pom, and force all other projects to use that latest version.
This will solve many of these dependency mismatch issues.
Don't worry, it's common in the java world, you're not alone. Just google "jar hell" and you can understand the issue in the broader context.
By the way mvn dependency:tree is your friend for isolating these dependency problems.

I agree with the answer of #JF Meier ，In Maven multi-module project, the dependency management node is usually defined in the parent POM file when doing unified version management. The content of dependencies node declared by the node class is about the resource version of unified definition. The resources in the directly defined dependencies node need not be introduced into the version phase. The contents of the customs are as follows:
in the parent pom
<dependencyManagement> 
    <dependencies > 
      <dependency > 
        <groupId>com.devzuz.mvnbook.proficio</groupId> 
        <artifactId>proficio-model</artifactId> 
        <version>${project.version}</version> 
      </dependency > 
</dependencies >
</dependencyManagement>
in your module ,you do not need to set the version
<dependencies > 
    <dependency > 
      <groupId>com.devzuz.mvnbook.proficio</groupId> 
       <artifactId>proficio-model</artifactId> 
    </dependency > 
  </dependencies > 
This will avoid the problem of inconsistency .

This question can't be answered in general.
In the past we usually just didn't use dependencies of different versions. If the version was changed, team-/company-wide refactoring was necessary. I doubt it is possible with most build tools.
But to answer your question..
Simple answer: Don't use two versions of one dependency within one compilation unit (usually a module)
But if you really have to do this, you could write a wrapper module that references to the legacy version of the library.
But my personal opinion is that within one module there should not be the need for these constructs because "one module" should be relatively small to be manageable. Otherwise it might be a strong indicator that the project could use some modularization refactoring. However, I know very well that some projects of "large-scale companies" can be a huge mess where no 'good' option is available. I guess you are talking about a situation where packageA is owned by a different team than packageB... and this is generally a very bad design decision due to the lack of separation and inherent dependency problems.

First of all, try to avoid the problem. As mentioned in #Henry's comment, don't use 3rd party libraries for trivial tasks.
However, we all use libraries. And sometimes we end up with the problem you describe, where we need two different versions of the same library. If library 'C' has removed and added some APIs between the two versions, and the removed APIs are needed by 'A', while 'B' needs the new ones, you have an issue.
In my company, we run our Java code inside an OSGi container. Using OSGi, you can modularize your code in "bundles", which are jar files with some special directives in their manifest file. Each bundle jar has its own classloader, so two bundles can use different versions of the same library. In your example, you could split your application code that uses 'packageA' into one bundle, and the code that uses 'packageB' in another. The two bundles can call each others APIs, and it will all work fine as long as your bundles do not use 'packageC' classes in the signature of the methods used by the other bundle (known as API leakage).
To get started with OSGi, you can e.g. take a look at OSGi enRoute.

Let me throw away a brick in order to get a gem first.
Alibaba is one of the largest E-Commerces in the world. And we tackle with these problems by creating an isolation container named Pandora. Its principle is simple: packaging those middle-wares together and load them with different ClassLoaders so that they can work well together even they referenced same packages with different versions. But this need a runtime environment provided by Pandora which is running as a tomcat process. I have to admit that this is a heavy plan.
Pandora is developed based on a fact that JVM identifies one class by class-loader plus classname.

How to handle version conflicts with my Java application using SPI extensions

I am writing a plugin API for a Java application, the idea being that eventually third parties will provide their own plugin extensions for the application and all the user needs to do is place the plugin jar into a plugins directory of the application. For the impatient, my question in short is how to handle possible version conflicts when the plugin relates to a different API than that on the system, see below for details of my situation and what I have thought about.
I have read a number of articles using service provider interfaces and have something working with that. The problem comes when considering how to deal with differing version combinations.
I am aware of the technique of when adding to an API adding extension interfaces, rather than changing the existing interface (eg. API 1.0 having MyInterface, API 1.1 adding MyInterface2 with the new methods, etc). With this technique if the user has the latest API then older plugins should work fine, but what happens if the user has an old API and newer plugins?
So as an example the user has API 1.0 only with MyInterface but installs binary plugin compiled against API 1.1 where the provider class implements MyInterface2. Whilst the application may only ever call plugins using MyInterface, what happens if the plugin internally calls MyInterface2? Will this cause an error or exception and when (IE. when the class is loaded or when the method from MyInterface2 is called). Also is this standard across JVMs or may it differ depending on the JVM used?
Finally, would it be better to use a plugin framework, would that be able to check version requirements? Searching the internet I find PF4J on github. A quick look in the source code seems to show it may support some sort of version checks.

Maximum Reusability for Two Implementations with Different Dependencies

I have a task that includes migrating an API Gateway from Zuul to Spring Cloud Gateway. There are two main versions currently: 1.0.1.RELEASE and 2.0.0.RC1. The first version is very basic and I'd have to manually implement filters related to rate limiting, authentication, etc...
While the second version has all the features we need with complete YML support. We have a strict rule in the company to never use beta or RC, and we need the first version of the gateway to be in production within a couple of weeks so there is not enough time to wait for the final release of version 2.
My team-leader specifically asked me to make 2 versions of using version 1.0.1 and 2.0.0 of SCG. How do you implement the module for maximum reusability? I mean I want switching between the two versions to be as easy as possible and I want to reuse as much of the logic as I can. The first thing that came to my mind is simply to create two separate projects. What do you think?

As I understand the question, you want an easy transition from the version 1.0.1.RELEASE to 2.0.0.RC1 of some dependency.
I would approach it as follows:
Create 3 modules (or projects):
api
bindings-1
bindings-2
The api module contains the API which you'll define to access functions of the dependency.
The bindings-1 and bindings-2 both implement what's defined in api, but based on the versions 1.0.1.RELEASE and 2.0.0.RC2 accordingly.
Your code will use the dependency only and exclusively via the api. No direct access to the classes and methods provided by the dependency. I would even not include the dependency as a compile-time dependency. You'll then import bindings-1 or bindings-2 depending on which version you want to use.
Having a separate api will require certain effort. It will seem overengineered. But if you don't do this, bindings to the dependency will diffuse in your code and switching from one version to another will be much more difficult.
With a dedicated api you will be forced to crystallize everything you need from the dependency in your api - in a version-independent manner.
I would also not develop bindings-1/bindings-2 as SCM branches. It's not like you'll be merging them, so why branches?

Removal of sun.misc.Unsafe in Java 9 will break Spring, Hibernate

I read here that Spring and many other popular libraries will break if Oracle removes sun.misc.Unsafe in Java 9. However, there are no static references to this class in Spring or Hibernate. So, is that claim true?
BTW there are 64 references to Unsafe in Java 8, but if Oracle removes that class they will update all of them and no library will be affected (unless they use Unsafe directly that is).

Mark Reinhold had a talk during JVM Language Summit 2015 titled The Secret History and Tragic Fate of sun.misc.Unsafe. Although these talks have plenty of disclaimers on them, you can see the proposed approach at 10:23, which is described in JEP260.
The general idea is:
replace existing functionality with safer, supported APIs
deprecate the previously existing Unsafe APIs that has been replaced
remove the deprecated code in the next version
Here is some relevant text from JEP260 (taken from October 20th 2015):
In JDK 9 we propose to:
Encapsulate all non-critical internal APIs by default: The modules that define them will not export their packages for outside use. (Access to such APIs will be available, as a last resort, via a command-line flag at both compile time and run time, unless those APIs are revised or removed for other reasons.)
Encapsulate critical internal APIs for which supported replacements exist in JDK 8, in the same manner and with the same last-resort workaround. (A supported replacement is one that is either part of the Java SE 8 standard (i.e., in a java.* or javax.* package) or else JDK-specific and annotated with #jdk.Exported (typically in a com.sun.* or jdk.* package).)
Not encapsulate critical internal APIs for which supported replacements do not exist in JDK 8 and, further, deprecate those which have supported replacements in JDK 9 with the intent to encapsulate them, or possibly even remove them, in JDK 10.
...
Critical internal APIs for which replacements are introduced in JDK 9 will be deprecated in JDK 9 and either encapsulated or removed in JDK 10.

Maybe the references are not in the core of Spring or Hibernate, but somewhere else. The document linked says with regard to Spring
Spring Framework (via Objenesis, with a fallback)
I tried to search for usages of Unsafe in the project I am currently working on, so there are still quite some libraries which may break.
result of quick search:
Guava
GWT
Netty
Jersey-Common
Infinispan
Jboss-Modules

This resource provides a proper understanding of the current status of JDK 9 and its features. The community started a discussion related to Unsafe and its future into the future of java. The given document is the effort of the community to react to JEP-260 that proposes hiding some internal APIs but leaving accessible some critical APIs, among witch Unsafe. As extracted from the document itself:
The critical internal APIs proposed to remain accessible in JDK 9 are:
sun.misc.Cleaner
sun.misc.{Signal,SignalHandler}
sun.misc.Unsafe (The functionality of many of the methods in this
class is now available via variable handles (JEP 193).)
sun.reflect.Reflection::getCallerClass (The functionality of this
method may be provided in a standard form via JEP 259.)
sun.reflect.ReflectionFactory
So to conclude, at least based on the given JEP, Unsafe should remain.

The answer is in the linked document. Spring does not have a direct dependency on Unsafe, but Spring depends on Objenesis and Objenesis depends on Unsafe.
Dependency for Objenesis on Unsafe: https://github.com/easymock/objenesis/blob/master/main/src/main/java/org/objenesis/instantiator/sun/UnsafeFactoryInstantiator.java
Spring's dependency on Objenesis is itself a bit strange. Spring's build script fetches the Objenesis binary and makes bytecode-level changes using the JarJar tool. You can see what it does in the following build script: https://github.com/spring-projects/spring-framework/blob/master/build.gradle (at time of writing, see lines 326-343, and 347).
This essentially means that Spring's "spring-core" binary ends up containing a load of classes under the org.springframework.objenesis.* package structure, but those classes were originally stored in source in Objenesis GitHub, published as a binary by the Objenesis team, fetched during Spring's build, repackaged to org.springframework.* packages and then republished as part of Spring. That's why you are having trouble finding them.
Spring uses Unsafe (via Objenesis) to create classes without first calling the constructor.

Targeting identical classes in different packages

I have created a library which supports an application, however in the newest version of the application the developer has changed the structure without changing the class names.
So version 1 of the application has classX in package A but version 2 has classX in package B. How can I develop my library in a way which allows supporting both of these in the same build?
Edit: My library is dependent on the application, not the other way around.

That is a bad decision, if you still want to make it work you need to provide skeleton classes with old structure and delegate calls to new version of class but it would get very dirty
better to not provide backward compatibility if you are firm with the renaming decision

Short answer: You can't.
Real answer: Your library should be able to exist independently of any application that uses it. The purpose of a library is to provide a set of reusable, modular code that you can use in any application. If your library is directly dependent on application classes, then it seems like a redesign should be seriously considered, as your dependencies are backwards. For example, have A.classX and B.classX both implement some interface (or extend some class) that your library provides, then have the application pass instances of those objects, or Class's for those objects, to the library.
If your "library" can't be designed this way then consider integrating it into application code, making it a direct part of the application, and come up with a better team workflow for you, the other developer, and others to work on the same project together.
Quick fix answer: Do not provide backward compatibility, as Jigar Joshi states in his answer.
Bad answer: You could hack a fragile solution together with reflection if you really had to. But please note that the "real answer" is going to last in the long run. You are already seeing the issues with the design you have currently chosen (hence your question), and a reflection based solution isn't going to prevent that from happening again (or even be reliable).