We Keep Coding

How to better structure the artifacts of a java library with plugins

Let's say I have a library, which provides two independent plugin interfaces, 2 implementations per plugin, and one parent POM file.
There are also some abstract tests in the "core", which the plugins have to implement and pass to be considered compliant.
From code perspective, the plugin interfaces don't depend on core at all.
Only core depends on plugins.
My assumptions are:
the core abstractions go into the "core" artifact and its tests are packaged in a test-jar.
each implementation of a "plugin" goes into a separate artifact (4 artifacts in this example).
the parent POM also goes into a separate artifact.
I have considered several options about how to structure the dependencies between each artifact, which can be boiled down to these 2:
Leave it at just 6 artifacts. Every "plugin" depends on "core". Every "plugin" and "core" all depend on parent artifact.
This makes it possible for the library users to only specify 2 artifacts in their pom.xml/build.gradle, because "core" is a transitive dependency.
BUT, given I have some changes to the "core" which cause a version bump, I would have to update every plugin implementaton to bump the dependency version. Also, if users didn't specify the core explicitly, they now depend on outdated core.
Extract plugin interfaces into separate artifacts, so that implementations no longer depend on core. Which now creates 8 artifacts.
Now, unlike the previous approach, library users can no longer skip the "core" dependency in their pom.xml/build.gradle - it is now a given that it has to be specified. Which means, they would have to depend on 3 artifacts.
But, overall, any update of the core no longer forces a cascading update of the plugins. The plugin implementations need version bumps only if their respective interface updates.
The downside is probably that I now have 2 more artifacts to maintain.
My questions are:
Which approach is the more correct one? Does it depend on project size or some other factors?
Are there other approaches?
Is it bad that users have to depend on plugins & "core" explicitly, even if plugins transitively bring "core" in the first approach?
Anything that is intrinsic to the problem and cannot be solved? (like, is it a given that 8 artifacts are to be maintained, with no way to minimize that?)
Is it correct to provide abstract tests in the "test-jar", if I want to make sure that all plugin implementations comply with the interface contracts? Or do I have to copy-paste the tests in each plugin implementation?
Reply to #vokail
Generally, If you release a new version of the core, you must release a new version of the plugin, right?
Currently the code is structured in such a way, that plugin have no dependencies on the core. With 1st scheme, if core updates, plugins must update. With 2nd scheme - if core updates, plugins don't care.
I think it's possible to have more than two plugins implementations
for plugin developers they need to use only this as dependency directly
True & true
plugin-api need only core-api
Currently, I cannot invert the dependency in such a way. Plugins know nothing about the core, except the plugins API.
As a note, there are 2 plugin APIs. Their code doesn't depend on core and their code doesn't depend on each other.
With 1st scheme, all plugin APIs are inside a single core artifact.
With 2nd scheme all plugin APIs are in separate artifacts (so it's 1 core artifact, and 2 separate API artifacts = 3 artifacts in total).
core-api can be implemented by more than one core-impl ( in the future )
Mhm... Don't see it in the future.
It's better to depend for my plugin implementation from an interface only, not from a core one
To clarify, this is what I meant.
From library user perspective, 1st scheme looks like this:
// Implementaton of "A" api, variant 1
implementation 'library:plugin-a1-impl:1.0.0'
// Implementaton of "B" api, variant 2
implementation 'library:plugin-b2-impl:1.0.0'
// Both plugins transitively bring in "library:core:1.0.0".
// But if for example core:1.1.0 is released, it has to be included explicitly
2nd scheme looks like this:
// Implementaton of "A" api, variant 1
// Transitively brings in "library:plugin-a-api" - a new artifact
implementation 'library:plugin-a1-impl:1.0.0'
// Implementaton of "B" api, variant 2
// Transitively brings in "library:plugin-b-api" - a new artifact
implementation 'library:plugin-b2-impl:1.0.0'
// Core has to be explicitly specified, nobody depends on it, only core depends on plugins
implementation 'library:core:1.0.0'
just do one artifact and let people depend on that only ( as example to minimize that ).
Currently there are separate projects that depend on the library, and they use different plugin implementations. Users pick between different implementation of the same APIs depending on shared dependencies.
For example, there's A, and there are 2 implementations: A-oranges, A-apples. If the project already uses oranges, it imports A-oranges. If it already uses apples, it imports A-apples.
In other words, the plugins are more like adapters between the library and external projects.
Another depiction of the differences between 2 options:
Squares represent ".jar" artifacts. Circles inside a square represent interfaces/classes and their dependencies on each other.
It could be said, that the code is DIP compliant - both core and plugin implementations depend on abstractions.
It's only a question of artifact structuring - is it worth extracting abstractions into separate artifacts as well?

I suppose there is an issue on how and how much often do you release a new version of the core and the plugin. Generally, If you release a new version of the core, you must release a new version of the plugin, right? If not so, please specify this.
I'm for solution 2, but with a little difference, as the following example:
As you can see I've introduced a plugin-api artifact, with only interfaces used by plugins, because:
I think it's possible to have more than two plugins implementations
for plugin developers they need to use only this as dependency directly
plugin-api need only core-api
core-api can be implemented by more than one core-impl ( in the future )
Following this approach you focus will be to design plugin-api better you can, stabilize it and then let plugin developers do the job.
What if:
core-impl change ? For example a bugfix or new release. Ask yourself: do I need to change core-api ? For example to provide a new feautre to plugin-api ? If so, release a new core-api and then release a new plugin-api
core-api change? Like before
plugin-api ? if plugin-api change, you need to change only plugin-impls
To answer on your questions:
Which approach is the more correct one? Does it depend on project size or some other factors?
There is no "correct one", depends for sure on project size ( just count how many feature/methos/interfaces you have in core-api and plugin-api ), how many developers works on it and how your release process works
Are there other approaches?
See one answer before, you can search from some big project like apache or eclipse foundation one to learn their patterns, but depends heavily on the subject and can be an huge task.
Is it bad that users have to depend on plugins & "core" explicitly, even if plugins transitively bring "core" in the first approach?
For my understanding, yes. It's better to depend for my plugin implementation from an interface only, not from a core one
Anything that is intrinsic to the problem and cannot be solved? (like, is it a given that 8 artifacts are to be maintained, with no way to minimize that?)
Well, If you are alone, this is an open source project used only by yourself, don't overengineering this, just do one artifact and let people depend on that only ( as example to minimize that ).
Is it correct to provide abstract tests in the "test-jar", if I want to make sure that all plugin implementations comply with the interface contracts? Or do I have to copy-paste the tests in each plugin implementation?
For me it's better to have a plugin-api and let plugin implementation declare only that, it's more clear and concise. For tests, I'm not sure if you plan to do tests on implementations by yourself, of "ask" to plugin developers to do the test. For sure copy-paste is not the right choice, you can use a command pattern or similar to make these tests, see here
After updated question, I'm still for solution 2, event if there are two separated plugin-api, is better to have different plugin-api.
It's only a question of artifact structuring - is it worth extracting abstractions into separate artifacts as well?
I think yes, in the long run. If you separate in different artifacts, you can change them independently, for example change something in plugin-apiA and this doesn't affect plugin-apiB. If you change the core, yes of course.
Note: for my diagram above I think can still be working, can't you make an abstract set of interfaces for plugin-api and have a common artifact for them ?

If plugin A and B are two distinct type of plugin, then the option 2 is a better pick however:
If A depends on core#v1
If B depends on core#v2
Then core#v2 have to be binary compatible with core#v1, otherwise it will fail when for example someone depends on an implementation of A and an implementation of B: there always have to upgrade the plugin version in any case.
You could probably use Java Module do hide the details (eg: only provide an interface that is likely to never changes) which will makes the solution 2 of Vokail useless in some sense: you don't need a core-impl because Java module will ensure you that, apart from your core module, no one access the details (the impl). This also allow you to reuse the same package.
If A and B interface are in core, then the likeness of a binary incompatibility fall down.

How to have common DTOs or entities across Microservices

Say I have 5 different microservices. Each use one common DTO, that is UserDTO.java or an entity User.java.
If these DTOs are to be accessed by all, I came up with two approaches
To place all these DTOs in all the microservices
To create a project say Project-commons and keep all the common DTOs there and then add the dependency of this project in all microservices
I know that the sharing common DTOs across microservices would kill the concept/principles of microservices, but I wanted to know is there any other approaches available at all?

Actually, sharing a common code library does not kill the concept/principle of keeping changes isolated to one microservice, as long as you strictly version that library. That way, one microservice team can decide it needs new functionality in the shared library, and can bump the version number of the library and add the new functionality. Existing microservices that use that library won't be affected until they independently decide to start using the newer version of that library.
So this is what we do. We have jar files that we share across our microservices, but we push specific versions of those jar files to Nexus and reference those jar files by version number in each microservice.
Note that this concept is no different than two microservices depending on a third party component, like an Apache Commons library. Just because Apache comes out with a new version of a library doesn't mean anyone's binaries change. Each codebase that depends on that library can decide if and when it moves to the new version independent of what version another codebase might be using or what the most recent version of the library might be.

How do big companies tackle with the package dependencies conflict problem?

Just as shown in the picture, one app (Java) referenced two third-party package jars (packageA and packageB), and they referenced packageC-0.1 and packageC-0.2 respectively. It would work well if packageC-0.2 was compatible with packageC-0.1. However sometimes packageA used something that could not be supported in packageC-0.2 and Maven can only use the latest version of a jar. This issue is also known as "Jar Hell".
It would be difficult in practice to rewrite package A or force its developers to update packageC to 0.2.
How do you tackle with these problems? This often happens in large-scale companies.
I have to declare that this problem is mostly occurred in BIG companies due to the fact that big company has a lot of departments and it would be very expensive to let the whole company update one dependency each time certain developers use new features of new version of some dependency jars. And this is not big deal in small companies.
Any response will be highly appreciated.
Let me throw away a brick in order to get a gem first.
Alibaba is one of the largest E-Commerces in the world. And we tackle with these problems by creating an isolation container named Pandora. Its principle is simple: packaging those middle-wares together and load them with different ClassLoaders so that they can work well together even they referenced same packages with different versions. But this need a runtime environment provided by Pandora which is running as a tomcat process. I have to admit that this is a heavy plan. Pandora is developed based on a fact that JVM identifies one class by class-loader plus classname.
If you know someone maybe know the answers, share the link with him/her.

We are a large company and we have this problem a lot. We have large dependency trees that over several developer groups. What we do:
We manage versions by BOMs (lists of Maven dependencyManagement) of "recommended versions" that are published by the maintainers of the jars. This way, we make sure that recent versions of the artifacts are used.
We try to reduce the large dependency trees by separating the functionality that is used inside a developer group from the one that they offer to other groups.
But I admit that we are still trying to find better strategies. Let me also mention that using "microservices" is a strategy against this problem, but in many cases it is not a valid strategy for us (mainly because we could not have global transactions on databases any more).

This is a common problem in the java world.
Your best options are to regularly maintain and update dependencies of both packageA and packageB.
If you have control over those applications - make time to do it. If you don't have control, demand that the vendor or author make regular updates.
If both packageA and packageB are used internally, you can use the following practise: have all internal projects in your company refer to a parent in the maven pom.xml that defines "up to date" versions of commonly used third party libraries.
For example:
<framework.jersey>2.27</framework.jersey>
<framework.spring>4.3.18.RELEASE</framework.spring>
<framework.spring.security>4.2.7.RELEASE</framework.spring.security>
Therefore, if your project "A" uses spring, if they use the latest version of your company's "parent" pom, they should both use 4.3.18.RELEASE.
When a new version of spring is released and desirable, you update your company's parent pom, and force all other projects to use that latest version.
This will solve many of these dependency mismatch issues.
Don't worry, it's common in the java world, you're not alone. Just google "jar hell" and you can understand the issue in the broader context.
By the way mvn dependency:tree is your friend for isolating these dependency problems.

I agree with the answer of #JF Meier ，In Maven multi-module project, the dependency management node is usually defined in the parent POM file when doing unified version management. The content of dependencies node declared by the node class is about the resource version of unified definition. The resources in the directly defined dependencies node need not be introduced into the version phase. The contents of the customs are as follows:
in the parent pom
<dependencyManagement> 
    <dependencies > 
      <dependency > 
        <groupId>com.devzuz.mvnbook.proficio</groupId> 
        <artifactId>proficio-model</artifactId> 
        <version>${project.version}</version> 
      </dependency > 
</dependencies >
</dependencyManagement>
in your module ,you do not need to set the version
<dependencies > 
    <dependency > 
      <groupId>com.devzuz.mvnbook.proficio</groupId> 
       <artifactId>proficio-model</artifactId> 
    </dependency > 
  </dependencies > 
This will avoid the problem of inconsistency .

This question can't be answered in general.
In the past we usually just didn't use dependencies of different versions. If the version was changed, team-/company-wide refactoring was necessary. I doubt it is possible with most build tools.
But to answer your question..
Simple answer: Don't use two versions of one dependency within one compilation unit (usually a module)
But if you really have to do this, you could write a wrapper module that references to the legacy version of the library.
But my personal opinion is that within one module there should not be the need for these constructs because "one module" should be relatively small to be manageable. Otherwise it might be a strong indicator that the project could use some modularization refactoring. However, I know very well that some projects of "large-scale companies" can be a huge mess where no 'good' option is available. I guess you are talking about a situation where packageA is owned by a different team than packageB... and this is generally a very bad design decision due to the lack of separation and inherent dependency problems.

First of all, try to avoid the problem. As mentioned in #Henry's comment, don't use 3rd party libraries for trivial tasks.
However, we all use libraries. And sometimes we end up with the problem you describe, where we need two different versions of the same library. If library 'C' has removed and added some APIs between the two versions, and the removed APIs are needed by 'A', while 'B' needs the new ones, you have an issue.
In my company, we run our Java code inside an OSGi container. Using OSGi, you can modularize your code in "bundles", which are jar files with some special directives in their manifest file. Each bundle jar has its own classloader, so two bundles can use different versions of the same library. In your example, you could split your application code that uses 'packageA' into one bundle, and the code that uses 'packageB' in another. The two bundles can call each others APIs, and it will all work fine as long as your bundles do not use 'packageC' classes in the signature of the methods used by the other bundle (known as API leakage).
To get started with OSGi, you can e.g. take a look at OSGi enRoute.

Let me throw away a brick in order to get a gem first.
Alibaba is one of the largest E-Commerces in the world. And we tackle with these problems by creating an isolation container named Pandora. Its principle is simple: packaging those middle-wares together and load them with different ClassLoaders so that they can work well together even they referenced same packages with different versions. But this need a runtime environment provided by Pandora which is running as a tomcat process. I have to admit that this is a heavy plan.
Pandora is developed based on a fact that JVM identifies one class by class-loader plus classname.

How to handle version conflicts with my Java application using SPI extensions

I am writing a plugin API for a Java application, the idea being that eventually third parties will provide their own plugin extensions for the application and all the user needs to do is place the plugin jar into a plugins directory of the application. For the impatient, my question in short is how to handle possible version conflicts when the plugin relates to a different API than that on the system, see below for details of my situation and what I have thought about.
I have read a number of articles using service provider interfaces and have something working with that. The problem comes when considering how to deal with differing version combinations.
I am aware of the technique of when adding to an API adding extension interfaces, rather than changing the existing interface (eg. API 1.0 having MyInterface, API 1.1 adding MyInterface2 with the new methods, etc). With this technique if the user has the latest API then older plugins should work fine, but what happens if the user has an old API and newer plugins?
So as an example the user has API 1.0 only with MyInterface but installs binary plugin compiled against API 1.1 where the provider class implements MyInterface2. Whilst the application may only ever call plugins using MyInterface, what happens if the plugin internally calls MyInterface2? Will this cause an error or exception and when (IE. when the class is loaded or when the method from MyInterface2 is called). Also is this standard across JVMs or may it differ depending on the JVM used?
Finally, would it be better to use a plugin framework, would that be able to check version requirements? Searching the internet I find PF4J on github. A quick look in the source code seems to show it may support some sort of version checks.

Java package naming for versioning external APIs

Are there any conventions of how to name Java packages containing classes that consume an external, versioned API?
Let's assume we have a major-minor semantic versioning scheme of a service and we have to implement a consumer that is compatible with and bound to a specific version of that API. What are the best practices to name the packages (and classes)?
Currently, we're using the scheme of: ${service}_${M}_${N} (with M = Major version, N = minor version). For example: com.example.theService_1_0..
But sonarqube is complaining that it does not match conventions.
Of course I can just disable this rule, but I wonder if there are any best-practices?
I'm looking for a general approach not only something specific to REST, because I've encountered consumer implementations for WebService, REST and CORBA. And I'm not sure, artifact-versioning (as in maven) works well here, because it relates to the version of the implementation and not the API.
I know there are questions around api versioning, but those are about the producer, not the consumer.

Yes, Java package names have a strong and ambiguous convention for indicating dependencies’ versions: don’t.
If you change your application to use a new version of an external API, you create a new version of your application. The version number you are looking for is your application’s version number. On many Java projects, dependencies’ versions are management by a Maven configuration file.
In any case, when classes use a particular API version, the class and package names have no business exposing this information, which would violate encapsulation, apart from anything else. These names have a different purpose.
Note that this is no different when you use HTTP/REST APIs or Java APIs. After all, I don’t think you’d name your class TheServiceWithLog4J_12_1_5. I hope not, at least.
You didn’t mention whether you have a requirement to support multiple versions of this external API at the same time. Even if you did, I still wouldn’t recommend exposing the API version number in the package name. Instead, use the package name to indicate why you have two versions, and the important difference.