I have a Java 11 project embedding Tomcat:
<dependency>
<groupId>org.apache.tomcat.embed</groupId>
<artifactId>tomcat-embed-core</artifactId>
<version>10.1.0</version>
</dependency>
The Tomcat-specific code is in a subproject with only two classes. When I compile using Maven 3.8.6 and Java 17 using -Xlint:all, I see the following warning for that subproject:
[WARNING] Cannot find annotation method 'value()' in type 'aQute.bnd.annotation.spi.ServiceConsumer': class file for aQute.bnd.annotation.spi.ServiceConsumer not found
Doing a bit of searching brings up similar (but not exact) things, such as Lombok Issue #2145, which hints that I may need to add some sort of extra dependency such as biz.aQute.bnd:bndlib or org.osgi:osgi.annotation. But even after adding those dependencies, the warning remains.
Where is this error coming from, and what does it mean? I don't have any #ServiceConsumer annotation in my source code, and I couldn't find any in the Tomcat classes I'm extending, either. How can I make it go away?
I filed Tomcat Bug 66299.
I discussed this on the Tomcat users mailing list users#tomcat.apache.org (thanks Mark), and here's what is happening:
Tomcat effectively has two builds:
What I call the Tomcat "primary build" uses Ant with build.xml, which compiles the source files, creates all the JARs and binaries, and publishes them to Maven Central (Nexus).
Any "secondary build" by third parties using the published JARs and POMs, using e.g. org.apache.tomcat.embed:tomcat-embed-core:10.1.0 with Maven.
The latest versions of direct dependencies are found in the Tomcat repository inside build.properties.default.
The primary build generates JPMS and OSGi metadata, so some classes are annotated with the bnd annotation aQute.bnd.annotation.spi.ServiceConsumer. Currently Tomcat gets this annotation from biz.aQute.bnd:biz.aQute.bnd:6.3.1, which is apparently some aggregate JAR; the same annotation can be found in the smaller biz.aQute.bnd:biz.aQute.bnd.annotation:6.3.1.
The aQute.bnd.annotation.spi.ServiceConsumer annotation source code uses the OSGi annotation org.osgi.annotation.bundle.Requirement. Currently this annotation can be found in org.osgi:osgi.annotation:8.1.0.
The bnd and OSGi annotations remain part of the compiled classes even though they are not used at runtime and are not technically needed in any secondary builds.
If you want to inform Maven and javac where these classes are so that they will not appear missing (if you are compiling with some variations of -Xlint), but that they nevertheless will not be needed at runtime (and technically aren't even needed at compile time in a secondary build) and should not be distributed in the resulting JAR, you can note them in your pom.xml file using the provided scope.
<dependency>
<groupId>biz.aQute.bnd</groupId>
<artifactId>biz.aQute.bnd.annotation</artifactId>
<version>6.3.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.osgi</groupId>
<artifactId>osgi.annotation</artifactId>
<version>8.1.0</version>
<scope>provided</scope>
</dependency>
Maven will download these artifacts during your build thereby removing the warning, but they will not be included in the resulting artifacts of your build.
I am currently porting an open source library to be JDK9+ compliant, and it depends on some of the Java EE Modules that have been deprecated in Java 9 and removed in Java 11: specifically, JAXB, JAX-WS and javax.annotation.
I added explicit dependencies to the third party implementations as suggested here:
<dependency>
<groupId>com.sun.xml.ws</groupId>
<artifactId>jaxws-ri</artifactId>
<version>2.3.0.1</version>
</dependency>
<dependency>
<groupId>com.sun.xml.bind</groupId>
<artifactId>jaxb-ri</artifactId>
<version>2.3.0.1</version>
</dependency>
<dependency>
<groupId>com.sun.activation</groupId>
<artifactId>javax.activation</artifactId>
<version>1.2.0</version>
</dependency>
However, I'd like my library to use them only if necessary (i.e., on JDK9+) and keep using the endorsed implementations on JDK8.
I can do so by adding the dependencies in a Maven profile to be activated only on JDK 9 and above, but what if I wanted to publish the jar file for my library on Maven Central? Should I publish two different jars, one with the Java EE third party implementations included, for JDK9+ and one without for JDK8?
Is there a way to produce a jar file that will use the third party implementations on JDK9+ and the endorsed ones on JDK8?
I have looked into multi-release jars, but looks like they are intended for jdk version-dependent implementations among project classes, not among dependencies.
Also, in case it's not possible to use the endorsed implementations on JDK 8, is there a way to reliably test that using the third party implementations does not introduce any regressions?
Is there a way to produce a jar file that will use the third party
implementations on JDK9+ and the endorsed ones on JDK8?
Unfortunately, no. When distributing a library via jar file, you cannot control how other jars and libraries will be listed in the classpath. This makes class loading non-deterministic for you. What that means for your situation is that if the aforementioned libraries are included in the classpath in a JDK8 environment, there is no way to determine or control which version of the classes get loaded.
Also, in case it's not possible to use the endorsed implementations on
JDK 8, is there a way to reliably test that using the third party
implementations does not introduce any regressions?
As the author, it would be up to you to execute tests across the different runtime environments to check for regressions.
Should I publish two different jars, one with the Java EE third party
implementations included, for JDK9+ and one without for JDK8?
This is a perfectly reasonable solution which others have used before as well. Take for example the sqlserver jdbc jars, which have different versions based on jre: https://learn.microsoft.com/en-us/sql/connect/jdbc/using-the-jdbc-driver?view=sql-server-2017
For the case described here, the JDK9+ version of the jar could declare the additional dependencies you mentioned in the question, whereas the JDK8 version would not.
One other option would be to have a single jar and to declare the dependecies you mentioned as provided. This would signal to the consumer that the runtime environment must include the classes in the declared dependencies. Some documentation would be called for to direct the consumer of the library as to when to they must explicitly add the jars declared as provided to their classpath.
IMHO, the clearest solution is two jars with a reference to the JRE version in the jar name and differing dependencies. It requires very little documentation (which most dont look at anyway). And it allows you to make changes in your library more freely.
On various projects I've been working on, I've seen diferent ways of specifying dependencies versions. On some projects, the package version is written on the same dependency declaration:
<dependency>
<groupId>org.apache.myfaces.extensions.validator.validation-modules</groupId>
<artifactId>myfaces-extval-property-validation</artifactId>
<version>2.0.7</version>
<scope>compile</scope>
</dependency>
On others, a property is used, as in:
<dependency>
<groupId>org.apache.myfaces.extensions.validator.validation-modules</groupId>
<artifactId>myfaces-extval-property-validation</artifactId>
<version>${versions.extval}</version>
<scope>compile</scope>
</dependency>
For multimodule projects, I can see a clear benefit in declaring versions on the parent pom to avoid duplication (and the potential confusion and errors that come with it), but on single module applications, would there be a benefit to use such a level of indirection?
What would be a best practice for this and why?
Thanks a lot :)
With a version property you can override it on the command line whereas with a fixed version you cannot.
So you can recompile your project with a newer version just by specifying it on the command line.
mvn -Dversions.extval=2.0.8 clean package
Or something.
apart from that, mostly used on multi-projects, and although there you have the dependency management section as well for versions.
In my office, the mere mention of the word Xerces is enough to incite murderous rage from developers. A cursory glance at the other Xerces questions on SO seem to indicate that almost all Maven users are "touched" by this problem at some point. Unfortunately, understanding the problem requires a bit of knowledge about the history of Xerces...
History
Xerces is the most widely used XML parser in the Java ecosystem. Almost every library or framework written in Java uses Xerces in some capacity (transitively, if not directly).
The Xerces jars included in the official binaries are, to this day, not versioned. For example, the Xerces 2.11.0 implementation jar is named xercesImpl.jar and not xercesImpl-2.11.0.jar.
The Xerces team does not use Maven, which means they do not
upload an official release to Maven Central.
Xerces used to be released as a single jar (xerces.jar), but was split into two jars, one containing the API (xml-apis.jar) and one containing the implementations of those APIs (xercesImpl.jar). Many older Maven POMs still declare a dependency on xerces.jar. At some point in the past, Xerces was also released as xmlParserAPIs.jar, which some older POMs also depend on.
The versions assigned to the xml-apis and xercesImpl jars by those who deploy their jars to Maven repositories are often different. For example, xml-apis might be given version 1.3.03 and xercesImpl might be given version 2.8.0, even though both are from Xerces 2.8.0. This is because people often tag the xml-apis jar with the version of the specifications that it implements. There is a very nice, but incomplete breakdown of this here.
To complicate matters, Xerces is the XML parser used in the reference implementation of the Java API for XML Processing (JAXP), included in the JRE. The implementation classes are repackaged under the com.sun.* namespace, which makes it dangerous to access them directly, as they may not be available in some JREs. However, not all of the Xerces functionality is exposed via the java.* and javax.* APIs; for example, there is no API that exposes Xerces serialization.
Adding to the confusing mess, almost all servlet containers (JBoss, Jetty, Glassfish, Tomcat, etc.), ship with Xerces in one or more of their /lib folders.
Problems
Conflict Resolution
For some -- or perhaps all -- of the reasons above, many
organizations publish and consume custom builds of Xerces in their
POMs. This is not really a problem if you have a small application and are only using Maven Central, but it quickly becomes an issue for enterprise software where Artifactory or Nexus is proxying multiple repositories (JBoss, Hibernate, etc.):
For example, organization A might publish xml-apis as:
<groupId>org.apache.xerces</groupId>
<artifactId>xml-apis</artifactId>
<version>2.9.1</version>
Meanwhile, organization B might publish the same jar as:
<groupId>xml-apis</groupId>
<artifactId>xml-apis</artifactId>
<version>1.3.04</version>
Although B's jar is a lower version than A's jar, Maven does not know
that they are the same artifact because they have different
groupIds. Thus, it cannot perform conflict resolution and both
jars will be included as resolved dependencies:
Classloader Hell
As mentioned above, the JRE ships with Xerces in the JAXP RI. While it would be nice to mark all Xerces Maven dependencies as <exclusion>s or as <provided>, the third-party code you depend on may or may not work with the version provided in JAXP of the JDK you're using. In addition, you have the Xerces jars shipped in your servlet container to contend with. This leaves you with a number of choices: Do you delete the servlet version and hope that your container runs on the JAXP version? Is it better to leave the servlet version, and hope that your application frameworks run on the servlet version? If one or two of the unresolved conflicts outlined above manage to slip into your product (easy to happen in a large organization), you quickly find yourself in classloader hell, wondering which version of Xerces the classloader is picking at runtime and whether or not it will pick the same jar in Windows and Linux (probably not).
Solutions?
We've tried marking all Xerces Maven dependencies as <provided> or as an <exclusion>, but this is difficult to enforce (especially with a large team) given that the artifacts have so many aliases (xml-apis, xerces, xercesImpl, xmlParserAPIs, etc.). Additionally, our third party libs/frameworks may not run on the JAXP version or the version provided by a servlet container.
How can we best address this problem with Maven? Do we have to exercise such fine-grained control over our dependencies, and then rely on tiered classloading? Is there some way to globally exclude all Xerces dependencies, and force all of our frameworks/libs to use the JAXP version?
UPDATE: Joshua Spiewak has uploaded a patched version of the Xerces build scripts to XERCESJ-1454 that allows for upload to Maven Central. Vote/watch/contribute to this issue and let's fix this problem once and for all.
There are 2.11.0 JARs (and source JARs!) of Xerces in Maven Central since 20th February 2013! See Xerces in Maven Central. I wonder why they haven't resolved https://issues.apache.org/jira/browse/XERCESJ-1454...
I've used:
<dependency>
<groupId>xerces</groupId>
<artifactId>xercesImpl</artifactId>
<version>2.11.0</version>
</dependency>
and all dependencies have resolved fine - even proper xml-apis-1.4.01!
And what's most important (and what wasn't obvious in the past) - the JAR in Maven Central is the same JAR as in the official Xerces-J-bin.2.11.0.zip distribution.
I couldn't however find xml-schema-1.1-beta version - it can't be a Maven classifier-ed version because of additional dependencies.
Frankly, pretty much everything that we've encountered works just fine w/ the JAXP version, so we always exclude xml-apis and xercesImpl.
You could use the maven enforcer plugin with the banned dependency rule. This would allow you to ban all the aliases that you don't want and allow only the one you do want. These rules will fail the maven build of your project when violated. Furthermore, if this rule applies to all projects in an enterprise you could put the plugin configuration in a corporate parent pom.
see:
http://maven.apache.org/plugins/maven-enforcer-plugin/
http://maven.apache.org/enforcer/enforcer-rules/bannedDependencies.html
I know this doesn't answer the question exactly, but for ppl coming in from google that happen to use Gradle for their dependency management:
I managed to get rid of all xerces/Java8 issues with Gradle like this:
configurations {
all*.exclude group: 'xml-apis'
all*.exclude group: 'xerces'
}
I guess there is one question you need to answer:
Does there exist a xerces*.jar that everything in your application can live with?
If not you are basically screwed and would have to use something like OSGI, which allows you to have different versions of a library loaded at the same time. Be warned that it basically replaces jar version issues with classloader issues ...
If there exists such a version you could make your repository return that version for all kinds of dependencies. It's an ugly hack and would end up with the same xerces implementation in your classpath multiple times but better than having multiple different versions of xerces.
You could exclude every dependency to xerces and add one to the version you want to use.
I wonder if you can write some kind of version resolution strategy as a plugin for maven. This would probably the nicest solution but if at all feasible needs some research and coding.
For the version contained in your runtime environment, you'll have to make sure it either gets removed from the application classpath or the application jars get considered first for classloading before the lib folder of the server get considered.
So to wrap it up: It's a mess and that won't change.
You should debug first, to help identify your level of XML hell. In my opinion, the first step is to add
-Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl
-Djavax.xml.transform.TransformerFactory=com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl
-Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl
to the command line. If that works, then start excluding libraries. If not, then add
-Djaxp.debug=1
to the command-line.
There is another option that hasn't been explored here: declaring Xerces dependencies in Maven as optional:
<dependency>
<groupId>xerces</groupId>
<artifactId>xercesImpl</artifactId>
<version>...</version>
<optional>true</optional>
</dependency>
Basically what this does is to force all dependents to declare their version of Xerces or their project won't compile. If they want to override this dependency, they are welcome to do so, but then they will own the potential problem.
This creates a strong incentive for downstream projects to:
Make an active decision. Do they go with the same version of Xerces or use something else?
Actually test their parsing (e.g. through unit testing) and classloading as well as not to clutter up their classpath.
Not all developers keep track of newly introduced dependencies (e.g. with mvn dependency:tree). This approach will immediately bring the matter to their attention.
It works quite well at our organization. Before its introduction, we used to live in the same hell the OP is describing.
Every maven project should stop depending on xerces, they probably don't really. XML APIs and an Impl has been part of Java since 1.4. There is no need to depend on xerces or XML APIs, its like saying you depend on Java or Swing. This is implicit.
If I was boss of a maven repo I'd write a script to recursively remove xerces dependencies and write a read me that says this repo requires Java 1.4.
Anything that actually breaks because it references Xerces directly via org.apache imports needs a code fix to bring it up to Java 1.4 level (and has done since 2002) or solution at JVM level via endorsed libs, not in maven.
What would help, except for excluding, is modular dependencies.
With one flat classloading (standalone app), or semi-hierarchical (JBoss AS/EAP 5.x) this was a problem.
But with modular frameworks like OSGi and JBoss Modules, this is not so much pain anymore. The libraries may use whichever library they want, independently.
Of course, it's still most recommendable to stick with just a single implementation and version, but if there's no other way (using extra features from more libs), then modularizing might save you.
A good example of JBoss Modules in action is, naturally, JBoss AS 7 / EAP 6 / WildFly 8, for which it was primarily developed.
Example module definition:
<?xml version="1.0" encoding="UTF-8"?>
<module xmlns="urn:jboss:module:1.1" name="org.jboss.msc">
<main-class name="org.jboss.msc.Version"/>
<properties>
<property name="my.property" value="foo"/>
</properties>
<resources>
<resource-root path="jboss-msc-1.0.1.GA.jar"/>
</resources>
<dependencies>
<module name="javax.api"/>
<module name="org.jboss.logging"/>
<module name="org.jboss.modules"/>
<!-- Optional deps -->
<module name="javax.inject.api" optional="true"/>
<module name="org.jboss.threads" optional="true"/>
</dependencies>
</module>
In comparison with OSGi, JBoss Modules is simpler and faster. While missing certain features, it's sufficient for most projects which are (mostly) under control of one vendor, and allow stunning fast boot (due to paralelized dependencies resolving).
Note that there's a modularization effort underway for Java 8, but AFAIK that's primarily to modularize the JRE itself, not sure whether it will be applicable to apps.
Apparently xerces:xml-apis:1.4.01 is no longer in maven central, which is however what xerces:xercesImpl:2.11.0 references.
This works for me:
<dependency>
<groupId>xerces</groupId>
<artifactId>xercesImpl</artifactId>
<version>2.11.0</version>
<exclusions>
<exclusion>
<groupId>xerces</groupId>
<artifactId>xml-apis</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>xml-apis</groupId>
<artifactId>xml-apis</artifactId>
<version>1.4.01</version>
</dependency>
My friend that's very simple, here an example:
<dependency>
<groupId>xalan</groupId>
<artifactId>xalan</artifactId>
<version>2.7.2</version>
<scope>${my-scope}</scope>
<exclusions>
<exclusion>
<groupId>xml-apis</groupId>
<artifactId>xml-apis</artifactId>
</exclusion>
</dependency>
And if you want to check in the terminal(windows console for this example) that your maven tree has no problems:
mvn dependency:tree -Dverbose | grep --color=always '(.* conflict\|^' | less -r
I'm using JBoss AS 7 with Maven, and also added RichFaces, that I'm almost sure, don't come with JBoss. So I added that to my pom.xml:
<dependency>
<groupId>org.richfaces.core</groupId>
<artifactId>richfaces-core-impl</artifactId>
<version>4.1.0.Final</version>
<scope>compile</scope>
</dependency>
But Richfaces also have its dependencies, so cssparser and sac also comes with compile scope, but they are also inside JBoss AS 7, so the following warning comes when I run JBoss:
Deployment "deployment.test.war" is using a private module ("org.w3c.css.sac:main") which may be changed or removed in future versions without notice.
Deployment "deployment.test.war" is using a private module ("net.sourceforge.cssparser:main") which may be changed or removed in future versions without notice.
I believe that this warning appears because I have this module both on my war and on JBoss, so I want to know: There is a way to change the scope of them to provided, in my POM? Even if they are inherited?
No, you can't change transitive dependencies' scopes. The best you can do is to exclude these dependencies using <exclusions> in your dependency declaration.