Maven shade plugin - don't relocate excluded/optional scope dependencies - java

I'm using maven shade plugin in order to create an uber jar while relocating all classes. That's because I'm getting an external jar and I don't want to have classpath collisions. So the idea is to create a new uber (relocated) jar and use it in my application.
So the shade plugin takes all the classes and relocates them to the new package prefix. My issue is that a far as I understand, it also does that for dependencies which the classes they depend on, aren't in the scope [*].
Let's say I'm relocating all com to shade.com:
<executions>
<execution>
<id>rename-all</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<shadedArtifactAttached>true</shadedArtifactAttached>
<keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope>
<relocations>
<relocation>
<pattern>com</pattern>
<shadedPattern>shade/com</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
So if one of my dependencies, A, has a <optional> dependency on B (which package com.B), the plugin would change the imports within A to point to shade.com.B. But com.B is optional, which means those classes won't be available in shade.com.B, because they didn't get into the relocation process. The actual classes available when using this jar would be the "normal" ones - com.B.
And then I get class not found exceptions on shade.com.B classes when I try to use the shaded jar in my application.
Am I missing something in my understanding? Is there any solution to this?
[*] Some examples: I'm not sure yet about the exact cases where this happens. In my case, I depend on spark-sqldependency. I dig into 3 of the classes which I saw this issue (there are many more):
org/apache/html/dom/HTMLIsIndexElementImpl imports org.w3c.dom.html.HTMLIsIndexElement which is in rt.jar - the import was relocated so now it points to shade.org.w3c.dom.html.HTMLIsIndexElement and can't be found.
io/netty/handler/codec/marshalling/ChannelBufferByteOutput imports org.jboss.marshalling.ByteOutput. Hence the relocation change this to import shade.org.jboss.marshalling.ByteOutput. But as can be seen here, the import for jboss-marshalling is marked as <optional>true</optional>, so it's not in the jar and ByteOutput itself won't be relocated, hence won't be available.
jasper-runtime is included here but excluded in the upper one, here. So the outcome is that the files in Hadoop-hdfs are getting relocation to their dependencies, even though these classes would never be available in the relocated path since they aren’t in the jar at all. e.g. org/apache/hadoop/hdfs/server/datanode/browseBlock_jsp now has a reference to “shade/org/apache/jasper/runtime/JspSourceDependent”.

Related

manage maven different cglib/asm versions

I have a multi-module project in maven, that uses (amongst others) glassfish-jersey, jersey-moxy, wicket-ioc, lucene and lamdbaj These all come with asm, but all with different versions.
Lately, I run into a lot of trouble when running my tests. Typical error I get is:
java.lang.VerifyError: class net.sf.cglib.core.DebuggingClassWriter overrides final method visit.(IILjava/lang/String;Ljava/lang/String;Ljava/lang/String;[Ljava/lang/String;)V
I read that this can be caused by different asm versions. Is there a way to 'sandbox' these different asm-versions in their dependencies, so they don't get mixed up?
Edit:
My current solution is to use jarjar, like this:
<build>
<plugins>
<plugin>
<groupId>org.sonatype.plugins</groupId>
<artifactId>jarjar-maven-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>jarjar</goal>
</goals>
<configuration>
<includes>
<include>cglib:cglib-nodep</include>
</includes>
<rules>
<rule>
<pattern>net.sf.cglib.asm.**</pattern>
<result>com.myproject.lambda4j.asm.#1</result>
</rule>
<rule>
<pattern>net.sf.cglib.**</pattern>
<result>com.myproject.lambda4j.cglib.#1</result>
</rule>
</rules>
</configuration>
</execution>
</executions>
</plugin>
Try explicitly adding some cglib v3.* on your class path which uses a recent version of ASM. The problem that you encounter is that cglib modifies the behavior of the ASM class writer by inheritance rather then by delegation. However, ASM enforced this latter best practice by making its ClassWriter's methods final from any version 4.* while it was still possible to override its methods in version 3. The error you encounter is a result of combining cglib 2.* with ASM 4.*.
Fortunately (for you), cglib has been rather static in its last versions, i.e. there were only minor API changes while the newer versions mostly consisted of updates of ASM. If you are lucky, this explicit use of cglib v3.* therefore solves your problem. This holds as long as none of your project dependencies has a direct dependency on ASM what seems reasonable for the dependencies you named like Jersey or Lucene.
If this does not work, you need to recompile some of your dependencies while using a tool like jarjar in order to repack the direct ASM dependencies into different name spaces in order to resolve these version conflicts. An alternative could be to isolate different ASM versions by some conditional child-first ClassLoader magic but this is not so recommendable since the effects are unpredictable and will also result in a performance penalty.

using maven coordinate style episodes in wsimport

I'm building (multiple) complex webservice with base XSD types from all kinds of standards (GML, SWE, XLINK, etc). Now, I would like to break up the compilation into more steps, preferrably one for each of the standards I'm using.
Advantages:
1) I can add create tooling libraries that I can re-use in all of my webservices on each of the standards.
2) I can make use of the power of JAXB2 basics plugin, which seems to work very nicely with the maven-jaxb2-plugin (org.jvnet.jaxb2.maven2) and create for instance interface bindings. This in contrast with the jaxws-maven-plugin plugin.
The final step would be using the org.jvnet.jax-ws-commons:maven-jaxb2-plugin to create the actual web service that I can implement in an EJB (or call as a client).
Now, the org.jvnet.jaxb2.maven2:maven-jaxb2-plugin plugin allows me to refer to episodes by means of their maven coordinate, as part of its like this:
<episodes>
<episode>
<groupId>org.example</groupId>
<artifactId>jaxb2-basics-test-episodes-a</artifactId>
</episode>
</episodes>
How can I do this by means of the org.jvnet.jax-ws-commons:maven-jaxb2-plugin? I've searched a lot, and experimented like this:
<plugin>
<groupId>org.jvnet.jax-ws-commons</groupId>
<artifactId>>maven-jaxb2-plugin</artifactId>
<version>2.1</version>
<executions>
<execution>
<goals>
<goal>wsimport</goal>
</goals>
</execution>
</executions>
<configuration>
<wsdlDirectory>src/main/resources/</wsdlDirectory>
<wsdlFiles>
<wsdlFile>example.wsdl</wsdlFile>
</wsdlFiles>
<xjcArgs>
<xjcArg>-b</xjcArg>
<xjcArg>../cpt-xsd/target/generated-sources/xjc/META-INF/sun-jaxb.episode</xjcArg>
</xjcArgs>
<verbose>true</verbose>
</configuration>
</plugin>
Which takes the episode file from the target dir of the (compiled) JAXB dependend project. This sometimes even fails in the maven build (why I did not figure out yet).
I've tried to use catalog files to make a mapping but (I think I saw somewhere a catalog mapping that took maven coordinates as destination), but haven't succeeded yet.
Are you aware of the OGC Schemas and Tools Project? (Disclaimer: I'm the author.)
Now, to your question. My guess would be that org.jvnet.jax-ws-commons:maven-jaxb2-plugin does not support the "Maven coordinates" as you call them. This was a feature I've specifically implemented for my org.jvnet.jaxb2.maven2:maven-jaxb2-plugin (disclaimer: I'm the author).
From the other hand, episode file is nothing but a JAXB binding file. So you can simply extract this file from the JAR artifact (for instance using the maven-dependency-plugin) and then include it more or less like you do it already. Just don't point to directories in other modules, this is not reliable.

Maven AppAssembler not finding class

Attempting to modify an existing Java/Tomcat app for deployment on Heroku following their tutorial and running into some issues with AppAssembler not finding the entry class. Running target/bin/webapp (or deploying to Heroku) results in Error: Could not find or load main class org.stopbadware.dsp.Main
Executing java -cp target/classes:target/dependency/* org.stopbadware.dsp.Main runs properly however. Here's the relevant portion of pom.xml:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>appassembler-maven-plugin</artifactId>
<version>1.1.1</version>
<configuration>
<assembleDirectory>target</assembleDirectory>
<programs>
<program>
<mainClass>org.stopbadware.dsp.Main</mainClass>
<name>webapp</name>
</program>
</programs>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>assemble</goal>
</goals>
</execution>
</executions>
</plugin>
My guess is mvn package is causing AppAssembler to not use the correct classpath, any suggestions?
Your artifact's packaging must be set to jar, otherwise the main class is not found.
<pom>
...
<packaging>jar</packaging>
...
</pom>
The artifact itself is added at the end of the classpath, so nothing other than a JAR file will have any effect.
Try:
mvn clean package jar:jar appassembler:assemble
Was able to solve this by adding "$BASEDIR"/classes to the CLASSPATH line in the generated script. Since the script gets rewritten on each call of mvn package I wrote a short script that calls mvn package and then adds the needed classpath entry.
Obviously a bit of a hack but after a 8+ hours of attempting a more "proper" solution this will have to do for now. Will certainly entertain any more elegant ways of correcting the classpath suggested here.
I was going through that tutorial some time ago and had very similar issue. I came with a bit different approach which works for me very nicely.
First of all, as it was mentioned before, you need to keep your POM's type as jar (<packaging>jar</packaging>) - thanks to that, appassembler plugin will generate a JAR file from your classes and add it to the classpath. So thanks to that your error will go away.
Please note that this tutorial Tomcat is instantiated from application source directory. In many cases that is enough, but please note that using that approach, you will not be able to utilize Servlet #WebServlet annotations as /WEB-INF/classes in sources is empty and Tomcat will not be able to scan your servlet classes. So HelloServlet servlet from that tutorial will not work, unless you add some additional Tomcat initialization (resource configuration) as described here (BTW, you will find more SO questions talking about that resource configuration).
I did a bit different approach:
I run a org.apache.maven.plugins:maven-war-plugin plugin (exploded goal) during package and use that generated directory as my source directory of application. With that approach my web application directory will have /WEB-INF/classes "populated" with classes. That in turn will allow Tomcat to perform scanning job correctly (i.e. Servlet #WebServlet annotations will work).
I also had to change a source of my application in the launcher class:
public static void main(String[] args) throws Exception {
// Web application is generated in directory name as specified in build/finalName
// in maven pom.xml
String webappDirLocation = "target/embeddedTomcatSample/";
Tomcat tomcat = new Tomcat();
// ... remaining code does not change
Changes to POM which I added - included maven-war-plugin just before appassembler plugin:
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.5</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>exploded</goal>
</goals>
</execution>
</executions>
</plugin>
...
Please note that exploded goal is called.
I hope that small change will help you.
One more comment on that tutorial and maven build: note that the tutorial was written to show how simple is to build an application and run it in Heroku. However, that is not the best approach to maven build.
Maven recommendation is that you should adhere to producing one artifact per POM. In your case there are should two artifacts:
Tomcat launcher
Tomcat web application
Both should be build as separate POMs and referenced as modules from your parent POM. If you look at the complexity of that tutorial, it does not make much sense to split that into two modules. But if your applications gets more and more complex (and the launcher gets some additional configurations etc.) it will makes a lot of sense to make that "split". As a matter of fact, there are some "Tomcat launcher" libraries already created so alternatively you could use of one them.
You can set the CLASSPATH_PREFIX environment variable:
export CLASSPATH_PREFIX=target/classes
which will get prepended to the classpath of the generated script.
The first thing is that you are using an old version of appassembler-maven-plugin the current version is 1.3.
What i don't understand why are you defining the
<assembleDirectory>target</assembleDirectory>
folder. There exists a good default value for that. So usually you don't need it. Apart from that you don't need to define an explicit execution which bounds to the package phase, cause the appassembler-maven-plugin is by default bound to the package phase.
Furthermore you can use the useWildcardClassPath configuration option to make your classpath shorter.
<configuration>
<useWildcardClassPath>true</useWildcardClassPath>
<repositoryLayout>flat</repositoryLayout>
...
</configruation>
And that the calling of the generated script shows the error is depending on the thing that the location of the repository where all the dependencies are located in the folder is different than in the generated script defined.

How to include external classes in my jar

I have a problem with a service I am trying to write. I am trying to create a service that runs in the background on a windows system but uses java. I have seen several ways of doing this, but decided on one method that seemed to meet my requirements. The service will check a database for items it needs to work on. When it finds an item in the DB that it needs to do it will run some system commands to take care of them.
I found a way to use the tomcat7.exe file to run a jar as a service and that worked pretty well for basic stuff. Anything I write and compile into my jar file "myService.jar" we'll can call it goes well enough. The problem is that we already have several classes written for accessing the DB and running commands that are precompiled in a library of classes called BGLib-1.0.jar.
I have used this library in writing several jenkins plugins and had no problems calling functions from it. They all work fine when I create an hpi file and deploy it in Jenkins. There the compiler (Eclipse using Maven) packages the BGLib jar in with the plugin jar and Jenkins figures out how to get them to see one another.
When I build my service jar, however, it doesn't work when I deploy it.
I run a command like this to install the Tomcat exe renames to myservice.exe:
d:\myService\bin>myService.exe //IS//myService --Install=D:\myService\bin\myService.exe --Description="run some commands
Java Service" --Jvm=auto --Classpath=D:\myService\jar\myService.jar;D:\myService\jar\BGLib-1.0.jar --StartMode=jvm --
StartClass=com.myCompany.myService.myService --StartMethod=windowsService --StartParams=start --StopMode=jvm --StopClass
=com.myCompany.myService.myService --StopMethod=windowsService --StopParams=stop --LogPath=D:\myService\logs --StdOutpu
t=auto --StdError=auto
When I deploy this with code solely within the myService.jar the service behaves as expected, but when I try to call functions within the BGLib-1.0.jar I get nothing. The jvm appears to crash or become unresponsive. Debugging is a little tricky but it looks like I am getting class not found errors.
I tried adding the entry below in the POM file to see if changing the classpath entry in the manifest would help, but it didn't change the manifest. I am still kind of clueless ass to how the manifest file works. Any documentation on that would be cool. I have been to Maven's site and it doesn't seem to have comprehensive documentation on the tags available. Is there something I need to change in the manifest to get my jar to see external classes? Or is there something I can add that will get Maven to compile the classes from that jar in with my jar?
thanks in advance.
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<mainClass>com.myCompany.myService.myService</mainClass>
<customClasspathLayout>BGLib-1.0.jar</customClasspathLayout>
</manifest>
</archive>
</configuration>
To answer mainly the question of the title, you can the shade plugin to include dependencies into your final jar. You can even even relocate the class files (e.g. change package name) within the final jar so that the included classes don't conflict with different versions of the shaded dependency on the classpath. Not sure if this is the best solution for your particular problem though.
You can use the maven-dependency-plugin unpack-dependencies goal to include the contents of a dependency in the resulting artifact.
An example of how to do this would be:
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>${project.artifactId}-fetch-deps</id>
<phase>generate-sources</phase>
<goals>
<goal>unpack-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.outputDirectory}</outputDirectory>
<stripVersion>true</stripVersion>
<excludeTransitive>true</excludeTransitive>
<includeArtifactIds>protobuf-java</includeArtifactIds>
</configuration>
</execution>
</executions>
</plugin>
This will expand the protobuf-java dependency (flatten it) and include the contents in the resulting artifact generated by your build.
Looks to me you actually want to use the appassembler-maven-plugin, otherwise I'd go for the maven-shade-plugin.

Using different package declarations in .java classes than their real file system locations, in Eclipse

I am currently developing a code generator, and I have built a test suite for it. The general ideia of how it is organized is depicted in the following image:
The Tests.java is a standard JUnit test class. I expect it to be on
package tests.system.bytecode.temp_tests;
as any Java developer would expect.
On the other hand, IOrderable.java and TreeSet.java are to be used by my code generator (after they are compiled to .class files, that is). But it is a bit troublesome for me to manage their current big big package declarations. The situation will only get worse as I add more tests and start to try to organize(nest) even more the tests.
It'd be perfect if IOrderable.java and TreeSet.java could have package declarations independent of their real paths, far removing future maintenance problems.
Is it possible to accomplish this with Eclipse? Maybe using Ant or Maven?
Thanks
Compiling .java classes in my Eclipse project with package declarations that don't correspond to their physical path in the project.
Should be not possible.
But you can separete the classes by different source folders.
In Eclipse you can have several source folders. When compiling they get mixed, so it would be what you need: one src folder for the normal classes and one src-gen Folder for the generated classes.
And with maven you can do the same.
There you have by default:
src/main/java
src/test/java
But you can add for example src/main/java2 or target/generated-src
Therefore you need the org.codehaus.mojo build-helper-maven-plugin. For example:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.7</version>
<executions>
<execution>
<id>add-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>${basedir}/target/generated-src</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>

Categories