I have several huge legacy applications that I am now working on. After months of testing, we finally reached deployment only to have a "failed to load webapplicationcontext" which foiled the whole endeavor. That specific failure was due to a name space conflict between two transitive dependencies. i.e., both jars had a class to load as: org.something.somethingelse.ClassName.
There are ~100 jars pulled in via maven for this single project. Several explicit, most transitive. Ideally, I would like to know every single jar I'm putting on my classpath. Practically, though, I don't have enough experience or time to look through every one of them for potential issues.
Is there a tool, technique, or eclipse/intelliJ feature that I can use to scan a set of jars for similar namespaces?
You can try with enforcer plugin. In a maven project, it's very usefull when you need to detect different jar depenndencies of same artifact with different version.
You can read this post too.
So there were a couple of different solutions here. I ended up using jhades (http://jhades.github.io/) to identify conflicts within the war, and then tattletale (a utility provided by JBOSS support) to identify conflicts between the war and the container.
I added 'exclude *' tags to all the explicit dependencies to prevent any transitive dependencies from loading. I added explicit dependencies for anything that still wasn't present. After ensuring that all compiled dependencies played nicely, I set any libraries identified by tattletale to provided and added the necessary module to standalone.xml. These things like hibernate, apache libs, servlet APIs etc.
The other thing I discovered which made this so difficult to identify in the first place is that JBOSS's classloader indexes libraries according to how the hosting file system orders them. On Windows, which is where we do 90% of our development, they are always loaded alphabetically. On linux, where we do our production deployments, the order is pseudo random. Our production servers are built from the same images, so a RHEL 3.4 server will load in the same order as another 3.4, but a 3.5 will load entirely differently. Thus, we did not see a failure until the stars aligned and we deployed to a 3.6 server. In production.
Hope this helps someone.
Related
We are running our Java EE applications in WAS 8.5 and Gradle 5.* to build them.
In the past we packaged our .war application in an .ear archive, which we then deployed on our server. We had to separate our libraries from our applications and include them as shared libraries, because in our experience it made deploying much slower and in some cases used up all system memory, crashing the server.
After some experimentation, we realized that we don't need to extract the dependencies into shared libraries, because we can include them in the lib folder of our .ear archive.
Currently, we get this done by defining the dependencies of our .war application as compileOnly and redefining them as earlib in the root project (which generates the .ear archive). I'm looking for a way to automate this procedure.
The script I used looks something like this:
project.configurations.named('deploy').getAllDependencies().withType(ProjectDependency.class).forEach({dependency ->
project.configurations.named('earlib').getAllDependencies()
.addAll(dependency.dependentProject.configurations.named('earlib').getAllDependencies())
})
// This loosely resembles the actual code I used. The thought process is right, it just might have a couple syntax errors.
// Obviously, I defined an `earlib` configuration in the subproject
I tried running this code in the configuration phase, as well as in the doFirst{} section of the ear task. They all had different problems.
The former didn't work, because it seems like in the configuration phase when this code ran, the dependencies weren't configured yet.
The latter didn't work, because I can't just add dependencies during runtime (thinking back, it sounds ridiculous that I even tried it).
My question is: Can I find a phase in the build lifecycle, where I can find and modify the dependencies? Is there another workaround to solve my problem?
The technical answer to your questions is that you can use either:
A configuration.incoming.beforeResolve hook to do it last minute, only when the configuration really needs to be resolved.
Use an afterEvaluate block, assuming all the other dependencies are not defined in an afterEvaluate themselves.
However, the right solution would be to leverage the dependency management engine of Gradle and effectively declare that your root project, the one building the EAR, has dependencies on the specific configurations of the subprojects.
Not knowing your full setup and details, I believe the above would still be the more correct solution, though you may have to filter the subproject artifacts from the resulting graph.
Ideas on how this works in recent Gradle version: https://docs.gradle.org/6.2/userguide/cross_project_publications.html Most of the things explained there should work with the latest 5.x versions.
why use Maven when you have such quantity of local jars?
So we have a client that have a lot of private jars and custom jars.
For example commons-langMyCompanyCustom.jar which is commons-lang.jar with 10 more classes in it.
So on their environment we use 100% Maven without local dependencies.
But on our site we have the jars for development in Eclipse and have Maven build with the public ones, but we do not have permission to add their jars in our organizational repository.
So we want to use the Maven good things like: compile,test, build uber-jar, add static code analysis, generate java-docs, sources-jars etc. not to do this thinks one by one with the help of Eclipse.
So we have 70 jar some of them are public if I get the effective pom on their environment I found 50 of them in Maven Central, but the other 20 are as I called "custom" jars. I searched for decision of course but found this:
<dependency>
<groupId>sample</groupId>
<artifactId>com.sample</artifactId>
<version>1.0</version>
<scope>system</scope>
<systemPath>${project.basedir}/src/main/resources/yourJar.jar</systemPath>
</dependency>
So for all 20 of them I have to add this in the development maven profile??
Is there a easy way like in Gradle where you can add all folder with its dependencies to the existing ones?
Also installing one by one in every developer's repo is not acceptable.
Please forget the system scope as mentioned before! Too problematic...
Ideally:
Ideally, all your developers have access to Repository Manager in your or their organization (if possible).
A central environment for your System Integration Testing, maybe?
Alternatively, you may have a central environment for testing where all the dependencies are provided. This approach can be used to simulate how a compilation would work as if it's in your client's environment. Plus you only setup jars once.
So on their environment we use 100% Maven without local dependencies.
But on our site we have the jars for development in Eclipse and have
Maven build with the public ones, but we do not have permission to add
their jars in our organizational repository.
According to what you're saying in the above-quoted excerpt I believe you want to have set in your build's pom.xml assuming that in the client setup the dependencies will be present.
Especially, as you indicate that the organization doesn't give you permission to add their jars in your repository, I would use the provided scope.
As stated in the Maven docs, the definition of a provided dependency is as followed:
This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scope provided because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.
So basically you assume that these dependencies will be present at your client's setup. However, this has some limitations. Meaning you can build solutions independently but cannot test it locally because you won't have the dependencies on your workstation.
If you won't even have access to the jars to configure your central environment ask if your client can provide a DEV/SIT environment.
None of the above? Inherit a parent pom.
To avoid the whole constant copy-paste process for every single (related) project, maven has the tools to centralize dependency and plugin configurations, one of such is by inheriting the configuration of a parent pom. As is explaining in the following documentation it is quite simple:
First you create a project with just a pom.xml where you define everything you wish to centralize (watch out, certain items have slight differences in their constructs);
Use as property of packaging tag the option pom: <packaging>pom</packaging>;
In the pom's that have to inherit these configurations set the parent configuration tags in <parent> ... </parent> (documentation is very clear with this);
Now everytime you update any "global" pom configuration only the parent version has to be updated on every project. As a result of this, you only need to configure everything once.
You can also apply this together with the abovementioned solutions, this way combining to find a solution that fits best to your needs.
But there is a big Maven world out there, so I advise a good read in its doc's to further acknowledge your possibilities. I remembered these situations because I've been in a similar situation you seem to be now.
Good luck!
Another alternative is the project RepoTree.
This one creates a Maven repository directory (not a server) from another directory which contains just the .jars. In other words, it creates the necessary .pom files and directory structure. It takes into account only the precise information from metadata contained in the archives (MANIFEST.MF, pom.xml).
Utility to recursively install artifacts from a directory into a local
Maven repository Based on Aether 1.7
This is 5 years old, but still should work fine.
TL;DR: MavenHoe creates a Maven repository server (not a directory) which serves the artefacts from a directory, guessing what you ask for if needed. The purpose is to avoid complicated version synchronizing - it simply takes whatever is closest to the requested G:A:V.
I have moved the MavenHoe project, which almost got lost with the decline of Google Code, to Github. Therefore I put it here for availability in the form of a full answer:
One of the options you have when dealing with conditions like that is to take whatever comes in form of a directory with .jar's and treat it as a repository.
Some time ago I have written a tool for that purpose. My situation was that we were building JBoss EAP and recompiled every single dependency.
That resulted in thousands of .jars which were most often the same as their Central counterpart (plus security and bug fixes).
I needed the tests to run against these artifacts rather than the Central ones. However, the Maven coordinates were the same.
Therefore, I wrote this "Maven repository/proxy" which provided the artifact if it found something that could be it, and if not, it proxied the request to Central.
It can derive the G:A:V from three sources:
MANIFEST.MF
META-INF/.../pom.xml
Location of the file in the directory, in combination with a configuration file like this:
jboss-managed.jar org/jboss/man/ jboss-managed 2.1.0.SP1 jboss-managed-2.1.0.SP1.jar
getopt.jar gnu-getopt/ getopt 1.0.12-brew getopt-1.0.12-brew.jar
jboss-kernel.jar org/jboss/microcontainer/ jboss-kernel 2.0.6.GA jboss-kernel-2.0.6.GA.jar
jboss-logging-spi.jar org/jboss/logging/ jboss-logging-spi 2.1.0.GA jboss-logging-spi-2.1.0.GA.jar
...
The first column is the filename in the .zip; Then groupId (with either slashes or dots), artifactId, version, artifact file name, respectively.
Your 70 files would be listed in this file.
See more information at this page:
https://rawgit.com/OndraZizka/MavenHoe/master/docs/README.html
The project is available here.
Feel free to fork and push further, if you don't find anything better.
I know that many subject speak about classpath loader and transitive dependancy but i m little confused.
I m working on a legacy application that use spring 2.0.5 and spring-ws and all work fine from the beginning of the project to now.
And recently we are faced to a runtime problem with a an exception like methodnotfoundexception.
In fact we see that spring-ws depend on spring-2.5. So maven transitive dependancy add spring 2.5 in my webinf/lib directory near spring 2.0.5
But i dont understand why all working fine during many years and now why weblogic decide to load spring2.5 before spring2.0.5 and cause this error?
We have correct the problem and now i m looking for same jar with different version and do build failure when i have a conflit to avoid dependancy problem in the future.
But i just want to understand why weblogic decide to load one or another jar ? Because if its alphabetical order, same jar will be loaded all the time, but in my exemple the order change...
So i don t understand clearly whats happened.
Thanks by advance;)
While it is documented that WEB-INF/classes is loaded before WEB-INF/lib, the documentation is not clear in cases like yours - where there are sibling copies of Spring in the WEB-INF/lib.
In Tomcat it is alphabetical, but looks like Websphere, Jboss and Weblogic are random. See https://stackoverflow.com/a/5817890/327426
From some threads on the Oracle forums, the recommendation is to run the Classloader Analysis Tool available at http://docs.oracle.com/cd/E23943_01/web.1111/e13706/classloading.htm#BABHJGJJ for your app and see the results. This tool will identify classloading conflicts.
See similar issue at https://community.oracle.com/thread/2387921?tstart=0
This post from 2009 http://www.coderanch.com/t/472496/BEA-Weblogic/Loading-order-jar-WEB-INF on Weblogic v9.2 states "Checking the logs I see the .jars in WEB-INF/lib are being loaded in reverse alphabetical order". This may or may not be true for your version
Another option is to use Maven to bypass the transitive import being packaged in the WAR
I'm using jarX that has embedded dependencies that conflict with my own dependencies, so I'm creating a classloader to isolate jarX's dependencies from my main classloader.
jarX is outside my app's classpath, but my classes that use jarX's classes are in my classpath, so when I instantiate my classes loaded via the custom classloader, I run into the class identity crisis in the form of ClassCastException as the JVM's version of my classes are considered different from those loaded by my custom classloader.
I found this blog post where they solved a similar problem by only interacting with the custom classloader loaded classes via reflection, which seems to solve this problem.
It just feels like it should be easier than this. Does anyone know a better way to handle this problem?
The easiest way is to open jarX, remove the offending classes, and done. It is a bad practice to embed dependencies in a JAR unless that is JAR is meant to be used only as a standalone runnable fat-jar. JARs that are meant to be used as libraries should not embed dependencies.
When you notice that people package third-party classes in their JARs, I'd recommend pointing out to them that this is generally not a good idea and to encourage them to refrain from doing so. If a project provides a runnable fat-jar including all dependencies, that is fine. But, it should not be the only JAR they provide. A plain JAR or set of JARs without any third-party code should also be offered. In the rare cases that third-party code was modified and must be included, it should be done under the package namespace of the provider, not of the original third-party.
Finally, for real solutions to building modular Java applications and handling classloader isolation, check out one of the several OSGi implementations or project Jigsaw.
Can you post which jar is it and what are the classes that it overlaps, with the full stacktrace? Have a look at this tool I wrote to generate a list of duplicate classes in the WAR, there is an option to exclude duplicates of the same size.
These are some measures that can be done to solve this:
Try to reduce the number of duplicates by doing a case by case analysis of why the overlap exists. Add maven exclusions for jars that are complete duplicates.
Check if there is a version of the same jar without the dependencies that you could use, which jar is it, xerces, etc?
If there is no jar without dependencies, you can you exclude the other jar that overlaps jarX and see if the application still works. This means all components that need the jar have a compatible version of the jarX library
Separate the application into two WARs each with the version of the library you need. This will reduce the number of libraries in which
These where measures that are likelly to be more maintainable long-term
If the previous measures do not work:
open the jar, delete the duplicate classes and publish in the maven repository with a different name jarX-patched
you can configure nexus to serve a patched jar instead of an unpatched jar transparently
If your container supports OSGI that would be even better, but if you don't use a OSGI container for development as well, then the application would not work in development.
While downloading Google Guice I noticed two main "types" of artifacts available on their downloads page:
guice-3.0.zip; and
guice-3.0-src.zip
Upon downloading them both and inspecting their contents, they seem to be two totally different "perspectives" of the Guice 3.0 release.
The guice-3.0.zip just contains the Guice jar and its dependencies. The guice-3.0-src.zip, however, did not contain the actual Guice jar, but it did contain all sorts of other goodness: javadocs, examples, etc.
So it got me thinking: there must be different "configurations" of jars that get released inside Java projects. Crossing this idea with what little I know from build tools like Ivy (which has the concept of artifact configurations) and Maven (which has the concept of artifact scopes), I am wondering what the relation is between artifact configuration/scope and the end deliverable (the jar).
Let's say I was making a utility jar called my-utils.jar. In its Ivy descriptor, I could cite log4j as a compile-time dependency, and junit as a test dependency. I could then specify which of these two "configurations" to resolve against at buildtime.
What I want to know is: what is the "mapping" between these configurations and the content of the jars that are produced in the end result?
For instance, I might package all of my compile configuration dependencies wind up in the main my-utils.jar, but would there ever be a reason to package my test dependencies into a my-utils-test.jar? And what kind of dependencies would go in the my-utils-src.jar?
I know these are a lot of tiny questions, so I guess you can sum everything up as follows:
For a major project, what are the typical varieties of jars that get released (such as guice-3.0.zip vs guice-3.0-src.zip, etc.), what are the typical contents of each, and how do they map back to the concept of Ivy configurations or Maven scopes?
The one you need to run is guice-3.0.zip. It has the .class files in the correct package structure.
The other JAR, guice-3.0-src.zip, has the .java source files and other things that you might find useful. A smart IDE, like IntelliJ, can use the source JAR to allow you to step into the Guice code with a debugger and see what's going on.
You can also learn a lot by reading the Guice source code. It helps to see how developers who are smarter than you and me write code.
I'd say that the best example I've found is the Efficient Java Matrix Library at Google Code. That has an extensive JUnit test suite that's available along with the source, the docs, and everything else that you need. I think it's most impressive. I'd like to emulate it myself.