We Keep Coding

Websphere: Shared libraries in common classloader earlier on classpath than application modules, even with parent last policy

Background:
I have the following problem: I have several WAR files I need to have deployed on same Websphere server. The WAR files use libraries that depend on having a specific version of XMLSec regisered as the XML Signature Provider (with the Java Security class). Currently I bundle this library with each WAR file (since the WAR files also need to work standalone and on Tomcat's without any special shared library configuration etc.). Each WAR files registers the provider with Security.addProvider() in a ServerContextListener. But this causes problems in the multi-WAR setup, because if one WAR file does the registration with Security.addProvider) and another WAR files tries to fetch it using the XMLSignatureFactory class (which is actually a javax.* class contained inside the XMLSec JAR itself, but which ultimately calls back to the global provider list configured with Security.addProvider), then it causes a ClassCastException inside XMLSignatureFactory, because this class does a cast of what it gets from Security into to its own version of the provider classes, which doesn't work. The exact stack trace is as follows:
Caused by: java.lang.ClassCastException:
org.apache.jcp.xml.dsig.internal.dom.DOMXMLSignatureFactory
incompatible with javax.xml.crypto.dsig.XMLSignatureFactory at
javax.xml.crypto.dsig.XMLSignatureFactory.findInstance(XMLSignatureFactory.java:202)
at
javax.xml.crypto.dsig.XMLSignatureFactory.getInstance(XMLSignatureFactory.java:292)
By the way this is not a case of conflict with different versions of XMLSec being in play or conflicts with Websphere's own version. There is only one version albeit it is loaded from different WAR's.
Of course the solution is to have the xmlsec library loaded with a common classloader so that there is only one version of the classes loaded that all WAR files see, which would avoid ClassCastExceptions etc.. But here is the rub: I also need to have each application loaded with the "parent last" policy - or rather, I need the JAR files inside each application to take precedence over Websphere's built-in version of the libraries (for instance Axis2 that I also include in the WAR filesetc.). Furter, I would prefer that I can keep the xmlsec library in each WAR files' WEB-INF/lib folder, so that the WAR files can still work stand-alone (i.e. in other environments which might not have the shared library configured etc.).
So basically I want to have a common class loader loading the XMLSec library, say, somewhere from disk. Let's denote that [SHARED XMLSEC]. Then I want each application classpath to ultimately appear like this:
App1: [SHARED XMLSEC][App1 WEB-inf/lib][Websphere libraries][JDK libraries]
App2: [SHARED XMLSEC][App2 WEB-inf/lib][Websphere libraries][JDK libraries]
etc.
In such a configuration it doesn't matter if App1+App2 themselves contain the XMLSec library since the shared one will take precedence so they will use the common one. At the same time, App1+App2 are still free to override other built-in Websphere libraries (Axis2).
Is it possible to realize this configuration and what options do I need to set? Do you see alternative ways to achieve the same objective?

Since you have a conflict between classes here, I would suggest going for isolated class loaders for each application. On the server side, setting the class loader policy to 'Multiple' should provide isolation between applications.
Once you have this set, configure class loading at the application level to the 'Parent last' configuration for both the applications.
The following Knowledge Center link has the relevant instructions [Steps 2,3 & 4 under the 'Procedure' section] :
http://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.multiplatform.doc/ae/trun_classload.html
[Note: The version of WAS in use is not specified in the question. The Knowledge Center link refers to version 8.5.5.]

Tomcat classloading doesn't seem to behave as documented

I'm running tomcat 6.0.23, and according to the classloader documentation webapps should look for classes in this order:
Bootstrap
System
WEB-INF/classes
WEB-INF/lib
Common
I have a webapp that uses hibernate and the hibernate jars are in it's WEB-INF/lib directory. When running on it's own, all works fine.
I also have a jar file that needs to sit in the tomcat/lib directory as it contains some classes that need to be loaded on startup (an object factory and the objects it creates). These classes use toplink for their JPA implementation and here is where I get a problem.
I need to put the toplink jars where they can be accessed on tomcat startup, so I've put them in the tomcat/lib directory. According to the order of classloading listed above, when the webapp that uses hibernate wants the hibernate implementation classes, it should find them in it's WEB-INF/lib directory, but what actually happens is it finds the toplink implementation classes from the tomcat/lib directory, and I get a class cast exception.
Can anyone please explain why my webapp class loaders aren't finding what they need in their WEB-INF/lib directory, or suggest a way to debug the classpath at runtime?
Thanks.

Be sure to read and understand the paragraph preceding the list you quoted:
"When a request to load a class from the web application's WebappX class loader is processed, this class loader will look in the local repositories first, instead of delegating before looking. There are exceptions. Classes which are part of the JRE base classes cannot be overriden. For some classes (such as the XML parser components in J2SE 1.4+), the J2SE 1.4 endorsed feature can be used."
What are these "hibernate implementation classes" that you're having problems with? (Hibernate's "implementation" classes would be completely different from Toplink's.) Are they javax.persistence classes? These may (or may not) fall under the category of "JRE base classes", which behave differently.
Edit: Based on your comment, this is just a typical cross-class-loader loading problem. Tomcat's class loading is working exactly as you expect it to. If you look into the JPA classes where this initialization is going on, you'll find a line like this:
Enumeration<URL> resources =
cl.getResources("META-INF/services/" + PersistenceProvider.class.getName());
That loads all PersistenceProviders from all ClassLoaders, including your Toplink one in the lib directory. It then immediately does this:
for ( PersistenceProvider provider : providers ) { ...
That's on line 77 of javax.persistence.Persistence, where your exception is coming from. That's because the PersistenceProvider class referred to on that line is from your webapp class loader, but the collection contains two instances: your Hibernate implementation from the same class loader and the Toplink one from a different class loader.
This global, static initialization is one major thing that's kept me from shifting to JPA. I still just use straight Hibernate because of problems like this.

Deploying 2 war files with common classes in JBoss

I have two war file app1.war and app2.war deployed in a single JBoss instance. Package names for java classes for both war files starts with com.myapp
To add further, there are some Classes that are common between the two apps while there are some that have same fully qualified class names but are different (Source Code has changed).
I want to know, if this could pose threat of any kind to the deployment scenario?

You could get class loading problems if your applications are not isolated, i.e. have their own class loading repository and class loaders. If you configure JBoss to isolate the applications from each other you should be fine (I don't know what is the default for your version but 4.2.3 that we use does not isolate apps by default).
To clarify that a bit:
If you have two classes with different implementations but the same FQCN you could get the wrong class from the class loader for the application that is loaded second. Even if the implementation was the same you could get class cast exceptions or other strange behavior if one app gets the class from the other app.

I had a similar situation with multiple apps.Look at my solution here

Best way is to isolate class loading for your application archives.
For JBoss 5.1.0 GA following worked for me.
Create jboss-classloading.xml file in WEB-INF folder.
Added following lines to this file
Here,
export-all="NON_EMPTY" => Makes sure the classes loaded for this app is not exported
import-all="true" => Imports and uses all of the class definition available.
parent-first="false" => If more than one class with same name is found, one defined under the application will be used first.
FYI. This also helped me embedding the log configuration of log4j in the application war file. Will need to place log4j.xml in WEB-INF/classes and have a log4j.jar in WEB-INF/lib folder.

There will be one class loader instance for each application or standalone module. In other words, classes in app1.war will be loaded in different class loader than the classes in app2.war. This is the default behavior of any Java EE server; So it really doesn't matter about having classes with the same package/names and/or different content. This is the default behavior of any Java EE server.
Having said that, if you tweak the class loader policy of the server or try to load classes (reflect) using anything other than Thread.currentThread().getContextClassLoader(), you could be asking for trouble.

Classloaders and sharing .jar files with Apache Tomcat

If I have classes that need to be shared between my webapp and Tomcat (e.g. a custom realm and principal), where should the .jar file containing those classes go?
Currently I'm putting the .jar in ${CATALINA_HOME}/lib. This result is a ClassCastException when assigning references from classes of the same type. Here's an example:
MyCustomPrincipal principal = (MyCustomPrincipal)FacesContext.getCurrentInstance().getExternalContext().getUserPrincipal();
The method above throws a ClassCastException. The method returns an actual MyCustomPrincipal type (since that's what my custom realm gave Tomcat when it performed authentication) that, apparently, was created by a different classloader. How do I fix this so both Tomcat and my webapp can use MyCustomPrincipal?
http://tomcat.apache.org/tomcat-6.0-doc/class-loader-howto.html
Any help is appreciated.
Andrew

It looks like you have 2 copies loaded, once in tomcat and once in your WEB-INF/lib jars or other classpath of your deployed application.
The reason you get classpath exception lies in the way a WAR looks for classes. Contrary to the normal Java rules, a war first looks inside the war for a class and only then passes the request to teh parent classloader.
A class's identity is dependent of the classloader and the same class loaded in 1 classloader will generate a classcast exception when it is casted in the other classloader.
The solution is to make sure that the war does not contain the classes which should be provided by the container. If you use maven you can mark these dependencies as 'provided', if you use ant, you have to split your classpath list in 2 and compile against both, but use only the ones you need for constructing the war.

Controlling the classpath in a servlet

My servlet application includes a number of library .jars, some of which contain embedded log4j.xml or log4j.properties files. I'd like to ensure that log4j finds my log4j.xml first! I've tried searching for some specification of the priorities of the various classpath elements in a servlet (e.g. does WEB-INF/classes always precede WEB-INF/lib?), or some way to configure or tweak the servlet's classloader so that a given resource directory appears early in the classpath. So far, I've drawn a blank. Any suggestions on ensuring that a servlet .war file loads the correct log4j.xml via the classloader?

Tomcat 8.5
Ditto Tomcat 8.0.
See documentation: Class Loader HOW-TO.
Tomcat 8.0
The answer is simple, taken from the Tomcat documentation page, Class Loader HOW-TO. In particular notice the use of the /WEB-INF/ directory/folder.
Therefore, from the perspective of a web application, class or resource loading looks in the following repositories, in this order:
Bootstrap classes of your JVM
/WEB-INF/classes of your web application
/WEB-INF/lib/*.jar of your web application
System class loader classes (described above)
Common class loader classes (described above)
If the web application class loader is configured with <Loader delegate="true"/> then the order becomes:
Bootstrap classes of your JVM
System class loader classes (described above)
Common class loader classes (described above)
/WEB-INF/classes of your web application
/WEB-INF/lib/*.jar of your web application
Tomcat 6
Excerpted from Tomcat 6 page, Class Loader HOW-TO.
Therefore, from the perspective of a web application, class or resource loading looks in the following repositories, in this order:
Bootstrap classes of your JVM
System class loader classes (described above)
/WEB-INF/classes of your web application
/WEB-INF/lib/*.jar of your web application
$CATALINA_HOME/lib
$CATALINA_HOME/lib/*.jar

As far as I understand the resource selection from the classpath is non-deterministic (from the point of view of the app developer). Even if the same file is loaded consistently the behaviour could change:
1. When you upgrade the version of your current container.
2. If you switch containers.
The simplest solution will be to remove embedded log4j config files from library jars. It is almost never a good idea to embed log4j config's as it leads to the problem you are seeing here...
Are they third party jars or jars you developed?

We the Spring Log4jConfigListener in our web.xml file.
You can specify as a context parameter the location of the log4j config file, i.e. you could set it as /WEB-INF/log4j.xml
Would this be an option for you? If you're not using Spring I know that you can set the Log4j location programatically which might also work.

In my experience, WEB-INF/classes typically takes precedence over jars in WEB-INF/lib, however, that also depends on the servlet container you use (I could never figure out the behavior of JRun, for instance). It would help immensely if you could tell me which container you're using.
Also, are you certain that the offending log4j configuration is in a jar in WEB-INF/lib? Typically, when I've run into classpath problems in a servlet container situation, it's because of libraries that reside outside of the web app.
The servlet specs recommend that web app classloaders load their own classes before delegating to the container's classloader (SRV.9.7.2), but since this is counter to the Java spec, not all vendors do this by default (in fact Tomcat is the only container I've used that does this by default). With that said, it's always possible to configure your container's web app classloading behavior. If you tell me which container you're using, I may be able to help you (specifically, I have done this successfully before on WebLogic, WebSphere, Glassfish and JRun)).

If you're unable to control the classpath, since Tomcat is setting it for you, are you at least able to set a system property for log4j.configuration? I believe that location pointed to by that property can be set outside of the classpath.
If not, another approach, although an ugly one, would be to explicitly run one of the configurators yourself in your application code.

You need to have log4j.properties in your CLASSPATH. The best place is under WEB-INF/classes.
You also have to make sure that you use your version of log4j.jar. So, put it in WEB-INF/lib, just to make sure you are not using one from tomcat folders, since it may cause strange classloading issues.