We Keep Coding

websphere classloader gives java.lang.LinkageError when loading slf4j LoggerFactory

[I originally posted this on the websphere forums bot didn't see a timely response. I'm re-posting here with slightly more content]
The Problem
I added a jar (call it "Foo.jar") to our enterprise application (i.e. ".ear"), created the manifest entry, etc., and found, to my surprise, that when my EJB instantiated FooClient, websphere threw a java.lang.LinkageError upon org.slf4j.LoggerFactory
I've added classloader debugging to the trace and did not find much. I was hoping to see "where WAS actually found the slf4j class". Here are the entries:
[9/24/14 16:23:01:417 CDT] 00000067 CompoundClass > loadClass org.slf4j.LoggerFactory this=com.ibm.ws.classloader.CompoundClassLoader#5c2687e4[app:yourappServer] Entry
[9/24/14 16:23:01:418 CDT] 00000067 CompoundClass < loadClass org.slf4j.LoggerFactory java.lang.LinkageError: org.slf4j.LoggerFactory Exit
I've checked my was8 directory and found various copies/versions of the slf4j jar.
/plugins
/osgiappbundles/com.ibm.ws.osgi.applications/aries/slf4j-api-1.5.6.jar
/slf4j-api-1.5.6.jar
/profiles/AppSrv01
/installedConnectors/activemq-rar-5.9.1.rar/slf4j-api-1.7.5.jar
/wstemp
/appdepl14769b0c2de/slf4j-api-1.7.5.jar3472682177251273594.tmp
/appdepl14769b0c2de/slf4j-api-1.7.5.jar7922145899030561292.tmp
Another developer told me "you won't be able to use Foo.jar" because we would need to put "Parent_last" and that's not an option for this project currently.
Additional Context
Websphere 8.0.0.7
RAD 9
Questions
java.lang.LinkageError : is this a 'classloader problem' or 'wrong version of class' problem? My experience with "wrong version of class" problems I usually see "NoSuchMethod" errors, not instantiation exceptions.
Any other way around this?
Updates
I read a bit about shared libraries and "isolated classloader" and this seemed the solution. However for some reason, my RAD/Eclipse UI seems broken as it does not 'save' when I associate the shared library with an application or module. It subsequently disappears.
thanks in advance

if you want to use newer library which is already there in websphere parent_last is the way to go
if not, remove the slf4j jar from the app, and use the version that is given by vendor

Check the version of SL4J expected by Foo.jar and the one you are using: they are surely different. Ensure the correct version has precedence in your classpath: you can put it in WEB-INF/lib but also check that you have just one version of it in WEB-INF/lib.
To discover which SL4J jar you are using you can run
java -verbose:class ....
and the JVM will dump out what it's loading, and from where.
You say "I usually see NoSuchMethod errors" instead of LinkageError; in this case the method exists but has some other incompatibility.

JAXBException caused by generated code

I am getting the following JAXBException of the form
class SomeClass nor any of its super class is known to this context.
The full stack trace looks like this:
javax.xml.ws.WebServiceException: javax.xml.bind.JAXBException: class com.myCompany.generatedCode.WebServiceOperationName nor any of its super class is known to this context.
at org.apache.axis2.jaxws.ExceptionFactory.createWebServiceException(ExceptionFactory.java:175)
at org.apache.axis2.jaxws.ExceptionFactory.makeWebServiceException(ExceptionFactory.java:70)
at org.apache.axis2.jaxws.ExceptionFactory.makeWebServiceException(ExceptionFactory.java:128)
at org.apache.axis2.jaxws.core.controller.impl.AxisInvocationController.execute(AxisInvocationController.java:586)
at org.apache.axis2.jaxws.core.controller.impl.AxisInvocationController.doInvoke(AxisInvocationController.java:130)
at org.apache.axis2.jaxws.core.controller.impl.InvocationControllerImpl.invoke(InvocationControllerImpl.java:93)
at org.apache.axis2.jaxws.client.proxy.JAXWSProxyHandler.invokeSEIMethod(JAXWSProxyHandler.java:364)
at org.apache.axis2.jaxws.client.proxy.JAXWSProxyHandler.invoke(JAXWSProxyHandler.java:185)
at $Proxy41.deleteAccount(Unknown Source)
at com.myCompany.myPackage.MyWebServiceClient.callSomeWebService(MyWebServiceClient.java:100)
(other classes specific to my application and framework)
...
Caused by:
javax.xml.bind.JAXBException: class com.myCompany.generatedCode.WebServiceOperationName nor any of its super class is known to this context.
at com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.getBeanInfo(JAXBContextImpl.java:556)
at com.sun.xml.internal.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:452)
at com.sun.xml.internal.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:314)
at com.sun.xml.internal.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:243)
at javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:75)
at com.ibm.xml.xlxp2.jaxb.marshal.MarshallerProxy.marshal(MarshallerProxy.java:100)
at org.apache.axis2.datasource.jaxb.JAXBDSContext$1.run(JAXBDSContext.java:470)
at org.apache.axis2.java.security.AccessController.doPrivileged(AccessController.java:76)
at org.apache.axis2.datasource.jaxb.JAXBDSContext.marshalByElement(JAXBDSContext.java:455)
at org.apache.axis2.datasource.jaxb.JAXBDSContext.marshal(JAXBDSContext.java:414)
at org.apache.axis2.jaxws.message.databinding.impl.JAXBBlockImpl._outputFromBO(JAXBBlockImpl.java:189)
at org.apache.axis2.jaxws.message.impl.BlockImpl.outputTo(BlockImpl.java:372)
at org.apache.axis2.jaxws.message.impl.BlockImpl.serialize(BlockImpl.java:296)
at org.apache.axiom.om.impl.llom.OMSourcedElementImpl.internalSerializeAndConsume(OMSourcedElementImpl.java:808)
at org.apache.axiom.om.impl.llom.OMElementImpl.internalSerialize(OMElementImpl.java:975)
at org.apache.axiom.om.impl.llom.OMElementImpl.internalSerializeAndConsume(OMElementImpl.java:1016)
at org.apache.axiom.soap.impl.llom.SOAPEnvelopeImpl.serializeInternally(SOAPEnvelopeImpl.java:271)
at org.apache.axiom.soap.impl.llom.SOAPEnvelopeImpl.internalSerialize(SOAPEnvelopeImpl.java:233)
at org.apache.axiom.om.impl.llom.OMElementImpl.internalSerializeAndConsume(OMElementImpl.java:1016)
at org.apache.axiom.om.impl.llom.OMNodeImpl.serializeAndConsume(OMNodeImpl.java:488)
at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:88)
at com.ibm.ws.websvcs.transport.http.SOAPOverHTTPSender.writeMessage(SOAPOverHTTPSender.java:3271)
at com.ibm.ws.websvcs.transport.http.SOAPOverHTTPSender.sendChunkedRequest(SOAPOverHTTPSender.java:888)
at com.ibm.ws.websvcs.transport.http.SOAPOverHTTPSender.sendSOAPRequest(SOAPOverHTTPSender.java:807)
at com.ibm.ws.websvcs.transport.http.SOAPOverHTTPSender.send(SOAPOverHTTPSender.java:611)
at com.ibm.ws.websvcs.transport.http.HTTPTransportSender.invoke(HTTPTransportSender.java:364)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:531)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:401)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:228)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:163)
at org.apache.axis2.jaxws.core.controller.impl.AxisInvocationController.execute(AxisInvocationController.java:581)
... 52 more
What triggers this is attempting to make a call to a web service operation named WebServiceOperationName. I've seen other tips for dealing with this error which usually involve adding a #XmlSeeAlso annotation or making adjustments to how marshalling and unmarshalling are done.
In this case, I am calling a webservice whose code is generated by Maven based on a WSDL (and supporting schemas). I don't have direct control over the code produced by Maven, and I don't have control over the marshall/unmarshall calls. Other webservice calls in the application don't seem to be having any problem.
Even stranger, I can't reproduce this on my local server, though it happens on the development server. The generated JAR for the webservice is the same, as it the rest of the application's code.
This exception has appeared suddenly this week (and consistently), and I suspect that something has changed, environmentally, but I'm not sure what. Any ideas?
A new observation:
The first time I attempt to run this code on the server after it's been started/restarted, it takes a while (About a minute) before it fails and throws an exception. Every time after that, the exception is almost instantaneous...
Further information:
This problem is present in WebSphere 7.0.0.23 (which is on the server), but not in WebSphere 7.0.0.7 (on the workstation).

The issue's been resolved. I was inspecting the jar that contained the class WAS was complaining about and noticed that some of the classes were duplicated at different levels of package hierarchy. It's worth pointing out that the class that was reported in the original error was not actually one of these duplicated classes. Also, the application did not reference different variants of the same class - the classes referenced were in the expected package. The extra duplicates one level up were not referenced anywhere (as far as I can tell).
It looked like a bad Maven configuration, but when I inspected the xjb binding files, I couldn't figure out how the types in that schema namespace got bound to two different packages. Either way, I cleanded up some of the binding files used by Maven, refactored common bindings into one file, had Maven rebuild the jar, and then tested, and suddenly the issue was gone!
Clearly there was a problem in WAS 7.0.0.23 since the original jar worked fine on WAS 7.0.0.7. It may have been caused by a problem with the Maven bindings which resulted in a strange-looking but technically valid jar.
Also interesting: one of our developers upgraded WAS to 7.0.0.25 and used the original "bad" jar and got a "no such operation" error instead of the JAXBException.

I have the same problem in Guidewire ClaimCenter, which is an insurance application that can be configured but the core can't be modified at all, It's Java based it has component included like axis2 but I created a seperate java project that wraps a webservice call, I created all the stub code with wsimport. It works well under my deve enviroment which is windows and jetty because is the Guidewire platform to configure it (no other option). we use WAS 7 as production server.
Caused by: javax.xml.ws.WebServiceException: javax.xml.bind.JAXBException: au.com.mycompany.policyinquiry.retrieveinsurancepolicydetails.v1.RetrieveInsurancePolicyDetailsResp is not known to this context
at org.apache.axis2.jaxws.ExceptionFactory.createWebServiceException(ExceptionFactory.java:175)
at org.apache.axis2.jaxws.ExceptionFactory.makeWebServiceException(ExceptionFactory.java:70)
at org.apache.axis2.jaxws.ExceptionFactory.makeWebServiceException(ExceptionFactory.java:128)
at org.apache.axis2.jaxws.marshaller.impl.alt.DocLitBareMinimalMethodMarshaller.demarshalResponse(DocLitBareMinimalMethodMarshaller.java:158)
at org.apache.axis2.jaxws.client.proxy.JAXWSProxyHandler.createResponse(JAXWSProxyHandler.java:499)
at org.apache.axis2.jaxws.client.proxy.JAXWSProxyHandler.invokeSEIMethod(JAXWSProxyHandler.java:377)
at org.apache.axis2.jaxws.client.proxy.JAXWSProxyHandler.invoke(JAXWSProxyHandler.java:185)
at $Proxy80.retrieveInsurancePolicyDetails(Unknown Source)
I tried to generate and compile the client in the server side just in case it was something related with different JDK platform, but that didn't solve the problem.
I found the next article was 6!
but for WAS 6 , I haven't tried though.
When the JAX-WS web service engine receives a message, it uses
the JAXB engine to convert the message into java beans.
The message indicates a failure occurred while converting xml
into an object of type
com.somecom.lib.business.xml.ComIdentifierXml. The failure
indicates that the JAXB engine was not initialized with
information about this user class.
.
In this use case, the class
com.somecom.lib.business.xml.ComIdentifierXml is packaged in a
JAR at the EAR level. The exception does not happen if the
class is packaged in a JAR located in the WAR module's
WEB-INF/lib directory.
SOLUTION:
Target Environment:
IBM server
IBM JDK 1.6
IBM Webpshere 7.0.0.25
The client to call the webservice is using pure JAVA no dependency on external library JAX-WS RI 2.1.6
In webspehre
1- Set for the application the class loading to parent last
2- Turn off webservice annotations with this: Set the com.ibm.websphere.webservices.DisableIBMJAXWSEngine property to true. this will disabled Websphere own implementation based on a modified Axis2
3- Deploy your application with third-party JAX-WS runtime
in my case i used Glassfish Metro 1.5 which includes JAX-WS RI 2.1.7 and is the same as my local environment with JDK 1.6.0_37
That's it
Cheers
Alex :)

WSDL parsing exception creating Webservice client with CXF JaxWSClientFactoryBean

I've written some code to create and run webservice client using CXF. I used JaxWsClientFactoryBean (not sure it's the best solution) to create client from .wsdl file.
The goal here was to do this programmatically avoiding Spring etc. Just pure code with Java and CXF.
JaxWsClientFactoryBean cfb = new JaxWsClientFactoryBean();
cfb.setAddress(getServiceProperty(intClass, PROPERTY_KEY_URL_SUFFIX));
cfb.setServiceClass(intClass);
cfb.setOutInterceptors(getOutInterceptors(intClass));
cfb.setServiceName(SERVICE_NAME);
cfb.setWsdlURL("classpath:wsdl/" + intClass.getSimpleName() + ".wsdl");
cfb.setEndpointName(ENDPOINT_NAME);
Client client = cfb.create();
ClientProxy cp = new ClientProxy(client);
I intService = (I)
Proxy.newProxyInstance(intClass.getClassLoader(), new Class[] { intClass }, cp);
I'm really not sure if this is done correctly, but it works when I run this code locally and when I deploy it on Tomcat.
Unfortunatelly I need to run this code on Weblogic and this results in strange exception:
Caused by: javax.wsdl.WSDLException: WSDLException: faultCode=PARSER_ERROR: org.w3c.dom.DOMException: HIERARCHY_REQUEST_ERR: An attempt was
made to insert a node where it is not permitted.
at org.apache.cxf.wsdl11.WSDLManagerImpl.loadDefinition(WSDLManagerImpl.java:235)
at org.apache.cxf.wsdl11.WSDLManagerImpl.getDefinition(WSDLManagerImpl.java:186)
at org.apache.cxf.wsdl11.WSDLServiceFactory.<init>(WSDLServiceFactory.java:92)
... 26 more
Caused by: org.w3c.dom.DOMException: HIERARCHY_REQUEST_ERR: An attempt was made to insert a node where it is not permitted.
at com.sun.org.apache.xerces.internal.dom.ParentNode.internalInsertBefore(ParentNode.java:356)
at com.sun.org.apache.xerces.internal.dom.ParentNode.insertBefore(ParentNode.java:284)
at com.sun.org.apache.xerces.internal.dom.CoreDocumentImpl.insertBefore(CoreDocumentImpl.java:399)
at com.sun.org.apache.xerces.internal.dom.NodeImpl.appendChild(NodeImpl.java:235)
at org.apache.cxf.staxutils.StaxUtils.readDocElements(StaxUtils.java:1019)
at org.apache.cxf.staxutils.StaxUtils.readDocElements(StaxUtils.java:939)
at org.apache.cxf.staxutils.StaxUtils.read(StaxUtils.java:866)
at org.apache.cxf.wsdl11.WSDLManagerImpl.loadDefinition(WSDLManagerImpl.java:226)
... 28 more
This happens during application deployment. It looks like there is something wrong with .wsdl file, but wait... It was working on Tomcat!
I think that there could be some difference in com.sun.org.apache.xerces.* classes implementation within Weblogic with its JRockit VM and standard JVM, but I have no idea how to solve it.
I spent many hours trying differend ways of client creation. Most of them worked locally and in Tomcat, but none on WebLogic.
Any hints what to try next? I'm kinda tired of this topic :D

I agree with your suspicion that the problem is related to the used version of Xerces. The stacktrace shows that the Sun implementation of Xerces which is derivative of the Apache Xerces is used in your case.
Please check the Apache CFX Application Server Configuration Guide instructions related to WebLogic.
WebLogic ClassLoading
In WebLogic Server, any .jar file present in the system classpath is loaded by the WebLogic Server system classloader. All applications running within a server instance are loaded in application classloaders which are children of the system classloader. In this implementation of the system classloader, applications cannot use different versions of third-party jars which are already present in the system classloader. Every child classloader asks the parent (the system classloader) for a particular class and cannot load classes which are seen by the parent.
For example, if a class called com.foo.Baz exists in both $CLASSPATH as well as the application EAR, then the class from the $CLASSPATH is loaded and not the one from the EAR. Since weblogic.jar is in the $CLASSPATH, applications can not override any WebLogic Server classes.
In order to use an alternate version of Xerces you have to create a FilteringClassLoader.
Usage of FilteringClassLoader
The FilteringClassLoader provides a mechanism for you to configure deployment descriptors to explicitly specify that certain packages should always be loaded from the application, rather than being loaded by the system classloader. This allows you to use alternate versions of applications such as Xerces and Ant.
The FilteringClassLoader sits between the application classloader and the system. It is a child of the system classloader and the parent of the application classloader. The FilteringClassLoader intercepts the loadClass(String className) method and compares the className with a list of packages specified in weblogic-application.xml file.
In conclusion, check the steps included in the Apache CFX Application Server Configuration Guide and take care to explicitly specify that the org.apache.xerces.* package is loaded from the application, rather than being loaded from the system classloader.
For example the weblogic-application.xml file in the META-INF should look like:
<?xml version="1.0" encoding="UTF-8"?>
<weblogic-application xmlns="http://www.bea.com/ns/weblogic/90">
<application-param>
<param-name>webapp.encoding.default</param-name>
<param-value>UTF-8</param-value>
</application-param>
<prefer-application-packages>
<package-name>javax.jws.*</package-name>
<package-name>org.apache.xerces.*</package-name>
</prefer-application-packages>
</weblogic-application>
I hope this helps.

WAS 6 steps to follow to support MQ 7 features

I have problem with EAR module deployed in WAS6.
To support the MQ 7 features in my App. I follow the below steps:
Put Class loader policy as PARENT_LAST.
Placed all MQ 7 related jars in the root of EAR.
EAR contains Web module. When I try to start the application, I got following exception:
javax.servlet.jsp.JspException: Can't get definitions factory from context.
at org.apache.struts.taglib.tiles.InsertTag.processDefinitionName(InsertTag.java:575)
at org.apache.struts.taglib.tiles.InsertTag.createTagHandler(InsertTag.java:474)
at org.apache.struts.taglib.tiles.InsertTag.doStartTag(InsertTag.java:436)
at com.ibm._jsp._home._jspx_meth_tiles_insert_0(_home.java:106)
at com.ibm._jsp._home._jspService(_home.java:81)
The War contains the following jars.
xstream-1.3.1.jar,xercesImpl.jar, xalan.jar,struts.jar, standard.jar,commons-validator.jar, commons-net-1.4.0.jar, commons-fileupload.jar, commons-digester.jar, commons-collections.jar, commons-beanutils.jar,resolver.jar,jstl.jar, jfreechart-1.0.2.jar, jcommon-1.0.5.jar, jaxen-full.jar, jakarta-oro.jar.
EAR contains the following Jars,
com.ibm.mqjms.jar, com.ibm.mq.jmqi.jar, com.ibm.mq.jar, com.ibm.mq.headers.jar, com.ibm.mq.commonservices.jar,log4j.jar,dhbcore.jar.
And I set the class-path attribute in my Manifest file of the WAR with log4j.jar
Please anyone suggest me how Websphere's classloading policy works for where I went wrong.
Karthik

Some time ago I did something similiar. I wanted to use a specific version of a library which was already used within the WebSphere Application Server. That is the reason why you have to put your libraries in the EAR file and set the application server to PARENT_LAST class loader order.
Correct me if I am wrong, but you also have to specify your custom MQ client libraries in Manifest of your WAR file. You only mentioned Log4J. It should look somehting like this:
Class-Path: com.ibm.mqjms.jar com.ibm.mq.jmqi.jar [...] log4j.jar
Anyway, you can always check what libraries are in the Classpath of you application if you log into the Integrated Solutions Console (aka Admin Console) and check the Troubleshooting section. There is a classloader viewer. Just click yourself through the tree and check which library path are mentioned and which you would expect.
Finally, as Dylan already mentioned in his comment: WebSphere Application Server version 6.1 runs out of support September 30, 2012. :)

Controlling the classpath in a servlet

My servlet application includes a number of library .jars, some of which contain embedded log4j.xml or log4j.properties files. I'd like to ensure that log4j finds my log4j.xml first! I've tried searching for some specification of the priorities of the various classpath elements in a servlet (e.g. does WEB-INF/classes always precede WEB-INF/lib?), or some way to configure or tweak the servlet's classloader so that a given resource directory appears early in the classpath. So far, I've drawn a blank. Any suggestions on ensuring that a servlet .war file loads the correct log4j.xml via the classloader?

Tomcat 8.5
Ditto Tomcat 8.0.
See documentation: Class Loader HOW-TO.
Tomcat 8.0
The answer is simple, taken from the Tomcat documentation page, Class Loader HOW-TO. In particular notice the use of the /WEB-INF/ directory/folder.
Therefore, from the perspective of a web application, class or resource loading looks in the following repositories, in this order:
Bootstrap classes of your JVM
/WEB-INF/classes of your web application
/WEB-INF/lib/*.jar of your web application
System class loader classes (described above)
Common class loader classes (described above)
If the web application class loader is configured with <Loader delegate="true"/> then the order becomes:
Bootstrap classes of your JVM
System class loader classes (described above)
Common class loader classes (described above)
/WEB-INF/classes of your web application
/WEB-INF/lib/*.jar of your web application
Tomcat 6
Excerpted from Tomcat 6 page, Class Loader HOW-TO.
Therefore, from the perspective of a web application, class or resource loading looks in the following repositories, in this order:
Bootstrap classes of your JVM
System class loader classes (described above)
/WEB-INF/classes of your web application
/WEB-INF/lib/*.jar of your web application
$CATALINA_HOME/lib
$CATALINA_HOME/lib/*.jar

As far as I understand the resource selection from the classpath is non-deterministic (from the point of view of the app developer). Even if the same file is loaded consistently the behaviour could change:
1. When you upgrade the version of your current container.
2. If you switch containers.
The simplest solution will be to remove embedded log4j config files from library jars. It is almost never a good idea to embed log4j config's as it leads to the problem you are seeing here...
Are they third party jars or jars you developed?

We the Spring Log4jConfigListener in our web.xml file.
You can specify as a context parameter the location of the log4j config file, i.e. you could set it as /WEB-INF/log4j.xml
Would this be an option for you? If you're not using Spring I know that you can set the Log4j location programatically which might also work.

In my experience, WEB-INF/classes typically takes precedence over jars in WEB-INF/lib, however, that also depends on the servlet container you use (I could never figure out the behavior of JRun, for instance). It would help immensely if you could tell me which container you're using.
Also, are you certain that the offending log4j configuration is in a jar in WEB-INF/lib? Typically, when I've run into classpath problems in a servlet container situation, it's because of libraries that reside outside of the web app.
The servlet specs recommend that web app classloaders load their own classes before delegating to the container's classloader (SRV.9.7.2), but since this is counter to the Java spec, not all vendors do this by default (in fact Tomcat is the only container I've used that does this by default). With that said, it's always possible to configure your container's web app classloading behavior. If you tell me which container you're using, I may be able to help you (specifically, I have done this successfully before on WebLogic, WebSphere, Glassfish and JRun)).

If you're unable to control the classpath, since Tomcat is setting it for you, are you at least able to set a system property for log4j.configuration? I believe that location pointed to by that property can be set outside of the classpath.
If not, another approach, although an ugly one, would be to explicitly run one of the configurators yourself in your application code.

You need to have log4j.properties in your CLASSPATH. The best place is under WEB-INF/classes.
You also have to make sure that you use your version of log4j.jar. So, put it in WEB-INF/lib, just to make sure you are not using one from tomcat folders, since it may cause strange classloading issues.