hadoop jaxb classpath issue - java

I am seeing this inside a hadoop job:
2014-08-21 09:26:57,216 ERROR org.apache.hadoop.mapred.Child: Error running child : java.lang.ExceptionInInitializerError
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:107)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:78)
at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:129)
at com.sun.proxy.$Proxy61.describeVehicle(Unknown Source)
at com.company.inventory.externalapi.externalsoapclient.ExternalClientImpl.getFromSoap(ExternalClientImpl.java:46)
at com.company.dataservices.external.hadoop.ExternalIngestionProcess.populateExternalCache(ExternalIngestionProcess.java:112)
at com.company.dataservices.external.hadoop.ExternalIngestionProcess.map(ExternalIngestionProcess.java:93)
at com.company.dataservices.external.hadoop.ExternalIngestionProcess.map(ExternalIngestionProcess.java:38)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.ClassCastException: com.sun.xml.bind.v2.runtime.JAXBContextImpl cannot be cast to com.sun.xml.internal.bind.api.JAXBRIContext
at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.<clinit>(SOAPFaultBuilder.java:550)
... 16 more
This issue looks similar to this issue, except I am not supplying any jax-b dependencies on my classpath. However, it does look like hadoop does.
Is there a way to force hadoop to use the jdk jaxb jars or only use it's own supplied jaxb jars, but not mix both?

Frankly speaking, it doesn't sound like a Hadoop bug. Proper solution, as you've already mentioned, is to have classpath fixed, but try this:
-Djavax.xml.bind.JAXBContext=com.sun.xml.internal.bind.v2.ContextFactory
Additional details

Related

Why Java doesn't recognize net.sf.jasperreports.engine.util.JRLoader class?

Good afternoon.
I am working on JDK 1.6.
I am trying to integrate JasperReports to the Java Web Application of company where I work. First, I created an simple Java Application for to test the reports generation, and I used the next libraries (jars):
Commons collections 3.2.1
Commons beanutils 1.9.0
Commons digester 2.1
Commons logging 1.2
Groovy all 1.8.0
Joda time 2.0
JasperReports 6.0.0
With these I was able to create a report but, when I transfered this libraries to the Java Web Application,happened the next exception:
javax.faces.FacesException: #{bean.functionCalled}: java.lang.reflect.InvocationTargetException
at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:118)
at org.primefaces.application.CleanupActionListener.processAction(CleanupActionListener.java:42)
...
Caused by: javax.faces.el.EvaluationException: java.lang.reflect.InvocationTargetException
at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:102)
at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102)
... 51 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
... 52 more
Caused by: java.lang.NoClassDefFoundError: net/sf/jasperreports/engine/util/JRLoader
... 85 more
Caused by: java.lang.ClassNotFoundException: net.sf.jasperreports.engine.util.JRLoader
at java.net.URLClassLoader$1.run(Unknown Source)
...
... 86 more
I found out that the cause of the problem is the commons loggin version, or the libraries combination.
By default, this Java Web Application has the next libraries:
Commons collections 3.2
Commons beanutils 1.6
Commons digester 1.8.1
Commons logging 1.1.1
Groovy all 1.5.4
Joda time 1.6
I'm making differents versions combinations but not working, the problem persist.
What you recommend me?
Thank you so much.
This is a missing jars in the class path issue.
I had this issue and found that with by adding the missing jars to the class path it resolved the issue.
From the below article java.dzone.com article here I found that the minimal jars necessary are...Now The versions numbers have changed but with these libraries only I managed to get it working.
jasperreports-3.5.3.jar and above version
commons-digester-2.1.jar
commons-collections-3.2.jar (commons-collections.jar)
commons-logging-1.2.jar
commons-beanutils.jar
iText-5.0.jar (used infor PDF exporting)

java.lang.UnsatisfiedLinkError: cqjnilinuxproxy (Not found in java.library.path)

I have webservices deployed on WAS that use Clearquest jars to connect to Clearquest. At runtime, I get the following errors.
java.lang.Exception: Cannot parse the output since it has an error. Error = ReasonCode = conflictCRVAP0049E (internal-error): Cannot instantiate protocol provider: cqjnilinuxproxy (Not found in java.library.path)
...java.lang.reflect.InvocationTargetException:
at com.ibm.rational.stp.cs.internal.util.StpExceptionImpl.realException(StpExceptionImpl.java:493)
at com.ibm.rational.stp.cs.internal.util.StpExceptionImpl.<init>(StpExceptionImpl.java:572)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:56)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39)
at java.lang.reflect.Constructor.newInstance(Constructor.java:527)
at com.ibm.rational.stp.client.internal.core.ProtocolSubprovider.setProtocolProvider(ProtocolSubprovider.java:614)
... 57 more
Caused by: java.lang.UnsatisfiedLinkError: cqjnilinuxproxy (Not found in java.library.path)
at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1090)
at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1054)
at java.lang.System.loadLibrary(System.java:510)
at com.rational.clearquest.cqjni.CQJNIBaseObj.<clinit>(Unknown Source)
at java.lang.J9VMInternals.initializeImpl(Native Method)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:235)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:202)
at com.ibm.rational.stp.client.internal.cqjni.CqJniProtocol.<init>(CqJniProtocol.java:2343)
... 62 more
I followed the technote https://www-304.ibm.com/support/docview.wss?uid=swg21515079 - "An exception occurs on 64-bit WebSphere servers when running reports from ClearQuest data sources that require the Java Native Interface (JNI).
As suggested, WAS 32-bit was installed instead on 64 bit. But the error still occurs. Can you please let me know where this library is or what is going wrong?
Thanks,
Aarthi
The ClearQuest libraries and jars need to be placed on the path so WebSphere can pick them up. Did you source cq_setup.csh prior to starting the WebSphere profile? You would need to source cq_setup.csh and then start the profile in the same shell.

HBase not starting after adding the jar file with Mapper/Reducer

I am trying to write a Mapper/Reducer for hbase and I added the jar. However after adding the jar file in lib directory, I cannot start hbase. I want to debug what is going wrong? How can I change the log level? Will it help?
Following is the exception:
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:143)
at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:217)
at org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:153)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:224)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2290)
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProtocolProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/security/UserGroupInformation;Lorg/apache/hadoop/conf/Configuration;Ljavax/net/SocketFactory;ILorg/apache/hadoop/io/retry/RetryPolicy;Ljava/util/concurrent/atomic/AtomicBoolean;)Lorg/apache/hadoop/ipc/ProtocolProxy;
at org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:420)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:316)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:178)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:665)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:601)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2625)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2607)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.hbase.util.FSUtils.getRootDir(FSUtils.java:1004)
at org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:562)
at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:364)
at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:307)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
... 7 more
So it seems the error was due to mismatch of hadoop libraries in the lib directory of HBase(which was hadoop--2.5.1) against my actual Hadoop installation(hadoop--2.6.0). My jar was looking for classes which it was not finding in the older version of the hadoop libraries due to which it was failing. this answer made me realize the issue. After I copied all the hadoop-*-2.6.0 jars in the lib directory, HBase started as expected. The same is also mentioned in HBase-Hadoop compatibility documentation.

No class def found java

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/http/protocol/HttpContext
at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:119)
at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:103)
at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:362)
at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:344)
at Gethtml_fromS3.main(Gethtml_fromS3.java:16)
Caused by: java.lang.ClassNotFoundException: org.apache.http.protocol.HttpContext
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 5 more
Can any one give me a solution for the above error?
Please check if you have httpcore jar. I got rid of this error, after adding httpcore-4.2.jar to my lib.
It's difficult to pinpoint the exact reason why your application raises this exception, but a few common pointers that might help you.
The exception is raised when you are using a class in your code that isn't available when it runs. In this case, it seems you are unable to use org.apache.http.protocol.HttpContext. Most likely, you created an application that does include the Amazon AWS jar, but lacks the Apache HTTPClient jar.
If you used gradle/maven to create your application, include it in your dependencies. It it's a plain Java program, ensure the required JARs are available in your classpath (e.g. the same place as you currently have the Amazon jar located).

Exception in thread "main" java.lang.NoClassDefFoundError:

I am trying to run a java program and I am getting the following run time error.The error is shown below.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/http/params/SyncBasicHttpParams
at org.apache.http.impl.client.DefaultHttpClient.createHttpParams(DefaultHttpClient.java:157)
at org.apache.http.impl.client.AbstractHttpClient.getParams(AbstractHttpClient.java:448)
at org.apache.http.impl.client.AbstractHttpClient.createClientConnectionManager(AbstractHttpClient.java:309)
at org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:466)
at org.apache.http.impl.client.AbstractHttpClient.createHttpContext(AbstractHttpClient.java:286)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:851)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at net.floodlightcontroller.core.internal.PacketStreamerClient.registerForPackets(PacketStreamerClient.java:90)
at net.floodlightcontroller.core.internal.PacketStreamerClient.main(PacketStreamerClient.java:51)
Caused by: java.lang.ClassNotFoundException: org.apache.http.params.SyncBasicHttpParams
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more
The error is java.lang.NoClassDefFoundError
Now the Obvious reason of NoClassDefFoundError is that a particular class is not available in Classpath, so we need to add that into Classpath or we need to check why it’s not available in Classpath if we are expecting it to be.
Now the files that I have added to the classpath are the following.
export CLASSPATH=$(JARS=(./lib/*.jar); IFS=:; echo "${JARS[*]}")
export CLASSPATH=$CLASSPATH:~/.m2/repository/org/apache/httpcomponents/httpclient/4.0.1/httpclient-4.0.1.jar
export CLASSPATH=$CLASSPATH:~/.m2/repository/org/apache/httpcomponents/httpcore/4.0.1/httpcore-4.0.1.jar
export CLASSPATH=$CLASSPATH:~/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar
export CLASSPAHT=$CLASSPATH:~/ms_thesis/ONOS/httpcore-4.1.jar
#export CLASSPATH=$CLASSPATH:~/ms_thesis/ONOS/lib/httpclient-4.2.jar
export CLASSPATH=$CLASSPATH:~/google-gson-2.2.4/gson-2.2.4.jar
What jar file should I add to find "org/apache/http/params/SyncBasicHttpParams" I am very new to java and don't know how to debug this issue.
According to the official documentation, the class that you seek is only in httpcore since 4.1. You'll want to pull a new JAR.
org/apache/http/params/SyncBasicHttpParams class is under httpcore-4.1-alpha1.jar , which can be downloaded from
http://mirrors.ibiblio.org/pub/mirrors/maven2/org/apache/httpcomponents/httpcore/4.1-alpha1/httpcore-4.1-alpha1.jar
In future if you need to find a jar for a class, then you can use this link, this is generally helpful:
http://www.findjar.com/
In your case it seems to br you missing out some necessory jar files to include in your project
otherwise you any one of your linked is giving an exception mainly because of static method invocation or intialization
Here's the link to get the jar file httpcore-4.1

Categories