I'm attempting to make a very simple Kafka Producer and am currently following the producer example except my producer does not have a partitioner class.
After exporting required files into a jar I transfer them to my Linux image and try to run it.
I get this exception:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.NoClassDefFoundError: org/apache/kafka/common/utils/Utils
at kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:103)
at kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:102)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:194)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:194)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:194)
at scala.collection.mutable.ArrayBuffer.map(ArrayBuffer.scala:44)
at kafka.client.ClientUtils$.parseBrokerList(ClientUtils.scala:102)
at kafka.producer.BrokerPartitionInfo.<init>(BrokerPartitionInfo.scala:32)
at kafka.producer.async.DefaultEventHandler.<init>(DefaultEventHandler.scala:41)
at kafka.producer.Producer.<init>(Producer.scala:60)
at kafka.javaapi.producer.Producer.<init>(Producer.scala:26)
at producers.HelloWorldProducer.main(HelloWorldProducer.java:20)
... 5 more
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.utils.Utils
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 19 more
After looking at the kafka jar I see that the utils is its own package now and not located within common.
What would be the best way to solve this issue?
The answer ended up being real silly ... I needed to use the kafka-clients-0.8.2.0.jar instead.
Related
I am having the following Stacktrace:
[2015-10-15 19:29:09,264] WARN
com.google.gerrit.server.plugins.PluginLoader : Cannot load plugin
gerrit-owners-autoassign-1.1
com.google.common.util.concurrent.ExecutionError:
com.google.common.util.concurrent.ExecutionError:
java.lang.NoClassDefFoundError:
com/google/gerrit/extensions/events/GitReferenceUpdatedListener$Update
at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) at
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
at
com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
at
com.google.inject.internal.FailableCache.get(FailableCache.java:48)
at
com.google.inject.internal.ConstructorInjectorStore.get(ConstructorInjectorStore.java:50)
at
com.google.inject.internal.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:136)
at
com.google.inject.internal.InjectorImpl.initializeBinding(InjectorImpl.java:533)
at
com.google.inject.internal.AbstractBindingProcessor$Processor$1.run(AbstractBindingProcessor.java:160)
at
com.google.inject.internal.ProcessedBindingData.initializeBindings(ProcessedBindingData.java:44)
at
com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:123)
at
com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107)
at
com.google.inject.internal.InjectorImpl.createChildInjector(InjectorImpl.java:226)
at
com.google.inject.internal.InjectorImpl.createChildInjector(InjectorImpl.java:233)
at
com.google.gerrit.server.plugins.ServerPlugin.startPlugin(ServerPlugin.java:190)
at
com.google.gerrit.server.plugins.ServerPlugin.start(ServerPlugin.java:170)
at
com.google.gerrit.server.plugins.PluginLoader.runPlugin(PluginLoader.java:461)
at
com.google.gerrit.server.plugins.PluginLoader.rescan(PluginLoader.java:390)
at
com.google.gerrit.server.plugins.PluginLoader.start(PluginLoader.java:295)
at
com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:74)
at com.google.gerrit.pgm.Daemon.start(Daemon.java:293) at
com.google.gerrit.pgm.Daemon.run(Daemon.java:205) at
com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:166)
at
com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:93)
at
com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:50)
at Main.main(Main.java:25) Caused by:
com.google.common.util.concurrent.ExecutionError:
java.lang.NoClassDefFoundError:
com/google/gerrit/extensions/events/GitReferenceUpdatedListener$Update
at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) at
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
at
com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
at
com.google.inject.internal.FailableCache.get(FailableCache.java:48)
at
com.google.inject.internal.MembersInjectorStore.get(MembersInjectorStore.java:68)
at
com.google.inject.internal.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:74)
at
com.google.inject.internal.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:29)
at
com.google.inject.internal.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:37)
at
com.google.inject.internal.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:33)
at
com.google.inject.internal.FailableCache$1.load(FailableCache.java:37)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 31 more Caused by: java.lang.NoClassDefFoundError:
com/google/gerrit/extensions/events/GitReferenceUpdatedListener$Update
at java.lang.Class.getDeclaredMethods0(Native Method) at
java.lang.Class.privateGetDeclaredMethods(Class.java:2615) at
java.lang.Class.getDeclaredMethods(Class.java:1860) at
com.google.inject.spi.InjectionPoint.getInjectionPoints(InjectionPoint.java:688)
at
com.google.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:380)
at
com.google.inject.internal.MembersInjectorStore.createWithListeners(MembersInjectorStore.java:93)
at
com.google.inject.internal.MembersInjectorStore.access$000(MembersInjectorStore.java:37)
at
com.google.inject.internal.MembersInjectorStore$1.create(MembersInjectorStore.java:45)
at
com.google.inject.internal.MembersInjectorStore$1.create(MembersInjectorStore.java:42)
at
com.google.inject.internal.FailableCache$1.load(FailableCache.java:37)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 46 more Caused by: java.lang.ClassNotFoundException:
com.google.gerrit.extensions.events.GitReferenceUpdatedListener$Update
at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 60 more
I am not really sure what I should look into to fix it.
This looks like a mismatch between the API version that the plugin was built against, and the API version of the Gerrit server.
java.lang.NoClassDefFoundError: com/google/gerrit/extensions/events/GitReferenceUpdatedListener$Update
The GitReferenceUpdatedListener.Update class was removed from the API in Gerrit version 2.8.
To fix this you should first check which version of Gerrit your server is running, and then make sure you've built the plugin against the API with the corresponding version.
I followed the instruction and ended up compiling the plugin with both changes submitted in here: https://github.com/vadims/gerrit-owners/issues/6 (look for the 2 forked repo) and it did work.
It was painful but the plugin does an awesome job so worth the effort!
I have an Infinispan Server, version 6.0.x, the one derived from JBoss 7.2, and it is working fine for caching.
However, when I try to monitor it by JMX I can't. This is URL I type in jconsole.bat :
service:jmx:remoting-jmx://MY.IP.ADDRESS.HERE:9999
But I get this error :
Exception in thread "VMPanel.connect" java.util.ServiceConfigurationError: javax.management.remote.JMXConnectorProvider: Provider org.jboss.remotingjmx.RemotingConnectorProvider could not be instantiated: java.lang.NoClassDefFoundError: org/jboss/logging/Logger
at java.util.ServiceLoader.fail(ServiceLoader.java:224)
at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
at javax.management.remote.JMXConnectorFactory.getConnectorAsService(JMXConnectorFactory.java:472)
at javax.management.remote.JMXConnectorFactory.newJMXConnector(JMXConnectorFactory.java:341)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
at sun.tools.jconsole.ProxyClient.tryConnect(ProxyClient.java:370)
at sun.tools.jconsole.ProxyClient.connect(ProxyClient.java:313)
at sun.tools.jconsole.VMPanel$2.run(VMPanel.java:292)
Caused by: java.lang.NoClassDefFoundError: org/jboss/logging/Logger
at org.jboss.remotingjmx.RemotingConnectorProvider.<clinit>(RemotingConnectorProvider.java:42)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:374)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
... 7 more
Caused by: java.lang.ClassNotFoundException: org.jboss.logging.Logger
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 14 more
Exception in thread "VMPanel.connect" java.util.ServiceConfigurationError: javax.management.remote.JMXConnectorProvider: Provider org.jboss.remotingjmx.RemotingConnectorProvider could not be instantiated: java.lang.NoClassDefFoundError: Could not initialize class org.jboss.remotingjmx.RemotingConnectorProvider
at java.util.ServiceLoader.fail(ServiceLoader.java:224)
at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
at javax.management.remote.JMXConnectorFactory.getConnectorAsService(JMXConnectorFactory.java:472)
at javax.management.remote.JMXConnectorFactory.newJMXConnector(JMXConnectorFactory.java:341)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
at sun.tools.jconsole.ProxyClient.tryConnect(ProxyClient.java:357)
at sun.tools.jconsole.ProxyClient.connect(ProxyClient.java:313)
at sun.tools.jconsole.VMPanel$2.run(VMPanel.java:292)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.jboss.remotingjmx.RemotingConnectorProvider
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:374)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
... 7 more
Maybe I should create some managed beans somewhere? How can I fix this?
Remark 1 : In the documentation they include the necearry configuration between <global></global>, but in my case the settings should respect JBoss Schema...
Remark 2 : I tried with JBoss 7.2, and it could show me graphs and so on, and by the way I used the same URL. The difference is that I used jconsole.bat of JBoss instead of Infinispan.
Please, any idea about the necessary configuration?
Thank you.
I'm a Windows 7 user, and this is the fix that worked for me.
Open jconsole.bat for editting.
At the top, turn on the echoing.
#echo on
Then go down and find this line
call :SearchForJars "%JBOSS_MODULEPATH%\system\layers\base\org\jboss\logging\main"
Insert the following line before it:
echo "%JBOSS_MODULEPATH%\system\layers\base\org\jboss\logging\main"
I'm not sure why that worked, but it did. If you get different ClassNotFound exceptions, try echo statements above the other jar searches.
I've got a word count example class called by a Jetty server's handle method. The job is submitted with:
JobClient.runJob(conf);
Sadly, the JobTracker sees nothing, the job does not get submitted, but it finishes and creates output. I've found out, that I have to inject my configurations with:
conf.addResource(new Path("/usr/local/hadoop/conf/mapred-site.xml"));
After adding this line, the JobTracker gets the job, but it fails with the following exception:
INFO: Task Id : attempt_201312071601_0005_m_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: WC2$Reduce
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:889)
at org.apache.hadoop.mapred.JobConf.getCombinerClass(JobConf.java:1049)
at org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1385)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:986)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:422)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: WC2$Reduce
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:881)
... 10 more
Caused by: java.lang.ClassNotFoundException: WC2$Reduce
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
... 11 more
My combiner class is the same as the reducer class. I've downloaded the Hadoop WordCount example without modifications. Only the mentioned addResource.
I am trying to join two datasets.. One of which is json..
I am relying on json-simple library to parse that json..
I am trying to use libjars.. So far .. for simple data processing.. the approach has worked.. but now i am getting the following error
Exception in thread "main" java.lang.NoClassDefFoundError: org/json/simple/parser/ParseException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
at org.apache.hadoop.mapreduce.lib.input.MultipleInputs.getMapperTypeMap(MultipleInputs.java:141)
at org.apache.hadoop.mapreduce.lib.input.DelegatingInputFormat.getSplits(DelegatingInputFormat.java:60)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1024)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1041)
at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:959)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
at org.select.Driver.run(Driver.java:130)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.select.Driver.main(Driver.java:139)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: org.json.simple.parser.ParseException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
I think I have implemented toolrunner.
hadoop jar domain_gold.jar org.select.Driver \
-libjars json-simple-1.1.1.jar $INPUT1 $INPUT2 $OUTPUT
The code
http://pastebin.com/7XnyVnkv
Place your third party JAR file on one of hadoop's default lib folder. For example, $HADOOP_HOME/share/hadoop/common/lib.
I am trying to run example from this particular site:
https://github.com/larsgeorge/hbase-book/tree/master/ch07
Now.. it compiles succesfully
But when I try to run an example, it gives me the following exception:
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113)
... 7 more
Caused by: java.lang.NoClassDefFoundError: org/json/simple/parser/JSONParser
at mapreduce.ParseJson2$ParseMapper.<init>(ParseJson2.java:48)
... 12 more
Caused by: java.lang.ClassNotFoundException: org.json.simple.parser.JSONParser
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
... 13 more
Now from the pom in this file, I can see that simple json is included but I am not sure why is it throwing this issue.?
I would say that the problem is that when run under map-reduce the jar you need is not in scope (i.e. not found). One way to fix that is to create a jar with all the dependencies and submit that as the job. Other options are to make sure it is available in each node and deploying the depdndecies to Hadoops' distribtued cache (but the first is the simplest - though it makes for slower job starts since the jar is bigger)