I am facing and issue when trying to read data form Mongo DB.
I have Spark 1.6.3 which has Scala 2.10.5
I am using the Mongo DB Connector Version 1.1 and package 2.10
Following is the dependencies i had used in my Mavan
<dependency>
<groupId>org.mongodb.spark</groupId>
<artifactId>mongo-spark-connector_2.10</artifactId>
<version>1.1.0</version>
</dependency>
Getting error like
java.lang.NoClassDefFoundError: com/mongodb/spark/rdd/api/java/JavaMongoRDD
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:175)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:708)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.mongodb.spark.rdd.api.java.JavaMongoRDD
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
But was able to find the class existing in the reference
Thanks in Advance
Add mongo-spark-connector_2.10's jar to your run time class path
It can be done it followting ways,
Add mongo-spark-connector jar to Spark lib folder
spark-submit with --jars=/path/mongo-spark-connector.jar
spark.driver.extraClassPath or it's alias --driver-class-path to set extra classpaths on the node running the driver.
spark.executor.extraClassPath to set extra class path on the Worker nodes.
If you add Driver class path then you will have need to add executor class paths to make jar available to executor on worker nodes
Try to use the latest version jar 2.2.8 which shall have the class
mongo-spark-connector_2.10:2.2.8
The reason for this issue is,
When i am trying to build using Maven , i am not using Fat Jar build using tools.
Instead of Maven build i tried the following to build a FAT Jar using Eclipse
File -> Export -> Runnable Jar -> Select Class , Jar file Name and choose Option 1 in Library handling. Then click finish.
This is a old type of Jar , but it fixed the issue for me.
Thanks all for the assistance.
Related
I have a custom plugin for Elasticsearch 6.x which has a dependency on httpclient and httpcore jar files. I see that these jars are part of Elasticsearch installation and hence want to use them instead of packaging these jars in my custom plugin.
So, I updated ES_CLASSPATH variable in $ES_HOME/bin/elasticsearch-env as follows:
ES_CLASSPATH="$ES_HOME/lib/*:$ES_HOME/modules/reindex/httpclient-4.5.2.jar:$ES_HOME/modules/reindex/httpcore-4.4.5.jar"
With above configuration I trying to start Elasticsearch server but I am getting below error:
Caused by: java.lang.IllegalStateException: jar hell! duplicate codebases between plugin and core: [file:/usr/share/elasticsearch/modules/reindex/httpclient-4.5.2.jar, file:/usr/share/elasticsearch/modules/reindex/httpcore-4.4.5.jar]
at org.elasticsearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:517) ~[elasticsearch-6.8.4.jar:6.8.4]
... 14 more
If I remove including these jars in ES_CLASSPATH variable and try to start Elasticsearch server, I am getting below error:
Caused by: java.lang.ClassNotFoundException: org.apache.http.HttpEntity
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_40]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_40]
at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:810) ~[?:1.8.0_40]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_40]
... 15 more
So how can I add httpclient and httpcore jars to my plugin classpath?
Thanks in advance.
P.S. I have posted this question couple of days back in ES forum but haven't got any reply. Hence posting it here.
I am trying to deploy a connect-standalone job to stream from an mssql server however am facing an issue (Kafka-Connect is part of my Ambari deployment, not docker). This is the properties file I am using:
name=JdbcSourceConnector
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
connection.user=ue
connection.password=pw
tasks.max=1
connection.url=jdbc:sqlserver://servername
topic.prefix=iblog
query=SELECT * FROM IB_WEBLOG_DUMMY_small
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.json.JsonConverter
poll.interval.ms=5000
table.poll.interval.ms=120000
mode=incrementing
incrementing.column.name=ID
I have added the jar file sqljdbc42.jar to /usr/share/java
and have run export CLASSPATH=/usr/share/java/*
however I still run into the error Failed to find any class that implements Connector and which name matches io.confluent.connect.jdbc.JdbcSourceConnector
Am I doing anything wrong or can I check something else?
Kafka-Connect is part of my Ambari deployment
That would imply you are using Hortonworks installation
You need to
git clone https://github.com/confluentinc/kafka-connect-jdbc/
Checkout a release branch that ideally matches your Kafka version. For example branch v3.1.2 is Kafka 0.10.1.1
mvn clean package will generate some folders in target/ of that project
SCP those files to all Kafka Connect workers in your cluster into /usr/hdp/current/kafka/.../share/java/kafka-connect-jdbc (create this, if not exists)
Restart Kafka processes to pick up the new CLASSPATH settings
You may need some extra Confluent packages that JDBC connect depends on
You need to include the kafka-connect-jdbc jar file, which contains the io.confluent.connect.jdbc.JdbcSourceConnector class.
If you are using maven, you can add it as a dependency:
[Add the following repo to your project if you haven't done so yet.]
<repository>
<id>confluent</id>
<url>http://packages.confluent.io/maven/</url>
</repository>
After this, add the following dependency:
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-connect-jdbc</artifactId>
<version>3.3.0 (or whatever version you want)</version>
</dependency>
https://github.com/confluentinc/kafka-connect-jdbc/issues/356
I too had the same problem. with Couchbase connector not found
ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:113) java.util.concurrent.ExecutionException: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches com.couchbase.connect.kafka.CouchbaseSourceConnector
Setting classpath was losing the existing classpath and I couldn't append as the classpath
I moved the required jar file from kafka-connect-couchase/*.jar files to /path/kafka_verison/libs/
libs is A folder where all the jar file stored.
I met the same issue, I resolved it by running connect-standalone in the root folder of confluent, in my case this was: /opt/confluent-5.0.1
Background: I am trying to add some code to a custom Oracle OAM plugin. The new code I am adding is to try to access what Oracle calls the "Identity Context". I got some test code example from https://docs.oracle.com/cd/E27559_01/admin.1112/e27239/id_context.htm#AIAAG7296, but had to include two additional jars IdentityContext.jar and jps-api.jar (for AppSecurityContext) and I put those files into the WebLogic Domain/lib directory on my managed servers then bounced the managed servers.
After that, I am still get a ClassNotFoundException but it looks lie a little different than I am used to:
Caused By: java.lang.ClassNotFoundException: oracle.security.jps.runtime.AppSecurityContext not found by SampleAuthPlugin [1]
So I don't understand the error since the two JARs I added have the AppSecurityContext class and the associated inner class classes, so I'm assuming that the "different" exception is telling me something (BTW, SampleAuthPlugin is my custom plugin, which has the code to access the Identity context).
I am kind of guessing that something (maybe the Oracle OAM plugin infrastructure ) is not using the normal WebLogic classpath, and so even though I put the JARs into the DOMAIN/lib, the SampleAuthPlugin is causing something to look for the AppSecurityContext class in a JAR in some different directory??
Thanks,
Jim
EDIT: Here a more extensive stacktrace - it looks like the SampleAuthPlugin IS using it's own (or a different/special) classloader)?
Caused By: java.lang.ClassNotFoundException: oracle.security.jps.runtime.AppSecurityContext not found by SampleAuthPlugin [1]
at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1550)
at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:77)
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1988)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at sample.SampleAuthPlugin$1.run(SampleAuthPlugin.java:606)
at java.security.AccessController.doPrivileged(Native Method)
at sample.SampleAuthPlugin.getIDContext(SampleAuthPlugin.java:605)
at sample.SampleAuthPlugin.ListContext(SampleAuthPlugin.java:584)
at sample.SampleAuthPlugin.process(SampleAuthPlugin.java:406)
At least for my situation, the following worked:
In my plugin JAR, I added the following in the META-INF/MANIFEST.MF:
Bundle-Classpath: .,jps-ee.jar, etc.
Then, I put the jps-ee.jar into the root of the plugin jar.
FYI, I built my plugin jar using Netbeans and for the 2nd step, I put the jps-ee.jar under the in the Netbeans project.
I am facing the below error while deploying the war file in Websphere-8.5.5
Caused by: java.lang.NoClassDefFoundError: org/apache/http/params/SyncBasicHttpParams
at com.couchbase.client.ViewConnection.createConnections(ViewConnection.java:120)
at com.couchbase.client.ViewConnection.<init>(ViewConnection.java:100)
at com.couchbase.client.CouchbaseConnectionFactory.createViewConnection(CouchbaseConnectionFactory.java:184)
at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:247)
at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:179)
at com.infosys.aham.services.utils.cache.CouchbaseCacheManager.instantiateCouchbaseManager(CouchbaseCacheManager.java:145)
at com.infosys.aham.services.utils.cache.CouchbaseCacheManager.<init>(CouchbaseCacheManager.java:55)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:56)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39)
at java.lang.reflect.Constructor.newInstance(Constructor.java:527)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147)
... 87 more
I am using Couchbase-Client -1.1.9 and http-core-4.1.3
The http-core jar is bundled in the war file /web-inf/lib folder.
Also while looking in the websphere EnterprizeApplication-->application -->Manage Module --> moduleName--> ClassLoaderViewer --> LoaderView
The http-core-4.1.3 is there in the table (jar is present in the classpath).
(I also tried with the http-core 4.1.1 jar.)
The application is still not able to find the class present in the given jar.
I am new to websphere, please help me out with this.
Thanks in advance.
The http-core classes are already included in WebSphere. To use different implementation create isolated shared library with conflicting libraries and map it to the application.
I'm developing code that executes a quartz job. When I execute this code on my local tomcat it is working fine, but when I try to do the same code inside a web application I get java.lang.NoClassDefFoundError: org/quartz/DisallowConcurrentExecution. I'm using quartz 2.2.1 and it's already in the class path. Here is the stack trace:
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at org.quartz.impl.JobDetailImpl.isConcurrentExectionDisallowed(JobDetailImpl.java:388)
at org.quartz.simpl.RAMJobStore.acquireNextTriggers(RAMJobStore.java:1428)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272)
Exception in thread "DefaultQuartzScheduler_QuartzSchedulerThread" java.lang.NoClassDefFoundError: org/quartz/DisallowConcurrentExecution
at org.quartz.impl.JobDetailImpl.isConcurrentExectionDisallowed(JobDetailImpl.java:388)
at org.quartz.simpl.RAMJobStore.acquireNextTriggers(RAMJobStore.java:1428)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272)
Caused by: java.lang.ClassNotFoundException: org.quartz.DisallowConcurrentExecution
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
... 3 more
I already read that I should check if I have multiply quartz-all-xxx.jar in my classpath and that I should use quartz-all-xxx.jar except of quartz-ver.jar.
Quartz Documentation:
The Quartz package includes a number of jar files, located in root directory of the distribution. The main Quartz library is named quartz-all-xxx.jar (where xxx is a version number). In order to use any of Quartz's features, this jar must be located on your application's classpath.
I can't find this quartz-all-xxx.jar in the quartz package and I also tried to find it as a single file. Does anyone know a download link or if it maybe got deleted?
Thanks in advance.
For quart version 2.2.1 you don't have quartz-all-xxx.jar. Refer below
Maven artifacts are available in the central Maven repository within a few hours after a new release.
As of version 2.2.0, all of the Quartz libraries are contained in the quartz-xxx.jar file (where xxx is a version number), except for the out-of-the-box jobs, which are in the quartz-jobs-xxx.jar file.
The group id is: org.quartz-scheduler. Substitute the version number you wish to use in the dependency snippets below (e.g., "2.2.0").
http://quartz-scheduler.org/downloads
Also refer pom.xml from maven