We Keep Coding

How to solve java.lang.ClassNotFoundException: org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer

I am using tinkerpop + Janus Graph + Spark
build.gradle
compile group: 'org.apache.tinkerpop', name: 'spark-gremlin', version: '3.1.0-incubating'
below is some critical configuration that we have
spark.serializer: org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer
In the logs corresponding long entry which refer the jar containing the above class is loaded
{"#timestamp":"2020-02-18T07:24:21.720+00:00","#version":1,"message":"Added JAR /opt/data/janusgraph/applib2/spark-gremlin-827a65ae26.jar at spark://gdp-identity-stage.target.com:38876/jars/spark-gremlin-827a65ae26.jar with timestamp 1582010661720","logger_name":"o.a.s.SparkContext","thread_name":"SparkGraphComputer-boss","level":"INFO","level_value":20000}
but my spark job submitted by SparkGraphComputer is failed, when we see executor logs, we saw
Caused by: java.lang.ClassNotFoundException: org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer
Why this exception is coming even though the corresponding jar is loaded?
Anyone, please suggest on this.
As I mention seeing this exception in spark executor when I opened one of the worker logs below complete exception
Spark Executor Command: "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el7_6.x86_64/bin/java" "-cp" "/opt/spark/spark-2.4.0/conf/:/opt/spark/spark-2.4.0/jars/*:/opt/hadoop/hadoop-3_1_1/etc/hadoop/" "-Xmx56320M" "-Dspark.driver.port=43137" "-XX:+UseG1GC" "-XX:+PrintGCDetails" "-XX:+PrintGCTimeStamps" "-Xloggc:/opt/spark/gc.log" "-Dtinkerpop.gremlin.io.kryoShimService=org.apache.tinkerpop.gremlin.hadoop.structure.io.HadoopPoolShimService" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler#gdp-identity-stage.target.com:43137" "--executor-id" "43392" "--hostname" "192.168.192.10" "--cores" "6" "--app-id" "app-20200220094335-0001" "--worker-url" "spark://Worker#192.168.192.10:36845"
========================================
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1713)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:64)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:281)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
at org.apache.spark.SparkEnv$.instantiateClass$1(SparkEnv.scala:259)
at org.apache.spark.SparkEnv$.instantiateClassFromConf$1(SparkEnv.scala:280)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:283)
at org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:200)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:221)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:65)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
... 4 more
when I am setting the spark. jars property on graph, am passing this jar location also
Jar which we created from the application is of fat jar type means it contains the actual code and all the required dependencies also, please see below screenshots .

If you look at the logs, you see this
java" "-cp" "/opt/spark/spark-2.4.0/conf/:/opt/spark/spark-2.4.0/jars/*:/opt/hadoop/hadoop-3_1_1/etc/hadoop/"
Unless you have the gremlin JARs in your /opt/spark/spark-2.4.0/jars/* folder on each Spark worker, then the class you're using doesn't exist.
The recommended way to include it for your specific application would be the Gradle Shadow plugin rather than --packages or spark.jars

Add jdbc postgre driver to jar

I'm building a java server program that connects to a psql database with JDBC drivers. I need to compile the program in a .jar file, but when I try to do so and run it, I get this exception. (the program has to run on a linux machine)
java.lang.ClassNotFoundException: org.postgresql.Driver
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at Server.connectToDatabase(Server.java:53)
at Server.(Server.java:33)
at Main.main(Main.java:10)
Since the program works fine on IntelliJ editor, I assume that the problem is that there isn't the jdbc driver in the jar file. How can I add it?

You need add PostgreDriver to your classpath and use command like following.
$Java -classpath postgre-jar+additional-jars-if-any Your-Main-class

STS Database Connection Ping Failed

I am trying to connect to a MySQL database so I will reverse engineer the tables into classes on a Spring Boot Project. When I enter the properties and , after adding the jar file, and trying to test the connection it says Ping failed!
Selecting Database perspective through STS
Trying to create you database connection
Connection profile with properties
The jar
Testing the connection
The exception
java.lang.ClassNotFoundException: Generic JDBC
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.eclipse.datatools.connectivity.drivers.jdbc.JDBCConnection.createConnection(JDBCConnection.java:327)
at org.eclipse.datatools.connectivity.DriverConnectionBase.internalCreateConnection(DriverConnectionBase.java:105)
at org.eclipse.datatools.connectivity.DriverConnectionBase.open(DriverConnectionBase.java:54)
at org.eclipse.datatools.connectivity.drivers.jdbc.JDBCConnection.open(JDBCConnection.java:96)
at org.eclipse.datatools.connectivity.drivers.jdbc.JDBCConnectionFactory.createConnection(JDBCConnectionFactory.java:53)
at org.eclipse.datatools.connectivity.internal.ConnectionFactoryProvider.createConnection(ConnectionFactoryProvider.java:83)
at org.eclipse.datatools.connectivity.internal.ConnectionProfile.createConnection(ConnectionProfile.java:359)
at org.eclipse.datatools.connectivity.ui.PingJob.createTestConnection(PingJob.java:76)
at org.eclipse.datatools.connectivity.ui.PingJob.run(PingJob.java:59)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)

Driver name should be com.mysql.jdbc.Driver

Where do I put the JDBC driver?

I'm working on IntelliJ with Tomcat and postgresql but I have this problem when I run my application:
java.lang.ClassNotFoundException: org.postgres.Driver
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1285)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264) and so on..
I have already put the JAR file in tomcat/lib but nothing changed.
Thanks for helping me.

You misspelled/shortened the qualified name of the PostgreSQL JDBC driver. The correct name is
org.postgresql.Driver

Make sure JDBC jar is added to the module dependencies: http://www.jetbrains.com/idea/webhelp/configuring-module-dependencies-and-libraries.html.

How to connect Java with Titan with embedded Cassandra

I installed titan-cassandra-0.4.4, and I can run cassandra and gremlin perfectly (The local node).
I want to test an algorithm in Java, so I need to connect Java to titan.
My pom is:
<dependency>
<groupId>com.thinkaurelius.titan</groupId>
<artifactId>titan-core</artifactId>
<version>0.4.4</version>
</dependency>
My codes are:
BaseConfiguration baseConfiguration = new BaseConfiguration();
baseConfiguration.setProperty("storage.backend", "cassandra");
baseConfiguration.setProperty("storage.hostname", "127.0.0.1");
baseConfiguration.setProperty("storage.tablename","test");
TitanGraph titanGraph = TitanFactory.open(baseConfiguration);
In the last line, the error is:
Exception in thread "main" java.lang.NoClassDefFoundError: com/netflix/astyanax/connectionpool/exceptions/ConnectionException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.thinkaurelius.titan.diskstorage.Backend.getStorageManager(Backend.java:202)
at com.thinkaurelius.titan.diskstorage.Backend.<init>(Backend.java:97)
at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:398)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.<init>(StandardTitanGraph.java:78)
at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:60)
at Connecting.main(Connecting.java:20)
Caused by: java.lang.ClassNotFoundException: com.netflix.astyanax.connectionpool.exceptions.ConnectionException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 8 more
I first start the cassandra, then titan gremlin.sh, and run the java code at last. I do not know if I was missing something?
Thanks in advance!

You should include a dependency for titan-casssandra.
<dependency>
<groupId>com.thinkaurelius.titan</groupId>
<artifactId>titan-casssandra</artifactId>
<version>0.4.4</version>
</dependency>
Also note if you're just getting started with Titan, version 0.4.4 is far out of date. Titan 1.0 was released in September 2015. You can find more information at http://titandb.io