We Keep Coding

pyspark cannot save data to ignite database: Py4JJavaError java.lang.ClassNotFoundException: org.apache.ignite.IgniteJdbcThinDriver

I am not so familiar with spark. I tried to solve it out but could not after a few days.
My system info is as follows:
versions:
python version: 3.9.13
pyspark version: 3.2.1
py4j version: 0.10.9.3
java version: OpenJDK Runtime Environment Zulu11.56+19-CA (build 11.0.15+10-LTS) OpenJDK 64-Bit Server VM Zulu11.56+19-CA (build 11.0.15+10-LTS, mixed mode)
OS version: macOS Big Sur 11.5.1 Mac M1 mini
Chip: Apple M1
environment variables:
SPARK_HOME: /opt/homebrew/Cellar/apache-spark/3.3.0/libexec
CLASSPATH: /Library/Java/Extensions/ignite-core-2.12.0.jar
JAVA_HOME: /Applications/zulu11.56.19-ca-jdk11.0.15-macosx_aarch64/zulu-11.jdk/Contents/Home
When I used a spark read data frame, where the spark is defined as
spark = SparkSession.builder.config(
"spark.jars", "/Library/Java/Extensions/ignite-core-2.12.0.jar"
).getOrCreate()
and then want to write the read data frame to ignite database
spark_data_frame.write.format("jdbc").option(
"driver", "org.apache.ignite.IgniteJdbcThinDriver"
).option(
"url", f"jdbc:ignite:thin://127.0.0.1:10800/"
).option(
"dbtable", "user"
).mode(
"append"
).save()
then it raises the following error:
virtual_env/lib/python3.9/site-packages/py4j/protocol.py, line326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o797.save.
: java.lang.ClassNotFoundException: org.apache.ignite.IgniteJdbcThinDriver
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at org.apache.spark.sql.execution.datasources.jdbc.DriverRegistry$.register(DriverRegistry.scala:46)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$1(JDBCOptions.scala:101)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$1$adapted(JDBCOptions.scala:101)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:101)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcOptionsInWrite.<init>(JDBCOptions.scala:218)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcOptionsInWrite.<init>(JDBCOptions.scala:222)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:46)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:110)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:110)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:106)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:457)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:106)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:93)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:91)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:128)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:848)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:382)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:355)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:247)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
When the call the last save() to save data to ignite database, it raises the problem. I am wondering that the spark already defined the spark.jars, which has the ignite jar file that includes the org.apache.ignite.IgniteJdbcThinDriver. Note that ignite needs the Java 8 or Java 11 and the pyspark can support Java 11. Therefore, the java is 11 and it is the ARM version. Not sure whether it is java problem or arm/intel problem or spark setting problem.
Thanks.

Why does Groovy compiler reject the dependency JSR308 Type Annotations Compiler?

I have been struggling to understand why groovyc does not want to compile my code for awhile and I finally narrowed it down to it disliking the dependency JSR308 Type Annotations Compiler, which uses as one of its package names java.lang.annotations.
When groovyc tries to compile my code with this jar in its classpath, the following is printed:
groovyc -cp jsr308-compiler-1.7.5.jar EmptyClass.groovy
>>> a serious error occurred: null
>>> stacktrace:
java.lang.ExceptionInInitializerError
at org.codehaus.groovy.classgen.Verifier.<clinit>(Verifier.java:137)
at org.codehaus.groovy.control.CompilationUnit.<init>(CompilationUnit.java:171)
at org.codehaus.groovy.tools.javac.JavaAwareCompilationUnit.<init>(JavaAwareCompilationUnit.java:66)
at org.codehaus.groovy.tools.javac.JavaAwareCompilationUnit.<init>(JavaAwareCompilationUnit.java:57)
at org.codehaus.groovy.tools.FileSystemCompiler.<init>(FileSystemCompiler.java:64)
at org.codehaus.groovy.tools.FileSystemCompiler.doCompilation(FileSystemCompiler.java:224)
at org.codehaus.groovy.tools.FileSystemCompiler.commandLineCompile(FileSystemCompiler.java:163)
at org.codehaus.groovy.tools.FileSystemCompiler.commandLineCompileWithErrorHandling(FileSystemCompiler.java:203)
at org.codehaus.groovy.tools.FileSystemCompiler.main(FileSystemCompiler.java:187)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:110)
at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:128)
Caused by: java.lang.SecurityException: Prohibited package name: java.lang.annotation
at java.lang.ClassLoader.preDefineClass(ClassLoader.java:662)
at java.lang.ClassLoader.defineClass(ClassLoader.java:761)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at org.codehaus.groovy.tools.RootLoader.oldFindClass(RootLoader.java:174)
at org.codehaus.groovy.tools.RootLoader.loadClass(RootLoader.java:146)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.codehaus.groovy.ast.ClassHelper.<clinit>(ClassHelper.java:129)
... 15 more
Running javac -cp jsr308-compiler-1.7.5.jar EmptyClass.java successfully compiles the dummy class.
java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1ubuntu1~16.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
groovy -version
Groovy Version: 3.0.0-beta-2 JVM: 1.8.0_222 Vendor: Private Build OS: Linux
From what I have read, the Type Annotations Project was integrated into JDK 8 back in 2014, so I have no idea why groovyc would not recognize this dependency's right to use java.lang.annotation as a package name.
[Update] As requested, here is the code for both EmptyClass.groovy and EmptyClass.java:
public class EmptyClass {
}

I have retried with multiple versions of groovy. It did not work with a 2.5 version, but with 2.0.0 my command actually technically compiles. Unfortunately, there are some other compatibility problems beyond this basic example and I would not recommend that anyone try to make groovy work with JSR-308 if it is not necessary.

Java 9 and tomcat 7.0.73

I am running my java application with java 9 and tomcat 7.0.73 and I am getting following in my catalina.out file.
Is there any work around to avoid it?
Can you please help me understand the impact of this error. My application is coming up properly.
Also why GC daemon thread is created during tomcat startup, is it configurable?
Here is error in catalina.out file.
Mar 10, 2018 2:09:39 AM org.apache.catalina.core.JreMemoryLeakPreventionListener lifecycleEvent
SEVERE: Failed to trigger creation of the GC Daemon thread during Tomcat start to prevent possible memory leaks. This is expected on non-Sun JVMs.
java.lang.ClassNotFoundException: sun.misc.GC
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:466)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:563)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:292)
at org.apache.catalina.core.JreMemoryLeakPreventionListener.lifecycleEvent(JreMemoryLeakPreventionListener.java:324)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:388)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:101)
at org.apache.catalina.startup.Catalina.load(Catalina.java:642)
at org.apache.catalina.startup.Catalina.load(Catalina.java:667)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:253)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:427)
I am using official oracle jdk
/usr/local/java/jdk9/bin/java --version
java 9.0.4
Java(TM) SE Runtime Environment (build 9.0.4+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.4+11, mixed mode)

For me the solution was: The JRE I used for the tomcat wasn't the right one.
Double click on the server
Runtime Environment
Choose the right JRE

java.lang.SecurityException when executing IBM Websphere runmqckm command

I'm trying to setup ssl for IBM MQ server and when I executed the command
runmqckm -keydb -create -db keydb.kdb -pw password -type cms -expire 1500 -stash
as one of the steps, it fails with the following exception.
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.J9VMInternals.ensureError(J9VMInternals.java:186)
at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:175)
at javax.crypto.Mac.getInstance(Unknown Source)
at com.ibm.security.cmskeystore.FileHeaderHashGeneratorFactory$FileHeaderHashGeneratorV4Impl.generateHash(FileHeaderHashGeneratorFactory.java:145)
at com.ibm.security.cmskeystore.CMSKeyStoreSpi.engineLoad(CMSKeyStoreSpi.java:1206)
at java.security.KeyStore.load(KeyStore.java:1226)
at com.ibm.gsk.ikeyman.keystore.KeyStoreProxyCreatorFactory$FileKeyStoreProxy.performLoad(KeyStoreProxyCreatorFactory.java:304)
at com.ibm.gsk.ikeyman.keystore.KeyStoreProxyCreatorFactory$CMSKeyStoreProxy.performLoad(KeyStoreProxyCreatorFactory.java:437)
at com.ibm.gsk.ikeyman.keystore.KeyStoreProxyCreatorFactory$FileKeyStoreProxy.load(KeyStoreProxyCreatorFactory.java:253)
at com.ibm.gsk.ikeyman.keystore.KeyStoreProxyCreatorFactory$AbstractKeyStoreProxy.getKeyStore(KeyStoreProxyCreatorFactory.java:193)
at com.ibm.gsk.ikeyman.keystore.KeyStoreItemFactory.getCMSKeyStoreItem(KeyStoreItemFactory.java:323)
at com.ibm.gsk.ikeyman.keystore.KeyStoreItemFactory.newCMSKeyStoreItem(KeyStoreItemFactory.java:294)
at com.ibm.gsk.ikeyman.keystore.KeyStoreItemFactory.createKeyStore(KeyStoreItemFactory.java:115)
at com.ibm.gsk.ikeyman.command.CommandFactory$CreateDbCommand.run(CommandFactory.java:402)
at com.ibm.gsk.ikeyman.command.Command.invoke(Command.java:203)
at com.ibm.gsk.ikeyman.ikeycmd.main(ikeycmd.java:62)
Caused by: java.lang.SecurityException: Cannot set up certs for trusted CAs
at javax.crypto.b.<clinit>(Unknown Source)
... 14 more
Caused by: java.lang.SecurityException: Jurisdiction policy files are not signed by trusted signers!
at javax.crypto.b.a(Unknown Source)
at javax.crypto.b.c(Unknown Source)
at javax.crypto.b.access$600(Unknown Source)
at javax.crypto.b$0.run(Unknown Source)
at java.security.AccessController.doPrivileged(AccessController.java:333)
... 15 more
My Websphere and Java version information are as below
Websphere
Name: WebSphere MQ
Version: 8.0.0.2
Level: p800-002-150519.TRIAL
BuildType: IKAP - (Production)
Platform: WebSphere MQ for Linux (x86-64 platform)
Mode: 64-bit
O/S: Linux 2.6.32-504.el6.x86_64
InstName: Installation1
InstDesc:
Primary: Yes
InstPath: /opt/mqm
DataPath: /var/mqm
MaxCmdLevel: 801
LicenseType: Production
Java
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
Can anyone please help me to resolve this issue?

I figured the solution myself and posting it below for the future reference of other.
Download IBM Unrestricted SDK JCE policy files from https://www-01.ibm.com/marketing/iwm/iwm/web/preLogin.do?source=jcesdk.
Copy both local_policy.jar and US_export_policy.jar files to <mq_installation_path>/java/jre64/jre/lib/security folder.
Give ownership and executable permission of both files to mqm user.

native library for attach API not available in this JRE

I am attempting to build a new project with gradle, junit, and jmock. When I run my build, I get this stack trace:
com.heavyweightsoftware.daybook.ws.TypeCodeWsTest STANDARD_ERROR
java.lang.IllegalStateException: Native library for Attach API not available
in this JRE
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.ru
nTestClass(JUnitTestClassExecuter.java:80)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.ex
ecute(JUnitTestClassExecuter.java:49)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.p
rocessTestClass(JUnitTestClassProcessor.java:64)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.process
TestClass(SuiteTestClassProcessor.java:50)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionD
ispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionD
ispatch.java:24)
at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(Con
textClassLoaderDispatch.java:32)
at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocat
ionHandler.invoke(ProxyDispatchAdapter.java:93)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestCl
ass(TestWorker.java:106)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionD
ispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionD
ispatch.java:24)
at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(Messa
geHub.java:360)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.
onExecute(ExecutorPolicy.java:54)
at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableE
xecutorImpl.java:40)
Caused by: java.lang.UnsatisfiedLinkError: no attach in java.library.path
... 14 more
Gradle Test Executor 1 finished executing tests.
com.heavyweightsoftware.daybook.ws.TypeCodeWsTest > testGetTypeCodes FAILED
java.lang.ExceptionInInitializerError
at com.heavyweightsoftware.daybook.ws.TypeCodeWsTest.testGetTypeCodes(Ty
peCodeWsTest.java:40)
Caused by:
java.lang.IllegalStateException: Native library for Attach API not avail
able in this JRE
at mockit.internal.startup.AgentLoader.getVirtualMachineImplementati
onFromEmbeddedOnes(AgentLoader.java:78)
at mockit.internal.startup.AgentLoader.loadAgent(AgentLoader.java:46
)
at mockit.internal.startup.Startup.verifyInitialization(Startup.java
:172)
at mockit.Invocations.<clinit>(Invocations.java:26)
... 1 more
Caused by:
java.lang.NoClassDefFoundError: Could not initialize class sun.tools
.attach.WindowsVirtualMachine
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unk
nown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at mockit.internal.startup.AgentLoader.getVirtualMachineImplemen
tationFromEmbeddedOnes(AgentLoader.java:70)
... 4 more
So I looked and found these questions: JMockit ERROR - Native library for Attach API not available in this JRE and jmockit: Native library for Attach API not available in this JRE error
And I've made the changes in eclipse, but this error happens even when running outside of eclipse and so it's not the eclipse settings.
Here's the output of java -version from the command prompt:
C:\Users\thom\git\daybook\WebServices>java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
This is part of gradle build --info:
Executing task ':test' (up-to-date check took 0.193 secs) due to:
Value of input property 'candidateClassFiles' has changed for task ':test'
Starting process 'Gradle Test Executor 1'. Working directory: C:\Users\thom\git\
daybook\WebServices Command: C:\java\bin\java.exe -Djava.security.manager=jarjar
.org.gradle.process.internal.child.BootstrapSecurityManager -Dfile.encoding=wind
ows-1252 -Duser.country=US -Duser.language=en -Duser.variant -ea -cp C:\Users\th
om\.gradle\caches\2.4\workerMain\gradle-worker.jar jarjar.org.gradle.process.int
ernal.launcher.GradleWorkerMain 'Gradle Test Executor 1'
Successfully started process 'Gradle Test Executor 1'
Gradle Test Executor 1 started executing tests.
Which tells me it's running java from c:\java\bin which is the right install. I looked and this directer has javac in it, which I think confirms that it's a JDK and not a JRE.
Any input, oh wise above wise?

I added c:\java\jre\bin to my path and the build started working again.