Flink Oracle JDBC sink connector not loading the driver - java

I am trying to create Flink JBDC sink to an oracle database. When run locally (from a junit test and minicluster) it works but when deployed in k8s it throws an exception saying it cannot find a suitable Driver. The Classpath is:
Classpath: /flink/lib/flink-cep-scala_2.12-1.13.5-stream1.jar:/flink/lib/flink-connector-jdbc_2.12-1.13.5.jar:/flink/lib/flink-csv-1.13.5-stream1.jar:/flink/lib/flink-json-1.13.5-stream1.jar:/flink/lib/flink-queryable-state-runtime_2.12-1.13.5-stream1.jar:/flink/lib/flink-shaded-netty-tcnative-dynamic-2.0.30.Final-13.0-stream1.jar:/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/flink/lib/flink-table-blink_2.12-1.13.5-stream1.jar:/flink/lib/flink-table_2.12-1.13.5-stream1.jar:/flink/lib/log4j-1.2-api-2.16.0.jar:/flink/lib/log4j-api-2.16.0.jar:/flink/lib/log4j-core-2.16.0.jar:/flink/lib/log4j-slf4j-impl-2.16.0.jar:/flink/lib/ojdbc8-21.5.0.0.jar:/flink/lib/vvp-flink-ha-kubernetes-flink113-1.4-20211013.091138-2.jar:/flink/lib/flink-dist_2.12-1.13.5-stream1.jar:::
I tried multiple things:
Included the driver in the flink/lib directory and the flink-connector-jdbc connector was packaged within the the jar and .withDriverName("oracle.jdbc.OracleDriver") /.withDriverName("oracle.jdbc.driver.OracleDriver")
Included both the driver and the connector into the flink/lib directory and .withDriverName("oracle.jdbc.OracleDriver") / .withDriverName("oracle.jdbc.driver.OracleDriver")
I also tried to change the classloading configuration to classloader.parent-first-patterns.additional: oracle.jdbc.
but nothing seems to be working for me. The exception is:
failure cause: java.io.IOException: unable to open JDBC writer
at org.apache.flink.connector.jdbc.internal.AbstractJdbcOutputFormat.open(AbstractJdbcOutputFormat.java:56)
at org.apache.flink.connector.jdbc.internal.JdbcBatchingOutputFormat.open(JdbcBatchingOutputFormat.java:115)
at org.apache.flink.connector.jdbc.internal.GenericJdbcSinkFunction.open(GenericJdbcSinkFunction.java:49)
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
at org.apache.flink.streaming.api.operators.StreamSink.open(StreamSink.java:46)
at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:442)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:585)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.call(StreamTaskActionExecutor.java:100)
at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:565)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:540)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: No suitable driver found for "jdbc:oracle:thin:#//SOMECONNECTION"
at org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider.getOrEstablishConnection(SimpleJdbcConnectionProvider.java:126)
at org.apache.flink.connector.jdbc.internal.AbstractJdbcOutputFormat.open(AbstractJdbcOutputFormat.java:54)
... 14 more
What am I missing?

There is no support in Flink 1.13 for Oracle via JDBC, that was only added in Flink 1.15

Related

Pool Manager Cannot load specified driver: oracle.jdbc.driver.OracleDriver

I use I-net designer 16 with a Oracle 12.2 Database Source. In report Designer i have no problem to view the report results ('ve put ojdbc8.jar into the driver path of Clear Reports) but, when I try to launch that report on my web-application (deployed on Apache Tomcat) that give me this error:
[Pool Manager] Cannot load specified driver: oracle.jdbc.driver.OracleDriver
; i've also copied the .jar file into the driver path on Clear-reports server.
Anyone have solved this problem?
Thanks

MySQL to PostgreSQL migration: mysql connector

I am trying to migrate from MySQL to PostgreSQL and I have a Java-related problem that I am not able to fix. Full disclosure: I know little or nothing about Java, but the migration uses a Java-based script, so for me it becomes a configuration problem.
Short version of the problem:
The migration tool throws this exception:
java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
mysql-connector-java-5.0.8-bin.jar is already in the "JAVA_HOME\jre\lib\ext" directory, and I don't know how to solve this depencency problem.
Long version of the problem:
I was trying to migrate from MySQL to PostgreSQL. I checked the official postgresql documentation and I chose the free tool from entreprisedb (that can be downloaded here) to start the migration.
From the installation readme, they tell you that the mysql connector is not installed by default, but they also tell you the steps to solve this problem:
To enable MySQL connectivity, download MySQL's freely available JDBC driver from:
http://www.enterprisedb.com/downloads/third-party-jdbc-drivers
Place the mysql-connector-java-5.0.8-bin.jar file in the "JAVA_HOME\jre\lib\ext" directory (in my case: "C:\Program Files\Java\jre1.8.0_60\lib\ext\mysql-connector-java-5.0.8-bin.jar").
After configuring the tool properly and executing the .bat, this is the error I get:
Connecting with source MySQL database server...
MTK-11009: Error Connecting Database "MySQL Server"
DB-null: java.sql.SQLException: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
Stack Trace:
com.edb.MTKException: MTK-11009: Error Connecting Database "MySQL Server"
at com.edb.dbhandler.mysql.MySQLConnection.<init>(MySQLConnection.java:48)
at com.edb.common.MTKFactory.createMTKConnection(MTKFactory.java:250)
at com.edb.MigrationToolkit.createNewSourceConnection(MigrationToolkit.java:5982)
at com.edb.MigrationToolkit.initToolkit(MigrationToolkit.java:3346)
at com.edb.MigrationToolkit.main(MigrationToolkit.java:1700)
Caused by: java.sql.SQLException: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
at com.edb.Utility.processException(Utility.java:327)
at com.edb.dbhandler.mysql.MySQLConnection.<init>(MySQLConnection.java:47)
... 4 more
...which, to my understanding, probably means that mysql-connector-java-5.0.8-bin.jar is not found.
All the links I've found online regarding the error are specific for Eclipse or other IDEs, so I have not yet been able to solve this dependency problem.
SOLUTION
With the help of a friend that masters Java, this is the solution he achieved:
To start looking for the problem, we opened the runMTK.bat. The execution line reads:
cscript //nologo "..\etc\sysconfig\runJavaApplication.vbs" "..\etc\sysconfig\edbmtk-49.config" "-Dprop=..\etc\toolkit.properties -classpath -jar edb-migrationtoolkit.jar %*"
So then we opened this runJavaApplication.vbs, and in order to know the JAVA_EXECUTABLE_PATH that the program was using, we add this line to the script:
Wscript.Echo "JAVA_EXECUTABLE_PATH = " & JAVA_EXECUTABLE_PATH
With that info, we discover that the script is using the Java folder under C:\Program Files (x86), instead of the one under C:\Program Files (where I dropped the mysql jar). So we copy the mysql-connector-java-5.0.8-bin.jar in the \ext folder of the x86, and now the script works.
Word of advice: the script is throwing errors in half of the exported tables, so all the hassle may not be worth it. BUT if anyone is interested in making this migration script work from A to Z (which has been quite a challenge), here are the details:
HOW TO
Free tool (from entreprisedb):
http://www.enterprisedb.com/downloads/postgres-postgresql-downloads
Extract the files from the zip and fun the installer (ppasmeta-9.5.0.5-windows-x64.exe) as administrator.
To enable MySQL connectivity, download MySQL's freely available JDBC driver from:
http://www.enterprisedb.com/downloads/third-party-jdbc-drivers
Place the mysql-connector-java-5.0.8-bin.jar file in the "JAVA_HOME\jre\lib\ext" directory (in my case: "C:\Program Files\Java\jre1.8.0_60\lib\ext\mysql-connector-java-5.0.8-bin.jar").
The Migration Toolkit documentation can be found:
here (online doc): https://www.enterprisedb.com/docs/en/9.4/migrate/toc.html
or here (pdf doc): http://get.enterprisedb.com/docs/Postgres_Plus_Migration_Guide_v9.5.pdf
First: modify C:\Program Files\PostgresPlus\edbmtk\etc\toolkit.properties (Info here):
SRC_DB_URL=jdbc:mysql://SOURCE-HOST-NAME/SOURCE-DB-NAME
SRC_DB_USER=********
SRC_DB_PASSWORD=********
TARGET_DB_URL=jdbc:edb://localhost:5444/DESTINATION-DB-NAME
TARGET_DB_USER=enterprisedb
TARGET_DB_PASSWORD=********
Then: execute C:\Program Files\PostgresPlus\edbmtk\bin\runMTK.bat (Info here).
runMTK.bat -sourcedbtype mysql -targetdbtype enterprisedb -allTables YOUR_DB_SCHEMA
// ...or with a limited subset of tables:
runMTK.bat -sourcedbtype mysql -targetdbtype enterprisedb -tables TABLE1,TABLE2,TABLE3 YOUR_DB_SCHEMA
In order to get this subset of tables from MySQL:
SELECT
GROUP_CONCAT(TABLE_NAME)
FROM
information_schema.tables
WHERE
TABLE_SCHEMA = 'your_db_name'

Which is the sybase driver(version) to be used in Weblogic 12c ? Where should I add the downloaded driver?

I am migrating my application from Weblogic 9 to 12c.
The connection pool - cvSybasepool created for my application is working fine on Web logic 9.
I had used com.sybase.jdbc.SybDriver to connect to Sybase database on Weblogic 9.
After replicating the connection pool(cvSybasepool) on Weblogic 12c, I get the following error:
<Jun 21, 2016 4:40:25 AM EDT> <Error> <Deployer> <BEA-149205> <Failed to initialize the application "cvSybasepool" due to error weblogic.application.ModuleException: weblogic.com
mon.resourcepool.ResourceSystemException: Cannot load driver class com.sybase.jdbc.SybDriver for datasource 'cvSybasepool'.
weblogic.application.ModuleException: weblogic.common.resourcepool.ResourceSystemException: Cannot load driver class com.sybase.jdbc.SybDriver for datasource 'cvSybasepool'.
at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:350)
at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:100)
at weblogic.application.internal.flow.ModuleStateDriver$1.next(ModuleStateDriver.java:175)
at weblogic.application.internal.flow.ModuleStateDriver$1.next(ModuleStateDriver.java:170)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42)
Truncated. see log file for complete stacktrace
Caused By: weblogic.common.resourcepool.ResourceSystemException: Cannot load driver class com.sybase.jdbc.SybDriver for datasource 'cvSybasepool'.
at weblogic.jdbc.common.internal.JDBCUtil.parseException(JDBCUtil.java:322)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.loadDriver(ConnectionEnvFactory.java:82)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.<init>(ConnectionEnvFactory.java:141)
at weblogic.jdbc.common.internal.ConnectionPool.initPooledResourceFactory(ConnectionPool.java:1110)
at weblogic.common.resourcepool.ResourcePoolImpl.start(ResourcePoolImpl.java:226)
Truncated. see log file for complete stacktrace
>
I suspect the Sybase JDBC driver jar is missing on Web logic 12c.
Also, When the Sybase driver provided by Web logic server is used, I get the following error
FILogger.log(232) - java.sql.SQLException: [FMWGEN][Sybase JDBC Driver][Sybase]A USE DATABASE statement is not allowed in a procedure or trigger. at weblogic.jdbc.sybasebase.ddcn.b(Unknown Source) at weblogic.jdbc.sybasebase.ddcn.a(Unknown Source) at weblogic.jdbc.sybasebase.ddcm.b(Unknown Source) at weblogic.jdbc.sybasebase.ddcm.a(Unknown Source)
Which driver needs to be used to connect to Sybase DB on Weblogic
12c? Jconn2.jar/Jconn3.jar?
Which jar needs to be added to Weblogic server?
Where do I need to add the jar and what configuration has be done?
I finally resolved the issue.
I could not use the sybase driver(weblogic.jdbc.sybase.SybaseDriver) bundled with weblogic as it was not complaint with my SQL code
Below are steps I followed.
Downloaded & copied JDBC driver JAR - jConnect.jar to
${WL_HOME}/server/lib/
Updated the setDomainEnv.cmd script in my domain's bin directory,
and prepended the JAR file to the PRE_CLASSPATH environment
variable(If the newly added JDBC driver JAR is required in other
domains, then edit the commEnv.cmd/sh script in WL_HOME/common/bin
and prepend your JAR file to the WEBLOGIC_CLASSPATH environment
variable.)
Restarted the server.
Update or re-create your datasource to use the sybase driver available with weblogic server 12c (weblogic.jdbc.sybase.SybaseDriver)

Connection with Sybase

I am new to Sybase ,I'm developing a web application with sybase and jboss, my app connect with the Sybase database. I have put the jconn4.jar in same directory with war file in deployments directory . My config in standalone.xml like this
<connection-url>jdbc:jtds:sybase://<myserver>:<myport>/<mydb></connection-url>
<driver-class>com.sybase.jdbc4.jdbc.SybDriver</driver-class>
<!--<driver-class>net.sourceforge.jtds.jdbc.Driver</driver-class> -->
<driver>jconn4.jar</driver>
and when i use com.sybase.jdbc4.jdbc.SybDriver got the error message
javax.resource.ResourceException: Wrong driver class [class com.sybase.jdbc4.jdbc.SybDriver] for this connection URL [jdbc:jtds:sybase://<myhost>:<myport>/<mydb>
...
when i use net.sourceforge.jtds.jdbc.Driver i got the message
java.lang.ClassNotFoundException: net.sourceforge.jtds.jdbc.Driver from [Module "deployment.jconn4.jar:main" from Service Module Loader]
I don't know what driver-class suitable for driver class jconn4.jar
You are using the driver url of the open source jTDS driver, but you are trying to use the Sybase driver. Use the right JDBC url for the Sybase driver: Check the documentation included with the Sybase driver for details (I can't be more specific because the Sybase documentation is usually hard to find online without access to the support portal). Or switch to jTDS, in which case you need to deploy it as a module to JBoss and reference the right driver module in the config.

DSE sqoop can't import date SQL Server to Cassandra table

Currently I have downloaded datastax enterprise edition 4.0 and used dse sqoop migrate SQL Server to Cassandra I am getting the following error:
ERROR:
sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.microsoft.sqlserver.jdbc.SQLServerDriver
java.lang.RuntimeException: Could not load db driver class: com.microsoft.sqlserver.jdbc.SQLServerDriver
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:653)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:542)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:565)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:194)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:178)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:262)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1235)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1060)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
I have put the sqljdbc4.jar file in the below paths
/usr/share/dse/sqoop/lib
/usr/share/dse/hadoop/lib
Still I am getting above error.
Have you tried placing the jar in /usr/share/dse/sqoop?

Categories