This is a weird error. After adding the selenium dependencies to the pom of my maven project and upload it to a lambda, it says it is unable to unzip the file. However after removing the dependencies, the lambda is able to unzip the file just fine (however it comes up with a class not found afterwards). I have tried removing the dependencies one by one but each one triggers the error.
Any ideas on how to solve this?
Class not found error
org/openqa/selenium/WebDriver: java.lang.NoClassDefFoundError
java.lang.NoClassDefFoundError: org/openqa/selenium/WebDriver
lambda cannot zip error
Calling the invoke API action failed with this message: Lambda was not able to unzip the file
The dependencies causing the issue
<dependency>
<groupId>org.seleniumhq.webdriver</groupId>
<artifactId>webdriver-common</artifactId>
<version>0.9.7376</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-chrome-driver</artifactId>
<version>3.141.59</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.141.59</version>
</dependency>
updated dependancies (for Vishal)
<dependency>
<groupId>org.seleniumhq.webdriver</groupId>
<artifactId>webdriver-common</artifactId>
<version>0.9.7376</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-chrome-driver</artifactId>
<version>3.141.59</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.141.59</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-api</artifactId>
<version>2.0rc2</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-remote-driver</artifactId>
<version>3.141.59</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-support</artifactId>
<version>3.141.59</version>
</dependency>
Configuration
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
<forceJavacCompilerUse>true</forceJavacCompilerUse>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
The shade plugin combines all dependencies with the developed code and plops them in one Uber JAR. The downside is that it can overwrite resource files, and doesn't play well with signed jars (in my experience at least).
I would recommend moving away from the shade plugin if at all possible.
That said, if you have to use it - you're issue may be with the combining the jar resources. There are many transformers that you can use to resolve this, and you'll need to investigate which one is really needed. I would start with something like this
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<configuration>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>${executable.classifier}</shadedClassifierName>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>fully.qualified.ClassName</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
You can find more tranformers on the Apache plugin here
The alternative that I would suggest is Spring Boot, which uses the Jar-in-Jar structure with a custom ClassLoader to load classes from the internal jar(s).
This is the easier method due to not having to re-write files as the Shade plugin approach and it handles the dependencies a little better.
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>1.3.6.RELEASE</version>
<configuration>
<classifier>${executable.classifier}</classifier>
<layout>ZIP</layout>
<mainClass>fully.qualified.ClassName</mainClass>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
Seriously, look at the simpler configuration!
NOTE: Most of this came from my own notes - version numbers may be a little old...
Try to tell your dependencies to output zip, maybe jar is messing up with things
Add this to maven-assembly-plugin configuration:
<formats>
<format>zip</format>
</formats>
For example:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
...
<configuration>
...
<formats>
<format>zip</format>
</formats>
</configuration>
</plugin>
Same is suggested here
I figured it out. The java selenium seemed to cause the major issue. Downgrading to 3.10 fixed the issue although I have no idea why.
Related
I've been trying to run my Maven Kotlin Ktor project with command line. The project contains the MySQL configuration too. Every time I tried to hit the following command, I got the following error.
Here is my command.
kotlin -cp mysql-connector-java-8.0.30.jar:dev_meet_dev_api.jar MainClassKt
The dev_meet_dev_api.jar file is my project jar and mysql-connector-java-8.0.30 is for MySQL connection with the database. My project is dependent on MySQL connector that's why I added the external dependency for MySQL connector.
As soon I hit the above command on terminal I've got the following error.
Caused by: java.lang.ClassNotFoundException: java.sql.Driver
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
Edit 1: Added the maven file.
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>spartons.com.devMeetdevApi</groupId>
<artifactId>KtorTesting</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<kotlin.version>1.7.10</kotlin.version>
<ktor.version>2.1.1</ktor.version>
<junit.version>4.12</junit.version>
<serialization.version>1.4.0</serialization.version>
<coroutines.version>1.6.4</coroutines.version>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<kotlin.compiler.incremental>true</kotlin.compiler.incremental>
<kotlin.compiler.jvmTarget>11</kotlin.compiler.jvmTarget>
<kotlin.code.style>official</kotlin.code.style>
<kotlin.compiler.incremental>true</kotlin.compiler.incremental>
</properties>
<dependencies>
<!-- Kotlin language dependencies -->
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib</artifactId>
<version>${kotlin.version}</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-reflect</artifactId>
<version>${kotlin.version}</version>
<scope>runtime</scope>
</dependency>
<!-- Kotlin's coroutines dependency -->
<dependency>
<groupId>org.jetbrains.kotlinx</groupId>
<artifactId>kotlinx-coroutines-core</artifactId>
<version>${coroutines.version}</version>
</dependency>
<!-- Ktor dependencies -->
<dependency>
<groupId>io.ktor</groupId>
<artifactId>ktor-server-core-jvm</artifactId>
<version>${ktor.version}</version>
</dependency>
<dependency>
<groupId>io.ktor</groupId>
<artifactId>ktor-server-jetty-jvm</artifactId>
<version>${ktor.version}</version>
</dependency>
<dependency>
<groupId>io.ktor</groupId>
<artifactId>ktor-server-content-negotiation-jvm</artifactId>
<version>${ktor.version}</version>
</dependency>
<dependency>
<groupId>io.ktor</groupId>
<artifactId>ktor-serialization-kotlinx-json-jvm</artifactId>
<version>${ktor.version}</version>
</dependency>
<dependency>
<groupId>io.ktor</groupId>
<artifactId>ktor-server-status-pages-jvm</artifactId>
<version>${ktor.version}</version>
</dependency>
<dependency>
<groupId>io.ktor</groupId>
<artifactId>ktor-server-cors-jvm</artifactId>
<version>${ktor.version}</version>
</dependency>
<!-- Kotlin serialization version -->
<dependency>
<groupId>org.jetbrains.kotlinx</groupId>
<artifactId>kotlinx-serialization-json</artifactId>
<version>${serialization.version}</version>
</dependency>
<!-- MySQL connector dependency -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.30</version>
</dependency>
</dependencies>
<build>
<resources>
<resource>
<directory>src/main/resources</directory>
</resource>
</resources>
<finalName>dev_meet_dev_api</finalName>
<plugins>
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<version>${kotlin.version}</version>
<executions>
<execution>
<id>compile</id>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
<configuration>
<compilerPlugins>
<plugin>kotlinx-serialization</plugin>
</compilerPlugins>
<args>
<arg>-opt-in=kotlin.RequiresOptIn</arg>
</args>
</configuration>
<dependencies>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-serialization</artifactId>
<version>${kotlin.version}</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.10.1</version>
<executions>
<execution>
<id>compile</id>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>default-compile</id>
<phase>none</phase>
</execution>
<execution>
<id>default-testCompile</id>
<phase>none</phase>
</execution>
</executions>
<configuration>
<release>11</release>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.3.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
<configuration>
<artifactSet>
<excludes>
<exclude>mysql:mysql-connector-java:jar:</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</plugin>
</plugins>
</build>
</project>
Edit 2: Added the java-11 modules
P.S I'm using Java 11. Projects works fine if I try to run with the IntelliJ.
You can do this directly in Maven! In your pom.xml add the driver as a dependency.
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.30</version>
</dependency>
Then you shade the dependency into the jar
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<includes>
<include>mysql:mysql-connector-java</include>
</includes>
</artifactSet>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
Then directly execute the jar. If this does not help may I know which Java version you are using?
Kotlin applications need the Kotlin runtime to run (mainly kotlin-stdlib.jar). It can be included
in the application jar or not.
If the runtime is not included, you must add it to the classpath:
java -cp /path/to/kotlin-stdlib.jar:app.jar MainClassKt
You can also use the kotlin script, which automatically adds it:
kotlin -cp app.jar MainClassKt
In your case, the runtime is included thanks to the maven-shade-plugin, so you can run the app like this:
java -cp mysql-connector-java-8.0.30.jar:dev_meet_dev_api.jar MainClassKt
Possibility 1
It is possible that some dependency that you manually added has a different version of java.sql.Driver.
It is overriding the java.sql.Driver dependency version that your code flow needs.
To debug if this is actually the cause of your problem, you can use IntelliJ maven dependency plugin.
It can show if there is any conflict in dependency causing the needed class to not load.
https://www.jetbrains.com/help/idea/work-with-maven-dependencies.html#maven_dependency_diagram
Also, please note that in my experience, If multiple dependencies are loading different versions of same class, then you may get different results in different setup, since order of class loading may differ. This can result in your code working in IntellijIdea, but not when running from terminal.
Possibility 2
Check if the jdbc driver version is compatible with java and database versions that you are using. Try upgrading or downgrading the jdbc version to see if that resolves issue.
Debug approach 1: if above ways don't fix the issue
Since you say that your code works in Intellij, start your application in debug mode and put a debug point in java.sql.Driver.
See which the library which contains this class and the dependency which contains this library.
That can help you debug the cause of problem.
Debug approach 2: if above ways don't fix the issue
Use remote debugging.
Start your jar application in console with remote debug enabled.
Refer this on steps to do it.
https://www.jetbrains.com/help/idea/tutorial-remote-debug.html#ecd9fef1
Using this, you can find the code flow and the cause of problem for program that runs in console.
I hope that this helps you.
I have a maven project and I would like to produce the jar file with all dependencies.
I use package shade plugin and the command mvn package to produce the jar file.
However, the produced jar file does not consider any of the dependencies that is pom.xml. The produced jar file keep give me exception:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Level
Here is the content of pom.xml:
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>myProject-core</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>myProject-parser</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>info.picocli</groupId>
<artifactId>picocli</artifactId>
<version>${picoli.version}</version>
</dependency>
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${shade.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/log4j/log4j -->
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.16</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.13.1</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.13.1</version>
</dependency>
</dependencies>
<profiles>
<profile>
<id>client</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${shade.version}</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>myProject-client-${project.version}</finalName>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>myProject.package.main</mainClass>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
</transformers>
<filters>
<!-- This is necessary to avoid a java.lang.SecurityException -->
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<!-- Jar file entry point -->
<mainClass>myProject.package.main</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>
Where is my mistake?
It is even does not consider <mainClass>myProject.package.main</mainClass> for the main class when I run the jar I have to specify the class name:
java -cp myApp.jar myProject.package.main
I looked over most of the questions and nothing solve my issue.
First of all, there are two plugins that you can use to create fat jars: maven-shade-plugin and maven-assembly-plugin.
The main difference between them is that maven-assembly just builds a jar containing all the dependency, while maven-shade also relocate these dependencies internally so that, if another project is depending on your jar, you won't risk having conflicts on dependencies.
If you don't expect yo use this artifact as a dependency of other projects, you can use maven-assembly-plugin for it. Here is an example of how to use it:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.1.1</version>
<configuration>
<finalName>${project.artifactId}</finalName>
<archive>
<manifest>
<mainClass>myProject.package.main</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<appendAssemblyId>false</appendAssemblyId>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
If you really need to shade your dependencies, try to modify the manifest config of the plugin with this, at least it should solve the problem that you-re having with identifying the main class:
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Main-Class>myProject.package.main</Main-Class>
<X-Compile-Source-JDK>${maven.compiler.source}</X-Compile-Source-JDK>
<X-Compile-Target-JDK>${maven.compiler.target}</X-Compile-Target-JDK>
</manifestEntries>
</transformer>
There are no groupId in the dependencies with org.apache.log4j, you have: org.apache.logging.log4j but its not the same as: org.apache.log4j
I trying to read the a file using :
final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSet<String> line = env.readTextFile("file:///pathtofile/myfile.txt");
I get following error:
java.lang.NoSuchMethodError: org.apache.flink.api.common.io.DelimitedInputFormat: method <init>(Lorg/apache/flink/core/fs/Path;)V not found
I'm using flink version 1.3.2, java version "1.8.0_91"
There is a conflict with dependencies. Apache Flink loads many classes by default into its classpath.
Please read this article https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/debugging_classloading.html the last section
Resolving Dependency Conflicts with Flink using the maven-shade-plugin
Apache Flink loads many classes by default into its classpath. If a user uses a different version of a library that Flink is using, often
IllegalAccessExceptions
or
NoSuchMethodError
are the result.
So, I suggest to play with your pom.xml and use maven-shade-plugin and add correct relocation, as we have in example
<relocation>
<pattern>org.codehaus.plexus.util</pattern>
<shadedPattern>org.shaded.plexus.util</shadedPattern>
<excludes>
<exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
<exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
</excludes>
</relocation>
Are you getting this error in IntelliJ or Dashboard, if you are getting this error in IntelliJ then make sure you use the same Flink version in your pom.xml and also add dependency shading in the build like this
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<compilerId>jdt</compilerId>
</configuration>
<dependencies>
<dependency>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-compiler-jdt</artifactId>
<version>0.21.0</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<relocations>
<relocation>
<pattern>org.codehaus.plexus.util</pattern>
<shadedPattern>org.shaded.plexus.util</shadedPattern>
<excludes>
<exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
<exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
</excludes>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>>
make sure to run maven clean install in the terminal after you make changes . On the other hand, If you are having this issue only in Dashboard not in intelliJ , then have a look here
I faced the similar issue, for me the problem was flink minor version mismatch. My local flink cluster was running flink-1.8.0 and my code expected the version to be flink-1.8.3. Switching to newer version solved this issue.
You need to check your build path, make sure the libs are there and import them properly
One possible cause for error "java.lang.NoSuchMethodError" is when we use different version of flink then what we have installed on our system. For me, I have Flink 1.3.2 and the version I was using was 1.1.1 . So I updated my pom file to have same version.
For the ones who're stuggling with the above issue, while using Flink 1.3.2, here's the entire pom which I was able to successfully build:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4.1</version>
<configuration>
<!-- get all project dependencies -->
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<!-- MainClass in mainfest make a executable jar -->
<archive>
<manifest>
<mainClass>FlinktoLambda</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<!-- bind to the packaging phase -->
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>FlinktoLambda</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<!--added newly-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<compilerId>jdt</compilerId>
</configuration>
<dependencies>
<dependency>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-compiler-jdt</artifactId>
<version>0.21.0</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>FlinktoLambda</mainClass>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
</transformers>
<relocations>
<relocation>
<pattern>org.codehaus.plexus.util</pattern>
<shadedPattern>org.shaded.plexus.util</shadedPattern>
<excludes>
<exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
<exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
</excludes>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<name>my-app</name>
<url>http://maven.apache.org</url>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>FlinktoLambda</groupId>
<artifactId>my-app</artifactId>
<exclusions>
<exclusion>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-remote_2.10</artifactId>
</exclusion>
<exclusion>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-actor_2.10</artifactId>
</exclusion>
<exclusion>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-slf4j_2.10</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table_2.10</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.10</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.10</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-scala_2.10</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala_2.10</artifactId>
<version>1.3.2</version>
</dependency>
</dependencies>
Please change your main class accordingly in the shade plugin.
Another source of this error could be a mismatch of scala version between Flink and your application code (or your application dependencies that uses scala).
For instance, in my case, I was using Flink 1.7.1 and I had to update my scala dependencies from 2.11 to 2.12; I updated the artifcatId of the concerned dependencies as follows: from flink-scala_2.11 to flink-scala_2.12, flink-table_2.11 to flink-table_2.12, etc.
See here for more info.
I have created executable Jar using Maven dependencies for connecting multiple database at runtime by passing the parameters.
I added dependencies as below:
<dependency>
<groupId>com.oracle.jdbc</groupId>
<artifactId>ojdbc7</artifactId>
<version>12.1.0.2</version>
</dependency>
<dependency>
<groupId>com.microsoft.sqlserver</groupId>
<artifactId>mssql-jdbc</artifactId>
<version>6.1.7.jre8-preview</version>
</dependency>
In my case, when I ran executable jar it connected successfully to Oracle database but gives No suitable driver not found for SQL server database.
When I changed the order of dependency shown above to SQL server first and Oracle second and built new executable jar and ran it, I could able to connect to SQL server database successfully but this time it gives error No suitable driver not found for Oracle database.
When I compared both the extracted executable jars, it has exactly same structure, numbers of files and size of files and didn't find any single difference. Hence not able to found what could be the problem while building executable jar so that a single jar would work for both databases.
Any suggestion would be really help me to resolve this strange behavior...
Thanks in advance.
The .pom file is as below.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany.dataload</groupId>
<artifactId>DataLoadUtilities</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>DataLoadUtilities</name>
<description>A utility project that started with fetching table metadata from SQL Server and Oracle</description>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>com.oracle.jdbc</groupId>
<artifactId>ojdbc7</artifactId>
<version>12.1.0.2</version>
</dependency>
<dependency>
<groupId>com.microsoft.sqlserver</groupId>
<artifactId>mssql-jdbc</artifactId>
<version>6.1.7.jre8-preview</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.mycompany.dataload.main.TableMetadataGeneratorForHive</mainClass>
</transformer>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<id>unpack-dependencies</id>
<phase>package</phase>
<goals>
<goal>unpack-dependencies</goal>
</goals>
<configuration>
<excludeScope>system</excludeScope>
<excludes>META-INF/*.SF</excludes>
<excludes>META-INF/*.DSA</excludes>
<excludes>META-INF/*.RSA</excludes>
<excludeGroupIds>junit,org.mockito,org.hamcrest</excludeGroupIds>
<outputDirectory>${project.build.directory}/classes</outputDirectory>
</configuration>
</execution>
</executions></plugin>
</plugins>
</build></project>
You are encountering this problem because of the way you have created your executable jar.
Each of ojdbc7-12.1.0.2.jar and mssql-jdbc-6.1.7.jre8-preview.jar contains a META-INF/services/java.sql.Driver file containing the name of the respective java.sql.Driver implementation classes for each vendor. It seems like you only get the first one that is encountered in the dependency list.
This is described in the javadoc for java.sql.DriverManager.
The Java 8 JAR File Specification - Service Provider provides more information that states that the META-INF/services/java.sql.Driver file:
... should contain a newline-separated list of unique concrete provider-class names
Therefore you can probably configure an AppendingTransformer in your configuration to merge the content of these files during shading.
Something along the lines of
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/services/java.sql.Driver</resource>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
ElasticSearch 2.4 client work well when running from Intelli Idea Java IDE. When same code run via jar file java -jar <jar-path> gives following error.
Exception in thread "main" NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{<remote-ip>}{<remote-ip>:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)
at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:288)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1226)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:86)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:56)
at com.creo.datawarehouse.es.op.imp.IndexOperationImp.createIndex(IndexOperationImp.java:66)
at com.creo.datawarehouse.es.ElasticSearch.initialise(ElasticSearch.java:27)
at com.creo.datawarehouse.script.App.main(App.java:37)
POM dependency:
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>2.4.0</version>
</dependency>
This question have not any answer and what is the jar version for 2.4.0 ES version in this question.
Please let me know the probable reason or solution for this.
Finally after hours of debugging, found its solution. Before mentioning those, quick key notes.
if els server is 2.4.1, maven jar should also be 2.4.1
Add log4j dependencies for more logs in details (found from this).
Add shade in plugin section.
Nothing to do with IDE (may run initially but to run prod, have to follow above).
But haven't got answer for why it was running in IDE?
Java Maven Dependency
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>2.4.1</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.5</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.5</version>
</dependency>
<build>
<plugins>
[---]
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.4</version>
<configuration>
<finalName>injector-2.4.1</finalName>
<appendAssemblyId>false</appendAssemblyId>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>org.elasticsearch.demo.workshop.injector.runner.Generate</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.1</version>
<executions>
<execution>
<phase>package</phase>
<goals><goal>shade</goal></goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.your.classss.App</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
[--]
</plugins>
</build>