I have a maven project and I would like to produce the jar file with all dependencies.
I use package shade plugin and the command mvn package to produce the jar file.
However, the produced jar file does not consider any of the dependencies that is pom.xml. The produced jar file keep give me exception:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Level
Here is the content of pom.xml:
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>myProject-core</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>myProject-parser</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>info.picocli</groupId>
<artifactId>picocli</artifactId>
<version>${picoli.version}</version>
</dependency>
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${shade.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/log4j/log4j -->
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.16</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.13.1</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.13.1</version>
</dependency>
</dependencies>
<profiles>
<profile>
<id>client</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${shade.version}</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>myProject-client-${project.version}</finalName>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>myProject.package.main</mainClass>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
</transformers>
<filters>
<!-- This is necessary to avoid a java.lang.SecurityException -->
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<!-- Jar file entry point -->
<mainClass>myProject.package.main</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>
Where is my mistake?
It is even does not consider <mainClass>myProject.package.main</mainClass> for the main class when I run the jar I have to specify the class name:
java -cp myApp.jar myProject.package.main
I looked over most of the questions and nothing solve my issue.
First of all, there are two plugins that you can use to create fat jars: maven-shade-plugin and maven-assembly-plugin.
The main difference between them is that maven-assembly just builds a jar containing all the dependency, while maven-shade also relocate these dependencies internally so that, if another project is depending on your jar, you won't risk having conflicts on dependencies.
If you don't expect yo use this artifact as a dependency of other projects, you can use maven-assembly-plugin for it. Here is an example of how to use it:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.1.1</version>
<configuration>
<finalName>${project.artifactId}</finalName>
<archive>
<manifest>
<mainClass>myProject.package.main</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<appendAssemblyId>false</appendAssemblyId>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
If you really need to shade your dependencies, try to modify the manifest config of the plugin with this, at least it should solve the problem that you-re having with identifying the main class:
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Main-Class>myProject.package.main</Main-Class>
<X-Compile-Source-JDK>${maven.compiler.source}</X-Compile-Source-JDK>
<X-Compile-Target-JDK>${maven.compiler.target}</X-Compile-Target-JDK>
</manifestEntries>
</transformer>
There are no groupId in the dependencies with org.apache.log4j, you have: org.apache.logging.log4j but its not the same as: org.apache.log4j
Related
This is a weird error. After adding the selenium dependencies to the pom of my maven project and upload it to a lambda, it says it is unable to unzip the file. However after removing the dependencies, the lambda is able to unzip the file just fine (however it comes up with a class not found afterwards). I have tried removing the dependencies one by one but each one triggers the error.
Any ideas on how to solve this?
Class not found error
org/openqa/selenium/WebDriver: java.lang.NoClassDefFoundError
java.lang.NoClassDefFoundError: org/openqa/selenium/WebDriver
lambda cannot zip error
Calling the invoke API action failed with this message: Lambda was not able to unzip the file
The dependencies causing the issue
<dependency>
<groupId>org.seleniumhq.webdriver</groupId>
<artifactId>webdriver-common</artifactId>
<version>0.9.7376</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-chrome-driver</artifactId>
<version>3.141.59</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.141.59</version>
</dependency>
updated dependancies (for Vishal)
<dependency>
<groupId>org.seleniumhq.webdriver</groupId>
<artifactId>webdriver-common</artifactId>
<version>0.9.7376</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-chrome-driver</artifactId>
<version>3.141.59</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.141.59</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-api</artifactId>
<version>2.0rc2</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-remote-driver</artifactId>
<version>3.141.59</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-support</artifactId>
<version>3.141.59</version>
</dependency>
Configuration
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
<forceJavacCompilerUse>true</forceJavacCompilerUse>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
The shade plugin combines all dependencies with the developed code and plops them in one Uber JAR. The downside is that it can overwrite resource files, and doesn't play well with signed jars (in my experience at least).
I would recommend moving away from the shade plugin if at all possible.
That said, if you have to use it - you're issue may be with the combining the jar resources. There are many transformers that you can use to resolve this, and you'll need to investigate which one is really needed. I would start with something like this
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<configuration>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>${executable.classifier}</shadedClassifierName>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>fully.qualified.ClassName</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
You can find more tranformers on the Apache plugin here
The alternative that I would suggest is Spring Boot, which uses the Jar-in-Jar structure with a custom ClassLoader to load classes from the internal jar(s).
This is the easier method due to not having to re-write files as the Shade plugin approach and it handles the dependencies a little better.
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>1.3.6.RELEASE</version>
<configuration>
<classifier>${executable.classifier}</classifier>
<layout>ZIP</layout>
<mainClass>fully.qualified.ClassName</mainClass>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
Seriously, look at the simpler configuration!
NOTE: Most of this came from my own notes - version numbers may be a little old...
Try to tell your dependencies to output zip, maybe jar is messing up with things
Add this to maven-assembly-plugin configuration:
<formats>
<format>zip</format>
</formats>
For example:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
...
<configuration>
...
<formats>
<format>zip</format>
</formats>
</configuration>
</plugin>
Same is suggested here
I figured it out. The java selenium seemed to cause the major issue. Downgrading to 3.10 fixed the issue although I have no idea why.
When trying to run .jar file have an error, cause of bouncy castle library is not signed or some kind of.
Exception in thread "main" java.lang.SecurityException: JCE cannot authenticate the provider BC
at javax.crypto.Cipher.getInstance(Cipher.java:656)
at com.bmw.cxml.CryptoUtils.aesDecrypt(CryptoUtils.java:54)
at com.bmw.cxml.CryptoUtils.aesDecryptCBC(CryptoUtils.java:19)
at com.bmw.cxml.ServiceDpParser.patchData(ServiceDpParser.java:37)
at com.bmw.SACXMLPatcher.patchFiles(SACXMLPatcher.java:36)
at com.bmw.App.main(App.java:16)
Caused by: java.util.jar.JarException: file:/C:/Users/Desktop/XMLChangerTool/target/XMLChangerTool-1.0-SNAPSHOT.jar has unsigned entries - com/bmw/App.class
at javax.crypto.JarVerifier.verifySingleJar(JarVerifier.java:502)
at javax.crypto.JarVerifier.verifyJars(JarVerifier.java:363)
at javax.crypto.JarVerifier.verify(JarVerifier.java:289)
at javax.crypto.JceSecurity.verifyProviderJar(JceSecurity.java:164)
at javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:190)
at javax.crypto.Cipher.getInstance(Cipher.java:652)
All libraries add as dependency in pom.xml file. pom looks like this
<dependencies>
<dependency>
<groupId>bouncycastle</groupId>
<artifactId>bcprov-jdk15</artifactId>
<version>140</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>18.0</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.6</version>
</dependency>
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<version>1.10</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src</sourceDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.bmw.App</mainClass>
</transformer>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version>
<configuration>
<archive>
<index>true</index>
<manifest>
<mainClass>com.bmw.App</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
I was trying solutions from common questions on stackoverflow, like this
Security.removeProvider("BC")
Security.addProvider(new BouncyCastleProvider())
and security.provider.11=org.bouncycastle.jce.provider.BouncyCastleProvider but it doesn't help.
Even trying to add plugins to maven, without any success
Maybe someone knows what can I try to fix this situation?
I don't know if thats a solution for you, but i've had to put the Bouncycastle Jars in the first position of the classpath before the other jars (i think one of them contained BC as a fat-jar), then the signing exception was gone.
Strangely the exception only occured with oracle jre, not adoptopenjdk.
I'm trying to make a fat uber jar that contains all the projects.
If I do "mvn package", I get an uber jar under "blah" project taget folder. (blah project has the main class.)
The uber jar contains all the projects (as folders not jars), but when I run it, it doesn't seem to recognize feature1 & feature2 projects.
parent pom:
<plugins>
<!-- download source code in Eclipse, best practice -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-eclipse-plugin</artifactId>
<version>2.10</version>
<configuration>
<downloadSources>true</downloadSources>
<downloadJavadocs>false</downloadJavadocs>
</configuration>
</plugin>
<!-- Set a compiler level -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<!-- Maven Shade Plugin -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.1</version>
<executions>
<!-- Run shade goal on package phase -->
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<!-- add Main-Class to manifest file -->
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.a.blah.main</mainClass>
</transformer>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
<modules>
<module>blahtask</module>
<module>feature1</module>
<module>feature2</module>
<module>blahmessaging</module>
<module>blah</module>
</modules>
pom for blah
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.39</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.a.blah</groupId>
<artifactId>blahtask</artifactId>
<version>1.0-SNAPSHOT</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.a.blah</groupId>
<artifactId>blahmessaging</artifactId>
<version>1.0-SNAPSHOT</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.a.fs</groupId>
<artifactId>feature1</artifactId>
<version>1.0-SNAPSHOT</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.a.fs</groupId>
<artifactId>feature2</artifactId>
<version>1.0-SNAPSHOT</version>
<scope>compile</scope>
</dependency>
I added dependencies for feature1 & feature2 above so that they are in the uber jar file. Is this wrong?
p.s. blahmessaging, feature1, & feature2 use classes/functions from blahtask.
It's really hard to find maven-shade-plugin examples with multiple projects. Hard to find how their poms files should be and how parent-child should be structured.
It turned out to be service loader issue...
I manually added class names that I'm using for serviceloader in feature1 & feature2 projects.
If you are using serviceloader, here is what I did.
feature1/src/main/resources/META-INF/services/"some super class"
feature2/src/main/resources/META-INF/services/"some super class"
if you open those two with a text editor, there is a sub class name for each that you need for serviceloader. I copied them and appended into the "some super class" file in the jar.
I trying to read the a file using :
final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSet<String> line = env.readTextFile("file:///pathtofile/myfile.txt");
I get following error:
java.lang.NoSuchMethodError: org.apache.flink.api.common.io.DelimitedInputFormat: method <init>(Lorg/apache/flink/core/fs/Path;)V not found
I'm using flink version 1.3.2, java version "1.8.0_91"
There is a conflict with dependencies. Apache Flink loads many classes by default into its classpath.
Please read this article https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/debugging_classloading.html the last section
Resolving Dependency Conflicts with Flink using the maven-shade-plugin
Apache Flink loads many classes by default into its classpath. If a user uses a different version of a library that Flink is using, often
IllegalAccessExceptions
or
NoSuchMethodError
are the result.
So, I suggest to play with your pom.xml and use maven-shade-plugin and add correct relocation, as we have in example
<relocation>
<pattern>org.codehaus.plexus.util</pattern>
<shadedPattern>org.shaded.plexus.util</shadedPattern>
<excludes>
<exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
<exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
</excludes>
</relocation>
Are you getting this error in IntelliJ or Dashboard, if you are getting this error in IntelliJ then make sure you use the same Flink version in your pom.xml and also add dependency shading in the build like this
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<compilerId>jdt</compilerId>
</configuration>
<dependencies>
<dependency>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-compiler-jdt</artifactId>
<version>0.21.0</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<relocations>
<relocation>
<pattern>org.codehaus.plexus.util</pattern>
<shadedPattern>org.shaded.plexus.util</shadedPattern>
<excludes>
<exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
<exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
</excludes>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>>
make sure to run maven clean install in the terminal after you make changes . On the other hand, If you are having this issue only in Dashboard not in intelliJ , then have a look here
I faced the similar issue, for me the problem was flink minor version mismatch. My local flink cluster was running flink-1.8.0 and my code expected the version to be flink-1.8.3. Switching to newer version solved this issue.
You need to check your build path, make sure the libs are there and import them properly
One possible cause for error "java.lang.NoSuchMethodError" is when we use different version of flink then what we have installed on our system. For me, I have Flink 1.3.2 and the version I was using was 1.1.1 . So I updated my pom file to have same version.
For the ones who're stuggling with the above issue, while using Flink 1.3.2, here's the entire pom which I was able to successfully build:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4.1</version>
<configuration>
<!-- get all project dependencies -->
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<!-- MainClass in mainfest make a executable jar -->
<archive>
<manifest>
<mainClass>FlinktoLambda</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<!-- bind to the packaging phase -->
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>FlinktoLambda</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<!--added newly-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<compilerId>jdt</compilerId>
</configuration>
<dependencies>
<dependency>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-compiler-jdt</artifactId>
<version>0.21.0</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>FlinktoLambda</mainClass>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
</transformers>
<relocations>
<relocation>
<pattern>org.codehaus.plexus.util</pattern>
<shadedPattern>org.shaded.plexus.util</shadedPattern>
<excludes>
<exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
<exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
</excludes>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<name>my-app</name>
<url>http://maven.apache.org</url>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>FlinktoLambda</groupId>
<artifactId>my-app</artifactId>
<exclusions>
<exclusion>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-remote_2.10</artifactId>
</exclusion>
<exclusion>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-actor_2.10</artifactId>
</exclusion>
<exclusion>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-slf4j_2.10</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table_2.10</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.10</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.10</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-scala_2.10</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala_2.10</artifactId>
<version>1.3.2</version>
</dependency>
</dependencies>
Please change your main class accordingly in the shade plugin.
Another source of this error could be a mismatch of scala version between Flink and your application code (or your application dependencies that uses scala).
For instance, in my case, I was using Flink 1.7.1 and I had to update my scala dependencies from 2.11 to 2.12; I updated the artifcatId of the concerned dependencies as follows: from flink-scala_2.11 to flink-scala_2.12, flink-table_2.11 to flink-table_2.12, etc.
See here for more info.
I am assembling the fat jar with maven assembly plugin and experience the following issue:
Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: parquet. Please find packages at http://spark-packages.org
at org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource(DataSource.scala:145)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:78)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:78)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:310)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:149)
at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:427)
at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:411)
at org.apache.spark.mllib.classification.impl.GLMClassificationModel$SaveLoadV1_0$.loadData(GLMClassificationModel.scala:77)
at org.apache.spark.mllib.classification.LogisticRegressionModel$.load(LogisticRegression.scala:183)
at org.apache.spark.mllib.classification.LogisticRegressionModel.load(LogisticRegression.scala)
at my.test.spark.assembling.TopicClassifier.load(TopicClassifier.java:35)
at my.test.spark.assembling.Main.main(Main.java:23)
Caused by: java.lang.ClassNotFoundException: parquet.DefaultSource
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5$$anonfun$apply$1.apply(DataSource.scala:130)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5$$anonfun$apply$1.apply(DataSource.scala:130)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5.apply(DataSource.scala:130)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5.apply(DataSource.scala:130)
at scala.util.Try.orElse(Try.scala:84)
at org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource(DataSource.scala:130)
... 11 more
And here is the pom.xml:
<groupId>my.test.spark</groupId>
<artifactId>assembling</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.0.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>
If I run it in Intellij IDEA the problem doesn't occur.
What else should I include to the jar to be able to find the class?
I found the solution to the problem. I tried to build the package with sbt assembly and experienced different but related problem. The solution I found here: https://stackoverflow.com/a/27532248/5520896 also helps with my original issue.
So what solves the problem is moving from maven assembly plugin to maven shade plugin and apply the transformer
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
So my final pom.xml plugin configuration is the following:
<plugin>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
Apparently what goes wrong with maven assembly is explained here: https://stackoverflow.com/a/21118824/5520896