I'm trying to write a DataFrame as follows to a CSV file on HDFS
df.write()
.format("com.databricks.spark.csv")
.option("header", "true")
.save("/user/cloudera/csv");
but I get the following error
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/csv/CSVFormat
...
Caused by: java.lang.ClassNotFoundException: org.apache.commons.csv.CSVFormat
... 21 more
My pom.xml has the following dependencies
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-csv_2.10</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-csv</artifactId>
<version>1.5</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.6.0</version>
</dependency>
I use spark 1.6.0 with scala 2.10.5 and use the following command to submit the job
spark-submit --jars /path/spark-csv_2.10-1.5.0.jar --class com.iris.Begin /path/CsvSolver.jar
I also have commons-csv/1.1 and commons-csv/1.5 in .m2 repository.
Could someone help me with this?
Just try to add those need jars to jars folder located in spark folder ...\spark\jars\
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-csv</artifactId>
<version>1.5</version>
</dependency>
Try adding this to the pom. If that doesn't work manually download the JAR from here https://mvnrepository.com/artifact/org.apache.commons/commons-csv/1.5 and add using --jars to spark submit. That would definitely solve the problem
It's better to build fat jar that will include all your dependencies (spark-core should be marked as provided) & submit only this jar without any additional --jars options.
In Maven you can generate fat jar by using Maven Assembly plugin with predefined profile jar-with-dependencies. Something like:
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>
Related
I am starting a project to read and analyze a JSON-file with JAVA 8. To have it run in Eclipse, I turned it into a maven project and added this dependency :
<dependency>
<groupId>org.glassfish</groupId>
<artifactId>javax.json</artifactId>
<version>1.1</version>
<scope>runtime</scope>
</dependency>
Within Eclipse , there is no problem, but when I run it from command line, I get this error:
Provider org.glassfish.json.JsonProviderImpl not found
In futur I want to run it on a server without an Eclipse installation.
How can I get it running ?
Please see this question. It will help you create a jar file that has all the dependencies built into it. I would not suggest rewriting things and to use libraries freely. You code much faster and in more bug free ways when you do not code it yourself.
Once you have done this you will be able to run java -jar on the jar file and your application will run. If you would like to just have the thing run you can download the jar and add it to the classpath variable that you pass the java command line.
thanks a lot. Following maven configuration gets it work for command line :
<plugins>
..
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>data.Parser</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.glassfish</groupId>
<artifactId>javax.json</artifactId>
<version>1.1</version>
<scope>runtime</scope>
</dependency>
</dependencies>
if you want simply deserialize using Gson. you use this sample program
import java.io.File;
import java.io.IOException;
import java.util.Map;
import org.apache.commons.io.FileUtils;
import com.google.gson.Gson;
public class Main {
public static void main(String[] args) throws IOException {
String json = FileUtils.readFileToString(new File("PATH_TO_JSON"), "UTF-8");
Gson deserializer = new Gson();
System.out.println(deserializer.fromJson(json, Map.class));
}
}
you will need following dependencies
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.6</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.5</version>
</dependency>
I am trying to use Java samplers in my tests.
I have a separate maven project where I create my extensions. After building the project I get a .jar lib. I include it in my maven plugin like this:
<dependencies>
<dependency>
<groupId>com.lazerycode.jmeter</groupId>
<artifactId>jmeter-maven-plugin</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.qiagen</groupId>
<artifactId>qa_toolkit</artifactId>
<version>RELEASE</version>
</dependency>
<dependency>
<groupId>com.qiagen</groupId>
<artifactId>JMeterExtensions</artifactId>
<version>jmeter3.2.3</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>com.lazerycode.jmeter</groupId>
<artifactId>jmeter-maven-plugin</artifactId>
<version>2.2.0</version>
<executions>
<execution>
<id>jmeter-tests</id>
<phase>verify</phase>
<goals>
<goal>jmeter</goal>
</goals>
</execution>
</executions>
<configuration>
<testFilesDirectory>${basedir}/src/test/jmeter/</testFilesDirectory>
<testFilesIncluded>
<jMeterTestFile>${jmxTest}</jMeterTestFile>
</testFilesIncluded>
<jmeterDirectory>${jmeter.home}</jmeterDirectory>
<jmeterExtensions>
<artifact>com.qiagen:JMeterExtensions:jmeter3.2.3</artifact>
</jmeterExtensions>
<propertiesUser>
<csvData>${basedir}/src/test/jmeter/${csvData}</csvData>
<threads>${threads}</threads>
<rampTime>${rampTime}</rampTime>
<loopCount>${loopCount}</loopCount>
<options>${options}</options>
<server>${server}</server>
<port>${port}</port>
<sleep>${sleep}</sleep>
<inputXmlFileDir>${inputXmlFileDir}</inputXmlFileDir>
<templatesCsv>${templatesCsv}</templatesCsv>
<xmlInputsCsv>${xmlInputsCsv}</xmlInputsCsv>
<reportScenariosCsv>${reportScenariosCsv}</reportScenariosCsv>
</propertiesUser>
<jMeterProcessJVMSettings>
<xms>2048</xms>
<xmx>2048</xmx>
<arguments>
<argument>-Xprof</argument>
<argument>-Xfuture</argument>
</arguments>
</jMeterProcessJVMSettings>
</configuration>
</plugin>
</plugins>
</build>
In my extensions i have some invalid transitive dependencies which i excluded from extensions pom.xml. I don't see them in the dependency tree.
When I run the tests, with the flag downloadExtensionDependencies on true, it looks like it tries to download all dependencies (also those excluded) and then the test fails because of that invalid dependency.
Failed to collect dependencies at org.springframework:spring-webmvc:jar:3.1.1.RELEASE -> jasperreports:jasperreports:jar:2.0.5 -> commons-collections:commons-collections:jar:3.2.1.redhat-7: Failed to read artifact descriptor for commons-collections:commons-collections:jar:3.2.1.redhat-7: Could not transfer artifact org.apache.commons:commons-parent:pom:22-redhat-2 from/to jaspersoft (http://www.jasperforge.org/maven2): www.jasperforge.org: Unknown host www.jasperforge.org -> [Help 1]
Do you have any ideas why is the plugin trying to download the excluded dependencies also?
Use version 2.6.0 of the plugin which has now better default values like not downloading optional dependencies.
And use this to exclude broken or excluded dependencies:
<excludedArtifacts>
<exclusion>commons-pool2:commons-pool2</exclusion>
<exclusion>commons-math3:commons-math3</exclusion>
<exclusion>com.sun.jdmk:jmxtools</exclusion>
<exclusion>com.sun.jmx:jmxri</exclusion>
</excludedArtifacts>
I have a project that uses Netty 4.0.29 and I have another dependency that pulls in netty 3.9.0. I put in an exclusion but it is still roping in 3.9.0 when I run copy-dependencies.
<dependency>
<groupId>com.ning</groupId>
<artifactId>async-http-client</artifactId>
<version>1.9.31</version>
<exclusions>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
</exclusions>
</dependency>
If I run mvn dependency:tree with this exclusion in place, I see that it is indeed excluded:
[INFO] +- com.ning:async-http-client:jar:1.9.31:compile
But when I run mvn clean dependency:copy-dependencies I see the jar 3.9.0 being copied along with the 4.0.29. According to the documentation and Google, this should not copy when there is an exclusion.
[INFO] Copying netty-3.9.0.Final.jar to /Users/udonom1/wk/141/coursecopy-api/target/dependency/netty-3.9.0.Final.jar
[INFO] Copying netty-all-4.0.29.Final.jar to /Users/udonom1/wk/141/coursecopy-api/target/dependency/netty-all-4.0.29.Final.jar
I tried excluding as suggested by the first answer below and that did not work.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.1</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>process-sources</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration> <outputDirectory>${project.build.directory}/lib</outputDirectory>
<overWriteReleases>false</overWriteReleases>
<overWriteSnapshots>false</overWriteSnapshots>
<overWriteIfNewer>true</overWriteIfNewer>
<excludeArtifactIds>io.netty:netty:3.9.0.Final</excludeArtifactIds>
</configuration>
</execution>
</executions>
</plugin>
I also added a dependency as further suggested:
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
<version>4.0.29.Final</version>
</dependency>
What am I doing wrong?
For those who are having the same issue. I used mvn -X and discovered that dependency:tree is omitting two other jars that are referencing netty. I added exclusions for those and I'm good to go. Spent a whole day on this.
If you are writing not library you have simple way to control versions of any dependency in your project - dependencyManagement block in root pom file, example:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
<version>4.0.29.Final</version>
</dependency>
</dependencies>
</dependencyManagement>
Additional bonuses from this block - you can omit version and scope for dependency in concrete dependency (with same group id, artifact id and packaging).
PS another look to your dependencies make me ask you: are you sure that this dependency have single maven artifact id? netty-all-4.0.29.Final.jar - seems that this artifact should have netty-all artifact id... If they have different artifact id's my recipe wouldn't help. In this case you should define build configuration for maven-dependency-plugin, example:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.10</version>
<configuration>
<excludeArtifactIds>io.netty:netty:3.9.0.Final</excludeArtifactIds>
</configuration>
</plugin>
</plugins>
</build>
or just use -DexcludeArtifactIds parameter in your maven call
I'm experiencing following error, on trying to run CucumberJVM tests using Maven Reporting Mojo.
[INFO] --- maven-cucumber-reporting:0.0.5:generate (execution) # testproject ---
About to generate
java.io.FileNotFoundException: /Users//IdeaProjects/testproject/target/cucumber.json (No such file or directory)
I have following dependencies in POM.xml file.
<dependency>
<groupId>net.masterthought</groupId>
<artifactId>maven-cucumber-reporting</artifactId>
<version>${maven.cucumber.reporting.version}</version>
</dependency>
<dependency>
<groupId>net.masterthought</groupId>
<artifactId>cucumber-reporting</artifactId>
<version>${cucumber.reporting.version}</version>
</dependency>
<dependency>
<groupId>com.googlecode.totallylazy</groupId>
<artifactId>totallylazy</artifactId>
<version>1077</version>
</dependency>
<dependency>
<groupId>com.beust</groupId>
<artifactId>jcommander</artifactId>
<version>1.25</version>
</dependency>
Repositories set to following.
<repositories>
<repository>
<id>repo.bodar.com</id>
<url>http://repo.bodar.com</url>
</repository>
<repository>
<id>sonatype-releases</id>
<url>https://oss.sonatype.org/content/repositories/releases/</url>
</repository>
</repositories>
Plugin set to following :
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.16</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
<plugin>
<groupId>net.masterthought</groupId>
<artifactId>maven-cucumber-reporting</artifactId>
<version>${maven.cucumber.reporting.version}</version>
<executions>
<execution>
<id>execution</id>
<phase>verify</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<projectName>${project.name}</projectName>
<outputDirectory>${project.build.directory}/cucumber-html-reports</outputDirectory>
<cucumberOutput>${project.build.directory}/cucumber.json</cucumberOutput>
<enableFlashCharts>false</enableFlashCharts>
</configuration>
</execution>
</executions>
</plugin>
I'm using Standard maven project structure, with feature files hosted in /src/test/resources, Step definitions under /src/test/main.
Would you please advise how to resolve this issue. I'm seeing this issue on MAC 64-bit OS.
Thanks in advance.
Best Regards,
Raja
The plugin generates the HTML report from json. So the first step is to have cucumber generate the json.
When I ran into the same issue I found the cause of this the problem is that the cucumber.json file does not exist.
I addressed this in my project by adding json:target under the #CucumberOptions annotation, this then generated the json. for example
#CucumberOptions(features = "src/test/resources/features/some.feature",
monochrome = false, format = {"pretty", "json:target/cucumber.json"})
The masterthought plugin then processes the generated JSON by looking for it at ${project.build.directory}/cucumber.json
At this point the Maven plugin generates the HTML from the JSON.
I'm a relatively new Java programmer, and I've only worked with Maven and Jackson on a couple projects.
In my current project, I am attempting to build a process I can run on Heroku (which I'm brand new to) once every 24 hours. Part of this program uses Jackson to deserialize json to use in an array. When run through my IDE (IntelliJ), everything works fine. However, when I run the program after a mvn package I get the following error:
Exception in thread "main" java.lang.VerifyError:
(class: org/codehaus/jackson/map/ObjectMapper, method: <init> signature: (Lorg/codehaus/jackson/JsonFactory;Lorg/codehaus/jackson/map/SerializerProvider;Lorg/codehaus/jackson/map/DeserializerProvider;Lorg/codehaus/jackson/map/SerializationConfig;Lorg/codehaus/jackson/map/DeserializationConfig;)V) Bad type in putfield/putstatic
at com.example.project.Main.main(Main.java:47)
This is obviously a showstopper since it keeps me from deploying to Heroku.
The line throwing the error is when I initialize an ObjectMapper:
ObjectMapper mapper = new ObjectMapper();
As for my POM, I was following this tutorial from Heroku. Here are the relevant sections from my POM:
...
<build>
<sourceDirectory>src/main/java</sourceDirectory>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>appassembler-maven-plugin</artifactId>
<version>1.1.1</version>
<configuration>
<assembleDirectory>target</assembleDirectory>
<programs><program>
<mainClass>com.example.project.Main</mainClass>
<name>main</name>
</program></programs>
</configuration>
<executions>
<execution>
<phase>package</phase><goals><goal>assemble</goal></goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.4</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals><goal>copy-dependencies</goal></goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mrbean</artifactId>
<version>1.9.13</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>1.9.13</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-asl</artifactId>
<version>0.9.5</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.2.2</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.13</version>
</dependency>
...
</dependencies>
...
So my process is a mvn clean install and a mvn package (where both succeed), and then sh target/bin/main (where we get the error).
If I can provide any more useful information, please let me know! Thanks in advance!
EDIT:
After running my program differently...
java -cp target/classes:"target/dependency/*" com.example.project.Main
I found that I essentially ran into the same problem as #Jithin from this post:
Jackson + Tomcat - java.lang.VerifyError: Cannot inherit from final class
The error was coming from having conflicting Jackson versions in my classpath/dependency tree (i.e., one of my other dependencies was also using Jackson). I found that, in my case, the only Jackson dependency I required was jackson-annotations. The program now runs like a charm! Hopefully this helps someone down the road.