I am trying to write a Spark program using Java to fetch records from oracle database and write it into S3 bucket. To access bucket using s3a:// convention, I have added hadoop-aws as dependency in the pom.xml.
I have added below dependencies in the pom.xml file :
<properties>
<java.version>1.8</java.version>
<scala.version>2.12</scala.version>
<spark.version>3.0.2</spark.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>com.github.noraui</groupId>
<artifactId>ojdbc7</artifactId>
<version>12.1.0.2</version>
</dependency>
<dependency>
<groupId>com.github.noraui</groupId>
<artifactId>ojdbc7</artifactId>
<version>12.1.0.2</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.11.690</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
</dependency>
I tried to run the program but got dependency error :
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/type/ReferenceType
at com.fasterxml.jackson.module.scala.modifiers.ScalaTypeModifierModule.$init$(ScalaTypeModifier.scala:35)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.<init>(DefaultScalaModule.scala:18)
at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<init>(DefaultScalaModule.scala:36)
at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<clinit>(DefaultScalaModule.scala)
After bit of research, I got to know the hadoop-aws has compilation dependency on com.fasterxml.jackson.core as mentioned here :
https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/2.8.5
So, I updated my pom.xml with the below dependencies :
<!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-annotations -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
</dependency>
But, I am still getting the same error. I am not sure what else I am missing here.
Related
I'm working on my first Java project with maven. I compiled the code but get the following error when trying to execute the jar:
java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/type/TypeReference
I have the following dependencies:
<dependencies>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.13.0</version>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.13</version>
</dependency>
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20211205</version>
</dependency>
</dependencies>
I've tried adding jackson-core as a separate dependency but this won't fix the issue.
I am getting the following error when try to get Issues from JIRA using JIRA Rest client.
java.lang.NoSuchMethodError:javax.ws.rs.core.Application.getProperties()Ljava/util/Map;
Below the pom i am using.
<properties>
<jersey-2-version>2.29</jersey-2-version>
<jaxrs.version>2.0.1</jaxrs.version>
</properties>
<dependencies>
<dependency>
<groupId>org.glassfish.jersey.containers</groupId>
<artifactId>jersey-container-servlet</artifactId>
<version>${jersey-2-version}</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.core</groupId>
<artifactId>jersey-common</artifactId>
<version>${jersey-2-version}</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.containers</groupId>
<artifactId>jersey-container-servlet-core</artifactId>
<version>${jersey-2-version}</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.inject</groupId>
<artifactId>jersey-hk2</artifactId>
<version>2.26</version>
</dependency>
<dependency>
<groupId>com.atlassian.jira</groupId>
<artifactId>jira-rest-java-client-api</artifactId>
<version>5.1.0</version>
</dependency>
<dependency>
<groupId>com.atlassian.jira</groupId>
<artifactId>jira-rest-java-client-core</artifactId>
<version>5.1.0</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.media</groupId>
<artifactId>jersey-media-json-jackson</artifactId>
<version>2.20</version>
</dependency>
<dependency>
<groupId>io.atlassian.fugue</groupId>
<artifactId>fugue</artifactId>
<version>4.7.2</version>
</dependency>
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20151123</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.9.9</version>
</dependency>
</dependencies>
I am trying to fetch issues and return it through a rest api.
SearchResult searchJqlPromise =restClient.getSearchClient().searchJql("issuetype = Bug AND resolution = Unresolved ORDER BY created DESC%2C priority DESC%2C updated DESC").claim();
entity = new GenericEntity<Iterable<Issue>>(searchJqlPromise.getIssues())
{
};
Can someone post me the exact versions of JSon and other libraries to be used.
Can you try to use the latest version and check ?
You can use the following dependencies in maven pom.xml.
<dependency>
<groupId>com.atlassian.jira</groupId>
<artifactId>jira-rest-java-client-api</artifactId>
<version>5.1.0</version>
</dependency>
<dependency>
<groupId>com.atlassian.jira</groupId>
<artifactId>jira-rest-java-client-core</artifactId>
<version>5.1.0</version>
</dependency>
References
You can refer below the links in mvnrepository.com.
https://mvnrepository.com/artifact/com.atlassian.jira/jira-rest-java-client-core/5.1.0
https://mvnrepository.com/artifact/com.atlassian.jira/jira-rest-java-client-api/5.1.0
I am attempting to use Apache Beam from Java as a data pipeline of sorts. I have written a simple class that sources from Google Pubsub and sinks to Google Bigquery, but I cannot get it to build for the life of me. I am using Maven to build and have added every Beam package I could find, but I still get "class file not found" errors.
Specifically:
[ERROR] /X:/Work/pipeline/backup-pipeline/src/main/java/PassthroughPipeline.java:[28,16] cannot access org.apache.beam.sdk.options.GcpOptions
class file for org.apache.beam.sdk.options.GcpOptions not found
[ERROR] /X:/Work/pipeline/backup-pipeline/src/main/java/PassthroughPipeline.java:[29,16] cannot access org.apache.beam.sdk.options.BigQueryOptions
class file for org.apache.beam.sdk.options.BigQueryOptions not found
[ERROR] /X:/Work/pipeline/backup-pipeline/src/main/java/PassthroughPipeline.java:[31,16] cannot access org.apache.beam.sdk.options.GcsOptions
class file for org.apache.beam.sdk.options.GcsOptions not found
Does anyone know what packages I need to add to resolve these? Google has unfortunately been no help.
The POM file that I have is based off of the example POM given by Apache for Wordcount, but with extra dependencies added. Below are the dependencies I put in it. I can provide the full file if needed, but it is quite monolithic.
<dependencies>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-apex</artifactId>
<version>${beam.version}</version>
<scope>runtime</scope>
</dependency>
<!--
Apex depends on httpclient version 4.3.5, project has a transitive dependency to httpclient 4.0.1 from
google-http-client. Apex dependency version being specified explicitly so that it gets picked up. This
can be removed when the project no longer has a dependency on a different httpclient version.
-->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.3.5</version>
<scope>runtime</scope>
<exclusions>
<exclusion>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</profile>
<profile>
<id>dataflow-runner</id>
<!-- Makes the DataflowRunner available when running a pipeline. -->
<dependencies>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>${beam.version}</version>
<scope>runtime</scope>
</dependency>
</dependencies>
</profile>
<profile>
<id>flink-runner</id>
<!-- Makes the FlinkRunner available when running a pipeline. -->
<dependencies>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-flink_2.10</artifactId>
<version>${beam.version}</version>
<scope>runtime</scope>
</dependency>
</dependencies>
</profile>
<profile>
<id>spark-runner</id>
<!-- Makes the SparkRunner available when running a pipeline. Additionally,
overrides some Spark dependencies to Beam-compatible versions. -->
<dependencies>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-spark</artifactId>
<version>${beam.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-hadoop-file-system</artifactId>
<version>${beam.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>${spark.version}</version>
<scope>runtime</scope>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>jul-to-slf4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-scala_2.10</artifactId>
<version>${jackson.version}</version>
<scope>runtime</scope>
</dependency>
</dependencies>
</profile>
</profiles>
<dependencies>
<!-- Adds a dependency on the Beam SDK. -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-core</artifactId>
<version>2.2.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-io-google-cloud-platform -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
<version>2.2.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-common-fn-api -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-common-fn-api</artifactId>
<version>2.2.0</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-extensions-google-cloud-platform-core -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
<version>2.2.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-io-common -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-common</artifactId>
<version>2.2.0</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-runners-parent -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-parent</artifactId>
<version>2.2.0</version>
<type>pom</type>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-runners-gcp-parent -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-gcp-parent</artifactId>
<version>2.2.0</version>
<type>pom</type>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-extensions-parent -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-extensions-parent</artifactId>
<version>2.2.0</version>
<type>pom</type>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-parent -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-parent</artifactId>
<version>2.2.0</version>
<type>pom</type>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-common-parent -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-common-parent</artifactId>
<version>2.2.0</version>
<type>pom</type>
</dependency>
<!-- https://mvnrepository.com/artifact/com.google.cloud.dataflow/google-cloud-dataflow-java-sdk-parent -->
<dependency>
<groupId>com.google.cloud.dataflow</groupId>
<artifactId>google-cloud-dataflow-java-sdk-parent</artifactId>
<version>2.2.0</version>
<type>pom</type>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-runners-reference -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-reference</artifactId>
<version>2.2.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-parent -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-parent</artifactId>
<version>2.2.0</version>
<type>pom</type>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-build-tools -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-build-tools</artifactId>
<version>2.2.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-runners-direct-java -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-direct-java</artifactId>
<version>2.2.0</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-runners-core-construction-java -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-core-construction-java</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.google.cloud.dataflow</groupId>
<artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
<version>[2.1.0, 2.99)</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-common-runner-api -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-common-runner-api</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>0.4.0</version>
</dependency>
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client</artifactId>
<version>${google-clients.version}</version>
<exclusions>
<!-- Exclude an old version of guava that is being pulled
in by a transitive dependency of google-api-client -->
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava-jdk5</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-bigquery</artifactId>
<version>${bigquery.version}</version>
<exclusions>
<!-- Exclude an old version of guava that is being pulled
in by a transitive dependency of google-api-client -->
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava-jdk5</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
<version>${google-clients.version}</version>
<exclusions>
<!-- Exclude an old version of guava that is being pulled
in by a transitive dependency of google-api-client -->
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava-jdk5</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-pubsub</artifactId>
<version>${pubsub.version}</version>
<exclusions>
<!-- Exclude an old version of guava that is being pulled
in by a transitive dependency of google-api-client -->
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava-jdk5</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>${joda.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.version}</version>
</dependency>
<!-- Add slf4j API frontend binding with JUL backend -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
<version>${slf4j.version}</version>
<!-- When loaded at runtime this will wire up slf4j to the JUL backend -->
<scope>runtime</scope>
</dependency>
<!-- Hamcrest and JUnit are required dependencies of PAssert,
which is used in the main code of DebuggingWordCount example. -->
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-all</artifactId>
<version>${hamcrest.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
</dependency>
</dependencies>
These classes:
org.apache.beam.sdk.options.GcpOptions
org.apache.beam.sdk.options.GcsOptions
org.apache.beam.sdk.options.BigQueryOptions
... are all in an earlier version of Apache Beam.
Given the dependencies in your pom.xml (specifically, the dependency on v2.2.0 of Apache Beam) the correct imports are:
org.apache.beam.sdk.extensions.gcp.options.GcpOptions
org.apache.beam.sdk.extensions.gcp.options.GcsOptions
org.apache.beam.sdk.io.gcp.bigquery.BigQueryOptions
#RunWith(SpringRunner.class)
#SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT)
#Import({ JerseyConfig.class })
public class BenchApplicationTest {
#Autowired
private TestRestTemplate restTemplate;
#Test
public void contextLoads() {
ResponseEntity<String> entity = this.restTemplate.getForEntity("/bench/healthcheck", String.class);
assertThat(entity.getStatusCode()).isEqualTo(HttpStatus.OK);
}
}
My application executes perfectly and i am able to test the api using Postman. But when i try the above method to execute the test case to test the API, it gives the following error.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'testRestTemplate': Initialization of bean failed; nested exception is java.lang.NoSuchFieldError: INSTANCE
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) ~[spring-beans-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) ~[spring-beans-4.3.4.RELEASE.jar:4.3.4.RELEASE]
.....
Caused by: java.lang.NoSuchFieldError: INSTANCE
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.<clinit>(SSLConnectionSocketFactory.java:144) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:962) ~[httpclient-4.5.2.jar:1.2.2]
I have tried to check the this error and the almost all of them suggest (like this link and link,) that there are multiple version of a jars(http-client/http-core) in my application. I have checked and i did not find multiple version of SSLConnectionSocketFactory in my application. If at all there are conflicting jars, how can i know which are conflicting. I do not manually add the jar, its a maven project. I am stuck with this problem for the last 1 week and I am now at my wits end.
Please find below my pom.xml
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.4.2.RELEASE</version>
</parent>
<dependencies>
<!-- Spring dependencies -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jersey</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jetty</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-json</artifactId>
<version>1.18.1</version>
</dependency>
<dependency>
<groupId>org.antlr</groupId>
<artifactId>antlr-complete</artifactId>
<version>3.5.2</version>
</dependency>
<dependency>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
</dependency>
<dependency>
<groupId>commons-beanutils</groupId>
<artifactId>commons-beanutils</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.3.2</version>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
</dependency>
<dependency>
<groupId>commons-validator</groupId>
<artifactId>commons-validator</artifactId>
<version>1.4.1</version>
</dependency>
<dependency>
<groupId>com.sun.jersey.contribs</groupId>
<artifactId>jersey-multipart</artifactId>
<version>1.19.3</version>
</dependency>
<dependency>
<groupId>net.sf.json-lib</groupId>
<artifactId>json-lib</artifactId>
<version>2.4</version>
</dependency>
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.7.2</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
</dependency>
<dependency>
<groupId>org.antlr</groupId>
<artifactId>antlr-complete</artifactId>
<version>3.5.2</version>
</dependency>
<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
<version>2.8.6</version>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.9.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc6</artifactId>
<version>11.2.0.4</version>
</dependency>
<dependency>
<groupId>org.owasp.esapi</groupId>
<artifactId>esapi</artifactId>
<version>2.1.0</version>
</dependency>
<dependency>
<groupId>com.sun.mail</groupId>
<artifactId>javax.mail</artifactId>
</dependency>
<dependency>
<groupId>org.freemarker</groupId>
<artifactId>freemarker</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-jcs-jcache</artifactId>
<version>2.0-beta-1</version>
</dependency>
<dependency>
<groupId>org.reflections</groupId>
<artifactId>reflections</artifactId>
<version>0.9.9</version>
</dependency>
<dependency>
<groupId>com.codahale.metrics</groupId>
<artifactId>metrics-healthchecks</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>com.googlecode.json-simple</groupId>
<artifactId>json-simple</artifactId>
</dependency>
</dependencies>
Any Help would be appreciated.
Quick answer
Quick check of your pom.xml shows that particular dependency org.owasp.esapi:esapi pulls old 3.1 version of httpclient in. Try adding exclusion section to this dependency as described below. If this does not help, work through dependency tree accordingly
Full answer
Run the following command under your project root folder
mvn dependency:tree -Dincludes=commons-httpclient
That will print your project's maven dependency tree with all pulled transitive dependencies with artifact id commons-httpclient, for example:
[INFO] your:project:jar:1.0-SNAPSHOT
[INFO] \- org.owasp.esapi:esapi:jar:2.1.0:compile
[INFO] \- org.owasp.antisamy:antisamy:jar:1.4.3:compile
[INFO] \- commons-httpclient:commons-httpclient:jar:3.1:compile
Than you could determine what is the source of wrong version of httpclient coming in (for our example let it be org.owasp.esapi:esapi) and then you can exclude that transitive dependency implicitly by amending your pom.xml:
<dependency>
<groupId>org.owasp.esapi</groupId>
<artifactId>esapi</artifactId>
<version>2.1.0</version>
<!-- change starts here -->
<exclusions>
<exclusion>
<groupId>commons-httpclient</groupId>
<artifactId>commons-httpclient</artifactId>
</exclusion>
</exclusions>
<!-- change ends here -->
</dependency>
Note, that you may require to filter dependency tree using wildcards like
mvn dependency:tree -Dincludes=*http*
or even work through plain tree output calling mvn dependency:tree without params
And finally you can end up with explicit adding required dependency to your pom.xml having wrong versions found and excluded as decried above:
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.2</version>
</dependency>
Update
As problem is not resolved by above. Handy code piece to determine all the files being loaded from classpath:
ClassLoader cl = ClassLoader.getSystemClassLoader();
URL[] urls = ((URLClassLoader)cl).getURLs();
for(URL url: urls){
System.out.println(url.getFile());
}
Can you put it into your contextLoads() right before restTemplate call and check/share the console output then?
I'm trying to upgrade spark-cassandra-connector from 1.4 to 1.5.
Everything seems fine but when I run test cases then It stuck between the process and log some error message saying:
Exception in thread "dag-scheduler-event-loop"
java.lang.NoClassDefFoundError: org/apache/spark/sql/DataFrame
My pom file looks like:
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector_2.10 -->
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>16.0.1</version>
</dependency>
<!-- Scala Library -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.10.5</version>
</dependency>
<!--Spark Cassandra Connector-->
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector-java_2.10</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>3.0.2</version>
</dependency>
<!--Spark-->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.5.0</version>
<exclusions>
<exclusion>
<groupId>net.java.dev.jets3t</groupId>
<artifactId>jets3t</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</project>
Thank you in advance!!
Can anyone please help me with this ?
If you need more info please let me know!!
Try to add dependency
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
Also make sure that your version spark-cassandra-connector is compatible with version of Spark you're using. I had the same error message even with all proper dependencies when was trying to use older spark-cassandra-connector with newer Spark version. Refer to this table: https://github.com/datastax/spark-cassandra-connector#version-compatibility