A Maven Java project using geotools 15.2 generates an error : NoSuchFieldError: METER
pom.xml
<dependency>
<groupId>org.geotools</groupId>
<artifactId>gt-data</artifactId>
<version>15.2</version>
</dependency>
<dependency>
<groupId>org.geotools</groupId>
<artifactId>gt-shapefile</artifactId>
<version>15.2</version>
</dependency>
<dependency>
<groupId>org.geotools</groupId>
<artifactId>gt-swing</artifactId>
<version>15.2</version>
</dependency>
When running the code, the error is generated when calling source.getFeatures(Filter.INCLUDE).
Stacktrace
java.lang.NoSuchFieldError: METER
at org.geotools.referencing.wkt.Parser.parseSpheroid(Parser.java:560)
at org.geotools.referencing.wkt.Parser.parseDatum(Parser.java:656)
at org.geotools.referencing.wkt.Parser.parseGeoGCS(Parser.java:867)
at org.geotools.referencing.wkt.Parser.parseProjCS(Parser.java:913)
at org.geotools.referencing.wkt.Parser.parseCoordinateReferenceSystem(Parser.java:225)
at org.geotools.referencing.wkt.Parser.parseCoordinateReferenceSystem(Parser.java:204)
I found out that SI.METER is linked to javax.measure but even if I add it to the project, the error is still present.
The result of mvn dependency:tree is
[INFO] +- com.vividsolutions:jts:jar:1.13:compile
[INFO] +- org.opengis:geoapi:jar:3.0.0:compile
[INFO] | \- javax.measure:jsr-275:jar:0.9.3:compile
[INFO] +- org.geotools:gt-data:jar:15.2:compile
[INFO] | +- org.geotools:gt-main:jar:15.2:compile
[INFO] | | \- org.geotools:gt-api:jar:15.2:compile
[INFO] | \- javax.media:jai_core:jar:1.1.3:compile
[INFO] +- org.geotools:gt-shapefile:jar:15.2:compile
[INFO] | \- org.jdom:jdom:jar:1.1.3:compile
[INFO] \- org.geotools:gt-swing:jar:15.2:compile
[INFO] +- org.geotools:gt-referencing:jar:15.2:compile
[INFO] | +- com.googlecode.efficient-java-matrix-library:core:jar:0.26:compile
[INFO] | +- commons-pool:commons-pool:jar:1.5.4:compile
[INFO] | +- org.geotools:gt-metadata:jar:15.2:compile
[INFO] | | \- org.geotools:gt-opengis:jar:15.2:compile
[INFO] | | \- net.java.dev.jsr-275:jsr-275:jar:1.0-beta-2:compile
[INFO] | +- jgridshift:jgridshift:jar:1.0:compile
[INFO] | \- net.sf.geographiclib:GeographicLib-Java:jar:1.44:compile
[INFO] +- org.geotools:gt-render:jar:15.2:compile
[INFO] | +- org.geotools:gt-coverage:jar:15.2:compile
[INFO] | | +- javax.media:jai_imageio:jar:1.1:compile
[INFO] | | +- it.geosolutions.imageio-ext:imageio-ext-tiff:jar:1.1.15:compile
[INFO] | | | +- it.geosolutions.imageio-ext:imageio-ext-utilities:jar:1.1.15:compile
[INFO] | | | +- it.geosolutions.imageio-ext:imageio-ext-geocore:jar:1.1.15:compile
[INFO] | | | | \- it.geosolutions.imageio-ext:imageio-ext-streams:jar:1.1.15:compile
[INFO] | | | \- javax.media:jai_codec:jar:1.1.3:compile
[INFO] | | +- org.jaitools:jt-zonalstats:jar:1.4.0:compile
[INFO] | | +- org.jaitools:jt-utils:jar:1.4.0:compile
[INFO] | | +- it.geosolutions.jaiext.affine:jt-affine:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.algebra:jt-algebra:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.bandmerge:jt-bandmerge:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.bandselect:jt-bandselect:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.bandcombine:jt-bandcombine:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.border:jt-border:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.buffer:jt-buffer:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.crop:jt-crop:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.iterators:jt-iterators:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.lookup:jt-lookup:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.mosaic:jt-mosaic:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.nullop:jt-nullop:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.rescale:jt-rescale:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.scale:jt-scale:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.stats:jt-stats:jar:1.0.11:compile
[INFO] | | | \- com.google.guava:guava:jar:17.0:compile
[INFO] | | +- it.geosolutions.jaiext.translate:jt-translate:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.utilities:jt-utilities:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.warp:jt-warp:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.zonal:jt-zonal:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.binarize:jt-binarize:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.format:jt-format:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.colorconvert:jt-colorconvert:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.errordiffusion:jt-errordiffusion:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.orderdither:jt-orderdither:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.colorindexer:jt-colorindexer:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.imagefunction:jt-imagefunction:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.piecewise:jt-piecewise:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.classifier:jt-classifier:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.rlookup:jt-rlookup:jar:1.0.11:compile
[INFO] | | +- it.geosolutions.jaiext.vectorbin:jt-vectorbin:jar:1.0.11:compile
[INFO] | | \- commons-io:commons-io:jar:2.1:compile
[INFO] | \- org.geotools:gt-cql:jar:15.2:compile
[INFO] \- com.miglayout:miglayout:jar:swing:3.7:compile
Am I missing a dependency ?
The answer was found with the help of #IanTurton
I upgraded the geotools version from 15.2 to 20.5. I also added the gt-epsg-hsql jar. The pom.xml is now like this :
<dependency>
<groupId>org.geotools</groupId>
<artifactId>gt-api</artifactId>
<version>20.5</version>
</dependency>
<dependency>
<groupId>org.geotools</groupId>
<artifactId>gt-shapefile</artifactId>
<version>20.5</version>
</dependency>
<dependency>
<groupId>org.geotools</groupId>
<artifactId>gt-swing</artifactId>
<version>20.5</version>
</dependency>
<dependency>
<groupId>org.geotools</groupId>
<artifactId>gt-epsg-hsql</artifactId>
<version>20.5</version>
</dependency>
I also updated my code to use org.locationtech.jts.geom.Geometry instead of com.vividsolutions
Related
I have a spark spring-boot application and upgrading spring boot from 1.X to 2.1.5.Now with spring-boot 2.1.15 upgrade , I am getting snakeyaml error in application startup as:
ERROR SpringApplication - Application run failed
java.lang.NoSuchMethodError: org.yaml.snakeyaml.Yaml.<init>(Lorg/yaml/snakeyaml/constructor/BaseConstructor;Lorg/yaml/snakeyaml/representer/Representer;Lorg/yaml/snakeyaml/DumperOptions;Lorg/yaml/snakeyaml/LoaderOptions;Lorg/yaml/snakeyaml/resolver/Resolver;)V
at org.springframework.boot.env.OriginTrackedYamlLoader.createYaml(OriginTrackedYamlLoader.java:71)
at org.springframework.beans.factory.config.YamlProcessor.process(YamlProcessor.java:162)
at org.springframework.boot.env.OriginTrackedYamlLoader.load(OriginTrackedYamlLoader.java:76)
at org.springframework.boot.env.YamlPropertySourceLoader.load(YamlPropertySourceLoader.java:50)
at org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.loadDocuments(ConfigFileApplicationListener.java:544)
at org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.load(ConfigFileApplicationListener.java:499)
at org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.loadForFileExtension(ConfigFileApplicationListener.java:475)
at org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.load(ConfigFileApplicationListener.java:445)
at org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.lambda$null$6(ConfigFileApplicationListener.java:427)
at java.lang.Iterable.forEach(Iterable.java:75)
at org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.lambda$load$7(ConfigFileApplicationListener.java:427)
at java.lang.Iterable.forEach(Iterable.java:75)
at org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.load(ConfigFileApplicationListener.java:424)
at org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.load(ConfigFileApplicationListener.java:323)
at org.springframework.boot.context.config.ConfigFileApplicationListener.addPropertySources(ConfigFileApplicationListener.java:204)
at org.springframework.boot.context.config.ConfigFileApplicationListener.postProcessEnvironment(ConfigFileApplicationListener.java:188)
at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEnvironmentPreparedEvent(ConfigFileApplicationListener.java:178)
at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEvent(ConfigFileApplicationListener.java:166)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127)
at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:76)
at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:53)
at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:342)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:305)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1204)
at com.capitalone.customercore.autolink.AutolinkApp.main(AutolinkApp.java:17)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
I did check the snakeyaml in dependency:tree and see org.yaml:snakeyaml:jar:1.23:compile
My dependency:tree is :
com.test.customercore:autolink-utility:jar:0.30-SNAPSHOT
[INFO] +- org.springframework.boot:spring-boot-starter:jar:2.1.15.RELEASE:compile
[INFO] | +- org.springframework.boot:spring-boot:jar:2.1.15.RELEASE:compile
[INFO] | +- org.springframework.boot:spring-boot-autoconfigure:jar:2.1.15.RELEASE:compile
[INFO] | +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] | +- org.springframework:spring-core:jar:5.1.16.RELEASE:compile
[INFO] | | \- org.springframework:spring-jcl:jar:5.1.16.RELEASE:compile
[INFO] | \- org.yaml:snakeyaml:jar:1.23:compile
[INFO] +- org.springframework:spring-expression:jar:5.3.17:compile
[INFO] [INFO] | +- com.amazonaws:aws-java-sdk-kinesisvideo:jar:1.11.511:compile
[INFO] | | \- io.netty:netty-handler:jar:4.1.50.Final:compile
[INFO] | | +- io.netty:netty-common:jar:4.1.50.Final:compile
[INFO] | | +- io.netty:netty-resolver:jar:4.1.50.Final:compile
[INFO] | | +- io.netty:netty-buffer:jar:4.1.50.Final:compile
[INFO] | | +- io.netty:netty-transport:jar:4.1.50.Final:compile
[INFO] | | \- io.netty:netty-codec:jar:4.1.50.Final:compile
[INFO] | [INFO] +- org.springframework.boot:spring-boot-starter-data-cassandra:jar:2.1.15.RELEASE:compile
[INFO] | \- org.springframework:spring-tx:jar:5.1.16.RELEASE:compile
[INFO] +- org.springframework.data:spring-data-cassandra:jar:2.1.18.RELEASE:compile
[INFO] | +- org.springframework:spring-context:jar:5.1.16.RELEASE:compile
[INFO] | | \- org.springframework:spring-aop:jar:5.1.16.RELEASE:compile
[INFO] | +- org.springframework:spring-beans:jar:5.1.16.RELEASE:compile
[INFO] | +- org.springframework.data:spring-data-commons:jar:2.1.18.RELEASE:compile
[INFO] | +- com.datastax.cassandra:cassandra-driver-core:jar:3.6.0:compile
[INFO] | | +- com.github.jnr:jnr-ffi:jar:2.1.7:compile
[INFO] | | | +- com.github.jnr:jffi:jar:1.2.16:compile
[INFO] | | | +- com.github.jnr:jffi:jar:native:1.2.16:runtime
[INFO] | | | +- org.ow2.asm:asm:jar:5.0.3:compile
[INFO] | | | +- org.ow2.asm:asm-commons:jar:5.0.3:compile
[INFO] | | | +- org.ow2.asm:asm-analysis:jar:5.0.3:compile
[INFO] | | | +- org.ow2.asm:asm-tree:jar:5.0.3:compile
[INFO] | | | +- org.ow2.asm:asm-util:jar:5.0.3:compile
[INFO] | | | \- com.github.jnr:jnr-x86asm:jar:1.0.2:compile
[INFO] | | \- com.github.jnr:jnr-posix:jar:3.0.44:compile
[INFO] | | \- com.github.jnr:jnr-constants:jar:0.9.9:compile
[INFO] | \- org.slf4j:slf4j-api:jar:1.7.30:compile
[INFO] +- org.springframework.boot:spring-boot-starter-test:jar:2.1.15.RELEASE:test
[INFO] | +- org.springframework.boot:spring-boot-test:jar:2.1.15.RELEASE:test
[INFO] | +- org.springframework.boot:spring-boot-test-autoconfigure:jar:2.1.15.RELEASE:test
[INFO] | +- com.jayway.jsonpath:json-path:jar:2.4.0:test
[INFO] | | \- net.minidev:json-smart:jar:2.3:compile
[INFO] | | \- net.minidev:accessors-smart:jar:1.2:compile
[INFO] | +- junit:junit:jar:4.12:test
[INFO] | +- org.assertj:assertj-core:jar:3.11.1:test
[INFO] | +- org.mockito:mockito-core:jar:2.23.4:test
[INFO] | | +- net.bytebuddy:byte-buddy:jar:1.9.16:compile
[INFO] | | +- net.bytebuddy:byte-buddy-agent:jar:1.9.16:test
[INFO] | | \- org.objenesis:objenesis:jar:2.6:compile
[INFO] | +- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] | +- org.hamcrest:hamcrest-library:jar:1.3:test
[INFO] | +- org.skyscreamer:jsonassert:jar:1.5.0:test
[INFO] | | \- com.vaadin.external.google:android-json:jar:0.0.20131108.vaadin1:test
[INFO] | +- org.springframework:spring-test:jar:5.1.16.RELEASE:test
[INFO] | \- org.xmlunit:xmlunit-core:jar:2.6.4:test
[INFO] +- org.springframework.boot:spring-boot-starter-log4j:jar:1.3.8.RELEASE:compile
[INFO] | +- org.slf4j:jcl-over-slf4j:jar:1.7.30:compile
[INFO] | +- org.slf4j:jul-to-slf4j:jar:1.7.30:compile
[INFO] | \- org.slf4j:slf4j-log4j12:jar:1.7.30:compile
[INFO] +- org.apache.logging.log4j:log4j-api:jar:2.17.1:compile
[INFO] +- com.test.api.customers:customers-relations-common:jar:1.1.28-RELEASE:compile
[INFO] | +- javax.inject:javax.inject:jar:1:compile
[INFO] | +- org.apache.commons:commons-lang3:jar:3.8.1:compile
[INFO] | +- org.apache.commons:commons-collections4:jar:4.1:compile
[INFO] | +- com.test.chassis.model:chassis-business-model:jar:3.1.1.RELEASE:compile
[INFO] | | +- com.test.chassis:chassis-annotations:jar:3.1.1.RELEASE:compile
[INFO] | | +- io.swagger:swagger-annotations:jar:1.5.21:compile
[INFO] | | +- javax.xml.bind:jaxb-api:jar:2.3.1:compile
[INFO] | | | \- javax.activation:javax.activation-api:jar:1.2.0:compile
[INFO] | | +- org.jooq:jool-java-8:jar:0.9.13:compile
[INFO] | | +- com.sun.xml.bind:jaxb-core:jar:2.3.0:compile
[INFO] | | \- com.sun.xml.bind:jaxb-impl:jar:2.3.0:compile
[INFO] | +- com.test.chassis.engine:chassis-model:jar:3.1.1.RELEASE:compile
[INFO] | | +- javax.ws.rs:javax.ws.rs-api:jar:2.1:compile
[INFO] | | +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] | | +- javax.el:javax.el-api:jar:3.0.0:compile
[INFO] | | \- org.glassfish.web:el-impl:jar:2.2:compile
[INFO] | +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.9.10:compile
[INFO] | | \- com.fasterxml.jackson.core:jackson-core:jar:2.9.10:compile
[INFO] | +- org.hamcrest:java-hamcrest:jar:2.0.0.0:compile
[INFO] | +- redis.clients:jedis:jar:2.9.3:compile
[INFO] | | \- org.apache.commons:commons-pool2:jar:2.6.2:compile
[INFO] | \- org.springframework.data:spring-data-redis:jar:2.1.18.RELEASE:compile
[INFO] | +- org.springframework.data:spring-data-keyvalue:jar:2.1.18.RELEASE:compile
[INFO] | +- org.springframework:spring-oxm:jar:5.1.16.RELEASE:compile
[INFO] | \- org.springframework:spring-context-support:jar:5.1.16.RELEASE:compile
[INFO] +- org.hibernate.validator:hibernate-validator:jar:6.1.5.Final:compile
[INFO] | +- jakarta.validation:jakarta.validation-api:jar:2.0.2:compile
[INFO] | +- org.jboss.logging:jboss-logging:jar:3.3.3.Final:compile
[INFO] | \- com.fasterxml:classmate:jar:1.4.0:compile
[INFO] +- org.apache.spark:spark-sql_2.11:jar:2.4.0:compile
[INFO] | +- com.univocity:univocity-parsers:jar:2.7.3:compile
[INFO] | +- org.apache.spark:spark-sketch_2.11:jar:2.4.0:compile
[INFO] | +- org.apache.spark:spark-catalyst_2.11:jar:2.4.0:compile
[INFO] | | +- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.1.0:compile
[INFO] | | +- org.codehaus.janino:janino:jar:3.0.16:compile
[INFO] | | +- org.codehaus.janino:commons-compiler:jar:3.0.16:compile
[INFO] | | \- org.antlr:antlr4-runtime:jar:4.7:compile
[INFO] | +- org.apache.spark:spark-tags_2.11:jar:2.4.0:compile
[INFO] | +- org.apache.orc:orc-core:jar:nohive:1.5.2:compile
[INFO] | | +- org.apache.orc:orc-shims:jar:1.5.2:compile
[INFO] | | \- io.airlift:aircompressor:jar:0.10:compile
[INFO] | +- org.apache.orc:orc-mapreduce:jar:nohive:1.5.2:compile
[INFO] | +- org.apache.parquet:parquet-column:jar:1.10.0:compile
[INFO] | | +- org.apache.parquet:parquet-common:jar:1.10.0:compile
[INFO] | | \- org.apache.parquet:parquet-encoding:jar:1.10.0:compile
[INFO] | +- org.apache.parquet:parquet-hadoop:jar:1.10.0:compile
[INFO] | | +- org.apache.parquet:parquet-format:jar:2.4.0:compile
[INFO] | | +- org.apache.parquet:parquet-jackson:jar:1.10.0:compile
[INFO] | | \- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] | +- org.apache.arrow:arrow-vector:jar:0.10.0:compile
[INFO] | | +- org.apache.arrow:arrow-format:jar:0.10.0:compile
[INFO] | | +- org.apache.arrow:arrow-memory:jar:0.10.0:compile
[INFO] | | +- com.carrotsearch:hppc:jar:0.7.2:compile
[INFO] | | \- com.vlkan:flatbuffers:jar:1.2.0-3f79e055:compile
[INFO] | +- org.apache.xbean:xbean-asm6-shaded:jar:4.8:compile
[INFO] | \- org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] +- org.apache.spark:spark-core_2.11:jar:2.4.1:compile
[INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.8:compile
[INFO] | +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] | | \- org.tukaani:xz:jar:1.5:compile
[INFO] | +- org.apache.avro:avro-mapred:jar:hadoop2:1.8.2:compile
[INFO] | | \- org.apache.avro:avro-ipc:jar:1.8.2:compile
[INFO] | +- com.twitter:chill_2.11:jar:0.9.3:compile
[INFO] | | \- com.esotericsoftware:kryo-shaded:jar:4.0.2:compile
[INFO] | | \- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] | +- com.twitter:chill-java:jar:0.9.3:compile
[INFO] | +- org.apache.spark:spark-launcher_2.11:jar:2.4.1:compile
[INFO] | +- org.apache.spark:spark-kvstore_2.11:jar:2.4.1:compile
[INFO] | +- org.apache.spark:spark-network-shuffle_2.11:jar:2.4.1:compile
[INFO] | +- org.apache.spark:spark-unsafe_2.11:jar:2.4.1:compile
[INFO] | +- javax.activation:activation:jar:1.1.1:compile
[INFO] | +- org.apache.curator:curator-recipes:jar:2.6.0:compile
[INFO] | | \- org.apache.curator:curator-framework:jar:2.6.0:compile
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
[INFO] | +- javax.servlet:javax.servlet-api:jar:4.0.1:compile
[INFO] | +- org.apache.commons:commons-math3:jar:3.4.1:compile
[INFO] | +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] | +- com.ning:compress-lzf:jar:1.0.3:compile
[INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.7.1:compile
[INFO] | +- org.lz4:lz4-java:jar:1.4.0:compile
[INFO] | +- com.github.luben:zstd-jni:jar:1.3.2-2:compile
[INFO] | +- org.roaringbitmap:RoaringBitmap:jar:0.5.11:compile
[INFO] | +- commons-net:commons-net:jar:3.1:compile
[INFO] | +- org.scala-lang:scala-library:jar:2.11.12:compile
[INFO] | +- org.json4s:json4s-jackson_2.11:jar:3.5.3:compile
[INFO] | | \- org.json4s:json4s-core_2.11:jar:3.5.3:compile
[INFO] | | +- org.json4s:json4s-ast_2.11:jar:3.5.3:compile
[INFO] | | +- org.json4s:json4s-scalap_2.11:jar:3.5.3:compile
[INFO] | | \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.6:compile
[INFO] | +- org.glassfish.jersey.core:jersey-client:jar:2.27:compile
[INFO] | | \- org.glassfish.hk2.external:javax.inject:jar:2.5.0-b42:compile
[INFO] | +- org.glassfish.jersey.core:jersey-common:jar:2.27:compile
[INFO] | | \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile
[INFO] | +- org.glassfish.jersey.core:jersey-server:jar:2.27:compile
[INFO] | | \- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.27:compile
[INFO] | +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.27:compile
[INFO] | +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.27:compile
[INFO] | +- com.clearspring.analytics:stream:jar:2.7.0:compile
[INFO] | +- io.dropwizard.metrics:metrics-core:jar:3.2.2:compile
[INFO] | +- io.dropwizard.metrics:metrics-jvm:jar:4.0.7:compile
[INFO] | +- io.dropwizard.metrics:metrics-json:jar:4.0.7:compile
[INFO] | +- io.dropwizard.metrics:metrics-graphite:jar:4.0.7:compile
[INFO] | | \- com.rabbitmq:amqp-client:jar:5.4.3:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.10.4:compile
[INFO] | +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.9.10:compile
[INFO] | | \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.9.10:compile
[INFO] | +- org.apache.ivy:ivy:jar:2.4.0:compile
[INFO] | +- oro:oro:jar:2.0.8:compile
[INFO] | +- net.razorvine:pyrolite:jar:4.13:compile
[INFO] | +- net.sf.py4j:py4j:jar:0.10.7:compile
[INFO] | \- org.apache.commons:commons-crypto:jar:1.0.0:compile
[INFO] +- org.apache.spark:spark-network-common_2.11:jar:2.4.6:compile
[INFO] | +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO] | \- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.10:compile
[INFO] +- commons-codec:commons-codec:jar:1.13:compile
[INFO] +- org.apache.logging.log4j:log4j-1.2-api:jar:2.17.1:compile
[INFO] +- commons-lang:commons-lang:jar:2.6:compile
[INFO] +- com.datastax.spark:spark-cassandra-connector-unshaded_2.11:jar:2.4.0:compile
[INFO] | +- org.joda:joda-convert:jar:1.2:compile
[INFO] | +- joda-time:joda-time:jar:2.10.6:compile
[INFO] | +- io.netty:netty-all:jar:4.1.50.Final:compile
[INFO] | +- com.twitter:jsr166e:jar:1.1.0:compile
[INFO] | \- org.scala-lang:scala-reflect:jar:2.11.12:compile
[INFO] +- org.cassandraunit:cassandra-unit:jar:3.1.1.0:test
[INFO] | +- org.apache.cassandra:cassandra-all:jar:3.9:test
[INFO] | | +- net.jpountz.lz4:lz4:jar:1.3.0:test
[INFO] | | +- commons-cli:commons-cli:jar:1.1:compile
[INFO] | | +- com.googlecode.concurrentlinkedhashmap:concurrentlinkedhashmap-lru:jar:1.4:test
[INFO] | | +- org.antlr:antlr:jar:3.5.2:test
[INFO] | | | \- org.antlr:ST4:jar:4.0.8:test
[INFO] | | +- org.antlr:antlr-runtime:jar:3.5.2:test
[INFO] | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.2:test
[INFO] | | +- com.googlecode.json-simple:json-simple:jar:1.1:test
[INFO] | | +- com.boundary:high-scale-lib:jar:1.0.6:test
[INFO] | | +- org.mindrot:jbcrypt:jar:0.3m:test
[INFO] | | +- com.addthis.metrics:reporter-config3:jar:3.0.0:test
[INFO] | | | \- com.addthis.metrics:reporter-config-base:jar:3.0.0:test
[INFO] | | +- com.thinkaurelius.thrift:thrift-server:jar:0.3.7:test
[INFO] | | | \- com.lmax:disruptor:jar:3.0.1:test
[INFO] | | +- org.apache.thrift:libthrift:jar:0.9.2:test
[INFO] | | +- org.apache.cassandra:cassandra-thrift:jar:3.9:test
[INFO] | | | +- de.jflex:jflex:jar:1.6.0:test
[INFO] | | | | \- org.apache.ant:ant:jar:1.7.0:test
[INFO] | | | | \- org.apache.ant:ant-launcher:jar:1.7.0:test
[INFO] | | | +- net.mintern:primitive:jar:1.0:test
[INFO] | | | +- com.github.rholder:snowball-stemmer:jar:1.3.0.581.1:test
[INFO] | | | \- com.googlecode.concurrent-trees:concurrent-trees:jar:2.4.0:test
[INFO] | | +- net.java.dev.jna:jna:jar:4.5.2:test
[INFO] | | +- com.github.jbellis:jamm:jar:0.3.0:test
[INFO] | | +- org.fusesource:sigar:jar:1.6.4:test
[INFO] | | +- org.eclipse.jdt.core.compiler:ecj:jar:4.4.2:test
[INFO] | | +- org.caffinitas.ohc:ohc-core:jar:0.4.3:test
[INFO] | | \- com.github.ben-manes.caffeine:caffeine:jar:2.6.2:test
[INFO] | \- com.google.guava:guava:jar:18.0:compile
[INFO] +- io.netty:netty:jar:3.10.6.Final:compile
[INFO] +- org.apache.hadoop:hadoop-client:jar:3.3.2:compile
[INFO] | +- org.apache.hadoop:hadoop-common:jar:3.3.2:compile
[INFO] | | +- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.1.1:compile
[INFO] | | +- org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.1.1:compile
[INFO] | | +- org.apache.httpcomponents:httpclient:jar:4.5.12:compile
[INFO] | | | \- org.apache.httpcomponents:httpcore:jar:4.4.13:compile
[INFO] | | +- commons-io:commons-io:jar:2.8.0:compile
[INFO] | | +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] | | +- jakarta.activation:jakarta.activation-api:jar:1.2.1:runtime
[INFO] | | +- javax.servlet.jsp:jsp-api:jar:2.1:runtime
[INFO] | | +- com.sun.jersey:jersey-servlet:jar:1.19:compile
[INFO] | | +- commons-beanutils:commons-beanutils:jar:1.9.4:compile
[INFO] | | +- org.apache.commons:commons-configuration2:jar:2.1.1:compile
[INFO] | | +- org.apache.commons:commons-text:jar:1.4:compile
[INFO] | | +- com.google.re2j:re2j:jar:1.1:compile
[INFO] | | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] | | +- com.google.code.gson:gson:jar:2.8.6:compile
[INFO] | | +- org.apache.hadoop:hadoop-auth:jar:3.3.2:compile
[INFO] | | | +- com.nimbusds:nimbus-jose-jwt:jar:9.8.1:compile
[INFO] | | | | \- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
[INFO] | | | \- org.apache.kerby:kerb-simplekdc:jar:1.0.1:compile
[INFO] | | | +- org.apache.kerby:kerb-client:jar:1.0.1:compile
[INFO] | | | | +- org.apache.kerby:kerby-config:jar:1.0.1:compile
[INFO] | | | | +- org.apache.kerby:kerb-common:jar:1.0.1:compile
[INFO] | | | | | \- org.apache.kerby:kerb-crypto:jar:1.0.1:compile
[INFO] | | | | +- org.apache.kerby:kerb-util:jar:1.0.1:compile
[INFO] | | | | \- org.apache.kerby:token-provider:jar:1.0.1:compile
[INFO] | | | \- org.apache.kerby:kerb-admin:jar:1.0.1:compile
[INFO] | | | +- org.apache.kerby:kerb-server:jar:1.0.1:compile
[INFO] | | | | \- org.apache.kerby:kerb-identity:jar:1.0.1:compile
[INFO] | | | \- org.apache.kerby:kerby-xdr:jar:1.0.1:compile
[INFO] | | +- org.apache.curator:curator-client:jar:4.2.0:compile
[INFO] | | +- org.apache.commons:commons-compress:jar:1.21:compile
[INFO] | | +- org.apache.kerby:kerb-core:jar:1.0.1:compile
[INFO] | | | \- org.apache.kerby:kerby-pkix:jar:1.0.1:compile
[INFO] | | | +- org.apache.kerby:kerby-asn1:jar:1.0.1:compile
[INFO] | | | \- org.apache.kerby:kerby-util:jar:1.0.1:compile
[INFO] | | +- org.codehaus.woodstox:stax2-api:jar:4.2.1:compile
[INFO] | | +- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:compile
[INFO] | | \- dnsjava:dnsjava:jar:2.1.7:compile
[INFO] | +- org.apache.hadoop:hadoop-hdfs-client:jar:3.3.2:compile
[INFO] | | \- com.squareup.okhttp:okhttp:jar:2.7.5:compile
[INFO] | | \- com.squareup.okio:okio:jar:1.6.0:compile
[INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.3.2:compile
[INFO] | +- org.apache.hadoop:hadoop-yarn-client:jar:3.3.2:compile
[INFO] | | \- org.jline:jline:jar:3.9.0:compile
[INFO] | \- org.apache.hadoop:hadoop-annotations:jar:3.3.2:compile
[INFO] \- commons-configuration:commons-configuration:jar:1.10:compile
[INFO] \- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] --------------------------------------------------------------------
how do I resolve snakeyaml error?
So, you had been using Snakeyaml with an earlier Spring Boot version, but with a newer Spring Boot version, Snakeyaml stopped working. This very much seems to be a version incompatibility issue.
Here: https://search.maven.org/artifact/org.springframework.boot/spring-boot-starter/2.1.15.RELEASE/jar
we can see this structure:
<?xml version="1.0" encoding="UTF-8"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starters</artifactId>
<version>2.1.15.RELEASE</version>
</parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<version>2.1.15.RELEASE</version>
<name>Spring Boot Starter</name>
<description>Core starter, including auto-configuration support, logging and YAML</description>
<url>https://projects.spring.io/spring-boot/#/spring-boot-parent/spring-boot-starters/spring-boot-starter</url>
<organization>
<name>Pivotal Software, Inc.</name>
<url>https://spring.io</url>
</organization>
<licenses>
<license>
<name>Apache License, Version 2.0</name>
<url>https://www.apache.org/licenses/LICENSE-2.0</url>
</license>
</licenses>
<developers>
<developer>
<name>Pivotal</name>
<email>info#pivotal.io</email>
<organization>Pivotal Software, Inc.</organization>
<organizationUrl>https://www.spring.io</organizationUrl>
</developer>
</developers>
<scm>
<connection>scm:git:git://github.com/spring-projects/spring-boot.git</connection>
<developerConnection>scm:git:ssh://git#github.com/spring-projects/spring-boot.git</developerConnection>
<url>https://github.com/spring-projects/spring-boot</url>
</scm>
<issueManagement>
<system>Github</system>
<url>https://github.com/spring-projects/spring-boot/issues</url>
</issueManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot</artifactId>
<version>2.1.15.RELEASE</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-autoconfigure</artifactId>
<version>2.1.15.RELEASE</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
<version>2.1.15.RELEASE</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>javax.annotation</groupId>
<artifactId>javax.annotation-api</artifactId>
<version>1.3.2</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>5.1.16.RELEASE</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
<version>1.23</version>
<scope>runtime</scope>
</dependency>
</dependencies>
</project>
where the interesting part from our perspective is:
<artifactId>snakeyaml</artifactId>
<version>1.23</version>
so, you will likely need to ensure that your Snakeyaml version is 1.23.
We can create a software solution for the above case by separating the above functionalities in three objects as below. 1) The Snake Box Factory The factory will produce the Snake Boxes. The factory could be
My code seems to be breaking on a transitive dependency issue that I cannot find. I am trying to migrate from JBoss EAP 7.3 with RedHat's PAM 7.9 product (which is JBPM 7.48) to JBoss EAP 7.4 with PAM 7.12 (ie JBPM 7.59) and am experiencing the issue in a servlet of mine that calls the kie-server via a kie-client API. I have read about and encountered RESTEasy issues before with JBoss, so based upon my error have removed the tags to hopefully avoid any inconsistencies - but still have an odd error.
This is the stack trace that I get which has been edited for brevity:
5:37:57,975 ERROR [io.undertow.request] (default task-4) UT005023: Exception handling request to /PamHelper/pam/helper/pamQuery: org.jboss.resteasy.spi.UnhandledException: java.lang.NoSuchFieldError: _typeCache
at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:82)
at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:346)
at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:193)
...
at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
...
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
...
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:841)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchFieldError: _typeCache
at org.kie.server.api.marshalling.json.FallbackableTypeFactory.withFallbackClassLoader(FallbackableTypeFactory.java:62)
at org.kie.server.api.marshalling.json.JSONMarshaller.configureMarshaller(JSONMarshaller.java:284)
at org.kie.server.api.marshalling.json.JSONMarshaller.<init>(JSONMarshaller.java:180)
at org.kie.server.api.marshalling.BaseMarshallerBuilder.build(BaseMarshallerBuilder.java:45)
at org.kie.server.api.marshalling.MarshallerFactory.getMarshaller(MarshallerFactory.java:50)
at org.kie.server.client.impl.AbstractKieServicesClientImpl.<init>(AbstractKieServicesClientImpl.java:84)
at org.kie.server.client.impl.KieServicesClientImpl.<init>(KieServicesClientImpl.java:74)
at org.kie.server.client.KieServicesFactory.newKieServicesClient(KieServicesFactory.java:101)
This is my dependency tree. I have guessing there is a conflict either in a RestEasy dependency or (from the error message) some odd problem initializing marshalling; there is no typeCache field either in my code or the JBPM source code.
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) # PamServer ---
[INFO] com.gok.ams:PamServer:war:1.0-SNAPSHOT
[INFO] +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.12.3:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.3:compile
[INFO] | \- com.fasterxml.jackson.core:jackson-core:jar:2.12.3:compile
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.3:compile
[INFO] +- jakarta.persistence:jakarta.persistence-api:jar:2.2.3:compile
[INFO] +- com.gok.ams:ams-pam-util:jar:0.0.12-SNAPSHOT:compile
[INFO] | +- org.hibernate:hibernate-core:jar:5.5.8.Final-redhat-00001:compile
[INFO] | | +- javax.persistence:javax.persistence-api:jar:2.2:compile
[INFO] | | +- org.javassist:javassist:jar:3.27.0.GA-redhat-00001:compile
[INFO] | | +- net.bytebuddy:byte-buddy:jar:1.11.12.redhat-00002:compile
[INFO] | | +- antlr:antlr:jar:2.7.7.redhat-7:compile
[INFO] | | +- org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:jar:1.1.1.Final:compile
[INFO] | | +- org.jboss:jandex:jar:2.2.3.Final-redhat-00001:compile
[INFO] | | +- com.fasterxml:classmate:jar:1.5.1.redhat-00001:compile
[INFO] | | +- javax.activation:javax.activation-api:jar:1.2.0:compile
[INFO] | | +- org.hibernate.common:hibernate-commons-annotations:jar:5.1.2.Final-redhat-00003:compile
[INFO] | | +- javax.xml.bind:jaxb-api:jar:2.3.1.redhat-00001:compile
[INFO] | | \- org.glassfish.jaxb:jaxb-runtime:jar:2.3.1.redhat-00002:compile
[INFO] | | +- org.glassfish.jaxb:txw2:jar:2.3.1.redhat-00002:compile
[INFO] | | +- com.sun.istack:istack-commons-runtime:jar:3.0.7.redhat-00001:compile
[INFO] | | +- org.jvnet.staxex:stax-ex:jar:1.8:compile
[INFO] | | \- com.sun.xml.fastinfoset:FastInfoset:jar:1.2.15:compile
[INFO] | \- org.apache.commons:commons-lang3:jar:3.12.0:compile
[INFO] +- com.gok.ams:ams-pam-workitemhandlers:jar:2.0.6.41-SNAPSHOT:compile
[INFO] | \- com.gok.ams:amsObjectModel:jar:0.0.65-SNAPSHOT:compile
[INFO] +- org.jbpm:jbpm-workitems-core:jar:7.59.0.Final-redhat-00006:compile
[INFO] | +- org.drools:drools-core:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | +- org.kie.soup:kie-soup-xstream:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | +- org.drools:drools-core-reflective:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | \- org.kie:kie-memory-compiler:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | \- org.drools:drools-core-dynamic:jar:7.59.0.Final-redhat-00006:runtime
[INFO] | +- org.jbpm:jbpm-flow:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | +- org.drools:drools-mvel:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | +- org.drools:drools-serialization-protobuf:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | +- org.kie.soup:kie-soup-commons:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | +- org.kie.soup:kie-soup-project-datamodel-commons:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | \- org.kie.soup:kie-soup-project-datamodel-api:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | +- org.kie:kie-dmn-feel:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.antlr:antlr4-runtime:jar:4.8.0.redhat-00002:compile
[INFO] | | | +- com.github.javaparser:javaparser-core:jar:3.13.10:compile
[INFO] | | | +- org.drools:drools-mvel-parser:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | \- ch.obermuhlner:big-math:jar:2.0.1:compile
[INFO] | | +- org.kie:kie-dmn-core:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-dmn-backend:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.drools:drools-ruleunit:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.drools:drools-alphanetwork-compiler:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | \- org.jpmml:pmml-model:jar:1.5.1:compile
[INFO] | | | \- org.jpmml:pmml-agent:jar:1.5.1:compile
[INFO] | | +- com.google.protobuf:protobuf-java:jar:3.6.1:compile
[INFO] | | \- org.mvel:mvel2:jar:2.4.12.Final-redhat-00001:compile
[INFO] | +- org.jbpm:jbpm-bpmn2:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | +- org.drools:drools-compiler:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | +- org.drools:drools-ecj:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | \- org.jbpm:jbpm-flow-builder:jar:7.59.0.Final-redhat-00006:compile
[INFO] | +- org.kie:kie-internal:jar:7.59.0.Final-redhat-00006:compile
[INFO] | +- org.kie:kie-dmn-api:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | \- org.kie:kie-dmn-model:jar:7.59.0.Final-redhat-00006:compile
[INFO] | +- commons-io:commons-io:jar:2.7.0.redhat-00003:compile
[INFO] | +- org.apache.commons:commons-exec:jar:1.3:compile
[INFO] | \- org.antlr:ST4:jar:4.0.7:compile
[INFO] | \- org.antlr:antlr-runtime:jar:3.5:compile
[INFO] | \- org.antlr:stringtemplate:jar:3.2.1:compile
[INFO] +- org.jbpm:jbpm-services-api:jar:7.59.0.Final-redhat-00006:compile
[INFO] +- org.kie:kie-api:jar:7.59.0.Final-redhat-00006:compile
[INFO] | \- org.kie.soup:kie-soup-maven-support:jar:7.59.0.Final-redhat-00006:compile
[INFO] +- org.kie.server:kie-server-client:jar:7.59.0.Final-redhat-00006:compile
[INFO] | +- org.optaplanner:optaplanner-core:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | +- org.drools:drools-canonical-model:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | +- org.drools:drools-model-compiler:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | \- org.drools:drools-mvel-compiler:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | +- org.apache.commons:commons-math3:jar:3.4.1.redhat-3:compile
[INFO] | | +- org.reflections:reflections:jar:0.9.11.redhat-00004:compile
[INFO] | | | \- com.google.guava:guava:jar:25.0.0.redhat-1:compile
[INFO] | | | +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] | | | +- org.checkerframework:checker-compat-qual:jar:2.0.0:compile
[INFO] | | | +- com.google.errorprone:error_prone_annotations:jar:2.1.3:compile
[INFO] | | | +- com.google.j2objc:j2objc-annotations:jar:1.1:compile
[INFO] | | | \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.14:compile
[INFO] | | \- com.thoughtworks.xstream:xstream:jar:1.4.18.redhat-00001:compile
[INFO] | | \- io.github.x-stream:mxparser:jar:1.2.2:compile
[INFO] | | \- xmlpull:xmlpull:jar:1.1.3.1:compile
[INFO] | +- org.kie.server:kie-server-api:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | +- org.optaplanner:optaplanner-persistence-xstream:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | \- org.optaplanner:optaplanner-persistence-common:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | +- org.optaplanner:optaplanner-persistence-jaxb:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | +- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.12.1.redhat-00002:compile
[INFO] | | +- org.kie:kie-pmml-dependencies:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-api:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-commons:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | | \- org.apache.commons:commons-text:jar:1.9:compile
[INFO] | | | +- org.kie:kie-pmml-compiler-api:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-compiler-core:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-compiler-commons:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-evaluator-api:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-evaluator-core:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-evaluator-assembler:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-kie-internal:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-models-drools-common:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-models-regression-model:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-models-regression-compiler:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-models-regression-evaluator:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-models-drools-tree-model:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-models-drools-tree-compiler:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-models-drools-tree-evaluator:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | | \- org.kie:kie-pmml-compiler-api:jar:tests:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-models-drools-scorecard-model:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-models-drools-scorecard-compiler:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-models-drools-scorecard-evaluator:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-models-mining-model:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-models-mining-compiler:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-models-mining-evaluator:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-models-clustering-model:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | +- org.kie:kie-pmml-models-clustering-compiler:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | | \- org.kie:kie-pmml-models-clustering-evaluator:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | \- org.drools:kie-pmml:jar:7.59.0.Final-redhat-00006:compile
[INFO] | +- jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6.redhat-00001:compile
[INFO] | +- org.kie.server:kie-server-common:jar:7.59.0.Final-redhat-00006:compile
[INFO] | | \- jakarta.servlet:jakarta.servlet-api:jar:4.0.3.redhat-00002:compile
[INFO] | +- jakarta.jms:jakarta.jms-api:jar:2.0.3.redhat-00001:compile
[INFO] | +- com.sun.xml.bind:jaxb-core:jar:2.3.0.redhat-1:compile
[INFO] | +- com.sun.xml.bind:jaxb-impl:jar:2.3.0.redhat-1:compile
[INFO] | \- org.slf4j:jcl-over-slf4j:jar:1.7.30.redhat-00001:runtime
[INFO] +- org.slf4j:slf4j-api:jar:1.7.26:compile
[INFO] +- org.jboss.resteasy:resteasy-jaxrs:jar:3.0.14.Final:compile
[INFO] | +- org.jboss.spec.javax.ws.rs:jboss-jaxrs-api_2.0_spec:jar:1.0.0.Final:compile
[INFO] | +- org.jboss.spec.javax.annotation:jboss-annotations-api_1.2_spec:jar:1.0.0.Final:compile
[INFO] | +- javax.activation:activation:jar:1.1.1:compile
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.3.6:compile
[INFO] | | +- org.apache.httpcomponents:httpcore:jar:4.3.3:compile
[INFO] | | \- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] | +- net.jcip:jcip-annotations:jar:1.0:compile
[INFO] | \- org.jboss.logging:jboss-logging:jar:3.1.4.GA:compile
[INFO] +- org.jboss.resteasy:resteasy-servlet-initializer:jar:6.0.0.Final:compile
[INFO] | +- org.jboss.resteasy:resteasy-core:jar:6.0.0.Final:compile
[INFO] | | +- jakarta.annotation:jakarta.annotation-api:jar:2.0.0:compile
[INFO] | | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:3.0.1:compile
[INFO] | | +- jakarta.activation:jakarta.activation-api:jar:2.0.0:compile
[INFO] | | +- jakarta.validation:jakarta.validation-api:jar:3.0.0:compile
[INFO] | | \- com.ibm.async:asyncutil:jar:0.1.0:compile
[INFO] | \- org.jboss.spec.javax.ws.rs:jboss-jaxrs-api_3.0_spec:jar:1.0.0.Final:compile
[INFO] +- org.jboss.resteasy:resteasy-client:jar:6.0.0.Final:compile
[INFO] | +- org.jboss.resteasy:resteasy-client-api:jar:6.0.0.Final:compile
[INFO] | +- org.jboss.resteasy:resteasy-core-spi:jar:6.0.0.Final:compile
[INFO] | +- commons-codec:commons-codec:jar:1.15:compile
[INFO] | \- org.reactivestreams:reactive-streams:jar:1.0.3:compile
[INFO] +- junit:junit:jar:4.13.2:test
[INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] \- org.mockito:mockito-all:jar:1.10.16:test
And this is the dependency part of my pom.xml not related to testing.
<properties>
<version.org.kie>7.59.0.Final-redhat-00006</version.org.kie>
<!--version.org.kie>7.48.0.Final-redhat-00004</version.org.kie-->
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
<version.org.slf4j>1.7.26</version.org.slf4j>
<jax-rs-version>3.0.14.Final</jax-rs-version>
<version.object.model>0.0.65-SNAPSHOT</version.object.model>
<version.utils>0.0.12-SNAPSHOT</version.utils>
<version.workItemHandlers>2.0.6.41-SNAPSHOT</version.workItemHandlers>
<version.jakarta.persistence-api>2.2.3</version.jakarta.persistence-api>
<version.resteasy>6.0.0.Final</version.resteasy>
<!--version.jackson>2.12.4.redhat-00001</version.jackson-->
<version.jackson>2.12.3</version.jackson>
</properties>
<dependencies>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
<version>${version.jackson}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${version.jackson}</version>
<!--scope>provided</scope-->
</dependency>
<dependency>
<groupId>jakarta.persistence</groupId>
<artifactId>jakarta.persistence-api</artifactId>
<version>${version.jakarta.persistence-api}</version>
</dependency>
<dependency>
<groupId>com.gok.ams</groupId>
<artifactId>ams-pam-util</artifactId>
<version>${version.utils}</version>
</dependency>
<dependency>
<artifactId>ams-pam-workitemhandlers</artifactId>
<version>${version.workItemHandlers}</version>
<groupId>com.gok.ams</groupId>
</dependency>
<dependency>
<groupId>org.jbpm</groupId>
<artifactId>jbpm-workitems-core</artifactId>
<version>${version.org.kie}</version>
</dependency>
<dependency>
<groupId>org.jbpm</groupId>
<artifactId>jbpm-services-api</artifactId>
<version>${version.org.kie}</version>
</dependency>
<dependency>
<groupId>org.kie</groupId>
<artifactId>kie-api</artifactId>
<version>${version.org.kie}</version>
</dependency>
<dependency>
<groupId>org.kie.server</groupId>
<artifactId>kie-server-client</artifactId>
<version>${version.org.kie}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${version.org.slf4j}</version>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jaxrs</artifactId>
<version>3.0.14.Final</version>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-servlet-initializer</artifactId>
<version>${version.resteasy}</version>
<exclusions>
<exclusion>
<artifactId>resteasy-jaxrs</artifactId>
<groupId>org.jboss.resteasy</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-client</artifactId>
<version>${version.resteasy}</version>
</dependency>
The other possibility that I can think of is that this is related to this issue which may not have been fixed in PAM ; I have downgraded from Java 11 to Java 8 to have my the code running on my laptop look more like my company's integration environment.
You've got a dependency on two different versions of RESTEasy. You don't want to use 6.0.0.Final as that is Jakarta REST 3.0 which is Jakarta EE 9.1. You want to use the latest 3.15.x version as that is what JBoss EAP 7.4 uses. You also want to make sure you mark those dependencies as <scope>provided</scope> as they are provided by JBoss EAP 7.4.
I don't think you need the org.jboss.resteasy:resteasy-servlet-initializer at either.
I have a java spark process which I call using -
spark-submit --class MyClass target/MyJar.jar
The last part of this process is to write it locally and then copy it to s3 as it need to have specific name (can also be written to s3 and mv'ed there, for the sake of the question the error stays the same).
The code compiles and runs but when it get to the code snippet below I get the following error -
java.lang.NoClassDefFoundError: com/amazonaws/services/s3/AmazonS3 at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:348) at
org.apache.spark.util.Utils$.classForName(Utils.scala:229) at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:700)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused
by: java.lang.ClassNotFoundException:
com.amazonaws.services.s3.AmazonS3 at
java.net.URLClassLoader.findClass(URLClassLoader.java:381) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
java.lang.ClassLoader.loadClass(ClassLoader.java:357)
public static void saveToS3(Dataset<Row> df, String outputBucket, String outputPath) throws IOException {
String tmpFile = "temp" + Long.toString(System.nanoTime());
df.coalesce(1).write().option("header", true).csv(tmpFile);
File directory = new File(tmpFile);
AmazonS3 s3client = new AmazonS3Client();
for (File file : directory.listFiles()) {
if (file.getName().startsWith("part_") || file.getName().endsWith("csv")) {
s3client.putObject(new PutObjectRequest(outputBucket, getS3path(outputPath), file));
}
file.delete();
}
directory.delete();
}
I use the following dependencies which might be relevant
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.7.4</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>2.7.2</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.mortbay.jetty</groupId>
<artifactId>servlet-api-2.5</artifactId>
</exclusion>
</exclusions>
</dependency>
Any idea?
Dependency tree -
[INFO] +- org.apache.spark:spark-core_2.11:jar:2.1.0:compile
[INFO] | +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile
[INFO] | | +- org.apache.avro:avro-ipc:jar:1.7.7:compile
[INFO] | | +- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile
[INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] | | \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] | +- com.twitter:chill_2.11:jar:0.8.0:compile
[INFO] | | \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile
[INFO] | | +- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] | | \- org.objenesis:objenesis:jar:2.1:compile
[INFO] | +- com.twitter:chill-java:jar:0.8.0:compile
[INFO] | +- org.apache.xbean:xbean-asm5-shaded:jar:4.4:compile
[INFO] | +- org.apache.hadoop:hadoop-client:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:compile
[INFO] | | | +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:compile
[INFO] | | | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
[INFO] | | | | | \- com.google.inject:guice:jar:3.0:compile
[INFO] | | | | | +- javax.inject:javax.inject:jar:1:compile
[INFO] | | | | | \- aopalliance:aopalliance:jar:1.0:compile
[INFO] | | | | \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile
[INFO] | | | \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:compile
[INFO] | | | \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:compile
[INFO] | | \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile
[INFO] | +- org.apache.spark:spark-launcher_2.11:jar:2.1.0:compile
[INFO] | +- org.apache.spark:spark-network-common_2.11:jar:2.1.0:compile
[INFO] | | \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO] | +- org.apache.spark:spark-network-shuffle_2.11:jar:2.1.0:compile
[INFO] | +- org.apache.spark:spark-unsafe_2.11:jar:2.1.0:compile
[INFO] | +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
[INFO] | | \- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] | +- org.apache.curator:curator-recipes:jar:2.4.0:compile
[INFO] | | +- org.apache.curator:curator-framework:jar:2.4.0:compile
[INFO] | | +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
[INFO] | | \- com.google.guava:guava:jar:14.0.1:compile
[INFO] | +- javax.servlet:javax.servlet-api:jar:3.1.0:compile
[INFO] | +- org.apache.commons:commons-lang3:jar:3.5:compile
[INFO] | +- org.apache.commons:commons-math3:jar:3.4.1:compile
[INFO] | +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.16:compile
[INFO] | +- org.slf4j:jul-to-slf4j:jar:1.7.16:compile
[INFO] | +- org.slf4j:jcl-over-slf4j:jar:1.7.16:compile
[INFO] | +- com.ning:compress-lzf:jar:1.0.3:compile
[INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
[INFO] | +- net.jpountz.lz4:lz4:jar:1.3.0:compile
[INFO] | +- org.roaringbitmap:RoaringBitmap:jar:0.5.11:compile
[INFO] | +- commons-net:commons-net:jar:2.2:compile
[INFO] | +- org.scala-lang:scala-library:jar:2.11.8:compile
[INFO] | +- org.json4s:json4s-jackson_2.11:jar:3.2.11:compile
[INFO] | | \- org.json4s:json4s-core_2.11:jar:3.2.11:compile
[INFO] | | +- org.json4s:json4s-ast_2.11:jar:3.2.11:compile
[INFO] | | +- com.thoughtworks.paranamer:paranamer:jar:2.6:compile
[INFO] | | \- org.scala-lang:scalap:jar:2.11.0:compile
[INFO] | | \- org.scala-lang:scala-compiler:jar:2.11.0:compile
[INFO] | | \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.1:compile
[INFO] | +- org.glassfish.jersey.core:jersey-client:jar:2.22.2:compile
[INFO] | | +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:compile
[INFO] | | +- org.glassfish.hk2:hk2-api:jar:2.4.0-b34:compile
[INFO] | | | +- org.glassfish.hk2:hk2-utils:jar:2.4.0-b34:compile
[INFO] | | | \- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.4.0-b34:compile
[INFO] | | +- org.glassfish.hk2.external:javax.inject:jar:2.4.0-b34:compile
[INFO] | | \- org.glassfish.hk2:hk2-locator:jar:2.4.0-b34:compile
[INFO] | | \- org.javassist:javassist:jar:3.18.1-GA:compile
[INFO] | +- org.glassfish.jersey.core:jersey-common:jar:2.22.2:compile
[INFO] | | +- javax.annotation:javax.annotation-api:jar:1.2:compile
[INFO] | | +- org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.22.2:compile
[INFO] | | \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile
[INFO] | +- org.glassfish.jersey.core:jersey-server:jar:2.22.2:compile
[INFO] | | +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.22.2:compile
[INFO] | | \- javax.validation:validation-api:jar:1.1.0.Final:compile
[INFO] | +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.22.2:compile
[INFO] | +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.22.2:compile
[INFO] | +- io.netty:netty-all:jar:4.0.42.Final:compile
[INFO] | +- io.netty:netty:jar:3.8.0.Final:compile
[INFO] | +- com.clearspring.analytics:stream:jar:2.7.0:compile
[INFO] | +- io.dropwizard.metrics:metrics-core:jar:3.1.2:compile
[INFO] | +- io.dropwizard.metrics:metrics-jvm:jar:3.1.2:compile
[INFO] | +- io.dropwizard.metrics:metrics-json:jar:3.1.2:compile
[INFO] | +- io.dropwizard.metrics:metrics-graphite:jar:3.1.2:compile
[INFO] | +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.6.5:compile
[INFO] | | +- org.scala-lang:scala-reflect:jar:2.11.7:compile
[INFO] | | \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.6.5:compile
[INFO] | +- org.apache.ivy:ivy:jar:2.4.0:compile
[INFO] | +- oro:oro:jar:2.0.8:compile
[INFO] | +- net.razorvine:pyrolite:jar:4.13:compile
[INFO] | +- net.sf.py4j:py4j:jar:0.10.4:compile
[INFO] | +- org.apache.spark:spark-tags_2.11:jar:2.1.0:compile
[INFO] | | \- org.scalatest:scalatest_2.11:jar:2.2.6:compile
[INFO] | | \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.2:compile
[INFO] | +- org.apache.commons:commons-crypto:jar:1.0.0:compile
[INFO] | \- org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] +- org.apache.hadoop:hadoop-aws:jar:2.7.2:compile
[INFO] | \- org.apache.hadoop:hadoop-common:jar:2.7.2:compile
[INFO] | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | +- commons-io:commons-io:jar:2.4:compile
[INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | +- javax.servlet.jsp:jsp-api:jar:2.1:runtime
[INFO] | +- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO] | +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] | | | \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO] | | | +- javax.xml.stream:stax-api:jar:1.0-2:compile
[INFO] | | | \- javax.activation:activation:jar:1.1:compile
[INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
[INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
[INFO] | +- com.sun.jersey:jersey-server:jar:1.9:compile
[INFO] | | \- asm:asm:jar:3.1:compile
[INFO] | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] | | +- commons-digester:commons-digester:jar:1.8:compile
[INFO] | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] | +- org.apache.avro:avro:jar:1.7.4:compile
[INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] | +- com.google.code.gson:gson:jar:2.2.4:compile
[INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.7.2:compile
[INFO] | | \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] | | +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] | | +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] | | \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] | +- com.jcraft:jsch:jar:0.1.42:compile
[INFO] | +- org.apache.curator:curator-client:jar:2.7.1:compile
[INFO] | +- org.apache.htrace:htrace-core:jar:3.1.0-incubating:compile
[INFO] | \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] | \- org.tukaani:xz:jar:1.0:compile
[INFO] +- org.apache.spark:spark-sql_2.10:jar:2.1.0:compile
[INFO] | +- com.univocity:univocity-parsers:jar:2.2.1:compile
[INFO] | +- org.apache.spark:spark-sketch_2.10:jar:2.1.0:compile
[INFO] | +- org.apache.spark:spark-core_2.10:jar:2.1.0:compile
[INFO] | | +- com.twitter:chill_2.10:jar:0.8.0:compile
[INFO] | | +- org.apache.spark:spark-launcher_2.10:jar:2.1.0:compile
[INFO] | | +- org.apache.spark:spark-network-common_2.10:jar:2.1.0:compile
[INFO] | | +- org.apache.spark:spark-network-shuffle_2.10:jar:2.1.0:compile
[INFO] | | +- org.apache.spark:spark-unsafe_2.10:jar:2.1.0:compile
[INFO] | | +- org.json4s:json4s-jackson_2.10:jar:3.2.11:compile
[INFO] | | | \- org.json4s:json4s-core_2.10:jar:3.2.11:compile
[INFO] | | | \- org.json4s:json4s-ast_2.10:jar:3.2.11:compile
[INFO] | | \- com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.6.5:compile
[INFO] | +- org.apache.spark:spark-catalyst_2.10:jar:2.1.0:compile
[INFO] | | +- org.codehaus.janino:janino:jar:3.0.0:compile
[INFO] | | +- org.codehaus.janino:commons-compiler:jar:3.0.0:compile
[INFO] | | \- org.antlr:antlr4-runtime:jar:4.5.3:compile
[INFO] | +- org.apache.spark:spark-tags_2.10:jar:2.1.0:compile
[INFO] | | \- org.scalatest:scalatest_2.10:jar:2.2.6:compile
[INFO] | +- org.apache.parquet:parquet-column:jar:1.8.1:compile
[INFO] | | +- org.apache.parquet:parquet-common:jar:1.8.1:compile
[INFO] | | \- org.apache.parquet:parquet-encoding:jar:1.8.1:compile
[INFO] | \- org.apache.parquet:parquet-hadoop:jar:1.8.1:compile
[INFO] | +- org.apache.parquet:parquet-format:jar:2.3.0-incubating:compile
[INFO] | \- org.apache.parquet:parquet-jackson:jar:1.8.1:compile
[INFO] +- com.amazonaws:aws-java-sdk:jar:1.7.4:compile
[INFO] | +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.2:compile
[INFO] | | \- org.apache.httpcomponents:httpcore:jar:4.2:compile
[INFO] | +- commons-codec:commons-codec:jar:1.3:compile
[INFO] | \- joda-time:joda-time:jar:2.9.9:compile (version selected from constraint [2.2,))
[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.7.3:compile
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.7.3:compile
[INFO] \- com.fasterxml.jackson.core:jackson-annotations:jar:2.7.3:compile
You appear to be using a really old version of AWS SDK (1.7.4) from 2014. Your apache spark is 2.1, which is from 2016.
Commit history for the AmazonS3Client only goes back to 1.9, so it very well may not exist in your AWS SDK.
The reason your code compiles is that your code itself isn't the problem, but one of your dependencies is trying to use a more recent version of S3 client that your old SDK doesn't have. In other words, this isn't a code problem, it's a dependency management problem. If you're not directly using the SDK at all, it's safe to use a more appropriate AWS SDK 1.11.X, if you are, you'll have to coax your code to that point.
Typically, the spark-core module would point out its dependency to a version of aws, but I assume it was specifically excluded from its own so you could provide your own version of the SDK for your use without any sort of conflict, with the caveat that if you use too old of a dependency, it just breaks.
After massaging my pom file I was able to get to a stable with the following pom file and packaging with -
mvn clean compile assembly:single;
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.group</groupId>
<artifactId>arifact</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<properties>
<java.version>1.8</java.version>
<jdk.version>1.8</jdk.version>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>2.8.2</version>
</dependency>
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.1.1</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.11.229</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>${jdk.version}</source>
<target>${jdk.version}</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>com.group.artifact.MainClass</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>
</project>
i am trying to write to hdfs, i am sure i have same version hdfs clients in my pom.xml, some other dependencies are causing the issue. following are the details. i am using CDH 5.5.1 hadoop 2.6.0
if i just specify
conf.set("fs.defaultFS",
"localhost:8020");
it throws
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost:8020/user/hdfs, expected: file:///
if i specify
conf.set("fs.default.name",
"localhost:8020");
it throws
org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
at org.apache.hadoop.ipc.Client.call(Client.java:1066)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at com.sun.proxy.$Proxy3.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:122)
maven dependency tree
+- junit:junit:jar:4.12:compile
[INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:compile
[INFO] +- com.googlecode.protobuf-java-format:protobuf-java-format:jar:1.4:compile
[INFO] +- org.apache.logging.log4j:log4j-core:jar:2.3:compile
[INFO] | \- org.apache.logging.log4j:log4j-api:jar:2.3:compile
[INFO] +- org.apache.avro:avro-tools:jar:1.7.7:compile
[INFO] | \- org.slf4j:slf4j-api:jar:1.6.4:compile
[INFO] +- org.apache.spark:spark-core_2.11:jar:1.5.0:compile
[INFO] | +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile
[INFO] | | +- org.apache.avro:avro-ipc:jar:1.7.7:compile
[INFO] | | +- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile
[INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] | | \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] | +- com.twitter:chill_2.11:jar:0.5.0:compile
[INFO] | | \- com.esotericsoftware.kryo:kryo:jar:2.21:compile
[INFO] | | +- com.esotericsoftware.reflectasm:reflectasm:jar:shaded:1.07:compile
[INFO] | | +- com.esotericsoftware.minlog:minlog:jar:1.2:compile
[INFO] | | \- org.objenesis:objenesis:jar:1.2:compile
[INFO] | +- com.twitter:chill-java:jar:0.5.0:compile
[INFO] | +- org.apache.spark:spark-launcher_2.11:jar:1.5.0:compile
[INFO] | +- org.apache.spark:spark-network-common_2.11:jar:1.5.0:compile
[INFO] | +- org.apache.spark:spark-network-shuffle_2.11:jar:1.5.0:compile
[INFO] | +- org.apache.spark:spark-unsafe_2.11:jar:1.5.0:compile
[INFO] | +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
[INFO] | | +- commons-codec:commons-codec:jar:1.3:compile
[INFO] | | \- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] | +- org.apache.curator:curator-recipes:jar:2.4.0:compile
[INFO] | | +- org.apache.curator:curator-framework:jar:2.4.0:compile
[INFO] | | \- com.google.guava:guava:jar:14.0.1:compile
[INFO] | +- org.eclipse.jetty.orbit:javax.servlet:jar:3.0.0.v201112011016:compile
[INFO] | +- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO] | +- org.apache.commons:commons-math3:jar:3.4.1:compile
[INFO] | +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] | +- org.slf4j:jul-to-slf4j:jar:1.7.10:compile
[INFO] | +- org.slf4j:jcl-over-slf4j:jar:1.7.10:compile
[INFO] | +- log4j:log4j:jar:1.2.17:compile
[INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.10:compile
[INFO] | +- com.ning:compress-lzf:jar:1.0.3:compile
[INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.1.7:compile
[INFO] | +- net.jpountz.lz4:lz4:jar:1.3.0:compile
[INFO] | +- org.roaringbitmap:RoaringBitmap:jar:0.4.5:compile
[INFO] | +- com.typesafe.akka:akka-remote_2.11:jar:2.3.11:compile
[INFO] | | +- com.typesafe.akka:akka-actor_2.11:jar:2.3.11:compile
[INFO] | | | \- com.typesafe:config:jar:1.2.1:compile
[INFO] | | +- io.netty:netty:jar:3.8.0.Final:compile
[INFO] | | \- org.uncommons.maths:uncommons-maths:jar:1.2.2a:compile
[INFO] | +- com.typesafe.akka:akka-slf4j_2.11:jar:2.3.11:compile
[INFO] | +- org.scala-lang:scala-library:jar:2.11.7:compile
[INFO] | +- org.json4s:json4s-jackson_2.11:jar:3.2.10:compile
[INFO] | | \- org.json4s:json4s-core_2.11:jar:3.2.10:compile
[INFO] | | +- org.json4s:json4s-ast_2.11:jar:3.2.10:compile
[INFO] | | \- org.scala-lang:scalap:jar:2.11.0:compile
[INFO] | | \- org.scala-lang:scala-compiler:jar:2.11.0:compile
[INFO] | | \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.1:compile
[INFO] | +- com.sun.jersey:jersey-server:jar:1.9:compile
[INFO] | | \- asm:asm:jar:3.1:compile
[INFO] | +- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO] | +- org.apache.mesos:mesos:jar:shaded-protobuf:0.21.1:compile
[INFO] | +- io.netty:netty-all:jar:4.0.29.Final:compile
[INFO] | +- com.clearspring.analytics:stream:jar:2.7.0:compile
[INFO] | +- io.dropwizard.metrics:metrics-core:jar:3.1.2:compile
[INFO] | +- io.dropwizard.metrics:metrics-jvm:jar:3.1.2:compile
[INFO] | +- io.dropwizard.metrics:metrics-json:jar:3.1.2:compile
[INFO] | +- io.dropwizard.metrics:metrics-graphite:jar:3.1.2:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.4.4:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.4.0:compile
[INFO] | | \- com.fasterxml.jackson.core:jackson-core:jar:2.4.4:compile
[INFO] | +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.4.4:compile
[INFO] | | +- org.scala-lang:scala-reflect:jar:2.11.2:compile
[INFO] | | \- com.thoughtworks.paranamer:paranamer:jar:2.6:compile
[INFO] | +- org.apache.ivy:ivy:jar:2.4.0:compile
[INFO] | +- oro:oro:jar:2.0.8:compile
[INFO] | +- org.tachyonproject:tachyon-client:jar:0.7.1:compile
[INFO] | | +- commons-lang:commons-lang:jar:2.4:compile
[INFO] | | +- org.apache.curator:curator-client:jar:2.1.0-incubating:compile
[INFO] | | +- org.tachyonproject:tachyon-underfs-hdfs:jar:0.7.1:compile
[INFO] | | \- org.tachyonproject:tachyon-underfs-local:jar:0.7.1:compile
[INFO] | +- net.razorvine:pyrolite:jar:4.4:compile
[INFO] | +- net.sf.py4j:py4j:jar:0.8.2.1:compile
[INFO] | \- org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] +- org.apache.spark:spark-sql_2.11:jar:1.5.0:provided
[INFO] | +- org.apache.spark:spark-catalyst_2.11:jar:1.5.0:provided
[INFO] | | \- org.codehaus.janino:janino:jar:2.7.8:provided
[INFO] | | \- org.codehaus.janino:commons-compiler:jar:2.7.8:provided
[INFO] | +- org.apache.parquet:parquet-column:jar:1.7.0:provided
[INFO] | | +- org.apache.parquet:parquet-common:jar:1.7.0:provided
[INFO] | | \- org.apache.parquet:parquet-encoding:jar:1.7.0:provided
[INFO] | | \- org.apache.parquet:parquet-generator:jar:1.7.0:provided
[INFO] | \- org.apache.parquet:parquet-hadoop:jar:1.7.0:provided
[INFO] | +- org.apache.parquet:parquet-format:jar:2.3.0-incubating:provided
[INFO] | \- org.apache.parquet:parquet-jackson:jar:1.7.0:provided
[INFO] +- org.bouncycastle:bcprov-jdk15on:jar:1.52:compile
[INFO] +- com.google.protobuf:protobuf-java:jar:3.0.0-beta-2:compile
[INFO] +- com.databricks:spark-avro_2.11:jar:2.0.1:compile
[INFO] | \- org.apache.avro:avro:jar:1.7.6:compile
[INFO] | \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] | \- org.tukaani:xz:jar:1.0:compile
[INFO] +- org.apache.hadoop:hadoop-client:jar:2.6.0-cdh5.5.1:compile
[INFO] | +- org.apache.hadoop:hadoop-common:jar:2.6.0-cdh5.5.1:compile
[INFO] | | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | | +- commons-net:commons-net:jar:3.1:compile
[INFO] | | +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] | | +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile
[INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-auth:jar:2.6.0-cdh5.5.1:compile
[INFO] | | | +- org.apache.httpcomponents:httpclient:jar:4.2.5:compile
[INFO] | | | | \- org.apache.httpcomponents:httpcore:jar:4.2.4:compile
[INFO] | | | \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] | | | +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] | | | +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] | | | \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] | | \- org.apache.htrace:htrace-core4:jar:4.0.1-incubating:compile
[INFO] | +- org.apache.hadoop:hadoop-hdfs:jar:2.6.0-cdh5.5.1:compile
[INFO] | | +- org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.4:compile
[INFO] | | +- xerces:xercesImpl:jar:2.9.1:compile
[INFO] | | | \- xml-apis:xml-apis:jar:1.3.04:compile
[INFO] | | \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.6.0-cdh5.5.1:compile
[INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.6.0-cdh5.5.1:compile
[INFO] | | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.6.0-cdh5.5.1:compile
[INFO] | | | \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.6.0-cdh5.5.1:compile
[INFO] | | \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.6.0-cdh5.5.1:compile
[INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:2.6.0-cdh5.5.1:compile
[INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.0-cdh5.5.1:compile
[INFO] | | \- org.apache.hadoop:hadoop-yarn-common:jar:2.6.0-cdh5.5.1:compile
[INFO] | | +- javax.servlet:servlet-api:jar:2.5:compile
[INFO] | | +- com.sun.jersey:jersey-client:jar:1.9:compile
[INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.8:compile
[INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.8.8:compile
[INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.6.0-cdh5.5.1:compile
[INFO] | +- org.apache.hadoop:hadoop-aws:jar:2.6.0-cdh5.5.1:compile
[INFO] | | \- com.amazonaws:aws-java-sdk:jar:1.7.4:compile
[INFO] | \- org.apache.hadoop:hadoop-annotations:jar:2.6.0-cdh5.5.1:compile
[INFO] +- org.apache.kafka:kafka_2.11:jar:0.9.0-kafka-2.0.0:compile
[INFO] | +- com.101tec:zkclient:jar:0.7:compile
[INFO] | +- com.yammer.metrics:metrics-core:jar:2.2.0:compile
[INFO] | +- net.sf.jopt-simple:jopt-simple:jar:4.9:compile
[INFO] | +- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:compile
[INFO] | \- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
[INFO] | \- jline:jline:jar:0.9.94:compile
[INFO] +- org.apache.kafka:kafka-clients:jar:0.9.0-kafka-2.0.0:compile
[INFO] +- commons-io:commons-io:jar:2.4:compile
[INFO] +- javax.xml.bind:jaxb-api:jar:2.2.11:compile
[INFO] +- org.jsoup:jsoup:jar:1.8.3:compile
[INFO] +- jdk.tools:jdk.tools:jar:1.7:compile
[INFO] +- commons-cli:commons-cli:jar:1.3.1:compile
[INFO] \- com.google.code.gson:gson:jar:2.5:compile
how do i resolve this issue.
Yes because of old hadoop-core.jar inside some dependency hierarchy cause this issue. After spending some time and research I found something useful:
Please use following dependencies in your pom.xml.
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>WordCount</groupId>
<artifactId>WordCount</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>3.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>3.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-common</artifactId>
<version>3.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-common</artifactId>
<version>3.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>3.2.0</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src</sourceDirectory>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
Sample java code which works well: AccessHDFS.java
package com.gpl.hadoop;
import java.io.InputStream;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
public class AccessHDFS {
public static void main(String args[]) {
String url = "hdfs://localhost:9000/user/hduser/input/test.txt";
FileSystem fs = null;
InputStream in = null;
try {
Configuration conf = new Configuration();
fs = FileSystem.get(URI.create(url), conf);
in = fs.open(new Path(url));
IOUtils.copyBytes(in, System.out, 4096, false);
} catch (Exception e) {
e.printStackTrace();
} finally {
IOUtils.closeStream(fs);
}
}
}
Alternatively you can add following repository with given dependency.
<repositories>
<repository> <id>mapr-releases</id>
<url>http://repository.mapr.com/maven/</url>
<snapshots><enabled>false</enabled></snapshots>
<releases><enabled>true</enabled></releases>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>2.4.1-mapr-1408</version>
</dependency>
</dependencies>
Make sure you do select Force Update of Snapshots/Releases option to update maven project from:
Project --> Maven --> Update Project --> Select Force Update of Snapshots/Releases --> ok
Check that you have the correct dependency and version in your case: 2.6.0
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.6.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.6.0</version>
</dependency>
This should resolve your error. You could also exclude, hadoop-core in the dependency. See this blog for more on the above solution.
My local environment: OSX 10.9.2, java1.6
I use java api to connect hbase and maven to manage my project, I added Hbase-0.94.17 and Hadoop-core-1.0.4 into pom.xml, when I ran my .java program to connect Hbase I got the following error:
SLF4J: slf4j-api 1.6.x (or later) is incompatible with this binding.
SLF4J: Your binding is version 1.5.5 or earlier.
SLF4J: Upgrade your binding to version 1.6.x.
Exception in thread "main" java.lang.NoSuchMethodError: org.slf4j.impl.StaticLoggerBinder.getSingleton()Lorg/slf4j/impl/StaticLoggerBinder;
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:128)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:107)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:295)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:269)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:281)
at org.apache.zookeeper.ZooKeeper.<clinit>(ZooKeeper.java:94)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:98)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:127)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:153)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:127)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1505)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:713)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:983)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:958)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:251)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:155)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:129)
at org.geogit.storage.hbase.MyLittleHBaseClient.main(MyLittleHBaseClient.java:29)
Here is what I got after ran mvm dependency:tree
INFO] --- maven-dependency-plugin:2.2:tree (default-cli) # geogit-hbase ---
[INFO] org.geogit:geogit-hbase:jar:0.8-SNAPSHOT
[INFO] +- org.geogit:geogit-core:jar:0.8-SNAPSHOT:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.5:compile
[INFO] | +- com.vividsolutions:jts:jar:1.13:compile
[INFO] | +- org.geotools:gt-opengis:jar:10.5:compile
[INFO] | | +- net.java.dev.jsr-275:jsr-275:jar:1.0-beta-2:compile
[INFO] | | +- java3d:vecmath:jar:1.3.2:compile
[INFO] | | +- commons-pool:commons-pool:jar:1.5.4:compile
[INFO] | | \- javax.media:jai_core:jar:1.1.3:compile
[INFO] | +- org.geotools:gt-referencing:jar:10.5:compile
[INFO] | | +- org.geotools:gt-metadata:jar:10.5:compile
[INFO] | | \- jgridshift:jgridshift:jar:1.0:compile
[INFO] | +- org.geotools:gt-epsg-hsql:jar:10.5:compile
[INFO] | | \- hsqldb:hsqldb:jar:1.8.0.7:compile
[INFO] | +- org.geotools:gt-main:jar:10.5:compile
[INFO] | | \- org.geotools:gt-api:jar:10.5:compile
[INFO] | +- org.geotools:gt-cql:jar:10.5:compile
[INFO] | +- net.sourceforge.findbugs:jsr305:jar:1.3.7:compile
[INFO] | +- com.google.inject:guice:jar:3.0:compile
[INFO] | | +- javax.inject:javax.inject:jar:1:compile
[INFO] | | \- aopalliance:aopalliance:jar:1.0:compile
[INFO] | +- com.google.inject.extensions:guice-multibindings:jar:3.0:compile
[INFO] | +- com.google.code.gson:gson:jar:2.2.2:compile
[INFO] | \- com.ning:compress-lzf:jar:0.9.8:compile
[INFO] +- org.apache.hbase:hbase-client:jar:0.98.0-hadoop2:compile
[INFO] | +- org.apache.hbase:hbase-common:jar:0.98.0-hadoop2:compile
[INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO] | | \- log4j:log4j:jar:1.2.17:compile
[INFO] | +- org.apache.hbase:hbase-protocol:jar:0.98.0-hadoop2:compile
[INFO] | +- commons-codec:commons-codec:jar:1.7:compile
[INFO] | +- commons-io:commons-io:jar:2.4:compile
[INFO] | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] | +- io.netty:netty:jar:3.6.6.Final:compile
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
[INFO] | | \- org.slf4j:slf4j-log4j12:jar:1.6.1:compile
[INFO] | +- org.cloudera.htrace:htrace-core:jar:2.04:compile
[INFO] | | \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
[INFO] | | \- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
[INFO] | +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
[INFO] | | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | | +- org.apache.commons:commons-math:jar:2.1:compile
[INFO] | | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | | +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] | | +- commons-net:commons-net:jar:3.1:compile
[INFO] | | +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] | | +- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO] | | +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO] | | | +- org.codehaus.jettison:jettison:jar:1.0.1:compile (version managed from 1.1)
[INFO] | | | | \- stax:stax-api:jar:1.0.1:compile
[INFO] | | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] | | | | \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO] | | | | \- javax.activation:activation:jar:1.1:compile
[INFO] | | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
[INFO] | | | \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
[INFO] | | +- commons-el:commons-el:jar:1.0:runtime
[INFO] | | +- net.java.dev.jets3t:jets3t:jar:0.6.1:compile
[INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile
[INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] | | +- org.apache.avro:avro:jar:1.7.4:compile
[INFO] | | | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] | | | \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile
[INFO] | | +- com.jcraft:jsch:jar:0.1.42:compile
[INFO] | | \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] | | \- org.tukaani:xz:jar:1.0:compile
[INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile
[INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
[INFO] | | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
[INFO] | | | +- com.sun.jersey:jersey-server:jar:1.9:compile
[INFO] | | | | \- asm:asm:jar:3.1:compile
[INFO] | | | \- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile
[INFO] | | \- com.google.inject.extensions:guice-servlet:jar:3.0:compile
[INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile
[INFO] | \- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] +- org.geogit:geogit-mongodb:jar:0.8-SNAPSHOT:compile
[INFO] | +- org.geogit:geogit-blueprints:jar:0.8-SNAPSHOT:compile
[INFO] | | +- com.tinkerpop.blueprints:blueprints-core:jar:2.4.0:compile
[INFO] | | | +- com.fasterxml.jackson.datatype:jackson-datatype-json-org:jar:2.1.2:compile
[INFO] | | | | +- com.fasterxml.jackson.core:jackson-core:jar:2.1.2:compile
[INFO] | | | | +- com.fasterxml.jackson.core:jackson-databind:jar:2.1.2:compile
[INFO] | | | | | \- com.fasterxml.jackson.core:jackson-annotations:jar:2.1.1:compile
[INFO] | | | | \- org.json:json:jar:20090211:compile
[INFO] | | | \- colt:colt:jar:1.2.0:compile
[INFO] | | | \- concurrent:concurrent:jar:1.3.4:compile
[INFO] | | \- com.tinkerpop.gremlin:gremlin-java:jar:2.4.0:compile
[INFO] | | \- com.tinkerpop:pipes:jar:2.4.0:compile
[INFO] | \- org.mongodb:mongo-java-driver:jar:2.11.3:compile
[INFO] +- com.google.guava:guava:jar:14.0.1:compile
[INFO] +- org.geogit:geogit-core:jar:tests:0.8-SNAPSHOT:test
[INFO] \- junit:junit:jar:4.10:test
[INFO] \- org.hamcrest:hamcrest-core:jar:1.1:test
And here I attach my pom.xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.geogit</groupId>
<artifactId>storage</artifactId>
<version>0.8-SNAPSHOT</version>
</parent>
<groupId>org.geogit</groupId>
<artifactId>geogit-hbase</artifactId>
<packaging>jar</packaging>
<name>HBase Storage for GeoGit objects</name>
<dependencies>
<dependency>
<groupId>org.geogit</groupId>
<artifactId>geogit-core</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase</artifactId>
<version>0.94.17</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.0.4</version>
</dependency>
<!-- is this necessary? -->
<!--
<dependency>
<groupId>com.boundlessgeo</groupId>
<artifactId>blongo</artifactId>
<version>0.1</version>
</dependency>
-->
<!-- is this necessary? -->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
<!-- Test scope dependencies -->
<dependency>
<groupId>org.geogit</groupId>
<artifactId>geogit-core</artifactId>
<version>${project.version}</version>
<scope>test</scope>
<classifier>tests</classifier>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
</plugins>
</build>
<profiles>
</profiles>
</project>
Have slf4j-api and choose a SLF4J binding of corresponding version, and add them as dependencies in your own POM. It is quite possibly some of your dependencies transitively resolve an incompatible version of SLF4J binding which caused the problem. Find which library is bringing in the unwanted SLF4J binding (you can easily do so by mvn dependency:tree), and add corresponding exclusion in your dependency to that lib.
(Normally SLF4J binding shouldn't be declared in library. It is in fact something wrong that you may want to report to the library developer once you found the actual cause)
Update
That's exactly what I am talking about:
From your dependency tree:
[INFO] +- org.geogit:geogit-core:jar:0.8-SNAPSHOT:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.5:compile
.....
[INFO] +- org.apache.hbase:hbase-client:jar:0.98.0-hadoop2:compile
.....
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
[INFO] | | \- org.slf4j:slf4j-log4j12:jar:1.6.1:compile
You got slf4j-log4j12 binding from hbase-client, and that's version 1.6.1
However, slf4j-api you have in your project is version 1.7.5. They are not compatible.
Proper way to solve is:
Exclude slf4j-log4j12 from your dependency of hbase-client
Declare a compatible SLF4J binding in your own project
You should also report this as a bug to zookeeper/hbase developer as it is not appropriate for a library to include SLF4J binding as compile scope dependency.