I m currently trying to write a Publisher for Kafka Messages using Spark and Java. The requirement is I will have a Dataset with few records. I need to only publish these records on Kafka queue on a topic.
Following is the publisher code written in Java:
Dataset dataset = messageDataSet.select("value");
dataset.selectExpr("CAST(value AS STRING)")
.write()
.format("kafka")
.option("topic","batch")
.option("kafka.bootstrap.servers","kafka01.qa.com:9092")
.save();
I have added following things is gradle , since the deployment guide of Streaming Kafka Integration says to have the gradle dependecies as provided I have also tried that.
dependencies {
compile group: 'com.util', name:'core-utilities-jdk8',version: '0.0.2-spark'
compile group: 'org.postgresql', name: 'postgresql', version: '42.2.8'
compile group: 'com.amazonaws', name: 'aws-java-sdk-s3', version: '1.11.687'
compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.9'
compile group: 'org.apache.spark', name: 'spark-core_2.12', version: "$sparkVersion"
compile group: 'org.apache.spark', name: 'spark-sql_2.12', version: "$sparkVersion"
compile group: 'com.databricks', name: 'spark-csv_2.11', version: '1.5.0'
compile group: 'commons-logging', name: 'commons-logging', version: '1.2'
compile group: 'org.apache.spark', name: 'spark-streaming-kafka-0-10_2.11', version: "$sparkVersion"
compile group: 'org.apache.spark', name: 'spark-streaming_2.12', version: "$sparkVersion"
compile group: 'org.apache.spark', name: 'spark-sql-kafka-0-10_2.12', version: "$sparkVersion"
annotationProcessor 'org.projectlombok:lombok:1.18.10'
compile group: 'org.projectlombok', name: 'lombok', version: '1.18.10'
testCompile group: 'junit', name: 'junit', version: '4.12'
testCompile group: 'org.mockito', name: 'mockito-all', version: '1.10.19'
compile 'org.apache.httpcomponents:httpmime:4.3.6'
compile group: 'org.apache.commons', name: 'commons-csv', version: '1.7'
compile group: 'com.fasterxml.jackson.module', name: 'jackson-module-scala_2.12', version: '2.10.1'
compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.10.1'
compile group: 'com.fasterxml.jackson.datatype', name: 'jackson-datatype-jsr310', version: '2.10.1'
compile group: 'com.jcraft', name: 'jsch', version: '0.1.55'
}
The dependecy I have tried as ProvidedCompile for Kafka and Spark Core but the error continues to be following :
User class threw exception: java.util.ServiceConfigurationError:
org.apache.spark.sql.sources.DataSourceRegister: Provider
org.apache.spark.sql.kafka010.KafkaSourceProvider could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:630)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:245)
at com.fileprocessor.jobs.Runner.publishMessage(Runner.java:200)
at com.fileprocessor.SparkFileProcessorMain.main(SparkFileProcessorMain.java:12)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684)
Caused by: java.lang.NoSuchMethodError:
org.apache.spark.internal.Logging.$init$(Lorg/apache/spark/internal/Logging;)V
at org.apache.spark.sql.kafka010.KafkaSourceProvider.<init>(KafkaSourceProvider.scala:44)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
I want to simply publish by dataset to Kafka Queues . Spark version is 2.4.4
I have went through the different results of Having transformenrs in Gradle , or downgrading the things for Kafka versions and scala . But nothing has helped . Anyone if have faced similar situation ?
Tried : spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.4 --class com.fileprocessor.SparkFileProcessorMain --master local[*] file-processor-spark-jobs-1.0-SNAPSHOT.jar
Or If any one have anyother way to write the similar code ?
Following gradle worked:
dependencies {
compile group: 'org.postgresql', name: 'postgresql', version: '42.2.8'
compile group: 'com.amazonaws', name: 'aws-java-sdk-s3', version: '1.11.687'
compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.9'
compile group: 'org.apache.spark', name: 'spark-core_2.12', version: "$sparkVersion"
compile group: 'org.apache.spark', name: 'spark-sql_2.12', version: "$sparkVersion"
compile group: 'com.databricks', name: 'spark-csv_2.11', version: '1.5.0'
compile group: 'commons-logging', name: 'commons-logging', version: '1.2'
compile group: 'org.apache.spark', name: 'spark-streaming-kafka-0-10_2.11', version: "$sparkVersion"
providedCompile group: 'org.apache.spark', name: 'spark-streaming_2.12', version: "$sparkVersion"
providedCompile group: 'org.apache.spark', name: 'spark-sql-kafka-0-10_2.11', version: "2.2.0"
annotationProcessor 'org.projectlombok:lombok:1.18.10'
compile group: 'org.projectlombok', name: 'lombok', version: '1.18.10'
testCompile group: 'junit', name: 'junit', version: '4.12'
testCompile group: 'org.mockito', name: 'mockito-all', version: '1.10.19'
compile 'org.apache.httpcomponents:httpmime:4.3.6'
compile group: 'org.apache.commons', name: 'commons-csv', version: '1.7'
compile group: 'com.fasterxml.jackson.module', name: 'jackson-module-scala_2.12', version: '2.10.1'
compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.10.1'
compile group: 'com.fasterxml.jackson.datatype', name: 'jackson-datatype-jsr310', version: '2.10.1'
compile group: 'com.jcraft', name: 'jsch', version: '0.1.55'
}
And spark submit : spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.0 --class com.fileprocessor.SparkFileProcessorMain --master local[*] file-processor-spark-jobs-1.0-SNAPSHOT.jar test bottomline false
Related
I have a rather strange issue with my Spring boot application which has been driving me around the bend.
So I have my Spring boot API which is in Github and uses Gradle to manage my dependencies. If I add a dependency and compile the project everything works fine. However, if I decide I do not need said dependency anymore and remove it from my build file, my Project will no longer compile stating it cant find packages / symbols for dependencies unrelated to what I removed.
Has anyone seen something like this before? As an FYI I'm using Intellij.
To give an example, here is my current build file:
plugins {
id "org.springframework.boot" version "2.4.2"
id 'io.spring.dependency-management' version '1.0.8.RELEASE'
id 'java'
id "com.commercehub.gradle.plugin.avro" version "0.21.0"
id "idea"
}
group 'org.example'
version '1.0-SNAPSHOT'
java {
sourceCompatibility = JavaVersion.VERSION_14
targetCompatibility = JavaVersion.VERSION_14
}
ext {
avroVersion = "1.10.1"
}
repositories {
mavenCentral()
jcenter()
maven {
url "https://packages.confluent.io/maven/"
}
}
avro {
createSetters = true
fieldVisibility = "PRIVATE"
}
apply plugin: "war"
dependencies {
providedRuntime 'org.springframework.boot:spring-boot-starter-tomcat'
implementation group: 'net.logstash.logback', name: 'logstash-logback-encoder', version: '6.6'
compile group: 'co.elastic.logging', name: 'logback-ecs-encoder', version: '0.5.2'
compile group: 'com.amazonaws', name: 'aws-java-sdk', version: '1.11.860'
implementation 'org.springframework.boot:spring-boot-starter-data-redis'
implementation group: 'org.springframework.boot', name: 'spring-boot-starter-cache', version: '2.4.3'
// compile 'redis.clients:jedis:3.5.1'
implementation group: 'com.mashape.unirest', name: 'unirest-java', version: '1.4.9'
compile group: 'io.springfox', name: 'springfox-swagger-ui', version: '3.0.0'
compile group: 'io.springfox', name: 'springfox-boot-starter', version: '3.0.0'
compile('org.springframework.boot:spring-boot-starter-data-elasticsearch')
compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-mongodb', version: '2.3.3.RELEASE'
compile group: 'org.springframework.data', name: 'spring-data-elasticsearch', version: '4.0.4.RELEASE'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-security', version: '2.3.3.RELEASE'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-jpa', version: '2.3.3.RELEASE'
compile group: 'org.springframework.security', name: 'spring-security-oauth2-client', version: '5.4.0'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-validation', version: '2.4.2'
compile group: 'org.springframework.kafka', name: 'spring-kafka', version: '2.6.5'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-oauth2-resource-server', version: '2.4.2'
compile group: 'mysql', name: 'mysql-connector-java', version: '8.0.21'
compile group: 'javax.validation', name: 'validation-api', version: '2.0.1.Final'
compile group: 'com.fasterxml.jackson.datatype', name: 'jackson-datatype-jsr310', version: '2.11.2'
compile group: 'io.jsonwebtoken', name: 'jjwt', version: '0.9.1'
compile group: 'org.openapitools', name: 'jackson-databind-nullable', version: '0.2.1'
compile group: 'commons-io', name: 'commons-io', version: '2.6'
compile group: 'org.apache.commons', name: 'commons-collections4', version: '4.4'
compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.11'
compile group: 'org.passay', name: 'passay', version: '1.6.0'
compile group: 'com.google.guava', name: 'guava', version: '30.0-jre'
// https://mvnrepository.com/artifact/com.auth0/java-jwt
compile group: 'com.auth0', name: 'java-jwt', version: '3.12.0'
compile group: 'io.confluent', name: 'kafka-schema-registry-client', version: '6.0.0'
compile group: 'io.confluent', name: 'kafka-avro-serializer', version: '6.0.0'
compile group: 'io.confluent', name: 'monitoring-interceptors', version: '6.0.0'
compile(group: 'io.confluent', name: 'kafka-streams-avro-serde', version: '6.0.0') {
exclude(module: 'log4j-over-slf4j')
}
compile "org.apache.avro:avro:1.10.1"
implementation "org.apache.avro:avro:${avroVersion}"
compileOnly 'org.projectlombok:lombok:1.18.12'
annotationProcessor 'org.projectlombok:lombok:1.18.12'
implementation 'com.amazonaws:aws-java-sdk-s3'
implementation 'org.springframework.boot:spring-boot-starter-web'
testCompile group: 'junit', name: 'junit', version: '4.12'
testCompileOnly 'org.projectlombok:lombok:1.18.12'
testAnnotationProcessor 'org.projectlombok:lombok:1.18.12'
testImplementation('org.springframework.boot:spring-boot-starter-test') {
exclude group: 'org.junit.vintage', module: 'junit-vintage-engine'
}
}
jar {
manifest {
attributes(
'Main-Class': 'com.test.TestApplication'
)
}
from {
configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) }
}
}
test {
useJUnitPlatform()
}
Now let's assume I don't want logstash so I remove it, this happens with any dependency
implementation group: 'net.logstash.logback', name: 'logstash-logback-encoder', version: '6.6'
Now my build file looks like this:
plugins {
id "org.springframework.boot" version "2.4.2"
id 'io.spring.dependency-management' version '1.0.8.RELEASE'
id 'java'
id "com.commercehub.gradle.plugin.avro" version "0.21.0"
id "idea"
}
group 'org.example'
version '1.0-SNAPSHOT'
java {
sourceCompatibility = JavaVersion.VERSION_14
targetCompatibility = JavaVersion.VERSION_14
}
ext {
avroVersion = "1.10.1"
}
repositories {
mavenCentral()
jcenter()
maven {
url "https://packages.confluent.io/maven/"
}
}
avro {
createSetters = true
fieldVisibility = "PRIVATE"
}
apply plugin: "war"
dependencies {
providedRuntime 'org.springframework.boot:spring-boot-starter-tomcat'
// REMOVED implementation group: 'net.logstash.logback', name: 'logstash-logback-encoder', version: '6.6'
compile group: 'co.elastic.logging', name: 'logback-ecs-encoder', version: '0.5.2'
compile group: 'com.amazonaws', name: 'aws-java-sdk', version: '1.11.860'
implementation 'org.springframework.boot:spring-boot-starter-data-redis'
implementation group: 'org.springframework.boot', name: 'spring-boot-starter-cache', version: '2.4.3'
// compile 'redis.clients:jedis:3.5.1'
implementation group: 'com.mashape.unirest', name: 'unirest-java', version: '1.4.9'
compile group: 'io.springfox', name: 'springfox-swagger-ui', version: '3.0.0'
compile group: 'io.springfox', name: 'springfox-boot-starter', version: '3.0.0'
compile('org.springframework.boot:spring-boot-starter-data-elasticsearch')
compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-mongodb', version: '2.3.3.RELEASE'
compile group: 'org.springframework.data', name: 'spring-data-elasticsearch', version: '4.0.4.RELEASE'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-security', version: '2.3.3.RELEASE'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-jpa', version: '2.3.3.RELEASE'
compile group: 'org.springframework.security', name: 'spring-security-oauth2-client', version: '5.4.0'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-validation', version: '2.4.2'
compile group: 'org.springframework.kafka', name: 'spring-kafka', version: '2.6.5'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-oauth2-resource-server', version: '2.4.2'
compile group: 'mysql', name: 'mysql-connector-java', version: '8.0.21'
compile group: 'javax.validation', name: 'validation-api', version: '2.0.1.Final'
compile group: 'com.fasterxml.jackson.datatype', name: 'jackson-datatype-jsr310', version: '2.11.2'
compile group: 'io.jsonwebtoken', name: 'jjwt', version: '0.9.1'
compile group: 'org.openapitools', name: 'jackson-databind-nullable', version: '0.2.1'
compile group: 'commons-io', name: 'commons-io', version: '2.6'
compile group: 'org.apache.commons', name: 'commons-collections4', version: '4.4'
compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.11'
compile group: 'org.passay', name: 'passay', version: '1.6.0'
compile group: 'com.google.guava', name: 'guava', version: '30.0-jre'
// https://mvnrepository.com/artifact/com.auth0/java-jwt
compile group: 'com.auth0', name: 'java-jwt', version: '3.12.0'
compile group: 'io.confluent', name: 'kafka-schema-registry-client', version: '6.0.0'
compile group: 'io.confluent', name: 'kafka-avro-serializer', version: '6.0.0'
compile group: 'io.confluent', name: 'monitoring-interceptors', version: '6.0.0'
compile(group: 'io.confluent', name: 'kafka-streams-avro-serde', version: '6.0.0') {
exclude(module: 'log4j-over-slf4j')
}
compile "org.apache.avro:avro:1.10.1"
implementation "org.apache.avro:avro:${avroVersion}"
compileOnly 'org.projectlombok:lombok:1.18.12'
annotationProcessor 'org.projectlombok:lombok:1.18.12'
implementation 'com.amazonaws:aws-java-sdk-s3'
implementation 'org.springframework.boot:spring-boot-starter-web'
testCompile group: 'junit', name: 'junit', version: '4.12'
testCompileOnly 'org.projectlombok:lombok:1.18.12'
testAnnotationProcessor 'org.projectlombok:lombok:1.18.12'
testImplementation('org.springframework.boot:spring-boot-starter-test') {
exclude group: 'org.junit.vintage', module: 'junit-vintage-engine'
}
}
jar {
manifest {
attributes(
'Main-Class': 'com.test.TestApplication'
)
}
from {
configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) }
}
}
test {
useJUnitPlatform()
}
Once I hit build I'll get non related logstash compilation errors such as in various classes:
java: package lombok does not exist
java: package org.openapitools.jackson.nullable does not exist
java: cannot find symbol
symbol: class Getter
java: package org.springframework.data.domain does not exist
java: package org.openapitools.jackson.nullable does not exist
I've tried reimporting my project, I've deleted it and pull down a fresh copy from Git..I'm at a loss now. Please if anyone has come across this let me know as google is failing me now :-/
Many thanks
EDIT: Please note for clarification, this seems to only happen when I have Intellij performing the build and Run.
If I switch the below to Gradle I've no issues remove dependencies as described above. However, I'd like to keep Intellij performing the build and run.
Edit:
Adding gradle-wrapper.properties in case it helps
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-6.7-bin.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
Also, following a suggestion below, I purged the cache in Intellij which Did solve the problem, However, once I remove another dependency I'm back to getting the errors above.
Second Edit
I'm not sure it matters but I noticed since Lombok is the main error I checked the annotation settings.
Why is there a default & a gradle imported, Would that be causing any of the issues?
I am trying to prepare a basic pipeline on jenkins, and although the local build works ok for the same task, it fails on jenkins with
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at org.gradle.internal.concurrent.CompositeStoppable.<clinit>(CompositeStoppable.java:37)
at org.gradle.internal.classloader.ClassLoaderUtils.tryClose(ClassLoaderUtils.java:51)
at org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:64)
at org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:37)
at org.gradle.launcher.GradleMain.main(GradleMain.java:23)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.gradle.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:31)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:108)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:62)
Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 12 more
The build file has the below configs for dependencies:
configurations {
all*.exclude group: 'com.google.guava', module:'guava-jdk5'
all*.exclude group: 'ch.qos.logback', module: 'logback-core'
all*.exclude group: 'ch.qos.logback', module: 'logback-classic'
all*.exclude module: 'slf4j-log4j12'
all*.exclude group: 'org.springframework.boot', module: 'spring-boot-starter-logging'
}
dependencies {
compile group: 'org.apache.commons', name: 'commons-collections4', version: '4.0'
compile group: 'org.slf4j', name:'slf4j-api', version: '1.7.29'
compile group: 'org.slf4j', name:'slf4j-log4j12', version:'1.5.6'
compile group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.10.0'
compile group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.10.0'
compile group: 'org.apache.logging.log4j', name: 'log4j-slf4j-impl', version: '2.10.0'
compile "com.google.guava:guava:23.0"
compile 'io.reactivex.rxjava2:rxjava:2.2.7'
compile group: 'org.springframework', name: 'spring-context', version: '5.0.4.RELEASE'
compile group: 'org.springframework', name: 'spring-web', version: '5.0.4.RELEASE'
compile group: 'org.springframework.security', name: 'spring-security-core', version: '5.0.4.RELEASE'
compile group: 'org.springframework.boot', name: 'spring-boot-autoconfigure', version: '2.0.0.RELEASE'
compile group: 'org.codehaus.groovy', name: 'groovy-all', version: '2.4.4'
testCompile group: 'org.spockframework', name: 'spock-core', version: '1.1-groovy-2.4'
testCompile group: 'org.spockframework', name: 'spock-spring', version: '1.1-groovy-2.4'
testCompile group: 'org.springframework.boot', name: 'spring-boot-starter-test', version: '2.0.0.RELEASE'
testCompile "cglib:cglib:2.2"
testCompile group: 'junit', name: 'junit', version: '4.12'
}
jenkins file
pipeline {
agent any
stages {
stage("Build") {
steps {
bat 'project/gradlew.bat -Djava.home=C:\\Program Files\\Java\\jdk1.8.0_172 -b project/build.gradle test'
}
}
}
}
I have tried to update the versions of the libs and enabled/disabled the exclusion parts, but the error still happens on jenkins, while on my local nothing unusual. can anyone point out what might be the reason and solution to that?
I try to run project with gretty. It throws exception:
10:47:39 WARN Failed startup of context o.a.g.JettyWebAppContext#52045dbe{/,file:/C:/IdeaProjects/example-tab/build/inplaceWebapp/,STARTING}
java.lang.RuntimeException: Error scanning entry module-info.class from jar file:/C:/Users/user1/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.10.0/fec5797a55b786184a537abd39c3fa1449d752d6/log4j-api-2.10.0.jar
at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:913) ~[jetty-annotations-9.2.24.v20180105.jar:9.2.24.v20180105]
at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:831) ~[jetty-annotations-9.2.24.v20180105.jar:9.2.24.v20180105]
at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:164) ~[jetty-annotations-9.2.24.v20180105.jar:9.2.24.v20180105]
at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:549) ~[jetty-annotations-9.2.24.v20180105.jar:9.2.24.v20180105]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) ~[jetty-util-9.2.24.v20180105.jar:9.2.24.v20180105]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) ~[jetty-util-9.2.24.v20180105.jar:9.2.24.v20180105]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171]
Caused by: java.lang.UnsupportedOperationException: null
at org.objectweb.asm.ClassVisitor.visitModule(ClassVisitor.java:129) ~[asm-6.1.1.jar:na]
at org.objectweb.asm.ClassReader.readModule(ClassReader.java:667) ~[asm-6.1.1.jar:na]
at org.objectweb.asm.ClassReader.accept(ClassReader.java:503) ~[asm-6.1.1.jar:na]
at org.objectweb.asm.ClassReader.accept(ClassReader.java:355) ~[asm-6.1.1.jar:na]
at org.eclipse.jetty.annotations.AnnotationParser.scanClass(AnnotationParser.java:974) ~[jetty-annotations-9.2.24.v20180105.jar:9.2.24.v20180105]
at org.eclipse.jetty.annotations.AnnotationParser.parseJarEntry(AnnotationParser.java:956) ~[jetty-annotations-9.2.24.v20180105.jar:9.2.24.v20180105]
at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:909) ~[jetty-annotations-9.2.24.v20180105.jar:9.2.24.v20180105]
... 6 common frames omitted
gradle:
plugins {
id 'java'
id 'war'
id "org.gretty" version "2.2.0"
}
group 'ru.example'
version '1.0-SNAPSHOT'
sourceCompatibility = 1.8
targetCompatibility = 1.8
compileJava.options.encoding = 'UTF-8'
ext {
SPRING_VERSION = '5.1.0.RELEASE'
JUNIT_VERSION = '5.3.1'
JACKSON_VERSION = '2.9.7'
}
repositories {
mavenCentral()
jcenter()
maven {
url 'https://oss.sonatype.org/content/repositories/releases/'
}
}
gretty {
springBootVersion = '2.0.5.RELEASE'
contextPath = "/"
}
dependencies {
annotationProcessor group: 'org.projectlombok', name: 'lombok', version: '1.18.2'
compileOnly group: 'org.projectlombok', name: 'lombok', version: '1.18.2'
compileOnly group: 'com.google.appengine', name: 'appengine-endpoints-deps', version: '1.9.65'
compile group: 'ch.qos.logback', name: 'logback-classic', version: '1.2.3'
compile group: 'org.springframework', name: 'spring-webmvc', version: SPRING_VERSION
compile group: 'org.springframework', name: 'spring-web', version: SPRING_VERSION
compile group: 'org.springframework', name: 'spring-jdbc', version: SPRING_VERSION
compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-jpa', version: '2.0.5.RELEASE'
compile group: 'com.mchange', name: 'c3p0', version: '0.9.5.2'
compile group: 'com.zaxxer', name: 'HikariCP', version: '3.2.0'
compile group: 'org.mariadb.jdbc', name: 'mariadb-java-client', version: '2.3.0'
compile group: 'com.fasterxml.jackson.core', name: 'jackson-core', version: JACKSON_VERSION
compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: JACKSON_VERSION
compile group: 'org.apache.httpcomponents', name: 'httpclient', version: '4.5.6'
compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.9.7'
testImplementation group: 'org.junit.jupiter', name: 'junit-jupiter-api', version: JUNIT_VERSION
testRuntimeOnly group: 'org.junit.jupiter', name: 'junit-jupiter-engine', version: JUNIT_VERSION
testCompile group: 'org.junit.jupiter', name: 'junit-jupiter-params', version: '5.3.1'
testCompile group: 'org.mockito', name: 'mockito-core', version: '2.22.0'
testCompile group: 'org.springframework', name: 'spring-test', version: SPRING_VERSION
testCompile group: 'org.hamcrest', name: 'hamcrest-library', version: '1.3'
testCompile group: 'com.jayway.jsonpath', name: 'json-path-assert', version: '2.4.0'
}
test {
useJUnitPlatform()
}
Scanning internet not helped. Please advice how to run it correctly. I had problems with downloading dependencies from jcenter(), may be, problem is here.
Validator write that my question have mostly code, but I have no Idea what to add to my explain, than, my apologies, I'll quote Steve Jobs:
“You can’t connect the dots looking forward; you can only connect them looking backward. So you have to trust that the dots will somehow connect in your future.”
You introduced a JEP-238 Jar file into your project: log4j-api-2.10.0.jar
Support for JEP-238 Jar files on any Java Runtime (doesn't matter if it's Java 7, Java 8, Java 9, Java 10, Java 11, or even the new Java 12-ea releases) was first introduced in Jetty 9.4.9.
You have 2 options:
Don't use JEP-238 Jar Files in your project.
Upgrade to Jetty 9.4.9 (or newer) to have support for JEP-238 Jar Files.
See past answers:
https://stackoverflow.com/a/45362629/775715
https://stackoverflow.com/a/52722120/775715
I was tried to build a web application by Gradle and pack a war file.
build.gradle
apply plugin: 'java'
apply plugin: 'eclipse-wtp'
apply plugin: 'war'
sourceCompatibility = 1.8
version = '1.0'
compileJava.options.encoding='UTF-8'
webAppDirName = 'fbs-report' //set webapp root
sourceSets {
main {
java {
srcDir 'src'
exclude '**/SlablistDload1Controller.java'
exclude '**/SlablistDload2Controller.java'
exclude '**/DormantDownloadController.java'
exclude '**/NewnewController.java'
}
resources {
srcDir 'config' //prorperties
}
}
}
repositories {
maven { url "http://maven.springframework.org/milestone" }
maven { url "http://repo.maven.apache.org/maven2" }
maven { url "http://repo1.maven.org/maven2/" }
maven { url "http://amateras.sourceforge.jp/mvn/" }
maven { url "http://repo.springsource.org/libs-release"}
mavenCentral()
}
configurations {
compile.exclude group: 'org.freemarker', module: 'freemarker'
compile.exclude group: 'javax.servlet', module: 'servlet-api'
compile.exclude group: 'jboss', module: 'javassist'
}
dependencies {
compile group: 'commons-collections', name: 'commons-collections', version: '3.2'
testCompile group: 'junit', name: 'junit', version: '4.+'
compile fileTree(include: ['*.jar'], dir: 'libs')
//spring framework
compile group: 'org.springframework', name: 'spring-context', version: '4.3.7.RELEASE'
compile group: 'org.springframework', name: 'spring-test', version: '4.3.7.RELEASE'
compile group: 'org.springframework', name: 'spring-core', version: '4.3.7.RELEASE'
compile group: 'org.springframework', name: 'spring-beans', version: '4.3.7.RELEASE'
compile group: 'org.springframework', name: 'spring-web', version: '4.3.7.RELEASE'
compile group: 'org.springframework', name: 'spring-webmvc', version: '4.3.7.RELEASE'
compile group: 'org.springframework', name: 'spring-jdbc', version: '4.3.7.RELEASE'
compile group: 'org.springframework', name: 'spring-aop', version: '4.3.7.RELEASE'
compile group: 'org.springframework', name: 'spring-tx', version: '4.3.7.RELEASE'
compile group: 'org.springframework', name: 'spring-context-support', version: '4.3.7.RELEASE'
compile group: 'org.springframework', name: 'spring-aspects', version: '4.3.7.RELEASE'
compile group: 'org.springframework', name: 'spring-oxm', version: '4.3.7.RELEASE'
compile group: 'org.springframework', name: 'spring-orm', version: '4.3.7.RELEASE'
compile group: 'org.springframework', name: 'spring-messaging', version: '4.3.7.RELEASE'
compile group: 'org.springframework', name: 'spring-websocket', version: '4.3.7.RELEASE'
compile group: 'org.springframework', name: 'spring-instrument', version: '4.3.7.RELEASE'
//spring data
compile group: 'org.springframework.data', name: 'spring-data-commons', version: '1.12.6.RELEASE'
compile group: 'org.springframework.data', name: 'spring-data-jpa', version: '1.11.1.RELEASE'
compile group: 'org.springframework.data', name: 'spring-data-rest-webmvc', version: '2.5.6.RELEASE'
compile group: 'org.springframework.data', name: 'spring-data-rest-core', version: '2.5.6.RELEASE'
compile group: 'org.springframework.data', name: 'spring-data-jdbc-core', version: '1.2.1.RELEASE'
compile group: 'org.springframework.data', name: 'spring-cql', version: '1.4.6.RELEASE'
compile group: 'org.springframework.data', name: 'spring-data-rest-parent', version: '2.5.6.RELEASE'
//spring-ws
compile group: 'org.springframework.ws', name: 'spring-xml', version: '2.4.0.RELEASE'
compile group: 'org.springframework.ws', name: 'spring-ws-security', version: '2.4.0.RELEASE'
compile group: 'org.springframework.ws', name: 'spring-ws-test', version: '2.4.0.RELEASE'
compile group: 'org.springframework.ws', name: 'spring-ws-support', version: '2.4.0.RELEASE'
//spring-boot
//compile group: 'org.springframework.boot', name: 'spring-boot-starter-test', version: '1.4.2.RELEASE'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-web', version: '1.5.2.RELEASE'
//compile group: 'org.springframework.boot', name: 'spring-boot-starter', version: '1.4.2.RELEASE'
//compile group: 'org.springframework.boot', name: 'spring-boot-autoconfigure', version: '1.4.2.RELEASE'
//compile group: 'org.springframework.boot', name: 'spring-boot-configuration-processor', version: '1.4.2.RELEASE'
//compile group: 'org.springframework.boot', name: 'spring-boot-starter-actuator', version: '1.4.2.RELEASE'
compile group: 'org.springframework.boot', name: 'spring-boot', version: '1.5.2.RELEASE'
//compile group: 'org.springframework.boot', name: 'spring-boot-starter-security', version: '1.4.2.RELEASE'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-jpa', version: '1.5.2.RELEASE'
//compile group: 'org.springframework.boot', name: 'spring-boot-starter-logging', version: '1.4.2.RELEASE'
//compile group: 'org.springframework.boot', name: 'spring-boot-starter-jdbc', version: '1.4.2.RELEASE'
//compile group: 'org.springframework.boot', name: 'spring-boot-starter-websocket', version: '1.4.2.RELEASE'
//compile group: 'org.springframework.boot', name: 'spring-boot-devtools', version: '1.4.2.RELEASE'
//compile group: 'org.springframework.boot', name: 'spring-boot-starter-aop', version: '1.4.2.RELEASE'
//compile group: 'org.springframework.boot', name: 'spring-boot-starter-log4j', version: '1.3.8.RELEASE'
//compile group: 'org.springframework.boot', name: 'spring-boot-test', version: '1.4.2.RELEASE'
//compile group: 'org.springframework.boot', name: 'spring-boot-gradle-plugin', version: '1.4.2.RELEASE'
//compile group: 'org.springframework.boot', name: 'spring-boot-deployment-tests', version: '1.4.2.RELEASE'
//compile group: 'org.springframework.boot', name: 'spring-boot-starter-parent', version: '1.4.2.RELEASE'
//hibernate
compile group: 'org.hibernate', name: 'hibernate-core', version: '5.2.5.Final'
compile group: 'org.hibernate', name: 'hibernate-validator', version: '5.3.3.Final'
compile group: 'org.hibernate', name: 'hibernate-entitymanager', version: '5.2.5.Final'
compile group: 'org.hibernate', name: 'hibernate-ehcache', version: '5.2.5.Final'
compile group: 'org.hibernate', name: 'hibernate-c3p0', version: '5.2.5.Final'
compile group: 'org.hibernate', name: 'hibernate-jpamodelgen', version: '5.2.5.Final'
compile group: 'org.hibernate', name: 'hibernate-search-orm', version: '5.5.5.Final'
compile group: 'org.hibernate', name: 'hibernate-tools', version: '5.1.0.Final'
compile group: 'org.hibernate', name: 'hibernate-testing', version: '5.2.5.Final'
compile group: 'org.hibernate', name: 'hibernate-infinispan', version: '5.2.5.Final'
compile group: 'org.hibernate', name: 'hibernate-validator-cdi', version: '5.3.3.Final'
compile group: 'org.hibernate', name: 'hibernate-validator-annotation-processor', version: '5.3.3.Final'
compile group: 'org.hibernate.hql', name: 'hibernate-hql-parser', version: '1.3.0.Final'
compile group: 'org.hibernate.hql', name: 'hibernate-hql-testing', version: '1.3.0.Final'
compile group: 'org.hibernate', name: 'hibernate-java8', version: '5.2.5.Final'
compile group: 'org.hibernate', name: 'hibernate-osgi', version: '5.2.5.Final'
compile group: 'org.hibernate', name: 'hibernate-validator-test-utils', version: '5.3.3.Final'
compile group: 'org.hibernate', name: 'hibernate-validator-osgi', version: '5.3.3.Final'
compile group: 'org.hibernate', name: 'hibernate-gradle-plugin', version: '5.2.5.Final'
compile group: 'org.hibernate', name: 'hibernate-validator-parent', version: '5.3.3.Final'
compile group: 'org.hibernate.javax.persistence', name: 'hibernate-jpa-2.1-api', version: '1.0.0.Final'
//compile group: 'javax.persistence', name: 'persistence-api', version: '1.0.2'
compile group: 'org.projectlombok', name: 'lombok', version: '1.16.16'
//apache common
compile group: 'javax.activation', name: 'activation', version: '1.1.1'
compile group: 'org.apache.activemq', name: 'activemq-core', version: '5.7.0'
compile group: 'org.apache.activemq.protobuf', name: 'activemq-protobuf', version: '1.1'
compile group: 'annogen', name: 'annogen', version: '0.1.0'
compile group: 'antlr', name: 'antlr', version: '2.7.7'
compile group: 'aopalliance', name: 'aopalliance', version: '1.0'
compile group: 'org.aspectj', name: 'aspectjrt', version: '1.8.10'
compile group: 'org.aspectj', name: 'aspectjweaver', version: '1.8.10'
compile group: 'org.apache.ws.commons.axiom', name: 'axiom-api', version: '1.2.20'
compile group: 'org.apache.axis2', name: 'axis2', version: '1.4.1'
compile group: 'org.apache.axis2', name: 'axis2-saaj', version: '1.4.1'
compile group: 'org.apache.commons', name: 'commons-pool2', version: '2.4.2'
compile group: 'org.apache.poi', name: 'poi', version: '3.9'
//compile group: 'org.quartz-scheduler', name: 'quartz', version: '2.2.3'
compile group: 'org.apache.commons', name: 'commons-dbcp2', version: '2.1.1'
compile group: 'net.sf.json-lib', name: 'json-lib', version: '2.4', classifier: 'jdk15'
//compile group: 'oro', name: 'oro', version: '2.0.8'
//compile group: 'org.freemarker', name: 'freemarker', version: '2.3.23'
//compile group: 'jboss', name: 'javassist', version: '3.22.0-CR1'
compile group: 'commons-fileupload', name: 'commons-fileupload', version: '1.3.2'
compile group: 'commons-io', name: 'commons-io', version: '2.5'
compile group: 'org.jfree', name: 'jcommon', version: '1.0.23'
compile group: 'org.jfree', name: 'jfreechart', version: '1.0.19'
compile group: 'net.minidev', name: 'json-smart', version: '1.0.9'
compile group: 'xerces', name: 'xercesImpl', version: '2.11.0'
}
When I execute $ gradle build, always fail in task "compilejava"
Here is part of the error mesage:
D:\fbs-UAT\fbs-report\src\model\MonitorModel.java:62: error: cannot find symbol
classes={
^
symbol: method classes()
location: #interface SqlResultSetMapping
D:\fbs-UAT\fbs-report\src\model\ConvertModel.java:114: error: cannot find symbol
#OneToMany(cascade= { CascadeType.ALL }, fetch=FetchType.EAGER, orphanRemoval=true)
^
symbol: method orphanRemoval()
location: #interface OneToMany
D:\fbs-UAT\fbs-report\src\model\MonitorModel.java:66: error: duplicate element '<any?>' in annotation #<any?>.
#ColumnResult(name="sysId", type=Long.class),
^
D:\fbs-UAT\fbs-report\src\model\MonitorModel.java:66: error: duplicate element '<any?>' in annotation #<any?>.
#ColumnResult(name="sysId", type=Long.class),
^
D:\fbs-UAT\fbs-report\src\model\MonitorModel.java:67: error: duplicate element '<any?>' in annotation #<any?>.
#ColumnResult(name="sysCreateUser", type=String.class),
^
D:\fbs-UAT\fbs-report\src\model\MonitorModel.java:67: error: duplicate element '<any?>' in annotation #<any?>.
#ColumnResult(name="sysCreateUser", type=String.class),
^
D:\fbs-UAT\fbs-report\src\model\MonitorModel.java:68: error: duplicate element '<any?>' in annotation #<any?>.
#ColumnResult(name="sysCreateTime", type=Date.class),
^
D:\fbs-UAT\fbs-report\src\model\MonitorModel.java:68: error: duplicate element '<any?>' in annotation #<any?>.
#ColumnResult(name="sysCreateTime", type=Date.class),
^
D:\fbs-UAT\fbs-report\src\MonitorModel.java:69: error: duplicate element '<any?>' in annotation #<any?>.
#ColumnResult(name="sysModifyUser", type=String.class),
^
D:\fbs-UAT\fbs-report\src\model\MonitorModel.java:69: error: duplicate element '<any?>' in annotation #<any?>.
#ColumnResult(name="sysModifyUser", type=String.class),
^
By the way, if takes away codes for JPA, it turns out Successful results.
If you add the dependency on JPA 1.0.2 the compilation will fail since some of your annotations depend on JPA 2.1.
Since 'org.hibernate.javax.persistence:hibernate-jpa-2.1-api:1.0.0.Final' already contains the JPA 2.1 API you don't need a separate dependency on javax.persistence:persistence-api
I am using the dependency section in my gradle script but I think I am crearting the duplication of the dependency in my script.I need the dependency during compilation time as well as during runtime.Do I need to define the dependency as a compile and runtime.
Because for runtime my script is having a below logic
runtime.configuration and that way I am putting all the jar file to lib folder
Can I do both the things in one section?I mean I want only one dependency section compile or runtime.
Please suggest
dependencies {
compile group: 'com.rohit.singh', name: 'common', version:'4.+'
compile group: 'com.rohit.singh', name: 'LicenseVerifier', version:'4.+'
compile group: 'com.rohit.singh', name: 'MegNativeJNI', version: '4.+', ext: 'so'
compile group: 'com.rohit.singh', name: 'NativeJNI', version: '4.+', ext: 'so'
compile(group: 'com.operasolutions', name: 'RiskAnalytics', version:'1.1') {
exclude(module: 'jyson')
}
compile group: 'org.springframework', name: 'spring-context', version:'3.2.5.RELEASE'
compile group: 'org.springframework', name: 'spring-beans', version:'3.2.5.RELEASE'
compile group: 'org.springframework', name: 'spring-core', version:'3.2.5.RELEASE'
compile group: 'org.springframework', name: 'spring-aop', version:'3.2.5.RELEASE'
compile group: 'org.springframework', name: 'spring-expression', version:'3.2.5.RELEASE'
compile group: 'org.apache.commons', name: 'commons-collections4', version:'4.0'
compile group: 'commons-io', name: 'commons-io', version:'2.4'
compile group: 'net.sf.supercsv', name: 'super-csv-dozer', version:'2.1.0'
compile group: 'net.sf.supercsv', name: 'super-csv', version:'2.1.0'
compile group: 'org.aspectj', name: 'aspectjtools', version:'1.6.2'
compile group: 'org.python', name: 'jython-standalone', version:'2.5.3'
compile group: 'com.google.code.gson', name: 'gson', version:'1.7.2'
compile group: 'log4j', name: 'log4j', version:'1.2.16'
compile group: 'com.xhaus', name: 'jyson', version:'1.0.2'
compile group: 'com.google.guava', name: 'guava', version:'17.0'
compile group: 'com.jamonapi', name: 'jamon', version:'2.4'
compile group: 'ch.qos.logback', name: 'logback-classic', version:'1.1.1'
compile group: 'ch.qos.logback', name: 'logback-core', version:'1.1.1'
compile group: 'org.slf4j', name: 'slf4j-api', version:'1.7.6'
compile group: 'org.codehaus.mojo', name: 'properties-maven-plugin', version:'1.0-alpha-2'
compile group: 'args4j', name: 'args4j', version:'2.0.28'
compile group: 'org.codehaus.jackson', name: 'jackson-mapper-asl', version:'1.9.5'
testCompile group: 'junit', name: 'junit', version:'4.11'
testCompile group: 'org.mockito', name: 'mockito-all', version:'1.9.5'
}
dependencies {
runtime group: 'com.rohit.singh', name: 'engine-common', version:'4.+'
runtime group: 'com.rohit.singh', name: 'engine-LicenseVerifier', version:'4.+'
runtime group: 'com.rohit.singh', name: 'MegNativeJNI', version: '4.+', ext: 'so'
runtime group: 'com.rohit.singh', name: 'NativeJNI', version: '4.+', ext: 'so'
runtime (group: 'com.operasolutions', name: 'RiskAnalytics', version:'1.1') {
exclude(module: 'jyson')
}
runtime group: 'org.springframework', name: 'spring-context', version:'3.2.5.RELEASE'
runtime group: 'org.springframework', name: 'spring-beans', version:'3.2.5.RELEASE'
runtime group: 'org.springframework', name: 'spring-core', version:'3.2.5.RELEASE'
runtime group: 'org.springframework', name: 'spring-aop', version:'3.2.5.RELEASE'
runtime group: 'org.springframework', name: 'spring-expression', version:'3.2.5.RELEASE'
runtime group: 'org.apache.commons', name: 'commons-collections4', version:'4.0'
runtime group: 'commons-io', name: 'commons-io', version:'2.4'
runtime group: 'net.sf.supercsv', name: 'super-csv-dozer', version:'2.1.0'
runtime group: 'net.sf.supercsv', name: 'super-csv', version:'2.1.0'
runtime group: 'org.aspectj', name: 'aspectjtools', version:'1.6.2'
runtime group: 'org.python', name: 'jython-standalone', version:'2.5.3'
runtime group: 'com.google.code.gson', name: 'gson', version:'1.7.2'
runtime group: 'log4j', name: 'log4j', version:'1.2.16'
runtime group: 'com.xhaus', name: 'jyson', version:'1.0.2'
runtime group: 'com.google.guava', name: 'guava', version:'17.0'
runtime group: 'com.jamonapi', name: 'jamon', version:'2.4'
runtime group: 'ch.qos.logback', name: 'logback-classic', version:'1.1.1'
runtime group: 'ch.qos.logback', name: 'logback-core', version:'1.1.1'
runtime group: 'org.slf4j', name: 'slf4j-api', version:'1.7.6'
runtime group: 'org.codehaus.mojo', name: 'properties-maven-plugin', version:'1.0-alpha-2'
runtime group: 'args4j', name: 'args4j', version:'2.0.28'
runtime group: 'org.codehaus.jackson', name: 'jackson-mapper-asl', version:'1.9.5'
}
You already defined your dependencies in compile scope. There is no need to define them also for runtime. Compile scope dependencies will automatically be available at runtime.
You can also put compile dependencies to lib folder and exclude any dependency you don't need there. But i wonder if there is any dependency to exclude and if it is worth to build that logic for it.
I would suggest to make your build script as simple as possible.