Issue with gradle with adding Open JavaFX package from Maven Central - java

So in my build.gradle file I have this dependencies added.
dependencies {
testCompile group: 'junit', name: 'junit', version: '4.12'
// https://mvnrepository.com/artifact/org.hibernate/hibernate-core
compile group: 'org.hibernate', name: 'hibernate-core', version: '5.3.6.Final'
// https://mvnrepository.com/artifact/mysql/mysql-connector-java
compile group: 'mysql', name: 'mysql-connector-java', version: '8.0.12'
// https://mvnrepository.com/artifact/javax.xml.bind/jaxb-api
compile group: 'javax.xml.bind', name: 'jaxb-api', version: '2.3.0'
// https://mvnrepository.com/artifact/javax.xml/jaxb-impl
compile group: 'javax.xml', name: 'jaxb-impl', version: '2.1'
// https://mvnrepository.com/artifact/org.openjfx/javafx-controls
compile group: 'org.openjfx', name: 'javafx-controls', version: '11'
}
The class of other dependencies were added except for javafx classes. It wasn't downloaded all I get is just the meta-inf directory.
I am using JDK 11, so I really need openjfx repo to use JavaFX. because in JDK 11, javafx is decoupled.
Update: I am using Intellij Idea

Each supported platform has its own version of JavaFx module artifacts, so you need to specify it too:
compile group: 'org.openjfx', name: 'javafx-controls', version: '11', classifier: 'linux' //'win', 'mac'
Additionally, you will need to explicitly add all transitive dependencies of the included javafx modules, because gradle cannot resolve platform specific modules by itself:
compile group: 'org.openjfx', name: 'javafx-base', version: '11', classifier: 'linux'
compile group: 'org.openjfx', name: 'javafx-graphics', version: '11', classifier: 'linux'
Check official documentation: https://openjfx.io/openjfx-docs/#gradle

Related

Selenium 4 : Getting java.lang.NoSuchMethodError: org.openqa.selenium.WebElement.getDomAttribute(Ljava/lang/String;)Ljava/lang/String;

I am using selenium "4.1.2" with chrome 97. While selecting value from drop down using select class, getting exception:
java.lang.NoSuchMethodError: org.openqa.selenium.WebElement.getDomAttribute(Ljava/lang/String;)Ljava/lang/String;
Below are dependencies in my project:
ext {
selenium = '4.1.2'
webdrivermanager = '5.0.3'
}
dependencies {
compile("org.seleniumhq.selenium:selenium-java:${selenium}")
// compile("io.github.bonigarcia:webdrivermanager:${webdrivermanager}")
testImplementation "org.seleniumhq.selenium:selenium-chrome-driver:${selenium}"
testImplementation "org.seleniumhq.selenium:selenium-firefox-driver:${selenium}"
testImplementation "org.seleniumhq.selenium:selenium-ie-driver:${selenium}"
testImplementation "org.seleniumhq.selenium:selenium-edge-driver:${selenium}"
testImplementation "org.seleniumhq.selenium:selenium-safari-driver:${selenium}"
testImplementation "org.seleniumhq.selenium:selenium-remote-driver:${selenium}"
testImplementation "org.seleniumhq.selenium:selenium-support:${selenium}"
testImplementation('org.junit.jupiter:junit-jupiter:5.5.1')
testImplementation 'org.hamcrest:hamcrest:2.1'
testImplementation 'org.hamcrest:hamcrest-library:2.1'
testCompile("org.junit.jupiter:junit-jupiter-api:5.7.2")
testRuntime 'org.junit.jupiter:junit-jupiter-engine:5.7.2'
testRuntime("org.junit.platform:junit-platform-launcher:1.7.2")
testCompile('io.github.bonigarcia:selenium-jupiter:3.3.4')
compile group: 'io.qameta.allure', name: 'allure-junit5', version: '2.11.0'
compile group: 'org.apache.pdfbox', name: 'pdfbox', version: '2.0.16'
implementation group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.17.1'
implementation group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.17.1'
compile group: 'io.qameta.allure', name: 'allure-gradle', version: '2.7.0'
compile 'org.apache.maven.plugins:maven-surefire-plugin:2.21.0'
compile('com.assertthat:selenium-shutterbug:1.5')
compile 'org.slf4j:slf4j-nop:1.7.25'
implementation group: 'javax.mail', name: 'mail', version: '1.4.7'
implementation group: 'javax.mail', name: 'javax.mail-api', version: '1.6.2'
runtimeClasspath group: 'javax.mail', name: 'javax.mail-api', version: '1.6.2'
compile group: 'net.lightbody.bmp', name: 'browsermob-core', version: '2.1.4'
compile group: 'org.postgresql', name: 'postgresql', version: '42.2.14'
compile group: 'org.mongodb', name: 'mongo-java-driver', version: '3.12.4'
compile group: 'ru.yandex.qatools.ashot', name: 'ashot', version: '1.5.4'
implementation group: 'org.json', name: 'json', version: '20201115'
implementation group: 'com.googlecode.json-simple', name: 'json-simple', version: '1.1.1'
testImplementation group: 'org.junit.platform', name: 'junit-platform-launcher', version: '1.7.2'
testImplementation group: 'org.junit.platform', name: 'junit-platform-runner', version: '1.7.2'
testImplementation group: 'org.junit.platform', name: 'junit-platform-surefire-provider', version: '1.3.2'
}
Any help in resolving this issue is highly appreciated.
This error message...
java.lang.NoSuchMethodError: org.openqa.selenium.WebElement.getDomAttribute(Ljava/lang/String;)Ljava/lang/String;
...is the result of dependency version conflict.
Deep Dive
As per #titusfortner comment in the discussion you need to crosscheck that everything related to Selenium is set to 4.x and that nothing that requires Selenium 3.x (e.g., Appium 7) is included in your dependencies as #asolntsev in the comment mentions:
All implementations of WebElement do override method getDomAttribute(). It never throws UnsupportedOperationException in real life.
Solution
You need to execute mvn dependency:tree or gradle dependencies and you will be able to trace the problem as described in this example.
I would try forcing below dependency in build.gradle
implementation (group: 'org.seleniumhq.selenium', name: 'selenium-api', version: '4.1.2') { force = true }
NoSuchMethod error occurs when dependency version conflict happens. If you are using lower version Appium, upgrading to following version,
implementation 'io.appium:java-client:8.0.0'
may resolve errors.

Spring Boot: Project won't compile after changing gradle dependencies

I have a rather strange issue with my Spring boot application which has been driving me around the bend.
So I have my Spring boot API which is in Github and uses Gradle to manage my dependencies. If I add a dependency and compile the project everything works fine. However, if I decide I do not need said dependency anymore and remove it from my build file, my Project will no longer compile stating it cant find packages / symbols for dependencies unrelated to what I removed.
Has anyone seen something like this before? As an FYI I'm using Intellij.
To give an example, here is my current build file:
plugins {
id "org.springframework.boot" version "2.4.2"
id 'io.spring.dependency-management' version '1.0.8.RELEASE'
id 'java'
id "com.commercehub.gradle.plugin.avro" version "0.21.0"
id "idea"
}
group 'org.example'
version '1.0-SNAPSHOT'
java {
sourceCompatibility = JavaVersion.VERSION_14
targetCompatibility = JavaVersion.VERSION_14
}
ext {
avroVersion = "1.10.1"
}
repositories {
mavenCentral()
jcenter()
maven {
url "https://packages.confluent.io/maven/"
}
}
avro {
createSetters = true
fieldVisibility = "PRIVATE"
}
apply plugin: "war"
dependencies {
providedRuntime 'org.springframework.boot:spring-boot-starter-tomcat'
implementation group: 'net.logstash.logback', name: 'logstash-logback-encoder', version: '6.6'
compile group: 'co.elastic.logging', name: 'logback-ecs-encoder', version: '0.5.2'
compile group: 'com.amazonaws', name: 'aws-java-sdk', version: '1.11.860'
implementation 'org.springframework.boot:spring-boot-starter-data-redis'
implementation group: 'org.springframework.boot', name: 'spring-boot-starter-cache', version: '2.4.3'
// compile 'redis.clients:jedis:3.5.1'
implementation group: 'com.mashape.unirest', name: 'unirest-java', version: '1.4.9'
compile group: 'io.springfox', name: 'springfox-swagger-ui', version: '3.0.0'
compile group: 'io.springfox', name: 'springfox-boot-starter', version: '3.0.0'
compile('org.springframework.boot:spring-boot-starter-data-elasticsearch')
compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-mongodb', version: '2.3.3.RELEASE'
compile group: 'org.springframework.data', name: 'spring-data-elasticsearch', version: '4.0.4.RELEASE'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-security', version: '2.3.3.RELEASE'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-jpa', version: '2.3.3.RELEASE'
compile group: 'org.springframework.security', name: 'spring-security-oauth2-client', version: '5.4.0'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-validation', version: '2.4.2'
compile group: 'org.springframework.kafka', name: 'spring-kafka', version: '2.6.5'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-oauth2-resource-server', version: '2.4.2'
compile group: 'mysql', name: 'mysql-connector-java', version: '8.0.21'
compile group: 'javax.validation', name: 'validation-api', version: '2.0.1.Final'
compile group: 'com.fasterxml.jackson.datatype', name: 'jackson-datatype-jsr310', version: '2.11.2'
compile group: 'io.jsonwebtoken', name: 'jjwt', version: '0.9.1'
compile group: 'org.openapitools', name: 'jackson-databind-nullable', version: '0.2.1'
compile group: 'commons-io', name: 'commons-io', version: '2.6'
compile group: 'org.apache.commons', name: 'commons-collections4', version: '4.4'
compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.11'
compile group: 'org.passay', name: 'passay', version: '1.6.0'
compile group: 'com.google.guava', name: 'guava', version: '30.0-jre'
// https://mvnrepository.com/artifact/com.auth0/java-jwt
compile group: 'com.auth0', name: 'java-jwt', version: '3.12.0'
compile group: 'io.confluent', name: 'kafka-schema-registry-client', version: '6.0.0'
compile group: 'io.confluent', name: 'kafka-avro-serializer', version: '6.0.0'
compile group: 'io.confluent', name: 'monitoring-interceptors', version: '6.0.0'
compile(group: 'io.confluent', name: 'kafka-streams-avro-serde', version: '6.0.0') {
exclude(module: 'log4j-over-slf4j')
}
compile "org.apache.avro:avro:1.10.1"
implementation "org.apache.avro:avro:${avroVersion}"
compileOnly 'org.projectlombok:lombok:1.18.12'
annotationProcessor 'org.projectlombok:lombok:1.18.12'
implementation 'com.amazonaws:aws-java-sdk-s3'
implementation 'org.springframework.boot:spring-boot-starter-web'
testCompile group: 'junit', name: 'junit', version: '4.12'
testCompileOnly 'org.projectlombok:lombok:1.18.12'
testAnnotationProcessor 'org.projectlombok:lombok:1.18.12'
testImplementation('org.springframework.boot:spring-boot-starter-test') {
exclude group: 'org.junit.vintage', module: 'junit-vintage-engine'
}
}
jar {
manifest {
attributes(
'Main-Class': 'com.test.TestApplication'
)
}
from {
configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) }
}
}
test {
useJUnitPlatform()
}
Now let's assume I don't want logstash so I remove it, this happens with any dependency
implementation group: 'net.logstash.logback', name: 'logstash-logback-encoder', version: '6.6'
Now my build file looks like this:
plugins {
id "org.springframework.boot" version "2.4.2"
id 'io.spring.dependency-management' version '1.0.8.RELEASE'
id 'java'
id "com.commercehub.gradle.plugin.avro" version "0.21.0"
id "idea"
}
group 'org.example'
version '1.0-SNAPSHOT'
java {
sourceCompatibility = JavaVersion.VERSION_14
targetCompatibility = JavaVersion.VERSION_14
}
ext {
avroVersion = "1.10.1"
}
repositories {
mavenCentral()
jcenter()
maven {
url "https://packages.confluent.io/maven/"
}
}
avro {
createSetters = true
fieldVisibility = "PRIVATE"
}
apply plugin: "war"
dependencies {
providedRuntime 'org.springframework.boot:spring-boot-starter-tomcat'
// REMOVED implementation group: 'net.logstash.logback', name: 'logstash-logback-encoder', version: '6.6'
compile group: 'co.elastic.logging', name: 'logback-ecs-encoder', version: '0.5.2'
compile group: 'com.amazonaws', name: 'aws-java-sdk', version: '1.11.860'
implementation 'org.springframework.boot:spring-boot-starter-data-redis'
implementation group: 'org.springframework.boot', name: 'spring-boot-starter-cache', version: '2.4.3'
// compile 'redis.clients:jedis:3.5.1'
implementation group: 'com.mashape.unirest', name: 'unirest-java', version: '1.4.9'
compile group: 'io.springfox', name: 'springfox-swagger-ui', version: '3.0.0'
compile group: 'io.springfox', name: 'springfox-boot-starter', version: '3.0.0'
compile('org.springframework.boot:spring-boot-starter-data-elasticsearch')
compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-mongodb', version: '2.3.3.RELEASE'
compile group: 'org.springframework.data', name: 'spring-data-elasticsearch', version: '4.0.4.RELEASE'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-security', version: '2.3.3.RELEASE'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-jpa', version: '2.3.3.RELEASE'
compile group: 'org.springframework.security', name: 'spring-security-oauth2-client', version: '5.4.0'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-validation', version: '2.4.2'
compile group: 'org.springframework.kafka', name: 'spring-kafka', version: '2.6.5'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-oauth2-resource-server', version: '2.4.2'
compile group: 'mysql', name: 'mysql-connector-java', version: '8.0.21'
compile group: 'javax.validation', name: 'validation-api', version: '2.0.1.Final'
compile group: 'com.fasterxml.jackson.datatype', name: 'jackson-datatype-jsr310', version: '2.11.2'
compile group: 'io.jsonwebtoken', name: 'jjwt', version: '0.9.1'
compile group: 'org.openapitools', name: 'jackson-databind-nullable', version: '0.2.1'
compile group: 'commons-io', name: 'commons-io', version: '2.6'
compile group: 'org.apache.commons', name: 'commons-collections4', version: '4.4'
compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.11'
compile group: 'org.passay', name: 'passay', version: '1.6.0'
compile group: 'com.google.guava', name: 'guava', version: '30.0-jre'
// https://mvnrepository.com/artifact/com.auth0/java-jwt
compile group: 'com.auth0', name: 'java-jwt', version: '3.12.0'
compile group: 'io.confluent', name: 'kafka-schema-registry-client', version: '6.0.0'
compile group: 'io.confluent', name: 'kafka-avro-serializer', version: '6.0.0'
compile group: 'io.confluent', name: 'monitoring-interceptors', version: '6.0.0'
compile(group: 'io.confluent', name: 'kafka-streams-avro-serde', version: '6.0.0') {
exclude(module: 'log4j-over-slf4j')
}
compile "org.apache.avro:avro:1.10.1"
implementation "org.apache.avro:avro:${avroVersion}"
compileOnly 'org.projectlombok:lombok:1.18.12'
annotationProcessor 'org.projectlombok:lombok:1.18.12'
implementation 'com.amazonaws:aws-java-sdk-s3'
implementation 'org.springframework.boot:spring-boot-starter-web'
testCompile group: 'junit', name: 'junit', version: '4.12'
testCompileOnly 'org.projectlombok:lombok:1.18.12'
testAnnotationProcessor 'org.projectlombok:lombok:1.18.12'
testImplementation('org.springframework.boot:spring-boot-starter-test') {
exclude group: 'org.junit.vintage', module: 'junit-vintage-engine'
}
}
jar {
manifest {
attributes(
'Main-Class': 'com.test.TestApplication'
)
}
from {
configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) }
}
}
test {
useJUnitPlatform()
}
Once I hit build I'll get non related logstash compilation errors such as in various classes:
java: package lombok does not exist
java: package org.openapitools.jackson.nullable does not exist
java: cannot find symbol
symbol: class Getter
java: package org.springframework.data.domain does not exist
java: package org.openapitools.jackson.nullable does not exist
I've tried reimporting my project, I've deleted it and pull down a fresh copy from Git..I'm at a loss now. Please if anyone has come across this let me know as google is failing me now :-/
Many thanks
EDIT: Please note for clarification, this seems to only happen when I have Intellij performing the build and Run.
If I switch the below to Gradle I've no issues remove dependencies as described above. However, I'd like to keep Intellij performing the build and run.
Edit:
Adding gradle-wrapper.properties in case it helps
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-6.7-bin.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
Also, following a suggestion below, I purged the cache in Intellij which Did solve the problem, However, once I remove another dependency I'm back to getting the errors above.
Second Edit
I'm not sure it matters but I noticed since Lombok is the main error I checked the annotation settings.
Why is there a default & a gradle imported, Would that be causing any of the issues?

Spark Submit Failed to run a Java Spark Job Accessing AWS S3 [NoSuch Method: ProviderUtils.excludeIncompatibleCredentialProviders]

I am getting java.lang.NoSuchMethodError: org.apache.hadoop.security.ProviderUtils.excludeIncompatibleCredentialProviders exception while submitting my spark job using spark-submit
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.security.ProviderUtils.excludeIncompatibleCredentialProviders(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/Class;)Lorg/apache/hadoop/conf/Configuration;
at org.apache.hadoop.fs.s3a.S3AUtils.getAWSAccessKeys(S3AUtils.java:740)
at org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider.<init>(SimpleAWSCredentialsProvider.java:58)
at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:600)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:260)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
Spark version Installed: 2.4.5
My Configuration: build.gradle:
buildscript {
repositories {
maven {
url "https://*********/****/content/repositories/thirdparty"
credentials {
username ****User
password ****Pwd
}
}
}
}
plugins {
id 'java'
id 'com.github.johnrengelman.shadow' version '1.2.3'
}
group 'com.felix'
version '1.0-SNAPSHOT'
sourceCompatibility = 1.8
repositories {
mavenCentral()
mavenLocal()
}
dependencies {
compile group: 'org.apache.spark', name: 'spark-hadoop-cloud_2.11', version: '2.4.2.3.1.3.0-79'
// https://mvnrepository.com/artifact/org.apache.spark/spark-sql
compileOnly group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.4.5'
// https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind
compileOnly group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.6.7.3'
// https://mvnrepository.com/artifact/org.apache.parquet/parquet-column
compileOnly group: 'org.apache.parquet', name: 'parquet-column', version: '1.10.1'
// https://mvnrepository.com/artifact/org.apache.parquet/parquet-hadoop
compileOnly group: 'org.apache.parquet', name: 'parquet-hadoop', version: '1.10.1'
// https://mvnrepository.com/artifact/org.apache.spark/spark-sketch
compileOnly group: 'org.apache.spark', name: 'spark-sketch_2.11', version: '2.4.5'
// https://mvnrepository.com/artifact/org.apache.spark/spark-core
compileOnly group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.4.5'
// https://mvnrepository.com/artifact/org.apache.spark/spark-catalyst
compileOnly group: 'org.apache.spark', name: 'spark-catalyst_2.11', version: '2.4.5'
// https://mvnrepository.com/artifact/org.apache.spark/spark-tags
compileOnly group: 'org.apache.spark', name: 'spark-tags_2.11', version: '2.4.5'
compileOnly group: 'org.apache.spark', name: 'spark-avro_2.11', version: '2.4.5'
// https://mvnrepository.com/artifact/org.apache.spark/spark-hive
compileOnly group: 'org.apache.spark', name: 'spark-hive_2.11', version: '2.4.5'
// https://mvnrepository.com/artifact/org.apache.xbean/xbean-asm6-shaded
compile group: 'org.apache.xbean', name: 'xbean-asm7-shaded', version: '4.15'
// https://mvnrepository.com/artifact/org.codehaus.janino/commons-compiler
compileOnly group: 'org.codehaus.janino', name: 'commons-compiler', version: '3.0.9'
// https://mvnrepository.com/artifact/org.codehaus.janino/janino
compileOnly group: 'org.codehaus.janino', name: 'janino', version: '3.0.9'
//HIVE Metastore
compile group: 'org.postgresql', name: 'postgresql', version: '42.2.9'
compile 'com.google.guava:guava:22.0'
compile group: 'org.apache.hadoop', name: 'hadoop-common', version: '3.2.1'
compile group: 'org.apache.hadoop', name: 'hadoop-aws', version: '3.2.1'
compile group: 'org.apache.hadoop', name: 'hadoop-client', version: '3.2.1'
compile group: 'com.amazonaws', name: 'aws-java-sdk-bundle', version: '1.11.271'
compile group: 'io.delta', name: 'delta-core_2.11', version: '0.5.0'
compile group: 'joda-time', name: 'joda-time', version: '2.10.5'
compile group: 'org.projectlombok', name: 'lombok', version: '1.16.6'
}
shadowJar {
zip64 true
}
Spark job:
df.distinct()
.withColumn("date", date_format(col(EFFECTIVE_PERIOD_START), "yyyy-MM-dd"))
.repartition(col("date"))
.write()
.format(fileFormat)
.partitionBy("date")
.mode(SaveMode.Append)
.option("fs.s3a.committer.name", "partitioned")
.option("fs.s3a.committer.staging.conflict-mode", "append")
.option("spark.sql.sources.commitProtocolClass", "org.apache.spark.internal.io.cloud.PathOutputCommitProtocol")
.option("spark.sql.parquet.output.committer.class", "org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter")
.option("compression", compressionCodecName.name().toLowerCase())
.save(DOWNLOADS_NON_COMPACT_PATH);
Executed script:
spark-submit --class com.felix.DataIngestionApplication --master local DataIngestion-1.0-SNAPSHOT-all.jar
From what I understand is the hadoop version is creating the issue All the hadoop-* JARs need to be 100% matching on versions. So I have ensured that all org.apache.hadoop dependencies are of the same version (3.2.1). But still it's giving this error.
I want to use hadoop version 3 or newer since that provides newer S3A committers like "PartitionedStagingCommitter". How does everybody using this with Spark 2.4.5?
How can I force/override hadoop version to use as 3.2.1 instead of hadoop versions in Spark/jars? When I looked at /usr/local/Cellar/apache-spark/2.4.5/libexec/jars/ I could see hadoop-common-2.7.3.jar, hadoop-client-2.7.3.jar etc. So how do we force hadoop newer version and therefore I could leverage new S3A comitters.?
Note: If I don't using spark-submit and instead run the application from IntelliJ with all dependencies as compile, then the app starts and executes without exception. I could see the data getting inserted in S3.
I have got this working by installing Spark distribution without Hadoop (user provided Hadoop version option) Then install hadoop version 3.2.1 (brew install hadoop) and create spark-env.sh from spark-env.sh.template file and add following line in spark-env.sh (/usr/local/spark-2.4.5/conf/):
export SPARK_DIST_CLASSPATH=$(hadoop classpath)
Now when I run spark-submit, the job executed without any issues
You can use force option in gradle.
compile(group: 'org.apache.hadoop', name: 'hadoop-aws', version: '3.1.1') {
force = true
}
compile(group: 'org.apache.hadoop', name: 'hadoop-common', version: '3.1.1') {
force = true
}

java.lang.RuntimeException: Error scanning entry module-info.class from jar

I try to run project with gretty. It throws exception:
10:47:39 WARN Failed startup of context o.a.g.JettyWebAppContext#52045dbe{/,file:/C:/IdeaProjects/example-tab/build/inplaceWebapp/,STARTING}
java.lang.RuntimeException: Error scanning entry module-info.class from jar file:/C:/Users/user1/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.10.0/fec5797a55b786184a537abd39c3fa1449d752d6/log4j-api-2.10.0.jar
at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:913) ~[jetty-annotations-9.2.24.v20180105.jar:9.2.24.v20180105]
at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:831) ~[jetty-annotations-9.2.24.v20180105.jar:9.2.24.v20180105]
at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:164) ~[jetty-annotations-9.2.24.v20180105.jar:9.2.24.v20180105]
at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:549) ~[jetty-annotations-9.2.24.v20180105.jar:9.2.24.v20180105]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) ~[jetty-util-9.2.24.v20180105.jar:9.2.24.v20180105]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) ~[jetty-util-9.2.24.v20180105.jar:9.2.24.v20180105]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171]
Caused by: java.lang.UnsupportedOperationException: null
at org.objectweb.asm.ClassVisitor.visitModule(ClassVisitor.java:129) ~[asm-6.1.1.jar:na]
at org.objectweb.asm.ClassReader.readModule(ClassReader.java:667) ~[asm-6.1.1.jar:na]
at org.objectweb.asm.ClassReader.accept(ClassReader.java:503) ~[asm-6.1.1.jar:na]
at org.objectweb.asm.ClassReader.accept(ClassReader.java:355) ~[asm-6.1.1.jar:na]
at org.eclipse.jetty.annotations.AnnotationParser.scanClass(AnnotationParser.java:974) ~[jetty-annotations-9.2.24.v20180105.jar:9.2.24.v20180105]
at org.eclipse.jetty.annotations.AnnotationParser.parseJarEntry(AnnotationParser.java:956) ~[jetty-annotations-9.2.24.v20180105.jar:9.2.24.v20180105]
at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:909) ~[jetty-annotations-9.2.24.v20180105.jar:9.2.24.v20180105]
... 6 common frames omitted
gradle:
plugins {
id 'java'
id 'war'
id "org.gretty" version "2.2.0"
}
group 'ru.example'
version '1.0-SNAPSHOT'
sourceCompatibility = 1.8
targetCompatibility = 1.8
compileJava.options.encoding = 'UTF-8'
ext {
SPRING_VERSION = '5.1.0.RELEASE'
JUNIT_VERSION = '5.3.1'
JACKSON_VERSION = '2.9.7'
}
repositories {
mavenCentral()
jcenter()
maven {
url 'https://oss.sonatype.org/content/repositories/releases/'
}
}
gretty {
springBootVersion = '2.0.5.RELEASE'
contextPath = "/"
}
dependencies {
annotationProcessor group: 'org.projectlombok', name: 'lombok', version: '1.18.2'
compileOnly group: 'org.projectlombok', name: 'lombok', version: '1.18.2'
compileOnly group: 'com.google.appengine', name: 'appengine-endpoints-deps', version: '1.9.65'
compile group: 'ch.qos.logback', name: 'logback-classic', version: '1.2.3'
compile group: 'org.springframework', name: 'spring-webmvc', version: SPRING_VERSION
compile group: 'org.springframework', name: 'spring-web', version: SPRING_VERSION
compile group: 'org.springframework', name: 'spring-jdbc', version: SPRING_VERSION
compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-jpa', version: '2.0.5.RELEASE'
compile group: 'com.mchange', name: 'c3p0', version: '0.9.5.2'
compile group: 'com.zaxxer', name: 'HikariCP', version: '3.2.0'
compile group: 'org.mariadb.jdbc', name: 'mariadb-java-client', version: '2.3.0'
compile group: 'com.fasterxml.jackson.core', name: 'jackson-core', version: JACKSON_VERSION
compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: JACKSON_VERSION
compile group: 'org.apache.httpcomponents', name: 'httpclient', version: '4.5.6'
compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.9.7'
testImplementation group: 'org.junit.jupiter', name: 'junit-jupiter-api', version: JUNIT_VERSION
testRuntimeOnly group: 'org.junit.jupiter', name: 'junit-jupiter-engine', version: JUNIT_VERSION
testCompile group: 'org.junit.jupiter', name: 'junit-jupiter-params', version: '5.3.1'
testCompile group: 'org.mockito', name: 'mockito-core', version: '2.22.0'
testCompile group: 'org.springframework', name: 'spring-test', version: SPRING_VERSION
testCompile group: 'org.hamcrest', name: 'hamcrest-library', version: '1.3'
testCompile group: 'com.jayway.jsonpath', name: 'json-path-assert', version: '2.4.0'
}
test {
useJUnitPlatform()
}
Scanning internet not helped. Please advice how to run it correctly. I had problems with downloading dependencies from jcenter(), may be, problem is here.
Validator write that my question have mostly code, but I have no Idea what to add to my explain, than, my apologies, I'll quote Steve Jobs:
“You can’t connect the dots looking forward; you can only connect them looking backward. So you have to trust that the dots will somehow connect in your future.”
You introduced a JEP-238 Jar file into your project: log4j-api-2.10.0.jar
Support for JEP-238 Jar files on any Java Runtime (doesn't matter if it's Java 7, Java 8, Java 9, Java 10, Java 11, or even the new Java 12-ea releases) was first introduced in Jetty 9.4.9.
You have 2 options:
Don't use JEP-238 Jar Files in your project.
Upgrade to Jetty 9.4.9 (or newer) to have support for JEP-238 Jar Files.
See past answers:
https://stackoverflow.com/a/45362629/775715
https://stackoverflow.com/a/52722120/775715

IntelliJ IDEA plugin dependency ClassNotFoundException

I created plugin with the following dependencies:
compile group: 'org.apache.httpcomponents', name: 'fluent-hc', version: '4.5.6'
compile group: 'com.google.code.gson', name: 'gson', version: '2.8.5'
compile group: 'joda-time', name: 'joda-time', version: '2.10'
compile group: 'com.google.guava', name: 'guava', version: '26.0-jre'
and it is working perfect.
Then I added more dependencies:
compile group: 'javax.validation', name: 'validation-api', version: '2.0.1.Final'
compile group: 'org.hibernate.validator', name: 'hibernate-validator', version: '6.0.13.Final'
compile group: 'javax.el', name: 'javax.el-api', version: '3.0.0'
compile group: 'org.glassfish', name: 'javax.el', version: '3.0.0'
and it is working perfect when I tested plugin with runIde task on local.
When I added jar file to IDEA I got exception: java.lang.NoClassDefFoundError: javax/validation/Validation
In both cases jar file does't contain any dependencies.
What did I miss?
UPDATE
6 years old question:
How does Intellij IDEA manage plugin dependencies?
Is it still valid? If so how I can put my dependencies in the plugin jar?
Im doing now:
gradle clean buildPlugin
Update build.gradle
jar {
from configurations.compile.collect { it.isDirectory() ? it : zipTree(it) }
}
Based on response here I need to submit zip file instead of jar. Zip contains all dependencies.

Categories