I set up Apache Spark maven dependency in pom.xml as follows
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>0.9.1</version>
</dependency>
But I found that this dependency use "hadoop-client-1.0.4.jar" and "hadoop-core-1.0.4.jar", and when I run my program I got the error "org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4", which shows that I need to switch hadoop version from 1.0.4 to 2.2.0.
Updates:
Is the following solution a correct method to solve this problem?
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>0.9.1</version>
<exclusions>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.2.0</version>
</dependency>
Many thanks for your help.
Recompile Spark for your Hadoop version, see "A Note About Hadoop Versions" here: http://spark.apache.org/docs/0.9.1/ . They conveniently give an example for 2.2.0
SPARK_HADOOP_VERSION=2.2.0 sbt/sbt assembly
This will create a new jar, $SPARK_HOME/assembly/target/scala-2.10/spark-assembly-*jar that you need to include into your pom.xml (instead of excluding Hadoop from the online jar).
If you're already hosting your own repository (e.g. on Nexus) then upload it there (this is what I do and it works great). If for some reason you can't upload to any repository, use Maven's install:install-file or one of the answers here Maven: add a dependency to a jar by relative path
Spark 1.2.0 depends on hadoop 2.2.0 be default.
If you can update your spark dependency to 1.2.0 (or newer) that will solve the problem.
Related
I am writing a SpringBoot 2.0.4 application, (Executable .jar) and I am trying to interface with an elastic search that I do not control. It is 6.2.2
Now I know Spring does not support that, and that's fine as I am not using any Spring classes, I am interfacing with elastic directly. However, The problem comes in trying to exclude elastic from spring in my pom. Here is my pom fragment:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-elasticsearch</artifactId>
</exclusion>
<exclusion>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>6.2.2</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>6.2.2</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-client</artifactId>
<version>6.2.2</version>
</dependency>
Now Eclipse has a yellow line under the versions for elasticsearch and transport, saying that the version is being overridden. But when I look at the dependency tree, I see all the elastic jars with the correct version of 6.2.2 -- EXCEPT for transport. That jar is not there at all. I looked in my executable jar, and it's definitely not there. Shockingly, my app throws a class not found error for org/elasticsearch/common/transport/InetSocketTransportAddress
How do I get that jar into my dependencies?
EDIT
So I added:
<properties>
<elasticsearch.version>6.2.2</elasticsearch.version>
</properties>
to my pom, and overrode the spring version, but the transport jar is not showing up in my dependency tree! How do I get this file into my executable jar?
I met problem when my project and jar which is used in my project did not used the same version.
Below is my pom.xml in my project:
<dependency>
<groupId>org.apache.sshd</groupId>
<artifactId>sshd-core</artifactId>
<version>1.6.0</version>
<type>jar</type>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.another.project.server</groupId>
<artifactId>exampleserver</artifactId>
<version>1.0</version>
<type>jar</type>
<scope>test</scope>
</dependency>
My exampleserver jar is using sshd-core version 0.14.0, but my project is using sshd-core version 1.6.0. So it will be failed because all implementation of sshd-core 0.14.0 and sshd-core version 1.6.0 is different.
Are there anyway to specify exactly sshd version for exampleserver ? Any help would be appreciated.
The unfortunate reality is that you can have just one version of a given jar when you compile/run the project. So you need to decide which version of sshd-core version is ok for all uses in your project (including transitive ones).
If you figure out that version x.y.z is a good fit both for your project and exampleserver you can set it through <dependencyManagement>. If this version happens to be 1.6.0 you can also use the exclusion mechanism described in the other answers.
If you do not find a version that fits for all purposes, you will have a hard time. You can try class shading or you need to rewrite the code.
The sshd version from exampleserver is defined in the POM of exampleserver. If this is not your project, you cannot change that version. What you could do, however, is exclude the sshd dependency from exampleserver, so that you will only have the 1.6.0 version on your classpath. E.g.:
<dependency>
<groupId>com.another.project.server</groupId>
<artifactId>exampleserver</artifactId>
<version>1.0</version>
<type>jar</type>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.apache.sshd</groupId>
<artifactId>sshd-core</artifactId>
</exclusion>
</exclusions>
</dependency>
You can use exclusions.
Since maven 2.x resolves dependencies transitively, it is possible for unwanted dependencies to be included in your project's classpath.
(https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html):
In your example
<dependency>
<groupId>com.another.project.server</groupId>
<artifactId>exampleserver</artifactId>
<version>1.0</version>
<type>jar</type>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.apache.sshd</groupId>
<artifactId>sshd-core</artifactId>
</exclusion>
</exclusions>
</dependency>
What Google Maven dependency could fix this error:
java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;J)V
at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:487)
at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127)
at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:94)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54)
at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:124)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.io.InputStream.read(InputStream.java:101)
Code:
Blob blob = storage.get(blobId);
if(blob.exists()) {
return true;
}
Your Google guava version is either too old (< 20.0) or mismatched (multiple jars versions). Make sure you don't have several versions in your dependency tree.
Use
mvn dependency:tree | less
to look for the guava versions.
Please add following dependencies to your project's POM:
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>23.6-jre</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.4.8</version>
</dependency>
In my case, I happened to include both
<dependency>
<groupId>com.google.collections</groupId>
<artifactId>google-collections</artifactId>
<version>1.0</version>
</dependency>
and
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>28.0-jre</version>
</dependency>
As it turns out, I cannot use both of these libraries. Removing google-collections fixed the issue for me.
Try inserting a dependency containing a newer version of guava at the top of your dependencies in your pom.xml containing your project.
E.g.
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>24.1.1-jre</version>
</dependency>
For anybody encountering anything like this, i had the following pom:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>3.2.1</version>
</dependency>
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20190722</version>
</dependency>
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.2</version>
</dependency>
And commenting out the maven-shade plugin fixed it for me...
I had the same issue in my Java/Kotlin application. When the app was ran via IntelliJ there were no issues. However, when the .Jar was ran the error message above was thrown.
I could not find a direct action item to take with the Guava issue defined by #Laurent Perez above so I did the following which resolved the issue with my .Jar file running:
Removed .Jar IntelliJ configurations and file from IntelliJ. Then re-added the .Jar following Step 3 from this deploy guide.
Other actions to try if the above does not work:
Rebuild project.
Invalidate IntelliJ cache and restart.
Restart computer.
It happens when you're missing guava library from your dependency. Add it using following in your pom.xml
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>24.0-jre</version>
</dependency>
Another reason could be there are multiple versions of google guava/collection library, you can't have both as mentioned by #sabbir in his comment, you ought to remove one of them.
Lastly it could be that some other dependency is dependent on the previous version of google guava/collections. Go to Dependency Hierarchy and search for google and see what all dependencies are dependent on the previous versions. In my case it was by version-maven-plugin, version 2.7
Here version-maven-plugin, version 2.7 uses google collections 1.0, and I require guava 24.0 Both can't work simuntaneously.
I excluded it in my pom.xml like this and then added a new dependency of guava
<dependency>
<groupId>org.codehaus.mojo</groupId>
<artifactId>versions-maven-plugin</artifactId>
<version>2.7</version>
<exclusions>
<exclusion>
<groupId>com.google.collections</groupId>
<artifactId>google-collections</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>24.0-jre</version>
</dependency>
Resultant: Now google-collections is gone.
I am facing this error while running my GWT application.
I have these jar files in my classpath: slf4j-api & slf4j-log4j12
Any idea what could be the reason?
This problem is due to a change in slf4j-log4j12 jar. From version 1.5.6 it doesn't allow to access the field org.slf4j.impl.StaticLoggerBinder.SINGLETON.
To resolve it, use the newest jars (or at least version 1.5.6 onward) for both slf4j-api & slf4j-log4j12.
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.5.6</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.5.6</version>
</dependency>
Finally resolved this problem in my SpringBoot application. If updating version is not helping this might help. Sometimes other libraries might bring different versions of this dependencies. These are the steps:
By error stack trace figure out which dependency is giving this issue
Get maven dependency plugin tree. Using this tree details find out if this library is coming as part of the some other dependency. In my case, the logback-classic and log4j-over-slf4j were giving this problem. They came together under spring-boot-starter-web
Use <exclusions><exclusion></exclusion></exclusions> in your pom.xml in that dependency for the libraries that giving this issue. In my case it looks like this:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<artifactId>logback-classic</artifactId>
<groupId>ch.qos.logback</groupId>
</exclusion>
<exclusion>
<artifactId>log4j-over-slf4j</artifactId>
<groupId>org.slf4j</groupId>
</exclusion>
</exclusions>
</dependency>
References:
http://www.slf4j.org/faq.html#IllegalAccessError
http://www.slf4j.org/codes.html#multiple_bindings
I need to import javax.jms.* classes. What is the right dependency to include into a Maven project? I'm trying javax.jms:jms:1.1, but no luck (it's pom, not jar).
ps. The only workaround I've found so far is: javax:javaee-api:6.0 (from Maven Central).
In ActiveMQ as well as some other projects like Qpid JMS we pull in the JMS spec classes from Apache Geronimo JARs, the 1.1 APIs are available in this dependency:
<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-jms_1.1_spec</artifactId>
<version>1.1.1</version>
</dependency>
For JMS 2 APIs you'd need to use a different dependency, for instance
<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-jms_2.0_spec</artifactId>
<version>1.0-alpha-2</version>
</dependency>
These are both Apache 2.0 licensed dependencies.
Another option which is not Apache licensed is here as others have pointed out.
<dependency>
<groupId>javax.jms</groupId>
<artifactId>javax.jms-api</artifactId>
<version>2.0.1</version>
</dependency>
The Sun license doesn't allow maven repositories to host this (and other) artifacts.
Here is the documentation explaining this and what you should do instead...
Maven - Guide to coping with Sun JARs
What it says is you need to download the JAR manually and then install it into your own local repository or nexus server.
The pom.xml files hosted at maven central for these artifacts contain information on where you can download the JARs from.
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-api</artifactId>
<version>6.0</version>
<scope>provided</scope>
</dependency>
I have successfully used this one:
<dependency>
<groupId>javax.jms</groupId>
<artifactId>jms</artifactId>
<version>1.1</version>
</dependency>
Go to Maven Search site and search for javax. Open the latest version for groupId javax and artifactId javaee-api
The current version is 7.0 [Maven dependency information]
If you just want the JMS libs, without the rest of javaee, use the following:
https://mvnrepository.com/artifact/javax.jms/javax.jms-api/2.0.1
<dependency>
<groupId>javax.jms</groupId>
<artifactId>javax.jms-api</artifactId>
<version>2.0.1</version>
</dependency>
According to mvnrepository, the dependency to add in the pom of your project is the following:
<dependency>
<groupId>jms</groupId>
<artifactId>jms</artifactId>
<version>1.1</version>
</dependency>
Check out the dependencies listed on grepcode.com.
I only discovered this site recently, and it rocks!
http://grepcode.com/search/?query=javax.jms.*
It looks like the Geronimo jars on maven central should sort your issues out.
This worked for myself
<dependency>
<groupId>javax.jms</groupId>
<artifactId>javax.jms-api</artifactId>
<version>2.0.1</version>
</dependency>