I am using maven for Eclipse to build a jar that will run on a remote server. My system is running OS X, the server is running CestOS.
For the project I need tensorflow library. Maven successfully resolves dependencies so I am able to run the project locally. However, on the server I am getting error that tensorflow library is not there because by default maven includes only macosx version. How can I force maven to substitute macosx version of tensorflow by the linux version during build?
TensorFlow java libraries for different platforms can be found here.
P.S. I already tried adding a dependency in pom with the system scope pointing to jar.
Try this in your POM:
<dependency>
<groupId>org.bytedeco.javacpp-presets</groupId>
<artifactId>tensorflow</artifactId>
<version>0.9.0-1.2</version>
</dependency>
<dependency>
<groupId>org.bytedeco.javacpp-presets</groupId>
<artifactId>tensorflow</artifactId>
<version>0.9.0-1.2</version>
<classifier>linux-x86_64</classifier>
</dependency>
Or linux-x86 instead, if your server is 32-bit.
Of course, defining a conditional dependency with profiles would be nice.
Judging by the jar names on the page you linked, the difference between the MacOs and Linux versions lies in the text after the version part on the jar name.
That is called the classifier (see Maven coordinates) and is an optional coordinate that gives an additional differentiation after the artifact version.
As already suggested by nandsito, and to expand on its answer, try this (untested, let me know and I'll update):
<profiles>
<profile>
<id>osx</id>
<dependencies>
<dependency>
<groupId>org.bytedeco.javacpp-presets</groupId>
<artifactId>tensorflow</artifactId>
<version>0.9.0-1.2</version>
<classifier>macosx-x86_64</classifier>
</dependency>
</dependencies>
</profile>
<profile>
<id>linux</id>
<dependencies>
<dependency>
<groupId>org.bytedeco.javacpp-presets</groupId>
<artifactId>tensorflow</artifactId>
<version>0.9.0-1.2</version>
<classifier>linux-x86_64</classifier>
</dependency>
</dependencies>
</profile>
</profiles>
And remove all the related <dependency> nodes elsewere in your POM (so that without the <profiles> part there would be no dependency for tensorflow).
After this change you'll necessarily have to specify a profile each time (as there will be no tensorflow dependency in the POM): when preparing the package on MacOs mvn clean package -Pmacos and when preparing the package on Centos mvn clean package -Plinux
Eclipse allows you to set a list of active profiles under Project properties > Maven (you can get to this window by right-clicking on the project folder in the Project explorer.
I think you have to setup a build system such as Jenkins and have configuration for each target environment/platform. When building for Linux, configure the build system to run this command mvn clean package -Djavacpp.platform=linux-x86_64, the key point here is the parameter -Djavacpp.platform, change it according to your target platform.
Related
Trying to use Maven to organize my project and I keep running into the following error. I know that this error means the file is present at compile time but for some reason it can't be found at runtime.
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/json/simple/parser/ParseException
So I'm working on a project in Java that will take a user query, search Google Images and then download some of the results onto my computer. To that end I've had to use some third party libraries like JSoup, Json-Simple, and Gson.
I initially added Jsoup to my classpath manually and it worked, but then I heard about Maven and started using it instead. My issue is that when I try to run my code I get the error above.
I'm just not sure how to resolve this. I've seen a bunch of other posts about similar errors and I've tried to modify my pom.xml accordingly but
I just can't get it to work. I've tried removing the ~/.m2 file, ran mvn clean, mvn install, mvn package, mvn compile, and it all works fine. But when it comes time to run, I keep getting that error.
Here's most of my pom.xml file.
<repositories>
<repository>
<id>central</id>
<name>Maven repository</name>
<url>http://repo1.maven.org/maven2</url>
</repository>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<!-- jsoup HTML parser library # https://jsoup.org/ -->
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.11.3</version>
<scope>provided</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/com.googlecode.json-simple/json-simple -->
<dependency>
<groupId>com.googlecode.json-simple</groupId>
<artifactId>json-simple</artifactId>
<version>1.1.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.google.code.gson/gson -->
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.5</version>
</dependency>
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>fully.qualified.MainClass</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>
FOUND SOLUTION: So I left out some parts of the pom.xml file to make it easier to read, and because all the other parts were generated by Maven itself so I figured there couldn't be an issue with anything there. But it adds a tag called "pluginManagement" that encloses all other plugins and apparently this does not allow the Shade plugin to run.
Between ngueno's guidance and this post Maven shade plugin is not called automatically for goal "package"
I was able to figure out my issue, though I'm still not entirely sure why it is an issue. Anyways, I figured I'd update this post in case someone else with a similar problem stumbles across it. This was on Mac OS btw, in case it makes a difference. Thanks for your help everyone.
Usually NoClassDefFound errors are related to missing libraries at runtime.
Since you are running using the terminal I supose you are building your project using Maven, and running the generated JAR,
I would recommend to you to use the maven-shade-plugin and generate an uber-jar as I explained on this question.
The purpose generating a uber-jar is to carry all the needed dependencies inside of it (available on the application classpath).
Implement the plugin and try to run using the new JAR.
PS: Remember to check this section related to Executable Jars
UPDATE: Remove the <scope>provided</scope> of your jsoup dependency, to enforce Maven to package it along your app, with the provided scope you are saying that this dependency will be provided by the JDK at runtime.
The jars that you identify in your dependencies must be present in the Runtime classpath.
This is not the classpath that is available when you compile the code;
it is the classpath on the host where you run the application.
You must install these jars on the target host.
Edit: More details
You must do the following:
Identify the runtime host.
Create a directory on the runtime host into which you will install the dependent jar files.
Include every jar in the classpath.
Consider abandoning the "roll-your-own" path.
If you use Spring Boot
(I like it,
I don't work for them).
One feature of spring boot is a reinvented "Fat Jar" that will include the dependencies inside one deliverable artifact (the fat jar) and will add them to the classpath at startup.
Edit:
The Spring boot executable jar file is not a "Fat Jar",
instead it includes the dependencies in a directory in the
executable jar and adds said jars to the classpath on startup.
Need to be pointed in the right direction on this perhaps, but if I add a "provided" dependency that is not included in the tomcat set of provided dependencies, running tomcat7:run from within eclipse fails with a classnotfoundexception on the class from the provided scope jar.
It needs to "provided" because it's a custom jar from a separate project that I've run mvn install on and for production am copying the jar to the $CATALINA_BASE/shared directory so that it's available (as a singleton) across applications/webapps.
<dependency>
<groupId>IndexFileAccessTracker</groupId>
<artifactId>IndexFileAccessTracker</artifactId>
<version>0.0.1-SNAPSHOT</version>
<scope>provided</scope>
</dependency>
Only way I see (with my limited knowledge of Maven and the Tomcat7 plugin) is to change the scope to compile when running tomcat from the plugin in Eclipse and then change the scope back to provided when running the package goal.
Are there solutions to this? I tried adding the dependency to the the tomcat maven plugin (keeping the main maven dependency as provided but got the same class not found error:
<!-- For Maven Tomcat Plugin -->
<plugin>
<groupId>org.apache.tomcat.maven</groupId>
<artifactId>tomcat7-maven-plugin</artifactId>
<version>2.2</version>
<configuration>
<path>/CounterWebApp</path>
</configuration>
<dependencies>
<dependency>
<groupId>IndexFileAccessTracker</groupId>
<artifactId>IndexFileAccessTracker</artifactId>
<version>0.0.1-SNAPSHOT</version>
<scope>compile</scope>
</dependency>
</dependencies>
</plugin>
Again, it needs to be provided in the main Maven dependency because I don't want it included in the deployed WAR.
Resolved by using profiles, similar to https://stackoverflow.com/a/5951630
...
</dependencies>
<profiles>
<profile>
<id>runineclipse</id>
<dependencies>
<dependency>
<groupId>IndexFileAccessTracker</groupId>
<artifactId>IndexFileAccessTracker</artifactId>
<version>0.0.1-SNAPSHOT</version>
<scope>compile</scope>
</dependency>
</dependencies>
</profile>
</profiles>
<build>
...
Then in my run/debug configuration just added runineclipse to the Profiles: box.
(On a side note, to do step through debugging I had to manually add the project to the Source tab.)
The build configuration was just the same package in the Goals: box; and I left the original dependency to have scope provided.
The tomcat7-maven-plugin and its run goal
Requires dependency resolution of artifacts in scope: test
Everythig that is on the compile classpath is also on the test classpath.
Thats why it is working with scope compile.
So the solution in your case would be to mark your dependency as test what even is (imo) semantically correct.
This will make the library available at local test-time, but not in the final artifact.
I am trying to use DataStax Enterprise 4.6 to write a Spark application in Java and run
it in DSE's Spark analytics mode.
The code for creating a Spark context using DSEConfHelper:
SparkConf conf = DseSparkConfHelper.enrichSparkConf(new SparkConf())
.setAppName( "My application");
To use DSEConfHelper we need to import com.datastax.bdp.spark.DseSparkConfHelper
which is located in dse.jar.
In my pom.xml I have included the dependency:
<dependency>
<groupId>com.datastax</groupId>
<artifactId>bdp</artifactId>
<version>4.6.0</version>
</dependency>
But Maven cannot download dse.jar.
Please help me.
The reference for code for creating a Spark context is taken from:
http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/spark/sparkJavaApi.html
Edit: This has been entirely superceded by the com.datastax.dse.dse-spark-dependencies artifact. Add it to your pom.xml:
<dependencies>
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>dse-spark-dependencies</artifactId>
<version>${dse.version}</version>
<scope>provided</scope>
</dependency>
<dependencies>
<repositories>
<repository>
<id>DataStax-Repo</id>
<url>https://repo.datastax.com/public-repos/</url>
</repository>
</repositories>
See https://github.com/datastax/SparkBuildExamples for Maven, SBT, and Gradle example projects.
Original, outdated answer:
You have to manually install dse.jar as of right now. There are two ways of doing this.
Option 1
Install the JAR file using mvn install:
$ mvn install:install-file -Dfile=<path-to-dse.jar> -DgroupId=com.datastax -DartficactId=bdp -Dversion=4.6.0
Option 2
Manually copy dse.jar from your install location to ${project.basedir}/lib/. Then modify your pom.xml:
<dependency>
<groupId>com.datastax</groupId>
<artifactId>bdp</artifactId>
<version>4.6.0</version>
<scope>system</scope>
<systemPath>${project.basedir}/lib/dse.jar</systemPath>
</dependency>
I don't really know why you're calling the artifact "bdp", but for these purposes it doesn't matter, and I just used it as well.
dse.jar is part of DSE installation. If you are working in windows environment, you can find it here dse.jar
register,download and extact to find the jar in lib folder. The use the above answer to add it in your maven project.
I'm developing an app using JOGL on Windows. I've been using Eclipse thus far but I have started writing a corresponding Maven POM file so I can automatic the build and packaging steps.
JOGL is not actively maintained in Maven so I have written a small script that imports the jars via install:install-file into my local repository :
set JOGL_VER=2.0
set JOGL_HOME=./jogl
set JOGL_LIB=%JOGL_HOME%/jar
set MVN_INSTALL=call mvn install:install-file
%MVN_INSTALL% -DgroupId=org.jogamp.gluegen -Dfile=%JOGL_LIB%/gluegen-rt-natives-windows-i586.jar \
-DartifactId=gluegen-rt-natives-windows-i586 -Dversion=%JOGL_VER% -Dpackaging=jar
%MVN_INSTALL% -DgroupId=org.jogamp.gluegen -Dfile=%JOGL_LIB%/gluegen.jar \
-DartifactId=gluegen -Dversion=%JOGL_VER% -Dpackaging=jar
%MVN_INSTALL% -DgroupId=org.jogamp.jogl -Dfile=%JOGL_LIB%/jogl-all-natives-windows-i586.jar \
-DartifactId=jogl-all-natives-windows-i586 -Dversion=%JOGL_VER% -Dpackaging=jar
%MVN_INSTALL% -DgroupId=org.jogamp.jogl -Dfile=%JOGL_LIB%/jogl-all.jar
-DartifactId=jogl-all -Dversion=%JOGL_VER% -Dpackaging=jar
This results in the following files in my repo
.m2\repository\org\jogamp\gluegen\gluegen\2.0\gluegen-2.0.jar
.m2\repository\org\jogamp\gluegen\gluegen\2.0\gluegen-2.0.pom
.m2\repository\org\jogamp\gluegen\gluegen-rt-natives-windows-i586\2.0\gluegen-rt-natives-windows-i586-2.0.jar
.m2\repository\org\jogamp\gluegen\gluegen-rt-natives-windows-i586\2.0\gluegen-rt-natives-windows-i586-2.0.pom
.m2\repository\org\jogamp\jogl\jogl-all\2.0\jogl-all-2.0.jar
.m2\repository\org\jogamp\jogl\jogl-all\2.0\jogl-all-2.0.pom
.m2\repository\org\jogamp\jogl\jogl-all-natives-windows-i586\2.0\jogl-all-natives-windows-i586-2.0.jar
.m2\repository\org\jogamp\jogl\jogl-all-natives-windows-i586\2.0\jogl-all-natives-windows-i586-2.0.pom
Note that since I specified 2.0, the files get suffixed with 2.0, e.g. gluegen-rt-natives-windows-i586-2.0.jar.
But now I want to use the exec:java command to run the app after a build, to ensure it functions i.e.
mvn exec:java
So I add the exec-maven-plugin to my pom.xml
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.2.1</version>
<configuration>
<mainClass>com.testapp.App</mainClass>
</configuration>
</plugin>
</plugins>
</build>
And I also add the dependencies to JOGL. Not that the native binaries are scoped runtime since I don't need them at compile time:
<dependency>
<groupId>org.jogamp.gluegen</groupId>
<artifactId>gluegen</artifactId>
<version>2.0</version>
</dependency>
<dependency>
<groupId>org.jogamp.jogl</groupId>
<artifactId>jogl-all</artifactId>
<version>2.0</version>
</dependency>
<dependency>
<groupId>org.jogamp.gluegen</groupId>
<artifactId>gluegen-rt-natives-windows-i586</artifactId>
<version>2.0</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.jogamp.jogl</groupId>
<artifactId>jogl-all-natives-windows-i586</artifactId>
<version>2.0</version>
<scope>runtime</scope>
</dependency>
But when I run this, I get the following error
Catched FileNotFoundException: C:\Users\xxx\.m2\repository\org\jogamp\gluegen\gluegen\2.0\gluegen-2.0-natives-windows-i586.jar (
The system cannot find the file specified), while TempJarCache.bootstrapNativeLib() of jar:file:/C:/Users/xxx/.m2/repository/org
/jogamp/gluegen/gluegen/2.0/gluegen-2.0-natives-windows-i586.jar!/ (file:/C:/Users/xxx/.m2/repository/org/jogamp/gluegen/gluegen
/2.0/ + gluegen-2.0-natives-windows-i586.jar)
[WARNING]
The issue is therefore straightforward. I installed the jars via install:install-file and the version 2.0 was appended after the artifact id e.g. gluegen-rt-natives-windows-i586-2.0.jar, but the exec:java expects the jar to be called gluegen-2.0-natives-windows-i586.jar.
Since the project builds I must assume the compile phase is correctly looking for the jar files using the expected file name, but exec is not.
Why is it dumping the version number in the middle of the artifact id and how do I make it work properly? The artifact id is named according to convention so I don't understand why it would be split like this.
Exec:java has nothing to do with your problem.
JOGL itself has some sort of complex internal mechanism for managing native code via JNI, and that mechanism makes assumptions about file names which are incompatible with the Maven rules for naming files in the repository.
You're going to have to use the maven-assembly-plugin to copy the dependencies to a tree that has the names and shapes required by JOGL and execute from there, or find a way to reconfigure JOGL to tolerate Maven naming conventions.
I am doing a project that has dependencies on some classes from the mahout and hadoop core jars. I was using javac with the classpath option to include them before, but someone suggested to me that I should use maven to build my project instead. However, I am not sure how to add the dependencies to these jar files which are located in my /usr/local directory.
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>0.20.205.0</version> <!-- or whatever version -->
</dependency>
<dependency>
<groupId>org.apache.mahout</groupId>
<artifactId>mahout-core</artifactId>
<version>0.5</version>
</dependency>
Add this to your pom:
<dependencies>
<dependency>
<groupId>org.apache.mahout</groupId>
<artifactId>mahout-core</artifactId>
<version>0.5</version>
</dependency>
<dependency>
<groupId>some.group</groupId>
<artifactId>hadoop</artifactId>
<version>some.version</version>
</dependency>
</dependencies>
If you have a copy of the jar to be used for say the hadoop example above, execute this command:
mvn install:install-file -Dfile=/some/path/my-hadoop.jar -DgroupId=some.group -DartifactId=hadoop -Dversion=some.version -Dpackaging=jar
Have a look at the maven documentation, especially the part on dependency management. If you want to use Maven you should get to know the basics (one of which is dependency management).
Basially you define your project's dependencies in the <dependencies> section of your pom. Look up maven central (the most common online repository) for the dependencies you want or search for other online repositories that might contain them.
If you can't find them, add the dependencies you want anyways (think of a sensible group id, artifact id and version) and try to compile. Maven will complain about the dependencies missing and provide a basic command to put those dependencies into the local repository. Copy those commands and fill in the appropriate path to the jar file and maven will deploy that dependency in your local repository.
Note that you should first look for the dependencies in an online repository since otherwise you'd have to manually deploy each new version in your local repo.