I am trying to use DataStax Enterprise 4.6 to write a Spark application in Java and run
it in DSE's Spark analytics mode.
The code for creating a Spark context using DSEConfHelper:
SparkConf conf = DseSparkConfHelper.enrichSparkConf(new SparkConf())
.setAppName( "My application");
To use DSEConfHelper we need to import com.datastax.bdp.spark.DseSparkConfHelper
which is located in dse.jar.
In my pom.xml I have included the dependency:
<dependency>
<groupId>com.datastax</groupId>
<artifactId>bdp</artifactId>
<version>4.6.0</version>
</dependency>
But Maven cannot download dse.jar.
Please help me.
The reference for code for creating a Spark context is taken from:
http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/spark/sparkJavaApi.html
Edit: This has been entirely superceded by the com.datastax.dse.dse-spark-dependencies artifact. Add it to your pom.xml:
<dependencies>
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>dse-spark-dependencies</artifactId>
<version>${dse.version}</version>
<scope>provided</scope>
</dependency>
<dependencies>
<repositories>
<repository>
<id>DataStax-Repo</id>
<url>https://repo.datastax.com/public-repos/</url>
</repository>
</repositories>
See https://github.com/datastax/SparkBuildExamples for Maven, SBT, and Gradle example projects.
Original, outdated answer:
You have to manually install dse.jar as of right now. There are two ways of doing this.
Option 1
Install the JAR file using mvn install:
$ mvn install:install-file -Dfile=<path-to-dse.jar> -DgroupId=com.datastax -DartficactId=bdp -Dversion=4.6.0
Option 2
Manually copy dse.jar from your install location to ${project.basedir}/lib/. Then modify your pom.xml:
<dependency>
<groupId>com.datastax</groupId>
<artifactId>bdp</artifactId>
<version>4.6.0</version>
<scope>system</scope>
<systemPath>${project.basedir}/lib/dse.jar</systemPath>
</dependency>
I don't really know why you're calling the artifact "bdp", but for these purposes it doesn't matter, and I just used it as well.
dse.jar is part of DSE installation. If you are working in windows environment, you can find it here dse.jar
register,download and extact to find the jar in lib folder. The use the above answer to add it in your maven project.
Related
I am a beginner to Maven project. In my project, I am getting the error Missing artifact com.oracle:ojdbc6:jar:11.2.0.3, even though the jar was present in my repository at the correct folder. Can anyone help with this, please?
Unfortunately, due to the binary license, there is no public repository with the Oracle Driver JAR, so you cannot just add it to your pom file.
You have to add this jar manually:
First, you have to download ojdbc6.jar from here
click jar (2.6 MB) on the middle of the page.
Then put ojdbc6.jar in some folder in your project (let's use lib).
Then you have to add this in your dependencies section in your pom.xml:
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc</artifactId>
<version>11.2.0</version>
<scope>system</scope>
<systemPath>${basedir}/lib/ojdbc6.jar</systemPath>
</dependency>
Another option is to install this jar in your local maven repository:
mvn install:install-file -Dfile=path/to/ojdbc6.jar -DgroupId=com.oracle -DartifactId=ojdbc6 -Dversion=11.2.0 -Dpackaging=jar
And then you will be able to reference this dependency like this:
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc6</artifactId>
<version>11.2.0</version>
</dependency>
You have to choose what's best for you.
Remove the ojdbc6 folder from the .m2 repository completely and then maven update the project in enclipse that solved my problem
<repositories>
<repository>
<id>spring-releases</id>
<url>https://repo.spring.io/libs-release</url>
</repository>
</repositories>
Should solve the issue if you are using spring boot
Once you face the issue . Check in your maven user settings path . This will be a path something like :
C:\Users\ user name\ .m2\repository
Open the location and go to oracle\ojdbc6\11.2.0.3 folder and put the .jar on that location .Return back to eclipse perform maven update and your issue will be gone.
I am using maven for Eclipse to build a jar that will run on a remote server. My system is running OS X, the server is running CestOS.
For the project I need tensorflow library. Maven successfully resolves dependencies so I am able to run the project locally. However, on the server I am getting error that tensorflow library is not there because by default maven includes only macosx version. How can I force maven to substitute macosx version of tensorflow by the linux version during build?
TensorFlow java libraries for different platforms can be found here.
P.S. I already tried adding a dependency in pom with the system scope pointing to jar.
Try this in your POM:
<dependency>
<groupId>org.bytedeco.javacpp-presets</groupId>
<artifactId>tensorflow</artifactId>
<version>0.9.0-1.2</version>
</dependency>
<dependency>
<groupId>org.bytedeco.javacpp-presets</groupId>
<artifactId>tensorflow</artifactId>
<version>0.9.0-1.2</version>
<classifier>linux-x86_64</classifier>
</dependency>
Or linux-x86 instead, if your server is 32-bit.
Of course, defining a conditional dependency with profiles would be nice.
Judging by the jar names on the page you linked, the difference between the MacOs and Linux versions lies in the text after the version part on the jar name.
That is called the classifier (see Maven coordinates) and is an optional coordinate that gives an additional differentiation after the artifact version.
As already suggested by nandsito, and to expand on its answer, try this (untested, let me know and I'll update):
<profiles>
<profile>
<id>osx</id>
<dependencies>
<dependency>
<groupId>org.bytedeco.javacpp-presets</groupId>
<artifactId>tensorflow</artifactId>
<version>0.9.0-1.2</version>
<classifier>macosx-x86_64</classifier>
</dependency>
</dependencies>
</profile>
<profile>
<id>linux</id>
<dependencies>
<dependency>
<groupId>org.bytedeco.javacpp-presets</groupId>
<artifactId>tensorflow</artifactId>
<version>0.9.0-1.2</version>
<classifier>linux-x86_64</classifier>
</dependency>
</dependencies>
</profile>
</profiles>
And remove all the related <dependency> nodes elsewere in your POM (so that without the <profiles> part there would be no dependency for tensorflow).
After this change you'll necessarily have to specify a profile each time (as there will be no tensorflow dependency in the POM): when preparing the package on MacOs mvn clean package -Pmacos and when preparing the package on Centos mvn clean package -Plinux
Eclipse allows you to set a list of active profiles under Project properties > Maven (you can get to this window by right-clicking on the project folder in the Project explorer.
I think you have to setup a build system such as Jenkins and have configuration for each target environment/platform. When building for Linux, configure the build system to run this command mvn clean package -Djavacpp.platform=linux-x86_64, the key point here is the parameter -Djavacpp.platform, change it according to your target platform.
I need to work with be.ac.ulg.montefiore.run.jahmm package for Hidden Markov Models in java.
My project is a mevenized project so I need to use the corresponding dependency.
<dependency>
<groupId>be.ac.ulg.montefiore.run.jahmm</groupId>
<artifactId>jahmm</artifactId>
<version>0.6.2</version>
</dependency>
The above dependency is not being resolved in my project.
Does anyone know how to help me?
Thank you.
Your dependency is not in the Maven Central Repository.
Find out the repository used by the authors to publish their artifacts and add this repository to your POM or to your settings.xml.
If it is not published to the usual online maven repositories, you will need to install it on your local machine first.
So download the sources, go to the top level and do the usual
mvn install
Once you installed jahmm locally, your project can resolve it from your maven cache.
Adding this repository should do the trick:
<repositories>
<repository>
<id>jsi</id>
<url>http://repo.springsource.org/libs-release-remote</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>be.ac.ulg.montefiore.run.jahmm</groupId>
<artifactId>jahmm</artifactId>
<version>0.6.2</version>
</dependency>
...
</dependencies>
All I want is the project file/jar for this project: http://thiagolocatelli.github.io/parse4j/
It says I need to do the following to obtain it:
Getting Started
Download the library manually
Maven
<project ...>
...
<dependencies>
<dependency>
<groupId>com.github.thiagolocatelli</groupId>
<artifactId>parse4j</artifactId>
<version>1.3</version>
</dependency>
</dependencies>
...
</project>
I have never used Maven, do not really know what it is. Can someone advice me how I obtain the project file?
Maven is a dependency manager. Lots of information about it if you're interested - just use your favorite search engine.
You can also download the jar file directly from http://search.maven.org/remotecontent?filepath=com/github/thiagolocatelli/parse4j/1.3/parse4j-1.3.jar
I am doing a project that has dependencies on some classes from the mahout and hadoop core jars. I was using javac with the classpath option to include them before, but someone suggested to me that I should use maven to build my project instead. However, I am not sure how to add the dependencies to these jar files which are located in my /usr/local directory.
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>0.20.205.0</version> <!-- or whatever version -->
</dependency>
<dependency>
<groupId>org.apache.mahout</groupId>
<artifactId>mahout-core</artifactId>
<version>0.5</version>
</dependency>
Add this to your pom:
<dependencies>
<dependency>
<groupId>org.apache.mahout</groupId>
<artifactId>mahout-core</artifactId>
<version>0.5</version>
</dependency>
<dependency>
<groupId>some.group</groupId>
<artifactId>hadoop</artifactId>
<version>some.version</version>
</dependency>
</dependencies>
If you have a copy of the jar to be used for say the hadoop example above, execute this command:
mvn install:install-file -Dfile=/some/path/my-hadoop.jar -DgroupId=some.group -DartifactId=hadoop -Dversion=some.version -Dpackaging=jar
Have a look at the maven documentation, especially the part on dependency management. If you want to use Maven you should get to know the basics (one of which is dependency management).
Basially you define your project's dependencies in the <dependencies> section of your pom. Look up maven central (the most common online repository) for the dependencies you want or search for other online repositories that might contain them.
If you can't find them, add the dependencies you want anyways (think of a sensible group id, artifact id and version) and try to compile. Maven will complain about the dependencies missing and provide a basic command to put those dependencies into the local repository. Copy those commands and fill in the appropriate path to the jar file and maven will deploy that dependency in your local repository.
Note that you should first look for the dependencies in an online repository since otherwise you'd have to manually deploy each new version in your local repo.