IntelliJ IDEA cannot import Spark mllib, when I added dependency in maven. With other parts of Spark no problems. In project Structure -> Libraries spark mllib is present.
import org.apache.spark.ml.feature.VectorAssembler; -> Cannot resolve symbol 'VectorAssembler'
pom.xml:
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.12</artifactId>
<version>3.0.0</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>3.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
</dependencies>
Project structure
I tried refreshing maven and clearing folder with repositories of maven. Nothing helped.
You specified mllib dependency as runtime - this means that dependency is required for execution, but not for compilation, so it won't be put into classpath for compiling your code. See this blog post for description of of different scopes available in Maven.
Replace all spark dependencies (mllib, core, sql) with just single dependency (also remove hadoop dependencies):
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_${spark.scala.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
where variables are defined as
<properties>
<spark.version>3.0.1</spark.version>
<spark.scala.version>2.12</spark.scala.version>
</properties>
I have several Java projects that pom.xml to define dependencies. It all works nicely but there's a lot of repetition so I am trying to move duplicate dependencies into a new parent pom file.
Let's say my parent pom.xml has the following:
<dependencyManagement>
<dependencies>
<!-- Hazelcast dependencies -->
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast</artifactId>
<version>${hazelcast.version}</version>
</dependency>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast-client</artifactId>
<version>${hazelcast.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
My child pom.xml has:
<dependencies>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast</artifactId>
</dependency>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast-client</artifactId>
</dependency>
</dependencies>
This isn't too bad as I'm only adding two artifacts. Is there a way to add the two dependencies with only one entry? I tried the following based on an online post, but it doesn't work but it does give an idea of what I was hoping for:
<dependencies>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast</artifactId>
<artifactId>hazelcast-client</artifactId>
</dependency>
</dependencies>
I'm also depending on several different Jetty artifacts so I'm trying to see if there's a way to reduce content. I know I can write out each artifact dependency separately in the child pom.xml files but I thought I'd see if there's a way to simplify.
Thanks
I am working on a library for some projects which relies on Spark and HBase.
So the POM of the library looks something link this:
<dependencies>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.7.4</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>1.1.2</version>
</dependency>
...
</dependencies>
And in the specific project that uses the central library (which is published on an internal Maven repository) I have this:
<dependencies>
<dependency>
<groupId>my.group.id</groupId>
<artifactId>myartifact</artifactId>
<version>LATEST</version>
</dependency>
</dependencies>
This, however does not then automatically include the dependencies that the library itself has. Therefore I would need to copy the dependency section of the library POM into the application POM.
Do you have any advice what might be missing/wrong?
Thanks and regards!
As part of learning, this is my first "spring nature" maven project,
In specific, I would like to understand the approach to know the list of dependencies that are required for any "spring nature" maven project, that I work in future.
For this project, here are the list of 21 dependencies that were just dumped into pom.xml without being told about, which dependency to use when, in this training video at 20:50?
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example.j2eeapp</groupId>
<artifactId>j2eeapplication</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>war</packaging>
<name>J2EE Applications Example</name>
<repositories>
<repository>
<id>prime-repo</id>
<name>PrimeFaces Maven Repository</name>
<url>http://repository.primefaces.org</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>4.1.8.Final</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-validator</artifactId>
<version>4.2.0.Final</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.webflow</groupId>
<artifactId>spring-webflow</artifactId>
<version>2.3.1.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.webflow</groupId>
<artifactId>spring-faces</artifactId>
<version>2.3.1.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-orm</artifactId>
<version>3.1.1.RELEASE</version>
</dependency>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc14</artifactId>
<version>10.2.0.1.0</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.16</version>
</dependency>
<dependency>
<groupId>com.sun.faces</groupId>
<artifactId>jsf-impl</artifactId>
<version>2.1.10</version>
</dependency>
<dependency>
<groupId>com.sun.facelets</groupId>
<artifactId>jsf-facelets</artifactId>
<version>1.1.14</version>
</dependency>
<dependency>
<groupId>com.sun.faces</groupId>
<artifactId>jsf-api</artifactId>
<version>2.1.10</version>
</dependency>
<dependency>
<groupId>commons-dbcp</groupId>
<artifactId>commons-dbcp</artifactId>
<version>20030825.184428</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.6.4</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.0.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-tx</artifactId>
<version>3.0.5.RELEASE</version>
</dependency>
<dependency>
<groupId>xml-apis</groupId>
<artifactId>xml-apis</artifactId>
<version>1.3.02</version>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-web</artifactId>
<version>3.1.3.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-config</artifactId>
<version>3.1.3.RELEASE</version>
</dependency>
<dependency>
<groupId>org.primefaces</groupId>
<artifactId>primefaces</artifactId>
<version>3.4</version>
</dependency>
<dependency>
<groupId>cglib</groupId>
<artifactId>cglib</artifactId>
<version>2.2.2</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>4.1.8.Final</version>
</dependency>
</dependencies>
</project>
So,
As of now, I do not have knowledge about bean/spring-webflow/hibernate etc...
What is the approach to know the dependencies required for my "spring nature" project?
Maven projects need dependencies instead of including jar files by build path.when you create a maven project, it requires the library files for the methods you use.you add the dependencies in pom.xml file and when you execute the maven build command the files are automatically downloaded from the internet and included in the project.
You can control the files to download.
you can just add dependencies of the library files you need in your source code.
you decide your dependencies by the methods you use in your source code.
for example.you are using sql database in your project you must need a jar file for the sql driver.
if you have a maven project you just have to add the dependency of the sql in pom.xml
Dependencies list depends on your project nature and usage of .jar relative files.
Simple if you want to use sql in your project then you only need to add sql dependencies in your pom.xml and in the same way if you use junit in your project then you will add junit dependencies in pom.xml in this way after tag
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.2</version>
</dependency>
no need of extra dependencies that are not being used in project.
Note that in maven dependencies repository you will see some extra dependencies. These are ones which maven used for itself or for other dependencies.
The "Spring nature" is related to how your IDE works with Spring plug-in. In eclipse, a "project nature" creates an association between the a project and a tool, plug-in, or feature set. By adding a nature to an eclipse project, you tell an eclipse plug-in that it is configured to use that project. By adding the "Spring Project Nature" to your project, you are enabling eclipse's spring plugin to work with your project.
Add dependencies on need basis. if you don't need hibernate or web-flow don't add it.Its like adding required toppings to your pizza. If you dont want mushroom, dont add it :)
I have a maven project where many of the version of child pom's dependencies are listed under <dependencyManagement> tag of the parent pom. This pom tree can have any depth.
My problem is, when a pom.xml is given how can I resolve the exact(inherited) version of it's dependencies using java?
The desired output is what is given when we run mvn dependency:resolve -DincludeTransitive=false on a pom.xml file
For an example, let's say parent pom has following dependencies defined in it under <dependencyManagement>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>group-a</groupId>
<artifactId>artifact-a</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>group-a</groupId>
<artifactId>artifact-b</artifactId>
<version>1.0</version>
<type>bar</type>
<scope>runtime</scope>
</dependency>
</dependencies>
</dependencyManagement>
and child pom.xml has following dependencies
<dependencies>
<dependency>
<groupId>group-a</groupId>
<artifactId>artifact-a</artifactId>
</dependency>
<dependency>
<groupId>group-a</groupId>
<artifactId>artifact-b</artifactId>
</dependency>
</dependencies>
How can i retrieve the versions of group-a:artifact-a and group-b:artifact-b using Java programmatically?
Thanks.
Try running
mvn dependency:tree -Doutput=/path/to/file
or
mvn dependency:tree -DoutputFile=/path/to/file
From maven docs: http://maven.apache.org/plugins/maven-dependency-plugin/tree-mojo.html