IntelliJ IDEA cannot import Spark mllib, when I added dependency in maven. With other parts of Spark no problems. In project Structure -> Libraries spark mllib is present.
import org.apache.spark.ml.feature.VectorAssembler; -> Cannot resolve symbol 'VectorAssembler'
pom.xml:
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.12</artifactId>
<version>3.0.0</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>3.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
</dependencies>
Project structure
I tried refreshing maven and clearing folder with repositories of maven. Nothing helped.
You specified mllib dependency as runtime - this means that dependency is required for execution, but not for compilation, so it won't be put into classpath for compiling your code. See this blog post for description of of different scopes available in Maven.
Replace all spark dependencies (mllib, core, sql) with just single dependency (also remove hadoop dependencies):
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_${spark.scala.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
where variables are defined as
<properties>
<spark.version>3.0.1</spark.version>
<spark.scala.version>2.12</spark.scala.version>
</properties>
Related
I am using concurrent tree library in two different modules. When I try to build the project (I am using buck to build the project), second module gives me the following errors:
error: package com.googlecode.concurrenttrees.radixinverted does not exist
import com.googlecode.concurrenttrees.radixinverted.ConcurrentInvertedRadixTree;
error: package com.googlecode.concurrenttrees.radixinverted does not exist
I checked all of the previous posts about how to import dependencies and the concurrent-tree maven library has been added successfully into the module and it detects all of its functions in a class. I tried everything that you think such as restarting and invalidating cache, opening a new project, etc but none of those work.
When I move my code under the first module it compiles without any problems but this is not the solution that I am looking for.
http://maven.apache.org/maven-v4_0_0.xsd">
4.0.0
<parent>
<groupId>org.onosproject</groupId>
<artifactId>onos-incubator</artifactId>
<version>1.10.0-SNAPSHOT</version>
</parent>
<artifactId>onos-incubator-api</artifactId>
<packaging>bundle</packaging>
<description>ONOS incubating core API</description>
<dependencies>
<dependency>
<groupId>com.googlecode.concurrent-trees</groupId>
<artifactId>concurrent-trees</artifactId>
<version>2.6.0</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava-testlib</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.easymock</groupId>
<artifactId>easymock</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.felix</groupId>
<artifactId>org.apache.felix.scr.annotations</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>RELEASE</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>compile</scope>
</dependency>
</dependencies>
When I run mvn clean install, it doesn't generate any errors but when I try to build the project using buck it generates error
I am working on a library for some projects which relies on Spark and HBase.
So the POM of the library looks something link this:
<dependencies>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.7.4</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>1.1.2</version>
</dependency>
...
</dependencies>
And in the specific project that uses the central library (which is published on an internal Maven repository) I have this:
<dependencies>
<dependency>
<groupId>my.group.id</groupId>
<artifactId>myartifact</artifactId>
<version>LATEST</version>
</dependency>
</dependencies>
This, however does not then automatically include the dependencies that the library itself has. Therefore I would need to copy the dependency section of the library POM into the application POM.
Do you have any advice what might be missing/wrong?
Thanks and regards!
As part of learning, this is my first "spring nature" maven project,
In specific, I would like to understand the approach to know the list of dependencies that are required for any "spring nature" maven project, that I work in future.
For this project, here are the list of 21 dependencies that were just dumped into pom.xml without being told about, which dependency to use when, in this training video at 20:50?
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example.j2eeapp</groupId>
<artifactId>j2eeapplication</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>war</packaging>
<name>J2EE Applications Example</name>
<repositories>
<repository>
<id>prime-repo</id>
<name>PrimeFaces Maven Repository</name>
<url>http://repository.primefaces.org</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>4.1.8.Final</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-validator</artifactId>
<version>4.2.0.Final</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.webflow</groupId>
<artifactId>spring-webflow</artifactId>
<version>2.3.1.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.webflow</groupId>
<artifactId>spring-faces</artifactId>
<version>2.3.1.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-orm</artifactId>
<version>3.1.1.RELEASE</version>
</dependency>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc14</artifactId>
<version>10.2.0.1.0</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.16</version>
</dependency>
<dependency>
<groupId>com.sun.faces</groupId>
<artifactId>jsf-impl</artifactId>
<version>2.1.10</version>
</dependency>
<dependency>
<groupId>com.sun.facelets</groupId>
<artifactId>jsf-facelets</artifactId>
<version>1.1.14</version>
</dependency>
<dependency>
<groupId>com.sun.faces</groupId>
<artifactId>jsf-api</artifactId>
<version>2.1.10</version>
</dependency>
<dependency>
<groupId>commons-dbcp</groupId>
<artifactId>commons-dbcp</artifactId>
<version>20030825.184428</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.6.4</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.0.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-tx</artifactId>
<version>3.0.5.RELEASE</version>
</dependency>
<dependency>
<groupId>xml-apis</groupId>
<artifactId>xml-apis</artifactId>
<version>1.3.02</version>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-web</artifactId>
<version>3.1.3.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-config</artifactId>
<version>3.1.3.RELEASE</version>
</dependency>
<dependency>
<groupId>org.primefaces</groupId>
<artifactId>primefaces</artifactId>
<version>3.4</version>
</dependency>
<dependency>
<groupId>cglib</groupId>
<artifactId>cglib</artifactId>
<version>2.2.2</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>4.1.8.Final</version>
</dependency>
</dependencies>
</project>
So,
As of now, I do not have knowledge about bean/spring-webflow/hibernate etc...
What is the approach to know the dependencies required for my "spring nature" project?
Maven projects need dependencies instead of including jar files by build path.when you create a maven project, it requires the library files for the methods you use.you add the dependencies in pom.xml file and when you execute the maven build command the files are automatically downloaded from the internet and included in the project.
You can control the files to download.
you can just add dependencies of the library files you need in your source code.
you decide your dependencies by the methods you use in your source code.
for example.you are using sql database in your project you must need a jar file for the sql driver.
if you have a maven project you just have to add the dependency of the sql in pom.xml
Dependencies list depends on your project nature and usage of .jar relative files.
Simple if you want to use sql in your project then you only need to add sql dependencies in your pom.xml and in the same way if you use junit in your project then you will add junit dependencies in pom.xml in this way after tag
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.2</version>
</dependency>
no need of extra dependencies that are not being used in project.
Note that in maven dependencies repository you will see some extra dependencies. These are ones which maven used for itself or for other dependencies.
The "Spring nature" is related to how your IDE works with Spring plug-in. In eclipse, a "project nature" creates an association between the a project and a tool, plug-in, or feature set. By adding a nature to an eclipse project, you tell an eclipse plug-in that it is configured to use that project. By adding the "Spring Project Nature" to your project, you are enabling eclipse's spring plugin to work with your project.
Add dependencies on need basis. if you don't need hibernate or web-flow don't add it.Its like adding required toppings to your pizza. If you dont want mushroom, dont add it :)
I have maven projects A and B. Project A uses maven embedder 3.2.5 and builds with it project B.
MavenCli mavenCli = new MavenCli();
mavenCli.doMain(new String[] {"package"}, "C:\\Users\\JohnDoe\\Desktop\\abc", System.out, System.out);
But I get the exception:
Plugin org.apache.maven.plugins:maven-compiler-plugin:3.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-compiler-plugin:jar:3.1: Could not transfer artifact org.apache.maven.plugins:maven-compiler-plugin:pom:3.1 from/to central (https://repo.maven.apache.org/maven2): No connector factories available
I looked into my local repository folder and there is indeed no plugin version 3.1. If I manually put it there the build is successful. But AFAIK maven automatically should download all necessary plugins. But it doesn't happen.
Adding these dependencies solved my problem
<dependency>
<groupId>org.eclipse.aether</groupId>
<artifactId>aether-connector-basic</artifactId>
<version>1.0.2.v20150114</version>
</dependency>
<dependency>
<groupId>org.eclipse.aether</groupId>
<artifactId>aether-transport-wagon</artifactId>
<version>1.0.2.v20150114</version>
</dependency>
<dependency>
<groupId>org.apache.maven.wagon</groupId>
<artifactId>wagon-http</artifactId>
<version>2.8</version>
</dependency>
<dependency>
<groupId>org.apache.maven.wagon</groupId>
<artifactId>wagon-provider-api</artifactId>
<version>2.8</version>
</dependency>
If you want to deploy, you also need this dependency:
<dependency
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.3</version>
</dependency>
use these dependencies
<dependency>
<groupId>org.apache.maven</groupId>
<artifactId>maven-embedder</artifactId>
<version>3.6.3</version>
</dependency>
<dependency>
<groupId>org.apache.maven</groupId>
<artifactId>maven-compat</artifactId>
<version>3.6.3</version>
</dependency>
<dependency>
<groupId>org.apache.maven.resolver</groupId>
<artifactId>maven-resolver-connector-basic</artifactId>
<version>1.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.maven.resolver</groupId>
<artifactId>maven-resolver-transport-http</artifactId>
<version>1.4.1</version>
</dependency>
use Apache Maven Artifact Resolver instead of Aether which is archived.
I updated my application (Spring Boot 2.3, JDK8/11) to the latest Maven Embedded 3.6.3 version with the following dependencies:
<!-- Maven Embedder -->
<dependency>
<groupId>org.apache.maven</groupId>
<artifactId>maven-embedder</artifactId>
<version>3.6.3</version>
</dependency>
<dependency>
<groupId>org.apache.maven</groupId>
<artifactId>maven-compat</artifactId>
<version>3.6.3</version>
</dependency>
<dependency>
<groupId>org.apache.maven.wagon</groupId>
<artifactId>wagon-http</artifactId>
<version>3.3.4</version>
</dependency>
<dependency>
<groupId>org.eclipse.aether</groupId>
<artifactId>aether-connector-basic</artifactId>
<version>1.1.0</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.eclipse.aether</groupId>
<artifactId>aether-transport-wagon</artifactId>
<version>1.1.0</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.30</version>
</dependency>
HTH
My pom.xml looks as follows:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>be.ugent.sop.p404</groupId>
<artifactId>P404</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<artifactId>P404EJB</artifactId>
<name>P404EJB</name>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-ejb-plugin</artifactId>
<configuration>
<ejbVersion>3.1</ejbVersion>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>2.40.0</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.29</version>
</dependency>
<dependency>
<groupId>org.eclipse.persistence</groupId>
<artifactId>org.eclipse.persistence.jpa</artifactId>
<version>2.5.2-M1</version>
</dependency>
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-api</artifactId>
<version>7.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
</dependency>
<dependency>
<groupId>org.databene</groupId>
<artifactId>contiperf</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.6</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.6</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
</dependency>
<dependency>
<groupId>org.databene</groupId>
<artifactId>contiperf</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-j2ee-deployment_1.1_spec</artifactId>
<version>1.1</version>
</dependency>
<dependency>
<groupId>com.googlecode.json-simple</groupId>
<artifactId>json-simple</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<version>1.4</version>
</dependency>
</dependencies>
<packaging>ejb</packaging>
As one of the examples of the problems I am receiving:
Description Resource Path Location Type
Project 'P404EJB' is missing required library: '/home/zeesteen/.m2/repository/com/googlecode/json-simple/json-simple/1.1.1/json-simple-1.1.1.jar' P404EJB Build path Build Path Problem
I assumed that after using the following command:
mvn clean install eclipse:eclipse
Maven was supposed to fetch libraries that are in the pom and not present on my machine yet? (The folders, for example "com" don't exist in the .m2/repository folder)
I have tried multiple solutions myself such as
- Removing the Maven nature, clean installing again, converting to a maven project
- Doing variations on the clean command
- Right clicking on the project and doing Maven > update Project
- Cleaning the project through eclipse
I got 124 Java Build Path Problems like the one mentioned above.
I found the mistake..
Maven didn't give errors because there was nothing wrong with my maven repo apparently. Eclipse had changed the location of my settings.xml maven file to a "nonexistent" location.. Resulting in all the problems. I manually changed it back and all problems are solved.
In Eclipse I did:
Window > Preferences > Maven > User Settings: Field User Settings was wrong
(eclipse changed it..) put it back to where I first made the settings, applied, cleaned the project and everything was fixed!
Please make sure that you have added Maven Dependencies under Libraries in Java Build Path.
Note : You can check the properties by Right-clicking the project and click Properties. If it is NOT added, you can click Add Library and select Maven Managed dependencies and finish