We Keep Coding

Maven dependency “Cannot resolve symbol VectorAssembler'” in IntelliJ IDEA

IntelliJ IDEA cannot import Spark mllib, when I added dependency in maven. With other parts of Spark no problems. In project Structure -> Libraries spark mllib is present.
import org.apache.spark.ml.feature.VectorAssembler; -> Cannot resolve symbol 'VectorAssembler'
pom.xml:
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.12</artifactId>
<version>3.0.0</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>3.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
</dependencies>
Project structure
I tried refreshing maven and clearing folder with repositories of maven. Nothing helped.

You specified mllib dependency as runtime - this means that dependency is required for execution, but not for compilation, so it won't be put into classpath for compiling your code. See this blog post for description of of different scopes available in Maven.
Replace all spark dependencies (mllib, core, sql) with just single dependency (also remove hadoop dependencies):
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_${spark.scala.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
where variables are defined as
<properties>
<spark.version>3.0.1</spark.version>
<spark.scala.version>2.12</spark.scala.version>
</properties>

where to define test maven dependencies

I want to supply test dependencies like spring, mockito and junit in every project to make it as easy as possible to write tests.
Should I use dependency management for this dependencies or define them in the parent pom?

Yes , You don't need to define the dependency in every project. Just define the dependency in parent pom. Child project will automatically inherits its parent pom. Also If you want to use different version of Mockito or anything. Just override the parent dependency in child one.

All the common dependencies can be mentioned in the parent pom file. There mainly 4 types of dependencies that can be mentioned in a pom file.
Library Dependencies created by ourselves
Module Dependencies from our own modules
3rd Party library Dependencies
Dependencies for tests
Example
<dependencies>
<!-- Library Dependencies created by ourselves -->
<dependency>
<groupId>it.myapp</groupId>
<artifactId>MyAppBootstrap</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>it.myapp.libs</groupId>
<artifactId>b2b_connecttion</artifactId>
</dependency>
<!-- Module Dependencies from our own modules-->
<dependency>
<groupId>it.myapp.mymodules</groupId>
<artifactId>RevenueManager</artifactId>
<version>${myapp.module.version}</version>
<classifier>classes</classifier>
</dependency>
<!-- 3rd Party Dependency -->
<dependency>
<groupId>com.sun.xml.ws</groupId>
<artifactId>jaxws-rt</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc7</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-vfs2</artifactId>
<scope>provided</scope>
</dependency>
<!-- Dependencies for tests -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.jmockit</groupId>
<artifactId>jmockit</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

there are two ways to achieve it -
you declare the dependencies in the parent pom in the <dependencies /> node, and each child will benefit from the dependency.
Add the dependencies in the parent pom under the <dependencyManagement /> node and in each child that requires it, add the dependency in the node. You can choose not to set the version of the dependency.
So for example, if you declare this in the parent pom:
<dependencies>
<dependency>
<groupId>org.abc</groupId>
<artifactId>xyz</artifactId>
<version>your_version</version>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.abc</groupId>
<artifactId>xyz</artifactId>
<version>your_version</version>
<scope>your_scope</scope>
</dependency>
</dependencies>
</dependencyManagement>

Any way to easily include multiple Maven artifacts from the same group?

I have several Java projects that pom.xml to define dependencies. It all works nicely but there's a lot of repetition so I am trying to move duplicate dependencies into a new parent pom file.
Let's say my parent pom.xml has the following:
<dependencyManagement>
<dependencies>
<!-- Hazelcast dependencies -->
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast</artifactId>
<version>${hazelcast.version}</version>
</dependency>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast-client</artifactId>
<version>${hazelcast.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
My child pom.xml has:
<dependencies>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast</artifactId>
</dependency>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast-client</artifactId>
</dependency>
</dependencies>
This isn't too bad as I'm only adding two artifacts. Is there a way to add the two dependencies with only one entry? I tried the following based on an online post, but it doesn't work but it does give an idea of what I was hoping for:
<dependencies>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast</artifactId>
<artifactId>hazelcast-client</artifactId>
</dependency>
</dependencies>
I'm also depending on several different Jetty artifacts so I'm trying to see if there's a way to reduce content. I know I can write out each artifact dependency separately in the child pom.xml files but I thought I'd see if there's a way to simplify.
Thanks

How do I decide the dependencies for my maven project?

As part of learning, this is my first "spring nature" maven project,
In specific, I would like to understand the approach to know the list of dependencies that are required for any "spring nature" maven project, that I work in future.
For this project, here are the list of 21 dependencies that were just dumped into pom.xml without being told about, which dependency to use when, in this training video at 20:50?
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example.j2eeapp</groupId>
<artifactId>j2eeapplication</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>war</packaging>
<name>J2EE Applications Example</name>
<repositories>
<repository>
<id>prime-repo</id>
<name>PrimeFaces Maven Repository</name>
<url>http://repository.primefaces.org</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>4.1.8.Final</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-validator</artifactId>
<version>4.2.0.Final</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.webflow</groupId>
<artifactId>spring-webflow</artifactId>
<version>2.3.1.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.webflow</groupId>
<artifactId>spring-faces</artifactId>
<version>2.3.1.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-orm</artifactId>
<version>3.1.1.RELEASE</version>
</dependency>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc14</artifactId>
<version>10.2.0.1.0</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.16</version>
</dependency>
<dependency>
<groupId>com.sun.faces</groupId>
<artifactId>jsf-impl</artifactId>
<version>2.1.10</version>
</dependency>
<dependency>
<groupId>com.sun.facelets</groupId>
<artifactId>jsf-facelets</artifactId>
<version>1.1.14</version>
</dependency>
<dependency>
<groupId>com.sun.faces</groupId>
<artifactId>jsf-api</artifactId>
<version>2.1.10</version>
</dependency>
<dependency>
<groupId>commons-dbcp</groupId>
<artifactId>commons-dbcp</artifactId>
<version>20030825.184428</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.6.4</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.0.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-tx</artifactId>
<version>3.0.5.RELEASE</version>
</dependency>
<dependency>
<groupId>xml-apis</groupId>
<artifactId>xml-apis</artifactId>
<version>1.3.02</version>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-web</artifactId>
<version>3.1.3.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-config</artifactId>
<version>3.1.3.RELEASE</version>
</dependency>
<dependency>
<groupId>org.primefaces</groupId>
<artifactId>primefaces</artifactId>
<version>3.4</version>
</dependency>
<dependency>
<groupId>cglib</groupId>
<artifactId>cglib</artifactId>
<version>2.2.2</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>4.1.8.Final</version>
</dependency>
</dependencies>
</project>
So,
As of now, I do not have knowledge about bean/spring-webflow/hibernate etc...
What is the approach to know the dependencies required for my "spring nature" project?

Maven projects need dependencies instead of including jar files by build path.when you create a maven project, it requires the library files for the methods you use.you add the dependencies in pom.xml file and when you execute the maven build command the files are automatically downloaded from the internet and included in the project.
You can control the files to download.
you can just add dependencies of the library files you need in your source code.
you decide your dependencies by the methods you use in your source code.
for example.you are using sql database in your project you must need a jar file for the sql driver.
if you have a maven project you just have to add the dependency of the sql in pom.xml

Dependencies list depends on your project nature and usage of .jar relative files.
Simple if you want to use sql in your project then you only need to add sql dependencies in your pom.xml and in the same way if you use junit in your project then you will add junit dependencies in pom.xml in this way after tag
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.2</version>
</dependency>
no need of extra dependencies that are not being used in project.
Note that in maven dependencies repository you will see some extra dependencies. These are ones which maven used for itself or for other dependencies.

The "Spring nature" is related to how your IDE works with Spring plug-in. In eclipse, a "project nature" creates an association between the a project and a tool, plug-in, or feature set. By adding a nature to an eclipse project, you tell an eclipse plug-in that it is configured to use that project. By adding the "Spring Project Nature" to your project, you are enabling eclipse's spring plugin to work with your project.
Add dependencies on need basis. if you don't need hibernate or web-flow don't add it.Its like adding required toppings to your pizza. If you dont want mushroom, dont add it :)

eclipse maven error: Archive for required library in project cannot be read or is not a valid ZIP file

In an eclipse maven project using spring web mvc, I am getting the following error in the Markers tab:
Archive for required library: 'D:/mypath/.m2/repository/javax/transaction/jta/1.0.1B/jta-1.0.1B.jar' in project 'DocumentManager' cannot be read or is not a valid ZIP file
I have checked, and the jar file is in fact present in the url indicated. I even copied a backup of the jar to overwrite the jar file in that location, but that did not get rid of the error either. I forced maven to update, which did not solve the problem. I also restarted eclipse to no effect.
At one point, jta-1.0.1B.jar had been jta-1.0.1B.jar.LatestUpdate, so I shortened the name to jta-1.0.1B.jar
In pom.xml, the location of the error message is line 2, which reads as follows:
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
When I download jta-1.0.1B.jar manually from this url, the error in pom.xml goes away but is replaced with new errors in .java class files indicating that 5 methods of the Document class are not valid.
I read this and this, but I am not using struts, I am already including 3.3.2.ga of hibernate, and I do not have a repository tag in pom.xml (which this said did not even solve the problem anyway), so I do not think this is a duplicate.
Can anyone suggest a way to fix this problem? If I need a repository tag, where do I put it? And what else do I change to accommodate its addition?
For anyone who is curious, my pom.xml is included for reference as follows:
<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<packaging>war</packaging>
<version>0.0.1-SNAPSHOT</version>
<description></description>
<build>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.5</source>
<target>1.5</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-war-plugin</artifactId>
<version>2.0.1</version>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
<version>2.5</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-beans</artifactId>
<version>${org.springframework.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-jdbc</artifactId>
<version>${org.springframework.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-web</artifactId>
<version>${org.springframework.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>${org.springframework.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-orm</artifactId>
<version>${org.springframework.version}</version>
</dependency>
<dependency>
<groupId>taglibs</groupId>
<artifactId>standard</artifactId>
<version>1.1.2</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>jstl</artifactId>
<version>1.1.2</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>3.3.2.GA</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.10</version>
</dependency>
<dependency>
<groupId>commons-dbcp</groupId>
<artifactId>commons-dbcp</artifactId>
<version>20030825.184428</version>
</dependency>
<dependency>
<groupId>commons-pool</groupId>
<artifactId>commons-pool</artifactId>
<version>1.5.4</version>
</dependency>
<dependency>
<groupId>commons-fileupload</groupId>
<artifactId>commons-fileupload</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>1.3</version>
</dependency>
<!-- dependency to fix JSPServletException -->
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>jsp-api</artifactId>
<version>6.0.32</version>
<scope>provided</scope>
</dependency>
</dependencies>
<properties>
<org.springframework.version>3.2.4.RELEASE</org.springframework.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<groupId>DocumentManager</groupId>
<artifactId>DocumentManager</artifactId>
</project>

To fix issues like that, let Maven download the files again:
Delete the folder D:/mypath/.m2/repository/javax/transaction/jta
Run Maven with -U so it tries broken downloads again.
That should try to download the file again and clean up any "residue" in your local repository.
If Eclipse still complains that the JAR file is corrupt:
Refresh your project (F5 or from the context menu)
Clean the project
Try to open the JAR file. Maybe it's really corrupt.
If the JAR file is corrupt, delete the folder again and run Maven once more. Note the URLs which Maven used to download the archive and contact the administrator of that site.

You can manually go to the repository specified in the error console
that is
Archive for required library: '**D:/mypath/.m2/repository/javax/transaction/jta/1.0.1B/jta-1.0.1B.jar**' in project 'DocumentManager' cannot be read or is not a valid ZIP file
Delete the JAR mentioned
Go to eclipse, right click the project and select run as->maven-install.
It will download the right jar file.

You should exclude JTA in your hibernate dependency.
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate</artifactId>
<version>3.2.3.ga</version>
<exclusions>
<exclusion>
<artifactId>javax.transaction</artifactId>
<groupId>jta</groupId>
</exclusion>
</exclusions>
</dependency>
After that, add the jta dependency.
<dependency>
<groupId>javax.transaction</groupId>
<artifactId>jta</artifactId>
<version>1.1</version>
</dependency>

The problem with me was fixed by going to Build Path-> Configure Build Path-> Libraries-> Remove the jta lib which has small cross mark on it.
I was able to successfully build my project but was getting this error even after project Refresh/Project Clean.

The maven central repository doesn't contain a jar file for the given groupId/artifactId only a pom.xml.
You are using Spring 3.2 so I suggest you simply upgrade hibernate to the most recent version (4.2.6.Final) which should have a reference to jta 1.1. Or simply include the 1.1 version in your pom which does have a jar in maven central.

To fix this issue simply.
select all libraries in "Maven Dependencies" of related project.
just do "Refresh" of mouse right button...
The reason is cache info, so just refresh the information.

Tried with 1.5.5 Release jar and it worked for me
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.5.RELEASE</version>
https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-parent/1.5.5.RELEASE