building dependencies for Apache Spark on Eclipse - java

I would like to use eclipse for an Apache Spark project, but building dependencies appears not to work. I keep getting
"Plugin execution not covered by lifecycle configuration: org.apache.maven.plugins:maven-plugin-plugin:3.2:descriptor (execution: default-descriptor, phase: generate-resources)"
Following the directions on the Apache spark site, https://spark.apache.org/docs/latest/programming-guide.html
I am running CDH5 cluster. using maven. Here is my generated pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven- 4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.1.0</version>
<packaging>jar</packaging>
<name>spark-core_2.10</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
</dependency>
<repositories>
<repository>
<id>Cloudera repository</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.1.0</version>
</dependency>
</dependencies>
I am a new Maven/POM user. How do i solve this issue and stop getting errors in my java code?

The error message seems to be related to m2e plugin. Take a look https://www.eclipse.org/m2e/documentation/m2e-execution-not-covered.html.
However the pom you expose is strange, the groupId and artifactId should be related to the path and name of your project, and not to the dependency that you try to use (in this case, spark-core_2.10).

Related

Setting maven to have a framework and other projects that use it

I created a Maven project that i should use as a framework. This framework has some dependencies:
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>SeleniumJavaFramework</groupId>
<artifactId>SeleniumJavaFramework</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>7</maven.compiler.source>
<maven.compiler.target>7</maven.compiler.target>
</properties>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java -->
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>4.0.0-alpha-6</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.testng/testng -->
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>7.1.0</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/junit/junit -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version>
<scope>compile</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/com.aventstack/extentreports -->
<dependency>
<groupId>com.aventstack</groupId>
<artifactId>extentreports</artifactId>
<version>4.1.6</version>
<scope>compile</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/log4j/log4j -->
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.poi/poi-ooxml -->
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml</artifactId>
<version>4.1.2</version>
</dependency>
</dependencies>
</project>
I'd like to create other Maven projects, separated from the framework project, that will have their own pom.xml with the framework dependency.
If possible, they should inherit dependencies from the framework project.
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>RicercaGoogle</groupId>
<artifactId>RicercaGoogle</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>TestOne</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>7</maven.compiler.source>
<maven.compiler.target>7</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>SeleniumJavaFramework</groupId>
<artifactId>SeleniumJavaFramework</artifactId>
<version>0.0.1-SNAPSHOT</version>
<scope>system</scope>
<systemPath>patofmyjar.jar</systemPath>
</dependency>
</dependencies>
</project>
Unfortunately I don't know where to start with the maven settings. I'm not even sure if the pom files are correct. I only know that i can't simply put the jar dependency into the test project and run it. Can you help me out?
Thank you
If you are working at single project, you can create a parent pom that include your TestOne app and Framework as two modules, so you'll have 3 poms (search for maven multimodule project). If your framework is a library for different projects, you need a repository (ie nexus), where your Framework can be deployed to. Then you can use it as a dependency in other projects (dependencies of Framework will be included automatically)
I suggest to follow the maven official guide for learning how inheritance works in maven and adapt this to your logic:
http://maven.apache.org/guides/introduction/introduction-to-the-pom.html#Project_Inheritance
Problem was that my classes where in the wrong folders.
They were in src/test/java while they had to be in src/main/java.
Also in the test project pom i removed scope and systemPath as suggested by Andrew Fomin
<dependency>
<groupId>SeleniumJavaFramework</groupId>
<artifactId>SeleniumJavaFramework</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
I had an error
Missing artifact SeleniumJavaFramework:SeleniumJavaFramework:jar:0.1.1-SNAPSHOT
because, as you can see, version was wrong.
Thank you all

how to add package "com.google.api.client.util.store" in netbeans? [duplicate]

This question already has answers here:
How to add a JAR in NetBeans
(4 answers)
Closed 6 years ago.
I'm following this tutorial to sync my java app with Google calendar. But the following line is giving error:
import com.google.api.client.util.store.FileDataStoreFactory;
So I searched how to add this dependence to my project. But I can only found this documentation page. But I Don't understand how to add this.
Other dependencies I've added in pom.xml file.
And I'm not using that Gradle for my project which is described in the tutorial which I mentioned.
My pom.xml file looks like:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany</groupId>
<artifactId>mavenproject2</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-calendar</artifactId>
<version>v3-rev226-1.22.0</version>
</dependency>
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client-java6</artifactId>
<version>1.12.0-beta</version>
</dependency>
<dependency>
<groupId>com.google.oauth-client</groupId>
<artifactId>google-oauth-client-jetty</artifactId>
<version>1.12.0-beta</version>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client-jackson2</artifactId>
<version>1.12.0-beta</version>
</dependency>
</dependencies>
<repositories>
<repository>
<id>google-api-services</id>
<url>https://oss.sonatype.org/content/repositories/releases/</url>
</repository>
<repository>
<id>google-api-services-beta</id>
<url>http://google-api-client-libraries.appspot.com/mavenrepo</url>
</repository>
</repositories>
</project>
You are Using an old version of the google-api-client. The documentation that you found is pointing the version 1.20 and your dependecies are 1.12.0-Beta. Try to use and check what dependecies you need and don't use outdated dependecies or in beta-version. Check always the maven repository
dependencies {
compile 'com.google.api-client:google-api-client:1.22.0'
compile 'com.google.oauth-client:google-oauth-client-jetty:1.22.0'
compile 'com.google.apis:google-api-services-calendar:v3-rev226-1.22.0'
}
Search this dependecies in the maven repository I think your dependecies are really different and outdated for the tutorial that are you trying to follow.
Gradle to Maven.
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client</artifactId>
<version>1.22.0</version>
</dependency>
<dependency>
<groupId>com.google.oauth-client</groupId>
<artifactId>google-oauth-client-jetty</artifactId>
<version>1.22.0</version>
</dependency>
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-calendar</artifactId>
<version>v3-rev225-1.22.0</version>
</dependency>
And use this Repository instead
<repository>
<id>central</id>
<url>http://repo1.maven.org/maven2/</url>
</repository>

Maven dependencies for Gson not being downloaded

I'm not that familiar with using Maven, so it is likely a user error in this case. My understanding of the elements in the POM file is that any "dependency" that is listed here will be retrieved from the Central Repository based upon the scope of the dependency. In my case, I'm attempting to use the Gson library from Google. It is located on the Central Repository and so it should be reachable by the Maven tool. I've executed "mvn -X compile" to determine if I can see the dependencies in the import. But I don't see them being downloaded during the compile. Any ideas as to what could be wrong with my configuration?
Below is my POM for my project.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.acumen.app</groupId>
<artifactId>CatalogConverter</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<name>CatalogConverter</name>
<url>http://maven.apache.org</url>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.3.1</version>
</dependency>
</dependencies>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
</project>

netbeans: dependencies for c3p0 and hibernate

Are these the correct dependencies for a hibernate and c3p0 JavaSE application? My pom.xml currently:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany</groupId>
<artifactId>crud</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<repositories>
<repository>
<id>unknown-jars-temp-repo</id>
<name>A temporary repository created by NetBeans for libraries and jars it could not identify. Please replace the dependencies in this repository with correct ones and delete this repository.</name>
<url>file:${project.basedir}/lib</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>4.1.8.Final</version>
</dependency>
<dependency>
<groupId>unknown.binary</groupId>
<artifactId>hibernate-jpa-2.0-api-1.0.1.Final</artifactId>
<version>SNAPSHOT</version>
</dependency>
<dependency>
<groupId>unknown.binary</groupId>
<artifactId>mysql-connector-java-5.1.23-bin</artifactId>
<version>SNAPSHOT</version>
</dependency>
<dependency>
<groupId>c3p0</groupId>
<artifactId>c3p0</artifactId>
<version>0.9.1.2</version>
</dependency>
</dependencies>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
</properties>
</project>
This is from netbeans, and I'm not quite clear on what to click on, even after referencing the Netbeans guide on maven.
Does this meet the hibernate and c3p0 requirements?
these dependencies were added by the IDE via the Library mechanism (Tools/Library Manager) and proper Maven GAV (GroupID-ArtifactID-version) could not be recognized (there are several strategies but apparently all failed). The generated pom snippet is the fallback that tries to make it all work for you locally. Ideally you would manually replace that with the proper GAV from a public repository.

java.lang.NoClassDefFoundError: com/google/gson/Gson

I noticed a strange problem. I am able to run my test cases using Junit but when I am running using maven One of the test case is failing. It is complaining that Gson class def is not found.
I am able to see the Gson jar in Maven dependencies.
So I doubted that classpath is not including the Gson. So I ran maven with -X and noticed some clues.
[DEBUG] Could not find metadata com.example.libraries:Symbology:1.0.0-SNAPSHOT/maven-metadata.xml in local (C:\Users\ra\.m2\repository)
[DEBUG] Skipped remote update check for com.example.libraries:Symbology:1.0.0-SNAPSHOT/maven-metadata.xml, locally cached metadata up-to-date.
[DEBUG] Could not find metadata com.example.libraries:Symbology:1.0.0-SNAPSHOT/maven-metadata.xml in local (C:\Users\ra\.m2\repository)
[DEBUG] Skipped remote update check for com.example.libraries:Symbology:1.0.0-SNAPSHOT/maven-metadata.xml, locally cached metadata up-to-date.
[WARNING] The POM for com.example.libraries:Symbology:jar:1.0.0-SNAPSHOT is invalid, transitive dependencies (if any) will not be available: 2 problems were encountered while building the effective model for com.example.libraries:Symbology:1.0.0-SNAPSHOT
[ERROR] 'dependencies.dependency.artifactId' for ::jar is missing. #
[ERROR] 'dependencies.dependency.groupId' for ::jar is missing. #
I have a project which depends on Symbology project and this in turn uses Gson.
But now from this log I am seeing transitive dependencies are not getting included. So Gson class is not being found.
Here is the Symbology pom:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example.libraries</groupId>
<artifactId>Symbology</artifactId>
<version>1.0.0-SNAPSHOT</version>
<packaging>jar</packaging>
<name>Symbology</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.1</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.16</version>
</dependency>
</dependencies>
Here is the Pom of my project which is calling symbology :
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example.libraries</groupId>
<artifactId>FGF</artifactId>
<version>1.0.0-SNAPSHOT</version>
<packaging>jar</packaging>
<distributionManagement>
<snapshotRepository>
<id>example.com</id>
<name>example.com-snapshots</name>
<url>http://example/artifactory/libs-snapshots-local</url>
</snapshotRepository>
</distributionManagement>
<name>FGF</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.16</version>
</dependency>
<dependency>
<groupId>com.example.libraries</groupId>
<artifactId>Category</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.example.libraries</groupId>
<artifactId>Time</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.example.libraries</groupId>
<artifactId>Display</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
**<dependency>
<groupId>com.example.libraries</groupId>
<artifactId>Symbology</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>**
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>3.2.0.BUILD-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>cglib</groupId>
<artifactId>cglib</artifactId>
<version>2.2.2</version>
</dependency>
<dependency>
<groupId>org.apache.directory.studio</groupId>
<artifactId>org.apache.commons.lang</artifactId>
<version>2.6</version>
</dependency>
</dependencies>
Figured Out the Problem Finally after spending lot of time.
The problem is when I am pushing the symbology artifact to repository it is not pushing the pom into repository. So when I am using the symbology artifact as dependency my project is not able to know what are the transitive dependencies.
So I pushed my pom into repository along with jar and now it is able to get all the dependencies.
HURRAYYYYYYYYYYYYYYYYY!!!!!!!!!!!!!!!!!!

Categories