I am following this tutorial. The code I have is this:
package com.ipponusa;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
public class SparkStringConsumer {
public static void main(String[] args) {
SparkConf conf = new SparkConf()
.setAppName("kafka-sandbox")
.setMaster("local[*]");
JavaSparkContext sc = new JavaSparkContext(conf);
JavaStreamingContext ssc = new JavaStreamingContext(sc, new Duration(2000));
// TODO: processing pipeline
ssc.start();
ssc.awaitTermination();
}
}
My Maven dependencies are:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka_2.10</artifactId>
<version>1.6.0</version>
</dependency>
The problem is that the code and imports all have errors. I don't know what dependencies I need, and I don't know how to find out what dependencies I need. I am new to Java.
So my question is, how am I supposed to know what dependencies I need?
Note: Neither IntelliJ or Netbeans "Find Maven Dependency" features found anything for me.
Edit: If I go here I can find the Classes I am looking for, but I can't find the dependencies there.
Edit: If I go here I can find a long list of dependencies, but I can't find the ones that have the classes I need.
Thanks in advance!
I think you should add the spark core too
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>2.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>2.1.0</version>
</dependency>
You can use the 'search by classname' feature of search.maven.org to find the library you need to import.
Also keep in mind that maven dependencies are transitive, i.e. a dependency of a dependency is included in your project too.
Not really Maven-specific, but you can look at the Spark source repo, search for the class you want, then look roughly at the folder that it is in.
For example, SparkContext exists under the core/ folder.
Note: spark-streaming-kafka depends on spark-streaming, which depends on spark-core, so if you included that, or the other core libraries like MLlib, SparkSQL, or GraphX, then core should be brought in as well.
Perhaps you needed to run mvn compile
Related
Trying to use AutoValue of package com.google.auto.value.AutoValue in my java class and get error "AutoValue cannot be resolved to a type"
I have added this in pom.xml
<!-- https://mvnrepository.com/artifact/com.google.auto.value/auto-value -->
<dependency>
<groupId>com.google.auto.value</groupId>
<artifactId>auto-value</artifactId>
<version>1.2</version>
<scope>provided</scope>
</dependency>
Java program :
package xxxx;
import ------
import com.google.auto.value.AutoValue;
#AutoValue
#DefaultCoder(SerializableCoder.class)
public abstract class ABC{
/**
* Manually create a test row.
*/
public static ABC create(List<Object> fields) {
return new AutoValue_ABC(fields); //error is here
}
public abstract List<Object> fields();
}
I tried the solutions like adding it by opening the project properties, browsing to Java Compiler -> Annotation Processing -> Factory Path, clicking on "Add External JARs" and then selecting 4 jar files - auto-service-1.0-rc1.jar ,
guava-16.0.1.jar ,
jsr-305-2.0.3.jar ,
auto-value-1.0-rc1.jar
Also tried adding m2e-apt plugin and Maven -> "Annotation processing" -> select "Automatically configure JDT APT..."
Still getting the errror "AutoValue cannot be resolved to a type"
Any Solutions?
Thanks in Advance.
Seems you are a bit outdated. Newest version of auto-value is 1.6.5.
You have to import not only auto-value, but also auto-value annotations. From official documentation of auto-value:
<dependency>
<groupId>com.google.auto.value</groupId>
<artifactId>auto-value-annotations</artifactId>
<version>1.6.2</version>
</dependency>
<dependency>
<groupId>com.google.auto.value</groupId>
<artifactId>auto-value</artifactId>
<version>1.6.2</version>
<scope>provided</scope>
</dependency>
That’s all you need for maven. For eclipse, you seem to have looked at Code Affine. Their article about auto-value is a bit confusing. You only need auto-value and auto-value-annotations. Guava, auto-service, and jsr are not necessary for use with auto-value.
At least, this worked for me.
I've set up JUnit in IntelliJ IDEA and have a bunch of tests with nothing in them. When I run them, they all pass as expected. However, when I type "assertEquals", it shows up in red. When I hover over it, it says "Cannot resolve method."
I've googled around and it looks like I need to do:
import static org.junit.Assert.*;
However, when I start typing import static org.junit., the next options are "*", "jupiter", or "platform"...
For reference, here's what a sample test looks like in my IDE:
#org.junit.jupiter.api.Test
void isButton() {
assertEquals()
}
Any idea how to fix this?
Thanks!
The full path to Assertions class is:
org.junit.jupiter.api.Assertions.assertEquals
Ensure you have added Jupiter API to your dependencies:
Gradle:
dependencies {
testCompile("org.junit.jupiter:junit-jupiter-api:5.9.0")
}
Maven:
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>5.9.0</version>
<scope>test</scope>
</dependency>
There is a nice guide for Intellij IDEA and JUnit 5.
Take a look at it: Using JUnit 5 in IntelliJ IDEA
Maven
Verify your dependency specified in your POM file. You should have following nested within your dependencies element.
<!-- https://mvnrepository.com/artifact/org.junit.jupiter/junit-jupiter -->
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>5.4.0-RC1</version>
<scope>test</scope>
</dependency>
If calling the assertions outside of your test classes, in your regular app classes, drop the <scope>test</scope> element.
Example class
Here is an example trivial test.
package work.basil.example;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* Unit test for simple App.
*/
public class AppTest
{
/**
* Rigorous Test :-)
*/
#Test
public void shouldAnswerWithTrue()
{
assertTrue( true );
}
}
New junit-jupiter artifact
Note that as of 5.4.0 of JUnit, we can specify the new and very convenient single Maven artifact of junit-jupiter which in turn will supply 8 libraries to your project.
The following is clearly bad practice but my question is out of pure curiosity: If I have a Maven project with two versions of the same library in the dependencies (e.g. jetty server in the pom.xml fraction below), then I import a class in my code (e.g. java code below), which version is the class picked up from?
<project>
....
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
<version>9.3.0.v20150612</version>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
<version>8.1.17.v20150415</version>
</dependency>
....
</project>
And the java code:
package test.hello;
import org.eclipse.jetty.server.Server;
public class Hello {
Server server;
public Hello() {
server = new Server();
}
}
Is the server object created from the old version class or the new version class? Maven does not complain about any ambiguity. It just loads a class; I don't know which.
Maven has nothing to do with loading jars or classes. Maven only gathers all the dependancy. Its upto your classloader to decide what to load when.
If you have two classes with the same binary name, and you want to know which one of them you are loading, you can only inspect the way that classloader tries to resolve a class name.
Summary
When trying XMLConfiguration configuration = new XMLConfiguration("config/config.xml"); with only commons-configuration 1.10 I need to add more depencies (namely commons-collections not newer than 3.2.1) to my maven setup. Why is that so and why doesn't maven simply resolve all needed dependencies?
Details
I am trying to get commons-configuration to work. First I wanted to use the latest version, 2.0-alpha2, which didn't work well at all since I was unable to configure Maven to download the correct ressources - but that is another story.
After I found out that version 1.10 is in fact "one point ten" (not "one point one zero") and thus the latest version of commons-configuration 1 (and covered by the tutorials), I decided to give it a try instead.
For my maven dependencies (integrated in eclipse) I used:
<dependency>
<groupId>commons-configuration</groupId>
<artifactId>commons-configuration</artifactId>
<version>1.10</version>
</dependency>
However, when trying out this example:
package main;
import java.util.Iterator;
import org.apache.commons.configuration.ConfigurationException;
import org.apache.commons.configuration.XMLConfiguration;
public class ConfigurationTest {
public static void main(String... args) {
try {
XMLConfiguration configuration =
new XMLConfiguration("config/config.xml");
Iterator<String> iterator = configuration.getKeys();
while (iterator.hasNext()) {
System.out.println(iterator.next());
}
} catch (ConfigurationException e) {
e.printStackTrace();
}
}
}
with the following config.xml:
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<configuration>
<property>value</property>
<nestedproperty>
<arrayvalue>0,1,2,3,4</arrayvalue>
<property>anothervalue</property>
</nestedproperty>
</configuration>
I got the error:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/collections/CollectionUtils
at org.apache.commons.configuration.XMLConfiguration.constructHierarchy(XMLConfiguration.java:640)
at org.apache.commons.configuration.XMLConfiguration.initProperties(XMLConfiguration.java:596)
at org.apache.commons.configuration.XMLConfiguration.load(XMLConfiguration.java:1009)
at org.apache.commons.configuration.XMLConfiguration.load(XMLConfiguration.java:972)
at org.apache.commons.configuration.XMLConfiguration$XMLFileConfigurationDelegate.load(XMLConfiguration.java:1647)
at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:324)
at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:261)
at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:238)
at org.apache.commons.configuration.AbstractHierarchicalFileConfiguration.load(AbstractHierarchicalFileConfiguration.java:184)
at org.apache.commons.configuration.AbstractHierarchicalFileConfiguration.<init>(AbstractHierarchicalFileConfiguration.java:95)
at org.apache.commons.configuration.XMLConfiguration.<init>(XMLConfiguration.java:261)
at main.ConfigurationTest.main(ConfigurationTest.java:12)
I first hoped they (not me, of course) just screwed up some maven dependencies and since I wouldn't bother which version to use anyway anymore (I didn't get 2.0 to work, remember?) I decided to go down to version 1.9 by replacing the maven dependency with:
<dependency>
<groupId>commons-configuration</groupId>
<artifactId>commons-configuration</artifactId>
<version>1.9</version>
</dependency>
That solved the problem pretty well, the test case is running:
property
nestedproperty.arrayvalue
nestedproperty.property
But when I tried to implement a similar example to the one referenced in Very simple Apache-commons configuration example throws NoClassDefFoundError and its follow-up question I got the exact same error which is referenced there - but the solution, importing org.apache.commons.beanutils.PropertyUtils is not working as I am missing the beanutils. So basically by downgrading I just switched from the error of missing the collections to missing beanutils.
There is a dependency overview where you can see which dependencies are used when you do what. I was a bit suprised to learn that version 1.10 now used other dependencies (namely the CollectionUtils) than 1.9 did in the constructor call. Since there were dependency problems in 1.10 as well as in 1.9 I just sticked to the newer version.
I found the CollectionUtils located in the following artifact (as I was pointed there by its maven repository):
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-collections4</artifactId>
<version>4.0</version>
</dependency>
Sadly that one (not obvious to me at first) doesn't define the class CollectionUtils in the package collections, but in the package collections4. It was hinted at this problem on the dependency overview, but they only mentioned possible problems with earlier versions... I appeared to be at a point of not thinking much about it anymore but simply changed the dependency to:
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.2.1</version>
</dependency>
I got everything to work (more or less, but the Exceptions I get now are not anymore depending on missing class definitions) after using these dependencies:
<dependencies>
<dependency>
<groupId>commons-configuration</groupId>
<artifactId>commons-configuration</artifactId>
<version>1.10</version>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.2.1</version>
</dependency>
<dependency>
<groupId>commons-beanutils</groupId>
<artifactId>commons-beanutils</artifactId>
<version>1.9.2</version>
</dependency>
</dependencies>
Why do I have to add the dependencies myself? I thought the whole point in using maven is to avoid having to do such things and in terms of javadocs and source files it does a pretty good job.
By now I am convinced that the dependencies are not included in the hierarchy by design (is that so?), probably to avoid overhead. However is there a way to either simply get all dependencies at once or even better to get all dependencies I need? And why is it designed this way?
If we analyse commons-configuration's POM we see that the commons-collections dependency is optional:
<dependencies>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.2.1</version>
<optional>true</optional>
</dependency>
...
Furthermore, from the Maven docs:
If a user wants to use functionality related to an optional
dependency, they will have to redeclare that optional dependency in
their own project.
This issue is explained on the Runtime dependencies page of the Commons Configuration website.
Quoting from that page:
A lot of dependencies are declared in the Maven POM. These are all needed during compile time. On runtime however you only need to add the dependencies to your classpath that are required by the parts of the Commons Configuration package you are using. The following table helps you to determine which dependencies you have to include based on the components you intend to use.
The other answers explain why this works from a Maven perspective. This answer is intended to provide a defence, of sorts, to the Commons Configuration folks. They did at least warn you!
In cases where the dependencies are on other Apache Commons components, they've taken the time to test with a variety of versions and have posted information on compatibility at the bottom of that page.
Maven tries to resolve all necessary dependencies for a library you're using in your pom. Well sometimes you have some dependencies which are only necessary for some specific features and you don't want to force the user of your dependency to download it if he doesn't use it. Then you're declaring your dependency as optional. This happened with commons-collections within commons-configuration. See commons-configuration-pom here
I get the following error:
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.of([Ljava/lang/Object;)Lcom/google/common/collect/ImmutableSet;
at com.google.gdata.wireformats.AltFormat$Builder.setAcceptableTypes(AltFormat.java:399)
at com.google.gdata.wireformats.AltFormat$Builder.setAcceptableXmlTypes(AltFormat.java:387)
at com.google.gdata.wireformats.AltFormat.<clinit>(AltFormat.java:49)
at com.google.gdata.client.Service.<clinit>(Service.java:558)
at testproject.TestProject.run(TestProject.java:22)
at testproject.TestProject.main(TestProject.java:31)
Java Result: 1
BUILD SUCCESSFUL (total time: 0 seconds)
This comes from the following code:
package testproject;
import com.google.gdata.client.youtube.YouTubeService;
import com.google.gdata.util.*;
import java.util.logging.*;
public class TestProject {
public static void main(String[] args) {
try {
YouTubeService service = new YouTubeService("Test", "developerKey");
service.setUserCredentials("root#gmail.com", "pa$$word");
} catch (AuthenticationException ex) {
Logger.getLogger(TestProject.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
At first, I included every library in http://code.google.com/p/gdata-java-client/downloads/list and also imported much more than I needed to.
I've since removed the libraries I deemed unnecessary (thanks thinksteep). So the libraries I'm currently including are the following libraries:
mail.jar
activation.jar
ant.jar
gdata-core-1.0.jar
gdata-media-1.0.jar
guava-11.0.1.jar
gdata-youtube-2.0.jar
gdata-youtube-met-2.0.jar
(There are probably a few libraries there which are not necessary... But I'm at my whit's end...)
I'm just trying to test getting a YouTube service so I can get things going on this project, but no dice. Oh, and I've also included this library: http://code.google.com/p/guava-libraries because before I was getting a NoClassDefFound error and including that library seemed to solve it. Thank you in advance for the help!
Oh, and I also followed every step exactly (or at least I think so) in the gdata getting started guide. My test build was successful by the end... Thanks again!
Adding more than required may cause issue too. java.lang.NoSuchMethodError error typically happens in case where runtime couldn't find required method with exact signature. Possible causes are:
1) There might be mulitple jars with same code, which may cause wrong class get loaded.
2) Incompatable version of jar, the jar you have in classpath might be older version/newer version.
Make sure none of those cases happening.
Issue with latest version of gdata still referencing older guava methods
Check Out
http://code.google.com/p/gdata-java-client/issues/detail?can=2&start=0&num=100&q=&colspec=ID%20Type%20Status%20Priority%20Milestone%20Owner%20Summary&groupby=&sort=&id=344
Solution
I switched to guava-r07.jar located at
http://code.google.com/p/guava-libraries/downloads/detail?name=guava-r07.zip&can=4&q=
This got me past
ContactsService service = new ContactsService("");
Jar's in use:
Default Eclipse plugin jar's
gdata-base-1.0.jar
gdata-client-1.0.jar
gdata-contacts-3.0.jar
gdata-core-1.0.jar
gdata-media-1.0.jar
guava-r07.jar
Apache (servlet-api.jar)
JavaMail (mail.jar)
JavaBeans Activation Framework (activation.jar)
I dont know if its still relevant but i had the same exception
there is a problem with guava 11.02.jar (currently latest version)
when using guava-10.0.1 (can be found here) everything went well.
The Required library jars are as follows.
gdata-client-1.0.jar
gdata-core-1.0.jar
gdata-media-1.0.jar
gdata-youtube-2.0.jar
guava-11.0.2.jar
java-mail-1.4.4.jar
I am using the above mentioned library . Please make use of it ; because the ultimate aim is to get the YouTubeService Object. Check below for the code snippet.
package com.baba.test;
/*
* Author : Somanath Nanda
*/
import java.net.MalformedURLException;
import java.net.URL;
import com.google.gdata.client.youtube.YouTubeQuery;
import com.google.gdata.client.youtube.YouTubeService;
public class Test {
private static final String CLIENT_ID = "XXXXXXXX.XXXXX.XXX.XXX";
private static final String DEVELOPER_KEY = "*********************************88";
public static void main(String[] args) throws MalformedURLException {
YouTubeService service = new YouTubeService(CLIENT_ID,DEVELOPER_KEY);
System.out.println("Service : "+service);
}
If you're using a build tool, such as Maven, then you could simply do something similar to the following example from a portion of the dependencies section in my pom.xml:
<!-- The mail dependency is required BEFORE the javaee-api dependency.
The gdata dependency (YouTube API) requires the mail dependency. -->
<dependency>
<groupId>javax.mail</groupId>
<artifactId>mail</artifactId>
<version>1.4</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-api</artifactId>
<version>6.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.google.gdata</groupId>
<artifactId>core</artifactId>
<version>1.47.1</version>
</dependency>
I have added googlecollection-exp.jar into my build path then the previous execption was gone.
Pay attention to this jar gdata-core-1.0.jar I have the same problem, and I realized I have problem with this jar gdata-core-1.0.jar, and I found from website the same jar gdata-core-1.0.jar, but the content is different. After I replaced the new gdata-core-1.0.jar, problem solved.
So it's tricky that the jar with the same name but their contents are not the same. you thought you have the jar, actually it's not the right one
It could be that some of your jars would be having dependency on google/guava jars and if they're not in build path or if multiple of them are there it might raise inconsistency hence the error. A quick solution could be add latest version of guava to your pom
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>24.0-jre</version>
</dependency>
Now check in dependency hierarchy if any of your Jar apart from guava is referring to any other older jar of guava/google-collections. If so then exclude it, something like this
<exclusions>
<exclusion>
<groupId>com.google.collections</groupId>
<artifactId>google-collections</artifactId>
</exclusion>
</exclusions>