I am trying out Spark Programming examples using Java 1.8 in Eclipse Luna and have the following code -
JavaPairRDD<String, Integer> counts = ones
.reduceByKey(new Function2<Integer, Integer, Integer>() {
#Override
public Integer call(Integer i1, Integer i2) {
return i1 + i2;
}
});
List<Tuple2<String, Integer>> output = counts.collect(); //Compilation Error
I am using M2Eclipse to build and create the jar and using spark-submit to execute the jar in my local. The jar is working and printing the correct output but Eclipse always shows the above mentioned line as a compilation error - The type Tuple2 is not generic; it cannot be parameterized with arguments <String, Integer>
Even the programming examples referred in the Spark webpage uses the same notation for Tuple2. https://spark.apache.org/docs/0.9.0/java-programming-guide.html
I am not able to understand why Eclipse is showing it as a compilation error since the return type of the collect call is a List<Tuple2<String,Integer>>
Any help is greatly appreciated.
As mentioned by #Holger in the comments, 2 scala-library jars were added to the build path. Removed the earlier version and compilation errors disappeared.
It helps on my problem on IntelliJ Idea, too. I called a function in which a parameter is a generic type with Tuple2 as a parameter. It always show an error but I can pass the compilation. It confused me for several days. After removing several dependent sharded-jars(in which it contains something related scala-libiary), the error disappeared.
Related
I am having the following problem:
I have an Enum that was originally declared with 5 elements.
public enum GraphFormat {
DOT,
GML,
PUML,
JSON,
NEO4J,
TEXT {
#Override
public String getFileExtension() {
return ".txt";
}
};
Now I need to add an additional element to it (NEO4J). When I run my code or try to debug it I am getting an exception because the value can't be found in the enum.
I am using IntelliJ as my IDE, and have cleaned the cache, force a rebuild, etc.. and nothing happens. When I look at the .class file created on my target folder, it also has the new element.
Any ideas on what could be causing this issue ?
I found my problem and want to share here what was causing it. My code was actually for a Maven plug-in which I was pointing to another project of mine to run it as a goal. However the pom.xml of my target test project was pointing to the original version of the plug-in instead of the one I am working on, and that version of course is outdated and does not include the new value. Thank you.
I have some fairly complex code that uses Javaslang. If I compile it into a jar, it runs fine. However, when I try to step into it in Eclipse for debugging, Eclipse flags it as a compilation error and dies when it reaches that line. The particularly weird part is that this worked a week ago, and the code has not changed in the interim.
Things I have tried:
clean project (including unchecking 'build automatically')
delete project from Eclipse, delete .project and .settings, re-import
from scratch
delete project from Eclipse, delete .project, .classpath, .settings, do mvn eclipse:eclipse, reimport
Maven builds this without errors [both within Eclipse and from the command line]. I can run the project this depends on and have it access this code from the JAR, so I know it works. I just cannot have Eclipse access the code from the project, either in 'run' or 'debug' mode.
Seq<Tuple2<StateProbabilityVector, ScenData>> resultStateProbs =
futures.
flatMap(Future::get).
toList();
// Update the target counts.
// THIS ENTIRE STATEMENT IS THE ERROR
Seq<Tuple2<ScenState, Probability>> result =
resultStateProbs.flatMap(tuple -> tuple.apply((spv, baTargetCount) ->
{
return spv.getStateProbList().
peek(sp -> logger.debug("Checking if {} > {}: {}",
sp.getProbability(),
intermediateMinProb,
sp.getProbability().greaterThan(intermediateMinProb))).
filter(sp -> sp.getProbability().greaterThan(intermediateMinProb)).
map(sp -> updateScenarioData(sp, baTargetCount, dupStateInfo));
}));
// signature for updateScenarioData
protected abstract Tuple2<ScenState, Probability> updateScenarioData(StateProbability stateProb,
ScenData scenData,
DSI dupStateInfo);
// truncated def of StateProbabilityVector
#Getter #ToString #Builder
public class StateProbabilityVector {
#NonNull
private final Seq<StateProbability> stateProbList;
}
So the types are all correct, but Eclipse claims:
> Type mismatch: cannot convert from Object to Iterable<? extends
> Object>
> Type mismatch: cannot convert from Seq<Object> to
> Seq<Tuple2<ScenState,Probability>>
As Nándor comments, this is probably down to a difference between the Eclipse compiler and javac, and the problem can probably be solved with a type witness in the right place. To find the right place, I would start by breaking up the functional method chain and extracting some local variables:
Seq<Tuple2<ScenState, Probability>> result =
resultStateProbs.flatMap(tuple -> {
Seq<Tuple2<ScenState, Probability>> filteredAndUpdated =
tuple.apply((spv, baTargetCount) -> {
Seq<StateProbability> stateProbList = spv.getStateProbList();
stateProbList.peek(sp -> {
logger.debug("Checking if {} > {}: {}", sp.getProbability(), intermediateMinProb, sp.getProbability().greaterThan(intermediateMinProb));
});
Seq<StateProbability> filtered = stateProbList.filter(sp ->
sp.getProbability().greaterThan(intermediateMinProb));
Seq<Tuple2<ScenState, Probability>> updated = filtered.map(sp ->
updateScenarioData(sp, baTargetCount, dupStateInfo));
return updated;
});
return filteredAndUpdated;
});
If you use Eclipse's extract variable refactoring, that by itself may tell you where it's inferring the wrong types, and explicitly declaring the correct types of the local variables might be enough to fix the problem all by itself.
If not, it should at least narrow the error down, and show you exactly where in the call chain Eclipse is having trouble. You can then probably fix it with type witnesses or, if all else fails, explicit casts, and then (with that type information added) perhaps inline the variables again, although this code is dense enough that I might leave them in.
Side notes:
peek() will only only debug the first StateProbability -- is that your intent?
consider adding a greaterThan() method to StateProbability so you don't have to repeatedly call getProbability().greaterThan(). (If the answer to #1 is "no", this method would also be a good place to put the debug statement.)
consider adding a method on SceneState that would return a prefiltered list, like Seq<StateProbability> SceneState.allGreaterThan(Probability).
I have the following method signature
JavaPairRDD<K,Object> countApproxDistinctByKey(double relativeSD)
from class
Class JavaPairRDD<K,V>
Javadoc here
In my code, I do the following
JavaPairRDD<String, String> mapToPair = ... some calculations ...
JavaPairRDD<String, Long> reachedDeviceRDD = mapToPair.countApproxDistinctByKey(0.01);
In Eclipse (Mars) this assignment throws the following error:
Type mismatch: cannot convert from JavaPairRDD<String,Object> to JavaPairRDD<String,Long>
which is correct, given the above signature!
But my problem is that I have been programming for 6+ months with IntellijIdea, and this error has never shown up since now.
It doesn't even come up with NetBeans and it doesn't show up compiling the project in Maven (i.e. javac).
It seems to me that the compiler is "casting automatically" the Object generic parameter to Long, which would be madness.
I don't know what I'm missing.
Additional info:
The classes are from the Apache Spark project, which is written in Scala.
** Edit **
You can find a code example here:
This code doesn't compile
List<String> pairs = new ArrayList<>();
System.out.println(pairs.stream().collect(Collectors.toMap(x -> x.split("=")[0], x -> x.split("=")[1])));
Compilation error is: The method split(String) is undefined for the type Object
error at System.out.println(pairs.stream().collect(Collectors.toMap(x -> x.split("=")[0], x -> x.split("=")[1])));
But this one compiles fine
List<String> pairs = new ArrayList<>();
Map<String,String> map = pairs.stream().collect(Collectors.toMap(x -> x.split("=")[0], x -> x.split("=")[1]));
System.out.println(map);
Can someone explain why?
MORE INFORMATION
It was intellij 12; jdk1.8.0_11; windows 64
I assume you are using an IDE (like Eclipse). Eclipse - for example - uses its own compiler and does not utilize the "javac" command (from JDK).
So, I can reproduce your problem, but only with Eclipse. Simply compiling this code on command line with "javac" works just fine.
The problem is very simple: The Eclipse compiler is not able to infer the type String for the collect method's arguments. So it simply infers Object (as this is the type, the compiler can safely assume). And an Object does not know the split method.
You can force the compiler to know about String by explicitely declaring the type inside the lambda:
List<String> pairs = new ArrayList<>();
System.out.println(pairs.stream().collect(Collectors.toMap((String x) -> x.split("=")[0], x -> x.split("=")[1])));
... or by explicitely declaring the correct types for the geneirc toMap method:
List<String> pairs = new ArrayList<>();
System.out.println(pairs.stream().collect(Collectors.<String, String, String> toMap(x -> x.split("=")[0], x -> x.split("=")[1])));
Versions of IntelliJ make different(Just red lines in source editor in IDE). The code should be compiled by JDK successfully.
IntelliJ 13 is OK for your code. IntelliJ 12 supports lambda expression poorly. I also met similar problems between two versions of IntelliJ when using lambda expression.
I'm trying to debug an java eclipse project with some problem!
I'm starting using CGLIB to make faster reflection calls using the index metod.
example
FastClass fastClass = FastClass.create(getClass());
int index = fastClass.getIndex("methodName", new Class[] { Object.class });
fastClass.invoke(index, this, new Object[] { obj } );
now when i try to put a breakpoint into a class that is called by fastreflection method this is the eclipse output.
I try to change compiler option on generate line number with no results.
I also upload an eclipse project (built with Juno version) that replicates the problem!!
http://www.filefactory.com/file/4zryz3gjgbyh/n/FastDebug.rar
Thanks!
I "resolved"(understand) the problem, but it is not a problem with Eclipse. When you launch
the program this line: FastClass.create(ReflectionTarget.class); ends up
creating an entirely new version of the compiled class removing all
non-essential stuff from the classfile to make it "fast" - that includes all
the line number / debug infos, which means the breakpoint cannot be set in it.
http://cglib.sourceforge.net/xref/net/sf/cglib/core/package-summary.html
There's no javadoc and you need to read the source but now i understand this is not a
problem but a feature of this method to make fast reflection!