We Keep Coding

Getting shortestPaths in GraphFrames with Java

I am new to Spark and GraphFrames.
When I wanted to learn about shortestPaths method in GraphFrame, GraphFrames documentation gave me a sample code in Scala, but not in Java.
In their document, they provided following (Scala code):
import org.graphframes.{examples,GraphFrame}
val g: GraphFrame = examples.Graphs.friends // get example graph
val results = g.shortestPaths.landmarks(Seq("a", "d")).run()
results.select("id", "distances").show()
and in Java, I tried:
import org.graphframes.GraphFrames;
import scala.collection.Seq;
import scala.collection.JavaConverters;
GraphFrame g = new GraphFrame(...,...);
Seq landmarkSeq = JavaConverters.collectionAsScalaIterableConverter(Arrays.asList((Object)"a",(Object)"d")).asScala().toSeq();
g.shortestPaths().landmarks(landmarkSeq).run().show();
or
g.shortestPaths().landmarks(new ArrayList<Object>(List.of((Object)"a",(Object)"d"))).run().show();
Casting to java.lang.Object was necessary since the API demands Seq<Object> or ArrayList<Object> and I could not pass ArrayList<String> to compile it right.
After running the code, I saw the message:
Exception in thread "main" org.apache.spark.sql.AnalysisException: You're using untyped Scala UDF, which does not have the input type information. Spark may blindly pass null to the Scala closure with primitive-type argument, and the closure will see the default value of the Java type for the null argument, e.g. `udf((x: Int) => x, IntegerType)`, the result is 0 for null input. To get rid of this error, you could:
1. use typed Scala UDF APIs(without return type parameter), e.g. `udf((x: Int) => x)`
2. use Java UDF APIs, e.g. `udf(new UDF1[String, Integer] { override def call(s: String): Integer = s.length() }, IntegerType)`, if input types are all non primitive
3. set spark.sql.legacy.allowUntypedScalaUDF to true and use this API with caution;
To follow the 3., I have added the code:
System.setProperty("spark.sql.legacy.allowUntypedScalaUDF","true");
but situation did not change.
Since there are limited number of sample code or stackoverflow questions about GraphFrames in Java, I could not find any useful information while seeking around.
Could anyone experienced in this area help me solve this problem?

This seems a bug in GraphFrames 0.8.0.
See Issue #367 in github.com

Scala Collections - type casting Any to Seq[T] using Converters

I am new to Scala and I am trying to replace the deprecated JavaConversions library with JavaConverters. The original Code looks like this:
addresses = {
import scala.collection.JavaConversions._
config.getConfigList("amqp.addresses").map(address ⇒
Address(
host = address.foo()
))(collection.breakOut)
}
When I replace the JavaConversions with JavaConvertors in the code above, I get a compilation error:
Type mismatch: expected Seq[Address], actual: Any
I understand what the exception means, but I am not sure how I can convert the code above to make is return a Seq[Address] and not an Any. Also, there is a asJava method in Converters to convert the scala list to java list, but not sure how I can use it here. Thoughts?

You would need to state the final type and explicitly convert the Java collection into a Scala one (asScala):
addresses: Seq[Address] = {
import scala.collection.JavaConverters._
config.getConfigList("amqp.addresses").asScala.map(address ⇒
Address(
host = address.foo()
))(collection.breakOut)
}

Is there any replacement or alternative way for using scala.collection.TraversableOnce.mkString() method in java 7

I am able to use mkString method in scala successfully. But while trying to do it with java, there does not exist mkString method for java7. So is there any way by which i can do the same thing in java.
Below is my code for reference:
val records: util.List[Tuple2[Void, Array[AnyRef]]] = dataSource.collect
import scala.collection.JavaConversions._
for (record <- records) {
println(record.f1.mkString(","))
}

You can use Arrays.deepToString to get the string representation of your array elements.
List<Tuple2<Void, Object[]>> records= dataSource.collect();
Tuple2<Void, Object[]> record = records.iterator().next();
System.out.println(Arrays.deepToString(record.f1));

How to convert a Java Collection/List to a Scala seq?

I'm trying to instantiate a Kafka Scala case class from Java code, and it has the following signature:
case class OffsetFetchRequest(groupId: String,
requestInfo: Seq[TopicAndPartition],
versionId: Short = OffsetFetchRequest.CurrentVersion,
correlationId: Int = 0,
clientId: String = OffsetFetchRequest.DefaultClientId)
I'm able to send all the requested parameters, except for the Seq[TopicAndPartition].
On the Java side, I have the following code:
OffsetFetchRequest offsetFetchRequest = new OffsetFetchRequest(
"someGroup",
topicAndPartitions,
(short)1,
1,
"clientId");
As expected, a java.util.Listis not compatible with a Scala Seq. However, I've tried all types of conversion methods in JavaConversions and JavaConverters, and I can't find anything that fits this case.
How can I create a Scala seq from a normal java.util.List or even a java.util.Collection? Or am I approaching this incorrectly?

Use scala.collection.JavaConversions.asScalaBuffer which would convert Java list to Scala buffer, of which toList method can be used to convert to immutable seq.
Alternative, you could use CyclicIterator as well.

Unable to iterate Java List in Scala

I'm using the Java Twitter4J library in a Scala project.
I'm calling the method
twitter.getFriendsStatuses()
This method returns a list of twitter4j.User objects containing statuses.
I try to iterate over them and it goes in an infinite loop over the first element:
val users:List[User] = twitter.getFriendsStatuses(userId, paging.getSinceId())
while( users.iterator.hasNext() ) {
println(users.iterator.next().getStatus())
}
Any ideas?

I guess users.iterator produces the new iterator each time it's evaluated. Try this:
val it = users.iterator
while(it.hasNext() ) {
println(it.next().getStatus())
}

If you use Scala 2.8, you could use JavaConversion to convert Java collection to Scala collection automatically.
Ex.
import scala.collection.JavaConversions._
// Java Collection
val arrayList = new java.util.ArrayList[Int]
arrayList.add(2)
arrayList.add(3)
arrayList.add(4)
// It will implicitly covert to Scala collection,
// so you could use map/foreach...etc.
arrayList.map(_ * 2).foreach(println)

What's wrong with just
users.foreach(user => println(user.getStatus()))
or even
users.map(_.getStatus()).foreach(println _)
or, if you're worried about traversing the collection twice
users.view.map(_.getStatus()).foreach(println _)
IOW: Why do you want to manage the iteration yourself (and possibly make mistakes), when you can just let someone else do the work for you?

I prefer scalaj-collection to scala.collection.JavaConversions. This makes the conversions explicit:
import scalaj.collection.Implicits._
val arrayList = new java.util.ArrayList[Int]
arrayList.add(2)
arrayList.add(3)
arrayList.add(4)
arrayList.asScala.map(_ * 2).foreach(println)
Available here: https://github.com/scalaj/scalaj-collection

I suggest using
scala.collection.JavaConverters._
and simply add .asScala to every object you wish to iterate