We Keep Coding

Getting shortestPaths in GraphFrames with Java

I am new to Spark and GraphFrames.
When I wanted to learn about shortestPaths method in GraphFrame, GraphFrames documentation gave me a sample code in Scala, but not in Java.
In their document, they provided following (Scala code):
import org.graphframes.{examples,GraphFrame}
val g: GraphFrame = examples.Graphs.friends // get example graph
val results = g.shortestPaths.landmarks(Seq("a", "d")).run()
results.select("id", "distances").show()
and in Java, I tried:
import org.graphframes.GraphFrames;
import scala.collection.Seq;
import scala.collection.JavaConverters;
GraphFrame g = new GraphFrame(...,...);
Seq landmarkSeq = JavaConverters.collectionAsScalaIterableConverter(Arrays.asList((Object)"a",(Object)"d")).asScala().toSeq();
g.shortestPaths().landmarks(landmarkSeq).run().show();
or
g.shortestPaths().landmarks(new ArrayList<Object>(List.of((Object)"a",(Object)"d"))).run().show();
Casting to java.lang.Object was necessary since the API demands Seq<Object> or ArrayList<Object> and I could not pass ArrayList<String> to compile it right.
After running the code, I saw the message:
Exception in thread "main" org.apache.spark.sql.AnalysisException: You're using untyped Scala UDF, which does not have the input type information. Spark may blindly pass null to the Scala closure with primitive-type argument, and the closure will see the default value of the Java type for the null argument, e.g. `udf((x: Int) => x, IntegerType)`, the result is 0 for null input. To get rid of this error, you could:
1. use typed Scala UDF APIs(without return type parameter), e.g. `udf((x: Int) => x)`
2. use Java UDF APIs, e.g. `udf(new UDF1[String, Integer] { override def call(s: String): Integer = s.length() }, IntegerType)`, if input types are all non primitive
3. set spark.sql.legacy.allowUntypedScalaUDF to true and use this API with caution;
To follow the 3., I have added the code:
System.setProperty("spark.sql.legacy.allowUntypedScalaUDF","true");
but situation did not change.
Since there are limited number of sample code or stackoverflow questions about GraphFrames in Java, I could not find any useful information while seeking around.
Could anyone experienced in this area help me solve this problem?

This seems a bug in GraphFrames 0.8.0.
See Issue #367 in github.com

Pyspark Create Model with Coefficient & Intercept

I am wondering if it is possible to construct a model(linear regression / logistic regression only with coefficient & intercept) For scikit-learn, things worked smoothly -- I could just set those variables for the model and predict worked.
For pyspark, I am having more trouble. I couldn't set those variables in scala. Since model takes in java_model as parameters, I am trying to create a java_model with pyspark/py4j and use it to create a pyspark model.
Here is what I am trying to do as a test.
from pyspark import SparkContext, SQLContext
from pyspark.mllib.linalg import DenseVector
sc = SparkContext.getOrCreate()
sql_ctx = SQLContext(sc)
vect = DenseVector([1.0, 2.0])
test = sc._jvm.org.apache.spark.ml.regression.LinearRegressionModel(vect, 1.0)
but then I get this error
AttributeError: 'numpy.ndarray' object has no attribute '_get_object_id'
Seems like vect has self.array which is ndarray, and py4j cannot convert it to java DenseVector. Has anyone tried a similar attempt?

What does com.mathworks.mde.cmdwin.CmdWin means in MATLAB? Is there any documentation regarding this?

function interrupt
import java.awt.event.KeyEvent
import java.lang.reflection.*
base = com.mathworks.mde.cmdwin.CmdWin.getInstance();
hCmd = base.getComponent(0).getViewport().getView();
cmdwin = handle(hCmd,'CallbackProperties');
argSig = javaArray('java.lang.Class',1);
argSig(1) = java.lang.Class.forName('java.awt.event.KeyEvent');
msTime = (8.64e7 * (now - datenum('1970', 'yyyy')));
args = javaArray('java.lang.Object',1);
args(1) = KeyEvent(cmdwin,KeyEvent.KEY_PRESSED,msTime,...
KeyEvent.CTRL_DOWN_MASK,KeyEvent.VK_C,KeyEvent.CHAR_UNDEFINED);
method = cmdwin.getClass().getDeclaredMethod('processKeyEvent',argSig);
method.setAccessible(true);
method.invoke(cmdwin,args);
The above code was pasted from this answer. I just need to understand or find API/Documentation about this line:
com.mathworks.mde.cmdwin.CmdWin.getInstance();
I saw similar stuff all over the internet. What is it and where can I find any source?

This is an unsupported and undocumented API to access the command window. More examples of its use can be found at Undocumented Matlab. You're accessing directly the Java components that MATLAB is built with, so it's best not to rely on these things to be stable or long-lived.

ColdFusion/Java - "The build method was not found"

We are using some org.apache classes as part of implementing WS Security for a webservice.
variables.paths = arrayNew(1);
variables.paths[1] = getDirectoryFromPath(getCurrentTemplatePath()) & "lib\wss4j-1.5.8.jar";
variables.paths[2] = getDirectoryFromPath(getCurrentTemplatePath()) & "lib\xmlsec-1.4.2.jar";
variables.loader = createObject("component","lib.javaloader.JavaLoader").init(loadPaths=variables.paths,loadColdFusionClassPath=true);
variables.WSConstantsObj = loader.create("org.apache.ws.security.WSConstants");
variables.messageClass = loader.create("org.apache.ws.security.message.WSSecUsernameToken");
variables.secHeaderClass = loader.create("org.apache.ws.security.message.WSSecHeader");
The following code:
<cfset var msg = getMessage()>
produces:
The following code:
<cfset var secHeader = getSecHeader()>
produces:
The following code:
<cfset var env = soapEnv.getDocumentElement()>
produces:
env.getOwnerDocument()
produces a huge structure (too big to include here), which you can view here.
However, the following code:
<cfset e = msg.build(env.GetOwnerDocument(),secHeader)>
throws the error:
The build method was not found.
Either there are no methods with the specified method name and argument types or the build method is overloaded with argument types that ColdFusion cannot decipher reliably. ColdFusion found 0 methods that match the provided arguments. If this is a Java object and you verified that the method exists, use the javacast function to reduce ambiguity.
However the Build() method certainly exists, as per the yellow highlight in the first screenshot.
The error message talks about "...use the javacast function to reduce ambiguity". If this were the problem, how would I apply this solution?

It's not that "build()" doesn't exist but that your signature is incorrect. The 2 arguments for the build method are:
env.GetOwnerDocument(),secHeader
We know that that secHeader is of the class
org.apache.ws.security.message.WSSecHeader
Because your cfdump indicates as much.
That likely means that "env.GetOwnerDocument" is not returning an object of the class
org.w3c.dom.Document
It may be returning an error or a primitive or another class. Instantiate envGetOwnerDocumet() and dump that out and check the class. I think it is screwing up the method signature. That's my best guess anyway.

Java, cannot find symbol : method methodName(org.bla.blabla.myClass)

I'm using Lucene APIs, and I get the following error on this line of my code:
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.Fieldable;
...
Document _document = new Document();
_document.add(new Field("type", document.getType()));
Error:
CollectionIndexer.java:34: cannot find symbol
symbol : method add(org.apache.lucene.document.Field)
location: class CollectionIndexer.Document
_document.add(new Field("type", document.getType()));
This is the documentation about the method:
http://lucene.apache.org/java/3_0_3/api/all/org/apache/lucene/document/Document.html#add(org.apache.lucene.document.Fieldable)
thanks
Update: javac -cp commons-digester-2.1/commons-digester-2.1.jar:lucene-core-3.0.3.jar myApp.java

When I'm stumped over this type of error, it is usually due to the fact that I've two definitions of InterfaceName, and accidentally imported the wrong one in one or more places.
(Happens for instance when I accidentally choose java.awt.List instead of java.util.List when auto-importing missing classes.)
Make sure that ...
symbol : method methodName(org.bla.blabla.myClass)
\____________________/
... this part ...
... matches the expected package / class.

The problem comes from the fact that your document.getType() method returns a String and there
is no constructor in the Field class that matches your call.
See http://lucene.apache.org/java/3_0_3/api/all/org/apache/lucene/document/Field.html.
If I test your code in my environment Eclipse says:
The constructor Field(String, String) is undefined
Maybe you could do as the following:
Document _document = new Document();
_document.add(new Field("type", document.getType().getBytes(), Store.YES);
// Or document.add(new Field("type", document.getType().getBytes(), Store.NO);
UPDATE after source code submission --------------------
The problem comes from the fact that in your class you have an inner-class called Document. There is a name conflict between your Document class and the Lucene's one. When you instanciate your document with the line Document _document = new Document(); you're actually instanciating YOUR Document class. That's why the compiler cannot find the add method.
Multiple solution:
a. Instanciate the Document prefixing it with the Lucene package name
org.apache.lucene.document.Document _document = new org.apache.lucene.document.Document();
b. Rename your inner class so that you don't have any name conflict.

Updated based on updates to question:
Make sure your curly braces around this line up and that there is not something else causing an issue.
Reduce the code down just to as few lines as possible to eliminate any other items that could be throughing the compiler off.
Compile without the commons-digester-2.1 if you can, to eliminate possible conflicts.
Break the line up so the a Field object is created on a separate line than adding the field to the document so that you can confirm there is no problem with your constructor call.