I have a Java class "Listings". I use this in my Java MapReduce job as below:
public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
Listings le = new Listings(value.toString());
...
}
I want to run the same job on Spark. So, I am writing this in Scala now. I imported the Java class:
import src.main.java.lists.Listings
I want to create a Listings object in Scala. I am doing this:
val file_le = sc.textFile("file// Path to file")
Listings lists = new Listings(file_le)
I get an error:
value lists is not a member of object src.main.java.lists.Listings
What is the right way to do this?
Based on what you've said, I think you may be forgetting the differences between Scala syntax and Java syntax.
Try this:
val lists: Listings = new Listings(SomeString)
Please note that specifying the type in Scala is completely optional. Also, use a var if you're going to be changing the value of lists.
The way you have it, Scala is trying to interpret it by its ability to call methods/access values of an object without the '.', so you're actually telling Scala this:
Listings.lists = new Listings(SomeString)
Related
I'm writing a program in Java for Spark 1.6.0 (so, please, don't supply Scala or Python code in your answers), and this is the code I'd like to implement:
double i = 0d;
JavaRDD<Vector> ideas = objects.map(
new Function<BSONObject, Vector>()
{
#Override public Vector call(final BSONObject t) throws Exception
{
double[] xy = new double[2];
xy[0] = i++;
xy[1] = ((Long)((Date)t.get("date")).toInstant().toEpochMilli()).doubleValue();
return Vectors.dense(xy);
}
}
);
but NetBeans shows an error: "Local variables referenced from an inner class must be final or effectively final".
I also tried to use Spark Accumulators, but if I call the value() method from the call() method I'm defining, a SparkException is raised during the job, telling me that "Task is not serializable", then the job fails.
So, how can I achieve my goal?
I apologize in advance if my English is not perfect (it's not my native language), and if my question could appear noob-ish, but I can't find any solution online.
Even if it compiled it wouldn't work as you expect. Each executor get its own copy of the variables referenced inside closure and any modifications are strictly local and are not propagated back to original source. Spark supports writable accumulators which can be used as follows:
Accumulator<Double> accum = sc.accumulator(0d);
objects.map(
...
accum.add(1d)
...
)
but these provide very weak guarantees (called at-least-once) when used inside transformations and, as you've already realized, are write-only from the worker perspective.
Regarding your code it looks like all you need is zipWithIndex:
objects.zipWithIndex().map(...)
I am working on a Java project that has some Clojure involved. I know how to run compile and run clojure code:
public static void main(String[] args) throws Exception {
RT.init();
runCode();
}
public static Object runCode() {
String str = "(ns my-ns)" +
"(defn add [a b] (+ a b))" +
"(println (add 1 2))";
Compiler.load(new StringReader(str));
/* I know how to invoke it: */
Var foo = RT.var("my-ns", "add");
return foo.invoke(1,2);
}
What would be very useful at the point is to have a way to iterate over forms in Java, and in some sense "analyze" the compiler output. Basic things I want to know is:
What is the text source of a form?
What function is being called in a form.
What arguments are being passed to the function (forms are ok)
Be able to do this on top level forms, or drill in as needed.
Is there a way to do this using the clojure compiler, or runtime (or other Java classes in Clojure?) I see such compiler methods as analyze, for example:
Expr target = analyze(C.EXPRESSION, RT.second(form));
Though its not clear to me yet how form was constructed, and there are no Javadoc :-). Do I need to go The Compiler Source and figure out how it works?
I have a code generator, which takes a syntax tree and converts it into a source file (text).
Basically, it traverses through all nodes of the tree, maps the node to text and appends the resulting texts to a StringBuilder.
Now I want the node to text mappers to be implemented using Xtend like this:
public class NodeXMapper
{
private XtendRunner xtendRunner = ...;
public String map(final NodeX aNode)
{
return xtendRunner.runScript("def String map(NodeX aNode) {
''' «aNode.fieldX» - «aNode.fieldY» '''
}", aNode);
}
}
xtendRunner.runScript(String aScript, final Object... aParams) is a method, which passes the parameters aParams to Xtend script aScript and returns the result.
How can I implement that method?
Update 1: Here I found this piece of code, which seems to run Xtend code in Java:
// setup
XtendFacade f = XtendFacade.create("my::path::MyExtensionFile");
// use
f.call("sayHello",new Object[]{"World"});
But I can't find XtendFacade class in the Type hiearchy view of Eclipse.
The interpreter you found was for the old Xtend1 language, which is not what you are looking for.
The new Xtend you are referring to is compiled, so there is no interpreter.
However, you could build an interpreted expression language using Xbase. See the documentation and Github for an example on how to do that. Then you could run the interpreter of your expression language from Java.
Problem Description
A somewhat contrived example to illustrate my question. Imagine we have some library of javascript functions that is already maintained and updated daily by an army of frontend devs. To be specific, imagine one such function looks like this:
function employeesForStore(store) {
var dictionary = {
"downtown": ["Joe", "Mary", "Steve"],
"uptown": ["Jules", "Vincent", "Matt"],
// and so on for hundreds of locations
};
return dictionary[store];
}
NOTE: Please ignore the details of this function's implementation. The actual function may be far more complex than simple JSON dictionary lookups, and assume we don't know any implementation details about the js function. All we know is it takes a String argument and returns and array of Strings.
Now we would like to take advantage of this function in our Java code. That is, in our Java code, we'd like to "load" this function, and then be able to call it multiple times, passing it String args and receiving String[] or ArrayList<String> results.
From searching SO and google so far, I understand that this will involve using:
javax.script.ScriptEngineManager
javax.script.ScriptEngine
and perhaps scriptEngine.getContext() for passing values into the function and receiving results.
I am a bit hazy on the details of the above, especially since most examples I've found involve running javascript code a single time, rather than making javascript function available to Java.
Example Code I'd Like To See
Assuming the js function is in the file "my_functions.js", load that file into Java so all of its functions will be available for use.
Call employeesForStore("downtown") and store its results in a native java String[] or List<String> in a variable called downtownResults.
Same as 2, but call employeesForStore("uptown") and store in variable uptownResults
Create an interface to act as a facade to your JavaScript code.
Here is an example using the Rhino implementation embedded in Oracle's Java 1.7 implementation:
package demo;
import java.io.*; import java.util.*;
import java.util.concurrent.atomic.AtomicReference;
import javax.script.*;
public class StoreData {
public static interface Stores {
public String[] employees(String store);
}
public static Stores stores() throws IOException, ScriptException {
ScriptEngineManager sem = new ScriptEngineManager();
ScriptEngine engine = sem.getEngineByName("JavaScript");
AtomicReference<Stores> ref = new AtomicReference<>();
engine.put("ref", ref);
String adapt = "ref.set("
+ "new Packages.demo.StoreData.Stores({employees:employeesForStore})"
+ ");";
try (Reader myFns = new FileReader("my_functions.js")) { // TODO encoding
engine.eval(myFns);
engine.eval(adapt);
return ref.get();
}
}
public static void main(String[] args) throws IOException, ScriptException {
List<String> employees = Arrays.asList(stores().employees("uptown"));
System.out.println(employees);
}
}
By specifying an interface we let Rhino coerce the JavaScript types to Java types (String, String[], etc.)
The JRE spec makes no guarantees about what scripting engines should be provided so it may be wise to rely on an external engine. I don't know if Nashorn will change this.
You can use Rhino API to execute JS code in java
This tutorial covers the examples requested.
The situation seems to be abnormal, but I was asked to build serializer that will parse an object into string by concatenating results of "get" methods. The values should appear in the same order as their "get" equivalent is declared in source code file.
So, for example, we have
Class testBean1{
public String getValue1(){
return "value1";
}
public String getValue2(){
return "value2";
}
}
The result should be:
"value1 - value2"
An not
"value2 - value1"
It can't be done with Class object according to the documentation. But I wonder if I can find this information in "*.class" file or is it lost? If such data exists, maybe, someone knows a ready to use tool for that purpose? If such information can't be found, please, suggest the most professional way of achieving the goal. I thought about adding some kind of custom annotations to the getters of the class that should be serialized.
If you want that you have to parse the source code, not the byte code.
There are a number of libraries that parse a source file into a node tree, my favorite is the javaparser (hosted at code.google.com), which, in a slightly modified version, is also used by spring roo.
On the usage page you can find some samples. Basically you will want to use a Visitor that listens for MethodDefinitions.
Although reflection does not anymore (as of java 7 I think) give you the methods in the order in which they appear in the source code, the class file appears to still (as of Java 8) contain the methods in the order in which they appear in the source code.
So, you can parse the class file looking for method names and then sort the methods based on the file offset in which each method was found.
If you want to do it in a less hacky way you can use Javassist, which will give you the line number of each declared method, so you can sort methods by line number.
I don't think the information is retained.
JAXB, for example, has #XmlType(propOrder="field1, field2") where you define the order of the fields when they are serialized to xml. You can implemenet something similar
Edit: This works only on concrete classes (the class to inspect has its own .class file). I changed the code below to reflect this. Until diving deeper into the ClassFileAnalyzer library to work with classes directly instead of reading them from a temporary file this limitation exists.
Following approach works for me:
Download and import following libarary ClassFileAnalyzer
Add the following two static methods (Attention! getClussDump() needs a little modification for writing out the class file to a temporary file: I removed my code here because it's very special at this point):
public static String getClassDump(Class<?> c) throws Exception {
String classFileName = c.getSimpleName() + ".class";
URL resource = c.getResource(classFileName);
if (resource == null) {
throw new RuntimeException("Works only for concreate classes!");
}
String absolutePath = ...; // write to temp file and get absolute path
ClassFile classFile = new ClassFile(absolutePath);
classFile.parse();
Info infos = new Info(classFile, absolutePath);
StringBuffer infoBuffer = infos.getInfos();
return infoBuffer.toString();
}
public static <S extends List<Method>> S sortMethodsBySourceOrder(Class<?> c, S methods) throws Exception {
String classDump = getClassDump(c);
int index = classDump.indexOf("constant_pool_count:");
final String dump = classDump.substring(index);
Collections.sort(methods, new Comparator<Method>() {
public int compare(Method o1, Method o2) {
Integer i1 = Integer.valueOf(dump.indexOf(" " + o1.getName() + lineSeparator));
Integer i2 = Integer.valueOf(dump.indexOf(" " + o2.getName() + lineSeparator));
return i1.compareTo(i2);
}});
return methods;
}
Now you can call the sortMethodsBySourceOrder with any List of methods (because sorting arrays is not very comfortable) and you will get the list back sorted.
It works by looking at the class dumps constant pool which in turn can be determined by the library.
Greetz,
GHad
Write your custom annotation to store ordering data, then use Method.getAnnotation(Class annotationClass)