I am trying to use Spark/GraphX using Clojure & Flambo.
Here is the code I ended up with:
In the project.clj file:
(defproject spark-tests "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.6.0"]
[yieldbot/flambo "0.5.0"]]
:main ^:skip-aot spark-tests.core
:target-path "target/%s"
:checksum :warn
:profiles {:dev {:aot [flambo.function]}
:uberjar {:aot :all}
:provided {:dependencies
[[org.apache.spark/spark-core_2.10 "1.3.0"]
[org.apache.spark/spark-core_2.10 "1.2.0"]
[org.apache.spark/spark-graphx_2.10 "1.2.0"]]}})
And then my Clojure core.clj file:
(ns spark-tests.core
(:require [flambo.conf :as conf]
[flambo.api :as f]
[flambo.tuple :as ft])
(:import (org.apache.spark.graphx Edge)
(org.apache.spark.graphx.impl GraphImpl)))
(defonce c (-> (conf/spark-conf)
(conf/master "local")
(conf/app-name "flame_princess")))
(defonce sc (f/spark-context c))
(def users (f/parallelize sc [(ft/tuple 3 ["rxin" "student"])
(ft/tuple 7 ["jgonzal" "postdoc"])
(ft/tuple 5 ["franklin" "prof"])]))
(defn edge
[source dest attr]
(new Edge (long source) (long dest) attr))
(def relationships (f/parallelize sc [(edge 3 7 "collab")
(edge 5 3 "advisor")]))
(def g (new GraphImpl users relationships))
When I run that code, I am getting the following error:
1. Caused by java.lang.ClassCastException
Cannot cast org.apache.spark.api.java.JavaRDD to
scala.reflect.ClassTag
Class.java: 3258 java.lang.Class/cast
Reflector.java: 427 clojure.lang.Reflector/boxArg
Reflector.java: 460 clojure.lang.Reflector/boxArgs
Disclaimer: I have no Scala knowledge.
Then I thought that it may be because Flambo returns a JavaRDD when we use f/parallelize. Then I tried to convert the JavaRDD into a simple RDD as used in the GraphX example:
(def g (new GraphImpl (.rdd users) (.rdd relationships)))
But the I am getting the same error but for the ParallelCollectionRDD class...
From there, I am have idea of what may be causing this. The Java API for the Graph class is here, the Scala API for the same class is here.
What I am not clear about is how to effectively use that class signature in Clojure:
org.apache.spark.graphx.Graph<VD,ED>
(Graph is an abstract class, but I tried using GraphImpl in this example)
What I am trying to do is to re-create that Scala example using Clojure.
Any hints would be highly appreciated!
Finally got it right (I think). Here is the code that appears to be working:
(ns spark-tests.core
(:require [flambo.conf :as conf]
[flambo.api :as f]
[flambo.tuple :as ft])
(:import (org.apache.spark.graphx Edge
Graph)
(org.apache.spark.api.java JavaRDD
StorageLevels)
(scala.reflect ClassTag$)))
(defonce c (-> (conf/spark-conf)
(conf/master "local")
(conf/app-name "flame_princess")))
(defonce sc (f/spark-context c))
(def users (f/parallelize sc [(ft/tuple 3 ["rxin" "student"])
(ft/tuple 7 ["jgonzal" "postdoc"])
(ft/tuple 5 ["franklin" "prof"])]))
(defn edge
[source dest attr]
(new Edge (long source) (long dest) attr))
(def relationships (f/parallelize sc [(edge 3 7 "collab")
(edge 5 3 "advisor")
(edge 7 3 "advisor")]))
(def g (Graph/apply (.rdd users)
(.rdd relationships)
"collab"
(StorageLevels/MEMORY_ONLY)
(StorageLevels/MEMORY_ONLY)
(.apply ClassTag$/MODULE$ clojure.lang.PersistentVector)
(.apply ClassTag$/MODULE$ java.lang.String)))
(println (.count (.edges g)))
What this code returns is 3 which seems to be exact. The main issue was that I was not creating the class using Graph/Apply. In fact, it appears that this is the way to create all the objects (looks to be the constructor...). I have no idea why this is what way, but this is probably due to my lack of Scala knowledge. If anybody knows, just tell me why :)
After that I only had to fill-in the gaps for the signature of the apply function.
One thing to note are the last two parameters:
scala.reflect.ClassTag<VD> evidence$17
scala.reflect.ClassTag<ED> evidence$18
This is used to instruct Scala of the vertex attribute type (VD) and the edge attribute type (ED). The type of ED is the type of the object I used as the third parameter of the Edge class. Then the type of VD is the type of the second parameter of the tuple function.
I ran lein new app hm, then in hm/src/hm edited core.clj to be:
(ns hm.core
(:gen-class)
(:use [hm.hashmap]))
(defn -main []
(def j (new hm.hashmap))
(-add j "foo" "bar")
(println j))
and hashmap.clj to be:
(ns hm.hashmap
(:gen-class
:methods [[hashmap [] java.util.HashMap]
[add [String String]]]))
(defn -hashmap []
(def h (new java.util.HashMap))
h)
(defn -add [this key value]
(. this put key value)
this)
The goal is to make a wrapper around the HashMap so I can understand Clojure and how it ties with Java. I'm fairly new to Clojure. However, when I compile this, I get a lot of ClassNotFoundException in hashmap.clj. How can I make this work?
Note: This is a direct answer to your question. I don't recommend that you learn Clojure this way.
You need to compile your classes before you can run them. In your project.clj add this to the map:
:aot [hm.hashmap]
Then you need to run lein compile in order to compile the classes. You should see output saying the hm.hashmap class was compiled. After that run lein run to invoke the "main "function in hm.core.
I removed the :methods part of your gen-class because you're already defining them below, and that was causing the weird java.lang., error. You're going to run into other errors, but this should be enough to get you passed this issue.
Your code has some other issues, but the immediate problem here is that the signature of add is incomplete. Your add returns this, a hm.hashmap.
To fix, change the signature to return an Object, or, with additional edit, a java.util.HashMap. If you want this to work as otherwise written, you'll also need to extend rather than encapsulate.
(ns hm.hashmap
(:gen-class
:extends java.util.HashMap
:methods [[add [String String] java.util.HashMap]]))
Finally change -main in core.clj to call the method using .add instead of trying to access the private -add.
...
(.add j "foo" "bar")
...
Then
lein clean
lein compile hm.core hm.hashmap
lein run
should print
#<hashmap {foo=bar}>
Note that you cannot, as far as I know, specify returning an hm.hashmap in the signature due to the timing of the symbol resolution. See GC Issue 81: compile gen-class fail when class returns self.
I am using Leiningen and Clojure and for the life of me I can't understand why Clojure makes it so difficult just to import namespaces correctly. This is the following error
This is what I have in my core.clj file:
; namespace macro
(ns animals.core
(:require animals.animal)
(:use animals.animal)
(:import (animals.animal Dog))
(:import (animals.animal Human))
(:import (animals.animal Arthropod))
(:import (animals.animal Insect)))
; make-animals will create a vector of animal objects
(defn make-animals []
(conj []
(Dog. "Terrier" "Canis lupis familiaris")
(Human. "Human" "Homo sapiens")
(Arthropod. "Brown Recluse" "Loxosceles reclusa")
(Insect. "Fire Ant" "Solenopsis conjurata")))
; print-animals will print all the animal objects
(defn print-animals [animals]
(doseq [animal animals]
(println animal)))
; move-animals will call the move action on each animal
(defn move-animals [animals]
(doseq [animal animals]
(animals.animal/move animal)))
; entry to main program
(defn -main [& args]
(let [animals make-animals]
(do
(println "Welcome to Animals!")
(println "-------------------")
(print-animals animals))))
Then, at the REPL, I enter the following (in the src/ directory of the lein project):
user> (require 'animals.core)
nil
user> (animals.core/-main)
ClassNotFoundException animals.core java.net.URLClassLoader$1.run (URLClassLoader.java:202)
Okay... what? Why?
For reference, here is my file animal.clj also in the animals directory:
(ns animals.animal)
(defprotocol Animal
"A simple protocol for animal behaviors."
(move [this] "Method to move."))
(defrecord Dog [name species]
Animal
(move [this] (str "The " (:name this) " walks on all fours.")))
(defrecord Human [name species]
Animal
(move [this] (str "The " (:name this) " walks on two legs.")))
(defrecord Arthropod [name species]
Animal
(move [this] (str "The " (:name this) " walks on eight legs.")))
(defrecord Insect [name species]
Animal
(move [this] (str "The " (:name this) " walks on six legs.")))
With your code pasted into a fresh Leiningen project, I get a different error due to a typo in -main: (let [animals make-animals] ...) should be (let [animals (make-animals)] ...). With this change, it all works fine:
user=> (require 'animals.core)
nil
user=> (animals.core/-main)
Welcome to Animals!
-------------------
#animals.animal.Dog{:name Terrier, :species Canis lupis familiaris}
#animals.animal.Human{:name Human, :species Homo sapiens}
#animals.animal.Arthropod{:name Brown Recluse, :species Loxosceles reclusa}
#animals.animal.Insect{:name Fire Ant, :species Solenopsis conjurata}
nil
Incidentally, it doesn't matter where exactly you invoke lein repl from as long as it's somewhere inside the project directory.
I'd venture a guess that there was something the matter with your namespace when you first tried to require it and now it won't load due to some namespace loading state in your REPL. You might want to try (require :reload 'animals.core) and if that doesn't work, restart your REPL. (You could also paste your entire REPL interaction up to the ClassNotFoundException somewhere if you run into it again.)
Also, about your ns form:
You shouldn't both :require and :use the same namespace; :use already :requires it.
It's more usual to use a single :import clause (in fact, a single clause per clause type); for example,
(:import (animals.animal Dog Human Arthropod Insect))
It's purely a matter of style in Clojure, but in ClojureScript it is in fact required by the language.
I'm currently learning Clojure, and I haven't worked with any Lisp dialect before.
So, trying to make a simple GUI test, I get a ClassCastException in seesaw.
core.clj file:
(ns veg-gui.core)
(use 'seesaw.core)
(def f (frame :title "Hello world!") )
(defn -main [& args]
( (config! f :content "Hello world.")
(-> f pack! show!) ) )
The full error.
Exception in thread "main" java.lang.ClassCastException: seesaw.core.proxy$javax
.swing.JFrame$Tag$a79ba523 cannot be cast to clojure.lang.IFn
at veg_gui.core$_main.doInvoke(core.clj:8)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.Var.invoke(Var.java:411)
at user$eval5181.invoke(NO_SOURCE_FILE:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6501)
at clojure.lang.Compiler.eval(Compiler.java:6477)
at clojure.core$eval.invoke(core.clj:2797)
at clojure.main$eval_opt.invoke(main.clj:297)
at clojure.main$initialize.invoke(main.clj:316)
at clojure.main$null_opt.invoke(main.clj:349)
at clojure.main$main.doInvoke(main.clj:427)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:419)
at clojure.lang.AFn.applyToHelper(AFn.java:163)
at clojure.lang.Var.applyTo(Var.java:532)
at clojure.main.main(main.java:37)
(Also, I'm running this with lein run
(defn -main [& args]
( (config! f :content "Hello world.")
(-> f pack! show!) ) )
You have a superfluous set of parens around your function body. Compare to the following:
(defn -main [& args]
(config! f :content "Hello world.")
(-> f pack! show!))
Seesaw is not included into Clojure core, so, as in any other programming language, you need to include this library into your project. With Leiningen it's pretty easy - just add [seesaw "x.y.z"] (where "x.y.z" is the version) to your project.clj. More details and examples on project home page.
UPD. I payed more attention to your exception and your code. You have an error at line:
( (config! f :content "Hello world.")
Unlike most other languages, in Lisp brackets play more important role than just grouping arguments. When you write:
(foo ...)
foo is expected to be function (instance of IFn). Thus, Clojure tries to treat expression (config! f :content "Hello world.") as function, which is not true, so exception is thrown. To fix it just remove additional brackets:
(defn -main [& args]
(config! f :content "Hello world.")
(-> f pack! show!))
Note, however, that writing expressions sequentially in Clojure is not always possible. Normally you have to use do to perform several actions one after another. Fortunately, defn includes do implicitly, so you don't need explicit one.
Most of the top google hits for "calling clojure from java" are outdated and recommend using clojure.lang.RT to compile the source code. Could you help with a clear explanation of how to call Clojure from Java assuming you have already built a jar from the Clojure project and included it in the classpath?
Update: Since this answer was posted, some of the tools available have changed. After the original answer, there is an update including information on how to build the example with current tools.
It isn't quite as simple as compiling to a jar and calling the internal methods. There do seem to be a few tricks to make it all work though. Here's an example of a simple Clojure file that can be compiled to a jar:
(ns com.domain.tiny
(:gen-class
:name com.domain.tiny
:methods [#^{:static true} [binomial [int int] double]]))
(defn binomial
"Calculate the binomial coefficient."
[n k]
(let [a (inc n)]
(loop [b 1
c 1]
(if (> b k)
c
(recur (inc b) (* (/ (- a b) b) c))))))
(defn -binomial
"A Java-callable wrapper around the 'binomial' function."
[n k]
(binomial n k))
(defn -main []
(println (str "(binomial 5 3): " (binomial 5 3)))
(println (str "(binomial 10042 111): " (binomial 10042 111)))
)
If you run it, you should see something like:
(binomial 5 3): 10
(binomial 10042 111): 49068389575068144946633777...
And here's a Java program that calls the -binomial function in the tiny.jar.
import com.domain.tiny;
public class Main {
public static void main(String[] args) {
System.out.println("(binomial 5 3): " + tiny.binomial(5, 3));
System.out.println("(binomial 10042, 111): " + tiny.binomial(10042, 111));
}
}
It's output is:
(binomial 5 3): 10.0
(binomial 10042, 111): 4.9068389575068143E263
The first piece of magic is using the :methods keyword in the gen-class statement. That seems to be required to let you access the Clojure function something like static methods in Java.
The second thing is to create a wrapper function that can be called by Java. Notice that the second version of -binomial has a dash in front of it.
And of course the Clojure jar itself must be on the class path. This example used the Clojure-1.1.0 jar.
Update: This answer has been re-tested using the following tools:
Clojure 1.5.1
Leiningen 2.1.3
JDK 1.7.0 Update 25
The Clojure Part
First create a project and associated directory structure using Leiningen:
C:\projects>lein new com.domain.tiny
Now, change to the project directory.
C:\projects>cd com.domain.tiny
In the project directory, open the project.clj file and edit it such that the contents are as shown below.
(defproject com.domain.tiny "0.1.0-SNAPSHOT"
:description "An example of stand alone Clojure-Java interop"
:url "http://clarkonium.net/2013/06/java-clojure-interop-an-update/"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.5.1"]]
:aot :all
:main com.domain.tiny)
Now, make sure all of the dependencies (Clojure) are available.
C:\projects\com.domain.tiny>lein deps
You may see a message about downloading the Clojure jar at this point.
Now edit the Clojure file C:\projects\com.domain.tiny\src\com\domain\tiny.clj such that it contains the Clojure program shown in the original answer. (This file was created when Leiningen created the project.)
Much of the magic here is in the namespace declaration. The :gen-class tells the system to create a class named com.domain.tiny with a single static method called binomial, a function taking two integer arguments and returning a double. There are two similarly named functions binomial, a traditional Clojure function, and -binomial and wrapper accessible from Java. Note the hyphen in the function name -binomial. The default prefix is a hyphen, but it can be changed to something else if desired. The -main function just makes a couple of calls to the binomial function to assure that we are getting the correct results. To do that, compile the class and run the program.
C:\projects\com.domain.tiny>lein run
You should see output shown in the original answer.
Now package it up in a jar and put it someplace convenient. Copy the Clojure jar there too.
C:\projects\com.domain.tiny>lein jar
Created C:\projects\com.domain.tiny\target\com.domain.tiny-0.1.0-SNAPSHOT.jar
C:\projects\com.domain.tiny>mkdir \target\lib
C:\projects\com.domain.tiny>copy target\com.domain.tiny-0.1.0-SNAPSHOT.jar target\lib\
1 file(s) copied.
C:\projects\com.domain.tiny>copy "C:<path to clojure jar>\clojure-1.5.1.jar" target\lib\
1 file(s) copied.
The Java Part
Leiningen has a built-in task, lein-javac, that should be able to help with the Java compilation. Unfortunately, it seems to be broken in version 2.1.3. It can't find the installed JDK and it can't find the Maven repository. The paths to both have embedded spaces on my system. I assume that is the problem. Any Java IDE could handle the compilation and packaging too. But for this post, we're going old school and doing it at the command line.
First create the file Main.java with the contents shown in the original answer.
To compile java part
javac -g -cp target\com.domain.tiny-0.1.0-SNAPSHOT.jar -d target\src\com\domain\Main.java
Now create a file with some meta-information to add to the jar we want to build. In Manifest.txt, add the following text
Class-Path: lib\com.domain.tiny-0.1.0-SNAPSHOT.jar lib\clojure-1.5.1.jar
Main-Class: Main
Now package it all up into one big jar file, including our Clojure program and the Clojure jar.
C:\projects\com.domain.tiny\target>jar cfm Interop.jar Manifest.txt Main.class lib\com.domain.tiny-0.1.0-SNAPSHOT.jar lib\clojure-1.5.1.jar
To run the program:
C:\projects\com.domain.tiny\target>java -jar Interop.jar
(binomial 5 3): 10.0
(binomial 10042, 111): 4.9068389575068143E263
The output is essentially identical to that produced by Clojure alone, but the result has been converted to a Java double.
As mentioned, a Java IDE will probably take care of the messy compilation arguments and the packaging.
As of Clojure 1.6.0, there is a new preferred way to load and invoke Clojure functions. This method is now preferred to calling RT directly (and supersedes many of the other answers here). The javadoc is here - the main entry point is clojure.java.api.Clojure.
To lookup and call a Clojure function:
IFn plus = Clojure.var("clojure.core", "+");
plus.invoke(1, 2);
Functions in clojure.core are automatically loaded. Other namespaces can be loaded via require:
IFn require = Clojure.var("clojure.core", "require");
require.invoke(Clojure.read("clojure.set"));
IFns can be passed to higher order functions, e.g. the example below passes plus to read:
IFn map = Clojure.var("clojure.core", "map");
IFn inc = Clojure.var("clojure.core", "inc");
map.invoke(inc, Clojure.read("[1 2 3]"));
Most IFns in Clojure refer to functions. A few, however, refer to non-function data values. To access these, use deref instead of fn:
IFn printLength = Clojure.var("clojure.core", "*print-length*");
IFn deref = Clojure.var("clojure.core", "deref");
deref.invoke(printLength);
Sometimes (if using some other part of the Clojure runtime), you may need to ensure that the Clojure runtime is properly initialized - calling a method on the Clojure class is sufficient for this purpose. If you do not need to call a method on Clojure, then simply causing the class to load is sufficient (in the past there has been a similar recommendation to load the RT class; this is now preferred):
Class.forName("clojure.java.api.Clojure")
EDIT This answer was written in 2010, and worked at that time. See Alex Miller's answer for more modern solution.
What kind of code are calling from Java? If you have class generated with gen-class, then simply call it. If you want to call function from script, then look to following example.
If you want to evaluate code from string, inside Java, then you can use following code:
import clojure.lang.RT;
import clojure.lang.Var;
import clojure.lang.Compiler;
import java.io.StringReader;
public class Foo {
public static void main(String[] args) throws Exception {
// Load the Clojure script -- as a side effect this initializes the runtime.
String str = "(ns user) (defn foo [a b] (str a \" \" b))";
//RT.loadResourceScript("foo.clj");
Compiler.load(new StringReader(str));
// Get a reference to the foo function.
Var foo = RT.var("user", "foo");
// Call it!
Object result = foo.invoke("Hi", "there");
System.out.println(result);
}
}
EDIT: I wrote this answer almost three years ago. In Clojure 1.6 there is a proper API exactly for the purpose of calling Clojure from Java. Please Alex Miller's answer for up to date information.
Original answer from 2011:
As I see it, the simplest way (if you don't generate a class with AOT compilation) is to use clojure.lang.RT to access functions in clojure. With it you can mimic what you would have done in Clojure (no need to compile things in special ways):
;; Example usage of the "bar-fn" function from the "foo.ns" namespace from Clojure
(require 'foo.ns)
(foo.ns/bar-fn 1 2 3)
And in Java:
// Example usage of the "bar-fn" function from the "foo.ns" namespace from Java
import clojure.lang.RT;
import clojure.lang.Symbol;
...
RT.var("clojure.core", "require").invoke(Symbol.intern("foo.ns"));
RT.var("foo.ns", "bar-fn").invoke(1, 2, 3);
It is a bit more verbose in Java, but I hope it's clear that the pieces of code are equivalent.
This should work as long as Clojure and the source files (or compiled files) of your Clojure code is on the classpath.
I agree with clartaq's answer, but I felt that beginners could also use:
step-by-step information on how to actually get this running
information that's current for Clojure 1.3 and recent versions of leiningen.
a Clojure jar that also includes a main function, so it can be run standalone or linked as a library.
So I covered all that in this blog post.
The Clojure code looks like this:
(ns ThingOne.core
(:gen-class
:methods [#^{:static true} [foo [int] void]]))
(defn -foo [i] (println "Hello from Clojure. My input was " i))
(defn -main [] (println "Hello from Clojure -main." ))
The leiningen 1.7.1 project setup looks like this:
(defproject ThingOne "1.0.0-SNAPSHOT"
:description "Hello, Clojure"
:dependencies [[org.clojure/clojure "1.3.0"]]
:aot [ThingOne.core]
:main ThingOne.core)
The Java code looks like this:
import ThingOne.*;
class HelloJava {
public static void main(String[] args) {
System.out.println("Hello from Java!");
core.foo (12345);
}
}
Or you can also get all the code from this project on github.
This works with Clojure 1.5.0:
public class CljTest {
public static Object evalClj(String a) {
return clojure.lang.Compiler.load(new java.io.StringReader(a));
}
public static void main(String[] args) {
new clojure.lang.RT(); // needed since 1.5.0
System.out.println(evalClj("(+ 1 2)"));
}
}
If the use case is to include a JAR built with Clojure in a Java application, I have found having a separate namespace for the interface between the two worlds to be beneficial:
(ns example-app.interop
(:require [example-app.core :as core])
;; This example covers two-way communication: the Clojure library
;; relies on the wrapping Java app for some functionality (through
;; an interface that the Clojure library provides and the Java app
;; implements) and the Java app calls the Clojure library to perform
;; work. The latter case is covered by a class provided by the Clojure lib.
;;
;; This namespace should be AOT compiled.
;; The interface that the java app can implement
(gen-interface
:name com.example.WeatherForecast
:methods [[getTemperature [] Double]])
;; The class that the java app instantiates
(gen-class
:name com.example.HighTemperatureMailer
:state state
:init init
;; Dependency injection - take an instance of the previously defined
;; interface as a constructor argument
:constructors {[com.example.WeatherForecast] []}
:methods [[sendMails [] void]])
(defn -init [weather-forecast]
[[] {:weather-forecast weather-forecast}])
;; The actual work is done in the core namespace
(defn -sendMails
[this]
(core/send-mails (.state this)))
The core namespace can use the injected instance to accomplish its tasks:
(ns example-app.core)
(defn send-mails
[{:keys [weather-forecast]}]
(let [temp (.getTemperature weather-forecast)] ...))
For testing purposes, the interface can be stubbed:
(example-app.core/send-mails
(reify com.example.WeatherForecast (getTemperature [this] ...)))
Other technique that works also with other languages on top of JVM is to declare an interface for functions you want to call and then use 'proxy' function to create instance that implemennts them.
You can also use AOT compilation to create class files representing your clojure code. Read the documentation about compilation, gen-class and friends in the Clojure API docs for the details about how to do this, but in essence you will create a class that calls clojure functions for each method invocation.
Another alternative is to use the new defprotocol and deftype functionality, which will also require AOT compilation but provide better performance. I don't know the details of how to do this yet, but a question on the mailing list would probably do the trick.