We Keep Coding

Dynamic (tree) structure of functions as process (and implementation in Clojure)

This will all sound probably very strange as I was not able to find an exact term for what I am trying to do.
I am developing an application that, given a set of rules (which are easily translated into functions) and input/output pairs (that are not so easily translated into the code), would allow to construct a tree of the rules to be applied to the given input to reach given output.
It is similar to expert system, except the goal is not to determine "better" (by some quality) rule/function tree (actually mapping input to output is secondary by itself) but to be able to build those trees according to some restrictions or business rules.
I am trying to do this in Clojure, but I will appreciate more general advice as well since I cannot even figure out how to call this sort of thing in short.
Going into details, supposedly I have a large flat map of details.
I have a long list of functions that I transformed to do almost the same: each function takes this large flat map of details, and applies the rule to whatever value(s) this rule concerns. Function has side effects (logs what it does) and, supposedly, single boolean output that is used by the said (to be) tree structure to determine what branch to go into (if tree splits on this function).
The idea is that I can take one function and declare it as a root of the tree. Then take either one another function and declare it to be next function to do, or take two functions and declare them as two next branches from the root depending on the root function output. And so on and so forth.
And I need to be able to do it as many times as I want, producing a tree that fits some requirements.
I will be doing all of the logic, but I need a structure that would allow me to apply the tree of functions (that I can even construct myself as long as I only need to specify it as something as simple as a list) to the given input map, without having to manually code the whole process tree for every tree I will be trying to do.
Real life example would be a large tree-like data structure (input that we can flat down easily) that every client can want to be described (side effect of functions) according to his own set of rules when it is processed (reaches output).
Does this "procedure" have a more common name than this long description?
Are there any functionalities withing Java/Clojure that can be used for it or should I try doing it myself?
For those who know Clojure, I basically need a variation of one of the (->) family that can take a tree of functions, like
(tree->
input-buffer side-effect-buffer output-buffer (f1 (f2 f4 (f5 f7)) (f3 f6)))
Edit below: adding examples:
This is just one part of a more general solution I am looking for:
A (mini)game that is based around alchemy (more generally, a mix of real chemistry and alchemy).
In this case, input is grouped measurable/observable characteristics of a concoction, for example:
(def concoction
{:observable {:color {:r 50 :g 50 :b 50} :opacity 50}
:measurable {:acidity 50 :density 50 :fluidity 50 :composition "TO DO"}
:testable {:duration 10 :known-effects [list of maps] :safety 10 :after "TO DO"}})
Output is a vector of maps each of which is similar to:
{:ingredient "ingredient-a" :amount 20 :application {:type "mix" :order 0}}
The (stand-alone) function in general consist of 3 parts:
Get one (or more) characteristics of the concoction.
Apply some restricted logic to the chosen characteristics (few entries from table of individual effects of ingredient on the resulting concoction, table of application types or huge table of combined effects of two or more ingredients).
Log processed characteristics into shared log/info output.
Append result of application of the logic to the output.
Return boolean (for now, it will be int later) that signals what level of success this step had in terms of producing output.
I changed logic around a bit so now I have only one function that applies a given piece of logic to the input (instead of having almost infinite amount of similar functions) similar to:
(defn apply-logic [input logic-to-apply]
(let [chalist (apply (:input logic-to-apply) input)
out (apply (:logic logic-to-apply) chalist)]
(info-out (apply (:format logic-to-apply) out))
(return-grade out chalist))))
; info-out uses info-output and output variables set with let somewhere outside
Then I would have a tree of logic to apply instead of functions:
(def tree-1112 '(logic1
(logic2
(logic3
(logic4 logic5)))
(logic6
(logic7)
(logic8
(logic9)))))
And a some sort of apply-tree-logic:
(defn apply-tree-logic [some-tree input]
(if (apply-logic input (take-root some-tree))
(apply-tree-logic (take-branch first some-tree) input)
(apply-tree-logic (take-branch last some-tree) input))
Practically if I could do exactly what I brought in these examples it would be pretty close to implementing it all myself.
But then it would take me ages to optimize all of this.

It sounds like what you are trying to do is similar in some respects to Plumbing.
A Graph is just a map from keywords to keyword functions. In this case, stats-graph represents the steps in taking a sequence of numbers (xs) and producing univariate statistics on those numbers (i.e., the mean m and the variance v). The names of arguments to each fnk can refer to other steps that must happen before the step executes. For instance, in the above, to execute :v, you must first execute the :m and :m2 steps (mean and mean-square respectively).

as far as i understood, you want to find all the paths in a graph from input node to output node, where every node of graph is some value, and every connection is the function application, and make a tree of them.
Here is some sketchy (and partial) solution for that:
let's say we want to get a list of arithmetic ops to make one number from another. We have functions description: a collection of pairs predicate to applicable functions. Predicate checks, if corresponding functions are applicable to some input:
(def fns
[[zero? {:add3 #(+ 3 %)}]
[#(== % 1) {:add2 #(+ 2 %) :sub10 #(- 10 %)}]
[even? {:mul3 #(* 3 %) :add2 #(+ 2 %) :add1 inc}]
[#(> % 50) {:sub49 #(- % 49)}]
[(constantly true) {:add1 inc}]])
(defn applicable-fns [fns input]
(some (fn [[pred f-map]]
(when (pred input)
f-map))
fns))
in repl:
(applicable-fns fns 1)
;; {:add2 #function[reactive-clj.core/fn--21334],
:sub10 #function[reactive-clj.core/fn--21336]}
as we can't look through all the numbers, let's just limit our domain to numbers from -100 to 100:
(defn in-domain? [i] (<= -100 i 100))
now to the function: clojure has a nice mechanism to traverse tree like nested structures: zippers
here is an example of a function, that would compute the functions' chain from input to output:
(defn get-path [input output next-fns domain-pred]
(loop [visited #{}
curr (z/zipper identity
#(map (fn [[k v]] [k (v (second %))])
(next-fns (second %)))
(constantly nil)
[nil input])]
(let [curr-out (-> curr z/node second)]
(cond (z/end? curr) nil
(or (visited curr-out) (not (domain-pred curr-out)))
(recur (conj visited curr) (-> curr z/remove z/next))
(= output curr-out) (conj (z/path curr) (z/node curr))
:else (recur (conj visited curr-out)
(z/next curr))))))
it's qoite a simple one (easier to understand, when you'll see input and output):
(get-path 1 21 (partial applicable-fns fns) in-domain?)
;; => [[nil 1] [:add2 3] [:add1 4] [:mul3 12] [:add2 14]
[:add2 16] [:add2 18] [:add2 20] [:add1 21]]
(get-path 51 29 (partial applicable-fns fns) in-domain?)
;; => [[nil 51] [:sub49 2] [:mul3 6] [:mul3 18] [:add2 20]
[:add2 22] [:add2 24] [:add2 26] [:add2 28] [:add1 29]]
so, these pairs is the result of depth-first search for the path. it's not the shortest one, but the first on, that was valid. You can read it as (-> 1 add2 add1 mul3 .... add1) => 21
if you really need all the paths from input to output, you should better read about some fancy algorithms for graphs with cycles (which can be really non a trivial task). But the most interesting question for me, is why do you need all the paths, and do you really need them? What is your final goal? how will you use this functions tree?

Clojure function evaluation

I was searching for a Clojure logarithm function and concluded there really isn't one for v1.3+. What took me so long to invoke Math/log was this:
user=> Math/log
CompilerException java.lang.RuntimeException: Unable to find static field: log in class java.lang.Math, compiling:(NO_SOURCE_PATH:0:0)
I tried variations like (use 'java.lang.Math) and so forth. It wasn't until I tried an example that I found it's there after all:
user=> (Math/log 10)
2.302585092994046
This must be by design, but what is the reason?

A static method imported from Java is not a Clojure function or any other kind of Clojure object. The compiler reports any misfit as a missing field, even where the context is looking for a function. All of these ...
Math/log
(Math/log)
(map Math/log (range 1 5))
... produce such errors.
For a corresponding Clojure function:
inc returns the function;
(inc) reports ... wrong number of args(0) ...;
(map inc (range 1 5)) returns (2 3 4 5).
You can wrap a Java method as a Clojure one:
(defn log [x] (Math/log x))
... getting the expected result:
(map log (range 1 5))
;(0.0 0.6931471805599453 1.0986122886681098 1.3862943611198906)
The Clojure Java Interop page recommends
(map #(Math/log %) (range 1 5))
... in such cases.

Clojure provides some forms for interoperating with Java classes, methods, and fields. When you access static field in a class, you can do it like this:
user=> Integer/MAX_VALUE
;=> 2147483647
user=> Math/PI
;=> 3.141592653589793
And when you want to invoke static method, you can use the following form
user=> (Math/log 10)
;=> 2.302585092994046
user=> (Math/sin 1)
;=> 0.8414709848078965
When you try to evaluate just Math/log, clojure think it's static field, which is not true.
user=> Math/log
;=> CompilerException java.lang.RuntimeException: Unable to find static field: log in class java.lang.Math, compiling:(/private/var/folders/jh/q738l9dn0hxg0vvwbty7m5hw0000gp/T/form-init145007798854806400.clj:1:5845)
You can see it from the error message(Unable to find static field:...) that the clojure REPL try to find static field log from Math class, then failed.

In Java, one uses parentheses to invoke a function:
println("hi");
Clojure also uses parentheses to invoke a function, but with a different placement:
(println "hi")
In either Java or Clojure, just typing:
println
is an error, because you don't have the parentheses to indicate a function call.
In your first example, you use the name of the function Math/log, but you had no parentheses to tell the REPL that you wanted to invoke a function.
Also, in Java, parentheses can indicate either a function call or a grouping operator:
println(2 + (3 + 4))
In Clojure, parentheses only indicate function calls, and never indicating simple grouping.

Converting JRuby RubyHash to Java Map?

I want to use JRuby to run some ruby scripts, get a result and continue using Java/Scala. Unfortunately, I don't know how to convert an object from a RubyHash to a Java or Scala Map.
Is there a simple way to do this?
Any suggestions on how to do it?

class org.jruby.RubyHash implements java.util.Map, so, it already is a Java Map, albeit a non-generic "raw" type.
Unfortunately, JRuby's Hash class only inherits the generic to_java method from Kernel, unlike Array which has its own version.
This works:
[1, 2, 3].to_java(:int)
This doesn't, although it looks like it should:
{ 'one' => 1, 'two' => 2 }.to_java(:string, :int)
[It might be worthwhile filing this as a feature request.]

org.jruby.RubyHash implements java.util.Map in a way that it attempts to convert Ruby objects to Java with "normal" Map operations.
e.g. if you have a hash such as: { 'a' => 1, 'b' => 2 } which (internally) means you have RubyString keys mapped to RubyFixnum values in a hash instance, on the Java side hash.get("a") will work fine and return 1 as a java.lang.Long

I would suggest to change your approach slightly: this would involve only changing the way you initialize objects in ruby, the rest should be transparent.
Instead of using pure Ruby hash objects (o=Hash.new or o={}), please use instead java.util.HashMap like in this snippet:
require 'java'
o = java.util.HashMap.new # you could also use o = Java::JavaUtil::HashMap.new
From then on, o would be a java object used as a hash by Ruby code, and can still be accessed using java methods.
Another mode complete snippet:
>> require 'java'
=> true
>> o = Java::JavaUtil::HashMap.new
=> {}
>> o[1] = 2
=> 2
>> o["cm"] = true
=> true
>> p o.keys # ruby methods/interface
[1, "cm"]
=> nil
>> p o.keySet().to_a # java methods
[1, "cm"]
=> nil

Dynamically generating high performance functions in clojure

I'm trying to use Clojure to dynamically generate functions that can be applied to large volumes of data - i.e. a requirement is that the functions be compiled to bytecode in order to execute fast, but their specification is not known until run time.
e.g. suppose I specify functions with a simple DSL like:
(def my-spec [:add [:multiply 2 :param0] 3])
I would like to create a function compile-spec such that:
(compile-spec my-spec)
Would return a compiled function of one parameter x that returns 2x+3.
What is the best way to do this in Clojure?

Hamza Yerlikaya has already made the most important point, which is that Clojure code is always compiled. I'm just adding an illustration and some information on some low-hanging fruit for your optimisation efforts.
Firstly, the above point about Clojure's code always being compiled includes closures returned by higher-order functions and functions created by calling eval on fn / fn* forms and indeed anything else that can act as a Clojure function. Thus you don't need a separate DSL to describe functions, just use higher order functions (and possibly macros):
(defn make-affine-function [a b]
(fn [x] (+ (* a x) b)))
((make-affine-function 31 47) 5)
; => 202
Things would be more interesting if your specs were to include information about the types of parameters, as then you could be interested in writing a macro to generate code using those type hints. The simplest example I can think of would be a variant of the above:
(defmacro make-primitive-affine-function [t a b]
(let [cast #(list (symbol (name t)) %)
x (gensym "x")]
`(fn [~x] (+ (* ~(cast a) ~(cast x)) ~(cast b)))))
((make-primitive-affine-function :int 31 47) 5)
; => 202
Use :int, :long, :float or :double (or the non-namespace-qualified symbols of corresponding names) as the first argument to take advantage of unboxed primitive arithmetic appropriate for your argument types. Depending on what your function's doing, this may give you a very significant performance boost.
Other types of hints are normally provided with the #^Foo bar syntax (^Foo bar does the same thing in 1.2); if you want to add them to macro-generated code, investigate the with-meta function (you'll need to merge '{:tag Foo} into the metadata of the symbols representing the formal arguments to your functions or let-introduced locals that you wish to put type hints on).
Oh, and in case you'd still like to know how to implement your original idea...
You can always construct the Clojure expression to define your function -- (list 'fn ['x] (a-magic-function-to-generate-some-code some-args ...)) -- and call eval on the result. That would enable you to do something like the following (it would be simpler to require that the spec includes the parameter list, but here's a version assuming arguments are to be fished out from the spec, are all called paramFOO and are to be lexicographically sorted):
(require '[clojure.walk :as walk])
(defn compile-spec [spec]
(let [params (atom #{})]
(walk/prewalk
(fn [item]
(if (and (symbol? item) (.startsWith (name item) "param"))
(do (swap! params conj item)
item)
item))
spec)
(eval `(fn [~#(sort #params)] ~#spec))))
(def my-spec '[(+ (* 31 param0) 47)])
((compile-spec my-spec) 5)
; => 202
The vast majority of the time, there is no good reason to do things this way and it should be avoided; use higher-order functions and macros instead. However, if you're doing something like, say, evolutionary programming, then it's there, providing the ultimate flexibility -- and the result is still a compiled function.

Even if you don't AOT compile your code, as soon as you define a function it gets compiled to bytecode on the fly.

Does Common Lisp have a something like java's Set Interface/implementing classes?

I need something like this, a collection of elements which contains no duplicates of any element. Does Common Lisp, specifically SBCL, have any thing like this?

For a quick solution, just use hash tables, as has been mentioned before.
However, if you prefer a more principled approach, you can take a look at FSet, which is “a functional set-theoretic collections library”. Among others, it contains classes and operations for sets and bags.
(EDIT:) The cleanest way would probably be to define your set-oriented operations as generic functions. A set of generic functions is basically equivalent to a Java interface, after all. You can simply implement methods on the standard HASH-TABLE class as a first prototype and allow other implementations as well.

Look at cl-containers. There is a set-container class.

You could use lists, though they can prove to be inefficient for representing large sets. This is done using ADJOIN or PUSHNEW to add a new element to a list, and DELETE or REMOVE to do the opposite.
(let ((set (list)))
(pushnew 11 set)
(pushnew 42 set)
(pushnew 11 set)
(print set) ; set={42,11}
(setq set (delete 42 set))
(print set)) ; set={11}
One thing to watch out for is all that these operators use EQL by default to test for potential duplicates in the set (much as Java uses the equals method). That's OK for sets holding numbers or characters, but for sets of other objects, a `deeper' equality test such as EQUAL should be specified as a :TEST keyword parameter, e.g. for a set of strings :-
(let ((set (list)))
(pushnew "foo" set :test #'equal)
(pushnew "bar" set :test #'equal)
(pushnew "foo" set :test #'equal) ; EQUAL decides that "foo"="foo"
(print set)) ; set={"bar","foo"}
Lisp's counterparts to some of Java's Set operations are:
addAll -> UNION or NUNION
containsAll -> SUBSETP
removeAll -> SET-DIFFERENCE or NSET-DIFFERENCE
retainAll -> INTERSECTION or NINTERSECTION

Yes, it has sets. See this section on "Sets" from Practical Common Lisp.
Basically, you can create a set with pushnew and adjoin, query it with member, member-if and member-if-not, and combine it with other sets with functions like intersection, union, set-difference, set-exclusive-or and subsetp.

Easily solvable using a hash table.
(let ((h (make-hash-table :test 'equalp))) ; if you're storing symbols
(loop for i from 0 upto 20
do (setf (gethash i h) (format nil "Value ~A" i)))
(loop for i from 10 upto 30
do (setf (gethash i h) (format nil "~A eulaV" i)))
(loop for k being the hash-keys of h using (hash-value v)
do (format t "~A => ~A~%" k v)))
outputs
0 => Value 0
1 => Value 1
...
9 => Value 9
10 => 10 eulaV
11 => 11 eulaV
...
29 => 29 eulaV
30 => 30 eulaV

Not that I'm aware of, but you can use hash tables for something quite similar.

Lisp hashtables are CLOS based. Specs here.

Personally, I would just implement a function which takes a list and return a unique set. I've drafted something together which works for me:
(defun make-set (list-in &optional (list-out '()))
(if (endp list-in)
(nreverse list-out)
(make-set
(cdr list-in)
(adjoin (car list-in) list-out :test 'equal))))
Basically, the adjoin function prepends an item to a list non-destructively if and only if the item is not already present in the list, accepting an optional test function (one of the Common Lisp "equal" functions). You can also use pushnew to do so destructively, but I find the tail-recursive implementation to be far more elegant. So, Lisp does export several basic functions that allow you to use a list as a set; no built-in datatype is needed because you can just use different functions for prepending things to a list.
My data source for all of this (not the function, but the info) has been a combination of the Common Lisp HyperSpec and Common Lisp the Language (2nd Edition).