Suppose I have a the following in Scala
object Foo {
var functions: List[String => String] = Nil // can be any type to any type.
def addFunc(f:String => String) = functions = f :: functions
}
At runtime, I am given Foo with some functions added. I now want to construct a new .class file implementing something like following in Scala:
object MyObject {
def process1(s:String) = // call Foo.functions(1)
}
I then want to save MyObject in bytecode that can be executed later on even when Foo is not there.
The above is just an example to show what I want to do . I am given the names MyObject, process1, and I have to generate an executable file MyObject.class. The source of MyObject is not needed (it could well have been Java source).
So, at a high level, we need to take memory "snapshot" of Foo.function(1), convert that snapshot into bytecode to store, and generate bytecode of MyObject using this.
All the bytecode engineering libraries I found are too low-level, so I was wondering if there is a higher level library that lets me deal with abstract objects such as functions etc.
Have you looked at the Tree model of ASM? I've only used the Event model before, but the Tree sounds like just what you're looking for. You'll find an overview in section 1.2.2 of the ASM user guide (a PDF--I don't think there's an HTML version, or I'd link that).
I will also recommend ASM framework. There is a paper from AOSD'07 about implementing common bytecode transformation patterns with ASM. Sections "Merging Two Classes into One" and "Inline Method" describing bytecode transformations very close to yours.
Related
I want to call Rust code from Java / Android, and I've found 3 variants to do so:
JNI
JNA
JNR FFI
JNI looks good and powerful enough, but you have to
write too much code by hand.
JNA, if not taking into consideration that it crashes on my machine, requires writing Rust struct data type description in Java by hand,
the same problem with JNR FFI.
So I wonder how difficult will be generate JNI code
for traits and struct with macros or a compiler plugin?
This compiler should match traits implementations for concrete struct,
and also struct
#[JNI]
struct Foo {
a: i32,
}
trait Boo {
fn f(&self, b: f64) -> f64;
}
#[JNI]
impl Boo for Foo {
fn f(&self, b: f64) -> f64 {
0f64
}
}
and create Java classes for struct and Java classes with native functions, plus generate pub no_mangle functions that wrap traits functions.
In order to provide #[jni] annotations that work like that you'd need to use a compiler plugin. It would be an awesome tool, but it doesn't exist yet, as far as I know.
There are bits and pieces of tooling lying around that might be helpful, if you want to create a project that does this.
Plugins are currently unstable, and don't work on non-nightly rust; you would probably want to use syntex, which provides a stable interface to compiler plugins. You could also write a raw plugin (see here for the API for those), but most people won't be able to use it.
There's rusty-cheddar, which generates c header files; you could take a look at that to see how it works. The author of that also seems to be working on a more general bindings-generation framework, but I don't know if it's active. You might be able to hook the output of cheddar up to something like JNAerator, but it probably won't create the prettiest interfaces on the java side.
There's also rust-bindgen and corrode, which work in the other direction; they translate c headers and arbitrary c code to rust respectively. I don't know if that's actually helpful.
JNI-sys provides low-level JNI bindings; rust-on-mobile is a small project that uses it. Also see First steps with Rust and Java, a blog post that shows some rudiments of getting things hooked up.
Finally, there's cbox, which lets you work around awkwardness with ownership and FFI.
Finally I created such project (link to github repository) to automate
binding creation.
You can use jnaerator to auto-generate your structure mappings for JNA.
Alternatively, if you want to use JNI (and compile some more native code) you should go with SWIG.
I am writing a scala application which loads up Groovy "plugin" classes at runtime. Once the plugins are loaded, standard scala types (like List and Option) are passed into them for processing.
Groovy naturally doesn't have syntactic sugar for scala types (particularly the Function* family), but I'd like a similar easy syntax. The best I have right now is to use the as operator to coerce groovy closures into scala types, e.g:
List<String> list = ... // Scala list
list.map({ it.toUpperCase() } as Function1<String,String>)
It would be nice to not have to include the as ... tail every time as it's bigger than the actual closure.
This question suggests changing the receiving method to not specify the parameter type, however that only works when it's a groovy method you can control. I'm using compiled scala/java classes.
I'm aware that there are various means for implicitly converting things in Groovy, however there are many small isolated groovy scripts being loaded independently. I don't know groovy well enough to understand which mechanism best suits this architecture so that all loaded plugins get this implicit conversion with minimal ceremony and imports. I have about 50+ plugins so streamlining this scala/groovy interop is worth it to me.
This is a simplified version of how I'm loading the groovy scripts (where Plugin is one of my types):
// Created once
val currentClassLoader = this.getClass.getClassLoader
val groovyClassLoader = new GroovyClassLoader(currentClassLoader)
...
// Many plugins created this way
val pluginFile = new java.io.File("path/to/some/plugin/MyPlugin.groovy")
val plugin: Plugin = groovyClassLoader.parseClass(pluginFile).asInstanceOf[Plugin]
// Use it
val list = List(1,2,3)
val answer = plugin.process(list)
Thanks in advance!
-Rohan
I checked the bytecode of Function1 scala produces and it is an interface, but by far not one with a single abstract method. This means it is useless for SAM conversion, ie. Java 8 would not be able to use lambdas to autoconvert to that. I also checked AbstractFunction1, and while it is an abstract class, it has no abstract method, so even that useless for extended SAM conversion as Groovy does it.
The only solution I see would be to write an extension module which does the conversion.
EDIT: Maybe scala-java8-compat can help. It targets allowing java8 lambdas for scala FunctionN, which would also allow SAM conversion for open blocks in case of Groovy 2.2.0 +
I know it's possible to do nice stuff with Reflection, such as invoking methods, or altering the values of fields. Is it possible to do heavier code modification, though, at runtime and programmatically?
For instance, if I have a method:
public void foo(){
this.bar = 100;
}
Can I write a program that modifies the innards of this method, notices that it assigns a constant to a field, and turns it into the following:
public int baz = 100;
public void foo(){
this.bar = baz;
}
Perhaps Java isn't really the language to do this kind of thing in - if not, I'm open to suggestions for languages that would allow me to basically reparse or inspect code in this way, and be able to alter it so precisely. I might be pipe dreaming here though, so please tell me if this is the case also.
Just adding a suggestion from a friend - Apache Commons' BCEL looks excellent:
http://commons.apache.org/bcel/manual.html
The Byte Code Engineering Library (Apache Commons BCEL™) is intended to
give users a convenient way to analyze, create, and manipulate (binary)
Java class files (those ending with .class). Classes are represented by
objects which contain all the symbolic information of the given class:
methods, fields and byte code instructions, in particular.
Such objects can be read from an existing file, be transformed by a
program (e.g. a class loader at run-time) and written to a file again.
An even more interesting application is the creation of classes from
scratch at run-time. The Byte Code Engineering Library (BCEL) may be
also useful if you want to learn about the Java Virtual Machine (JVM)
and the format of Java .class files.
You are looking for software that allows you to do bytecode manipulation, there are several frameworks to achieve this, but the two most known currently are:
ASM
javassist
When performing bytecode modifications at runtime in Java classes keep in mind the following:
If you change a class's bytecode after a class has been loaded by a classloader, you'll have to find a way to reload it's class definition (either through classloading tricks, or using hotswap functionalities)
If you change the classes interface (example add new methods or fields) you will be able only to reach them through reflection.
It's probably fair to say that Java wasn't designed with this purpose in mind, but you can do it potentially. How and when depends a little on the ultimate aim of the exercise. A couple of options:
At the source code level, you can use the Java Compiler API to
compile arbitrary code into a class file (which you can then load).
At the bytecode level, you can write an agent that installs a
ClassFileTransformer to arbitrarily alter a class "on the fly"
as it is loaded. In practice, if you do this, you will also probably
make use of a library such as BCEL (Bytecode Engineering
Library) to make manipulating the class easier.
You want to investigate program transformation systems (PTS), which provide general facilities for parsing and transforming languages at the source level. PTS provide rewrite rules that say in effect, "if you see this pattern, replace it by that pattern" using the surface syntax of the target language. This is done using full parsers so the rewrite rule really operates on language syntax and not text; such rewrite rules obviously won't attempt to modify code-like text in comments, unlike tools based on regexps.
Our DMS Software Reengineering Toolkit is one of these. It provides not only the usual parsing, AST building and prettyprinting (reproducing compilable source code complete with comments), but also supports symbol tables and control and data flow analysis. These are needed for almost any interesting transformations. DMS also has front ends for a variety of dialects of Java as well as many other languages.
Bytecode transformers exist because they are much easier to build; it is pretty easy to "parse" bytecode. Of course, you can't make permanent source changes with a bytecode transformer, so it is lot less useful.
You mean like this?
String script1 = "println(\"OK!\");";
eval( script1 );
script1 += "println(\"... well, maybe NOT OK after all\");";
eval( script2 );
Output:
OK!
OK!
... well, maybe NOT OK after all
... use a scripting extension to Java. Groovy and other things like that would probably allow you to do what you want. I've written a scripting extension which integrates with Java through reflection almost seamlessly myself; contact me if you're interested in the details.
Java claims to be object oriented and typesafe, and Scala even more so.
Internal Class fields are represented by class called Field, which you can obtain a reference to via the Reflection API.
My question: do these languages provide any way to obtain that Field reference in a typesafe way? (And if not, why on earth not? Seems like a glaring deficiency)
It would be extremely useful when mapping an Object to some external representation, for example to html fields in a template, or to column names in a database, to keep the reference names automatically in sync.
Ideally I'd like to say something like:
&(SomeClass.someField).name()
to get the name of the field declaration, similar to how java enums let you say:
MyEnum.SOME_INSTANCE.name()
[update:] after reading feedback that this functionality would somehow violate the intent of the Reflection API, I agree that Reflection is designed for things that aren't known at compile time, and that's exactly why it's so absurd to have to use it to learn things that are known at compile time, namely the Fields of the very class that it's compiling!
The compiler provides this for enums, so if the compiler is able to access the enum Field's reference to allow MyEnum.SOME_INSTANCE.name(), then there's no logical reason why it shouldn't also be able to provide this same functionality to ordinary Classes.
Is there any technological reason why this functionality couldn't be there for ordinary classes? I don't see why not, and I disagree that this functionality would "complicate" things... on the contrary it would vastly simplify the present cumbersome Reflection API techniques. Why force developers into Reflection to find out something that is known at compile time?
[update #2] as for the utility of this feature, have you ever tried using the Criteria API in JPA or Hibernate to dynamically construct a query? Have you seen the absurd work-arounds people have come up with to try to avoid having to pass in an unsafe String representation of the field to query against?
[update #3] Finally, a new JVM language called Ceylon has heeded the call and makes this trivial to do!
My question: do these languages provide any way to obtain that Field reference in a typesafe way?
Compile-time typesafe? Not that I'm aware of, at least in Java. The normal purpose of reflection in Java is for code to be able to deal with types it has no knowledge of before-hand - it's rare (in my experience) to be in a position where you want to be able to refer to a field in a known type. It does happen, but it's not very common.
(And if not, why on earth not? Seems like a glaring deficiency)
Every feature needs to be designed, implemented, tested, and has to meet the balance of providing more value than the added complexity in the language.
Personally I can think of features I'd much rather see in Java than this.
It is a pity that Java still misses this feature. This feature would not add additional complexity because it would interfere with other aspects of the language. Furthermore, being a feature that would be rarely used is not excuse. Every language is full of features and most projects make use of a small subset of them.
I really don't understand why the language allows me to do this:
Field field = MyClass.class.getField("myField"); // verbose syntax, evaluate at runtime, not type-safe, must deal with Reflective operation exceptions
But it doesn't let me do (something like) this:
Field field = MyClass::myField; // compact syntax, evaluated at compile-time, type-safe, no exceptions!
(the "::" operator is just a suggestion, borrowed from java 8 or c++)
In Scala you can you might use macros for it. See the following:
Example:
class Car(val carName: String);
object Main {
def main(args: Array[String]): Unit = {
println(FieldNameMacro.getFieldName[Car](_.carName))
}
}
So this prints the field name "carName". If you rename the field "carName" to "cName" it would print "cName" instead.
Macro:
In this case actually the expression tree of "_.carName" is passed to the macro handler, rather a executable method. In our macro we can look into this expression tree and find out the name of the field that we are referring to.
import scala.reflect.macros.whitebox.Context
import scala.language.experimental.macros
import scala.reflect.runtime.universe._
import scala.reflect.ClassTag
object FieldNameMacro {
def getFieldNameImpl[T](c: Context)(block: c.Expr[T => AnyRef]): c.Expr[String] = {
import c.universe._
// here we look inside the block-expression and
// ... bind the TermName "carName" to the value name
val Expr(Function(_, Select(_, TermName(name: String)))) = block;
// return the name as a literal expression
c.Expr(Literal(Constant(name)));
// Uncomment this to get an idea of what is "inside" the block-expression
// c.Expr(Literal(Constant(showRaw(block))));
}
def getFieldName[T](block: (T) => AnyRef): String = macro getFieldNameImpl[T]
}
I took some inspiration from http://blogs.clariusconsulting.net/kzu/linq-beyond-queries-strong-typed-reflection/. The post is about the same issue but with respect to C#.
Shortcomings
Beware that the Macro has to be called exactly as above. As for instance the following usage will lead to a compiler exception (actually it is a Scala match exception within the macro).
object Main {
def main(args: Array[String]): Unit = {
val block = (car : Car) => car.carName;
println(FieldNameMacro.getFieldName[Car](block))
}
}
The problem is that a different expression tree is passed to the macro handler. For more details about this problem have a look at Scala Macro get value for term name
In Scala 2.11 we can use this:
object R_ {
def apply[T](x: (T) => AnyRef): (Class[T], Method) = macro impl
def impl(c: whitebox.Context)(x: c.Tree) = { import c.universe._
val q"((${_: TermName}:${a: Type}) => ${_: TermName}.${p: TermName})" = x
val typeDef = a.typeSymbol
val propertyDef = p.toString
q"(classOf[$typeDef], classOf[$typeDef].getMethod($propertyDef))"
}
}
Usage:
class User(val name: String)
object Test extends App {
println(R_((a: User) => a.name))
}
And result will be:
(class mref.User,public java.lang.String mref.User.name())
Why on earth not is because you don't know the type of the field at compile-time when using reflection. This is the whole point of reflection: to give you access to class information at runtime. Of course you'll get a runtime error if you use the wrong type, but that doesn't really help much.
Unfortunately, as tricky as it is to keep names the same, it's typically even trickier to keep types the same, so it's probably not worth it for the application you have in mind.
Right now there's no way to do what you want with reasonable effort in either Scala or Java. Scala could add this information to its manifests (or somewhere else), but it doesn't presently and it's not clear to me that it's worth the effort.
In Kotlin it looks like this:
SomeClass::someField.name
It is very close to your ideal:
&(SomeClass.someField).name()
I would like to create some reverse egineered design docs based on Java code (not bytecode), instead of writing my own interpreter, what tools and APIs are available to traverse Java code, using Java code?
Reflection is on bytecode, and is limited to the method level, I want to "objectize" also the method code.
Java doc is ignoring the code itself and only based on comments, automatic UML sequnces are too strict
E.g. an API like this (forgive my ignorance of official Programming Languages Structure terms):
JavaCodeDom jcd = new JavaCodeDom(new File(pathToJavaSource), CompilerEnum.Java16)
List <ClassSrc> classes = jcd.getClasses();
ClassSrc cls = classes.get(0);
Map<MethodSignatureSrc,MethodSrc> methods = cls.getMethodsMap();
MethodSrc main = mothds.get(new MethodSignatureSrc(Modifiers.Public, Modifiers.Static, ReturnTypes.Void, "main", new MethodParams(String[].class))
List<StatementSrc> statements = main.getStatements();
for(StatementSrc statement : statements){
if(statement.getType()==StatementTypes.Assignment()){
AssignmentStatementSrc assignment = (AssignmentStatementSrc)statement;
Identifier src = assignment.getAssigneeVariable();
ExpressinoSrc = assignment.getAssignmentValue();
}
}
List<AnnotationsSrc> annotations = cls.getAnnotations();
There are several such APIs in existence (and delivered with the JDK), some of them build in in the Java Compiler (javac).
The most extensive is the Compiler Tree API, which gets you access to individual expressions (and subexpressions) in the Java source.
The language model API models types and members of types (constructors, methods, fields) - it is used by the compiler tree API and also for annotation processing. It does not give access to the contents of the methods.
Of course, on runtime you have the Reflection API (java.lang.Class and java.lang.reflect.*, together with java.lang.annotation).
To use the compiler tree API, you have to invoke the compiler, with the compiler API.
Additionally, there is the Doclet API for Javadoc, which gives you a similar view like the language model API, but additionally with the documentation comments (and parsed tags).
I once used a combination of Doclet API and Compiler Tree API to format source code beautifully (this is not online, sadly).
BCEL supports reading an manipulating Java class files. (I have not used it myself, but saw it used successfully in a third-party product.)
The Byte Code Engineering Library is intended to give users a convenient
possibility to analyze, create, and manipulate (binary) Java class files
(those ending with .class). Classes are represented by objects which
contain all the symbolic information of the given class: methods,
fields and byte code instructions, in particular.
If you're just interested in decompiling, you might find it sufficient to decompile to source code. Here's a comparison of several options for Java.
I seems ANTLR is one option, but I haven't used it
This seems to answer my question: How to generate AST from Java source-code? ( Spoon )