I'm trying to figure out how to load two co-dependent Groovy scripts in java at runtime. If I have two groovy scripts like:
A.groovy
import B
class A {
A() {
B b = new B()
}
}
B.groovy
import A
class B {
B() {
A a = new A()
}
}
I would like to load them as java classes, but when I run:
ClassLoader parent = getClass().getClassLoader();
GroovyClassLoader loader = new GroovyClassLoader(parent);
loader.parseClass(new File("A.groovy"));
I get the error:
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
A.groovy: 1: unable to resolve class B
# line 1, column 1.
import B
I certainly understand the reason for the error, but is there any way to load these classes at runtime despite their co-dependency? Thanks!
GroovyClassLoader must be enabled to find B.groovy on the classpath. Normally that means you change the classpath of you application to include the root of the scripts. Since there is no package here for the scripts and since you use new File("A.groovy"), I would assume that it is here the current directory.
If you don't want to change the classpath of the application, you can also call addURL to add the path containing the scripts.
One more thing to mention.... parseClass will always create a newly parsed class. You might want to try a standard loadClass call instead to avoid compiling the file multiple times. But of course that works only after you fixed the lookup for GroovyClassLoader, because using loadClass, GroovyClassLoader will also have to look for A.groovy in the same manner it does have to look for B.groovy
Related
I'm attaching my Java agent dynamically to a java process which instruments the code. Basically it adds a static call to every start of method:
//method start
AgentClass.staticMethod();
//method body
AgentClass lies in the agent's .jar. But after instrumentation, the process starts executing the new code and it throws a NoClassDefFoundError, it cannot find AgentClass.
I tried to istrument the classes in a way to include a try-catch block and load AgentClass with forName like this:
try {
AgentClass.staticMethod();
} catch(NoClassDefFoundError e) {
Class.forName("AgentClass");
}
But then I got several errors related to the recalculation of stack frames like:
Caused by: java.lang.VerifyError: Inconsistent stackmap frames at branch target 20
I solved this by using visitMaxs() (I am using ASM library).Then I got this: StackMapTable error: bad offset.
This was solved by using GOTO instead of RETURN but then I got: ClassFormatError: Illegal local variable table in method.
Is there any easier way to solve my initial NoClassDefFoundError error?
UPDATE: My agent classes are loaded with the Application Classloader(sun.misc.Launcher$AppClassLoader), and the process which I wanted to instrument loads classes with a custom URL classloader.
UPDATE2:
This is what I wanted to transform into bytecode:
try {
AgentClass agent = AgentClass.staticMethod();
} catch (Throwable e) {
try {
Class.forName("AgentClass");
} catch (ClassNotFoundException ex) {
}
}
My MethodVisitor(I am not very good at bytecode, so the bytecode was generated automatically by ASM, using a TraceClassVisitor.):
protected MethodVisitor createVisitor(MethodVisitor mv,final String name,final String desc,int access,String signature,String[]exceptions){
int variablesCount = (8 & access) != 0 ? 0 : 1;
Type[]args=Type.getArgumentTypes(desc);
for(int i=0;i<args.length; ++i){
Type arg=args[i];
variablesCount+=arg.getSize();
}
final int varCount=variablesCount;
return new MethodVisitor(458752,mv){
public void visitCode(){
Label label0=new Label();
Label label1=new Label();
Label label2=new Label();
this.mv.visitTryCatchBlock(label0,label1,label2,"java/lang/Throwable");
Label label3=new Label();
Label label4=new Label();
Label label5=new Label();
this.mv.visitTryCatchBlock(label3,label4,label5,"java/lang/ClassNotFoundException");
this.mv.visitLabel(label0);
this.mv.visitLineNumber(42,label0);
this.mv.visitMethodInsn(Opcodes.INVOKESTATIC,"AgentClass","staticMethod","()LAgentClass;",false);
this.mv.visitVarInsn(Opcodes.ASTORE,varCount);
this.mv.visitLabel(label1);
this.mv.visitLineNumber(48,label1);
Label label6=new Label();
this.mv.visitJumpInsn(Opcodes.GOTO,label6);
this.mv.visitLabel(label2);
this.mv.visitLineNumber(43,label2);
this.mv.visitFrame(Opcodes.F_SAME1,0,null,1,new Object[]{"java/lang/Throwable"});
this.mv.visitVarInsn(Opcodes.ASTORE,0);
this.mv.visitLabel(label3);
this.mv.visitLineNumber(45,label3);
this.mv.visitLdcInsn("AgentClass");
this.mv.visitMethodInsn(Opcodes.INVOKESTATIC,"java/lang/Class","forName","(Ljava/lang/String;)Ljava/lang/Class;",false);
this.mv.visitInsn(Opcodes.POP);
this.mv.visitLabel(label4);
this.mv.visitLineNumber(47,label4);
this.mv.visitJumpInsn(Opcodes.GOTO,label6);
this.mv.visitLabel(label5);
this.mv.visitLineNumber(46,label5);
this.mv.visitFrame(Opcodes.F_FULL,1,new Object[]{"java/lang/Throwable"},1,new Object[]{"java/lang/ClassNotFoundException"});
this.mv.visitVarInsn(Opcodes.ASTORE,1);
this.mv.visitLabel(label6);
this.mv.visitLineNumber(49,label6);
this.mv.visitFrame(Opcodes.F_CHOP,1,null,0,null);
this.mv.visitInsn(Opcodes.RETURN);
this.mv.visitLocalVariable("e","Ljava/lang/Throwable;",null,label3,label6,0);
this.mv.visitMaxs(1, 2);
super.visitCode();
}
...
}
}
UPDATE 3
This is how I attach my agent during runtime:
final VirtualMachine attachedVm = VirtualMachine.attach(String.valueOf(processID));
attachedVm.loadAgent(pathOfAgent, argStr);
attachedVm.detach();
For now my guess is that your class loader hierarchy is something like:
boot class loader
platform class loader
system/application class loader
custom URL class loader
Or maybe:
boot class loader
platform class loader
system/application class loader
custom URL class loader
I.e. the application class loader and the custom URL class loader are siblings or in some other way in different parts of the class loader hierarchy, i.e. the classes loaded in one of them are unknown to the other one.
The way to solve this would be to find a common ancestor and make sure the classes needed for your instrumentation scheme are loaded there. I usually use the bootstrap class loader. Before I explain to you how to programmatically add classes to the bootstrap class loader, please try adding your agent JAR to the bootstrap class path manually on the Java command line via -Xbootclasspath/a:/path/to/your/agent.jar and see if the custom URL class loader then finds the class. I would be very surprised if that would not work. Then please report back and we can continue.
Please also explain how you attach the instrumentation agent:
via -javaagent:/path/to/your/agent.jar or
via hot-attachment during runtime (if so, please show the code)
Update after some clarifying OP comments:
It is possible to add a JAR (not single classes) to the bootstrap class path by calling method Instrumentation.appendToBootstrapClassLoaderSearch(JarFile). In your agent's premain or (for hot-attachment) agentmain methods the JVM passes you an Instrumentation instance you can use for that purpose.
Caveat: You need to add the JAR before any of the classes you need on the bootstrap classpath have been imported or used by other, already loaded classes (including the agent class itself). So if in your case the AgentClass method called by the other class in the sibling class loader happens to reside inside the same class housing the premain and agentmain methods, you want to factor that method (and all others that might be called from outside) into another utility class. Also, do not directly refer to that class from the agent main class, rather first make the agent add its own JAR to the boot class path and then call any methods in there via reflection rather than directly from the agent main class. After the agent main class has done its job, other classes can refer to the classes that are now on the bootstrap class path directly, the problem is solved.
One problem remains, though: How does the agent find out the JAR path to add to the bootstrap class path? That is up to you. You can set a system property on the command line, read the path from a file, hard-code, hand it over as an agent configuration string passed to premain/agentmain via attachedVm.loadAgent(agentPath, configString) (in this case configString containing the agent path again) or whatever. Alternatively, create an inner JAR as a resource inside the main agent JAR, containing the classes to be put on the bootstrap class loader. The agent can load the resource, save it into a temporary file and then add the temp-file path to the bootstrap class path. This is a bit complicated, but clean and thus quite popular among agent developers. Sometimes this scheme is referred to as a "trampoline agent" approach.
I want to use a 'Util' groovy script inside another groovy script. I don't want to load 'Util' class inside my 'main' groovy script every time. So using evaluate or GroovyShell don't fit my case.
My java application fetches the 'main' groovy script body from a database, parse it and call test() method from 'main' script every time.
java code :
GroovyShell groovyShell = new GroovyShell();
Script parsedScript = groovyShell.parse(scriptBody);
ResultPojo result = (ResultPojo) parsedScript.invokeMethod("test", null);
'main' script
public int test(){
// this will not work at the moment
int result = GroovyUtils.sum();
return result;
}
A 'Util' class will be located in the database too. 'Util' classes will be somehow loaded on application startup and they will be reloaded every X minutes.
class GroovyUtils{
static int sum() {
return 2+1;
}
}
Like i said i don't want to 'parse' the GroovyUtils class inside 'main' script because this is time costly.
Ideally i want to import GroovyUtils script when i need it.
import groovy.GroovyUtils;
public int test(){
int result = GroovyUtils.sum();
return result;
}
But in order to import the script, the script need to be saved in the same folder that the java application runs. The java application is deployed on a remote application server in .war format.
Can i somehow load GroovyUtils dynamically to CLASSPATH without saving it, so i can import it from my 'main' script?
Any suggestions? My main concerns is speed and reloadability.
if you'd like to create a delivery process through the database you can do it by extending GroovyClassLoader and implementing public Class loadClass(name, lookupScriptFiles, preferClassOverScript, resolve) method that will search classes in some table in a database.
Let me simplify your goal and exclude database.
There is a standard behavior of classloaders: search and load classes among the classpath
The GroovyClassLoader allows to add new paths to a classpath at runtime, so it will search additionally classes in specified folder or jar file.
classloader keeps parsed classes in memory and groovy classloader provides protected method to remove class definition by name: removeClassCacheEntry(java.lang.String)
and finally example:
/myprj/classes/util/MyClass.groovy
package util
class MyClass{
def echo(msg){ println msg }
}
code to run main script
//create shell and init classloader just once
GroovyShell gs = new GroovyShell()
gs.getClassLoader().addClasspath("/myprj/classes/")
//forces classloader to recompile on file change
//this is alternative to removeClassCacheEntry
//but in some specific cases this reload will not work
gs.getClassLoader().setShouldRecompile​(true)
Script script = gs.parse('''
import util.MyClass
new MyClass().echo("hello world")
''')
script.run() // prints 'hello world'
//removeClassCacheEntry is alternative to setShouldRecompile​
//you can use it to remove compiled class from this classloader
println gs.getClassLoader().getLoadedClasses() // outputs util.MyClass, and Script1
gs.getClassLoader().removeClassCacheEntry("util.MyClass")
println gs.getClassLoader().getLoadedClasses() // outputs Script1
returning to the database: you could have a daemon thread that scans database for groovy code changes and exports modified sources into a folder that was defined as additional classpath and triggers removeClassCacheEntry for the classloader. So, next access to removed class will force to parse it by GroovyClassLoader.
NOTE: by using dynamic class loading you could have situation when two versions of same class present in memory and they will not be comparible and assignable to each other. So, you could have the error like: could not assign MyClass to MyClass
General idea: I'm writing on a loader for java that allows dynamically reloading classes to allow for changing the implementation, without restarting the entire program to keep the main application running and minimize downtimes. Every external piece of code is grouped by "modules", each module has a main class with a "onEnable, postEnable, onDisable" entry/exit point and can consist of any amount of classes. To load a module, the class containing the entry point is specified, then loaded. I'll reference them as "modules" and "additional classes" in the following, "module" being the class containing the above mentioned functions by implementing the "public interface Module", "additional classes" refer to everything the module would use on runtime but isn't a Module by itself (e.g. we have a Module called "Car implements Module", and that module requires a class "Engine" to function -> "Car" is the module, "Engine" is an additional class")
Code of what I'm doing to load a module initially (name is a String containing the full classname including path, example given later):
Class<?> clazz = mainLoader.loadClass(name);
Module module = (Module) clazz.newInstance();
addLoadedModule(module);
enableLoadedModule(module);
And here's how I reload the module when it's already existing, so that I can override the implementation. "m" is an instance of the current implementation of the Module that is supposed to be reloaded.
boolean differs = false;
Class<?> newClass = null;
try (URLClassLoader cl = new URLClassLoader(urls, mainLoader.getParent()))
{
// Try to load the class and check if it differs from the already known one
newClass = cl.loadClass(m.getClass().getName());
differs = m.getClass() != newClass;
}
catch (IOException | ClassNotFoundException e)
{
// Class couldn't be found, abort.
e.printStackTrace();
return;
}
if (!differs)
{
// New class == old class -> no need to reload it
return;
}
Module module = null;
try
{
// Try to instantiate the class
module = (Module) newClass.newInstance();
}
catch (InstantiationException | IllegalAccessException e)
{
// Can't instantiate, abort
e.printStackTrace();
return;
}
// Check versions, only reload if the new implementation's version differs from the current one. Version is a custom annotation, don't worry about that; the version check works fine
Version oldVersion = m.getClass().getAnnotation(Version.class);
Version newVersion = module.getClass().getAnnotation(Version.class);
if (oldVersion.equals(newVersion))
{
return;
}
// And if everything went well, disable and remove the old module from the list, then add and enable the new module.
disableModule(m);
modules.remove(m);
modules.put(module, false);
enableLoadedModule(module);
This is the mainLoader, urls is an URL[] pointing to the location containing the external classes to load:
mainLoader = new URLClassLoader(urls, this.getClass().getClassLoader());
The problem arises when I try to RE-load an implementation, that requires multiple classes:
Module of class A requires class B to function. This is what happens when I try to dynamically load, then reload class A:
load A -> "Sure, but I'll need B with it." -> automatically loads B -> "Here ya go, A works fine now."
reload A -> "Sure, but I'll need B with it." -> crashes because B couldn't be found
Both classes are located in the exact same folder, structure like this:
Class A implements Module: com/foo/bar/A.class
Class B: com/foo/bar/B.class
urls: ["com/foo/bar/"]
I call the function with load("com.foo.bar.A"), which works when attempting to load it the first time, but fails when trying to reload it as described above.
It works fine when trying to load a "single class module", the problem arises when the module relies on an additional external class. I tried using different classloaders to use as the parent for the URLClassLoader in the reloading process, those being the sysloader, Module.class.getClassLoader(), mainLoader (using that one, it won't ever find the new class definition because it already knows about it and therefor won't even attempt to load it from the drive again) and the mainLoader.getParent(), the classloader of the old module, and the parent of the modules classloader.
I'm probably just overseeing something obvious, but I can't figure out why it would manage to load the "extra" classes the first time, but fail when I reload the base class...
If you need any debug outputs or exact errors let me know, I replaced the debug outputs with comments explaining what does what so I got a fairly detailed log of what's happening when, but I didn't seem it to be necessary as it goes through the entire "check and then load" process just fine, it crashes when trying to enable the module. The "onEnable" method of the module requires the additional class B, that's where it fails. As I said, if you need the implementation of the classes A and B, Module, any other code or the debug outputs let me know and I'll add them in as requested.
There's a few things you can try:
Create an extension of UrlClassLoader so that you can track when it loads a class and what class loader is used to load the class.
Your other issue is make sure none of these classes are available on the "default" class path as that will cause that version to use. You are not overriding the default class loading behaviour which is to check the parent for the class first.
The other issue you're probably facing relates to the way the VM caches classes - I'm not entirely sure how this works - but from what I've experienced it seems that once a class is loaded it puts it in a shared storage space so that it does not load the class again. This shared space class will not be unloaded until the class loader that loaded it goes unreachable.
The solution lies in the classloader being closed and deleted as soon as the loading of the initial class is done, due to the class loader being only existant in the try/catch clause. I solved the issue by storing the classloader in a map until a new implementation of the module is loaded, then I can discard the old loader and store the new one instead.
I'm trying to run Groovy scripts inside my Java Application using the GroovyClassLoader and the GroovyScriptEngineImpl, and I want to isolate the Groovy script from the parent application context.
What I mean is that when I'm running my Groovy script, I don't want it to inherit from the dependencies loaded in my Java application in order to be able to, for example, load Gson 2.5.5 in my script, even if my Java application is using Gson 3.4.1.
// Replacing the context class loader with the system one
ClassLoader initialCL = Thread.currentThread().getContextClassLoader();
Thread.currentThread().setContextClassLoader(ClassLoader.getSystemClassLoader().getParent());
// Creating my GroovyClassLoader and GroovyScriptEngine
GroovyClassLoader groovyCL = new GroovyClassLoader();
GroovyScriptEngineImpl scriptEngine = new GroovyScriptEngineImpl(groovyCL);
// Compiling and running my Groovy script
CompiledScript compiledScript = scriptEngine.compile("println \"hello\"");
compiledScript.eval();
// Going back to my initial classloader
Thread.currentThread().setContextClassLoader(initialCL);
This way, the isolation is indeed working, but the content of the script is not executed at all (even printing a line in the console for example), and I'm getting no error anywhere.
If I don't update my context classloader before creating the new GroovyClassLoader, the script is working fine, but it's inheriting from the parent dependencies.
Do you have any idea?
Thanks :)
UPDATE : After a bit more testing, It seems like the compilation is working properly, but the evaluation of the compiled script isn't doing anything. Indeed, I'm getting an error when trying to compile a script that doesn't have all the dependencies needed even tho they're present in my Java application classpath.
Okay, finally figured it out, and this is an interesting one.
I said that there was no error, and while it wasn't printed, I actually had one (quite obvious since it wasn't working...). I managed to get the error by changing slightly my method to execute Groovy scripts using a GroovyShell instead of my GroovyScriptEngineImpl and GroovyClassLoader:
Thread.currentThread().setContextClassLoader(ClassLoader.getSystemClassLoader().getParent());
GroovyClassLoader groovyCL = new GroovyClassLoader();
new GroovyShell(groovyCL).evaluate("println(\"test\")");
And here's the error I'm finally getting and that was, for some reason, hidden during my previous executions, not using the GroovyShell:
groovy.lang.GroovyRuntimeException: Failed to create Script instance for class: class Script1. Reason: java.lang.ClassCastException: Script1 cannot be cast to groovy.lang.GroovyObject
So what's the problem?
Well actually, the classloader that will be used to compile the script and the one used to evaluate the compiled script are not the same: groovyCL is used to compile the script, and the "current thread" classloader is used to create the GroovyShell object and evaluate the script.
This means that the groovy.lang.GroovyObject from both classloaders are not compatible, since the class is defined in both classloaders (even tho they're the exact same class codewise).
Then, when trying to cast the Script1 object created by groovyCL, the GroovyShell (or the mechanism I used before with GroovyScriptEngineImpl etc...) will encounter ClassCastException.
This whole thing can lead to funny errors such as:
java.lang.ClassCastException: groovy.lang.GroovyShell cannot be cast to groovy.lang.GroovyShell
The solution
So, what you want to do is to create your GroovyClassLoader instance, and, using this classloader, create all the objects of the workflow using reflection:
GroovyClassLoader groovyCL = new GroovyClassLoader();
Class groovyShellClazz = groovyCL.loadClass(GroovyShell.class.getName());
Object groovyShellObj = groovyShellClazz.newInstance();
Method evaluateMethod = groovyShellClazz.getMethod("evaluate", String.class);
evaluateMethod.invoke(groovyShellObj, "println(\"test\")");
This requires a bit more work, but the script will be compiled and evaluated by the same classloader and the problem will be fixed.
I still have to work on adapting this solution to my initial situation using the GroovyScriptEngineImpl but it's the exact same method :)
Let us have a Groovy/Java application that should use a set of classes, defined in external *.jar-files (suppose they are located near the main executable jar).
So, the main class (let us call it Main) should load plugin.jar file at runtime and call some instance method on the class, defined in that jar (for some convention, suppose the class has the name as its jar - Plugin in our case).
The Main class could not know which plugins it has until it is runned. Let's throw away the CLASSPATH and java -jar run arguments and just do the magic with code only.
So, how this could be done and how the plugin.jar should be created (using Eclipse in my case) in order to be correctly loaded?
PS: yeah, i do compile my groovy sources into jar file. But i need to perform class loading and invoke exactly on-the-fly.
The secret was really simple!
Using URLClassLoader does the trick.
So, Groovy code:
ClassLoader loader = new URLClassLoader((URL[]) [
new File("C:\\Users\\errorist\\workspace\\javatest1\\bin\\").toURI().toURL()
])
Class c = loader.loadClass("src.SomeClass1")
c.invokeMethod("main", (String[]) ["Hello", "World"])
And the Java one:
File file = new File("C:\\Users\\errorist\\workspace\\javatest1\\bin\\");
URL[] urls = new URL[] { file.toURI().toURL() };
ClassLoader loader = new URLClassLoader(urls);
Class c = loader.loadClass("src.SomeClass1");
c.invokeMethod("main", new String[] { "Hello", "World!" });
The OSGi framework supports dynamic loading of plug-ins. There are multiple implementations, including Equinox, which underpins Eclipse itself.