General idea: I'm writing on a loader for java that allows dynamically reloading classes to allow for changing the implementation, without restarting the entire program to keep the main application running and minimize downtimes. Every external piece of code is grouped by "modules", each module has a main class with a "onEnable, postEnable, onDisable" entry/exit point and can consist of any amount of classes. To load a module, the class containing the entry point is specified, then loaded. I'll reference them as "modules" and "additional classes" in the following, "module" being the class containing the above mentioned functions by implementing the "public interface Module", "additional classes" refer to everything the module would use on runtime but isn't a Module by itself (e.g. we have a Module called "Car implements Module", and that module requires a class "Engine" to function -> "Car" is the module, "Engine" is an additional class")
Code of what I'm doing to load a module initially (name is a String containing the full classname including path, example given later):
Class<?> clazz = mainLoader.loadClass(name);
Module module = (Module) clazz.newInstance();
addLoadedModule(module);
enableLoadedModule(module);
And here's how I reload the module when it's already existing, so that I can override the implementation. "m" is an instance of the current implementation of the Module that is supposed to be reloaded.
boolean differs = false;
Class<?> newClass = null;
try (URLClassLoader cl = new URLClassLoader(urls, mainLoader.getParent()))
{
// Try to load the class and check if it differs from the already known one
newClass = cl.loadClass(m.getClass().getName());
differs = m.getClass() != newClass;
}
catch (IOException | ClassNotFoundException e)
{
// Class couldn't be found, abort.
e.printStackTrace();
return;
}
if (!differs)
{
// New class == old class -> no need to reload it
return;
}
Module module = null;
try
{
// Try to instantiate the class
module = (Module) newClass.newInstance();
}
catch (InstantiationException | IllegalAccessException e)
{
// Can't instantiate, abort
e.printStackTrace();
return;
}
// Check versions, only reload if the new implementation's version differs from the current one. Version is a custom annotation, don't worry about that; the version check works fine
Version oldVersion = m.getClass().getAnnotation(Version.class);
Version newVersion = module.getClass().getAnnotation(Version.class);
if (oldVersion.equals(newVersion))
{
return;
}
// And if everything went well, disable and remove the old module from the list, then add and enable the new module.
disableModule(m);
modules.remove(m);
modules.put(module, false);
enableLoadedModule(module);
This is the mainLoader, urls is an URL[] pointing to the location containing the external classes to load:
mainLoader = new URLClassLoader(urls, this.getClass().getClassLoader());
The problem arises when I try to RE-load an implementation, that requires multiple classes:
Module of class A requires class B to function. This is what happens when I try to dynamically load, then reload class A:
load A -> "Sure, but I'll need B with it." -> automatically loads B -> "Here ya go, A works fine now."
reload A -> "Sure, but I'll need B with it." -> crashes because B couldn't be found
Both classes are located in the exact same folder, structure like this:
Class A implements Module: com/foo/bar/A.class
Class B: com/foo/bar/B.class
urls: ["com/foo/bar/"]
I call the function with load("com.foo.bar.A"), which works when attempting to load it the first time, but fails when trying to reload it as described above.
It works fine when trying to load a "single class module", the problem arises when the module relies on an additional external class. I tried using different classloaders to use as the parent for the URLClassLoader in the reloading process, those being the sysloader, Module.class.getClassLoader(), mainLoader (using that one, it won't ever find the new class definition because it already knows about it and therefor won't even attempt to load it from the drive again) and the mainLoader.getParent(), the classloader of the old module, and the parent of the modules classloader.
I'm probably just overseeing something obvious, but I can't figure out why it would manage to load the "extra" classes the first time, but fail when I reload the base class...
If you need any debug outputs or exact errors let me know, I replaced the debug outputs with comments explaining what does what so I got a fairly detailed log of what's happening when, but I didn't seem it to be necessary as it goes through the entire "check and then load" process just fine, it crashes when trying to enable the module. The "onEnable" method of the module requires the additional class B, that's where it fails. As I said, if you need the implementation of the classes A and B, Module, any other code or the debug outputs let me know and I'll add them in as requested.
There's a few things you can try:
Create an extension of UrlClassLoader so that you can track when it loads a class and what class loader is used to load the class.
Your other issue is make sure none of these classes are available on the "default" class path as that will cause that version to use. You are not overriding the default class loading behaviour which is to check the parent for the class first.
The other issue you're probably facing relates to the way the VM caches classes - I'm not entirely sure how this works - but from what I've experienced it seems that once a class is loaded it puts it in a shared storage space so that it does not load the class again. This shared space class will not be unloaded until the class loader that loaded it goes unreachable.
The solution lies in the classloader being closed and deleted as soon as the loading of the initial class is done, due to the class loader being only existant in the try/catch clause. I solved the issue by storing the classloader in a map until a new implementation of the module is loaded, then I can discard the old loader and store the new one instead.
Related
I'm attaching my Java agent dynamically to a java process which instruments the code. Basically it adds a static call to every start of method:
//method start
AgentClass.staticMethod();
//method body
AgentClass lies in the agent's .jar. But after instrumentation, the process starts executing the new code and it throws a NoClassDefFoundError, it cannot find AgentClass.
I tried to istrument the classes in a way to include a try-catch block and load AgentClass with forName like this:
try {
AgentClass.staticMethod();
} catch(NoClassDefFoundError e) {
Class.forName("AgentClass");
}
But then I got several errors related to the recalculation of stack frames like:
Caused by: java.lang.VerifyError: Inconsistent stackmap frames at branch target 20
I solved this by using visitMaxs() (I am using ASM library).Then I got this: StackMapTable error: bad offset.
This was solved by using GOTO instead of RETURN but then I got: ClassFormatError: Illegal local variable table in method.
Is there any easier way to solve my initial NoClassDefFoundError error?
UPDATE: My agent classes are loaded with the Application Classloader(sun.misc.Launcher$AppClassLoader), and the process which I wanted to instrument loads classes with a custom URL classloader.
UPDATE2:
This is what I wanted to transform into bytecode:
try {
AgentClass agent = AgentClass.staticMethod();
} catch (Throwable e) {
try {
Class.forName("AgentClass");
} catch (ClassNotFoundException ex) {
}
}
My MethodVisitor(I am not very good at bytecode, so the bytecode was generated automatically by ASM, using a TraceClassVisitor.):
protected MethodVisitor createVisitor(MethodVisitor mv,final String name,final String desc,int access,String signature,String[]exceptions){
int variablesCount = (8 & access) != 0 ? 0 : 1;
Type[]args=Type.getArgumentTypes(desc);
for(int i=0;i<args.length; ++i){
Type arg=args[i];
variablesCount+=arg.getSize();
}
final int varCount=variablesCount;
return new MethodVisitor(458752,mv){
public void visitCode(){
Label label0=new Label();
Label label1=new Label();
Label label2=new Label();
this.mv.visitTryCatchBlock(label0,label1,label2,"java/lang/Throwable");
Label label3=new Label();
Label label4=new Label();
Label label5=new Label();
this.mv.visitTryCatchBlock(label3,label4,label5,"java/lang/ClassNotFoundException");
this.mv.visitLabel(label0);
this.mv.visitLineNumber(42,label0);
this.mv.visitMethodInsn(Opcodes.INVOKESTATIC,"AgentClass","staticMethod","()LAgentClass;",false);
this.mv.visitVarInsn(Opcodes.ASTORE,varCount);
this.mv.visitLabel(label1);
this.mv.visitLineNumber(48,label1);
Label label6=new Label();
this.mv.visitJumpInsn(Opcodes.GOTO,label6);
this.mv.visitLabel(label2);
this.mv.visitLineNumber(43,label2);
this.mv.visitFrame(Opcodes.F_SAME1,0,null,1,new Object[]{"java/lang/Throwable"});
this.mv.visitVarInsn(Opcodes.ASTORE,0);
this.mv.visitLabel(label3);
this.mv.visitLineNumber(45,label3);
this.mv.visitLdcInsn("AgentClass");
this.mv.visitMethodInsn(Opcodes.INVOKESTATIC,"java/lang/Class","forName","(Ljava/lang/String;)Ljava/lang/Class;",false);
this.mv.visitInsn(Opcodes.POP);
this.mv.visitLabel(label4);
this.mv.visitLineNumber(47,label4);
this.mv.visitJumpInsn(Opcodes.GOTO,label6);
this.mv.visitLabel(label5);
this.mv.visitLineNumber(46,label5);
this.mv.visitFrame(Opcodes.F_FULL,1,new Object[]{"java/lang/Throwable"},1,new Object[]{"java/lang/ClassNotFoundException"});
this.mv.visitVarInsn(Opcodes.ASTORE,1);
this.mv.visitLabel(label6);
this.mv.visitLineNumber(49,label6);
this.mv.visitFrame(Opcodes.F_CHOP,1,null,0,null);
this.mv.visitInsn(Opcodes.RETURN);
this.mv.visitLocalVariable("e","Ljava/lang/Throwable;",null,label3,label6,0);
this.mv.visitMaxs(1, 2);
super.visitCode();
}
...
}
}
UPDATE 3
This is how I attach my agent during runtime:
final VirtualMachine attachedVm = VirtualMachine.attach(String.valueOf(processID));
attachedVm.loadAgent(pathOfAgent, argStr);
attachedVm.detach();
For now my guess is that your class loader hierarchy is something like:
boot class loader
platform class loader
system/application class loader
custom URL class loader
Or maybe:
boot class loader
platform class loader
system/application class loader
custom URL class loader
I.e. the application class loader and the custom URL class loader are siblings or in some other way in different parts of the class loader hierarchy, i.e. the classes loaded in one of them are unknown to the other one.
The way to solve this would be to find a common ancestor and make sure the classes needed for your instrumentation scheme are loaded there. I usually use the bootstrap class loader. Before I explain to you how to programmatically add classes to the bootstrap class loader, please try adding your agent JAR to the bootstrap class path manually on the Java command line via -Xbootclasspath/a:/path/to/your/agent.jar and see if the custom URL class loader then finds the class. I would be very surprised if that would not work. Then please report back and we can continue.
Please also explain how you attach the instrumentation agent:
via -javaagent:/path/to/your/agent.jar or
via hot-attachment during runtime (if so, please show the code)
Update after some clarifying OP comments:
It is possible to add a JAR (not single classes) to the bootstrap class path by calling method Instrumentation.appendToBootstrapClassLoaderSearch(JarFile). In your agent's premain or (for hot-attachment) agentmain methods the JVM passes you an Instrumentation instance you can use for that purpose.
Caveat: You need to add the JAR before any of the classes you need on the bootstrap classpath have been imported or used by other, already loaded classes (including the agent class itself). So if in your case the AgentClass method called by the other class in the sibling class loader happens to reside inside the same class housing the premain and agentmain methods, you want to factor that method (and all others that might be called from outside) into another utility class. Also, do not directly refer to that class from the agent main class, rather first make the agent add its own JAR to the boot class path and then call any methods in there via reflection rather than directly from the agent main class. After the agent main class has done its job, other classes can refer to the classes that are now on the bootstrap class path directly, the problem is solved.
One problem remains, though: How does the agent find out the JAR path to add to the bootstrap class path? That is up to you. You can set a system property on the command line, read the path from a file, hard-code, hand it over as an agent configuration string passed to premain/agentmain via attachedVm.loadAgent(agentPath, configString) (in this case configString containing the agent path again) or whatever. Alternatively, create an inner JAR as a resource inside the main agent JAR, containing the classes to be put on the bootstrap class loader. The agent can load the resource, save it into a temporary file and then add the temp-file path to the bootstrap class path. This is a bit complicated, but clean and thus quite popular among agent developers. Sometimes this scheme is referred to as a "trampoline agent" approach.
I'm currently trying to load classes into my application so that I can then filter out those that don't contain any test / #Test-methods. I want to run these tests in my application afterwards.
So far, so good - except it seems like the URLClassLoader I'm using (or probably any ClassLoader) doesn't actually reload classes that are located on my applications classpath.
More precisely, a user of my application starts by selecting a number of .java source files. Those are then copied to a temporary location and a number of regex match/replace pairs are applied to the copy of the original source files. Next, the copies are compiled, and I then want to load those compiled .class-files into my application so I can run them as tests.
In most cases, this works as desired.
Unfortunately, if the fully qualified class name (FQCN) of any of the selected files is identical to the FQCN of a class that is part of this application (such as tests for this application), the ClassLoader ignores the specified path (to %temp%\myfolder\) and the (updated) version of the class file located there, but instead uses the already present/loaded version of the class.
After some research, this behaviour can be expected according to the docs (emphasis mine):
• The loadClass method in ClassLoader performs these tasks, in order, when called to load a class:
- If a class has already been loaded, it returns it.
- Otherwise, it delegates the search for the new class to the parent class loader.
- If the parent class loader does not find the class, loadClass calls the method findClass to find and load the class.
I tried using findClass, but it's unfortunately not visible.
Hence my question - does anyone know how to force java / the ClassLoader to actually load the class from the specified path, ignoring any - FQCN-wise - identical existing classes?
A classloader first delegates to its parent classloader, which is how it determines "if a class has already been loaded". If you want to force a classloader to load a new class, one way is to insert another classloader in the chain which fails to load/find the same class. Very quick, incomplete example:
class FirewallLoader extends ClassLoader {
public FirewallLoader(ClassLoader parent) {
super(parent);
}
public loadClass(String name, boolean resolve) {
if (name.equals("some.class.Xyz")) {
throw ClassNotFoundException();
}
super.loadClass(name, resolve);
}
}
You make the "FirewallLoader" the parent or your URLClassLoader, and then your URLClassLoader will load new versions of whatever classes the Firewall loader filters out.
I have a project with multiple modules in Android Studio. A module may have a dependency on another module, for example:
Module PhoneApp -> Module FeatureOne -> Module Services
I've included my annotation processing in the root module but the android-apt annotation processing occurs only at the top most level (PhoneApp) so that it should theoretically have access to all the modules at compile time. However, what I'm seeing in the generated java file is only the classes annotated in PhoneApp and none from the other modules.
PhoneApp/build/generated/source/apt/debug/.../GeneratedClass.java
In the other modules, I am finding a generated file in the intermediates directory that contains only the annotated files from that module.
FeatureOne/build/intermediates/classes/debug/.../GeneratedClass.class
FeatureOne/build/intermediates/classes/debug/.../GeneratedClass.java
My goal is to have a single generated file in PhoneApp that allows me to access the annotated files from all modules. Not entirely sure why the code generation process is running for each and failing to aggregate all annotations at PhoneApp. Any help appreciated.
Code is fairly simple and straight forward so far, checkIsValid() omitted as it works correctly:
Annotation Processor:
#Override
public boolean process(Set<? extends TypeElement> annotations, RoundEnvironment roundEnv) {
try {
for (Element annotatedElement : roundEnv.getElementsAnnotatedWith(GuiceModule.class)) {
if (checkIsValid(annotatedElement)) {
AnnotatedClass annotatedClass = new AnnotatedClass((TypeElement) annotatedElement);
if (!annotatedClasses.containsKey(annotatedClass.getSimpleTypeName())) {
annotatedClasses.put(annotatedClass.getSimpleTypeName(), annotatedClass);
}
}
}
if (roundEnv.processingOver()) {
generateCode();
}
} catch (ProcessingException e) {
error(e.getElement(), e.getMessage());
} catch (IOException e) {
error(null, e.getMessage());
}
return true;
}
private void generateCode() throws IOException {
PackageElement packageElement = elementUtils.getPackageElement(getClass().getPackage().getName());
String packageName = packageElement.isUnnamed() ? null : packageElement.getQualifiedName().toString();
ClassName moduleClass = ClassName.get("com.google.inject", "Module");
ClassName contextClass = ClassName.get("android.content", "Context");
TypeName arrayOfModules = ArrayTypeName.of(moduleClass);
MethodSpec.Builder methodBuilder = MethodSpec.methodBuilder("juice")
.addParameter(contextClass, "context")
.addModifiers(Modifier.PUBLIC, Modifier.STATIC)
.returns(arrayOfModules);
methodBuilder.addStatement("$T<$T> collection = new $T<>()", List.class, moduleClass, ArrayList.class);
for (String key : annotatedClasses.keySet()) {
AnnotatedClass annotatedClass = annotatedClasses.get(key);
ClassName className = ClassName.get(annotatedClass.getElement().getEnclosingElement().toString(),
annotatedClass.getElement().getSimpleName().toString());
if (annotatedClass.isContextRequired()) {
methodBuilder.addStatement("collection.add(new $T(context))", className);
} else {
methodBuilder.addStatement("collection.add(new $T())", className);
}
}
methodBuilder.addStatement("return collection.toArray(new $T[collection.size()])", moduleClass);
TypeSpec classTypeSpec = TypeSpec.classBuilder("FreshlySqueezed")
.addModifiers(Modifier.PUBLIC, Modifier.FINAL)
.addMethod(methodBuilder.build())
.build();
JavaFile.builder(packageName, classTypeSpec)
.build()
.writeTo(filer);
}
This is just for a demo of annotation processing that works with Guice, if anyone is curious.
So how can I get all the annotated classes to be included in the generated PhoneApp .java file from all modules?
It's never too late to answer a question on SO, so...
I have faced a very similar complication during one of tasks at work.
And I was able to resolve it.
Short version
All you need to know about generated classes from moduleB in moduleA is package and class name. That can be stored in some kind of MyClassesRegistrar generated class placed in known package. Use suffixes to avoid names clashing, get registrars by package. Instantiate them and use data from them.
Lond version
First of all - you will NOT be able to include your compile-time-only dependency ONLY at topmost module (lets call it "app" module as your typical android project structure does). Annotation processing just does not work that way and, as far as I could find out - nothing can be done about this.
Now to the details. My task was this:
I have human-written annotated classes. I'll name them "events". At compile time I need to generate helper-classes for those events to incorporate their structure and content (both statically-available (annotation values, consts, etc) and runtime available (I am passing event objects to those helpers when using latter). Helper class name depends on event class name with a suffix so I don't know it until code generation finished.
So after helpers are generated I create a factory and generate code to provide new helper instance based on MyEvent.class provided. Here's the problem: I only needed one factory in app module, but it should be able to provide helpers for events from library module - this can't be done straightforward.
What I did was:
skip generating factory for modules that my app module depends upon;
in non-app modules generate a so-called HelpersRegistrar implementation(s):
– they all share same package (you'll know why later);
– their names don't clash because of suffix (see below);
– differentiation between app module and library-module is done via javac "-Amylib.suffix=MyModuleName" param, that user MUST set - this is a limitation, but a minor one. No suffix must be specified for app module;
– HelpersRegistrar generated implementation can provide all I need for future factory code generating: event class name, helper class name, package (these two share package for package-visibility between helper and event) - all Strings, incorporated in POJO;
in app module I generate helpers - as usual, then I obtain HelperRegistrars by their package, instantiate them, run through their content to enrich my factory with code that provides helpers from other modules. All I needed for this was class names and a package.
Voilà! My factory can provide instances of helpers both from app module and from other modules.
The only uncertainty left is order of creating and running processor-class instances in app module and in other modules. I have not found any solid info on this, but running my example shows that compiler (and, therefore, code generation) first runs in module that we depend upon, and then - in app module (otherwise compilation of app module will be f..cked). This gives us reason to expect known order of code processor executions in different modules.
Another, slightly similar, approach is this: skip registrars, generate factories in all modules and write factory in app module to use other factories, that you get and name same way as registrars above.
Example can be seen here: https://github.com/techery/janet-analytics - this is a library where I applied this approach (the one without registrars since I have factories, but that can be not the case for you).
P. S.: suffix param can be switched to simpler "-Amylibraryname.library=true" and factories/registrars names can be autogenerated/incremented
Instead of using Filer to save generated file, use regular java file writing instead. You will need to serialize objects to temp files when processing because even static variables won't save in between modules. Configure gradle to delete the temp files before compilation.
I am trying to write an Eclipse plugin that can run JUnit tests and do something with the results. My plugin loads a given class correctly, but fails to run the JUnit tests and gives an error: initializationError(className): No runnable methods. When I run the test class using Result result = JUnitCore.runClasses(className.class); Failure failure : result.getFailures(); from within the same Eclipse instance, however, I don't get any errors.
I think my problem is the one that #gubby describes in the question java.lang.Exception: No runnable methods exception in running JUnits, but I don't know how to implement his suggestion to a solution which reads: "Solution is to load JUnitCore in the same ClassLoader as the tests themselves."
Here is a reduced version of my implementation (please assume that everything except the loading of the runnable methods work):
ClassLoader classLoader = ClassLoaderHelper.getClassLoader(FileFinder.getCurrentProject());
Class clazz = classLoader.loadClass(fileName.substring(0, fileName.indexOf(".class")));
Result result = JUnitCore.runClasses(clazz);
Failure failure : result.getFailures()
The code to get the ClassLoader is the following:
public static ClassLoader getClassLoader(IProject project) {
String[] classPathEntries = null;
try {
project.open(null);
IJavaProject javaProject = JavaCore.create(project);
classPathEntries = JavaRuntime.computeDefaultRuntimeClassPath(javaProject);
} catch (CoreException e1) {
e1.printStackTrace();
}
List<URL> urlList = new ArrayList<URL>();
for (String entry : classPathEntries) {
IPath path = new Path(entry);
URL url;
try {
url = path.toFile().toURI().toURL();
urlList.add(url);
} catch (MalformedURLException e) {
e.printStackTrace();
}
}
ClassLoader parentClassLoader = project.getClass().getClassLoader();
URL[] urls = (URL[]) urlList.toArray(new URL[urlList.size()]);
URLClassLoader classLoader = new URLClassLoader(urls, parentClassLoader);
return classLoader;
}
For Eclipse plugins, you have basically two options to have one plugin share a class loader with another.
Buddy Class Loading. Note that this breaks loose coupling, but it's easy to "implement" as you simply add two statements in the two respective plugins' MANIFEST.MF and an export statement as well. The following rules apply (from the link given above).
The bundle Y must specify the registered buddy policy (i.e. Eclipse-BuddyPolicy: registered)
The bundle X must specify the symbolic name of Y in the Eclipse-RegisterBuddy header (i.e Eclipse-RegisterBuddy: Y)
The bundle X must be dependent on a package exported by bundle Y. This can happen through either a Require-Bundle or Import-Package constraint.
Fragments: You can "attach" a fragment to a plugin. Both share the same class loader. Usually, this technique is used for things like plugin i18n, but it's also the recommended practice for adding unit tests to plugins. This way, the tests don't have to go into the same plugin, and possibly unneeded test classes or dependencies won't go in the production code.
There's a wizard for fragments in Eclipse, but they're basically plugins themselves, which declare a "host plugin".
So, you could consider putting your code into a fragment and attach that to the respective plugin containing the code-under-test. Or, if you need to reuse your code for different plugins and do not care about loose coupling, use Buddy Class Loading.
Also, check whether you have all the right dependencies in the plugin (e.g., org.junit). You need the JUnit contained in the Java Development Tools.
I'm trying to figure out how to load two co-dependent Groovy scripts in java at runtime. If I have two groovy scripts like:
A.groovy
import B
class A {
A() {
B b = new B()
}
}
B.groovy
import A
class B {
B() {
A a = new A()
}
}
I would like to load them as java classes, but when I run:
ClassLoader parent = getClass().getClassLoader();
GroovyClassLoader loader = new GroovyClassLoader(parent);
loader.parseClass(new File("A.groovy"));
I get the error:
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
A.groovy: 1: unable to resolve class B
# line 1, column 1.
import B
I certainly understand the reason for the error, but is there any way to load these classes at runtime despite their co-dependency? Thanks!
GroovyClassLoader must be enabled to find B.groovy on the classpath. Normally that means you change the classpath of you application to include the root of the scripts. Since there is no package here for the scripts and since you use new File("A.groovy"), I would assume that it is here the current directory.
If you don't want to change the classpath of the application, you can also call addURL to add the path containing the scripts.
One more thing to mention.... parseClass will always create a newly parsed class. You might want to try a standard loadClass call instead to avoid compiling the file multiple times. But of course that works only after you fixed the lookup for GroovyClassLoader, because using loadClass, GroovyClassLoader will also have to look for A.groovy in the same manner it does have to look for B.groovy