We Keep Coding

Java ClassLoader - force reloading already loaded classes

I'm currently trying to load classes into my application so that I can then filter out those that don't contain any test / #Test-methods. I want to run these tests in my application afterwards.
So far, so good - except it seems like the URLClassLoader I'm using (or probably any ClassLoader) doesn't actually reload classes that are located on my applications classpath.
More precisely, a user of my application starts by selecting a number of .java source files. Those are then copied to a temporary location and a number of regex match/replace pairs are applied to the copy of the original source files. Next, the copies are compiled, and I then want to load those compiled .class-files into my application so I can run them as tests.
In most cases, this works as desired.
Unfortunately, if the fully qualified class name (FQCN) of any of the selected files is identical to the FQCN of a class that is part of this application (such as tests for this application), the ClassLoader ignores the specified path (to %temp%\myfolder\) and the (updated) version of the class file located there, but instead uses the already present/loaded version of the class.
After some research, this behaviour can be expected according to the docs (emphasis mine):
• The loadClass method in ClassLoader performs these tasks, in order, when called to load a class:
- If a class has already been loaded, it returns it.
- Otherwise, it delegates the search for the new class to the parent class loader.
- If the parent class loader does not find the class, loadClass calls the method findClass to find and load the class.
I tried using findClass, but it's unfortunately not visible.
Hence my question - does anyone know how to force java / the ClassLoader to actually load the class from the specified path, ignoring any - FQCN-wise - identical existing classes?

A classloader first delegates to its parent classloader, which is how it determines "if a class has already been loaded". If you want to force a classloader to load a new class, one way is to insert another classloader in the chain which fails to load/find the same class. Very quick, incomplete example:
class FirewallLoader extends ClassLoader {
public FirewallLoader(ClassLoader parent) {
super(parent);
}
public loadClass(String name, boolean resolve) {
if (name.equals("some.class.Xyz")) {
throw ClassNotFoundException();
}
super.loadClass(name, resolve);
}
}
You make the "FirewallLoader" the parent or your URLClassLoader, and then your URLClassLoader will load new versions of whatever classes the Firewall loader filters out.

JVM locks JAR files on startup

I've implemented a custom ClassLoader which is set via system property -Djava.system.class.loader=com.MyClassLoader. The CL contains a static initializer that invokes some code (before any class is loaded) manipulating the byte code of a class file within a jar (maven dependency) using the javassist library. This works fine, except that i cannot replace the old jar with the new one since the JVM is locking the file and only releases it when it terminates. Why is that and how can I enforce the JVM to release the lock?
Here is a little code snippet:
public class CustomClassLoader extends ClassLoader {
static {
...
modifyJar();
}
private static void modifyJar(){
URLClassLoader urlClassLoader = (URLClassLoader) Thread.currentThread().getContextClassLoader();
URL[] urls = urlClassLoader.getURLs();
for(URL url : urls) {
//find matching jar and modify byte code
}
replaceJarFile(metaData);
}
private static void replaceJarFile(JarMetaData jmd){
//add modified class to new jar file
JarFile jar = new JarFile(jmd.getJarFile());
...
//this method call returns false, jar is locked by another process (the JVM)
if(oldJarFile.delete()){
...
}
}
}
OS: Windows 10
JDK version: 1.8.0_131

It has never been specified that changing the jars in use by a JVM should be possible. And the JVM is locking the jars for quiet a long time now. Changing the classes within a jar in use would also bear the semantic problem of how to handle modifications to already loaded classes or even an ongoing class loading overlapping with a write.
Since modifying the jar would be a permanent change, the most reasonable approach for that would be doing it before starting the JVM, e.g. in a different JVM.
But if you want to change class definitions on-the-fly, you should write a Java Agent. For those JVMs supporting Java Agents, the Instrumentation API offers everything needed, e.g. transforming classes at load time or even redefining already loaded classes.
It also offers a standard way of adding jar files to the bootstrap or system class path, whereas assuming that the application class loader is a subclass of URLClassLoader will start to fail with Java 9.

URLClassLoader finding X but not Y in same folder

General idea: I'm writing on a loader for java that allows dynamically reloading classes to allow for changing the implementation, without restarting the entire program to keep the main application running and minimize downtimes. Every external piece of code is grouped by "modules", each module has a main class with a "onEnable, postEnable, onDisable" entry/exit point and can consist of any amount of classes. To load a module, the class containing the entry point is specified, then loaded. I'll reference them as "modules" and "additional classes" in the following, "module" being the class containing the above mentioned functions by implementing the "public interface Module", "additional classes" refer to everything the module would use on runtime but isn't a Module by itself (e.g. we have a Module called "Car implements Module", and that module requires a class "Engine" to function -> "Car" is the module, "Engine" is an additional class")
Code of what I'm doing to load a module initially (name is a String containing the full classname including path, example given later):
Class<?> clazz = mainLoader.loadClass(name);
Module module = (Module) clazz.newInstance();
addLoadedModule(module);
enableLoadedModule(module);
And here's how I reload the module when it's already existing, so that I can override the implementation. "m" is an instance of the current implementation of the Module that is supposed to be reloaded.
boolean differs = false;
Class<?> newClass = null;
try (URLClassLoader cl = new URLClassLoader(urls, mainLoader.getParent()))
{
// Try to load the class and check if it differs from the already known one
newClass = cl.loadClass(m.getClass().getName());
differs = m.getClass() != newClass;
}
catch (IOException | ClassNotFoundException e)
{
// Class couldn't be found, abort.
e.printStackTrace();
return;
}
if (!differs)
{
// New class == old class -> no need to reload it
return;
}
Module module = null;
try
{
// Try to instantiate the class
module = (Module) newClass.newInstance();
}
catch (InstantiationException | IllegalAccessException e)
{
// Can't instantiate, abort
e.printStackTrace();
return;
}
// Check versions, only reload if the new implementation's version differs from the current one. Version is a custom annotation, don't worry about that; the version check works fine
Version oldVersion = m.getClass().getAnnotation(Version.class);
Version newVersion = module.getClass().getAnnotation(Version.class);
if (oldVersion.equals(newVersion))
{
return;
}
// And if everything went well, disable and remove the old module from the list, then add and enable the new module.
disableModule(m);
modules.remove(m);
modules.put(module, false);
enableLoadedModule(module);
This is the mainLoader, urls is an URL[] pointing to the location containing the external classes to load:
mainLoader = new URLClassLoader(urls, this.getClass().getClassLoader());
The problem arises when I try to RE-load an implementation, that requires multiple classes:
Module of class A requires class B to function. This is what happens when I try to dynamically load, then reload class A:
load A -> "Sure, but I'll need B with it." -> automatically loads B -> "Here ya go, A works fine now."
reload A -> "Sure, but I'll need B with it." -> crashes because B couldn't be found
Both classes are located in the exact same folder, structure like this:
Class A implements Module: com/foo/bar/A.class
Class B: com/foo/bar/B.class
urls: ["com/foo/bar/"]
I call the function with load("com.foo.bar.A"), which works when attempting to load it the first time, but fails when trying to reload it as described above.
It works fine when trying to load a "single class module", the problem arises when the module relies on an additional external class. I tried using different classloaders to use as the parent for the URLClassLoader in the reloading process, those being the sysloader, Module.class.getClassLoader(), mainLoader (using that one, it won't ever find the new class definition because it already knows about it and therefor won't even attempt to load it from the drive again) and the mainLoader.getParent(), the classloader of the old module, and the parent of the modules classloader.
I'm probably just overseeing something obvious, but I can't figure out why it would manage to load the "extra" classes the first time, but fail when I reload the base class...
If you need any debug outputs or exact errors let me know, I replaced the debug outputs with comments explaining what does what so I got a fairly detailed log of what's happening when, but I didn't seem it to be necessary as it goes through the entire "check and then load" process just fine, it crashes when trying to enable the module. The "onEnable" method of the module requires the additional class B, that's where it fails. As I said, if you need the implementation of the classes A and B, Module, any other code or the debug outputs let me know and I'll add them in as requested.

There's a few things you can try:
Create an extension of UrlClassLoader so that you can track when it loads a class and what class loader is used to load the class.
Your other issue is make sure none of these classes are available on the "default" class path as that will cause that version to use. You are not overriding the default class loading behaviour which is to check the parent for the class first.
The other issue you're probably facing relates to the way the VM caches classes - I'm not entirely sure how this works - but from what I've experienced it seems that once a class is loaded it puts it in a shared storage space so that it does not load the class again. This shared space class will not be unloaded until the class loader that loaded it goes unreachable.

The solution lies in the classloader being closed and deleted as soon as the loading of the initial class is done, due to the class loader being only existant in the try/catch clause. I solved the issue by storing the classloader in a map until a new implementation of the module is loaded, then I can discard the old loader and store the new one instead.

Reflections returning Set with null elements

I'm using Reflections to find classes that have an specific annotation. My project structure is the following
One WAR package:
WEB-INF/classes/...packages.../ClassAnnoted1.class
One JAR package that is included by the war that has a class that executes this code:
Reflections reflections= new Reflections(ClasspathHelper.forWebInfClasses(servletContext))
Set set= reflections.getTypesAnnotatedWith(CustomAnnotation.class)
CustomAnnotation is also present on the JAR package.
the set size is correct (ie if I have 3 classes with the annotation in my WAR the jar, the set size comes back as 3), but all elements inside it are null instead of Class. I need to get the class and check the annotation parameters inside the class of the JAR.
Anyone got any idea of why this is happening?
EDIT:
Reflections reflections= new Reflections("com.my.customAnnotededClasses"); //package that my annoted class is in
Set set= reflections.getTypesAnnotatedWith(CustomAnnotation.class);
Also does not work, in this case the set length is zero instead of the number of classes with the annotation.
EDIT 2:
Ok, the real problem was that I was packaging my whole application as an EAR so I had the following:
EAR
----> WAR
----> JAR
The jar was included in the EAR lib folder and not on the WAR lib folder. So the jar classes couldn't see the war classes, once i made the WAR depend on the JAR directly like this:
EAR
----> WAR
---------> JAR
It started working. But the original question still stands, there might be situations where I want the Jar classes included in the EAR instead of the WAR (if i have multiple wars that need to use my jar for instance).

I guess I can't do it using the reflections library. So I did it by hand:
public static List<Class<?>> getClassesAnnotatedWith(Class annotation, ServletContext servletContext) {
List<Class<?>> webClasses, jarClasses;
webClasses= getClassesAnnotedWithFromClassLoader(annotation, servletContext.getClassLoader());
jarClasses= getClassesAnnotedWithFromClassLoader(annotation, Thread.currentThread().getContextClassLoader());
for (Class<?> jarClass : jarClasses) {
Class<?> elementToAdd= null;
for (Class<?> webClass : webClasses) {
if ( ! jarClass.getName().equals(webClass.getName())) {
elementToAdd= jarClass;
}
}
if(elementToAdd != null) {
webClasses.add(elementToAdd);
}
}
return webClasses;
}
private static List<Class<?>> getClassesAnnotedWithFromClassLoader(Class annotation, ClassLoader classLoader) {
List<Class<?>> classes= new ArrayList<Class<?>>();
Class<?> classLoaderClass= classLoader.getClass();
while (! classLoaderClass.getName().equals("java.lang.ClassLoader")) {
classLoaderClass= classLoaderClass.getSuperclass();
}
try {
Field fldClasses= classLoaderClass.getDeclaredField("classes");
fldClasses.setAccessible(true);
Vector<Class<?>> classesVector= (Vector<Class<?>>) fldClasses.get(classLoader);
for (Class c : classesVector) {
if (c.isAnnotationPresent(annotation)) {
classes.add(c);
}
}
} catch (Exception e) { }
return classes;
}
I get the ClassLoader from my WAR package through the ServletContext object. There is also a protection in case a class is defined in both the WAR and the JAR with the annotation and same name (you should probably check if the packages are the same too though).
Note that you should probably never use this code in your own projects (maybe only for debugging). It involves reflecting the ClassLoader class to make the "classes" property public. This property might not exists in Java 9 for example, so beware. This might also have some security problems if you are interacting modules written by third parties.

i had one a similar problem. are you sure, you included the annotation-classes into your classpath? if they are not loaded, they will somehow be found but not really returned and without any exception or anything

The Reflections library gave me various problems. Now I am using the reflection part of the Guava library: until now, no unexpected behavior has occurred.
In any case, I think that it is very rare that the source of the problem is the Java classloader.
Maybe try to load the class CustomAnnotation.class before to use it in the Reflections API.

Your code should work on conventional environments.
However, in different environments, such as osgi, you get:
1) urls with different protocol (bundle/vfs/...)
2) different class loader.
In the first case, you should a) add the relevant UrlType (see the DefaultUrlTypes in Vfs for examples), or b) use different method to get the urls (see other methods in ClasspathHelper and examine the returned URL list)
In the second case, you should a) pass the customClassLoader to Reflections constructor or ConfigurationBuilder in order resolving will happen, or b) query the store directly reflections.getStore().get(TypeAnnotationsScanner.class)
see also #8339845, JbossIntegration

Load Java classes based on a classpath in a properties file

My application uses JDBC database drivers. I load these from a jar file, db2jcc.jar in the case of DB2 which I'm currently working with. With this jar in the classpath, everything is fine, but I have a requirement to find the jar from a property in the application's config file instead - for example,
database.driver=/opt/IBM/db2/V9.5/java/db2jcc.jar
I can load the class via a URLClassLoader ok, but the problem is that I need to treat the object thus created as an explicit DB2XADataSource. For example:
URLClassLoader dbClassLoader = new URLClassLoader(new URL[]{driverJar});
xaClass = dbClassLoader.loadClass("com.ibm.db2.jcc.DB2XADataSource");
DB2XADataSource dataSource = (DB2XADataSource) xaClass.newInstance();
dataSource.setCurrentSchema(DATABASE_SCHEMA); // <- dataSource has to be a
dataSource.setDatabaseName(DATABASE_NAME); // DB2XADataSource to do this
(rearranged and renamed somewhat; I actually do the loadClass in the constructor of the class that contains this code, while the newInstance is in one of its methods.)
I guess I'm getting into a classloader tangle because the classloader that loaded my class is trying to find DB2XADataSource in order to do the cast, but the URL classloader is not above it in the tree. The trouble is, it being long after I should have stopped working for the day (here in the UK) I can't think how best to solve it in a vaguely neat and sane manner.
Ideas?
Thanks.

The simplest approach is to just use the java.beans API (or direct reflection if you must) to invoke the setter methods.
Alternatively: Your database code requires to link to the dynamically loaded code. Therefore, dynamically load your database code. How much is up to you. You might load almost everything except the "bootstrap".

Yep - the class can't load its own dependencies. You could do some ClassLoader magic, but I imagine it would get messy very quickly.
One way to reduce the amount of reflection would be to put any code that depends on DB2XADataSource into an implementation that is invoked via an interface available to the calling ClassLoader.
//in mydb2driver.jar
public class MyDb2Driver implements IDriver {
private DB2XADataSource dataSource = new DB2XADataSource();
public void init() {
dataSource.setCurrentSchema(DATABASE_SCHEMA);
}
//etc.
}
This is loaded with your driver:
database.driver=/opt/IBM/db2/V9.5/java/db2jcc.jar:/foo/mydb2driver.jar
Invoking code is in the regular classpath:
public interface IDriver {
public void init();
//etc.
}
...
URLClassLoader dbClassLoader = new URLClassLoader(new URL[]{driverJar});
xaClass = dbClassLoader.loadClass("foo.MyDb2Driver");
IDriver dataSource = (IDriver) xaClass.newInstance();
dataSource.init();