We Keep Coding

Interactions between code in two unnamed modules

Premise: I have had 2 classes loaded by different class loaders - one from the app classloader and the other, a custom class loader. The class loaded by the custom class loader is trying to reflectively access a private field in the class loaded via the appclass loader. In this case, I get a runtime error along the lines of "Failed to access class A from class B; A is in unnamed module of loader 'app' and B is in unnamed module of loader 'Custom'. The error goes away after adding an '--add-opens java.base/java.lang=ALL-UNNAMED'
Questions : I understand that there can be up to one unnamed module per classloader in the VM.
What part of the JLS or JDK implementation specific to hotspot VM talks about interactions between two unnamed modules across loaders ? I can see a lot of unnamed to named module interactions etc in the document, but not between two unnamed.
I understand why add-opens is solving the issue in general( the code
is being invoked reflectively and end calls into JDK happen via
java.lang.reflect APIs?). However, unclear again as to how the
add-opens works across loaders - does --add-opens=ALL-UNNAMED expected
to open the packages in the specified module to all unnamed modules
across loaders in the VM or only to that loader?
Using Java 17 + hotspot VM.

After a lot of trial and error, I finally got a reproducer - but I had to make quite a few assumptions.
Reproducer:
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.util.concurrent.Callable;
class A {
private static final String TARGET = "S3CR3T";
public static void main(String[] args) throws Throwable {
System.out.println("B should get " + TARGET);
Class<?> bClass = injectClass(A.class.getClassLoader(), "B",
A.class.getResourceAsStream("B.class").readAllBytes());
Method doIt = bClass.getMethod("doIt", Class.class);
doIt.setAccessible(true);
doIt.invoke(null, A.class);
}
private static Class<?> injectClass(ClassLoader loader, String name, byte[] code)
throws Throwable {
Method m = ClassLoader.class.getDeclaredMethod("defineClass", String.class, byte[].class,
int.class, int.class);
try {
m.setAccessible(true);
} catch (RuntimeException e) {
return new CustomClassLoader(loader, name, code).loadClass(name);
}
return (Class<?>) m.invoke(loader, name, code, 0, code.length);
}
}
class B {
public static void doIt(Class<?> target) throws Throwable {
accessField("parameter", () -> target);
accessField("classForName", () -> Class.forName("A"));
accessField("direct", B::getAClass);
}
private static Class<?> getAClass() {
return A.class;
}
private static void accessField(String desc, Callable<Class<?>> targetSupplier) {
try {
Class<?> target = targetSupplier.call();
Field f = target.getDeclaredField("TARGET");
f.setAccessible(true);
Object value = f.get(null);
System.out.println(desc + ": Got " + value);
} catch (Throwable t) {
System.err.println(desc + ": Got exception when trying to access private field: ");
t.printStackTrace(System.err);
}
}
}
class CustomClassLoader extends ClassLoader {
private final String name;
private final byte[] code;
private final ClassLoader delegate;
CustomClassLoader(ClassLoader delegate, String name, byte[] code) {
super("Custom", null);
this.name = name;
this.code = code;
this.delegate = delegate;
}
#Override
protected Class<?> findClass(String name) throws ClassNotFoundException {
if (name.equals(this.name)) {
Class<?> result = findLoadedClass(name);
if (result == null) {
result = defineClass(name, code, 0, code.length);
}
return result;
}
return delegate.loadClass(name);
}
}
Assumptions:
A is NOT public
This is supported by the error message - because now A and B are in different runtime packages, B can't access a non-public class in the other runtime package.
The error message matches.
B references A directly.
Reflective access (Class.forName) would still succeed.
Before creating a new ClassLoader you (or the library you use) first tries to use ClassLoader.defineClass.
This would explain why adding --add-opens java.base/java.lang=ALL-UNNAMED would exhibit different behavior. (Not an uncommon thing
Running this code yields:
B should get S3CR3T
parameter: Got S3CR3T
classForName: Got S3CR3T
direct: Got exception when trying to access private field:
java.lang.IllegalAccessError: failed to access class A from class B (A is in unnamed module of loader 'app'; B is in unnamed module of loader 'Custom' #3b07d329)
at Custom//B.getAClass(A.java:82)
at Custom//B.accessField(A.java:70)
at Custom//B.doIt(A.java:65)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at A.main(A.java:17)
while running it with --add-opens java.base/java.lang=ALL-UNNAMED yields
B should get S3CR3T
parameter: Got S3CR3T
classForName: Got S3CR3T
direct: Got S3CR3T
I already hinted in my assumptions about the cause for this:
A and B are in different runtime packages, and A is not public - which means A is not accessible to B.
This has nothing to do with modules.
To answer your other questions:
The canonical reference for this part is JVMS §5.3.6: read the part at the end in bold, then from the beginning
5.3.6. Modules and Layers
The Java Virtual Machine supports the organization of classes and interfaces into modules. The membership of a class or interface C in a module M is used to control access to C from classes and interfaces in modules other than M (§5.4.4).
Module membership is defined in terms of run-time packages (§5.3). A program determines the names of the packages in each module, and the class loaders that will create the classes and interfaces of the named packages; it then specifies the packages and class loaders to an invocation of the defineModules method of the class ModuleLayer. Invoking defineModules causes the Java Virtual Machine to create new run-time modules that are associated with the run-time packages of the class loaders.
Every run-time module indicates the run-time packages that it exports, which influences access to the public classes and interfaces in those run-time packages. Every run-time module also indicates the other run-time modules that it reads, which influences access by its own code to the public types and interfaces in those run-time modules.
We say that a class is in a run-time module iff the class's run-time package is associated (or will be associated, if the class is actually created) with that run-time module.
A class created by a class loader is in exactly one run-time package and therefore exactly one run-time module, because the Java Virtual Machine does not support a run-time package being associated with (or more evocatively, "split across") multiple run-time modules.
A run-time module is implicitly bound to exactly one class loader, by the semantics of defineModules. On the other hand, a class loader may create classes in more than one run-time module, because the Java Virtual Machine does not require all the run-time packages of a class loader to be associated with the same run-time module.
In other words, the relationship between class loaders and run-time modules need not be 1:1. For a given set of modules to be loaded, if a program can determine that the names of the packages in each module are found only in that module, then the program may specify only one class loader to the invocation of defineModules. This class loader will create classes across multiple run-time modules.
Every run-time module created by defineModules is part of a layer. A layer represents a set of class loaders that jointly serve to create classes in a set of run-time modules. There are two kinds of layers: the boot layer supplied by the Java Virtual Machine, and user-defined layers. The boot layer is created at Java Virtual Machine startup in an implementation-dependent manner. It associates the standard run-time module java.base with standard run-time packages defined by the bootstrap class loader, such as java.lang. User-defined layers are created by programs in order to construct sets of run-time modules that depend on java.base and other standard run-time modules.
A run-time module is implicitly part of exactly one layer, by the semantics of defineModules. However, a class loader may create classes in the run-time modules of different layers, because the same class loader may be specified to multiple invocations of defineModules. Access control is governed by a class's run-time module, not by the class loader which created the class or by the layer(s) which the class loader serves.
The set of class loaders specified for a layer, and the set of run-time modules which are part of a layer, are immutable after the layer is created. However, the ModuleLayer class affords programs a degree of dynamic control over the relationships between the run-time modules in a user-defined layer.
If a user-defined layer contains more than one class loader, then any delegation between the class loaders is the responsibility of the program that created the layer. The Java Virtual Machine does not check that the layer's class loaders delegate to each other in accordance with how the layer's run-time modules read each other. Moreover, if the layer's run-time modules are modified via the ModuleLayer class to read additional run-time modules, then the Java Virtual Machine does not check that the layer's class loaders are modified by some out-of-band mechanism to delegate in a corresponding fashion.
There are similarities and differences between class loaders and layers. On the one hand, a layer is similar to a class loader in that each may delegate to, respectively, one or more parent layers or class loaders that created, respectively, modules or classes at an earlier time. That is, the set of modules specified to a layer may depend on modules not specified to the layer, and instead specified previously to one or more parent layers. On the other hand, a layer may be used to create new modules only once, whereas a class loader may be used to create new classes or interfaces at any time via multiple invocations of the defineClass method.
It is possible for a class loader to define a class or interface in a run-time package that was not associated with a run-time module by any of the layers which the class loader serves. This may occur if the run-time package embodies a named package that was not specified to defineModules, or if the class or interface has a simple binary name (§4.2.1) and thus is a member of a run-time package that embodies an unnamed package (JLS §7.4.2). In either case, the class or interface is treated as a member of a special run-time module which is implicitly bound to the class loader. This special run-time module is known as the unnamed module of the class loader. The run-time package of the class or interface is associated with the unnamed module of the class loader. There are special rules for unnamed modules, designed to maximize their interoperation with other run-time modules, as follows:
A class loader's unnamed module is distinct from all other run-time modules bound to the same class loader.
A class loader's unnamed module is distinct from all run-time modules (including unnamed modules) bound to other class loaders.
Every unnamed module reads every run-time module.
Every unnamed module exports, to every run-time module, every run-time package associated with itself.
(Emphasis mine)
Your other 2 questions:
I understand why add-opens is solving the issue in general( the code is being invoked reflectively and end calls into JDK happen via java.lang.reflect APIs?)
No, java.lang and java.lang.reflect are two different packages. They have nothing to do with each other - except that they are in the same module, part of the platform...
It is more likely that you or a library that you use first tries to hack into java.lang.ClassLoader.defineClass - which resides in the java.lang package - and falls back to creating their own ClassLoader.
However, unclear again as to how the add-opens works across loaders - does --add-opens=ALL-UNNAMED expected to open the packages in the specified module to all unnamed modules across loaders in the VM or only to that loader?
Good question. ALL-UNNAMED implies ALL unnamed modules.
We can test this with this code (and reusing the CustomClassLoader from the reproducer:
import java.lang.reflect.Method;
public class C {
public static void main(String[] args) throws Throwable {
Class<?> d = new CustomClassLoader(C.class.getClassLoader(), "D",
C.class.getResourceAsStream("D.class").readAllBytes()).loadClass("D");
Method doIt = d.getDeclaredMethod("doIt");
doIt.setAccessible(true);
doIt.invoke(null);
}
}
class D {
public static void doIt() throws Throwable {
System.out.println("is java.lang open to me: "
+ Object.class.getModule().isOpen("java.lang", D.class.getModule()));
System.out.print("Try to set ClassLoader.defineClass accessible: ");
Method m = ClassLoader.class.getDeclaredMethod("defineClass", String.class, byte[].class,
int.class, int.class);
m.setAccessible(true);
System.out.println("Done");
}
}
When we run this code with --add-opens java.base/java.lang=ALL-UNNAMED this yields:
is java.lang open to me: true
Try to set ClassLoader.defineClass accessible: Done
So the answer is: Yes, ALL-UNNAMED opens the package to all unnamed modules, for every ClassLoader.

Unable to cast to interface class with URLClassLoader and reflection

I have a jar:
/home/cole/lib/a.jar
And in this jar I have the following interface/classes (horrible names for illustration purposes only!):
CreatorInterface.java
Base.java (implements CreatorInterface.java)
AbstractBase.java (extends Base.java)
Implementation.java (extends AbstractBase.java)
In a separate project I have the following code:
final URL[] jars = new URL[] {
new File("/home/cole/lib/a.jar").toURL();
}
final URLClassLoader classLoader = new URLClassLoader(jars, null);
final Class<?> implementation = classLoader.loadClass("Implementation");
final CreatorInterface object = (CreatorInterface)implementation.newInstance();
However when I run the above, I get the following:
java.lang.ClassCastException: Implementation cannot be cast to CreatorInterface
Given Implementation is ultimately an instance of a class that implements CreatorInterface, why do I get the ClassCastException?
Update 1
This isn't a question about using URLClassLoader, the class is found ok, the problem appears to be in the instantiation. For example, the following code works fine:
final Object object = implementation.newInstance();
Update 2
As #davidxxx answered, I have the interface class twice (once in the jar and once in the project using it). Although the interface was the same, this was the cause of the issue.
However to make it work, I needed to fix my URLClassLoader like this, to avoid a ClassNotFoundException:
final ClassLoader parent = this.getClass().getClassLoader();
final URLClassLoader classLoader = new URLClassLoader(jars, parent);

This exception :
java.lang.ClassCastException: Implementation cannot be cast to
CreatorInterface
makes me think that you have very probably two distinct CreatorInterface classes : one included in the jar and another other coming from the client program that tries to load it.
Even if the two classes have the same name (qualified names), these are different classes for each classloader as here you use two unassociated classloaders.
You have the current classloader of the program that you run and this other classloader as you specified null as parent classloader :
final URLClassLoader classLoader = new URLClassLoader(jars, null);
So as you try to assign the object created by reflection to the CreatorInterface variable, the cast fails because two distinct CreatorInterface were loaded by each classloader and are used : one coming from the classloader of your client code and another coming from the the instantiated classloader.
Using a single classloader would solve the issue but a best practice would be including the jar in the classpath of the project and to ensure to have a single version of the classes provided in the jar.
To decouple things you should probably split the jar into 2 jars : an API jar that contains only the interface and an implementation jar that depends on the API jar and that contains other classes.
In the client code, add in the classpath only the API jar to be able to assign to a variable with the interface declared type.
About your second point :
This isn't a question about using URLClassLoader, the class is found ok, the problem appears to be in the instantiation. For example, the
following code works fine:
final Object object = implementation.newInstance();
In this case you don't refer the interface type.
You indeed assign the Implementation object to an Object and not to a CreatorInterface variable.
The correct/consistent interfaces and subclasses are loaded by the classloader but here you never give a chance to provoke a ClassCastException as you never assign it to a type of a duplicate class but Object that is defined a single time.
So the problem previously encountered cannot occur.
About the third point :
However to make it work, I needed to fix my URLClassLoader like this,
to avoid a ClassNotFoundException:
final ClassLoader parent = this.getClass().getClassLoader();
final URLClassLoader classLoader = new URLClassLoader(jars, parent);
It works because here you create a classloader associated to the parent classloader.
In fact if you did :
final URLClassLoader classLoader = new URLClassLoader(jars);
It would produce the same result as the URLClassLoader object created would use by default the delegation to the parent classloader (here the classloader that started your application).

Difference in functionality between instance created by classloader and new keyword

I am little bit confused in class loading and initializing concept
1: Class.forName("test.Employee").newInstance();
2: ClassLoader.getSystemClassLoader().loadClass("test.Employee").newInstance();
3: new test.Employee();
Every line of above written code is creating an instance of Employee class but I don't understand what is the difference in all three methods.

The core differences between the three approaches come down to how the classes are located at runtime and what you can do with them.
For example...
Class.forName("test.Employee").newInstance();
Will use the current class's ClassLoader to search the class named Employee in the test package. This would allow you to discover classes that might not be available at compile time and which are loaded dynamically into the same class loader context. This will also search it's parent class loaders if the class is not found within the current context...
ClassLoader.getSystemClassLoader().loadClass("test.Employee").newInstance();
This will use the "system" ClassLoader, this typically the one that launched the main application.
Using either of these two methods is a great way to generate dynamic applications where the actual implementation of a Class is not known at compile type. The problem here is it can affect visibility and restrict what you can do with the loaded classes.
For example, while you may have loaded the test.Employee class and created an instance of it, unless you have a reference to test.Employee at compile time, you want be able to cast it. This is typically where interfaces come in very handy.
Equally, you could create your own ClassLoader instance and load classes or jars at runtime to provide, for example, plugins, factories or managers where the implementation is unknown at compile time. The functionality for these would, typically, be described through the use of interfaces.
Examples would include java.awt.Toolkit and JDBC java.sql.Driver
At the end of the day, the ClassLoader mechanism is providing a means by which a class file can be loaded and instantiated into the current JVM. The new keyword does a similar job, but the results are pre-determined at compile time
ClassLoaders are a very powerful feature and provide a lot of functionality, but can also be down right confusion, especially the way that they chain together
You might find...
The basics of Java class loaders
How ClassLoader Works in Java
of some help

You cannot create a instance of an Object unless class is loaded into the memory. In all three cases class is loaded and then instance is created.
class is loaded by Class.forName("test.Employee")
class is loaded by ClassLoader.getSystemClassLoader().loadClass("test.Employee")
class is loaded automatically as Employee class is referenced for 1st time.

Just to illustrate it with an example and complete the other answers:
public class ClassLoaderTest {
public ClassLoaderTest() throws InstantiationException, IllegalAccessException, ClassNotFoundException {
System.out.println("Current CL: "+getClass().getClassLoader());
System.out.println("Parent CL: "+getClass().getClassLoader().getParent());
// ClassTest class is defined in the current CL so I can dynamically create an instance
// from the current CL and assign it (forName uses the current CL)
ClassTest c1 = (ClassTest)Class.forName("ClassTest").newInstance();
System.out.println("CL using forName: "+c1.getClass().getClassLoader());
// the new keyword creates an instance using the current CL but doesn't have the
// advantages of creating instances dynamically
ClassTest c2 = (ClassTest) new ClassTest();
System.out.println("CL using new: "+c2.getClass().getClassLoader());
// Here we are indicating to use the system CL that in this case is the parent of the current CL
Object c3 = ClassLoader.getSystemClassLoader().loadClass("ClassTest").newInstance();
System.out.println("CL using system CL: "+c3.getClass().getClassLoader());
// This won't work because the ClassTest is defined in the current CL but I'm trying to assign it to a
// dynamically created instance of ClassTest associated to the system CL so:
// java.lang.ClassCastException: ClassTest cannot be cast to ClassTest
// ClassTest c4 = (ClassTest)ClassLoader.getSystemClassLoader().loadClass("ClassTest").newInstance();
}
public static void main(String[] args) throws InstantiationException, IllegalAccessException, ClassNotFoundException {
CustomClassLoader cl = new CustomClassLoader(Test.class.getClassLoader());
cl.loadClass("ClassLoaderTest").newInstance();
}
}
The output in my case is:
Current CL: CustomClassLoader#1cfb549
Parent CL: sun.misc.Launcher$AppClassLoader#8ed465
CL using forName: CustomClassLoader#1cfb549
CL using new: CustomClassLoader#1cfb549
CL using system CL: sun.misc.Launcher$AppClassLoader#8ed465
I'm using this custom ClassLoader (CL): www.javablogging.com/java-classloader-2-write-your-own-classloader/

Getting a list of all the bootstrap classes on the JVM?

There is a method called findBootstrapClass for a ClassLoader that returns a Class if it is bootstrapped. Is there a way to find classes has been loaded?

You could try to first get the bootstrap class loader by e.g. calling
ClassLoader bootstrapLoader = ClassLoader.getSystemClassLoader().getParent();
and then get the classes of this class loader as explained here: How can I list all classes loaded in a specific class loader.
But note, that getting the bootstrap class loader is not reliable, because it may not explicitly exist. So ClassLoader.getSystemClassLoader().getParent() may return null, as explained in the Javadoc of ClassLoader#getParent():
Returns the parent class loader for delegation. Some implementations
may use null to represent the bootstrap class loader. This method will
return null in such implementations if this class loader's parent is
the bootstrap class loader.

Java example with ClassLoader

I have small problem. I learn java SE and find class ClassLoader. I try to use it in below code:
I am trying to use URLClassLoader to dynamically load a class at runtime.
URLClassLoader urlcl = new URLClassLoader(new URL[] {new URL("file:///I:/Studia/PW/Sem6/_repozytorium/workspace/Test/testJavaLoader.jar")});
Class<?> classS = urlcl.loadClass("michal.collection.Stack");
for(Method field: classS.getMethods()) {
System.out.println(field.getName());
}
Object object = classS.newInstance();
michal.collection.Stack new_name = (michal.collection.Stack) object;
The java virtual machine does not see me class, and I get the following exception:
Exception in thread "main" java.lang.Error: Unresolved compilation problems: michal cannot be resolved to a type michal cannot be resolved to a type at Main.main(Main.java:62)
Do you know how I can solve this problem?

The above answers are both wrong, they don't understand the root problem. Your main refers to the Stack class which was loaded by one class loader. Your urlclassloader is attempting to load a class with the same name. You cannot cast the loaded to the referred because they are not the same, they belong to different classloaders. You can print the has code of each to see they are different. An equality test will also show the cclass references to be different. Your problem is probably because dependent classes referenced by sstack can be found, which will result in NoClassDefErrors etc. Your main will probably fail with a classcastexception.

Class<?> classS = urlcl.loadClass("michal.collection.Stack");
[...]
Object object = classS.newInstance();
michal.collection.Stack new_name = (michal.collection.Stack) object;
So you're attempting to dynamically load a class and then you statically refer to it. If you can already statically link to it, then its loaded and you can't load it again. You'll need to access the methods by reflection.
What you would usually do is have the loaded class implement an interface from the parent class loader. After an instance is created (usually just a single instance), then you can refer to it through a reference with a type of the interface.
public interface Stack {
[...]
}
[...]
URLClassLoader urlcl = URLClassLoader.newInstance(new URL[] {
new URL(
"file:///I:/Studia/PW/Sem6/_repozytorium/workspace/Test/testJavaLoader.jar"
)
});
Class<?> clazz = urlcl.loadClass("michal.collection.StackImpl");
Class<? extends Stack> stackClass = clazz.asSubclass(Stack.class);
Constructor<? extends Stack> ctor = stackClass.getConstructor();
Stack stack = ctor.newInstance();
(Usual Stack Overflow disclaimer about not so much as compiling.)
You'll need to add error handling to taste. URLClassLoader.newInstance adds a bit of refinement to URLClassLoader. Class.newInstance has completely broken exception handling and should be avoided.

You can't refer to the dynamically-loaded type by name in the code, since that has to be resolved at compile-time. You'll need to use the newInstance() function of the Class object you get back from loadClass().