We Keep Coding

Java Module Layers: Access classes in custom ModuleLayer from unnamed module

We have a plugin system with multiple module layers:
The PluginLoader module inside the boot layer instantiates a custom ModuleLayer for each Module at runtime with ModuleLayer::defineModulesWithOneLoader.
We now have a set of non-modular plugins that need access to the modular plugins inside the custom layers. They will be loaded after the custom layers have been instantiated. How can this be achieved?
Simply putting them on the classpath doesn't work, since the modular classes will be loaded a second time instead of using the (already loaded) classes from the modular plugins.
Our next try was creating a dummy plugin and loading it last inside a layer that has all other layers (and therefore modular plugins) as parents. It doesn't seem to be possible to add URLs to the class loader generated by the defineModulesWithOneLoader method. Instead we created a custom URLClassLoader as a parent of this class loader and added the non-modular JARs via URLClassLoader::addURL.
This works for the non-modular plugin classes. However, if a class already loaded in one of the modular plugins is referenced from such a class, it will also attempt to load the class with our custom URLClassLoader (which fails, since the parent is null) instead of the module's internal class loader, which delegates to parent layers. Is there a way to load our non-modular plugin's classes inside the unnamed module of a class loader that behaves like the JPMS' default class loader (search parent layers first) but still loads the classes inside the non-modular JAR, if they aren't found in the parent layers?
It is possible to solve this by using automatic modules, but we want to avoid this for now since it leads to a slew of other necessary changes (e.g. resolve split packages) which we currently don't have the resources for.

Spring boot runnable jar can't find classloader set via java.system.class.loader jvm parameter

In a module structure like this:
project
|
|- common module
|- app module
Where app module has common module as a dependency, I have a custom classloader class defined in the common module. The app module has a -Djava.system.class.loader=org.project.common.CustomClassLoader jvm parameter set to use that custom classloader defined in common module.
Running a spring boot project within IDEA this works perfectly. The custom classloader is found, set as a system classloader and everything works.
Compiling a runnable jar (using default spring-boot-maven-plugin without any custom properties), the jar itself has all the classes and within it's lib directory is the common jar which has the custom classloader. However running the jar with the -Djava.system.class.loader=org.project.common.CustomClassLoader results in the following exception
java.lang.Error: org.project.common.CustomClassLoader
at java.lang.ClassLoader.initSystemClassLoader(java.base#12.0.2/ClassLoader.java:1989)
at java.lang.System.initPhase3(java.base#12.0.2/System.java:2132)
Caused by: java.lang.ClassNotFoundException: org.project.common.CustomClassLoader
at jdk.internal.loader.BuiltinClassLoader.loadClass(java.base#12.0.2/BuiltinClassLoader.java:583)
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(java.base#12.0.2/ClassLoaders.java:178)
at java.lang.ClassLoader.loadClass(java.base#12.0.2/ClassLoader.java:521)
at java.lang.Class.forName0(java.base#12.0.2/Native Method)
at java.lang.Class.forName(java.base#12.0.2/Class.java:415)
at java.lang.ClassLoader.initSystemClassLoader(java.base#12.0.2/ClassLoader.java:1975)
at java.lang.System.initPhase3(java.base#12.0.2/System.java:2132)
Why does this happen? Is it because in the runnable jar the classloader class is in a jar in lib directory so the classloader is trying to get set before the lib classes were added to the classpath? Is there anything I can do besides moving the classloader from common to all the other modules that need it?
EDIT: I've tried moving the custom classloader class from common module to app but I am still getting the same error. What is going on here?

Running a spring boot project within IDEA this works perfectly. The custom classloader is found, set as a system classloader and everything works.
Because IDEA puts your modules on the class path and one of them contains the custom class loader.
Is it because in the runnable jar the classloader class is in a jar in lib directory so the classloader is trying to get set before the lib classes were added to the classpath?
Kind of. The lib classes are not "added to the class path", but the runnable Spring Boot app's own custom class loader knows where to find and how to load them.
For a deeper understanding of java.system.class.loader, please read the Javadoc for ClassLoader.getSystemClassLoader() (slightly reformatted with added enumeration):
If the system property java.system.class.loader is defined when this method is first invoked then the value of that property is taken to be the name of a class that will be returned as the system class loader.
The class is loaded using the default system class loader and must define a public constructor that takes a single parameter of type ClassLoader which is used as the delegation parent.
An instance is then created using this constructor with the default system class loader as the parameter.
The resulting class loader is defined to be the system class loader.
During construction, the class loader should take great care to avoid calling getSystemClassLoader(). If circular initialization of the system class loader is detected then an IllegalStateException is thrown.
The decisive factor here is #3: The user-defined system class loader is loaded by the default system class loader. The latter of course has no clue about how to load something from a nested JAR. Only later, after the JVM is fully initialised and Spring Boot's special application class loader kicks in, can those nested JARs be read.
I.e. you are having a chicken vs. egg problem here: In order to find your custom class loader during JVM initialisation, you would need to use the Spring Boot runnable JAR class loader which has not been initialised yet.
If you want to know how what the Javadoc above describes is done in practice, take a look at the OpenJDK source code of ClassLoader.initSystemClassLoader().
Is there anything I can do besides moving the classloader from common to all the other modules that need it?
Even that would not help if you insist in using the runnable JAR. What you could do is either of these:
Run your application without zipping it up into a runnable JAR, but as a normal Java application with all application modules (especially the one containing the custom class loader) on the class path.
Extract your custom class loader into a separate module outside of the runnable JAR and put it on the class path when running the runnable JAR.
Set your custom class loader via Thread.setContextClassLoader() or so instead of trying to use it as a system class loader, if that would be a viable option.
Update 2020-10-28: In the document "The Executable Jar Format" I found this under "Executable Jar Restrictions":
System classLoader: Launched applications should use Thread.getContextClassLoader() when loading classes (most libraries and frameworks do so by default). Trying to load nested jar classes with ClassLoader.getSystemClassLoader() fails. java.util.Logging always uses the system classloader. For this reason, you should consider a different logging implementation.
This confirms what I wrote above, especially my last bullet point about using the thread context class loader.

Assuming you want to add custom jar to the classpath with Spring, do the following:
Generate the jar file with the maven jar plugin
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>libs/</classpathPrefix>
<mainClass>
com.demo.DemoApplication
</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
While running the application from the command line, use the below command
java -cp target/demo-0.0.1-SNAPSHOT.jar -Dloader.path=<Path to the Custom Jar file> org.springframework.boot.loader.PropertiesLauncher
This should launch your app while loading the Custom Classloader as well
In short, the trick is, to use the -Dloader.path along with org.springframework.boot.loader.PropertiesLauncher

An application launched on the lines of -
java -cp ./lib/* com.example.Main
would ideally be sufficient.
Will need some clarity on how the application is being used.
Is the main class itself being attempted to be launched from a custom class loader (assuming its possible to do so) or whether post launch specific application related classes are required to be loaded with a custom class-loader (and associated privileges)?
Have asked those questions in the comments above (planning to update the answers here once have more clarity).
PS: Haven't really factored the use of 'modules' yet but believe the above syntax would still hold for the newer jdk's (after jdk 8).

How to get a Module object from a ModuleReference

Using the code:
ModuleFinder.of(Paths.get(path)).findAll()
I am able to retrieve a Set<ModuleReference> of all the .jars in the path folder.
My next step would be getting a Module from a ModuleReference but there's no method that returns that, I can get a ModuleDescriptor but even that one doesn't help. Is there a way to do this?

If you desire to access the module content you should open the ModuleReference that you've attained.
This would provide you access to the ModuleReader which
is intended for cases where access to the resources in a module is
required
A resource in a module is identified by an abstract name that is a
'/'-separated path string. For example, module java.base
may have a resource "java/lang/Object.class" that, by convention,
is the class file for java.lang.Object. A module reader may treat
directories in the module content as resources (whether it does or not is
module reader specific). Where the module content contains a directory
that can be located as a resource then its name ends with a slash ('/'). The directory can also be located with a name that drops the trailing slash.
Do keep in mind though, that the docs also specify :
A ModuleReader is open upon creation and is closed by invoking the
close
method. Failure to close a module reader may result in a resource
leak. The try-with-resources statement provides a useful construct to
ensure that module readers are closed.
One way to get the Module from the resources would be to access it using the Class#getModule as:
Module module = com.foo.bar.YourClass.class.getModule();
Edit: I've learned with time a better way to use the ModuleFinder to access a Module as suggested by #Alan as well could possibly be :
ModuleFinder finder = ModuleFinder.of(path);
ModuleLayer parent = ModuleLayer.boot();
Configuration configuration = parent.configuration().resolve(finder, ModuleFinder.of(), Set.of("curious")); // 'curious' being the name of the module
ClassLoader systemClassLoader = ClassLoader.getSystemClassLoader();
ModuleLayer layer = parent.defineModulesWithOneLoader(configuration, systemClassLoader);
Module m = layer.findModule("curious").orElse(null);

The classes in java.lang.module are more model world. Look at j.l.module.Configuration and also j.l.ModuleLayer to see how to create a configuration and instantiate it in the Java virtual machine as a layer of modules. There is a code fragment in the ModuleLayer javadoc that might get you going, just bewarned that this is an advanced topic and best to master the basics first.

URLClassLoader can't handle jar:file urls?

I want to load a library in separated classloader because don't want to add directly as dependency to do not conflict with other versions in the project.
I created a loader:
public LibLoader(String resourcePath) {
//resourcePath="/lib/Log4JHack-1.0.jar"
URL url = getClass().getResource(resourcePath);
loader = new URLClassLoader(new URL[] {url}, getClass().getClassLoader());
}
url = [file:/D:/..../lib/Log4JHack-1.0.jar]
if url is a file, then it works well.
url = [jar:file:/C:/..../Log4JHackLoader-1.0.jar!/lib/Log4JHack-1.0.jar]
if url is a jar:file (jar inside jar), then it don't work:
ERROR StatusLogger Unable to open jar:jar:file:/C:/Users/Dani/.m2/repository/hu/daniel/hari/log4jhack/Log4JHackLoader/1.0/Log4JHackLoader-1.0.jar!/lib/Log4JHack-1.0.jar!/META-INF/log4j-provider.properties java.net.MalformedURLException: no !/ in spec
at java.net.URL.<init>(URL.java:620)
at java.net.URL.<init>(URL.java:483)
at java.net.URL.<init>(URL.java:432)
at java.net.JarURLConnection.parseSpecs(JarURLConnection.java:175)
at java.net.JarURLConnection.<init>(JarURLConnection.java:158)
at sun.net.www.protocol.jar.JarURLConnection.<init>(JarURLConnection.java:81)
at sun.net.www.protocol.jar.Handler.openConnection(Handler.java:41)
at java.net.URL.openConnection(URL.java:972)
at java.net.URL.openStream(URL.java:1038)
at org.apache.logging.log4j.util.ProviderUtil.loadProvider(ProviderUtil.java:79)
at org.apache.logging.log4j.util.ProviderUtil.<init>(ProviderUtil.java:66)
at org.apache.logging.log4j.util.ProviderUtil.lazyInit(ProviderUtil.java:122)
at org.apache.logging.log4j.util.ProviderUtil.hasProviders(ProviderUtil.java:106)
at org.apache.logging.log4j.LogManager.<clinit>(LogManager.java:91)
at hu.daniel.hari.log4jpattern.logrenderer.log4j.log4j.capture.log4j2.StringLoggerCapture.<clinit>(StringLoggerCapture.java:34)
at hu.daniel.hari.log4jpattern.logrenderer.log4j.Log4j2Hack.doRender(Log4j2Hack.java:30)
at hu.daniel.hari.log4jpattern.logrenderer.log4j.Log4j2Hack.render(Log4j2Hack.java:23)
at hu.daniel.hari.log4jpattern.logrenderer.log4j.renderer.AbstractLog4jRendererAdapter.render(AbstractLog4jRendererAdapter.java:25)
at hu.daniel.hari.log4jpattern.logrenderer.service.LogRendererServiceImpl.getOutput(LogRendererServiceImpl.java:44)
at hu.daniel.hari.log4jpattern.logrenderer.service.LogRendererServiceImpl.render(LogRendererServiceImpl.java:37)
at hu.daniel.hari.log4jpattern.logrenderer.TestMain.main(TestMain.java:14)
Caused by: java.lang.NullPointerException: no !/ in spec
at sun.net.www.protocol.jar.Handler.parseAbsoluteSpec(Handler.java:171)
at sun.net.www.protocol.jar.Handler.parseURL(Handler.java:151)
at java.net.URL.<init>(URL.java:615)
... 20 more
Since I want to pack the loadable Log4JHack-1.0.jar
to Log4JHackLoader-1.0.jar, I need a solution for loading from inside jar.

It's not 100% clear to me what you're trying to do here. Why are you trying to include conflicting dependencies in your classpath?
In any case, this is a known limitation of UrlClassLoader. Have you considered extracting the nested jar as a temporary file on the file system, and then pointing your class loader to it?

The github.com/squark-io/nested-jar-classloader project is able to load classes in nested jar files, but has a lot of external dependencies. I added this same mechanism in my project (sourceforge.net/projects/mdiutilities/) but in this case without external dependencies.
These two projects do not work by making a temporary copy of the nested jar files, but directly load the classes bytes.

Is there any way to load a component from filesystem in Oracle Commerce(ATG)?

I'm trying to find out if we can load a oracle commerce component from file system. Generally we assemble all the code into an ear file and deploy it, however, I got a requirement where in I have to store some components in file system rather than packaging them along with ear file.
I know that we can use URLClassloader to load a class as shown below,
File classDir = new File("A:\\LodeeModule\\classes");
URL[] url = { classDir.toURI().toURL() };
ClassLoader loader = new URLClassLoader(url);
for (File file : classDir.listFiles()) {
String filename = file.getName().replace(".class", "");
loader.loadClass("com.buddha.testers." + filename).getConstructor().newInstance();
}
but how can we use the same for an component which has to be resolved by Nucleus at later point of time? Is there any way to instruct Nucleus to resolve component from file system?

You should just be able to add the JAR that contains the components classes to the CLASSPATH system variable used by the application server instance.
Then in the component configuration just define the implementing class as you normally would
$class=some.class.path.class
If you are using Jboss EAP 6+ on a newer version of ATG (11.0+) you might have some more trouble, you have to jump through some more hoops due to its classloader
https://docs.jboss.org/author/display/AS7/Class+Loading+in+AS7
Essentially you would need to define a jboss module containing your jar files, and define a dependency between the ear's "module" and the module containing your classes.
Alternatively you can define a ClassLoaderService that will manage the classes for your JARs
To do this, you need to define a new ClassLoaderService, so create a new properties file as you would with any other component.
/my/custom/ClassLoaderService.properties
$class=atg.nucleus.ServicesManifestClassLoaderService
$description=Custom Class Loader Service.
# The files to go into the classpath of the classloader
classpathFiles=\
/path/to/my/jars/lib/someClasses.jar,\
/path/to/my/jars/lib/someOtherClasses.jar
loggingDebug=false
Then in the actual component that you need these classes for add this line;
$classloader=/my/custom/ClassLoaderService

I think you're looking for the atg.dynamo.data-dir property. If you specify that property dynamo will look at that location for the "server configs" or properties files. This allows you to separate the configs from the ear file.
Note: You can still include configs in the ear, I believe they will still have first precedence
It's usually specified when you start the server, something like:
run.sh -c <your server> -Datg.dynamo.data-dir=/data/something/serverconfigs
This feature is largely undocumented, but many people know about it.
See http://docs.oracle.com/cd/E24152_01/Platform.10-1/ATGPlatformProgGuide/html/s0302developmentmodeandstandalonemode01.html
EDIT:
I mistook what you were originally asking. You might want to take a look at the disposable class loader that ATG provides, but keep in mind this is only intended for development purposes.