We have a plugin system with multiple module layers:
The PluginLoader module inside the boot layer instantiates a custom ModuleLayer for each Module at runtime with ModuleLayer::defineModulesWithOneLoader.
We now have a set of non-modular plugins that need access to the modular plugins inside the custom layers. They will be loaded after the custom layers have been instantiated. How can this be achieved?
Simply putting them on the classpath doesn't work, since the modular classes will be loaded a second time instead of using the (already loaded) classes from the modular plugins.
Our next try was creating a dummy plugin and loading it last inside a layer that has all other layers (and therefore modular plugins) as parents. It doesn't seem to be possible to add URLs to the class loader generated by the defineModulesWithOneLoader method. Instead we created a custom URLClassLoader as a parent of this class loader and added the non-modular JARs via URLClassLoader::addURL.
This works for the non-modular plugin classes. However, if a class already loaded in one of the modular plugins is referenced from such a class, it will also attempt to load the class with our custom URLClassLoader (which fails, since the parent is null) instead of the module's internal class loader, which delegates to parent layers. Is there a way to load our non-modular plugin's classes inside the unnamed module of a class loader that behaves like the JPMS' default class loader (search parent layers first) but still loads the classes inside the non-modular JAR, if they aren't found in the parent layers?
It is possible to solve this by using automatic modules, but we want to avoid this for now since it leads to a slew of other necessary changes (e.g. resolve split packages) which we currently don't have the resources for.
Related
Both Sealed Packages/Jars and the Java Module System disallow spliting packages across several jars.
Does that mean that all packages contained within a Module are implicitly sealed? If not what does explicitly sealing the jar change?
Yes, packages within modules are always implicitly sealed. This is specified in the documentation of the Package class:
A Package automatically defined for classes in a named module has the following properties:
The name of the package is derived from the binary names of the classes. Since classes in a named module must be in a named package, the derived name is never empty.
The package is sealed with the module location as the code source, if known.
The specification and implementation titles, versions, and vendors are unspecified.
Any annotations on the package are read from package-info.class as specified above.
I also made a quick test to verify that isSealed() on a module’s package did indeed return true. However, it must be noted that the relationship between (named) modules and packages is of a fundamental nature and hence, independent of the fact that isSealed() returns true. The latter is just the natural way for the old API to interact with this new feature.
A sealed package of an unnamed module only affects the runtime package of the particular class loader, as each class loader can have a package of the same name, which is considered a different runtime package.
In contrast, each package of a named module must unambiguously belong to a single module within the entire module layer and a module layer can span multiple class loaders, e.g. the boot layer spans the bootstrap loader, the platform loader, and the application loader.
This affects the class loading process. The old way of loading, which is still used for the unnamed module, features class loader delegation where each loader queries its parent loader first. When the parent loader failed, the class path entries are queried linearly for the class until a class is found.
When a module layer is initialized, all package names and their owning modules are recorded, so when an attempt to load a class is made, the package name can be derived from the qualified name and mapped to a module even before the class is loaded and therefore, only the module’s known source needs to be queried.
In a module structure like this:
project
|
|- common module
|- app module
Where app module has common module as a dependency, I have a custom classloader class defined in the common module. The app module has a -Djava.system.class.loader=org.project.common.CustomClassLoader jvm parameter set to use that custom classloader defined in common module.
Running a spring boot project within IDEA this works perfectly. The custom classloader is found, set as a system classloader and everything works.
Compiling a runnable jar (using default spring-boot-maven-plugin without any custom properties), the jar itself has all the classes and within it's lib directory is the common jar which has the custom classloader. However running the jar with the -Djava.system.class.loader=org.project.common.CustomClassLoader results in the following exception
java.lang.Error: org.project.common.CustomClassLoader
at java.lang.ClassLoader.initSystemClassLoader(java.base#12.0.2/ClassLoader.java:1989)
at java.lang.System.initPhase3(java.base#12.0.2/System.java:2132)
Caused by: java.lang.ClassNotFoundException: org.project.common.CustomClassLoader
at jdk.internal.loader.BuiltinClassLoader.loadClass(java.base#12.0.2/BuiltinClassLoader.java:583)
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(java.base#12.0.2/ClassLoaders.java:178)
at java.lang.ClassLoader.loadClass(java.base#12.0.2/ClassLoader.java:521)
at java.lang.Class.forName0(java.base#12.0.2/Native Method)
at java.lang.Class.forName(java.base#12.0.2/Class.java:415)
at java.lang.ClassLoader.initSystemClassLoader(java.base#12.0.2/ClassLoader.java:1975)
at java.lang.System.initPhase3(java.base#12.0.2/System.java:2132)
Why does this happen? Is it because in the runnable jar the classloader class is in a jar in lib directory so the classloader is trying to get set before the lib classes were added to the classpath? Is there anything I can do besides moving the classloader from common to all the other modules that need it?
EDIT: I've tried moving the custom classloader class from common module to app but I am still getting the same error. What is going on here?
Running a spring boot project within IDEA this works perfectly. The custom classloader is found, set as a system classloader and everything works.
Because IDEA puts your modules on the class path and one of them contains the custom class loader.
Is it because in the runnable jar the classloader class is in a jar in lib directory so the classloader is trying to get set before the lib classes were added to the classpath?
Kind of. The lib classes are not "added to the class path", but the runnable Spring Boot app's own custom class loader knows where to find and how to load them.
For a deeper understanding of java.system.class.loader, please read the Javadoc for ClassLoader.getSystemClassLoader() (slightly reformatted with added enumeration):
If the system property java.system.class.loader is defined when this method is first invoked then the value of that property is taken to be the name of a class that will be returned as the system class loader.
The class is loaded using the default system class loader and must define a public constructor that takes a single parameter of type ClassLoader which is used as the delegation parent.
An instance is then created using this constructor with the default system class loader as the parameter.
The resulting class loader is defined to be the system class loader.
During construction, the class loader should take great care to avoid calling getSystemClassLoader(). If circular initialization of the system class loader is detected then an IllegalStateException is thrown.
The decisive factor here is #3: The user-defined system class loader is loaded by the default system class loader. The latter of course has no clue about how to load something from a nested JAR. Only later, after the JVM is fully initialised and Spring Boot's special application class loader kicks in, can those nested JARs be read.
I.e. you are having a chicken vs. egg problem here: In order to find your custom class loader during JVM initialisation, you would need to use the Spring Boot runnable JAR class loader which has not been initialised yet.
If you want to know how what the Javadoc above describes is done in practice, take a look at the OpenJDK source code of ClassLoader.initSystemClassLoader().
Is there anything I can do besides moving the classloader from common to all the other modules that need it?
Even that would not help if you insist in using the runnable JAR. What you could do is either of these:
Run your application without zipping it up into a runnable JAR, but as a normal Java application with all application modules (especially the one containing the custom class loader) on the class path.
Extract your custom class loader into a separate module outside of the runnable JAR and put it on the class path when running the runnable JAR.
Set your custom class loader via Thread.setContextClassLoader() or so instead of trying to use it as a system class loader, if that would be a viable option.
Update 2020-10-28: In the document "The Executable Jar Format" I found this under "Executable Jar Restrictions":
System classLoader: Launched applications should use Thread.getContextClassLoader() when loading classes (most libraries and frameworks do so by default). Trying to load nested jar classes with ClassLoader.getSystemClassLoader() fails. java.util.Logging always uses the system classloader. For this reason, you should consider a different logging implementation.
This confirms what I wrote above, especially my last bullet point about using the thread context class loader.
Assuming you want to add custom jar to the classpath with Spring, do the following:
Generate the jar file with the maven jar plugin
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>libs/</classpathPrefix>
<mainClass>
com.demo.DemoApplication
</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
While running the application from the command line, use the below command
java -cp target/demo-0.0.1-SNAPSHOT.jar -Dloader.path=<Path to the Custom Jar file> org.springframework.boot.loader.PropertiesLauncher
This should launch your app while loading the Custom Classloader as well
In short, the trick is, to use the -Dloader.path along with org.springframework.boot.loader.PropertiesLauncher
An application launched on the lines of -
java -cp ./lib/* com.example.Main
would ideally be sufficient.
Will need some clarity on how the application is being used.
Is the main class itself being attempted to be launched from a custom class loader (assuming its possible to do so) or whether post launch specific application related classes are required to be loaded with a custom class-loader (and associated privileges)?
Have asked those questions in the comments above (planning to update the answers here once have more clarity).
PS: Haven't really factored the use of 'modules' yet but believe the above syntax would still hold for the newer jdk's (after jdk 8).
I have two JPMS layers:
Boot layer with module A loaded by ClassLoaders$AppClassLoader#4fca772d
Child layer with module B that provides cervices and loaded by Loader#6b58b9e9
The parent classloader of Loader#6b58b9e9 is ClassLoaders$AppClassLoader#4fca772d.
In module A I have the following code:
ServiceLoader<ModuleAInterface> sl = ServiceLoader.load(ModuleAInterface.class);
However, the services of Module B are found only when context class loader is Loader#6b58b9e9 and not found when context class loader is ClassLoaders$AppClassLoader#4fca772d.
The question - is it possible to get services of module B in module A without knowing class loader of module B in such configuration.
looking at the code of java.util.ServiceLoader in jdk 14 (see screenshot) it looks like it follows the same logic as class loading when there are multiple ModuleLayer, as described in this stackoverflow answer
which means that ServiceLoader will will first look at services in its own ModuleLayer then in its parent ModuleLayer and continue from child to parent in a recursive manner
is it possible to get services of module B in module A without knowing class loader of module B in such configuration.
no
but module B can see the services in module A
I am trying to understand how JPMS works.
From here
The classpath is not completely gone yet. All JARs (modular or not)
and classes on the classpath will be contained in the Unnamed Module.
Similar to automatic modules, it exports all packages and reads all
other modules. But it does not have a name, obviously. For that
reason, it cannot be required and read by named application modules.
The unnamed module in turn can access all other modules.
Please, note ...on the classpath will be contained in the Unnamed Module. Module is singular.
From here
For compatibility, all code on the classpath is packaged up as a
special unnamed module, with no hidden packages and full access to the
whole JDK.
Again unnamed module. Module is singular.
Do I understand right that there is always only one unnamed module in JPMS? Does it mean that applications that were developed before Java9 and not updated for Java9 will be loaded as one unnamed module?
Do I understand right that there is always only one unnamed module in JPMS?
In short
Generally speaking, no. But let's put it this way: If you place some or even all JARs on the class path and your application does not create class loaders to load any additional content, then there is only one unnamed module you need to care about.
In more detail
Every ClassLoader has its own unnamed module that it uses to represent classes that it loaded from the class path. This is necessary because the module system requires everything to be in a module.
As nullpointer's answer explains in detail, an application will by default use three separate class loaders. It is possible that it might spin up its own class loaders, for example to load plugins. If it doesn't do that, though, all application code will end up in the system/application class loader and hence in the same unnamed module. That's why there is typically only one you need to care about.
Does it mean that applications that were developed before Java9 and not updated for Java9 will be loaded as one unnamed module?
This has nothing to do with whether code (application, frameworks, libraries) targets Java 9 - it only depends on which path you place a JAR, on the class path or the module path.
If it's on the class path, it ends up in the unnamed module together with other class path content. This is true for plain JARs without module descriptor but also for modular JARs that contain one.
If it's on the module path, it gets its own module. If it's a modular JAR, it gets an explicit module as those described throughout the State of the Module System - plain JARs get turned into automatic modules (note the plural: one automatic module per JAR).
Do I understand right that there is always only one unnamed module in JPMS?
Yes, there is one unnamed module. The unnamed module is very similar to the existing concept of the unnamed package.
In implementations of the Java SE platform that use a hierarchical file system for storing packages, one typical strategy is to associate an unnamed package with each directory; only one unnamed package is observable at a time, namely the one that is associated with the "current working directory". The precise meaning of "current working directory" depends on the host system.
Does it mean that applications that were developed before Java9 and not updated for Java9 will be loaded as one unnamed module?
Yes, for those jars placed on the classpath would be treated as a single unnamed module. The bottom up migration with the concept of unnamed modules illustrates this with a similar example as:
Suppose, e.g., that the application shown above had originally been
built for Java SE 8, as a set of similarly-named JAR files placed on
the class path. If we run it as-is on Java SE 9 then the types in the
JAR files will be defined in the unnamed module.
The actual question that can arise here is
Which class loader is the unnamed module associated?
The State of Module System about unnamed module states a clarification instead about this.
Every class loader, it turns out, has its own unique unnamed module,
which is returned by the new ClassLoader::getUnnamedModule method.
If a class loader loads a type that is not defined in a named module then
that type is considered to be in that loader’s unnamed module, i.e.,
the getModule method of the type’s Class object will return its
loader’s unnamed module. The module colloquially referred to as “the
unnamed module” is, then, simply the unnamed module of the application
class loader, which loads types from the classpath when they are in
packages not defined by any known module.
The ClassLoader as revised in Java-9 states that:
The Java run-time has the following built-in class loaders:
Bootstrap class loader: The virtual machine's built-in class loader...
Platform class loader: ...
To allow for upgrading/overriding of modules defined to the platform
class loader, and where upgraded modules read modules defined to class
loaders other than the platform class loader and its ancestors, then
the platform class loader may have to delegate to other class loaders,
the application class loader for example. In other words, classes in
named modules defined to class loaders other than the platform class
loader and its ancestors may be visible to the platform class loader.
System class loader: It is also known as application class loader and is distinct from the platform class loader. The system
class loader is typically used to define classes on the application
class path, module path, and JDK-specific tools. The platform class
loader is a parent or an ancestor of the system class loader that all
platform classes are visible to it.
I have a problem with this linkage error, we have projects with the same code and works :
this.plateformTransactionManager = new DataSourceTransactionManager();
this.plateformTransactionManager.setDataSource(dataSource);
DefaultTransactionDefinition def = new DefaultTransactionDefinition();
def.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRED);
TransactionStatus status = plateformTransactionManager.getTransaction(def);
Stack trace at runtime :
java.lang.LinkageError: loader constraint violation: when resolving method org.springframework.jdbc.datasource.DataSourceTransactionManager.getTransaction(Lorg/springframework/transaction/TransactionDefinition;) Lorg/springframework/transaction/TransactionStatus;" the class loader (instance of org/apache/catalina/loader/WebappClassLoader) of the current class, com/as24/referentiel/daos/ExternalUserDAO, and the class loader (instance of rg/apache/catalina/loader/StandardClassLoader) for resolved class, org/springframework/jdbc/datasource/DataSourceTransactionManager, have different Class Objects for the type org/springframework/transaction/TransactionDefinition used in the Signature
I read it's a maven dependancies error but we used Spring only on version 3.0.7.REALEASE
I don't find any soutions on the web ...
This is a problem of shared libraries between between server and web application.
In java a class is identified by its name (including package name) and its class loader (which loaded it). So if you have the same MyBean.class inside web application WEB-INF/classes and also inside ${catalina.home}/lib, these will be viewed as two different classes (i.e. myBean instanceof MyBean == false). When JVM approaches situation where some class is using class XYZ in its method signatures and XYZ is loaded by a different classloader than a class XYZ known by the current class loader, then LinkageError is raised.
You problem is that you have Spring dependencies (at least spring-tx) on the shared / system class loader and also within web application dependencies.
To solve your issue you need to do one of the following:
remove Spring dependencies from server class-loaders
remove duplicate dependency from the web application (mark it as provided dependency in POM)
add duplicate dependencies on endorsed class-loader (by that web application class-loader will ignore the web application dependency).