We Keep Coding

How to add external JARs to Spring application without restarting JVM?

I have a Spring Boot application which copies external JAR files to a folder, depending on certain conditions. These JARs can contain many Spring components (i.e. classes annotated or meta-annotated with #Component) and the Spring application should be able scan and instantiate for these beans. Is it possible, based on certain conditions, to dynamically load the contents of the JAR files and make them available to the Spring application context? I am fully aware of the security implications this has.
I have read about the different types of Launchers which Spring provides for its executable JAR format, such as JarLauncher and PropertiesLauncher, but it looks like that these launchers do not detect changes to the classpath, but instead only scan the directories once for JAR files.
The following simple application demonstrates the problem:
// .../Application.java
#SpringBootApplication
public class Application {
public static void main(String[] args) {
System.out.println("Please copy JAR files and press Enter ...");
System.in.read();
SpringApplication.run(Application.class, args);
}
}
Replace the default JarLauncher with PropertiesLauncher:
// build.gradle
tasks.named('bootJar') {
manifest {
attributes 'Main-Class': 'org.springframework.boot.loader.PropertiesLauncher',
'Start-Class': 'com.example.customlauncher.Application'
}
}
Specify the location to the external JARs in the properties file of the PropertiesLauncher:
# .../resources/loader.properties
loader.path=file:/path/to/dir
The application is a Spring Initializer Gradle application and packaged by running the bootJar task: ./gradlew bootJar.
It is then started with the following command:
java -jar build/libs/customlauncher-0.0.1-SNAPSHOT.jar
This works if the JAR file is already present at the specified location (/path/to/dir), but it does not work if the java command is executed while the directory is empty and the JAR file is then copied while the app waits for the user to copy the files and press Enter ↲.
There are a couple of related questions, but it looks like they all assume that the JAR files already exist at the time of starting the JVM:
How to put a directory first on the classpath with Spring Boot?
Spring Boot Executable Jar with Classpath
SpringBoot external jar not load
Is there a way to achieve this without too many awkard hacks? Or is recommended to utilize something like OSGi? Am I looking at this completely wrong and there is a better way to have JARs on the classpath that do not need always need loading (if the JAR is "disabled", it should not be loaded/compiled by the JVM, should not be picked up by Spring, etc.)?

It looks like this is possible if the JAR files are copied before starting the Spring application. It feels hackish, but it works. Use at your own risk!
You need two classes, one for bootstrapping the external JARs, which will then start the second via a manually created PropertiesLauncher. The bootstrapping class can be a plain old regular Java class (but it can be a Spring Boot Application too) and only the second class needs to be a SpringBootApplication.
// BootstrapApplication.java
public class BootstrapApplication {
public static void main(String[] args) {
System.out.println("Please copy JAR files and press Enter ...");
System.in.read();
PropertiesLauncher.main(args);
}
}
// Application.java
#SpringBootApplication
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}
In the gradle file, we can switch back to the default JarLauncher, by removing the bootJar task manifest configuration and applying settings via the springBoot configuration block. mainClass will end up as Start-Class in the MANIFEST.MF file.
// build.gradle
springBoot {
mainClass = 'com.example.customlauncher.BootstrapApplication'
}
In the properties file for the loader, a new property needs to be set, which points to the real application class. The settings in this file are only picked up by PropertiesLauncher and ignored by JarLauncher. In other words: JarLauncher delegates to Start-Class from the manifest file and PropertiesLauncher delegates to loader.main from its properties file.
# .../resources/loader.properties
loader.path=file:/path/to/dir
loader.main=com.example.customlauncher.Application
Spring (Boot) will first call the main method of BootstrapApplication, as specified in the MANIFEST.MF file (controlled via springBoot configuration block in the build.gradle file). In the implementation of this main method, a new PropertiesLauncher is created with the main class set to the "real" application (i.e. Application).
Executing the application is still done via the same invocation:
java -jar build/libs/customlauncher-0.0.1-SNAPSHOT.jar
Any JAR files added to /path/to/dir after the JVM has started, but before calling PropertiesLauncher#main in BootstrapApplication are then available in the classpath and application context as seen from Application.

Kafka Connect cannot cast custom storage sink partitioner to Partitioner interface

I need to create a custom partitioner for the kafka connect S3 sink plugin.
I've extended the HourlyPartitioner in a custom class using kotlin:
class RawDumpHourlyPartitioner<T> : HourlyPartitioner<T>() {
...
}
and changed my connector config accordingly to use the custom class:
"partitioner.class": "co.myapp.RawDumpHourlyPartitioner",
I've then created our jar (we use shadow) and included it into a custom docker image based on the kafka connect image (the image version is the same as the dependencies we use in the project):
FROM gradle:6.0-jdk8 as builder
WORKDIR /app
ADD . .
RUN gradle clean shadowJar
FROM confluentinc/cp-kafka-connect:5.3.2
COPY --from=builder /app/build/libs/kafka-processor-0.1-all.jar /usr/share/java/kafka/kafka-processor.jar
When the connector starts I get this error:
ERROR WorkerSinkTask{id=staging-raw-dump-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
java.lang.ClassCastException: co.myapp.RawDumpHourlyPartitioner cannot be cast to io.confluent.connect.storage.partitioner.Partitioner
To double check I've created a java file that tries to instantiate the class and it didn't throw any error:
import io.confluent.connect.storage.partitioner.Partitioner;
public class InstantiateTest {
public static void main(String[] args) throws ClassNotFoundException, IllegalAccessException, InstantiationException {
Class<? extends Partitioner<?>> partitionerClass =
(Class<? extends Partitioner<?>>) Class.forName("co.myapp.RawDumpHourlyPartitioner");
Partitioner<?> partitioner = partitionerClass.newInstance();
}
}

Looking at the kafka connect guide it says:
A Kafka Connect plugin is simply a set of JAR files where Kafka Connect can find an implementation of one or more connectors, transforms, and/or converters. Kafka Connect isolates each plugin from one another so that libraries in one plugin are not affected by the libraries in any other plugins. This is very important when mixing and matching connectors from multiple providers.
This means that since I'm using the S3 sink connector, I have to put my jar with the custom partitioner in the directory of the S3 plugin.
Moving the jar file to /usr/share/java/kafka-connect-s3 solved the issue
In the comments I've mentioned that my jar also includes a custom subject name strategy that we use in the main kafka-connect config (the env variables), in that case the jar needs to be in the /usr/share/java/kafka folder
Update: as cricket_007 mentioned it's better to put the custom partitioner jar into the /usr/share/java/kafka-connect-storage-common folder which is where all the other partitioners are

Depending on which Sink you use, We need to push partitioner class there, as in our case when we were using Confluent Kafka 5.5 , and connector class Azure Gen2 Storage.
For that we need to write custom partitioner similar to following Repo in Github.
Then We place the custom JAR in following path:
/usr/share/confluent-hub-components/confluentinc-kafka-connect-azure-data-lake-gen2-storage/lib/
After which our connector class working successfully!

Launch Spring Boot WicketWebApplication inside another java Application without main

i want to use Spring Boot with Spring Security and Apache Wicket 8.0 as web view.
It`s easy to do it with SpringBoot and just simply run jar file, but i want to do something like this:
Minecraft Server (Spigot, Bukkit, etc..) Scanning /plugins/ folder, and looking for .jar files.
All files must contain plugin.yml in root of .jar archive. This plugin.yml file contains a path to class file, for example - BukkitMain and this class MUST extend JavaPlugin.
Then MinecraftServer core is executing onEnable() method of BukkitMain.class (so, if we override this method - server code will execute all code inside this method)
For Example:
public class BukkitMain extends JavaPlugin {
#Override
public void onEnable(){
//SpringApplicationBuilder.run(....);
}
}
So, i can use Spring-boot-plugin in build part of maven .pom file, but it's packing all class files and resources to BOOT-INF inside jar and it's requiring main(String args[]) method to run SpringApplication. It`s bad variant, because i can't access my BukkitMain to run it.
When i use Maven Compiler Plugin and packing it to jar - it's all ok, but there is new problem. My BukkitMain trying to run SpringApplicationBuilder, but i'm getting ClassNotFoundException, cause there is no SpringApplicationBuilder inside jar.

FileNotFoundException in test case run with maven using Spring resource loader

Our program is sending mails. For sending mails it includes an attachment.
The project is set up with maven. The JUnit test case loads the spring configuration first. Spring is loading the File with its DefaultResourceLoader. When running this directly within the workspace its running fine, however as a maven test it fails.
Spring Configuration for resource loader:
<property name="defaultResourceLoader">
<bean class="org.springframework.core.io.DefaultResourceLoader" />
</property>
ImplementingClass:
#Service
public class SomeClassA {
#Autowired
private DefaultResourceLoader defaultResourceLoader;
#Value("${mailLogoPath}")
private String mailLogo;
public void someMethod(){
String filePath = defaultResourceLoader.getResource(mailLogo).getFile().getAbsolutePath();
}
}
The exception points to the actual problem and difference:
java.io.FileNotFoundException: class path resource [templates/efmaillogo.jpg] cannot be resolved to absolute file path because it does not reside in the file system: jar:file:/C:/IBM/workspace/efMy10/ef-mvxsrv-reactor/ef-mvxsrv-service-resources/target/ef-mvxsrv-service-resources-1.5.23-SNAPSHOT.jar!/templates/efmaillogo.jpg
at org.springframework.util.ResourceUtils.getFile(ResourceUtils.java:204)
at org.springframework.core.io.AbstractFileResolvingResource.getFile(AbstractFileResolvingResource.java:52)
at SomeClassA.someMethod(SomeClassA.java:9)
Spring tries to look the file up within the jar. When running it as a junit test within eclipse, everything is working, because spring finds the file directly within the workspace. However when running the same unit test with maven it fails, because it doesn't find the file, because its in a jar in a different module. The file in the other module is needed by different modules, so I don't want to move it. On the application server where everything is deployed its working as well, because the .ear file is expanded.
I'm wondering if there is a different way to access the file with Spring, so that I don't have to skip this test case with maven.
Update
Tried doing a lookup for the file with:
String filePath = defaultResourceLoader.getResource(mailLogo).getURL.getFile();
However it now fails when actually sending the mail with Transport.send(msg).
java.io.FileNotFoundException: file:\C:\Users\uhe.m2\repository\com\clavisit\ef\mvxsrv\ef-mvxsrv-service-resources\1.5.23-SNAPSHOT\ef-mvxsrv-service-resources-1.5.23-SNAPSHOT.jar!\templates\efmaillogo.jpg (The filename, directory name, or volume label syntax is incorrect.)
at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:1141)
at javax.mail.Transport.send0(Transport.java:195)
at javax.mail.Transport.send(Transport.java:124)
at com.clavisit.ef.ep.service.integration.handler.mail.SendMail.sendMessage(SendMail.java:153)

String filePath = defaultResourceLoader.getResource(mailLogo).getURL().getFile();
The above code change must work as per the discussion in this thread http://www.coderanch.com/t/474047/Spring/Spring-cannot-find-file-classpath

Spring Dependency Injection and Plugin Jar

I have web application running with a default impl of a backend service. One should be able to implement the interface and drop the jar into the plugins folder (which is not in the apps classpath). Once the server is restarted, the idea is to load the new jar into the classloader, and have it take part in dependency injection. I am using Spring DI using #Autowired. The new plugin service impl will have #Primary annotation. So given two impls of the interface, the primary should be loaded.
I got the jar loaded into the classloader and can invoke the impl manually. But I haven't been able to get to to participate in the Dependency Injection, and have it replace the default impl.
Here's a simplified example:
#Controller
public class MyController {
#Autowired
Service service;
}
//default.jar
#Service
DefaultService implements Service {
public void print() {
System.out.println("printing DefaultService.print()");
}
}
//plugin.jar not in classpath yet
#Service
#Primary
MyNewService implements Service {
public void print() {
System.out.println("printing MyNewService.print()");
}
}
//For lack of better place, I loaded the plugin jar from the ContextListener
public class PluginContextLoaderListener extends org.springframework.web.context.ContextLoaderListener {
#Override
protected void customizeContext(ServletContext servletContext,
ConfigurableWebApplicationContext wac) {
System.out.println("Init Plugin");
PluginManager pluginManager = PluginManagerFactory.createPluginManager("plugins");
pluginManager.init();
//Prints the MyNewService.print() method
Service service = (Service) pluginManager.getService("service");
service.print();
}
}
<listener>
<listener-class>com.plugin.PluginContextLoaderListener</listener-class>
</listener>
Even after I have loaded the jar into the classloader, DefaultService is still being injected as service. Any idea how I get the plugin jar to participate into the spring's DI lifecycle?
Edited:
To put it simply, I have a war file that has a few plugin jars in a plugins directory inside the war. Based on a value from a configuration file that the app looks at, when the app is started, I want to load that particular plugin jar and run the application with it. That way, I can distribute the war to anyone, and they can choose which plugin to run based on a config value without having to to repackage everything. This is the problem I am trying to solve.

It seems like all You need is to create the Spring ApplicationContext properly. I think it's possible without classpath mingling. What matters most are the locations of the Spring configuration files within the classpath. So put all Your plugin jar's into WEB-INF/lib and read on.
Let's start with the core module. We'll make it to create it's ApplicationContext from files located at classpath*:META-INF/spring/*-corecontext.xml.
Now we'll make all plugins to have their config files elsewhere. I.e. 'myplugin1' will have its config location like this: classpath*:META-INF/spring/*-myplugin1context.xml. And anotherplugin will have the configs at classpath*:META-INF/spring/*-anotherplugincontext.xml.
What You see is a convension. You can also use subdirectiries if You like:
core: classpath*:META-INF/spring/core/*.xml
myplugin1: classpath*:META-INF/spring/myplugin1/*.xml
anotherplugin: classpath*:META-INF/spring/anotherplugin/*.xml
What matters is that the locations have to be disjoint.
All that remains is to pass the right locations to the ApplicationContext creator. For web applications the right place for this would be to extend the ContextLoaderListener and override the method customizeContext(ServletContext, ConfigurableWebApplicationContext).
All that remains is to read Your config file (its location can be passed as servlet init parameter). Than You need to construct the list of config locations:
String locationPrefix = "classpath*:META-INF/spring/";
String locationSiffix = "/*.xml";
List<String> configLocations = new ArrayList<String>();
configLocations.add(locationPrefix + "core" + locationSiffix);
List<String> pluginsTurnedOn = getPluginsTurnedOnFromConfiguration();
for (String pluginName : pluginsTurnedOn) {
configLocations.add(locationPrefix + pluginName + locationSiffix);
}
applicationContext.setConfigLocations(configLocations.toArray(new String[configLocations.size()]));
This way You can easily manage what is and what is not loaded into Spring ApplicationContext.
Update:
To make it work there's one more hidden assumption I made that I'm about to explain now. The base package of the core module and each plugin should also be disjoint. That is i.e.:
com.mycompany.myapp.core
com.mycompany.myapp.myplugin1
com.mycompany.myapp.anotherplugin
This way each module can use <context:componet-scan /> (on equivalent in JavaConfig) easily to add classpath scanning for it's own classes only. The core module should not contain any package scanning of any plugin packages. The plugins should extend configuration of ApplicationContext to add their own packages to classpath scanning.

If you restart the server, I see no reason why you can't just add the JAR to the WEB-INF/lib and have it in the CLASSPATH. All the complication of a custom class loader and context listener goes away, because you treat it just like any other class under Spring's control.
If you do it this way because you don't want to open or modify a WAR, why not put it in the server /lib directory? Let the server class loader pick it up. This makes all plugin classes available to all deployed apps.
The answer depends on how important the separate /plugin directory is. If it's key to the solution, and you can't add the JAR to the server's /lib directory, then that's that. I've got nothing. But I think it'd be worthwhile to at least revisit the solution you have to make sure that it's the only way to accomplish what you want.