I want to create a Java program that can be extended with plugins. How can I do that and where should I look for?
I have a set of interfaces that the plugin must implement, and it should be in a jar. The program should watch for new jars in a relative (to the program) folder and registered them somehow.
Although I do like Eclipse RCP, I think it's too much for my simple needs.
Same thing goes for Spring, but since I was going to look at it anyway, I might as well try it.
But still, I'd prefer to find a way to create my own plugin "framework" as simple as possible.
I've done this for software I've written in the past, it's very handy. I did it by first creating an Interface that all my 'plugin' classes needed to implement. I then used the Java ClassLoader to load those classes and create instances of them.
One way you can go about it is this:
File dir = new File("put path to classes you want to load here");
URL loadPath = dir.toURI().toURL();
URL[] classUrl = new URL[]{loadPath};
ClassLoader cl = new URLClassLoader(classUrl);
Class loadedClass = cl.loadClass("classname"); // must be in package.class name format
That has loaded the class, now you need to create an instance of it, assuming the interface name is MyModule:
MyModule modInstance = (MyModule)loadedClass.newInstance();
Look into OSGi.
On one hand, OSGi provides all sorts of infrastructure for managing, starting, and doing lots of other things with modular software components. On the other hand, it could be too heavy-weight for your needs.
Incidentally, Eclipse uses OSGi to manage its plugins.
I recommend that you take a close look at the Java Service Provider (SPI) API. It provides a simple system for finding all of the classes in all Jars on the classpath that expose themselves as implementing a particular service. I've used it in the past with plugin systems with great success.
Although I'll second the accepted solution, if a basic plugin support is needed (which is the case most of the time), there is also the Java Plugin Framework (JPF) which, though lacking proper documentation, is a very neat plugin framework implementation.
It's easily deployable and - when you get through the classloading idiosynchrasies - very easy to develop with. A comment to the above is to be aware that plugin loadpaths below the plugin directory must be named after the full classpath in addition to having its class files deployed in a normal package path named path. E.g.
plugins
`-com.my.package.plugins
`-com
`-my
`-package
`-plugins
|- Class1.class
`- Class2.class
At the home-grown classloader approach:
While its definitely a good way to learn about classloaders there is something called "classloader hell", mostly known by people who wrestled with it when it comes to use in bigger projects. Conflicting classes are easy to introduce and hard to solve.
And there is a good reason why eclipse made the move to OSGi years ago.
So, if its more then a pet project, take a serious look into OSGi. Its worth looking at.
You'll learn about classloaders PLUS an emerging technolgy standard.
Have you considered building on top of Eclipse's Rich Client Platform, and then exposing the Eclipse extension framework?
Also, depending on your needs, the Spring Framework might help with that and other things you might want to do: http://www.springframework.org/
Related
I want to make a java application that supports plug ins. Now my core will use jars for certain processes. If my plug ins where to also use these jars, do the plug ins of my application need to configure their build path to include the jars they would also use or is their a way so that the jars can be imported similar to how I import packages from the main application
Guice and Spring are tools for dependency injection, which means that creating objects is easier with them because they take care of instantiating objects and placing them into other objects that depends on them.
Now, when we talk about plugins, we usually are talking too about dynamically loading new classes into a running app. Think on eclipse IDE. Its architecture was designed from the beginning to be "pluggable", like, you can download jars and eclipse will add them to the running application without the need of application restart.
In this case, if you want to build pluggable apps, in a sense of dynamic classloading, I'd recommend you not to go through this path, but to research subjects such as OSGI. One popular OSGI framework is http://felix.apache.org/
Another approach for application extension (we may call this pluggable too, somehow, I guess), depending on how your app is organized and what it does, is to develop a DSL (http://en.wikipedia.org/wiki/Domain-specific_language) for it and extend it letting people adding scripts to it. Isn't something like this when a browser let you add pieces of funcionality written in javascript? Groovy makes DSL easier in some aspects, for java programmers. (see http://docs.codehaus.org/display/GROOVY/Writing+Domain-Specific+Languages)
If you want dynamic plugable systems OSGI can give you this, but OSGI its IMMO a over-complicated technology, use only if you are really sure that needs this dynamic plug-ability.
Other option for builds extensible systems its use de ServiceProvider mechanism, this is a core java mechanism, for example its the one that JDBC implementations use, you can put a JDBC driver in your classpath and the application can find it and use it without needing that you explicitly import the driver classes in your code.
This is an example of using ServiceProvider in your owns applications: http://docs.oracle.com/javase/tutorial/ext/basics/spi.html#limitations-of-the-service-loader-api
Its of course more limited than OSGI, but its very easy to use when you get the idea, and you don't need any external library because its a java core mechanism.
EDIT: about the libraries.
In runtime: With ServiceProvicer there is no separate classloaders (you can implement off course, but by default, in OSGI its implemented this separation), in runtime if your plugin need X class and this class is in the classpath all is ok, the limitation its that the main application and all the plugins use this version of the dependency (guice 3 for example) and you cannot have one plugin using X version and other plugin using X+2 version if this version are not compatible. (this is the famous hell .jar, and one of the principal motivations behind jigsaw project for example).
In compile time, include the dependency in your pom, ant build file, gradle build file or whatever build system your use as usual.
I'm trying to make a plugin-based application which means, besides the basic framework, other functionalities should be added with a plugin style.
The majar problem of such system, i think, is how to load new plugins at the runtime. I don't want to recomplie the whole project when a new plugin installed. Just like Eclipse, after new installation of plugin, the user only need to restart it.
So my first idea about that is using java reflection to load the class at the runtime. there might be some plugin configuration files, the system reads them and load the plugin classes with reflection. Maybe i can use the spring framework as well, because its Inversion of Control is just match my request.
Another idea i'm investigating is using the ejb container. when the plugins are ejbs, i can just pack them into jar and deploy them in an ejb container, then i can use jndi to access them. but it only works when the plugin are ejbs.
anyway, i'm a rookie in system design. so i post this topic and want to hear of your opinions.
btw. is there any good book about system design you want to recommand?
thanks alot in advance!
You may first look at OSGI and his implementation (Equinox for Eclipse and Apache Felix) but it would be easier if you give us more information of what you are trying to do.
Plug-in to solve what problem?
Designers of frameworks usually have deep domain knowledge that informs their design choices. Frameworks come about after two or three attempts to solve some problem; the earlier implementations give clues about how to abstract what's important into a framework.
Eclipse solves the problem of an expandable IDE (poorly, in my opinion). EJBs are transactional, distributed components that run in a container.
You've given no indication of what kind of problem you want to solve. Until you do, your ideas won't get very far.
What are the best practices for using ServiceLoader in an Environment with multiple ClassLoaders? The documentation recommends to create and save a single service instance at initialization:
private static ServiceLoader<CodecSet> codecSetLoader = ServiceLoader.load(CodecSet.class);
This would initialize the ServiceLoader using the current context classloader. Now suppose this snippet is contained in a class loaded using a shared classloader in a web container and multiple web applications want to define their own service implementations. These would not get picked up in the above code, it might even be possible that the loader gets initialized using the first webapps context classloader and provide the wrong implementation to other users.
Always creating a new ServiceLoader seems wasteful performance wise since it has to enumerate and parse service files each time. Edit: This can even be a big performance problem as shown in this answer regarding java's XPath implementation.
How do other libraries handle this? Do they cache the implementations per classloader, do they reparse their configuration everytime or do they simply ignore this problem and only work for one classloader?
I personally do not like the ServiceLoader under any circumstances. It's slow and needlessly wasteful and there is little you can do to optimize it.
I also find it a bit limited -- you really have to go out of your way if you want to do more than search by type alone.
xbean-finder's ResourceFinder
ResourceFinder is a self-contained java file capable of replacing ServiceLoader usage. Copy/paste reuse is no problem. It's one java file and is ASL 2.0 licensed and available from Apache.
Before our attention spans get too short, here's how it can replace a ServiceLoader
ResourceFinder finder = new ResourceFinder("META-INF/services/");
List<Class<? extends Plugin>> impls = finder.findAllImplementations(Plugin.class);
This will find all of the META-INF/services/org.acme.Plugin implementations in your classpath.
Note it does not actually instantiate all the instances. Pick the one(s) you want and you're one newInstance() call away from having an instance.
Why is this nice?
How hard is it to call newInstance() with proper exception handling? Not hard.
Having the freedom to instantiate only the ones you want is nice.
Now you can support constructor args!
Narrowing search scope
If you want to just check specific URLs you can do so easily:
URL url = new File("some.jar").toURI().toURL();
ResourceFinder finder = new ResourceFinder("META-INF/services/", url);
Here, only the 'some.jar' will be searched on any usage of this ResourceFinder instance.
There's also a convenience class called UrlSet which can make selecting URLs from the classpath very easy.
ClassLoader webAppClassLoader = Thread.currentThread().getContextClassLoader();
UrlSet urlSet = new UrlSet(webAppClassLoader);
urlSet = urlSet.exclude(webAppClassLoader.getParent());
urlSet = urlSet.matching(".*acme-.*.jar");
List<URL> urls = urlSet.getUrls();
Alternate "service" styles
Say you wanted to apply the ServiceLoader type concept to redesign URL handling and find/load the java.net.URLStreamHandler for a specific protocol.
Here's how you might layout the services in your classpath:
META-INF/java.net.URLStreamHandler/foo
META-INF/java.net.URLStreamHandler/bar
META-INF/java.net.URLStreamHandler/baz
Where foo is a plain text file that contains the name of the service implementation just as before. Now say someone creates a foo://... URL. We can find the implementation for that quickly, via:
ResourceFinder finder = new ResourceFinder("META-INF/");
Map<String, Class<? extends URLStreamHandler>> handlers = finder.mapAllImplementations(URLStreamHandler.class);
Class<? extends URLStreamHandler> fooHandler = handlers.get("foo");
Alternate "service" styles 2
Say you wanted to put some configuration information in your service file, so it contains more than just a classname. Here's an alternate style that resolves services to properties files. By convention one key would be the class names and the other keys would be injectable properties.
So here red is a properties file
META-INF/org.acme.Plugin/red
META-INF/org.acme.Plugin/blue
META-INF/org.acme.Plugin/green
You can look things up similarly as before.
ResourceFinder finder = new ResourceFinder("META-INF/");
Map<String,Properties> plugins = finder.mapAllProperties(Plugin.class.getName());
Properties redDefinition = plugins.get("red");
Here's how you could use those properties with xbean-reflect, another little library that can give you framework-free IoC. You just give it the class name and some name value pairs and it will construct and inject.
ObjectRecipe recipe = new ObjectRecipe(redDefinition.remove("className").toString());
recipe.setAllProperties(redDefinition);
Plugin red = (Plugin) recipe.create();
red.start();
Here's how that might look "spelled" out in long form:
ObjectRecipe recipe = new ObjectRecipe("com.example.plugins.RedPlugin");
recipe.setProperty("myDateField","2011-08-29");
recipe.setProperty("myIntField","100");
recipe.setProperty("myBooleanField","true");
recipe.setProperty("myUrlField","http://www.stackoverflow.com");
Plugin red = (Plugin) recipe.create();
red.start();
The xbean-reflect library is a step beyond the built-in JavaBeans API, but a bit better without requiring you to go all the way to a full-on IoC framework like Guice or Spring. It supports factory methods and constructor args and setter/field injection.
Why is the ServiceLoader so limited?
Deprecated code in the JVM damages the Java language itself. Many things are trimmed to the bone before being added to the JVM, because you cannot trim them after. The ServiceLoader is a prime example of that. The API is limited and OpenJDK implementation is somewhere around 500 lines including javadoc.
There's nothing fancy there and replacing it is easy. If it doesn't work for you, don't use it.
Classpath scope
APIs aside, in pure practicality narrowing the scope of the URLs searched is the true solution to this problem. App Servers have quite a lot of URLs all by themselves, not including the jars in your application. Tomcat 7 on OSX for example has about 40~ URLs in the StandardClassLoader alone (this is the parent to all webapp classloaders).
The bigger your app server the longer even a simple search will take.
Caching doesn't help if you intend to search for more than one entry. As well, it can add some bad leaks. Can be a real lose-lose scenario.
Narrow the URLs down to the 5 or 12 that you really care about and you can do all sorts of service loading and never notice the hit.
Have you tried using the two argument version so that you can specify which classloader to use? Ie, java.util.ServiceLoader.load(Class, ClassLoader)
Mu.
In a 1x WebContainer <-> Nx WebApplication system, the ServiceLoader instantiated in the WebContainer will not pick up any classes defined in WebApplications, just those in the container. A ServiceLoader instantiated in a WebApplication will detect classes defined in the application in addition to those defined in the container.
Keep in mind WebApplications will need to be kept separate, are designed that way, things will break if you try and circumvent that, and they are not the method and system available to extend the container - if your library is a simple Jar, just drop it into the appropriate extension folder of the container.
I really like Neil's answer in the link I added in my comment. Due to I have same experences in my recent project.
"Another thing to bear in mind with ServiceLoader is to try to abstract the lookup mechanism. The publish mechanism is quite nice and clean and declarative. But the lookup (via java.util.ServiceLoader) is as ugly as hell, implemented as a classpath scanner that breaks horribly if you put the code into any environment (such as OSGi or Java EE) that does not have global visibility. If your code gets tangled up with that then you'll have a hard time running it on OSGi later. Better to write an abstraction that you can replace when the time comes."
I actually met this problem in OSGi environment actually it's just eclipse in our project. But I luckily fixed it in a timely fashion. My workaround is using one class from the plugin I want to load ,and get classLoader from it. That will be a valid fix. I didn't use the standard ServiceLoader, But my process is quite similiar, use a properties to define the plugin classes I need to load. And I know there is another way to know each plugin's classloader. But at least I don't need to use that.
Honest, I don't like the generics used in ServiceLoader. Because it limited that one ServiceLoader can only handle classes for one interface. Well is it really useful? In my implementation, it don't force you by this limitation. I just use one implementation of loader to load all the plugin classes. I don't see the reason to use two or more. Due to the consumer can know from the config files about the relationships between interfaces and implementation.
This question seem to be more complicated than I first anticipated. As I see
it, there are 3 possible strategies for dealing with ServiceLoaders.
Use a static ServiceLoader instance and only support loading classes from
the same classloader as the one holding the ServiceLoader reference. This
would work when
The service configuration and implementation are in a shared classloader
and all child classloaders are using the same implementation. The example
in the documentation is geared towards theis use case.
Or
Configuration and implementation are put into each child classloader and
deployed along each webapp in WEB-INF/lib.
In this scenario it is not possible to deploy the service in a shared classloader
and let each webapp choose its own service implementation.
Initialize the ServiceLoader on each access passing the context classloader of
the current thread as the second parameter. This approach is taken be the JAXP
and JAXB apis, although they are using their own FactoryFinder implementation
instead of ServiceLoader. So it is possible to bundle a xml parser with a webapp
and have it automatically get picked up for example by DocumentBuilderFactory#newInstance.
This lookup has a performance impact, but in the case of xml parsing
the time to look up the implementation is small compared to the time needed to
actually parse a xml document. In the library I'm envisioning the factory
itself is pretty simple so the lookup time would dominate the performance.
Somehow cache the implementation class with the context classloader as the key.
I'm not entirely sure if this is possible in all the above cases without
causing any memory leaks.
In conclusion, I will probably be ignoring this problem and require that the library
gets deployed inside each webapp, i.e. option 1b above.
I'm trying to make a simple application that loads and runs some classes during runtime. For example, let's say I have this config:
module1.classpath=module1.jar,somelibs1.jar
module1.class=com.blabla.Module1
module2.classpath=module2.jar,somelibs2.jar
module2.class=com.blabla.Module2
Then I need to load libraries specified in module1.classpath and run the module1.class with that libraries loaded. Afterwards I need to load module2.classpath and run module2.class with those libraries.
How do I handle the case when somelibs1.jar and somelibs2.jar have the same classes inside? Basically I'd like to run module1.jar using exclusively somelibs1.jar and module2.jar using exclusively somelibs2.jar. How do I implement that?
I'm guessing I need to create a separate classloader for each of my classes and push the jars in that classloaders. However I'd appreciate some example or at least a confirmation that it is a right way to do that.
This seems to be a pretty good use case for OSGI. I would recommend using OSGI for this as everything you nees is provided by OSGI out-of-box.
But if for some reason you can't use OSGI, then what you need to do is to have a classloader for each module. Load the moduleX.class by a ClassLoaderX, and moduleX.classpath should be added in to ClassLoaderX's path. You can use a set of simple URLClassLoader for this.
Thanks for question. Very interesting.
It seems to you can't use several versions of the same class in one instance of JVM. I've never had this task and I don't know how to implement this.
But let's play. I don't know what is exotic application do you develop. May be you can run many JVMs and each JVM will have exclusive CLASSPATH.
Write application which can run (for example using Runtime.exec()) another JVM and make a conversation to it via some channel (may be network).
I am writing an application that will ship in several different versions (initially around 10 variations of the code base will exist, and will need to be maintained). Of course, 98% or so of the code will be the same amongst the different systems, and it makes sense to keep the code base intact.
My question is - what would be the preferred way to do this? If I for instance have a class (MyClass) that is different in some versions (MyClassDifferent), and that class is referenced at a couple of places. I would like for that reference to change depending on what version of the application I am compiling, rather than having to split all the classes referring to MyClassDifferent too. Preprocessor macros would be nice, but they bloat the code and afaik there are only proof of concept implementations available?
I am considering something like a factory-pattern, coupled with a configuration file for each application. Does anyone have any tips or pointers?
You are on the right track: Factory patterns, configuration etc.
You could also put the system specific features in separate jar files and then you would only need to include the appropriate jar alongside your core jar file.
I'd second your factory approach and you should have a closer look at maven or ant (depending on what you are using).
You can deploy the different configuration files that determine which classes are used based on parameters/profiles.
Preprocessor makros like C/C++ have are not available directly for java. Although maybe it's possible to emulate this via build scripts. But I'd not go down that road. My suggestion is stick with the factory approach.
fortunately you have several options
1) ServiceLoader (builtin in java6) put your API class like MyClass in a jar, the compile your application against this API. Then put a separate implementation of MyClass in a separate jar with /META-INF/services/com.foo.MyClass. . Then you can maintain several version of your application simply keeping a "distribution" of jars. Your "main" class is just a bunch of ServiceLoader calls
2) same architecture of 1) but replacing META-INF services with Spring or Guice config
3) OSGI
4) your solution
Look up the AbstractFactory design pattern, "Dependency Injection", and "Inversion of Control". Martin Fowler writes about these here.
Briefly, you ship JAR files with all the needed components. For each service point that can be customized, you define an Interface for the service. Then you write one or more implementations of that Interface. To create a service object, you ask an AbstractFactory for it, eg:
AbstractFactory factory = new AbstractFactory();
...
ServiceXYZ s = factory.newServiceXYZ();
s.doThis();
s.doThat();
Inside your AbstractFactory you construct the appropriate ServiceXYZ object using the Java reflection method Class.classForName(), and SomeClassObject.newInstance(). (Doing it this way means you don't have to have the ServiceXYZ class in the jar files unless it makes sense. You can also build the objects normally.)
The actual class names are read in from a properties file unique to each site.
You can roll your own solution easily enough, or use a framework like Spring, Guice, or Pico.