We Keep Coding

How to handle version conflicts with my Java application using SPI extensions

I am writing a plugin API for a Java application, the idea being that eventually third parties will provide their own plugin extensions for the application and all the user needs to do is place the plugin jar into a plugins directory of the application. For the impatient, my question in short is how to handle possible version conflicts when the plugin relates to a different API than that on the system, see below for details of my situation and what I have thought about.
I have read a number of articles using service provider interfaces and have something working with that. The problem comes when considering how to deal with differing version combinations.
I am aware of the technique of when adding to an API adding extension interfaces, rather than changing the existing interface (eg. API 1.0 having MyInterface, API 1.1 adding MyInterface2 with the new methods, etc). With this technique if the user has the latest API then older plugins should work fine, but what happens if the user has an old API and newer plugins?
So as an example the user has API 1.0 only with MyInterface but installs binary plugin compiled against API 1.1 where the provider class implements MyInterface2. Whilst the application may only ever call plugins using MyInterface, what happens if the plugin internally calls MyInterface2? Will this cause an error or exception and when (IE. when the class is loaded or when the method from MyInterface2 is called). Also is this standard across JVMs or may it differ depending on the JVM used?
Finally, would it be better to use a plugin framework, would that be able to check version requirements? Searching the internet I find PF4J on github. A quick look in the source code seems to show it may support some sort of version checks.

Dynamic class loading in OSGi

I have a whole bunch of framework modules that work fine on OSGi, all the services and components are finding one another and running just fine.
There is however one framework that does some dynamic stuff regarding classes. Basically at some point you give it a class name and it performs Class.forName() and then reflection magic happens.
This works great when running in a standard jvm and using SPI to wire together the frameworks but it fails in OSGi because of course that random class "test.MyTest" that you are trying to approach via the framework is not visible to said framework.
It will throw a "java.lang.ClassNotFoundException: test.MyTest not found by framework"
So my question: how can I solve this lack of visibility for the framework that needs to see all? Import-Package: *?
UPDATE
Assuming OSGi hasn't changed much since 2010 on this front, the article http://njbartlett.name/2010/08/30/osgi-readiness-loading-classes.html is very interesting. I have currently added support for both actively registering classes and a domain factory to be injected via OSGi.
Apart from that the default resolving uses context classloader anyway so if all else fails that will be used to try and load the class.
UPDATE
I have added support for the suggested DynamicImport-Package as well which is easier for small projects.

You can use DynamicImport-Package:*. This will allow the bundle to see all classes. The problem is that you have no real control over what exactly is exposed. So this is normally a last resort and not the recommended way.
You should first try to use Thread.currentThread().setContextClassLoader() and set it to the classloader of the class you provide to the framework. Sometimes the frameworks also consult this classloader.
The even better way is to find a method in the framework that allows to provide the user classloader.
If you have control over the code then avoid Class.forName(). Instead let the user either give you a class object instead of a class name or let the user give you the combination of a class name and the classloader to use. Both ways work perfectly in and outside OSGi.

How does Java find a specification implementation?

I don't understand how java knows an implementation of any spec..
For example, I have a Spring App with JSF, if I place the Mojarra jar on classpath the app works, but I have not done nothing but that add it to classpath, all without configuration
If I take out the Mojarra jar, then my app doesn't works.
Even, How it can differenciate between Mojarra or MyFaces without any change?
I just want to know how Java finds the implementations, not why it doesn't work if I take out the jars ;)

Usually the Java spec classes are divided in two parts:
An API part(jar) that contains mainly interfaces usually provided by the spec guys.
An implementation part(jar) that contain the implementations of these interfaces.
An example of the above is JDBC: javax.sql.* classes are provided by sun/oracle, but the implementation is done in the JDBC drivers provided by each db vendor. In this case some kind of configuration is needed in order to "map" the API to its implementation because you use API classes only.
Your example with JSF is a little different in that both Mojarra and MyFaces contains the API classes from javax.faces.* package, basically there are different classes with the same names. So there is no need for configuration, the classes have the same name in both libraries and are loaded by the class loader whenever needed from the Mojarra or MyFaces jars whichever is on the classpath.

I have found MyFaces and Mojarra Implementation for JSF. I have realized that both has same pachage structure so it means that all are compliance with JSF spec. In other word which implementation jar is given into classpath classpath loader will handle it.
It is all about Java Class Loader mechanism of JVM. You can find detailed discussion at Java Class Loaders
Here is the picture of both JSF implemantation package structure.

I've seen the mojarra jar (JSF Reference Implementation) and I found that it uses The Service Loader API, since it places the javax.servlet.ServletContainerInitializer, javax.enterprise.inject.spi.Extension and com.sun.faces.spi.injectionprovider inside the /META-INF/services folder in the javax.faces-2.2.0.jar.
This is how java distinguishes between an implementation and another (mojarra or myfaces).

Runtime code injection

SLF4J has a nice mechanism, where the implementation is chosen at runtime, depending of what is available in the classpath. I would like to use such feature in several projects, for example to choose the communication layer or to choose a mock implementation.
I had a look at slf4j source to see how it's done and I could just write something similar. Before I start, I would like to know if some lightweight FOSS library exists for this kind of injection.

Unless you need specific configuration abilities as provided by Pico or Guice, you may get what you need from java.util.ServiceLoader.
Basically, all you have to do is to package your service implementation in a JAR file, include a text file with a list of all implementation classes in "META-INF/services/" and on you go.

Have you looked at Weld, CDI is part of the EE6 spec but the Weld implementation also supports running in a Java SE environment. It has exactly what you are looking for, here is a link to the relative documentation:
http://seamframework.org/Weld one maven dependency for your SE app.
http://docs.jboss.org/weld/reference/1.1.0.Final/en-US/html/environments.html#d0e5333 bootstrapping the Weld container in SE.
Producer methods to vary implementation at runtime:
http://docs.jboss.org/weld/reference/1.1.0.Final/en-US/html/producermethods.html
Plus (in my very biased opinion) Weld rocks ;)

SLF4J's "mechanism" is simply that its API jar is compiled with code that refers to a class that is only provided by one of its "implementation" jars. No framework or library of any kind is needed for this. Simply write one module which is compiled against a class not in that module. Then your "implementation" modules provide that class when included in the project.
Edit: Oh, and this is basically OSGi writ small (very, very small). If you're going to use this kind of thing on a large scale, look to an OSGi container or Eclipse Virgo.

Every java programmer should know how to use Spring.

Java ServiceLoader with multiple Classloaders

What are the best practices for using ServiceLoader in an Environment with multiple ClassLoaders? The documentation recommends to create and save a single service instance at initialization:
private static ServiceLoader<CodecSet> codecSetLoader = ServiceLoader.load(CodecSet.class);
This would initialize the ServiceLoader using the current context classloader. Now suppose this snippet is contained in a class loaded using a shared classloader in a web container and multiple web applications want to define their own service implementations. These would not get picked up in the above code, it might even be possible that the loader gets initialized using the first webapps context classloader and provide the wrong implementation to other users.
Always creating a new ServiceLoader seems wasteful performance wise since it has to enumerate and parse service files each time. Edit: This can even be a big performance problem as shown in this answer regarding java's XPath implementation.
How do other libraries handle this? Do they cache the implementations per classloader, do they reparse their configuration everytime or do they simply ignore this problem and only work for one classloader?

I personally do not like the ServiceLoader under any circumstances. It's slow and needlessly wasteful and there is little you can do to optimize it.
I also find it a bit limited -- you really have to go out of your way if you want to do more than search by type alone.
xbean-finder's ResourceFinder
ResourceFinder is a self-contained java file capable of replacing ServiceLoader usage. Copy/paste reuse is no problem. It's one java file and is ASL 2.0 licensed and available from Apache.
Before our attention spans get too short, here's how it can replace a ServiceLoader
ResourceFinder finder = new ResourceFinder("META-INF/services/");
List<Class<? extends Plugin>> impls = finder.findAllImplementations(Plugin.class);
This will find all of the META-INF/services/org.acme.Plugin implementations in your classpath.
Note it does not actually instantiate all the instances. Pick the one(s) you want and you're one newInstance() call away from having an instance.
Why is this nice?
How hard is it to call newInstance() with proper exception handling? Not hard.
Having the freedom to instantiate only the ones you want is nice.
Now you can support constructor args!
Narrowing search scope
If you want to just check specific URLs you can do so easily:
URL url = new File("some.jar").toURI().toURL();
ResourceFinder finder = new ResourceFinder("META-INF/services/", url);
Here, only the 'some.jar' will be searched on any usage of this ResourceFinder instance.
There's also a convenience class called UrlSet which can make selecting URLs from the classpath very easy.
ClassLoader webAppClassLoader = Thread.currentThread().getContextClassLoader();
UrlSet urlSet = new UrlSet(webAppClassLoader);
urlSet = urlSet.exclude(webAppClassLoader.getParent());
urlSet = urlSet.matching(".*acme-.*.jar");
List<URL> urls = urlSet.getUrls();
Alternate "service" styles
Say you wanted to apply the ServiceLoader type concept to redesign URL handling and find/load the java.net.URLStreamHandler for a specific protocol.
Here's how you might layout the services in your classpath:
META-INF/java.net.URLStreamHandler/foo
META-INF/java.net.URLStreamHandler/bar
META-INF/java.net.URLStreamHandler/baz
Where foo is a plain text file that contains the name of the service implementation just as before. Now say someone creates a foo://... URL. We can find the implementation for that quickly, via:
ResourceFinder finder = new ResourceFinder("META-INF/");
Map<String, Class<? extends URLStreamHandler>> handlers = finder.mapAllImplementations(URLStreamHandler.class);
Class<? extends URLStreamHandler> fooHandler = handlers.get("foo");
Alternate "service" styles 2
Say you wanted to put some configuration information in your service file, so it contains more than just a classname. Here's an alternate style that resolves services to properties files. By convention one key would be the class names and the other keys would be injectable properties.
So here red is a properties file
META-INF/org.acme.Plugin/red
META-INF/org.acme.Plugin/blue
META-INF/org.acme.Plugin/green
You can look things up similarly as before.
ResourceFinder finder = new ResourceFinder("META-INF/");
Map<String,Properties> plugins = finder.mapAllProperties(Plugin.class.getName());
Properties redDefinition = plugins.get("red");
Here's how you could use those properties with xbean-reflect, another little library that can give you framework-free IoC. You just give it the class name and some name value pairs and it will construct and inject.
ObjectRecipe recipe = new ObjectRecipe(redDefinition.remove("className").toString());
recipe.setAllProperties(redDefinition);
Plugin red = (Plugin) recipe.create();
red.start();
Here's how that might look "spelled" out in long form:
ObjectRecipe recipe = new ObjectRecipe("com.example.plugins.RedPlugin");
recipe.setProperty("myDateField","2011-08-29");
recipe.setProperty("myIntField","100");
recipe.setProperty("myBooleanField","true");
recipe.setProperty("myUrlField","http://www.stackoverflow.com");
Plugin red = (Plugin) recipe.create();
red.start();
The xbean-reflect library is a step beyond the built-in JavaBeans API, but a bit better without requiring you to go all the way to a full-on IoC framework like Guice or Spring. It supports factory methods and constructor args and setter/field injection.
Why is the ServiceLoader so limited?
Deprecated code in the JVM damages the Java language itself. Many things are trimmed to the bone before being added to the JVM, because you cannot trim them after. The ServiceLoader is a prime example of that. The API is limited and OpenJDK implementation is somewhere around 500 lines including javadoc.
There's nothing fancy there and replacing it is easy. If it doesn't work for you, don't use it.
Classpath scope
APIs aside, in pure practicality narrowing the scope of the URLs searched is the true solution to this problem. App Servers have quite a lot of URLs all by themselves, not including the jars in your application. Tomcat 7 on OSX for example has about 40~ URLs in the StandardClassLoader alone (this is the parent to all webapp classloaders).
The bigger your app server the longer even a simple search will take.
Caching doesn't help if you intend to search for more than one entry. As well, it can add some bad leaks. Can be a real lose-lose scenario.
Narrow the URLs down to the 5 or 12 that you really care about and you can do all sorts of service loading and never notice the hit.

Have you tried using the two argument version so that you can specify which classloader to use? Ie, java.util.ServiceLoader.load(Class, ClassLoader)

Mu.
In a 1x WebContainer <-> Nx WebApplication system, the ServiceLoader instantiated in the WebContainer will not pick up any classes defined in WebApplications, just those in the container. A ServiceLoader instantiated in a WebApplication will detect classes defined in the application in addition to those defined in the container.
Keep in mind WebApplications will need to be kept separate, are designed that way, things will break if you try and circumvent that, and they are not the method and system available to extend the container - if your library is a simple Jar, just drop it into the appropriate extension folder of the container.

I really like Neil's answer in the link I added in my comment. Due to I have same experences in my recent project.
"Another thing to bear in mind with ServiceLoader is to try to abstract the lookup mechanism. The publish mechanism is quite nice and clean and declarative. But the lookup (via java.util.ServiceLoader) is as ugly as hell, implemented as a classpath scanner that breaks horribly if you put the code into any environment (such as OSGi or Java EE) that does not have global visibility. If your code gets tangled up with that then you'll have a hard time running it on OSGi later. Better to write an abstraction that you can replace when the time comes."
I actually met this problem in OSGi environment actually it's just eclipse in our project. But I luckily fixed it in a timely fashion. My workaround is using one class from the plugin I want to load ,and get classLoader from it. That will be a valid fix. I didn't use the standard ServiceLoader, But my process is quite similiar, use a properties to define the plugin classes I need to load. And I know there is another way to know each plugin's classloader. But at least I don't need to use that.
Honest, I don't like the generics used in ServiceLoader. Because it limited that one ServiceLoader can only handle classes for one interface. Well is it really useful? In my implementation, it don't force you by this limitation. I just use one implementation of loader to load all the plugin classes. I don't see the reason to use two or more. Due to the consumer can know from the config files about the relationships between interfaces and implementation.

This question seem to be more complicated than I first anticipated. As I see
it, there are 3 possible strategies for dealing with ServiceLoaders.
Use a static ServiceLoader instance and only support loading classes from
the same classloader as the one holding the ServiceLoader reference. This
would work when
The service configuration and implementation are in a shared classloader
and all child classloaders are using the same implementation. The example
in the documentation is geared towards theis use case.
Or
Configuration and implementation are put into each child classloader and
deployed along each webapp in WEB-INF/lib.
In this scenario it is not possible to deploy the service in a shared classloader
and let each webapp choose its own service implementation.
Initialize the ServiceLoader on each access passing the context classloader of
the current thread as the second parameter. This approach is taken be the JAXP
and JAXB apis, although they are using their own FactoryFinder implementation
instead of ServiceLoader. So it is possible to bundle a xml parser with a webapp
and have it automatically get picked up for example by DocumentBuilderFactory#newInstance.
This lookup has a performance impact, but in the case of xml parsing
the time to look up the implementation is small compared to the time needed to
actually parse a xml document. In the library I'm envisioning the factory
itself is pretty simple so the lookup time would dominate the performance.
Somehow cache the implementation class with the context classloader as the key.
I'm not entirely sure if this is possible in all the above cases without
causing any memory leaks.
In conclusion, I will probably be ignoring this problem and require that the library
gets deployed inside each webapp, i.e. option 1b above.