We Keep Coding

JVM Security Manager File permissions - custom policy

I've found a somehow unexpected behaviour using JVM Security Manager custom policies.
repo: https://github.com/pedrorijo91/jvm-sec-manager
in branch master, go into the /code folder:
custom policy file grants File read permission for file ../allow/allow.txt
no permission for the file ../deny/deny.txt
the code in the HelloWorld.java tries to read both files
There's a run.sh script to run the command
Now everything works as expected: the allowed file reads, but the other throws a security exception: java.security.AccessControlException: access denied ("java.io.FilePermission" "../deny/deny.txt" "read")
But if I move both files (../allow/allow.txt and ../deny/deny.txt) to the code folder (changing the custom policy and the java code to use those files), I get no exception. (branch 'unexpected')
Is the current directory a special case or something else is happening?

Brief explanation
This behaviour is documented in a number of places:
FilePermission's overview;
the Permissions in the JDK document; and
the URLClassLoader#getPermissions(CodeSource) method.
The latter two reiterate the closing note of the first one, which states that:
Code can always read a file from the same directory it's in (or a subdirectory of that directory); it does not need explicit permission to do so.
In other words, if
(HelloWorld.class.getProtectionDomain().getCodeSource().implies(
new CodeSource(new URL("file:" + codeDir),
(Certificate[]) null)) == true)
then HelloWorld will by default be granted read access to the denoted directory and its descendants. Particularly for the code directory itself this should make some intuitive sense, as otherwise the class would be unable to even access public-access classes within its very package.
The full story
It is basically up to the ClassLoader: If it statically assigned any Permissions to the ProtectionDomain to which it mapped the class--which applies to both java.net.URLClassLoader and sun.misc.Launcher$AppClassLoader (the OpenJDK-specific default system class loader)--these permissions will always be accorded to the domain, regardless of the Policy in effect.
Workarounds
The typical "quick-n'-dirty" workaround to anything authorization-related is to extend SecurityManager and override the methods irking you; i.e. in this case the checkRead group of methods.
For a more thorough solution that doesn't reduce the flexibility of AccessController and friends, on the other hand, you would have to write a class loader that at the very least overrides URLClassLoader#getPermissions(CodeSource) and/or restricts loaded classes' domains' CodeSources down to the file level (code sources of domains assigned by default by URLClassLoader and AppClassLoader imply (recursively) the .class file's classpath entry (JAR or directory)). For further granularity, your loader might as well assign instances of your own domain subclass, and/or domains encapsulating code sources of your own subclass, overriding respectively ProtectionDomain#implies(Permission) and/or CodeSource#implies(CodeSource); the former could for example be made to support "negative permission" semantics, and the latter could base code source implication on arbitrary logic, potentially decoupled from physical code location (think e.g. "trust levels").
Clarification as per the comments
To prove that under a different class loader these permissions actually matter, consider the following example: There are two classes, A and B; A has the main method, which simply calls a method on B. Additionally, the application is launched using a different system class loader, which a) assigns domains on a per-class basis (rather than on a per-classpath-entry basis, as is the default) to classes it loads, without b) assigning any permissions to these domains.
Loader:
package com.example.q45897574;
import java.io.BufferedInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.security.AccessController;
import java.security.CodeSource;
import java.security.PermissionCollection;
import java.security.Permissions;
import java.security.PrivilegedAction;
import java.security.ProtectionDomain;
import java.security.cert.Certificate;
import java.util.LinkedHashSet;
import java.util.Set;
import java.util.regex.Pattern;
public class RestrictiveClassLoader extends URLClassLoader {
private static final Pattern COMMON_SYSTEM_RESOURCE_NAMES = Pattern
.compile("(((net\\.)?java)|(java(x)?)|(sun|oracle))\\.[a-zA-Z0-9\\.\\-_\\$\\.]+");
private static final String OWN_CLASS_NAME = RestrictiveClassLoader.class.getName();
private static final URL[] EMPTY_URL_ARRAY = new URL[0], CLASSPATH_ENTRY_URLS;
private static final PermissionCollection NO_PERMS = new Permissions();
static {
String[] classpathEntries = AccessController.doPrivileged(new PrivilegedAction<String>() {
#Override
public String run() {
return System.getProperty("java.class.path");
}
}).split(File.pathSeparator);
Set<URL> classpathEntryUrls = new LinkedHashSet<>(classpathEntries.length, 1);
for (String classpathEntry : classpathEntries) {
try {
URL classpathEntryUrl;
if (classpathEntry.endsWith(".jar")) {
classpathEntryUrl = new URL("file:jar:".concat(classpathEntry));
}
else {
if (!classpathEntry.endsWith("/")) {
classpathEntry = classpathEntry.concat("/");
}
classpathEntryUrl = new URL("file:".concat(classpathEntry));
}
classpathEntryUrls.add(classpathEntryUrl);
}
catch (MalformedURLException mue) {
}
}
CLASSPATH_ENTRY_URLS = classpathEntryUrls.toArray(EMPTY_URL_ARRAY);
}
private static byte[] readClassData(URL classResource) throws IOException {
try (InputStream in = new BufferedInputStream(classResource.openStream());
ByteArrayOutputStream out = new ByteArrayOutputStream()) {
while (in.available() > 0) {
out.write(in.read());
}
return out.toByteArray();
}
}
public RestrictiveClassLoader(ClassLoader parent) {
super(EMPTY_URL_ARRAY, parent);
for (URL classpathEntryUrl : CLASSPATH_ENTRY_URLS) {
addURL(classpathEntryUrl);
}
}
#Override
protected Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
if (name == null) {
throw new ClassNotFoundException("< null >", new NullPointerException("name argument must not be null."));
}
if (OWN_CLASS_NAME.equals(name)) {
return RestrictiveClassLoader.class;
}
if (COMMON_SYSTEM_RESOURCE_NAMES.matcher(name).matches()) {
return getParent().loadClass(name);
}
Class<?> ret = findLoadedClass(name);
if (ret != null) {
return ret;
}
return findClass(name);
}
#Override
protected Class<?> findClass(String name) throws ClassNotFoundException {
String modifiedClassName = name.replace(".", "/").concat(".class");
URL classResource = findResource(modifiedClassName);
if (classResource == null) {
throw new ClassNotFoundException(name);
}
byte[] classData;
try {
classData = readClassData(classResource);
}
catch (IOException ioe) {
throw new ClassNotFoundException(name, ioe);
}
return defineClass(name, classData, 0, classData.length, constructClassDomain(classResource));
}
#Override
protected PermissionCollection getPermissions(CodeSource codesource) {
return NO_PERMS;
}
private ProtectionDomain constructClassDomain(URL codeSourceLocation) {
CodeSource cs = new CodeSource(codeSourceLocation, (Certificate[]) null);
return new ProtectionDomain(cs, getPermissions(cs), this, null);
}
}
A:
package com.example.q45897574;
public class A {
public static void main(String... args) {
/*
* Note:
* > Can't we set the security manager via launch argument?
* No, it has to be set here, or bootstrapping will fail.
* > Why?
* Because our class loader's domain is unprivileged.
* > Can't it be privileged?
* Yes, but then everything under the same classpath entry becomes
* privileged too, because our loader's domain's code source--which
* _its own_ loader creates, thus escaping our control--implies _the
* entire_ classpath entry. There are various workarounds, which
* however fall outside of this example's scope.
*/
System.setSecurityManager(new SecurityManager());
B.b();
}
}
B:
package com.example.q45897574;
public class B {
public static void b() {
System.out.println("success!");
}
}
Unprivileged test:
Make sure nothing is granted at the policy level; then run (assuming a Linux-based OS--modify classpath as appropriate):
java -cp "/home/your_user/classpath/" \
-Djava.system.class.loader=com.example.q45897574.RestrictiveClassLoader \
-Djava.security.debug=access=failure com.example.q45897574.A
You should get a NoClassDefFoundError, along with a failed FilePermission for com.example.q45897574.A.
Privileged test:
Now grant the necessary permission to A (again make sure to correct both the codeBase (code source URL) and permission target name):
grant codeBase "file:/home/your_user/classpath/com/example/q45897574/A.class" {
permission java.io.FilePermission "/home/your_user/classpath/com/example/q45897574/B.class", "read";
};
...and re-run. This time execution should complete successfully.

Using java securityManager blocks me from reading files

In my java code I call another 3rd party java class.
I want to catch that latter System.exit() exit code
So I use security-manager as suggested in this post
The problem is that I cannot read files now, I get permissions error
as seen in that post.
How can I catch the exit code and still read files?
Published class MyClass {
class MySecurityManager extends SecurityManager {
#Override
public void checkExit(int status) {
throw new SecurityException();
}
}
public void foo() {
MySecurityManager secManager = new MySecurityManager();
System.setSecurityManager(secManager);
try {
ConfigValidator.main(new String[]{"-dirs", SdkServiceConfig.s.PROPERTIES_FILE_PATH});
new FileInputStream(new File("/Users/eladb/WorkspaceQa/sdk-service/src/main/resources/convert_conditions.sh"));
} catch (SecurityException e) {
//Do something if the external code used System.exit()
String a = "1";
}
catch (Exception e) {
logger.error("failed converting properties file to proto", e);
}
}
}

You have two separate problems: Your trusted code cannot read the file, while the untrusted third-party library can still call System#exit unhindered. The former can be easily circumvented by granting further privileges to the trusted code; the latter is a tad trickier to address.
A bit of background
Privilege assignment
Code (the ProtectionDomains encapsulated by a thread's AccessControlContext) generally gets assigned Permissions in two ways: Statically, by the ClassLoader, upon class definition, and/or dynamically, by the Policy in effect. Other, less frequently encountered alternatives, exist as well: DomainCombiners, for instance, can modify AccessControlContexts' domains (and therefore the effective permissions of their respective code that is subject to authorization) on the fly, and custom domain implementations may use their own logic for permission implication, possibly disregarding or altering the semantics of the policy. By default the permission set of a domain is the union of its static and dynamic permissions. As for how exactly classes are mapped to domains, it is, for the most part, up to the loader's implementation. By default, all classes, JAR'ed or otherwise, residing beneath the same class path entry, are grouped under the same domain. More restrictive class loaders may choose to e.g. allocate a domain per class, which could be used to prevent communication even between classes within the same package.
Privilege evaluation
Under the default SecurityManager, for a privileged operation (an invocation of any method having a SecurityManager#checkXXX within its body) to succeed, every domain (of every class of every method) of the effective AccessControlContext must have been assigned, as explained above, the permission being checked. Recall however that the context need not necessarily represent "the truth" (the actual call stack)—system code gets optimized away early on, while AccessController#doPrivileged calls, along with the DomainCombiner potentially coupled to the AccessControlContext can modify the context's domains, and the authorization algorithm in its entirety, consequently.
Problem and workarounds
The issue with System#exit is that the corresponding permission (RuntimePermission("exitVM.*")) is one amongst few that are statically assigned by the default application class loader (sun.misc.Launcher$AppClassLoader) to all domains associated with classes loaded from the class path.
A number of alternatives come to mind:
Installing a custom SecurityManager which denies the particular right based on, e.g., the class attempting to terminate the JVM process.
Loading the third-party library from a "remote" location (a directory outside of the class path), so that it gets treated as "untrusted" code by its class loader.
Authoring and installing a different application class loader, which does not assign the extraneous permission.
Plugging a custom domain combiner into the access control context, which replaces, at the time of an authorization decision, all third-party domains with equivalent ones that do not have the offending permission.
I should, for the sake of completeness, note that at the Policy level, unfortunately, nothing can be done to negate statically assigned permissions.
The first option is overall the most convenient one, but I will not explore it further because:
The default SecurityManager is quite flexible, thanks to the handful of components (AccessController et al.) it interacts with. The background section in the beginning aimed to serve as a reminder of that flexibility, which "quick-n'-dirty" method overrides tend to cripple.
Careless modifications of the default implementation might cause (system) code to misbehave in curious ways.
Frankly, because it's boring―it's the one-size-fits-all solution perpetually advocated, while the fact that the default manager was standardized in 1.2 for a reason has long been forgotten.
The second alternative is easy to implement but impractical, complicating either development or the build process. Assuming you are not planning to invoke the library solely reflectively, or aided by interfaces present on the class path, it would have to be present initially, during development, and relocated before execution.
The third is, at least in the context of a standalone Java SE application, fairly straightforward and should not pose too much of a burden on performance. It is the approach I will favour herein.
The last option is the most novel and least convenient. It is hard to securely implement, has the greatest potential for performance degradation, and burdens client code with ensuring presence of the combiner prior to every delegation to untrusted code.
Proposed solution
The custom ClassLoader
The following is to be used as the replacement of the default application loader, or alternatively as the context class loader, or the loader used to load at least the untrusted classes. There is nothing novel to this implementation—all it does is prevent delegation to the default application class loader when the class in question is assumed to not be a system one. URLClassLoader#findClass, in turn, does not assign RuntimePermission("exitVM.*") to the domains of the classes it defines.
package com.example.trusted;
import java.io.File;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.regex.Pattern;
public class ClasspathClassLoader extends URLClassLoader {
private static final Pattern SYSTEM_CLASS_PREFIX = Pattern.compile("((java(x)?|sun|oracle)\\.).*");
public ClasspathClassLoader(ClassLoader parent) {
super(new URL[0], parent);
String[] classpath = System.getProperty("java.class.path").split(File.pathSeparator);
for (String classpathEntry : classpath) {
try {
if (!classpathEntry.endsWith(".jar") && !classpathEntry.endsWith("/")) {
// URLClassLoader assumes paths without a trailing '/' to be JARs by default
classpathEntry += "/";
}
addURL(new URL("file:" + classpathEntry));
}
catch (MalformedURLException mue) {
System.err.println(MessageFormat.format("Erroneous class path entry [{0}] skipped.", classpathEntry));
}
}
}
#Override
protected Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
Class<?> ret;
synchronized (getClassLoadingLock(name)) {
ret = findLoadedClass(name);
if (ret != null) {
return ret;
}
if (SYSTEM_CLASS_PREFIX.matcher(name).matches()) {
return super.loadClass(name, resolve);
}
ret = findClass(name);
if (resolve) {
resolveClass(ret);
}
}
return ret;
}
}
If you also wish to fine-tune the domains assigned to loaded classes, you will additionally have to override findClass. The following variant of the loader is a very crude attempt at doing so. constructClassDomain therein merely creates one domain per class path entry (which is more or less the default), but can be modified to do something different.
package com.example.trusted;
import java.io.BufferedInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.IOException;
import java.lang.ref.WeakReference;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.ByteBuffer;
import java.security.AccessController;
import java.security.CodeSource;
import java.security.PrivilegedAction;
import java.security.ProtectionDomain;
import java.security.cert.Certificate;
import java.text.MessageFormat;
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Pattern;
public final class ClasspathClassLoader extends URLClassLoader {
private static final Pattern SYSTEM_CLASS_PREFIX = Pattern.compile("((java(x)?|sun|oracle)\\.).*");
private static final List<WeakReference<ProtectionDomain>> DOMAIN_CACHE = new ArrayList<>();
// constructor, loadClass same as above
#Override
protected Class<?> findClass(String name) throws ClassNotFoundException {
URL classOrigin = getClassResource(name);
if (classOrigin == null) {
return super.findClass(name);
}
URL classCodeSourceOrigin = getClassCodeSourceResource(classOrigin);
if (classCodeSourceOrigin == null) {
return super.findClass(name);
}
return defineClass(name, readClassData(classOrigin), constructClassDomain(classCodeSourceOrigin));
}
private URL getClassResource(String name) {
return AccessController.doPrivileged((PrivilegedAction<URL>) () -> getResource(name.replace(".", "/") + ".class"));
}
private URL getClassCodeSourceResource(URL classResource) {
for (URL classpathEntry : getURLs()) {
if (classResource.getPath().startsWith(classpathEntry.getPath())) {
return classpathEntry;
}
}
return null;
}
private ByteBuffer readClassData(URL classResource) {
try {
BufferedInputStream in = new BufferedInputStream(classResource.openStream());
ByteArrayOutputStream out = new ByteArrayOutputStream();
int i;
while ((i = in.read()) != -1) {
out.write(i);
}
return ByteBuffer.wrap(out.toByteArray());
}
catch (IOException ioe) {
throw new RuntimeException(ioe);
}
}
private ProtectionDomain constructClassDomain(URL classCodeSourceResource) {
ProtectionDomain ret = getCachedDomain(classCodeSourceResource);
if (ret == null) {
CodeSource cs = new CodeSource(classCodeSourceResource, (Certificate[]) null);
DOMAIN_CACHE.add(new WeakReference<>(ret = new ProtectionDomain(cs, getPermissions(cs), this, null)));
}
return ret;
}
private ProtectionDomain getCachedDomain(URL classCodeSourceResource) {
for (WeakReference<ProtectionDomain> domainRef : DOMAIN_CACHE) {
ProtectionDomain domain = domainRef.get();
if (domain == null) {
DOMAIN_CACHE.remove(domainRef);
}
else if (domain.getCodeSource().implies(new CodeSource(classCodeSourceResource, (Certificate[]) null))) {
return domain;
}
}
return null;
}
}
The "unsafe" code
package com.example.untrusted;
public class Test {
public static void testExitVm() {
System.out.println("May I...?!");
System.exit(-1);
}
}
The entry point
package com.example.trusted;
import java.security.AccessControlException;
import java.security.Permission;
import com.example.untrusted.Test;
public class Main {
private static final Permission EXIT_VM_PERM = new RuntimePermission("exitVM.*");
public static void main(String... args) {
System.setSecurityManager(new SecurityManager());
try {
Test.testExitVm();
}
catch (AccessControlException ace) {
Permission deniedPerm = ace.getPermission();
if (EXIT_VM_PERM.implies(deniedPerm)) {
ace.printStackTrace();
handleUnauthorizedVmExitAttempt(Integer.parseInt(deniedPerm.getName().replace("exitVM.", "")));
}
}
}
private static void handleUnauthorizedVmExitAttempt(int exitCode) {
System.out.println("here let me do it for you");
System.exit(exitCode);
}
}
Testing
Packaging
Place the loader and the main class in one JAR (let's call it trusted.jar) and the demo untrusted class in another (untrusted.jar).
Assigning privileges
The default Policy (sun.security.provider.PolicyFile) is backed by the file at <JRE>/lib/security/java.policy, as well as any of the files referenced by the policy.url.n properties in <JRE>/lib/security/java.security. Modify the former (the latter should hopefully be empty by default) as follows:
// Standard extensions get all permissions by default
grant codeBase "file:${{java.ext.dirs}}/*" {
permission java.security.AllPermission;
};
// no default permissions
grant {};
// trusted code
grant codeBase "file:///path/to/trusted.jar" {
permission java.security.AllPermission;
};
// third-party code
grant codeBase "file:///path/to/untrusted.jar" {
permission java.lang.RuntimePermission "exitVM.-1", "";
};
Note that it is virtually impossible to get components extending the security infrastructure (custom class loaders, policy providers, etc.) to work properly without granting them AllPermission.
Running
Run:
java -classpath "/path/to/trusted.jar:/path/to/untrusted.jar" -Djava.system.class.loader=com.example.trusted.ClasspathClassLoader com.example.trusted.Main
The privileged operation should succeed.
Next comment out the RuntimePermission under untrusted.jar, within the policy file, and re-run. The privileged operation should fail.
As a closing note, when debugging AccessControlExceptions, running with -Djava.security.debug=access=domain,access=failure,policy can help track down offending domains and policy configuration issues.

How to set my ClassLoader as the JVM's ClassLoader for loading ALL classes (jar classes included)?

I am writing a game engine in which I need to analyze every single class that is mentioned in my program. As this is a game engine, it is to be attached to a client's project as a JAR file. From within that JAR file I need to be able to scan every class that is being used by the client.
So I thought that I should create a custom ClassLoader! By overriding ClassLoader I can take a look at every class that is being used as it is loaded.
I started playing around a bit with ClassLoaders. Here is what I did:
(I stole this classloader from JavaWorld just to play around with it)
public class SimpleClassLoader extends ClassLoader {
private HashMap<String, Class> classes = new HashMap<String, Class>();
public SimpleClassLoader() {
}
/**
* This sample function for reading class implementations reads
* them from the local file system
*/
private byte getClassImplFromDataBase(String className)[] {
System.out.println(" >>>>>> Fetching the implementation of "+className);
byte result[];
try {
FileInputStream fi = new FileInputStream("store\\"+className+".impl");
result = new byte[fi.available()];
fi.read(result);
return result;
} catch (Exception e) {
/*
* If we caught an exception, either the class wasnt found or it
* was unreadable by our process.
*/
return null;
}
}
/**
* This is a simple version for external clients since they
* will always want the class resolved before it is returned
* to them.
*/
public Class loadClass(String className) throws ClassNotFoundException {
return (loadClass(className, true));
}
/**
* This is the required version of loadClass which is called
* both from loadClass above and from the internal function
* FindClassFromClass.
*/
public synchronized Class loadClass(String className, boolean resolveIt)
throws ClassNotFoundException {
Class result;
byte classData[];
System.out.println(className);
System.out.println(" >>>>>> Load class : "+className);
/* Check our local cache of classes */
result = (Class)classes.get(className);
if (result != null) {
System.out.println(" >>>>>> returning cached result.");
return result;
}
/* Check with the primordial class loader */
try {
result = super.findSystemClass(className);
System.out.println(" >>>>>> returning system class (in CLASSPATH).");
return result;
} catch (ClassNotFoundException e) {
System.out.println(" >>>>>> Not a system class.");
}
/* Try to load it from our repository */
classData = getClassImplFromDataBase(className);
if (classData == null) {
throw new ClassNotFoundException();
}
/* Define it (parse the class file) */
result = defineClass(classData, 0, classData.length);
if (result == null) {
throw new ClassFormatError();
}
if (resolveIt) {
resolveClass(result);
}
classes.put(className, result);
System.out.println(" >>>>>> Returning newly loaded class.");
return result;
}
}
Then I decided to test it:
public class Test
{
public static void main(String args[]) throws ClassNotFoundException
{
SimpleClassLoader s = new SimpleClassLoader();
Thread.currentThread().setContextClassLoader(s);
Foo myfoo = new Foo(); //local class
ArrayList myList = new ArrayList(); //class from JAR file
//both should have been loaded from my SimpleClassLoader
System.out.println(s + "\n\tshould be equal to\n" + myfoo.getClass().getClassLoader());
System.out.println("\tand also to: \n" + myList.getClass().getClassLoader());
/*
OUTPUT:
SimpleClassLoader#57fee6fc
should be equal to
sun.misc.Launcher$AppClassLoader#51f12c4e
and also to:
null
***
bizarre results: why are ArrayList and Foo not being loaded by my classloader?
*/
}
}
Is creating a custom ClassLoader the correct approach to the problem described at the top?
Why is the system class loader being invoked? How do I force the JVM to use MY classloader, for ALL threads? (not just the first thread, I want every new thread created to automatically use my classloader)
It is my understanding that via the "delegation system", the parent ClassLoader gets the first try at loading the class; this may be the reason my ClassLoader isn't loading anything. If I am correct, how to I disable this feature? How do I get MY classloader to do the loading?

The instrumentation api would be a natural fit for what I think you're trying to do
http://docs.oracle.com/javase/6/docs/api/java/lang/instrument/Instrumentation.html
It lets you intercept classes as they are loaded by the JVM - normally to modify them, but you are given the byte array to play with so I can see no reason why you couldn't perform any required analysis without actually modifying.
This would let you analyse classes before they are loaded. I'm not entirely clear on what it is you are trying to achieve, but if you can analyse the classes after they are loaded then you could simply open and parse the contents of all jars on the classpath.

How do I create a parent-last / child-first ClassLoader in Java, or How to override an old Xerces version that was already loaded in the parent CL?

I would like to create a parent-last / child-first class loader, e.g. a class loader that will look for classes in the child class loder first, and only then delegate to it's parent ClassLoader to search for classes.
Clarification:
I know now that to get complete ClassLoading seperation I need to use something like a URLClassLoader passing null as it's parent, thanks to this answer to my previous question
However the current question comes to help me resolve this issue:
My code + dependent jars are being loaded into an existing system, using a ClassLoader that sets that System's ClassLoader as it's parent (URLClassLoader)
That System uses some libraries of a version not compatible with the one I need (e.g. older version of Xerces, that doesn't allow me to run my code)
My code runs perfectly fine if runs stand alone, but it fails if runs from that ClassLoader
Howerver I do need access to many other classes within the parent ClassLoader
Therefore I want to allow me to Override, the parent classloader "jars" with my own: If a class I call is found in the child class loader (e.g. I provided a newer version of Xerces with my own jars, instead of the one users by the ClassLoader that loaded my code and jars.
Here is the System's code that loads my code + Jars (I can't change this one)
File addOnFolder = new File("/addOns");
URL url = addOnFolder.toURL();
URL[] urls = new URL[]{url};
ClassLoader parent = getClass().getClassLoader();
cl = URLClassLoader.newInstance(urls, parent);
Here is "my" code (taken fully from the Flying Sauser "Hello World" code demo):
package flyingsaucerpdf;
import java.io.*;
import com.lowagie.text.DocumentException;
import org.xhtmlrenderer.pdf.ITextRenderer;
public class FirstDoc {
public static void main(String[] args)
throws IOException, DocumentException {
String f = new File("sample.xhtml").getAbsolutePath();
System.out.println(f);
//if(true) return;
String inputFile = "sample.html";
String url = new File(inputFile).toURI().toURL().toString();
String outputFile = "firstdoc.pdf";
OutputStream os = new FileOutputStream(outputFile);
ITextRenderer renderer = new ITextRenderer();
renderer.setDocument(url);
renderer.layout();
renderer.createPDF(os);
os.close();
}
}
This works standalone (running main) but fails with this error when loaded through the parent CL:
org.w3c.dom.DOMException: NAMESPACE_ERR: An attempt is made to
create or change an object in a way
which is incorrect with regard to
namespaces.
probably because the parent system uses Xerces of an older version, and even though I provide the right Xerces jar in the /addOns folder, since it's classes were already loaded and used by the parent System, it doesn't allow my own code to use my own jar due to the direction of the delegation. I hope this makes my question clearer, and I'm sure it has been asked
before. (Perhaps I don't ask the right question)

Today is your lucky day, as I had to solve this exact problem. I warn you though, the innards of class loading are a scary place. Doing this makes me think that the designers of Java never imagined that you might want to have a parent-last classloader.
To use just supply a list of URLs containing classes or jars to be available in the child classloader.
/**
* A parent-last classloader that will try the child classloader first and then the parent.
* This takes a fair bit of doing because java really prefers parent-first.
*
* For those not familiar with class loading trickery, be wary
*/
private static class ParentLastURLClassLoader extends ClassLoader
{
private ChildURLClassLoader childClassLoader;
/**
* This class allows me to call findClass on a classloader
*/
private static class FindClassClassLoader extends ClassLoader
{
public FindClassClassLoader(ClassLoader parent)
{
super(parent);
}
#Override
public Class<?> findClass(String name) throws ClassNotFoundException
{
return super.findClass(name);
}
}
/**
* This class delegates (child then parent) for the findClass method for a URLClassLoader.
* We need this because findClass is protected in URLClassLoader
*/
private static class ChildURLClassLoader extends URLClassLoader
{
private FindClassClassLoader realParent;
public ChildURLClassLoader( URL[] urls, FindClassClassLoader realParent )
{
super(urls, null);
this.realParent = realParent;
}
#Override
public Class<?> findClass(String name) throws ClassNotFoundException
{
try
{
// first try to use the URLClassLoader findClass
return super.findClass(name);
}
catch( ClassNotFoundException e )
{
// if that fails, we ask our real parent classloader to load the class (we give up)
return realParent.loadClass(name);
}
}
}
public ParentLastURLClassLoader(List<URL> classpath)
{
super(Thread.currentThread().getContextClassLoader());
URL[] urls = classpath.toArray(new URL[classpath.size()]);
childClassLoader = new ChildURLClassLoader( urls, new FindClassClassLoader(this.getParent()) );
}
#Override
protected synchronized Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException
{
try
{
// first we try to find a class inside the child classloader
return childClassLoader.findClass(name);
}
catch( ClassNotFoundException e )
{
// didn't find it, try the parent
return super.loadClass(name, resolve);
}
}
}
EDIT: Sergio and ɹoƃı have pointed out that if you call .loadClass with the same classname, you will get a LinkageError. While this is true, the normal use-case for this classloader is to set it as the thread's classloader Thread.currentThread().setContextClassLoader() or via Class.forName(), and that works as-is.
However, if .loadClass() was needed directly, this code could be added in the ChildURLClassLoader findClass method at the top.
Class<?> loaded = super.findLoadedClass(name);
if( loaded != null )
return loaded;

The following code is what I use. It has the advantage over the other answer that it doesn't break the parent chain (you can follow getClassLoader().getParent()).
It also has an advantage over tomcat's WebappClassLoader by not reinventing the wheel and not depending on other objects. It re-uses code from URLClassLoader as much as possible.
(it doesn't yet honor the system class loader, but when I get that fixed I'll update the answer)
It honors the system class loader (for java.* classes, endorsed dir, etc.). It also works when security is turned on and the classloader doesn't have access to its parent (yes, this situation is weird, but possible).
public class ChildFirstURLClassLoader extends URLClassLoader {
private ClassLoader system;
public ChildFirstURLClassLoader(URL[] classpath, ClassLoader parent) {
super(classpath, parent);
system = getSystemClassLoader();
}
#Override
protected synchronized Class<?> loadClass(String name, boolean resolve)
throws ClassNotFoundException {
// First, check if the class has already been loaded
Class<?> c = findLoadedClass(name);
if (c == null) {
if (system != null) {
try {
// checking system: jvm classes, endorsed, cmd classpath, etc.
c = system.loadClass(name);
}
catch (ClassNotFoundException ignored) {
}
}
if (c == null) {
try {
// checking local
c = findClass(name);
} catch (ClassNotFoundException e) {
// checking parent
// This call to loadClass may eventually call findClass again, in case the parent doesn't find anything.
c = super.loadClass(name, resolve);
}
}
}
if (resolve) {
resolveClass(c);
}
return c;
}
#Override
public URL getResource(String name) {
URL url = null;
if (system != null) {
url = system.getResource(name);
}
if (url == null) {
url = findResource(name);
if (url == null) {
// This call to getResource may eventually call findResource again, in case the parent doesn't find anything.
url = super.getResource(name);
}
}
return url;
}
#Override
public Enumeration<URL> getResources(String name) throws IOException {
/**
* Similar to super, but local resources are enumerated before parent resources
*/
Enumeration<URL> systemUrls = null;
if (system != null) {
systemUrls = system.getResources(name);
}
Enumeration<URL> localUrls = findResources(name);
Enumeration<URL> parentUrls = null;
if (getParent() != null) {
parentUrls = getParent().getResources(name);
}
final List<URL> urls = new ArrayList<URL>();
if (systemUrls != null) {
while(systemUrls.hasMoreElements()) {
urls.add(systemUrls.nextElement());
}
}
if (localUrls != null) {
while (localUrls.hasMoreElements()) {
urls.add(localUrls.nextElement());
}
}
if (parentUrls != null) {
while (parentUrls.hasMoreElements()) {
urls.add(parentUrls.nextElement());
}
}
return new Enumeration<URL>() {
Iterator<URL> iter = urls.iterator();
public boolean hasMoreElements() {
return iter.hasNext();
}
public URL nextElement() {
return iter.next();
}
};
}
#Override
public InputStream getResourceAsStream(String name) {
URL url = getResource(name);
try {
return url != null ? url.openStream() : null;
} catch (IOException e) {
}
return null;
}
}

By reading the source code of either Jetty or Tomcat, both of which provide parent-last class loaders to implement webapp semantics.
https://github.com/apache/tomcat/blob/7.0.93/java/org/apache/catalina/loader/WebappClassLoaderBase.java
Which is to say, by overriding the findClass method in your ClassLoader class. But why reinvent the wheel when you can steal it?
Reading your various updates, I see that you ran into some classic problems with the XML SPI system.
The general problem is this: if you create a completely isolated class loader, then it's hard to use the objects it returns. If you allow sharing, you can have problems when the parent contains the wrong versions of things.
It is to deal with all this lunacy that OSGi was invented, but that's a big pill to swallow.
Even in webapps, the class loaders exempt some packages from the 'local-first' processing on the assumption that the container and the webapp have to agree on the API between them.

(see at the bottom for an update on a solution I found)
It seems that AntClassLoader has a support for parent first/last, (didn't test it yet)
http://svn.apache.org/repos/asf/ant/core/trunk/src/main/org/apache/tools/ant/AntClassLoader.java
Here is a snippet
/**
* Creates a classloader for the given project using the classpath given.
*
* #param parent The parent classloader to which unsatisfied loading
* attempts are delegated. May be <code>null</code>,
* in which case the classloader which loaded this
* class is used as the parent.
* #param project The project to which this classloader is to belong.
* Must not be <code>null</code>.
* #param classpath the classpath to use to load the classes.
* May be <code>null</code>, in which case no path
* elements are set up to start with.
* #param parentFirst If <code>true</code>, indicates that the parent
* classloader should be consulted before trying to
* load the a class through this loader.
*/
public AntClassLoader(
ClassLoader parent, Project project, Path classpath, boolean parentFirst) {
this(project, classpath);
if (parent != null) {
setParent(parent);
}
setParentFirst(parentFirst);
addJavaLibraries();
}
Update:
Found this as well, when as a last resort I started guessing class names in google (this is what ChildFirstURLClassLoader produced) - but it seems to be incorrect
Update 2:
The 1st Option (AntClassLoader) is very coupled to Ant (requires a Project context and not easy to pass a URL[] to it
The 2nd Option (from an OSGI project in google code) wasn't quite what I needed as it searched parent classloader before the system classloader (Ant class loader does it correctly by the way). The problem as I see it, think that your parent classloader includes a jar (that it shouldn't have) of a functionality that wasn't on JDK 1.4 but was added in 1.5, this has no harm as the parent last class loader (regular delegation model, e.g. URLClassLoader) will always load first the JDK's classes, but here the child first naive implementation seems to unveil the old, redundant jar in the parent class loader, shadowing the JDK / JRE own implementation.
I have yet to find a certified, fully tested, mature Parent Last / Child First correct implementation that is not coupled to a specific solution (Ant, Catalina/Tomcat)
Update 3 - I found it!
I WAS looking in the wrong place,
All I did was add META-INF/services/javax.xml.transform.TransformerFactory and restored the JDK's com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl instead of the old Xalan's org.apache.xalan.processor.TransformerFactoryImpl
The only reason I don't "accept my own answer" yet, is that I don't know if the META-INF/services approach has the same classloader delegation as regular classes (e.g. is it parent-first / child-last or parent-last / child-first?)

URLClassLoader had this constructor public URLClassLoader(URL[], ClassLoader) that allows you to override the parent classloader of an URLClassLoader. You can just load your classloader through an URLClassLoader with an overridden parent classloader.

You can override findClass() and loadClass() to implement a child first class loader:
/**
* Always throws {#link ClassNotFoundException}. Is called if parent class loader
* did not find class.
*/
#Override
protected final Class findClass(String name)
throws ClassNotFoundException
{
throw new ClassNotFoundException();
}
#Override
protected Class loadClass(String name, boolean resolve)
throws ClassNotFoundException
{
synchronized (getClassLoadingLock(name)){
/*
* Check if we have already loaded this class.
*/
Class c = findLoadedClass(name);
if (c == null){
try {
/*
* We haven't previously loaded this class, try load it now
* from SUPER.findClass()
*/
c = super.findClass(name);
}catch (ClassNotFoundException ignore){
/*
* Child did not find class, try parent.
*/
return super.loadClass(name, resolve);
}
}
if (resolve){
resolveClass(c);
}
return c;
}
}

Problems with loading resources during execution

Here's the background of the underlying problem, I am collaborating with a group on a project which uses Swt libraries and I am trying to package the software for deployment. As it turns out SWT is very platform/architecture dependent. I would like to be able to package all six jars (linux, mac, win and 32/64-bit) into the same package and use the appropriate library depending on the system. I realize that it is a tough challenge however, switching to Swing (or anything else) isn't really an option right now.
I have found a number of relevant threads (#Aaron Digulla's thread and #mchr's thread) which provided me valuable insights regarding the problem at hand. I have tried to implement the solution proposed by #Alexey Romanov here. With one difference, as the loadSwtJar() method he proposes is not static, I instantiate the object, and immediately following that, run the method before anything else is done to the object.
It appears as the loading procedure doesn't work properly. My reasoning for this statement is as follows:
If all Swt jars are removed from the classpath of the executable jar file, then Exception in thread "main" java.lang.NoClassDefFoundError: org/eclipse/swt/events/MouseListener is thrown which is caused by: java.lang.ClassNotFoundException: org.eclipse.swt.events.MouseListener
to me this means that the libraries are not found on classpath, am I mistaken?
If swt jars are left on the classpath then the first jar file is used by the system during execution. Meaning if gtk-linux-x86_64 happens to be the first swt jar on the list of jars then the system tries to use that, regardless if the system is win32 or Mac OSX.
I have tried to add some output to see if the loadSwtJar() method is choosing the right jar, and the output seems right on all platforms I have tried, as in the right package is selected (and the files do exist in the runnable jar). But nevertheless the right library is not loaded hence execution errors occur:
Exception in thread "main" java.lang.reflect.InvocationTargetException caused by for ex: Caused by: java.lang.UnsatisfiedLinkError: Cannot load 32-bit SWT libraries on 64-bit JVM
(Note that this is the error I get on my Linux machine if I change the order of appearance of 64-bit and 32 bit swt libraries on the build.xml file)
So, what seems to be the problem here? Am I missing out on some detail, or is it simply not possible to check system properties and load an appropriate library accordingly?
Finally below is an excerpt of my build file, figured it might help finding the source of the problem.
Thanks in advance,
EDIT: After a long debug session with a colleague, the problem is resolved (except an annoying bug regarding Thread management on MacOS as I mentioned here). It involved tweaking with the ANT build as well as the way the main class was written. (The main class, as it turns out, was extending & implementing references from the SWT library which meant that the code wouldn't compile at all, wrapped the main class with another class and loaded the SWT jars from there which seemed to be enough to tackle the problem)
Thanks and regards to everyone who contributed, especially #Aaron. Really appreciated!

Here is a copy of the latest version of my Main class. Let me know if that works for you. I tested it on Linux (32/64bit) and Windows (32bit).
package de.pdark.epen.editor;
import java.io.File;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
import java.net.URL;
import java.net.URLClassLoader;
import org.apache.commons.lang.SystemUtils;
import org.apache.commons.lang.exception.ExceptionUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import ch.qos.logback.classic.LoggerContext;
import ch.qos.logback.core.util.StatusPrinter;
import de.pdark.epen.exceptions.WikiException;
public class Main
{
public final static String VERSION = "V0.9 (13.05.2010)"; //$NON-NLS-1$
private final static Logger log = LoggerFactory.getLogger (Main.class);
private static final String ORG_ECLIPSE_SWT_WIDGETS_SHELL = "org.eclipse.swt.widgets.Shell"; //$NON-NLS-1$
/**
* #param args
*/
#SuppressWarnings({"nls", "PMD.SystemPrintln"})
public static void main (String[] args)
{
String msg = "Starting ePen "+VERSION;
System.out.println (msg);
log.info (msg);
LoggerContext lc = (LoggerContext)LoggerFactory.getILoggerFactory ();
StatusPrinter.print (lc);
int rc = 1;
try
{
Main main = new Main ();
main.run (args);
rc = 0;
}
catch (Throwable t) //NOPMD
{
ExceptionUtils.printRootCauseStackTrace (t);
}
finally
{
System.out.println ("Done.");
log.info ("Exit {}", rc);
System.exit (rc); //NOPMD
}
}
#SuppressWarnings({"nls", "PMD.SystemPrintln", "PMD.SignatureDeclareThrowsException"})
private void run (String[] args) throws Exception
{
if (!SystemUtils.isJavaVersionAtLeast (150))
{
System.out.println ("Version="+SystemUtils.JAVA_VERSION_INT);
throw new WikiException ("Need at least Java 5 but this Java is only "+SystemUtils.JAVA_VERSION);
}
loadSwtJar ();
URLClassLoader cl = (URLClassLoader) getClass().getClassLoader(); //NOPMD
Class<?> c = cl.loadClass ("de.pdark.epen.editor.EPenEditor");
Class<?> shellClass = cl.loadClass (ORG_ECLIPSE_SWT_WIDGETS_SHELL);
Constructor<?> ctor = c.getConstructor (shellClass);
Object obj = ctor.newInstance (new Object[] { null });
Method run = c.getMethod ("run", args.getClass ()); //$NON-NLS-1$
run.invoke (obj, new Object[] { args });
}
#SuppressWarnings({"nls", "PMD"})
private void loadSwtJar ()
{
try {
Class.forName (ORG_ECLIPSE_SWT_WIDGETS_SHELL);
// Already on classpath
return;
} catch (ClassNotFoundException e) {
// Add the JAR
}
String osName = SystemUtils.OS_NAME.toLowerCase ();
String osArch = SystemUtils.OS_ARCH.toLowerCase ();
String swtFileNameOsPart =
osName.contains("win") ? "win32" :
osName.contains("mac") ? "macosx" :
osName.contains("linux") || osName.contains("nix") ? "linux" :
null;
String swtFileNameUiPart =
osName.contains("win") ? "win32" :
osName.contains("mac") ? "cocoa" :
osName.contains("linux") || osName.contains("nix") ? "gtk" :
null;
if (null == swtFileNameOsPart)
{
throw new RuntimeException ("Can't determine name of SWT Jar from os.name=[" + osName + "] and os.arch=["
+ osArch + "]");
}
String swtFileNameArchPart = osArch.contains ("64") ? ".x86_64" : ".x86";
if(".x86".equals(swtFileNameArchPart) && "macosx".equals(swtFileNameOsPart)) {
swtFileNameArchPart = "";
}
String swtFileName = "org.eclipse.swt." + swtFileNameUiPart + "." + swtFileNameOsPart + swtFileNameArchPart + "-3.6.0.jar";
File file = new File ("swt", swtFileName);
if (!file.exists ())
{
throw new RuntimeException ("Can't locate SWT Jar " + file.getAbsolutePath ());
}
try
{
URLClassLoader classLoader = (URLClassLoader) getClass ().getClassLoader ();
Method addUrlMethod = URLClassLoader.class.getDeclaredMethod ("addURL", URL.class);
addUrlMethod.setAccessible (true);
URL swtFileUrl = file.toURI ().toURL ();
log.info ("Adding {} to the classpath", swtFileUrl);
addUrlMethod.invoke (classLoader, swtFileUrl);
}
catch (Exception e)
{
throw new RuntimeException ("Unable to add the swt jar to the class path: " + file.getAbsoluteFile (), e);
}
}
}

You could use Java Web Start as a bootstrap mechanism for your multi platform SWT application. See a corresponding entry in SWT FAQ.
Alternatively, you could put SWT native libraries for each platform into a separate folders and specify them -Djava.library.path in your platform-specific startup script.