We Keep Coding

Why can I not look up classes from sun.misc.DelegatingClassLoader

From this question, I found that it is not possible to look up a class from a sun.misc.DelegatingClassLoader, i.e. looking up a class on its own class loader like
Class<?> accessor = ...
accessor.getClassLoader().findClass(accessor.getName());
throws a ClassNotFoundException. The delegating class loader is used for loading runtime-generated accessor classes for converting Java's reflective JNI calls into Java invocations.
For some strange reason, I am not able to find the source of the DelegatingClassLoader anywhere in the JDK sources even though it is clearly available in my build where it seems to be an empty implementation of a subclass of the standard ClassLoader from looking at the class's byte code.
If the DelegatingClassLoader is however really only a simple subclass, I do not understand why it is not possible to look up a class by its name. Is there some VM magic involved? It seems like the
private native final Class<?> findLoadedClass0(String name);
method does not return the loaded class for a DelegatingClassLoader. The private classes field does however include the loaded class.
I was not able to find any information on how these class loaders are supposed to be different from any other class loaders. I am looking for an explanation why the above lookup does not work but throws an exception.

The source for DelegatingClassLoader is for whatever reason located in ClassDefiner.java. Interestingly, this code has a comment that references bug 4474172, which suggests that this class loader is known to the JVM and has special behavior:
The second solution is to make the virtual machine "magically"
delegate class loading for these newly-fabricated class loaders to
their parent loaders, without making an upcall to Java. The
disadvantage is that this is another hack in the JVM [...]. This
option has been chosen.

Gaining access to package-access members by creating the same package name

I have a question about a sneaky way to gain access to package-access members that occurred to me. Specifically, I want to extend a class - let's call it com.acme.Foo - to add some functionality. This is pure addition: all current methods of Foo would be supported just by delegating to the superclass's method. However, there is no accessible constructor of Foo, so I can't extend the class and my implementation won't pass the "isA" test for being a Foo. There is no interface that expresses Foo-ness that I can use instead of inheritance.
This is where the sneaky thought occurred to me: Foo has a package-access constructor, so why not just create a package com.acme in my source folder and create an InheritableFoo class in that package:
package com.acme;
public class InheritableFoo extends Foo {
public InheritableFoo() {
super();
}
}
Now I can implement my extension to Foo by extending InheritableFoo, since that DOES have an accessible constructor.
This feels all wrong, and somehow unethical: I'm not respecting the decision the original programmer made when he or she decided to not expose a constructor for Foo. But it could be the case that they simply adopted the "deny everything until there's a reason for it" approach. Either way, is there anything fundamentally wrong with this idea? Am I going to cause myself problems in the future if I go this route, and if so, why?

You are correct regarding the fact that you can "sneak" in a class of your own into another package and get access to package restricted elements.
This will however only as long as the author of com.acme.foo distribute his code as source or an unsealed package. Should the author of this package distribute it as a sealed jar (which is quite common I believe) your approach will no longer work.
From Sealing Packages within a JAR File
Packages within JAR files can be optionally sealed, which means that
all classes defined in that package must be archived in the same JAR
file.

Is it possible to "reset" a class loader?

I have to dynamically load a class with the same name from some JAR, but different implementation, multiple times.
I'm creating an evaluator backend and I have to dynamically load classes and test them.
The tests are JUnit classes that instantiate the classes which should be tested, this is a simple example:
package evaluator.tests;
import static org.junit.Assert.*;
import org.junit.*;
import evaluator.tested.*;
public class KTest {
private K tested;
#Before
public void setup() {
tested = new K();
}
#Test
public void returnsTrueTest() {
assertTrue(tested.returnsTrue());
}
}
The rest of my application would need to receive JAR files from users which would contain implementations of the K class which is being tested above. Then, KTest would have to run on their K classes, not the ones in the application.
I know how to dynamically load a class, but I don't know how to make a test work with it, and not the one which I made.
One of the solutions I came up with was to isolate testing in a completely new class, e.g. Evaluation, create a new class loader in that class, and make it load all referenced classes. After creating the class loader, it would load the K class from the JAR file.
This would mean that each time a user submits his JAR, a separate Evaluation would be instantiated, it would create its own class loader and start the JUnit test. When that happens, the test would use the user's implementation of K, and not the default one.
Is this possible, and how can it be done?
I read that class loaders always ask their parent whether a class is already loaded. This would mean that I would have to somehow "flush" all the classes that I loaded dynamically from the JAR file in Evaluation, so that they would be unloaded and then loaded again in another Evaluation.
Load class K, test class K, unload class K, repeat with different K.

Yes you can do this.
If you for example use java.net.URLClassLoader with null as parent: new URLClassLoader( urlArray , null ). Then the bootstrap ClassLoader will be used as parent for your ClassLoader.
Here is an example Class which simply uses a new classloader two reload a class.
package com.anarsoft.agent.regression;
import java.net.URL;
import java.net.URLClassLoader;
public class TestClassLoading {
public static boolean field = false;
public static void main(String[] args) throws Exception
{
URL[] urlArray = new URL[] { TestClassLoading.class.getProtectionDomain().getCodeSource().getLocation().toURI().toURL() };
URLClassLoader firstClassloader = new URLClassLoader( urlArray , null );
Class firstClass = firstClassloader.loadClass("com.anarsoft.agent.regression.TestClassLoading");
firstClass.getField("field").setBoolean(null,true);
System.out.println(firstClass.getField("field").getBoolean(null)); // true
URLClassLoader secondClassloader = new URLClassLoader( urlArray , null );
Class secondClass = secondClassloader.loadClass("com.anarsoft.agent.regression.TestClassLoading");
System.out.println(secondClass.getField("field").getBoolean(null)); // false
// the static field is false since its a new loaded class
}
}

I've made a blog post on this subject, in which the problematics is explained a bit further.
After some research I think I've found a way to accomplish exactly what I want. I haven't implemented this system yet (will edit the answer once I do), so I'd like some feedback from people who are more experienced in this regard.
This is how I understand URLClassLoaders (and classloaders in general) work:
URLClassLoader.loadClass() gets automatically called if it’s the current classloader (for the method/class executing). The call is first delegated to its parent classloader, and if nothing is found it uses its own custom findClass() to load it off one of the URLs it has. loadClass() here (and the classloading logic) is just inherited from the regular ClassLoader.
The default behavior of this method (loadClass()) is to first delegate the search to the parent, and if parent cannot find the class, only then to call its own findClass(). This method (findClass()) is, by default (in ClassLoader), left unimplemented, and you are supposed to implement it yourself, by essentially getting the bytecodes of the class from somewhere (e.g. a file or a network) and calling defineClass() on them.
Once you call defineClass() on some class, and only then, are you registered as the classloader of this class. In all other cases classloading is either delegated to your parent (and you are not the classloader of the class you are loading, paradoxically), or you throw a ClassNotFoundException. You cannot change the classloader of some class at runtime, it is set once it is loaded, and constant.
All of my test classes will try to getClass().getClassLoader().loadClass() all of the classes that they reference - including my custom test classes (this is the regular behavior of all classes, not just my tests, to be clear). As long as they’re using standard classes, and other classes from my application that are not to-be-tested classes, this classloading method should be delegated further on to the application classloader. However, as soon as they try to load a to-be-tested class, they need to get their own, specifically loaded, custom version of it.
The use case of my application is that a user submits a JAR with some class, then a test that expects this class to have certain methods is ran on the class from that JAR (using JUnit), and then the results are sent back to the user.
The solution is as follows:
Implement basic versions of the to-be-tested classes so that the tests would compile and run without any submitted JARs. This could (more likely, should) be done by leveraging polymorphism, but is not planned at the moment (this means that the user should most likely extend the “basic” version of the class himself, locally, before sending the class to be tested).
Extend URLClassLoader and re-work the classloading logic.
Do the necessary checks (is the class already loaded) that exist in the default implementation.
Try to findClass yourself, if it’s not in your URLs, throw a ClassNotFoundException.
If the class has already been loaded, or if has been loaded just now from some URL, return the class.
If the class has neither been loaded before (by this classloader), nor is it in one of the URLs, delegate the search to your parent.
If the parent returns the class, return the class, if not, throw ClassNotFoundException (it will actually be thrown by the parent).
For each JAR file with a class that is sent:
Instantiate the custom URLClassLoader
Add the JAR to it and the specific test class as well
Ask it to load the test class. At this point my custom classloading logic kicks in, and it loads my test straight from the disk - anew, without delegating to its parent classloader. Why? It calls defineClass() on my test class, which sets this custom URLClassLoader as the parent of the test class
Give the test class to JUnit, which then instantiates it and begins testing
Once one of my tests are running, each time they reference any class, they will call their own custom classloader’s loadClass(). First they will search the URLs - so in case they’re referencing a to-be-tested class, it will be loaded from there. In case they are referencing some other application class, or a system class, the custom classloading logic will just delegate this call and an already-loaded (presumably) class would be returned.
As I’ve said - I haven’t implemented this yet, and I would really like if you’d point out my mistakes in the comments.
Resources from which I've gathered this information:
http://www.onjava.com/pub/a/onjava/2003/11/12/classloader.html
http://www.devx.com/Java/Article/31614
http://www.onjava.com/pub/a/onjava/2005/01/26/classloading.html
http://docs.oracle.com/javase/7/docs/api/java/lang/ClassLoader.html#loadClass%28java.lang.String,%20boolean%29

You need to make sure that jar you are loading classes from is unknown to JVM system class loader (not on the classpath). Then system class loader will not be loading classes from the jar thus making your custom class loader to load from this jar.

You can fool it.
Cass that you want to reset should not be in the classpath
Can implement a interface or extend a class that is in the classpath (useful to type cast or need to use reflection to 'discover' and call methods, OR the calling class calls another method that has the code the calls the class(es) under test
every time you want to test, rename the jar, load from the jar, using a URL class loader sub class. On init of your loader null is fine.
So if you load a1.jar first time, load from a2.jar next time and make sure a1.jar is moved to another folder

Different behaviour of Class literal and Class.forName for package local classes

Only for example lets consider the class ClassFileAssembler from the sun.reflect package.
This class is a package local class:
class ClassFileAssembler implements sun.reflect.ClassFileConstants {...
So we can not use even its name ClassFileAssembler, we can not import it directly - it will lead to a compiler error.
However we can create a package named sun.reflect in our project and use the ClassFileAssembler name internally in this package - Java compiler will think that we are inside the ClassFileAssembler's package.
If so, why not to try to get a reference to a class object, i.e. ClassFileAssembler.class?
Class<ClassFileAssembler> classFileAssemblerClass = ClassFileAssembler.class;
Unexpectedly this code leads to a run-time error: java.lang.IllegalAccessError: tried to access class sun.reflect.ClassFileAssembler from class sun.reflect.Test.
However we still able to get the ClassFileAssembler class object:
Class<ClassFileAssembler> aClass = (Class<ClassFileAssembler>)Class.forName("sun.reflect.ClassFileAssembler");
It works fine and gives us a full class description.
So, the questions are:
1) What is the difference between techniques, how Class.forName0 retrieves reference to class object, and how .class does it?
2) Why do they have such different security checks?
3) What's the reason to protect .class reference in such way?
4) Do these techniques use different class loaders?

Class.forName don't care about whether a class is package local or not. It is when you attempt to use that class that access is checked. BTW if you do setAccessible(true) you can by pass these access restrictions.
The Reflection library allows you to do many things you cannot do in Java code. The Java has rules as to what you can and cannot do. e.g. you cannot set a final field outside a constructor or more than once. Note: the JVM doesn't have this restriction and at runtime you can use reflections to change it.
The reason this class is package local is to restrict access of the class to code outside this package. This doesn't mean you cannot access it if you really try, but it is less likely you will access it without serious thought being put into it. e.g. when I import classes in my IDE it often suggests classes from com.sun.* which are unlikely to be the right choice. (MY IDE can be set up to ignore these, but I often seem for find some new package I don't want)
The reason Reflections can do this is to support functionality such a Serialization. With Serialization you need to be able to serialize class outside the package of the Serialization library and obtain fields and reset them when deserializing. Reflections is also used by many Inversion of Control libraries though I suspect this is not what they had in mind when they design it.

If you check the javadoc of Class#forName, you will see that:
Note that this method does not check whether the requested class is accessible to its caller.

there is no difference. but you cannot access the static field .class of the package private (no modifier) class ClassFileAssembler.
everyone could access the Class instances, but the fields are protected.
in fact no one designed to protect .class reference this way, it's side effect of protecting other fields.
i dont think so.

Can't access protected member of superclass from same package in different jar

I'm having a strange problem that I can't figure out that popped up when trying to pluginize my program. An additional problem is that I'm not able to create a simple test case, because every time I try it works. There must be some complication I'm missing. But I'll try to describe the situation as clearly as possible, in case it sounds familiar to anyone.
I have a base class called Seed which is part of the main application and loaded by the system classloader. I have a plugin which contains a class Road which is a subclass of Seed. It is loaded at runtime from a separate jar file. The class Road references the field Seed.garden, which is defined as:
protected final Garden garden;
Note that I don't get compilation errors. I also don't get runtime errors when the plugin jar is included on the system classpath. Only when my main application loads the plugin using a new classloader (that has the system classloader as its parent) do I get the error. The error is:
java.lang.IllegalAccessError: tried to access field package.Seed.garden from class package.Road$4
It must have something to do with the fact that the subclass has been loaded by a different class loader than the superclass, but I can't find any official reason why that shouldn't work. Also, like I said, when I try to reproduce the problem with a simple test case (that includes the separate jars, loading the subclass with a different classloader, etc.), I don't get the error.
It also doesn't seem likely that I'm violating the access rules since it works when the classes are loaded by the same classloader, and I don't get compilation errors.
I'm out of ideas! Does anyone recognise this problem, or have some pointers for me for directions in which to look? Help!

OK, so with the help of axtavt and other respondents I figured out what the problem is. The other answers helped, but they didn't get it exactly right, which is why I'm answering my own question. The problem turned out to be the concept of "runtime packages", defined in the Java Virtual Machine specification as follows:
5.3 Creation and Loading
...
At run time, a class or interface is determined not by its name alone, but by a pair: its fully qualified name and its defining class loader. Each such class or interface belongs to a single runtime package. The runtime package of a class or interface is determined by the package name and defining class loader of the class or interface.
...
5.4.4 Access Control
...
A field or method R is accessible to a class or interface D if and only if any of the following conditions is true: ...
R is protected and is declared in a class C, and D is either a subclass of C or C itself.
R is either protected or package private (that is, neither public nor protected nor private), and is declared by a class in the same runtime package as D.
The first clause explains why Road is allowed to access Seed.garden, since Road is a subclass of Seed, and the second clause explains why Road$4 is not allowed to access it, despite being in the same package as Road, since it is not in the same runtime package, having been loaded by a different class loader. The restriction is not actually a Java language restriction, it is a Java VM restriction.
So the conclusion for my situation is that the exception occurs due to a legitimate restriction of the Java VM, and I'm going to have to work around it, probably by making the fields public, which is not a problem in this case since they are final, and not secret, or perhaps by exporting Seed.garden to Road$4 via Road, which does have access.
Thank you everyone for your suggestions and answers!

Sounds like you have a class identity crisis, having two different class loaders loading the same class in the class hierarchy or similar. Read up some on the java class loaders. Here is a good one for introduction, for "class identity crisis" see figure 2: http://www.ibm.com/developerworks/java/library/j-dyn0429/

I should add that Road$4 is an anonymous inner class of Road...
Someone else thought this was a bug as well back in 1998:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4116802
An inner class has no greater access to members of another class than
a top-level class, except for those members that are declared within
an enclosing, enclosed, or sibling class. It is a common misconception
that an inner class has unrestricted access to inherited members of
its enclosing classes. This is not true.
I would probably research that fact a bit more though because this was reported originally against Java 1.2, but I seem to remember from my reading that this is true today as well.
EDIT:
I confirmed this to be true:
http://docs.oracle.com/javase/tutorial/java/javaOO/summarynested.html
The scope for an anonymous inner class is only the point where it is defined. So it will not have access to inherited members, even if the outer class does.

This is permission error, so it depends on the framework you use to run your runtime.
Just to clarify this is indeed this, make the parent member public, and then try to run. In case everything is ok, then restore your code, and according to the runtime you use we need to configure the correct security access.