For a project, we need a way to run user scripts that can come with attached JAR files with additional classes.
What are my options when I want to write a couple of tests to make sure normal script don't leave anything dangling behind?
I specifically need to know: Are all classes from the attached JARs "unloaded"?
Note: I'm not looking for the 100% super-watertight solution that works across all versions of Java from 1.0 to 7. Right now, I just need to be better than "I have no idea".
The likely best option is to ensure your loaded jars are loaded by a specific class loader, and then to discard that class loader (after discarding all the objects).
As far as unit testing the unloading, if you go with this option, you need to extend your testing framework and customized class loaders to have a "create class loader on demand" flag. Then you load the class once with the flag on, discard the class loader, and attempt to load the class again with the flag off. If the class is truly not reachable, the second attempt should throw a class not found exception. You then wrap your unit tests to pass if they fall into the exception, and fail if they succeed in hitting the line after the second load attempt.
If you are disposed to use more than pure-Java tools, an OSGi container might be a consideration. Most of the established OSGi container implementations explicitly test class unloading.
I wouldn't try to unit test this. Instead, I'd run the JVM with -XX:-TraceClassUnloading and look to see if the classes in question show up in the trace output.
It looks like what you want to test is that hose scripts don't have a classloader leak.
To do that, I'd create a WeakReference to the ClassLoader used to load that JAR, then run the script, then call System.gc() and afterwards assertNull(reference.get())
This depends entirely on the way you allow the scripts to run. Do they have access to the classes of the rest of the application?
The typical way to leak memory in Java is to have a static reference. A static reference is only static within the ClassLoader of the class that contains it. So, if you load your user scripts using a ClassLoader you manage yourself (and you should do this anyway), then the references (static or not) inside will be eligable for GC as soon as your classloader itself it.
The only way they could work around this, is to add a reference to one of their objects into one of yours. So you have to be very careful with the API you expose. Another way is if they would make a static reference to their class in a class from another ClassLoader.
I don't see a way to fully automate testing for this. But I suppose you could trace the class unloading with any decent profiler.
Related
Is there a way to configure the JVM to block instances of a class being created?
I'd like to do this to ensure no service running in the JVM is allowed to create instances of a class that has been identified as a security risk in a CVE, lets call that class BadClass.
NOTE: I'm looking for a general solution, so the following is purely additional information. I would normally address this by switching the library out, or upgrading it to a version that doesn't have the exploit, but it's part of a larger library that wont be addressing the issue for some time. So I'm not even using BadClass anywhere, but want to completely block it.
I do not know a JVM parameter, but here's some alternatives that might pout you in a position that solve your requirements:
You can write a CustomClassLoader that gives you fine control on what to do. Normal use cases would be plugin loading etc. In your case this is more security governance on devops level.
If you have a CICD pipeline with integration tests you could also start the JVM with -verbose:class parameter and see which classes are loaded when running your tests. Seem a bit hacky, but maybe suits your use case. Just throwing everything into the game, it's up to you judging about the best fit.
Depending on your build system (Maven?) you could restrict building applications just on your private cached libs. So you should have full control on it and put a library - review layer in between. This would also share responsibility between devs and the repository admins.
A distinct non-answer: Do not even try!
What if that larger library that has this dependency wants to call that method? What should happen then?
In other words, what is your blocking supposed to do?
Throw some Error instance, that leads to a teardown of the JVM?
Return null, so that (maybe much later) other code runs into a NPE?
Remember: that class doesn't exist in a void. There is other code invoking it. That code isn't prepared for you coming in, and well, doing what again?!
I think there are no good answers to these questions.
So, if you really want to "manipulate" things:
Try sneaking in a different version of that specific class into your classpath instead. Either an official one, that doesn't have the security issue, or something that complies to the required interface and that does something less harmful. Or, if you dare going down that path, do as the other answer suggests and get into "my own classloader" business.
In any case, your first objective: get clean on your requirements here. What does blocking mean?!
Have you considered using Java Agent?
It can intercept class loading in any classloader, and manipulate it's content before the class is actually loaded. Then, you may either modify the class to remove/fix it's bugs, or return dummy class that would throw error in static initializer.
I am looking for a way to replace a jar with old functionality with a new one without stopping the whole application. I have googled for this a little bit (Unloading classes in java?, Dynamically loadable and unloadable application modules in Java - how?), and have found 3 possible solutions:
OSGI, which looks too complicated for my purposes: class unloading is the only thing I need from this powerful specification.
Own implementation, which implies creating own Classloader, loading jars with its help, and when time comes to dismiss the jar - expose Classloader instance to GC and hope that GC will remove it together with all loaded jars, which may not happen immediately.
Copy the stuff application servers do :) That is the purpose of my question.
Is there any explanation on how Tomcat unloads the classes (wars)? Is it possible to reuse this in my application?
PS
Just while I was writing this post 2 strange ideas came into my head:
may be we can start another instance of application with new set of jars and switch input and output data flows from old one to new one. If there is a place to preserve state - it can be more quick than full restart.
may be Spring may help? It can dynamically register beans, but this looks like somewhat ugly solution, and it cannot unload them anyway; may be, Spring Boot may help?
A class can be unloaded only when it and its class loader are unreachable and thus eligible for garbage collection. Since the class loader can reach every class it loaded and every class loaded by a class loader can reach its class loader and every object which is an instance of a class can reach its class, this means that the class loader, all the classes it loaded and all the objects which are an instance of those classes must all be unreachable and eligible to be garbage collected. Only then, can a class be "unloaded" when that complete object graph is garbage collected.
I can only partially answer you questions, however better than nothing... You are right, modularity is a quite a tedious topic (at least until there's Java 9).
Ad idea 2 (Spring): I already spent some thoughts on Spring Boot over here and came to the conclusion that I'll stick either to OSGI (it's worth getting into it) or to a pure Microservice architecture.
I suppose that you are not doing whole redeployment of application, but don't want to restart whole application every time while you are developing it
If you want to change functionality of methods without changing class or method signatures you can use Spring Loaded
Add dependency to your project https://mvnrepository.com/artifact/org.springframework/springloaded/1.2.6.RELEASE
Set JVM option -javaagent:path\springloaded-version.jar -noverify
This option can be set in several ways depending on how you run tomcat
I have Broadleaf eccomerce web site and I added this option to MAVEN_OPTS in batch script that runs embedded tomcat or you can configure that in Run configuration for Tomcat VM options
read http://docs.spring.io/spring-boot/docs/current/reference/html/howto-hotswapping.html
I am looking for a way to reload a class into Java at runtime. The motivation is to make debugging more efficient. The application is a typical client/server design that synchronously processes requests. A "handler" object is instantiated for each request. This is the only class I intend to dynamically replace. Since each request deals with a fresh instance, reloading this class won't have any side-effects. In short, I do not want to restart the entire application every time there is a change to this module.
In my design, the Java process becomes aware that a .class file has been updated in the classpath in between requests. When this happens, the "handler" class is unloaded and a new one is loaded.
I know I can use the classLoader interface to load in a new class. I seem to be having trouble finding the proper way of "unloading".
Classes will be unloaded and garbage collected like any other object, if there is no remaining reference to them. That means there must be no reachable instance of the class (as loaded by that particular classloader instance) and the classloader instance itself must be eligible for garbage collection as well.
So basically, all you have to do is to create a new classloader instance to load the new version of the class, and make sure that no references to instances of the old version remain.
I believe that you actually need to have a hierarchy of classloaders, and in order to reload you actually get rid of the low level classloader (by normall GC means), and hence all the classes it loaded. So far as I know this technique is used by Java EE app servers for reloading applications, and there's all manner of fun results when framework code loaded in one classloader wants to use classes loaded somewhere else.
As of 2015 also java's class reloading is a missing feature.
Use OSGi to create a class reloading application.
Use jrebel for testing. There are a few others which does the same thing.
Use application server and externalize the parts which you want to reload into a separate web application. Then keep deploying/undeploying. You will eventually get some perm gen space overflow kind of errors due to dangling old ClassLoader instances.
Use a script runner to execute parts of changeable code. JSR-223 Java Scripting API support for the scripting language "Java".
I had written a series about class reloading. But all of those methods are not good for production.
The blog and source codes in google sources
IMHO this class reloading is messy in java and its not worth trying it. But I would very much like this to be a specification in java.
I'm quite restricted in the platform I'm currently working on (JDK 1.3, BD-J). One JAR file I would like to use attempts to perform a self-integrity check on load and if it fails it goes into an inoperable state. It's quite difficult to find out why this is happening but most sources point to that it cannot find/access it self through the BD-J structure, so it dies.
This rules out using it at load time and instead to load it in the application itself. This is quite a large library so I have to create quite an amount of interfaces so I can cast a loaded object to it and potentially use it. This is where my problem lies.
The interfaces are loaded on normal load time and the library is then loaded during run time and casted to the previously loaded interfaces, is this a problem? I'm receiving ClassCastException
I've based the interfaces off the libraries public methods as best I can, but when I attempt to cast to an interface I receive the ClassCastException. Note: It all loads fine, I can access constructors and read the method names. Just when casting it for it to be useable it fails.
The interface packages are different in my project to that of the toolkit, does this matter?
I'm running out of ideas, is there something I have overlooked?
Thanks.
I'm not sure I fully grok what your problem is - maybe some more details about what the class hierarchy looks like would help in figuring out the situation. From what you wrote I can guess two possible scenarios:
.1. The classes you want to use do not implement any interface.
In this case no matter what you name your interfaces, it will not work, since the classes you're loading do not implement them. You're stuck with using reflection if you can't load that jar as part of the boot classpath.
.2. The classes you want implement some interface that you're trying to replicate.
In this case you interface implementation must match the exact qualified name of the interface the classes are implementing. Normally, when loading the classes from the jar, the class loader will pick up the interfaces from the system class loader first, thus loading your interfaces, and everything should work.
If they use some crazy internal class loader, though, they might still try to load their own interfaces. You could try to figure out if that's the case by using "-XX:+TraceClassLoading", although I don't know if the 1.3 jre will understand that option.
Now if you're willing to experiment more, you could also try another approach. Write your own class loader that loads both the classes from that jar and the code you want to run. That way, your code would be able to directly refer to the classes in that jar, but to start your application the "main" method will have to be one that initializes this classloader, loads the "real" main class using reflection, and executes its main() method also via reflection.
Most probably the classes are loaded by different class loaders. http://mindprod.com/jgloss/classloader.html may give some idea.
I have a Tomcat with some applications running. I cannot restart the Tomcat but I would like to monitor the usage of class files.
I would like to log if a specified class is used. Is this possible?
How could I accomplish it?
Class Usage: If an object for this class is instantiated or methods are called etc.
Clarification: I cannot restart the application. Every solution with recompiling the running code are not acceptable. That makes the problem so hard.
Remote debugging/JMX is not enabled yet. It would be a similar effort like recompiling the application to activate it.
Platform is RHEL, 64 Bit.
I would like to log if a specified class is used; i.e. if an object for this class is instantiated or methods are called etc.
A memory profiler would tell you if a reachable instance of a class exists at the instant you run the profiler. An execution profiler could tell you that a method or constructor is called during some interval ... though it might also miss a call, due to the way that profilers work.
The webapp's classloader could in theory tell you if a class has been loaded, but I doubt there is a way to call the classloader's method that doesn't involve a restart. Also, there is no way to know if a method has EVER been called or an instance has EVER been created apart from adding monitoring hooks to the class. And adding those hooks would entail a restart.
And of course there are other ways that a class could be "used" that don't entail constructing instances or calling its methods.
So depending on what you are really trying to figure out, you may be out of luck.
I think you need profiling for that.
Profiler will allow you to see which classes are used.
Or the programmers fav - System.out.println :)
If you have access to the source files, you could simply put some log statements in the constructor of the class (assuming instantiation implies class use).
If object instantiation is not what you mean by "the usage of class files", then perhaps you mean that you'd like to know when a class is loaded by the JVM? If so, static initialization blocks may be able to help; you could put the log statements there.
If you are on Solaris or OS X, then a DTrace probe might on option.
Other than that, if you cannot attach a debugger or profiler or other monitoring tool to the already running JVM (which depends on options you would have needed to specify on startup), you are out of luck.
If you want to know it programatically then I don't think it is trivially possible.
But you can do some reflection to get to know, by calling this method.
ClassLoader#findLoadedClass()
If the method returns null, that means the class is not loaded and hence not in use.
The problem of course, is that the method is protected, so you need to use reflection.
You can define a static block in the class
class X {
static {
System.out.println("class X has been loaded");
}
...
}
that will be called once when the class object is loaded.
If you cannot edit the code, and you have a JRE 6, you might try the jmap tool
jmap -histo <pid>
it will print a histogram of your heap, including all loaded classes.
Using jps you can find out the pid
With the JRockit JVM you can start your application with -Xverbose:class=info to get information about class loading.