We Keep Coding

Starting Instrumentation Agent after VM Startup

I was hoping for someone to explain this item since I might be getting this wrong:
I was reading about Java Agent Instrumentation which says that the agent can start after VM startup. So if I want to dynamically replace some class (without brining down the app) is this what I am going to go for using agent-main? Or do I need to do something more here?
I know people might ask "Are you talking about JRebel" - not really because I want to do something simple and JRebel is an overkill.
instrument docs - Java docs for Instrumentation
I understand all the instrumentation overrides, but I am slightly confused how I can hook this agent with -agent argument after the app has started.

First your agent class needs to specify an agentmain method like:
public class MyAgent {
public static void agentmain(final String args, final Instrumentation inst) {
try {
System.out.println("Agent loaded.");
} catch (Exception e) {
// Catch and handle every exception as they would
// otherwise be ignored in an agentmain method
e.printStackTrace();
}
}
}
Compile it and pack it inside a jar-file for example. If you choose the jar-variant then it must specify the Agent-Class key in its manifest-file (MANIFEST.MF). It points to the class implementing the agentmain method. It could look like:
Manifest-Version: 1.0
Agent-Class: package1.package2.MyAgent
If it is located inside those packages, as an example.
After that you can load the agent via the VirtualMachine#loadAgent method (documentation). Note that the mechanism used by those classes are part of the Attach library of Java. They decided, as most users don't need it, to not directly add it to the systems path but you can just add it. It is located at
pathToYourJDKInstallation\jre\bin\attach.dll
And it needs to be somewhere where the system property java.library.path is pointing at. You could for example just copy it to your .../Windows/System32 folder or adjust the property or stuff like that.
As an example, if you want to inject an agent-jar inside another currently running jar, you could use a method like this:
public void injectJarIntoJar(final String processIdOfTargetJar,
final String pathToAgentJar, final String[] argumentsToPass) {
try {
final VirtualMachine vm = VirtualMachine.attach(processIdOfTargetJar);
vm.loadAgent(pathToAgentJar, argumentsToPass.toString());
vm.detach();
} catch (AttachNotSupportedException | AgentLoadException
| AgentInitializationException | IOException e) {
System.err.println("Unable to inject jar into target jar.");
}
}
With the same technique you can inject dll-libraries (if they implement the corresponding agent-methods via the native agent interface) into jars.
Actually, if that helps you, I have written some small library for that kind of stuff some time ago. See Mem-Eater-Bug, the corresponding class is Injector.java and the whole project has a small Wiki.
It has an example showing how to use that technique to manipulate a SpaceInvaders game written as Java application.

So apparently you want to reload classes at runtime. Such that your project can react to changes of the code without restarting.
To achieve this you need to prepare your project and write a very clean architecture, it involves using interfaces, factory-patterns, proxy-patterns and a routine that checks for updates and then destroys and rebuilds all current objects.
Unfortunately this might not be an easy task, but it is doable, depending on the size of your project and the amount of code that should react dynamically to changes.
I found this article helpful, let me explain how it works. You can easily load a class with ClassLoader.loadClass(...) and you can also use that to reload a class, very easy. However at the time you have compiled your code classes are some kind of hardwired already. So your old code will continue to create instances of the old classes although you have reloaded the class.
This is the reason why we need some kind of architecture that allows exchanging the old class with the new class. Also it is pretty obvious that current instances of the old class can not automatically be transferred to the new version as everything could have changed. So you will also need a custom method that collects and rebuilds those instances.
The approach described in the article uses an Interface instead of an actual class in the first place. This allows to easily exchange the class behind that interface without breaking the code that uses the interface.
Then you need a factory where you ask for instances of that Interface. The factory can now check if the underlying class-file has changed, if so it reloads it and obtains a reference to the new class version. It can now always create an instance of the interface which uses the up-to-date class.
The factory, by that, is also able to collect all created instances in order to exchange them later, if the code base has changed. But the factory should reference them using WeakReference (documentation), else you have a big memory leak because the Garbage Collector would not be able to delete instances because the factory holds references to them.
Okay, now we are able to always obtain up-to-date implementations of an Interface. But how can we easily exchange existing instances. The answer is by using a proxy-pattern (explanation).
It is simple, you have a proxy class which is the actual object you are working with. It has all the methods of the Interface and upon calling methods it simply forwards to the real class.
Your factory, as it has a list of all current instances using WeakReference, can now iterate the list of proxies and exchange their real class with a new up-to-date version of the object.
Existing proxies that are used all around your project will now automatically use the new real version as the proxy itself has not changed, only its internal reference to the real target has changed.
Now some sample code to give you a rough idea.
The interface for the objects you want to monitor:
public interface IExample {
void example();
}
The real class, which you want to rebuild:
public class RealExample implements IExample {
#Override
public void example() {
System.out.println("Hi there.");
}
}
The proxy class that you will actually use:
public class ProxyExample implements IExample {
private IExample mTarget;
public ProxyExample(final IExample target) {
this.mTarget = target;
}
#Override
public void example() {
// Forward to the real implementation
this.mRealExample.example();
}
public void exchangeTarget(final IExample target) {
this.mTarget = target;
}
}
The factory you will mainly use:
public class ExampleFactory {
private static final String CLASS_NAME_TO_MONITOR = "somePackage.RealExample";
private final List<WeakReference<ProxyExample>> mInstances;
private final URLClassLoader mClassLoader;
public ExampleFactory() {
mInstances = new LinkedList<>();
// Classloader that will always load the up-to-date version of the class to monitor
mClassLoader = new URLClassLoader(new URL[] {getClassPath()}) {
public Class loadClass(final String name) {
if (CLASS_NAME_TO_MONITOR.equals(name)) {
return findClass(name);
}
return super.loadClass(name);
}
};
}
private IExample createRealInstance() {
return (IExample) this.mClassLoader.loadClass(CLASS_NAME_TO_MONITOR).newInstance();
}
public IExample createInstance() {
// Create an up-to-date instance
final IExample instance = createRealInstance();
// Create a proxy around it
final ProxyExample proxy = new ProxyExample(instance);
// Add the proxy to the monitor
this.mInstances.add(proxy);
return proxy;
}
public void updateAllInstances() {
// Iterate the proxies and update their references
// Use a ListIterator to easily remove instances that have been cleared
final ListIterator<WeakReference<ProxyExample>> instanceIter =
this.mInstances.listIterator();
while (instanceIter.hasNext()) {
final WeakReference<ProxyExample> reference = instanceIter.next();
final ProxyExample proxy = reference.get();
// Remove the instance if it was already cleared,
// for example by the garbage collector
if (proxy == null) {
instanceIter.remove();
continue;
}
// Create an up-to-date instance for exchange
final IExample instance = createRealInstance();
// Update the target of the proxy instance
proxy.exchangeTarget(instance);
}
}
}
And finally how to use it:
public static void main(final String[] args) {
final ExampleFactory factory = new ExampleFactory();
// Get some instances using the factory
final IExample example1 = factory.createInstance();
final IExample example2 = factory.createInstance();
// Prints "Hi there."
example1.example();
// Update all instances
factory.updateAllInstances();
// Prints whatever the class now contains
example1.example();
}

Attaching an agent at runtime requires use of the attach API which is contained in the tools.jar until Java 8 and is contained in its own module starting from Java 9. The location of the tools.jar and the name of its classes is system-dependent (OS, version, vendor) and as of Java 9 it does not exist at all but must be resolved via its module.
If you are looking for an easy way to access this functionality, try out Byte Buddy which has a subproject byte-buddy-agent for this. Create a Java agent as you are used to it but add anAgent-Main entry where you would put your Pre-Main in the manifest. Also, name the entry method agentmain, not premain.
Using byte-buddy-agent, you can write a program:
class AgentLoader {
public static void main(String[] args) {
String processId = ...
File agentJar = ...
ByteBuddyAgent.attach(processId, agentJar);
}
}
and you are done.

Issue with ASM getMergedType and getCommonSuperClass

I use ASM to update the class stack map, but when asm getMergedType, the following exception occurs:
java.lang.RuntimeException:
java.io.IOException: Resource not found for IntefaceImplA.
If without asm modify the class method, it does work fine.
I have defined two interfaces A and B: IntefaceImplA and
IntefaceImplB.
My environment source code:
IntefaceA.java
public interface IntefaceA {
void inteface();
}
IntefaceImplA.java
public class IntefaceImplA implements IntefaceA {
#Override
public void inteface() {
}
}
IntefaceImplB.java
public class IntefaceImplB implements IntefaceA {
#Override
public void inteface() {
}
}
Test.java
public class Test {
public IntefaceA getImpl(boolean b) {
IntefaceA a = b ? new IntefaceImplA() : new IntefaceImplB();
return a;
}
}
Main.java
public class Main {
public static void main(String args[]) {
....
if (a instance of Test) {
..
...
}
}
}
After I compiled a runner jar, and delete the IntefaceImplA.class and IntefaceA.class manually from the jar. why i wanna to delete those classes files, since the spring always like to do this stuff.
the runner jar can be run normal without ASM, but use Asm will occur exception. since the asm wanna to getMergedType for IntefaceImplA and IntefaceImplB, but IntefaceImplA was deleted by me.
After investigate the ASM ClassWriter source code i found below code:
protected String getCommonSuperClass(String type1, String type2)
{
ClassLoader classLoader = this.getClass().getClassLoader();
Class c;
Class d;
try {
c = Class.forName(type1.replace('/', '.'), false, classLoader);
d = Class.forName(type2.replace('/', '.'), false, classLoader);
} catch (Exception var7) {
throw new RuntimeException(var7.toString());
}
if(c.isAssignableFrom(d)) {
return type1;
} else if(d.isAssignableFrom(c)) {
return type2;
} else if(!c.isInterface() && !d.isInterface()) {
do {
c = c.getSuperclass();
} while(!c.isAssignableFrom(d));
return c.getName().replace('.', '/');
} else {
return "java/lang/Object";
}
}
Actually, I deleted the related class file, the classloader cannot find the class. but without asm the Program does work normal.
Should I enhance the override to the getCommonSuperClass method, if occur exception then return java/lang/Object for it? that's funny

Generally, overriding getCommonSuperClass to use a different strategy, e.g. without loading the class, is a valid use case. As it’s documentation states:
The default implementation of this method loads the two given classes and uses the java.lang.Class methods to find the common super class. It can be overridden to compute this common super type in other ways, in particular without actually loading any class, or to take into account the class that is currently being generated by this ClassWriter, which can of course not be loaded since it is under construction.
Besides the possibility that either or both arguments are classes you are currently constructing (or changing substantially), it might be the case that the context of the code transforming tool is not the context in which the classes will eventually run, so they don’t have to be accessible via Class.forName in that context. Since Class.forName uses the caller’s context, which is ASM’s ClassWriter, it is even possible that ASM can’t access the class despite it is available in the context of the code using ASM (if different class loaders are involved).
Another valid scenario is to have a more efficient way to resolve the request by using already available meta information without actually loading the class.
But, of course, it is not a valid resolution to just return "java/lang/Object". While this is indeed a common super type of every argument, it isn’t necessarily the right type for the code. To stay with your example,
public IntefaceA getImpl(boolean b) {
IntefaceA a = b ? new IntefaceImplA() : new IntefaceImplB();
return a;
}
the common super type of IntefaceImplA and IntefaceImplB is not only required to verify the validity of assigning either type to it, it is also the result type of the conditional expression, which must be assignable to the return type of the method. If you use java/lang/Object as common super type, a verifier will reject the code as it can’t be assignable to IntefaceA.
The original stackmap, very likely reporting IntefaceA as common super, will be accepted by the verifier as that type is identical to the method’s return type, so it can be considered assignable, even without loading the type. The test, whether either, IntefaceImplA and IntefaceImplB, is assignable to that specified common type, might be postponed to the point where these types are actually loaded and since you said, you deleted IntefaceA, this can never happen.
A method whose declared return type is absent, can’t work at all. The only explanation of your observation that “without asm the program does work normal”, is, that this method was never invoked at all during your test. You most probably created a time bomb in your software by deleting classes in use.
It’s not clear why you did this. Your explanation “since the spring always like to do this stuff” is far away from being comprehensible.
But you can use the overriding approach to get the same behavior as with the unmodified code. It just doesn’t work by return java/lang/Object. You could use
#Override
protected String getCommonSuperClass(String type1, String type2) {
if(type1.matches("IntefaceImpl[AB]") && type2.matches("IntefaceImpl[AB]"))
return "IntefaceA";
return super.getCommonSuperClass(type1, type2);
}
Of course, if you deleted more class files, you have to add more special cases.
An entirely different approach is not to use the COMPUTE_FRAMES option. This option implies that ASM will recompute all stack map frames from scratch, which is great for the lazy programmer, but implies a lot of unnecessary work if you are just doing little code transformations on an existing class and, of course, creates the requirement to have a working getCommonSuperClass method.
Without that option, the ClassWriter will just reproduce the frames the ClassReader reports, so all unchanged methods will also have unchanged stack maps. You will have to care about the methods whose code you change, but for a lot of typical code transformation tasks, you can still keep the original frames. E.g. if you just redirect method calls to signature-compatible targets or inject logging statements which leave the stack in the same state it was before them, you can still keep the original frames, which happens automatically. Note the existence of the ClassWriter(ClassReader,int) constructor, which allows an even more efficient transfer of the methods you don’t change.
Only if you change the branch structure or insert code with branches, you have to care for frames. But even then, it’s often worth learning how to do this, as the automatic calculation is quiet expensive while you usually have the necessary information already when doing a code transformation.

instantiating a Scala class using reflection Java's `newInstance`

For some special use-case I have a small utility to load Java classes from jars using a dynamic class loader DynamicClassLoader. This works fine for Java classes contained in jars. Loading Scala classes from a jar also works without problems. However, instantiating the loaded Scala class leads to the following exception. It looks like the Scala class has private default constructor? Note the compiled Scala class name ending with $
java.lang.IllegalAccessException: Class XXX can not access a member of class ScalaClassYYY$ with modifiers "private"
The snippet below illustrates the idea of what I'm trying to achieve and gives a bit more context. The exception happens at the annotated line:
// deploy and register the new code
byte[] jarBytes = (byte[]) ((Object) message.getAttachment("jar"));
String registerClassName = message.getAttachment("register");
logger.debug("the register is '" + registerClassName + "'");
DynamicClassLoader loader = new DynamicClassLoader(jarBytes);
Class<?> registerClass = loader.lookUp(registerClassName);
// ===> this is where the java.lang.IllegalAccessException happens
IRegisterExecutor registerExecutor = (IRegisterExecutor) registerClass.newInstance();
registerExecutor.register();
Any ideas how to fix?

Obviously, you need to make the default constructor public (it won't work for Java classes without a public default constructor either). E.g.
class ScalaClassYYY() {
...
}
or if you want primary constructor to take some arguments,
class ScalaClassYYY(arg1: Int) {
def this() = this(0)
}
But from
Note the compiled Scala class name ending with $
it seems like you are actually trying to instantiate a Scala object:
object ScalaClassYYY { ... }
In this case, you shouldn't create a new instance and instead use the existing one:
(IRegisterExecutor) registerClass.getField("MODULE$").get(null);
EDIT:
I don't see in your answer how you add a default public constructor to a Scala class that does NOT require any parameters.
A class (not an object) that doesn't require any parameters has a default public constructor already (my first example).
Actually in Java all classes by default offer a public default constructor
No. Only those classes which have no constructors which take arguments.
remove the "(it won't work for Java classes without a public default constructor either)" because it is wrong
The documentation for Class.newInstance() says
IllegalAccessException - if the class or its nullary constructor is not accessible.
So I am pretty sure it's right. If it does work for Java classes without a public default constructor, this seems to be a major bug in the class loader you use. You can test it with a Java class which looks like this:
public class TestClass implements IRegisterExecutor {
public TestClass(int dummy) {}
// some implementation for IRegisterExecutor methods to get it to compile
}

How do I use Powermockito to mock the construction of new objects when testing a method in an anonymous class?

I woud like to write a JUnit test to verify that the code below uses a BufferedInputStream:
public static final FilterFactory BZIP2_FACTORY = new FilterFactory() {
public InputStream makeFilter(InputStream in) {
// a lot of other code removed for clarity
BufferedInputStream buffer = new BufferedInputStream(in);
return new CBZip2InputStream(buffer);
}
};
(FilterFactory is an interface.)
My test thus far looks like this:
#Test
public void testBZIP2_FactoryUsesBufferedInputStream() throws Throwable {
InputStream in = mock(InputStream.class);
BufferedInputStream buffer = mock(BufferedInputStream.class);
CBZip2InputStream expected = mock(CBZip2InputStream.class);
PowerMockito.spy(InputHelper.BZIP2_FACTORY); // This line fails
whenNew(BufferedInputStream.class).withArguments(in).thenReturn(buffer);
whenNew(CBZip2InputStream.class).withArguments(buffer).thenReturn(expected);
InputStream observed = InputHelper.BZIP2_FACTORY.makeFilter(in);
assertEquals(expected, observed);
}
The call to PowerMockito.spy raises an exception with this message:
org.mockito.exceptions.base.MockitoException:
Mockito cannot mock this class: class edu.gvsu.cis.kurmasz.io.InputHelper$1
Mockito can only mock visible & non-final classes.
What should I be using instead of PowerMocktio.spy to set up the calls to whenNew?

The message is pretty obvious: You can't mock non-visible and final classes. Short answer : Create a named class of your anonymous one, and test this class instead!
Long answer, let's dig why !
An anonymous class is final
You instantiate an anonymous class of FilterFactory, when the compiler sees an anonymous class, it creates a final and package visible class. So the anonymous class is not mockable through standard mean i.e. through Mockito.
Mocking anonymous class : possible but BRITTLE if not HACKY
OK, now suppose you want to be able to mock this anonymous class through Powermock. Current compilers compile anonymous class with following scheme :
Declaring class + $ + <order of declaration starting with 1>
Mocking anonymous class possible but brittle (And I mean it)
So supposing the anonymous class is the eleventh to be declared, it will appear as
InputHelper$11.class
So you could potentially prepare for test the anonymous class:
#RunWith(PowerMockRunner.class)
#PrepareForTest({InputHelper$11.class})
public class InputHelperTest {
#Test
public void anonymous_class_mocking works() throws Throwable {
PowerMockito.spy(InputHelper.BZIP2_FACTORY); // This line fails
}
}
This code will compile, BUT will eventually be reported as an error with your IDE. The IDE probably doesn't know about InputHelper$11.class. IntelliJ who doesn't use compiled class to check the code report so.
Also the fact that the anonymous class naming actually depends on the order of the declaration is a problem, when someone adds another anonymous class before, the numbering could change.
Anonymous classes are made to stay anonymous, what if the compiler guys decide one day to use letters or even random identifiers!
So mocking anonymous classes through Powermock is possible but brittle, don't ever do that in a real project!
EDITED NOTE : The Eclipse compiler has a different numbering scheme, it always uses a 3 digit number :
Declaring class + $ + <pad with 0> + <order of declaration starting with 1>
Also I don't think the JLS clearly specify how the compilers should name anonymous classes.
You don't reassign the spy to the static field
PowerMockito.spy(InputHelper.BZIP2_FACTORY); // This line fails
whenNew(BufferedInputStream.class).withArguments(in).thenReturn(buffer);
whenNew(CBZip2InputStream.class).withArguments(buffer).thenReturn(expected);
InputStream observed = InputHelper.BZIP2_FACTORY.makeFilter(in);
PowerMockito.spy returns the spy, it doesn't change the value of InputHelper.BZIP2_FACTORY. So you would need to actually set via reflection this field. You can use the Whiteboxutility that Powermock provide.
Conclusion
Too much trouble to just test with mocks that the anonymous filter uses a BufferedInputStream.
Alternative
I would rather write the following code:
An input helper that will use the named class, I don't use the interface name to make clear to the user what is the intent of this filter!
public class InputHelper {
public static final BufferedBZIP2FilterFactory BZIP2_FACTORY = new BufferedBZIP2FilterFactory();
}
And now the filter itself :
public class BufferedBZIP2FilterFactory {
public InputStream makeFilter(InputStream in) {
BufferedInputStream buffer = new BufferedInputStream(in);
return new CBZip2InputStream(buffer);
}
}
Now you can write a test like this :
#RunWith(PowerMockRunner.class)
public class BufferedBZIP2FilterFactoryTest {
#Test
#PrepareForTest({BufferedBZIP2FilterFactory.class})
public void wraps_InputStream_in_BufferedInputStream() throws Exception {
whenNew(CBZip2InputStream.class).withArguments(isA(BufferedInputStream.class))
.thenReturn(Mockito.mock(CBZip2InputStream.class));
new BufferedBZIP2FilterFactory().makeFilter(anInputStream());
verifyNew(CBZip2InputStream.class).withArguments(isA(BufferedInputStream.class));
}
private ByteArrayInputStream anInputStream() {
return new ByteArrayInputStream(new byte[10]);
}
}
But could eventually avoid powermock stuff for this test scenario if you force the CBZip2InputStream to only accept BufferedInputStream. Usually using Powermock means something is wrong with the design. In my opinion Powermock is great for legacy softwares, but can blind developers when designing new code; as they are missing the point of OOP's good part, I would even say they are designing legacy code.
Hope that helps !

Old post, but you don't need to create a named class - use wildcards instead as mentioned in this post powermock mocking constructor via whennew() does not work with anonymous class
#PrepareForTest(fullyQualifiedNames = "com.yourpackage.containing.anonclass.*")

You need to run the test using the PowerMockito runner, and you need to tell the framework which class(es) should have custom behaviour. Add the following class annotations on your test class:
#RunWith(PowerMockRunner.class)
#PrepareForTest({ BufferedInputStream.class })

I just came around the same problem. So according to the documentation of constructor mocking you need to prepare the class, which will create the evil class(es). In your case the evil classes are BufferedInputStream and CBZip2InputStream, and the creator of them is an anonymous class, which cannot be defined in PrepareForTest annotation. So I had to do the same as you did (hmm, just saw your comment), I moved the anonymous class to named class.

Pushing to stack reference to the class begin generated with ASM

I'm using ASM Java library to generate a class X from scratch. Inside one of the static methods of this class I need to push a reference to X.class. Since there isn't yet a X.class I can't use visitLdcInsn. Is there a way to do it?
Well, it's possible (and I'm currently using it) to generate the following code (new X().getClass()), but I'm sure that's not the cleanest way to do it.

With generated code you usually don't need to push the class onto the stack. Anything you can do with a method call is usually available in byte code.
Say you have to call a method with a class, you can push it onto the stack whether it exists or not.
Something I use is the ASMifier. This is useful because you can start with a class which compiles and does what you want as a template and get it to dump all the code needed to recreate the class. This means you don't really need to write most of the code yourself.
public class Main {
public static void main(String... args) throws IOException {
ASMifierClassVisitor cv = new ASMifierClassVisitor(new PrintWriter(System.out));
ClassReader cr = new ClassReader("X");
cr.accept(cv, 0);
}
}
class X {
{
System.out.println("Inside class "+X.class);
}
}
prints
// lots of code
mv.visitLdcInsn(Type.getType("LX;"));
// more code.