I'm currently developing a Java Agent in order to facilitate the dynamic instrumentation of new and legacy Java Applications.
It occurred to me that, as far as IDE debugging is concerned, Java Agents could be perhaps considered a special case as they are required to be injected into a target JVM process in order to be ran. Which thus naturally gives rise to the question of how one would go about debugging, testing and profiling an Agent-type application.
A cursory search for existing solutions turned up a few command line based options (i.e YourKit, JIP, etc) however many of them are ALSO Java Agents under the hood. Which if utilized would lead to, at least in my view, the rather odd scenario of an Agent debugging/profiling another Agent. I am aware that Agents can be stacked in a hierarchical arrangement, however i'm unsure if Agent Applications can be debugged by stacking Agents in this manor.
As stated in Java How To ... The -javaagent: Option:
An agent is just an interceptor in front of your main method, executed
in the same JVM and loaded by the same system classloader, and
governed by the same security policy and context.
The name is misleading, since the word agent usually suggests
something working remotely and separately from the primary entity. But
it turns out the java agent as used in -javaagent: is much simpler
than that.
One java application may have any number of agents by using
-javaagent: option any number of times. Agents are invoked in the same order as specified in options.
Each agent may also take String-valued args. I guess that's the reason
why we have to use this option multiple times for multiple agents.
Otherwise, we could've just done something like:
-javaagent agent1.jar:agent2.jar
, which is incorrect.
So, by putting the profiler agent (e.g. YourKit, JIP, etc.) before your own agent will give the debugging control to you.
Related
EDIT: Oracle accept my bug report requesting an enhancement as JDK-8138944 : Support command line arguments to the JVM passed to self-contained app launchers.
The problem
My team is working on an open source Java SE project, ImageJ, that currently has a custom native launcher written in cross-platform C. We would like to move away from this launcher, switching to a more modern, industry standard and maintainable deployment mechanism. JavaFX self-contained applications are the most promising solution we have found so far.
One great feature of ImageJ's current native launcher is its ability to customize how the JVM is launched. For example, you can write:
ImageJ --debugger=8000 myFile.png
And the launcher will invoke the JVM with flag:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=localhost:8000
while keeping the myFile.png as an argument to the Java main class.
But from the docs, we cannot see a way to accomplish something similar with the JavaFX packaging tool.
Considerations
I know that the UserJvmOptionsService provides a means to configure how the JVM is started (via Java Preferences API internally, which the JavaFX launcher reads before JVM startup). This is nice for providing the user with a friendly dialog box where they can tweak maximum heap size and other common parameters. Certainly we could add a remote debugging toggle and port settings to such a dialog box, and/or support JVM configuration via the CLI—but it would then require a restart of the application.
Ideally we would not need to support this scenario at all, and instead would handle all command line arguments after the JVM starts up, in Java. For example, in most cases, system properties of the form -Dfoo=bar can be supported by simply parsing the arg and setting the system property at runtime, as long as it is done early enough in the application's startup cycle. But there are clearly many cases where doing it after JVM startup is too late:
The debugging example above (you can't enable remote debugging after startup).
Heap size tuning
Garbage collection tuning
Verbose class loading (-verbose)
Interpreted mode (-Xint)
Lots of other examples
Our users expect to be able to pass these settings on the CLI, and have the Java runtime operate accordingly—and in the case of ImageJ, it is particularly important for backward compatibility.
Possible solutions
We could retain the native C launcher, replacing the native executable that the Java packaging tool installs. But this strikes me as highly fragile, and largely defeats the purpose of switching to JavaFX deployment, since we would still need to maintain, build and test the custom native launcher across several different platforms.
Alternately, we could have the application main class be a very thin CLI option parser, which then spawns a second instance of the JVM. This would keep the bootstrapping logic in pure Java, which would be far more maintainable than the current native C code, while fully leveraging the JavaFX deployment scheme's cross-platform bundling capabilities. But that seems like a big hack with potentially challenging side effects.
Finally, the question
Has anyone achieved support for JVM CLI arguments with JavaFX self-contained application deployment? If so, how did you accomplish it? If not, any alternative suggestions?
You can modify the launch arguments to the JVM by modifying the jvm user overrides config file that the API writes to:
• Mac ~/Library/Application Support/[app.preferences.id]/packager/jvmuserargs.cfg
• Windows C:\Users[username]\AppData\Roaming[app.preferences.id]\packager\jvmuserargs.cfg
• Linux ~/.local/[app.preferences.id]/packager/jvmuserargs.cfg
NOTE: This is an internal implementation detail and subject to change.
Rather than having any kind of two-phase launch system, where you see what options you have and then launch a replacement JVM, I would suggest you create your own native launcher by copying and editing the platform java.c launcher. You can see what they are doing in the open JDK project.
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/914cd9b482c8/src/share/bin/java.c
There are quite a few places where they are looking for various options and converting them into init arguments for the JVM itself. Take a look at the functions ParseArguments, AddApplicationOptions, TranslateApplicationArgs, CheckJvmType, AddOption etc.
I am no C programmer but I have maintained my own launcher on quite a few occasions: such as a launcher that loaded a specific class path from a one-entry-per-line text file that could be checked into source control and one that used a different entry point to main(). It doesn't change that much and you can isolate your changes quite easily, so that they are easy to reapply on later versions of java.c. For your purposes, you would not really need to change every time someone makes a change to java.c, you only really need to change it when JavaVMInitArgs or some other critical aspect of the invocation interface changes.
I am sure that if you proposed and contributed a more elegant option-handling solution, maybe one that behaved differently when argv[0] is not 'java', then maybe the open jdk team would adopt your approach and maintain it for you, or for all us. I am sure there are lots of developers out there needing features like these.
I've been looking at various java performance monitoring tools.
I worked out a solution that uses perf4j + aspectj, so that I can define which classes to monitor with an aspectj pointcut expression. However aspectj also (AFAIK), does not allow you to change the pointcut expression at runtime. So if I need to monitor new classes I would have to change the aop.xml and recompile (or atleast restart the app if using AspectJ load time weaving).
I have not found anything so far that can instrument classes at runtime without needing a restart of the application. Is there a tool/technology out there which can do that?
These types of tools typically don't instrument classes at runtime. Instead they use the JVMTI interface (if you don't understand what this is, google it).
Prime examples of products that use it:
Yourkit, JProfiler
There's a few open source tools, but I haven't really found any of them to be nearly as polished as their commercial counterparts.
Found a perfect little library BTrace, which does exactly what I needed. It works by manipulating the byte code of the instrumented class at runtime, no need to restart your application. Check out the user guide to see how easy it is to set it up.
Another thing worth mentioning about Btrace is that, the authors have made a lot effort to make it safe to work in a production environment.
To guarantee that the tracing actions are "read-only" (i.e., the trace actions don't change the state of the program traced) and bounded (i.e., trace actions terminate in bounded time), a BTrace program is allowed to do only a restricted set of actions.
I suggest using Eclipse's (or NetBeans) debugger/profiler.
It is very powerful.
There is also an entire project dedicated to these tools:
http://www.eclipse.org/projects/project.php?id=tptp.performance
and some interesting plugins:
http://code.google.com/a/eclipselabs.org/p/jvmmonitor/
As Matt said, YouKit Java Profiler can instrument Java byte-code. YourKit allows to write own probes and insert them into the running Java apps. Probe API allows to access method parameters, return values, catch exceptions, etc.
The docs is here:
http://www.yourkit.com/docs/11/help/probes.jsp
We have a situation where we would like to be able to create a zip file containing as much information as possible about a currently running Java program (which may be on its way down) to allow for post-mortem forensic analysis. We currently deploy to Java 5 but Java 6 features are interesting too.
So far I've thought of:
A programmatically generated thread dump. This appears to work better in Java 6.
The logged log events for the last X minutes. We currently use logback or java.util.logging.
Some serialized objects.
External environment - all system properties.
What else would be useful of JVM information?
Would it be possible in a generic way to walk the call stacks and see the arguments? (or does this require JVMTI or equivalent). It is a IBM JVM so we cannot use jvisualvm and the Attach API.
You could go all the way and capture a full heap dump? I realize you are on an IBM JVM, but this page seems to indicate there is a way.
I am writing a profiler with the aid of the JVM TI.
In C++ I have written a simple agent, which writes the information collected to a socket. With Java Swing I have built a simple GUI which reads these data from a socket to visualize it.
However I am facing some usability issues. I would like to provide the functionality to start profiling a Java application on request. There is the Attach API which provides the possibility to inject an agent into a running JVM.
But to start a new Java program and inject the agent is a little bit more complicated. One way would be, to make a call to the command line and start the Java program from the GUI Profiler:
java -agentlib:agent Program
I kind of dislike this idea, because it is somehow hacky but I see no other way, do you?
To summarize I need two ways to start profiling a JVM:
Start a Java applicatiom from the scratch and start profiling it directly
Attach to a running JVM and inject the agent to start profiling it
Further, I would be in need to distinguish the different JVMs which I inspect, but how to do that? There no unique identifier for the different JVMs. The Attach API gives the possibility to list the different JVMs with their name and id, but what to do in the first case? Is it possible to inject the agent with arguments?
You can also generate your own GUID in the Agent_OnLoad and use that for logging. this way if your some of your processes have short lives and others long lives you can distinguish between recycled PIDS.
I solved the problem by using the local process identification (pid) and the network address to uniquely identfy the JVM.
How does one secure the Java environment when running on a machine you don't control? What is to stop someone from creating a java agent or native JVMTI agent and dumping bytecode or re-writing classes to bypass licensing and/or other security checks? Is there any way to detect if any agents are running from Java code? From JNI? From a JVMTI agent?
If you don't control the environment, then I'm sorry - you're really stuck. Yes, you could look for trivial JVMTI agents via some sort of cmdline sniffing, but that's the least of your worries. Think about java/lang/Classloader.defineClass() being compromised directly. That's easy to do if you own the box - just replace the .class file in rt.jar. In fact, until JVMTI came around, that was a typical way that profilers and monitoring tools instrumented Java code.
Going back to JVMTI - the "Late attach" feature also allows for JVMTI agents to be loaded on the fly. That might not have happened when you scanned the first time around.
Bottom line - if someone can change the bytes of the JRE on disk, they can do anything they want. Is it ethical, no? Can they get caught? Possibly, but you'll never win the war.
It looks like I can go with a combination of checks inside some custom JNI native code.
1.) cmd line sniffing to search for agents.
2.) Ensure that the cmd-line parameter -XX:+DisableAttachMechanism exists. (this will prevent people from attaching to my running VM)
I remember I once made almost a silent Java Agent. I guess you better look for port scanners or something around that.
Java 2 security, signing of jars etc, gives some level of control over what gets loaded into your application.
However in the end if a malicious person has access to a machine such that they can write to disk then in all probability they have plenty of capacity to do harm without resorting to clever Java hacks.
Turn this round, in any language what can you do to detect Trojans?
Careful access control to the machines you care about is non-trivial but essential if you are serious about such issues. Security specialists may seem paranoid, but that often means that they really understand the risks.
If you can't control the platform, you can't control the software upon it.
Even if you could shut down all the avenues of inspection you've listed, Java is open source. They could just take the source code and recompile it with the necessary changes built-in.
Also, try to remember that while it is your code, it's their machine. They have a right to inspect your code to verify that running it on their machine does what they expect it to do, and doesn't perform "extra" actions which they might find undesirable. Less trustworthy companies in the past have scanned for non-relevant files, copied sensitive information back to their home servers, etc.
I would look at the command line and see, if there are any "-agent" parameters. All profilers, debuggers and other code modificators use this for introspection. You could also check for unusual jars on the bootclasspath, since those might also provide a threat (but be aware that you then also must deliver a custom JVM, since some software like Quicktime adds itself to the bootclasspath of ALL java apps running... (I couldn't belive my eyes when I saw that...))
Basically this is a loosing battle.
Have a look at how visualvm in the Sun JDK works and how it can attach to a running process and redefine whatever it pleases. It is extremely hard to detect that in a portable way, and unless you can do so, you might as well give up on this approach.
The question is, what is it you want to avoid?