Is there a way to get notified when a method from a Different JVM is called.
Dev Env: JDK8, Windows 10 (later on cloud for deployment).
I have couple of Java applications running, One in App Server and another is standalone batch process.
Whenever a core java class method is called on either of these JVM's e.g. PrintStream.print, I need to get handle to input string and log it somewhere else.
I tried with
1. Java bytecode manipulation libraries e.g. Javassist, to transform byte code using Instrumentation, but it allows to have handle and manipulate User Defined classes / Third party library classes only - not java., sun. etc... (even if we do it somehow, it says - it violates JRE binary licence - Official Javadoc says this process of instrumenting the rt.jar class violates the JRE binary code license - so this may not be the go ahead approach.
https://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html )
Reflections - Can be used when you are on the same JVM, not sure if it works on different JVM.
Appreciate suggestions.
I want to use the #FunctionalInterface from Java 8 in my code, but I want to be able to use the generated class files with Java 6. I think then that I should the source version to 1.8, and the target version to 1.6.
I would be using #FunctionalInterface just for documentation, but I note that it has #Retention(RetentionPolicy.RUNTIME). If no one ever uses that annotation, will it cause problems?
If someone iterates over the annotations of my object at runtime, will it cause a missing class exception? But if that is true, how is it that how Google Guava can declare the JSR 305 annotation dependency to have a Maven <scope> of provided, which means annotations such as javax.annotation.Nonnull are missing at runtime, too, in Guava, without causing problems?
Let me ask it another way: if I use Google Guava in my project but don't include a JSR 305 dependency, do I really risk some error if I use reflection on the code? If so, what error will occur? If no error will occur, then analogously can I use the #FunctionalInterface annotation in source compiled with Java version 1.8 yet targeted to version 1.6 without any risk of runtime errors, even using reflection?
I think then that I should [set] the source version to 1.8, and the target version to 1.6.
Actually, it is not possible to compile Java source files of newer source versions for older JVM target versions. Oracles and OpenJDKs javac will reject a compilation attempt where the -source version is higher than the -target version. (However, I couldn't find a specification denying it, even the manual doesn't mention that). The sole idea of javacs cross-compiling feature is that you can compile your old e.g. 1.6 Java files still for the old 1.6 JVM even when you are using a newer JDK for compilation.
The issue you are describing is the sort of reason for this. Since Java is using a sort of lazy dependency loading, the compiler can't guarantee that there will be an appropriated class at runtime for all the dependencies. This also applies to the standard library.
However, there are (unofficial) tools to compile the newer source idioms or byte code to older byte code versions. But that doesn't go for the standard library. If you wanna use newer classes, you have to provide them on your own. For this purpose, there exist some back ports for specific parts of the standard library.
Specifically about your annotation question:
I was not able to find any reliable specification to what should/might happen if the JVM encounters an annotated construct for which it could not retrieve the class file (I searched the Java virtual machine specification SE 8). However, I found a somewhat related reference in the Java language specification SE 8:
An annotation is a marker which associates information with a program construct, but has no effect at run time.
From JLS 9.7
This statement rather indicates that an annotation (present or not) should not have an influence on the execution of a JVM. Therefore, a exception (such as NoClassDefFoundError) because of a missing annotation were rather against this.
Finally, though the answers of this question, I found even more specific statements:
An annotation that is present in the binary form may or may not be available at run time via the reflection libraries of the Java SE platform.
From JLS 9.6.4.2
And
Adding or removing annotations has no effect on the correct linkage of the binary representations of programs in the Java programming language.
From JLS 13.5.7
This quite clearly states that missing annotations will not cause an error, but instead will be just ignored if examined by reflection.
And if you deliver a class annotated with a Java 1.8 standard library annotation, and it will be (somehow) executed on e.g. Java 1.6 JVM where that annotation is just not present, then this specifications denies that any error is generated.
This is also supported by the following test which I wrote: (notice the usage of reflection)
#TestAnno
public class Test {
public static void main(String[] args) {
Annotation[] annos = Test.class.getAnnotations();
for (Annotation a : annos) {
System.out.println(a);
}
}
}
#Retention(RetentionPolicy.RUNTIME)
#interface TestAnno {
}
If compiled, it yields a Test.class and a TestAnno.class. When executed the program outputs:
#TestAnno()
Because that is the one annotation applied to Test. Now, if the TestAnno.class is removed without any modifications to Test.class (which refers to TestAnno with LTestAnno; sequence in the byte code) and Test is executed again, it just does not output anything. So my JVM is indeed ignoring the missing annotation and does not generate any error or exception (Tested with a OpenJDK version 1.8.0_131 on Linux).
As with any class loading situation, if the class isn't needed (or rather, doesn't need to be loaded), it doesn't matter if the class doesn't exist at runtime. Runtime annotations normally have the same problem, since if they're retained at runtime, it usually means that there's logic based on them, meaning their classes are loaded too.
But #FunctionalInterface doesn't have runtime logic, so...
Why does #FunctionalInterface have a RUNTIME retention? Apparently not for any particularly compelling reason, just a side effect of it also being #Documented annotation.
So if you want to make sure there are no potential problems if someone (or some tool more likely (I don't mean a "tool", like a co-worker)) decides to enumerate the annotations in your classes, I guess you'd need to remove the annotations at pre-processing.
According this topic:
Reasons of getting a java.lang.VerifyError
java.lang.VerifyError obtains if version of execution jvm newer than jvm that was used for compilation.
Always we can fix this problem using following jvm option: -XX:-UseSplitVerifier.
According this:
https://stackoverflow.com/a/16467026/2674303
to use this option is 'perfectly safe'.
Thus I don't understand why does java.lang.VerifyError is problem that prevents succesful compilation. Please clarify. Maybe it is not safe for libraries which instrument bytecode?
The Q&A's that you link to refer to a specific kind of verification error that it is safe to work around by using an alternative verifier. However, there are other kinds of verify error that you should not ignore ... and that you cannot deal with that way.
In short, the advice in the linked question does not apply in general. In general:
Switching to the alternative verifier probably won't help.
If you disable the verifier entirely, you risk running code that may violate the JVM's runtime type-safety (etc) constraints. That can lead to security issues and / or heap corruption and hard JVM crashes.
If you have a specific VerifyError that you need advice on, please include the full exception message and stacktrace, and describe the situation in which it occurs. Note that Andrey's answer is correct that a common cause of verification errors is bugs in code that is doing "bytecode engineering" for various purposes. Often, the fix is to change to a different version of the corresponding dependency.
VerifyError happens when your byte code is incorrect in some way. For example, when it tries to read from an uninitialized variable or assigns a primitive value to a field of type Object. An instrumentation library may have bugs leading to generation of such malformed bytecode. If you can share the exact details of the error, we could probably be more specific about the exact cause.
The goal is to force tools and libraries generate correct StackMapTable attribute whenever they manipulate Java bytecode, so that JVM does not have to do slow and complicated type inferencing phase of verification, but rather do only fast and simple type checking phase.
-XX:-UseSplitVerifier has been deprecated in Java 8, it won't help any more.
VerifyError is thrown during class loading by JVM, so it appears on runtime level, not compile level.
Not always we can fix probem by using old verifier. There is a particular case when this helps. It only helps when class is propher for older verifier and is missing StackMapTable which was introduced in JVM 1.6 and are obligatory in JVM 1.7
If you want to get rid VerifyError and UseSplitVerifier does not help that mean that your class is incorrect from JVM point of view. You can still turn off whole verification, but this can cause problems.
Bytecode manipulation is always danger when you don't know what are your'e doing. Now main bytecode manipulation libraries support StackMapFrames calculations so they could generate bytecode which is propher for every VM. But still, user can generate bytecode which is incorrect on class file level, in that case JVM will still thrown VerifyError on load, SplitVerifier won't help. Only disabling verification will force VM to load that class, but could be some error when it will be executing.
When using the same JDK (i.e. the same javac executable), are the generated class files always identical? Can there be a difference depending on the operating system or hardware? Except of the JDK version, could there be any other factors resulting in differences? Are there any compiler options to avoid differences? Is a difference only possibly in theory or does Oracle's javac actually produce different class files for the same input and compiler options?
Update 1 I'm interested in the generation, i.e. compiler output, not whether a class file can be run on various platforms.
Update 2 By 'Same JDK', I also mean the same javac executable.
Update 3 Distinction between theoretical difference and practical difference in Oracle's compilers.
[EDIT, adding paraphrased question]
"What are the circumstances where the same javac executable,when run on a different platform, will produce different bytecode?"
Let's put it this way:
I can easily produce an entirely conforming Java compiler that never produces the same .class file twice, given the same .java file.
I could do this by tweaking all kinds of bytecode construction or by simply adding superfluous attributes to my method (which is allowed).
Given that the specification does not require the compiler to produce byte-for-byte identical class files, I'd avoid depending such a result.
However, the few times that I've checked, compiling the same source file with the same compiler with the same switches (and the same libraries!) did result in the same .class files.
Update: I've recently stumbled over this interesting blog post about the implementation of switch on String in Java 7. In this blog post, there are some relevant parts, that I'll quote here (emphasis mine):
In order to make the compiler's output predictable and repeatable, the maps and sets used in these data structures are LinkedHashMaps and LinkedHashSets rather than just HashMaps and HashSets. In terms of functional correctness of code generated during a given compile, using HashMap and HashSet would be fine; the iteration order does not matter. However, we find it beneficial to have javac's output not vary based on implementation details of system classes .
This pretty clearly illustrates the issue: The compiler is not required to act in a deterministic manner, as long as it matches the spec. The compiler developers, however, realize that it's generally a good idea to try (provided it's not too expensive, probably).
There is no obligation for the compilers to produce the same bytecode on each platform. You should consult the different vendors' javac utility to have a specific answer.
I will show a practical example for this with file ordering.
Let's say that we have 2 jar files: my1.jar and My2.jar. They're put in the lib directory, side-by-side. The compiler reads them in alphabetical order (since this is lib), but the order is my1.jar, My2.jar when the file system is case insensitive , and My2.jar, my1.jar if it is case sensitive.
The my1.jar has a class A.class with a method
public class A {
public static void a(String s) {}
}
The My2.jar has the same A.class, but with different method signature (accepts Object):
public class A {
public static void a(Object o) {}
}
It is clear that if you have a call
String s = "x";
A.a(s);
it will compile a method call with different signature in different cases. So, depending on your filesystem case sensitiveness, you will get different class as a result.
Short Answer - NO
Long Answer
They bytecode need not be the same for different platform. It's the JRE (Java Runtime Environment) which know how exactly to execute the bytecode.
If you go through the Java VM specification you'll come to know that this needs not to be true that the bytecode is same for different platforms.
Going through the class file format, it shows the structure of a class file as
ClassFile {
u4 magic;
u2 minor_version;
u2 major_version;
u2 constant_pool_count;
cp_info constant_pool[constant_pool_count-1];
u2 access_flags;
u2 this_class;
u2 super_class;
u2 interfaces_count;
u2 interfaces[interfaces_count];
u2 fields_count;
field_info fields[fields_count];
u2 methods_count;
method_info methods[methods_count];
u2 attributes_count;
attribute_info attributes[attributes_count];
}
Checking about the minor and major version
minor_version, major_version
The values of the minor_version and
major_version items are the minor and major version numbers of this
class file.Together, a major and a minor version number determine the
version of the class file format. If a class file has major version
number M and minor version number m, we denote the version of its
class file format as M.m. Thus, class file format versions may be
ordered lexicographically, for example, 1.5 < 2.0 < 2.1. A Java
virtual machine implementation can support a class file format of
version v if and only if v lies in some contiguous range Mi.0 v
Mj.m. Only Sun can specify what range of versions a Java virtual
machine implementation conforming to a certain release level of the
Java platform may support.1
Reading more through the footnotes
1 The Java virtual machine implementation of Sun's JDK release 1.0.2
supports class file format versions 45.0 through 45.3 inclusive. Sun's
JDK releases 1.1.X can support class file formats of versions in the
range 45.0 through 45.65535 inclusive. Implementations of version 1.2
of the Java 2 platform can support class file formats of versions in
the range 45.0 through 46.0 inclusive.
So, investigating all this shows that the class files generated on different platforms need not be identical.
Firstly, there's absolutely no such guarantee in the spec. A conforming compiler could stamp the time of compilation into the generated class file as an additional (custom) attribute, and the class file would still be correct. It would however produce a byte-level different file on every single build, and trivially so.
Secondly, even without such nasty tricks about, there's no reason to expect a compiler to do exactly the same thing twice in a row unless both its configuration and its input are identical in the two cases. The spec does describe the source filename as one of the standard attributes, and adding blank lines to the source file could well change the line number table.
Thirdly, I've never encountered any difference in build due to the host platform (other than that which was attributable to differences in what was on the classpath). The code which would vary based on platform (i.e., native code libraries) isn't part of the class file, and the actual generation of native code from the bytecode happens after the class is loaded.
Fourthly (and most importantly) it reeks of a bad process smell (like a code smell, but for how you act on the code) to want to know this. Version the source if possible, not the build, and if you do need to version the build, version at the whole-component level and not on individual class files. For preference, use a CI server (such as Jenkins) to manage the process of turning source into runnable code.
I believe that, if you use the same JDK, the generated byte code will always be the same, without relation with the harware and OS used. The byte code production is done by the java compiler, that uses a deterministic algorithm to "transform" the source code into byte code. So, the output will always be the same. In these conditions, only a update on the source code will affect the output.
Overall, I'd have to say there is no guarantee that the same source will produce the same bytecode when compiled by the same compiler but on a different platform.
I'd look into scenarios involving different languages (code-pages), for example Windows with Japanese language support. Think multi-byte characters; unless the compiler always assumes it needs to support all languages it might optimize for 8-bit ASCII.
There is a section on binary compatibility in the Java Language Specification.
Within the framework of Release-to-Release Binary Compatibility in SOM
(Forman, Conner, Danforth, and Raper, Proceedings of OOPSLA '95), Java
programming language binaries are binary compatible under all relevant
transformations that the authors identify (with some caveats with
respect to the addition of instance variables). Using their scheme,
here is a list of some important binary compatible changes that the
Java programming language supports:
•Reimplementing existing methods, constructors, and initializers to
improve performance.
•Changing methods or constructors to return values on inputs for which
they previously either threw exceptions that normally should not occur
or failed by going into an infinite loop or causing a deadlock.
•Adding new fields, methods, or constructors to an existing class or
interface.
•Deleting private fields, methods, or constructors of a class.
•When an entire package is updated, deleting default (package-only)
access fields, methods, or constructors of classes and interfaces in
the package.
•Reordering the fields, methods, or constructors in an existing type
declaration.
•Moving a method upward in the class hierarchy.
•Reordering the list of direct superinterfaces of a class or
interface.
•Inserting new class or interface types in the type hierarchy.
This chapter specifies minimum standards for binary compatibility
guaranteed by all implementations. The Java programming language
guarantees compatibility when binaries of classes and interfaces are
mixed that are not known to be from compatible sources, but whose
sources have been modified in the compatible ways described here. Note
that we are discussing compatibility between releases of an
application. A discussion of compatibility among releases of the Java
SE platform is beyond the scope of this chapter.
Java allows you write/compile code on one platform and run on different platform.
AFAIK; this will be possible only when class file generated on different platform is same or technically same i.e. identical.
Edit
What i mean by technically same comment is that. They don't need to be exactly same if you compare byte by byte.
So as per specification .class file of a class on different platforms don't need to match byte-by-byte.
For the question:
"What are the circumstances where the same javac executable,when run on a different platform, will produce different bytecode?"
The Cross-Compilation example shows how we can use the Javac option:-target version
This flag generates class files which are compatible with the Java version we specify while invoking this command. Hence the class files will differ depending on the attributes we supply during the compaliation using this option.
Most probably, the answer is "yes", but to have precise answer, one does need to search for some keys or guid generation during compiling.
I can't remember the situation where this occurs. For example to have ID for serializing purposes it is hardcoded, i.e. generated by programmer or IDE.
P.S. Also JNI can matter.
P.P.S. I found that javac is itself written in java. This means that it is identical on different platforms. Hence it would not generate different code without a reason. So, it can do this only with native calls.
There are two questions.
Can there be a difference depending on the operating system or hardware?
This is a theoretical question, and the answer is clearly, yes, there can be. As others have said, the specification does not require the compiler to produce byte-for-byte identical class files.
Even if every compiler currently in existence produced the same byte code in all circumstances (different hardware, etc.), the answer tomorrow might be different. If you never plan to update javac or your operating system, you could test that version's behavior in your particular circumstances, but the results might be different if you go from, for example, Java 7 Update 11 to Java 7 Update 15.
What are the circumstances where the same javac executable, when run on a different platform, will produce different bytecode?
That's unknowable.
I don't know if configuration management is your reason for asking the question, but it's an understandable reason to care. Comparing byte codes is a legitimate IT control, but only to determine if the class files changed, not top determine if the source files did.
I would put it another way.
First, I think the question is not about being deterministic:
Of course it is deterministic: randomness is hard to achieve in computer science, and there is no reason a compiler would introduce it here for any reason.
Second, if you reformulate it by "how similar are bytecode files for a same sourcecode file ?", then No, you can't rely on the fact that they will be similar.
A good way of making sure of this, is by leaving the .class (or .pyc in my case) in your git stage. You'll realize that among different computers in your team, git notices changes between .pyc files, when no changes were brought to the .py file (and .pyc recompiled anyway).
At least that's what I observed. So put *.pyc and *.class in your .gitignore !
I have a post-compilation step that manipulates the Java bytecode of generated classes. I'd like to make life as painless as possible for library consumers, so I'm looking at ways I can make this process automatic and (if possible) compiler agnostic.
The Annotation Processing API provides many of the desired features (automatic service discovery; supported by Eclipse). Unfortunately, this is aimed at code generators and doesn't support manipulation of existing artefacts:
The initial inputs to the tool are
considered to be created by the zeroth
round; therefore, attempting to create
a source or class file corresponding
to one of those inputs will result in
a FilerException.
The Decorator pattern recommended by the API is not an option.
I can see how to perform the step with a runtime agent/instrumentation, but this is a worse option than a manual build step as it would require anyone even peripherally touched by the API to configure their JVMs in a non-obvious manner.
Is there a way to plug into or wrap the compiler tool as invoked by javac? Has anyone successfully subverted the annotation processors to manipulate bytecode, no matter what the doc says?
The Groovy compiler is the only bytecode compiler which allows to hook into the compilation process (example: Generate bytecode to support the Singleton pattern)
The Annotation Processing API is not meant to change the code. As you have already found out, all you can do is install a classloader, examine the bytecode at runtime and manipulate it. It's braindead but it works. This follows the general "we're afraid that a developer could try something stupid" theme which you will find throughout Java. There is no way to extend javac. The relevant classes are either private, final or will change with the next version of Java.
Another option is to write annotated Java, for example you write a class "ExampleTpl.java". Then, you use a precompiler which expands the annotations in that file to get "Example.java". In the rest of the code, you use Example and ignore ExampleTpl.
For Eclipse, there is a bug report to automate this step. I'm not aware of any other work in this area.
It can be done.
Take a look at my blog post Roman Numerals, in our Java where an annotation processor is used to rewrite code. Limitation being that it works with Sun's javac only.