Are there currently (Java 6) things you can do in Java bytecode that you can't do from within the Java language?
I know both are Turing complete, so read "can do" as "can do significantly faster/better, or just in a different way".
I'm thinking of extra bytecodes like invokedynamic, which can't be generated using Java, except that specific one is for a future version.
After working with Java byte code for quite a while and doing some additional research on this matter, here is a summary of my findings:
Execute code in a constructor before calling a super constructor or auxiliary constructor
In the Java programming language (JPL), a constructor's first statement must be an invocation of a super constructor or another constructor of the same class. This is not true for Java byte code (JBC). Within byte code, it is absolutely legitimate to execute any code before a constructor, as long as:
Another compatible constructor is called at some time after this code block.
This call is not within a conditional statement.
Before this constructor call, no field of the constructed instance is read and none of its methods is invoked. This implies the next item.
Set instance fields before calling a super constructor or auxiliary constructor
As mentioned before, it is perfectly legal to set a field value of an instance before calling another constructor. There even exists a legacy hack which makes it able to exploit this "feature" in Java versions before 6:
class Foo {
public String s;
public Foo() {
System.out.println(s);
}
}
class Bar extends Foo {
public Bar() {
this(s = "Hello World!");
}
private Bar(String helper) {
super();
}
}
This way, a field could be set before the super constructor is invoked which is however not longer possible. In JBC, this behavior can still be implemented.
Branch a super constructor call
In Java, it is not possible to define a constructor call like
class Foo {
Foo() { }
Foo(Void v) { }
}
class Bar() {
if(System.currentTimeMillis() % 2 == 0) {
super();
} else {
super(null);
}
}
Until Java 7u23, the HotSpot VM's verifier did however miss this check which is why it was possible. This was used by several code generation tools as a sort of a hack but it is not longer legal to implement a class like this.
The latter was merely a bug in this compiler version. In newer compiler versions, this is again possible.
Define a class without any constructor
The Java compiler will always implement at least one constructor for any class. In Java byte code, this is not required. This allows the creation of classes that cannot be constructed even when using reflection. However, using sun.misc.Unsafe still allows for the creation of such instances.
Define methods with identical signature but with different return type
In the JPL, a method is identified as unique by its name and its raw parameter types. In JBC, the raw return type is additionally considered.
Define fields that do not differ by name but only by type
A class file can contain several fields of the same name as long as they declare a different field type. The JVM always refers to a field as a tuple of name and type.
Throw undeclared checked exceptions without catching them
The Java runtime and the Java byte code are not aware of the concept of checked exceptions. It is only the Java compiler that verifies that checked exceptions are always either caught or declared if they are thrown.
Use dynamic method invocation outside of lambda expressions
The so-called dynamic method invocation can be used for anything, not only for Java's lambda expressions. Using this feature allows for example to switch out execution logic at runtime. Many dynamic programming languages that boil down to JBC improved their performance by using this instruction. In Java byte code, you could also emulate lambda expressions in Java 7 where the compiler did not yet allow for any use of dynamic method invocation while the JVM already understood the instruction.
Use identifiers that are not normally considered legal
Ever fancied using spaces and a line break in your method's name? Create your own JBC and good luck for code review. The only illegal characters for identifiers are ., ;, [ and /. Additionally, methods that are not named <init> or <clinit> cannot contain < and >.
Reassign final parameters or the this reference
final parameters do not exist in JBC and can consequently be reassigned. Any parameter, including the this reference is only stored in a simple array within the JVM what allows to reassign the this reference at index 0 within a single method frame.
Reassign final fields
As long as a final field is assigned within a constructor, it is legal to reassign this value or even not assign a value at all. Therefore, the following two constructors are legal:
class Foo {
final int bar;
Foo() { } // bar == 0
Foo(Void v) { // bar == 2
bar = 1;
bar = 2;
}
}
For static final fields, it is even allowed to reassign the fields outside of
the class initializer.
Treat constructors and the class initializer as if they were methods
This is more of a conceptional feature but constructors are not treated any differently within JBC than normal methods. It is only the JVM's verifier that assures that constructors call another legal constructor. Other than that, it is merely a Java naming convention that constructors must be called <init> and that the class initializer is called <clinit>. Besides this difference, the representation of methods and constructors is identical. As Holger pointed out in a comment, you can even define constructors with return types other than void or a class initializer with arguments, even though it is not possible to call these methods.
Create asymmetric records*.
When creating a record
record Foo(Object bar) { }
javac will generate a class file with a single field named bar, an accessor method named bar() and a constructor taking a single Object. Additionally, a record attribute for bar is added. By manually generating a record, it is possible to create, a different constructor shape, to skip the field and to implement the accessor differently. At the same time, it is still possible to make the reflection API believe that the class represents an actual record.
Call any super method (until Java 1.1)
However, this is only possible for Java versions 1 and 1.1. In JBC, methods are always dispatched on an explicit target type. This means that for
class Foo {
void baz() { System.out.println("Foo"); }
}
class Bar extends Foo {
#Override
void baz() { System.out.println("Bar"); }
}
class Qux extends Bar {
#Override
void baz() { System.out.println("Qux"); }
}
it was possible to implement Qux#baz to invoke Foo#baz while jumping over Bar#baz. While it is still possible to define an explicit invocation to call another super method implementation than that of the direct super class, this does no longer have any effect in Java versions after 1.1. In Java 1.1, this behavior was controlled by setting the ACC_SUPER flag which would enable the same behavior that only calls the direct super class's implementation.
Define a non-virtual call of a method that is declared in the same class
In Java, it is not possible to define a class
class Foo {
void foo() {
bar();
}
void bar() { }
}
class Bar extends Foo {
#Override void bar() {
throw new RuntimeException();
}
}
The above code will always result in a RuntimeException when foo is invoked on an instance of Bar. It is not possible to define the Foo::foo method to invoke its own bar method which is defined in Foo. As bar is a non-private instance method, the call is always virtual. With byte code, one can however define the invocation to use the INVOKESPECIAL opcode which directly links the bar method call in Foo::foo to Foo's version. This opcode is normally used to implement super method invocations but you can reuse the opcode to implement the described behavior.
Fine-grain type annotations
In Java, annotations are applied according to their #Target that the annotations declares. Using byte code manipulation, it is possible to define annotations independently of this control. Also, it is for example possible to annotate a parameter type without annotating the parameter even if the #Target annotation applies to both elements.
Define any attribute for a type or its members
Within the Java language, it is only possible to define annotations for fields, methods or classes. In JBC, you can basically embed any information into the Java classes. In order to make use of this information, you can however no longer rely on the Java class loading mechanism but you need to extract the meta information by yourself.
Overflow and implicitly assign byte, short, char and boolean values
The latter primitive types are not normally known in JBC but are only defined for array types or for field and method descriptors. Within byte code instructions, all of the named types take the space 32 bit which allows to represent them as int. Officially, only the int, float, long and double types exist within byte code which all need explicit conversion by the rule of the JVM's verifier.
Not release a monitor
A synchronized block is actually made up of two statements, one to acquire and one to release a monitor. In JBC, you can acquire one without releasing it.
Note: In recent implementations of HotSpot, this instead leads to an IllegalMonitorStateException at the end of a method or to an implicit release if the method is terminated by an exception itself.
Add more than one return statement to a type initializer
In Java, even a trivial type initializer such as
class Foo {
static {
return;
}
}
is illegal. In byte code, the type initializer is treated just as any other method, i.e. return statements can be defined anywhere.
Create irreducible loops
The Java compiler converts loops to goto statements in Java byte code. Such statements can be used to create irreducible loops, which the Java compiler never does.
Define a recursive catch block
In Java byte code, you can define a block:
try {
throw new Exception();
} catch (Exception e) {
<goto on exception>
throw Exception();
}
A similar statement is created implicitly when using a synchronized block in Java where any exception while releasing a monitor returns to the instruction for releasing this monitor. Normally, no exception should occur on such an instruction but if it would (e.g. the deprecated ThreadDeath), the monitor would still be released.
Call any default method
The Java compiler requires several conditions to be fulfilled in order to allow a default method's invocation:
The method must be the most specific one (must not be overridden by a sub interface that is implemented by any type, including super types).
The default method's interface type must be implemented directly by the class that is calling the default method. However, if interface B extends interface A but does not override a method in A, the method can still be invoked.
For Java byte code, only the second condition counts. The first one is however irrelevant.
Invoke a super method on an instance that is not this
The Java compiler only allows to invoke a super (or interface default) method on instances of this. In byte code, it is however also possible to invoke the super method on an instance of the same type similar to the following:
class Foo {
void m(Foo f) {
f.super.toString(); // calls Object::toString
}
public String toString() {
return "foo";
}
}
Access synthetic members
In Java byte code, it is possible to access synthetic members directly. For example, consider how in the following example the outer instance of another Bar instance is accessed:
class Foo {
class Bar {
void bar(Bar bar) {
Foo foo = bar.Foo.this;
}
}
}
This is generally true for any synthetic field, class or method.
Define out-of-sync generic type information
While the Java runtime does not process generic types (after the Java compiler applies type erasure), this information is still attcheched to a compiled class as meta information and made accessible via the reflection API.
The verifier does not check the consistency of these meta data String-encoded values. It is therefore possible to define information on generic types that does not match the erasure. As a concequence, the following assertings can be true:
Method method = ...
assertTrue(method.getParameterTypes() != method.getGenericParameterTypes());
Field field = ...
assertTrue(field.getFieldType() == String.class);
assertTrue(field.getGenericFieldType() == Integer.class);
Also, the signature can be defined as invalid such that a runtime exception is thrown. This exception is thrown when the information is accessed for the first time as it is evaluated lazily. (Similar to annotation values with an error.)
Append parameter meta information only for certain methods
The Java compiler allows for embedding parameter name and modifier information when compiling a class with the parameter flag enabled. In the Java class file format, this information is however stored per-method what makes it possible to only embed such method information for certain methods.
Mess things up and hard-crash your JVM
As an example, in Java byte code, you can define to invoke any method on any type. Usually, the verifier will complain if a type does not known of such a method. However, if you invoke an unknown method on an array, I found a bug in some JVM version where the verifier will miss this and your JVM will finish off once the instruction is invoked. This is hardly a feature though, but it is technically something that is not possible with javac compiled Java. Java has some sort of double validation. The first validation is applied by the Java compiler, the second one by the JVM when a class is loaded. By skipping the compiler, you might find a weak spot in the verifier's validation. This is rather a general statement than a feature, though.
Annotate a constructor's receiver type when there is no outer class
Since Java 8, non-static methods and constructors of inner classes can declare a receiver type and annotate these types. Constructors of top-level classes cannot annotate their receiver type as they most not declare one.
class Foo {
class Bar {
Bar(#TypeAnnotation Foo Foo.this) { }
}
Foo() { } // Must not declare a receiver type
}
Since Foo.class.getDeclaredConstructor().getAnnotatedReceiverType() does however return an AnnotatedType representing Foo, it is possible to include type annotations for Foo's constructor directly in the class file where these annotations are later read by the reflection API.
Use unused / legacy byte code instructions
Since others named it, I will include it as well. Java was formerly making use of subroutines by the JSR and RET statements. JBC even knew its own type of a return address for this purpose. However, the use of subroutines did overcomplicate static code analysis which is why these instructions are not longer used. Instead, the Java compiler will duplicate code it compiles. However, this basically creates identical logic which is why I do not really consider it to achieve something different. Similarly, you could for example add the NOOP byte code instruction which is not used by the Java compiler either but this would not really allow you to achieve something new either. As pointed out in the context, these mentioned "feature instructions" are now removed from the set of legal opcodes which does render them even less of a feature.
As far as I know there are no major features in the bytecodes supported by Java 6 that are not also accessible from Java source code. The main reason for this is obviously that the Java bytecode was designed with the Java language in mind.
There are some features that are not produced by modern Java compilers, however:
The ACC_SUPER flag:
This is a flag that can be set on a class and specifies how a specific corner case of the invokespecial bytecode is handled for this class. It is set by all modern Java compilers (where "modern" is >= Java 1.1, if I remember correctly) and only ancient Java compilers produced class files where this was un-set. This flag exists only for backwards-compatibility reasons. Note that starting with Java 7u51, ACC_SUPER is ignored completely due to security reasons.
The jsr/ret bytecodes.
These bytecodes were used to implement sub-routines (mostly for implementing finally blocks). They are no longer produced since Java 6. The reason for their deprecation is that they complicate static verification a lot for no great gain (i.e. code that uses can almost always be re-implemented with normal jumps with very little overhead).
Having two methods in a class that only differ in return type.
The Java language specification does not allow two methods in the same class when they differ only in their return type (i.e. same name, same argument list, ...). The JVM specification however, has no such restriction, so a class file can contain two such methods, there's just no way to produce such a class file using the normal Java compiler. There's a nice example/explanation in this answer.
Here are some features that can be done in Java bytecode but not in Java source code:
Throwing a checked exception from a method without declaring that the method throws it. The checked and unchecked exceptions are a thing which is checked only by the Java compiler, not the JVM. Because of this for example Scala can throw checked exceptions from methods without declaring them. Though with Java generics there is a workaround called sneaky throw.
Having two methods in a class that only differ in return type, as already mentioned in Joachim's answer: The Java language specification does not allow two methods in the same class when they differ only in their return type (i.e. same name, same argument list, ...). The JVM specification however, has no such restriction, so a class file can contain two such methods, there's just no way to produce such a class file using the normal Java compiler. There's a nice example/explanation in this answer.
GOTO can be used with labels to create your own control structures (other than for while etc)
You can override the this local variable inside a method
Combining both of these you can create create tail call optimised bytecode (I do this in JCompilo)
As a related point you can get parameter name for methods if compiled with debug (Paranamer does this by reading the bytecode
Maybe section 7A in this document is of interest, although it's about bytecode pitfalls rather than bytecode features.
In Java language the first statement in a constructor must be a call to the super class constructor. Bytecode does not have this limitation, instead the rule is that the super class constructor or another constructor in the same class must be called for the object before accessing the members. This should allow more freedom such as:
Create an instance of another object, store it in a local variable (or stack) and pass it as a parameter to super class constructor while still keeping the reference in that variable for other use.
Call different other constructors based on a condition. This should be possible: How to call a different constructor conditionally in Java?
I have not tested these, so please correct me if I'm wrong.
Something you can do with byte code, rather than plain Java code, is generate code which can loaded and run without a compiler. Many systems have JRE rather than JDK and if you want to generate code dynamically it may be better, if not easier, to generate byte code instead of Java code has to be compiled before it can be used.
I wrote a bytecode optimizer when I was a I-Play, (it was designed to reduce the code size for J2ME applications). One feature I added was the ability to use inline bytecode (similar to inline assembly language in C++). I managed to reduce the size of a function that was part of a library method by using the DUP instruction, since I need the value twice. I also had zero byte instructions (if you are calling a method that takes a char and you want to pass an int, that you know does not need to be cast I added int2char(var) to replace char(var) and it would remove the i2c instruction to reduce the size of the code. I also made it do float a = 2.3; float b = 3.4; float c = a + b; and that would be converted to fixed point (faster, and also some J2ME did not support floating point).
In Java, if you attempt to override a public method with a protected method (or any other reduction in access), you get an error: "attempting to assign weaker access privileges". If you do it with JVM bytecode, the verifier is fine with it, and you can call these methods via the parent class as if they were public.
Android 3.5.1
I was using the WebView and I noticed that when I override some of the methods all the parameters are nullable types:
webview.webViewClient = object : WebViewClient() {
override fun shouldOverrideUrlLoading(view: WebView?, request: WebResourceRequest?): Boolean {
return super.shouldOverrideUrlLoading(view, request)
}
}
Which means I have to use the safe call operator to use them. However, when I looked at the WebViewClient class that I have overridden the method from they are not specified as nullable annotation in the Java code.
public boolean shouldOverrideUrlLoading(WebView view, WebResourceRequest request) {
return shouldOverrideUrlLoading(view, request.getUrl().toString());
}
So I am left thinking do I remove the nullability from the overridden method or keep them?
The source of this issue comes from Interoperability between Java and Kotlin. There are some basic language level differences between Java and Kotlin which causes interoperability issues. Android Studio provides some Lint checks to warn them, such as Unknown Nullness. (reference)
By taking a look at details of Unknown nullness Lint check from android.com, we see that:
To improve referencing code from Kotlin, consider adding
explicit nullness information here with either #NonNull or #Nullable.
and on developer.android.com:
If you use Kotlin to reference an unannotated name member that is defined in a Java class (e.g. a String), the compiler doesn't know whether the String maps to a String or a String? in Kotlin. This ambiguity is represented via a platform type, String!.
and on kotlinlang.org:
Any reference in Java may be null, which makes Kotlin's requirements of strict null-safety impractical for objects coming from Java. Types of Java declarations are treated specially in Kotlin and called platform types.
Therefore, when we override a Java method that its arguments are not annotated with nullity annotations, the IDE adds nullable sign (?) for arguments in Kotlin class. It leads to avoid throwing NullPointerException when the method is called in Java by passing a null value for one of the arguments.
webview.webViewClient = object : WebViewClient() {
override fun shouldOverrideUrlLoading(
view: WebView, // <- potential to throw NPE before executing the function block!
request: WebResourceRequest // <- as well!
): Boolean {
return super.shouldOverrideUrlLoading(view, request)
}
}
In a nutshell, we SHOULD NOT remove ? sign from function arguments, when the overridden method is defined in a Java class.
Unlike Kotlin , Java objects by default can accept null values
#Nullable annotation is just used for operations like code analysers (for eg. if the #Nullable parameter is not handled inside the method then it will show warning)
#NonNull annotation is used to specify that the value received can't/won't be null
if(#NonNull){
can omit ? check
}else if(#Nullable){
Mandatory to put ? check
}else(No annotation){
Not mandatory but put on safer side .
Passing null from Java into Kotlin fun without ? will lead to NPE
if(putting ? check){
java equivalent Kotlin param (#Nullable Webview view)
} else{
java equivalent Kotlin param (#NonNull Webview view)
}
}
Also Refer this : https://kotlinlang.org/docs/reference/java-to-kotlin-interop.html#null-safety
If a virtual method in Java doesn't specify nullability of its parameters somehow, for example with the #Nullable/#NotNull annotations, you are free to choose the nullability either way when overriding that method in Kotlin.
But how should you choose?
First, you can consult the method documentation and check the method contract. Does it specify that the method can be called with nulls, and what would these nulls mean when passed to the method?
In this particular case,
WebViewClient.shouldOverrideUrlLoading
method doc page doesn't say anything about nulls, so it can be taken as
an evidence that its parameters are supposed to be non-nullable.
Second, if you are still unsure about the nullability after consulting the docs, you can consider what would you do with the null parameter value, if you receive one. If the only reasonable thing in this situation is to throw an exception, you can delegate that check to the parameter checking code generated by Kotlin—by declaring parameters as non-nullable.
They are not specified as nullable annotation in the Java code.
If that's true note that you risk throwing a NullPointerException if not specified as nullable annotation in the Java code and assign a null value.
so remove the nullability from the overridden method if not specified as nullable annotation in the Java code.
For more detail read this also this
On the language-level, this can be generalized:
For proper Java interoperability, the Kotlin code should reflect the annotations of the Java code.
The linter only complains about lacking annotations in the other direction, for Kotlin interoperability.
See this recent article on How to write Java friendly Kotlin code?
Null reference is pretty obvious exception for everybody now because for everything has started with native development on C/C++. Reference to the objects in memory might be missing or cleaned by different reasons. Java was designed in way of those native languages, which assume null pointers everywhere.
Managing all mutable states are getting fun with thousand of microservices. This cause a lot of workaround for Nullable reference. - Optional Objects - Mock of Null Object - Wrappers around references - Annotations, etc. And all this for avoiding changing state of somewhere allocated object.
Finally, Kotlin is not first here. Scala, with immutable states had excellent experience in usage and supporting application. So answering this question and summarize Java was designed in this way from its parent C++, and you should expect null values everywhere. We still checking reference for null, even it's not annotated #Nullable, because of this reason. And in the same way Kotlin handles Java usage, and that is why you need to handle null values in overridden methods.
I am trying to mock some resources that are generated dynamically. In order to generate these resources, we must pass in a class argument. So for example:
FirstResourceClass firstResource = ResourceFactory.create(FirstResourceClass.class);
SecondResourceClass secondResource = ResourceFactory.create(SecondResource.class);
This is well and good until I tried to mock. I am doing something like this:
PowerMockito.mockStatic(ResourceFactory.class);
FirstResourceClass mockFirstResource = Mockito.mock(FirstResourceClass.class);
SecondResourceClass mockSecondResource = Mockito.mock(SecondResourceClass.class);
PowerMockito.when(ResourceFactory.create(Matchers.<Class<FirstResourceClass>>any()).thenReturn(mockFirstResource);
PowerMockito.when(ResourceFactory.create(Matchers.<Class<SecondResourceClass>>any()).thenReturn(mockSecondResource);
It seems like the mock is being injected into the calling class, but FirstResourceClass is being send mockSecondResource, which throws a compile error.
The issue is (I think) with the use of any() (which I got from this question). I believe I have to use isA(), but I'm not sure how to make that method call, as it requires a Class argument. I have tried FirstResourceClass.class, and that gives a compile error.
You want eq, as in:
PowerMockito.when(ResourceFactory.create(Matchers.eq(FirstResourceClass.class)))
.thenReturn(mockFirstResource);
any() ignores the argument, and isA will check that your argument is of a certain class—but not that it equals a class, just that it is an instanceof a certain class. (any(Class) has any() semantics in Mockito 1.x and isA semantics in 2.x.)
isA(Class.class) is less specific than you need to differentiate your calls, so eq it is. Class objects have well-defined equality, anyway, so this is easy and natural for your use-case.
Because eq is the default if you don't use matchers, this also works:
PowerMockito.when(ResourceFactory.create(FirstResourceClass.class))
.thenReturn(mockFirstResource);
Note that newer versions of Mockito have deprecated the Matchers name in favor of ArgumentMatchers, and Mockito.eq also works (albeit clumsily, because they're "inherited" static methods).
Are there any annotations in java which mark a method as unsupported? E.g. Let's say I'm writing a new class which implements the java.util.List interface. The add() methods in this interface are optional and I don't need them in my implementation and so I to do the following:
public void add(Object obj) {
throw new UnsupportedOperationException("This impl doesn't support add");
}
Unfortunately, with this, it's not until runtime that one might discover that, in fact, this operation is unsupported.
Ideally, this would have been caught at compile time and such an annotation (e.g. maybe #UnsupportedOperation) would nudge the IDE to say to any users of this method, "Hey, you're using an unsupported operation" in the way that using #Deprecated flags Eclipse to highlight any uses of the deprecated item.
Although on the surface this sounds useful, in reality it would not help much. How do you usually use a list? I generally do something like this:
List<String> list = new XXXList<String>();
There's already one indirection there, so if I call list.add("Hi"), how should the compiler know that this specific implementation of list doesn't support that?
How about this:
void populate(List<String> list) {
list.add("1");
list.add("2");
}
Now it's even harder: The compiler would need to verify that all calls to that function used lists that support the add() operation.
So no, there is no way to do what you are asking, sorry.
You can do it using AspectJ if you are familiar with it. You must first create a point-cut, then give an advice or declare error/warning joint points matching this point cut. Of course you need your own #UnsupportedOperation annotation interface. I gave a simple code fragment about this.
// This the point-cut matching calls to methods annotated with your
// #UnsupportedOperation annotation.
pointcut unsupportedMethodCalls() : call(#UnsupportedOperation * *.*(..));
// Declare an error for such calls. This causes a compilation error
// if the point-cut matches any unsupported calls.
declare error: unsupportedMethodCalls() : "This call is not supported."
// Or you can just throw an exception just before this call executed at runtime
// instead of a compile-time error.
before() : unsupportedMethodCalls() {
throw new UnsupportedOperationException(thisJoinPoint.getSignature()
.getName());
}
(2018) While it may not be possible to detect it at compile time, there could be an alternative. i.e. The IDE (or other tools) could use the annotation to warn the user that such a method is being used.
There actually is a ticket for this: JDK-6447051
From a technical point of view, it shouldn't be much harder to implement than inspections that detect an illegal use of an #NotNull or a #Nullable accessor.
try this annotation #DoNotCall
https://errorprone.info/api/latest/com/google/errorprone/annotations/DoNotCall.html