[TL;DR: the following JVM bytecode instructions seems not to work:
iconst_0
istore 6
...sequential
iinc 6 1
jsr L42
...
; L42
iload 6
ifeq L53 ; Always branches!!!
astore 8
iinc 6 -1
; L53
LDC 100
ISUB ; ERROR, returnAddress is at the top of the stack
A test .class can be found here (with slightly more complex logic). If you want to know more about why I'm seeing these instructions, please keep reading.]
I'm writing a Whitespace compiler targeting JVM bytecode. Although being an esoteric language, Whitespace describes an interesting set of assembly instructions to a stack machine, which maps nicely to the JVM.
Whitespace has labels, which are both targets for jump (goto/jump-if-zero/jump-if-negative) and function calls. The relevant instructions (with names given by me, in the spec they are given as combinations of space, tabs and newlines) are:
mark <label>: sets a label for the following instruction
jump[-if-neg|-if-zero] <label>: jumps unconditionally or conditionally to the given label
call <label>: call a function pointed by label
end <label>: ends a function, returning to the caller.
My compiler outputs the whole Whitespace program in the main method of a class. The simplest way to implement call and end is using the JSR and RET opcodes, which are made to implement subroutines. After a JSR operation the stack will contain a returnAddress reference that should be stored in a variable for later use in end.
However, as mark can be either call-ed or jump-ed into, the stack may or may not contain the returnAddress reference. I decided to use a boolean variable (call-bit, at address 6) to store how the mark was reached, and then test if it should store the top of the stack into a local variable (return-address, at address 8). The implementation for each instruction is as follows:
; ... initialization
iconst_0
istore 6 ; local variable #6 holds the call bit
# call
iinc 6 1 ; sets the call bit
jsr Lxxx ; jumps to the given label, pushing a returnAddress to the stack
# mark
; Lxxx
iload 6 ; loads the call bit
ifeq Lxxx-end ; SHOULD jump to mark's end if the call bit is not set
; call bit is set: mark was call-ed and returnAddress is in the stack
astore 8 ; stores returnAddress to local variable #8
iinc 6 -1 ; resets the call bit
; Lxxx-end
# end
ret 8 ; returns using the stored returnAddress
The problem: ifeq ALWAYS branches. I also tried reversing the logic (call-bit -> jump-bit, ifeq->ifne), and even simply switching to ifne (which would be wrong)... but the if always branches to the end. After a call, the returnAddress stays in the stack and the next operation blows up.
I've used ASM's analyzer to watch the stack to debug all this, but have just asserted this behavior and can't find what I'm doing wrong. My one suspicion is that there's more to iinc, or to ifeq than my vain philosophy can imagine. I'll admit that I've only read the instruction set page and ASM's pertinent documentation for this project, but I'd hope that someone can bring a solution from the top of their mind.
In this folder there are the relevant files, including the executable class and the original Whitespace, as well as the output of javap -c and ASM analysis.
Found a possible reason: the problem is not during execution, but with the verifier. When it seemed that it "always branched", was in fact the verifier testing all possible outcomes of an if so it could be sure the stack would look like the same. My code relies on a reference (returnAddress) maybe or maybe not being present on the stack and the verifier can't check that.
That said, the sample code does not run with the -noverify flag, but other, simpler examples that failed verification did execute correctly.
Related
This question already has an answer here:
In Java Lambda's why is getClass() called on a captured variable
(1 answer)
Closed 1 year ago.
I'm looking at some decompiled code and seeing .getClass();, i.e. nothing being done with its return value.
public String forLocale(Locale locale, String keyword) {
Stream var10000 = getLocaleMappingList(locale, this.getSupportedLocales());
Map var10001 = this.translations;
var10001.getClass();
Map<String, String> translation = (Map)var10000.map(var10001::get).filter((m) -> {
return m.containsKey(keyword);
}).findFirst().orElse(this.translations.get(FALLBACK));
Preconditions.checkState(translation.containsKey(keyword), keyword + " is not a valid translation key");
return (String)translation.get(keyword);
}
What is that for? Was it in the original code like that? (So far I haven't seen an instance of decompiled code not matching up at least line-wise to source code.)
It kind of looks like an assertion, but then what is achieved by doing that as opposed to letting things go wrong at var10001::get? Or is it more about performance?
Update
Here's the bytecode. Cool thing to learn how to do!
// access flags 0x1
public forLocale(Ljava/util/Locale;Ljava/lang/String;)Ljava/lang/String;
L0
LINENUMBER 184 L0
ALOAD 1
ALOAD 0
INVOKEVIRTUAL com/spotify/i18n/Translations.getSupportedLocales ()Ljava/util/Set;
INVOKESTATIC com/spotify/i18n/Translations.getLocaleMappingList (Ljava/util/Locale;Ljava/util/Collection;)Ljava/util/stream/Stream;
ALOAD 0
GETFIELD com/spotify/i18n/Translations.translations : Ljava/util/Map;
DUP
INVOKEVIRTUAL java/lang/Object.getClass ()Ljava/lang/Class;
POP
INVOKEDYNAMIC apply(Ljava/util/Map;)Ljava/util/function/Function; [
// handle kind 0x6 : INVOKESTATIC
java/lang/invoke/LambdaMetafactory.metafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite;
// arguments:
(Ljava/lang/Object;)Ljava/lang/Object;,
// handle kind 0x9 : INVOKEINTERFACE
java/util/Map.get(Ljava/lang/Object;)Ljava/lang/Object; itf,
(Ljava/util/Locale;)Ljava/util/Map;
]
This looks like decompiled code, and my guess is that the decompiler hasn't generated Java code that is equivalent to the original source code.
The literal meaning of
var10001.getClass();
is to return the Class object for the type of the object that var10001 refers to. But the value that is returned appears to be discarded, so the call (apparently) doesn't achieve anything. Hence, my tentative conclusion that the decompiler has stuffed up.
You may need to read the (disassembled) bytecodes directly to discern what they are actually doing. (Or you could try a different decompiler.)
UPDATE
It is plausible that getClass() is called solely for the side-effect of checking for null. (I've never seen that idiom ... but it would work.) I wouldn't expect it to make the code faster, but it would make it more compact.
However, if this is being done in the (original) source code, it would appear to be unnecessary. A couple of lines later, the code takes var10001::get and passes it as an argument in a Stream.map call. I'm pretty sure that that evaluating var10001::get will entail checking that var10001 is not null.
int a; // why can't I put this in the condition itself
if((a = readData()) > 0){
// do something..
}
I'm wondering why java and javascript don't allow me to declare my variable in the (condition). I understand that it has to do with the fact that a variable declaration has to be the first thing on the line -except for for loops- but what is the underlying reason ? In C++ it seems like they can do it. I don't know C++ so apologizes if I misunderstood.
The reason is simplicity of scope rules. If you allowed variable declaration within a condition, you'd have to define the scope of such a variable, and what it means for a declaration to have a value. Neither is straightforward. Requiring variables in an expression to be already defined is simple, and has no downside. Allowing that is complicated, and would obfuscate the source. That is a posteriori reasoning, of course. It might just be that the language designers had other reasons in the moment.
Why do you care? Bottom line: because the JLS says so.
According to the Java Language Specification (https://docs.oracle.com/javase/specs/jls/se8/html/jls-14.html#jls-14.9), an if statement has the following form:
if ( Expression ) Statement [else ...]
A variable assignment, aka an AssignmentExpression, is just one of many sub-types of the abstract Expression (https://docs.oracle.com/javase/specs/jls/se8/html/jls-15.html#jls-15.26), so you can use its result (which happens to be the value that was assigned) in your if statement.
Its simpel:
A If-statement ist defined as:
if (Boolean) {...
We can now substitute:
if (var1 operator var2){... // with "_ operator _" => Boolean
If we now put i=1 as var1 we would get:
if ((i=1) operator var2){... // compile error: i cannot resolved to a local variable
or int i = 1:
if ((int i = 1) operator var2){...
The obvious problem is: int i = 1 is a type declaration and not a variable and we learned above thet the if clause only accept Boolean variables or expression which lead to a Boolean vaiable...!
I wasn't satisfied with the answer here, so like any normal person would do, I learned java byte code. Well, sort of... the rudimentary. So if anyone read this keep in mind that my findings have some extrapolations in them and could be inaccurate.. However it makes sens why it doesn't happen java (much less in javascript where my head was at when asking this) and it is ho so simple.
So now the byte code part:
here is what I got for this:
int a = 9;
System.out.println(a); // just here to prevent some optimization
if((a = 18) > 15){
System.out.println(a);
}
Code:
0: bipush 9
2: istore_1
// -- removed the System.out.print --
10: bipush 18
12: dup
13: istore_1
14: bipush 15
16: if_icmple 26
Here is what happens:
0: push 9 onto the stack
2: store it in the var a
10: push 18 onto stack
12: duplicate 18 on top of the stack
13: store it in var a
14 : push 15
16: the if statement if value1 is less than or equal to value2, branch to instruction at branchoffset.
So since the content of the condition is evaluated before the if statement itself, if we declared int a in the condition, depending on where we define the scope of int a to be, there would be scoping issues.
If it was inside the if block:
If we suppose int a would be scoped into the if block, it must still be evaluated before reaching it and thus being out of scope !
In other words:
while((int a = 9) > 15)
int a = 9 would be evaluated before reaching the while condition, thus being out of the scope we just defined. We wouldn't enter the while block, but the value of a is well defined! Makes sens why it doesn't happen, doesn't it ?
If we go the other way and says that it is just a shortcut for this:
int a;
while((a = 9) > 15)
Then the language wouldn't be consistent and would bring many confusing code! See the for loop, the variable defined in it is scoped in it not outside for example.
It still doesn't make any sens why it's not possible to do in javascript though. However I don't care enough to find out.
I hope I made it clear, at least it makes some sens to me and I learned something. On a side note, I don't think my question really warranted so much down votes, even tho the answer seems obvious to me now. Not every question on stackoverflow has to be a practical one..
In JVM specs, The following example
void createThreadArray() {
Thread threads[];
int count = 10;
threads = new Thread[count];
threads[0] = new Thread();
}
produces
Method void createThreadArray()
0 bipush 10 // Push int constant 10
2 istore_2 // Initialize count to that
3 iload_2 // Push count, used by anewarray
4 anewarray class #1 // Create new array of class Thread
7 astore_1 // Store new array in threads
8 aload_1 // Push value of threads
9 iconst_0 // Push int constant 0
10 new #1 // Create instance of class Thread
13 dup // Make duplicate reference...
14 invokespecial #5 // ...for Thread's constructor
// Method java.lang.Thread.<init>()V
17 aastore // Store new Thread in array at 0
18 return
My question is, why are we doing istore_2 then iload_2 in this method in the start? Can't we just use the value pushed by bipush 10 onto the stack for creating new array of objects? What's the design consideration behind this?
javac is not an optimizing compiler. Optimization like removal of the unnecessesary local variable count is done in the JVM runtime when the runtime detects that it is a hotspot.
By using literal translation it is very easy to construct the bytecode compiler. Any optimization analysis is hard, and is already implemented in the runtime, so javac simply does not do it.
As it was already told, Javac does not optimize this pattern.
However, not because it could not do so, but because it must not do so.
For every local variable in source code Javac reserves a slot in local variable array in the class file (JVMS §2.6.1). The liveness range of this variable along with its local variable index is stored in LocalVariableTable attribute (JVMS §4.7.13).
This information is necessary for debugging purposes. Local variable array provides mapping from variables in source code to variables in bytecode. E.g. you may set a breakpoint in your code at the line
threads = new Thread[count];
and query the value of count variable that is mapped to locals[2] in the bytecode.
I'm modifying a Java class bytecode through an hexadecimal editor, and I want to force a method to always return true.
Replaced all its bytecode with nops to keep the size intact (original size is 1890).
Execute a pop to restore the stack height since it receives an argument.
Return true with iconst_1 followed by ireturn.
public static boolean test(java.lang.String);
descriptor: (Ljava/lang/String;)Z
flags: ACC_PUBLIC, ACC_STATIC
Code:
stack=5, locals=12, args_size=1
0: nop
1: nop
2: nop
3: nop
4: nop
[...]
1886: nop
1887: nop
1888: pop
1889: iconst_1
1890: ireturn
But when executing it, I'm getting the following error
java.lang.VerifyError: (class: com/example/test/TestBytecode, method: test signature: (Ljava/lang/String;)Z) Inconsistent stack height 0 != 1
NOTE: with or without pop the result is exactly the same.
The pop is unnecessary, since the arguments are not on the stack at first. They are only pushed onto the stack when using *load instructions as if they were local variables, which can happen at any time.
pop pops a value from the stack, but the string that is passed as argument is in "local variable" 0.
You should be able to safely omit that pop.
Also, you should be able to leave out all nops, and instead just replace instruction 0 with iconst_1, instruction 1 with ireturn, and leave the entire rest of the method unchanged.
This way you'd be doing less work and probably even increasing performance.
If you are using Java 7 or later, probably the JVM is validating your bytecode with the stack map frames. (Check this question/answer for an explanation of the stack map frames)
If you are using Java 7, try using -XX:-UseSplitVerifier in the command line when you run your class.
If you are using java 8, then you will also have modify the stack map frames; doing so is not trivial, so I better recommend you to use a bytecode manipulation library like javassist.
UPDATE
Based on the comment of #Holger, he is right. However as far as I've seen you are filling unwanted op codes with NOPs rather than removing them.
As you probably already know, the machine instructions are located in an attribute called code; this attribute can have "sub-attributes" (attributes of code itself). One of these is the attribute StackMapTable, which is an "array" (a table) of "stack maps".
Replacing this attribute with zeros won't be enough, you'll have to:
set the number of entries of that table (the stack map table) to zero
delete the only entry on that table (and offset the following attributes and other fields)
still want to do it by hand? :-)
The problem is as follows; the method, in Java code, is:
Rule foo()
{
return sequence(foo(), x());
}
This will provoke a parsing loop which of course should be avoided; however, this is legal:
Rule foo()
{
return sequence(x(), foo());
}
Now, somewhere else in the code I do have access to a RuleMethod, which is a class extending MethodNode, and therefore I do have access to the following information:
ruleMethod.name: foo; (defined in MethodNode)
ruleMethod.desc: ()Lorg/parboiled/Rule; (defined in MethodNode)
ruleMethod.ownerClass: com.github.fge.grappa.experiments.SelfReferringRule.MyParser (defined in RuleMethod
And the bytecode of the first code extract above is as follows:
Method 'foo':
0 L0
1 ALOAD 0
2 ALOAD 0
3 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.foo ()Lorg/parboiled/Rule;
4 ALOAD 0
5 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.x ()Lorg/parboiled/Rule;
6 ICONST_0
7 ANEWARRAY java/lang/Object
8 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.sequence (Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Lorg/parboiled/Rule;
9 ARETURN
10 L1
Which means I have each and every information available to me to be able to spot, at least in the bytecode above, that foo() is the first argument of the sequence() invocation, since the constructor accepts three arguments and there are three elements on the stack.
But of course I can't "eye inspect" at runtime. Therefore I need a way to do this...
It looks like what I need is a MethodVisitor and somewhat visitInsn(), then see what arguments there are and detect appropriately...
But I don't have the slightest idea where to start; searching around on the net seems to only give examples of how to modify byte code, not detect such situations :/
Where do I start?
Analysis is generally much easier using the tree api as it allows you to easily backtrack and provides support for flow analysis.
If I understand your problem correctly, all you need to do (if all you wish to support is simple cases such as your example) is scan backwards from the call to sequence. As you know the code compiles what's on the stack must be valid, so just count back three method calls / field gets / etc.
If you want to support more complex scenarios where the inputs are assigned to variables by branch statements you will need some sort of flow analysis.
Create a MethodVistor and while you are in visitCode() of the method foo() look for visitMethodInsn() and if the name argument in the visitMethodInsn() is foo you know that you have a recursive call to the method.
In your bytecode listing you have three INVOKEVIRTUAL instructions, these instructions are visited in order by the visitMethodInsn() function. If you want to check the sequence you can keep track of the order in which the method calls are made. You will see foo() first followed by x() and then finally sequence().
3 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.foo ()Lorg/parboiled/Rule;
4 ALOAD 0
5 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.x ()Lorg/parboiled/Rule;
6 ICONST_0
7 ANEWARRAY java/lang/Object
8 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.sequence (Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Lorg/parboiled/Rule;
If I get your intention correctly, you want to detect direct left recursion, which parboiled cannot handle. However, left recursion can be handled:
In essence, the parser must detect a left recursion and fail there, but remember that a left recursion occured. When the rule which was used in a recursive way succeeds, the result is saved as "seed" and the parsing process is restarted at the original input position. This time, when the left recursion happens, the seed is used instead of failing. This process is repeated.
For explanation see http://www.vpri.org/pdf/tr2007002_packrat.pdf for the paper. The algorithm can easily be adapted to PEG parsers.
A parsing library similar to parboiled using this technique is https://github.com/ruediste/lambda-peg-parser