I'm modifying a Java class bytecode through an hexadecimal editor, and I want to force a method to always return true.
Replaced all its bytecode with nops to keep the size intact (original size is 1890).
Execute a pop to restore the stack height since it receives an argument.
Return true with iconst_1 followed by ireturn.
public static boolean test(java.lang.String);
descriptor: (Ljava/lang/String;)Z
flags: ACC_PUBLIC, ACC_STATIC
Code:
stack=5, locals=12, args_size=1
0: nop
1: nop
2: nop
3: nop
4: nop
[...]
1886: nop
1887: nop
1888: pop
1889: iconst_1
1890: ireturn
But when executing it, I'm getting the following error
java.lang.VerifyError: (class: com/example/test/TestBytecode, method: test signature: (Ljava/lang/String;)Z) Inconsistent stack height 0 != 1
NOTE: with or without pop the result is exactly the same.
The pop is unnecessary, since the arguments are not on the stack at first. They are only pushed onto the stack when using *load instructions as if they were local variables, which can happen at any time.
pop pops a value from the stack, but the string that is passed as argument is in "local variable" 0.
You should be able to safely omit that pop.
Also, you should be able to leave out all nops, and instead just replace instruction 0 with iconst_1, instruction 1 with ireturn, and leave the entire rest of the method unchanged.
This way you'd be doing less work and probably even increasing performance.
If you are using Java 7 or later, probably the JVM is validating your bytecode with the stack map frames. (Check this question/answer for an explanation of the stack map frames)
If you are using Java 7, try using -XX:-UseSplitVerifier in the command line when you run your class.
If you are using java 8, then you will also have modify the stack map frames; doing so is not trivial, so I better recommend you to use a bytecode manipulation library like javassist.
UPDATE
Based on the comment of #Holger, he is right. However as far as I've seen you are filling unwanted op codes with NOPs rather than removing them.
As you probably already know, the machine instructions are located in an attribute called code; this attribute can have "sub-attributes" (attributes of code itself). One of these is the attribute StackMapTable, which is an "array" (a table) of "stack maps".
Replacing this attribute with zeros won't be enough, you'll have to:
set the number of entries of that table (the stack map table) to zero
delete the only entry on that table (and offset the following attributes and other fields)
still want to do it by hand? :-)
Related
This question already has an answer here:
In Java Lambda's why is getClass() called on a captured variable
(1 answer)
Closed 1 year ago.
I'm looking at some decompiled code and seeing .getClass();, i.e. nothing being done with its return value.
public String forLocale(Locale locale, String keyword) {
Stream var10000 = getLocaleMappingList(locale, this.getSupportedLocales());
Map var10001 = this.translations;
var10001.getClass();
Map<String, String> translation = (Map)var10000.map(var10001::get).filter((m) -> {
return m.containsKey(keyword);
}).findFirst().orElse(this.translations.get(FALLBACK));
Preconditions.checkState(translation.containsKey(keyword), keyword + " is not a valid translation key");
return (String)translation.get(keyword);
}
What is that for? Was it in the original code like that? (So far I haven't seen an instance of decompiled code not matching up at least line-wise to source code.)
It kind of looks like an assertion, but then what is achieved by doing that as opposed to letting things go wrong at var10001::get? Or is it more about performance?
Update
Here's the bytecode. Cool thing to learn how to do!
// access flags 0x1
public forLocale(Ljava/util/Locale;Ljava/lang/String;)Ljava/lang/String;
L0
LINENUMBER 184 L0
ALOAD 1
ALOAD 0
INVOKEVIRTUAL com/spotify/i18n/Translations.getSupportedLocales ()Ljava/util/Set;
INVOKESTATIC com/spotify/i18n/Translations.getLocaleMappingList (Ljava/util/Locale;Ljava/util/Collection;)Ljava/util/stream/Stream;
ALOAD 0
GETFIELD com/spotify/i18n/Translations.translations : Ljava/util/Map;
DUP
INVOKEVIRTUAL java/lang/Object.getClass ()Ljava/lang/Class;
POP
INVOKEDYNAMIC apply(Ljava/util/Map;)Ljava/util/function/Function; [
// handle kind 0x6 : INVOKESTATIC
java/lang/invoke/LambdaMetafactory.metafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite;
// arguments:
(Ljava/lang/Object;)Ljava/lang/Object;,
// handle kind 0x9 : INVOKEINTERFACE
java/util/Map.get(Ljava/lang/Object;)Ljava/lang/Object; itf,
(Ljava/util/Locale;)Ljava/util/Map;
]
This looks like decompiled code, and my guess is that the decompiler hasn't generated Java code that is equivalent to the original source code.
The literal meaning of
var10001.getClass();
is to return the Class object for the type of the object that var10001 refers to. But the value that is returned appears to be discarded, so the call (apparently) doesn't achieve anything. Hence, my tentative conclusion that the decompiler has stuffed up.
You may need to read the (disassembled) bytecodes directly to discern what they are actually doing. (Or you could try a different decompiler.)
UPDATE
It is plausible that getClass() is called solely for the side-effect of checking for null. (I've never seen that idiom ... but it would work.) I wouldn't expect it to make the code faster, but it would make it more compact.
However, if this is being done in the (original) source code, it would appear to be unnecessary. A couple of lines later, the code takes var10001::get and passes it as an argument in a Stream.map call. I'm pretty sure that that evaluating var10001::get will entail checking that var10001 is not null.
int a; // why can't I put this in the condition itself
if((a = readData()) > 0){
// do something..
}
I'm wondering why java and javascript don't allow me to declare my variable in the (condition). I understand that it has to do with the fact that a variable declaration has to be the first thing on the line -except for for loops- but what is the underlying reason ? In C++ it seems like they can do it. I don't know C++ so apologizes if I misunderstood.
The reason is simplicity of scope rules. If you allowed variable declaration within a condition, you'd have to define the scope of such a variable, and what it means for a declaration to have a value. Neither is straightforward. Requiring variables in an expression to be already defined is simple, and has no downside. Allowing that is complicated, and would obfuscate the source. That is a posteriori reasoning, of course. It might just be that the language designers had other reasons in the moment.
Why do you care? Bottom line: because the JLS says so.
According to the Java Language Specification (https://docs.oracle.com/javase/specs/jls/se8/html/jls-14.html#jls-14.9), an if statement has the following form:
if ( Expression ) Statement [else ...]
A variable assignment, aka an AssignmentExpression, is just one of many sub-types of the abstract Expression (https://docs.oracle.com/javase/specs/jls/se8/html/jls-15.html#jls-15.26), so you can use its result (which happens to be the value that was assigned) in your if statement.
Its simpel:
A If-statement ist defined as:
if (Boolean) {...
We can now substitute:
if (var1 operator var2){... // with "_ operator _" => Boolean
If we now put i=1 as var1 we would get:
if ((i=1) operator var2){... // compile error: i cannot resolved to a local variable
or int i = 1:
if ((int i = 1) operator var2){...
The obvious problem is: int i = 1 is a type declaration and not a variable and we learned above thet the if clause only accept Boolean variables or expression which lead to a Boolean vaiable...!
I wasn't satisfied with the answer here, so like any normal person would do, I learned java byte code. Well, sort of... the rudimentary. So if anyone read this keep in mind that my findings have some extrapolations in them and could be inaccurate.. However it makes sens why it doesn't happen java (much less in javascript where my head was at when asking this) and it is ho so simple.
So now the byte code part:
here is what I got for this:
int a = 9;
System.out.println(a); // just here to prevent some optimization
if((a = 18) > 15){
System.out.println(a);
}
Code:
0: bipush 9
2: istore_1
// -- removed the System.out.print --
10: bipush 18
12: dup
13: istore_1
14: bipush 15
16: if_icmple 26
Here is what happens:
0: push 9 onto the stack
2: store it in the var a
10: push 18 onto stack
12: duplicate 18 on top of the stack
13: store it in var a
14 : push 15
16: the if statement if value1 is less than or equal to value2, branch to instruction at branchoffset.
So since the content of the condition is evaluated before the if statement itself, if we declared int a in the condition, depending on where we define the scope of int a to be, there would be scoping issues.
If it was inside the if block:
If we suppose int a would be scoped into the if block, it must still be evaluated before reaching it and thus being out of scope !
In other words:
while((int a = 9) > 15)
int a = 9 would be evaluated before reaching the while condition, thus being out of the scope we just defined. We wouldn't enter the while block, but the value of a is well defined! Makes sens why it doesn't happen, doesn't it ?
If we go the other way and says that it is just a shortcut for this:
int a;
while((a = 9) > 15)
Then the language wouldn't be consistent and would bring many confusing code! See the for loop, the variable defined in it is scoped in it not outside for example.
It still doesn't make any sens why it's not possible to do in javascript though. However I don't care enough to find out.
I hope I made it clear, at least it makes some sens to me and I learned something. On a side note, I don't think my question really warranted so much down votes, even tho the answer seems obvious to me now. Not every question on stackoverflow has to be a practical one..
[TL;DR: the following JVM bytecode instructions seems not to work:
iconst_0
istore 6
...sequential
iinc 6 1
jsr L42
...
; L42
iload 6
ifeq L53 ; Always branches!!!
astore 8
iinc 6 -1
; L53
LDC 100
ISUB ; ERROR, returnAddress is at the top of the stack
A test .class can be found here (with slightly more complex logic). If you want to know more about why I'm seeing these instructions, please keep reading.]
I'm writing a Whitespace compiler targeting JVM bytecode. Although being an esoteric language, Whitespace describes an interesting set of assembly instructions to a stack machine, which maps nicely to the JVM.
Whitespace has labels, which are both targets for jump (goto/jump-if-zero/jump-if-negative) and function calls. The relevant instructions (with names given by me, in the spec they are given as combinations of space, tabs and newlines) are:
mark <label>: sets a label for the following instruction
jump[-if-neg|-if-zero] <label>: jumps unconditionally or conditionally to the given label
call <label>: call a function pointed by label
end <label>: ends a function, returning to the caller.
My compiler outputs the whole Whitespace program in the main method of a class. The simplest way to implement call and end is using the JSR and RET opcodes, which are made to implement subroutines. After a JSR operation the stack will contain a returnAddress reference that should be stored in a variable for later use in end.
However, as mark can be either call-ed or jump-ed into, the stack may or may not contain the returnAddress reference. I decided to use a boolean variable (call-bit, at address 6) to store how the mark was reached, and then test if it should store the top of the stack into a local variable (return-address, at address 8). The implementation for each instruction is as follows:
; ... initialization
iconst_0
istore 6 ; local variable #6 holds the call bit
# call
iinc 6 1 ; sets the call bit
jsr Lxxx ; jumps to the given label, pushing a returnAddress to the stack
# mark
; Lxxx
iload 6 ; loads the call bit
ifeq Lxxx-end ; SHOULD jump to mark's end if the call bit is not set
; call bit is set: mark was call-ed and returnAddress is in the stack
astore 8 ; stores returnAddress to local variable #8
iinc 6 -1 ; resets the call bit
; Lxxx-end
# end
ret 8 ; returns using the stored returnAddress
The problem: ifeq ALWAYS branches. I also tried reversing the logic (call-bit -> jump-bit, ifeq->ifne), and even simply switching to ifne (which would be wrong)... but the if always branches to the end. After a call, the returnAddress stays in the stack and the next operation blows up.
I've used ASM's analyzer to watch the stack to debug all this, but have just asserted this behavior and can't find what I'm doing wrong. My one suspicion is that there's more to iinc, or to ifeq than my vain philosophy can imagine. I'll admit that I've only read the instruction set page and ASM's pertinent documentation for this project, but I'd hope that someone can bring a solution from the top of their mind.
In this folder there are the relevant files, including the executable class and the original Whitespace, as well as the output of javap -c and ASM analysis.
Found a possible reason: the problem is not during execution, but with the verifier. When it seemed that it "always branched", was in fact the verifier testing all possible outcomes of an if so it could be sure the stack would look like the same. My code relies on a reference (returnAddress) maybe or maybe not being present on the stack and the verifier can't check that.
That said, the sample code does not run with the -noverify flag, but other, simpler examples that failed verification did execute correctly.
The problem is as follows; the method, in Java code, is:
Rule foo()
{
return sequence(foo(), x());
}
This will provoke a parsing loop which of course should be avoided; however, this is legal:
Rule foo()
{
return sequence(x(), foo());
}
Now, somewhere else in the code I do have access to a RuleMethod, which is a class extending MethodNode, and therefore I do have access to the following information:
ruleMethod.name: foo; (defined in MethodNode)
ruleMethod.desc: ()Lorg/parboiled/Rule; (defined in MethodNode)
ruleMethod.ownerClass: com.github.fge.grappa.experiments.SelfReferringRule.MyParser (defined in RuleMethod
And the bytecode of the first code extract above is as follows:
Method 'foo':
0 L0
1 ALOAD 0
2 ALOAD 0
3 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.foo ()Lorg/parboiled/Rule;
4 ALOAD 0
5 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.x ()Lorg/parboiled/Rule;
6 ICONST_0
7 ANEWARRAY java/lang/Object
8 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.sequence (Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Lorg/parboiled/Rule;
9 ARETURN
10 L1
Which means I have each and every information available to me to be able to spot, at least in the bytecode above, that foo() is the first argument of the sequence() invocation, since the constructor accepts three arguments and there are three elements on the stack.
But of course I can't "eye inspect" at runtime. Therefore I need a way to do this...
It looks like what I need is a MethodVisitor and somewhat visitInsn(), then see what arguments there are and detect appropriately...
But I don't have the slightest idea where to start; searching around on the net seems to only give examples of how to modify byte code, not detect such situations :/
Where do I start?
Analysis is generally much easier using the tree api as it allows you to easily backtrack and provides support for flow analysis.
If I understand your problem correctly, all you need to do (if all you wish to support is simple cases such as your example) is scan backwards from the call to sequence. As you know the code compiles what's on the stack must be valid, so just count back three method calls / field gets / etc.
If you want to support more complex scenarios where the inputs are assigned to variables by branch statements you will need some sort of flow analysis.
Create a MethodVistor and while you are in visitCode() of the method foo() look for visitMethodInsn() and if the name argument in the visitMethodInsn() is foo you know that you have a recursive call to the method.
In your bytecode listing you have three INVOKEVIRTUAL instructions, these instructions are visited in order by the visitMethodInsn() function. If you want to check the sequence you can keep track of the order in which the method calls are made. You will see foo() first followed by x() and then finally sequence().
3 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.foo ()Lorg/parboiled/Rule;
4 ALOAD 0
5 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.x ()Lorg/parboiled/Rule;
6 ICONST_0
7 ANEWARRAY java/lang/Object
8 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.sequence (Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Lorg/parboiled/Rule;
If I get your intention correctly, you want to detect direct left recursion, which parboiled cannot handle. However, left recursion can be handled:
In essence, the parser must detect a left recursion and fail there, but remember that a left recursion occured. When the rule which was used in a recursive way succeeds, the result is saved as "seed" and the parsing process is restarted at the original input position. This time, when the left recursion happens, the seed is used instead of failing. This process is repeated.
For explanation see http://www.vpri.org/pdf/tr2007002_packrat.pdf for the paper. The algorithm can easily be adapted to PEG parsers.
A parsing library similar to parboiled using this technique is https://github.com/ruediste/lambda-peg-parser
I have a large Java app that processes a large collection of data files, using a try/catch within an actionPerformed (sample code below). It runs out of memory when I get to about 1000 files inside the loop.
Each file load legitimately takes about 1MB of storage, but I've looked carefully and don't see any place where that storage is being hung on to. Each file load is doing just the same thing (ie assigning the same vars), so it ought to be re-using, not accumulating.
I tried inserting an explicit gc call into the loop, which (according to visualvm) succeeds only in smoothing out the spikes in memory use (see image below).
The odd thing is the behavior of memory use: as the attached image makes clear, the usage climbs while the loading loop is working, persists at the plateau while inside the try, but the gc outside the try causes all the memory to reclaimed (the cliff at the end of the plateau).
Is there something about try/catch that influences gc behavior? Any hints about things to check in my code to find a possible leak that I might have introduced?
I've spent many hours on this with a variety of memory/heap management tools, and tracing code, and it's really got me bewildered. If it were a true memory leak in my code, why would the final gc clean up everything?
Many thanks for any suggestions/ideas.
if (message == MenuCommands.TRYLOADINGFILES){
try {
File dir = new File(<directory with 1015 files in it>);
File [] cskFiles = dir.listFiles(ioUtilities.cskFileFilter);
for (int i=0; i<cskFiles.length; i++){
loadDrawingFromFile(cskFiles[i], true);
if (i % 10 == 0) System.gc();
}
DebugUtilities.pauseForOK("pausing inside try");
}
catch (Exception e1){
e1.printStackTrace();
}
DebugUtilities.pauseForOK("pausing outside try");
System.gc();
DebugUtilities.pauseForOK("pausing after gc, outside try");
}
where
public static pauseForOK(String msg){
JOptionPane.showMessageDialog(null, msg, "OK", JOptionPane.INFORMATION_MESSAGE);
}
Followup based on suggestion from Peter, below. histo:live shows almost NO change no matter when run (at pgm startup, before any actions taken, after all files read (when visualvm reports GB of storage being used), after final gc, when visualvm says it's back down to initial stg use). From startup to running the first four categories about double, and the amount of Char stg goes up by about the amount expected for one file processing, but not much else changes.
According to it, it looks like nothing is sticking around. Here are the first 30 or so lines of the histo from just after when the file load loop finishes (before the final gc outside the try).
num #instances #bytes class name
----------------------------------------------
1: 67824 9242064 <methodKlass>
2: 67824 9199704 <constMethodKlass>
3: 6307 7517424 <constantPoolKlass>
4: 6307 6106760 <instanceKlassKlass>
5: 46924 5861896 [C
6: 5618 4751200 <constantPoolCacheKlass>
7: 10590 3944304 [S
8: 19427 3672480 [I
9: 15280 1617096 [B
10: 33996 1584808 [Ljava.lang.Object;
11: 2975 1487144 <methodDataKlass>
12: 40028 1280896 java.util.Hashtable$Entry
13: 45791 1098984 java.lang.String
14: 31079 994528 java.util.HashMap$Entry
15: 10580 973472 [Ljava.util.HashMap$Entry;
16: 6750 817344 java.lang.Class
17: 10427 583912 java.util.HashMap
18: 1521 523224 javax.swing.JPanel
19: 10008 516344 [[I
20: 8291 457176 [Ljava.security.ProtectionDomain;
21: 4022 431800 [Ljava.util.Hashtable$Entry;
22: 774 377712 com.sun.java.swing.plaf.windows.WindowsScrollBarUI$WindowsArrowButton
23: 689 369704 [J
24: 13931 334344 java.util.ArrayList
25: 7625 305000 java.util.WeakHashMap$Entry
26: 8611 275552 java.lang.ref.WeakReference
27: 8501 272032 java.security.AccessControlContext
28: 16144 258304 javax.swing.event.EventListenerList
29: 6141 245640 com.sun.tools.visualvm.attach.HeapHistogramImpl$ClassInfoImpl
30: 426 245376 <objArrayKlassKlass>
31: 3937 220472 java.util.Hashtable
32: 13395 214320 java.lang.Object
33: 2267 199496 javax.swing.text.html.InlineView
It shows basically this same thing no matter at what point in the process it's run. Got basically the same result even without the :live argument. Yet the program definitely will run out of memory if it runs on enough files.
One other item: I took two snapshots using visualvm's Memory Sampling, one at pgm starup and one on the plateau of memory use; the delta shows the expected increase in storage use, including an increase in the count of some structures that's exactly the same as the number of files processed. As each file processing creates one of those structures, it's as if all that intermediate storage is being kept around while inside the try, but can be cleared out afterward.
What's going on?
++++++++++++
Update 22:00 EDT Sunday
Thanks to #Peter Lowrey, #Vampire, and others for suggestions. Tried all those ideas and nothing works. Tried setting -XX:NewSize=1GB and -XX:NewRatio=3, but it didn't help.
The try/catch was a holdover from the original code and is (I belatedly realized) irrelevant in the example. Getting rid of it entirely changes nothing. Just the simple for-loop loading the files causes the same memory growth pattern, followed by the drop
to initial values when the final gc is done.
Following up on #Vampire's suggestion, I tried this variation (with the loads inline, rather than in a block):
loadDrawingFromFile(thecskFile, true);
loadDrawingFromFile(thecskFile, true);
... 20 times
DebugUtilities.pauseForOK("after 20 loads, before gc");
System.gc();
DebugUtilities.pauseForOK("after gc outside try");
The 20 file loads produced proportionally the same amount of growth in Used Heap space (about 400MB) as in the full example, then after the System.gc() above, the heap space used drops instantly back to program initialization levels, just as before.
When that happened I tried an even more basic approach
loadDrawingFromFile(thecskFile, true);
DebugUtilities.pauseForOK("after load ");
System.gc();
.. repeated 20 times
Turns out this work, in the sense that the memory usage never goes 50 MB even after 20 file loads.
So this seems to have to do with threads and thread interruption. Which leads me to mention one more fact: this is an application that runs off a GUI that's started with:
SwingUtilities.invokeLater(new Runnable() {
public void run() { ... }
}
I'm not that familiar with threads and the Swing Utilities, so perhaps this is some form of naive mistake, but it seems to come down to the fact that a lot of non-live objects are not being touched by the gc until the ShowMessageDialog interrupts something.
Additional suggestions welcome.
I suspect you don't have a memory leak. Instead you have having premature promotion of large objects.
If you are creating large objects, e.g. byte[], these go straight in the tenured space. These are only cleaned up on a major or full collections. Most likely you are only triggering minor collections do the large objects are not being freed until a full collection is triggered.
I guess Peter is right, but in case he isn't: You can run out of file descriptors by not closing streams in loadDrawingFromFile. IIRC it manifests itself also by OOM, while you can have tons of free memory. I guess it's not what's happening in your case as the exception message should state it clearly.