I have a large Java app that processes a large collection of data files, using a try/catch within an actionPerformed (sample code below). It runs out of memory when I get to about 1000 files inside the loop.
Each file load legitimately takes about 1MB of storage, but I've looked carefully and don't see any place where that storage is being hung on to. Each file load is doing just the same thing (ie assigning the same vars), so it ought to be re-using, not accumulating.
I tried inserting an explicit gc call into the loop, which (according to visualvm) succeeds only in smoothing out the spikes in memory use (see image below).
The odd thing is the behavior of memory use: as the attached image makes clear, the usage climbs while the loading loop is working, persists at the plateau while inside the try, but the gc outside the try causes all the memory to reclaimed (the cliff at the end of the plateau).
Is there something about try/catch that influences gc behavior? Any hints about things to check in my code to find a possible leak that I might have introduced?
I've spent many hours on this with a variety of memory/heap management tools, and tracing code, and it's really got me bewildered. If it were a true memory leak in my code, why would the final gc clean up everything?
Many thanks for any suggestions/ideas.
if (message == MenuCommands.TRYLOADINGFILES){
try {
File dir = new File(<directory with 1015 files in it>);
File [] cskFiles = dir.listFiles(ioUtilities.cskFileFilter);
for (int i=0; i<cskFiles.length; i++){
loadDrawingFromFile(cskFiles[i], true);
if (i % 10 == 0) System.gc();
}
DebugUtilities.pauseForOK("pausing inside try");
}
catch (Exception e1){
e1.printStackTrace();
}
DebugUtilities.pauseForOK("pausing outside try");
System.gc();
DebugUtilities.pauseForOK("pausing after gc, outside try");
}
where
public static pauseForOK(String msg){
JOptionPane.showMessageDialog(null, msg, "OK", JOptionPane.INFORMATION_MESSAGE);
}
Followup based on suggestion from Peter, below. histo:live shows almost NO change no matter when run (at pgm startup, before any actions taken, after all files read (when visualvm reports GB of storage being used), after final gc, when visualvm says it's back down to initial stg use). From startup to running the first four categories about double, and the amount of Char stg goes up by about the amount expected for one file processing, but not much else changes.
According to it, it looks like nothing is sticking around. Here are the first 30 or so lines of the histo from just after when the file load loop finishes (before the final gc outside the try).
num #instances #bytes class name
----------------------------------------------
1: 67824 9242064 <methodKlass>
2: 67824 9199704 <constMethodKlass>
3: 6307 7517424 <constantPoolKlass>
4: 6307 6106760 <instanceKlassKlass>
5: 46924 5861896 [C
6: 5618 4751200 <constantPoolCacheKlass>
7: 10590 3944304 [S
8: 19427 3672480 [I
9: 15280 1617096 [B
10: 33996 1584808 [Ljava.lang.Object;
11: 2975 1487144 <methodDataKlass>
12: 40028 1280896 java.util.Hashtable$Entry
13: 45791 1098984 java.lang.String
14: 31079 994528 java.util.HashMap$Entry
15: 10580 973472 [Ljava.util.HashMap$Entry;
16: 6750 817344 java.lang.Class
17: 10427 583912 java.util.HashMap
18: 1521 523224 javax.swing.JPanel
19: 10008 516344 [[I
20: 8291 457176 [Ljava.security.ProtectionDomain;
21: 4022 431800 [Ljava.util.Hashtable$Entry;
22: 774 377712 com.sun.java.swing.plaf.windows.WindowsScrollBarUI$WindowsArrowButton
23: 689 369704 [J
24: 13931 334344 java.util.ArrayList
25: 7625 305000 java.util.WeakHashMap$Entry
26: 8611 275552 java.lang.ref.WeakReference
27: 8501 272032 java.security.AccessControlContext
28: 16144 258304 javax.swing.event.EventListenerList
29: 6141 245640 com.sun.tools.visualvm.attach.HeapHistogramImpl$ClassInfoImpl
30: 426 245376 <objArrayKlassKlass>
31: 3937 220472 java.util.Hashtable
32: 13395 214320 java.lang.Object
33: 2267 199496 javax.swing.text.html.InlineView
It shows basically this same thing no matter at what point in the process it's run. Got basically the same result even without the :live argument. Yet the program definitely will run out of memory if it runs on enough files.
One other item: I took two snapshots using visualvm's Memory Sampling, one at pgm starup and one on the plateau of memory use; the delta shows the expected increase in storage use, including an increase in the count of some structures that's exactly the same as the number of files processed. As each file processing creates one of those structures, it's as if all that intermediate storage is being kept around while inside the try, but can be cleared out afterward.
What's going on?
++++++++++++
Update 22:00 EDT Sunday
Thanks to #Peter Lowrey, #Vampire, and others for suggestions. Tried all those ideas and nothing works. Tried setting -XX:NewSize=1GB and -XX:NewRatio=3, but it didn't help.
The try/catch was a holdover from the original code and is (I belatedly realized) irrelevant in the example. Getting rid of it entirely changes nothing. Just the simple for-loop loading the files causes the same memory growth pattern, followed by the drop
to initial values when the final gc is done.
Following up on #Vampire's suggestion, I tried this variation (with the loads inline, rather than in a block):
loadDrawingFromFile(thecskFile, true);
loadDrawingFromFile(thecskFile, true);
... 20 times
DebugUtilities.pauseForOK("after 20 loads, before gc");
System.gc();
DebugUtilities.pauseForOK("after gc outside try");
The 20 file loads produced proportionally the same amount of growth in Used Heap space (about 400MB) as in the full example, then after the System.gc() above, the heap space used drops instantly back to program initialization levels, just as before.
When that happened I tried an even more basic approach
loadDrawingFromFile(thecskFile, true);
DebugUtilities.pauseForOK("after load ");
System.gc();
.. repeated 20 times
Turns out this work, in the sense that the memory usage never goes 50 MB even after 20 file loads.
So this seems to have to do with threads and thread interruption. Which leads me to mention one more fact: this is an application that runs off a GUI that's started with:
SwingUtilities.invokeLater(new Runnable() {
public void run() { ... }
}
I'm not that familiar with threads and the Swing Utilities, so perhaps this is some form of naive mistake, but it seems to come down to the fact that a lot of non-live objects are not being touched by the gc until the ShowMessageDialog interrupts something.
Additional suggestions welcome.
I suspect you don't have a memory leak. Instead you have having premature promotion of large objects.
If you are creating large objects, e.g. byte[], these go straight in the tenured space. These are only cleaned up on a major or full collections. Most likely you are only triggering minor collections do the large objects are not being freed until a full collection is triggered.
I guess Peter is right, but in case he isn't: You can run out of file descriptors by not closing streams in loadDrawingFromFile. IIRC it manifests itself also by OOM, while you can have tons of free memory. I guess it's not what's happening in your case as the exception message should state it clearly.
Related
I have a newbie question, so I started looking at Java Affinity library and I have the following code:
public static void main(String[] args){
AffinityLock l = AffinityLock.acquireLock(5);
Thread.currentThread().setName("Testing");
System.out.println("\nThe assignment of CPUs is\n" + AffinityLock.dumpLocks());
while(!Thread.currentThread().isInterrupted()){
}
}
and I have output:
The assignment of CPUs is
0: General use CPU
1: Reserved for this application
2: Reserved for this application
3: Reserved for this application
4: Reserved for this application
5: Thread[Testing,5,main] alive=true
...
But if I go ps -alF, I can see the process is not running on PSR 5. Am I missing something obvious?
Thanks a lot!!
- Mag
A thread could be executed on different cores. Please read this fascinating discussion.
In this particular case you can stop the debugger in the method net.openhft.affinity.LockInventory.set(CpuLayout cpuLayout) and do some steps to find out what happens during initialization of logicalCoreLocks array.
It looks like the dumped data was actual only at the moment of initialization, but it's not when being printed to the console.
I'm modifying a Java class bytecode through an hexadecimal editor, and I want to force a method to always return true.
Replaced all its bytecode with nops to keep the size intact (original size is 1890).
Execute a pop to restore the stack height since it receives an argument.
Return true with iconst_1 followed by ireturn.
public static boolean test(java.lang.String);
descriptor: (Ljava/lang/String;)Z
flags: ACC_PUBLIC, ACC_STATIC
Code:
stack=5, locals=12, args_size=1
0: nop
1: nop
2: nop
3: nop
4: nop
[...]
1886: nop
1887: nop
1888: pop
1889: iconst_1
1890: ireturn
But when executing it, I'm getting the following error
java.lang.VerifyError: (class: com/example/test/TestBytecode, method: test signature: (Ljava/lang/String;)Z) Inconsistent stack height 0 != 1
NOTE: with or without pop the result is exactly the same.
The pop is unnecessary, since the arguments are not on the stack at first. They are only pushed onto the stack when using *load instructions as if they were local variables, which can happen at any time.
pop pops a value from the stack, but the string that is passed as argument is in "local variable" 0.
You should be able to safely omit that pop.
Also, you should be able to leave out all nops, and instead just replace instruction 0 with iconst_1, instruction 1 with ireturn, and leave the entire rest of the method unchanged.
This way you'd be doing less work and probably even increasing performance.
If you are using Java 7 or later, probably the JVM is validating your bytecode with the stack map frames. (Check this question/answer for an explanation of the stack map frames)
If you are using Java 7, try using -XX:-UseSplitVerifier in the command line when you run your class.
If you are using java 8, then you will also have modify the stack map frames; doing so is not trivial, so I better recommend you to use a bytecode manipulation library like javassist.
UPDATE
Based on the comment of #Holger, he is right. However as far as I've seen you are filling unwanted op codes with NOPs rather than removing them.
As you probably already know, the machine instructions are located in an attribute called code; this attribute can have "sub-attributes" (attributes of code itself). One of these is the attribute StackMapTable, which is an "array" (a table) of "stack maps".
Replacing this attribute with zeros won't be enough, you'll have to:
set the number of entries of that table (the stack map table) to zero
delete the only entry on that table (and offset the following attributes and other fields)
still want to do it by hand? :-)
[TL;DR: the following JVM bytecode instructions seems not to work:
iconst_0
istore 6
...sequential
iinc 6 1
jsr L42
...
; L42
iload 6
ifeq L53 ; Always branches!!!
astore 8
iinc 6 -1
; L53
LDC 100
ISUB ; ERROR, returnAddress is at the top of the stack
A test .class can be found here (with slightly more complex logic). If you want to know more about why I'm seeing these instructions, please keep reading.]
I'm writing a Whitespace compiler targeting JVM bytecode. Although being an esoteric language, Whitespace describes an interesting set of assembly instructions to a stack machine, which maps nicely to the JVM.
Whitespace has labels, which are both targets for jump (goto/jump-if-zero/jump-if-negative) and function calls. The relevant instructions (with names given by me, in the spec they are given as combinations of space, tabs and newlines) are:
mark <label>: sets a label for the following instruction
jump[-if-neg|-if-zero] <label>: jumps unconditionally or conditionally to the given label
call <label>: call a function pointed by label
end <label>: ends a function, returning to the caller.
My compiler outputs the whole Whitespace program in the main method of a class. The simplest way to implement call and end is using the JSR and RET opcodes, which are made to implement subroutines. After a JSR operation the stack will contain a returnAddress reference that should be stored in a variable for later use in end.
However, as mark can be either call-ed or jump-ed into, the stack may or may not contain the returnAddress reference. I decided to use a boolean variable (call-bit, at address 6) to store how the mark was reached, and then test if it should store the top of the stack into a local variable (return-address, at address 8). The implementation for each instruction is as follows:
; ... initialization
iconst_0
istore 6 ; local variable #6 holds the call bit
# call
iinc 6 1 ; sets the call bit
jsr Lxxx ; jumps to the given label, pushing a returnAddress to the stack
# mark
; Lxxx
iload 6 ; loads the call bit
ifeq Lxxx-end ; SHOULD jump to mark's end if the call bit is not set
; call bit is set: mark was call-ed and returnAddress is in the stack
astore 8 ; stores returnAddress to local variable #8
iinc 6 -1 ; resets the call bit
; Lxxx-end
# end
ret 8 ; returns using the stored returnAddress
The problem: ifeq ALWAYS branches. I also tried reversing the logic (call-bit -> jump-bit, ifeq->ifne), and even simply switching to ifne (which would be wrong)... but the if always branches to the end. After a call, the returnAddress stays in the stack and the next operation blows up.
I've used ASM's analyzer to watch the stack to debug all this, but have just asserted this behavior and can't find what I'm doing wrong. My one suspicion is that there's more to iinc, or to ifeq than my vain philosophy can imagine. I'll admit that I've only read the instruction set page and ASM's pertinent documentation for this project, but I'd hope that someone can bring a solution from the top of their mind.
In this folder there are the relevant files, including the executable class and the original Whitespace, as well as the output of javap -c and ASM analysis.
Found a possible reason: the problem is not during execution, but with the verifier. When it seemed that it "always branched", was in fact the verifier testing all possible outcomes of an if so it could be sure the stack would look like the same. My code relies on a reference (returnAddress) maybe or maybe not being present on the stack and the verifier can't check that.
That said, the sample code does not run with the -noverify flag, but other, simpler examples that failed verification did execute correctly.
The problem is as follows; the method, in Java code, is:
Rule foo()
{
return sequence(foo(), x());
}
This will provoke a parsing loop which of course should be avoided; however, this is legal:
Rule foo()
{
return sequence(x(), foo());
}
Now, somewhere else in the code I do have access to a RuleMethod, which is a class extending MethodNode, and therefore I do have access to the following information:
ruleMethod.name: foo; (defined in MethodNode)
ruleMethod.desc: ()Lorg/parboiled/Rule; (defined in MethodNode)
ruleMethod.ownerClass: com.github.fge.grappa.experiments.SelfReferringRule.MyParser (defined in RuleMethod
And the bytecode of the first code extract above is as follows:
Method 'foo':
0 L0
1 ALOAD 0
2 ALOAD 0
3 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.foo ()Lorg/parboiled/Rule;
4 ALOAD 0
5 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.x ()Lorg/parboiled/Rule;
6 ICONST_0
7 ANEWARRAY java/lang/Object
8 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.sequence (Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Lorg/parboiled/Rule;
9 ARETURN
10 L1
Which means I have each and every information available to me to be able to spot, at least in the bytecode above, that foo() is the first argument of the sequence() invocation, since the constructor accepts three arguments and there are three elements on the stack.
But of course I can't "eye inspect" at runtime. Therefore I need a way to do this...
It looks like what I need is a MethodVisitor and somewhat visitInsn(), then see what arguments there are and detect appropriately...
But I don't have the slightest idea where to start; searching around on the net seems to only give examples of how to modify byte code, not detect such situations :/
Where do I start?
Analysis is generally much easier using the tree api as it allows you to easily backtrack and provides support for flow analysis.
If I understand your problem correctly, all you need to do (if all you wish to support is simple cases such as your example) is scan backwards from the call to sequence. As you know the code compiles what's on the stack must be valid, so just count back three method calls / field gets / etc.
If you want to support more complex scenarios where the inputs are assigned to variables by branch statements you will need some sort of flow analysis.
Create a MethodVistor and while you are in visitCode() of the method foo() look for visitMethodInsn() and if the name argument in the visitMethodInsn() is foo you know that you have a recursive call to the method.
In your bytecode listing you have three INVOKEVIRTUAL instructions, these instructions are visited in order by the visitMethodInsn() function. If you want to check the sequence you can keep track of the order in which the method calls are made. You will see foo() first followed by x() and then finally sequence().
3 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.foo ()Lorg/parboiled/Rule;
4 ALOAD 0
5 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.x ()Lorg/parboiled/Rule;
6 ICONST_0
7 ANEWARRAY java/lang/Object
8 INVOKEVIRTUAL com/github/fge/grappa/experiments/SelfReferringRule$MyParser.sequence (Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Lorg/parboiled/Rule;
If I get your intention correctly, you want to detect direct left recursion, which parboiled cannot handle. However, left recursion can be handled:
In essence, the parser must detect a left recursion and fail there, but remember that a left recursion occured. When the rule which was used in a recursive way succeeds, the result is saved as "seed" and the parsing process is restarted at the original input position. This time, when the left recursion happens, the seed is used instead of failing. This process is repeated.
For explanation see http://www.vpri.org/pdf/tr2007002_packrat.pdf for the paper. The algorithm can easily be adapted to PEG parsers.
A parsing library similar to parboiled using this technique is https://github.com/ruediste/lambda-peg-parser
for 13K users I have the following memory dump. I will paste the top 7 consumers. Netty seems to consume too much memory. Is this normal ?
(Netty Version:3.2.7, implementing IdleStateAwareChannelUpstreamHandler,Total Memory Netty Memory Usage:2.5GB minimum )
num #instances #bytes class name
----------------------------------------------
1: 23086640 923465600 org.jboss.netty.util.internal.ConcurrentHashMap$Segment
2: 28649817 916794144 java.util.concurrent.locks.ReentrantLock$NonfairSync
3: 23086640 554864352 [Lorg.jboss.netty.util.internal.ConcurrentHashMap$HashEntry;
4: 118907 275209504 [I
5: 5184704 207388160 java.util.concurrent.ConcurrentHashMap$Segment
6: 5184704 130874832 [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
7: 1442915 115433200 [Lorg.jboss.netty.util.internal.ConcurrentHashMap$Segment;
It looks like the memory usage is not normal.
Here are some facts about Netty internal memory usage
One channel has two ReentrantLocks, (one read lock,one write lock)
Channel stores all channel references in a
org.jboss.netty.util.internal.ConcurrentHashMap internally, and automatically
removes on close (This is to assign unique channel ids).
ChannelGroup stores channel references in a org.jboss.netty.util.internal.ConcurrentHashMap on add() and automatically removes on close.
There will be one ConcurrentHashMap$HashEntry per item stored in org.jboss.netty.util.internal.ConcurrentHashMap.
so you can calculate the expected memory usage, if your handlers are not leaking any references.