We Keep Coding

Apache Ignite bus_adraln error during installation of cache

I have developers that came to me with an interesting issue in regards to Apache Ignite and Java. This is on a HPUX 11.31 IA platform with Apache-Ignite version 1.3.0-incubating and our HP-UX Java build is 1.7.0.13-jinteg_2015_05_22_03_42-b00. The below error is what has been passed to me but I can't find that Apache-Ignite is supported or tested on HPUX and the BUS_ADRALN error seems to always result in C++ search hits. Anyway if some light could be shed on what our issue is it would be greatly appreciated as I have spent two days trying to find anything on it.
ERROR
"siginfo:si_signo=SIGBUS: si_errno=0, si_code=1 (BUS_ADRALN), si_addr=1fffffffa98e8fe9"
this is thrown after the call to create the readyQueue cache in Ignite. This fatal exception means there is an "invalid address alignment" when trying to memory during the installation of cache.
The developers also sent me a couple logs if that would be of assistance.
A bug in Ignite's Jira for the same issue has been created, the bug has more detailed information about the setup and stack causing the error.
https://issues.apache.org/jira/browse/IGNITE-1493

Chris,
This looks very interesting. When writting a class during marshalling Ignite first writes an internal byte flag, followed by some other data. Therefore offset 17 at the moment of crash mentioned in the ticket makes sense: 16 + 1 (our byte).
I have several assumptions here:
1) We are out of bounds of the array.
2) Byte array element length is not 1.
3) Something else (alignment problem? JMV bug?).
Can you please do the following if possible:
1) Provide GridUnsafeDataOutput.bytes array length at the moment of crash. From the crash report I see that there is some array of length 4096 on the stack. Most probably this is GridUnsafeDataOutput.bytes, but we need to be 100% sure.
2) Run the following program on affected machine and provide the output:
public static void main(String[] args) {
sun.misc.Unsafe unsafe = org.apache.ignite.internal.util.GridUnsafe.unsafe();
System.out.println(unsafe.arrayBaseOffset(byte[].class));
System.out.println(unsafe.arrayIndexScale(byte[].class));
}
This way we will know runtime JVM array parameters.

Missing gdb symbols in Backtrace?

I am running this test because I want to see the stacktrace of a program.
Below is my program:
public class NanoTime {
public static void main (String[] args) {
long StartTime = System.nanoTime();
StringBuffer buffer = new StringBuffer();
for(int i=0; i<1000; i++) {
buffer.append("a"); }
long EndTime = System.nanoTime();
long totalTime = EndTime-StartTime;
System.out.println("Total time of calculation ="+ totalTime);
}
}
Now I am using OpenJDK built with debug-level set to slowdebug and also another one set to fastdebug.
I get this output:
[New Thread 0x7ffff7fd3700 (LWP 22532)]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7fd3700 (LWP 22532)]0x00007fffe10002b4 in ?? ()
(gdb) bt
#0 0x00007fffe10002b4 in ?? ()
#1 0x0000000000000202 in ?? ()
#2 0x00007fffe1000160 in ?? ()
#3 0x00007ffff66bd1a9 in execute_internal_vm_tests () at /home/bionix/Openjdk8/hotspot/src/share/vm/prims/jni.cpp:5128
#4 0x00007ffff7fd2550 in ?? ()
#5 0x00007ffff6b08bf8 in VM_Version::get_cpu_info_wrapper () at /home/bionix/Openjdk8/hotspot/src/cpu/x86/vm/vm_version_x86.cpp:395
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
I am confused at the question marks there, as I expected native methods name.
Note: I am even disabling the JIT Compiler : gdb --args java -Xint Test

In ordinary code, gdb relies on debug information (and to a lesser extent the "linker" symbols) to find the names of functions as it unwinds. Debugging information is described in various standards, DWARF being the current best one for Linux. Compilers emit the debugging information that is then read by gdb.
For just-in-time compilers like OpenJDK, there is no agreed-upon solution to emitting debugging information for debuggers to read. And so, as you've found, gdb generally has no idea what is going on.
In fact, as you can see from your trace, gdb can't even really unwind the entire stack. That's what this means:
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
Modern compilers and ABIs tend to require some extra debugging information to unwind as well -- and, again, there's no agreement on how this should work for JIT compilation. GDB has some heuristics it uses to try to unwind when this information isn't available, but as you can see, they sometimes fail.
So, that's the bad news.
The good news is that gdb provides some ways to write unwinders and debug info readers for JITs. And, someone is working on this for OpenJDK. I wasn't able to quickly find the source, but I did find this thread which explains it a little.

This is because jvm is using template interpreter. For that interpreter java bytecode handlers are translated into machine instructions for the current platform during start time. Still you can get the full stacktrace, more info: template interpreter demo
There is also cpp interpreter and that is the interpreter you would expect to see in your stacktraces. But cpp interpreter is not supported anymore and is only available with zero jvm variant.
Also there is an article about the interpreters.

Possible try/catch and memory management issue?

I have a large Java app that processes a large collection of data files, using a try/catch within an actionPerformed (sample code below). It runs out of memory when I get to about 1000 files inside the loop.
Each file load legitimately takes about 1MB of storage, but I've looked carefully and don't see any place where that storage is being hung on to. Each file load is doing just the same thing (ie assigning the same vars), so it ought to be re-using, not accumulating.
I tried inserting an explicit gc call into the loop, which (according to visualvm) succeeds only in smoothing out the spikes in memory use (see image below).
The odd thing is the behavior of memory use: as the attached image makes clear, the usage climbs while the loading loop is working, persists at the plateau while inside the try, but the gc outside the try causes all the memory to reclaimed (the cliff at the end of the plateau).
Is there something about try/catch that influences gc behavior? Any hints about things to check in my code to find a possible leak that I might have introduced?
I've spent many hours on this with a variety of memory/heap management tools, and tracing code, and it's really got me bewildered. If it were a true memory leak in my code, why would the final gc clean up everything?
Many thanks for any suggestions/ideas.
if (message == MenuCommands.TRYLOADINGFILES){
try {
File dir = new File(<directory with 1015 files in it>);
File [] cskFiles = dir.listFiles(ioUtilities.cskFileFilter);
for (int i=0; i<cskFiles.length; i++){
loadDrawingFromFile(cskFiles[i], true);
if (i % 10 == 0) System.gc();
}
DebugUtilities.pauseForOK("pausing inside try");
}
catch (Exception e1){
e1.printStackTrace();
}
DebugUtilities.pauseForOK("pausing outside try");
System.gc();
DebugUtilities.pauseForOK("pausing after gc, outside try");
}
where
public static pauseForOK(String msg){
JOptionPane.showMessageDialog(null, msg, "OK", JOptionPane.INFORMATION_MESSAGE);
}
Followup based on suggestion from Peter, below. histo:live shows almost NO change no matter when run (at pgm startup, before any actions taken, after all files read (when visualvm reports GB of storage being used), after final gc, when visualvm says it's back down to initial stg use). From startup to running the first four categories about double, and the amount of Char stg goes up by about the amount expected for one file processing, but not much else changes.
According to it, it looks like nothing is sticking around. Here are the first 30 or so lines of the histo from just after when the file load loop finishes (before the final gc outside the try).
num #instances #bytes class name
----------------------------------------------
1: 67824 9242064 <methodKlass>
2: 67824 9199704 <constMethodKlass>
3: 6307 7517424 <constantPoolKlass>
4: 6307 6106760 <instanceKlassKlass>
5: 46924 5861896 [C
6: 5618 4751200 <constantPoolCacheKlass>
7: 10590 3944304 [S
8: 19427 3672480 [I
9: 15280 1617096 [B
10: 33996 1584808 [Ljava.lang.Object;
11: 2975 1487144 <methodDataKlass>
12: 40028 1280896 java.util.Hashtable$Entry
13: 45791 1098984 java.lang.String
14: 31079 994528 java.util.HashMap$Entry
15: 10580 973472 [Ljava.util.HashMap$Entry;
16: 6750 817344 java.lang.Class
17: 10427 583912 java.util.HashMap
18: 1521 523224 javax.swing.JPanel
19: 10008 516344 [[I
20: 8291 457176 [Ljava.security.ProtectionDomain;
21: 4022 431800 [Ljava.util.Hashtable$Entry;
22: 774 377712 com.sun.java.swing.plaf.windows.WindowsScrollBarUI$WindowsArrowButton
23: 689 369704 [J
24: 13931 334344 java.util.ArrayList
25: 7625 305000 java.util.WeakHashMap$Entry
26: 8611 275552 java.lang.ref.WeakReference
27: 8501 272032 java.security.AccessControlContext
28: 16144 258304 javax.swing.event.EventListenerList
29: 6141 245640 com.sun.tools.visualvm.attach.HeapHistogramImpl$ClassInfoImpl
30: 426 245376 <objArrayKlassKlass>
31: 3937 220472 java.util.Hashtable
32: 13395 214320 java.lang.Object
33: 2267 199496 javax.swing.text.html.InlineView
It shows basically this same thing no matter at what point in the process it's run. Got basically the same result even without the :live argument. Yet the program definitely will run out of memory if it runs on enough files.
One other item: I took two snapshots using visualvm's Memory Sampling, one at pgm starup and one on the plateau of memory use; the delta shows the expected increase in storage use, including an increase in the count of some structures that's exactly the same as the number of files processed. As each file processing creates one of those structures, it's as if all that intermediate storage is being kept around while inside the try, but can be cleared out afterward.
What's going on?
++++++++++++
Update 22:00 EDT Sunday
Thanks to #Peter Lowrey, #Vampire, and others for suggestions. Tried all those ideas and nothing works. Tried setting -XX:NewSize=1GB and -XX:NewRatio=3, but it didn't help.
The try/catch was a holdover from the original code and is (I belatedly realized) irrelevant in the example. Getting rid of it entirely changes nothing. Just the simple for-loop loading the files causes the same memory growth pattern, followed by the drop
to initial values when the final gc is done.
Following up on #Vampire's suggestion, I tried this variation (with the loads inline, rather than in a block):
loadDrawingFromFile(thecskFile, true);
loadDrawingFromFile(thecskFile, true);
... 20 times
DebugUtilities.pauseForOK("after 20 loads, before gc");
System.gc();
DebugUtilities.pauseForOK("after gc outside try");
The 20 file loads produced proportionally the same amount of growth in Used Heap space (about 400MB) as in the full example, then after the System.gc() above, the heap space used drops instantly back to program initialization levels, just as before.
When that happened I tried an even more basic approach
loadDrawingFromFile(thecskFile, true);
DebugUtilities.pauseForOK("after load ");
System.gc();
.. repeated 20 times
Turns out this work, in the sense that the memory usage never goes 50 MB even after 20 file loads.
So this seems to have to do with threads and thread interruption. Which leads me to mention one more fact: this is an application that runs off a GUI that's started with:
SwingUtilities.invokeLater(new Runnable() {
public void run() { ... }
}
I'm not that familiar with threads and the Swing Utilities, so perhaps this is some form of naive mistake, but it seems to come down to the fact that a lot of non-live objects are not being touched by the gc until the ShowMessageDialog interrupts something.
Additional suggestions welcome.

I suspect you don't have a memory leak. Instead you have having premature promotion of large objects.
If you are creating large objects, e.g. byte[], these go straight in the tenured space. These are only cleaned up on a major or full collections. Most likely you are only triggering minor collections do the large objects are not being freed until a full collection is triggered.

I guess Peter is right, but in case he isn't: You can run out of file descriptors by not closing streams in loadDrawingFromFile. IIRC it manifests itself also by OOM, while you can have tons of free memory. I guess it's not what's happening in your case as the exception message should state it clearly.

Why does this method print 4?

I was wondering what happens when you try to catch an StackOverflowError and came up with the following method:
class RandomNumberGenerator {
static int cnt = 0;
public static void main(String[] args) {
try {
main(args);
} catch (StackOverflowError ignore) {
System.out.println(cnt++);
}
}
}
Now my question:
Why does this method print '4'?
I thought maybe it was because System.out.println() needs 3 segments on the call stack, but I don't know where the number 3 comes from. When you look at the source code (and bytecode) of System.out.println(), it normally would lead to far more method invocations than 3 (so 3 segments on the call stack would not be sufficient). If it's because of optimizations the Hotspot VM applies (method inlining), I wonder if the result would be different on another VM.
Edit:
As the output seems to be highly JVM specific, I get the result 4 using
Java(TM) SE Runtime Environment (build 1.6.0_41-b02)
Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
Explanation why I think this question is different from Understanding the Java stack:
My question is not about why there is a cnt > 0 (obviously because System.out.println() requires stack size and throws another StackOverflowError before something gets printed), but why it has the particular value of 4, respectively 0,3,8,55 or something else on other systems.

I think the others have done a good job at explaining why cnt > 0, but there's not enough details regarding why cnt = 4, and why cnt varies so widely among different settings. I will attempt to fill that void here.
Let
X be the total stack size
M be the stack space used when we enter main the first time
R be the stack space increase each time we enter into main
P be the stack space necessary to run System.out.println
When we first get into main, the space left over is X-M. Each recursive call takes up R more memory. So for 1 recursive call (1 more than original), the memory use is M + R. Suppose that StackOverflowError is thrown after C successful recursive calls, that is, M + C * R <= X and M + C * (R + 1) > X. At the time of the first StackOverflowError, there's X - M - C * R memory left.
To be able to run System.out.prinln, we need P amount of space left on the stack. If it so happens that X - M - C * R >= P, then 0 will be printed. If P requires more space, then we remove frames from the stack, gaining R memory at the cost of cnt++.
When println is finally able to run, X - M - (C - cnt) * R >= P. So if P is large for a particular system, then cnt will be large.
Let's look at this with some examples.
Example 1: Suppose
X = 100
M = 1
R = 2
P = 1
Then C = floor((X-M)/R) = 49, and cnt = ceiling((P - (X - M - C*R))/R) = 0.
Example 2: Suppose that
X = 100
M = 1
R = 5
P = 12
Then C = 19, and cnt = 2.
Example 3: Suppose that
X = 101
M = 1
R = 5
P = 12
Then C = 20, and cnt = 3.
Example 4: Suppose that
X = 101
M = 2
R = 5
P = 12
Then C = 19, and cnt = 2.
Thus, we see that both the system (M, R, and P) and the stack size (X) affects cnt.
As a side note, it does not matter how much space catch requires to start. As long as there is not enough space for catch, then cnt will not increase, so there are no external effects.
EDIT
I take back what I said about catch. It does play a role. Suppose it requires T amount of space to start. cnt starts to increment when the leftover space is greater than T, and println runs when the leftover space is greater than T + P. This adds an extra step to the calculations and further muddies up the already muddy analysis.
EDIT
I finally found time to run some experiments to back up my theory. Unfortunately, the theory doesn't seem to match up with the experiments. What actually happens is very different.
Experiment setup:
Ubuntu 12.04 server with default java and default-jdk. Xss starting at 70,000 at 1 byte increments to 460,000.
The results are available at: https://www.google.com/fusiontables/DataSource?docid=1xkJhd4s8biLghe6gZbcfUs3vT5MpS_OnscjWDbM
I've created another version where every repeated data point is removed. In other words, only points that are different from the previous are shown. This makes it easier to see anomalies. https://www.google.com/fusiontables/DataSource?docid=1XG_SRzrrNasepwZoNHqEAKuZlHiAm9vbEdwfsUA

This is the victim of bad recursive call. As you are wondering why the value of cnt varies, it is because the stack size depends on the platform. Java SE 6 on Windows has a default stack size of 320k in the 32-bit VM and 1024k in the 64-bit VM. You can read more here.
You can run using different stack sizes and you will see different values of cnt before the stack overflows-
java -Xss1024k RandomNumberGenerator
You don't see the value of cnt being printed multiple times even though the value is greater than 1 sometimes because your print statement is also throwing error which you can debug to be sure through Eclipse or other IDEs.
You can change the code to the following to debug per statement execution if you'd prefer-
static int cnt = 0;
public static void main(String[] args) {
try {
main(args);
} catch (Throwable ignore) {
cnt++;
try {
System.out.println(cnt);
} catch (Throwable t) {
}
}
}
UPDATE:
As this getting a lot more attention, let's have another example to make things clearer-
static int cnt = 0;
public static void overflow(){
try {
overflow();
} catch (Throwable t) {
cnt++;
}
}
public static void main(String[] args) {
overflow();
System.out.println(cnt);
}
We created another method named overflow to do a bad recursion and removed the println statement from the catch block so it doesn't start throwing another set of errors while trying to print. This works as expected. You can try putting System.out.println(cnt); statement after cnt++ above and compile. Then run multiple times. Depending on your platform, you may get different values of cnt.
This is why generally we do not catch errors because mystery in code is not fantasy.

The behavior is dependent upon the stack size (which can be manually set using Xss. The stack size is architecture specific. From JDK 7 source code:
// Default stack size on Windows is determined by the executable (java.exe
// has a default value of 320K/1MB [32bit/64bit]). Depending on Windows version, changing
// ThreadStackSize to non-zero may have significant impact on memory usage.
// See comments in os_windows.cpp.
So when the StackOverflowError is thrown, the error is caught in catch block. Here println() is another stack call which throws exception again. This gets repeated.
How many times it repeates? - Well it depends on when JVM thinks it is no longer stackoverflow. And that depends on the stack size of each function call (difficult to find) and the Xss. As mentioned above default total size and size of each function call (depends on memory page size etc) is platform specific. Hence different behavior.
Calling the java call with -Xss 4M gives me 41. Hence the correlataion.

I think the number displayed is the number of time the System.out.println call throws the Stackoverflow exception.
It probably depend on the implementation of the println and the number of stacking call it is made in it.
As an illustration:
The main() call trigger the Stackoverflow exception at call i.
The i-1 call of main catch the exception and call println which trigger a second Stackoverflow. cnt get increment to 1.
The i-2 call of main catch now the exception and call println. In println a method is called triggering a 3rd exception. cnt get increment to 2.
this continue until println can make all its needed call and finally display the value of cnt.
This is then dependent of the actual implementation of println.
For the JDK7 either it detect cycling call and throws the exception earlier either it keep some stack resource and throw the exception before reaching the limit to give some room for remediation logic either the println implementation doesn't make calls either the ++ operation is done after the println call thus is by pass by the exception.

main recurses on itself until it overflows the stack at recursion depth R.
The catch block at recursion depth R-1 is run.
The catch block at recursion depth R-1 evaluates cnt++.
The catch block at depth R-1 calls println, placing cnt's old value on the stack. println will internally call other methods and uses local variables and things. All these processes require stack space.
Because the stack was already grazing the limit, and calling/executing println requires stack space, a new stack overflow is triggered at depth R-1 instead of depth R.
Steps 2-5 happen again, but at recursion depth R-2.
Steps 2-5 happen again, but at recursion depth R-3.
Steps 2-5 happen again, but at recursion depth R-4.
Steps 2-4 happen again, but at recursion depth R-5.
It so happens that there is enough stack space now for println to complete (note that this is an implementation detail, it may vary).
cnt was post-incremented at depths R-1, R-2, R-3, R-4, and finally at R-5. The fifth post-increment returned four, which is what was printed.
With main completed successfully at depth R-5, the whole stack unwinds without more catch blocks being run and the program completes.

After digging around for a while, I can't say that I find the answer, but I think it's quite close now.
First, we need to know when a StackOverflowError will be thrown. In fact, the stack for a java thread stores frames, which containing all the data needed for invoking a method and resume. According to Java Language Specifications for JAVA 6, when invoking a method,
If there is not sufficient memory available to create such an activation frame, an StackOverflowError is thrown.
Second, we should make it clear what is "there is not sufficient memory available to create such an activation frame". According to Java Virtual Machine Specifications for JAVA 6,
frames may be heap allocated.
So, when a frame is created, there should be enough heap space to create a stack frame and enough stack space to store the new reference which point to the new stack frame if the frame is heap allocated.
Now let's go back to the question. From the above, we can know that when a method is execute, it may just costs the same amount of stack space. And invoking System.out.println (may) needs 5 level of method invocation, so 5 frames need to be created. Then when StackOverflowError is thrown out, it has to go back 5 times to get enough stack space to store 5 frames' references. Hence 4 is print out. Why not 5? Because you use cnt++. Change it to ++cnt, and then you will get 5.
And you will notice that when the size of stack go to a high level, you will get 50 sometimes. That is because the amount of available heap space need to be taken into consideration then. When the stack's size is too large, maybe heap space will run out before stack. And (maybe) the actual size of stack frames of System.out.println is about 51 times of main, therefore it goes back 51 times and print 50.

This is not exactly an answer to the question but I just wanted to add something to the original question that I came across and how I understood the problem:
In the original problem the exception is caught where it was possible:
For example with jdk 1.7 it is caught at first place of occurence.
but in earlier versions of jdk it looks like the exception is not being caught at the first place of occurence hence 4, 50 etc..
Now if you remove the try catch block as following
public static void main( String[] args ){
System.out.println(cnt++);
main(args);
}
Then you will see all the values of cnt ant the thrown exceptions (on jdk 1.7).
I used netbeans to see the output, as the cmd will not show all the output and exception thrown.

Translate goto statements to if, switch, while, break, etc

Is there a way to mechanically translate goto statements to if, switch, while, break, and continue statements, etc, or with function calls, objects, anything?

While it is not a good idea, it is possible using loops and swith-case. In the following example the goto variable decides what label (0, 1, 2 or default) to goto when you get to a continue.
int goTo=0;
while(true){
switch(goTo){
case 0:
doSomething();
goTo = 1;
continue;
case 1:
doSomethingElse();
goTo = 2;
continue;
case 2:
doSOmethingDifferent();
goTo = 0;
continue;
default:
return;
}
}

I thought this would be worth sharing here. I saw this on Reddit one day, it's an implementation of goto to an arbitrary line number (in the same .java file) via a custom class loader. It's a fun piece of code. http://steike.com/tmp/goto.zip . I take no credit for it.
Edit:
For those who are curious but don't want to download the zip and run it, for the following file:
public class GotoDemo {
public static void main(String[] args) {
int i = 5;
System.out.println(i);
i = i - 1;
if (i >= 0) {
GotoFactory.getSharedInstance().getGoto().go(4);
}
try {
System.out.print("Hell");
if (Math.random() < 2) throw new Exception();
System.out.println("World!");
} catch (Exception e) {
System.out.print("o ");
GotoFactory.getSharedInstance().getGoto().go(13);
}
}
}
It will print:
3
2
1
0
Hello World!

Considering the complexity that goto's can create based on where its jumping between, it's very doubtful.

The Java language doesn't support the features necessary to simulate gotos to arbitrary position (not even within the same method). You can implement a few simple uses of goto using the constructs you mentioned, but can't implement all possible goto uses this way. (*)
On the byte code level you could probably implement goto (at least within a method), but that byte code could not be mapped back to valid Java code in this case.
As for gotos that cross method or object boundaries: That's definitely a big no-no on the JVM level. The entire Java security model depends on the fact that code is verifyable and thus has only defined entry points (also known as "methods").
(*) Disclaimer: this assumes that you don't want to completely restructure the method to implement the effect of goto, which could also invoke code duplication and obfuscating the "normal" flow of the method. Since you can implement a turing machine in a Java method you can definitely implement "goto" in a Java method ;-)

Sure: (abbreviating slightly for clarity)
int goTo = 0; boolean done = false;
while (!done) {
switch (goTo) {
default:
case 1: System.out.println("We're at line 1!"); goTo = 2; break;
case 2: System.out.println("We're going to line 4!"); goTo = 4; break;
case 3: System.out.println("We're at line 3 and we're done!"); done = true; break;
case 4: System.out.println("We're at 4, going to 2! Screw you, line 3!"); goTo = 2; break;
}
}
Why you would want to do such a thing is beyond me, but hey, you can...

Yes, using a combination of the methods you mentioned...it is possible (anything is possible really, just figuring out how to do it properly is a pain in the ass).
Keep in mind that goto's can result in extremely complex execution paths...and therefore may result in unsightly large amounts of duplicate code in whatever is generated.

In practice, I imagine any given example goto can be translated to something else, particularly if method extraction is an allowed transformation. Is this an abstract question or do you actually have so many goto's that you really need an actual tool? Perhaps the java code itself was machine-translated from something?
I used to put one actual goto into every program I wrote just to annoy the purists.