See here for original question.
I'm writing a Java service, using Jetty for Webserving and SQLite for database storage. Source is available here.
I found that, although the service ran stably from my laptop, when deployed to an EC2 instance it would crash with no obvious error message in output somewhere between 1 hour and 2 days from starting up. I added some logging options to the startup call (-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=...), and got this crashlog.
This is the first time I've encountered such a file, and at first glance it doesn't seem very obvious which part of it refers to the actual error and which parts are just giving contextual information, so I'd really appreciate any good guides to understanding it. In particular, it looks like it's trying to interact with ZipFiles, which I don't use in my project.
This answer links to a blog which was -1'd in the answer comments
This answer links to some Oracle documentation, which I would normally be overjoyed at, but it appears to be a general debugging guide - can't see anything in there about hs_err files (though that may become clear on further reading)
This result from Google claims that "whatever Java code you execute, JVM should never crash. If it does, it is just a JVM bug. So try to log a defect with Sun with all details and hopefully they will look into that.", which sounds pretty absolutist - I think it's highly unlikely that my little toy project has uncovered a JVM bug!
Crashes in Java_java_util_zip_ZipFile_getEntry are most often caused by concurrent access to .zip files, e.g. a file is overwritten while an open instance exists.
See JDK-8042197, JDK-8031691 for details.
BTW, as to original question, here is a presentation on JVM crash dump analysis.
Looking at the crashlog, it appears that something you were unpacking (jar file or zip file) resulted in a JVM crash.
If this occurs during startup, try adding this to your JVM command line to see which file it was possibly working on.
java -verbose:class [more command line]
Related
How is the java utility that begins the process of launching a class told to "spill its guts" on what it's doing as it tries to load classes?
In particular, what file paths is it TRYING to access, only to perhaps discover whatever it's looking for is not there, at least as it interprets the specification given? There was a way to get that information, but I can't find it now.
Note that this is Java version "1.8.0_333" on Windows 10.
I've tried every flag known to me, via the -h and -X flags, and I strongly suspect what I'm looking for is (was) an X flag that's been removed, just as the -X help output warns. And so, there must be an OS way to figure this out, I sure hope!
You might ask why? Whatever for? What are you trying to do? Well, that's the bulk of this question's text. To wit:
As one of the very early users of Java (I started with 1.1) way back in the '90s, I had an issue moving an application suite I'd written for my company on Linux to MS Windows and I got it working by using Cygwin. Along the way, this same sort of issue came up and I quite vividly recall having found a mechanism for getting the Java launcher to articulate just what file specifications - paths - it was actually using in searching for the appropriate class. And through using this, I found that the CLASSPATH was being specified incorrectly, and with some experimentation, I got it working reliably. Now I need to do that again!
This flag I'd used was immensely helpful in figuring out just what the file specification format CLASSPATH needed to be (we're not talking semicolons here) this combination of OS, Java, and Cygwin. After some hours of what I hope was reasonable hunting, I'm wondering if this capability has been removed at some point? Either that or "I'm looking for the wrong thing." Heck, since the source is available (I think!), maybe some brave soul has hacked the java utility to do such a thing?
It may help to understand that for this application I wrote for my company, it was a major goal to have the source work pretty much the same on all Windows and Linux / Unix systems (and at the time, macOS), and just use a configuration file to tell the code what's different. And that wasn't easy to figure out, but with this flag, it wasn't that hard, either.
But, unfortunately, I haven't needed this knowledge since I figured it all out all those years ago, and apparently, this little kernel of knowledge is very hard to find today. Or, it's no longer pertinent to the modern version(s).
I don't think this has anything much to do with the actual problem, but it may help in people's thinking if they understood the scenario: The current situation is that I have a fully functional installation of this software on Windows 7 to use as a comparison for how to configure things on Windows 10 (and hopefully younger). The Windows 7 is running a pretty modern Cygwin installation and very nearly the most modern Java - just a sub-version away from the new installation from last week on a Windows 10 box. (Everything's bright and shiny new on the new box.)
The required format for CLASSPATH on the nearly identical but fully functional Windows 7 system is:
CLASSPATH="C:/opt/OurInstallationDir/lib"
And that's it.
This value is picked up in several places as the code later needs to launch Java itself to do some unusual things. However, the java command that gets it all going is launched from a C program - not that that matters for this problem - but the C program (compiled under Cygwin, but perfectly runnable from any Windows environment) helps ensure that the Java environment is secured (policy file contents and so forth) before getting into Java, else it refuses. And this program on Windows 11 launches Java just fine, it's just giving it a CLASSPATH that isn't useful, apparently, even though the files are there where they should be, etc.
Configuring things as before just doesn't work, even from the command line. No version of specifying CLASSPATH seems to work if it's more than a dot; the only thing that works, is being in the /lib directory when starting and using "-cp ." ... But that's just not going to fly for so many reasons! To be a little more clear, I've tried reversing the slashes, using /cygdrive/c/, and whatever else I could think of. But, at least we know that if you're in the directory and use -cp, it will find and launch the program. So, there's nothing wrong with the Java, just pointing the java utility at it.
Again: How is the java utility that begins the process of launching a class told to "spill its guts" on what it's doing as it's trying to load classes?
You use this construct on the JVM:
java -XX:+TraceClassPaths -cp "C:\opt\SomeDirectory\lib" myClass
I was able to get confirmation of what Java was using, not only for my CLASSPATH, but "internally" by using the above.
The fact that it echoed back both what I was doing and what it was doing somehow gave me the insight to check everything about it. Java itself doesn't work (at all) if it's installed in a location that it thinks has a link in it, and it's own fetches go right back to the system disk specification.
From that I found that Java on Windows won't take a CLASSPATH that has a link in it!
Simply ensuring that the whole tree was specified "from the top" of the drive it's on works. If it's not, it won't.
It's now working happily using the syntax noted above.
This is quite different from every other application I've seen on Windows. But, well, it's Java!
This really came from a pointer from Mark Rotteveel who commented above about this article: How to track when class is loaded and destroyed in jvm? And therein I learned how to get the list of all the options the presently in-use JVM supports. All Java developers should be aware of this in my opinion, so thanks to Mark for that.
i have noticed that signed applets are loaded much slower with the latest plugin (included in java 8u31 and 7u75). I have debugged the situation quite a lot and i found out that the problem is directly related to the size of the jar files that are referenced in the jnlp file. The problem is that each time the applet starts, there is some 're-indexing' of the cached jar files that takes time.
To reproduce the issue i did this:
I created a minimal applet and in the jnlp file i used to deploy it, I added several irrelevant .jar files (that are not even referenced, so the classloader does not load them) of considerable size (e.g. 30MB). Of course i am using versioning in the jnlp and capture all http traffic to make sure that the delays are not because of traffic (either re-downloading or certificate revocation checks, etc). I run the applet with the trace enabled and then went through the xml trace log file and found out where the delays come: they are always from the JarSigningVerifier ....
Has anyone else seen something like that?
It is very easy to see and reproduce this behavior and i wonder if there is something i am overlooking. Having worked on applets for the past years extensively, i am totally lost at what can be happening. I can verify that reverting to the previous version of the plugin (and every other version before) works as expected.
I have submitted a bug report with oracle, but i haven't heard back still. Any info or idea will help,
TIA
Same here. I thought already I'm getting crazy. Thanks for sharing this.
We are using Java Web Start, but it's sharing the same problem of re-indexing all jar files (in our case it's an app with quite some jars, so starting takes ages).
Aside from the fact that Oracle suddenly decided to check the certificate of the deploy TLS, which caused some hickup on Linux and Macs (we used a StartSSL certificate there, which isn't included in the Java keystore - on Windows it works as it uses the platforms root certs, too).
And, to make it even worse, on Windows x86 the 8u31 silently dies if -XX:+DoEscapeAnalysis or -XX:+OptimizeStringConcat is present in the JVM arguments, though both parameters are standard in Java 8 (but not in 7, that's why they've been included, still). The 64bit engine doesn't have that problem.
The next thing they changed is they now overwrite the start icons if they've been changed (we changed them to put the 64bit engine's path in there), so it stubbornly changes the path back to the 32bit engine every time.
The behaviour of Oracle is not helpful at all, as they didn't announce any of these changes in their changelogs, let alone announcing the certs changes a few days ahead.
I would like to hear from anyone who's sharing the problems and possible workarounds.
Patric
Have you been able to solve this issue? Have you had a reaction from Oracle?
UPDATE START
I've tried everything I can think of and haven't managed to solve the
issue, so I posted my own question on this issue.
A similar bug report can be found here and is backported to at
least Java 8u51 which I tested. Yet again they managed to increase
startup time for our application.
END UPDATE
We are using Java Web Start for an Eclipse RCP-based application with jars that are all signed.
Startup time is 8s within the IDE, Java version doesn't seem to matter here. With web start the story is different. It becomes (much) worse with every Java update:
7u?? (<60): +2s (10s)
7u60: +5s (13s)
7u75: +15s (23s)
8u31: +12s (20s)
8u40: +21s (29s)
8u51: +23s (31s)
Have you tried your jnlp without versioning? In my experience Java jnlp is very buggy specially if you change the jnlp default values. Versioning support is disabled by support, so try it without versioning and see if it still slow?
For me, I noticed some bugs when I used update="background" value, so I no longer change the default update method. My theory is that Oracle only tests jnlp against default values before they release new Java versions.
I have made a java program that creates PDF files based to GnuPdf. It runs perfectly when run using native java code (on windows or iSeries QSH), however, when run through an RPGLE interface, the program crashes (at what seems like random intervals) when processing images. I tracked down one of these down to loading an image from a .jar file and removed the call from the code. It worked for a while but is now crashing for images loaded from IFS. Maybe RPGLE is locking the files somehow, and ideas? The code is called from a Service Program.
Here is the stacktrace
java.lang.NullPointerException
at gnu.jpdf.PDFImage.write(PDFImage.java:286)
at gnu.jpdf.PDFOutput.write(PDFOutput.java:114)
at gnu.jpdf.PDFDocument.write(PDFDocument.java:307)
at gnu.jpdf.PDFJob.end(PDFJob.java:182)
at com.mysite.pdf.PdfDocumentStateValid.endDocument(PdfDocumentStateValid.java:657)
at com.mysite.pdf.PdfDocument.endDocument(PdfDocument.java:36)
java.io.IOException: Descriptor not valid.
at java.lang.Throwable.<init>(Throwable.java:196)
at java.lang.Exception.<init>(Exception.java:41)
at java.io.IOException.<init>(IOException.java:40)
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:112)
at gnu.jpdf.PDFOutput.<init>(PDFOutput.java:96)
at gnu.jpdf.PDFDocument.write(PDFDocument.java:302)
at gnu.jpdf.PDFJob.end(PDFJob.java:182)
at java.awt.PrintJob.finalize(PrintJob.java:60)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:116)
at java.lang.ref.Finalizer.access$100(Finalizer.java:47)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:193)
Changing my answer here now that I can see the stack trace. The problem happens when you hit native code (native meaning you are basically drilling into OS level or OS custom code).
You are getting an IOException of Descriptor not valid (and by that I'm assuming it means the file descriptor (ie, FileDescriptor). Now the big difference between running it in QShell and running it from RPG is that the Java code invoked from RPG is probably invoked under a different ID and/or privilege level. You may have to make modifications to the iSeries to your program to grant it authority for Java to have the right authorities to do what it needs. (I know you would think that is something the SecurityManager in Java would have picked up...but I know wierd things can happens sometimes when you are using a custom JVM (read IBM's) on a custom OS (i5/OS). You are left to the mercy of the vendor (in this case IBM). Give that a shot (the permissions thing) and see if that solves your problem.
Also...I googled and just found this related to iSeries: https://www-304.ibm.com/support/docview.wss?uid=nas379538999e744aad1862575b0006e28ab
So it might be that the jt400 library used by the OS may have a flaw and you may need to PTF it and/or your JVM too. Just another thought of something to try.
I'm having some problems tracing the reason that I can't load a java(fx) applet.
The java plugin console for the browser has always been my sole source of information for problem solving. The reason it isn't much help at the moment is that the applet starts to load, downloading the jar files and outputting a percentage to the console along with the occasional statement to say it's on the next jar file. At some point in time (different percentages each time) the console just closes unexpectedly and the applet stops loading. I know this isn't much to go on but I was wondering if there was any way that the output from the console could be stored to a file on the local machine. To debug this particular problem, changing security permissions temporarily would be acceptable.
The problem has occured on every pc i've tried, however if I keep refreshing after failure it seems to cache the jar files previously downloaded and thus get further through the loading process until it eventually works. My issue now is that on a particular customers network refreshing does not resolve the issue. I thought it may be a permissions issue writing to disk but I've tried an administrator account and still no luck. I've also tried a variety of browsers. It might also be worth noting that they go through a proxy server - when the applet tries to load it asks for the credentials for logging on to the proxy which seems to authorise fine.
If anyone has a suggestion on what I could try it would be gratefully received.
Thanks,
James
Two points:
You should be able to get a stack trace through the applet console viewer (which runs in the system tray on PC's) and if you open that up it will show you your stack trace for debugging.
Can you reproduce this problem using the Java Applet Viewer tool? This will allow you to write unit tests, and debug much more easily.
I forget where it is, but somewhere in the java control panel (one of the options under the advanced tab I believe) there's an option to turn on logging. This will log all output to the java console to a file as well. I've used this when trying to debug issues similar to this.
There's some info here on where the files will appear:
http://download.oracle.com/javase/1.5.0/docs/guide/deployment/deployment-guide/tracing_logging.html
The problem ended up being some JS code that was making calls to the applet before the applet had initialised.
Recently two of our clients have reported problems with our applets. Looking at the java plugin console it is full of ClassNotFoundException so none of our code is executed.
I've been able to reproduce the stack trace using a virtual pc image with 0 free space on disk, but the problem goes away as I restore some disk space, and the users tell me that their disk is not full; they are able to create new files.
Our applet requires java 6, and the problem has appeared with updates 1, 10 and 14 of the jre. We have also tried different browsers (IE and Firefox), clearing the browser and java caches, ...
How can I debug or trace what is the jvm doing to load our applet?
I suppose that the problem lies on some security directive on windows so I'm using Sysinternal's Process Monitor to log the activity but I don't really know where to look at.
The Java cache is most likely messed up. Open Java in the Control Panel and get rid of all temporary files to see if it goes away.
Logging in the Java Console for loading applets can be enabled which helps quite alot (even if the plugin is extremely cryptic). See http://java.sun.com/javase/6/docs/technotes/guides/deployment/deployment-guide/tracing_logging.html
I've been looking at the plugin source from the JDK and I've found that there is additional debug info available in the logs setting the environment variable JPI_PLUGIN2_DEBUG.
Unfortunately I still only see ClassNotFound exceptions.
I've found a piece of the plugin code that swallows all exceptions, so maybe my users problem is there...
had similar problems a while ago. In our case, the problem seems to be how the applet tag is set up on the web page. If it is in a wrong order or contains the codebase attribute, it fails with 6u10+. This works for us:
<applet name="DMGANTT" archive="DMGantt.zip" code="dm.applet.DMGanttApplet"
width='100%' height='100%' mayscript="mayscript">
We had a similar problem with one of our clients. We discovered that it was a strange bug in some versions of Java related to proxy configuration on the client. See this article for the details
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6723715
Also, our applet failed with new versions for a different reason.
Now, in similar cases I ask them to check javatester.org, so that I can see if applets (in general) load OK in their browser. If that page loads OK, its a problem on our applet (or page). Else it's a problem in their configuration. I help them anyway but it's easier to debug.