I'm making an application in Java using Eclipse Indigo. When I run it using Eclipse the Task Manager shows javaw.exe is using 50mb of memory. When I export the application as a runnable .jar and execute the .jar the Task Manager shows javaw.exe is using 500mb.
Why is this? How could I fix this?
Edit: I'm using a Windows 7 64 bit, and my system says I have Java 1.7 installed. Apparently the memory problem is caused by a while loop. I'll study what's inside the while loop causing the problem.
Edit: Problem found. At one point in the while loop new BufferedImage instances where created, instead of replacing the same BufferedImage.
Without any additional details about your code, I would suggest using a profiler to analyze the problem. I know YourKit and the one that is available for NetBeans are very good.
Once you run you app from the profiler, you should initially look at the objects and listeners created by your application's packages. If the issue is not there, you can expand your search to other packages until you identify things that are growing out-of-control, and then look at the code that handles those entities.
When you run certain parts of the code multiple times and still see memory utilization after that code stopped running, then you might have a leak and may consider nulling or emptying variables/listeners on exit.
It should be a good starting point, but please report your results back, so we know how it goes. By the way, what operating system are you using and what version of java?
--Luiz
You need to profile your code to get the exact answer, but from my experience when I see similar things I often equate it to garbage collecting. For example, I ran the same job and gave one job 10 gigs and the other 2 gigs..Both ran and completed but the 10gigs one used more memory(and finished faster) while the second(2gig) one, I believe, garbage collected so it still completed but took a bit more time with less memory. I'm a bit new to java so I maybe assuming the garbage collecting but I have seen what you are talking about.
You need to profile your code(check out jconsole, which is included with java, or visualVM)..
That sounds most peculiar.
I can think of two possible explanations:
You looked at the wrong javaw.exe instance. Perhaps you looked at the instance that is running Eclipse ... which is likely to be that big, or bigger.
You have (somehow) managed to configure Java to run with a large heap by default. On Linux you could do this with a wrapper script, a shell function or a shell alias. You can do at least the first of those on Windows.
I don't think it is the JAR file itself. AFAIK, you can't set JVM parameters in a JAR file. It is possible that you've somehow included a different version of something in the JAR file, but that's a bit of a stretch ...
If none of these ideas help, try profiling.
Problem found. At one point in the while loop new BufferedImage instances where created, instead of replacing the same BufferedImage.
Ah yes. BufferedImage uses large amounts of out-of-heap memory and that needs to be managed carefully.
But this doesn't explain why your application used more memory when run from the JAR than when launched from Eclipse ... unless you were telling the application to do different things.
Related
An application I have uses Java Agents with need large jar Libraries (the biggest one is PDFBox - all in all 11MB). They were running for 3 years without any issue with the jars in jvm/lib/ext.
During an upgrade to Domino 9.0.1FP6 the administrator forgot to reinstall the jars in jvm/lib/ext - with obvious repercussions. (Such an annoyance that IBM just completely replaces the whole jvm sometimes without being gentle to the jars)
Upon request, I changed the code by including the jars directly into the Java Agents. Things worked well for 2-3 days, and now we're getting OutOfMemory errors.
As far as I understand it, the jars get loaded onto the Java Heap when the agents get started, but the garbage collection is working slower than the continuous loading of the jars into the heap. I couldn't find any precise documentation by IBM on this matter.
We've increased JavaMaxHeapSize in the notes.ini of the servers but that didn't bring the expected results.
I'm dismissing the possibility that I have forgotten a recycle() in my code because it run beforehand with no memory leaks for three years.
I have thought of the possibility of running a separate Agent that checks total memory usage and then runs Sytem.gc() but I'm not convinced since I have no guarantee that the garbage collector will actually fire.
Apart from the obvious move of putting back the jars in jvm/lib/ext, is there an alternative that I haven't considered?
And is there anywhere some sort of documentation about how these classes get loaded into the Heap, and whether there's a possibility that the jars erroneously are not recognized as being garbage-collectible?
It's a memory leak bug - see http://www-01.ibm.com/support/docview.wss?uid=swg1LO49880 for details.
You need to go back to placing the jar files in jvm/lib/ext.
Here's my issue. I have an existing .jar file that I must use in my program. The program, however, is written in Python.
Since my program is taking a long time to run (a named entity tagger on a large development corpus) I profiled it using cProfiler and lined profiled it using line_profiler. It seems that 92% of the time is spent on this task.
I am currently using the following code:
import subprocess as sub
sub.call(["java", "-jar", "-Xmx512m", "MyFile.jar",
featuresFileName, numIterations, featureCutOff])
I read somewhere about subprocess vs Popen and other bits and pieces, but couldn't find a good solution that does not require subprocess or os calls (of course, there may not be any).
I'd really appreciate some advice on the fastest way to run a .jar file from within a Python script. Note, however, that I cannot modify the Java code nor do I have access to speak to the developer of that code.
Alternatively, and I don't know if this will help or if I'm simply grasping at straws here, but perhaps there is a way to keep the process called in sub.call() above in the background, somehow keeping the JVM running (?) so that I can simply invoke the jar file. Maybe that can help reduce startup costs? BTW I am a total Java newbie (mostly C++,C#,Python experience) so my question could make no sense whatsoever - I apologize in advance...
You could try porting your Python to Jython, and then run it all natively in the same JVM (that may or may not work). That way you have effectively zero start up time, and the JVM has enough time to leverage its JIT over time to ideally give you better performance overall.
That indicates that most of the time is spent in this process. It may not be the startup time which is the problem. It may be what it does once it has started.
The only way around this I can think of is to run the process in the background, multiple time concurrently if that is an option. (concurrently rather than running one after another)
Try with "-client" option. It should reduce JVM startup time.
By analysing the manifest file of the jar file you can find out the class name of the jar file which is used. So then you could in principle write your own small java daemon which is listening for new arguments to arrive and calls the main() function of the appropriate class. But it is really worth the effort only if startup costs are the issue.
I assume the latest update version of java would provide better performance.
I am looking for a way to implement isolation of software components from endless loops or memory leaks. Android isolates each app in it's own process, Google Chrome isolates each tab in it's own process.
My primary drawback is that java takes so long to start and also I would like to reduce memory consumption.
Is there any alternate build or more controlled startup that will accomplish this?
If quick startup is your goal, Java on a PC may not be your best bet. It's going to take a few seconds because that's how long it takes to load the VM from disk.
If you want your app to start more quickly it's easy to get a splash screen up, just create a module that only loads your splash screen, waits for it to fully display then uses reflection to link to your "Real" main module.
(Use reflection because otherwise it will pull in your entire program through references before it starts the main one--at least that's how it used to work).
If you're talking about run-time performance, you won't get quicker by changing languages, Java's about as fast as you can get. You MIGHT be able to get a boost by converting to C/C++ and rewriting it to suit those platforms (Less OO, stack allocations instead of heap, etc), but otherwise none of the other languages in general usage are close to Java in speed.
If you really need the quick startup, depending on what you are doing there may be some tricks. I've seen projects that try to keep a Java VM running in your toolbar and allow you to make requests (tell it to start an app). This was faster but made additional requirements of the user (Loading this additional tool)
Another possibility--if you are constantly starting up/shutting down small tasks and that's the reason the startup bothers you then you can definitely speed it up by keeping it running invisibly. Just have your Java app open a socket and listen for commands then create a little .EXE or shell script that can start your program if it's not running or send commands to that socket if it is. This would completely eliminate startups after the first run.
In general, Java has a much longer startup time than other languages. If you are sticking with Java on a desktop app, a lot of stuff like startup time is determined by the JRE installed on the client's computer, which you can't control.
As to "endless memory leaks"... Java doesn't leak memory. If your program does, fix it.
This is a second answer because it's completely different and my other got too long :)
Try compiling it--I think GCC can compile it. This could almost completely eliminate your startup. I believe Jikes used to be a windows java compiler by IBM, but I don't know if it's still maintained.
Note that compiled code will probably run slower than JVM code for long-running apps.
I have a game implemented in Java that was having a problem when running too much code from scripts: depending on the script language, the game could have these "hiccups" where the game would freeze for a couple frames every now and then, making the game "jerky" at times.
After some research, I discovered that was happening when the Garbage Collector decided to run. After some more research and testing, I discovered that using the incremental garbage collector (by using the -Xincgc VM parameter) fixed the problem. Yes, I am aware it makes the GC run about 10% slower, but it is the cost I pay.
With that background information, now what I want to do is package the game in a .jar like I have been doing so far, but I know no way of making the application use the incremental GC without using -Xincgc, and I didn't want to create .bat/.sh files where before the user only had to double-click the .jar.
Is there any way to make the .jar run with the incremental GC without needing some kind of loader (.bat/.sh) or wrapper around it? Is there some way to include that parameter in the .jar?
No, there's no way to do that. VM arguments can only be supplied as command-line parameters to JVM invocation: Java Tool
You can definitely do this with JNLP, which can be configured to add a shortcut to the desktop and automatically keep jars up to date, among other things.
From just a jar file, you can't do this. Adding Java WebStart can allow you to add java VM commands. If you are not against wrapping the jar in an executable Launch4J will wrap the jar in a executable as well. It only makes a windows exe, but it can make it on linux or windows. I haven't found a good alternative for linux.
I don't know of too many java apps outside of internal coorporate or dev stuff that people run via clicking a jar. Even a shortcut can specify this. Fancier options exist that let you create an exe, run as a service, or even wrap the whole thing in an installer... if we are talking windows, similar things exist for linux.
Besides Java Web Start, you could consider wrapping with JSMooth. It gives an EXE which in turn invokes the JVM - you can provide arguments there.
One option - I always wondered - is to re-run the jar program with Runtime.exec() call when the parameters are inadequate? Never tried it but seems possible.
I'm not experienced with java applications but I found out that finding static pointers etc. to these applications' memory addresses is often (nearly) impossible, apparently because of the java engine that handles the code (correct me if this way of naming it is wrong please).
Now, I've used VisualVM (https://visualvm.dev.java.net/) and it's great. I can select my java process and create a heap dump. It then shows me all classes and their values.
Can I use this method to continousely poll the heap dump and receive object values, for example the X Y and Z of a game? How would I programmatically interact with such application, and if this should not be done with VisualVM, what would be an alternative?
Edit: this is what I need to do:
I need to be able to find all classes with properties that have a certain value. For example: I'd search for the X coordinate (a float) and it should return the class "PlayerCoordsHandler" (just an example) and the corresponding float with it's value... or alternatively just a way to find this same float again (after restarting for example). This process does not have to be programmatic, aslong as requesting the value of the now known property (x float) can be retrieved programmatically (for example with a command line utility or reading from a file).
Edit2:
The target application is a windows executable (but made with java) and launches it's own java VM. It's not possible to add java parameters for debugging. This does not seem to be required though, as VirtualVM is able to debug the process just fine. Anyone knows how?
Thanks in advance.
It looks like you want to debug running Java applications.
The "official" Java debugger is JDB. I believe it's part of the JDK. It has the ability to set breakpoints, examine heaps, list and display and even change variables, show running threads and so on. The usual debugger stuff. But it's command line, which makes it a pain in the neck to work with.
Instead, it makes a lot of sense to use an IDE with integrated debugger. I use Eclipse. You can do all the usual debuggery things, including displaying windows with variables. You can set conditional breakpoints and there's much more. Specifically in answer to your question, you can set up watch expressions, which will be evaluated during the program's execution and their displays refreshed with new values when they change.
You may not want to run your Java app inside the IDE; or it may be running in a Web application server. That's no problem for JDB or Eclipse (or other IDEs, like NetBeans or IntelliJ Idea): They can connect to a running JVM and debug remotely with the same level of convenience.
A program being debugged like this, remotely or otherwise, run somewhat more slowly than if it were not. Your game, while being debugged, will run at rather bad-looking FPS; but it should still respond more or less normally to gameplay interaction.
Remote debugging:
To be able to attach your EclipseNetBeans debugger to a running Java process you need to start that process with the following Java options…
-Xdebug -Xrunjdwp:transport=dt_socket,address=3704,server=y,suspend=n
Have a look at YourKit. You can monitor CPU, memory and threads live, and generate dumps whenever you want. It can even compare different memory dumps to show you which objects were added/removed.
It's not free though, it has a 15 day (or 30 day?) fully functional eval period. If free is not a real concern it's definitely a great tool.
I good starting point is the jps and jstat tools added in Java 6 (i think). jps gives you the pid and main class for each application. jstat give you more details about process
Triggering a heapdump is usefull for post-mortem analysis of say memory leaks, but as the Java garbage collector moves objects around, you cannot use the memory values of a heapdump to reliably access those objects.
If you need a way to query internal values from outside of the application you could look into setting up an RMI service API via which you can retrieve the values you need.
Another method (if you just need to test something) could be to connect to the process via de Java debugging API.
If you know the JRE location that is used, you could rename java.exe and write a (C/C++) wrapper that adds the debug options listed by Carl and calls the renamed_java.exe in turn.
Another posibility might be to add or update classes in the .jar file of the application. You do not need the source to do this.
Tom, are you trying to reverse engineer an application that specifically tries to obfuscate its working? If so you might get further if you contact the manufacturer and ask them what possibilities they see for what you try to achieve?
You can easily generate a heap dump by creating your own JMX connection to the JVM, just like VisualVM does it. Analyzing the heapdump is very possible (the data is there and totally disconnected from the JVM so there is no interference from the gc).
However, unless it is a very specific scenario you are looking for you are probably much better off giving the heapdump to MAT and find a good workflow in there to use.
Edit: In this particular case it is probably better to create some kind of specific API to access the values from the outside (and maybe publish the values as MBeans using JMX). Taking a heap dump is way to much work if all you want to do is monitoring a few values.
Edit2: Based on your edits, it seems to me like you could really benefit from publishing your own MBean over JMX. I have to run for a meeting but, unless someone else does it while I am away, I will try to remember to give you some pointers later. Either in an edit of this one or in a new post.
If you want to poll the values of specific objects while your Java application is running you would probably find that using JMX is a better and more efficient approach rather than using a heap dump. With JMX you can define what values should be exposed and use tools such as VisualVM or JConsole to view them at runtime.
With VisualVM and heapdump you can find all classes with certain property by OQL:
var out = "";
var cls = filter(heap.classes(), "/java./(it.name)")
while (cls.hasNext()) {
var cl = cls.next();
var fls = cl.fields;
while (fls.hasMoreElements()) {
var fl = fls.nextElement();
if (/size/(fl.name)) {
out = toHtml(cl) + "." + fl.name + "()\n";
}
}
}
out.toString()
and write custom logging for BTrace
It is alternative for debugging.
FusionReactor could be a good alternative. For example;
VisualVM doesn’t give you a lot of insides on application memory
except for the total Heap allocation. Heap is a good metric to start
with, but I feel this is not enough to troubleshoot the actual cause
of a memory-related issue.
FusionReactor will display all of the memory spaces it detects, which
depends on the version of Java you’re running:
Heap allocation Non-Heap allocation CodeHeap (profiled and
non-profiled methods) Compressed Class Space FusionReactor also shows
the amount of memory that each generation takes Eden Space Old Space
Survivor Space
https://www.fusion-reactor.com/blog/java-visualvm-alternatives/