I am trying to create large RDF/HDT files, which in turn means reading large files into memory, etc. Now, that is not really an issue since the server has 516GB of memory, around 510GB of which are free.
I am using the rdfhdt library to create the files, which works just fine. However, for one specific file, I keep getting an OutOfMemoryError, with no real reason as to why. Here is the stack trace:
Exception in thread "main" java.lang.OutOfMemoryError
at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at org.rdfhdt.hdt.util.string.ByteStringUtil.append(ByteStringUtil.java:238)
at org.rdfhdt.hdt.dictionary.impl.section.PFCDictionarySection.load(PFCDictionarySection.java:123)
at org.rdfhdt.hdt.dictionary.impl.section.PFCDictionarySection.load(PFCDictionarySection.java:87)
at org.rdfhdt.hdt.dictionary.impl.FourSectionDictionary.load(FourSectionDictionary.java:83)
at org.rdfhdt.hdt.hdt.impl.HDTImpl.loadFromModifiableHDT(HDTImpl.java:441)
at org.rdfhdt.hdt.hdt.writer.TripleWriterHDT.close(TripleWriterHDT.java:96)
at dk.aau.cs.qweb.Main.makePredicateStores(Main.java:137)
at dk.aau.cs.qweb.Main.main(Main.java:69)
I am running the Jar file with the tag -Xmx200G. The strange thing is, when looking in 'top', it shows VIRT to be 213G (as expected). However, every time RES climbs to just about 94GB, it crashes with the error above, which I think is strange since it should have more than 100GB left to use. I looked in this question, as the problem seems to be similar to mine, although on a different scale. However, using -verbose:gc and -XX:+PrintGCDetails doesn't seem to give me any indication as to what is wrong, and there is about 500G of swap space available as well.
The, perhaps, strangest thing however is the fact that the specific file I have issues with is not even the largest files. For scale, it has about 83M triples to write, and for other files, up to 200M triples have not been an issue. I am using Java version 1.8.0_66 and Ubuntu version 14.04.3 LTS.
So my question is, if anyone can explain what I am doing wrong? It seems very strange to me that larger files have no issue, but this one does. Please let me know if you need any other information.
Due to Java's maximum array length, a ByteArrayOutputStream can not hold more than 2GB of data. This is true regardless of your current amount of RAM or memory limits. Here's the code you're hitting:
private static int hugeCapacity(int minCapacity) {
if (minCapacity < 0) // overflow
throw new OutOfMemoryError();
You'll have to rewrite your code to not try to keep that much data in a single array.
This seems to be a common issue, however the existing solutions didn't work for me.
I am trying to perform topic modeling in R with the help of the mallet package.
The corpus consists of forum comments and is app. 50 MB large. It is divided in 41.975 files.
Here's my script:
documents_65 <- mallet.read.dir("~/20170315_F65/tm")
# Loading corpus and stopwords
mallet.instances <- mallet.import(id.array = documents_65$id,
text.array = documents_65$text,
"~/stopwords.txt",
token.regexp = "\\p{L}[\\p{L}\\p{P}]+\\p{L}")
topic.model <- MalletLDA(num.topics = 1000)
topic.model$loadDocuments(mallet.instances)
vocabulary <- topic.model$getVocabulary()
word.freqs <- mallet.word.freqs(topic.model)
topic.model$setAlphaOptimization(10, 30)
topic.model$train(1000)
topic.model$maximize(10)
doc.topics <- mallet.doc.topics(topic.model, smoothed=T,
normalized=T)
After this I get the error:
Error in .jcall(wrapper, "[D", "flat_double") :
java.lang.OutOfMemoryError: Java heap space
As many have suggested, I tried
options(java.parameters = "-Xmx1000m")
After that I tried the mallet.doc.topics command again and got the same error. I tried to assign the heap space in the Control Panel (as suggested here: https://www.youtube.com/watch?v=b-D24vnuUMM).
I also tried
gc()
Output:
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 2407281 128.6 6619081 353.5 7415161 396.1
Vcells 32395876 247.2 48628929 371.1 48628929 371.1
as well as assigning more heap space through the command line with this:
java -Xmx2048m
I even tried to peak into the assigned memory with the XLConnect-package and the following commands:
xlcMemoryReport ()
Output:
Amount of free memory in the Java Virtual Machine (JVM): 308.0197 MB
xlcFreeMemory()
.jcall(.jnew("java/lang/Runtime"), "J", "totalMemory")
.jcall(.jnew("java/lang/Runtime"), "J", "maxMemory")
I hope I am not missing something really basic. I can imagine that my corpus is hard to handle and requires a lot of memory (mostly because of the large amount of files).
UPDATE:
I couldn't resolve this issue in R and switched to mallet directly. There I found the batch file with specs for mallet, in which one could see that the java memory allotted to mallet was limited to 1 GB. Since my corpus was around 1.3 GB it wasn't possible at all to load it into Mallet. In the batch file I could easily change this and kept working with mallet directly.
I don't know, if it possible to access the batch file from R.
We have to assume you're running a 64 bit JVM. The max heap you can allocate to a 32 bit JVM is 1.6 GB.
Are you running JDK 1.8? You should be. Perm gen is gone, replaced by managed meta space.
Maybe you can profile the app using VisualVM.
Run Java with the command-line option -Xmx, which sets the maximum size of the heap.
Click Here to see Details
I Hope its helpful for you!
I am trying to connect R to Teradata to pull data directly into R for analysis. However, I am getting the error of,
Error in .jcall(rp, "I", "fetch", stride, block) :
java.lang.OutOfMemoryError: Java heap space
I have tried to set up my R options to increase the max heap size of JVM by doing:
options(java.parameters = "-Xmx8g")
I have also tried to initialize java parameters with rJava function .jinit as: .jinit(parameters="-Xmx8g").
But still failed.
The calculated size of the data should be approximately 3G (actually less than 3G).
You need to make sure you're allocating additional memory before loading rJava or any other packages. Wipe the environment first (via rm(list = ls())), restart R/Rstudio if you must, and modify the options at the beginning of your script.
options(java.parameters = "-Xmx8000m")
See for example https://support.snowflake.net/s/article/solution-using-r-the-following-error-is-returned-javalangoutofmemoryerror-gc-overhead-limit-exceeded
I somehow had this problem in a not reproducible manner, partly solved it with -Xmx8g but run in to problems randomly.
I now found an option with a different garbage collector by using
options(java.parameters = c("-XX:+UseConcMarkSweepGC", "-Xmx8192m"))
library(xlsx)
at the beginning of the script and before any other package is loaded since other packages can load some java things by themselves and the options have to be set before any Java is loaded.
So far, the problem didn't occurred again.
Only sometimes in a long session it can still happen. But in this case a session restart normally solves the problem.
Running the following two lines of code (before any packages are loaded) worked for me on a Mac:
options(java.parameters = c("-XX:+UseConcMarkSweepGC", "-Xmx8192m"))
gc()
This essentially combines two proposals previously posted herein: Importantly, only running the first line alone (as suggested by drmariod) did not solve the problem in my case. However, when I was additionally executing gc() just after the first line (as suggested by user2961057) the problem was solved.
Should it still not work, restart your R session, and then try (before any packages are loaded) instead options(java.parameters = "-Xmx8g") and directly after that execute gc(). Alternatively, try to further increase the RAM from "-Xmx8g" to e.g. "-Xmx16g" (provided that you have at least as much RAM).
EDIT: Further solutions: While I had to use the rJava for model estimations in R (explaining y from a large number of X), I kept receiving the above 'OutOfMemory' Errors even if I scaled up to "-Xmx60000m" (the machine I am using has 64 GB RAM). The problem was that some model specifications were simply too big (and would have required even more RAM). One solution that may help in this case is scaling the size of the problem down (e.g. by reducing the number of X's in the model), or – if possible – splitting the problem into independent pieces, estimating each separately, and putting those pieces together again.
I added garbage collection and that solved the issue for me. I am connecting to Oracle databases using RJDBC.
simply add gc()
When running the following code in Matlab, the physical memory usage of Matlab increases until it reaches 90% of my system memory, and crashes the program. This code is part of a larger optimization scheme that will have many more iterations, but it has been simplified here.
import org.parent.modeling.*
for i = 1:100
rTool = Tool('C:/etc');
model = rTool.getModel();
model.initSystem();
rTool.setModel(model);
rTool.run();
end
rTool and model are stored as Java objects in the workspace.
By selectively removing steps from the bottom to top, I've determined that the rTool.run(); step is causing an incremental memory increase during each iteration.
I've tried using a variety of memory clearing techniques with no success
clear classes, clear java, clear rTool model, clear all
I have checked for fragmentation by running the memory command, but it is insignificant.
Below is the actual code that corresponds to the API documentation (https://simtk.org/api_docs/opensim/api_docs/classOpenSim_1_1RRATool.html)
for i = 1:1:4
import org.opensim.modeling.*
rraTool = RRATool('C:\...filename.xml');
model = rraTool.getModel();
model.initSystem();
rraTool.setModel(model);
rraTool.run();
end
Any suggestions for reducing the physical memory increase?
Matlab: 2013
It is difficult to figure out what you are actually doing here, let alone why it is leaking memory. From what I have been able to deduce, you can call Java from MatLab, and you can call C++ directly from Matlab. I cannot figure out if you are calling the C++ library directly, or calling it via a Java (JNI / JNA) wrapper. And if you are doing the latter ... why.
For the Matlab -> C++ case, this page has some tips on avoiding memory leaks: http://www.mathworks.com.au/help/matlab/matlab_external/memory-management-issues.html
For the Matlab -> Java -> C++ case, the leaks could be due to bugs in the Java wrapper or its associated C++-side wrapper code, or a problem in the way that you are using the wrapper. For instance, there could be some dispose method that you need to call. The details should be in the Java API docs ... which you have not been able to locate ... or in the Java source code itself.
I was hoping that there were some general memory clearing Java commands I could use.
There are no such magic commands. You need to figure out where the memory is leaking and fix that.
I have to find a memory leak in a Java application. I have some experience with this but would like advice on a methodology/strategy for this. Any reference and advice is welcome.
About our situation:
Heap dumps are larger than 1 GB
We have heap dumps from 5 occasions.
We don't have any test case to provoke this. It only happens in the (massive) system test environment after at least a weeks usage.
The system is built on a internally developed legacy framework with so many design flaws that they are impossible to count them all.
Nobody understands the framework in depth. It has been transfered to one guy in India who barely keeps up with answering e-mails.
We have done snapshot heap dumps over time and concluded that there is not a single component increasing over time. It is everything that grows slowly.
The above points us in the direction that it is the frameworks homegrown ORM system that increases its usage without limits. (This system maps objects to files?! So not really a ORM)
Question: What is the methodology that helped you succeed with hunting down leaks in a enterprise scale application?
It's almost impossible without some understanding of the underlying code. If you understand the underlying code, then you can better sort the wheat from chaff of the zillion bits of information you are getting in your heap dumps.
Also, you can't know if something is a leak or not without know why the class is there in the first place.
I just spent the past couple of weeks doing exactly this, and I used an iterative process.
First, I found the heap profilers basically useless. They can't analyze the enormous heaps efficiently.
Rather, I relied almost solely on jmap histograms.
I imagine you're familiar with these, but for those not:
jmap -histo:live <pid> > histogram.out
creates a histogram of the live heap. In a nutshell, it tells you the class names, and how many instances of each class are in the heap.
I was dumping out heap regularly, every 5 minutes, 24hrs a day. That may well be too granular for you, but the gist is the same.
I ran several different analyses on this data.
I wrote a script to take two histograms, and dump out the difference between them. So, if java.lang.String was 10 in the first dump, and 15 in the second, my script would spit out "5 java.lang.String", telling me it went up by 5. If it had gone down, the number would be negative.
I would then take several of these differences, strip out all classes that went down from run to run, and take a union of the result. At the end, I'd have a list of classes that continually grew over a specific time span. Obviously, these are prime candidates for leaking classes.
However, some classes have some preserved while others are GC'd. These classes could easily go up and down in overall, yet still leak. So, they could fall out of the "always rising" category of classes.
To find these, I converted the data in to a time series and loaded it in a database, Postgres specifically. Postgres is handy because it offers statistical aggregate functions, so you can do simple linear regression analysis on the data, and find classes that trend up, even if they aren't always on top of the charts. I used the regr_slope function, looking for classes with a positive slope.
I found this process very successful, and really efficient. The histograms files aren't insanely large, and it was easy to download them from the hosts. They weren't super expensive to run on the production system (they do force a large GC, and may block the VM for a bit). I was running this on a system with a 2G Java heap.
Now, all this can do is identify potentially leaking classes.
This is where understanding how the classes are used, and whether they should or should not be their comes in to play.
For example, you may find that you have a lot of Map.Entry classes, or some other system class.
Unless you're simply caching String, the fact is these system classes, while perhaps the "offenders", are not the "problem". If you're caching some application class, THAT class is a better indicator of where your problem lies. If you don't cache com.app.yourbean, then you won't have the associated Map.Entry tied to it.
Once you have some classes, you can start crawling the code base looking for instances and references. Since you have your own ORM layer (for good or ill), you can at least readily look at the source code to it. If you ORM is caching stuff, it's likely caching ORM classes wrapping your application classes.
Finally, another thing you can do, is once you know the classes, you can start up a local instance of the server, with a much smaller heap and smaller dataset, and using one of the profilers against that.
In this case, you can do unit test that affects only 1 (or small number) of the things you think may be leaking. For example, you could start up the server, run a histogram, perform a single action, and run the histogram again. You leaking class should have increased by 1 (or whatever your unit of work is).
A profiler may be able to help you track the owners of that "now leaked" class.
But, in the end, you're going to have to have some understanding of your code base to better understand what's a leak, and what's not, and why an object exists in the heap at all, much less why it may be being retained as a leak in your heap.
Take a look at Eclipse Memory Analyzer. It's a great tool (and self contained, does not require Eclipse itself installed) which 1) can open up very large heaps very fast and 2) has some pretty good automatic detection tools. The latter isn't perfect, but EMA provides a lot of really nice ways to navigate through and query the objects in the dump to find any possible leaks.
I've used it in the past to help hunt down suspicious leaks.
This answer expands upon #Will-Hartung's. I applied to same process to diagnose one of my memory leaks and thought that sharing the details would save other people time.
The idea is to have postgres 'plot' time vs. memory usage of each class, draw a line that summarizes the growth and identify the objects that are growing the fastest:
^
|
s | Legend:
i | * - data point
z | -- - trend
e |
( |
b | *
y | --
t | --
e | * -- *
s | --
) | *-- *
| -- *
| -- *
--------------------------------------->
time
Convert your heap dumps (need multiple) into a format this is convenient for consumption by postgres from the heap dump format:
num #instances #bytes class name
----------------------------------------------
1: 4632416 392305928 [C
2: 6509258 208296256 java.util.HashMap$Node
3: 4615599 110774376 java.lang.String
5: 16856 68812488 [B
6: 278914 67329632 [Ljava.util.HashMap$Node;
7: 1297968 62302464
...
To a csv file with a the datetime of each heap dump:
2016.09.20 17:33:40,[C,4632416,392305928
2016.09.20 17:33:40,java.util.HashMap$Node,6509258,208296256
2016.09.20 17:33:40,java.lang.String,4615599,110774376
2016.09.20 17:33:40,[B,16856,68812488
...
Using this script:
# Example invocation: convert.heap.hist.to.csv.pl -f heap.2016.09.20.17.33.40.txt -dt "2016.09.20 17:33:40" >> heap.csv
my $file;
my $dt;
GetOptions (
"f=s" => \$file,
"dt=s" => \$dt
) or usage("Error in command line arguments");
open my $fh, '<', $file or die $!;
my $last=0;
my $lastRotation=0;
while(not eof($fh)) {
my $line = <$fh>;
$line =~ s/\R//g; #remove newlines
# 1: 4442084 369475664 [C
my ($instances,$size,$class) = ($line =~ /^\s*\d+:\s+(\d+)\s+(\d+)\s+([\$\[\w\.]+)\s*$/) ;
if($instances) {
print "$dt,$class,$instances,$size\n";
}
}
close($fh);
Create a table to put the data in
CREATE TABLE heap_histogram (
histwhen timestamp without time zone NOT NULL,
class character varying NOT NULL,
instances integer NOT NULL,
bytes integer NOT NULL
);
Copy the data into your new table
\COPY heap_histogram FROM 'heap.csv' WITH DELIMITER ',' CSV ;
Run the slop query against size (num of bytes) query:
SELECT class, REGR_SLOPE(bytes,extract(epoch from histwhen)) as slope
FROM public.heap_histogram
GROUP BY class
HAVING REGR_SLOPE(bytes,extract(epoch from histwhen)) > 0
ORDER BY slope DESC
;
Interpret the results:
class | slope
---------------------------+----------------------
java.util.ArrayList | 71.7993806279174
java.util.HashMap | 49.0324576155785
java.lang.String | 31.7770770326123
joe.schmoe.BusinessObject | 23.2036817108056
java.lang.ThreadLocal | 20.9013528767851
The slope is bytes added per second (since the unit of epoch is in seconds). If you use instances instead of size, then that's the number of instances added per second.
My one of the lines of code creating this joe.schmoe.BusinessObject was responsible for the memory leak. It was creating the object, appending it to an array without checking if it already existed. The other objects were also created along with the BusinessObject near the leaking code.
Can you accelerate time? i.e. can you write a dummy test client that forces it to do a weeks worth of calls/requests etc in a few minutes or hours? These are your biggest friend and if you don't have one - write one.
We used Netbeans a while ago to analyse heap dumps. It can be a bit slow but it was effective. Eclipse just crashed and the 32bit Windows tools did as well.
If you have access to a 64bit system or a Linux system with 3GB or more you will find it easier to analyse the heap dumps.
Do you have access to change logs and incident reports? Large scale enterprises will normally have change management and incident management teams and this may be useful in tracking down when problems started happening.
When did it start going wrong? Talk to people and try and get some history. You may get someone saying, "Yeah, it was after they fixed XYZ in patch 6.43 that we got weird stuff happening".
I've had success with IBM Heap Analyzer. It offers several views of the heap, including largest drop-off in object size, most frequently occurring objects, and objects sorted by size.
There are great tools like Eclipse MAT and Heap Hero to analyze heap dumps. However, you need to provide these tools with heap dumps captured in the correct format and correct point in time.
This article gives you multiple options to capture heap dumps. However, in my opinion, first 3 are effective options to use and others are good options to be aware.
1. jmap
2. HeapDumpOnOutOfMemoryError
3. jcmd
4. JVisualVM
5. JMX
6. Programmatic Approach
7. IBM Administrative Console
7 Options to capture Java Heap dumps
If it's happening after a week's usage, and your application is as byzantine as you describe, perhaps you're better off restarting it every week ?
I know it's not fixing the problem, but it may be a time-effective solution. Are there time windows when you can have outages ? Can you load balance and fail over one instance whilst keeping the second up ? Perhaps you can trigger a restart when memory consumption breaches a certain limit (perhaps monitoring via JMX or similar).
I've used jhat, this is a bit harsh, but it depends on the kind of framework you had.