When i run the below program i got the exception when for loop begins its execution at i=1031521. How to over come memory usage of this type?
class wwww
{
public static void main(String args[])
{
String abc[]=new String[4194304];
String wwf="";
int s_count=524286;
for(int i=0;i<4194304;i++)
{
System.out.println("----------enter--------"+i);
abc[i]=""+i;
System.out.println("----------exit--------"+i);
}
}
}
The exception is:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.
java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390
)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at wwww.main(wwww.java:12)
This is because your your uses up all the heap space allocated to your jvm.
You can use argument while running the program to specify the heap size that you would like to allocate.
This is an example:
java -Xmx256m MyClass
Here a maximum of 256 MB of heap space will be allocated
How to over come memory usage of this type?
Don't perform memory usage of this type. You are creating 4194304 strings, of the general form ""+i. You don't need 4194304 strings of that form all at once. You only need one of them at a time, if any, and you can create it every time you need it.
You could either:
Increase the heap size that you give to your program. This is done via the -Xmx command-line argument to java.
Re-engineer the program to use less memory. Do you really need to keep all those strings in memory at once?
Related
I have a spring batch application, it consumes ~16GB Memory & 75% of CPU(4core X2.5Ghz) and at times it throws out of memory exception.
I want to optimize the Heap allocation & Garbage collection and tried with the following JVM options so resolve the out of memory exception.
I could not understand some of these parameters as I copy pasted directly from an article
JAVA_OPTS="-server -Xmx20480m -Xms512m -XX:+UseConcMarkSweepGC
-XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=30 -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:ParallelCMSThreads=2 -XX:+UseCMSCompactAtFullCollection -XX:+DisableExplicitGC -XX:MaxHeapFreeRatio=70 -XX:MinHeapFreeRatio=40 -XX:MaxTenuringThreshold=0 -XX:NewSize=450m -XX:MaxNewSize=650m"
would it really optimize the Heap allocation & Garbage collection and resolve the out of memory exception.?
First, you need to take a heap dump of the process when it is throwing the OOM error. You can do that by adding -XX:+HeapDumpOnOutOfMemoryError JVM option. After when you have the heap dump try using any of the following tool to analyze your heap dump. Locate which object is growing in the memory and then optimize it. Heap dump analyze tools are :
Eclipse Memory Analyzer
Heap Hero
jxray
This error is usually thrown when there is insufficient space to allocate an object on the Java heap or if the Java process is spending more than 98% of its time doing garbage collection and if it is recovering less than 2% of the heap and has been doing so far the last 5 garbage collection cycles.
I would first use a Java profiler to determine what methods are allocating large numbers of objects on the heap and make sure that they are no longer referenced after they are not needed. If this doesn't fix the issue and I have confirmed that I need all the objects, the other option would be to increase the max heap size of the program.
This could also happen when you are using too many 'String' objects or updating those strings again and again.
Strings are stored in a hashed string pool, which resides in the Heap space. When you manipulate a string, a new string is formed and stored in a different pool (hashed pools) but the original string is not deleted until the garbage collector does it.
If we use StringBuilder or StringBuffer (both are mutable, unlike strings), the space is better utilised.
Read more about strings immutability and why stringbuilder should be preferred when you need a lot of string manipulations to be performed.
StringBuilder-StringBuffer-Strings in java
Why strings are immutable in java?
While decoding an Base64 encoded string to byte array (Have to do this as I have a key which can act on byte array to decrypt), I am getting outOfMemory. What are the effective ways to handle this problem? Should I chunk my input encoded String into partitions of size and then decode it or any other suggestions which are effective please suggest.
Code which was causing the issue.
byte[] encrypted = Base64.decodeBase64(strEncryptedEncodedData);
Stack Trace
DefaultQuartzScheduler_Worker-3
at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48)
at java.lang.StringCoding$StringEncoder.encode([CII)[B (StringCoding.java:300)
at java.lang.StringCoding.encode(Ljava/lang/String;[CII)[B (StringCoding.java:344)
at java.lang.String.getBytes(Ljava/lang/String;)[B (String.java:918)
at org.apache.commons.codec.binary.StringUtils.getBytesUnchecked(Ljava/lang/String;Ljava/lang/String;)[B (StringUtils.java:156)
at org.apache.commons.codec.binary.StringUtils.getBytesUtf8(Ljava/lang/String;)[B (StringUtils.java:129)
at org.apache.commons.codec.binary.BaseNCodec.decode(Ljava/lang/String;)[B (BaseNCodec.java:306)
at org.apache.commons.codec.binary.Base64.decodeBase64(Ljava/lang/String;)[B (Base64.java:669)
Eclipse Memory Analyzer memory usage:
Edit1: Max allowed XMX is 1 GB.
Edit2: JDK version"1.8.0_91"
try to increase max heap size to the JVM using option like this
-Xmx4096m
Please specify the java version you use for this code.
There are more than 10 different types of OutOfMemoryError as listed below and yours might be “10.java.lang.OutOfMemoryError: Direct buffer memory” type. Please verify your exception stack trace to find this matching string to confirm the same. If you see different type, please share it.
I verified that “java.lang.StringCoding$StringEncoder” class you shared in your exception trace uses java.nio.ByteBuffer and other related classes. You can validate the import sections in the below url.
http://cr.openjdk.java.net/~sherman/7040220/webrev/src/share/classes/java/lang/StringCoding.java.html
Java applications can access native memory (not heap memory) for buffer operations (direct byte) to perform speed operations. Some portion of memory is allocated to JVM from native memory for these direct byte buffer operations. If its size is not enough, you can increase it using VM flag –XX:MaxDirectMemorySize= (eg. -XX:MaxDirectMemorySize=10M). Increasing heap memory by using –Xmx flag would not solve this type of outofmemory. Please try MaxDirectMemorySize flag and see whether it solves your problem.
If you want to know more details about this OutOfMemoryError, you can read Java Performance Optimization: How to avoid the 10 OutOfMemoryErrors book.
1.java.lang.OutOfMemoryError: Java heap space
2.java.lang.OutOfMemoryError: Unable to create new native thread
3.java.lang.OutOfMemoryError: Permgen space
4.java.lang.OutOfMemoryError: Metaspace
5.java.lang.OutOfMemoryError: GC overhead limit exceeded
6.java.lang.OutOfMemoryError: Requested array size exceeds VM limit
7.java.lang.OutOfMemoryError: request "size" bytes for "reason". Out of swap space?
8.java.lang.OutOfMemoryError: Compressed class space
9.java.lang.OutOfMemoryError: "reason" "stack trace" (Native method)
10.java.lang.OutOfMemoryError: Direct buffer memory
I am mass-processing very large files. I am calling the following method on each URI in each line:
public String shortenUri(String uri) {
uri = uri
.replace("http://www.lemon-model.net/lemon#", "lemon:")
.replace("http://babelnet.org/rdf/", "bn:")
.replace("http://purl.org/dc/", "dc:")
.replace("http://www.w3.org/1999/02/22-rdf-syntax-ns#", "rdf:");
return uri;
}
Strangely, this leads to the following error:
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.regex.Pattern$BnM.optimize(Pattern.java:5411)
at java.util.regex.Pattern.compile(Pattern.java:1711)
at java.util.regex.Pattern.<init>(Pattern.java:1351)
at java.util.regex.Pattern.compile(Pattern.java:1054)
at java.lang.String.replace(String.java:2239)
at XYZ.shortenUri(XYZ.java:217)
I did increase Xmsand Xmx but it did not help. Strangely, I could also not observe an increased memory usage when monitoring the process. Any suggestions on increasing the performance and memory consumption here?
A quote from Oracle:
Excessive GC Time and OutOfMemoryError
The parallel collector will throw an OutOfMemoryError if too much time is being spent in garbage collection: if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError will be thrown. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small. If necessary, this feature can be disabled by adding the option -XX:-UseGCOverheadLimit to the command line.
The first thing you could try is to increase the heap size even more, for example, fot a few GB with -Xmx4G.
Another option might be to prevent the creation of too many objects by not using the replace method. Instead you could create the Pattern and Matcher objects as needed (see below).
The third option I see is to disable this feature altogether with -XX:-UseGCOverheadLimit
private static final Pattern PURL_PATTERN = Pattern.compile("http://purl.org/dc/");
// other patterns
public static String shortenUri(String uri) {
// other matchers
Matcher matcher = PURL_PATTERN.matcher(uri);
return matcher.replaceAll("dc:");
}
My maths says the following Java program would need approx 8GB (2147483645 * 4 bytes) of RAM:
package foo;
public class Foo {
public static void main(String[] args) throws Exception {
int[] arr = new int[Integer.MAX_VALUE-2];
Thread.sleep(500000L);
}
}
This is backed up by observing the program when running:
But unless you set the max heap to around 12.5GB, the program fails to start:
$ java -Xmx12000m -cp ./ foo.Foo
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at foo.Foo.main(Foo.java:5)
$ java -Xmx12500m -cp ./ foo.Foo
//this works
Can understand the need for a bit of wiggle-room but why do we need so much?
Its because of the MinHeapFreeRatio default value (which is 40%). If you want to "need" less then you have to specify it: e.g. 5%
-XX:MinHeapFreeRatio=5
Also, you need to change memory allocated to the young generation as it plays a important role in memory allocation:
After total available memory, the second most influential factor affecting garbage collection performance is the proportion of the heap dedicated to the young generation.
try this:
java -Xmx9g -XX:MinHeapFreeRatio=1 -XX:MaxNewSize=256m -cp ./ foo.Foo
https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/sizing.html
in my application i am comparing two docx files and creating one html comparison file, when i tried with below 150 or 170 lines of file then there is no issue, while i try to compare the big files like 200 lines or more than that then that time it showing the
java.lang.OutOfMemoryError: Java heap space error,
can any one please help on this?
You are running out of memory because you aren't using the Docx4jDriver class, which makes the diff problem more tractable by doing a paragraph level diff first.
Use it like so:
Body newerBody = ((Document)newerPackage.getMainDocumentPart().getJaxbElement()).getBody();
Body olderBody = ((Document)olderPackage.getMainDocumentPart().getJaxbElement()).getBody();
// 2. Do the differencing
java.io.StringWriter sw = new java.io.StringWriter();
Docx4jDriver.diff( XmlUtils.marshaltoW3CDomDocument(newerBody).getDocumentElement(),
XmlUtils.marshaltoW3CDomDocument(olderBody).getDocumentElement(),
sw);
// 3. Get the result
String contentStr = sw.toString();
System.out.println("Result: \n\n " + contentStr);
Body newBody = (Body) org.docx4j.XmlUtils
.unmarshalString(contentStr);
you can make the heap space bigger with -Xmx and -Xmx as VM Arguments
Here more about Heap Size Tuning or here Heap size
Try increasing the Java heap size using the command line arguments -Xmx<maximum heap size> and -Xms<minimum heap size>.
Also in your code, test that you actually have increased the heap size with the following:
long heapSize = Runtime.getRuntime().totalMemory();
System.out.println("Heap Size = " + heapSize);
Do this before calling Differencer.diff on line 117.
Try profiling your application rather than making assumptions or intelligent guess. You can use visualvm or console that ships with the Jdk.
Also, you can take a heap dump of your application using jmap and then use either jhat or eclipse mat (I prefer this, google it out) to see what's consuming the memory and look out for any unusual behavior.