We Keep Coding

OutOfMemoryError with apache commons Base64 static method decodeBase64

While decoding an Base64 encoded string to byte array (Have to do this as I have a key which can act on byte array to decrypt), I am getting outOfMemory. What are the effective ways to handle this problem? Should I chunk my input encoded String into partitions of size and then decode it or any other suggestions which are effective please suggest.
Code which was causing the issue.
byte[] encrypted = Base64.decodeBase64(strEncryptedEncodedData);
Stack Trace
DefaultQuartzScheduler_Worker-3
at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48)
at java.lang.StringCoding$StringEncoder.encode([CII)[B (StringCoding.java:300)
at java.lang.StringCoding.encode(Ljava/lang/String;[CII)[B (StringCoding.java:344)
at java.lang.String.getBytes(Ljava/lang/String;)[B (String.java:918)
at org.apache.commons.codec.binary.StringUtils.getBytesUnchecked(Ljava/lang/String;Ljava/lang/String;)[B (StringUtils.java:156)
at org.apache.commons.codec.binary.StringUtils.getBytesUtf8(Ljava/lang/String;)[B (StringUtils.java:129)
at org.apache.commons.codec.binary.BaseNCodec.decode(Ljava/lang/String;)[B (BaseNCodec.java:306)
at org.apache.commons.codec.binary.Base64.decodeBase64(Ljava/lang/String;)[B (Base64.java:669)
Eclipse Memory Analyzer memory usage:
Edit1: Max allowed XMX is 1 GB.
Edit2: JDK version"1.8.0_91"

try to increase max heap size to the JVM using option like this
-Xmx4096m

Please specify the java version you use for this code.
There are more than 10 different types of OutOfMemoryError as listed below and yours might be “10.java.lang.OutOfMemoryError: Direct buffer memory” type. Please verify your exception stack trace to find this matching string to confirm the same. If you see different type, please share it.
I verified that “java.lang.StringCoding$StringEncoder” class you shared in your exception trace uses java.nio.ByteBuffer and other related classes. You can validate the import sections in the below url.
http://cr.openjdk.java.net/~sherman/7040220/webrev/src/share/classes/java/lang/StringCoding.java.html
Java applications can access native memory (not heap memory) for buffer operations (direct byte) to perform speed operations. Some portion of memory is allocated to JVM from native memory for these direct byte buffer operations. If its size is not enough, you can increase it using VM flag –XX:MaxDirectMemorySize= (eg. -XX:MaxDirectMemorySize=10M). Increasing heap memory by using –Xmx flag would not solve this type of outofmemory. Please try MaxDirectMemorySize flag and see whether it solves your problem.
If you want to know more details about this OutOfMemoryError, you can read Java Performance Optimization: How to avoid the 10 OutOfMemoryErrors book.
1.java.lang.OutOfMemoryError: Java heap space
2.java.lang.OutOfMemoryError: Unable to create new native thread
3.java.lang.OutOfMemoryError: Permgen space
4.java.lang.OutOfMemoryError: Metaspace
5.java.lang.OutOfMemoryError: GC overhead limit exceeded
6.java.lang.OutOfMemoryError: Requested array size exceeds VM limit
7.java.lang.OutOfMemoryError: request "size" bytes for "reason". Out of swap space?
8.java.lang.OutOfMemoryError: Compressed class space
9.java.lang.OutOfMemoryError: "reason" "stack trace" (Native method)
10.java.lang.OutOfMemoryError: Direct buffer memory

Repeated replace calls lead to java.lang.OutOfMemoryError

I am mass-processing very large files. I am calling the following method on each URI in each line:
public String shortenUri(String uri) {
uri = uri
.replace("http://www.lemon-model.net/lemon#", "lemon:")
.replace("http://babelnet.org/rdf/", "bn:")
.replace("http://purl.org/dc/", "dc:")
.replace("http://www.w3.org/1999/02/22-rdf-syntax-ns#", "rdf:");
return uri;
}
Strangely, this leads to the following error:
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.regex.Pattern$BnM.optimize(Pattern.java:5411)
at java.util.regex.Pattern.compile(Pattern.java:1711)
at java.util.regex.Pattern.<init>(Pattern.java:1351)
at java.util.regex.Pattern.compile(Pattern.java:1054)
at java.lang.String.replace(String.java:2239)
at XYZ.shortenUri(XYZ.java:217)
I did increase Xmsand Xmx but it did not help. Strangely, I could also not observe an increased memory usage when monitoring the process. Any suggestions on increasing the performance and memory consumption here?

A quote from Oracle:
Excessive GC Time and OutOfMemoryError
The parallel collector will throw an OutOfMemoryError if too much time is being spent in garbage collection: if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError will be thrown. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small. If necessary, this feature can be disabled by adding the option -XX:-UseGCOverheadLimit to the command line.
The first thing you could try is to increase the heap size even more, for example, fot a few GB with -Xmx4G.
Another option might be to prevent the creation of too many objects by not using the replace method. Instead you could create the Pattern and Matcher objects as needed (see below).
The third option I see is to disable this feature altogether with -XX:-UseGCOverheadLimit
private static final Pattern PURL_PATTERN = Pattern.compile("http://purl.org/dc/");
// other patterns
public static String shortenUri(String uri) {
// other matchers
Matcher matcher = PURL_PATTERN.matcher(uri);
return matcher.replaceAll("dc:");
}

string literals and permanent generation memory area

When we say that interned strings are stored in permanent generation area then does the same applies for string literals also? Or it is only for strings interned by inter()?
Actually blog posts usually say that string pool contains reference to string object while actual string object is somewhere in heap. also there is very much confusion that whether permanent generation is IN heap or outside of it. (i used jcosole it is showing permanent gen different from heap.many posts say it as a part of heap and many say it is different)
Edit:
Also when I ran:
public class stringtest2{
public static void main(String args[]){
int i=0;
List<String> list=new ArrayList<String>();
while(true){
String s="hello"+i;
String s1=i+"hello";
String s2=i+"hello"+i;
System.out.println(s);
s.intern();
s1.intern();
s2.intern();
list.add(s);
list.add(s1);
list.add(s2);
i++;
}
}
}
I was expecting Java.lang.OutOfMemoryError: PermGen space But i got :
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2760)
at java.util.Arrays.copyOf(Arrays.java:2734)
at java.util.ArrayList.ensureCapacity(ArrayList.java:167)
at java.util.ArrayList.add(ArrayList.java:351)
at stringtest2.main(stringtest2.java:20)
Shouldn't it be Java.lang.OutOfMemoryError: PermGen space

When we say that interned strings are stored in permanent generation area then does the same applies for string literals also?
Literal strings are interned. So yes, in Java 6-.
From Java 7, interned strings are not stored in permanent generation any longer. They are stored in the main part of the heap like any other objects you would create.
Shouldn't it be Java.lang.OutOfMemoryError: PermGen space
The exception you get is caused by the creation of an array which lives on the heap. To try to get an "out of permgen memory" error, you could try to remove the list.add() lines. Note however that interned strings can be garbage collected so even doing that will still not cause the exception you expect.
Cf RFE 6962931:
In JDK 7, interned strings are no longer allocated in the permanent generation of the Java heap, but are instead allocated in the main part of the Java heap (known as the young and old generations), along with the other objects created by the application. This change will result in more data residing in the main Java heap, and less data in the permanent generation, and thus may require heap sizes to be adjusted. Most applications will see only relatively small differences in heap usage due to this change, but larger applications that load many classes or make heavy use of the String.intern() method will see more significant differences.

All String literals are interned automatically, as described in the String JavaDoc:
All literal strings and string-valued constant expressions are interned.
I would expect the behaviour to be consistent between strings you manually intern and any string literals.

s.intern() returns a reference to the interned string but you are not using it.
Try s = s.intern()

java heap memory usage exception when using FOR lOOP

When i run the below program i got the exception when for loop begins its execution at i=1031521. How to over come memory usage of this type?
class wwww
{
public static void main(String args[])
{
String abc[]=new String[4194304];
String wwf="";
int s_count=524286;
for(int i=0;i<4194304;i++)
{
System.out.println("----------enter--------"+i);
abc[i]=""+i;
System.out.println("----------exit--------"+i);
}
}
}
The exception is:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.
java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390
)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at wwww.main(wwww.java:12)

This is because your your uses up all the heap space allocated to your jvm.
You can use argument while running the program to specify the heap size that you would like to allocate.
This is an example:
java -Xmx256m MyClass
Here a maximum of 256 MB of heap space will be allocated

How to over come memory usage of this type?
Don't perform memory usage of this type. You are creating 4194304 strings, of the general form ""+i. You don't need 4194304 strings of that form all at once. You only need one of them at a time, if any, and you can create it every time you need it.

You could either:
Increase the heap size that you give to your program. This is done via the -Xmx command-line argument to java.
Re-engineer the program to use less memory. Do you really need to keep all those strings in memory at once?

Does Immutability of Strings in Java cause Out Of Memory

I have written a simple Java program that reads a million rows from the Database and writes them to a File.
The max memory that this program can use is 512M.
I frequently notice that this program runs Out Of Memory for more than 500K rows.
Since the program is a very simple program it is easy to find out that this doesn't have a memory leak. the way the program works is that it fetches a thousand rows from the Database, writes them to a file using Streams and then goes and fetches the next thousand rows. The size of each row varies but none of the rows is huge. On taking a dump while the program is running the older string are easily seen on the heap. These String in heap are unreachable which means they are waiting to get Garbage collected. I also believe that the GC doesn't necessarily run during the execution of this program which leaves String's in the heap longer than they should.
I think the solution would be to use long Char Arrays(or Stringbuffer) instead of using String objects to store the lines that are returned by the DB. The assumption is that I can overwrite the contents of a Char Array which means the same Char Array can be used across multiple iterations without having to allocate new Space each time.
Pseudocode :
Create an Array of Arrays using new char[1000][1000];
Fill the thousand rows from DB to the Array.
Write Array to File.
Use the same Array for next thousand rows
If the above pseudocode fixes my problem then in reality the Immutable nature of the String class hurts the Java programmer as there is no direct way to claim the space used up by a String even though the String is no longer in use.
Are there any better alternatives to this problem ?
P.S : I didn't do a static analysis alone. I used yourkit profiler to test a heap dump. The dump clearly says 96% of the Strings have NO GC Roots which means they are waiting to get Garbage collected. Also I don't use Substring in my code.

Immutability of the class String has absolutely nothing to do with OutOfMemoryError. Immutability means that it cannot ever change, only that.
If you run out of memory, it is simply because the garbage collector was unable to find any garbage to collect.
In practice, it is likely that you are holding references to way too many Strings in memory (for instance, do you have any kind of collection holding strings, such as List, Set, Map?). You must destroy these references to allow the garbage collector to do its job and free up some memory.

The simple answer to this question is 'no'. I suspect you're hanging onto references longer than you think.
Are you closing those streams properly ? Are you intern()ing those strings. That would result in a permanent copy being made of the string if it doesn't exist already, and taking up permgen space (which isn't collected). Are you taking substring() of a larger string ? Strings make use of the flyweight pattern and will share a character array if created using substring(). See here for more details.
You suggest that garbage collection isn't running. The option -verbose:gc will log the garbage collections and you can see immediately what's going on.

The only thing about strings which can cause an OutOfMemoryError is if you retain small sections of a much larger string. If you are doing this it should be obvious from a heap dump.
When you take a heap dump I suggest you only look at live objects, in which case any retained objects you don't need is most likely to be a bug in your code.