I have written simple code in java as follows
public static void main(String[] args) {
String st ="Java";
StringBuffer sb = new StringBuffer(st);
long startTime = System.currentTimeMillis();
Runtime r = Runtime.getRuntime();
long startm = r.freeMemory();
System.out.println("Before running program free memory is: "+startm);
for(int i=0; i<10000; i++)
st+="is";
long endm = r.freeMemory();
System.out.println("After program free memory is: "+endm);
}
However the problem is when I run the program, free memory is increasing after loop is finished but it should be less than in beginning of the program. My output is as follows:
Before running program free memory is: 61089768
After program free memory is: 123747064
Please tell me why is that?
Your code is producing a lot of object garbage. The line
st+="is";
translates to:
st = new StringBuffer().append(st).append("is").toString();
This means each loop produces at least two objects of garbage.
Before entering the loop, there is already garbage on the object heap from the Java startup. Your loop triggers the garbage collector, freeing not only the unneeded objects from your program, but also all other objects left behind by the Java startup.
Start your program with the parameters -XX:+PrintGCDetails and you will see when the GC operates.
Related
Here is the function:
String gen() {
String raw_content = "";
String line;
for (int i=0; i<10000; i++) {
line = randString();
raw_content += line + "\n";
}
return raw_content;
}
When I call gen() for 100 times in main(), my program will stuck.
I suspect this is related to memory leak caused by Java String. So will the no-longer used memory be freed by JVM automatically? How to fix this?
Thanks!
To make a long story short, in java (and other JVM languages), you don't have to care about memory allocation at all. You really shouldn't be worrying about it - at some time after all references to it have been lost, it'll be freed up by the garbage collecting thread. See: Garbage Collection in Java.
Your problem has less to do with memory and more that your function is just really time intensive (as Hot Licks said in comment). Strings in Java are immutable, so when you say raw_content += line + "\n"; you're really creating a new string of raw_content + line + "\n" and setting raw_content equal to that. If rand_string() returns long results, this will become an egregiously long string. If you really want to perform this function, StringBuilders are the way to go to at least reduce it from O(N^2) to O(N). If you're just looking for a memory exercise, you don't have to actually do any changes - just read the above article.
So I understand that strings in Java are immutable. I'm interested in how to repeatedly update a certain string.
Ex:
public static void main(String[] args) {
String myString = "hey";
for (int i = 1; i <= 9; i++) {
myString += "hey";
}
}
Now, this won't work in Java because I've already declared and assigned myString. How do people get around Java's immutable strings (as in the above example)?
The only thing I can think to do is declare another string. Unfortunately, this just delays my problem, as the second time through the loop, I'll be reassigning an already assigned string:
public static void main(String[] args) {
String myString = "hey";
String secondString;
for (int i = 1; i <= 10; i++) {
secondString += "hey";
}
}
Any suggestions / explanations are much appreciated!
Thanks,
Mariogs
A Quick Answer
You should use a StringBuilder for this sort of thing. It is designed to put Strings together without constantly copying or holding on to older Strings.
public class SimpleGrowingString {
private StringBuilder stringBuilder;
public SimpleGrowingString() {
this.stringBuilder = new Stringbuilder();
}
public void addToString(String str) {
this.stringBuilder.append(str);
}
public String getString() {
return this.stringBuilder.toString();
}
}
A Not So Quick Answer:
Immutable?
While Strings are immutable, you can re-assign a String variable.
The variable will then reference (point to) the new String assigned to it and the old value will be marked for Garbage Collection and only hang about in RAM until the Garbage Collector gets off its arse and around to clearing it out. That is, as long as there are no other references to it (or subsections of it) still about somewhere.
Immutable means that you cannot change a String itself not that you cannot reassign what the variable that was pointing to its value now is.
eg.
String str = "string one";
The String "string one" exists in memory and can not be changed, modified, cut up, added to etc.
It is immutable.
If I then say:
str = "a different string";
Then the variable str now references a different piece of data in memory; the String "a different string".
The original String "string one" is still the exact same String that it was before we've just told the handle we had for it to point to something else. The old String is still floating around in memory but now it's headless and we no longer have any way to actually access that value.
This leads to the idea of Garbage Collection.
Garbage. Garbage Everywhere.
The Garbage Collector runs every now and again and cleans out old, unnecessary data that's no longer being used.
It decides what is and isn't useful, among other ways, by checking if there are any valid handles/variables currently pointing at the data. If there's nothing using it and there's no way for us to even access it anymore it's useless to us and it gets dumped.
But you can't really ever rely on the Garbage Collector to clean out thing on time, quickly or even get it to run when you want it to. It does its own thing, in its own time. You are better off trying to minimise its workload than assuming it's going to clean up after you all the time.
And now that you have, an admittedly very basic, grounding in Garbage Collection we can talk about why you don't add Strings together:
String con+cat+en+a+tion
One big issue with using + for Strings (and the reason that StringBuilder and StringBuffer were designed) is that it creates an awful lot of Strings. Strings all over the place! They may end up as candidates for Garbage Collection relatively quickly depending on your usage, but they still can lead to bloat if handled incorrectly, especially when loops get involved, as the Garbage Collector runs whenever it damn well feels like and we can't really control that we can't say that things are not getting out of hand unless we ourselves stop them getting that way.
Doing the simple String concatenation of:
"a String" + " another String"
actually leads to having three Strings in memory:
"a String", " another String" and "a String another String"
Expand this out to a simple, fairly short loop and you can see how things can get out of hand pretty quickly:
String str = "";
for (int i=0; i<=6; i++) {
str += "a chunk of RAM ";
}
Which at each loops means we have in memory:
0:
"a chunk of RAM "
1:
"a chunk of RAM "
"a chunk of RAM a chunk of RAM"
2:
"a chunk of RAM "
"a chunk of RAM a chunk of RAM"
"a chunk of RAM a chunk of RAM a chunk of RAM"
3:
"a chunk of RAM "
"a chunk of RAM a chunk of RAM "
"a chunk of RAM a chunk of RAM a chunk of RAM "
"a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM "
4:
"a chunk of RAM "
"a chunk of RAM a chunk of RAM "
"a chunk of RAM a chunk of RAM a chunk of RAM "
"a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM "
"a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM"
5:
"a chunk of RAM "
"a chunk of RAM a chunk of RAM "
"a chunk of RAM a chunk of RAM a chunk of RAM "
"a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM "
"a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM"
"a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM "
6:
"a chunk of RAM "
"a chunk of RAM a chunk of RAM "
"a chunk of RAM a chunk of RAM a chunk of RAM "
"a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM "
"a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM"
"a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM "
"a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM a chunk of RAM "
And so on and so on... You can see where that's going and how quickly it's getting there.
Moral of the story
If you are looping with and combining Strings use a StringBuilder or StringBuffer it's what they were made for.
"Sidenote".substring(4);
Concatenating Strings isn't the only way to end up with a lot of wasted RAM because of String's immutability.
Just as we can't add to the end of a String neither can we chop off the tail.
String moderatelyLongString = "a somewhat lengthy String that rambles on and on about nothing in particular for quite a while even though nobody's listening";
If perhaps we wanted to make use of only a part of this String we could use the String.substring() method:
String snippedString = moderatelyLongString.substring(0, 13);
System.out.println(snippedString):
>> a somewhat le
Ok, so what's wrong with that?
Well, if we wanted to dump the first, long String but hang onto the short bit you might think that we can just say:
moderatelyLongString = null;
You may think that will leave the long String abandoned and alone, crying in a corner waiting for the GC to come and kick it out into the cold but you'd be wrong.
Since we've still got a hold of a couple of characters of the longer chain in snippedString the entire moderatelyLongString stays on the heap wasting space.
If you wanted to do this but shed the useless weight what you would want to do is copy the shortened part but not retain a tie to the lengthy bit:
String aNicerMorePoliteShortString = new String(moderatelyLongString.substring(0, 13));
This makes a copy of the short String taken from the long that is its own stand alone array of characters and has nothing to do with that pestering hanger-on that is the long String.
Now doing this will, this time, mark the long String as available for Collection as we have no remaining ties to it:
moderatelyLongString = null;
However
If you just wanted to display a single, different String in a loop on every iteration what you want to do is just (re)use a single variable so that all of the older Strings in memory get released as soon as possible and become available for Garbage Collection as quickly as they can be. Declare your variable outside of the loop and then reassign it inside on every iteration:
String whatYouWantToUse;
for (int i=0; i<100; i++) {
whatYouWantToUse = someStringyGettyMethod();
howYouWantToUseIt(whatYouWantToUse);
}
Each time this loop loops it is assigning a new value to the variable which throws the older value onto the pile of waste for the Garbage Collector to clean up in time, or, you know, whenever it could be bothered to...
Arguably, a better way to do the above method is to never try to hold onto the String at all — just pass it straight though from where we get it from to where it's wanted:
for (int i=0; i<100; i++) {
howYouWantToUseIt(someStringyGettyMethod());
}
But watch out for over optimising this sort of thing as readability is almost always more important than compactness.
Most compilers are smarter than we'll ever be, or than I will be at least. They can find all the great shortcuts and minifications that can be done to your code and apply their wizardry in a more magnificent way than we mortals could hope to achieve.
If you try to streamline your code too much then all you're left with is two varieties of unreadable code instead of one useful, fast and optimised version and the other maintainable and something Johnny with the off-putting habit of sniffling every 25 seconds two desks over can follow.
this won't work in Java because I've already declared and assigned myString
You are wrong, it will still work but each time you append to the string it will generate a new string.
If you dont want to generate new string when you append/add to it then StringBuilder is the solution.
sample:
public static void main(String args[]) {
StringBuilder sb = new StringBuilder("hey");
for (int i = 1; i <= 9; i++) {
sb.append("hey");
}
}
Being immutable doesnt mean that it wont work, it just means that the object you created wont be modified.. but the String variable can be assigned to another object (the new string created by concatenating the previous strings on += "hey").
If you want to do it like a mutable object, then just use StringBuilder append() method.
While Strings in Java are immutable, your first example above will work because it creates a new String object every time through the loop, and assigns the newly created string to myString:
public static void main(String[] args) {
String myString = "hey";
for (int i = 1; i <= 9; i++) {
myString += "hey";
}
System.out.println(myString); // prints "heyheyheyheyheyheyheyheyheyhey"
}
While this works, it's inefficient due to the object creation. For a loop with more iterations, or when concatenating longer strings, performance might be a concern – so there are better ways to do it.
One better approach to concatenating Strings in Java is to use StringBuilder. Here's your code adapted to use StringBuilder:
public static void main(String[] args) {
StringBuilder builder = new StringBuilder(50); // estimated length
for (int i = 1; i <= 9; i++) {
builder.append("hey");
}
String myString = builder.toString(); // convert to String when needed
System.out.println(myString); // prints "heyheyheyhey..."
}
StringBuilder has a backing array as a buffer, which is expanded whenever the appended length exceeds the size of the buffer. In this case, we start with an initial allocation of 50 characters.
In a real world situation, you could set the initial size of the StringBuilder buffer based on the size of the input, to minimise the need for expensive buffer expansions.
When you execute the working code in your question, you will simply create a new string in memory each time you append to it.
This means that every time you append something to your string it will be a new string object in memory, implying that it also has a new memory address.
This is because strings are immutable indeed.
If you only want to create a string object once, you should use a StringBuilder, otherwise this solution works fine.
StringBuilders are recommended for building string that you will - as you do - modify a lot. Because modifying a string a lot (i.e., creating many new strings) does a lot of reading and writing in your memory.
Your code works perfectly fine. Although its not recommended to work on strings like you do.
Have a look at Java's StringBuilder: http://docs.oracle.com/javase/7/docs/api/java/lang/StringBuffer.html
With the aid of a StringBuilder, you can modify the string.
I have a big txt file with integers in it. Each line in file has two integer numbers separated by whitespace. Size of a file is 63 Mb.
Pattern p = Pattern.compile("\\s");
try (BufferedReader reader = new BufferedReader(new FileReader(filePath))) {
String line;
while ((line = reader.readLine()) != null) {
String[] tokens = p.split(line);
String s1 = new String(tokens[0]);
String s2 = new String(tokens[1]);
int startLabel = Integer.valueOf(s1) - 1;
int endLabel = Integer.valueOf(s2) - 1;
Vertex fromV = vertices.get(startLabel);
Vertex toV = vertices.get(endLabel);
Edge edge = new Edge(fromV, toV);
fromV.addEdge(edge);
toV.addEdge(edge);
edges.add(edge);
System.out.println("Edge from " + fromV.getLabel() + " to " + toV.getLabel());
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:2694)
at java.lang.String.<init>(String.java:203)
at java.lang.String.substring(String.java:1913)
at java.lang.String.subSequence(String.java:1946)
at java.util.regex.Pattern.split(Pattern.java:1202)
at java.util.regex.Pattern.split(Pattern.java:1259)
at SCC.main(SCC.java:25)
Why am I getting this exception? How can I change my code to avoid it?
EDIT:
I've already increase heap size to 2048m.
What is consuming it? That's what I would want to know also.
For all I know jvm should allocate memory to list of vertices, set of edges, buffer for buffered reader and one small string "line". I don't see where this outOfMemory coming from.
I read about string.split() method. I think it's causing memory leak, but I don't know what should I do about it.
What you should try first is reduce the file to small enough that it works. That will allow you to appraise just how large a problem you have.
Second, your problem is definitely unrelated to String#split since you are using it on just one line at a time. What is consuming your heap are the Vertex and Edge instances. You'll have to redesign this towards a smaller footprint, or completely overhaul your algorithms to be able to work with only a part of the graph in memory, the rest on the disk.
P.S. Just a general Java note: don't write
String s1 = new String(tokens[0]);
String s2 = new String(tokens[1]);
you just need
String s1 = tokens[0];
String s2 = tokens[1];
or even just use tokens[0] directly instead of s1, since it's about as clear.
Easiest way: increase your heap size:
Add -Xmx512m -Xms512m (or even more) arguments to jvm
Increase the heap memory limit, using the -Xmx JVM option.
More info here.
You are getting this exception because your program is storing too much data in the java heap.
Although your exception is showing up in the Pattern.split() method, the actual culprit could be any large memory user in your code, such as the graph you are building. Looking at what you provided, I suspect the graph data structure is storing much redundant data. You may want to research a more space-efficient graph structure.
If you are using the Sun JVM, try the JVM option -XX:+HeapDumpOnOutOfMemoryError to create a heap dump and analyze that for any heavy memory users, and use that analysis to optimize your code. See Using HeapDumpOnOutOfMemoryError parameter for heap dump for JBoss for more info.
If that's too much work for you, as others have indicated, try increasing the JVM heap space to a point where your program no longer crashes.
When ever you get an OOM while trying to parse stuff, its just that the method you are using is not scalable. Even though increasing the heap might solve the issue temporarily, it is not scalable. Example, if tomorrow your file size increases by an order or magnitude, you would be back in square one.
I would recommend trying to read the file in pieces, cache x lines of the file, read off it, clear the cache and re-do the process.
You can use either ehcache or guava cache.
The way you parse the string could be changed.
try (Scanner scanner = new Scanner(new FileReader(filePath))) {
while (scanner.hasNextInt()) {
int startLabel = scanner.nextInt();
int endLabel = scanner.nextInt();
scanner.nextLine(); // discard the rest of the line.
// use start and end.
}
I suspect the memory consumption is actually in the data structure you build rather than how you read the data, but this should make it more obvious.
I knowingly created the following class to cause out of memory error
public class Test1
{
public static void main(String[] args)
{
StringBuffer sb = new StringBuffer();
while(true)
{
Test1 a = new Test1();
sb.append(a.toString());
}
}
}
As I expected this above class fails with what I wanted...
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuffer.append(Unknown Source)
at Test1.main(Test1.java:10)
but this:
public class Test1
{
public static void main(String[] args)
{
StringBuffer sb = new StringBuffer();
while(true)
{
Test1 a = new Test1();
System.out.println(sb.toString());
sb.append(a.toString());
}
}
}
Does not crash. Runs just fine, by printing the object address over and over again on console.
My question is:
What difference a simple SOP made?
Your assumption that there is no OutOfMemoryError is likely to be incorrect. It is just massively delayed. Printing a string that is getting bigger and bigger on the out stream takes so much time, that your loop may take an hour to run out of memory.
You can double-check this, by printing only every 10th, 100th, 1000th time. You'll see the error will occur the earlier the less IO you generate. Probably you'll see a curve like this in jconsole:
As you can see, the heap is slowly but steadily going up. Even if I try to force garbage collection (15:02 and 15:07), I cannot free all memory anymore. But since I'm still only at 5% of my heap, I'll stop running your code now :-)
Just tested a bit, the actual bottleneck is sb.toString(). This of course takes time proportional to the length of the string(buffer), so every next loop takes a tiny bit longer to execute. Before you run out of memory, after a few thousands loops, one loop will take a couple of seconds just to create the string.
Replacing sb.toString() by a long counter, makes it crash 'instantly' aswell. Removing the System.out.println() has little effect on speed.
On my computer java -Xmx2m Test1 > /dev/null takes about 8 minutes to crash. With a normal heap size this could take days. (Feel free to try it.)
Sorry I can't post code but I have a bufferedreader with 50000000 bytes set as the buffer size. It works as you would expect for half an hour, the HDD light flashing every two minutes or so, reading in the big chunk of data, and then going quiet again as the CPU processes it. But after about half an hour (this is a very big file), the HDD starts thrashing as if it is reading one byte at a time. It is still in the same loop and I think I checked free ram to rule out swapping (heap size is default).
Probably won't get any helpful answers, but worth a try.
OK I have changed heap size to 768mb and still nothing. There is plenty of free memory and java.exe is only using about 300mb.
Now I have profiled it and heap stays at about 200MB, well below what is available. CPU stays at 50%. Yet the HDD starts thrashing like crazy. I have.. no idea. I am going to rewrite the whole thing in c#, that is my solution.
Here is the code (it is just a throw-away script, not pretty):
BufferedReader s = null;
HashMap<String, Integer> allWords = new HashMap<String, Integer>();
HashSet<String> pageWords = new HashSet<String>();
long[] pageCount = new long[78592];
long pages = 0;
Scanner wordFile = new Scanner(new BufferedReader(new FileReader("allWords.txt")));
while (wordFile.hasNext()) {
allWords.put(wordFile.next(), Integer.parseInt(wordFile.next()));
}
s = new BufferedReader(new FileReader("wikipedia/enwiki-latest-pages-articles.xml"), 50000000);
StringBuilder words = new StringBuilder();
String nextLine = null;
while ((nextLine = s.readLine()) != null) {
if (a.matcher(nextLine).matches()) {
continue;
}
else if (b.matcher(nextLine).matches()) {
continue;
}
else if (c.matcher(nextLine).matches()) {
continue;
}
else if (d.matcher(nextLine).matches()) {
nextLine = s.readLine();
if (e.matcher(nextLine).matches()) {
if (f.matcher(s.readLine()).matches()) {
pageWords.addAll(Arrays.asList(words.toString().toLowerCase().split("[^a-zA-Z]")));
words.setLength(0);
pages++;
for (String word : pageWords) {
if (allWords.containsKey(word)) {
pageCount[allWords.get(word)]++;
}
else if (!word.isEmpty() && allWords.containsKey(word.substring(0, word.length() - 1))) {
pageCount[allWords.get(word.substring(0, word.length() - 1))]++;
}
}
pageWords.clear();
}
}
}
else if (g.matcher(nextLine).matches()) {
continue;
}
words.append(nextLine);
words.append(" ");
}
Have you tried removing the buffer size and trying it out with the defaults?
It may be not that the file buffering isn't working, but that your program is using up enough memory that your virtual memory system is page swapping to disk. What happens if you try with a smaller buffer size? What about larger?
I'd bet that you are running out of heap space and you are getting stuck doing back to back GC's. Have you profiled the app to see what is going on during that time? Also, try running with -verbose:gc to see garbage collection as it happens. You could also try starting with a larger heap like"
-Xms1000m -Xmx1000m
That will give you 1gb of heap so if you do use that all up, it should be much later than it is currently happening.
It appears to me that if the file you are reading is very large, then the following lines could result in a large portion of the file being copied to memory via a StringBuilder. If the process' memory footprint becomes too large, you will likely swap and/or throw your garbage collector into a spin.
...
words.append(nextLine);
words.append(" ");
Hopefully this may help: http://www.velocityreviews.com/forums/t131734-bufferedreader-and-buffer-size.html
Before you assume there is something wrong with Java and reading IO, I suggest you write a simple program which just reads the file as fast as it can. You should be able to read the file at 20 MB/s or more regardless of file size with default buffering. You should be able to do this by stripping down your application to just read the file. Then you can prove to yourself how long it takes to read the file.
You have used quite a lot of expensive operations. Perhaps you should look at how you can make your parser more efficient using a profiler. e.g.
word.substring(0, word.length() - 1)
is the same as
word
so the first if clause and the second are the same.