Suppose a thread in my program reading a file from disk, and it encountered an Error(outOfMemory) and thread got killed without a chance to execute closing of streams given in finally. Will it keep that stream open even after that thread kills?
The finally block will still be executed. However, if the JVM is out of memory, there's a chance that there will be a problem closing the stream, resulting in another out of memory error thrown from within the finally block. If that happens, the stream will likely not be closed until the JVM exits.
In most case it should be closed. But it mostly depends on memory left when hiting close method and reference retention you have on the stream.
OOM are raised when trying to use more Heap memory than JVM is allowed to. But it doesn't mean you have no memory available at all. After OOM is raised, a lot of memory can be available due to many reasons : the process just try to allocate a BIG array that don't fit into memory, many intermediate allocated objects may have been discarded due to raised exception, GC may have run deeper collection than usual incremental ones, Stack memory can be used to process stream closing, etc.
Then, most streams are closed when garbage collected. Generally, you open and close a stream in the scope of method, then when exited there's no more reference over it. Thus the reference become eligible to garbage collection and may close automatically (however you have to wait for GC to collect it).
Most software good practice are based on "best effort". Don't try to think/do too much. Make the "best effort" to clean up and let it crash.
What are you suppose to do about a non-closed stream while your entire JVM is going away ?
In your case ("stream handling"), "best effort" is done trough usage of try-with-resources statement.
If you are worry about overhead of non-closed streams, you just have to use try-with-resources statement ("best effort" application) and MUST focus on reference retention which the real cause of "memory leak" in Java (as most Stream are closed when garbage collected).
The real problem about "non-closed streams" is related to limitation OS apply about number of "file descriptor/handler" that a process can have at a given time.
Thread aren't supposed to be "killed" and if so, you may quickly run into troubles as monitor aren't freed (which will cause more damage through your VM).
Related
ResultSet fetches records from database.
After using the resultset object we finally close the resultset.
Question is , once rs.close() is called, will it free the delete the fetched records from memory?
or
when JVM is facing shortage of space, garabage collector will be called to delete the resultSet?
If JVM is invoking GC when it faces shortage of memory, is it a good practice to call the Garbage collector manually in the java program to free up the space?
Result Sets are often implemented by using a database cursor. Calling resultSet.close() will release that cursor, so it will immediately free resources in the database.
The data read by a Result Set is often received in blocks of records. Calling resultSet.close() might "release" the last block, making it eligible for GC, but that would happen anyway once the resultSet itself goes out of scope and becomes eligible for GC, and that likely happens right after calling close(), so it really doesn't matter if calling close() releases Java memory early.
Java memory is only freed by a GC run. You don't control when that happens (calling System.gc() is only a hint, you don't have control).
You're considering the wrong things. What you should focus on is:
Making sure resources1 are always closed as soon a possible to free up database and system resources.
This is best done using try-with-resources.
Making sure you don't keep too much data, e.g. don't create objects for every row retrieved if you can process the data as you get it.
This is usually were memory leaks occur, not inside the JDBC driver.
1) E.g. ResultSet, Statement, Connection, InputStream, OutputStream, Reader, Writer, etc.
ResultSet.close() will immediately release all resources, except Blob, Clob and NClob objects. Release means resources will be freed when Garbage Collector decides so. Usually we don't have to worry about it.
However, some memory used by JDBC may remain used.
Suppose that the driver has some sort of cache built in, and that cache is connection-scoped. To release that memory, you'd have to close JDBC Connection.
E.g. MySQL JDBC has default fetch size of 0, meaning it loads entire table into memory and keeps it in the memory for all of your statements. What's the scope of this in-memory buffer? ;)
Anyway, if you suspect memory issues, have a look at your JDBC driver specifics.
Rule of thumb, explicit GC is never good idea. But for a quick look to determine if ResultSet.close()/Connection.close() release any resources, give it a try: inspect used/free memory, close(), gc(), inspect memory again. Without explicit GC you will hardly see any changes.
Explicit GC is a burden on JVM as it has to frequently check the memory usage and decide when to trigger it. Where as, setting the appropriate GC as per application requirement would be sufficient to handle the above scenarios.
ResultSet.close will mark the resources for garbage collection i.e. freed up the reference to mark the memory blocks as non-reachable. Also, for a jdbc, connection needs to be closed so that memory holding the connection cache can also be marked for gc.
This question already has answers here:
Error java.lang.OutOfMemoryError: GC overhead limit exceeded
(22 answers)
Closed 6 months ago.
After a lot of effort I can't seem to overcome the problem of getting a
GC overhead limit exceeded
error in my Java program.
It occurs inside a large method that contains large string manipulation, many lists of objects and accesses to DB.
I have tried the following:
after the use of each ArrayList, I have added: list=new ArrayList<>(); list=null;
for the strings, instead of having e.g. 50 appends (str+="....") I try to have one append with the total text
after each DB access I close the statements and the resultSets.
This method is called from main like this:
for(int i=0; i<L; i++) {
cns = new Console(i);
cns.processData();//this is the method
cns=null;
}
When this loop gets executed 1 or 2 times, everything is ok. For L>=3 it's almost certain that I will get the garbage collector error.
Shouldn't the fact that I have a cns=null after each execution of the method, force the GC and free everything from the previous execution?
Should I also delete all private attributes of the object before setting it to null? Maybe putting a Thread.sleep() could force the GC after each loop?
There's actually no reason to set cns to null at the end of each loop. You're setting it to a new Console() at the beginning of the loop anyway - if anything could be freed by setting it to null, it's also going to be freed by setting it to a new object.
You can try a System.gc(); call to suggest the system do a garbage collection, but I don't know if that would help you or make it worse. The system IS already attempting garbage collection - if it wasn't, you wouldn't get this error.
You don't show us exactly how you're building your Strings, but keep in mind that += isn't the only culprit. If you have something like String s = "Hello " + "to the" + " world";, that's just as bad as putting that on three lines and using +=. If it's an issue, StringBuilder may be your friend.
You can read the answers at Error java.lang.OutOfMemoryError: GC overhead limit exceeded for some other suggestions on how to avoid this error. It seems that for some people it's triggered when you're almost, but not quite, out of memory. So increasing the amount of memory available to Java may (or may not) help.
Basically, an "GC overhead limit exceeded" is a symptom of having too much reachable data. The heap is filling up with things that cannot be garbage collected ... 'cos they are not garbage! The JVM is running the GC again and again in an attempt to make space. Eventually, it decides that too much time is being spent garbage collecting, and it gives up. This is usually the correct thing to do.
The following ideas from your question (and the other answer) are NOT solutions.
Forcing the GC to run by calling System.gc() won't help. The GC is already running too often.
Assigning null to cns won't help. It immediately gets something else assigned to it. Besides, there is no evidence that the Console object is occupying much memory.
(Note that the constructor for the java.io.Console class is not public, so your example is not meaningful as written. Perhaps you are actually calling System.getConsole()? Or perhaps this is a different Console class?)
Clearing private attributes of an object before setting it to null is unlikely to make any difference. If an object is not reachable, then the values of its attributes are irrelevant. The GC won't even look at them.
Calling Thread.sleep() will make no difference. The GC runs when it thinks it needs to.
The real problem is ... something that we can't determine from the evidence that you have provided. Why is there so much reachable data?
In general terms, the two most likely explanations are as follows:
Your application (or some library you are) is accumulating more and more objects in some data structure that is surviving beyond a single iteration of the for loop. In short, you have a memory leak.
To solve this, you need to find the storage leak; see How to find a Java Memory Leak. (The leak could be related to database connections, statements or resultsets not being closed, but I doubt it. The GC should find and close these resources if they have become unreachable.)
Your application simply needs more memory. For example, if a single call to processData needs more memory than is available, you will get an OOME no matter what you try to get the GC to do. It cannot delete reachable objects, and it obviously cannot find enough garbage to fast enough.
To solve this, first see if there are ways modify the program so that it needs less (reachable) memory, Here are a couple of ideas:
If you are building a huge string to represent the output before writing it to an OutputStream, Writer or similar. You would save memory if you wrote directly to the output sink.
In some cases, consider using StringBuilder rather than String concatenation when assembling large strings. Particularly when the concatenations are looped.
However, note that 1) in Java 8 and earlier javac already emits StringBuilder sequences for you for concatenation expressions, and 2) in Java 9+, javac emits invokedynamic code that is better than using StringBuilder; see
JDK 9/JEP 280: String Concatenations Will Never Be the Same
If that doesn't help, increase the JVM's heap size or split the problem into smaller problems. Some problems just need lots of memory to solve.
This question already has answers here:
Catching java.lang.OutOfMemoryError?
(14 answers)
Closed 8 years ago.
How good is a try catch at catching out of memory exceptions? I don't have any experience writing software that at the low level manages its own memory, but I could imagine an approach to doing that.
I don't know the way Java actually handles memory exceptions. Is it possible that the program managing memory runs out of memory? When could I try to catch an out of memory exception and it would fail to catch the exception?
Thanks!
You do not have to worry about any implicit allocations that happen as part of catching the throwable, per se. It is always possible to catch them. The JVM even keeps pre-allocated instances of OOM errors available for when trouble happens, just so that they themselves never fail to allocate.
However, there may well be secondary concerns:
Any allocation could be the proverbial straw that breaks the camel's back, so you likely won't know just where your code is going to throw an OOM error. It could even happen in a completely different thread from the one you're doing your memory-consuming work in, thus crashing completely different parts of the JVM.
Depending on just what you're going to do when you catch it, you may be allocating more memory (such as a LogRecord or StringBuilder, the latter of which could even happen implicitly as part of syntactical string concatenation), which could run out of memory again.
These concerns only apply if you're running out of memory "the normal way", however; that is, by allocating lots of "normal" objects. In contrast, if the operation running out of memory is, for instance, the single allocation of, say, a 10 GB array, then they do not pose a problem.
How good is a try catch at catching out of memory exceptions?
It works just fine at one level ... but at another level it can be risky, and / or futile. See below.
Is it possible that the program managing memory runs out of memory?
The "program" managing memory is the garbage collector. AFAIK, it always has enough memory for its own purposes. Though if it didn't it would have no alternative but to hard crash the JVM.
When could I try to catch an out of memory exception and it would fail to catch the exception?
Only if the JVM crashes. Or, if the OOME is thrown on a different thread stack to the one where you are trying to catch it.
OK, so why is catching OOME risky and / or futile.
The first reason is that an OOME can be thrown on any thread stack without any warning. When you catch the exception, the handler (typically) has no way to know what was happening "up stack", and exactly how it failed. Hence, it has no way of knowing if the application's execution state has been damaged or compromised. (Was the thread in the middle of updating something important? Was it about to notify some other thread ... and the thread will now never get the notification?)
The second reason is that OOMEs are often a result of a Java storage leak ... caused by something "hang onto" objects when it shouldn't do. If you catch an OOME and attempt to resume ... when the problem was caused by a leak ... the chances are that the offending objects will still be reachable, and another OOME will follow pretty soon. In other words, your application could get stuck in a state where it is continually throwing and recovering from OOMEs. At best this is going to kill performance ... because the last thing that the JVM typically does before an OOME is to perform a full (stop the world) garbage collection. And that takes a significant time.
Note that is not to say that you should never catch OOMEs. Indeed, catching an OOME, reporting it and then shutting down is often a good strategy.
No, the thing that is risky / futile is to catch the OOME and then attempt to recover and continue running.
In java Out Of Memory is not an exception it is an error. You can do anything at program level but it is a system limitation. You can escape it by increasing default heap memory size.
export JVM_ARGS="-Xmx1024m -XX:MaxPermSize=256m"
An OutOfMemory is an error not an exception. Errors are specifically designed to diferenciate things that can be catch, and hopefully recover, by your program to low level errors that are non recoverable.
You only catch some thing if you think that something can be done to recover from this exception/error, you can do nothing to recover from a OutOfMemory.
First of all,out of memory is Error in java not an Exception.We can handle only exceptions in java by using try-catch construct or throws clause.
Both Error and Exception classes extends Throwable class.But Error is irrecoverable condition and Exception is hand able.
For your situation ,go through Garbage Collection in java .
I have developed a J2ME web browser application, it is working fine. I am testing its memory consumption. It seems to me that it has a memory leak, because the green curve that represents the consumed memory of the memory monitor (of the wireless toolkit) reaches the maximum allocated memory (which is 687768 bytes) every 7 requests done by the browser, (i.e. when the end user navigates in the web browser from one page to other for 7 pages) after that the garbage collector runs and frees the allocated memory.
My question is:
is it a memory leak when the garbage collector runs automatically every 7 page navigation?
Do I need to run the garbage collector (System.gc()) manually one time per request to prevent the maximum allocated memory to be reached?
Please guide me, thanks
To determine if it is a memory leak, you would need to observe it more.
From your description, i.e. that once the maximum memory is reached, the GC kicks in and is able to free memory for your application to run, it does not sound like there is a leak.
Also you should not call GC yourself since
it is only an indication
could potentially affect the underlying algorithm affecting its performance.
You should instead focus on why your application needs so much memory in such a short period.
My question is: is it a memory leak when the garbage collector runs automatically every 7 page navigation?
Not necessarily. It could also be that:
your heap is too small for the size of problem you are trying to solve, or
your application is generating (collectable) garbage at a high rate.
In fact, given the numbers you have presented, I'm inclined to think that this is primarily a heap size issue. If the interval between GC runs decreased over time, then THAT would be evidence that pointed to a memory leak, but if the rate stays steady on average, then it would suggest that the rate of memory usage and reclamation are in balance; i.e. no leak.
Do I need to run the garbage collector (System.gc()) manually one time per request to prevent the maximum allocated memory to be reached?
No. No. No.
Calling System.gc() won't cure a memory leak. If it is a real memory leak, then calling System.gc() will not reclaim the leaked memory. In fact, all you will do is make your application RUN A LOT SLOWER ... assuming that the JVM doesn't ignore the call entirely.
Direct and indirect evidence that the default behaviour of HotSpot JVMs is to honour System.gc() calls:
"For example, the default setting for the DisableExplicitGC option causes JVM to honor Explicit garbage collection requests." - http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/topic/com.ibm.websphere.express.doc/info/exp/ae/rprf_hotspot_parms.html
"When JMX is enabled in this way, some JVMs (such as Sun's) that do distributed garbage collection will periodically invoke System.gc, causing a Full GC." - http://static.springsource.com/projects/tc-server/2.0/getting-started/html/ch11s07.html
"It is best to disable explicit GC by using the flag -XX:+DisableExplicitGC." - http://docs.oracle.com/cd/E19396-01/819-0084/pt_tuningjava.html
And from the Java 7 source code:
./openjdk/hotspot/src/share/vm/runtime/globals.hpp
product(bool, DisableExplicitGC, false, \
"Tells whether calling System.gc() does a full GC") \
where the false is the default value for the option. (And note that this is in the OS / M/C independent part of the code tree.)
I wrote a library that makes a good effort to force the GC. As mentioned before, System.gc() is asynchronous and won't do anything by itself. You may want to use this library to profile your application and find the spots where too much garbage is being produced. You can read more about it in this article where I describe the GC problem in detail.
That is (semi) normal behavior. Available (unreferenced) storage is not collected until the size of the heap reaches some threshold, triggering a collection cycle.
You can reduce the frequency of GC cycles by being a bit more "heap aware". Eg, a common error in many programs is to parse a string by using substring to not only parse off the left-most word, but also shorten the remaining string by substringing to the right. Creating a new String for the word is not easily avoided, but one can easily avoid repeatedly substringing the "tail" of the original string.
Running System.GC will accomplish nothing -- on most platforms it's a no-op, since it's so commonly abused.
Note that (outside of brain-dead Android) you can't have a true "memory leak" in Java (unless there's a serious JVM bug). What's commonly referred to as a "leak" in Java is the failure to remove all references to objects that will never be used again. Eg, you might keep putting data into a chain and never clear pointers to the stuff on the far end of the chain that is no longer going to be used. The resulting symptom is that the MINIMUM heap used (ie, the size immediately after GC runs) keeps rising each cycle.
Adding to the other excellent answers:
Looks like you are confusing memory leak with garbage collection.
Memory leak is when unused memory cannot be garbage collected because it still has references somewhere (although they're not used for anything).
Garbage collection is when a piece of software (the garbage collector) frees unreferenced memory automatically.
You should not call the garbage collector manually because that would affect its performance.
I am reading "Effective Java".
In the discussion about finalize, he says
C++ destructors are also used to reclaim other nonmemory resources.
In Java, the try finally block is generally used for this purpose.
What are nonmemory resources?
Is a database connection a nonmemory resource? Doesn't the object for holding the database connection occupy some memory?
Database connections, network connections, file handles, mutexes, etc. Something which needs to be released (not just garbage-collected) when you're finished with it.
Yes, these objects typically occupy some memory, but the critical point is that they also have (possibly exclusive) access to some resource in addition to memory.
Is a database connection a non memory resource?
Yes, that's one of the most common examples. Others are file handles, native GUI objects (e.g. Swing or AWT windows) and sockets.
Doesn't the Object for holding the database connection occupy some memory?
Yes, but the point is that the non-memory part of the resource needs to be released as well and is typically much scarcer than the comparatively small amount of memory the object uses. Typically, such objects have a finalize() method that releases the non-memory resource, but the problem is that this finalizers will only run when the objects are garbage collected.
Since the objects are small, there may be plenty of available heap memory so that the garbage collector runs rarely. And in between runs of the garbage collector, the non-memory resources are not released and you may run out of them.
This may even cause problems with only a single object: for example, if you want to move a file between filesystems by opening it, opening the target file, copying the data and then deleting the original file, the delete will fail if the file is still opened - and it is almost certain to be if you only set the reference to the input stream to null and don't call close() explicitly, because it's very unlikely that the garbage collector would have run at exactly the right point between the object becoming eligible for garbage collection and the call to delete()
Another important peace on
Java Automatic Memory Management which touches on some of the essentials.
The question is better answered the other way around, in my view- 'why don't I need to release memory manually'.
This raises the question, 'why do I need to release any resources?'
Fundamentally, your running program uses many forms of resources to execute and do work (CPU cycles, memory locations, disk access, etc.). Almost all of these suffer from 'scarcity', that is, there is a fixed pool of any such resource available, if all resource is allocated then the OS can't satisfy requests and generally your program can't continue and dies very ungracefully- possibly making the whole system unstable. The only one that comes to mind that isn't scarce is CPU cycles, you can issue as many of these as you like, you're only limited by the rate at which you can issue them, they aren't consumed in the same sense that memory or file handles are.
So, any resource you use (memory, file handles, database connexions, network sockets, etc.) comes from a fixed amount of such resource (avoiding the word 'pool') and as your program (and, bear-in-mind other programs, not to mention the OS itself) allocates these resources, the amount available decreases.
If a program requests and is allocated resources and never releases them to be used elsewhere, eventually (often soon) the system will run out of such resources. At which point, either the system halts, or sometimes the offending program can be killed abruptly.
Pre-90s, resource management (at least in mainstream development) was a problem that every programmer had to deal with explicitly. Some resource allocation management isn't too hard, mostly because the allocation is already abstracted (e.g. file handles or network sockets) and one can obtain the resource, use it and explicitly release it when it's no longer wanted.
However, managing memory is very hard, particularly as memory allocation cannot (in non-trivial situations) be calculated at design-time, whereas, say, database connexions can feasibly be managed this way. (There's no way of knowing how much memory you will use, it's very difficult/ impossible to know when an allocation of memory is no longer in use). Also, memory allocations tend to hang-around for a while, where most other resource allocations are limited to a narrow scope, often within a single try-block, or method, at most usually a class. Therefore, vendors developed methods of abstracting memory allocation and bringing it under a single management system, handled by the executing environment, not the program.
This is the difference between managed environments (e.g. Java, .NET) and unmanaged ones (e.g. C, C++ run directly through the OS). In C/C++ memory allocation is done explicitly (with malloc()/new and associated reallocation), which leads to all sorts of problems- how much do I need? How do I calculate when I need more/less? How do I release memory? How do I make sure I'm not using memory that's already been released? How do I detect and manage situations where a memory allocation request fails? How do I avoid writing over memory (perhaps not even my own memory)? All this is extremely difficult and leads to memory leaks, core dumps and all sorts of semi-random, unreproducible errors.
So, Java implements automatic memory-management. The programmer simply allocates new a object and is neither interested, nor should be in terms of what or where memory is allocated (this is also why there isn't much in the way of pointers in managed environments):
object thing = new Object();
and that's all that needs to be done. The JVM will keep track of what memory is available, when it needs allocating, when it can be released (as it's no longer in use), providing ways of dealing with out of memory situations as gracefully as possible (and limiting any problems to the executing thread/ JVM and not bringing down the entire OS).
Automatic memory management is the standard with most programming now, as memory management is by far the most difficult resource to manage (mainly as others are abstracted away to some extent already, database connection pools, socket abstractions etc).
So, to answer the question, yes, you need to manage all resources, but in Java you don't need to (and can't) explicitly manage memory yourself (though it's worth considering in some situations, e.g. designing a cache). This leaves all other resources that you do need to explicitly manage (and these are the non-memory resources, i.e. everything except object instantiation/destruction).
All these other resources are wrapped in a memory resource, clearly, but that's not the issue here. For instance, there are a finite number of database connexion you are allowed to open, a finite number of file handles you may create. You need to manage the allocation of these. The use of the finally block allows you to ensure resources are deallocated, even when an exception occurs.
e.g.
public void server()
{
try
{
ServerSocket serverSocket = new ServerSocket(25);
}
catch (Exception exception)
{
// Something went wrong.
}
finally
{
// Clear up and deallocate the unmanaged resource serverSocket here.
// The close method will internally ensure that the network socket is actually flushed, closed and network resources released.
serverSocket.close();
// The memory used by serverSocket will be automatically released by the JVM runtime at this point, as the serverSocket has gone out-of-scope- it is never used again, so can safely be deallocated.
}
}