In Spark, I'm getting java.lang.OutOfMemoryError: Java heap space error when reading a String of around 1 GB from the HDFS from within a function. The executor memory I use is 6 GB though. To increase the user memory, I even decreased spark.memory.fraction to just 0.3, but I am still getting the same error. It seems as though decreasing that value had no effect. I am using Spark 1.6.1 and compiling with Spark 1.6 core library. Am I doing something wrong here?
Please see SparkConf
Spark Executor OOM: How to set Memory Parameters on Spark
Once a app is running the next most likely error you will see is an OOM on a spark executor. Spark is an extremely powerful tool for doing in-memory computation but it’s power comes with some sharp edges. The most common cause for an executor OOM’ing is that the application is trying to cache or load too much information into memory. Depending on your use case there are several solutions to this:
Increase the storage fraction variable, spark.storage.memoryFraction. This can be set as above on either the command line or in the SparkConf object. This variable sets exactly how much of the JVM will be dedicated to the caching and storage of RDD’s. You can set it as a value between 0 and 1, describing what portion of executor JVM memory will be dedicated for caching RDDs. If you have a job that will require very little shuffle memory but will utilize a lot of cached RDD’s increase this variable (example: Caching an RDD then performing aggregates on it.)
If all else fails you may just need additional ram on each worker.
Then increase the amount of ram the application requests by setting spark.executor.memory variable either on the command line or in the SparkConf object.
In your case somehow seems like memory fraction setting was not applied. as advised in comment you can print all settings applied like this to cross check.
logger.info(sparkContext.getConf.getAll.mkString("\n")
if its not applied, you can set this grammatically and try to see the effect.
val conf = new SparkConf()
.set("spark.memory.fraction", "1")
.set("spark.testing.memory", maxOnHeapExecutionMemory.toString)
…
as described in the test
UPDATE :
Please go through this nice post to understand more in detail
Gist of above the post is :
You can see 3 main memory regions on the diagram:
1) Reserved Memory : Memory reserved by the system, and its size is
hard coded
2) User Memory (in Spark 1.6 “Java Heap” – “Reserved Memory”) * (1.0
– spark.memory.fraction)
This is the memory pool that remains after the allocation of Spark
Memory, and it is completely up to you to use it in a way you like.
User Memory and its completely up to you what would be stored in this
RAM and how, Spark makes completely no accounting on what you do there
and whether you respect this boundary or not. Not respecting this
boundary in your code might cause OOM error.
3) Spark Memory (“Java Heap” – “Reserved Memory”) *
spark.memory.fraction, --> Memory pool managed by Spark. Further
divided in to
|--> Storage Memory
|--> Execution Memory
Related
My KAFKA Stream java application goes to ERROR status due to an out of memory problem.
I use windowed aggregation, mainly in order to calculate median values:
a 1 second windows
.windowedBy(TimeWindows.of(Duration.ofSeconds(1)).advanceBy(Duration.ofMillis(999)).grace(Duration.ofMillis(1))) with .suppress(Suppressed.untilWindowCloses(Suppressed.BufferConfig.unbounded().withMaxBytes(10000).withLoggingDisabled()))
30 seconds windows without suppress .windowedBy(TimeWindows.of(Duration.ofSeconds(30)).advanceBy(Duration.ofSeconds(2)).grace(Duration.ofMillis(1)))
I have also a steate store:`
StoreBuilder<KeyValueStore<String, Gateway>> kvStoreBuilder = Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(AppConfigs.GW_STATE_STORE_NAME),
Serdes.String(),
JsonSerdes.getGatewaySerde()
);
// add state store to StreamBuilder
builder.addStateStore(kvStoreBuilder);`
Eclipse memory analyzer says that:
One instance of ‘org.apache.kafka.streams.state.internals.InMemoryTimeOrderedKeyValueBuffer’ loaded by ‘jdk.internal.loader.ClassLoaders$AppClassLoader # 0xf00d8558’ occupies 238,753,712 (90.51%) bytes. The memory is accumulated in one instance of ‘java.util.HashMap$Node[]’, loaded by ‘’, which occupies 238,749,768 (90.51%) bytes.
Can anyone explain which should be the root cause ?
The error is from suppress() that use the in-memory store (InMemoryTimeOrderedKeyValueBuffer). suppress() does not support RocksDB atm (cf https://issues.apache.org/jira/browse/KAFKA-7224).
Your suppress() config seems to be incorrect:
Suppressed.BufferConfig.unbounded().withMaxBytes(10000).withLoggingDisabled()
The configs unbounded() and withMaxBytes() contradict each other: do you want an unbounded or bounded buffer? -- In your case, the second withMaxBytes() overwrites the first one. Thus, you only provide 10,000 bytes for the suppress buffer. Because you use untilWindowCloses(), Kafka Streams will need to shut down if it runs out of memory, because it's neither allows to early emit (untilWindowClose()) not allowed to use more memory (withMaxBytes(...)).
For untilWindowClose() you should use unbounded(). If you want to bound memory, you should not use untilWindowClose().
You need to tune the rocks DB configuration, please read this https://medium.com/#grinfeld_433/kafka-streams-and-rocksdb-in-the-space-time-continuum-and-a-little-bit-of-configuration-40edb5ee9ed7
If you are using java>=8, set metaspace otherwise it will eat all your server RAM. http://karunsubramanian.com/websphere/one-important-change-in-memory-management-in-java-8/
If using dockers, limit the max memory configs.
There is a bug in old kafka and they recommend to update version.
https://issues.apache.org/jira/browse/KAFKA-8637
I was able to call ObjectHeap.iterateObjectsOfKlass (with the help of SA) to obtain all objects belonging to a certain class. The result is exactly what I have expected, but the performance is not.
It took me >800 seconds to get my result, during which the target VM is suspended. The target VM heap is about 2GB. I know iterateObjectsOfKlass will call iterateExact.
My question is: do these methods iterate/traverse the entire heap just to obtain objects for 1 class? I am disappointed since my expectation is that with a single class, the result should return within 10 seconds.
HotSpot Serviceability Agent is really powerful technology, but indeed very slow. I have explained how it works in this answer.
JVM has no means to quickly find all instances of the specific class. So, yes, it has to scan the entire heap. Moreover, in order to read memory of a foreign process, SA uses ptrace system call for every single word of data. That's why it is so slow.
You have several options to scan heap faster:
Create a coredump of a foreign process and then run SA tool against the coredump. This is much faster than to read memory of a suspended process. See the related question.
Inject a JVMTI agent into a running process using Dynamic Attach mechanism. The agent can scan heap of a local JVM using IterateOverInstancesOfClass function. This will be dramatically faster comparing to SA, because it will be just reading from within the same process without any syscalls or whatever. I believe it will take just a few seconds for a 2GB heap.
I'm working with datastax 3.1 on a single node with 4Go of RAM.
I have not change anything in cassandra-en.sh and cassandra.yaml except the "--Xss" (because of my java version which require a little more)
So by default Cassandra set to 1Go my -Xms and -Xmx parameters: -Xms1024M -Xmx1024M
But while inserting my data after around 200 000 rows (in 3 different column_families), Solr and cassandra logs keep repeat this kind of warning:
WARN StorageService Flushing CFS(Keyspace='OpsCenter',
ColumnFamily='rollups60') to relieve memory pressure 17:58:07
WARN GCInspector Heap is 0.8825103486201678 full. You may need to reduce
memtable and/or cache sizes. Cassandra will now flush up to the two
largest memtables to free up memory. Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this
automatically
So, OK my heap is full, but why after flushing, is my heap still full ?
If I stop inserting data at this point. Warning keep repeating.
If I stop and restart cassandra. No problem raise
It looks like memory leak issue right?
So where should I look at?
Thanks for futur help.
One thing that's a memory hog is Solr's caches. Take a look at your solrconfig.xml file inside the "conf" dir of each of your Solr cores, and look at the value configured for caches such as:
<filterCache class="solr.FastLRUCache"
size="100"
initialSize="0"
autowarmCount="0"/>
There may be multiple entries like this one. Make sure that, at least the autowarmCount and initialSize are set to 0. Further more, lower the "size" value to something small, like 100 or something. All these values refer to number of entries in the cache.
Another thing that may help is configuring Solr to do hard-commits more often. Look for an entry such as:
<!-- stuff ommited for brevity -->
<autoCommit>
<maxDocs>5000</maxDocs>
<maxTime>15000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
The above settings will commit to disk each time 5000 documents have been added or 15 seconds have passed since the last commit, which ever comes first. Also set openSearcher to false.
Finally, look for these entries and set them as follows:
<ramBufferSizeMB>16</ramBufferSizeMB>
<maxBufferedDocs>5000</maxBufferedDocs>
Now, making all this modifications on Solr at once will surely make it run a lot slower. Try instead to make them incrementally, until you get rid of the memory error. Also, it may simply be that you need to allocate more memory to your Java process. If you say the machine has 4 Gb of RAM, why not try your test with -Xmx2g or -Xmx3g ?
Cassandra is trying to clear up heap space, however flushing memtables doesn't flush Solr heap data structures.
For the index size you have, combined with possibly queries that load the Lucene field caches there is not enough heap space allocated. The best advice is to allocate more heap space.
To view the field cache memory usage:
http://www.datastax.com/docs/datastax_enterprise3.1/solutions/dse_search_core_status
I'm writing a Java/Swing application with ~30 class my probleme is when i run my programe it load more than 150 M of the memory, is that normal ? since the application have 4 threads, parse some XML files, load some icon file, and drow some Jfreechat charts.
if not how can i do to minimize the amount of memory used by the application, is affecting some variables to null help? is loading the XML files once to use them in all the application life cycle help or i have to load them evry time i need them? is there some other tips that help me?
PS: im devlopping with a 8G memory computer in case that can affect the memory used by my program.
EDIT: it appeared that the program don't occupy all the 150MB because i get this value from the top command on linux, by running this code in my application as vilmantas advises me:
long free = Runtime.getRuntime().freeMemory();
long total = Runtime.getRuntime().totalMemory();
long max = Runtime.getRuntime().maxMemory();
long used = total - free;
I found that he occupy much less than that (~40MB) so i decide to run it with "-Xmx40M" argument and i reduce more than 40% of memory usage in the Top command.
The problem who are occupying the rest of memory since JVM (as i know) have his own process ? and how to make this operation automatic**?** because when choosing a not appropriate value you can get a memory exception as i have by running with "-Xmx30M" argument:
Exception in thread "Thread-2" java.lang.OutOfMemoryError: Java heap space
It is. This is Java, usually your VM/GC will do the job for you. Worry about memory usage when and if it becomes a problem.
If you want, there are several tools that can help you analyze what is going on. How to monitor Java memory usage?
Setting variables to null can help preventing memory leaks, if the referring variable's life cycle is greater than the referred instance. So that variables that should hold-on through the whole application life cycle are better not hold references to temporary objects that are used for a short time.
Loading the XMLs only once can help if you're good with loading its information only once. Meaning, if the XML is changed otherwise than through your application and you need to get the update - you'll have to reload the XML (and if the deprecated XML info is no longer needed - get rid of it).
You could use java memory heap analyzer like http://www.eclipse.org/mat/ to identify the parts of your application that use up most of the memory. You can then either optimize your data structures, or decide release parts of the data by setting all references to it to null.
Unintended references to data that is not needed anymore are also refered as "memory leaks". Settings those references to null will cause the garbage collector to remove it from java memory heap.
Along that line, you might find WeakReferences helpful.
Where do you observe those 150M? Is that how much your JVM process occupies (e.g. visible in the top command on linux/unix) or is it really the memory used (and necessary) by your application?
Try writing the following 4 values when your application runs:
long free = Runtime.getRuntime().freeMemory();
long total = Runtime.getRuntime().totalMemory();
long max = Runtime.getRuntime().maxMemory();
long used = total - free;
If the value for "used" is much lower than 150M, you may add java start parameter e.g. "-Xmx30M" to limit the heap size of your application to 30MB. Note that the JVM process will still occupy a little bit more than 30MB in such case.
The memory usage by JVM is somewhat tricky.
I use the following lines to get the memory usage with Java :
Free_Memory=Run_Time.freeMemory()/1048576; // 1024 x 1024 = 1K x 1K = 1Meg
Total_Memory=Run_Time.totalMemory()/1048576; // 992 Total on a 4 GB PC
The Free_Memory I got was : 900, but it is way off, when Free_Memory goes down to around 600, my program ran out of memory and generated heap overflow message.
So I looked at the => Windows Task Manager : Performance : Physical Memory : Free, it's down to 1, 2 or 0, which is a more accurate reflection of my memory situation, and according to it, my Total Memory is : 4089, which is correct, while Java's Total_Memory=992 is incorrect.
So, my question now is : In my Java program how to get the memory usage numbers reflected in the Windows Task Manager : Performance : Physical Memory ? I need to depend on those numbers.
The JVM doesn't allow Java to consume all available system memory. Instead the JVM grabs a fixed chunk and allocates all of your objects within that chunk. If this area fills up, you're out of memory! There are commandline options to alternate the max/initial memory usage of the JVM.
The more important issue is that you should not be relying on tracking free/max memory. What are you doing that relies on tracking memory?
UPDATE:
Try 64bit if you need more memory than 1.5GB
If you're trying to track memory running out then consider figuring our WHY your program does this and if it can be prevented through different algorithms or better management of objects. When the memory reaches zero what do you expect to do? Popup a dialog and tell the user they're screwed and exit the program? I can understand a grateful shutdown but warning the user to run with a large -Xmx is not going to cut it.
If you want detailed Windows stats, you can use WMI and a .vbs script, executed via cscript.exe.
This link details a script that pulls more detailed memory stats than you could possibly want.
Execute this via the usual Process/Runtime combination, and simply read back what figures you require. These are system level stats, and not for the VM (although WMI can pull back per-process stats as well).