I'm tying to load a graph of several hundred million nodes using the neo4j-admin import tool to load the data from csv. The import will run for about two hours but then crashes with the following error:
Exception in thread "Thread-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.String.substring(String.java:1969)
at java.util.Formatter.parse(Formatter.java:2557)
at java.util.Formatter.format(Formatter.java:2501)
at java.util.Formatter.format(Formatter.java:2455)
at java.lang.String.format(String.java:2940)
at org.neo4j.unsafe.impl.batchimport.input.BadCollector$RelationshipsProblemReporter.getReportMessage(BadCollector.java:209)
at org.neo4j.unsafe.impl.batchimport.input.BadCollector$RelationshipsProblemReporter.message(BadCollector.java:195)
at org.neo4j.unsafe.impl.batchimport.input.BadCollector.processEvent(BadCollector.java:93)
at org.neo4j.unsafe.impl.batchimport.input.BadCollector$$Lambda$110/603650290.accept(Unknown Source)
at org.neo4j.concurrent.AsyncEvents.process(AsyncEvents.java:137)
at org.neo4j.concurrent.AsyncEvents.run(AsyncEvents.java:111)
at java.lang.Thread.run(Thread.java:748)
I've been trying to adjust my max and initial heap size settings in a few different ways. First I tried simply creating a HEAP_SIZE= variable before running the command to load the data as described here and I tried setting the heap size on the JVM like this:
export JAVA_OPTS=%JAVA_OPTS% -Xms100g -Xmx100g
but whatever I setting I use when the import starts I get the same report:
Available resources:
Total machine memory: 1.48 TB
Free machine memory: 95.00 GB
Max heap memory : 26.67 GB
Processors: 48
Configured max memory: 1.30 TB
High-IO: true
As you can see, I'm building this on a large server that should have plenty of resources available. I'm assuming I'm not setting the JVM parameters correctly for Neo4j but I can't find anything online showing me the correct way to do this.
What might be causing my GC memory error and how can I resolve it? Is this something I can resolve by throwing more resources at the JVM and if so, how do I do that so the neo4j-admin import tool can use it?
RHEL 7 Neo4j CE 3.4.11 Java 1.8.0_131
The issue was resolved by increasing the maximum heap memory. The problem was I wasn't setting the heap memory allocation correctly.
It turns out there was a simple solution; it was just a matter of when I tried to set the heap memory. Initially, I had tried the command export JAVA_OPTS='-server -Xms300g -Xmx300g' at the command line then run my bash script to call neo4j-admin import. This was not working, neo4j-admin import continued to use the same heap space configuration regardless.
The solution was to simple include the command to set the heap memory in the shell script that called neo4j-admin import. My shell script ended up looking like this:
#!/bin/bash
export JAVA_OPTS='-server -Xms300g -Xmx300g'
/usr/local/neo4j-community-3.4.11/bin/neo4j-admin import \
--ignore-missing-nodes=true \
--database=mag_cs2.graphdb \
--multiline-fields=true \
--high-io=true \
This seems super obvious but it took me almost a week to realize what I needed to change. Hopefully, this saves someone else the same headache.
Related
I am running a Nutch 1.16, Hadoop 2.83, Solr 8.5.1 crawler setup that is running fine up to a few million indexed pages. Then I am running into Java Heap Space issues during the MapReduce job and I just cannot seem to find the correct way to up that heap space. I have tried:
Passing -D mapreduce.map.memory.mb=24608 -D mapreduce.map.java.opts=-Xmx24096m when starting nutch crawl.
Editing NUTCH_HOME/bin/crawl commonOptions mapred.child.java.opts to -Xmx16000m
Setting HADOOP_HOME/etc/hadoop/mapred-site.xml mapred.child.java.opts to -Xmx160000m -XX:+UseConcMarkSweepGC
Copying said mapred-site.xml into my nutch/conf folder
None of that seems to change anything. I run into the same Heap Space error at the same point in the crawling process. I have tried reducing the fetcher threads back to 12 from 25 and switching off parsing while fetching. Nothing changed and I am out of ideas. I have 64GB RAM so thats really not an issue. Please help ;)
EDIT: fixed filename to mapred-site.xml
Passing -D ...
The heap space needs to be set also for the reduce task using "mapreduce.reduce.memory.mb" and "mapreduce.reduce.java.opts". Note that the script bin/crawl was recently improved in this regard, see NUTCH-2501 and the recent bin/crawl script.
3./4. Setting/copying hadoop-site.xml
Shouldn't this be set in "mapred-site.xml"?
I think that my Java Program exceeded the usage of memory allowed... this is the error that shows up in Eclipse:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at Main.main(Main.java:228)
before than trying an implementation with better memory handling, is there any way basically to augment the memory to allocate for the program ? I would like to see the program terminating and showing the result at least.
You can allocate more memory to the process by using the -Xmx flag:
java -Xmx2G YourProgram
Will allocate 2 Gigabytes of memory to the process.
You can do this in Eclipse by going to Run Configurations, the Arguments section and adding -Xmx 2G to the VM arguments.
Increasing memory size is good, but you should also consider that probably your program has some memory leak and you really should bother about it.
You can increase heap size using the cmd-line flags
For example:
java -Xmx6g myprogram
You can get a full list (or a nearly full list, anyway) by typing java -X.
As I don't think your program is using that much memory, I suspect your program is having a memory leak somewhere.
If you could give Main.java:200 till Main.java:250 we could check for any leaks.
If you are sure your program is using that amount of memory you can either run
java -Xmx2G -jar foo.jar in the Command Prompt (Windows) or Terminal (Mac OS X or Linux)
or, if you're running your program in Eclipse under Linux, do the following:
1) Make sure Eclipse is closed
2) Open your favorite text editor at your eclipse.ini file (Default location: /usr/lib/eclipse/eclipse.ini)
3) Search for --Xmx512M and increase this number (i.e. --Xmx2G)
4) Save the file and restart Eclipse
For any explaination about --Xmx and --Xms I refer to Eclipse memory settings when getting "Java Heap Space" and "Out of Memory"
Happy coding :) -Charlie
My specs:
-Ubuntu 64bit
-Neo4j 2.0
-32 GB of Ram
-AMD FX-8350 Eight COre Processor
The problem:
I'm making a request to my Neo4j server with the following query:
MATCH (being:my_label_2) RETURN being
And gives me this error:
OutOfMemoryError
Requested array size exceeds VM limit
StackTrace:
java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
java.lang.StringCoding.encode(StringCoding.java:344)
java.lang.String.getBytes(String.java:916)
org.neo4j.server.rest.repr.OutputFormat.toBytes(OutputFormat.java:194)
org.neo4j.server.rest.repr.OutputFormat.formatRepresentation(OutputFormat.java:147)
org.neo4j.server.rest.repr.OutputFormat.response(OutputFormat.java:130)
org.neo4j.server.rest.repr.OutputFormat.ok(OutputFormat.java:67)
org.neo4j.server.rest.web.CypherService.cypher(CypherService.java:101)
java.lang.reflect.Method.invoke(Method.java:606)
org.neo4j.server.rest.transactional.TransactionalRequestDispatcher.dispatch(TransactionalRequestDispatcher.java:139)
org.neo4j.server.rest.security.SecurityFilter.doFilter(SecurityFilter.java:112)
This works fine with "my_label_1" which returns around 30k results
What I believe is the problem:
I don't have enough memory allocated to my JVM
Attempts made to fix/things I've found online:
I read what the manual says to do
And what the Ubuntu Forums say to do
So I've tried going to my neo4 folder (with cd as usual) and running it with the arguments this way:
sudo bin/neo4j start -Xmx4096M
However that didn't work. When Neo4j starts it does warn me that I might not have enough space with:
WARNING: Max 1024 open files allowed, minimum of 40 000 recommended. See the Neo4j manual.
Using additional JVM arguments: -server -XX:+DisableExplicitGC -Dorg.neo4j.server.properties=conf/neo4j-server.properties -Djava.util.logging.config.file=conf/logging.properties -Dlog4j.configuration=file:conf/log4j.properties -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
Question
I know I'm definitely using the arguments wrong, I honestly don't have much experience with JVM configurations. How should I approach this, am I missing something?
You should put JVM setting into the conf/neo4j-wrapper.conf file. It should look like this:
user#pc:> head -n 7 neo4j-enterprise-2.0.0/conf/neo4j-wrapper.conf
wrapper.java.additional=-Dorg.neo4j.server.properties=conf/neo4j-server.properties
wrapper.java.additional=-Djava.util.logging.config.file=conf/logging.properties
wrapper.java.additional=-Dlog4j.configuration=file:conf/log4j.properties
# Java Additional Parameters
wrapper.java.additional=-XX:+UseConcMarkSweepGC
wrapper.java.additional=-XX:+CMSClassUnloadingEnabled
Note that you can configure different aspects of neo4j via different files, so it's better to read description to every file in that conf/ directory in order to get familiar with what can be done and how exactly.
I am trying to run the vFileServe of project anarchy, but i always have this error even though i set a parameter on my java
Here is my error
Loading deploy parameters from Source\Vision\Samples\Engine\FileServe\FileServe_
android_arm_vs2010_anarchy.vcxproj_Dev.deploy
Creating apkFile ../../../../../Bin/android_arm_vs2010_anarchy/Dev/libFileServe.
apk for library ../../../../../Bin/android_arm_vs2010_anarchy/Dev/libFileServe.s
o
Android platform (SDK) android-10
Removing dir AndroidTemp\armeabi-v7a
Making a raw (so uncompressed) dir for the gdb server, native .so etc...
Done
Generating classes.dex file
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Command finished with error code '1'
Here is my java parameter
I tried putting -XX:MaxHeapSize=256m -Xmx512m also but doesnt work either
Whats wrong here? Thanks
I don't know about this particular software but limiting JVM heap size to 512Mb max may be insufficient.
Try setting initial heap size to 512 Mb and maximum to 2 Gb with -Xms512m -Xmx2g
Fo some reason the environment variable of java has been removed. I have to set it again. That solve the problem
I'm running a WEKA classifier (J48 with an input .arff file composed of 3 fields, field 1 has ~27k distinct attributes, field 2 ~ 500k values) in a latest generation Macbook Pro with 8GB RAM.
I increased the java heap space to the maximum possible using the -Xmx parameter:
java -Xmx7G -cp weka-3-6-10/weka.jar weka.classifiers.trees.J48 -t
myfiles/loc_linear.arff -i
however when I run the classifier (after about 10 minutes) I get the error "Exception in thread "main" java.lang.OutOfMemoryError: Java heap space".
Evidently 8GB RAM is not enough with my input file. Does this mean the only solution to this is having a more powerful hardware (e.g. 16GB RAM or a very powerful server/cluster)?
Would there be any workaround to this issue? (e.g. reducing the input file? If so, which would be in your opinion the criteria to apply in the reduction?). Any other ideas or suggestions?
If you are running the Weka GUI on a Mac OS X machine, you can edit a plist configuration file. I followed instructions from the Weka mailing list.
cd into /Applications/weka-XXX.app/Contents , or wherever your weka executable was installed.
There will be a file called Info.plist there. I suggest you save a copy of that file to another location, as you'll need to change it in the next step.
Open the weka-XXX.app/Contents/Info.plist (XML) file in your favorite text editor and look for a block that says "VMOptions". There should be a value that says "-Xmx256M" which specifies the memory. Change that value to something bigger, like "-Xmx1024M".
Start Weka.
From your cited line of code it seems you are running Weka from the simple command line interface. If that is the case, then the answer is the same as this [question] (Increase heap to avoid Out of Memory Error in WEKA.)
You can't increase the heap size from the command line interface. Instead I believe you should increase the heap size in the RunWeka.ini file as stated in Weka's instructions