I am having issues assembling Spark using the sbt on my machine.
Attempting the assembly without allocating extra memory either runs out or times out on the garbage collector; the exact issue has been different at different times I have tried. However, any attempt to modify the allocated memory, either through Xmx or Xms, and whether giving more or restricting, fails as the sbt doesn't recognize -Xmx or -Xms as a command.
Here is a sample of the kind of command I have been trying (from the source directory of Spark):
sbt -Xmx2g assembly
Here is the error I have been receiving:
java.util.concurrent.ExecutionException:java.lang.OutOfMemoryError: GC overhead limit exceeded
Use 'last' for the full log.
Not a valid command: Xmx2g
Not a valid project: Xmx2g
Expected ':' (if selecting a configuration)
Not a valid key: Xmx2g
Xmx2g
^
I am running 64-bit Java, version 1.8.0_20.
Try creating a new environment variable SBT_OPTS, with the value "-XX:MaxPermSize=1024m". That should give sbt the memory it needs without producing your error.
Check the docs: Building Spark with Maven.
Setting up Maven’s Memory Usage
You’ll need to configure Maven to use more memory than usual by setting MAVEN_OPTS. We recommend the following settings:
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
Then you can build it with:
mvn -DskipTests package
Related
I'm trying to replicate the openie4.0 program by:
https://github.com/knowitall/openie
The environment is
Ubuntu: 18.04,
Scala: 2.10.2,
Java: 1.7.0_80,
sbt: 0.13.18,
I managed to get to the step of "running with sbt:"
'''
sbt 'run-main edu.knowitall.openie.OpenIECli'
'''
However, I kept facing with the problem of
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
when I step into to create the stand-alone jar, when using:
sbt clean compile assembly
while trying
"sbt -J-Xmx2700M clean compile assembly" (or larger, say 10g, I have 64g on my computer)
The above out of memeory problem continues plus:
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
[error] Not a valid command: J-Xmx10g
[error] Not a valid project ID: J-Xmx10g
[error] Expected ':' (if selecting a configuration)
[error] Not a valid key: J-Xmx10g
[error] J-Xmx10g
[error]
This is super confusing why this command isn't a valid one and I didn't find a solution for this.
Later on I tried to set the universal JAVA_OPTS, not working.
Then I take the suggestions from user972946 (last one), which suggest its overriden since the default settings.
How to set heap size for sbt?.
Then trying after fixing as he suggested,but still the same.....
env JAVA_OPTS="-Xmx41200m" sbt clean compile assembly
I also took the suggestion from Mike and Cassio from the above link...not working at all.
Really appreciate on any suggestions on this.
Many thanks.
HZ
If you take a look at the following line:
[error] Not a valid command: J-Xmx10g
Your sbt cannot resolve this as a valid option, this can be issued since you're using java 7 (make sure java7 accepts -Xmx$size as a valid argument), the same thing goes for sbt 0.13 (which is pretty old). there also must be a file or something related to deployments (a guide or a CI/CD file or something) that you can take help from on how to run the project. (if you use java 8 everything should be fine probably)
According to Baeldung:
Try exporting this environment variable before running it:
export SBT_OPTS="-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=10G -Xmx10G -Xms10G"
Alternatively, SBT allows us to specify the JVM options on a file named .jvmopts. create a file called .jvmopts in the root of your project and put this inside:
-Xms1g
-Xmx4g
I've been able to create dashboards for small amounts of log data (3mb) with JMeter. However, when trying to create dashboards with large amounts of data (35mb), jmeter will throw a java.lang.OutOfMemoryError: Java Heap Space.
So far I've tried to create an environment variable called JVM_ARGS=-Xms1024m -Xmx10240m but I still do not have enough space.
Is there anything else I can try to create these dashboards? Or is there a way to reduce the number of entries that get written to the log file?
Thank you!
There are 2 possibilities :
Option 1 : your JVM options are not taken into account. Show the first lines or all content of jmeter.log
Option 2 : you have added some dynamic parameter to your http requests that has created a lot of different (name) SampleResult
Edit 8 october 2018:
Root cause was point Option 2
Make sure you've really created the environment variable and it has the anticipated value, double check this by running the following command in the terminal window where you will be launching JMeter from:
echo %JVM_ARGS% for Windows
echo $JVM_ARGS for Linux/Unix/MacOS
You should see your increased JVM heap settings
Make sure to use either jmeter.bat for Windows or jmeter.sh for other operating systems wrapper script
Make sure to use 64-bit version of JRE as 32-bit will not be able to allocate more than 3G heap
Make sure you can execute java command with your 10G heap
java -Xms1024m -Xmx10240m -version
you should see your Java version
Try running ApacheJMeter.jar executable directly:
java -Xms1024m -Xmx10240m -jar ApacheJMeter.jar -g result.jtl -o destination_folder
If nothing helps be aware that you can generate tables/charts using JMeterPluginsCMD Command Line Tool (it is not a part of standard JMeter installation, can be installed using JMeter Plugins Manager)
Though not a direct programming question, it is related to Maven and JVM
While I was going thru this link - http://maven.apache.org/archives/maven-1.x/reference/command-line.html
Update of links: https://wiki.openstack.org/wiki/Documentation/Troubleshooting
https://maven.apache.org/docs/3.0/release-notes.html
it says about MAVEN_OPTS
"Specify additional options using the MAVEN_OPTS environment variable. It is for passing parameters to the Java VM when running Maven. For example, to increase the amount of memory to 1024 Meg for the entire run of Maven, use:
MAVEN_OPTS=-Xmx1024m "
Questions:
How Maven is able to set the JVM properties?
Why does Maven need JVM? It just build the war and JVM is needed only when u deploy the WAR to appserver.right?
Does Maven starts the JVM and stop it after use? IF it stops it after use, what's the point in setting more heap size?
My understanding after research:
(1) How/Why Maven is able to set the JVM properties?
Maven will be able to set the JVM properties using MAVEN_OPTS.
It is to pass the parameter to JVM for that run of Maven
(2) Why does Maven need JVM? Maven is just a build tool..
Maven has to build the WAR file, right? To build the WAR file, you need to compile the code,
and sometimes run the test cases. To compile the code, you need JVM. To run test cases you need JVM.
Don't know where else we need JVM when we do a mvn clean deploy.
[eg of javac passing JVM parameters otherwise: javac -d build/classes -classpath ... -J-Xms256m -J-Xmx1024m java-source-files]
(3) Does Maven starts the JVM and stop it after use? IF it stops it after use, what's the point in setting more heap size?
Yes , Maven starts the JVM and stop after its use. JVM is needed during its run. like if maven build needs lot of memory [like if you do XML processing] then
setting the heap memory using MAVEN_OPTS will be helpful.
Please clarify on my understanding especially on the second question
mvn is a shell script/batch file, and it uses MAVEN_OPTS to do what you'd expect.
Maven is Java. Maven tasks are written in Java.
...? Maven does a lot, and some of those things use memory. It's just a Java program. When it ends the JVM that was running it also ends.
To further clarify point 1, this is what happens at the end of the script:
exec "$JAVACMD" \
$MAVEN_OPTS \
-classpath "${M2_HOME}"/boot/plexus-classworlds-*.jar \
"-Dclassworlds.conf=${M2_HOME}/bin/m2.conf" \
"-Dmaven.home=${M2_HOME}" "-Dmaven.multiModuleProjectDirectory=${MAVEN_PROJECTBASEDIR}" \
${CLASSWORLDS_LAUNCHER} "$#"
Note: why read Maven 1 docs?
Maven is a program written in Java. This program is used to compile files, copy files, make jar files, and a whole lot of other tasks. As any Java program it needs a JVM to run. And as any Java program it uses memory. And as any Java program using memory, it can cause ot of memory errors if you don't start the JVM with enough memory.
mvn is able to set JVM properties because it's a simple shell script that starts a Java program: Maven.
And yes, once mvn exits, the JVM it started to run all the tasks it needs to run exits too.
I am using the Fortify SCA plugin with Maven 3.2.1 to scan a pretty large Java webapp.
I have a custom .bat file that sets up my all my environment variables and makes a call to mvn.bat to start the scan.
Then mvn.bat reads my pom.xml and finds the custom profiles for clean, translate, and scan and then calls sourceanalyzer.
The trouble is, it never seems like sourceanalyzer uses the full amount of memory that I grant it in either the custom bat file or the pom file. This machine has 16GB of RAM, and when the scan is done 18-20 hours later it will print "memory used: 317 MB" and the report has a bunch of Out of Memory warnings. This machine is doing nothing besides this scan, and while it's running the Task Manager shows that something is using a bunch of memory.
The error message is "Scan progress is slowing due to JVM garbage collection."
My MAVEN_OPTS:
-Xmx4096m
-XX:MaxPermSize=1024m
-Dfortify.sca.64bit=true
-Dfortify.sca.Xmx=8000m
-DskipTests=true
-Dfortify.sca.verbose=true
I need to figure out how to both speed this scan up and remove the memory warnings.
Thanks
You can try using the SCA memory variable option. Set:
SCA_VM_OPTS=-Xmx8000M
You can also try posting Fortify issues to their online forum at https://protect724.hp.com. The support group monitors those forums.
I have a big problem with my jenkins server: I Cant build a maven project, because the java vm cannot start:
Checkout:workspace / /var/lib/jenkins/jobs/SchwarzGoldTool/workspace - hudson.remoting.LocalChannel#b7193fc
Using strategy: Default
Last Built Revision: Revision c2d18fd7a5d7f112163e9440a8e7256a44e32f46 (origin/HEAD, origin/master)
Checkout:workspace / /var/lib/jenkins/jobs/SchwarzGoldTool/workspace - hudson.remoting.LocalChannel#b7193fc
Fetching changes from 1 remote Git repository
Fetching upstream changes from git://.../tsc.git
Seen branch in repository origin/HEAD
Seen branch in repository origin/master
Commencing build of Revision 2b4654302e8222509db5808c9071ec95daf0b495 (origin/HEAD, origin/master)
Checking out Revision 2b4654302e8222509db5808c9071ec95daf0b495 (origin/HEAD, origin/master)
Warning : There are multiple branch changesets here
Parsing POMs
Modules changed, recalculating dependency graph
[SchwarzGoldTool] $ java -Xmx512M -Xms512M -cp /var/lib/jenkins/plugins/maven-plugin/WEB-INF/lib/maven3-agent-1.2.jar:/var/lib/jenkins/tools/Maven_3.0.3/boot/plexus-classworlds-2.4.jar org.jvnet.hudson.maven3.agent.Maven3Main /var/lib/jenkins/tools/Maven_3.0.3 /var/run/jenkins/war/WEB-INF/lib/remoting-2.11.jar /var/lib/jenkins/plugins/maven-plugin/WEB-INF/lib/maven3-interceptor-1.2.jar 58359
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
ERROR: Failed to launch Maven. Exit code = 1
Finished: FAILURE
I tried to add -Xmx and -Xms to the VM (as you can see) but that doesnt work either... someone has a idea whats going on there?
The problem is caused by Jenkins failed to reserve enough heap space to kick off a maven build. From what you said, it seems like there are few things which share your VM limit, (I included the estimation of the memory required to run each process)
The VM's OS (~200~300mb)
Jenkins (~min 256mb)
Webcontainers (~256~512mb)
etc. (~100mb)
My memory estimation is rather conservative, but still, it's easily adds up to over 1gb, which leave the available heap for Jenkins to reserve to less than the Xms (512m), hence, failed to kick off a build
Ideally you should increase the softlimit on your VM to a higher value. If that is not feasible, my advice will be to reduce the memory management of the build by change the job configuration in Jenkins to something like this, (Xmx512m, Xms128m) so that Jenkins can kick off a build with only 128m free heap. But this setting may cause out of memory error in the later stage, when the build required a heap which is below Xmx but above the available heap.
Alternatively, you can memory manage other processes I mentioned above or you can setup some configure the virtual memory on your VM.