increase jvm heap space while runnig from hadoop unix - java

I am running a java class test.java from hadoop command :
$ hadoop test
I am using a stringBuilder, and its size is going out of memory :
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder. java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572 )
at java.lang.StringBuffer.append(StringBuffer.java:320)
at java.io.StringWriter.write(StringWriter.java:60)
at org.json.JSONObject.quote(JSONObject.java:1287)
at org.json.JSONObject.writeValue(JSONObject.java:1597)
at org.json.JSONObject.write(JSONObject.java:1649)
at org.json.JSONObject.writeValue(JSONObject.java:1574)
at org.json.JSONArray.write(JSONArray.java:929)
at org.json.JSONObject.writeValue(JSONObject.java:1576)
at org.json.JSONObject.write(JSONObject.java:1649)
at org.json.JSONObject.writeValue(JSONObject.java:1574)
at org.json.JSONObject.write(JSONObject.java:1632)
at org.json.JSONObject.toString(JSONObject.java:1443)
I know in java we can run a java program by providing a heap space size :
java -Xmx4G test
How can I do this while running with hadoop, if I run it like :
$ hadoop -Xmx4G test
it throws exception :
Caused by: java.lang.ClassNotFoundException: java
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: java. Program will exit.
Error: No command named `-Xmx4g' was found. Perhaps you meant `hadoop Xmx4g'
I would like to change the heap size permanently.

Usually you do one of these in the terminal before executing the class or put it in a bash script.
export HADOOP_HEAPSIZE=4096
hadoop test
Or,
export HADOOP_OPTS="$HADOOP_OPTS -Xmx4G"
hadoop test
Or, if the above doesn't work, try modifying HADOOP_OPTS in $HADOOP_HOME/conf/hadoop-env.sh

Related

java.lang.NoClassDefFoundError when using weka.jar file in my own java file

I'm having problems executing my java program. The code is placed in the file RunMining.java and contains multiple import weka.[..] lines. The weka library (weka.jar) is placed in the same folder.
I compile it:
$ javac -cp weka.jar RunMining.java
This creates three files: RunMining.class, RunMining$1.class, RunMining$classifierType.class
When i run it: java -cp weka.jar RunMining i get the following error:
Exception in thread "main" java.lang.NoClassDefFoundError: RunMining
Caused by: java.lang.ClassNotFoundException: RunMining
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
My RunMining.java file only contains one class called RunMining, which contains the main method. I would prefer not to add the weka.jar file to my CLASSPATH since this file will be compiled using a makefile and executed using a script, on different machines.
Any suggestions, to how i can run the compiled files?
You need to add the classpath of your compiled files as follows:
java -cp weka.jar;. RunMining
This assumes your files are compiled in the root directory which your javac command suggests.

How to run Netflix Astyanax in linux

I'm a newbie in running Java code in Linux environment. I'm trying to get the Astyanax to work on the linux machine.
I followed the instruction on https://github.com/Netflix/astyanax/wiki/Getting-Started
I have cloned the code from github, and ./gradlew build runs successfully.
I'm trying to run the AstClient in the astyanax-examples. I tried the following command:
java -cp "/home/shbash6/astyanax/astyanax-examples/build/libs/;/home/shbash6/astyanax/astyanax-cassandra/build/libs/;/home/shbash6/astyanax/astyanax-core/build/libs/;/home/shbash6/astyanax/astyanax-thrift/build/libs/" com.netflix.astyanax.examples.AstClient
under holder:
/home/shbash6/astyanax/astyanax-examples/build/classes/main
but I got the following exception:
Exception in thread "main" java.lang.NoClassDefFoundError: com/netflix/astyanax/examples/AstClient
Caused by: java.lang.ClassNotFoundException: com.netflix.astyanax.examples.AstClient
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: com.netflix.astyanax.examples.AstClient. Program will exit.
Could anyone tell me how to fix this issue? Thanks!
Use : classpath separators on *nix platforms
java -cp "/home/shbash6/astyanax/astyanax-examples/build/libs/*:/home/shbash6/astyanax/astyanax-cassandra/build/libs/*:/home/shbash6/astyanax/astyanax-core/build/libs/*:/home/shbash6/astyanax/astyanax-thrift/build/libs/*" com.netflix.astyanax.examples.AstClient

SNMP4J.jar - java.lang.ClassNotFoundException

I'm trying to check some SNMP example, so
i've downloaded snmp4j.jar
compiled SNMP.java using javac -cp snmp4j.jar SNMP.java (it's OK)
tried to start it using java -cp snmp4j.jar SNMP, but
it reports:
root#comp:~/workspace_c/SNMP# java -cp snmp4j.jar SNMP
Exception in thread "main" java.lang.NoClassDefFoundError: SNMP
Caused by: java.lang.ClassNotFoundException: SNMP
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: SNMP. Program will exit.
UPDATE:
And when I import snmp4j.jar to the java project, it works normally in ECLIPSE...
UPDATE#2: main part of SNMP.java
import org.snmp4j.CommunityTarget;
...
public class SNMP {
public static void main(String[] args) {
...
}
}
what i'm doing wrong? it's first time i've met snmp4j so i know noting about it
If I understand correctly, SNMP is your example class, which isn't inside the jar. In that case, you need to add to the classpath both the jar and your current directory:
java -cp .:snmp4j.jar SNMP
As far as I can see you are trying to run your application with SNMP4J.jar (upper case) but when you compiled your application you probably used lower case version.
Your prompt shows that your are using unix OS that has case sensitive file system. So, check the case of the JAR file name. I believe that all letters are in lower case.

Java: Exception in thread "main" java.lang.NoClassDefFoundError

To compile my java app, I use from terminal:
javac -cp commons-digester-2.1/commons-digester-2.1.jar:lucene-core-3.0.3.jar CollectionIndexer.java
To run the app I use:
java -classpath commons-digester-2.1/commons-digester-2.1.jar:lucene-core-3.0.3.jar CollectionIndexer
However I get the following error:
Exception in thread "main" java.lang.NoClassDefFoundError: commons-digester-2/1/commons-digester-2/1/jar:lucene-core-3/0/3/jar
Caused by: java.lang.ClassNotFoundException: commons-digester-2.1.commons-digester-2.1.jar:lucene-core-3.0.3.jar
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
I guess I'm defining correctly the classpath.. or what ?
Try -cp instead of classpath. Also add the current directory to the classpath.
$ java -cp .:commons-digester-2.1/commons-digester-2.1.jar:lucene-core-3.0.3.jar CollectionIndexer
or
$ export CLASSPATH=.:commons-digester-2.1/commons-digester-2.1.jar:lucene-core-3.0.3.jar
$ java CollectionIndexer
If on Windows, use backslashes and a semi-colon separator:
java -cp .;commons-digester-2.1\commons-digester-2.1.jar;lucene-core-3.0.3.jar CollectionIndexer
I would suggest you to set the classpath first with
SET CLASSPATH command and then execute your program as
java CollectionIndexer.

Calling Java .jar Program from Perl using System()

The title pretty much says it all, I am trying to get a java program (.jar file) to run from a Perl program. I read another post on Stackoverflow saying this syntax is correct:
system("java filename.jar");
However this is giving me the following error. I'm not sure if the problem is that it shows the filename as "filename/jar" instead of "filename.jar"
Exception in thread "main" java.lang.NoClassDefFoundError: filename/jar
Caused by: java.lang.ClassNotFoundException: filename.jar
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
You need to call it as java -jar filename.jar.
If you leave out the -jar, then java thinks you're trying to run a class named filename.jar, which it tries to load from the filename/jar file. Since you don't have that file, java throws the error that you see.
You have to use the -jar option to launch a jar.
java [ options ] -jar file.jar [ argument ... ]
Resources :
sun.com - java

Categories