We Keep Coding

How do I run a JAR file on an EC2 instance?

I wrote a basic MapReduce program on my MacBook utilizing Apache's resource here:
https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html
After I finished, I Exported a jar of my project and transferred it to my EC2 instance through SSH.
After that, I ran this command through the terminal of my EC2 instance:
/usr/local/hadoop/bin/hadoop jar test.jar com.map.reduce games.tar.gz output
Where /usr/local/hadoop/bin/hadoop is where hadoop is installed on the EC2, test.jar is the transfered jar file and com.map.reduce is the package name where all of my classes are hosted. games.tar.gz is the directory I will be working with and output is where I want to see my results.
But I am getting the exception:
Exception in thread "main" java.lang.ClassNotFoundException:
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:466)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:566)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:374)
at org.apache.hadoop.util.RunJar.run(RunJar.java:311)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
I am wondering if this is an issue with the JARS I am using locally. Any help is appreciated.

Error running Cordova android platform

I'm trying to create my first cordova hybrid app. I run it with the "cordova browser" pratform and the app run fine. Now, when i try to build the app for android the console show me the next error :
$ cordova build android
ANDROID_HOME=C:\Users\MyUser\AppData\Local\Android\sdk
JAVA_HOME=C:\Program Files\java\jdk1.8.0_121
Unzipping C:\Users\MyUser\.gradle\wrapper\dists\gradle-2.13-all\7hsc6vr6mi3i6i5m7 q9hj4ci1q\gradle-2.13-all.zip to C:\Users\MyUser\.gradle\wrapper\dists\gradle-2.1 3-all\7hsc6vr6mi3i6i5m7q9hj4ci1q
Exception in thread "main" java.lang.RuntimeException: java.util.zip.ZipExceptio n: error in opening zip file
at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAcc essManager.java:78)
at org.gradle.wrapper.Install.createDist(Install.java:47)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:129)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:48)
Caused by: java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:219)
at java.util.zip.ZipFile.<init>(ZipFile.java:149)
at java.util.zip.ZipFile.<init>(ZipFile.java:163)
at org.gradle.wrapper.Install.unzip(Install.java:160)
at org.gradle.wrapper.Install.access$400(Install.java:29)
at org.gradle.wrapper.Install$1.call(Install.java:70)
at org.gradle.wrapper.Install$1.call(Install.java:47)
at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAcc essManager.java:65)
... 3 more
Error: cmd: Command failed with exit code 1 Error output:
Exception in thread "main" java.lang.RuntimeException: java.util.zip.ZipExceptio n: error in opening zip file
at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAcc essManager.java:78)
at org.gradle.wrapper.Install.createDist(Install.java:47)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:129)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:48)
Caused by: java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:219)
at java.util.zip.ZipFile.<init>(ZipFile.java:149)
at java.util.zip.ZipFile.<init>(ZipFile.java:163)
at org.gradle.wrapper.Install.unzip(Install.java:160)
at org.gradle.wrapper.Install.access$400(Install.java:29)
at org.gradle.wrapper.Install$1.call(Install.java:70)
at org.gradle.wrapper.Install$1.call(Install.java:47)
at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAcc essManager.java:65)
... 3 more
Before, i was fixin others errors but with this, i don't know what i need to do. If a run the command "cordova requirements android" the console show that I have installed everything that I need.
The output for that command is the next:
$ cordova requirements android
Error loading cordova-browser
Requirements check results for android:
Java JDK: installed 1.8.0
Android SDK: installed
Android target: installed android-14,android-16,android-18,android-22,android-23,android-24,android-25,Google Inc.:Google APIs:22,Google Inc.:Google APIs:23,Google Inc.:Google APIs:24
Gradle: installed
My project is on "React JS" and his libraries.

In your case, find this:
C:\Users\MyUser.gradle\wrapper\dists\gradle-2.13-
all\7hsc6vr6mi3i6i5m7
now when you find it,delete the gradle-2.2.1-all.zip because this file is corrupted and then we'll replace for this http://services.gradle.org/distributions/gradle-2.2.1-all.zip
when you finish all try again
cordova build android

EMR-4.2.0 Error during running of custom jar (command-runner)

I am running sqoop installation script in AWS - EMR-4.2.0 Version, followed this documentation.
After created cluster (at Steps), I have submitted my sqoop script as an arguments and s3://elasticmapreduce/libs/script-runner/script-runner.jar/ command-runner.jar as a jar file, but getting error like this. Can you help me pls what is the cause and problem?
Error:
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Cannot run program "s3://bmsgcm/spark/install-sqoop.sh" (in directory "."): error=2, No such file or directory
at com.amazonaws.emr.command.runner.ProcessRunner.exec(ProcessRunner.java:139)
at com.amazonaws.emr.command.runner.CommandRunner.main(CommandRunner.java:13)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Cannot run program "s3://bmsgcm/spark/install-sqoop.sh" (in directory "."): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
at com.amazonaws.emr.command.runner.ProcessRunner.exec(ProcessRunner.java:92)
... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:187)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
... 8 more

command-runner.jar can only read local files. You can add a bootstrap script to copy files from S3 to local file system.

Piggybox is correct. Unlike script-runner.jar that was used on the 2.x and 3.x EMR AMIs, command-runner.jar can only run local commands. A bootstrap script is the best way to do this.
For instance, if you have a few spark drivers on S3, and you have a shell script (also on S3) to copy them to the master node for later use in a job flow step with spark-submit, then you might have had a step like this:
Steps=[
{
'Name': 'Install My Spark Drivers',
'ActionOnFailure':'TERMINATE_JOB_FLOW',
'HadoopJarStep':
'Jar': 's3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar',
'Args': [
's3://my-bucket/spark-driver-install.sh',
]
}
},
...other steps...
]
Which, as you've experienced, will fail on EMR 4.x if you just swap command-runner.jar for script-runner.jar, there.
Instead, make a bootstrap action to call the script, like:
BootstrapActions=[
{
'Name': 'Install My Spark Drivers',
'ScriptBootstrapAction': {
'Path': 's3://my-bucket/spark-driver-install.sh',
'Args': []
}
}
]
The above example is expressed as boto3 run_job_flow kwargs. It's not immediately obvious to me how to accomplish the same thing in the web console, though.

For executing a script you can use script-runner. I was also facing the same issue. My script had ^M characters which was causing this issue. Removing those worked.

throwing error executing jar file after executing dos2unix command in linux

I have a jar file which is copied from a windows machine onto unix machine....usually we convert all the files that were copied from windows using dos2unix command..
When I convert the jar file to unix format using dos2unix...I am getting the below error...
Exception in thread "main" java.io.IOException: Error opening job jar: hadoop-examples-2.0.0-mr1-cdh4.3.0.jar
at org.apache.hadoop.util.RunJar.main(RunJar.java:135)
Caused by: java.util.zip.ZipException: invalid END header (bad central directory offset)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:127)
at java.util.jar.JarFile.<init>(JarFile.java:135)
at java.util.jar.JarFile.<init>(JarFile.java:72)
at org.apache.hadoop.util.RunJar.main(RunJar.java:133)
It ran successfully before running dos2unix command on it...
Any idea why this happened ?

Don't do that. A jar file is the same as a zip, it's a binary. dos2unix is for converting line endings in ASCII files (not binary).

Running wordcount.jar on hadoop in windows using command line

I am trying to run a simple wordcount program on hadoop, but facing an error as below.
Exception in thread "main" java.io.IOException: Error opening job jar: /user/asiapac/bmohanty6/wordcount/wordcount.jar
at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
Caused by: java.io.FileNotFoundException: \user\asiapac\bmohanty6\wordcount\wordcount.jar (The system cannot find the path specified)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(Unknown Source)
at java.util.zip.ZipFile.<init>(Unknown Source)
at java.util.jar.JarFile.<init>(Unknown Source)
at java.util.jar.JarFile.<init>(Unknown Source)
at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
I am using below command.
$ bin/hadoop jar /user/asiapac/bmohanty6/wordcount/wordcount.jar WordCount /user/asiapac/bmohanty6/wo
rdcount/input /user/asiapac/bmohanty6/wordcount/output
I am using Cygwin, hadoop-0.20.2 with pseudo node set up. I have also uploaded the wordcount.jar to my DFS. See below my DFS screenshot
I am able to run the same wordcount program in eclipse successfully. I have created the wordcount.jar file via eclipse as per this tutorial. I searched a lot in web but could not understand how to solve this. Please help me.

You need to add / before user:
bin/hadoop jar /user/asiapac/bmohanty6/wordcount/wordcount.jar WordCount /user/asiapac/bmohanty6/wordcount/input /user/asiapac/bmohanty6/wordcount/output
This makes them fully-qualified paths. If you omit / before user, Hadoop will search from the current directory.