We Keep Coding

How to run my map-reduce program with AWS?

I`m trying to run a map-reduce program using Hadoop framework.
I need to run the program on an AmazonElasticMapReduce instance but I keep getting the following error:
Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.transform.JsonErrorUnmarshaller: method (Ljava/lang/Class;)V not found
I tried to fix my pom file and adding the AWS SDK, changing its versions and adding the core separately.
my pom.xml file:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.3</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-ec2</artifactId>
<version>1.10.2</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-emr</artifactId>
<version>1.9.0</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.11.5</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<version>1.11.5</version>
</dependency>
me trying to create the instance:
AWSCredentials creds = new PropertiesCredentials(new FileInputStream(propertiesFilePath));
AmazonElasticMapReduce mapReduce = new AmazonElasticMapReduceClient(creds);
I expect the program to run on AWS instance (and probably get a bunch more error in my code that I can debug).

NoClassDefFoundError: org/apache/spark/sql/DataFrame in spark-cassandra-connector

I'm trying to upgrade spark-cassandra-connector from 1.4 to 1.5.
Everything seems fine but when I run test cases then It stuck between the process and log some error message saying:
Exception in thread "dag-scheduler-event-loop"
java.lang.NoClassDefFoundError: org/apache/spark/sql/DataFrame
My pom file looks like:
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector_2.10 -->
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>16.0.1</version>
</dependency>
<!-- Scala Library -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.10.5</version>
</dependency>
<!--Spark Cassandra Connector-->
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector-java_2.10</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>3.0.2</version>
</dependency>
<!--Spark-->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.5.0</version>
<exclusions>
<exclusion>
<groupId>net.java.dev.jets3t</groupId>
<artifactId>jets3t</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</project>
Thank you in advance!!
Can anyone please help me with this ?
If you need more info please let me know!!

Try to add dependency
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
Also make sure that your version spark-cassandra-connector is compatible with version of Spark you're using. I had the same error message even with all proper dependencies when was trying to use older spark-cassandra-connector with newer Spark version. Refer to this table: https://github.com/datastax/spark-cassandra-connector#version-compatibility

Spring boot with websphere configuration classes not working

I am getting below error when running the app with springboot tomcat. The below error message is repeating continuously. Please help.
2015-09-22 03:37:09.477 ERROR 20112 --- [ main]
c.i.w.naming.java.javaURLContextFactory : NMSV0307E: A java: URL name
was used, b ut Naming was not configured to handle java: URL names.
The likely cause is a user in error attempting to specify a java: URL
name in a non- J2EE client or server environment. Throwing
ConfigurationException.
Dependencies
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>log4j-over-slf4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.aredis</groupId>
<artifactId>aredis</artifactId>
<version>1.4</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<!-- <version>1.2.16</version> -->
</dependency>
<dependency>
<groupId>net.sf.joesnmp</groupId>
<artifactId>joesnmp</artifactId>
<version>0.3.4</version>
</dependency>
<dependency>
<groupId>com.ibm.ws.admin</groupId>
<artifactId>adminClient</artifactId>
<version>8.5.0</version>
</dependency>
<dependency>
<groupId>com.ibm.ws.runtime</groupId>
<artifactId>ibmRuntime</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>com.ibm.websphere</groupId>
<artifactId>ibmorb</artifactId>
<version>8.5</version>
</dependency>
<dependency>
<groupId>com.ibm.websphere</groupId>
<artifactId>ibmorbapi</artifactId>
<version>8.5</version>
</dependency>
<dependency>
<groupId>com.ibm.websphere</groupId>
<artifactId>orb</artifactId>
<version>8.5</version>
</dependency>
</dependencies>
<properties>
<start-class>com.springboot.main.ConfigInitializer</start-class>
<jersey-version>1.5</jersey-version>
</properties>

On Spring Boot v1.5.6.RELEASE I had to do this to load WebSphere classes on the classpath:
/src/main/resources/application.properties:
spring.jmx.enabled=false
/src/main/resources/spring.properties:
spring.jndi.ignore=true

Apache spark maven tomcat:run

I want to run Apache Spark from a spring project with multiple maven module.
Spring project running correctly without apache spark dependency but when I added spark dependency I see the following:
java.lang.ClassCastException: org.springframework.web.servlet.DispatcherServlet cannot be cast to javax.servlet.Servlet
spark dependencies:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.2.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.10</artifactId>
<version>1.2.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>0.20.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.5.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-assembly-jar</artifactId>
<version>1.3.0</version>
</dependency>

Here is what works for me. Reason is spark engine has transitive dependencies to a particular version of servlet api and implementation. In the classpath it finds first smth different and complains about it.
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>${spark.version}</version>
<exclusions>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.glassfish</groupId>
<artifactId>javax.servlet</artifactId>
</exclusion>
<exclusion>
<groupId>org.eclipse.jetty.orbit</groupId>
<artifactId>javax.servlet</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.10</artifactId>
<version>${spark.version}</version>
</dependency>
Also in order to use web service as a driver I believe it's required to use compile scope of the spark libraries.

Cannot run Arquillian Tests for Google App Engine

I'm trying to run Arquillian test for GAE container, however I am getting this error:
com.google.apphosting.utils.config.AppEngineConfigException: Could not locate /tmp/servlet-test.war/WEB-INF/appengine-web.xml
at com.google.apphosting.utils.config.AppEngineWebXmlReader.getInputStream(AppEngineWebXmlReader.java:140)com.google.apphosting.utils.config.AppEngineWebXmlReader.getInputStream(AppEngineWebXmlReader.java:140)
All my other non-Arquillian tests works fine, not sure why this happens, should I put appengine-web.xml into the test folder? Or having it in the /src/main/webapp/WEB-INF should be fine?
Also in the Maven surefire report I can see this:
com.mycompany.AnnotatedEchoServletTestCase Time elapsed: 1.131 sec <<< ERROR!
java.lang.NoSuchMethodError: com.google.appengine.tools.development.DevAppServer.start()Ljava/util/concurrent/CountDownLatch;
at org.jboss.arquillian.container.appengine.embedded.AppEngineEmbeddedContainer.doDeploy(AppEngineEmbeddedContainer.java:74)
I'm just running this AnnotatedEchoServletTestCase with GAE container.
My app have this dependency in the app:
<!-- Added for GAE Arquillian Testing -->
<dependency>
<groupId>org.jboss.spec</groupId>
<artifactId>jboss-javaee-6.0</artifactId>
<version>1.0.0.Final</version>
<type>pom</type>
<scope>provided</scope>
<exclusions>
<exclusion>
<groupId>org.jboss.resteasy</groupId>
<artifactId>jaxrs-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.jboss.arquillian.junit</groupId>
<artifactId>arquillian-junit-container</artifactId>
<version>${version.arquillian}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>${version.slf4j}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jboss.shrinkwrap.descriptors</groupId>
<artifactId>shrinkwrap-descriptors-impl-javaee</artifactId>
</dependency>
<dependency>
<groupId>org.jboss.arquillian.container</groupId>
<artifactId>arquillian-gae-embedded</artifactId>
<version>1.0.0.Beta7</version>
</dependency>
<!-- End GAE Arquillian Testing -->
What could be missing in my configuration?