Spark-Cassandra Maven project with java source making scala-lib calls

Spark-Cassandra Maven project with java source making scala-lib calls - java

I am fairly new to both Spark and Cassandra and need some guidance. I am setting up a maven project that uses Spark v1.3.1 and Cassandra v2.0.14. I am attempting the following:
1) Make a connection with Oracle DB using the following approach for data input; leveraging DataFrames new to Spark 1.3.0: http://www.sparkexpert.com/2015/03/28/loading-database-data-into-spark-using-data-sources-api/
2) Using spark-cassandra-connector to make the connection between the latter; found on github.
3) Once i have the DB data in DataFrame i should be able to convert to JavaRDD type and push to Cassandra keyspace as illustrated here: http://www.datastax.com/dev/blog/accessing-cassandra-from-spark-in-java
4) In short: [Oracle DB]<---[Spark]---[spark-cassandra-connector]--->[Cassandra]
The problem i'm having comes during (step 1 from above) the Scala-lib call in my Java code; more specifically during load function call: DataFrame jdbcDF = sqlContext.load(“jdbc”, options);
Runtime Error:
java.lang.ClassNotFoundException: scala.collection.GenTraversableOnce$class”
The above errors comes despite having tried several different version of the recommended 2.10.X Scala in my pom.xml file. From my prior research i'm thinking it might be a Spark-Scala compatibility issue. I've also read that i need to include the scala-lib.jar in my classpath but i am not certain how to do this with maven. Any ideas on any of this? I've included the pom.xml and java code below:
POM.XML
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.dev</groupId>
<artifactId>spark-cassandra</artifactId>
<version>0.0.1-SPARK-CASSANDRA</version>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.35</version>
</dependency>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc6</artifactId>
<version>11.2.0</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>1.0.0-rc4</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector-java_2.10</artifactId>
<version>1.0.0-rc4</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>2.1.5</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>com.dev.cassandra</groupId>
<artifactId>spark-cassandra</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.10.3</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-compiler</artifactId>
<version>2.10.3</version>
</dependency>
<!--
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-reflect</artifactId>
<version>2.10.0-M1</version>
</dependency>
-->
<!--
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-swing</artifactId>
<version>2.10.0-M1</version>
</dependency>
-->
</dependencies>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.1.5</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.3</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
<mainClass>com.dev.cassandra.Main</mainClass>
<cleanupDaemonThreads>false</cleanupDaemonThreads>
<compilerArgument>-Xlint:all</compilerArgument>
<showWarnings>true</showWarnings>
<showDeprecation>true</showDeprecation>
</configuration>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<executions>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>scala-test-compile</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<!-- Plugin to create a single jar that includes all dependencies
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>com.dev.cassandra.Main</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
-->
</plugins>
</build>
</project>
JAVA CODE:
package com.dev.cassandra;
import java.io.Serializable;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.sql.*;
import org.apache.spark.*;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.*;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SQLContext;
import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;
import oracle.jdbc.*;
import com.datastax.spark.connector.cql.CassandraConnector;
import static com.datastax.spark.connector.CassandraJavaUtil.*;
public class Main implements Serializable {
private static final org.apache.log4j.Logger LOGGER = org.apache.log4j.Logger.getLogger(Main.class);
private static final String JDBC_DRIVER = "oracle.jdbc.driver.OracleDriver";
private static final String JDBC_USERNAME = "XXXXXO01";
private static final String JDBC_PWD = "XXXXXO01";
private static final String JDBC_CONNECTION_URL =
"jdbc:oracle:thin:" + JDBC_USERNAME + "/" + JDBC_PWD + "#CONNECTION VALUES";
private transient SparkConf conf;
private Main(SparkConf conf) {
this.conf = conf;
}
private void run() {
JavaSparkContext sc = new JavaSparkContext(conf);
SQLContext sqlContext = new SQLContext(sc);
generateData(sc);
compute(sc);
showResults(sc);
sc.stop();
}
private void generateData(JavaSparkContext sc) {
SQLContext sqlContext = new org.apache.spark.sql.SQLContext(sc);
System.out.println("AFTER SQL CONTEXT");
//Data source options
Map<String, String> options = new HashMap<>();
options.put("driver", JDBC_DRIVER);
options.put("url", JDBC_CONNECTION_URL);
options.put("dbtable","(SELECT * FROM XXX_SAMPLE_TABLE WHERE ROWNUM <=5)");
CassandraConnector connector = CassandraConnector.apply(sc.getConf());
try{
Class.forName(JDBC_DRIVER);
System.out.println("BEFORE jdbcDF");
//Load JDBC query result as DataFrame
DataFrame jdbcDF = sqlContext.load("jdbc", options);
System.out.println("AFTER jdbcDF");
List<Row> tableRows = jdbcDF.collectAsList();
System.out.println("AFTER tableRows");
for (Row tableRow : tableRows) {
System.out.println();
LOGGER.info(tableRow);
System.out.println();
}
}catch(Exception e){
//Handle errors for Class.forName
e.printStackTrace();
}
}
private void compute(JavaSparkContext sc) {
}
private void showResults(JavaSparkContext sc) {
}
public static void main(String[] args) throws InterruptedException
{
if (args.length != 2) {
System.err.println("Syntax: com.datastax.spark.dev.cassandra <Spark Master URL> <Cassandra contact point>");
System.exit(1);
}
//JavaSparkContext sc = new JavaSparkContext(new SparkConf().setAppName("SparkJdbcDs").setMaster("local[*]"));
SparkConf conf = new SparkConf().setAppName("SparkJdbcDs").setMaster("local[*]");
//SparkConf conf = new SparkConf();
//conf.setAppName("SparkJdbcDs");
//conf.setMaster(args[0]);
//conf.set("spark.cassandra.connection.host", args[1]);
Main app = new Main(conf);
app.run();
}
}
Thanks in advance!

The pom.xml is requesting the Scala 2.11 versions of some of the Spark JARs
<artifactId>spark-core_2.11</artifactId>
<artifactId>spark-sql_2.11</artifactId>
and the Scala 2.10 versions of another Spark JAR and the Cassandra connector JARs.
<artifactId>spark-streaming_2.10</artifactId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<artifactId>spark-cassandra-connector-java_2.10</artifactId>
(Based on the naming convention for Scala Artifact IDs, which end with the version of Scala you want them built for.)
These need to be consistent (a) with each other and (b) with the Scala version you're actually using.

Related

Some cucumber methods do not work correctly (Cucumber Feature Wrapper)

I have a question about Cucumber library, I was taking a course of selenium with cucumber and testNg, but I have run into some problems because some methods no longer exist
For Example:
I cant used "CucumberFeatureWrapper" what could be the relative to this? Image: https://i.stack.imgur.com/3JMcD.png
The same way happens when i need to used .provideFeatures(); in testNgCucumberRunner.provideFeatures Image: https://i.stack.imgur.com/L76xQ.png
POM:
e<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>groupId</groupId>
<artifactId>CRMFramework</artifactId>
<version>1.0-SNAPSHOT</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>10</source>
<target>10</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M5</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
<suiteXmlFiles>
<suiteXmlFile>src/test/java/TestNg.xml</suiteXmlFile>
</suiteXmlFiles>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
<plugin>
<groupId>net.masterthought</groupId>
<artifactId>maven-cucumber-reporting</artifactId>
<version>5.4.0</version>
<executions>
<execution>
<id>execution</id>
<phase>verify</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<projectName>ExecuteAutomation</projectName>
<!-- output directory for the generated report -->
<outputDirectory>${project.build.directory}/cucumber-reports</outputDirectory>
<inputDirectory>${project.build.directory}/cucumber-json-report.json</inputDirectory>
<jsonFiles>
<!-- supports wildcard or name pattern -->
<param>**/*.json</param>
</jsonFiles>
<mergeFeaturesWithRetest>true</mergeFeaturesWithRetest>
<mergeFeaturesById>true</mergeFeaturesById>
<checkBuildResult>false</checkBuildResult>
<skipEmptyJSONFiles>true</skipEmptyJSONFiles>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<properties>
<maven.compiler.source>15</maven.compiler.source>
<maven.compiler.target>15</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.141.59</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/net.sourceforge.jexcelapi/jxl -->
<dependency>
<groupId>net.sourceforge.jexcelapi</groupId>
<artifactId>jxl</artifactId>
<version>2.6.12</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java</artifactId>
<version>6.9.1</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-core</artifactId>
<version>6.9.1</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit</artifactId>
<version>6.9.1</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-testng</artifactId>
<version>6.9.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.testng/testng -->
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>7.3.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/io.cucumber/datatable-dependencies -->
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>datatable-dependencies</artifactId>
<version>1.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.aventstack/extentreports -->
<dependency>
<groupId>com.aventstack</groupId>
<artifactId>extentreports</artifactId>
<version>5.0.6</version>
</dependency>
<dependency>
<groupId>org.jetbrains</groupId>
<artifactId>annotations</artifactId>
<version>RELEASE</version>
<scope>compile</scope>
</dependency>
</dependencies>
TestRunner
import com.aventstack.extentreports.gherkin.model.Feature;
import com.crm.framework.utilities.ExtentReport;
import io.cucumber.testng.*;
import org.testng.annotations.Test;
import org.testng.ITestContext;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.DataProvider;
import java.util.List;
//json:target/cucumber-reports/cucumberTestReport.json
#CucumberOptions(
features = {"src/test/java/features/"},
glue = {"steps"},
plugin = {"json:target/cucumber-json-report.json",
"pretty", "html:target/cucumber-report-html"})
public class TestRunner {
private TestNGCucumberRunner testNGCucumberRunner;
#BeforeClass(alwaysRun = true)
public void setUpClass() {
testNGCucumberRunner = new TestNGCucumberRunner(this.getClass());
}
#Test(dataProvider = "features")
public void LoginTest(CucumberFeatureWrapper cucumberFeatureWrapper) throws ClassNotFoundException {
//Insert the Feature Name
//ExtentReport.startFeature("login").assignAuthor("DiegoHM").assignDevice("Chrome").assignCategory("Regression");
}
#DataProvider
public Object[] features(ITestContext context) {
// return testNGCucumberRunner.
return testNGCucumberRunner.provideFeatures();
}
#AfterClass(alwaysRun = true)
public void afterClass() {
testNGCucumberRunner.finish();
}
}

You're using classes that don't exist anymore. You can find out which classes are in a library by using ctrl/cmd + clicking on a package name in most modern IDEs.
So try using:
package io.cucumber.examples.testng;
import io.cucumber.testng.CucumberOptions;
import io.cucumber.testng.FeatureWrapper;
import io.cucumber.testng.PickleWrapper;
import io.cucumber.testng.TestNGCucumberRunner;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
#CucumberOptions(....)
public class RunCucumberByCompositionTest {
private TestNGCucumberRunner testNGCucumberRunner;
#BeforeClass(alwaysRun = true)
public void setUpClass() {
testNGCucumberRunner = new TestNGCucumberRunner(this.getClass());
}
#Test(groups = "cucumber", description = "Runs Cucumber Scenarios", dataProvider = "scenarios")
public void scenario(PickleWrapper pickle, FeatureWrapper cucumberFeature) {
testNGCucumberRunner.runScenario(pickle.getPickle());
}
#DataProvider
public Object[][] scenarios() {
return testNGCucumberRunner.provideScenarios();
}
#AfterClass(alwaysRun = true)
public void tearDownClass() {
testNGCucumberRunner.finish();
}
}
From:
https://github.com/cucumber/cucumber-jvm/blob/main/examples/java-calculator-testng/src/test/java/io/cucumber/examples/testng

FindAnnotation class is missing when using resteasy-jackson-provider in jetty application

I am trying to use RestEasy in a Jetty application. I am using Jackson for serializing/deserializing my POJOs. I need my serialization to be polymorphic since I have endpoints that handle a POJO type that has sub-types. I thought I could use resteasy-jackson-provider to provide my MessageBodyWriter and MessageBodyReader implementations.
When I call an endpoint I get the following error:
java.lang.NoClassDefFoundError: org/jboss/resteasy/util/FindAnnotation
at org.jboss.resteasy.plugins.providers.jackson.ResteasyJacksonProvider.isWriteable(ResteasyJacksonProvider.java:42)
at org.jboss.resteasy.core.providerfactory.ResteasyProviderFactoryImpl.resolveMessageBodyWriter(ResteasyProviderFactoryImpl.java:1322)
at org.jboss.resteasy.core.providerfactory.ResteasyProviderFactoryImpl.getMessageBodyWriter(ResteasyProviderFactoryImpl.java:1293)
at org.jboss.resteasy.core.ServerResponseWriter.lambda$writeNomapResponse$2(ServerResponseWriter.java:112)
at org.jboss.resteasy.core.interception.jaxrs.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:405)
at org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:232)
at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:97)
at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:70)
at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:578)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:508)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:252)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:153)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:363)
at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:156)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:238)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:249)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:60)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:561)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:334)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:104)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:679)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:597)
at java.base/java.lang.Thread.run(Thread.java:834)
Here's my POM file:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.foster.app</groupId>
<artifactId>jettyRestEasyApiService</artifactId>
<version>3.0.0</version>
<packaging>jar</packaging>
<name>Jetty-RESTEasy Api Service</name>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<version.jetty>9.4.7.v20170914</version.jetty>
</properties>
<dependencies>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlet</artifactId>
<version>${version.jetty}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.9.9</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.9.9</version>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-core</artifactId>
<version>4.2.0.Final</version>
<type>jar</type>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jackson-provider</artifactId>
<version>3.9.0.Final</version>
</dependency>
</dependencies>
<build>
<finalName>JettyRestEasyApiService</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.target}</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.1.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<archive>
<manifest>
<mainClass>com.foster.app.resteasy.rest.App</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Here's my Main class:
import java.util.logging.Level;
import java.util.logging.Logger;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.servlet.DefaultServlet;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;
import org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher;
public class App {
private static final Logger LOGGER = Logger.getLogger(App.class.getName());
private static final String APPLICATION_PATH = "/jetty/resteasy";
private static final String CONTEXT_ROOT = "/";
private static final Server JETTY_SERVER = new Server(8080);
public App() {}
public static void main(String[] args) throws Exception {
try {
new App().run();
} catch (Exception ex) {
LOGGER.log(Level.SEVERE, "Application Exception", ex);
throw ex;
} finally {
if (JETTY_SERVER != null) {
JETTY_SERVER.destroy();
}
}
}
public void run() throws Exception {
final ServletContextHandler context = new ServletContextHandler(JETTY_SERVER, CONTEXT_ROOT);
final ServletHolder restEasyServlet = new ServletHolder(new HttpServletDispatcher());
restEasyServlet.setInitParameter("resteasy.servlet.mapping.prefix", APPLICATION_PATH);
restEasyServlet.setInitParameter("javax.ws.rs.Application", JaxRsActivator.class.getCanonicalName());
context.addServlet(restEasyServlet, APPLICATION_PATH + "/*");
final ServletHolder defaultServlet = new ServletHolder(new DefaultServlet());
context.addServlet(defaultServlet, CONTEXT_ROOT);
JETTY_SERVER.start();
LOGGER.log(Level.INFO, "Started Server");
JETTY_SERVER.join();
LOGGER.log(Level.INFO, "Joined Server");
}
}
Here's my Application class:
import java.util.HashSet;
import java.util.Set;
import javax.ws.rs.core.Application;
public class JaxRsActivator extends Application {
#Override
public Set<Class<?>> getClasses() {
Set<Class<?>> resources = new HashSet<>();
resources.add(MessageResource.class);
resources.add(TestResource.class);
return resources;
}
}
Here are my POJO classes:
import com.fasterxml.jackson.annotation.JsonSubTypes;
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import java.io.Serializable;
import java.util.Date;
#JsonTypeInfo(
use = JsonTypeInfo.Id.NAME,
include = JsonTypeInfo.As.PROPERTY,
property = "$type")
#JsonSubTypes({
#JsonSubTypes.Type(
value = PojoDog.class,
name = "PojoDog"),
#JsonSubTypes.Type(
value = PojoCat.class,
name = "PojoCat")
})
public abstract class Pojo implements Serializable {
public String property1;
public String property2;
public int property3;
public Date property4;
}
public class PojoCat extends Pojo {
public long sebastian;
}
public class PojoDog extends Pojo {
public String clifford;
}
Here's my endpoint:
#Path("/test")
public class TestResource {
#GET
#Path("pojo")
#Produces(MediaType.APPLICATION_JSON)
public Response getPojo() {
Pojo pojo = new PojoDog();
pojo.property1 = "Hello";
pojo.property2 = "World";
pojo.property3 = 5;
pojo.property4 = new Date();
((PojoDog)pojo).clifford = "I am a dog.";
return Response.ok(pojo).build();
}
}

The class org.jboss.resteasy.util.FindAnnotation is part of the resteasy-jaxrs artifact.
See search:
https://search.maven.org/search?q=fc:org.jboss.resteasy.util.FindAnnotation
That pointed me to see that you have mixed resteasy versions.
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-core</artifactId>
<version>4.2.0.Final</version>
<type>jar</type>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jackson-provider</artifactId>
<version>3.9.0.Final</version>
</dependency>
That's 100% not valid, you need to use the same resteasy version everywhere.
So from that, I took your code and made a few simple changes, and it worked.
I upgraded to Jetty 9.4.20.v20190813.
Upgraded to org.jboss.resteasy/resteasy-jackson2-provider/4.2.0.Final
Removed the two Jackson dependencies from the pom entirely, relied on them coming from transitive dependency on resteasy-jackson2-provider
<!-- REMOVED : no point having them here
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.9.9</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.9.9</version>
</dependency>
-->
I initialized the ServletContextHandler slightly differently (don't pass server into constructor, as that's the job of the handler tree).
final ServletContextHandler context = new ServletContextHandler();
final ServletHolder restEasyServlet = new ServletHolder(new HttpServletDispatcher());
context.setContextPath(CONTEXT_ROOT);
restEasyServlet.setInitParameter("resteasy.servlet.mapping.prefix", APPLICATION_PATH);
restEasyServlet.setInitParameter("javax.ws.rs.Application", JaxRsActivator.class.getCanonicalName());
context.addServlet(restEasyServlet, APPLICATION_PATH + "/*");
final ServletHolder defaultServlet = new ServletHolder("default", DefaultServlet.class); // the name "default" here is important
context.addServlet(defaultServlet, "/"); // this is not context-root, it's the default url-pattern
HandlerList handlers = new HandlerList();
handlers.addHandler(context);
handlers.addHandler(new DefaultHandler()); // to report errors that don't match the root context better
JETTY_SERVER.setHandler(handlers);
JETTY_SERVER.start();

AWS lambda + spring boot = not wiring component

I´m trying to use spring-boot inside my AWS lambda application to make calls to a SOAP web-service. But looks like it isn´t autowiring my SOAP component.
Here´s my code:
#SpringBootApplication(scanBasePackages={"com.fenix"})
#EnableAutoConfiguration
#ComponentScan
public class Aplicacao extends SpringBootServletInitializer {
public static void main(String[] args) {
SpringApplication.run(Aplicacao.class, args);
}
#Override
protected SpringApplicationBuilder configure(SpringApplicationBuilder springApplicationBuilder) {
return springApplicationBuilder.sources(Aplicacao.class);
}
}
#Configuration
public class Beans {
#Bean
public Jaxb2Marshaller marshaller() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setPackagesToScan("com.tranzaxis.schemas", "org.radixware.schemas", "org.xmlsoap.schemas", "com.compassplus.schemas");
return marshaller;
}
#Bean
public TranClient tranClient(Jaxb2Marshaller marshaller) {
TranClient client = new TranClient();
client.setDefaultUri("http://rhel72.tx:12301?wsdl");
client.setMarshaller(marshaller);
client.setUnmarshaller(marshaller);
return client;
}
#Bean(name = "Tran")
public TranClient getTranClient() {
return tranClient(marshaller());
}
}
public class PostMovimentacao extends Handler implements Service {
#Autowired
#Qualifier("Tran")
private TranClient tranClient;
#Inject
private PessoaCompassBO pessoaCompassBO;
private CompassConfig compassConfig;
private static final Logger LOGGER = Logger.getLogger(PostMovimentacao.class);
#Override
protected ResponseEntity execute(ApiRequest request, Context context) throws HttpException {
MovimentacaoRequest movimentacaoRequest = new MovimentacaoRequest();
movimentacaoRequest.setOrigem(669L);
movimentacaoRequest.setDestino(657L);
movimentacaoRequest.setValor(BigDecimal.valueOf(1L));
TranInvoke invoke = tranClient.movimentacaoFinanceira(movimentacaoRequest, compassConfig); -->NullPointer here
return ResponseEntity.of(Optional.of(invoke), Optional.empty(), HttpStatus.SC_OK);
}
#Override
public void setup() {
try {
compassConfig = CompassConfig.build();
} catch (InvalidConfigException e) {
LOGGER.error("Compass config error", e);
}
}
}
Here´s my pom.xml:
<build>
<finalName>integrador-compass</finalName>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>1.5.6.RELEASE</version>
<configuration>
<layout>MODULE</layout>
</configuration>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.0</version>
<configuration>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.target}</target>
<encoding>${project.build.sourceEncoding}</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.1.11</version>
<configuration>
<targetClasses>
<param>com.fenix.*</param>
</targetClasses>
<excludedClasses>
<excludedClasse>com.fenix.handler.request*</excludedClasse>
<excludedClasse>com.fenix.handler.response*</excludedClasse>
<excludedClasse>com.fenix.model*</excludedClasse>
</excludedClasses>
<avoidCallsTo>
<avoidCallsTo>java.util.logging</avoidCallsTo>
<avoidCallsTo>org.apache.log4j</avoidCallsTo>
<avoidCallsTo>org.slf4j</avoidCallsTo>
<avoidCallsTo>org.apache.commons.logging</avoidCallsTo>
</avoidCallsTo>
<timestampedReports>false</timestampedReports>
<outputFormats>
<outputFormat>XML</outputFormat>
<outputFormat>HTML</outputFormat>
</outputFormats>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.16</version>
<configuration>
<includes>
<include>**/*Tests.java</include>
<include>**/*Test.java</include>
</includes>
</configuration>
</plugin>
</plugins>
<extensions>
<extension>
<groupId>org.springframework.build</groupId>
<artifactId>aws-maven</artifactId>
<version>5.0.0.RELEASE</version>
</extension>
</extensions>
</build>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.3.0.RELEASE</version>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.1.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.ws</groupId>
<artifactId>spring-ws-core</artifactId>
</dependency>
<dependency>
<groupId>com.fenix</groupId>
<artifactId>compass-api</artifactId>
<version>0.0.15</version>
</dependency>
<dependency>
<groupId>com.fenix</groupId>
<artifactId>lambda-commons</artifactId>
<version>1.6.2</version>
</dependency>
<dependency>
<groupId>commons-validator</groupId>
<artifactId>commons-validator</artifactId>
<version>1.5.1</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>2.6.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<version>3.6.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.mashape.unirest</groupId>
<artifactId>unirest-java</artifactId>
<version>1.4.9</version>
</dependency>
</dependencies>
Has anyone already used this configuration? I added spring-boot cause it was the only way to make my SOAP calls. If I try to make call just using code from WSDL, when it tries to connect to server, I got an error saying that request was empty. With spring-boot it doesn´t need to connect first, it just sends the request and it works fine.
Any help is welcome.
Thanks a lot.

Apparently, the Spring Boot application context is not being initialized in your code...If you really want to use Spring Boot in your Lambda Function you could try something like:
pom.xml
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.4.0.RELEASE</version>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<version>1.11.181</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-lambda</artifactId>
<version>1.11.181</version>
</dependency>
</dependencies>
Service Example:
#Component
public class MyService {
public void doSomething() {
System.out.println("Service is doing something");
}
}
Some Bean Example:
#Component
public class MyBean {
#Autowired
private MyService service;
public void executeService() {
service.doSomething();
}
}
A Lambda Handler Example:
#SpringBootApplication
public class LambdaHandler implements RequestHandler<Request, Response> {
private ApplicationContext getApplicationContext(String [] args) {
return new SpringApplicationBuilder(LambdaHandler.class)
.web(false)
.run(args);
}
public Response handleRequest(Request input, Context context) {
ApplicationContext ctx = getApplicationContext(new String[]{});
MyBean bean = ctx.getBean(MyBean.class);
bean.executeService();
return new Response();
}
}

Amazon Web Service Lambda using Java

Hi I am doing a small POC with AWS Lambda using Java,
I want to test a simple Lambda function call, which will accept a string
I am using Google's Gson to convert String to Json and use it in my demo program.
I referred this documentation for my demo -
http://docs.aws.amazon.com/lambda/latest/dg/get-started-step4-optional.html
and
http://docs.aws.amazon.com/lambda/latest/dg/java-create-jar-pkg-maven-and-eclipse.html
Here's my Project directory structure -
I am getting an error when I call the Lambda function with the following String -
"{'name':'Aniruddha','age':'25'}"
Here's the error -
{
"errorMessage": "com/google/gson/GsonBuilder",
"errorType": "java.lang.NoClassDefFoundError",
"stackTrace": [
"example.Hello.myHandler(Hello.java:11)",
"sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)",
"sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)",
"sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)",
"java.lang.reflect.Method.invoke(Method.java:498)"
]
}
Here's my Handler function -
package example;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.LambdaLogger;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
public class Hello {
public String myHandler(String request, Context context) {
GsonBuilder builder = new GsonBuilder();
Gson gson = builder.create();
QueryParams queryParams = gson.fromJson(request, QueryParams.class);
System.out.println("Name is: "+queryParams.getName()+" Age is: "+queryParams.getAge());
LambdaLogger logger = context.getLogger();
logger.log("received a Lambda request");
String message = "Hey there! " + queryParams.getName() + " you are " + queryParams.getAge() + " years old";
return message;
}
}
This is my PoJo -
package example;
public class QueryParams {
String name;
int age;
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public int getAge() {
return age;
}
public void setAge(int age) {
this.age = age;
}
}
This is the pom.xml -
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>doc-examples</groupId>
<artifactId>lambda-java-example</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>lambda-java-example</name>
<dependencies>
<!-- https://mvnrepository.com/artifact/com.amazonaws/aws-lambda-java-core -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.google.code.gson/gson -->
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>1.4</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>

Alright as #JBNizet advised (check question's comments),
It solved the issue.
Here's the pom.xml we need -
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>doc-examples</groupId>
<artifactId>lambda-java-example</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>lambda-java-example</name>
<dependencies>
<!-- https://mvnrepository.com/artifact/com.amazonaws/aws-lambda-java-core -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.google.code.gson/gson -->
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>1.4</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>

storm hdfs connector ... trying to write data into hdfs using storm

I am trying to write data into HDFS using "storm-hdfs connector 0.1.3".
The github URL: https://github.com/ptgoetz/storm-hdfs,I have added this dependency into my maven project.
<dependency>
<groupId>com.github.ptgoetz</groupId>
<artifactId>storm-hdfs</artifactId>
<version>0.1.3-SNAPSHOT</version>
<scope>provided</scope>
</dependency>
The sample topology to write the data into HDFS is provided in the storm-hdfs project itself. I just modified it to match to my file locations. The HdfsFileTopology is :
package my.company.app;
import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.StormSubmitter;
import backtype.storm.spout.SpoutOutputCollector;
import backtype.storm.task.OutputCollector;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.topology.base.BaseRichBolt;
import backtype.storm.topology.base.BaseRichSpout;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
import backtype.storm.tuple.Values;
import org.apache.storm.hdfs.bolt.HdfsBolt;
import org.apache.storm.hdfs.bolt.AbstractHdfsBolt;
import org.apache.storm.hdfs.bolt.SequenceFileBolt;
import org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat;
import org.apache.storm.hdfs.bolt.format.DelimitedRecordFormat;
import org.apache.storm.hdfs.bolt.format.FileNameFormat;
import org.apache.storm.hdfs.bolt.format.RecordFormat;
import org.apache.storm.hdfs.bolt.rotation.FileRotationPolicy;
import org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy;
import org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy.Units;
import org.apache.storm.hdfs.bolt.rotation.TimedRotationPolicy;
import org.apache.storm.hdfs.bolt.sync.CountSyncPolicy;
import org.apache.storm.hdfs.bolt.sync.SyncPolicy;
import org.apache.storm.hdfs.common.rotation.MoveFileAction;
import org.yaml.snakeyaml.Yaml;
import java.io.FileInputStream;
import java.io.InputStream;
import java.util.HashMap;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
public class HdfsFileTopology {
static final String SENTENCE_SPOUT_ID = "sentence-spout";
static final String BOLT_ID = "my-bolt";
static final String TOPOLOGY_NAME = "test-topology";
public static void main(String[] args) throws Exception {
Config config = new Config();
config.setNumWorkers(1);
SentenceSpout spout = new SentenceSpout();
// sync the filesystem after every 1k tuples
SyncPolicy syncPolicy = new CountSyncPolicy(1000);
// rotate files when they reach 5MB
FileRotationPolicy rotationPolicy = new TimedRotationPolicy(1.0f, TimedRotationPolicy.TimeUnit.MINUTES);
FileNameFormat fileNameFormat = new DefaultFileNameFormat()
.withPath("/users/storm/")
.withExtension(".txt");
// use "|" instead of "," for field delimiter
RecordFormat format = new DelimitedRecordFormat()
.withFieldDelimiter("|");
Yaml yaml = new Yaml();
InputStream in = new FileInputStream(args[1]);
Map<String, Object> yamlConf = (Map<String, Object>) yaml.load(in);
in.close();
config.put("hdfs.config", yamlConf);
HdfsBolt bolt = new HdfsBolt()
.withConfigKey("hdfs.config")
.withFsUrl(args[0])
.withFileNameFormat(fileNameFormat)
.withRecordFormat(format)
.withRotationPolicy(rotationPolicy)
.withSyncPolicy(syncPolicy)
.addRotationAction(new MoveFileAction().toDestination("/dest2/"));
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout(SENTENCE_SPOUT_ID, spout, 1);
// SentenceSpout --> MyBolt
builder.setBolt(BOLT_ID, bolt, 4)
.shuffleGrouping(SENTENCE_SPOUT_ID);
if (args.length == 2) {
LocalCluster cluster = new LocalCluster();
cluster.submitTopology(TOPOLOGY_NAME, config, builder.createTopology());
waitForSeconds(120);
cluster.killTopology(TOPOLOGY_NAME);
cluster.shutdown();
System.exit(0);
} else if (args.length == 3) {
StormSubmitter.submitTopology(args[0], config, builder.createTopology());
} else{
System.out.println("Usage: HdfsFileTopology [topology name] <yaml config file>");
}
}
public static void waitForSeconds(int seconds) {
try {
Thread.sleep(seconds * 1000);
} catch (InterruptedException e) {
}
}
public static class SentenceSpout extends BaseRichSpout {
private ConcurrentHashMap<UUID, Values> pending;
private SpoutOutputCollector collector;
private String[] sentences = {
"my dog has fleas",
"i like cold beverages",
"the dog ate my homework",
"don't have a cow man",
"i don't think i like fleas"
};
private int index = 0;
private int count = 0;
private long total = 0L;
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("sentence", "timestamp"));
}
public void open(Map config, TopologyContext context,
SpoutOutputCollector collector) {
this.collector = collector;
this.pending = new ConcurrentHashMap<UUID, Values>();
}
public void nextTuple() {
Values values = new Values(sentences[index], System.currentTimeMillis());
UUID msgId = UUID.randomUUID();
this.pending.put(msgId, values);
this.collector.emit(values, msgId);
index++;
if (index >= sentences.length) {
index = 0;
}
count++;
total++;
if(count > 20000){
count = 0;
System.out.println("Pending count: " + this.pending.size() + ", total: " + this.total);
}
Thread.yield();
}
public void ack(Object msgId) {
this.pending.remove(msgId);
}
public void fail(Object msgId) {
System.out.println("**** RESENDING FAILED TUPLE");
this.collector.emit(this.pending.get(msgId), msgId);
}
}
public static class MyBolt extends BaseRichBolt {
private HashMap<String, Long> counts = null;
private OutputCollector collector;
public void prepare(Map config, TopologyContext context, OutputCollector collector) {
this.counts = new HashMap<String, Long>();
this.collector = collector;
}
public void execute(Tuple tuple) {
collector.ack(tuple);
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
// this bolt does not emit anything
}
#Override
public void cleanup() {
}
}
}
I compile the project using maven (group id : my.company.app) and the build is successful but when I submit the jar file to storm it throws error.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/storm/hdfs/bolt/format/FileNameFormat
Caused by: java.lang.ClassNotFoundException: org.apache.storm.hdfs.bolt.format.FileNameFormat
Even though I have included the class,why is it throwing an error that the class was not found?
And any help in how to write data to HDFS using storm will be appreciated.
As requested the pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany.app</groupId>
<artifactId>hdfs_example</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<name>hdfs_example</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<repositories>
<repository>
<id>Codehaus</id>
<url>http://repository.codehaus.org</url>
</repository>
<repository>
<id>Codehaus.Snapshots</id>
<url>http://snapshots.repository.codehaus.org</url>
<snapshots><enabled>true</enabled></snapshots>
</repository>
<repository>
<id>github-releases</id>
<url>http://oss.sonatype.org/content/repositories/github-releases/</url>
</repository>
<repository>
<id>clojars.org</id>
<url>http://clojars.org/repo</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.8.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.9.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.easytesting</groupId>
<artifactId>fest-assert-core</artifactId>
<version>2.0M8</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jmock</groupId>
<artifactId>jmock</artifactId>
<version>2.6.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>0.9.2-incubating</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.2.1</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>15.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.2.0</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.github.ptgoetz</groupId>
<artifactId>storm-hdfs</artifactId>
<version>0.1.3-SNAPSHOT</version>
<scope>provided</scope>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/java</sourceDirectory>
<testSourceDirectory>test/main/java</testSourceDirectory>
<resources>
<resource>
<directory>${basedir}/multilang</directory>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>1.4</version>
<configuration>
<createDependencyReducedPom>true</createDependencyReducedPom>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass></mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
<!--
<plugin>
<groupId>com.theoryinpractise</groupId>
<artifactId>clojure-maven-plugin</artifactId>
<version>1.3.12</version>
<extensions>true</extensions>
<configuration>
<sourceDirectories>
<sourceDirectory>src/clj</sourceDirectory>
</sourceDirectories>
</configuration>
<executions>
<execution>
<id>compile</id>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>test</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
-->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.2.1</version>
<executions>
<execution>
<goals>
<goal>exec</goal>
</goals>
</execution>
</executions>
<configuration>
<executable>java</executable>
<includeProjectDependencies>true</includeProjectDependencies>
<includePluginDependencies>false</includePluginDependencies>
<classpathScope>compile</classpathScope>
<mainClass>${storm.topology}</mainClass>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
</plugins>
</build>
</project>

I had same issue and after checking pom.xml in detail I realized that in dependency of storm-hdfs <version>0.1.3-SNAPSHOT</version> has scope defined as included which I think means that we have to add jar into storm jar and maven wont do it during packaging.
I changed version to what is available in maven repo and removed the scope which forced maven to download jar and include it in storm jar during build.
Here is my pom.xml for reference(with some basic details removed):
src/main/resources/
false
core-site.xml
hdfs-site.xml
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.2</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>1.4</version>
<configuration>
<createDependencyReducedPom>true</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.company.main</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>0.9.2-incubating</version>
<!-- keep storm out of the jar-with-dependencies -->
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-kafka</artifactId>
<version>0.9.2-incubating</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<!-- Utilities -->
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.2.1</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>15.0</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.8.1.1</version>
<exclusions>
<exclusion>
<groupId>javax.jms</groupId>
<artifactId>jms</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jdmk</groupId>
<artifactId>jmxtools</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jmx</groupId>
<artifactId>jmxri</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</exclusion>
<exclusion>
<groupId>com.101tec</groupId>
<artifactId>zkclient</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- our cluster hadoop version -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.4.0</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- our cluster hadoop version -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.4.0</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- apache hdfs-bolt related dependencies -->
<dependency>
<groupId>com.github.ptgoetz</groupId>
<artifactId>storm-hdfs</artifactId>
<version>0.1.2</version>
<exclusions>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
</exclusion>
</exclusions>
</dependency>

We Keep Coding

Java is a programming language and computing platform first released by Sun Microsystems in 1995.

Spark-Cassandra Maven project with java source making scala-lib calls - java

Related

Some cucumber methods do not work correctly (Cucumber Feature Wrapper)

FindAnnotation class is missing when using resteasy-jackson-provider in jetty application

AWS lambda + spring boot = not wiring component

Amazon Web Service Lambda using Java

storm hdfs connector ... trying to write data into hdfs using storm

Categories

Resources