Hi I am doing a small POC with AWS Lambda using Java,
I want to test a simple Lambda function call, which will accept a string
I am using Google's Gson to convert String to Json and use it in my demo program.
I referred this documentation for my demo -
http://docs.aws.amazon.com/lambda/latest/dg/get-started-step4-optional.html
and
http://docs.aws.amazon.com/lambda/latest/dg/java-create-jar-pkg-maven-and-eclipse.html
Here's my Project directory structure -
I am getting an error when I call the Lambda function with the following String -
"{'name':'Aniruddha','age':'25'}"
Here's the error -
{
"errorMessage": "com/google/gson/GsonBuilder",
"errorType": "java.lang.NoClassDefFoundError",
"stackTrace": [
"example.Hello.myHandler(Hello.java:11)",
"sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)",
"sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)",
"sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)",
"java.lang.reflect.Method.invoke(Method.java:498)"
]
}
Here's my Handler function -
package example;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.LambdaLogger;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
public class Hello {
public String myHandler(String request, Context context) {
GsonBuilder builder = new GsonBuilder();
Gson gson = builder.create();
QueryParams queryParams = gson.fromJson(request, QueryParams.class);
System.out.println("Name is: "+queryParams.getName()+" Age is: "+queryParams.getAge());
LambdaLogger logger = context.getLogger();
logger.log("received a Lambda request");
String message = "Hey there! " + queryParams.getName() + " you are " + queryParams.getAge() + " years old";
return message;
}
}
This is my PoJo -
package example;
public class QueryParams {
String name;
int age;
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public int getAge() {
return age;
}
public void setAge(int age) {
this.age = age;
}
}
This is the pom.xml -
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>doc-examples</groupId>
<artifactId>lambda-java-example</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>lambda-java-example</name>
<dependencies>
<!-- https://mvnrepository.com/artifact/com.amazonaws/aws-lambda-java-core -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.google.code.gson/gson -->
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>1.4</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Alright as #JBNizet advised (check question's comments),
It solved the issue.
Here's the pom.xml we need -
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>doc-examples</groupId>
<artifactId>lambda-java-example</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>lambda-java-example</name>
<dependencies>
<!-- https://mvnrepository.com/artifact/com.amazonaws/aws-lambda-java-core -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.google.code.gson/gson -->
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>1.4</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Related
I have a question about Cucumber library, I was taking a course of selenium with cucumber and testNg, but I have run into some problems because some methods no longer exist
For Example:
I cant used "CucumberFeatureWrapper" what could be the relative to this? Image: https://i.stack.imgur.com/3JMcD.png
The same way happens when i need to used .provideFeatures(); in testNgCucumberRunner.provideFeatures Image: https://i.stack.imgur.com/L76xQ.png
POM:
e<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>groupId</groupId>
<artifactId>CRMFramework</artifactId>
<version>1.0-SNAPSHOT</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>10</source>
<target>10</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M5</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
<suiteXmlFiles>
<suiteXmlFile>src/test/java/TestNg.xml</suiteXmlFile>
</suiteXmlFiles>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
<plugin>
<groupId>net.masterthought</groupId>
<artifactId>maven-cucumber-reporting</artifactId>
<version>5.4.0</version>
<executions>
<execution>
<id>execution</id>
<phase>verify</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<projectName>ExecuteAutomation</projectName>
<!-- output directory for the generated report -->
<outputDirectory>${project.build.directory}/cucumber-reports</outputDirectory>
<inputDirectory>${project.build.directory}/cucumber-json-report.json</inputDirectory>
<jsonFiles>
<!-- supports wildcard or name pattern -->
<param>**/*.json</param>
</jsonFiles>
<mergeFeaturesWithRetest>true</mergeFeaturesWithRetest>
<mergeFeaturesById>true</mergeFeaturesById>
<checkBuildResult>false</checkBuildResult>
<skipEmptyJSONFiles>true</skipEmptyJSONFiles>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<properties>
<maven.compiler.source>15</maven.compiler.source>
<maven.compiler.target>15</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.141.59</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/net.sourceforge.jexcelapi/jxl -->
<dependency>
<groupId>net.sourceforge.jexcelapi</groupId>
<artifactId>jxl</artifactId>
<version>2.6.12</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java</artifactId>
<version>6.9.1</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-core</artifactId>
<version>6.9.1</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit</artifactId>
<version>6.9.1</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-testng</artifactId>
<version>6.9.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.testng/testng -->
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>7.3.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/io.cucumber/datatable-dependencies -->
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>datatable-dependencies</artifactId>
<version>1.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.aventstack/extentreports -->
<dependency>
<groupId>com.aventstack</groupId>
<artifactId>extentreports</artifactId>
<version>5.0.6</version>
</dependency>
<dependency>
<groupId>org.jetbrains</groupId>
<artifactId>annotations</artifactId>
<version>RELEASE</version>
<scope>compile</scope>
</dependency>
</dependencies>
TestRunner
import com.aventstack.extentreports.gherkin.model.Feature;
import com.crm.framework.utilities.ExtentReport;
import io.cucumber.testng.*;
import org.testng.annotations.Test;
import org.testng.ITestContext;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.DataProvider;
import java.util.List;
//json:target/cucumber-reports/cucumberTestReport.json
#CucumberOptions(
features = {"src/test/java/features/"},
glue = {"steps"},
plugin = {"json:target/cucumber-json-report.json",
"pretty", "html:target/cucumber-report-html"})
public class TestRunner {
private TestNGCucumberRunner testNGCucumberRunner;
#BeforeClass(alwaysRun = true)
public void setUpClass() {
testNGCucumberRunner = new TestNGCucumberRunner(this.getClass());
}
#Test(dataProvider = "features")
public void LoginTest(CucumberFeatureWrapper cucumberFeatureWrapper) throws ClassNotFoundException {
//Insert the Feature Name
//ExtentReport.startFeature("login").assignAuthor("DiegoHM").assignDevice("Chrome").assignCategory("Regression");
}
#DataProvider
public Object[] features(ITestContext context) {
// return testNGCucumberRunner.
return testNGCucumberRunner.provideFeatures();
}
#AfterClass(alwaysRun = true)
public void afterClass() {
testNGCucumberRunner.finish();
}
}
You're using classes that don't exist anymore. You can find out which classes are in a library by using ctrl/cmd + clicking on a package name in most modern IDEs.
So try using:
package io.cucumber.examples.testng;
import io.cucumber.testng.CucumberOptions;
import io.cucumber.testng.FeatureWrapper;
import io.cucumber.testng.PickleWrapper;
import io.cucumber.testng.TestNGCucumberRunner;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
#CucumberOptions(....)
public class RunCucumberByCompositionTest {
private TestNGCucumberRunner testNGCucumberRunner;
#BeforeClass(alwaysRun = true)
public void setUpClass() {
testNGCucumberRunner = new TestNGCucumberRunner(this.getClass());
}
#Test(groups = "cucumber", description = "Runs Cucumber Scenarios", dataProvider = "scenarios")
public void scenario(PickleWrapper pickle, FeatureWrapper cucumberFeature) {
testNGCucumberRunner.runScenario(pickle.getPickle());
}
#DataProvider
public Object[][] scenarios() {
return testNGCucumberRunner.provideScenarios();
}
#AfterClass(alwaysRun = true)
public void tearDownClass() {
testNGCucumberRunner.finish();
}
}
From:
https://github.com/cucumber/cucumber-jvm/blob/main/examples/java-calculator-testng/src/test/java/io/cucumber/examples/testng
I´m trying to use spring-boot inside my AWS lambda application to make calls to a SOAP web-service. But looks like it isn´t autowiring my SOAP component.
Here´s my code:
#SpringBootApplication(scanBasePackages={"com.fenix"})
#EnableAutoConfiguration
#ComponentScan
public class Aplicacao extends SpringBootServletInitializer {
public static void main(String[] args) {
SpringApplication.run(Aplicacao.class, args);
}
#Override
protected SpringApplicationBuilder configure(SpringApplicationBuilder springApplicationBuilder) {
return springApplicationBuilder.sources(Aplicacao.class);
}
}
#Configuration
public class Beans {
#Bean
public Jaxb2Marshaller marshaller() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setPackagesToScan("com.tranzaxis.schemas", "org.radixware.schemas", "org.xmlsoap.schemas", "com.compassplus.schemas");
return marshaller;
}
#Bean
public TranClient tranClient(Jaxb2Marshaller marshaller) {
TranClient client = new TranClient();
client.setDefaultUri("http://rhel72.tx:12301?wsdl");
client.setMarshaller(marshaller);
client.setUnmarshaller(marshaller);
return client;
}
#Bean(name = "Tran")
public TranClient getTranClient() {
return tranClient(marshaller());
}
}
public class PostMovimentacao extends Handler implements Service {
#Autowired
#Qualifier("Tran")
private TranClient tranClient;
#Inject
private PessoaCompassBO pessoaCompassBO;
private CompassConfig compassConfig;
private static final Logger LOGGER = Logger.getLogger(PostMovimentacao.class);
#Override
protected ResponseEntity execute(ApiRequest request, Context context) throws HttpException {
MovimentacaoRequest movimentacaoRequest = new MovimentacaoRequest();
movimentacaoRequest.setOrigem(669L);
movimentacaoRequest.setDestino(657L);
movimentacaoRequest.setValor(BigDecimal.valueOf(1L));
TranInvoke invoke = tranClient.movimentacaoFinanceira(movimentacaoRequest, compassConfig); -->NullPointer here
return ResponseEntity.of(Optional.of(invoke), Optional.empty(), HttpStatus.SC_OK);
}
#Override
public void setup() {
try {
compassConfig = CompassConfig.build();
} catch (InvalidConfigException e) {
LOGGER.error("Compass config error", e);
}
}
}
Here´s my pom.xml:
<build>
<finalName>integrador-compass</finalName>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>1.5.6.RELEASE</version>
<configuration>
<layout>MODULE</layout>
</configuration>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.0</version>
<configuration>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.target}</target>
<encoding>${project.build.sourceEncoding}</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.1.11</version>
<configuration>
<targetClasses>
<param>com.fenix.*</param>
</targetClasses>
<excludedClasses>
<excludedClasse>com.fenix.handler.request*</excludedClasse>
<excludedClasse>com.fenix.handler.response*</excludedClasse>
<excludedClasse>com.fenix.model*</excludedClasse>
</excludedClasses>
<avoidCallsTo>
<avoidCallsTo>java.util.logging</avoidCallsTo>
<avoidCallsTo>org.apache.log4j</avoidCallsTo>
<avoidCallsTo>org.slf4j</avoidCallsTo>
<avoidCallsTo>org.apache.commons.logging</avoidCallsTo>
</avoidCallsTo>
<timestampedReports>false</timestampedReports>
<outputFormats>
<outputFormat>XML</outputFormat>
<outputFormat>HTML</outputFormat>
</outputFormats>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.16</version>
<configuration>
<includes>
<include>**/*Tests.java</include>
<include>**/*Test.java</include>
</includes>
</configuration>
</plugin>
</plugins>
<extensions>
<extension>
<groupId>org.springframework.build</groupId>
<artifactId>aws-maven</artifactId>
<version>5.0.0.RELEASE</version>
</extension>
</extensions>
</build>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.3.0.RELEASE</version>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.1.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.ws</groupId>
<artifactId>spring-ws-core</artifactId>
</dependency>
<dependency>
<groupId>com.fenix</groupId>
<artifactId>compass-api</artifactId>
<version>0.0.15</version>
</dependency>
<dependency>
<groupId>com.fenix</groupId>
<artifactId>lambda-commons</artifactId>
<version>1.6.2</version>
</dependency>
<dependency>
<groupId>commons-validator</groupId>
<artifactId>commons-validator</artifactId>
<version>1.5.1</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>2.6.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<version>3.6.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.mashape.unirest</groupId>
<artifactId>unirest-java</artifactId>
<version>1.4.9</version>
</dependency>
</dependencies>
Has anyone already used this configuration? I added spring-boot cause it was the only way to make my SOAP calls. If I try to make call just using code from WSDL, when it tries to connect to server, I got an error saying that request was empty. With spring-boot it doesn´t need to connect first, it just sends the request and it works fine.
Any help is welcome.
Thanks a lot.
Apparently, the Spring Boot application context is not being initialized in your code...If you really want to use Spring Boot in your Lambda Function you could try something like:
pom.xml
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.4.0.RELEASE</version>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<version>1.11.181</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-lambda</artifactId>
<version>1.11.181</version>
</dependency>
</dependencies>
Service Example:
#Component
public class MyService {
public void doSomething() {
System.out.println("Service is doing something");
}
}
Some Bean Example:
#Component
public class MyBean {
#Autowired
private MyService service;
public void executeService() {
service.doSomething();
}
}
A Lambda Handler Example:
#SpringBootApplication
public class LambdaHandler implements RequestHandler<Request, Response> {
private ApplicationContext getApplicationContext(String [] args) {
return new SpringApplicationBuilder(LambdaHandler.class)
.web(false)
.run(args);
}
public Response handleRequest(Request input, Context context) {
ApplicationContext ctx = getApplicationContext(new String[]{});
MyBean bean = ctx.getBean(MyBean.class);
bean.executeService();
return new Response();
}
}
I'm following AWS lambda's "create a jar using mvn":
http://docs.aws.amazon.com/lambda/latest/dg/java-create-jar-pkg-maven-no-ide.html
and some reason when I try to do either "mvn clean install" nor "mvn package" run properly. the project is using java 8.
Here's my pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>lambda</groupId>
<artifactId>test</artifactId>
<packaging>jar</packaging>
<version>0.0.0.1-SNAPSHOT</version>
<name>lambda-test</name>
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-events</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>com.jcraft</groupId>
<artifactId>jsch</artifactId>
<version>0.1.53</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.3.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
and my code:
package lambda.test;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.LambdaLogger;
import com.google.gson.Gson;
import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.Session;
import java.util.Objects;
import java.util.Properties;
import java.util.Vector;
public class Test {
private static final Gson gson = new Gson();
public static String myHandler(Tester tester, Context context) {
LambdaLogger logger = context.getLogger();
logger.log("received: " + tester);
try {
JSch jSch = new JSch();
Session session = jSch.getSession(username, sftpHost, sftpPort);
session.setPassword(password);
Properties config = new Properties();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
session.connect(15000);
ChannelSftp channel = (ChannelSftp) session.openChannel("sftp");
channel.connect();
channel.cd("Inbox");
Vector<ChannelSftp.LsEntry> list = channel.ls(".");
logger.log("the list ==> \n" + gson.toJson(list));
} catch (Exception e) {
e.printStackTrace();
logger.log("the exception: " + e.getMessage());
}
return String.valueOf(tester);
}
public static class Tester {
private final String name;
private final Integer id;
public Tester(String name, Integer id) {
this.name = name;
this.id = id;
}
public String getName() {
return name;
}
public Integer getId() {
return id;
}
#Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Tester tester = (Tester) o;
return Objects.equals(name, tester.name) &&
Objects.equals(id, tester.id);
}
#Override
public int hashCode() {
return Objects.hash(name, id);
}
#Override
public String toString() {
return gson.toJson(this);
}
}
}
Maybe you must set the java -version of your project, and tell to the maven compiler the value of these setting like below :
<project>
[...]
<build>
[...]
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.3</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
[...]
</build>
[...]
</project>
I think it will working better right now!
You can look at https://maven.apache.org/plugins/maven-compiler-plugin/examples/compile-using-different-jdk.html and other sections.
I am fairly new to both Spark and Cassandra and need some guidance. I am setting up a maven project that uses Spark v1.3.1 and Cassandra v2.0.14. I am attempting the following:
1) Make a connection with Oracle DB using the following approach for data input; leveraging DataFrames new to Spark 1.3.0: http://www.sparkexpert.com/2015/03/28/loading-database-data-into-spark-using-data-sources-api/
2) Using spark-cassandra-connector to make the connection between the latter; found on github.
3) Once i have the DB data in DataFrame i should be able to convert to JavaRDD type and push to Cassandra keyspace as illustrated here: http://www.datastax.com/dev/blog/accessing-cassandra-from-spark-in-java
4) In short: [Oracle DB]<---[Spark]---[spark-cassandra-connector]--->[Cassandra]
The problem i'm having comes during (step 1 from above) the Scala-lib call in my Java code; more specifically during load function call: DataFrame jdbcDF = sqlContext.load(“jdbc”, options);
Runtime Error:
java.lang.ClassNotFoundException: scala.collection.GenTraversableOnce$class”
The above errors comes despite having tried several different version of the recommended 2.10.X Scala in my pom.xml file. From my prior research i'm thinking it might be a Spark-Scala compatibility issue. I've also read that i need to include the scala-lib.jar in my classpath but i am not certain how to do this with maven. Any ideas on any of this? I've included the pom.xml and java code below:
POM.XML
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.dev</groupId>
<artifactId>spark-cassandra</artifactId>
<version>0.0.1-SPARK-CASSANDRA</version>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.35</version>
</dependency>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc6</artifactId>
<version>11.2.0</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>1.0.0-rc4</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector-java_2.10</artifactId>
<version>1.0.0-rc4</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>2.1.5</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>com.dev.cassandra</groupId>
<artifactId>spark-cassandra</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.10.3</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-compiler</artifactId>
<version>2.10.3</version>
</dependency>
<!--
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-reflect</artifactId>
<version>2.10.0-M1</version>
</dependency>
-->
<!--
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-swing</artifactId>
<version>2.10.0-M1</version>
</dependency>
-->
</dependencies>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.1.5</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.3</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
<mainClass>com.dev.cassandra.Main</mainClass>
<cleanupDaemonThreads>false</cleanupDaemonThreads>
<compilerArgument>-Xlint:all</compilerArgument>
<showWarnings>true</showWarnings>
<showDeprecation>true</showDeprecation>
</configuration>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<executions>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>scala-test-compile</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<!-- Plugin to create a single jar that includes all dependencies
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>com.dev.cassandra.Main</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
-->
</plugins>
</build>
</project>
JAVA CODE:
package com.dev.cassandra;
import java.io.Serializable;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.sql.*;
import org.apache.spark.*;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.*;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SQLContext;
import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;
import oracle.jdbc.*;
import com.datastax.spark.connector.cql.CassandraConnector;
import static com.datastax.spark.connector.CassandraJavaUtil.*;
public class Main implements Serializable {
private static final org.apache.log4j.Logger LOGGER = org.apache.log4j.Logger.getLogger(Main.class);
private static final String JDBC_DRIVER = "oracle.jdbc.driver.OracleDriver";
private static final String JDBC_USERNAME = "XXXXXO01";
private static final String JDBC_PWD = "XXXXXO01";
private static final String JDBC_CONNECTION_URL =
"jdbc:oracle:thin:" + JDBC_USERNAME + "/" + JDBC_PWD + "#CONNECTION VALUES";
private transient SparkConf conf;
private Main(SparkConf conf) {
this.conf = conf;
}
private void run() {
JavaSparkContext sc = new JavaSparkContext(conf);
SQLContext sqlContext = new SQLContext(sc);
generateData(sc);
compute(sc);
showResults(sc);
sc.stop();
}
private void generateData(JavaSparkContext sc) {
SQLContext sqlContext = new org.apache.spark.sql.SQLContext(sc);
System.out.println("AFTER SQL CONTEXT");
//Data source options
Map<String, String> options = new HashMap<>();
options.put("driver", JDBC_DRIVER);
options.put("url", JDBC_CONNECTION_URL);
options.put("dbtable","(SELECT * FROM XXX_SAMPLE_TABLE WHERE ROWNUM <=5)");
CassandraConnector connector = CassandraConnector.apply(sc.getConf());
try{
Class.forName(JDBC_DRIVER);
System.out.println("BEFORE jdbcDF");
//Load JDBC query result as DataFrame
DataFrame jdbcDF = sqlContext.load("jdbc", options);
System.out.println("AFTER jdbcDF");
List<Row> tableRows = jdbcDF.collectAsList();
System.out.println("AFTER tableRows");
for (Row tableRow : tableRows) {
System.out.println();
LOGGER.info(tableRow);
System.out.println();
}
}catch(Exception e){
//Handle errors for Class.forName
e.printStackTrace();
}
}
private void compute(JavaSparkContext sc) {
}
private void showResults(JavaSparkContext sc) {
}
public static void main(String[] args) throws InterruptedException
{
if (args.length != 2) {
System.err.println("Syntax: com.datastax.spark.dev.cassandra <Spark Master URL> <Cassandra contact point>");
System.exit(1);
}
//JavaSparkContext sc = new JavaSparkContext(new SparkConf().setAppName("SparkJdbcDs").setMaster("local[*]"));
SparkConf conf = new SparkConf().setAppName("SparkJdbcDs").setMaster("local[*]");
//SparkConf conf = new SparkConf();
//conf.setAppName("SparkJdbcDs");
//conf.setMaster(args[0]);
//conf.set("spark.cassandra.connection.host", args[1]);
Main app = new Main(conf);
app.run();
}
}
Thanks in advance!
The pom.xml is requesting the Scala 2.11 versions of some of the Spark JARs
<artifactId>spark-core_2.11</artifactId>
<artifactId>spark-sql_2.11</artifactId>
and the Scala 2.10 versions of another Spark JAR and the Cassandra connector JARs.
<artifactId>spark-streaming_2.10</artifactId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<artifactId>spark-cassandra-connector-java_2.10</artifactId>
(Based on the naming convention for Scala Artifact IDs, which end with the version of Scala you want them built for.)
These need to be consistent (a) with each other and (b) with the Scala version you're actually using.
I have defined an aspect which should be executed when the method getFirstName() is called on the User object. But it does not happen.
My Aspect class:
#Component
#Aspect
public class JpaGetFirstNameAspect {
#Pointcut("execution(* de.playground.model.User.getFirstName(..))")
public void pointCutSetFirstName() {
}
#Before("pointCutGetFirstName()")
public void beforeGetFirstName(JoinPoint joinPoint) {
System.out.println(">>>> Before retrieving first name ... " + joinPoint.getSignature().getName());
}
#After("pointCutGetFirstName()")
public void afterGetFirstName() {
System.out.println(">>>> After execution of getFirstName method ... ");
}
#AfterReturning(pointcut = "pointCutGetFirstName()", returning = "firstName")
public void afterReturningGetFirstName(JoinPoint joinPoint, String firstName) {
System.out.println("<<<< " + joinPoint.getSignature().getName());
System.out.println("<<<< returned first name of user " + firstName);
}
#AfterThrowing(pointcut = "pointCutGetFirstName()", throwing = "exc")
public void afterThrowingGetFirstName(Exception exc) {
System.out.println("|||| Retrieved Exception from getFirstName method ... " + exc.toString());
}
}
My application-context.xml:
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context" xmlns:tx="http://www.springframework.org/schema/tx" xmlns:aop="http://www.springframework.org/schema/aop"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd
http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop.xsd
">
<context:annotation-config />
<context:load-time-weaver />
<context:component-scan base-package="de.playground.service" />
</beans>
My testclass:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(locations = { "file:src/test/resources/META-INF/spring/application-context.xml" })
public class AspectIntegrationTest {
#Autowired
private GenericApplicationContext context;
private ClassPathXmlApplicationContext ctx;
public AspectIntegrationTest() {
}
#AfterClass
public static void tearDownClass() {
}
#BeforeClass
public static void setUpClass() {
}
#Before
public void setUp() throws Exception {
ctx = new ClassPathXmlApplicationContext();
}
#After
public void tearDown() {
}
#Test
public void testHijackingUser() {
User user1 = new User();
user1.setCreatedOn(new Date());
user1.setLastModified(new Date());
user1.setFirstName("Max");
user1.setLastName("Mustermann");
user1.setUsername("max.mustermann");
user1.setPassword("start123");
System.out.println(">user 1 = " + user1.getFirstName());
assertNotNull(user1);
}
}
My aop.xml:
<!DOCTYPE aspectj PUBLIC
"-//AspectJ//DTD//EN" "http://www.eclipse.org/aspectj/dtd/aspectj.dtd">
<aspectj>
<weaver options="-Xset:weaveJavaxPackages=true -verbose -showWeaveInfo">
<!-- only weave classes in this package -->
<include within="de.playground.service.*" />
</weaver>
<aspects>
<!-- use only this aspect for weaving -->
<aspect name="de.playground.aspect.JpaGetFirstNameAspect" />
</aspects>
</aspectj>
My pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>de.playground.platform</groupId>
<artifactId>playground-dev</artifactId>
<version>0.0.2-SNAPSHOT</version>
</parent>
<artifactId>playground-script</artifactId>
<dependencies>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjrt</artifactId>
<version>${version.aspectj}</version>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>${version.aspectj}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>${version.spring}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>${version.spring}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-aop</artifactId>
<version>${version.spring}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<version>${version.spring}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<properties>
<version.aspectj>1.8.5</version.aspectj>
<version.spring>4.1.6.RELEASE</version.spring>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<forkMode>once</forkMode>
<argLine>-javaagent:"${settings.localRepository}/org/springframework/spring-instrument/${version.spring}/spring-instrument-${version.spring}.jar"</argLine>
<useSystemClassloader>true</useSystemClassloader>
</configuration>
</plugin>
</plugins>
</build>
</project>
Got it working. My updated pom.xml
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<forkMode>once</forkMode>
<argLine>-javaagent:"${settings.localRepository}/org/springframework/spring-instrument/${version.spring}/spring-instrument-${version.spring}.jar"</argLine>
<useSystemClassloader>true</useSystemClassloader>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>aspectj-maven-plugin</artifactId>
<version>1.7</version>
<configuration>
<showWeaveInfo>true</showWeaveInfo>
<source>1.7</source>
<target>1.7</target>
<Xlint>ignore</Xlint>
<complianceLevel>1.7</complianceLevel>
<encoding>UTF-8</encoding>
<verbose>false</verbose>
<aspectLibraries>
<aspectLibrary>
<groupId>org.springframework</groupId>
<artifactId>spring-aspects</artifactId>
</aspectLibrary>
</aspectLibraries>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>test-compile</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjrt</artifactId>
<version>${version.aspectj}</version>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjtools</artifactId>
<version>${version.aspectj}</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>