RDD not serializable Cassandra/Spark connector java API - java

so I previously had some questions on how to query cassandra using spark in a java maven project here: Querying Data in Cassandra via Spark in a Java Maven Project
Well my question was answered and it worked, however I've run into an issue (possibly an issue). I'm trying to now use the datastax java API. Here is my code:
package com.angel.testspark.test2;
import org.apache.commons.lang3.StringUtils;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import java.io.Serializable;
import static com.datastax.spark.connector.CassandraJavaUtil.*;
public class App
{
// firstly, we define a bean class
public static class Person implements Serializable {
private Integer id;
private String fname;
private String lname;
private String role;
// Remember to declare no-args constructor
public Person() { }
public Integer getId() { return id; }
public void setId(Integer id) { this.id = id; }
public String getfname() { return fname; }
public void setfname(String fname) { this.fname = fname; }
public String getlname() { return lname; }
public void setlname(String lname) { this.lname = lname; }
public String getrole() { return role; }
public void setrole(String role) { this.role = role; }
// other methods, constructors, etc.
}
private transient SparkConf conf;
private App(SparkConf conf) {
this.conf = conf;
}
private void run() {
JavaSparkContext sc = new JavaSparkContext(conf);
createSchema(sc);
sc.stop();
}
private void createSchema(JavaSparkContext sc) {
JavaRDD<String> rdd = javaFunctions(sc).cassandraTable("tester", "empbyrole", Person.class)
.where("role=?", "IT Engineer").map(new Function<Person, String>() {
#Override
public String call(Person person) throws Exception {
return person.toString();
}
});
System.out.println("Data as Person beans: \n" + StringUtils.join("\n", rdd.toArray()));
}
public static void main( String[] args )
{
if (args.length != 2) {
System.err.println("Syntax: com.datastax.spark.demo.JavaDemo <Spark Master URL> <Cassandra contact point>");
System.exit(1);
}
SparkConf conf = new SparkConf();
conf.setAppName("Java API demo");
conf.setMaster(args[0]);
conf.set("spark.cassandra.connection.host", args[1]);
App app = new App(conf);
app.run();
}
}
here is my error:
Exception in thread "main" org.apache.spark.SparkException: Job aborted: Task not serializable: java.io.NotSerializableException: com.angel.testspark.test2.App
at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:781)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:724)
at org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:554)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Now I KNOW exactly where my error is. It is System.out.println("Data as Person beans: \n" + StringUtils.join("\n", rdd.toArray())); because I need to convert rdd to an Array. However, the API documentation SAID i should be able to do this... this is code copied and pasted from the documentation. Why can I not serialize the RDD to an array?
I've already inserted dummy data into my cassandra using the insertions in my post that I included in the link above.
Also, a previous error that I solved was when i changed all of my getters and setters to lowercase. When I used capitals in them, it produced an error. Why can't I use capitals in my getters and setters here?
Thanks,
Angel

Changing public class App to public class App implements Serializable should fix the error. Because a java inner class will keep a reference to the outer class, your Function object will have a reference to App. As Spark needs to serialize your Function object, it requires App is also serializable.

Related

Is it possible to use IntelliJ's 'Analyze Data Flow to Here' feature with Java Lombok?

I have recently done an experiment to see how we can use Lombok to reduce boilerplate in our code.
The issue:
When creating a simple data class with a builder through Lombok annotations, in IntelliJ IDEA, I cannot right click a field, then select Analyze Data Flow to Here.
This is using the latest IntelliJ Lombok Plugin. IntelliJ Ultimate 2019.2.3.
Is there any fix for this or is it simply not supported?
Example 1 - no lombok:
public class Person {
private String name;
private int age;
private Person() {
}
public Person(Builder builder) {
name = builder.name;
age = builder.age;
}
public String getName() {
return name;
}
public int getAge() {
return age;
}
public static class Builder {
private String name;
private int age;
public Builder name(String val) {
this.name = val;
return this;
}
public Builder age(int val) {
this.age = val;
return this;
}
public Person build() {
return new Person(this);
}
}
}
public class Main {
public static void main(String[] args) {
Person person = new Person.Builder().name("tom").age(3).build();
}
}
With the above code, when I right click the "name" variable and select analyse dataflow to here, I am able to see the dataflow. As shown in screenshot:
Example 2 - with Lombok:
import lombok.Builder;
import lombok.Getter;
import lombok.NoArgsConstructor;
#NoArgsConstructor
#Builder
#Getter
public class Person {
private String name;
private int age;
}
public class Main {
public static void main(String[] args) {
Person person = Person.builder().name("tom").age(3).build();
}
}
With the above code example, selecting 'analyse data flow to here' on the name field will show the variable name, but with no tree to expand as shown in the screenshot.
"Analyze data flow to here" will not work with generated code provided by Lombok annotations.

How to fix " Failed to instantiate 'className' using constructor NO_CONSTRUCTOR with arguments" in immutable class

I use MongoDBRepository in spring boot, and when I save some object in database everything is ok. but when I find object by id spring does not allow do that.
I try to change VehicleRoutingProblemSolution type to Object type, but VehicleRoutingProblemSolution have other object field PickupService and it without default constructor to. And yes, this class has immutable... I can't create default constructors, what can I do?
import com.fasterxml.jackson.annotation.JsonProperty;
import com.graphhopper.jsprit.core.problem.solution.VehicleRoutingProblemSolution;
import org.springframework.data.annotation.Id;
import org.springframework.data.mongodb.core.mapping.Document;
#Document(collection = "vrp_solutions")
public class VrpSolutionHolder {
// Specifies the solution id
#Id
#JsonProperty("id")
private String id;
// Specifies the solution id
#JsonProperty("solution")
private VehicleRoutingProblemSolution vehicleRoutingProblemSolution;
// Created at timestamp in millis
#JsonProperty("created_at")
private Long created_at = System.currentTimeMillis();
public VrpSolutionHolder(String id, VehicleRoutingProblemSolution vehicleRoutingProblemSolution) {
this.id = id;
this.vehicleRoutingProblemSolution = vehicleRoutingProblemSolution;
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public VehicleRoutingProblemSolution getVehicleRoutingProblemSolution() {
return vehicleRoutingProblemSolution;
}
public void setVehicleRoutingProblemSolution(VehicleRoutingProblemSolution vehicleRoutingProblemSolution) {
this.vehicleRoutingProblemSolution = vehicleRoutingProblemSolution;
}
public Long getCreated_at() {
return created_at;
}
public void setCreated_at(Long created_at) {
this.created_at = created_at;
}
}
org.springframework.web.util.NestedServletException: Request
processing failed; nested exception is
org.springframework.data.mapping.model.MappingInstantiationException:
Failed to instantiate
com.graphhopper.jsprit.core.problem.solution.VehicleRoutingProblemSolution
using constructor NO_CONSTRUCTOR with arguments
I ran into the exact same problem. A persistent immutable class containing other class instances, throwing that aforementioned exception when retrieved by this repository method:
public interface ProjectCodeCacheRepository extends MongoRepository<CachedCode, String> {
public CachedCode findByCode(String code);
public List<CachedCode> findByClientId(UUID clientId);
}
...
List<CachedCode> cachedForClient = this.codeCacheRepo.`**findByClientId**`(clientId);
...
Following Erwin Smouts hints, this is nicely fixed by giving it a special constructor annotated org.springframework.data.annotation.PersistenceConstructor like so:
#Document(collection="cachedcodes")
public class CachedCode {
#PersistenceConstructor
public CachedCode(String code, UUID clientId, LocalDateTime expiration) {
this.code = code;
this.clientId = clientId;
this.expiration = expiration;
}
public CachedCode(String code, UUID clientId, long secondsExpiring) {
this.code = code;
this.clientId = clientId;
this.expiration = LocalDateTime.now().plusSeconds(secondsExpiring);
}
public UUID getClientId( ) {
return this.clientId;
}
public String getCode() {
return this.code;
}
public boolean hasExpired(LocalDateTime now) {
return (expiration.isBefore(now));
}
...
#Id
private final String code;
private final UUID clientId;
private final LocalDateTime expiration;
}
So, you should check if your VehicleRoutingProblemSolution has a) a constructor that matches the database fields (check in mongo client) and b) is annotated to be the one used by the driver (or whichever piece of Spring magic under the hood).
If your framework tool requires (visible) no-arg constructors (plus accompanying setters), and the class you have is required to stay as is, then you could roll your own, say, MutableVehicleRoutingProblemSolution where in the setters you could have :
this.vehicleRoutingProblemSolution = new VehicleRoutingProblemSolution(vehicleRoutingProblemSolution.getId(), newSolution);
Thus your MutableVehicleRoutingProblemSolution wraps around the existing VehicleRoutingProblemSolution.
Hacky smell to it, but it fits the requirements.
(Or you could try to find a tool that is able to use, not annotations on the contained fields, but annotations on constructor arguments.)
This is a problem where the corresponding class does not have a no-arg constructor like - I was facing an issue with java.io.File.
Solution:
In general - change the declaration to Object class and convert where we are using the class.
from
class MyClass{
File myfile;
}
to
class MyClass{
Object myFile;
}
For anyone using lombok, you need to remove the #Builder annotation on your class and use #Data instead, or follow the above solution to provide a specialized constructor
Oddly, I received this when I attempted to decorate a custom interface with ...
#Document(collection = "Person")
Example:
package test.barry.interfaces;
import org.springframework.data.mongodb.core.mapping.Document;
import org.springframework.data.mongodb.core.query.Query;
import org.springframework.data.mongodb.core.query.UpdateDefinition;
#Document(collection = "Person")
public interface CustomRepository
{
void updatex(Query filterPredicate, UpdateDefinition updatePredicate);
}

SpringBoot deserialization without default constructor

During the last hours I read many StackOverflow questions and articles, but none of the advices helped. What I tried:
Add #JsonCreator and #JsonProperty to both Person and Employee classes (link)
Add #JsonDeserialize(using = EmployeeDeserialize.class) to Employee class (link)
Add Lombok as dependency, set lombok.anyConstructor.addConstructorProperties=true and add #Data / #Value annotation to both Person and Employee classes (link)
Finally, I did the deserialization manually:
String json = "{\"name\": \"Unknown\",\"email\": \"please#work.now\",\"salary\":1}";
ObjectMapper objectMapper = new ObjectMapper();
Employee employee = objectMapper.readValue(json, Employee.class);
In this way I could deserialize the JSON, but as soon as I started my spring-boot-starter-web project and called
http://localhost:8080/print?name=unknown&email=please#work.now&salary=1
I got the good old BeanInstantiationException
Failed to instantiate [Employee]: No default constructor found
I run out of ideas. Does anybod know why this worked when I did the deserialization manually? And why it throws exception when I call the REST endpoint?
#SpringBootApplication
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}
#RestController
public class EmployeeController {
#GetMapping("print")
public void print(Employee employee) {
System.out.println(employee);
}
}
public class Person {
private final String name;
#JsonCreator
public Person(#JsonProperty("name") String name) {
this.name = name;
}
public String getName() {
return name;
}
}
public class Employee extends Person {
private final String email;
private final int salary;
#JsonCreator
public Employee(
#JsonProperty("name") String name,
#JsonProperty("email") String email,
#JsonProperty("salary") int salary) {
super(name);
this.email = email;
this.salary = salary;
}
public String getEmail() {
return email;
}
public int getSalary() {
return salary;
}
}
You’re implementing JSON deserialisation, yet you’re not using any JSON.
Change to use #PostMapping on your controller method and use something like Postman or cURL to send the JSON to your /print endpoint.

neo4j :: Getting java.lang.IllegalArgumentException: Class class com.my.domain.Actor is not a valid entity class. Please check the entity mapping

I was trying to learn Neo4j-OGM(Version:3.1.0).But I stucked with this Exception.Even tried with older versions but no use.And googled for help but couldn't find anything.Neo4j is the only choice for my project.I don't find anything wrong with my code. Can anyone help me with this? Thanks :) Here is my code..Sorry for this lengthy question.
#NodeEntity(label="Film")
public class Movie {
#GraphId
Long id;
#Property(name="title")
private String name;
public Movie(String name){
this.name=name;
}
public Movie(){}
}
#NodeEntity
public class Actor {
#Id
#GeneratedValue
private Long id;
#Property(name="name")
private String fullName;
#Relationship(type="ACTED_IN", direction=Relationship.OUTGOING)
private List<Role> filmography;
public Actor(String name){
this.fullName=name;
this.filmography=new ArrayList<>();
}
public Actor(){}
public void addRole(Role r){
this.filmography.add(r);
}
}
#RelationshipEntity(type="ACTED_IN")
public class Role {
#Id #GeneratedValue private Long relationshipId;
#Property private String title;
#StartNode private Actor actor;
#EndNode private Movie movie;
public Role(){}
public Role(String title,Actor actor,Movie movie){
this.actor=actor;
this.title=title;
this.movie=movie;
}
}
public class Main{
public static void main(String[] a){
Movie m1=new Movie("M1");
Actor a1=new Actor("A1");
Actor a2=new Actor("A2");
Movie m2=new Movie("M2");
Role r1=new Role("R1",a1,m1);
Role r2=new Role("R2",a2,m1);
Role r3=new Role("R3",a2,m2);
a1.addRole(r1);
a2.addRole(r2);
a2.addRole(r3);
Configuration configuration = new Configuration.Builder()
.uri("bolt://localhost")
.credentials("neo4j", "admin")
.build();
SessionFactory sessionFactory = new SessionFactory(configuration, "com.my.domain");
Session session=sessionFactory.openSession();
session.beginTransaction();
session.save(a1);
session.save(a2);
}
}
Check your package scanning in SessionFactory eg. new SessionFactory(configuration, "com.my.domain");
If declared package is not your entity package then this error also occurs
Try to check if bean for the class actor is loaded in the spring context properly. If it is not there in the context at run time due to wrong configuration (for example: #EntityScan is not defined with proper path), this exception can occur.

Spring Mongo DB with Annotation based configurations

I am learning Spring Boot and I am trying to make a very simple app that fetches data from Mongo DB by using Dynamic Queries. I am using Intellij as my IDE.
FILE: application.properties (inside resource folder)
spring.mongo.host=127.0.0.1
spring.mongo.port=27017
spring.mongo.databaseName=spring
FILE: person.java
#Document (collection = "person")
public class Person {
#Id
String id;
int age;
String name;
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public int getAge() {
return age;
}
public void setAge(int age) {
this.age = age;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}
FILE: MyRepo.java
#Repository
public interface MyRepo extends PagingAndSortingRepository<Person, String> {
public List<Person> findAllByName(String name);
}
FILE: Config.java
#Configuration
#EnableMongoRepositories(basePackages = {"mongo.customQueries"})
public class Config {
}
FILE: Main.java
public class Main {
#Autowired
public static MyRepo myRepo;
public static void main(String[] args) {
ApplicationContext context = new AnnotationConfigApplicationContext(Config.class);
MyRepo myRepo = context.getBean(MyRepo.class);
System.out.println(myRepo.findAllByName("Avishek"));
}
}
When I run the project, I get an error
Exception in thread "main" org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type [mongo.customQueries.MyRepo] is defined
What is it that I am missing here? Why is my MyRepo bean not created as most of the examples in net are doing so.
The problem is you want to annotation the MyRepo in the Main class, please remove it as below:
public class Main {
public static void main(String[] args) {
ApplicationContext context = new AnnotationConfigApplicationContext(Config.class);
MyRepo myRepo = context.getBean(MyRepo.class);
System.out.println(myRepo.findAllByName("Avishek"));
}
}
If someone could just give me a simple example to run Dynamic Queries
in Spring boot with mongo. Some examples similar to that of above. Or
how can I make the above example correct.
You can see working example here. And find explanations here.

Categories