Deserialise a POJO in Kafka Streams - java

My Kafka topic has messages of this format
user1,subject1,80|user1,subject2,90
user2,subject1,70|user2,subject2,100
and so on.
I have created User POJO as below.
class User implements Serializable{
/**
*
*/
private static final long serialVersionUID = -253687203767610477L;
private String userId;
private String subject;
private String marks;
public User(String userId, String subject, String marks) {
super();
this.userId = userId;
this.subject = subject;
this.marks = marks;
}
public String getUserId() {
return userId;
}
public void setUserId(String userId) {
this.userId = userId;
}
public String getSubject() {
return subject;
}
public void setSubject(String subject) {
this.subject = subject;
}
public String getMarks() {
return marks;
}
public void setMarks(String marks) {
this.marks = marks;
}
}
Further I have created default key value serialization
streamProperties.put(
StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
streamProperties.put(
StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
I am trying to find count by userID as follows. Also I need User object to perform some other functionalities.
KTable<String, Long> wordCount = streamInput
.flatMap(new KeyValueMapper<String, String, Iterable<KeyValue<String,User>>>() {
#Override
public Iterable<KeyValue<String, User>> apply(String key, String value) {
String[] userObjects = value.split("|");
List<KeyValue<String, User>> userList = new LinkedList<>();
for(String userObject: userObjects) {
String[] userData = userObject.split(",");
userList.add(KeyValue.pair(userData[0],
new User(userData[0],userData[1],userData[2])));
}
return userList;
}
})
.groupByKey()
.count();
I am getting the below error
Caused by: org.apache.kafka.streams.errors.StreamsException: A serializer (key: org.apache.kafka.common.serialization.StringSerializer / value: org.apache.kafka.common.serialization.StringSerializer) is not compatible to the actual key or value type (key type: java.lang.String / value type: com.example.testing.dao.User). Change the default Serdes in StreamConfig or provide correct Serdes via method parameters.
I think I need to provide correct Serde for User Class.

The problem is with Value Serdes.
There are two version of function groupBy:
KStream::KGroupedStream<K, V> groupByKey();
KStream::KGroupedStream<K, V> groupByKey(final Grouped<K, V> grouped);
First version under the hood call second with Grouped with default Serdes (In your case it was for key and value StringSerde
Your flatMap map message to KeyValue<String, User> type so value was of type User.
Solution in your case would be instead using groupByKey() call groupByKey(Grouped.with(keySerde, valSerde));, with proper Serdes.

Related

Spring Enum list empty in #RequestBody

I have the below enum with two values, and i have a search api with many fields, One of these fields is a list of StatusEnum. So i created a dto that contains this field.
The problem when i send the data the list status is always empty
json exp: {"status":["En activité"],"startDate":null,"endDate":null}
public enum StatusEnum {
INACTIVITY, ENDACTIVITY;
private static Map<String, StatusEnum > namesMap = new HashMap<>(2);
static {
namesMap.put("En activité", INACTIVITY);
namesMap.put("En fin d'activité", ENDACTIVITY);
}
#JsonCreator
public static StatusEnum forValue(String value) {
return namesMap.get(StringUtils.lowerCase(value));
}
#JsonValue
public String toValue() {
for (Entry<String, StatusEnum > entry : namesMap.entrySet()) {
if (entry.getValue() == this)
return entry.getKey();
}
return null;
}
}
#PostMapping
public ResponseEntity<List<Object>> search(#RequestBody SearchDTO search) { }
public class SearchDTO {
private Date startDate;
private Date endDate
private List<StatusEnum> status;
//getter and setter
}
#JsonCreator
public static StatusEnum forValue(String value) {
return namesMap.get(StringUtils.lowerCase(value));
}
Problem is the usage of #lowerCase in forValue!
Your keys in your map aren't lower-cased. That's why namesMap.get can't find anything.

Modelmapper is not recognizing UUIDs

Hey I just began playing around with Modelmapper to map jOOQ records to POJOs.
This is the schema for the table whose records I am attempting to convert (Postgresql)
CREATE TABLE IF NOT EXISTS actor(
actor_id UUID DEFAULT uuid_generate_v4(),
first_name VARCHAR(256) NOT NULL,
last_name VARCHAR(256) NOT NULL,
PRIMARY KEY(actor_id)
);
Here is what the POJO looks like:
#JsonDeserialize(builder = Actor.Builder.class)
public class Actor {
private final UUID actorId;
private final String firstName;
private final String lastName;
private Actor(final Builder builder) {
actorId = builder.actorId;
firstName = builder.firstName;
lastName = builder.lastName;
}
public static Builder newBuilder() {
return new Builder();
}
public UUID getActorId() {
return actorId;
}
public String getFirstName() {
return firstName;
}
public String getLastName() {
return lastName;
}
#JsonIgnoreProperties(ignoreUnknown = true)
public static final class Builder {
private UUID actorId;
private String firstName;
private String lastName;
private Builder() {
}
public Builder withActorId(final UUID val) {
actorId = val;
return this;
}
public Builder withFirstName(final String val) {
firstName = val;
return this;
}
public Builder withLastName(final String val) {
lastName = val;
return this;
}
public Actor build() {
return new Actor(this);
}
}
}
I am creating a ModelMapper bean in my application and registering a UUID converter to it.
#Bean
public ModelMapper modelMapper() {
final ModelMapper mapper = new ModelMapper();
Provider<UUID> uuidProvider = new AbstractProvider<UUID>() {
#Override
public UUID get() {
return UUID.randomUUID();
}
};
final Converter<String, UUID> uuidConverter = new AbstractConverter<>() {
#Override
protected UUID convert(final String source) {
return UUID.fromString(source);
}
};
mapper.createTypeMap(String.class, UUID.class);
mapper.addConverter(uuidConverter);
mapper.getTypeMap(String.class, UUID.class).setProvider(uuidProvider);
mapper.getConfiguration()
.setSourceNameTokenizer(NameTokenizers.UNDERSCORE)
.addValueReader(new RecordValueReader())
.setDestinationNameTransformer(NameTransformers.builder("with"))
.setDestinationNamingConvention(NamingConventions.builder("with"));
mapper.validate();
return mapper;
}
I then use the model mapper to map the ActorRecord from the jOOQ autogenerated code to the POJO
public Optional<Actor> getActor(final UUID actorId) {
return Optional.ofNullable(dsl.selectFrom(ACTOR)
.where(ACTOR.ACTOR_ID.eq(actorId))
.fetchOne())
.map(e -> modelMapper.map(e, Actor.Builder.class).build());
}
This works except the UUID is always null. For example:
{"actor_id":null,"first_name":"John","last_name":"Doe"}
However when I change the following in the Builder:
public Builder withActorId(final String val) {
actorId = UUID.fromString(val);
return this;
}
It works! Unfortunately this does not work with an overloaded method:
public Builder withActorId(final String val) {
actorId = UUID.fromString(val);
return this;
}
public Builder withActorId(final UUID val) {
actorId = val;
return this;
}
As this also returns null.
You can see from the autogenerated jOOQ code it should be handling a UUID:
/**
* The column <code>public.actor.actor_id</code>.
*/
public final TableField<ActorRecord, UUID> ACTOR_ID = createField(DSL.name("actor_id"), org.jooq.impl.SQLDataType.UUID.nullable(false).defaultValue(org.jooq.impl.DSL.field("uuid_generate_v4()", org.jooq.impl.SQLDataType.UUID)), this, "");
I am not sure what I am exactly missing. I do not want to create a custom converter for each of my entities as I have a lot of them and they all contain (at least 1) UUID. Ideally I want to configure the ModelMapper to know about UUID and whenever it sees one it can handle it. Thanks!
NOTE: I also tried this with Lombok #Data object and it does not work either.
#JsonDeserialize(builder = Actor.ActorBuilder.class)
#Data
public class Actor {
private UUID actorId;
private String firstName;
private String lastName;
#JsonPOJOBuilder(withPrefix = "with")
public static class ActorBuilder {
}
}
UUID.fromString(val) is not allowed. I had same problem yesterday. Try to put to ModelMapper configurations UUID converted to String.

What is the best way to get the result through Java8 function?

I need to filter elements and then sort based on certain column. Post that I would need to find the unique entries based on combination of columns. Since it is file processing, pipe(|) is used as delimiter to denote the column value.
String s1= "12|Thor|Asgaurd|1000000|Avenger|Active"
String s2= "234|Iron man|New York|9999999|Avenger|Active"
String s3= "420|Loki|Asgaurd|||Inactive"
String s4= "12|Thor|Asgaurd Bank|1000000|Avenger HQ|Active"
Data first needs to be filtered based on the Active/Inactive status. Then it needs to be sorted based on 4th column. Lastly, the uniqueness needs to be maintained by combining column 1,2,3.
Expected Output =
"234|Iron man|New York|9999999|Avenger|Active"
"12|Thor|Asgaurd|1000000|Avenger|Active"
Creating a model class and parsing the string is the way to go, but if for some reaseon you don't want to do that you can do it this way:
import java.util.Comparator;
import java.util.stream.Collectors;
import java.util.stream.Stream;
List<String> result = Stream.of(s1, s2, s3, s4)
.filter(s -> s.split("\\|")[5].equals("Active"))
.sorted(Comparator.comparing(e -> e.split("\\|")[4]))
.collect(Collectors.toList());
First of all you should create an Object which represents your String data. Something like this:
public class MyObject {
private int id;
private String name;
private String location;
private Integer value;
private String category;
private String state;
public MyObject(String entry) {
String[] parts = entry.split("\\|");
if (parts.length != 6) {
throw new IllegalArgumentException("entry has not 6 parts");
}
id = Integer.parseInt(parts[0]);
name = parts[1];
location = parts[2];
try {
value = Integer.parseInt(parts[3]);
} catch (NumberFormatException ignored) {
}
category = parts[4];
state = parts[5];
}
// getters
#Override
public String toString() {
return String.join("|", String.valueOf(id), name, location, String.valueOf(value), category, state);
}
}
With this you can create a Stream of objects from your Strings and to the filter, sort and distinct operations afterwards:
Collection<MyObject> result = Stream.of(s1, s2, s3, s4)
.map(MyObject::new)
.filter(o -> "Active".equals(o.getState()))
.sorted(Comparator.comparing(MyObject::getValue).reversed())
.collect(Collectors.toMap(o -> Arrays.asList(o.getId(), o.getName()),
Function.identity(), (o1, o2) -> o1, LinkedHashMap::new))
.values();
result.forEach(System.out::println);
After the map operation you filter the values by state and sort them by column 4 (value in my case). At the end you collect all the values in a map for the distinct operation. Add all values you need distinction for to the Arrays.asList(). As values the map takes all the original values (Function.identity()). For duplicates we keep the first value ((o1, o2) -> o1) and we are using a LinkedHashMap to keep the order of the items. At the end wee use only the values of the map.
If you need a List instead of a Collection use new ArrayList(result).
The result will be this:
234|Iron man|New York|9999999|Avenger|Active
12|Thor|Asgaurd|1000000|Avenger|Active
It seems like you're unable to filter while everything is string only.
Try this,
create a new model class which can hold your columns.
Ex:
class MyData{
private String name;
private String city;
private String distance;
private String organization;
private String status;
//And create Getter Setter method for all above fields.
}
Now came to your main class where you can play with your code stuff.
Map<MyData> map = new HashMap<MyData>();
MyData myData = new MyData();
myData.setName("Thor");
myData.setCity("Asgaurd");
myData.setDistance("1000000");
myData.setOrganization("Avenger");
myData.setStatus("Active");
map.put(12, myData);
//Same thing for all other data (note: use the loop for data insertion in map)
Map<String, MyData> sorted = map.entrySet().stream().sorted(comparingByValue()).collect(toMap(e -> e.getKey(), e -> e.getValue().getName(), (e1, e2) -> e2,LinkedHashMap::new));
System.out.println("map after sorting by values: " + sorted);
You can solve your task this way:
Firstly, just create POJO(Plain Old Java Object) and override the toString() method.
class MarvelPerson {
private Integer id;
private String name;
private String origin;
private Integer point = null;
private String faction;
private String status;
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getOrigin() {
return origin;
}
public void setOrigin(String origin) {
this.origin = origin;
}
public Integer getPoint() {
return point;
}
public void setPoint(Integer point) {
this.point = point;
}
public String getFaction() {
return faction;
}
public void setFaction(String faction) {
this.faction = faction;
}
public String getStatus() {
return status;
}
public void setStatus(String status) {
this.status = status;
}
#Override
public String toString() {
StringBuilder builder = new StringBuilder();
builder.append(id);
builder.append("|");
builder.append(name);
builder.append("|");
builder.append(origin);
builder.append("|");
if(point != null) {
builder.append(point);
}
builder.append("|");
if(faction != null) {
builder.append(faction);
}
builder.append("|");
builder.append(status);
return builder.toString();
}
}
Then, you should write the parser from string to MarvelPerson. Side note: Carefully, my implementation is pretty basic, and I suppose it should be modified because I may not have foreseen some corner cases.
class PersonParser {
static MarvelPerson parse(String data) {
MarvelPerson person = new MarvelPerson();
String[] array = data.split("\\|", -1);
person.setId(Integer.parseInt(array[0]));
person.setName(array[1]);
person.setOrigin(array[2]);
if(!array[3].isEmpty()) {
person.setPoint(Integer.parseInt(array[3]));
}
if(!array[4].isEmpty()) {
person.setFaction(array[4]);
}
person.setStatus(array[5]);
return person;
}
}
And then your solution:
public class Test {
public static void main(String[] args) {
List<MarvelPerson> list = new ArrayList<>();
list.add(PersonParser.parse("12|Thor|Asgaurd|1000000|Avenger|Active"));
list.add(PersonParser.parse("234|Iron man|New York|9999999|Avenger|Active"));
list.add(PersonParser.parse("420|Loki|Asgaurd|||Inactive"));
list.add(PersonParser.parse("12|Thor|Asgaurd Bank|1000000|Avenger HQ|Actie"));
list.stream()
.filter(marvelPerson -> marvelPerson.getStatus().equals("Active"))
.sorted((o1, o2) -> o1.getPoint() <= o2.getPoint() ? 1 : -1)
.forEach(marvelPerson -> {
System.out.println(marvelPerson.toString());
});
}
}
The output to be printed:
234|Iron man|New York|9999999|Avenger|Active
12|Thor|Asgaurd|1000000|Avenger|Active

Spring Boot - JSON Object Array to Java Array

I have an endpoint in spring boot that consumes this JSON as an example:
{
"userId": 3,
"postBody": "This is the body of a post",
"postTitle": "This is the title of a post",
"created": null,
"tagList": ["tag1", "tag2", "tag3"]
}
The endpoint:
#RequestMapping(value="/newPost", method = RequestMethod.POST, produces="application/json", consumes = "application/json")
#ResponseBody
public ResponseEntity newPost(#RequestBody Map<String, Object> body) throws Exception {
I know the issue here is the Request body is being saved as a Map of objects which is fine for all the other attributes except the tagList. How can I get tagList to be an array of Strings in Java?
Thanks.
A mixutre of Ankur and Jose's answers solved this, thanks for the fast responses guys!
You should probably create a Java class which represents the input JSON and use it in the method newPost(.....). For example:-
public class UserPostInfo {
private int userId;
private String postBody;
private String postTitle;
private Date created;
private List<String> tagList;
}
Also, include the getter/setter methods in this class.
If you want to modify the behavior of JSON parsing, you can use Annotations to change field names, include only non-null values, and stuff like this.
If you don't want to use a custom POJO you could also just handle the deserialization into a Map yourself. Just have your controller accept a String and then use Jackson's ObjectMapper along with TypeReference to get a map.
#RequestMapping(value="/newPost", method = RequestMethod.POST, produces="application/json", consumes = "application/json")
#ResponseBody
public ResponseEntity newPost(#RequestBody String body) throws Exception {
ObjectMapper mapper = new ObjectMapper();
TypeReference<HashMap<String,Object>> typeRef = new TypeReference<HashMap<String,Object>>() {};
HashMap<String,Object> map = mapper.readValue(body, typeRef);
}
The resulting HashMap will use an ArrayList for the tag list:
You can create a custom Java POJO for the request that uses String[] versus List<String>. Here I did it for you using the site jsonschema2pojo.
package com.stackoverflow.question;
import com.fasterxml.jackson.annotation.*;
import java.util.HashMap;
import java.util.Map;
#JsonInclude(JsonInclude.Include.NON_NULL)
#JsonPropertyOrder({
"userId",
"postBody",
"postTitle",
"created",
"tagList"
})
public class MyRequest {
#JsonProperty("userId")
private int userId;
#JsonProperty("postBody")
private String postBody;
#JsonProperty("postTitle")
private String postTitle;
#JsonProperty("created")
private Object created;
#JsonProperty("tagList")
private String[] tagList = null;
#JsonIgnore
private Map<String, Object> additionalProperties = new HashMap<String, Object>();
#JsonProperty("userId")
public int getUserId() {
return userId;
}
#JsonProperty("userId")
public void setUserId(int userId) {
this.userId = userId;
}
#JsonProperty("postBody")
public String getPostBody() {
return postBody;
}
#JsonProperty("postBody")
public void setPostBody(String postBody) {
this.postBody = postBody;
}
#JsonProperty("postTitle")
public String getPostTitle() {
return postTitle;
}
#JsonProperty("postTitle")
public void setPostTitle(String postTitle) {
this.postTitle = postTitle;
}
#JsonProperty("created")
public Object getCreated() {
return created;
}
#JsonProperty("created")
public void setCreated(Object created) {
this.created = created;
}
#JsonProperty("tagList")
public String[] getTagList() {
return tagList;
}
#JsonProperty("tagList")
public void setTagList(String[] tagList) {
this.tagList = tagList;
}
#JsonAnyGetter
public Map<String, Object> getAdditionalProperties() {
return this.additionalProperties;
}
#JsonAnySetter
public void setAdditionalProperty(String name, Object value) {
this.additionalProperties.put(name, value);
}
}

java cassandra object mapping annotations

Can you provide an example for mapping collections using datastax api annotations to use Map.
class pojo {
#PartitionKey(value = 0)
#Column(name = "user_id")
String userId;
#Column(name = "attributes")
// How to map it
Map<String, String> attributes;
}
error log :
2015-08-03 16:33:34,568 INFO com.jpma.jpmc.slot.persistance.DAOFactory main - Cassandra Cluster Details: ConnectionCfg [userName=test, password=test, port=9042, seeds=[Ljava.lang.String;#1a85bd75, keySpace=test]
java.lang.Class
2015-08-03 16:33:34,646 DEBUG com.datastax.driver.mapping.EntityMapper main - Preparing query INSERT INTO "test"."user_event_date"("user_id","entry_date","entry_time","app","attributes","user_ip","user_authschemes") VALUES (?,?,?,?,?,?,?);
com.datastax.driver.core.exceptions.InvalidQueryException: Unknown identifier attributes
Based on the error message you are seeing, I'm guessing that attributes is not defined in your table definition. Would you mind editing your post with that?
But when I build my CQL table like this (note the compound partition key of itemid and version):
CREATE TABLE products.itemmaster (
itemid text,
version int,
productid uuid,
supplierskumap map<uuid, text>,
PRIMARY KEY ((itemid,version), productid)
);
...insert this row:
INSERT INTO products.itemmaster (itemid,version,productid,supplierskumap)
VALUES ('item1',1,26893749-dcfc-42c7-892c-bee8c9cff630,
{1351f82f-5dc5-4328-82f4-962429c92a2b:'86CCG123'});
...and I build my POJO like this:
#Table(keyspace = "products", name = "itemmaster")
public class Product {
#PartitionKey(0)
private String itemid;
#PartitionKey(1)
private int version;
#ClusteringColumn
private UUID productid;
#Column(name="supplierskumap")
private Map<UUID,String> suppliersku;
public UUID getProductid() {
return productid;
}
public void setProductid(UUID _productid) {
this.productid = _productid;
}
public int getVersion() {
return this.version;
}
public void setVersion(int _version)
{
this.version = _version;
}
public String getItemid() {
return itemid;
}
public void setItemid(String _itemid) {
this.itemid = _itemid;
}
public Map<UUID, String> getSuppliersku() {
return suppliersku;
}
public void setSuppliersku(Map<UUID, String> _suppliersku) {
this.suppliersku = _suppliersku;
}
}
...with this constructor and getProd method on my data access object (dao):
public ProductsDAO()
{
session = connect(CASSANDRA_NODES, USERNAME, PASSWORD);
prodMapper = new MappingManager(session).mapper(Product.class);
}
public Product getProd(String itemid, int version, UUID productid) {
return prodMapper.get(itemid,version,sku);
}
...then this main class successfully queries my table and maps my Map:
private static void main(String[] args) {
ProductsDAO dao = new ProductsDAO();
Product prod = dao.getProd("item1", 1, UUID.fromString("26893749-dcfc-42c7-892c-bee8c9cff630"));
System.out.println(
prod.getProductid() + " - " +
prod.getItemid() + " - " +
prod.getSuppliersku().get(UUID.fromString("1351f82f-5dc5-4328-82f4-962429c92a2b")));
dao.closeCassandra();
}
...and produces this output:
26893749-dcfc-42c7-892c-bee8c9cff630 - item1 - 86CCG123
NOTE: Edit made to the above example to support a compound partition key.

Categories