We Keep Coding

Kafka Custom Deserializer

So far, I've been able to create a KStream with the help of a topic.
KStream<String, Object> testqa2 = builder.stream("testqa2", Consumed.with(Serdes.String(), Serdes.String()))
.mapValues(value -> {
System.out.println(value);
return value;
});
It doesn't print anything, so on debbuging - I realized I am just creating my KStream. There is no data in it.
I am having a litte trouble creating serializer/deserializer for worker class.
package com.copart.mwa.Avro;
public class Worker {
private static String WorkerActivityName;
private static String WorkerSid;
private static String WorkerPreviousActivityName;
private static String WorkerPreviousActivitySid;
public String getWorkerActivityName() {
return WorkerActivityName;
}
public void setWorkerActivityName(String workerActivityName) {
WorkerActivityName = workerActivityName;
}
public static String getWorkerSid() {
return WorkerSid;
}
public void setWorkerSid(String workerSid) {
WorkerSid = workerSid;
}
public String getWorkerPreviousActivityName() {
return WorkerPreviousActivityName;
}
public void setWorkerPreviousActivityName(String workerPreviousActivityName) {
WorkerPreviousActivityName = workerPreviousActivityName;
}
public String getWorkerPreviousActivitySid() {
return WorkerPreviousActivitySid;
}
public void setWorkerPreviousActivitySid(String workerPreviousActivitySid) {
WorkerPreviousActivitySid = workerPreviousActivitySid;
}
#Override
public String toString() {
return "Worker(" + WorkerSid + ", " + WorkerActivityName + ")";
} }
And the message from the producer to the consumer is a JSON
{
"WorkerActivityName": "Available",
"EventType": "worker.activity.update",
"ResourceType": "worker",
"WorkerTimeInPreviousActivityMs": "237",
"Timestamp": "1626114642",
"WorkerActivitySid": "WAc9030ef021bc1786d3ae11544f4d9883",
"WorkerPreviousActivitySid": "WAf4feb231e97c1878fecc58b26fdb95f3",
"WorkerTimeInPreviousActivity": "0",
"AccountSid": "AC8c5cd8c9ba538090da104b26d68a12ec",
"WorkerName": "Dorothy.Finegan#Copart.Com",
"Sid": "EV284c8a8bc27480e40865263f0b42e5cf",
"TimestampMs": "1626114642204",
"P": "WKe638256376188fab2a98cccb3c803d67",
"WorkspaceSid": "WS38b10d521442ecb74fcc263d5a4d726e",
"WorkspaceName": "Copart-MiPhone",
"WorkerPreviousActivityName": "Unavailable(RNA)",
"EventDescription": "Worker Dorothy.Finegan#Copart.Com updated to Available Activity",
"ResourceSid": "WKe638256376188fab2a98cccb3c803d67",
"WorkerAttributes": "{\"miphone_dept\":[\"USA_YRD_OPS\"],\"languages\":[\"en\"],\"home_region\":\"GL\",\"roles\":[\"supervisor\"],\"miphone_yards\":[\"81\"],\"miphone_enabled\":true,\"miphone_states\":[\"IL\"],\"home_state\":\"IL\",\"skills\":[\"YD_SELLER\",\"YD_TITLE\"],\"home_division\":\"Northern\",\"miphone_divisions\":[\"Northern\"],\"miphone_functions\":[\"outbound_only\"],\"full_name\":\"Dorothy Finegan\",\"miphone_regions\":[\"GL\"],\"home_country\":\"USA\",\"copart_user_id\":\"USA3204\",\"home_yard\":\"81\",\"home_dept\":\"USA_YRD_OPS\",\"email\":\"dorothy.finegan#copart.com\",\"home_dept_category\":\"OPS\",\"contact_uri\":\"client:Dorothy_2EFinegan_40Copart_2ECom\",\"queue_activity\":\"Available\",\"teams\":[],\"remote_employee\":false,\"miphone_call_center_units\":[\"USA_YRD_OPS|81\"],\"miphone_call_center_teams\":[]}"
}
I want to implemenet a customer deserializer where
"WorkspaceSid": "WS38b10d521442ecb74fcc263d5a4d726e", is the key and the remaining values of the other attributes act as the value for the key-value pair.
Thanks,
Anmol

It doesn't print anything
If there is data in testqa2 topic, and you have auto.offset.reset=earliest, then it should.
having a litte trouble creating serializer/deserializer for worker class
Kafka has bulit-in JSON serializers that you can build a Serde for. You don't need to make your own.
"WorkspaceSid", is the key
use selectKey, or map if you want to modify the key, not mapValues
Serializer<JsonNode> jsonNodeSerializer = new JsonSerializer();
Deserializer<JsonNode> jsonNodeDeserializer = new JsonDeserializer();
final Serde<JsonNode> jsonNodeSerde = Serdes.serdeFrom(jsonNodeSerializer,jsonNodeDeserializer);
KStream<String, JsonNode> testqa2 = builder.stream("testqa2", Consumed.with(Serdes.String(), jsonSerde))
.selectKey((k, json) -> json.get("WorkspaceSid"))
.print(Printed.toSysOut());
Alternatively, fix your producer code to get the Sid from the value, and set the key there...
If you want to use Avro, you wouldn't write a Worker class - you would generate it from an Avro schema.

Jackson JSON key as value in Java

I'm using Jackson in Spring MVC application. I want to use a String value as key name for Java POJO --> JSON
"record": {
"<Dynamic record name String>": {
"value": {
....
}
}
}
So the dynamic record name String could be "abcd","xyz" or any other string value. How can I define my "record" POJO to have a key like that ?

Unfortunately, you cannot have dynamic fields in Java classes (unlike some other languages), so you have two choices:
Using Maps
Using JSON objects (i.e. JsonNode in case of Jackson)
Suppose, you have a data like this:
{
"record": {
"jon-skeet": {
"name": "Jon Skeet",
"rep": 982706
},
"darin-dimitrov": {
"name": "Darin Dimitrov",
"rep": 762173
},
"novice-user": {
"name": "Novice User",
"rep": 766
}
}
}
Create two classes to capture it, one for user and another for the object itself:
User.java:
public class User {
private String name;
private Long rep;
public String getName() { return name; }
public void setName(String name) { this.name = name; }
public Long getRep() { return rep; }
public void setRep(Long rep) { this.rep = rep; }
#Override
public String toString() {
return "User{" +
"name='" + name + '\'' +
", rep=" + rep +
'}';
}
}
Data.java:
public class Data {
private Map<String, User> record;
public Map<String, User> getRecord() { return record; }
public void setRecord(Map<String, User> record) { this.record = record; }
#Override
public String toString() {
return "Data{" +
"record=" + record +
'}';
}
}
Now, parse the JSON (I assume there is a data.json file in the root of your classpath):
public class App {
public static void main(String[] args) throws Exception {
final ObjectMapper objectMapper = new ObjectMapper();
System.out.println(objectMapper.readValue(App.class.getResourceAsStream("/data.json"), Data.class));
System.out.println(objectMapper.readTree(App.class.getResourceAsStream("/data.json")));
}
}
This will output:
Data{record={jon-skeet=User{name='Jon Skeet', rep=982706}, darin-dimitrov=User{name='Darin Dimitrov', rep=762173}, novice-user=User{name='Novice User', rep=766}}}
{"record":{"jon-skeet":{"name":"Jon Skeet","rep":982706},"darin-dimitrov":{"name":"Darin Dimitrov","rep":762173},"novice-user":{"name":"Novice User","rep":766}}}
In case of a Map you can use some static classes, like User in this case, or go completely dynamic by using Maps of Maps (Map<String, Map<String, ...>>. However, if you find yourself using too much maps, consider switching to JsonNodes. Basically, they are the same as Map and "invented" specifically for highly dynamic data. Though, you'll have some hard time working with them later...
Take a look at a complete example, I've prepared for you here.

This is in Kotlin but I have found a solution to the same problem using Jackson.
You don't need the root node "record", so you will need to get rid of it or start one node deeper(you're on your own there) but to turn the list of records that are children of their id into a list of records with id in the object follows:
val node = ObjectMapper().reader().readTree(json)
val recordList = mutableListOf<Record>()
node.fields().iterator().forEach {
val record = record(
it.key,
it.value.get("name").asText(),
it.value.get("rep").asText()
)
recordList.add(event)
}
node.fields() returns a map of children(also maps)
iterating through the parent map you will get the id from the key and then the nested data is in the value (which is another map)
each child of fields is key : value where
key = record id
value = nested data (map)
This solution, you don't need multiple classes to deserialize a list of classes.

I have my data in this format:
{
"0" : {"a": {}}, {"b": {}}, ...
"1" : {"c": {}}, {"d": {}}, ...
.
.
.
}
I am able to capture it into a map using the dynamic capture feature of jackson by using #JsonAnySetter annotation.
public class Destination{
Map<String, Object> destination = new LinkedHashMap<>();
#JsonAnySetter
void setDestination(String key, Object value) {
destination.put(key, value);
}
}

How to transform a Java object into JSON tree format with jackson

I want to use Jackson to convert a Java object into JSON format. I have a class which looks pretty much the following structure
public Class Event
{
String type;
String timestamp;
String hostname;
String service;
Payload payload;
}
I have the getters and setters for the above fields and also the getters/setters in Payload class.
Here is the json format, i want
{
"type":"end",
"time":"2016-08-01 11:11:11:111",
"origin":{
"hostname":"<hostname>",
"service":"<service>"
},
"version":"1.0"
"data":{ .... }
}
I can't seem to find a jackson way to get the above format, don't know how to put the whole payload object in "data" node and how to put the hostname, service in the "origin" node.

from your question, this is one approach that should showcase on how to solve it. Since you only posted 1 class, I am changing the payload to be a map. It works the same way with other classes as well.
Consider this example:
public class JacksonTest {
public static void main(String[] args) throws JsonProcessingException {
Event e = new Event();
e.type="end";
e.service="<service>";
e.hostname = "<hostname>";
e.timestamp = LocalDateTime.now().toString();
Map<String,String> payload = new HashMap<>();
payload.put("param1", "xyz");
e.payload = payload;
String writeValueAsString = new ObjectMapper().writeValueAsString(e);
System.out.println(writeValueAsString);
}
public static class Event {
#JsonProperty
String type;
#JsonProperty("time")
String timestamp;
#JsonIgnore
String hostname;
#JsonIgnore
String service;
#JsonProperty("data")
Map<String, String> payload;
#JsonProperty("origin")
Map<String,String> getOrigin() {
Map<String,String> tmp = new HashMap<>();
tmp.put("hostname", hostname);
tmp.put("service", service);
return tmp;
}
#JsonProperty("version")
private String getVersion() {
return "1.0";
}
}
}
I annotate the Event class with the necessary properties I want and the names they should have. Since you want the hostname and service to be in a nested setting and not create a new object for it (a new object would be easier as you could just have that serialised), I ignore those and instead use a getter to create the necessary structure as a map.
The output is:
{
"type":"end",
"time":"2016-08-19T16:45:18.072",
"data":{"param1":"xyz"},
"origin":{
"hostname":"<hostname>",
"service":"<service>"
},
"version":"1.0"
}
Regads,
Artur

Flattening Java Bean to a Map

I am stuck at converting Java Bean to Map. There are many resources on the internet, but unfortunately they all treat converting simple beans to Maps. My ones are a little bit more extensive.
There's simplified example:
public class MyBean {
private String firstName;
private String lastName;
private MyHomeAddress homeAddress;
private int age;
// getters & setters
}
My point is to produce Map<String, Object> which, in this case, is true for following conditions:
map.containsKey("firstName")
map.containsKey("lastName")
map.containsKey("homeAddress.street") // street is String
map.containsKey("homeAddress.number") // number is int
map.containsKey("homeAddress.city") // city is String
map.containsKey("homeAddress.zipcode") // zipcode is String
map.containsKey("age")
I have tried using Apache Commons BeanUtils. Both approaches BeanUtils#describe(Object) and BeanMap(Object) produce a Map which "deep level" is 1 (I mean that there's only "homeAddress" key, holding MyHomeAddress object as a value). My method should enter the objects deeper and deeper until it meets a primitive type (or String), then it should stop digging and insert key i.e. "order.customer.contactInfo.home".
So, my question is: how can it be easliy done (or is there already existing project which would allow me to do that)?
update
I have expanded Radiodef answer to include also Collections, Maps Arrays and Enums:
private static boolean isValue(Object value) {
final Class<?> clazz = value.getClass();
if (value == null ||
valueClasses.contains(clazz) ||
Collection.class.isAssignableFrom(clazz) ||
Map.class.isAssignableFrom(clazz) ||
value.getClass().isArray() ||
value.getClass().isEnum()) {
return true;
}
return false;
}

Here's a simple reflective/recursive example.
You should be aware that there are some issues with doing a conversion the way you've asked:
Map keys must be unique.
Java allows classes to name their private fields the same name as a private field owned by an inherited class.
This example doesn't address those because I'm not sure how you want to account for them (if you do). If your beans inherit from something other than Object, you will need to change your idea a little bit. This example only considers the fields of the subclass.
In other words, if you have
public class SubBean extends Bean {
this example will only return fields from SubBean.
Java lets us do this:
package com.acme.util;
public class Bean {
private int value;
}
package com.acme.misc;
public class Bean extends com.acme.util.Bean {
private int value;
}
Not that anybody should be doing that, but it's a problem if you want to use String as the keys, because there would be two keys named "value".
import java.lang.reflect.*;
import java.util.*;
public final class BeanFlattener {
private BeanFlattener() {}
public static Map<String, Object> deepToMap(Object bean) {
Map<String, Object> map = new LinkedHashMap<>();
try {
putValues(bean, map, null);
} catch (IllegalAccessException x) {
throw new IllegalArgumentException(x);
}
return map;
}
private static void putValues(Object bean,
Map<String, Object> map,
String prefix)
throws IllegalAccessException {
Class<?> cls = bean.getClass();
for (Field field : cls.getDeclaredFields()) {
if (field.isSynthetic() || Modifier.isStatic(field.getModifiers()))
continue;
field.setAccessible(true);
Object value = field.get(bean);
String key;
if (prefix == null) {
key = field.getName();
} else {
key = prefix + "." + field.getName();
}
if (isValue(value)) {
map.put(key, value);
} else {
putValues(value, map, key);
}
}
}
private static final Set<Class<?>> VALUE_CLASSES =
Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
Object.class, String.class, Boolean.class,
Character.class, Byte.class, Short.class,
Integer.class, Long.class, Float.class,
Double.class
// etc.
)));
private static boolean isValue(Object value) {
return value == null
|| value instanceof Enum<?>
|| VALUE_CLASSES.contains(value.getClass());
}
}

You could always use the Jackson Json Processor. Like this:
import com.fasterxml.jackson.databind.ObjectMapper;
//...
ObjectMapper objectMapper = new ObjectMapper();
//...
#SuppressWarnings("unchecked")
Map<String, Object> map = objectMapper.convertValue(pojo, Map.class);
where pojo is some Java bean. You can use some nice annotations on the bean to control the serialization.
You can re-use the ObjectMapper.

Using String to find Class in java?

I have made a class named Entity, and have the following code:
Entity zombie1 = new Entity();
I get input 'zombie' from a scanner, and then concatenate a number, based on level on the end of that, leaving 'zombie1' as the string... I want to be able to use that string and call
zombie1.shoot("shotgun");
but I can't seem to find a solution. I'd just do a if statement but I want to be able to create as many zombies as I want and not have to put in more if statements every single time.
I've read articles using reflection and forString but that doesn't seem to be what i'm looking for.
Any help would be nice.

Possible solutions are to use a Map<String, Entity> to be able to store and retrieve entities based on specific Strings. If you have a limited number of sub-types of Entity such as Zombies, Vampires, Victims, etc, you could have a Map<String, List<Entity>>, allowing you to map a String to a specific type of entity and then get that type by number.
e.g.,
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
public class Foo002 {
private static final String ZOMBIE = "zombie";
public static void main(String[] args) {
Map<String, List<Entity>> entityMap = new HashMap<String, List<Entity>>();
entityMap.put(ZOMBIE, new ArrayList<Entity>());
entityMap.get(ZOMBIE).add(new Entity(ZOMBIE, "John"));
entityMap.get(ZOMBIE).add(new Entity(ZOMBIE, "Fred"));
entityMap.get(ZOMBIE).add(new Entity(ZOMBIE, "Bill"));
for (Entity entity : entityMap.get(ZOMBIE)) {
System.out.println(entity);
}
}
}
class Entity {
private String type;
private String name;
public Entity(String type, String name) {
this.type = type;
this.name = name;
}
public String getType() {
return type;
}
public String getName() {
return name;
}
#Override
public String toString() {
return type + ": " + name;
}
}

This is not your best bet. Your best bet is to have a Map;
// PLEASE LOOK INTO WHICH MAP WOULD BE BEST FOR YOUR CASE OVERALL
// HASHMAP IS JUST AN EXAMPLE.
Map<String, Entity> zombieHoard = new HashMap<String, Entity>;
String getZombieID( int id )
{
return String.format( "zombie%s", id );
}
String createZombie() {
String zid = getZombieID( Map.size() );
Map.put( zid, new Entity() );
return zid;
}
void sendForthTheHoard() {
createZombie();
createZombie();
String currentZombie = createZombie();
zombieHoard.get( currentZombie ).shoot( "blow-dryer" );
zombieHoard.get( getZombieID( 1 ) ).eatBrains();
}

Put your zombies in an ArrayList. Example:
ArrayList<Entity> zombies = new ArrayList<Entity>();
Entity zombie1 = new Entity();
zombies.add(zombie1);
Entity zombie2 = new Entity();
zombies.add(zombie2);
etc...
Then when it is time to call a certain zombie to the following:
zombies.get(1).shoot("shotgun");

If you are talking about dynamically invoking a method on an object, you can use Reflection to get the method object and invoke it (Note: I may have inadvertantly mixed up some C# syntax in this Java):
Entity zombie1 = new Entity();
Method shootMethod = Entity.class.getMethod("shoot", new Class[] { string.class });
shootMethod.invoke(zombie1, new Object[] { "shotgun" });