Kafka Consumer stuck at deserialization - java

I created a simple producer-consumer app that using a custom Serializer and Deserializer.
After adding a new method to the Message class that I produce, the consumer started being stack at deserialization. My producer is using the new class (with the new method) and the consumer is using the old class (without the method).
I didn't add any new data members to the message that is sent!
I have multiple class messages that all are based on the following base class:
public enum MessageType{
A,
B,
C,
}
public class Message implements Serializable {
protected long id;
protected MessageType type;
public Message(int id , MessageType type) {
this.id=id;
this.type=type;
}
Each one of the messages classes add some data members that are relevant to that message type.
For example:
public class MessageA extends Message{
private String name;
public MessageA(int id, String name) {
super(id,MessageType.A);
this.name = name;
}
My Serializer:
public class MessageSerializer implements Serializer<Message> {
#Override
public byte[] serialize(String topic, Message msg) {
return SerializationUtils.serialize(msg);
}
#Override
public void configure(Map<String, ?> configs, boolean isKey) {}
#Override
public void close() {}
}
My Deserializer :
public class MessageDeserializer implements Deserializer<Message> {
#Override
public void configure(Map<String, ?> configs, boolean isKey) {}
#Override
public Message deserialize(String topic, byte[] data) {
if(data == null)
return null;
log.info("Deserializing msg");
Message msg= (Message) SerializationUtils.deserialize(data);
if(msg== null){
log.warn("Deserialization failed, got null as a result.");
return null;
}
log.info("Deserialization complete");
return msg;
}
#Override
public void close() {}
}
I added a new method to MessageC class which is a subclass of Message. The new class version is available only on the producer and not on the consumer. Although I didn't change the schema of the class, can that change cause a problem in the deserialization in the consumer?
After producing a MessageC message, my consumer printed "Deserializing msg" but it is stuck/failed since it didn't print any exception or "Deserialization complete".
Can JSONSerializer/Deserializer handles those type of fixes? If I'll use JSONSerialzier it should care only regarding the schema of the class, right?

If you use JsonSerializer it should not be affected by changes to the methods - only changes to the data fields. Other serializers can serialize the whole object including the methods.

As #Menelaos suggested, the solution was to use JsonSerializer. You can find an implementation in here .

Related

Multiple channel Notification system - LLD -

You have to design the classes for building a notification system that supports multiple channels such as email, SMS, Whatsapp. It should be easily extensible.
My design :
class Message {
NotificationType type ; //email, sms
msgId;
String content ;
}
MessagingServiceImpl {
static {
//map from notification type to the respective handler
map.put("SMS",new SMSHandler());
map.put("Email",new EmailHandler();
}
void processMessage(Message message) {
Handler handler = map.get(message.getNotificationType();
handler.handleMessage();
}
}
public abstract class Handler {
public abstract void handle(Mesage message) ;
}
public EmailHandler extends Handler {
public void handle(Message message) {
System.out.println("Sending email"): // similar class for phone.
}
Note: This design was rejected in the interview.
Questions:
Should we make Message abstract - EmailMessage,SMSMessage etc ?
Note that the content of the message varies depending on the channel. For eg. In email,the content will be large. For SMS,it is much smaller. So,should Message be astract?
How can you support a new channel eg..telegram messages with minimal changes?
Of course I'm not in the mind of your reviewer, but he could argue about you breaking the "O" of the "SOLID" principles, which is the Open-Closed principle and states that:
"software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification".
The problem with your solution is that if you need to support a new channel you need to add another instance to the NotificationType Enum.
Let's start from the interfaces:
public interface Message {
//no need to define any methods here
}
public interface MessageHandler {
void handleMessage(Message message);
Class<?> getSupportedMessageType();
}
public interface MessageService {
void processMessage(Message message);
}
It is already possible to define a definitive version of the MessageServiceImpl class which will not need to change each time we need to support a new channel.
public class MessageServiceImpl implements MessageService
{
private final Map<Class<?>, MessageHandler> handlers = new HashMap<>();
public MessageServiceImpl(ICollection<MessageHandler> handlers) {
foreach(MessageHandler handler : handlers) {
this.handlers.put(handler.getSupportedMessageType(), handler);
}
}
public void processMessage(Message message) {
if (!this.handlers.containsKey(message.getType())) throw new InvalidArgumentException();
this.handlers.get(message.getType()).handleMessage(message);
}
With the help of a dependency injection framework it might be also possible to let DI handle new message handlers implementations.
Then we can define an abstract class to abstract the message handler:
public abstract class AbsMessageHandler<T implements Message> implements MessageHandler {
private final Class<T> supportedMessageType;
protected AbsMessageHandler(Class<T> supportedMessageType) {
this.supportedMessageType = supportedMessageType;
}
protected abstract void handleMessageInternal(T message);
public Class<?> getSupportedMessageType() { return supportedMessageType; }
public void handleMessage(Message message)
{
if (message == null) throw new InvalidArgumentException();
if (message.getType() != getSupportedMessageType()) throw new InvalidArgumentException();
handleMessageInternal((T) message);
}
}
The only thing left to define are the messages and message handlers implementations:
public EmailMessage implements Message {
private string from;
private string to;
private string cc;
private string subject;
private string messageBody;
//getters and setters
}
public EmailMessageHandler extends AbsMessageHandler<EmailMessage> {
public EmailMessage() {
super(EmailMessage.class);
}
protected void handleMessageInternal(EmailMessage message) {
// do what you like
}
}
Each time you need to add support for a new channel you only need to add a new couple of classes, one to implement the Message interface and one to extend the AbsMessageHandler abstract class.

What's a good way to store class properties and load at run time?

I have a messaging producer (RabbitMQ) and depending on what kind of message i have to send, i need to change the routing key and exchange at runtime.
Given this i'd implemented a strategy to load each class with specific properties, but it's not appear a good solution.
For example:
public class MyProducerStrategy1 extends RabbitMessagingProducer {
private static final String ROUTING_KEY = "order1";
private static final String EXCHANGE = "myexchange1";
#Override
String getRoutingKey() {
return ROUTING_KEY;
}
#Override
String getExchange() {
return EXCHANGE;
}
#Override
public void sendMessage(Message message) {
super.sendMessage(message);
}
}
public class MyProducerStrategy2 extends RabbitMessagingProducer {
private static final String ROUTING_KEY = "fullfilment";
private static final String EXCHANGE = "myexchange2";
#Override
String getRoutingKey() {
return ROUTING_KEY;
}
#Override
String getExchange() {
return EXCHANGE;
}
#Override
public void sendMessage(Message message) {
super.sendMessage(message);
}
}
public abstract class RabbitMessagingProducer implements MessagingProducerStrategy {
#Autowired
private RabbitTemplate rabbitTemplate;
abstract String getRoutingKey();
abstract String getExchange();
#Override
public void sendMessage(Message message) {
rabbitTemplate.convertAndSend(getExchange(), getRoutingKey(), message);
}
}
Does it make sense? or there's another approach to load there properties and have maybe one class?
Do not create multiple class for this scenario.
load the files from a property file based on the message.
Another option is to create a static map of values with message types as the key and routing key as the value. Fetch the values from the map based on the message type.

How to serialize the custom object you are sending in JMS template Convert and Send method

I am using the below mentioned code to publish into a topic. It is of the format convertAndSend(Destination destination, Object message)
Event event;
jmsTemplate.convertAndSend(topic, event);
My current Event interface is something like this,
public interface Event {
public boolean isEmpty();
public AcEventDatafileTransaction getDatafileTransaction();
public AcEventObjectTransaction getObjectTransaction();
boolean isDatafileTransaction();
boolean isObjectTransaction();
boolean isRdbmsTransaction();
String getTransactionId();
}
Its implementation is something like this,
public class EventPublisherImpl implements Event {
private final AcTransactionRecord transactionRecord;
private final Ac ac;
private final String[] actualTemplates;
private final String[] curveTemplates;
public AcEventPublisherImpl(final Ac ac,
final String[] actualTemplates,
final String[] curveTemplates,
final AcTransactionRecord acTransactionRecord) {
this.ac = ac;
this.transactionRecord = acTransactionRecord;
this.actualTemplates = actualTemplates;
this.curveTemplates = curveTemplates;
}
#Override
public boolean isEmpty() {
return transactionRecord.isEmpty();
}
#Override
public AcEventDatafileTransaction getDatafileTransaction() {
if (isDatafileTransaction()) {
return new AcEventDatafileTransactionPublisherImpl(transactionRecord.getDatafileTransaction());
}
return null;
}
#Override
public AcEventObjectTransaction getObjectTransaction() {
if (isObjectTransaction()) {
return new AcEventObjectTransactionPublisherImpl(ac, actualTemplates, curveTemplates, transactionRecord.getObjectOperations());
}
return null;
}
#Override
public boolean isDatafileTransaction() {
return transactionRecord.getType() == AcTransactionRecord.DATAFILE_TRANS;
}
#Override
public boolean isObjectTransaction() {
return transactionRecord.getType() == AcTransactionRecord.OBJECT_TRANS;
}
#Override
public boolean isRdbmsTransaction() {
return transactionRecord.getType() == AcTransactionRecord.RDBMS_TRANS;
}
#Override
public String getTransactionId() {
if (transactionRecord != null) {
return Integer.toString(transactionRecord.getNumber());
}
return "";
}
}
How do I convert it into a serializable form?
My whole objective is to publish Event object into topic. For that I am using convertAndSend method. That method requires an object of a serialized class, which I currently dont have. It is what I am trying to achieve.
I have looked at examples where a class with few attributes is serialized using message converter https://examples.javacodegeeks.com/enterprise-java/spring/spring-framework-jmstemplate-example/
Please suggest a method through code or point to the appropriate example and how that helps me.
The overloaded methods convertAndSend() and receiveAndConvert() in JmsTemplate delegate the conversion process to an instance of the MessageConverter interface. This interface defines a simple contract to convert between Java objects and JMS messages. The default implementation SimpleMessageConverter supports conversion between String and TextMessage, byte[] and BytesMesssage, and java.util.Map and MapMessage.
So, to support custom type we have to create custom converter which implements MessageConverter to convert Event object to one of supported message type. The simplest solution is to register MappingJackson2MessageConverter provided by spring which support converting object to TextMessage in form of json.
#Bean // Serialize message content to json using TextMessage
public MessageConverter jacksonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("_type");
return converter;
}
You can take a look how the serialization process is done on the source code in case you want to create your own converter.
To receive the message, the easiest way is to use #JmsListener annotation, spring will implicitly convert the message to its java type using the registered converter
#Component
public class Receiver {
#JmsListener(destination = "dest")
public void receiveMessage(Event event) {
// do whatever you need with the event
}
}
another way is to use javax.jms.MessageListener but you need to manually convert the message yourself.
#Component
public class ExampleListener implements MessageListener {
#Autowired
private ObjectMapper objectMapper;
public void onMessage(Message message) {
if (message instanceof TextMessage) {// we set the converter targetType to text
try {
String json = ((TextMessage) message).getText(); // here you have your event object as json string
Event event = objectMapper.readValue(json, Event.class); // convert back to event object
}
catch (JMSException ex) {
throw new RuntimeException(ex);
}
}
else {
throw new IllegalArgumentException("Message must be of type TextMessage");
}
}
Ref:
MappingJackson2MessageConverter
https://spring.io/guides/gs/messaging-jms/

spring state machine - manage long running processes

I need some advice for using spring state machine for long running processes. I want to design some flow. Let's say I have next states: Start->step1->step2->step3->finish. I have a controller that can send events to the state machine for managing transitions between states. I have a StateMachinePersister. I have a converter from StateMachineContext to byte[] and back. Sounds good for my business goal. So everything should work fine.
But I have a problem? I can't understand how to manage cases when I will decide to change the flow. I mean that if I have a Production environment where some processes persisted in "step2" state. But I am forced to change the flow. let say that I want to add a step or remove a step in the flow. I think I will have problems during state machine deserialization.
So the question is: may be spring state machine is not suitable for me, or there are some recipes how I can manage such cases?
I have an entity which I want to manage states, transitions etc.
#Entity
#Access(AccessType.FIELD)
#Table(name = "processes", indexes = #Index(columnList = "currentState"))
public class Process extends AbstractPersistable<Long> implements ContextEntity<ProcessState, ProcessEvent, Long> { // NOSONAR
private static final long serialVersionUID = 8848887579564649636L;
#JsonIgnore
StateMachineContext<ProcessState, ProcessEvent> stateMachineContext; // NOSONAR
#Enumerated(EnumType.STRING)
ProcessState currentState;
#Override
public void setStateMachineContext(StateMachineContext<ProcessState, ProcessEvent> stateMachineContext) {
if (stateMachineContext == null) {
throw new IllegalStateException("stateMachineContext can't be null");
}
this.currentState = stateMachineContext.getState();
this.stateMachineContext = stateMachineContext;
}
#Override
public StateMachineContext<ProcessState, ProcessEvent> getStateMachineContext() {
return stateMachineContext;
}
...
}
I have StateMachinePersist bean which is responsible for initializing stateMachineContext for particular process.
#Bean
public StateMachinePersist> persist() {
return new StateMachinePersist>() {
#Override
public StateMachineContext<ProcessState, ProcessEvent> read(
ContextEntity<ProcessState, ProcessEvent, Serializable> process) throws Exception {
return process.getStateMachineContext();
}
#Override
public void write(StateMachineContext<ProcessState, ProcessEvent> context,
ContextEntity<ProcessState, ProcessEvent, Serializable> process) throws Exception {
process.setStateMachineContext(context);
}
};
}
I have StateMachineAdapter which is responsible for persisting and restoring state machine
public class DefaultStateMachineAdapter<S, E, T> {
final StateMachineFactory<S, E> stateMachineFactory;
final StateMachinePersister<S, E, T> persister;
public DefaultStateMachineAdapter(StateMachineFactory<S, E> stateMachineFactory, StateMachinePersister<S, E, T> persister) {
this.stateMachineFactory = stateMachineFactory;
this.persister = persister;
}
public StateMachine<S, E> restore(T contextObject) throws Exception {
StateMachine<S, E> stateMachine = stateMachineFactory.getStateMachine();
return persister.restore(stateMachine, contextObject);
}
public void persist(StateMachine<S, E> stateMachine, T order) throws Exception {
persister.persist(stateMachine, order);
}
public StateMachine<S, E> create() {
StateMachine<S, E> stateMachine = stateMachineFactory.getStateMachine();
stateMachine.start();
return stateMachine;
}
}
I have StateMachineContextConverter which is responsible for serialization/deserialization of StateMachineContext. I have used Kryo for this operations.
public class StateMachineContextConverter implements AttributeConverter<StateMachineContext, byte[]> {
#Override
public byte[] convertToDatabaseColumn(StateMachineContext attribute) {
return serialize(attribute);
}
#Override
public StateMachineContext convertToEntityAttribute(byte[] dbData) {
return deserialize(dbData);
}
}
I have controller which is responsible to switch states
public class ProcessEventController {
final DefaultStateMachineAdapter<ProcessState, ProcessEvent, ContextEntity<ProcessState, ProcessEvent, ? extends Serializable>> processStateMachineAdapter;
public ProcessEventController(DefaultStateMachineAdapter<ProcessState, ProcessEvent, ContextEntity<ProcessState, ProcessEvent, ? extends Serializable>> processStateMachineAdapter) {
this.processStateMachineAdapter = processStateMachineAdapter;
}
#RequestMapping(path = "/processes/{id}/{event}", method = RequestMethod.POST)
#Transactional
public HttpEntity<Void> receiveEvent(#PathVariable("id") Process process, #PathVariable("event") ProcessEvent event) throws Exception {
StateMachine<ProcessState, ProcessEvent> stateMachine = processStateMachineAdapter.restore(process);
if (stateMachine.sendEvent(event)) {
processStateMachineAdapter.persist(stateMachine, process);
return ResponseEntity.accepted().build();
} else {
return ResponseEntity.unprocessableEntity().build();
}
}
}
Obviously you can't modify existing running machine but as you're already working with persistence I assume you're at least stopping machines.
Look statemachine-examples-datajpa which is using existing machine apis to store config in a DB. We have StateMachineModelFactory which pretty much allows you to store your stuff anywhere if any build-in implementation is not suitable for you. This sample will build new machine instances every time when new events are sent. i.e. you can go to a DB using build-in editor and add new transition(s) without restarting a main java process.
This gives you some flexibility but if you need to change your running code then things will get impossible.

How to design a type safe message API in Java?

I have a Java client which wants to communicate with a device through messages over serial communication. The client should be able to use a clean API, abstracting the ugly details of the serial communication. The client can send many types of messages through that API and gets responses. I'm searching for advice which way is best to implement this API.
For simplicity, say we have only two message types: HelloMessage which triggers a HelloResponse and InitMessage which triggers an InitResponse (in reality, there are many more)
Designing the API (that is, the Java abstraction of the device) I could have:
One method per message type:
public class DeviceAPI {
public HelloResponse sendHello(HelloMessage){...}
public InitResponse sendInit(InitMessage){...}
... and many more message types ....
This is nicely type safe. (It could also be many times the same send() method, overloaded, but that's about the same). But it is very explicit, and not very flexible - we cannot add messages without modification of the API.
I could also have a single send method, which takes all message types:
class HelloMessage implements Message
class HelloResponse implements Response
...
public class DeviceAPI {
public Response send(Message msg){
if(msg instanceof HelloMessage){
// do the sending, get the response
return theHelloResponse
} else if(msg instanceof ...
This simplifies the API (only one method) and allows for additional Message types to be added later without changing the API. At the same time, it requires the Client to check the Response type and cast it to the right type.
Client code:
DeviceAPI api = new DeviceAPI();
HelloMessage msg = new HelloMessage();
Response rsp = api.send(msg);
if(rsp instanceOf HelloResponse){
HelloResponse hrsp = (HelloResponse)rsp;
... do stuff ...
This is ugly in my opinion.
What do you recommend? Are there other approaches which give cleaner results?
References welcome! How did others solve this?
Here is a way to do it in type-safe (and extensible) way using generics:
public interface MessageType {
public static final class HELLO implements MessageType {};
}
public interface Message<T extends MessageType> {
Class<T> getTypeClass();
}
public interface Response<T extends MessageType> {
}
public class HelloMessage implements Message<MessageType.HELLO> {
private final String name;
public HelloMessage(final String name) {
this.name = name;
}
#Override
public Class<MessageType.HELLO> getTypeClass() {
return MessageType.HELLO.class;
}
public String getName() {
return name;
}
}
public class HelloResponse implements Response<MessageType.HELLO> {
private final String name;
public HelloResponse(final String name) {
this.name = name;
}
public String getGreeting() {
return "hello " + name;
}
}
public interface MessageHandler<T extends MessageType, M extends Message<T>, R extends Response<T>> {
R handle(M message);
}
public class HelloMessageHandler
implements MessageHandler<MessageType.HELLO, HelloMessage, HelloResponse> {
#Override
public HelloResponse handle(final HelloMessage message) {
return new HelloResponse(message.getName());
}
}
import java.util.HashMap;
import java.util.Map;
public class Device {
#SuppressWarnings("rawtypes")
private final Map<Class<? extends MessageType>, MessageHandler> handlers =
new HashMap<Class<? extends MessageType>, MessageHandler>();
public <T extends MessageType, M extends Message<T>, R extends Response<T>>
void registerHandler(
final Class<T> messageTypeCls, final MessageHandler<T, M, R> handler) {
handlers.put(messageTypeCls, handler);
}
#SuppressWarnings("unchecked")
private <T extends MessageType, M extends Message<T>, R extends Response<T>>
MessageHandler<T, M, R> getHandler(final Class<T> messageTypeCls) {
return handlers.get(messageTypeCls);
}
public <T extends MessageType, M extends Message<T>, R extends Response<T>>
R send(final M message) {
MessageHandler<T, M, R> handler = getHandler(message.getTypeClass());
R resposnse = handler.handle(message);
return resposnse;
}
}
public class Main {
public static void main(final String[] args) {
Device device = new Device();
HelloMessageHandler helloMessageHandler = new HelloMessageHandler();
device.registerHandler(MessageType.HELLO.class, helloMessageHandler);
HelloMessage helloMessage = new HelloMessage("abhinav");
HelloResponse helloResponse = device.send(helloMessage);
System.out.println(helloResponse.getGreeting());
}
}
To add support for a new message type, implement MessageType interface to create a new message type, implement Message, Response and MessageHandler interfaces for the new MessageType class and register the handler for the new message type by calling Device.registerHandler.
I've got a fully working example now of what you want:
To define the types of messages:
public interface MessageType {
public static class INIT implements MessageType { }
public static class HELLO implements MessageType { }
}
Base Message and Response classes:
public class Message<T extends MessageType> {
}
public class Response<T extends MessageType> {
}
Create custom init messages and responses:
public class InitMessage extends Message<MessageType.INIT> {
public InitMessage() {
super();
}
public String getInit() {
return "init";
}
}
public class InitResponse extends Response<MessageType.INIT> {
public InitResponse() {
super();
}
public String getInit() {
return "init";
}
}
Create custom hello messages and responses:
public class HelloMessage extends Message<MessageType.HELLO> {
public HelloMessage() {
super();
}
public String getHello() {
return "hello";
}
}
public class HelloResponse extends Response<MessageType.HELLO> {
public HelloResponse() {
super();
}
public String getHello() {
return "hello";
}
}
The DeviceAPI:
public class DeviceAPI {
public <T extends MessageType, R extends Response<T>, M extends Message<T>> R send(M message) {
if (message instanceof InitMessage) {
InitMessage initMessage = (InitMessage)message;
System.out.println("api: " + initMessage.getInit());
return (R)(new InitResponse());
}
else if (message instanceof HelloMessage) {
HelloMessage helloMessage = (HelloMessage)message;
System.out.println("api: " + helloMessage.getHello());
return (R)(new HelloResponse());
}
else {
throw new IllegalArgumentException();
}
}
}
Note that it does require an instanceof-tree, but you need that to handle what kind of message it is.
And a working example:
public static void main(String[] args) {
DeviceAPI api = new DeviceAPI();
InitMessage initMsg = new InitMessage();
InitResponse initResponse = api.send(initMsg);
System.out.println("client: " + initResponse.getInit());
HelloMessage helloMsg = new HelloMessage();
HelloResponse helloResponse = api.send(helloMsg);
System.out.println("client: " + helloResponse.getHello());
}
Output:
api: init
client: init
api: hello
client: hello
UPDATE: Added example on how to get input from the messages the client wants to send.
You could have a system of message handlers, and your DeviceAPI could choose which handler is suitable for the incoming message; and delegate it to the appropriate message handler:
class DeviceAPI {
private List<Handler> msgHandlers = new ArrayList<Handler>();
public DeviceAPI(){
msgHandlers.add(new HelloHandler());
//Your other message handlers can be added
}
public Response send(Message msg) throws Exception{
for (Handler handler : msgHandlers) {
if (handler.canHandle(msg)){
return handler.handle(msg);
}
}
throw new Exception("No message handler defined for " + msg);
}
}
The HelloHandler would look like:
interface Handler<T extends Message, U extends Response> {
boolean canHandle(Message message);
U handle(T message);
}
class HelloHandler implements Handler<HelloMessage, HelloResponse> {
#Override
public boolean canHandle(Message message) {
return message instanceof HelloMessage;
}
#Override
public HelloResponse handle(HelloMessage message) {
//Process your message
return null;
}
}
Ditto for your other messages. I'm sure you could make it more elegant, but the idea still remains the same - donot have one monster method with ifs; instead use polymorphism.
I don't think this is ugly at all:
if(rsp instanceOf HelloResponse){
HelloResponse hrsp = (HelloResponse)rsp;
...
else if ...
as long as you don't have like 100 different responses. You can encapsulate many kind of responses in one, depending on the data they have. For example:
class GenericResponse{
private String message;
private int responseType;
...
}
I have developed some multiplayer games and this is a good way to do it.
In case you have too many different types of messages, you just can use generic java types, like the example of skiwi above.
hope it helps

Categories