I created a Springboot application to push message to a Kafka Topic. The application is working fine. What I am trying is to handle the exceptions when there is a failure while sending the messages to Kafka Topic. I am using the Error Channel to track the errors while sending a message. But the actual issue is, I am able to see the error message, but I am not able to see the Actual payload which got failed in the error message. Actually , I want to log that Payload.
The JSON Message that I am trying to send : {"key1":"value1"}
Service class :
#AllArgsConstructor
#EnableBinding(Source.class)
public class SendMessageToKafka {
private final Source source;
public void sendMessage(String sampleMessage) {
source.output.send(MessageBuilder.withPayLoad(sampleMessage).build());
}
#ServiceActivator(inputChannel = "errorChannel")
public void errorHandler(ErrorMessage em) {
System.out.println(em);
}
}
application.yml:
spring:
cloud:
stream:
bindings:
output:
producer:
error-channel-enabled: true
With the above configuration, when the Kafka server is down, the control is coming to the errorHandler method and printing the message. But I am not able to see the actual payload which is {"key1":"value1"} from the error message. How can I retrieve that from the error message?
You can filter the payload based on type, e.g. KafkaSendFailureException (https://docs.spring.io/spring-kafka/docs/2.2.0.RELEASE/reference/html/_spring_integration.html indicates the ErrorMessage payload is this type.)
After that what worked in my case is to cast it down to the original sent message as follows (analyze the objects via breakpoint to determine the appropriate value type, e.g. byte[]):
#ServiceActivator(inputChannel = "errorChannel")
public void errorHandler(ErrorMessage em) {
log.debug("got error message over errorChannel: {}", em);
if (null != em.getPayload() && em.getPayload() instanceof KafkaSendFailureException) {
KafkaSendFailureException kafkaSendFailureException = (KafkaSendFailureException) em.getPayload();
if (kafkaSendFailureException.getRecord() != null && kafkaSendFailureException.getRecord().value() != null
&& kafkaSendFailureException.getRecord().value() instanceof byte[]) {
log.warn("error channel message. Payload {}", new String((byte[])(kafkaSendFailureException.getRecord().value())));
}
}
}
Related
I have a spring boot application which will publish message on azure Queue. I have one more azure queueTrigger function written in Java which will listen to the same queue to which spring boot application has published a message. The queueTrigger function not able to detected messages published on queue.
Here is my publisher code
public static void addQueueMessage(String connectStr, String queueName, String message) {
try {
// Instantiate a QueueClient which will be
// used to create and manipulate the queue
QueueClient queueClient = new QueueClientBuilder()
.connectionString(connectStr)
.queueName(queueName)
.buildClient();
System.out.println("Adding message to the queue: " + message);
// Add a message to the queue
queueClient.sendMessage(message);
} catch (QueueStorageException e) {
// Output the exception message and stack trace
System.out.println(e.getMessage());
e.printStackTrace();
}
}
Here is my queueTrigger function app code
#FunctionName("queueprocessor")
public void run(
#QueueTrigger(name = "message",
queueName = "queuetest",
connection = "AzureWebJobsStorage") String message,
final ExecutionContext context
) {
context.getLogger().info(message);
}
I'm passing same connection-String and queueName, still doesn't work. If i run function on my local machine then it gets triggered but with error error image
As the official doc suggests,
Functions expect a base64 encoded string. Any adjustments to the encoding type (in order to prepare data as a base64 encoded string) need to be implemented in the calling service.
Update sender code to send base64 encoded message.
String encodedMsg = Base64.getEncoder().encodeToString(message.getBytes())
queueClient.sendMessage(encodedMsg);
I currently have 4 queues:
test-queue
test-queue-short-term-dead-letter
test-queue-long-term-dead-letter
test-queue-parking-lot
When a message comes into test-queue, I do a check to see if the message is in the correct format. If it isn't I want to send the message directly to the parking lot queue.
I can't use AmqpRejectAndDontRequeue() because it will automatically send the message to the configured DLQ (test-queue-short-term-dead-letter).
Using RabbitTemplate.convertAndSend() with another exception such as BadRequestException doesn't work. The message goes to the parking lot queue as expected, however the same message will stay in the test-queue
Using RabbitTemplate.convertAndSend() on it's own won't work as the program continues execution.
All queues are bound to a single direct exchange, each with unique routing keys. The test-queue is configured with the following arguments:
x-dead-letter-exchange: ""
x-dead-letter-routing-key: <shortTermDeadLetterKey>
Receiver:
#RabbitListener(queues = "test-queue")
public void receiveMessage(byte[] person) {
String personString = new String(person);
if (!personString.matches(desiredRegex)) {
rabbitTemplate.convertAndSend("test-exchange", "test-queue-parking-lot",
"invalid person");
log.info("Invalid person");
}
...some other code which I dont want to run as the message has arrived in the incorrect format
}
The problem was solved by manually acknowledging the message and returning from the method.
#RabbitListener(queues = "test-queue")
public void receiveMessage(byte[] person, Channel channel,
#Header(AmqpHeaders.DELIVERY_TAG) long tag) throws Exception) {
String personString = new String(person);
if (!personString.matches(desiredRegex)) {
rabbitTemplate.convertAndSend("test-exchange", "test-queue-parking-lot",
"invalid person");
log.info("Invalid person");
channel.basicAck(tag, false);
return;
}
...some other code which I dont want to run as the message has arrived in the incorrect format
}
I created Rabbitmq with x-message-ttl,x-dead-letter-exchange,x-dead-letter-routing-key. The queue name is tempQueue.
I integrated Rabitmq with JMSTemplate, when I sent the message to above-created queue got the error like.
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - inequivalent arg 'x-message-ttl' for queue 'tempQueue' in vhost '/': received none but current is the value '10000' of type 'long', class-id=50, method-id=10).
My code is :
public void sample(){
jmsTemplate.send("tempQueue", new MessageCreator() {
#Override
public Message createMessage(Session session) throws JMSException {
BytesMessage message = session.createBytesMessage();
try {
message.writeBytes("Welcome".getBytes());
} catch (Exception e) {
e.printStackTrace();
}
return message;
}
});
}
if I sent normally created queue, without any additional properties like x-message-ttl ,its working fine.
for RabbitTemplate both cases are working fine, but integrate with JMSTemplate extra arguements are not working..
I am currently using Spring Cloud Stream with Kafka binders with a GlobalChannelInterceptor to perform message-logging for my Spring Boot microservices.
I have:
a producer to publish messages to a SubscribableChannel
a consumer to listen from the Stream (using the #StreamListener annotation)
Throughout the process when a message is published to the Stream from the producer and listened by the consumer, it is observed that the preSend method was triggered twice:
Once at producer side - when the message is published to the Stream
Once at consumer side - when the message is listened from the Stream
However, for my logging purposes, I only need to intercept and log the message at consumer side.
Is there any way to intercept the SCS message ONLY at one side (e.g. consumer side)?
I would appreciate any thoughts on this matter. Thank you!
Ref:
GlobalChannelInterceptor documentation - https://docs.spring.io/spring-integration/api/org/springframework/integration/config/GlobalChannelInterceptor.html
EDIT
Producer
public void sendToPushStream(PushStreamMessage message) {
try {
boolean results = streamChannel.pushStream().send(MessageBuilder.withPayload(new ObjectMapper().writeValueAsString(message)).build());
log.info("Push stream message {} sent to {}.", results ? "successfully" : "not", StreamChannel.PUSH_STREAM);
} catch (JsonProcessingException ex) {
log.error("Unable to parse push stream message.", ex);
}
}
Producer's streamChannel
public interface StreamChannel {
String PUSH_STREAM = "PushStream";
#Output(StreamChannel.PUSH_STREAM)
SubscribableChannel pushStream();
}
Consumer
#StreamListener(StreamChannel.PUSH_STREAM)
public void handle(Message<PushStreamMessage> message) {
log.info("Incoming stream message from {}, {}", streamChannel.pushStream(), message);
}
Consumer's streamChannel
public interface StreamChannel {
String PUSH_STREAM = "PushStream";
#Input(StreamChannel.PUSH_STREAM)
SubscribableChannel pushStream();
}
Interceptor (Common Library)
public class GlobalStreamInterceptor extends ChannelInterceptorAdapter {
#Override
public Message<?> preSend(Message<?> msg, MessageChannel mc) {
log.info("presend " + msg);
return msg;
}
#Override
public void postSend(Message<?> msg, MessageChannel mc, boolean sent) {
log.info("postSend " + msg);
}
}
Right, why don't follow GlobalChannelInterceptor options and don't apply
An array of simple patterns against which channel names will be matched.
?
So, you may have something like this:
#GlobalChannelInterceptor(patterns = Processor.INPUT)
Or use a custom name of input channel to your SCSt app.
I'm sending a single message that produces multiple messages, two of which arrive on the same JMS endpoint.
runner.send(sendMessageBuilder -> sendMessageBuilder.endpoint(inputMessage.getEndpoint())
.messageType(MessageType.XML)
.payload(inputMessage.getPayload())
.header(JMSOUTPUTCORRELATIONID, correlationId));
for(OutputMessage outputMessage : inputMessage.getOutputMessages()) {
runner.receive(receiveMessageBuilder -> receiveMessageBuilder.endpoint(outputMessage.getEndpoint())
.schemaValidation(false)
.payload(outputMessage.getPayload())
.header(JMSOUTPUTCORRELATIONID, correlationId));
}
When validating two messages on the same endpoint I'm having trouble finding a way to match them to their respective expected outputs.
I was wondering if Citrus has a built in way to do this or if I could build in a condition that checks the other expected outputs if the first one fails.
I've added a custom validator.
List<OutputMessage> outputMessages = inputMessage.getOutputMessages();
while(outputMessages.size() > 0) {
OutputMessage outputMessage = outputMessages.get(0);
runner.receive(receiveMessageBuilder -> receiveMessageBuilder.endpoint(outputMessage.getEndpoint())
.schemaValidation(true)
.validator(new MultipleOutputMessageValidator(outputMessages))
.header(JMSOUTPUTCORRELATIONID, correlationId));
}
The validator is provided with the the list of expected outputs that have not yet been validated. It will then try to validate each of the expected outputs in the list against the received message and if the validation is succesful removes that expected output from the list.
public class MultipleOutputMessageValidator extends DomXmlMessageValidator {
private static Logger log = LoggerFactory.getLogger(MultipleOutputMessageValidator.class);
private List<OutputMessage> controlMessages;
public MultipleOutputMessageValidator(List<OutputMessage> controlMessages) {
this.controlMessages = controlMessages;
}
#Override
public void validateMessagePayload(Message receivedMessage, Message controlMessage, XmlMessageValidationContext validationContext, TestContext context) throws ValidationException {
Boolean isValidated = false;
for (OutputMessage message : this.controlMessages) {
try {
super.validateMessagePayload(receivedMessage, message, validationContext, context);
isValidated = true;
controlMessages.remove(message);
break;
} catch (ValidationException e) {
// Do nothing for now
}
}
if (!isValidated) {
throw new ValidationException("None of the messages validated");
}
}
}
You should use JMS message selectors so you can "pick" one of the messages from that queue based on a technical identifier. This selector can be a JMS message header for instance (in your case the header JMSOUTPUTCORRELATIONID). This way you make sure to receive the message that you want to validate first.
Example usage:
receive(action -> action.endpoint(someEndpoint)
.selector("correlationId='Cx1x123456789' AND operation='getOrders'"));
Citrus message selector support is described here