How can i handle exception on streaming processing with quarkus + kafka + smallrye?
My code is very similar to the imperative producer example on quarkus guide (https://quarkus.io/guides/kafka#imperative-usage)
import io.smallrye.reactive.messaging.annotations.Channel;
import io.smallrye.reactive.messaging.annotations.Emitter;
import javax.inject.Inject;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.Consumes;
import javax.ws.rs.core.MediaType;
#Path("/prices")
public class PriceResource {
#Inject #Channel("price-create") Emitter<Double> priceEmitter;
#POST
#Consumes(MediaType.TEXT_PLAIN)
public void addPrice(Double price) {
priceEmitter.send(price);
}
}
I wanted something similar the vanilla Kafka library, that gives the option to handle the callback of each record requested to send.
ProducerRecord<String, String> record = new ProducerRecord<>("topic-name", key, value);
producer.send(record, new Callback() {
#Override
public void onCompletion(RecordMetadata metadata, Exception exception) {
logger.info(record.toString());
if (exception != null) {
logger.error("Producer exception", exception);
}
}
});
Tks
There is a section of the docs on Acknowlegement
#Incoming("i")
#Outgoing("j")
#Acknowledgment(Acknowledgment.Strategy.MANUAL)
public CompletionStage<Message<String>> manualAck(Message<String> input) {
return CompletableFuture.supplyAsync(input::getPayload)
.thenApply(Message::of)
.thenCompose(m -> input.ack().thenApply(x -> m));
}
Related
I'm trying to figure out how to include custom headers in the Spring Message<?> used in Spring Cloud Stream with the Kafka Binder. My goal is to include some custom header data that would be added on in one producer (function) class, passed to kafka and then consumed by another class in a different service (with the customer header data).
I feel like I am missing something as I can seem to get it to work using the TestChannelBinder e.g.
import org.springframework.messaging.Message;
import org.springframework.stereotype.Component;
import java.util.function.Function;
#Component
#Slf4j
public class BaseStream implements Function<Message<String>, String> {
#Override
public String apply(Message<String> transactionMessage) {
log.debug("Converted Message: {} ", transactionMessage);
return transactionMessage.getPayload();
}
}
Test class with Test Binder:
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.cloud.stream.binder.test.InputDestination;
import org.springframework.cloud.stream.binder.test.OutputDestination;
import org.springframework.cloud.stream.binder.test.TestChannelBinderConfiguration;
import org.springframework.context.annotation.Import;
import org.springframework.integration.support.MessageBuilder;
import org.springframework.kafka.test.context.EmbeddedKafka;
import org.springframework.test.context.TestPropertySource;
#SpringBootTest
#TestPropertySource("classpath:testStream.properties")
#Import(TestChannelBinderConfiguration.class)
public class TestForStream {
#Autowired
InputDestination inputDestination;
#Autowired
OutputDestination outputDestination;
#Test
void contextLoads() {
inputDestination.send(MessageBuilder
.withPayload("Test Payload")
.setHeader("customHeader", "headerSpecificData")
.build());
}
}
testStream.properties
spring.cloud.function.definition=baseStream
spring.cloud.stream.bindings.baseStream-in-0.destination=test-in
spring.cloud.stream.bindings.baseStream-out-0.destination=test-out
spring.cloud.stream.bindings.baseStream-in-0.group=test-group-base
Log when running:
Converted Message: GenericMessage [payload=Test Payload, headers={id=5c6d1082-c084-0b25-4afc-b5d97bf537f9, customHeader=headerSpecificData, contentType=application/json, timestamp=1639398696800, target-protocol=kafka}]
Which is what I am looking to do. But when I try to test it for the kafka bider it seems to include the Message<String> object in the payload as a JSON string, which I thought would be parsed into the requested input of the function BaseStream.
Just wondering if someone could maybe see where i'm going wrong with my testing as I have tried various things to get this to work, and seeing as it works with a test binder I would assume it works for the Kafka Binder.
Test Class for Kafka Binder Test:
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.integration.support.MessageBuilder;
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.core.ProducerFactory;
import org.springframework.kafka.support.serializer.JsonSerializer;
import org.springframework.kafka.test.EmbeddedKafkaBroker;
import org.springframework.kafka.test.context.EmbeddedKafka;
import org.springframework.kafka.test.utils.KafkaTestUtils;
import org.springframework.test.context.TestPropertySource;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
#EmbeddedKafka(partitions = 1, brokerProperties = { "listeners=PLAINTEXT://localhost:9092", "port=9092"})
#SpringBootTest
#TestPropertySource("classpath:testStream.properties")
public class TestForStream {
public static CountDownLatch latch = new CountDownLatch(1);
#Autowired
public EmbeddedKafkaBroker broker;
#Test
void contextLoads() {
sleep(5);//Included this as it takes some time to init>
sendMessage("test-in", MessageBuilder
.withPayload("Test Payload")
.setHeader("customHeader", "headerSpecificData")
.build());
}
public <T> ProducerFactory<String, T> createProducerFactory() {
Map<String, Object> configs = new HashMap<>(KafkaTestUtils.producerProps(broker));
configs.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configs.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
//Is JsonSerializer correct for a message?
return new DefaultKafkaProducerFactory<>(configs);
}
public <T> void sendMessage(String topic, T listObj) {
try {
KafkaTemplate<String, T> kafkaTemplate = new KafkaTemplate<>(createProducerFactory());
kafkaTemplate.send(new ProducerRecord<>(topic, listObj));
}catch (Exception e){
e.printStackTrace();
}
}
public void sleep(long time){
try {
latch.await(time, TimeUnit.SECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
Log of kafka binder test for message:
Converted Message: GenericMessage [payload={"payload":"Test Payload","headers":{"customHeader":"headerSpecificData","id":"d540a3ca-28db-b137-fc86-c25cc4b7eb8b","timestamp":1639399810476}}, headers={deliveryAttempt=1, kafka_timestampType=CREATE_TIME, kafka_receivedTopic=test-in, target-protocol=kafka, kafka_offset=0, scst_nativeHeadersPresent=true, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer#79580279, id=1cf2d382-df29-2672-4180-07da99e58244, kafka_receivedPartitionId=0, kafka_receivedTimestamp=1639399810526, contentType=application/json, __TypeId__=[B#24c79350, kafka_groupId=test-group-base, timestamp=1639399810651}]
So here the message has been included in the payload and the kafka headers included in the headers as expected.
I have tried spring.cloud.stream.kafka.binder.headers and headerMode to see if they would change anything but to no avail.
Edit:
Using springCloudVersion = 2020.0.3
I was using:
public <T> void sendMessage(String topic, T listObj) {
try {
KafkaTemplate<String, T> kafkaTemplate = new KafkaTemplate<>(createProducerFactory());
kafkaTemplate.send(new ProducerRecord<>(topic, listObj));
}catch (Exception e){
e.printStackTrace();
}
}
To send the message which was putting the message as the value.
What I should've been using:
public void sendMessage(String topic, Message<?> listObj) {
try {
KafkaTemplate<String, Message<?>> kafkaTemplate = new KafkaTemplate<>(createProducerFactory());
kafkaTemplate.setDefaultTopic(topic);
kafkaTemplate.send(listObj);
}catch (Exception e){
e.printStackTrace();
}
}
I’m trying to add a Prometheus metrics exporter to my Java app. The app is currently using javax.ws.rs to define REST endpoints.
For example:
Import javax.ws.rs.*;
Import javax.ws.rs.core.MediaType;
Import javax.ws.rs.core.Response;
#GET
#Path(“/example”)
#Timed
Public Response example(#QueryParam(“id”) Integer id) {
return Response.ok(“testing”)
}
All the examples I found for setting up Prometheus in Java are using Spring. They suggest the following:
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import io.prometheus.client.exporter.HTTPServer;
import java.io.IOException;
#SpringBootApplication
public class App {
public static void main(String[] args) {
SpringApplication.run(App.class, args);
try {
HTTPServer server = new HTTPServer(8081);
} catch (IOException e) { e.printStackTrace(); }
}
}
Is there a way I can simply define a new endpoint in my current setup, for example:
#GET
#Path(“/metrics”)
#Timed
Public Response example {
return Response.ok(“return prom metrics here”)
}
Without having to introduce Spring into the stack?
This can be done as follows:
import io.prometheus.client.Counter;
import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.exporter.TextFormat;
CollectorRegistry registry = new CollectorRegistry();
Counter exCounter = Counter.build().name(“example”).register(registry);
#GET
#Path(“/metrics”)
Public String getMetrics() {
Writer writer = new StringWriter();
try {
TextFormat.write004(writer, registry.metricFamilySamples());
return writer.toString();
} catch (IOException e) {
return “error”;
}
}
I'm trying to listen my Gmail inbox for incoming mails. Every time new mail arrives, I want to see it's subject and content.
So far, I have this:
import java.io.IOException;
import javax.mail.BodyPart;
import javax.mail.Folder;
import javax.mail.internet.ContentType;
import javax.mail.internet.MimeBodyPart;
import javax.mail.internet.MimeMessage;
import javax.mail.internet.MimeMultipart;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.commons.mail.util.MimeMessageParser;
import org.springframework.context.support.ClassPathXmlApplicationContext;
import org.springframework.integration.channel.DirectChannel;
import org.springframework.integration.mail.transformer.MailToStringTransformer;
import org.springframework.messaging.Message;
import org.springframework.messaging.MessageHandler;
import org.springframework.messaging.MessagingException;
public class GmailInboundImapIdleAdapterTestApp {
private static Log logger = LogFactory.getLog(GmailInboundImapIdleAdapterTestApp.class);
public static void main (String[] args) throws Exception {
#SuppressWarnings("resource")
ClassPathXmlApplicationContext ac = new ClassPathXmlApplicationContext("/META-INF/spring/integration/gmail-imap-idle-config.xml");
DirectChannel inputChannel = ac.getBean("receiveChannel", DirectChannel.class);
inputChannel.subscribe(new MessageHandler() {
public void handleMessage(Message<?> message){
MimeMessage mm = (MimeMessage) message.getPayload();
try {
System.out.println("Subject: "+mm.getSubject());
System.out.println("Body: "+readPlainContent(mm));
}
catch (javax.mail.MessagingException e) {
System.out.println("MessagingException: "+e.getMessage());
e.printStackTrace();
}
catch (Exception e) {
System.out.println("Exception: "+e.getMessage());
e.printStackTrace();
}
}
});
}
private static String readHtmlContent(MimeMessage message) throws Exception {
return new MimeMessageParser(message).parse().getHtmlContent();
}
private static String readPlainContent(MimeMessage message) throws Exception {
return new MimeMessageParser(message).parse().getPlainContent();
}
}
It can read the mail subject correctly. But no luck with mail body.javax.mail.FolderClosedException hit me. How to fix this?
As Gary said: simple-content="true" or since recently autoCloseFolder = false: https://docs.spring.io/spring-integration/docs/5.2.0.RELEASE/reference/html/mail.html#mail-inbound
Starting with version 5.2, the autoCloseFolder option is provided on the mail receiver. Setting it to false doesn’t close the folder automatically after a fetch, but instead an IntegrationMessageHeaderAccessor.CLOSEABLE_RESOURCE header (see MessageHeaderAccessor API for more information) is populated into every message to producer from the channel adapter. It is the target application’s responsibility to call the close() on this header whenever it is necessary in the downstream flow:
I'm trying to confirm the value of an HTTP response header with Spring 5 WebClient, but only if the web call responds with an HTTP 200 status code. In this use case if authentication is not successful, the API call returns with an HTTP 401 without the response header present. I have the following code below which functionally works, but it is making the web call twice (because I'm blocking twice). Short of just blocking on the HTTP response header only, and putting a try/catch for an NPE when the header isn't present, is there any "cleaner" way to do this?
import java.net.URI;
import java.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.WebApplicationType;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.http.HttpMethod;
import org.springframework.http.HttpStatus;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.util.LinkedMultiValueMap;
import org.springframework.web.reactive.function.BodyInserters;
import org.springframework.web.reactive.function.client.ClientRequest;
import org.springframework.web.reactive.function.client.ClientResponse;
import org.springframework.web.reactive.function.client.ExchangeFunction;
import org.springframework.web.reactive.function.client.ExchangeFunctions;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
#SpringBootApplication
public class ContentCheckerApplication {
private static final Logger LOGGER = LoggerFactory.getLogger(ContentCheckerApplication.class);
private ExchangeFunction exchange = ExchangeFunctions.create(new ReactorClientHttpConnector());
public static void main(String[] args) {
SpringApplication app = new SpringApplication(ContentCheckerApplication.class);
// prevent SpringBoot from starting a web server
app.setWebApplicationType(WebApplicationType.NONE);
app.run(args);
}
#Bean
public CommandLineRunner myCommandLineRunner() {
return args -> {
// Our reactive code will be declared here
LinkedMultiValueMap<String, String> formData = new LinkedMultiValueMap<String, String>();
formData.add("username", args[2]);
formData.add("password", args[3]);
ClientRequest request = ClientRequest.method(HttpMethod.POST, new URI(args[0]+"/api/token"))
.body(BodyInserters.fromFormData(formData)).build();
Mono<ClientResponse> mresponse = exchange.exchange(request);
Mono<String> mnewToken = mresponse.map(response -> response.headers().asHttpHeaders().getFirst("WSToken"));
LOGGER.info("Blocking for status code...");
HttpStatus statusCode = mresponse.block(Duration.ofMillis(1500)).statusCode();
LOGGER.info("Got status code!");
if (statusCode.value() == 200) {
String newToken = mnewToken.block(Duration.ofMillis(1500));
LOGGER.info("Auth token is: " + newToken);
} else {
LOGGER.info("Unable to authenticate successfully! Status code: "+statusCode.value());
}
};
}
}
Thanks to comments from #M. Deinum to guide me, I have the following code which is workable now.
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.WebApplicationType;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.http.HttpStatus;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.util.LinkedMultiValueMap;
import org.springframework.web.reactive.function.BodyInserters;
import org.springframework.web.reactive.function.client.ExchangeFunction;
import org.springframework.web.reactive.function.client.ExchangeFunctions;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.core.publisher.Mono;
#SpringBootApplication
public class ContentCheckerApplication {
private static final Logger LOGGER = LoggerFactory.getLogger(ContentCheckerApplication.class);
private ExchangeFunction exchange = ExchangeFunctions.create(new ReactorClientHttpConnector());
public static void main(String[] args) {
SpringApplication app = new SpringApplication(ContentCheckerApplication.class);
// prevent SpringBoot from starting a web server
app.setWebApplicationType(WebApplicationType.NONE);
app.run(args);
}
#Bean
public CommandLineRunner myCommandLineRunner() {
return args -> {
// Change some Netty defaults
ReactorClientHttpConnector connector = new ReactorClientHttpConnector(
options -> options.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 2000)
.compression(true)
.afterNettyContextInit(ctx -> {
ctx.addHandlerLast(new ReadTimeoutHandler(1500, TimeUnit.MILLISECONDS));
}));
LinkedMultiValueMap<String, String> formData = new LinkedMultiValueMap<String, String>();
formData.add("username", args[2]);
formData.add("password", args[3]);
WebClient webClient = WebClient.builder().clientConnector(connector).build();
Mono<String> tokenResult = webClient.post()
.uri( args[0] + "/api/token" )
.body( BodyInserters.fromFormData(formData))
.exchange()
.onErrorMap(ContentCheckerApplication::handleAuthTokenError)
.map(response -> {
if (HttpStatus.OK.equals(response.statusCode())) {
return response.headers().asHttpHeaders().getFirst("WSToken");
} else {
return "";
}
});
LOGGER.info("Subscribing for the result and then going to sleep");
tokenResult.subscribe(ContentCheckerApplication::handleAuthTokenResponse);
Thread.sleep(3600000);
};
}
private static Throwable handleAuthTokenError(Throwable e) {
LOGGER.error("Exception caught trying to process authentication token. ",e);
ContentCheckerApplication.handleAuthTokenResponse("");
return null;
}
private static void handleAuthTokenResponse(String newToken) {
LOGGER.info("Got status code!");
if (!newToken.isEmpty()) {
LOGGER.info("Auth token is: " + newToken);
} else {
LOGGER.info("Unable to authenticate successfully!");
}
System.exit(0);
}
}
Hi i have an issue with spring websockets, this is the scenario:
a standalone application is sending (remote) some data like date Date, procedence String, and weight BigDecimal this data is sending via TCP to socket,
after that this data is saving into database at this point all is fine, but in the next step (websocket) i cannot show this information in a webpage, the weight data must be showed (live) in the screen
this is my websocket configuration:
import java.util.List;
import org.springframework.context.annotation.Configuration;
import org.springframework.messaging.converter.MessageConverter;
import org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolver;
import org.springframework.messaging.handler.invocation.HandlerMethodReturnValueHandler;
import org.springframework.messaging.simp.config.ChannelRegistration;
import org.springframework.messaging.simp.config.MessageBrokerRegistry;
import org.springframework.web.socket.config.annotation.EnableWebSocketMessageBroker;
import org.springframework.web.socket.config.annotation.StompEndpointRegistry;
import org.springframework.web.socket.config.annotation.WebSocketMessageBrokerConfigurer;
import org.springframework.web.socket.config.annotation.WebSocketTransportRegistration;
#Configuration
#EnableWebSocketMessageBroker
public class WebSocketConfiguration implements WebSocketMessageBrokerConfigurer {
#Override
public void registerStompEndpoints(final StompEndpointRegistry registry) {
registry.addEndpoint("/indicator").withSockJS();
}
#Override
public void configureClientInboundChannel(final ChannelRegistration registration) {
}
#Override
public void configureClientOutboundChannel(final ChannelRegistration registration) {
}
#Override
public void configureMessageBroker(final MessageBrokerRegistry registry) {
}
#Override
public void configureWebSocketTransport(WebSocketTransportRegistration wstr) {
}
#Override
public void addArgumentResolvers(List<HandlerMethodArgumentResolver> list) {
}
#Override
public void addReturnValueHandlers(List<HandlerMethodReturnValueHandler> list) {
}
#Override
public boolean configureMessageConverters(List<MessageConverter> list) {
return Boolean.TRUE;
}
}
this is my another class that receive data form socket and process information and send to websocket:
import com.mcss.mcontrols.helper.ByteHelper;
import com.spc.basweb.Constants;
import com.spc.basweb.transmissor.dto.Transmission;
import org.apache.log4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.ApplicationListener;
import org.springframework.messaging.core.MessageSendingOperations;
import org.springframework.messaging.simp.broker.BrokerAvailabilityEvent;
import com.spc.basweb.service.BroadcastingService;
import com.spc.basweb.service.DataProcessorService;
import java.io.IOException;
import org.springframework.integration.annotation.MessageEndpoint;
import org.springframework.integration.annotation.ServiceActivator;
import org.springframework.integration.annotation.Transformer;
#MessageEndpoint
public class BroadcastingServiceImpl implements BroadcastingService, ApplicationListener<BrokerAvailabilityEvent> {
private static final Logger LOGGER = Logger.getLogger(BroadcastingServiceImpl.class);
private final MessageSendingOperations<String> messagingTemplate;
private String processedData;
#Autowired
DataProcessorService dataProcessorService;
#Autowired
public BroadcastingServiceImpl(final MessageSendingOperations<String> messagingTemplate) {
this.messagingTemplate = messagingTemplate;
}
#Override
public String getProcessedData() {
return processedData;
}
#Override
#ServiceActivator(inputChannel = "broadcaster")
public String broadcast(byte[] bytes) {
try {
Transmission t = (Transmission) ByteHelper.toObject(bytes);
LOGGER.debug(t.getProcedence() + " " + t.getDate() + " " + t.getWeight());
String rm = this.dataProcessorService.processData(t);
this.messagingTemplate.convertAndSend(Constants.END_POINT_READ, this.dataProcessorService.getWeighing().getWeight().toString());
return rm;
} catch (IOException | ClassNotFoundException ex) {
LOGGER.error("Error de transmision de objetos", ex);
}
return DataProcessorService.NOT_OK_RESPONSE;
}
#Override
public void onApplicationEvent(BrokerAvailabilityEvent e) {
LOGGER.debug("Application event");
}
#Transformer(outputChannel = "broadcaster")
public String convert(String response) {
return response;
}
}
in the debbuger i'm getting this information:
30-03-2016 15:07:20 DEBUG SimpleBrokerMessageHandler:277 - Processing MESSAGE destination=/read session=null payload=3003
in another class (Controller) i'm using the same method:
this.messagingTemplate.convertAndSend(Constants.END_POINT_READ, "3500");
and sending "manually" the information an is showing correctly. and i'm getting in debbuger this message:
30-03-2016 15:05:18 DEBUG SimpleBrokerMessageHandler:277 - Processing MESSAGE destination=/read session=dfR45V77 payload=3500
the difference is in session value but i don't know what this session is having null in the process, what am i doing wrong some clarification o help is welcome
First of all I don't see the configureMessageBroker implementation, so it isn't clear how that may work at all...
From other hand if you see such a difference, try to debug the code in the SimpMessagingTemplate.
I only see headerAccessor.setSessionId(sessionId); in the SimpleBrokerMessageHandler when it does
logger.debug("Broadcasting to " + subscriptions.size() + " sessions.");