#Value("${amazon.sqs.queue.endpoint}")
private String endpoint;
#Value("${amazon.sqs.queue.name}")
private String queueName;
#Autowired
private SQSListener sqsListener;
#Bean
public DefaultMessageListenerContainer jmsListenerContainer() throws JMSException {
SQSConnectionFactory sqsConnectionFactory = SQSConnectionFactory.builder()
.withAWSCredentialsProvider(new DefaultAWSCredentialsProviderChain())
.withEndpoint(endpoint)
.withAWSCredentialsProvider(new ClasspathPropertiesFileCredentialsProvider("AwsCredentials-sqs.properties"))
.withNumberOfMessagesToPrefetch(10).build();
DefaultMessageListenerContainer dmlc = new DefaultMessageListenerContainer();
dmlc.setConnectionFactory(sqsConnectionFactory);
dmlc.setDestinationName(queueName);
dmlc.setConcurrentConsumers(1);
dmlc.setMaxConcurrentConsumers(100);
dmlc.setMessageListener(sqsListener);
return dmlc;
}
#Component
public class SQSListener implements MessageListener {
private static final Logger LOGGER = LoggerFactory.getLogger(SQSListener.class);
#Override
public void onMessage(Message message) {
try {
// Cast the recei
ved message as TextMessage and print the text to screen.
System.out.println("Received: " + ((TextMessage) message).getText());
} catch (JMSException e) {
e.printStackTrace();
}
}
}
I added a file in s3 then the message was sent to sqs queue.
after getting this message can I get the actual data, that was uploaded in s3?
It's not clear from your question what the message contains, but if the message contains the S3 bucket and key, then yes, you can just use the S3 API to download this.
Add the following dependency
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
</dependency>
This logic will create the S3 client and download the object.
AwsCredentials credentials = AwsBasicCredentials.create(
"<AWS Access Key>",
"<AWS Secret>"
);
S3Client s3client = S3Client.builder()
.region(Region.US_EAST_1) // or whatever region you're in
.credentialsProvider(() -> credentials) // credentials created above (or preferably injected)
.build();
GetObjectRequest getObjectRequest = GetObjectRequest.builder()
.bucket("the bucket") // the bucket that contains the object, from message maybe?
.key("the key") // the key to the object, from message maybe?
.build();
ResponseInputStream<GetObjectResponse> responseInputStream = s3client.getObject(getObjectRequest);
This will give you an InputStream that you can read from.
Related
Currently I am making logic to consume Message using Rabbitmq. However, contrary to expectations, it takes too long to consume the message.
If you look at the Queued messages graph in the picture above, you can see Unacked and Ready stacking up.
Looking at the message rates below, the publish speed is fast, but the consumer ack speed is too slow.
I'm not sure if the Rabbitmq Configuration I've developed is wrong or if I forgot to set the listener configuration.
The rabbitmq message I receive is a callback message.
Any help would be greatly appreciated.
This is Rabbitmq configuration and RabbitListener configuration
#Configuration
#Profile({ProfileConfig.RABBITMQ})
public class RabbitmqConfig {
#Value("${rabbitmq.queue.name}")
private String queueName;
#Value("${rabbitmq.exchange.name}")
private String exchangeName;
#Value("${rabbitmq.routing.key.callback}")
private String routingKey;
#Value("${rabbitmq.fetch-count}")
private Integer fetchCount;
#Bean
Queue queue() {
return new Queue(queueName, true);
}
#Bean
DirectExchange directExchange() {
return new DirectExchange(exchangeName);
}
#Bean
Binding binding(DirectExchange directExchange, Queue queue) {
return BindingBuilder.bind(queue).to(directExchange).with(routingKey);
}
#Bean
public RabbitListenerContainerFactory<SimpleMessageListenerContainer> prefetchOneContainerFactory(
SimpleRabbitListenerContainerFactoryConfigurer configurer, ConnectionFactory factory)
{
SimpleRabbitListenerContainerFactory simpleFactory = new SimpleRabbitListenerContainerFactory();
configurer.configure(simpleFactory, factory);
simpleFactory.setPrefetchCount(fetchCount);
return simpleFactory;
}
}
#RabbitListener(queues = {"${rabbitmq.queue.name}"}, concurrency = "3", containerFactory = "prefetchOneContainerFactory")
public void receiveMessage(final String message, Channel channel, #Header(AmqpHeaders.DELIVERY_TAG) long tag) {
try {
JSONParser parser = new JSONParser();
JSONObject json = (JSONObject) parser.parse(message);
String messageType = json.get("messageType").toString();
log.debug("Receive Queue Key={}, Message = {}", messageType, message);
AsyncType asyncType = AsyncType.valueOf(messageType);
executeMessage(asyncType, message);
} catch (Exception e) {
traceService.removeTraceId();
traceService.printErrorLog(log, "Fail to deal receive message.", e, PrintStackPolicy.ALL);
} finally {
try {
channel.basicAck(tag, false);
}
catch (IOException e) {
traceService.printErrorLog(log, "Fail to send ack to RabbitMQ", e, PrintStackPolicy.ALL);
}
}
}
The goal is to consume messages to Rabbitmq faster.
However, the current consumption speed is too slow.
I'm not able to send message to sqs-queue from my java application.
I am running roribio16/alpine-sqs docker image for SQS in my local and I've created a standard queue.
Code snippets:
#Data
#Configuration
#EnableSqs
public class SQSConfig {
private final String amazonAWSAccessKey;
private final String amazonAWSSecretKey;
private final String amazonSQSRegion;
public SQSConfig(#Value("${aws.accesskey}") String amazonAWSAccessKey,
#Value("${aws.secretkey}") String amazonAWSSecretKey,
#Value("${sqs.aws.region}") String amazonSQSRegion) {
this.amazonAWSAccessKey = amazonAWSAccessKey;
this.amazonAWSSecretKey = amazonAWSSecretKey;
this.amazonSQSRegion = amazonSQSRegion;
}
#Bean
public QueueMessagingTemplate queueMessagingTemplate() {
return new QueueMessagingTemplate(amazonSQSClient());
}
#Bean
#Primary
public AmazonSQSAsync amazonSQSClient() {
return AmazonSQSAsyncClientBuilder.standard()
.withRegion(amazonSQSRegion)
.withCredentials(new AWSStaticCredentialsProvider(new DefaultAWSCredentialsProviderChain().getCredentials()))
.build();
}
private AWSStaticCredentialsProvider amazonAWSCredentials() {
return new AWSStaticCredentialsProvider(new BasicAWSCredentials(amazonAWSAccessKey, amazonAWSSecretKey));
}
}
MessageProducerService:
#Component
public class SQSMessageProducerService {
#Value("${sqs.endpoint}")
private String amazonSQSEndpoint;
#Autowired
private QueueMessagingTemplate queueMessagingTemplate;
public void sendMessage(String messageBody) {
Message messageBuilt = MessageBuilder.withPayload(messageBody).build();
try {
queueMessagingTemplate.send(amazonSQSEndpoint, messageBuilt);
} catch (MessagingException e) {
System.out.println(e);
}
System.out.println("message sent: " + messageBody);
}
}
Calling using -
sqsMessageProducerService.sendMessage(userEvent.toString());
Error stack -
InvalidAction; see the SQS docs. (Service: AmazonSQS; Status Code:
400; Error Code: InvalidAction; Request ID:
00000000-0000-0000-0000-000000000000)
Vars from properties file -
sqs.endpoint=${SQS_ENDPOINT:http://localhost:9324/test-sns-queue}
sqs.aws.region=${SQS_AWS_REGION:us-west-2}
Also, SQS instance is up in my local and can be accessed using http://localhost:9325/.
Is there some config that I'm missing here?
I think the problem might be that the endpoint is http://localhost:9324/queue/test-sns-queue
I try to receive a protobuf message out of RabbitMQ using Spring Integration.
My integration flow:
public class FacadeIntegrationFlowAdapter extends IntegrationFlowAdapter {
#SuppressWarnings("rawtypes")
private final Facade facade;
private final FacadeProperties facadeProperties;
#SuppressWarnings("unchecked")
#Override
protected IntegrationFlowDefinition<?> buildFlow() {
return from(facadeProperties.getQueueName())
.handle(facade::getNewMessages);
}
}
The getNewMessages method:
#Override
public ExchangeResponse getNewMessages(Message<ExchangeRequest> message) {
ExchangeRequest request = message.getPayload();
log.info("Receiving new message: " + request.toString());
This is how I send the message to the queue. It's so simple to make the test easy to follow.
ExchangeRequest request = ExchangeRequest.newBuilder()
.addAllAuthors(List.of("author1", "author2"))
.addAllBooks(List.of("book1", "book2"))
.build();
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setUsername("user");
connectionFactory.setPassword("password");
connectionFactory.setHost("localhost");
connectionFactory.setPort(24130);
try {
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
var basicProperties = new AMQP.BasicProperties().builder()
.contentType("application/x-protobuf")
.type(request.getDescriptorForType().getFullName())
.build();
channel.basicPublish(
"facade-exchange", "facade-routing-key", basicProperties, request.toByteArray());
} catch (IOException e) {
Unfortunately, I keep getting the exception:
com.google.protobuf.InvalidProtocolBufferException: Type of the Any message does not match the given class.
However, when I change the getNewMessages method to the following, all seems fine.
#Override
public ExchangeResponse getNewMessages(Message message) {
try {
Any payload = (Any) message.getPayload();
ByteString value = payload.getValue();
ExchangeRequest request = ExchangeRequest.parseFrom(value);
log.info("Receiving new message: " + request.toString());
Where do I make a mistake? Tx!
My application registers a listeners to a SQS queue (queue itself is populated by a SNS topic).
When I start the application, message consumer is working as expected but after a while it stops to receive any messages. Can it be that consumer is shutting down after a while?
Suggestions or comments would be much appreciated.
SQSConnection:
#Bean
public SQSConnection amazonSQSConnection(
#Value("${aws.access.key}") String accessKey,
#Value("${aws.secret.key}") String secretKey) throws JMSException {
BasicAWSCredentials basicAWSCredentials = new BasicAWSCredentials(accessKey, secretKey);
AmazonSQSClientBuilder client = AmazonSQSClientBuilder
.standard()
.withRegion(Regions.GovCloud)
.withCredentials(new AWSStaticCredentialsProvider(basicAWSCredentials));
SQSConnectionFactory connectionFactory = new SQSConnectionFactory(new ProviderConfiguration(), client);
return connectionFactory.createConnection();
}
Consummer:
#Bean
public MessageConsumer workOrderChangeConsumer(
SQSConnection connection,
WorkOrderKittingService workOrderKittingService,
AuthenticationProvider authProvider,
#Value("${app.user.name}") String appUserName,
#Value("${aws.sqs.workorder.change.queue}") String woChangeQueue) throws JMSException {
Session session = connection.createSession(false, Session.CLIENT_ACKNOWLEDGE);
Queue queue = session.createQueue(woChangeQueue);
WorkOrderChangeIngestor workOrderChangeIngestor = new WorkOrderChangeIngestor(
workOrderKittingService,
authProvider,
appUserName);
MessageConsumer consumer = session.createConsumer(queue);
consumer.setMessageListener(workOrderChangeIngestor);
connection.start();
return consumer;
}
You're trying to manage the connection lifecycle by yourself.
I recommend you to let spring manage that for you, by using spring-cloud-aws
https://docs.spring.io/spring-cloud-aws/docs/2.2.3.RELEASE/reference/html/#receiving-a-message
You can create a listener through annotations:
#Component
public class MyMessageHandler {
#SqsListener("queueName")
void handle(String message) {
...
throw new MyException("something went wrong");
}
#MessageExceptionHandler(MyException.class)
void handleException(MyException e) {
...
}
}
I'm trying to send an email with file attachments in Spring Boot.
This is a basic gmail SMTP server application properties config:
This is my EmailService:
EmailService
When I call this method with mailMessageDto object passed, there is no exception thrown. Nothing happens, e-mail isn't sent.
I have debugged on javaMailSender.send(messsage) line of code and everything seems fine.
Update
spring.mail.properties.mail.smtp.ssl.enable=false
should be false not true spring.mail.properties.mail.smtp.socketFactory.class=javax.net.ssl.SSLSocketFactory
step 1. add dependencies in porm.xml
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-mail</artifactId>
</dependency>
step 2. add configuration code in application.properties
spring.mail.host=smtp.gmail.com
spring.mail.port=465
spring.mail.username=username
spring.mail.password=password
spring.mail.protocol=smtps
spring.mail.properties.mail.smtp.auth=true
spring.mail.properties.mail.smtp.connectiontimeout=5000
spring.mail.properties.mail.smtp.timeout=5000
spring.mail.properties.mail.smtp.writetimeout=5000
spring.mail.properties.mail.smtp.starttls.enable=true
step 3. add code in controller
masterconroller.java
#GetMapping("/sendmail")
#ResponseBody
String home() {
try {
masterServiceImpl.sendEmail("path");
return "Email Sent!";
} catch (Exception ex) {
return "Error in sending email: " + ex;
}
}
step 4. add code in MasterServiceImpl.java
#Autowired
private JavaMailSender javaMailSender;
public void sendEmail(String path) throws Exception{
MimeMessage message = javaMailSender.createMimeMessage();
MimeMessageHelper helper = new MimeMessageHelper(message, true);
helper.setTo("xyz#gmail.com");
helper.setText("<html><body><h1>hello Welcome!</h1><body></html>", true);
FileSystemResource file = new FileSystemResource(new File(path));
helper.addAttachment("testfile", file);
helper.addAttachment("test.png", new ClassPathResource("test.jpeg"));
helper.setSubject("Hi");
javaMailSender.send(message);
}
I propose you to apply SRP to sendMessageWithAttachment() method by extracting functionality around adding attachments:
private void addAttachments(MailMessageDto message, MimeMessageHelper helper) {
message.getFiles().forEach(file -> addAttachment(file, helper));
}
This method streams over all files and adds every file by using addAttachment():
private void addAttachment(File file, MimeMessageHelper helper) {
String fileName = file.getName();
try {
helper.addAttachment(fileName, file);
log.debug("Added a file atachment: {}", fileName);
} catch (MessagingException ex) {
log.error("Failed to add a file atachment: {}", fileName, ex);
}
}
This will log an error for each failed attachment. Can you try this approach?