We Keep Coding

AWS Lambda not removing messages from the queue

I am triggering a Lambda function from an SQS event with the following code:
#Override
public Void handleRequest(SQSEvent sqsEvent, Context context) {
for (SQSMessage sqsMessage : sqsEvent.getRecords()) {
final String body = sqsMessage.getBody();
try {
//do stuff here
} catch (Exception ex) {
//send to DLQ
}
}
return null;
}
The "do stuff" is calling another Lambda function with the following code:
private final AWSLambda client;
private final String functionName;
public LambdaService(AWSLambdaAsync client, String functionName) {
this.client = client;
this.functionName = functionName;
}
public void runWithPayload(String payload) {
logger.info("Invoking lambda {} with payload {}", functionName, payload);
final InvokeRequest request = new InvokeRequest();
request.withFunctionName(functionName).withPayload(payload);
final InvokeResult invokeResult = client.invoke(request);
final Integer statusCode = invokeResult.getStatusCode();
logger.info("Invoked lambda {} with payload {}. Got status code {} and response payload {}",
functionName,
payload,
statusCode,
StandardCharsets.UTF_8.decode(invokeResult.getPayload()).toString());
if(statusCode.equals(200) == false) {
throw new IllegalStateException(String.format("There was an error executing the lambda function %s with payload %s", functionName, payload));
}
}
I am using the following libraries:
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.2.0</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-events</artifactId>
<version>2.2.6</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-sqs</artifactId>
<version>1.11.505</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-lambda</artifactId>
<version>1.11.505</version>
</dependency>
The problem is that it looks like the SQS message is not removed from the queue and it gets reprocessed over and over. It happens every 30 seconds which is exactly the value of Default Visibility Timeout. Now, as far as I know, if the lambda consuming the sqs messages is terminating properly it should automatically delete the message from the queue, but this is not happening.
I don't think there is any error happening in the lambda because I am not getting any message in the DLQ (and I have a catch-all block) and I cannot see any stacktrace in the logs in Cloudwatch. I am bit confused about what's happening here, anyone has some good idea?

Unless something changed recently, I don't think the AWS SDK for Java automatically deletes the message from the queue. You need to write the code to do that.
I would love to be proven wrong on that one, please share the doc excerpt I missed.
Code sample :
https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-sqs-messages.html
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-getting-started-java.html

Request not interrupted when HystrixTimeoutException occured

I have a simple Eureka Server, Config Server, Zuul Gateway, and a test service(named aService below) registed in eureka.
Besides, a implemention of FallbackProvider is registed and timeoutInMilliseconds for default command is 10000.
I send a request to aService, in which will sleep 15 seconds and print tick per second.After 10 seconds a HystrixTimeoutException occured and my custom fallbackResponse accessed, but the tick still go on until 15 seconds end.
My question is, abviously, why is the request not interrupted?Could someone please explain what hystrix and zuul do after HystrixTimeout?
Dependency version:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-commons-dependencies</artifactId>
<version>Edgware.SR2</version>
</dependency>
<dependency>
<groupId>com.netflix.zuul</groupId>
<artifactId>zuul-core</artifactId>
<version>1.3.0</version>
</dependency>
Some of my hystrix configurations:
zuul.servletPath=/
hystrix.command.default.execution.isolation.strategy=THREAD
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds=10000
hystrix.command.aService.execution.isolation.strategy=THREAD
ribbon.ReadTimeout=60000
ribbon.ConnectTimeout=3000
Some of my FallbackProvider:
#Component
public class ServerFallback implements FallbackProvider {
#Override
public String getRoute() {
return "*";
}
#Override
public ClientHttpResponse fallbackResponse() {
// some logs
return simpleClientHttpResponse();
}
#Override
public ClientHttpResponse fallbackResponse(Throwable cause) {
// some logs
return simpleClientHttpResponse();
}
}
``

When using zuul with ribbon(default), the executionIsolationStrategy in HytrixCommandProperties will be overrided by AbstractRibbonCommand,which is SEMAPHORE by default.In this isolation strategy, request will not interrupted immediatelly.See ZuulProxy fails with “RibbonCommand timed-out and no fallback available” when it should do failover

Connection pooling in Spring Boot and mongo db

I am going through spring boot application and mongoDb connection POC.
I have added following dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
Also I have gone through mongoB properties with properties: https://docs.spring.io/spring-boot/docs/current/reference/html/common-application-properties.html
Can you please how do we define connection pooling mechanism here?

You cannot do this out of the box with application properties. You need to make use of MongoClientOptions to configure various aspects of connection pool.
Have a look at the documentation for various options available.
Here is a simple example.
#Bean(name="mongoTempl")
public MongoTemplate mongoTempl() throws Exception {
return new MongoTemplate(createMongoClient(new ServerAddress(host, port))
,dbName);
}
Mongo createMongoClient(ServerAddress serverAddress) {
final MongoClientOptions options = MongoClientOptions.builder()
.threadsAllowedToBlockForConnectionMultiplier(...)
.connectionsPerHost(...)
.connectTimeout(...)
.maxWaitTime(...)
.socketKeepAlive(...)
.socketTimeout(...)
.heartbeatConnectTimeout(...)
.minHeartbeatFrequency(...)
.build();
return new MongoClient(serverAddress, options);
}

You can use also MongoClientSettingsBuilderCustomizer like in this spring sample
#Bean
public MongoClientSettingsBuilderCustomizer customizer() {
return (builder) -> builder.applyToConnectionPoolSettings(
(connectionPool) -> {
connectionPool.maxSize(10);
connectionPool.minSize(2);
connectionPool.maxConnectionIdleTime(5, TimeUnit.MINUTES);
connectionPool.maxWaitTime(2, TimeUnit.MINUTES);
connectionPool.maxConnectionLifeTime(30, TimeUnit.MINUTES);
connectionPool.addConnectionPoolListener();
});
}

Change default Mongo connection pool size in spring-boot

I want to change the default size of connection pool provided by java mongodb driver which is 100 according to mongo docs.
Below is the mongo client bean which I used to customize the connection pool size (refered this question). I set both min and max connectionPerHost attributes to 1 and ran 10 parallel worker threads which interact with the DB to make sure that my change is applied.
#Bean
public Mongo mongo() throws Exception {
MongoClientOptions.Builder clientOptions = new MongoClientOptions.Builder();
clientOptions.minConnectionsPerHost(1);
clientOptions.connectionsPerHost(1);
MongoClient mongoClient = new MongoClient(new MongoClientURI(env.getProperty("mongodbhost"), clientOptions));
return mongoClient;
}
Then I calculated the starting and ending time spots of each worker thread. So that I know for sure the threads are working parallely and my connection pool size haven't changed by these configuration.
Could anyone help me to get through this please? any help would be highly appreciated!

You can configure connection parameters by uri.
spring.data.mongodb.uri=mongodb://localhost:27017/?connectTimeoutMS=300000&minPoolSize=0&maxPoolSize=10&maxIdleTimeMS=900000
Please see the following documentation for other parameters.
https://docs.mongodb.com/manual/reference/connection-string/#connections-connection-options

With updated Spring boot(2.0.0 +) and Mongo DB java(3.9 +) driver versions following code can be used for creating configurable mongo template in spring boot.
Most of the configurations that were earlier part of MongoClientOptions are moved to MongoClientSettings.
import com.mongodb.*;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import com.mongodb.connection.*;
import org.springframework.data.mongodb.core.MongoTemplate;
#Configuration
public class MongoConfig {
//-- variables
#Bean(name = "mongoTemplate")
public MongoTemplate getMongoTemplate(){
MongoTemplate mongoTemplate = new MongoTemplate(getMongoClient(), mongoDatabaseName);
return mongoTemplate;
}
private MongoClient getMongoClient(){
List<ServerAddress> serverAddressList = new ArrayList<>();
String[] hostPortList = mongoHostPortList.split(",");
for (String serverAddress : hostPortList) {
String[] hostPortArr = serverAddress.split(":");
serverAddressList.add(new ServerAddress(hostPortArr[0], Integer.parseInt(hostPortArr[1])));
}
MongoClientSettings mongoSettingsProperties = getMongoClientSettings();
MongoClient mongoClient = MongoClients.create(mongoSettingsProperties);
return mongoClient;
}
private MongoClientSettings getMongoClientSettings() {
return MongoClientSettings.builder()
.applicationName(appName)
.applyToSslSettings(sslBuilder ->
SslSettings.builder().
enabled(sslEnabled).
invalidHostNameAllowed(false).build())
.applyToConnectionPoolSettings(connPoolBuilder ->
ConnectionPoolSettings.builder().
maxWaitTime(maxWaitTime, MILLISECONDS).
maxSize(connectionPoolMinSize).
maxSize(connectionPoolMaxSize).build())
.applyToSocketSettings(socketBuilder ->
SocketSettings.builder().
connectTimeout(connectionTimeout,MILLISECONDS).build())
.readPreference(ReadPreference.secondaryPreferred())
.build();
}
}
Above code is verified with dependencies -
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
<version>2.3.1.RELEASE</version>
</dependency>
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>3.11.2</version>
</dependency>

You can configure connection pool size via MongoDb uri parameters. Details - https://stackoverflow.com/a/50407284/6629515

Spring Boot Redis configuration not working

I am developing a Spring Boot [web] REST-style application with a ServletInitializer (since it needs to be deployed to an existing Tomcat server). It has a #RestController with a method that, when invoked, needs to write to a Redis pub-sub channel. I have the Redis server running on localhost (default port, no password). The relevant part of the POM file has the required starter dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
When I deploy the WAR and hit the endpoint http://localhost:8080/springBootApp/health, I get this response:
{
"status": "DOWN",
"diskSpace": {
"status": "UP",
"total": 999324516352,
"free": 691261681664,
"threshold": 10485760
},
"redis": {
"status": "DOWN",
"error": "org.springframework.data.redis.RedisConnectionFailureException: java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out"
}
}
I added the following to my Spring Boot application class:
#Bean
JedisConnectionFactory jedisConnectionFactory() {
return new JedisConnectionFactory();
}
#Bean
public RedisTemplate<String, Object> redisTemplate() {
RedisTemplate<String, Object> template = new RedisTemplate<String, Object>();
template.setConnectionFactory(jedisConnectionFactory());
return template;
}
I also tried adding the following to my #RestController before executing some test Redis code, but I get the same error as above in the stack trace:
#Autowired
private RedisTemplate<String, String> redisTemplate;
Edit (2017-05-09)
My understanding is that Spring Boot Redis starter assumes the default values of spring.redis.host=localhost and spring.redis.port=6379, I still added the two to application.properties, but that did not fill the gap.
Update (2017-05-10)
I added an answer to this thread.

I done a simple example with redis and spring boot
First I installed redis on docker:
$ docker run --name some-redis -d redis redis-server --appendonly yes
Then I Used this code for receiver :
import java.util.concurrent.CountDownLatch;
public class Receiver {
private static final Logger LOGGER = LoggerFactory.getLogger(Receiver.class);
private CountDownLatch latch;
#Autowired
public Receiver(CountDownLatch latch) {
this.latch = latch;
}
public void receiveMessage(String message) {
LOGGER.info("Received <" + message + ">");
latch.countDown();
}
}
And this is my spring boot app and my listener:
#SpringBootApplication
// after add security library then it is need to use security configuration.
#ComponentScan("omid.spring.example.springexample.security")
public class RunSpring {
private static final Logger LOGGER = LoggerFactory.getLogger(RunSpring.class);
public static void main(String[] args) throws InterruptedException {
ConfigurableApplicationContext contex = SpringApplication.run(RunSpring.class, args);
}
#Autowired
private ApplicationContext context;
#RestController
public class SimpleController{
#RequestMapping("/test")
public String getHelloWorld(){
StringRedisTemplate template = context.getBean(StringRedisTemplate.class);
CountDownLatch latch = context.getBean(CountDownLatch.class);
LOGGER.info("Sending message...");
Thread t = new Thread(new Runnable() {
#Override
public void run() {
for (int i = 0 ; i < 100 ; i++) {
template.convertAndSend("chat", i + " => Hello from Redis!");
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
});
t.start();
try {
latch.await();
} catch (InterruptedException e) {
e.printStackTrace();
}
return "hello world 1";
}
}
///////////////////////////////////////////////////////////////
#Bean
RedisMessageListenerContainer container(RedisConnectionFactory connectionFactory,
MessageListenerAdapter listenerAdapter) {
RedisMessageListenerContainer container = new RedisMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.addMessageListener(listenerAdapter, new PatternTopic("chat"));
return container;
}
#Bean
MessageListenerAdapter listenerAdapter(Receiver receiver) {
return new MessageListenerAdapter(receiver, "receiveMessage");
}
#Bean
Receiver receiver(CountDownLatch latch) {
return new Receiver(latch);
}
#Bean
CountDownLatch latch() {
return new CountDownLatch(1);
}
#Bean
StringRedisTemplate template(RedisConnectionFactory connectionFactory) {
return new StringRedisTemplate(connectionFactory);
}
}
The important point is the redis IP. if you installed it on docker like me then
you should set ip address in application.properties like this:
spring.redis.host=172.17.0.4
I put all my spring examples on github here
In addition I used redis stat to monitor redis. it is simple monitoring.

Spring data redis properties are updated, e.g. spring.redis.host is now spring.data.redis.host.

You need to configure your redis server information using the application.properties:
# REDIS (RedisProperties)
spring.redis.cluster.nodes= # Comma-separated list of "host:port"
spring.redis.database=0 # Database index
spring.redis.url= # Connection URL,
spring.redis.host=localhost # Redis server host.
spring.redis.password= # Login password of the redis server.
spring.redis.ssl=false # Enable SSL support.
spring.redis.port=6379 # Redis server port.
Spring data docs: https://docs.spring.io/spring-boot/docs/current/reference/html/common-application-properties.html#REDIS

This was a proxy related problem, where even access to localhost was somehow being curtailed. Once I disabled the proxy settings, Redis health was UP! So the problem is solved. I did not have to add any property to application.properties and neither did I have to explicitly configure anything in the Spring Boot application class, because Spring Boot and the Redis Starter auto-configures based on Redis defaults (as applicable in my development environment). I just added the following to the pom.xml:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
and the following to the #RestController annotated class, and Spring Boot auto-wired as needed (awesome!).
#Autowired
private RedisTemplate<String, String> redisTemplate;
To publish a simple message to a channel, this single line of code was sufficient for validating the setup:
this.redisTemplate.convertAndSend(channelName, "hello world");
I appreciate all the comments, which were helpful in backing up my checks.