I have an application that calls external service and in case of connection errors, application retries for certain number of times.
There are certain instances where the application had to be shutdown on receipt of SIGTERM; During the shutdown process, camel waits for the in-flight messages to be completed which is good but odd times with connection errors, the retry also kicks in causing long delays for shutdown. is there a way to stop the retries while application is shutting down?
Per this link from the year 2012, Graceful shutdown was made aggressive after timeout but seems it is not any more with v2.24.0.
Related
When the java app in the middle of the service process restarts, waits for the process and after finishing that continued or does abort the process.
Depends on which process we are talking about, in-flight http requests are killed, but this behaviour can be configured, see https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#web.graceful-shutdown.
Currently, I have set up a spring boot application, it has 3 pods runnings.
And I have a Kafka consumer that processes a particular task for 20 minutes.
When the Kafka rebalances during that time then the same is consumed again, so I have set the Redis key whenever the message comes first, so when it rebalances it checks that key exists and discards that event as that old process is still running.
But now I have a scenario that a particular pod that is running can get restarted anytime, no when the application restarts and the same message is consumed that then, I want that message to be reprocessed, but as the Redis key exists it discard this event, but the old process is not running.
I have to re-process the message on application restart and discard it in case of Kafka rebalance. How can I handle this scenario?
Avoid the rebalance by increasing max.poll.interval.ms
Kafka is not really suited for this kind of long-running task; consider moving the task to a DB (or even Redis) when received from Kafka, and process from there.
Another solution would be to set max.poll.records to 1, then pause the listener container before running the task on another thread, and then resume the container when the job finishes.
Pausing the container will keep the consumer alive and avoid the rebalance.
When I send a stop signal(either with kill -SIGINT <pid> or System.exit(0) or environment.getApplicationContext().getServer().stop()) to the application, it waits for the shutdownGracePeriod (by default 30 sec or whatever I configure in .yml file) and also it does not accept new request. However, my requirement is to make the server wait for the ongoing request to complete before stopping. The ongoing request may take 30 sec or 30 minutes, it is unknown. Can somebody suggest me the way to achieve this?
Note: I've referred to the below links but could not achieve.
How to shutdown dropwizard application?
shutdownGracePeriod
We've used in-app healthcheck combined with some external load balancer service and prestop scripts. A healthcheck is turned off by the prestop script, then healthcheck says it is unhealthy so no new requests are sent by the load balancer (but existing ones are processed), only after draining period a stop signal is sent to the application.
Even this though has a specified time limit. I don't know how you would monitor requests that last an unknown amount of time.
My tomcat server keeps processing some requests for more than 10 minutes. I stopped client which had triggered those requests but then also tomcat keeps processing those requests.
I have tried different settings for connectionTimeout property in server.xml file of tomcat but it is not working.
I would like to know how to configure tomcat such that tomcat kills/ stops processing requests which take longer than certain time like 10 seconds or 1 minute, etc.
From The Apache Tomcat Connector - Generic HowTo
Timeouts
JK can also use a timeout on request replies. This timeout does not
measure the full processing time of the response. Instead it controls,
how much time between consecutive response packets is allowed.
In most cases, this is what one actually wants. Consider for example
long running downloads. You would not be able to set an effective
global reply timeout, because downloads could last for many minutes.
Most applications though have limited processing time before starting
to return the response. For those applications you could set an
explicit reply timeout. Applications that do not harmonise with reply
timeouts are batch type applications, data warehouse and reporting
applications which are expected to observe long processing times.
If JK aborts waiting for a response, because a reply timeout fired,
there is no way to stop processing on the backend. Although you free
processing resources in your web server, the request will continue to
run on the backend - without any way to send back a result once the
reply timeout fired.
I posted this on the AWS support forums but haven't received a response, so hoping you guys have an idea...
We have an auto-scaling group which boots up or terminates an instance based on current load. What I'd like to be able to do it detect, on my current EC2 instance, that it's about to be shut down and to finish my work.
To describe the situation in more detail. We have an auto-scaling group, and each instance reads content from a single SQS. Each instance will be running multiple threads, each thread is reading from the same SQS queue and processing the data as needed.
I need to know when this instance will be about to shut down, so I can stop new threads from reading data, and block the shutdown until the remaining data has finished processing.
I'm not sure how I can do this in the Java SDK, and I'm worried my instances will be terminated without my data being processed correctly.
Thanks
Lee
When it wants to scale down, AWS Auto Scaling will terminate your EC2 instances without warning.
There's no way to allow any queue workers to drain before terminating.
If your workers are processing messages transactionally, and you're not deleting messages from SQS until after they have been successfully processed, then this shouldn't be a problem. The processing will stop when the instance is terminated, and the transaction will not commit. The message won't be deleted from the SQS queue, and can be picked up and processed by another worker later on.
The only kind of draining behavior it supports is HTTP connection draining from an ELB: "If connection draining is enabled for your load balancer, Auto Scaling waits for the in-flight requests to complete or for the maximum timeout to expire, whichever comes first, before terminating instances".