we experience the following error pattern:
sometimes we have GAE app request processing lasting long, this throws DeadlineExceededException as GAE has a limit for 1 min. This is a described by docs, ok.
apart from DeadlineExceededException we get A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may be throwing exceptions during the initialization of your application. (Error code 104)
subsequent requests coming to GAE app within next few milliseconds fail with the same Error code 104
Questions:
Why #2 is reported?
How can we avoid #3? Is it a bug in GAE? What is the mechanism of such failure?
Thanks for help.
As Bruyere kindly pointed, related threads killing in result of timeout exception is detailed here:
If concurrent requests are enabled through the "threadsafe" flag, every other running concurrent request is killed with error code 104:
Related
the requests suddenly stopped and I don't know why this happen.
I am using java11.
{"#type":"type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry", "statusDetails":"client_disconnected_before_any_response"}
this my app engine configs
<precompilation-enabled>false</precompilation-enabled>
<sessions-enabled>true</sessions-enabled>
<threadsafe>true</threadsafe>
<instance-class>F4</instance-class>
<automatic-scaling>
<max-concurrent-requests>50</max-concurrent-requests>
<max-pending-latency>8s</max-pending-latency>
<min-pending-latency>4s</min-pending-latency>
<min-instances>1</min-instances>
<min-idle-instances>1</min-idle-instances>
</automatic-scaling>
Error you're getting means as per documentation that the client disconnected before load balancer could reply:
client_disconnected_before_any_response - The connection to the client
was broken before the load balancer sent any response.
And the Error code 123 is a Deadline Exceeded Error
HARD_REQUEST_TIME_LIMIT- Request exceeded a deadline, causing the
instance to shut down
Here are some known issues that cause this issue:
High latency, this means that your startup code for instances may take too long, what is recommended in this case is to set a “min_idle_instances” just remember that this might cause an increase in your billing.
Running multiple requests in parallel, this can cause a thread deadlock and result in timeouts; in this case it is recommend to use load balancers.
As also mentioned in this stackoverflow Answer by LundinCast
You've most likely set your App Engine service scaling element to "autoscaling" (or didn't define it, autoscaling being the default
value) in the app.yaml file.
Instances in autoscaling have a deadline of 10min, as documented here. You'll need to re-deploy your service with an updated
app.yaml file setting the scaling element to "manual scaling" or
"basic scaling" to allow your tasks to run to up to 24h.
You might also check this similar cases for more information
how to solve "Process terminated because the request deadline was exceeded. (Error code 123)" in google api?
Google App Engine: Intermittent Issue: Process terminated because the request deadline was exceeded. (Error code 123)
I have created a spring boot web application and deployed war of the same to tomcat container.
The application connects to mongoDB using Async connections. I am using mongodb-driver-async library for that.
At startup everything works fine. But as soon as load increases, It shows following exception in DB connections:
org.springframework.web.context.request.async.AsyncRequestTimeoutException: null
at org.springframework.web.context.request.async.TimeoutDeferredResultProcessingInterceptor.handleTimeout(TimeoutDeferredResultProcessingInterceptor.java:42)
at org.springframework.web.context.request.async.DeferredResultInterceptorChain.triggerAfterTimeout(DeferredResultInterceptorChain.java:75)
at org.springframework.web.context.request.async.WebAsyncManager$5.run(WebAsyncManager.java:392)
at org.springframework.web.context.request.async.StandardServletAsyncWebRequest.onTimeout(StandardServletAsyncWebRequest.java:143)
at org.apache.catalina.core.AsyncListenerWrapper.fireOnTimeout(AsyncListenerWrapper.java:44)
at org.apache.catalina.core.AsyncContextImpl.timeout(AsyncContextImpl.java:131)
at org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapter.java:157)
I am using following versions of software:
Spring boot -> 1.5.4.RELEASE
Tomcat (installed as standalone binary) -> apache-tomcat-8.5.37
Mongo DB version: v3.4.10
mongodb-driver-async: 3.4.2
As soon as I restart the tomcat service, everything starts working fine.
Please help, what could be the root cause of this issue.
P.S.: I am using DeferredResult and CompletableFuture to create Async REST API.
I have also tried using spring.mvc.async.request-timeout in application and configured asynTimeout in tomcat. But still getting same error.
It's probably obvious that Spring is timing out your requests and throwing AsyncRequestTimeoutException, which returns a 503 back to your client.
Now the question is, why is this happening? There are two possibilities.
These are legitimate timeouts. You mentioned that you only see the exceptions when the load on your server increases. So possibly your server just can't handle that load and its performance has degraded to the point where some requests can't complete before Spring times them out.
The timeouts are caused by your server failing to send a response to an asynchronous request due to a programming error, leaving the request open until Spring eventually times it out. It's easy for this to happen if your server doesn't handle exceptions well. If your server is synchronous, it's okay to be a little sloppy with exception handling because unhandled exceptions will propagate up to the server framework, which will send a response back to the client. But if you fail to handle an exception in some asynchronous code, that exception will be caught elsewhere (probably in some thread pool management code), and there's no way for that code to know that there's an asynchronous request waiting on the result of the operation that threw the exception.
It's hard to figure out what might be happening without knowing more about your application. But there are some things you could investigate.
First, try looking for resource exhaustion.
Is the garbage collector running all the time?
Are all CPUs pegged at 100%?
Is the OS swapping heavily?
If the database server is on a separate machine, is that machine showing signs of resource exhaustion?
How many connections are open to the database? If there is a connection pool, is it maxed out?
How many threads are running? If there are thread pools in the server, are they maxed out?
If something's at its limit then possibly it is the bottleneck that is causing your requests to time out.
Try setting spring.mvc.async.request-timeout to -1 and see what happens. Do you now get responses for every request, only slowly, or do some requests seem to hang forever? If it's the latter, that strongly suggests that there's a bug in your server that's causing it to lose track of requests and fail to send responses. (If setting spring.mvc.async.request-timeout appears to have no effect, then the next thing you should investigate is whether the mechanism you're using for setting the configuration actually works.)
A strategy that I've found useful in these cases is to generate a unique ID for each request and write the ID along with some contextual information every time the server either makes an asynchronous call or receives a response from an asynchronous call, and at various checkpoints within asynchronous handlers. If requests go missing, you can use the log information to figure out the request IDs and what the server was last doing with that request.
A similar strategy is to save each request ID into a map in which the value is an object that tracks when the request was started and what your server last did with that request. (In this case your server is updating this map at each checkpoint rather than, or in addition to, writing to the log.) You can set up a filter to generate the request IDs and maintain the map. If your filter sees the server send a 5xx response, you can log the last action for that request from the map.
Hope this helps!
Asynchroneus tasks are arranged in a queue(pool) which is processed in parallel depending on the number of threads allocated. Not all asynchroneus tasks are executed at the same time. Some of them are queued. In a such system getting AsyncRequestTimeoutException is normal behaviour.
If you are filling up the queues with asynchroneus tasks that are unable to execute under pressure. Increasing the timeout will only delay the problem. You should focus instead on the problem:
Reduce the execution time(through various optimizations) of asynchroneus task. This will relax the pooling of async tasks. It oviously requires coding.
Increase the number of CPUSs allocated in order to be able to run more efficiently the parallel tasks.
Increase the number of threads servicing the executor of the driver.
Mongo Async driver is using AsynchronousSocketChannel or Netty if Netty is found in the classpath. In order to increase the number of the worker threads servicing the async comunication you should use:
MongoClientSettings.builder()
.streamFactoryFactory(NettyStreamFactoryFactory(io.netty.channel.EventLoopGroup eventLoopGroup,
io.netty.buffer.ByteBufAllocator allocator))
.build();
where eventLoopGroup would be io.netty.channel.nio.NioEventLoopGroup(int nThreads))
on the NioEventLoopGroup you can set the number of threads servicing your async comunication
Read more about Netty configuration here https://mongodb.github.io/mongo-java-driver/3.2/driver-async/reference/connecting/connection-settings/
My code is written in jsp for importing product from CSV file .
The process is working fine for less then 300 - 400 products importing. But when I try with large number of rows like 1000 or more, it gives me below error because of its long process.
I am using Google App Engine with Google cloud SQL. This problem does not occur only with this product import system. But I observed many times, it is due to long process time execution.
Note : Working well in local system but problem occurs after deploying on google app server
500 Server Error
Error: Server Error
The server encountered an error and could not complete your request.Please try again in 30 seconds.
Is there any special configuration in jsp or servlet file ?
Is this issue related to any session expiry time ? (i also tried with session.setMaxInactiveInterval(500000))
Is there any configuration on google-app-engine ?
Any help would be appreciated.
Thanks in advance..!!
if your import takes longer than 60 seconds your issue is documented at [1].
The Google App Engine request timer (Java/Python/Go) ensures that requests have a finite lifespan and do not get caught in an infinite loop. Currently, the deadline for requests to frontend instances is 60 seconds. (Backend instances have no corresponding limit.) Every request, including warmup (request to /_ah/warmup) and loading requests ("loading_request=1" log header), is subject to this restriction.
If a request fails to return within 60 seconds and a DeadlineExceededError is thrown and not caught, the request is aborted and a 500 internal server error is returned. If the DeadlineExceededError is caught but a response is not produced quickly enough (you have less than a second), the request is aborted and a 500 internal server error is returned.
Hope this helps.
Ciao
Paolo
Links:
[1] - https://developers.google.com/appengine/articles/deadlineexceedederrors
So we run a Hibernate, Spring, Spring Webflow stack. From what I've read so far it might also be important to know we use c3p0-0.9.1.2.
Over the last couple of days we've noticed the server suddenly stop. Users cannot log into the website, nothing appears to happen, the browser simply sits loading the page forever. The server logs also simply halt.
When we notice this we shutdown the tomcat instance and all of a sudden quite a few of the following errors get logged;
13:05:57.492 [TP-Processor7] WARN o.h.util.JDBCExceptionReporter - SQL Error: 0, SQLState: null
13:05:57.492 [TP-Processor7] ERROR o.h.util.JDBCExceptionReporter - An SQLException was provoked by the following failure: java.lang.InterruptedException
Any ideas what these mean? Google hasn't been too helpful. Are we leaking db connections somewhere and the pool cannot gain a new session?
We have just put in a couple new Spring Webflow flows and are experiencing a slightly increased amount of website traffic but we haven't seen this behaviour before.
I suspect those InterruptExceptions come from the actual shutdown of those threads by the container, and simply indicate that those threads are existant when Tomcat shuts down.
Instead, I would grab a thread dump from Tomcat when it next freezes. I would also get a DBA to tell you what's happening in the database. From the above I'm guessing you're hung on a database resource, but a thread dump and analysis from a DBA will certainly point you in the right direction.
Here's a Thread Dump JSP as an alternative means of generating thread dumps.
I'm getting the following error:
javax.servlet.jsp.JspException: Broken pipe
Now I have seen questions/answers with respects to the socket exception, but this error is coming from a different package. Any help is greatly appreciated.
BTW, I am seeing quite a lot of these errors in a struts web app Weblogic Node logs and I am thinking that it has to do with end users closing their web browser before the page reloads/executes the next step (database transaction which takes quite a bit of time to execute, anywhere from 30 seconds to 4 mins).
I am thinking that it has to do with end users closing their web browser before the page reloads/executes the next step
You are entirely correct. This exception will be thrown when the client aborts the current request by navigating away, closing the tab/window, refreshing the request, etc while the request is still running. In other words, the client abruptly closed the connection and the server side can't write/flush any byte to it anymore. It has normally an IOException as the root cause, usually in flavor of a servletcontainer specific subclass like ClientAbortException in case of Tomcat and clones. If you investigate the entire stacktrace in the server logs, you'll find it somewhere at the bottom.
I am sure the underlying package uses pipes internally to transfer the result from a to b. Now B (the ServletOutputStream) closes, and the other end of the pipe notifies this by throwing this exception.
The HTTPRequest is handled by a chain of servlets which are connected to each other using pipes. When the browser abandons the connection and the socket gets closed, that is being caught by the servlet chaining managment layer. The servlet probably is indirectly catching the socket closed exception and is thowing it as the broken pipe. Look at any wrapped exception for more details.
It happens when the user clicks stop, or logs off, or otherwise prematurely aborts the connection, We can ignore this exception.