The problem:
I am having some strange behaviour from a Jetty server (rest over https) when some client connections are closed (client-side) before the server has had time to reply. Normally this is well managed and expected by a webserver/application server but in a specific instance something breaks the server that stops replying.
I am trying to reproduce programmatically and locally the issue, opening a client connection and closing it before the server has had time to reply, but I do not have much experience with a situation like this, normally the clients I write are expected to not die immediately.
I am not interested in the language/application I have to use to replicate my case, it can be a Java program, a netcat command, telnet, dotnetcore... The only limit I have is that it should run on a Kubernetes pod, if possible.
I am trying to use Java to open a socket then close it immediately, or to create an Http client and stop it immediately after a request sent, but with no luck at the moment.
At the same time I am looking at netcat, but I fear it's too low level for a rest request.
Related
I have a java client app and a java server app.
My client can have network slowdown.
My client performs SOAP webservices to my server app. The problem is that sometimes the client reach its timeout (40sec) because the network is really, really bad.
For the client app this request is a fail, and it retry the same call a bit later. But the server had already integrated the data from client, and I get violated keys error from my ORM.
I do not want to prolong the timeout on the client side.
My question is: when the client timeout, is there a way to rollback everything on the server side ?
Thanks
One of the options to solve it is to set some flag/status in the database when request is accepted by server. Something like inProcessing. And change this flag to Complete after successful data processing.
When client will retry the same call later you can check this flag and if flag=inProcessing or Complete don't do any date processing.
I have classic http client/server application where the server serves the clients data at their will but also performs some kind of call-backs to the list of clients' addresses it has. My two questions are :
1- How would the server know if a client is down (the client did not disconnect but the connection got suddenly interrupted) ?
2- Is there a way to know from the server-side if the process at client-side listening on the call-back port is still up (i.e. client call-back socket is still open) ?
1- How would the server know if a client is down (the client did not disconnect but the connection got suddenly interrupted) ?
Option #1: direct communication
Client tells server "I'm alive" at a periodic interval. You could make your client to ping your server at a configurable interval, and if the server does not receive the signal for a certain time, it'll mark the client as down. Client could even tell server more info(e.g. It's status) in each heartbeat if necessary, this is also the way used in many distributed systems(e.g. Hadoop/Hbase).
Option #2: distributed coordination service
You could treat all clients connected to a server as a group, and use a 3rd party distributed coordination service like Zookeeper to facilitate the membership management. Client registers itself to Zookeeper as a new member of the group right after booting up, and leaves the group if it's down. Zookeeper notifies the server whenever the membership changes.
2- Is there a way to know from the server-side if the process at client-side listening on the call-back port is still up (i.e. client call-back socket is still open) ?
I think this can only be done by the way Option #1 listed above. It could be either the way clients tell server "My callback port is OK" at a fixed interval, or the server asks clients "Are your callback port OK?" and wait its response at a fixed interval
You would have to establish some sort of protocol; and simply spoken: the server keeps track of "messages" that it tried to sent to clients.
If that "send" is acknowledged, fine; if not: then the server might do a limited number of retries; and then regard that client as "gone"; and then drop any other messages for that client.
1- How would the server know if a client is down (the client did not disconnect but the connection got suddenly interrupted) ?
A write to the client will fail.
2- Is there a way to know from the server-side if the process at client-side listening on the call-back port is still up (i.e. client call-back socket is still open
A write to the client will fail.
The write won't necessarily fail immediately, due to TCP buffering, but the write will eventually provoke retries and retry timeouts that will cause a subsequent read or write to fail.
In Java the failure will manifest itself as an IOException: connection reset.
I have a REST service that calls another remote service.
Most of the time the communication works fine, but occasionally, I encounter
org.apache.cxf.jaxrs.client.ClientWebApplicationException:
org.apache.cxf.interceptor.Fault: Could not send Message.
org.apache.cxf.interceptor.Fault: Could not send Message.
SocketException invoking https://url: Unexpected end of file from server
I did some research and found it's the remote server shut down the connection unexpectedly.
It really puzzles me, because everything (input, header, etc) is the same and I was testing with a small amount of requests only like (50-100), I have tried both in sequence and in parallel, only a few will encounter this issue.
Why would this happen? Do I have to ask the remote server to reconfigure, or do I have to implement a retry pattern in my service in this case?
Any hint?
Thanks
P.S
I am using
org.apache.cxf.jaxrs.client.WebClient
to invoke the remote service. Will it make any difference if I switch to HttpClient?
I am encountering an interesting issue wherein a TCP connection for a HTTP 1.1 POST request is being closed immediately following the request (ie, before the response can be sent by the server).
A few details about the test environment:
Client - Windows XP, Internet Explorer 8, Flash player 12.
Server - Java 7
Prior to the aforementioned behaviour, we have several longstanding TCP connections, each being reused for multiple HTTP requests; we open a long poll and when this poll completes, open another. We see several hours of well behaved and reused TCP connections opening polls as the previous poll closes.
Eventually -- sometimes after 12 or more hours of normal behaviour -- a poll on a long standing connection will send the HTTP POST and immediately send a TCP FIN before the server can write the response.
The client behaviour is to keep a poll open at all times, so at this point we try to open a new poll.
A new TCP connection is then opened by the client sending another HTTP POST, with the same behaviour; the request is sent, followed by a FIN from the client.
This behaviour can continue for several minutes, until the server can finally respond to kill the client. (The server detects the initial closed connection by encountering an IO Exception, the next time it can communicate with the client, the response is to tell the client to close)
Edit: We are opening connections only through the Flash client, and are not delving into low level TCP code. While Steffen Ullrich is correct, and the single sided shutdown is possible and should be dealt with, what is not clear is why a single sided shutdown is occurring at this (seemingly arbitrary) point. We are not calling close from the application to instigate this behaviour.
My questions are:
Under what circumstances would a TCP connection for a HTTP request be terminated prior to the response being received? I understand this is bad behaviour, and an incomplete HTTP transaction, so presumably something lower down is terminating the connection for an unknown reason.
Are there any diagnostics that could be used to help understand the problem? (We are currently monitoring server and client side activity with Wireshark.)
Notes:
In Wireshark, the behaviour we see is:
Longstanding TCP connection (#1) serving multiple HTTP requests.
HTTP request is made over #1.
Server ACKs the request.
Client sends FIN to close connection #1. Server responds with FIN,ACK. (The expected traffic would be the server sending the HTTP response). Around this point the server experiences an IO Exception.
Client opens connection #2 and sends HTTP request.
Behaviour continues as from 3.
Sending a request immediatly followed by a FIN is not a connection close, but shutdown of writing shutdown(socket,SHUT_WR). The client tells the server this way that it will not send any more data, but it might still receive data. It's not that uncommon.
I have been trying to figure out how to implement a simple xml rpc server and client with apache xml-rpc (http://ws.apache.org/xmlrpc/) but i haven't been successfull. I implemented a server and a client as specified here in the section Webserver: http://ws.apache.org/xmlrpc/server.html
The only thing i did differently was this "phm.addHandler("Calculator",org.apache.xmlrpc.demo.Calculator.class);" instead of this "phm.load(Thread.currentThread().getContextClassLoader(), "MyHandlers.properties");". When i start the server it start properly but I can't see the service when running netstat. Then when I run a request from the client it fails because it cannot find the class to call (i.e. the client doesn't really connect to the server). I have also observed that the client can start (not sending any requests) without any errors even when the server is not running.
Anyone have any ideas. I'm really lost.
Are you trying to run both client and server on the same computer? ...that may be the reason for netstat not working. (It cannot sniff packets on the loopback address.)
If you're still interested I can post a working example.