We Keep Coding

how java unirest to set content-length

java.
I use Unirest.post to post my multipart data. but server shows error to me:
multipart: NextPart: EOF.
I find that, if I set Content-Length I can solve this.
Here the code:
String buff = "my data";
HttpResponse<String> res = Unirest.post(url)
.header("Content-Type", multipart.getContentType().getValue())
.header("Content-Length", String.valueOf(buff.length()))
.body(buff).asString();
But after I add .header("Content-Length", String.valueOf(buff.length())), run java I get error:
org.apache.http.client.ClientProtocolException
How can I solve this?

You need to remove the earlier set content length so that you can set a new one.
#John Rix's answer This code helped solved the problem
private static class ContentLengthHeaderRemover implements HttpRequestInterceptor{
#Override
public void process(HttpRequest request, HttpContext context) throws HttpException, IOException {
request.removeHeaders(HTTP.CONTENT_LEN);// fighting org.apache.http.protocol.RequestContent's ProtocolException("Content-Length header already present");
}
}
HttpClient client = HttpClients.custom()
.addInterceptorFirst(new ContentLengthHeaderRemover())
.build();

OkHttp POST error "connection reset by peer" for unauthorized call and large payload

I've been struggling with the following issue:
I have a spring boot application which allows a user to post JSON content to an API endpoint. To use this endpoint, the user has to authenticate himself via basic authentication. Moreover, I use OkHttp (3.6.0) as an HTTP client.
Now if I post a large payload (> 4 MB) while being unauthorized, the OkHttp client fails with the following error:
java.net.SocketException: Connection reset by peer: socket write error
To reproduce the issue, I created a minimal client and server:
Server (Spring Boot Web App)
#SpringBootApplication
#RestController
public class App {
#PostMapping
public String create(#RequestBody Object obj) {
System.out.println(obj);
return "success";
}
public static void main(String[] args) {
SpringApplication.run(App.class);
}
}
Client (OkHttp 3.6.0)
public class Main {
public static void main(String[] args) {
OkHttpClient client = new OkHttpClient
.Builder()
.build();
Request request = new Request.Builder()
.url("http://localhost:8080")
.header("Content-Type", "application/json")
.post(RequestBody.create(MediaType.parse("application/json"), new File("src/main/java/content.json")))
// .post(RequestBody.create(MediaType.parse("application/json"), new File("src/main/java/content-small.json")))
.build();
try {
Response response = client.newCall(request).execute();
System.out.println(response);
} catch (IOException e) {
e.printStackTrace();
}
}
}
Instead of the previously mentioned exception ("java.net.SocketException: Connection reset by peer: socket write error"), I would expect the response to be a default error message with HTTP status code 401, e.g. {"timestamp":1508767498068,"status":401,"error":"Unauthorized","message":"Full authentication is required to access this resource","path":"/"}. This is the result I get when using cURL and Postman as clients.
When I'm using less payload (content-small.json; approx. 1KB) instead of the large payload (content.json; approx. 4881KB), I receive the expected response, i.e. Response{protocol=http/1.1, code=401, message=, url=http://localhost:8080/}.
The issue is actually embedded in a larger project with Eureka and Feign clients. Threfore, I would like to continue using OkHttp client and I need the expected behavior.
My problem analysis
Of course, I tried to solve this problem myself for quite some time now. The IOException occurs when the request body is written to the HTTP stream:
if (permitsRequestBody(request) && request.body() != null) {
Sink requestBodyOut = httpStream.createRequestBody(request, request.body().contentLength());
BufferedSink bufferedRequestBody = Okio.buffer(requestBodyOut);
request.body().writeTo(bufferedRequestBody);
bufferedRequestBody.close();
}
My assumption is that the server closes the connection as soon as it receives the headers (as the request is unauthorized), but the client continues trying to write to the stream although it is already closed.
Update
I've also implemented a simple client with Unirest which shows the same behavior. Implementation:
public class UnirestMain {
public static void main(String[] args)
throws IOException, UnirestException {
HttpResponse response = Unirest
.post("http://localhost:8080")
.header("Content-Type", "aplication/json")
.body(Files.readAllBytes(new File("src/main/java/content.json").toPath()))
// .body(Files.readAllBytes(new File("src/main/java/content-small.json").toPath()))
.asJson();
System.out.println(response.getStatus());
System.out.println(response.getStatusText());
System.out.println(response.getBody());
}
}
Expected output: {"path":"/","error":"Unauthorized","message":"Full authentication is required to access this resource","timestamp":1508769862951,"status":401}
Actual output: java.net.SocketException: Connection reset by peer: socket write error

Cucumber Test Step time of execution in Java

If i have Cucumber scenario like this :
Scenario: Correct response should be returned.
When User sent "xml example" request
Then Response should have some data "
And for this first step in ma TestStep class :
#When("^User sent \"(.*)\" request")
public void sendRequest(String requestName) throws Exception
{
//code for SOAP request
}
If service is not available, test will run for hours. I would like to stop it after some period of time. Any suggestions?

If your lib is not terminating request after provided timeout, you can use alternative approach with ExecutorService. The main idea is submit task with request to SOAP service to ExecutorService and then call get(time, TimeUnit.SECONDS) with timeout from Future.
Example:
Future<String> future = executor.submit(new SoapRequeset());
future.get(5L, TimeUnit.SECOND); //will throw exception if result not available after timeout

Solution with ExecutorService can be used but it was not acceptable for my project. I had to override method from my lib. This is what i did:
#Override
public SOAPMessage sendSOAPMessage(SOAPMessage request) throws Exception
{
SOAPConnection soapConnection = SOAPConnectionFactory.newInstance().createConnection();
URL endpoint =
new URL(new URL(serviceUrl), "endpoint spec.",
new URLStreamHandler()
{
#Override
protected URLConnection openConnection(URL url) throws IOException
{
URL target = new URL(url.toString());
URLConnection connection = target.openConnection();
// Connection settings
connection.setConnectTimeout(5000);
connection.setReadTimeout(5000);
return (connection);
}
});
return soapConnection.call(request, endpoint);
}
so when i called sendSOAPMessage my test step will break down after 5 seconds if there was no response in that period of time.

How to properly clean up after using an HttpClient in a unit test

In a unit test using an Apache HttpClient to fire requests, I have seen the following setup and cleanup code:
private HttpClient httpClient;
private HttpRequestBase httpRequest;
#Before
public void setUp() throws Exception {
httpClient = new DefaultHttpClient();
}
#After
public void closeRequests() {
if (httpRequest != null) {
httpRequest.releaseConnection();
httpRequest = null;
}
}
The tests than e.g. send get requests and check the response:
#Test
public void getSomething() throws Exception {
httpGet = new HttpGet("http://some/url");
HttpResponse response = httpclient.execute(httpGet);
assertThat(response.getStatusLine().getStatusCode(), is(HttpStatus.SC_OK));
}
Now my question is: Do these tests properly clean up after themselves? From what I understand, the releaseConnection() call only hands back the connections to the client's connection manager but doesn't actually close it.
So shouldn't the tests rather do this:
#After
public void closeConnections() {
httpClient.getConnectionManager().shutdown();
}
And would this properly close all connections even without calling releaseConnection() on the http request instances?

Yes, you should. Shutting down the connection manager (or closing the client with HC 4.3 and newer) is the right thing to do in integration tests.

Getting a ConnectionClosedException with Http.Core talking to Http.Client 4?

I am trying my hand at using http.core & client 4.3. In general it works well, and is quite pleasant to deal with. However, I am getting a ConnectionClosedException on one of my transfers and I can't see why. Others work just fine as far as I can tell.
Everything follows the examples in a pretty straight forward way. If it didn't, it was re-written to as much as possible in an effort to get rid of this.
There are 2 servers, both running the same code [A & B]
A HttpClient sends a request "AX" (POST) to B
B HttpService receives the "AX" post, processes it
B HttpClient sends a reply "BR" (POST) to A on a different port
Later This should happen after the connection to A is closed, or as close as possible
Right now the code doesn't actually care
A receives the reply from B (on a different thread) and does things
In the problem scenario, A is running as the server, and B is sending a POST. Sorry it isn't always clear, since in one transaction both sides end up running server and client code.
A Sends POST to B:8080. Get back a proper response inline, everything ok.
POST Connection to B:8080 gets closed properly
B sends new POST (like an ACK) to A (ex... B:53991 => A:9000).
A Processs everything. No issues
A rasies ConnectionClosedException
Since I don't why it's happening for sure, I tried to put everything I think is relevant in there. My only thought right now is that it has something to with making sure I add/change connection control headers, but I can't see how that would affect anything.
Stack Trace from machine "A", when the reply from B comes
org.apache.http.ConnectionClosedException: Client closed connection
at org.apache.http.impl.io.DefaultHttpRequestParser.parseHead(DefaultHttpRequestParser.java:133)
at org.apache.http.impl.io.DefaultHttpRequestParser.parseHead(DefaultHttpRequestParser.java:54)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
at org.apache.http.impl.DefaultBHttpServerConnection.receiveRequestHeader(DefaultBHttpServerConnection.java:131)
at org.apache.http.protocol.HttpService.handleRequest(HttpService.java:307)
at com.me.HttpRequestHandlerThread.processConnection(HttpRequestHandlerThread.java:45)
at com.me.net.http.HttpRequestHandlerThread.run(HttpRequestHandlerThread.java:70)
com.me.ExceptionHolder: Client closed connection
at com.me.log.Log.logIdiocy(Log.java:77)
at com.me.log.Log.error(Log.java:54)
at com.me.net.http.HttpRequestHandlerThread.run(HttpRequestHandlerThread.java:72)
Caused by: org.apache.http.ConnectionClosedException: Client closed connection
at org.apache.http.impl.io.DefaultHttpRequestParser.parseHead(DefaultHttpRequestParser.java:133)
at org.apache.http.impl.io.DefaultHttpRequestParser.parseHead(DefaultHttpRequestParser.java:54)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
at org.apache.http.impl.DefaultBHttpServerConnection.receiveRequestHeader(DefaultBHttpServerConnection.java:131)
at org.apache.http.protocol.HttpService.handleRequest(HttpService.java:307)
at com.me.net.http.HttpRequestHandlerThread.processConnection(HttpRequestHandlerThread.java:45)
at com.me.net.http.HttpRequestHandlerThread.run(HttpRequestHandlerThread.java:70)
This is the code running on B, the "client" in this scenario. It is trying to POST the reply acknowledging that the first POST from A was received properly. There really isn't much to transmit, and the response should only be an HTTP 200:
try (CloseableHttpClient client = HttpClients.createDefault()) {
final HttpPost post = new HttpPost(url);
post.setHeaders(/* create application specific headers */);
ByteArrayEntity entity = new ByteArrayEntity(IOUtils.toByteArray(myStream));
post.setEntity(entity);
ResponseHandler<Void> responseHandler = new ResponseHandler<Void>() {
#Override
public Void handleResponse(HttpResponse response) throws ClientProtocolException, IOException {
StatusLine status = response.getStatusLine();
if (!NetUtil.isValidResponseCode(response)) {
throw new ClientProtocolException("Unexpected Error! Oops");
}
// consume the response, if there is one, so the connection will close properly
EntityUtils.consumeQuietly(response.getEntity());
return null;
}
};
try {
client.execute(post, responseHandler);
} catch (ClientProtocolException ex) {
// logic to queue a resend for 10 minutes later. not triggered
throw ex;
}
}
On A: This is called async because the response doesn't come in over the same http connection.
The main request handler does a lot more work, but it is amazing how little code there is actually controlling the HTTP in the handler/server side. Great library... that I am misusing somehow. This is the actual handler, with everything simplified a bit, validation removed, etc.
public class AsyncReceiverHandler implements HttpRequestHandler {
#Override
public void handle(HttpRequest request, HttpResponse response, HttpContext context) throws HttpException, IOException {
// error if not post, other logic. not touching http. no errors
DefaultBHttpServerConnection connection = (DefaultBHttpServerConnection) context.getAttribute("connection");
Package pkg = NetUtil.createPackageFrom(connection); // just reads sender ip/port
NetUtil.copyHttpHeaders(request, pkg);
try {
switch (recieive(request, pkg)) {
case EH_OK:
response.setStatusCode(HttpStatus.SC_OK);
break;
case OHNOES_BAD_INPUT:
response.setStatusCode(HttpStatus.SC_BAD_REQUEST);
response.setEntity(new StringEntity("No MDN entity found in request body"));
// bunch of other cases, but are not triggered. xfer was a-ok
}
} catch (Exception ex) {
//log
}
}
private MyStatus receiveMdn(HttpRequest request, Package pkg) throws Exceptions..., IOException {
// validate request, get entity, make package, no issues
HttpEntity resEntity = ((HttpEntityEnclosingRequest) request).getEntity();
try {
byte[] data = EntityUtils.toByteArray(resEntity);
// package processing logic, validation, fairly quick, no errors thrown
} catch (Exceptions... ex) {
throw ExceptionHolder(ex);
}
}
}
This is the request handler thread. This and the server are taken pretty much verbatim from the samples. The service handler just starts the service and accept()s the socket. When it gets one, it creates a new copy of this, and calls start():
public HttpRequestHandlerThread(final HttpService httpService, final HttpServerConnection conn, HttpReceiverModule ownerModule) {
super();
this.httpService = httpService;
this.conn = (DefaultBHttpServerConnection) conn;
}
private void processConnection() throws IOException, HttpException {
while (!Thread.interrupted() && this.conn.isOpen()) {
/* have the service create a handler and pass it the processed request/response/context */
HttpContext context = new BasicHttpContext(null);
this.httpService.handleRequest(this.conn, context);
}
}
#Override
public void run() {
// just runs the main logic and reports exceptions.
try {
processConnection();
} catch (ConnectionClosedException ignored) {
// logs error here (and others).
} finally {
try { this.conn.shutdown(); } catch (IOException ignored) {}
}
}
}

Well, this seems stupid now, and really obvious. I ignored the issue for a while and moved on to other things, and the answer bubbled up from the subconscious, as they will.
I added this header back and it all cleared up:
post.setHeader("Connection", "close, TE")
Somehow the line to set the Connection header got removed, probably accidentally by me. A lot of them get set, and it was still there, just wrong in this code path. Basically, the server expects this connection to close immediately but the header was reverting to the default keep-alive. Since the client closes the connection as soon as it is done with it this was surprising the server, who was told otherwise, and rightly compliained :D In the reverse path everything was OK.
Since I had just changed the old stack to use HttpComponents I didn't look at headers and such, and I just assumed I was using it wrong. The old stack didn't mind it.