I've created a simple server Java application using Gradle. As an embedded server , I am using Jetty . I am also using a Gretty plugin as it supports the latest Jetty version.
The project runs just fine . And I have tried to stress test it. As a part of my test I need to check the response time and therefore I am sending "Connection:Close" header via curl.
My response is a long JSON string , and I see only part of it , after which the connection hangs . I would like to know Why is it happening , and how can I work around it.
NOTE :
When sending Connection:Keep-alive header , everything is fine
When response from the server is not a long string , but smaller . It works just fine (doesn't hang)
Tried the standard Jetty plugin from gradle , the result was the same.
HOW TO TEST :
Build and run my project from console ./gradlew appRun
From bash console run curl -H "Connection:Close" -i "http://localhost:8080/Environment/example"
See the partial response and the connection still alive...
Seems like you are confusing the persistent connection modes between HTTP/1.0 and HTTP/1.1.
Either that, or you are using a really old version of curl that still defaults to HTTP/1.0.
HTTP/1.0 has no persistent connections by default, so to use persistent connections we send Connection: keep-alive.
HTTP/1.1 uses persistent connections by default, so to disable it we can send Connection: close
Using HTTP/1.0, with Connection: close is like sending this ...
GET /Environment/example HTTP/1.0
Host: localhost:8080
Connection: close
... which produces an invalid header value for Connection per the HTTP/1.0 spec
Lets use the verbose features of curl to see what's really going on Connection wise...
Example: HTTP/1.1 with normal operation:
$ curl --verbose --http1.1 http://apache.org/ -so /dev/null
* Trying 88.198.26.2...
* Connected to apache.org (88.198.26.2) port 80 (#0)
> GET / HTTP/1.1
> Host: apache.org
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 06 May 2016 12:05:39 GMT
< Server: Apache/2.4.7 (Ubuntu)
< Last-Modified: Fri, 06 May 2016 11:10:20 GMT
< ETag: "cf64-5322a812896a8"
< Accept-Ranges: bytes
< Content-Length: 53092
< Vary: Accept-Encoding
< Cache-Control: max-age=3600
< Expires: Fri, 06 May 2016 13:05:39 GMT
< Content-Type: text/html
<
{ [1125 bytes data]
* Connection #0 to host apache.org left intact
Notice that it says it kept the connection intact?
Example: HTTP/1.1 with manual Connection: close operation:
$ curl --verbose --http1.1 --header "Connection: close" http://apache.org/ -so /dev/null
* Trying 140.211.11.105...
* Connected to apache.org (140.211.11.105) port 80 (#0)
> GET / HTTP/1.1
> Host: apache.org
> User-Agent: curl/7.43.0
> Accept: */*
> Connection: close
>
< HTTP/1.1 200 OK
< Date: Fri, 06 May 2016 12:06:35 GMT
< Server: Apache/2.4.7 (Ubuntu)
< Last-Modified: Fri, 06 May 2016 11:10:20 GMT
< ETag: "cf64-5322a812896a8"
< Accept-Ranges: bytes
< Content-Length: 53092
< Vary: Accept-Encoding
< Cache-Control: max-age=3600
< Expires: Fri, 06 May 2016 13:06:35 GMT
< Connection: close
< Content-Type: text/html
<
{ [1106 bytes data]
* Closing connection 0
Ah, the HTTP response headers say that the server will close, and curl saw the connection being closed. What we wanted.
Example: HTTP/1.0 with normal operation:
$ curl --verbose --http1.0 http://apache.org/ -so /dev/null
* Trying 140.211.11.105...
* Connected to apache.org (140.211.11.105) port 80 (#0)
> GET / HTTP/1.0
> Host: apache.org
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 06 May 2016 12:08:27 GMT
< Server: Apache/2.4.7 (Ubuntu)
< Last-Modified: Fri, 06 May 2016 11:10:20 GMT
< ETag: "cf64-5322a812896a8"
< Accept-Ranges: bytes
< Content-Length: 53092
< Vary: Accept-Encoding
< Cache-Control: max-age=3600
< Expires: Fri, 06 May 2016 13:08:27 GMT
< Connection: close
< Content-Type: text/html
<
{ [4002 bytes data]
* Closing connection 0
See how the HTTP response headers say that the server will close?
Curl also saw the connection being closed.
That's what we expect with normal HTTP/1.0 operation.
Example: HTTP/1.0 with persistent connection:
$ curl --verbose --http1.0 --header "Connection: keep-alive" http://apache.org/ -so /dev/null
* Trying 88.198.26.2...
* Connected to apache.org (88.198.26.2) port 80 (#0)
> GET / HTTP/1.0
> Host: apache.org
> User-Agent: curl/7.43.0
> Accept: */*
> Connection: keep-alive
>
< HTTP/1.1 200 OK
< Date: Fri, 06 May 2016 12:08:37 GMT
< Server: Apache/2.4.7 (Ubuntu)
< Last-Modified: Fri, 06 May 2016 11:10:20 GMT
< ETag: "cf64-5322a812896a8"
< Accept-Ranges: bytes
< Content-Length: 53092
< Vary: Accept-Encoding
< Cache-Control: max-age=3600
< Expires: Fri, 06 May 2016 13:08:37 GMT
< Keep-Alive: timeout=30, max=100
< Connection: Keep-Alive
< Content-Type: text/html
<
{ [3964 bytes data]
* Connection #0 to host apache.org left intact
Yup, the server indicates that it will use Keep-Alive too (per HTTP/1.0 spec), and curl even concurs and says the connection is left intact.
Related
Using certbot fails to generate certificate with this error:
org.shredzone.acme4j.exception.AcmeException: Failed to pass the challenge for domain www.
mysampledomain123.com, ... Giving up.
I manually checked the challenge file and got
http://www.mysampledomain123.com/.well-known/acme-challenge/jU--PkDrn5tDZw2RN6NNJHbPD00ovHFkLFvN3mJdeQX
Inside the file:
jU--PkDrn5tDZw2RN6NNJHbPD00ovHFkLFvN3mJdeQX.tuMr-UijwpsJ1KVZkdWTYgodWZ2SxxKdB7_CMAAEfpg
And here's the complete HTTP response header:
Accept-Ranges: bytes
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/plain;charset=iso-8859-1
Date: Sun, 16 Feb 2020 14:15:22 GMT
Server: nginx/1.14.0 (Ubuntu)
Transfer-Encoding: chunked
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
X-Powered-By: MyServer
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 999
X-RateLimit-Reset: 0
I'm wondering whether the problem is with the HTTP response headers or the content itself.
Any ideas would be appreciated.
I'm am developing an application with the Jersey framework that acts as a proxy to the docker HTTP API. I'm able to upload images successfully to Docker but I am unable to forward the response from Docker to the client without an error. I believe it has something to do with the Transfer-Encoding=[chunked] header in the response.
Response
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> POST /proxy/images/load HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:8080
> Accept: */*
> Content-Type: application/octet-stream
> Content-Length: 477637632
> Expect: 100-continue
>
< HTTP/1.1 100
< HTTP/1.1 200
< Transfer-Encoding: chunked
< Docker-Experimental: false
< Api-Version: 1.26
< Server: Docker/1.13.1 (linux)
< Date: Mon, 23 Jul 2018 11:20:14 GMT
< Content-Type: application/json
< Content-Length: 54
<
* Problem (2) in the Chunked-Encoded data
* Closing connection 0
curl: (56) Problem (2) in the Chunked-Encoded data
code
#POST
#Path("/load2")
//#Consumes(MediaType.APPLICATION_OCTET_STREAM)
public Response loadimages(byte[] input, #Context UriInfo uriInfo) {
WebTarget webTarget = ClientBuilder.newClient().target("http://127.0.0.1:5555/images").path("load");
Response response = webTarget.request(MediaType.APPLICATION_JSON).post(Entity.entity(input, MediaType.APPLICATION_OCTET_STREAM));
return response;
}
I've written a query tool which at it's heart uses this block to gather info from an external url:
AsyncHttpClientConfig proxiedCF = new DefaultAsyncHttpClientConfig.Builder().setUserAgent(pickUserAgent()).build();
AsyncHttpClient asyncHttpClient = new DefaultAsyncHttpClient(proxiedCF);
Future<Response> f = asyncHttpClient.prepareGet(url).setProxyServer(new ProxyServer.Builder(pickProxyServer(), 80)).execute();
It works fine. However it works even when an invalid proxy is provided, which is a bit suspicious and I feel my elaborate proxy configuration is not used at all.
I stumbled upon this by having pickProxyServer() return a String "1.1.1.1", which is obviously not a valid web proxy.
I use SLF4J for logging and it looks pretty normal:
20:31:47.454 [AsyncHttpClient-7-1] DEBUG o.a.n.channel.NettyConnectListener - Using new Channel '[id: 0x03359938, L:/10.0.0.101:59775 - R:/1.1.1.1:80]' for 'GET' to '[[url removed by me]]'
20:31:47.586 [AsyncHttpClient-7-1] DEBUG o.a.netty.handler.HttpHandler -
Request DefaultFullHttpRequest(decodeResult: success, version: HTTP/1.1, content: EmptyByteBufBE)
GET [[url removed by me]] HTTP/1.1
Host: [[url removed by me]]
Accept: */*
User-Agent: burning_dandelion
Response DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
HTTP/1.1 200 OK
Date: Fri, 13 Oct 2017 18:31:49 GMT
Expires: Fri, 13 Oct 2017 18:31:49 GMT
Cache-Control: private, max-age=3600
Content-Type: text/xml; charset=ISO-8859-1
P3P: CP="This is not a P3P policy!"
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2017-10-13-18; expires=Fri, 20-Oct-2017 18:31:49 GMT; path=/; domain=[[url removed by me]]
Set-Cookie: NID=114=qiVBv02cmXYHh2RfLQbhBfESWIoaGlf3d2jlSbAdQ8yWPDsCpOeK9aYbvfq0HWsER68W1oE53jiriM_fivTc1bJi1F2sfCi0wMptKI-9U3ueVKITtFvYYZx2T0rJf1kQ; expires=Sat, 14-Apr-2018 18:31:49 GMT; path=/; domain=[[url removed by me]]; HttpOnly
Accept-Ranges: none
Vary: Accept-Encoding
Transfer-Encoding: chunked
20:31:47.587 [AsyncHttpClient-7-1] DEBUG o.a.netty.channel.ChannelManager - Adding key: ProxyPartitionKey(proxyHost=1.1.1.1, proxyPort=80, secured=false, targetHostBaseUrl=[[url removed by me]]:80 for channel [id: 0x03359938, L:/10.0.0.101:59775 - R:/1.1.1.1:80]
Can someone point me towards my error? Obviously, I want an I/O exception or any kind of notice when an invalid proxy is called upon.
Ok it's the middle of the night but I found something. When the port is anything but :80, the proxy is correctly used. This does the trick for me.
I have implemented an application on GAE which can be accessible through https://<my_app_id>.appspot.com. Now I have a custom domain registered with Register.com. As described in GAE documentation I have mapped my custom domain to https://<my_app_id>.appspot.com and I see my application getting served from my custom domain. But I see requests are failed with error "SSL required to perform this operation". But I don't have any SSL certificate. And Can I serve my application without SSL? I mean just using http
UPDATED:
Response on executing url curl -v example.org/_ah/api/myapi/v1/package/ -o /dev/null
* Adding handle: conn: 0x1fa7e80
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x1fa7e80) send_pipe: 1, recv_pipe: 0
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to connect() to mydomain.com port 80 (#0)
* Trying ipaddress...
* Connected to mydomain.com (ipaddress) port 80 (#0)
> GET /_ah/api/myapp/v1/package/ HTTP/1.1
> User-Agent: curl/7.30.0
> Host: mydomain.com
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: Fri, 01 Jan 1990 00:00:00 GMT
< Date: Tue, 23 Jun 2015 12:26:50 GMT
< Vary: X-Origin
< Content-Type: text/html; charset=UTF-8
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
* Server GSE is not blacklisted
< Server: GSE
< Alternate-Protocol: 80:quic,p=0
< Accept-Ranges: none
< Vary: Origin,Accept-Encoding
< Transfer-Encoding: chunked
<
{ [data not shown]
Warning: Failed to create the file /dev/null: No such file or directory
* Failed writing body (0 != 9)
* Failed writing data
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
* Closing connection 0
curl: (23) Failed writing body (0 != 9)
There two things you've missed from docs:
Google Cloud Endpoints requires SSL. If you need to access your backend API in a system not supporting SSL, you'll need to either update the system to support SSL or use a proxy.
and
Google Cloud Endpoints does not support custom domains.
See https://cloud.google.com/appengine/docs/java/endpoints/
I'm doing a POST to a restlet and need to return a zip file. But although the created file is zip, the method returns gibberish.
I tried wrapping the FileRepresentation as was suggested here:
new org.restlet.engine.application.EncodeRepresentation(org.restlet.data.Encoding.ZIP, representation);
And also tried adding a Produces annotation like this:
#Produces({"application/x-zip-compressed"})
But neither works. The representation returns as gibberish string, and the Content-Type header stays application/octet-stream. What am I missing?
These are the request headers. Note the Accept-Encoding: gzip, deflate:
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36
Origin: chrome-extension://hgmloofddffdnphfgcellkdfbfbjeloo
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8,he;q=0.6
The response headers:
Vary: Accept-Encoding
Last-Modified: Wed, 06 May 2015 14:49:03 GMT
Content-Disposition: attachment; filename=_backup_20150506.zip; size=162191
Date: Wed, 06 May 2015 14:49:03 GMT
Accept-Ranges: bytes
Server: Restlet-Framework/2.2.1
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Set-Cookie: JSESSIONID=5F10BBBDC58D5C3D6C0474FA12C44FB9; Path=/AppName/; Domain=localhost
Content-Encoding: gzip
Content-Type: application/octet-stream
Transfer-Encoding: chunked
EDIT: I also tried changing the media type when creating the representation:
MediaType mt = MediaType.APPLICATION_ZIP;
FileRepresentation fr = new FileRepresentation(file, mt);
The response content type changed to Content-Type: application/zip but the returned value is still a gibberish string.
The right way to do that is what you used:
public class MyServerResource extends ServerResource {
#Post
public Representation test(Representation repr) {
FileRepresentation outputRepresentation
= new FileRepresentation(new File("(...)"),
MediaType.APPLICATION_ZIP);
return outputRepresentation;
}
}
So this should work.
Using curl with such code, here is what I have:
$ curl -X POST http://localhost:8182/test > mycontent.zip
$ unzip mycontent.zip
Archive: mycontent.zip
extracting: test.txt
In addition, here is what I have with the verbose mode of curl:
curl -X POST --verbose http://localhost:8182/test
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8182 (#0)
> POST /test HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:8182
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-type: application/zip
< Last-modified: Thu, 07 May 2015 08:08:59 GMT
< Content-length: 134
* Server Restlet-Framework/2.3.1 is not blacklisted
< Server: Restlet-Framework/2.3.1
< Accept-ranges: bytes
< Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
< Date: Thu, 07 May 2015 08:19:26 GMT
<
Notice that you can use the header Disposition if you want to configure hints within the download dialog of your browser.
Otherwise, "enable GZIP compression of the JSON response entity on Resltet" corresponds to automatic compression of the whole response content by Restlet. Browsers support this and can directly uncompress the content before displaying it. I don't think that isn't really what you need / expect. If it's the case, you could be interested in this link: https://templth.wordpress.com/2015/02/23/optimizing-restlet-server-applications/.
Hope it helps you,
Thierry