We Keep Coding

Retrofit client and response with Transfer-Encoding: chunked

I'm working on a Android sample app that get film list from http://www.omdbapi.com/.
The REST service is:
http://www.omdbapi.com/?s=star&apikey=d497e644
I'm using Retrofit to write the client.
public interface OmdbApi {
#Streaming
#GET("./")
#Headers({"Cache-control: no-cache"})
Call<Search> search(#Query("s") String search, #Query("apikey") String apiKey);
#Streaming
#GET("./")
#Headers({"Cache-control: no-cache"})
Call<FilmDetail> getFilm(#Query("i") String uid, #Query("apikey") String apiKey);
}
The complete source code is available here.
When I run the application, I obtain the following response (taken from logcat):
OK http://www.omdbapi.com/?s=star&apikey=d497e644 (108ms)
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Cache-Control: public, max-age=86400
Expires: Sat, 26 May 2018 14:28:18 GMT
Last-Modified: Fri, 25 May 2018 05:39:04 GMT
Vary: *, Accept-Encoding
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Access-Control-Allow-Origin: *
CF-Cache-Status: HIT
Server: cloudflare
CF-RAY: 4208b00c817b3db9-MXP
Connection: Keep-Alive
And the following error:
java.net.ProtocolException: unexpected end of stream
at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:455)
at okio.RealBufferedSource.read(RealBufferedSource.java:47)
okio.RealBufferedSource.exhausted(RealBufferedSource.java:57)
okio.InflaterSource.refill(InflaterSource.java:102)
okio.InflaterSource.read(InflaterSource.java:62)
okio.GzipSource.read(GzipSource.java:80)
okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:237)
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
at okhttp3.RealCall.execute(RealCall.java:77)
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180)
retrofit2.ExecutorCallAdapterFactory$ExecutorCallbackCall.execute(ExecutorCallAdapterFactory.java:91)
com.abubusoft.filmfinder.service.repository.FilmRepository.lambda$findFilm$0$FilmRepository(FilmRepository.java:18)
com.abubusoft.filmfinder.service.repository.FilmRepository$$Lambda$0.run(Unknown Source:20)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
at java.lang.Thread.run(Thread.java:764)
After a bit of investigation, I found that the problem is that the response has Transfer-Encoding: chunked. How can I resolve this problem?
Thank you.

After more investigation:
I remove the #Streaming annotations.. they aren't useful.
Tests that I did was on an emulator on a PC behind a firewall.
OkHttp had some problem about it many years ago, but now chunked transfer is completely managed.
On the same machine, invoking the same URL in a browser does not present any problems
I finally try on another machine, with a JUnit test and in Android emulator and I had no problem.
At last, I'm sure that code works, so is the environment that I used to develop the app has some problem with transfer-encoding chunked.
Complete source code is available here:
android app
junit test
I have to find exactly what is the problem, I'm sure now that it not Retrofit or my client definition or OkHttp.
I hope my experience can help other developers.
Byez

Google Drive API responds with 416 HTTP code

For some reason the previously working code stopped working and server started to respond with 416.
Here are the logs of HTTP client during failing interaction:
-------------- REQUEST --------------
GET https://www.googleapis.com/drive/v3/files/0B02Nopv3SQOvOVNKaDIwTEZ3MHd?alt=media
Accept-Encoding: gzip
Authorization: <Not Logged>
Range: bytes=0-33554431
User-Agent: My app Google-API-Java-Client Google-HTTP-Java-Client/1.22.0 (gzip)
-------------- RESPONSE --------------
HTTP/1.1 416 Requested range not satisfiable
Alt-Svc: quic=":443"; ma=2592000; v="39,38,37,35"
Server: UploadServer
Cache-Control: private, max-age=0
Content-Range: bytes */0
X-GUploader-UploadID: AEnB2UqBx9B09Lnr8tG761gdoz3DkhHSNO_OzHh1LkU6B2908v17rnBGQZSNW4ZVTjbRdFtvPWWIqZGdtSrTo6ZWN7YW9nxf6d
Vary: X-Origin
Vary: Origin
Expires: Mon, 11 Sep 2017 15:23:20 GMT
Content-Length: 225
Date: Mon, 11 Sep 2017 15:23:20 GMT
Content-Type: application/json; charset=UTF-8
I was trying to download a file which is around 200000 bytes, so I thought meaning of "chuck size" changed somewhere, so it could not give 33554431 bytes of a 282177 byte file. Tried changing that to a smaller value, but no success.
Drive.Files.Get get = drive.files().get(file.getId())
MediaHttpDownloader downloader = get.getMediaHttpDownloader()
downloader.directDownloadEnabled = false
localFile.newOutputStream()
get.executeMediaAndDownloadTo(stream)
Direct download does not work either, it just downloads "0" bytes.
Does anyone know how to overcome this issue?

416 Range Not
Satisfiable
error means the server is not able to serve the requested ranges. The
most likely reason is that the document doesn't contain such ranges,
or that the Range header value, though syntactically correct, doesn't
make sense.
One of the resolutions that may provide from this forum is to:
Add "Accept-Ranges: none" to our response headers.

It appeared to be a web interface when using Firefox. It uploaded "empty" files in certain cases.
https://productforums.google.com/forum/#!topic/drive/S03wEknc75g;context-place=forum/drive

Json doesn't work with specified URL

I am a new programmer i am trying to build an app with Json.
If i use this URL doesn't work . http://zsuzsafodraszat.hostzi.com/boltok.json
if i Use this, my app working. https://api.myjson.com/bins/3zm8i
Both Json files exactly the same.
Can you help me what i am doing wrong ? Maybe bad extension or web000 is not a good service for Json ? Can you give me some good free json hosting ? Thanks

Those 2 urls do not have the same content or the same headers. You can see this if run curl commands from the command line:
$ curl -i "http://zsuzsafodraszat.hostzi.com/boltok.json"
HTTP/1.1 200 OK
Date: Wed, 13 Apr 2016 22:52:50 GMT
Server: Apache
Last-Modified: Wed, 13 Apr 2016 16:48:23 GMT
Accept-Ranges: bytes
Content-Length: 1020
Connection: close
Content-Type: application/json
??{"Aldi":"http://catalog.aldi.com/emag/hu_HU/print/Online_katalogus_04_07/Online_katalogus_04_07.pdf",
"Lidl":"http://www.lidl.hu/statics/lidl-hu/ds_doc/HU_HHZ_kw14_2016.pdf",
"Spar":"http://ajanlatok.spar.hu/view/download/?d=1279",
"Penny":"https://view.publitas.com/16538/136265/pdfs/016f82fb5b00bc97b5a8c35f512d89b01cd3e3ce.pdf",
"Coop":"https://view.publitas.com/2556/133497/pdfs/16603d7e9bf30e8a8a4efec7f01d3fa2caf92fe0.pdf",
"Auchan":"http://www.lidl.hu/statics/lidl-hu/ds_doc/HU_HHZ_kw14_2016.pdf"}
$ curl -i "https://api.myjson.com/bins/3zm8i"
HTTP/1.1 200 OK
Server: nginx/1.5.8
Date: Wed, 13 Apr 2016 22:52:56 GMT
Content-Type: application/json
Content-Length: 500
Connection: keep-alive
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
{"Aldi":"http://catalog.aldi.com/emag/hu_HU/print/Online_katalogus_04_07/Online_katalogus_04_07.pdf","Lidl":"http://www.lidl.hu/statics/lidl-hu/ds_doc/HU_HHZ_kw14_2016.pdf","Spar":"http://ajanlatok.spar.hu/view/download/?id=1279","Penny":"https://view.publitas.com/16538/136265/pdfs/016f82fb5b00bc97b5a8c35f512d89b01cd3e3ce.pdf","Coop":"https://view.publitas.com/2556/133497/pdfs/16603d7e9bf30e8a8a4efec7f01d3fa2caf92fe0.pdf","Auchan":"http://www.lidl.hu/statics/lidl-hu/ds_doc/HU_HHZ_kw14_2016.pdf"}
As you can see, one of them has a couple of junk bytes at the beginning that my terminal is displaying as question marks. Also the http headers are different. The Content-Lengths are wildly different too. Did you use something other than a plain text editor to create the json payload in the failing example?
Try removing the junk characters and adding these http headers:
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

netty: the http file example can not work with apache bench when keep-alive is enabled

The http file example under test:
https://github.com/netty/netty/tree/4.1/example/src/main/java/io/netty/example/http/file
I compiled the above example with netty-4.1.0.beta8.
My test result:
$ ab -k -n 2 -c 1 -v 6 http://127.0.0.1:8080/test.sh
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 127.0.0.1 (be patient)...INFO: POST header ==
---
GET /test.sh HTTP/1.0
Connection: Keep-Alive
Host: 127.0.0.1:8080
User-Agent: ApacheBench/2.3
Accept: */*
---
LOG: header received:
HTTP/1.1 200 OK
content-length: 462
content-type: application/octet-stream
date: Fri, 26 Feb 2016 06:34:52 GMT
expires: Fri, 26 Feb 2016 06:35:52 GMT
cache-control: private, max-age=60
last-modified: Fri, 19 Feb 2016 02:35:40 GMT
connection: keep-alive
LOG: Response code = 200
LOG: header received:
**MY TEST.SH CONTENT**
WARNING: Response code not 2xx (500)
apr_poll: The timeout specified has expired (70007)
Total of 1 requests completed
It seemed that my test.sh content is considered to be the second request's headers.
My further test:
$ ab -k -n 1 -c 1 -v 6 http://127.0.0.1:8080/test.sh
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 127.0.0.1 (be patient)...INFO: POST header ==
---
GET /test.sh HTTP/1.0
Connection: Keep-Alive
Host: 127.0.0.1:8080
User-Agent: ApacheBench/2.3
Accept: */*
---
LOG: header received:
HTTP/1.1 200 OK
content-length: 462
content-type: application/octet-stream
date: Fri, 26 Feb 2016 06:39:02 GMT
expires: Fri, 26 Feb 2016 06:40:02 GMT
cache-control: private, max-age=60
last-modified: Fri, 19 Feb 2016 02:35:40 GMT
connection: keep-alive
LOG: Response code = 200
..done
Server Software:
Server Hostname: 127.0.0.1
Server Port: 8080
Document Path: /test.sh
Document Length: 0 bytes
Concurrency Level: 1
Time taken for tests: 0.005 seconds
Complete requests: 1
Failed requests: 0
Write errors: 0
Keep-Alive requests: 1
Total transferred: 263 bytes
HTML transferred: 0 bytes
Requests per second: 209.25 [#/sec] (mean)
Time per request: 4.779 [ms] (mean)
Time per request: 4.779 [ms] (mean, across all concurrent requests)
Transfer rate: 53.74 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 5 5 0.0 5 5
Waiting: 5 5 0.0 5 5
Total: 5 5 0.0 5 5
You can see the document length is 0 byte.
I run similar commands against www.google.com and it works fine.
Could you please help on this? Thanks in advance.

This is a bug in ApacheBench.
ApacheBench does not follow the http specifications fully, and assumes headers have a certain capitalization. Because the headers returned by the netty application doesn't have the character case ApacheBench expects, it assumes a default of 0 bytes for the content length.
Http specification 4.2:
HTTP header fields, which include general-header (section 4.5),
request-header (section 5.3), response-header (section 6.2), and
entity-header (section 7.1) fields, follow the same generic format as
that given in Section 3.1 of RFC 822 [9]. Each header field consists
of a name followed by a colon (":") and the field value. Field names
are case-insensitive. The field value MAY be preceded by any amount of
LWS, though a single SP is preferred. Header fields can be extended
over multiple lines by preceding each extra line with at least one SP
or HT. Applications ought to follow "common form", where one is known
or indicated, when generating HTTP constructs, since there might exist
some implementations that fail to accept anything beyond the common forms.
Because the keep-alive switch of ApacheBench forces it to assume the server supports keep alive, it doesn't detect a "missing" header for the keep alive packet.
This problem if however something that should be solved in ApacheBench, as it is a bug in that application that causes it to miss the proper content-length.

Restlent returns nothing to a partial request

I building a REST-API with restlet to serve a mp3 file. But the File is created and served at the same time. More informations about that here.
It work perfectly but I only tested it in a desktop environment. When I fired up my iPad to test the API, it starts playing the File but after some seconds it stops, send a new Request and starts playing the File from the beginning.
After some researcher I find out that the iPad sends a Partial Request and therefore expects a partial response.
So I modified the Response-Header to fulfill the requirements.
I use curl to test the API :
curl -v -r 0-1 http://localhost:12345/api/path/file.mp3
Request-Header
GET /api/path/file.mp3 HTTP/1.1
Range: bytes=0-1
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
Host: localhost:12345
Accept: */*
Response-Header
HTTP/1.1 206 Partial Content
Date: Tue, 29 Oct 2013 12:18:11 GMT
Accept-Ranges: bytes
Server: Restlet-Framework/2.1.0
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Length: 2
Content-Range: bytes 0-1/19601021
Content-Type: audio/mpeg; charset=UTF-8
Expires: Tue, 29 Oct 2013 12:18:07 GMT
Last-Modified: Tue, 29 Oct 2013 12:18:07 GMT
Error-Message
But there is no Data that comes back from the server. Curl keeps the connection open to the server and the iPad sends a new Request.
When I shut the server down curl gives me:
transfer closed with 1 bytes remaining to read
Closing connection #0
curl: (18) transfer closed with 1 bytes remaining to read
Code
And here is the code I use to return the data. As you can see, this method is just for testing and should always return 2 bytes.
private InputRepresentation rangeGetRequest() throws IOException {
final byte[] bytes = new byte[2];
bytes[0] = 68;
bytes[1] = 68;
final InputRepresentation inputRepresentation = new InputRepresentation(new ByteArrayInputStream(bytes), MediaType.AUDIO_MPEG);
return inputRepresentation;
}
I have no idea what to do. I try to write my own InputStream that returns the 2 bytes, but without success.
Or is the InputRepresentation not suitable for this area of application?
Thanks in advance

Ok, I solved the problem, it was a stupid mistake.
To modify the response-header, I used
getResponse().setEntity(new StringRepresentation(" "));
getResponse().getEntity().setSize(19601021);
getResponse().getEntity().setRange(new Range(0, 2));
getResponse().getEntity().setModificationDate(new Date());
getResponse().getEntity().setExpirationDate(new Date());
getResponse().getEntity().setMediaType(MediaType.AUDIO_MPEG);
The first line was a relic from an old test. If I just use the actual InputRepresentation that holds the two bytes and it work just fine.
What do I learn from that, first tidy up your code, then ask questions ^^