I am creating a Java application which 'streams' a video file over http to a browser (currently Chrome v24.x). This video is sent to FFmpeg and the output of this is sent over HTTP.
Now, once the file is completely encoded the file is served using chunked transfer, and responding to range requests. Example headers:
Request
GET /file/9fe6b502-c127-47c2-b6d2-83ea58676a8d HTTP/1.1 : Host: localhost:1234 : Connection: keep-alive : Accept-Encoding: identity;q=1, *;q=0 : User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.56 Safari/537.17 : Accept: */* : Referer: http://localhost:1234/media/9fe6b502-c127-47c2-b6d2-83ea58676a8d : Accept-Language: en-US,en;q=0.8 : Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 : Cookie: plushContainerWidth=100%25; plushNoTopMenu=0 : Range: bytes=0- :
Response
HTTP/1.1 206 Partial Content : Connection: close : Date: Mon, 28 Jan 2013 14:51:52 GMT : Content-Type: video/mp4 : Etag: "9fe6b502-c127-47c2-b6d2-83ea58676a8d" : Accept-Ranges: bytes : Content-Range: bytes 0-625825/625826 : Content-Length: 625826 : Transfer-Encoding: chunked :
Data sent is 625826 bytes, excluding header data and chunked overhead.
Now, this works just fine!
The trouble is when the GET request happens before the file encoding has completed. I have tried to start sending the file straight away, over HTTP, using just chunked transfer with no content length attributes or ranges because they are not currently known. This causes the browser to wait for the full file (and not start playing until the transfer has completed). In addition, when the file transfer is completed, the browser reports a video error that the file could not be played. Example headers:
Request
Request : GET /file/9fe6b502-c127-47c2-b6d2-83ea58676a8d HTTP/1.1 : Host: localhost:1234 : Connection: keep-alive : Accept-Encoding: identity;q=1, *;q=0 : User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.56 Safari/537.17 : Accept: */* : Referer: http://localhost:1234/media/9fe6b502-c127-47c2-b6d2-83ea58676a8d : Accept-Language: en-US,en;q=0.8 : Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 : Cookie: plushContainerWidth=100%25; plushNoTopMenu=0 : Range: bytes=0-
Response
HTTP/1.1 200 OK : Connection: close : Date: Mon, 28 Jan 2013 14:51:29 GMT : Content-Type: video/mp4 : Transfer-Encoding: chunked
Data sent is 625826 bytes, excluding header data and chunked overhead.
Does anyone have any ideas what's going wrong, or how to start playing a video without knowing the full length of the file?
Thanks for your time,
James.
//EDIT
As the request for the incomplete file states 'Range: bytes=0-' - can I reply with a partial content of 'n' bytes (say, 1000 bytes) with Content-Range: bytes 0-999/* ?
//EDIT 2
As requested, here is my code for outputting the file. This is condensed as the code actually spans several classes.
File f = new File(_filename);
RandomAccessFile raf = new RandomAccessFile(f, "r");
ChunkedOutputStream cos = new ChunkedOutputStream(_out, 1024 * 100);
byte[] bytes;
while (true){
lBufferSizeMax = Math.min(lBufferSize, fc.length() - lCompleted);
bytes = new byte[(int)lBufferSizeMax];
lCurrentRead = fc.read(bytes);
if (lCurrentRead == 0){
break;
}
cos.write(bytes);
}
Be aware that ffmpeg is likely to rewrite portions of the file when encoding is ending. That means the metadata in the video file will be incomplete until the very end so there is no use trying to stream the beginning of the file before that time. You can check with replacing the output filename in your ffmpeg command line with a single dash and redirect stdout to a file. Instead of ffmpeg ... out.mov do ffmpeg ... - > out.mov. If you get an error message saying the format needs seekable output you are affected.
I have done almost exactly what you are trying to do, the results are in my Git repo. The container format I use, where I can play games with the metadata, is FLV. Unfortunately that is not what the iPad/iPhone devices out there like to see.
Related
For some reason the previously working code stopped working and server started to respond with 416.
Here are the logs of HTTP client during failing interaction:
-------------- REQUEST --------------
GET https://www.googleapis.com/drive/v3/files/0B02Nopv3SQOvOVNKaDIwTEZ3MHd?alt=media
Accept-Encoding: gzip
Authorization: <Not Logged>
Range: bytes=0-33554431
User-Agent: My app Google-API-Java-Client Google-HTTP-Java-Client/1.22.0 (gzip)
-------------- RESPONSE --------------
HTTP/1.1 416 Requested range not satisfiable
Alt-Svc: quic=":443"; ma=2592000; v="39,38,37,35"
Server: UploadServer
Cache-Control: private, max-age=0
Content-Range: bytes */0
X-GUploader-UploadID: AEnB2UqBx9B09Lnr8tG761gdoz3DkhHSNO_OzHh1LkU6B2908v17rnBGQZSNW4ZVTjbRdFtvPWWIqZGdtSrTo6ZWN7YW9nxf6d
Vary: X-Origin
Vary: Origin
Expires: Mon, 11 Sep 2017 15:23:20 GMT
Content-Length: 225
Date: Mon, 11 Sep 2017 15:23:20 GMT
Content-Type: application/json; charset=UTF-8
I was trying to download a file which is around 200000 bytes, so I thought meaning of "chuck size" changed somewhere, so it could not give 33554431 bytes of a 282177 byte file. Tried changing that to a smaller value, but no success.
Drive.Files.Get get = drive.files().get(file.getId())
MediaHttpDownloader downloader = get.getMediaHttpDownloader()
downloader.directDownloadEnabled = false
localFile.newOutputStream()
get.executeMediaAndDownloadTo(stream)
Direct download does not work either, it just downloads "0" bytes.
Does anyone know how to overcome this issue?
416 Range Not
Satisfiable
error means the server is not able to serve the requested ranges. The
most likely reason is that the document doesn't contain such ranges,
or that the Range header value, though syntactically correct, doesn't
make sense.
One of the resolutions that may provide from this forum is to:
Add "Accept-Ranges: none" to our response headers.
It appeared to be a web interface when using Firefox. It uploaded "empty" files in certain cases.
https://productforums.google.com/forum/#!topic/drive/S03wEknc75g;context-place=forum/drive
I'm trying to use JMeter to test a restful endpoint for uploading files, but I'm getting a 400 error. The one thing that jumps out at me is the boundary value; it's not the same I provided as the one shown in the request. I'm able to use the endpoint in the browser without issue, and I've replicated the headers revealed in the developer tools in FF.
Here is the relevant info from JMeter:
Result Tab:
Thread Name: Asdf - Load Test 1-1
Sample Start: 2017-06-05 08:47:46 EDT
Load time: 159
Connect Time: 28
Latency: 159
Size in bytes: 438
Sent bytes:821003
Headers size in bytes: 438
Body size in bytes: 0
Sample Count: 1
Error Count: 1
Data type ("text"|"bin"|""):
Response code: 400
Response message: Bad Request
Response headers:
HTTP/1.1 400 Bad Request
Date: Mon, 05 Jun 2017 12:47:46 GMT
Server: Apache/2.4.25 (Win64) OpenSSL/1.0.2k
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
Strict-Transport-Security: max-age=31536000 ; includeSubDomains
X-XSS-Protection: 1; mode=block
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
Cache-Control: no-cache, must-revalidate
Content-Length: 0
Connection: close
HTTPSampleResult fields:
ContentType:
DataEncoding: null
Request Tab:
POST https://localhost/my/rest/endpoint
POST data:
--9amm365-gMmimP70lvs9jIvlIxOfkocUN
Content-Disposition: form-data; name="parseable"; filename="asdf.docx"
Content-Type: application/vnd.openxmlformats-
officedocument.wordprocessingml.document
Content-Transfer-Encoding: binary
--9amm365-gMmimP70lvs9jIvlIxOfkocUN--
[no cookies]
Request Headers:
Connection: keep-alive
Content-Type: multipart/form-data; boundary=--Uc_2uLvcVgc7SqvzIJxR3encUKw-
f7w9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.5
Accept: application/json, text/plain, /
X-Request-OnBehalfOf: some-user-uuid
X-Requested-With: XMLHttpRequest
Content-Length: 820532
Host: localhost
User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_121)
Response Data Tab is blank.
Server side implementation:
#POST
#Override
#EnhancedDetail
#Consumes(MediaType.MULTIPART_FORM_DATA)
#Path("my/rest/endpoint")
public Response uploadProduct(#PathParam("id") final String id, MultiPart multipart) throws IOException {
return processMultiFileUpload(id, multipart, MultiPartUploadType.DRAFT, false);
}
If you are building your request manually like it is described in Testing REST API File Uploads in JMeter article - make sure you have Use multipart/form-data for POST box of the HTTP Request sampler is not checked.
And vice versa, if you tick this box, you won't need to override the Content-Type header
I believe the easiest way would be just recording your file upload event using HTTP(S) Test Script Recorder (just make sure your asdf.docx file is copied to the "bin" folder of your JMeter installation)
I figured it out. I changed the Client Implementation on the HTTP Request to Java. Everything I've read said use HttpClient4, but I tried Java and it worked.
I'm doing a POST to a restlet and need to return a zip file. But although the created file is zip, the method returns gibberish.
I tried wrapping the FileRepresentation as was suggested here:
new org.restlet.engine.application.EncodeRepresentation(org.restlet.data.Encoding.ZIP, representation);
And also tried adding a Produces annotation like this:
#Produces({"application/x-zip-compressed"})
But neither works. The representation returns as gibberish string, and the Content-Type header stays application/octet-stream. What am I missing?
These are the request headers. Note the Accept-Encoding: gzip, deflate:
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36
Origin: chrome-extension://hgmloofddffdnphfgcellkdfbfbjeloo
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8,he;q=0.6
The response headers:
Vary: Accept-Encoding
Last-Modified: Wed, 06 May 2015 14:49:03 GMT
Content-Disposition: attachment; filename=_backup_20150506.zip; size=162191
Date: Wed, 06 May 2015 14:49:03 GMT
Accept-Ranges: bytes
Server: Restlet-Framework/2.2.1
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Set-Cookie: JSESSIONID=5F10BBBDC58D5C3D6C0474FA12C44FB9; Path=/AppName/; Domain=localhost
Content-Encoding: gzip
Content-Type: application/octet-stream
Transfer-Encoding: chunked
EDIT: I also tried changing the media type when creating the representation:
MediaType mt = MediaType.APPLICATION_ZIP;
FileRepresentation fr = new FileRepresentation(file, mt);
The response content type changed to Content-Type: application/zip but the returned value is still a gibberish string.
The right way to do that is what you used:
public class MyServerResource extends ServerResource {
#Post
public Representation test(Representation repr) {
FileRepresentation outputRepresentation
= new FileRepresentation(new File("(...)"),
MediaType.APPLICATION_ZIP);
return outputRepresentation;
}
}
So this should work.
Using curl with such code, here is what I have:
$ curl -X POST http://localhost:8182/test > mycontent.zip
$ unzip mycontent.zip
Archive: mycontent.zip
extracting: test.txt
In addition, here is what I have with the verbose mode of curl:
curl -X POST --verbose http://localhost:8182/test
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8182 (#0)
> POST /test HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:8182
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-type: application/zip
< Last-modified: Thu, 07 May 2015 08:08:59 GMT
< Content-length: 134
* Server Restlet-Framework/2.3.1 is not blacklisted
< Server: Restlet-Framework/2.3.1
< Accept-ranges: bytes
< Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
< Date: Thu, 07 May 2015 08:19:26 GMT
<
Notice that you can use the header Disposition if you want to configure hints within the download dialog of your browser.
Otherwise, "enable GZIP compression of the JSON response entity on Resltet" corresponds to automatic compression of the whole response content by Restlet. Browsers support this and can directly uncompress the content before displaying it. I don't think that isn't really what you need / expect. If it's the case, you could be interested in this link: https://templth.wordpress.com/2015/02/23/optimizing-restlet-server-applications/.
Hope it helps you,
Thierry
I building a REST-API with restlet to serve a mp3 file. But the File is created and served at the same time. More informations about that here.
It work perfectly but I only tested it in a desktop environment. When I fired up my iPad to test the API, it starts playing the File but after some seconds it stops, send a new Request and starts playing the File from the beginning.
After some researcher I find out that the iPad sends a Partial Request and therefore expects a partial response.
So I modified the Response-Header to fulfill the requirements.
I use curl to test the API :
curl -v -r 0-1 http://localhost:12345/api/path/file.mp3
Request-Header
GET /api/path/file.mp3 HTTP/1.1
Range: bytes=0-1
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
Host: localhost:12345
Accept: */*
Response-Header
HTTP/1.1 206 Partial Content
Date: Tue, 29 Oct 2013 12:18:11 GMT
Accept-Ranges: bytes
Server: Restlet-Framework/2.1.0
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Length: 2
Content-Range: bytes 0-1/19601021
Content-Type: audio/mpeg; charset=UTF-8
Expires: Tue, 29 Oct 2013 12:18:07 GMT
Last-Modified: Tue, 29 Oct 2013 12:18:07 GMT
Error-Message
But there is no Data that comes back from the server. Curl keeps the connection open to the server and the iPad sends a new Request.
When I shut the server down curl gives me:
transfer closed with 1 bytes remaining to read
Closing connection #0
curl: (18) transfer closed with 1 bytes remaining to read
Code
And here is the code I use to return the data. As you can see, this method is just for testing and should always return 2 bytes.
private InputRepresentation rangeGetRequest() throws IOException {
final byte[] bytes = new byte[2];
bytes[0] = 68;
bytes[1] = 68;
final InputRepresentation inputRepresentation = new InputRepresentation(new ByteArrayInputStream(bytes), MediaType.AUDIO_MPEG);
return inputRepresentation;
}
I have no idea what to do. I try to write my own InputStream that returns the 2 bytes, but without success.
Or is the InputRepresentation not suitable for this area of application?
Thanks in advance
Ok, I solved the problem, it was a stupid mistake.
To modify the response-header, I used
getResponse().setEntity(new StringRepresentation(" "));
getResponse().getEntity().setSize(19601021);
getResponse().getEntity().setRange(new Range(0, 2));
getResponse().getEntity().setModificationDate(new Date());
getResponse().getEntity().setExpirationDate(new Date());
getResponse().getEntity().setMediaType(MediaType.AUDIO_MPEG);
The first line was a relic from an old test. If I just use the actual InputRepresentation that holds the two bytes and it work just fine.
What do I learn from that, first tidy up your code, then ask questions ^^
I am trying to download an image from URL. I use the Apache Commons library:
org.apache.commons.io.FileUtils
.copyURLToFile(
new URL(
"https://lh3.ggpht.com/AXYMUV5cpne2vE9U3X8x87HgrbwijwiG50_yOnehU2MUPKDoJky-BKFOPZzZ07Pug8U=h230"),
new File("test.png"));
While opening the image it says not an PNG file. Is this a good way to download images from url to local?
Updated .
Thankyou guys for your answers i undestand that i need to get the connection.getContentType(); and then save it as jpg or png as it is.
if you check the url, It is JPEG file. Try to execute program after renaming file name to test.jpeg..
Checking for the content type see this SO Question.
Then use this to save the file to the right type.
Running wget on this URL and looking at the HTTP response, looks like the file type is explicitly specified as JPEG:
--2012-06-27 16:55:30-- https://lh3.ggpht.com/AXYMUV5cpne2vE9U3X8x87HgrbwijwiG50_yOnehU2MUPKDoJky-BKFOPZzZ07Pug8U=h230
Resolving lh3.ggpht.com... 74.125.237.108, 74.125.237.106, 74.125.237.107, ...
Connecting to lh3.ggpht.com|74.125.237.108|:443... connected.
HTTP request sent, awaiting response...
HTTP/1.0 200 OK
Access-Control-Allow-Origin: *
ETag: "v1"
Expires: Thu, 28 Jun 2012 06:50:40 GMT
Content-Disposition: inline;filename="unnamed.jpg"
Content-Type: image/jpeg
X-Content-Type-Options: nosniff
Date: Wed, 27 Jun 2012 06:50:40 GMT
Server: fife
Content-Length: 46597
X-XSS-Protection: 1; mode=block
Cache-Control: public, max-age=86400, no-transform
Age: 290
Connection: Keep-Alive
Length: 46597 (46K) [image/jpeg]