I'm trying to get a JSON from a server to use it in a Python code. For test purposes, I did POST by curl:
$ curl -u trial:trial -H "Content-Type: application/json"
-X POST -d '{"BP_TSM":"22"}' http://some-host --trace-ascii -
My Java code seems to correctly handle creating JSON as a response. Please look at the result of curl command:
== Info: About to connect() to localhost port 8080 (#0)
== Info: Trying ::1...
== Info: Connected to localhost (::1) port 8080 (#0)
== Info: Server auth using Basic with user 'trial'
=> Send header, 224 bytes (0xe0)
0000: POST /get/auth HTT
0040: P/1.1
0047: Authorization: Basic dHJpYWw6dHJpYWw=
006e: User-Agent: curl/7.29.0
0087: Host: localhost:8080
009d: Accept: */*
00aa: Content-Type: application/json
00ca: Content-Length: 15
00de:
=> Send data, 15 bytes (0xf)
0000: {"BP_TSM":"22"}
== Info: upload completely sent off: 15 out of 15 bytes
<= Recv header, 23 bytes (0x17)
0000: HTTP/1.1 202 Accepted
<= Recv header, 34 bytes (0x22)
0000: Server: Payara Micro #badassfish
<= Recv header, 32 bytes (0x20)
0000: Content-Type: application/json
<= Recv header, 37 bytes (0x25)
0000: Date: Thu, 22 Mar 2018 14:30:43 GMT
<= Recv header, 21 bytes (0x15)
0000: Content-Length: 108
<= Recv header, 29 bytes (0x1d)
0000: X-Frame-Options: SAMEORIGIN
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 108 bytes (0x6c)
0000: {"title":"Free Music Archive - Albums","message":"","total":"112
0040: 59","total_pages":2252,"page":1,"limit":"5"}
{"title":"Free Music Archive - Albums","message":"","total":"11259","total_pages
":2252,"page":1,"limit":"5"}== Info: Connection #0 to host localhost left intact
Now I would like Python script be able to receive the same message that curl did. I wrote the following Python code (note I'm not Python developer):
import pickle
import requests
import codecs
import json
from requests.auth import HTTPBasicAuth
from random import randint
req = requests.get('server/get/auth', auth=HTTPBasicAuth('trial', 'trial'))
return pickle.dumps(req)
Unfortunately, I get error message 'unicode' object has no attribute 'copy' when return pickle.dumps(req) command is executed. I also tried using return json.dumps(req) but this time I get another error:
Traceback (most recent call last):
File "/tmp/tmp8DfLJ7/usercode.py", line 16, in the_function
return json.dumps(req)
File "/usr/lib64/python2.7/json/__init__.py", line 244, in dumps
return _default_encoder.encode(obj)
File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
File "/usr/lib64/python2.7/json/encoder.py", line 184, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <Response [405]> is not JSON serializable
Do I have some error in Python code or is it fault of my Java server returning incorrect JSON?
There are a number of errors in your Python code.
You are using request.get to POST. Instead, use request.post.
You are not passing the BP_TSM json string into your request. Use data= in your request.post.
You are not emulating the -H switch to curl. Use headers= in your request.post.
You are using pickle for no apparent reason. Don't do that.
You are using a return statement when you are not in a function. Don't do that. If you want to print to stdout, use print() or sys.stdout.write() instead.
If you actually want to use the returned variables from the JSON (as opposed to simply printing to stdout), you shoud invoke req.json().
Here is a version of your code with problems addressed.
import requests
import json
import sys
from requests.auth import HTTPBasicAuth
data = '{"BP_TSM": "22"}' # curl -d
headers = {'content-type': 'application/json'} # curl -H
auth = HTTPBasicAuth('trial', 'trial') # curl -u
req = requests.post( # curl -X POST
'http://httpbin.org/post',
auth=auth,
data=data,
headers=headers)
sys.stdout.write(req.text) # Display JSON on stdout
returned_data = req.json()
my_ip = returned_data["origin"] # Query value from JSON
print("My public IP is", my_ip)
You're trying to dumps a Response object.
Try returning req.json() or calling json.loads(req.text)
In order to load the Json string, you'll need to use json.loads(req.text).
You must also ensure that the req string is valid json.
eg
'{"FOO":"BAR"}'
You can use requests.json() method to get json response as dict
req = requests.get('http://yourdomain.com/your/path', auth=HTTPBasicAuth('trial', 'trial'))
mydict = req.json()
Related
I have a small demo to play with customized status code.
The interesting part is the request will always hang there if the status is below 200, like 105, 199, etc. But works for any status greater than 200, like 209, 789 etc.
Http status code registry, refer to https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml
Spring boot: 1.5.4.RELEASE with embedded tomcat
Java: 8
Controller:
#RestController
public class DemoController {
#GetMapping("/hello")
public ResponseEntity get() {
return ResponseEntity.status(105).build();
}
}
Can anyone give me a clear explanation?
I create a gist at here: https://gist.github.com/pengisgood/dbea1fcdc45c2bb5809871c7f020b800
Update:
I also create a small demo to reproduce it at here:
https://github.com/pengisgood/springboot-customize-status-code
Update:
After I run curl -v localhost:8080/hello, I can see the status, but the response doesn't finish. Refer to the gif below:
I also ran into this issue and found that it is not Spring that creates this behavior. It is Tomcat.
curl -v --header "Expect: 100-continue" http://localhost:8080
Calling any configured endpoint like this will return an extra response code that doesn't terminate the request.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0*
Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.56.1
> Accept: */*
> Expect: 100-continue
>
< HTTP/1.1 100
< HTTP/1.1 200
< Set-Cookie: JSESSIONID=9355141A10CF546E9A9A43F5A5C0B1A4; Path=/; HttpOnly
< Content-Type: text/html;charset=ISO-8859-1
< Content-Length: 58
< Date: Tue, 31 Jul 2018 17:27:52 GMT
<
{ [58 bytes data]
100 58 100 58 0 0 58 0 0:00:01 --:--:-- 0:00:01 82<html>
<body>
<h2>Hello Heroku!</h2>
</body>
</html>
* Connection #0 to host localhost left intact
note the HTTP/1.1 100
This response came from this project https://devcenter.heroku.com/articles/create-a-java-web-application-using-embedded-tomcat which doesn't have spring. If I modify the HelloServlet to include a response code of 100 it just hangs.
Looking deeper:
https://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.3
The specification makes it clear that the 100 response is supposed to happen in the same request. The reason it hangs is because it is expecting the client to respond with the content of the request.
Looking at the wiki for other 1XX response codes, it appears to also be true that some information is returned without closing the request. My guess is that Tomcat expects all 1xx response codes to act in this manner.
As far as I can tell the Spring DispacherServlet is handling the different return codes in exactly the same way. I think what's happening is curl is just leaving the connection open because the response is in the 1xx range.
This article provides a good primer on status codes. This sentence in particular is relevant:
100–199 Codes in the 100s are informational, indicating that the client should respond with some other action.
If you run curl with --trace you'll see that the 105 response does actually arrive:
curl -v -trace http://localhost:8080/hello
Trying ::1...
TCP_NODELAY set
Connected to localhost (::1) port 8080 (#0)
> GET /hello HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 105
< Date: Tue, 19 Sep 2017 18:07:04 GMT
^C
So what I think is happening is the response is returned, the client should respond with some other action (but doesn't) so it looks like the thing has hung.
Probably the real question here is why are you trying to return a 105 status and what do you expect to happen?
For some reason the previously working code stopped working and server started to respond with 416.
Here are the logs of HTTP client during failing interaction:
-------------- REQUEST --------------
GET https://www.googleapis.com/drive/v3/files/0B02Nopv3SQOvOVNKaDIwTEZ3MHd?alt=media
Accept-Encoding: gzip
Authorization: <Not Logged>
Range: bytes=0-33554431
User-Agent: My app Google-API-Java-Client Google-HTTP-Java-Client/1.22.0 (gzip)
-------------- RESPONSE --------------
HTTP/1.1 416 Requested range not satisfiable
Alt-Svc: quic=":443"; ma=2592000; v="39,38,37,35"
Server: UploadServer
Cache-Control: private, max-age=0
Content-Range: bytes */0
X-GUploader-UploadID: AEnB2UqBx9B09Lnr8tG761gdoz3DkhHSNO_OzHh1LkU6B2908v17rnBGQZSNW4ZVTjbRdFtvPWWIqZGdtSrTo6ZWN7YW9nxf6d
Vary: X-Origin
Vary: Origin
Expires: Mon, 11 Sep 2017 15:23:20 GMT
Content-Length: 225
Date: Mon, 11 Sep 2017 15:23:20 GMT
Content-Type: application/json; charset=UTF-8
I was trying to download a file which is around 200000 bytes, so I thought meaning of "chuck size" changed somewhere, so it could not give 33554431 bytes of a 282177 byte file. Tried changing that to a smaller value, but no success.
Drive.Files.Get get = drive.files().get(file.getId())
MediaHttpDownloader downloader = get.getMediaHttpDownloader()
downloader.directDownloadEnabled = false
localFile.newOutputStream()
get.executeMediaAndDownloadTo(stream)
Direct download does not work either, it just downloads "0" bytes.
Does anyone know how to overcome this issue?
416 Range Not
Satisfiable
error means the server is not able to serve the requested ranges. The
most likely reason is that the document doesn't contain such ranges,
or that the Range header value, though syntactically correct, doesn't
make sense.
One of the resolutions that may provide from this forum is to:
Add "Accept-Ranges: none" to our response headers.
It appeared to be a web interface when using Firefox. It uploaded "empty" files in certain cases.
https://productforums.google.com/forum/#!topic/drive/S03wEknc75g;context-place=forum/drive
I am a new programmer i am trying to build an app with Json.
If i use this URL doesn't work . http://zsuzsafodraszat.hostzi.com/boltok.json
if i Use this, my app working. https://api.myjson.com/bins/3zm8i
Both Json files exactly the same.
Can you help me what i am doing wrong ? Maybe bad extension or web000 is not a good service for Json ? Can you give me some good free json hosting ? Thanks
Those 2 urls do not have the same content or the same headers. You can see this if run curl commands from the command line:
$ curl -i "http://zsuzsafodraszat.hostzi.com/boltok.json"
HTTP/1.1 200 OK
Date: Wed, 13 Apr 2016 22:52:50 GMT
Server: Apache
Last-Modified: Wed, 13 Apr 2016 16:48:23 GMT
Accept-Ranges: bytes
Content-Length: 1020
Connection: close
Content-Type: application/json
??{"Aldi":"http://catalog.aldi.com/emag/hu_HU/print/Online_katalogus_04_07/Online_katalogus_04_07.pdf",
"Lidl":"http://www.lidl.hu/statics/lidl-hu/ds_doc/HU_HHZ_kw14_2016.pdf",
"Spar":"http://ajanlatok.spar.hu/view/download/?d=1279",
"Penny":"https://view.publitas.com/16538/136265/pdfs/016f82fb5b00bc97b5a8c35f512d89b01cd3e3ce.pdf",
"Coop":"https://view.publitas.com/2556/133497/pdfs/16603d7e9bf30e8a8a4efec7f01d3fa2caf92fe0.pdf",
"Auchan":"http://www.lidl.hu/statics/lidl-hu/ds_doc/HU_HHZ_kw14_2016.pdf"}
$ curl -i "https://api.myjson.com/bins/3zm8i"
HTTP/1.1 200 OK
Server: nginx/1.5.8
Date: Wed, 13 Apr 2016 22:52:56 GMT
Content-Type: application/json
Content-Length: 500
Connection: keep-alive
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
{"Aldi":"http://catalog.aldi.com/emag/hu_HU/print/Online_katalogus_04_07/Online_katalogus_04_07.pdf","Lidl":"http://www.lidl.hu/statics/lidl-hu/ds_doc/HU_HHZ_kw14_2016.pdf","Spar":"http://ajanlatok.spar.hu/view/download/?id=1279","Penny":"https://view.publitas.com/16538/136265/pdfs/016f82fb5b00bc97b5a8c35f512d89b01cd3e3ce.pdf","Coop":"https://view.publitas.com/2556/133497/pdfs/16603d7e9bf30e8a8a4efec7f01d3fa2caf92fe0.pdf","Auchan":"http://www.lidl.hu/statics/lidl-hu/ds_doc/HU_HHZ_kw14_2016.pdf"}
As you can see, one of them has a couple of junk bytes at the beginning that my terminal is displaying as question marks. Also the http headers are different. The Content-Lengths are wildly different too. Did you use something other than a plain text editor to create the json payload in the failing example?
Try removing the junk characters and adding these http headers:
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
I building a REST-API with restlet to serve a mp3 file. But the File is created and served at the same time. More informations about that here.
It work perfectly but I only tested it in a desktop environment. When I fired up my iPad to test the API, it starts playing the File but after some seconds it stops, send a new Request and starts playing the File from the beginning.
After some researcher I find out that the iPad sends a Partial Request and therefore expects a partial response.
So I modified the Response-Header to fulfill the requirements.
I use curl to test the API :
curl -v -r 0-1 http://localhost:12345/api/path/file.mp3
Request-Header
GET /api/path/file.mp3 HTTP/1.1
Range: bytes=0-1
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
Host: localhost:12345
Accept: */*
Response-Header
HTTP/1.1 206 Partial Content
Date: Tue, 29 Oct 2013 12:18:11 GMT
Accept-Ranges: bytes
Server: Restlet-Framework/2.1.0
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Length: 2
Content-Range: bytes 0-1/19601021
Content-Type: audio/mpeg; charset=UTF-8
Expires: Tue, 29 Oct 2013 12:18:07 GMT
Last-Modified: Tue, 29 Oct 2013 12:18:07 GMT
Error-Message
But there is no Data that comes back from the server. Curl keeps the connection open to the server and the iPad sends a new Request.
When I shut the server down curl gives me:
transfer closed with 1 bytes remaining to read
Closing connection #0
curl: (18) transfer closed with 1 bytes remaining to read
Code
And here is the code I use to return the data. As you can see, this method is just for testing and should always return 2 bytes.
private InputRepresentation rangeGetRequest() throws IOException {
final byte[] bytes = new byte[2];
bytes[0] = 68;
bytes[1] = 68;
final InputRepresentation inputRepresentation = new InputRepresentation(new ByteArrayInputStream(bytes), MediaType.AUDIO_MPEG);
return inputRepresentation;
}
I have no idea what to do. I try to write my own InputStream that returns the 2 bytes, but without success.
Or is the InputRepresentation not suitable for this area of application?
Thanks in advance
Ok, I solved the problem, it was a stupid mistake.
To modify the response-header, I used
getResponse().setEntity(new StringRepresentation(" "));
getResponse().getEntity().setSize(19601021);
getResponse().getEntity().setRange(new Range(0, 2));
getResponse().getEntity().setModificationDate(new Date());
getResponse().getEntity().setExpirationDate(new Date());
getResponse().getEntity().setMediaType(MediaType.AUDIO_MPEG);
The first line was a relic from an old test. If I just use the actual InputRepresentation that holds the two bytes and it work just fine.
What do I learn from that, first tidy up your code, then ask questions ^^
I am creating a Java application which 'streams' a video file over http to a browser (currently Chrome v24.x). This video is sent to FFmpeg and the output of this is sent over HTTP.
Now, once the file is completely encoded the file is served using chunked transfer, and responding to range requests. Example headers:
Request
GET /file/9fe6b502-c127-47c2-b6d2-83ea58676a8d HTTP/1.1 : Host: localhost:1234 : Connection: keep-alive : Accept-Encoding: identity;q=1, *;q=0 : User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.56 Safari/537.17 : Accept: */* : Referer: http://localhost:1234/media/9fe6b502-c127-47c2-b6d2-83ea58676a8d : Accept-Language: en-US,en;q=0.8 : Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 : Cookie: plushContainerWidth=100%25; plushNoTopMenu=0 : Range: bytes=0- :
Response
HTTP/1.1 206 Partial Content : Connection: close : Date: Mon, 28 Jan 2013 14:51:52 GMT : Content-Type: video/mp4 : Etag: "9fe6b502-c127-47c2-b6d2-83ea58676a8d" : Accept-Ranges: bytes : Content-Range: bytes 0-625825/625826 : Content-Length: 625826 : Transfer-Encoding: chunked :
Data sent is 625826 bytes, excluding header data and chunked overhead.
Now, this works just fine!
The trouble is when the GET request happens before the file encoding has completed. I have tried to start sending the file straight away, over HTTP, using just chunked transfer with no content length attributes or ranges because they are not currently known. This causes the browser to wait for the full file (and not start playing until the transfer has completed). In addition, when the file transfer is completed, the browser reports a video error that the file could not be played. Example headers:
Request
Request : GET /file/9fe6b502-c127-47c2-b6d2-83ea58676a8d HTTP/1.1 : Host: localhost:1234 : Connection: keep-alive : Accept-Encoding: identity;q=1, *;q=0 : User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.56 Safari/537.17 : Accept: */* : Referer: http://localhost:1234/media/9fe6b502-c127-47c2-b6d2-83ea58676a8d : Accept-Language: en-US,en;q=0.8 : Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 : Cookie: plushContainerWidth=100%25; plushNoTopMenu=0 : Range: bytes=0-
Response
HTTP/1.1 200 OK : Connection: close : Date: Mon, 28 Jan 2013 14:51:29 GMT : Content-Type: video/mp4 : Transfer-Encoding: chunked
Data sent is 625826 bytes, excluding header data and chunked overhead.
Does anyone have any ideas what's going wrong, or how to start playing a video without knowing the full length of the file?
Thanks for your time,
James.
//EDIT
As the request for the incomplete file states 'Range: bytes=0-' - can I reply with a partial content of 'n' bytes (say, 1000 bytes) with Content-Range: bytes 0-999/* ?
//EDIT 2
As requested, here is my code for outputting the file. This is condensed as the code actually spans several classes.
File f = new File(_filename);
RandomAccessFile raf = new RandomAccessFile(f, "r");
ChunkedOutputStream cos = new ChunkedOutputStream(_out, 1024 * 100);
byte[] bytes;
while (true){
lBufferSizeMax = Math.min(lBufferSize, fc.length() - lCompleted);
bytes = new byte[(int)lBufferSizeMax];
lCurrentRead = fc.read(bytes);
if (lCurrentRead == 0){
break;
}
cos.write(bytes);
}
Be aware that ffmpeg is likely to rewrite portions of the file when encoding is ending. That means the metadata in the video file will be incomplete until the very end so there is no use trying to stream the beginning of the file before that time. You can check with replacing the output filename in your ffmpeg command line with a single dash and redirect stdout to a file. Instead of ffmpeg ... out.mov do ffmpeg ... - > out.mov. If you get an error message saying the format needs seekable output you are affected.
I have done almost exactly what you are trying to do, the results are in my Git repo. The container format I use, where I can play games with the metadata, is FLV. Unfortunately that is not what the iPad/iPhone devices out there like to see.