When doing calls to Exact-on-line API to get authenticated we run into the problem that getting the first refresh-token fails. We're not sure why. This is what we get back from Exact:
Http code: 400
JSON Data:
{
error='invalid_request',
description='Handle could not be extracted',
uri='null',
state='null',
scope='null',
redirectUri='null',
responseStatus=400,
parameters={}
}
We use this Java code based on library org.apache.oltu.oauth2.client (1.0.2):
OAuthClientRequest oAuthRequest = OAuthClientRequest //
.tokenLocation(BASE_URL + "/api/oauth2/token") //
.setGrantType(GrantType.AUTHORIZATION_CODE) //
.setClientId(clientId) //
.setClientSecret(clientSecret) //
.setRedirectURI(REDIRECT_URI) //
.setCode(code) //
.buildBodyMessage();
OAuthClient client = new OAuthClient(new URLConnectionClient());
OAuthJSONAccessTokenResponse oauthResponse = client.accessToken(oAuthRequest, OAuth.HttpMethod.POST);
We did do the first step (getting the 'code' as used in setCode(...)) using a localhost-redirect as displayed in https://support.exactonline.com/community/s/knowledge-base#All-All-DNO-Content-gettingstarted There we copy the code from the address-bar of our browser and store it in a place the next computer-step can read it again.
This is due to the fact that the code was copied from your browsers address-bar. There you will find a URL-encoded version of the code (visible in the '%21' often) which when passed into the setCode verbatim will fail the subsequent calls.
Suggestion: URL-decode the value or setup a small temporary localhost-HTTP-server using Undertow or the like to catch the code that was send to you localhost-URL:
Undertow server = Undertow.builder() //
.addHttpListener(7891, "localhost") //
.setHandler(new HttpHandler() {
#Override
public void handleRequest(final HttpServerExchange exchange) throws Exception {
String code = exchange.getQueryParameters().get("code").getFirst();
LOG.info("Recieved code: {}.", code);
LOG.info("Store code");
storeCode(code);
LOG.info("Code stored");
exchange.getResponseHeaders().put(Headers.CONTENT_TYPE, "text/plain");
exchange.getResponseSender().send( //
"Thanks for getting me the code: " + code + "\n" //
+ "Will store it for you and get the first refreshToken..." //
+ "Please have a look at " + OAUTH_STATE_INI
+ " for the new code & refreshToken in a minute" //
);
done.add("done");
}
}).build();
server.start();
NB: Do make sure the redirect URL is correct in your Exact-app-settings
I am working on a crawler and I have the following question: it works with simple HTTP requests, but not HTTPS, and I need to make an HTTPS request. I changed the port to 443 and try to send the same request, but I get 400 error. Obviously, I need to change something else, but I do not know what. I open socket and this is how I make the request:
String request
= "GET " + file
+ (file.endsWith("robots.txt") ? " HTTP/1.0\r\n" : " HTTP/1.1\r\n")
// " HTTP/1.1\r\n"
+ "User-Agent: " + CrawlerConfig.USER_AGENT + "\r\n"
// + ((!CrawlerConfig.SAVE_IMAGES) ? "Accept: text/html\r\n" : "")
// + "Accept: text/*\r\n"
+ (file.endsWith("robots.txt") ? "Connection: close\r\n" : "")
+ "Host: " + host + "\r\n" + "\r\n"/*
* + body
*/;
outStream.write(request.getBytes("US-ASCII"));
outStream.flush();
Try to send an OPTIONS request on the resource, needed request headers should be returned, some may be missing.
I have a java client that reads JSON from a web service. The first part of thejson response (offset 1 - 20) is of the format below. I can currently parse the first page of the data into a file (though i will love to parse into db eventually). How can I get the client to parse the next page (offset 21 - 40) ? The second part of the file has more information intentionally omitted for brevity sake. I have used the apache http client but it seems there is no method to handle this ? Will be grateful if pointed to the right directoin.
{
"meta": {
"limit": 15,
"next": "/abc/api/defapi/lkk/?username=abc&api_key=xyz&limit=15&offset=60&format=json",
"offset": 30,
"previous": "/abc/api/defapi/lkk/?username=abc&api_key=xyz&limit=15&offset=30&format=json",
"total_count": 312213
},
"objects": [
{
"availability": "COMPLETE",
"confidentiality": "",
"cpe_id": [
{
"name": "cpe:/o:sun:sunos:5.11::express"
}
NOTE:truncated for brevity
Many thanks all the answers and pointers to my question. The code below does what I wanted i.e. grabs json files and save on disk, incrementally. I think it is still very basic hence I would appreciate all criticisms on better approaches, coding styles. Still learning !!
while (count < totalCount ) {
MyClient check = new MyClient();
check.checkSecurity();
System.out.println(" ************ Starting with count " + count + " ***********************");
URI uriBuilder = new URIBuilder(url)
.addParameter("limit", String.valueOf(limit))
.addParameter("offset", String.valueOf(offset))
.build();
System.out.println("The URL built is --> " + uriBuilder);
HttpGet hget = new HttpGet(uriBuilder);
System.out.println(hget.getRequestLine());
CloseableHttpClient httpclient = HttpClients.createDefault();
hget.addHeader("Content-Type", "application/json");
HttpResponse response = httpclient.execute(hget);
URL vdbUrl = uriBuilder.toURL();
System.out.println("Response status : " + response.getStatusLine());
HttpEntity entity2 = response.getEntity();
do {
vdbInput = new File("C:/myfiles/entry_0" + filenamePostfix + ".json");
FileUtils.copyURLToFile(vdbUrl, vdbInput);
System.out.println("File successfully written from page " + count );
System.out.println("page is : " + vdbUrl);
filenamePostfix++;
count++;
offset = offset + 20;
} while (filenamePostfix < count) ;
EntityUtils.consume(entity2);
}
}
i have a domain with fields like URL, lastModified etc. Now i want to view the information i get from this code into the list page. How can i do tht
def url = new URL("http://www.google.com")
HttpURLConnection connection = (HttpURLConnection)url.openConnection()
// long contentLength = Long.parseLong(connection.getHeaderField("Content-Length"));
connection.setRequestMethod("GET")
connection.connect()
connection.getURL()
println("Date: " + new Date(connection.getDate()))
println("LastModified Date:" + new Date(connection.getLastModified()))
if (connection.responseCode == 200 || connection.responseCode == 201){
def returnMessage = connection.content
//print out the full response
println "${returnMessage}";
System.out.println(connection.getURL());
} else {
println "Error Connecting to " + url
println "Couldn't connect to url"
}
From your controller, you can use the render method,
render returnMessage
(see http://grails.org/doc/2.0.x/ref/Controllers/render.html)
or pass it in a model to a gsp page and then print it from gsp,
return [returnMsg: returnMessage]
....(in .gsp)
${returnMsg}
(see http://grails.org/doc/2.0.x/guide/theWebLayer.html#GSPBasics)
Yes, it's a long question with a lot of detail... So, my question is: How can I stream an upload to Vimeo in segments?
For anyone wanting to copy and debug on their own machine: Here are the things you need:
My code here.
Include the Scribe library found here
Have a valid video file (mp4) which is at least greater than 10 MB and put it in the directory C:\test.mp4 or change that code to point wherever yours is.
That's it! Thanks for helping me out!
Big update: I've left a working API Key and Secret for Vimeo in the code here. So as long as you have a Vimeo account, all the code should work just fine for you once you've allowed the application and entered your token. Just copy the code from that link into a project on your favorite IDE and see if you can fix this with me. I'll give the bounty to whoever gives me the working code. Thanks! Oh, and don't expect to use this Key and Secret for long. Once this problem's resolved I'll delete it. :)
Overview of the problem: The problem is when I send the last chunk of bytes to Vimeo and then verify the upload, the response returns that the length of all the content is the length of only the last chunk, not all the chunks combined as it should be.
SSCCE Note: I have my entire SSCCE here. I put it somewhere else so it can be C ompilable. It is NOT very S hort (about 300 lines), but hopefully you find it to be S elf-contained, and it's certainly an E xample!). I am, however, posting the relevant portions of my code in this post.
This is how it works: When you upload a video to Vimeo via the streaming method (see Upload API documentation here for setup to get to this point), you have to give a few headers: endpoint, content-length, and content-type. The documentation says it ignores any other headers. You also give it a payload of the byte information for the file you're uploading. And then sign and send it (I have a method which will do this using scribe).
My problem: Everything works great when I just send the video in one request. My problem is in cases when I'm uploading several bigger files, the computer I'm using doesn't have enough memory to load all of that byte information and put it in the HTTP PUT request, so I have to split it up into 1 MB segments. This is where things get tricky. The documentation mentions that it's possible to "resume" uploads, so I'm trying to do that with my code, but it's not working quite right. Below, you'll see the code for sending the video. Remember my SSCCE is here.
Things I've tried: I'm thinking it has something to do with the Content-Range header... So here are the things I've tried in changing what the Content-Range header says...
Not adding content range header to the first chunk
Adding a prefix to the content range header (each with a combination of the previous header):
"bytes"
"bytes " (throws connection error, see the very bottom for the error) --> It appears in the documentation that this is what they're looking for, but I'm pretty sure there are typos in the documentation because they have the content-range header on their "resume" example as: 1001-339108/339108 when it should be 1001-339107/339108. So... Yeah...
"bytes%20"
"bytes:"
"bytes: "
"bytes="
"bytes= "
Not adding anything as a prefix to the content range header
Here's the code:
/**
* Send the video data
*
* #return whether the video successfully sent
*/
private static boolean sendVideo(String endpoint, File file) throws FileNotFoundException, IOException {
// Setup File
long contentLength = file.length();
String contentLengthString = Long.toString(contentLength);
FileInputStream is = new FileInputStream(file);
int bufferSize = 10485760; // 10 MB = 10485760 bytes
byte[] bytesPortion = new byte[bufferSize];
int byteNumber = 0;
int maxAttempts = 1;
while (is.read(bytesPortion, 0, bufferSize) != -1) {
String contentRange = Integer.toString(byteNumber);
long bytesLeft = contentLength - byteNumber;
System.out.println(newline + newline + "Bytes Left: " + bytesLeft);
if (bytesLeft < bufferSize) {
//copy the bytesPortion array into a smaller array containing only the remaining bytes
bytesPortion = Arrays.copyOf(bytesPortion, (int) bytesLeft);
//This just makes it so it doesn't throw an IndexOutOfBounds exception on the next while iteration. It shouldn't get past another iteration
bufferSize = (int) bytesLeft;
}
byteNumber += bytesPortion.length;
contentRange += "-" + (byteNumber - 1) + "/" + contentLengthString;
int attempts = 0;
boolean success = false;
while (attempts < maxAttempts && !success) {
int bytesOnServer = sendVideoBytes("Test video", endpoint, contentLengthString, "video/mp4", contentRange, bytesPortion, first);
if (bytesOnServer == byteNumber) {
success = true;
} else {
System.out.println(bytesOnServer + " != " + byteNumber);
System.out.println("Success is not true!");
}
attempts++;
}
first = true;
if (!success) {
return false;
}
}
return true;
}
/**
* Sends the given bytes to the given endpoint
*
* #return the last byte on the server (from verifyUpload(endpoint))
*/
private static int sendVideoBytes(String videoTitle, String endpoint, String contentLength, String fileType, String contentRange, byte[] fileBytes, boolean addContentRange) throws FileNotFoundException, IOException {
OAuthRequest request = new OAuthRequest(Verb.PUT, endpoint);
request.addHeader("Content-Length", contentLength);
request.addHeader("Content-Type", fileType);
if (addContentRange) {
request.addHeader("Content-Range", contentRangeHeaderPrefix + contentRange);
}
request.addPayload(fileBytes);
Response response = signAndSendToVimeo(request, "sendVideo on " + videoTitle, false);
if (response.getCode() != 200 && !response.isSuccessful()) {
return -1;
}
return verifyUpload(endpoint);
}
/**
* Verifies the upload and returns whether it's successful
*
* #param endpoint to verify upload to
* #return the last byte on the server
*/
public static int verifyUpload(String endpoint) {
// Verify the upload
OAuthRequest request = new OAuthRequest(Verb.PUT, endpoint);
request.addHeader("Content-Length", "0");
request.addHeader("Content-Range", "bytes */*");
Response response = signAndSendToVimeo(request, "verifyUpload to " + endpoint, true);
if (response.getCode() != 308 || !response.isSuccessful()) {
return -1;
}
String range = response.getHeader("Range");
//range = "bytes=0-10485759"
return Integer.parseInt(range.substring(range.lastIndexOf("-") + 1)) + 1;
//The + 1 at the end is because Vimeo gives you 0-whatever byte where 0 = the first byte
}
Here's the signAndSendToVimeo method:
/**
* Signs the request and sends it. Returns the response.
*
* #param service
* #param accessToken
* #param request
* #return response
*/
public static Response signAndSendToVimeo(OAuthRequest request, String description, boolean printBody) throws org.scribe.exceptions.OAuthException {
System.out.println(newline + newline
+ "Signing " + description + " request:"
+ ((printBody && !request.getBodyContents().isEmpty()) ? newline + "\tBody Contents:" + request.getBodyContents() : "")
+ ((!request.getHeaders().isEmpty()) ? newline + "\tHeaders: " + request.getHeaders() : ""));
service.signRequest(accessToken, request);
printRequest(request, description);
Response response = request.send();
printResponse(response, description, printBody);
return response;
}
And here's some (an example... All of the output can be found here) of the output from the printRequest and printResponse methods: NOTE This output changes depending on what the contentRangeHeaderPrefix is set to and the first boolean is set to (which specifies whether or not to include the Content-Range header on the first chunk).
We're sending the video for upload!
Bytes Left: 15125120
Signing sendVideo on Test video request:
Headers: {Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes%200-10485759/15125120}
sendVideo on Test video >>> Request
Headers: {Authorization=OAuth oauth_signature="zUdkaaoJyvz%2Bt6zoMvAFvX0DRkc%3D", oauth_version="1.0", oauth_nonce="340477132", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336004", Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes: 0-10485759/15125120}
Verb: PUT
Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d
sendVideo on Test video >>> Response
Code: 200
Headers: {null=HTTP/1.1 200 OK, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0}
Signing verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d request:
Headers: {Content-Length=0, Content-Range=bytes */*}
verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d >>> Request
Headers: {Authorization=OAuth oauth_signature="FQg8HJe84nrUTdyvMJGM37dpNpI%3D", oauth_version="1.0", oauth_nonce="298157825", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336015", Content-Length=0, Content-Range=bytes */*}
Verb: PUT
Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d
verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d >>> Response
Code: 308
Headers: {null=HTTP/1.1 308 Resume Incomplete, Range=bytes=0-10485759, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0}
Body:
Bytes Left: 4639360
Signing sendVideo on Test video request:
Headers: {Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes: 10485760-15125119/15125120}
sendVideo on Test video >>> Request
Headers: {Authorization=OAuth oauth_signature="qspQBu42HVhQ7sDpzKGeu3%2Bn8tM%3D", oauth_version="1.0", oauth_nonce="183131870", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336015", Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes%2010485760-15125119/15125120}
Verb: PUT
Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d
sendVideo on Test video >>> Response
Code: 200
Headers: {null=HTTP/1.1 200 OK, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0}
Signing verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d request:
Headers: {Content-Length=0, Content-Range=bytes */*}
verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d >>> Request
Headers: {Authorization=OAuth oauth_signature="IdhhhBryzCa5eYqSPKAQfnVFpIg%3D", oauth_version="1.0", oauth_nonce="442087608", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336020", Content-Length=0, Content-Range=bytes */*}
Verb: PUT
Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d
4639359 != 15125120
verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d >>> Response
Success is not true!
Code: 308
Headers: {null=HTTP/1.1 308 Resume Incomplete, Range=bytes=0-4639359, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0}
Body:
Then the code goes on to complete the upload and set video information (you can see that in my full code).
Edit 2: Tried removing the "%20" from the content-range and received this error making connection. I must use either "bytes%20" or not add "bytes" at all...
Exception in thread "main" org.scribe.exceptions.OAuthException: Problems while creating connection.
at org.scribe.model.Request.send(Request.java:70)
at org.scribe.model.OAuthRequest.send(OAuthRequest.java:12)
at autouploadermodel.VimeoTest.signAndSendToVimeo(VimeoTest.java:282)
at autouploadermodel.VimeoTest.sendVideoBytes(VimeoTest.java:130)
at autouploadermodel.VimeoTest.sendVideo(VimeoTest.java:105)
at autouploadermodel.VimeoTest.main(VimeoTest.java:62)
Caused by: java.io.IOException: Error writing to server
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:622)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:634)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1317)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
at org.scribe.model.Response.<init>(Response.java:28)
at org.scribe.model.Request.doSend(Request.java:110)
at org.scribe.model.Request.send(Request.java:62)
... 5 more
Java Result: 1
Edit 1: Updated the code and output. Still need help!
I think your problem could simply be the result of this line:
request.addHeader("Content-Range", "bytes%20" + contentRange);
Try and replace "bytes%20" by simply "bytes "
In your output you see the corresponding header has incorrect content:
Headers: {
Content-Length=15125120,
Content-Type=video/mp4,
Content-Range=bytes%200-10485759/15125120 <-- INCORRECT
}
On the topic of Content-Range...
You're right that an example final block of content should have a range like 14680064-15125119/15125120. That's part of the HTTP 1.1 spec.
Here
String contentRange = Integer.toString(byteNumber + 1);
you start from 1 and not from 0 at the first iteration.
Here
request.addHeader("Content-Length", contentLength);
you put the entire file content length and not the length of the current chunk.
The vimeo API page says:
"The final step is to call vimeo.videos.upload.complete to queue up the video for transcoding. This call will return the video_id, which you can then use in other calls (to set the title, description, privacy, etc.). If you do not call this method, the video will not be processed."
I added this bit of code to the end and got it to work:
request = new OAuthRequest(Verb.PUT, "http://vimeo.com/api/rest/v2");
request.addQuerystringParameter("method", "vimeo.videos.upload.complete");
request.addQuerystringParameter("filename", video.getName());
request.addQuerystringParameter("ticket_id", ticket);
service.signRequest(token, request);
response = request.send();
Check this :
String contentRange="bytes "+lastBytesSend+"-"+ ((totalSize - lastBytesSend)-1)+"/"+totalSize ;
request.addHeader("Content-Range",contentRange);