We Keep Coding

Microsoft Graph: Requesting an Extension returns http 400 bad request

I added an open extension to an event in a calendar and am trying to read it back.
Here is the url:
https://graph.microsoft.com/v1.0/users/{userid}/calendars/{calendarId}=/events?$expand=Extensions($filter=Id eq 'c.i.m.p.server.entities.outlook.Event')
I cannot get this to work in a Java program. The following combinations do work:
It works my Java program if I remove the $expand... parameter. I can also ask for certain fields, that works too.
The request works in Postman (I just have to set the token)
The request works in Graph Explorer when I log in as the owner of the calendar
Here is the extension (inside one of the events) when I use Postman to read the event. It is the last item in the event:
"extensions#odata.context": "https://graph.microsoft.com/v1.0/$metadata#users('{userid}')/calendars('{calendarId}')/events('{eventId})/extensions",
"extensions": [
{
"#odata.type": "#microsoft.graph.openTypeExtension",
"id": "Microsoft.OutlookServices.OpenTypeExtension.c.i.m.p.server.entities.outlook.Event",
"extensionName": "c.i.m.p.server.entities.outlook.Event",
"adherentId": "12346",
"timeSlotID": "346463"
}
]
Here is the Java code (Java 8, using java.io and java.net libraries):
private static void doSomething(String _accessToken) throws IOException {
String urlString = "https://graph.microsoft.com/v1.0/users/{userId}/calendars/{calendarId}/events?$expand=Extensions($filter=Id eq 'c.i.m.p.server.entities.outlook.Event')";
URL url = new URL(urlString);
Proxy webProxy
= new Proxy(Proxy.Type.HTTP, new InetSocketAddress({proxy-address}, {port}));
HttpURLConnection connection = (HttpURLConnection) url.openConnection(webProxy);
// Set the appropriate header fields in the request header.
connection.setRequestProperty("Authorization", "Bearer " + _accessToken);
connection.setRequestProperty("Accept", "application/json");
connection.setDoOutput(true);
connection.setReadTimeout(5000);
connection.setRequestMethod(HttpMethod.GET);
try {
connection.connect();
int responseCode = connection.getResponseCode();
System.out.println("execute(), response code = " + responseCode);
String responseMessage = connection.getResponseMessage();
System.out.println("execute(), response Message = " + responseMessage);
String responseString = null;
try {
InputStream ins = connection.getInputStream();
BufferedReader br=new BufferedReader(new InputStreamReader(ins));
StringBuffer sb=new StringBuffer();
String line;
while ((line=br.readLine()) != null) {
sb.append(line);
}
responseString = sb.toString();
} catch (Exception e) {
System.out.println("Could not get input stream from response, error is " + e.toString());
}
System.out.println("execute(), httpResult = " + responseString);
} catch (IOException e) {
System.out.println(".execute(), IOException : " + e.toString());
} finally {
connection.disconnect();
}
}
How do I fix this? Thanks!

400 means bad request. It could be because of url encoding. Url encode the query string.
Something like
String query = "Extensions($filter=Id eq 'c.i.m.p.server.entities.outlook.Event'";
String url = "https://graph.microsoft.com/v1.0/users/{userId}/calendars/{calendarId}/events?
$expand=" + URLEncoder.encode(query, StandardCharsets.UTF_8.name());
Alternatively you could use graph service java api based on your need which will help abstract all the interactions for you or you could use any of the rest clients available.

First of all, you should provide more info on the error - Stacktrace and error message. But 400 code indicates that was a user mistake, meaning that you are sending an invalid request. Since you say that postman request works then compare all the headers that are sent by postman and see if your code misses some hearer. As for the code, instead of coding your own Http client functionality I would suggest using 3d party Http client. Here are a few suggestions:
Apache Http client - very popular and well known 3d party Http Client
OK Http client - Open-source Http client. Here is tutorial
MgntUtils Http client - very simple 3d party HttpClient: Provided in MgntUtils Open source library (written by me). Very simple in use. Take a look at Javadoc. Library itself provided as Maven artifacts and on Git (including source code and Javadoc).

Http POST in android app without data

I'm trying to send a video url in a http POST request but it's not working for me, I think I've (almost?) the necessary code to make it work, or else I'm missing something very simple?
public void postVideoURL() throws IOException {
String encodedUrl = URLEncoder.encode("http://video.ted.com/talks/podcast/DavidBrooks_2011.mp4", "UTF-8");
URL obj = new URL("http://10.50.0.105:8060/launch/dev?url="+encodedUrl);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
//add request header
con.setRequestMethod("POST");
// Send post request
con.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(con.getOutputStream());
System.out.println(con.getResponseCode());
System.out.println(con.getResponseMessage());
wr.flush();
wr.close();
wr.write("");
}
Any tips to lead me to the right direction?
Here is what I'm trying to do but in C#
using System.Net;
using System.Web;
class Program
{
static void Main()
{
string rokuIp = "192.168.0.6";
string channelId = "dev";
string videoUrl = "http://video.ted.com/talks/podcast/DavidBrooks_2011.mp4";
// Note that the query string parameters should be url-encoded
string requestUrl = $"http://{rokuIp}:8060/launch/{channelId}?url={HttpUtility.UrlEncode(videoUrl)}";
using (var wc = new WebClient())
{
// POST the query string with no data
wc.UploadString(requestUrl, "");
}
}
}
The following Post command to use in terminal works, this is essentially what I want to do, but in Java:
curl -d "" "http://10.50.0.46:8060/launch/12?url=http%3A%2F%2Fvideo.ted.com%2Ftalks%2Fpodcast%2FDavidBrooks_2011.mp4"

You are never writing anything to the output stream. You have to call wr.write() to write your data to the stream.
Also, you can't encode the URL like that inside the String. You need to concatenate the two Strings together after you've encoded the url separately. Like this:
String encodedUrl = URLEncoder.encode("http://video.ted.com/talks/podcast/DavidBrooks_2011.mp4");
URL obj = new URL("http://10.50.0.105:8060/launch/dev?url="+encodedUrl);

Getting JSON response as part of Rest call in Java

I am trying to make Rest service call in Java. I am new to web and Rest service. I have Rest service which returns JSON as response. I have the following code but I think it's incomplete because I don't know how to process output using JSON.
public static void main(String[] args) {
try {
URL url = new URL("http://example.com:7000/test/db-api/processor");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setInstanceFollowRedirects(false);
connection.setRequestMethod("PUT");
connection.setRequestProperty("Content-Type", "application/json");
OutputStream os = connection.getOutputStream();
//how do I get json object and print it as string
os.flush();
connection.getResponseCode();
connection.disconnect();
} catch(Exception e) {
throw new RuntimeException(e);
}
}
I am new to Rest services and JSON.

Since this is a PUT request you're missing a few things here:
OutputStream os = conn.getOutputStream();
os.write(input.getBytes()); // The input you need to pass to the webservice
os.flush();
...
BufferedReader br = new BufferedReader(new InputStreamReader(
(conn.getInputStream()))); // Getting the response from the webservice
String output;
System.out.println("Output from Server .... \n");
while ((output = br.readLine()) != null) {
System.out.println(output); // Instead of this, you could append all your response to a StringBuffer and use `toString()` to get the entire JSON response as a String.
// This string json response can be parsed using any json library. Eg. GSON from Google.
}
Have a look at this to have a more clear idea on hitting webservices.

Your code is mostly correct, but there is mistake about OutputStream.
As R.J said OutputStream is needed to pass request body to the server.
If your rest service doesn't required any body you don't need to use this one.
For reading the server response you need use InputStream(R.J also show you example) like that:
try (InputStream inputStream = connection.getInputStream();
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();) {
byte[] buf = new byte[512];
int read = -1;
while ((read = inputStream.read(buf)) > 0) {
byteArrayOutputStream.write(buf, 0, read);
}
System.out.println(new String(byteArrayOutputStream.toByteArray()));
}
This way is good if you don't want to depends on third-part libraries. So I recommend you to take a look on Jersey - very nice library with huge amount of very useful feature.
Client client = JerseyClientBuilder.newBuilder().build();
Response response = client.target("http://host:port").
path("test").path("db-api").path("processor").path("packages").
request().accept(MediaType.APPLICATION_JSON_TYPE).buildGet().invoke();
System.out.println(response.readEntity(String.class));

Since your Content-Type is application/json, you could directly cast the response to a JSON object for example
JSONObject recvObj = new JSONObject(response);

JsonKey jsonkey = objectMapper.readValue(new URL("http://echo.jsontest.com/key/value/one/two"), JsonKey.class);
System.out.println("jsonkey.getOne() : "+jsonkey.getOne())

uploading files to a dataset in CKAN / datahub.io through a Java client

I am testing the uploading of files to a dataset on CKAN / datahub.io through a Java client of the API.
public String uploadFile()
throws CKANException {
String returned_json = this._connection.MultiPartPost("", "");
System.out.println("r: " + returned_json);
return returned_json;
}
and
protected String MultiPartPost(String path, String data)
throws CKANException {
URL url = null;
try {
url = new URL(this.m_host + ":" + this.m_port + path);
} catch (MalformedURLException mue) {
System.err.println(mue);
return null;
}
String body = "";
HttpClient httpclient = new DefaultHttpClient();
try {
String fileName = "D:\\test.jpg";
FileBody bin = new FileBody(new File(fileName),"image/jpeg");
StringBody comment = new StringBody("Filename: " + fileName);
MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("bin", bin);
reqEntity.addPart("comment", comment);
HttpPost postRequest = new HttpPost("http://datahub.io/api/storage/auth/form/2013-01-24T130158/test.jpg");
postRequest.setEntity(reqEntity);
postRequest.setHeader("X-CKAN-API-Key", this._apikey);
HttpResponse response = httpclient.execute(postRequest);
int statusCode = response.getStatusLine().getStatusCode();
System.out.println("status code: " + statusCode);
BufferedReader br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
String line;
while ((line = br.readLine()) != null) {
body += line;
}
System.out.println("body: " + body);
} catch (IOException ioe) {
System.out.println(ioe);
} finally {
httpclient.getConnectionManager().shutdown();
}
return body;
}
2 responses I get to my POST request:
a 413 error ("request entity too large") when the jpeg I try to upload is 2.83 Mb. This disappears when I shrink the file to a smaller size. Is there a limit to file size uploads?
a 500 error ("internal server error"). This is where I am stuck. It might have to do with the fact that my dataset on datahub.io is not "datastore enabled"? (I see a disabled "Data API" button next to my resource files in the dataset, with a tooltip saying:
"Data API is unavailable for this resource as DataStore is disabled"
=> is it a possible reason for this 500 error? If so, how could I enable it from the client side? (pointers to Python code would be useful!)
Thx!
PS: the dataset I am using for testing purposes: http://datahub.io/dataset/testapi

Only someone with access to the exception log could tell you why the 500 is occurring.
However, I'd check your request is the same as what you'd get from the python client that was written alongside the datastore: https://github.com/okfn/ckanclient/blob/master/ckanclient/init.py#L546
You're sending the "bin" image buffer and "comment" file_key in your multipart request. Note the file_key must be changed for every upload, so add in a timestamp or something. And maybe you need to add in a Content-Type: for the binary.

I have been going through the same kind of troubles as the poster of this question. After quite a bit of trial and error, I came up with a solution to the problem. In my case, I had some control over the CKAN repository that I wanted to upload to. If you don't, your problem might be impossible to solve...
I assume you are using the 1.8 version of CKAN?
First of all, check whether the CKAN repository has been set up to allow file upload and if not, configure it to allow that. This can be done on the server using the steps posted here: http://docs.ckan.org/en/ckan-1.8/filestore.html#local-file-storage
The 413 error that you mentioned should be adressed next. This has to do with the general configuration of the server. In my case, the CKAN was hosted through nginx. I added a "client_max_body_size 100M" line to the nginx.conf file. See this post for instance: http://recursive-design.com/blog/2009/11/18/nginx-error-413-request-entity-too-large/
Then there is only the 500 error left. At the time of this writing, the api documentation of CKAN is still a little immature... It does indeed say that you have to build a request like you have made for file upload. However, this request is just to ask for permission for the file upload. If your credentials check out for file upload (not every user may be allowed to upload files), the response holds an object telling you where to send your file to... Because of the unclear api, you ended up merging these two requests.
The following scenario shows a follow up of two requests to handle the file upload. It might be that some steps in the scenario work out differently in your case, because of a repository that has been set up a little differently. If you get error messages, please be sure to check the response's body for clues!
Here is the authentication request that I used:
String body = "";
String generatedFilename=null;
HttpClient httpclient = new DefaultHttpClient();
try {
// create new identifier for every file, use time
SimpleDateFormat dateFormatGmt = new SimpleDateFormat("yyyyMMMddHHmmss");
dateFormatGmt.setTimeZone(TimeZone.getTimeZone("GMT"));
String date=dateFormatGmt.format(new Date());
generatedFilename=date +"/"+filename;
HttpGet getRequest = new HttpGet(this.CKANrepos+ "/api/storage/auth/form/"+generatedFilename);
getRequest.setHeader(CKANapiHeader, this.CKANapi);
HttpResponse response = httpclient.execute(getRequest);
int statusCode = response.getStatusLine().getStatusCode();
BufferedReader br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
String line;
while ((line = br.readLine()) != null) {
body += line;
}
if(statusCode!=200){
throw new IllegalStateException("File reservation failed, server responded with code: "+statusCode+
"\n\nThe message was: "+body);
}
}finally {
httpclient.getConnectionManager().shutdown();
}
Now, if all goes well, the server responds with a json object holding the parameters to use when doing the actual file upload. In my case, the object looked like:
{file_key:"some-filename-to-use-when-uploading"}
Be sure to check the json object though, as I'm given to understand that there may be custom ckan repositories that require more or different parameters.
These responses can then be used in the actual file upload:
File file = new File("/tmp/file.rdf");
String body = "";
HttpClient httpclient = new DefaultHttpClient();
try {
FileBody bin = new FileBody(file,"application/rdf+xml");
MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("file", bin);
reqEntity.addPart("key", new StringBody(filename));
HttpPost postRequest = new HttpPost(this.CKANrepos+"/storage/upload_handle");
postRequest.setEntity(reqEntity);
postRequest.setHeader(CKANapiHeader, this.CKANapi);
HttpResponse response = httpclient.execute(postRequest);
int statusCode = response.getStatusLine().getStatusCode();
BufferedReader br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
String line;
while ((line = br.readLine()) != null) {
body += line;
}
if(statusCode!=200){
getWindow().showNotification("Upload Statuscode: "+statusCode,
body,
Window.Notification.TYPE_ERROR_MESSAGE);
}
}finally {
httpclient.getConnectionManager().shutdown();
}
as you can see, the file_key property has now been transformed into the simple 'key' property. I don't know why.
This will get your file uploaded. The response to this upload request will hold a json object telling you where the file got uploaded to. edit: actually it seems that my ckan responded with a simple html page to tell me that the file got uploaded... I had to parse the page to confirm that the file was uploaded correctly :(
In my case, the file was at
this.CKANrepos +"/storage/f/"+location
where location is the filename returned in the authentication phase.
In the previous code fragments:
//the location of your ckan repository, including /api and possibly version, e.g.
this.CKANrepos = "http://datahub.io/api/3/";
this.CKANapiHeader="X-CKAN-API-Key";
this.CKANapi = "your ckan api key here";

Getting a compressed version of web page

I am using HttpClient 4.1 to download a web page. I would like to get a compressed version:
HttpGet request = new HttpGet(url);
request.addHeader("Accept-Encoding", "gzip,deflate");
HttpResponse response = httpClient.execute(request,localContext);
HttpEntity entity = response.getEntity();
response.getFirstHeader("Content-Encoding") shows "Content-Encoding: gzip"
however, entity.getContentEncoding() is null.
If I put:
entity = new GzipDecompressingEntity(entity);
I get:
java.io.IOException: Not in GZIP format
It looks like the resulting page is plain text and not compressed even though "Content-Encoding" header shows it's gzipped.
I have tried this on several URLs (from different websites) but get the same results.
How can I get a compressed version of a web page?

Don't use HttpClient if you don't want your API to handle mundane things like unzipping.
You can use the basic URLConnection class to fetch the compressed stream, as demonstrated by the following code :
public static void main(String[] args) {
try {
URL url = new URL("http://code.jquery.com/jquery-latest.js");
URLConnection con = url.openConnection();
// comment next line if you want to have something readable in your console
con.addRequestProperty("Accept-Encoding", "gzip,deflate");
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String l;
while ((l=in.readLine())!=null) {
System.out.println(l);
}
} catch (Exception e) {
e.printStackTrace();
}
}