We Keep Coding

HTTP 403 Forbidden coming when using HttpClient authentication mechanism for Sharepoint using NTLM

HTTP/1.1 403 FORBIDDEN
Response content length: 729
Result: soap:ServerException of type 'Microsoft.SharePoint.SoapServer.SoapServerException' was thrown.The security validation for this page is invalid. Click Back in your Web browser, refresh the page, and try your operation again.0x8102006d
My concern is how to get authenticated by proxy and also at the same time, to NTLM configured sharepoint in same session? Please help me.
Thanks in advance.
AuthCache authCache = new BasicAuthCache();
HttpHost targetHost = new HttpHost(<sharepointserverhostname>);
NTLMSchemeFactory f = new NTLMSchemeFactory();
HttpContext ctx = new BasicHttpContext();
AuthScheme ns = f.create(ctx);
authCache.put(targetHost, ns);
HttpClientContext localContext = HttpClientContext.create();
localContext.setAuthCache(authCache);
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(
new AuthScope(<proxyserverip>, 8080), new UsernamePasswordCredentials ("testdomain\phanigandeed", "Jul#2014"));
credsProvider.setCredentials(
AuthScope.ANY, new NTCredentials("phanigandeed", "Jul#2014", "", "testdomain"));
CloseableHttpClient httpclient = HttpClients.custom().setDefaultCredentialsProvider(credsProvider).build();
String listName = "PhaniList";
String description = "TestDescription";
String templateID = "101";
// for safety reasons, I had to remove the actual server details.
String endpointURL = <serviceurl>;
String result = "Failed";
String username = "phanigandeed";
String psWord = "Jul#2014";
String domainName = "vsnl";
String XML_DATA = new String("<?xml version=\"1.0\" encoding=\"utf-8\"?><soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\"><soap:Body><AddList xmlns=\"http://schemas.microsoft.com/sharepoint/soap/\"><listName>"
+ listName
+ "</listName><description>"
+ description
+ "</description><templateID>"
+ templateID
+ "</templateID></AddList></soap:Body></soap:Envelope>");
HttpPost httpPost = new HttpPost(endpointURL);
httpPost.setHeader(new BasicHeader("Content-Type",
"text/xml;charset=UTF-8"));
try {
StringEntity s = new StringEntity(XML_DATA, "UTF-8");
httpPost.setEntity(s);
System.out.println("executing request" + httpPost.getRequestLine());
HttpResponse response = httpclient.execute(httpPost, localContext);
HttpEntity entity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
if (entity != null) {
System.out.println("Response content length: "
+ entity.getContentLength());
result = EntityUtils.toString(entity);
System.out.println("Result: " + result);
entity.consumeContent();
}
System.out.println("result: " + result);
return;
} catch (Exception e) {
e.printStackTrace();
System.out.println("Sharepoint Create Library failed :"
+ e.getMessage());
return;
}

You need connect to SharePoint and retrieve a digest for authenticating any further requests, that is why it is throwing an exception about your security validation being invalid.
String digest;
try
{
string url = "https://Your.SharePoint.Site";
HttpClient client = new HttpClient(new HttpClientHandler() { UseDefaultCredentials = true });
client.BaseAddress = new System.Uri(url);
string cmd = "_api/contextinfo";
client.DefaultRequestHeaders.Add("Accept", "application/json;odata=verbose");
client.DefaultRequestHeaders.Add("ContentType", "application/json");
client.DefaultRequestHeaders.Add("ContentLength", "0");
StringContent httpContent = new StringContent("");
var response = client.PostAsync(cmd, httpContent).Result;
if (response.IsSuccessStatusCode)
{
string content = response.Content.ReadAsStringAsync().Result;
JsonObject val = JsonValue.Parse(content).GetObject();
JsonObject d = val.GetNamedObject("d");
JsonObject wi = d.GetNamedObject("GetContextWebInformation");
digest = wi.GetNamedString("FormDigestValue");
}
}
catch
{
MessageDialog dialog = new MessageDialog("Authentication to server failed. Please try again.");
dialog.ShowAsync();
}
Later you will need to add the digest to the header of your next request to pass along the authentication which would look something like this. This is C# and sorry I don't know the Java specific syntax but you need the Digest to work with Web Services in all platforms:
client.DefaultRequestHeaders.Add("X-RequestDigest", digest);

I have had issues with HTTPClient working with Sharepoint/IIS and NTLM authentication. I was able to get past these issue when I switched to Java HTTPURLConnection. Please see the following posts for reference
HTTP Client and NTLM issue
HTTP Call to Sharepoint fails
Hope this helps you.

how to set parameters to server from android application?

I have the following code to connect from my android application to zappos api server and search for some stuff. But It either returns error 404 or We are unable to process the request from the input feilds given.
When I execute the same query it works on the web browser.
The query is:
http://api.zappos.com/Search&term=boots&key=<my_key_inserted_here>
Code:
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost("http://api.zappos.com/Search");
NameValuePair keypair = new BasicNameValuePair("key",KEY);
NameValuePair termpair = new BasicNameValuePair("term",data);
List<NameValuePair> params = new ArrayList<NameValuePair>(2);
params.add(keypair);
params.add(termpair);
post.setEntity(new UrlEncodedFormEntity(params));
HttpResponse response = client.execute(post);
String str;
StringBuilder sb = new StringBuilder();
HttpEntity entity =response.getEntity();
if (entity != null) {
DataInputStream in = new DataInputStream(entity.getContent());
while (( str = in.readLine()) != null){
sb.append(str);
}
in.close();
}
Log.i("serverInterface","response from server is :"+sb.toString());
What am I doing wrong?

If I am correct, what you want to do is a GET request with parameters.
Then,the code would looks like something like that:
HttpClient client = new DefaultHttpClient();
HttpGet get = new HttpGet("http://api.zappos.com/Search");
HttpParams params = new BasicHttpParams();
params.setParameter("key", "KEY");
params.setParameter("term", "data");
get.setParams(params);
HttpResponse response;
response = client.execute(get);
String str;
StringBuilder sb = new StringBuilder();
HttpEntity entity = response.getEntity();
if (entity != null) {
DataInputStream in;
in = new DataInputStream(entity.getContent());
while ((str = in.readLine()) != null) {
sb.append(str);
}
in.close();
}
Log.i("serverInterface", "response from server is :" + sb.toString());

I found an answer to the question based on ALL of your help. I got the hint that I must search how to connect to REST service and I also used this result. This is the exact result I was looking for. Sadly it resembles too much to what I'm trying to achieve that I think whoever asked it might be applying to the same position :(

500 internal server error when using app engine, trying to upload an image

I'm trying to upload an image from android to app engine(Java). Im using BlobstoreService to store images. Here is what my project looks like:
On the server side, I do the following to generate the upload url
BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
String uploadURL = blobstoreService.createUploadUrl("/upload");
resp.getWriter().print(uploadURL);
On the client(Android) side, I get this when printed
http://mydomain.com/_ah/upload/AMmfu6YehP-o-VUwRXmCfJzYOZ1jvXiD33fF-LhZw5nO6XIwCE0cD0-zTwvg9beO0gXeNjuNG_wMNwnX9Yr5I3BVUdyolby4bnXQyHBWA-fpJNfR7sfqqukOIuMEa3EubgNWTj7q0bxY/ALBNUaYAAAAAUsDnWeK48TX7bblzmE0jUNgf22o4-IXP/
Please note the last character in the url, the '/'. When I use this as url to build httpurl 'HttpPost(mUploadURL)', I get Illegal character in path at index 231(which is '/') of url. So my question is -- what am I doing wrong here?
So, I removed the '/' from url as below:
HttpClient httpClient = new DefaultHttpClient();
HttpPost postRequest = null;
postRequest = new HttpPost(mUploadURL.substring(0, mUploadURL.length()-2));
try {
MultipartEntity reqEntity = new MultipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE);
reqEntity.addPart("title", new StringBody("new title"));
try{
ByteArrayOutputStream bos = new ByteArrayOutputStream();
Log.d("test", "mPhoto: " + mPhoto);
mPhoto.compress(CompressFormat.JPEG, 75, bos);
byte[] data = bos.toByteArray();
ByteArrayBody bab = new ByteArrayBody(data, "image/jpeg", "forest.jpg");
reqEntity.addPart("picture", bab);
}
catch(Exception e){
Log.d("test", "e: " + e);
reqEntity.addPart("picture", new StringBody(""));
}
postRequest.setEntity(reqEntity);
HttpResponse response = httpClient.execute(postRequest);
BufferedReader reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent(), "UTF-8"));
String sResponse;
StringBuilder s = new StringBuilder();
while ((sResponse = reader.readLine()) != null) {
s = s.append(sResponse);
}
Log.d("test", "after post: " + s.toString());
return s.toString();
} catch (Exception ee) {
Log.d("test", "ee: " + ee);
return "";
}
After post, I get this as result
<html><head><meta http-equiv="content-type" content="text/html;charset=utf-8"><title>500 Server Error</title></head><body text=#000000 bgcolor=#ffffff><h1>Error: Server Error</h1><h2>The server encountered an error and could not complete your request.<p>Please retry your last submission.</p><p>If the problem persists, please contact the person responsible for the application you're using, or, if you are that person, report your problem and mention this error message and the query that caused it.</h2></body></html>
My second question is -- what is wrong in the multipart post code? I use app engine as backend. Any help would be appreciated!

uploading files to a dataset in CKAN / datahub.io through a Java client

I am testing the uploading of files to a dataset on CKAN / datahub.io through a Java client of the API.
public String uploadFile()
throws CKANException {
String returned_json = this._connection.MultiPartPost("", "");
System.out.println("r: " + returned_json);
return returned_json;
}
and
protected String MultiPartPost(String path, String data)
throws CKANException {
URL url = null;
try {
url = new URL(this.m_host + ":" + this.m_port + path);
} catch (MalformedURLException mue) {
System.err.println(mue);
return null;
}
String body = "";
HttpClient httpclient = new DefaultHttpClient();
try {
String fileName = "D:\\test.jpg";
FileBody bin = new FileBody(new File(fileName),"image/jpeg");
StringBody comment = new StringBody("Filename: " + fileName);
MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("bin", bin);
reqEntity.addPart("comment", comment);
HttpPost postRequest = new HttpPost("http://datahub.io/api/storage/auth/form/2013-01-24T130158/test.jpg");
postRequest.setEntity(reqEntity);
postRequest.setHeader("X-CKAN-API-Key", this._apikey);
HttpResponse response = httpclient.execute(postRequest);
int statusCode = response.getStatusLine().getStatusCode();
System.out.println("status code: " + statusCode);
BufferedReader br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
String line;
while ((line = br.readLine()) != null) {
body += line;
}
System.out.println("body: " + body);
} catch (IOException ioe) {
System.out.println(ioe);
} finally {
httpclient.getConnectionManager().shutdown();
}
return body;
}
2 responses I get to my POST request:
a 413 error ("request entity too large") when the jpeg I try to upload is 2.83 Mb. This disappears when I shrink the file to a smaller size. Is there a limit to file size uploads?
a 500 error ("internal server error"). This is where I am stuck. It might have to do with the fact that my dataset on datahub.io is not "datastore enabled"? (I see a disabled "Data API" button next to my resource files in the dataset, with a tooltip saying:
"Data API is unavailable for this resource as DataStore is disabled"
=> is it a possible reason for this 500 error? If so, how could I enable it from the client side? (pointers to Python code would be useful!)
Thx!
PS: the dataset I am using for testing purposes: http://datahub.io/dataset/testapi

Only someone with access to the exception log could tell you why the 500 is occurring.
However, I'd check your request is the same as what you'd get from the python client that was written alongside the datastore: https://github.com/okfn/ckanclient/blob/master/ckanclient/init.py#L546
You're sending the "bin" image buffer and "comment" file_key in your multipart request. Note the file_key must be changed for every upload, so add in a timestamp or something. And maybe you need to add in a Content-Type: for the binary.

I have been going through the same kind of troubles as the poster of this question. After quite a bit of trial and error, I came up with a solution to the problem. In my case, I had some control over the CKAN repository that I wanted to upload to. If you don't, your problem might be impossible to solve...
I assume you are using the 1.8 version of CKAN?
First of all, check whether the CKAN repository has been set up to allow file upload and if not, configure it to allow that. This can be done on the server using the steps posted here: http://docs.ckan.org/en/ckan-1.8/filestore.html#local-file-storage
The 413 error that you mentioned should be adressed next. This has to do with the general configuration of the server. In my case, the CKAN was hosted through nginx. I added a "client_max_body_size 100M" line to the nginx.conf file. See this post for instance: http://recursive-design.com/blog/2009/11/18/nginx-error-413-request-entity-too-large/
Then there is only the 500 error left. At the time of this writing, the api documentation of CKAN is still a little immature... It does indeed say that you have to build a request like you have made for file upload. However, this request is just to ask for permission for the file upload. If your credentials check out for file upload (not every user may be allowed to upload files), the response holds an object telling you where to send your file to... Because of the unclear api, you ended up merging these two requests.
The following scenario shows a follow up of two requests to handle the file upload. It might be that some steps in the scenario work out differently in your case, because of a repository that has been set up a little differently. If you get error messages, please be sure to check the response's body for clues!
Here is the authentication request that I used:
String body = "";
String generatedFilename=null;
HttpClient httpclient = new DefaultHttpClient();
try {
// create new identifier for every file, use time
SimpleDateFormat dateFormatGmt = new SimpleDateFormat("yyyyMMMddHHmmss");
dateFormatGmt.setTimeZone(TimeZone.getTimeZone("GMT"));
String date=dateFormatGmt.format(new Date());
generatedFilename=date +"/"+filename;
HttpGet getRequest = new HttpGet(this.CKANrepos+ "/api/storage/auth/form/"+generatedFilename);
getRequest.setHeader(CKANapiHeader, this.CKANapi);
HttpResponse response = httpclient.execute(getRequest);
int statusCode = response.getStatusLine().getStatusCode();
BufferedReader br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
String line;
while ((line = br.readLine()) != null) {
body += line;
}
if(statusCode!=200){
throw new IllegalStateException("File reservation failed, server responded with code: "+statusCode+
"\n\nThe message was: "+body);
}
}finally {
httpclient.getConnectionManager().shutdown();
}
Now, if all goes well, the server responds with a json object holding the parameters to use when doing the actual file upload. In my case, the object looked like:
{file_key:"some-filename-to-use-when-uploading"}
Be sure to check the json object though, as I'm given to understand that there may be custom ckan repositories that require more or different parameters.
These responses can then be used in the actual file upload:
File file = new File("/tmp/file.rdf");
String body = "";
HttpClient httpclient = new DefaultHttpClient();
try {
FileBody bin = new FileBody(file,"application/rdf+xml");
MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("file", bin);
reqEntity.addPart("key", new StringBody(filename));
HttpPost postRequest = new HttpPost(this.CKANrepos+"/storage/upload_handle");
postRequest.setEntity(reqEntity);
postRequest.setHeader(CKANapiHeader, this.CKANapi);
HttpResponse response = httpclient.execute(postRequest);
int statusCode = response.getStatusLine().getStatusCode();
BufferedReader br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
String line;
while ((line = br.readLine()) != null) {
body += line;
}
if(statusCode!=200){
getWindow().showNotification("Upload Statuscode: "+statusCode,
body,
Window.Notification.TYPE_ERROR_MESSAGE);
}
}finally {
httpclient.getConnectionManager().shutdown();
}
as you can see, the file_key property has now been transformed into the simple 'key' property. I don't know why.
This will get your file uploaded. The response to this upload request will hold a json object telling you where the file got uploaded to. edit: actually it seems that my ckan responded with a simple html page to tell me that the file got uploaded... I had to parse the page to confirm that the file was uploaded correctly :(
In my case, the file was at
this.CKANrepos +"/storage/f/"+location
where location is the filename returned in the authentication phase.
In the previous code fragments:
//the location of your ckan repository, including /api and possibly version, e.g.
this.CKANrepos = "http://datahub.io/api/3/";
this.CKANapiHeader="X-CKAN-API-Key";
this.CKANapi = "your ckan api key here";

Google Reader editing API Authentication problem

I have problem with authentication GReader editing API. I can do the HTTPS (https://www.google.com/accounts/ClientLogin) for authentication and google is returning three tokens (SID, LSID, AUTH), but no HSID.
When I try add a new feed http://www.google.com/reader/api/0/subscription/quickadd?ck=1290452912454&client=scroll with POST data T=djj72HsnLS8293&quickadd=blog.martindoms.com/feed/ without HSID in Cookie, is response status code 401. With SID and HSID in Cookie everything works properly.
What is and where can I find this HSID string?
Thaks for your answers.
My code:
public void addNewFeed() throws IOException {
HttpPost requestPost = new HttpPost("http://www.google.com/reader/api/0/subscription/quickadd?ck=1290452912454&client=scroll");
getSid();
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>();
DefaultHttpClient client = new DefaultHttpClient();
requestPost.addHeader("Cookie", "SID=" + _sid + "; HSID=" + _hsid);
try {
nameValuePairs.add(new BasicNameValuePair("T", _token));
nameValuePairs.add(new BasicNameValuePair("quickadd", "blog.martindoms.com/feed/"));
requestPost.setEntity(new UrlEncodedFormEntity(nameValuePairs));
HttpResponse response = client.execute(requestPost);
InputStream in = response.getEntity().getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder str = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
str.append(line + "\n");
}
System.out.println(str.toString());
in.close();
} catch (Exception ex) {
ex.printStackTrace();
}
}

Looks like you might be using old info as a reference. Google switched to using auth now.
You'll need to replace getSid() with a getAuth() function.
Then this line
requestPost.addHeader("Cookie", "SID=" + _sid + "; HSID=" + _hsid);
should now be this
requestPost.addHeader("Authorization", "GoogleLogin auth=" + _auth);