We Keep Coding

401 for PostRequest with HttpURLConnection

UPDATE
My problem turned out to be due to some super-class weirdness. However, the urlEncode in this example is wrong. In this case it was also unnecessary. If needed it should be done by each key and value without encoding & or =.
Other than that, this code is correct.
The (scala) code below fails because the server does not seem to receive the body.
The warning on the server log is:
WARNING *** OAuthTokenProcessor caught a OAuthRequestProblemException with message OAuthProblemException{error='invalid_request', description='Missing grant_type parameter value', uri='null', state='null', scope='null', redirectUri='null', responseStatus=0, parameters={}}
I am certain that the params are correct. The url is correct, The headers are correct
(This call already works in Postman and Python)
Any hints deeply appreciated!
private def doPostRequest(): Unit ={
try {
val connectionforPost = (new URL(url)).openConnection.asInstanceOf[HttpURLConnection]
val params = "grant_type=password&client_id=xxxx&client_secret=seccret&username=admin&password=xxxx"
val encodedString = URLEncoder.encode(params,"UTF-8")
import java.nio.charset.StandardCharsets
val postData = encodedString.getBytes(StandardCharsets.UTF_8)
val contentlength = Integer.toString(encodedString.length)
connectionforPost.setConnectTimeout(50000)
connectionforPost.setReadTimeout(50000)
connectionforPost.setRequestMethod("POST")
connectionforPost.setDoOutput(true)
connectionforPost.setDoInput(true)
connectionforPost.setInstanceFollowRedirects(false)
connectionforPost.setRequestProperty("User-agent", "test bot")
connectionforPost.setRequestProperty("Content-Type", "application/x-www-form-urlencoded")
connectionforPost.setRequestProperty("Accept", "*/*")
connectionforPost.setRequestProperty("Connection", "keep-alive")
connectionforPost.setRequestProperty("Content-Length", contentlength)
connectionforPost.setRequestProperty("Accept-Encoding", "gzip, deflate")
connectionforPost.setUseCaches(false)
import java.io.DataOutputStream
val wr = new DataOutputStream(connectionforPost.getOutputStream)
wr.write(postData)
if (wr != null) wr.close()
} catch{
case e:Exception => logger.error(e.getMessage,e)
throw e
}
}

It appears that the server complains that you are not logged in. You get an authentication problem exception. How it usually works is that you send a request to the server to login with the user name and password. If the server authenticates you (recognizes your user name and password as valid ones) it sends you a response with an authentication token in the headers. You may see those 2 headers in your response to successful login:
access-control-expose-headers=[Authorization], Authorization=[Bearer ....]
This means that in all your subsequent requests you will need to add a header "Authorization" that will hold the value "Bearer ....". Then the server will recognize this request as coming from an authenticated user.
Also, I suggest that you may use a 3d party HTTP client. Some well-known clients are Appache HTTP client and OK HTTP client. I use my own Http client that I wrote. It is also available as part of a MgntUtils Open Source Library. Here is the link to Maven artifacts and it is also available on Github with source code and JavaDoc. And here is a javadoc page for HttpClient. Here is the code sample on how you may obtain the token and use it:
try {
HttpClient loginClient = new HttpClient();
loginClient.setConnectionUrl("http://your_url/login");
loginClient.setRequestProperty("accept", "application/json;charset=UTF-8");
loginClient.setContentType("application/json");
String result = loginClient.sendHttpRequest(HttpMethod.POST, "{ \"username\": \"your_user_name\", \"password\": \"Your_password\"}");
System.out.println(result);
System.out.println("HTTP " + loginClient.getLastResponseCode() + " " + loginClient.getLastResponseMessage());
System.out.println("Response headers: " + loginClient.getLastResponseHeaders());
String accessControlExposeHeader = loginClient.getLastResponseHeader("access-control-expose-headers").get(0);
String accessKey = loginClient.getLastResponseHeader(accessControlExposeHeader).get(0);
HttpClient workingClient = new HttpClient();
workingClient.setRequestProperty("accept", "application/json;charset=UTF-8");
workingClient.setContentType("application/json");
workingClient.setRequestProperty(accessControlExposeHeader, accessKey);
workingClient.setConnectionUrl("http://yourUrl/yourPath");
System.out.println(workingClient.sendHttpRequest(HttpMethod.GET));
} catch (IOException e) {
System.out.println(TextUtils.getStacktrace(e));
}

How to create a cookie and use it in HttpURLConnection?

I have the following python code which creates a cookie and adds it to the session. What would be the equivalent java code for it using HttpURLConnection? I basically want to do a HTTP POST request using the generated cookie.
session = requests.session()
session.auth = (username, password)
try:
token = session.get(SITEMINDER_URL % server, verify=False)
session.cookies.update(dict(SMSESSION=json.loads(token.content)['SMSESSION']))
except Exception as ex:
raise Exception("Failed in authenticating with siteminder", ex)
response = session.post(api_url, headers=headers, verify=False, json=data)

You would use something like this:
HttpURLConnection httpconn = < some source to get a HttpURLConnection >;
String cookieName = "SMSESSION"; // note this is the default but SM can use other prefixes
String cookieValue = < your token content >;
httpurl.setRequestProperty("Cookie", cookieName + "=" + cookieValue);
Also, from the javadocs: NOTE: HTTP requires all request properties which can legally have multiple instances with the same key to use a comma-separated list syntax which enables multiple properties to be appended into a single property
Which leads me to pointing out that using the HttpUrlConnection directly is really clumsy. I recommend you look at an HTTP client library such as Apache HTTP Client http://hc.apache.org/httpcomponents-client-ga/

In my opinion, you can just create an HttpUrlConnection object assigning a List of Cookies as follows:
List<String> cookies = new ArrayList<>();
//Or using a map With entries: Key and value for each cookie
cookies.add("User-Agent=MyUserAgent"); //etc...
URL site = new URL("https://myurl.com");
HttpsURLConnection conn = (HttpsURLConnection) site.openConnection();
for (String string: cookies) {
conn.setRequestProperty("Cookie", string);
}
However this is the simplest but not the best way to do so.

To get higher abstraction for Cookie use CookieManager and CookieStore classes. Here is an example:
HttpURLConnection connection
CookieManager cookieManager = new CookieManager();
HttpCookie cookie = new HttpCookie("cookieName","cookieValue");
cookieManager.getCookieStore().add(null,cookie);
connection.setRequestProperty("Cookie", String.join( ";", cookieManager.getCookieStore().getCookies()));

Try This:
URL url = new URL("http://www.example.com");
HttpURLConnection conn = (HttpURLConnection)url.openConnection();
conn.setRequestProperty("Cookie", "name1=value1; name2=value2");
conn.connect();

How to enable cookies when opening a URL's connection

I am trying to use the following code to get the redirected URL and then do some processing on it. But when I print the redirected link, it goes to a page that informs the absence of cookies. how is it possible to enable cookies while opening the url's connection?
String url = "http://dx.doi.org/10.1137/0210059";
URLConnection con = new URL( url ).openConnection();
con.getInputStream();
String redirctedURL= con.getURL().toString();
System.out.println(redirctedURL);

When using java UrlConnection's, you should handle the cookies by yourself, to read and set the cookies, you can use the setRequestProperty() and getHeaderField() of URLConnection.
The remaining part is parsing the cookies by yourself, an example how the can be done is as follows:
Map<String, String> cookie = new HashMap<>();
public URLConnection doConnctionWithCookies(URL url) {
StringBuilder builder = new StringBuilder();
builder.append("&");
for(Map.Entry<String,String> entry : cookie.entrySet()) {
builder.append(urlenEncode(entry.getKey()))
.append("=")
.append(urlenEncode(entry.getValue()))
.append("&");
}
builder.setLength(builder.length() - 1);
URLConnection con = url.openConnection();
con.setRequestProperty("Cookie", builder.toString());
con.connect();
// Parse cookie headers
List<String> setCookie = con.getHeaderFields().get("set-cookie");
// HTTP Spec state header fields MUST be case INsentive, I expet the above to work with all servers
if(setCookie == null)
return con;
// Parse all the cookies
for (String str : setCookie) {
String[] cookieKeyValue = str.split(";")[0].split("=",2);
if (cookieKeyValue.length != 2) {
continue;
}
cookie.put(urlenDecode(cookieKeyValue[0]), urlenDecode(cookieKeyValue[1]));
}
return con;
}
public String urlenEncode(String en) {
return URLEncoder.encode(en, "UTF-8");
}
public String urlenDecode(String en) {
return URLDecoder.decode(en, "UTF-8");
}
The above implementation is a very stupid and crude way of implementation cookies, while it works, it totally ignores the fact that cookies can also have a host parameter to prevent identification cookies be shared across multiple hosts.
A better way than doing it yourself can be using a library dedicated for the task like Apache HttpClient.

Authenticate with CouchDB and insert new database via HTTP PUT in Android app

I am implementing an Android app that should upload data to CouchDB. Since I have restricted the admin access to one account, I have to authenticate before inserting a new database. And this is what I am currently struggling with: Authenticate and insert a new database. Operating via Terminal and using curl, everything is working out fine the following way:
> curl -X PUT http://admin_name:admin_password#url:port/database_to_be_inserted
First approach
My first approach was to simply do the same via HTTP PUT in my code like that:
private boolean putJSON(String json, String url) {
// url = http://admin_name:admin_password#url:port/database_to_be_inserted
HttpClient client = new DefaultHttpClient();
HttpPut put = new HttpPut(url);
try {
StringEntity stringEntity = new StringEntity(json,"utf-8");
put.setEntity(stringEntity);
put.setHeader("Content-type", "application/json; charset=utf-8");
put.setHeader("Accept", "application/json");
HttpResponse response = client.execute(put);
// ... buffered input reading on response...
return true;
} catch (IOException e) {
e.printStackTrace();
return false;
}
}
However, doing so I retrieve the following error and JSON array:
Authentication error: Unable to respond to any of these challenges: {}
{"error":"unauthorized","reason":"You are not a server admin."}
The point is, that using the same method for inserting a new user works out perfectly. So, if I am using the above method with a correctly formatted user JSON-Dictionary and the following url, the user is inserted correctly.
http://admin_name:admin_password#url:port/_users/org.couchdb.user:user_name
This should prove, that I am using the right admin data at least, shouldn't it?
Second approach
So, by now, I am trying to authenticate using the "Authorization" option in my HTTP PUT's header:
private boolean putDatabase(String userName, String password, String url) {
// url = "http://url:port/database_to_be_inserted"
HttpClient client = new DefaultHttpClient();
HttpPut put = new HttpPut(url);
String authenticationData = userName+":"+password;
String encoding = Base64.encodeToString(authenticationData.getBytes(Charset.forName("utf-8")), Base64.DEFAULT);
put.setHeader("Authorization", "Basic " + encoding);
try {
put.setHeader("Content-type", "application/json; charset=utf-8");
put.setHeader("Accept", "application/json");
HttpResponse response = client.execute(put);
// ... buffered input reading on response...
return true;
} catch (IOException e) {
e.printStackTrace();
return false;
}
}
Still no success in inserting the database. The response I am parsing says:
Host not found
I have double checked the admin name, password, and url and everything seems correct. Does anyone of you see why this might not work out?

Ok, the answer is simple: The above code (at least the one of my second approach) is working fine. My mistake was to not explicitly specify the port via which the CouchDB should be accessed. This is, how I accidentally called the method:
putDatabase("adminName", "adminPassword", "http://url/database_to_be_inserted");
However, this is how I should have called it:
putDatabase("adminName", "adminPassword", "http://url:port/database_to_be_inserted");
Who is using iriscouch like me and does not know which port to specify here, can look it up in the config file. Using Futon this can be found in the entry "httpd > port" here:
> http://your_url_spec.iriscouch.com/_utils/config.html
More general and without Futon this can be found (and if you wish so edited) via command-line in the local.ini of your own CouchDB installation:
~$ cat etc/couchdb/local.ini

Download File from server that uses Icefaces form based authentication

I am a newbie to ICEfaces and i have a requirement where i need to download a document from a given url (http://ipaddress/formexec?objectid=201).
This URL uses a form based authentication that is deployed through ICEFaces.
i tracked the request of this URL and i get the following line:
&ice.submit.partial=false&ice.event.target=loginForm%3Aj_id33&ice.event.captured=loginForm%3Aj_id33
Is there any libraries or code to download the document by successfully passing the username and password.

You need to extract the jsessionid from the Set-Cookie response header and append it as URL attribute to the subsequent requests as http://example.com/path/page.jsf;jsessionid=XXX.
Here's a kickoff example with help of "plain vanilla" java.net.URLConnection:
// Prepare stuff.
String loginurl = "http://example.com/login";
String username = "itsme";
String password = "youneverguess";
URLConnection connection = null;
InputStream response = null;
// First get jsessionid (do as if you're just opening the login page).
connection = new URL(loginurl).openConnection();
response = connection.getInputStream(); // This will actually send the request.
String cookie = connection.getHeaderField("Set-Cookie");
String jsessionid = cookie.split(";")[0].split("=")[1]; // This assumes JSESSIONID is first field (normal case), you may need to change/finetune it.
String jsessionidurl = ";jsessionid=" + jsessionid;
response.close(); // We're only interested in response header. Ignore the response body.
// Now do login.
String authurl = loginurl + "/j_security_check" + jsessionidurl;
connection = new URL(authurl).openConnection();
connection.setDoOutput(true); // Triggers POST method.
PrintWriter writer = new PrintWriter(new OutputStreamWriter(connection.getOutputStream()));
writer.write("j_username=" + URLEncoder.encode(username, "UTF-8")
+ "&j_password=" + URLEncoder.encode(password, "UTF-8"));
writer.close();
response = connection.getInputStream(); // This will actually send the request.
response.close();
// Now you can do any requests in the restricted area using jsessionid. E.g.
String downloadurl = "http://example.com/download/file.ext" + jsessionidurl;
InputStream download = new URL(downloadurl).openStream();
// ...
To achieve the same with less bloated code, consider Apache Commons HttpComponents Client.

Form based auth is not much different from other requests. All you have to do is to submit an a request to the auth form providing required parameters, such as user and password and in some cases additional token that you would have to get from the source page. Then you need to get cookies from the auth response or session id parameter and copy them to your next request that will fetch the data.