I'm trying to use the Apache HttpComponents Client to simulate, in Java, a login process on a website. It's failing at the moment and I'm not sure why. I'm using Firebug in Firefox to try and understand the equivalent browser events and I'm sure it's logging in by a particular POST method on a form.
Here's my code. At the end I expect a whole bunch of cookies to be set, but they are not, and the responses I'm getting to subsequent GET requests prove that the login failed.
String email = ...
String password = ...
String url = ...
DefaultHttpClient client = new DefaultHttpClient();
List<NameValuePair> loginParams = new ArrayList<NameValuePair>(2);
loginParams.add(new BasicNameValuePair("logindetails[email]", email));
loginParams.add(new BasicNameValuePair("logindetails[password]", password));
UrlEncodedFormEntity loginEntity = new UrlEncodedFormEntity(loginParams);
HttpPost loginPost = new HttpPost(url);
loginPost.setEntity(loginEntity);
HttpResponse loginResponse = client.execute(loginPost);
HttpEntity loginRepsonseEntity = loginResponse.getEntity();
loginRepsonseEntity.consumeContent();
for (Cookie cookie: client.getCookieStore().getCookies()) {
System.out.println(cookie.getName() + " -> " + cookie.getValue());
}
Where should I start looking for the problem? I'd like to verify that the POST is the same as the one my browser is making -- what's the best way to do that?
We use the Apache HttpClient stuff every day where I work, it's wonderful. What you're doing, where you are encoding the data yourself, looks a little weird to me. There is an easier way. I'll assume you're doing a POST.
String email = ...
String password = ...
String url = ...
PostMethod post = new PostMethod(url);
NameValuePair[] data = {
new NameValuePair("logindetails[email]", email),
new NameValuePair("logindetails[password]", password)
};
post.setRequestBody(data);
// execute method and handle any error responses.
HttpClient httpclient = new HttpClient();
int statusCode = httpclient.executeMethod(post);
// continue as neccessary
I took that code (and modified it to fit yours better) from the Apache HttpClient site. Unless you have a good reason, you shouldn't need to bother packaging up the fields yourself with the UrlEncodedFormEntity.
As for debugging this kind of stuff, it can be easiest to watch using WireShark to watch the packets go back and forth. Any of these will let you watch the requests go back and forth (as long as you don't use HTTPS) so you can easily tell what's being sent incorrectly by comparing it to what a browser sends. It's not the user friendliest program in the world, but it's invaluable when trying to figure out this kind of stuff.
You can use Firebug in Firefox to look at the Net-tab to investigate how the post is done when you use a web page. Looking at your code, my guess is that your parameter names are strange. logindetails[email] and logindetails[password] look weird.
This would be the equivalent of having:
<form action="...url" method="post">
email: <input type="text" name="logindetails[email]"/><br/>
password: <input type="password" name="logindetails[password]"/><br/>
<input type="submit" value="submit"/>
</form>
Make sure that's what you intended...
Look at the wonderful cURL utility -> http://curl.haxx.se/docs/manpage.html.
It's ubiquitous in Unix/Linux and is available under Cygwin on Windows.
It lets you achieve what you are trying to do in Java in a few keystrokes from command line.
Related
In our webapp, we have to send a POST request via HttpClient to an endpoint on our network, which will receive this and do some work with it. We are having trouble with character encoding, and I am having difficulties finding an answer to my question.
We have used the postMethod.getParams().setContentCharset("UTF-8") method when sending the request, but on the receiving end, it seems like the characters are still encoded in ISO 8859-1. I have determined this because when I inspect the String on the receiving side, it has garbage characters in it that go away once I follow the steps found at https://stackoverflow.com/a/16549329/1130549. Is there any extra steps I need to take on the sending end to ensure that I am actually writing characters in UTF-8 as expected? All we are doing now is using postMethod.addParameter(paramKey, paramValue) with native String objects.
Edit: Here is a very simple example of how we're sending the POST request. For what it's worth, the values are being taken from an XMLBeans object.
PostMethod postMethod = new PostMethod(url);
postMethod.getParams().setContentCharset("UTF-8");
postMethod.addParameter("key1", "value1");
postMethod.addParameter("key2", "value2");
HttpClient httpClient = new HttpClient();
int status = httpClient.executeMethod(postMethod);
EDIT
Simpler solution is to encode the value
postMethod.addParameter("key1", URLEncoder.encode("value1","UTF-8"));
To encode properly UTF-8, you can execute differently, using StringEntity and NameValuePair, e.g.:
try (CloseableHttpClient httpClient = HttpClients.custom().build()) {
URIBuilder uriBuilder = new URIBuilder(url);
HttpHost target = new HttpHost(uriBuilder.getHost(), uriBuilder.getPort(), uriBuilder.getScheme());
List<NameValuePair> nameValuePairs = new ArrayList<>();
nameValuePairs.add(new BasicNameValuePair("key1", "value1"));
nameValuePairs.add(new BasicNameValuePair("key2", "value2"));
String entityValue = URLEncodedUtils.format(nameValuePairs, StandardCharsets.UTF_8.name());
StringEntity entity = new StringEntity(entityValue, StandardCharsets.UTF_8.name());
post.setEntity(entity);
httpClient.execute(target, post);
First of all, you do need to make sure that the string that you are actually writing is encoded in UTF-8. I realized that you already know that but still double-check that it is so, as it would be the prime suspect of your problem. Also, I would recommend trying a much simpler HTTP client. Apache HTTP client (I believe that's the library that you are using) is an excellent library. But due to covering a very wide range of options it tends to be a bit bulky. So, or simple requests I would suggest a lightweight HTTP client that maybe not that comprehensive as Apache library but offers simplicity as a trade-off. Here how your code may look like:
private static void testHttpClient() {
HttpClient client = new HttpClient();
// client.setContentType("text/html; charset=utf-8");
client.setContentType("application/json; charset=utf-8");
client.setConnectionUrl("http://www.my-url.com");
String content = null;
try {
String myMessage = getMyMessage() // get the string that you want to send
content = client.sendHttpRequest(HttpMethod.POST, myMessage);
} catch (IOException e) {
content = client.getLastResponseMessage() + TextUtils.getStacktrace(e, false);
}
System.out.println(content);
}
It looks much more simple, I think. Also in the same library, there is another utility that allows you to convert any string in any language into a sequence of unicodes and vice-versa. This helped me numerous times to diagnose encoding thorny issues. For instance, if you see some gibberish symbols that could be a wrong display of a valid character or actual character loss. Here is an example of how it works:
result = "Hello World";
result = StringUnicodeEncoderDecoder.encodeStringToUnicodeSequence(result);
System.out.println(result);
result = StringUnicodeEncoderDecoder.decodeUnicodeSequenceToString(result);
System.out.println(result);
The output of this code is:
\u0048\u0065\u006c\u006c\u006f\u0020\u0057\u006f\u0072\u006c\u0064
Hello World
That might help you to check if the string you passed is valid or not. The library is called MgntUtils and could be found at Maven Central or at Github It comes as maven artifact and with sources and Javadoc. Javadoc could be found separately here
Disclaimer: The MgntUtils library is written by me
I am trying to get some data from a webpage that needs login in a page. I could do with htmlunit but gave me problems import these libraries on Android. So I am trying to do it with apache http client.
HttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost("https://inside.cineca.it/cgi-bin/uinside/marcature.pl");
BasicNameValuePair usernameBasicNameValuePair = new BasicNameValuePair("j_username", "user");
BasicNameValuePair passwordBasicNameValuePAir = new BasicNameValuePair("j_password", "pass");
List<NameValuePair> nameValuePairList = new ArrayList<NameValuePair>();
nameValuePairList.add(usernameBasicNameValuePair);
nameValuePairList.add(passwordBasicNameValuePAir);
UrlEncodedFormEntity urlEncodedFormEntity = new UrlEncodedFormEntity(nameValuePairList);
HttpResponse httpResponse = httpClient.execute(httpPost);
System.out.println(EntityUtils.toString(httpResponse.getEntity()));
But is giving me error. I am affraid i should manage the redirect but i don't know how. I would be happy if someone could give me some advice.
Best regards
You're performing an http POST to a login page. Maybe the server that serves you back the login page doesn't know to handle the post. Anyway, I am sure your intention is to perform the login actually. You need to figure out what is the server API where you should post the credentials and process the result. Performing a quick investigation with Chrome Network profiler, the login API for above link actually resides at: https://idp-is.cineca.it/idp/Authn/Multilogin. That is the URL that you should be using and there is where you need to POST.
Also, make sure the request you're making is done on a non-UI thread as this is a common mistake done by beginners.
I'm posting data to a website form using Apache's HttpClient class. The form is retrieved using the following lines of code:
HttpGet get = new HttpGet(url);
HttpResponse response = client.execute(get);
The website that I'm retrieving the form from requires authentication to access the form. If the request isn't authenticated, the website redirects the request to a login form page that will subsequently redirect back to the original page on successful authentication.
I want to cleanly detect whether or not the GET request returns the login page or the desired form page so that I can either POST login data or form data. The only way I can think of to do this is by reading from the content InputStream of the entity of the response and parsing each line. But that seems somewhat convoluted. I haven't worked with the Apache HttpComponents api before so I'm not sure if this would be the only and best way to accomplish what I want to accomplish.
EDIT: To clarify question, I'm asking if there is a set way to handle forms with Apache's HttpClient. I somewhat know how to achieve what I'm looking to do, but it looks very ugly and I'm hoping there is an easier and faster way to achieve it. For example, if there was some way to do the following:
HttpGet get = new HttpGet(url);
HttpResponse response = client.execute(get);
if(parseElements(response.getEntity()).hasFormWithId("login")) {
// post authentication data
} else {
// post actual form data
}
Because of my inexperience with Apache's HttpClient api, I'm not sure if what I'm looking for in the API is too abstract for the intent of the API.
You can modify the behavior of the HttpClient by setting the HttpClient Parameters
DefaultHttpClient client = new DefaultHttpClient();
client.setDefaultHttpParams(client.getParams().setBoolean(ClientPNames.HANDLE_REDIRECTS, false));
Which disables handling redirects automatically.
See also:
Automatic redirect handling
HTTP Authentication
DefaultHttpClient API
I am trying to find a solution to this the whole evening now...
I write an app which requests data from a web server. The Server answers in JSON format.
Everything works well except when I enter a umlaut like ä into my App.
In the following I assume the request URL is http://example.com/?q= and I am searching for "Jäger"
The correct call would then be h++p://example.com/?q=J%C3%A4ger
(Sorry for plus-signs but the spam protection doesnt let me post it correctly.)
So my problem is now:
When I give my URL String encoded or unencoded over to HttpGet it will always result in a doublee-encoded URL.
The Request to my Server is then http://example.com/?q=J%25C3%25A4ger (It encodes the percent signs)
which leads to the server searching in database for J%C3%A4ger what is obviously wrong.
So my question is how can I achive that if the user enters "Jäger" my app calls the correctly encoded URL?
Thanks for any help!
Here is the currently used code... Ist probably the worst possible idea I had...
URI url = new URI("http", "//example.com/?q=" + ((EditText)findViewById(R.id.input)).getText().toString(), null);
Log.v("MyLogTag", "API Request: " + url);
HttpGet httpGetRequest = new HttpGet(url);
// Execute the request in the client
HttpResponse httpResponse;
httpResponse = defaultClient.execute(httpGetRequest);
Update: Sorry, HttpParams isn't meant for request parameters but for configuring HttpClient.
On Android, you might want to use Uri.Builder, like suggested in this other SO answer:
Uri uri = new Uri.Builder()
.scheme("http")
.authority("example.com")
.path("someservlet")
.appendQueryParameter("param1", foo)
.appendQueryParameter("param2", bar)
.build();
HttpGet request = new HttpGet(uri.toString());
// This looks very tempting but does NOT set request parameters
// but just HttpClient configuration parameters:
// HttpParams params = new BasicHttpParams();
// params.setParameter("q", query);
// request.setParams(params);
HttpResponse response = defaultClient.execute(request);
String json = EntityUtils.toString(response.getEntity());
Outside of Android, your best bet is building the query string manually (with all the encoding hassles) or finding something similar to Android's Uri.Builder.
I have to upload a file to a server which only exposes a jsf web page with file upload button (over http). I have to automate a process (done as java stand alone process) which generates a file and uploads the file to the server.Sadly the server to where the file has to be uploaded does not provide a FTP or SFTP. Is there a way to do this?
Thanks,
Richie
When programmatically submitting a JSF-generated form, you need to make sure that you take the following 3 things in account:
Maintain the HTTP session (certainly if website has JSF server side state saving turned on).
Send the name-value pair of the javax.faces.ViewState hidden field.
Send the name-value pair of the button which is virtually to be pressed.
Otherwise the action will possibly not be invoked at all. For the remnant it's not different from "regular" forms. The flow is basically as follows:
Send a GET request on the page with the form.
Extract the JSESSIONID cookie.
Extract the value of the javax.faces.ViewState hidden field from the response. If necessary (for sure if it has a dynamically generated name and thus possibly changes every request), extract the name of input file field and the submit buttonas well. Dynamically generated IDs/names are recognizeable by the j_id prefix.
Prepare a multipart/form-data POST request.
Set the JSESSIONID cookie (if not null) on that request.
Set the name-value pair of javax.faces.ViewState hidden field and the button.
Set the file to be uploaded.
You can use any HTTP client library to perform the task. The standard Java SE API offers java.net.URLConnection for this, which is pretty low level. To end up with less verbose code, you could use Apache HttpClient to do the HTTP requests and manage the cookies and Jsoup to extract data from the HTML.
Here's a kickoff example, assuming that the page has only one <form> (otherwise you need to include an unique identifier of that form in Jsoup's CSS selectors):
String url = "http://localhost:8088/playground/test.xhtml";
String viewStateName = "javax.faces.ViewState";
String submitButtonValue = "Upload"; // Value of upload submit button.
HttpClient httpClient = new DefaultHttpClient();
HttpContext httpContext = new BasicHttpContext();
httpContext.setAttribute(ClientContext.COOKIE_STORE, new BasicCookieStore());
HttpGet httpGet = new HttpGet(url);
HttpResponse getResponse = httpClient.execute(httpGet, httpContext);
Document document = Jsoup.parse(EntityUtils.toString(getResponse.getEntity()));
String viewStateValue = document.select("input[type=hidden][name=" + viewStateName + "]").val();
String uploadFieldName = document.select("input[type=file]").attr("name");
String submitButtonName = document.select("input[type=submit][value=" + submitButtonValue + "]").attr("name");
File file = new File("/path/to/file/you/want/to/upload.ext");
InputStream fileContent = new FileInputStream(file);
String fileContentType = "application/octet-stream"; // Or whatever specific.
String fileName = file.getName();
HttpPost httpPost = new HttpPost(url);
MultipartEntity entity = new MultipartEntity();
entity.addPart(uploadFieldName, new InputStreamBody(fileContent, fileContentType, fileName));
entity.addPart(viewStateName, new StringBody(viewStateValue));
entity.addPart(submitButtonName, new StringBody(submitButtonValue));
httpPost.setEntity(entity);
HttpResponse postResponse = httpClient.execute(httpPost, httpContext);
// ...
Try using HttpClient, here's an article that I think describes what you want, towards the bottom there's a section titled "Using HttpClient-Based FileUpload".
Hope this helps.
Probably that webpage just sends a POST request to the server with the contents of the form. You can easily send such a POST request yourself from Java, without using that page. For example this article shows an example of sending POST requests from Java
What you'll need to do is to examine the HTML on the page and work out what parameters are needed to post the form. It'll probably look something like this:
<form action="/RequestURL">
<input type=file name=file1>
<input type=textbox name=value1>
</form>
Based on that you can write some code to do a POST request to the url:
String data = URLEncoder.encode("value1", "UTF-8") + "=" + URLEncoder.encode("value1", "UTF-8");
data += "&" + URLEncoder.encode("file1", "UTF-8") + "=" + URLEncoder.encode(FileData, "UTF-8");
// Send data
URL url = new URL("http://servername.com/RequestURL");
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
wr.close();
Remember that the person who wrote the page might do some checks to make sure the POST request came from the same site. In that case you might be in trouble, and you might need to set the user agent correctly.
You could try to use HtmlUnit for this. It provides a very simply API for simulating browser actions. I already used this approach for similar requirements. It's very easy. You should give it a try.