We Keep Coding

How to extract parameters from a URL regardless the way it written by?

I want a java way to extract the parameters of a URL regardless the way these parameters are written in it, in the regular way like( https://www.facebook.com/Doly.mohamed.Smile9?ref=stream&hc_location=stream ) it's so easy because all i have to do is :
URL url = new URL("www.blabla....etc");
String query = url.getQuery();
try{
String [] params = query.split("&");
for(int i= 0 ; i < params.length; i++){
String [] split = params[i].split("=");
parameters.put(split[0], split[1]);
}
}catch(NullPointerException ex){}
so the parameters values would be :
key = ref value = stream , key = hc_location value = stream
but what shall i do if the URL has parameters written in another way or if the URL does't has it's parameters written in it like in the case of the doPost() way.
and is there is a way to get the extraPathInfo from a URL without using servlets?

You could do that easily with Apache's HTTP utils.
URIBuilder uriBuilder = new URIBuilder(uriString);
List<NameValuePair> urlParameters = uriBuilder.getQueryParams();
String uriWithoutParameters = uriBuilder.clearParameters().toString();
Now you could, for example, easily convert the GET request to a POST request, using other classes from the http utils API.

There is a difference between GET and POST urls
In GET url, parameters are part of URL and in POST they are part of Request-body.
So in POST, the URL may or may not contain the request params, and unless you don't have them in the URL its not possible to extract.
The POST request method is designed to request that a web server
accept the data enclosed in the request message's body for storage.1
It is often used when uploading a file or submitting a completed web
form.
So unless you have the POST request's body. Its difficult to extract the Parameter.

Typically you need HTTP request parameters on HTTP server side. Java HTTP server will parse the request and pass it as ServletRequest object to Servlet.service method. ServletRequest has methods to access the request parameters.

Using InputStreamEntity for buidling a http put request with httpclient won't work unless I pass the content length explicitely

I am trying to do a http request in scala using httpclient from org.apache.httpcomponents version 4.23. In particular I want to do a put using an InputStreamEntity to build the request in order to avoid copying over a large (~100Mb) byte array in memory. Here is the snippet:
val req = new HttpPut(url)
req setEntity new InputStreamEntity(contentStream, -1/*contentlength*/)
val client = new DefaultHttpClient(connManager, httpParams)
val resp = client execute req
In the code url, connManager, httpParams are defined elsewhere. The result of the code is the creation of a file on the desired location with NO content. I am testing with a contentStream which has 3 bytes. Creating the InputStreamEntity with content length as argument set explicitly to 3 will result in the code to create the file the right way. For good reasons in production I won't know the length of the stream hence I want to use negative numbers to make sure the entire stream is sent until, as advertized by the api of InputStreamEntity, the end of the stream is reached.
What am I doing wrong? Why am I getting an empty file when not explicitely setting the content length?

Not setting the content-length, will result in HTTP Client switching to chunked transfer-encoding
For this to work, the http server you are posting to must be HTTP 1.1 compliant. Is it ?

Put request parameters not getting set

This may be standard stuff but unable to get it wokring.
I'm using org.apache.commons.httpclient.methods for making Http request from my Java code. In one instance I've to make a PUT request and pass some parameters. I'm doing it the following way:
PutMethod putMethod = new PutMethod(url);
putMethod.getParams().setParameter("param1", "param1Value");
putMethod.getParams().setParameter("param2", "param2Value");
httpClient.executeMethod(putMethod);
But at the server, when it tries to read these parameters - it can only get null.
However, When I modify my url as url?param1=param1Value&param2=param2Value it works.
How do I get it working using setParameter method?

To add Query Params to PutMethod, follow this method.
NameValuePair[] putParameters = new NameValuePair[2];
putParameters[0] = new NameValuePair(Param1, value1);
putParameters[1] = new NameValuePair(Param2, value2);
HttpClient client = new HttpClient();
PutMethod putMethod = new PutMethod(url);
putMethod.setQueryString(putParameters);
Then Call,
int response = client.executeMethod(putMethod);
Instead of putMethod.setQueryString(putParameters); you could also use
putMethod.setRequestBody(EncodingUtil.formUrlEncode(putParameters, "UTF-8"));
(This is deprecated)
GetMethod, PostMethod have slight differences when adding Query Params compared to the above code.
For More Code Examples : http://www.massapi.com/class/pu/PutMethod.html
Hope this helps.

your server side code has to support the PUT method
for example if its a Servlet you can include the method
doPUT(); // your put request will be delivered to this method
if you use REST based frameworks such as jersey
you can use
#PUT
Response yourPutMethod(){..}

In Java, using HttpPost (apache commons HttpClient) when sending JSON POST - should I URL encode the body?

I'm sending a RESTful JSON POST request using Apache HttpClient (to a 3rd party API)
Should I URL encode the JSON body?
And if something in the content is already URL encoded (e.g. I send HTML that has some links with URL encoded chars, e.g. #22) should I expect to get the content as is on the other side without it being decoded?
E.g. if i'm doing something like this
String html = "<a href='http://example.com?charOfTheDay=%22'>click me</a>";
// Build the JSON object
JSONObject jsonObj = new JSONObject();
jsonObj.put("html", html);
jsonObj.put("otherKey",otherValue);
//...
// Create the POST object and add the parameters
HttpPost httpPost = new HttpPost(url);
StringEntity entity = new StringEntity(jsonObj.toString(), HTTP.UTF_8);
entity.setContentType("application/json");
httpPost.setEntity(entity);
HttpClient client = new DefaultHttpClient();
HttpResponse response = client.execute(httpPost);
Should I expect getting the same value on the receiving end, after getting the value of the "html" key?
e.g. on the receiving end
//after parsing the request string to a JSON object
String html = inputJsonObject.get("html")
// should return "<a href='http://example.com?charOfTheDay=%22'>click me</a>"
Are there any other steps I need to do to make sure what I send is what is received "as is"?

There's sort of two contexts you have to worry about here:
JSON
You have to make sure that the JSON you generate is valid JSON (duh). This means making sure that all the { } and [ ] syntax is in the right place and making sure that the field values you are inserting into the JSON object are safely escaped (like escaping that HTML snippet in your question --some of the quote characters would need to be escaped). BUT because you're using a standard JSON Java library, you don't have to worry about this...it will take care of all this for you.
HTTP
The JSON string then has to be inserted into the HTTP request body. No escaping needs to be done here--just insert the raw JSON string. HTTP, as a protocol, will accept anything inside the request/response bodies, including raw binary data.
Anyway that was kind of long, but I hope it helped.

The Content-Type in your http header should be application/json, so you oughtn't URL encode the body of the http request.
URL encoding is meant to prevent users from using characters that are special in representing URLs (such as '/').
You don't have to worry about links in the content being decoded either, unless you use a Content-Type in your http header that suggests that the server should decode the body, such as application/x-www-form-urlencoded

How can i programmatically upload a file to a website?

I have to upload a file to a server which only exposes a jsf web page with file upload button (over http). I have to automate a process (done as java stand alone process) which generates a file and uploads the file to the server.Sadly the server to where the file has to be uploaded does not provide a FTP or SFTP. Is there a way to do this?
Thanks,
Richie

When programmatically submitting a JSF-generated form, you need to make sure that you take the following 3 things in account:
Maintain the HTTP session (certainly if website has JSF server side state saving turned on).
Send the name-value pair of the javax.faces.ViewState hidden field.
Send the name-value pair of the button which is virtually to be pressed.
Otherwise the action will possibly not be invoked at all. For the remnant it's not different from "regular" forms. The flow is basically as follows:
Send a GET request on the page with the form.
Extract the JSESSIONID cookie.
Extract the value of the javax.faces.ViewState hidden field from the response. If necessary (for sure if it has a dynamically generated name and thus possibly changes every request), extract the name of input file field and the submit buttonas well. Dynamically generated IDs/names are recognizeable by the j_id prefix.
Prepare a multipart/form-data POST request.
Set the JSESSIONID cookie (if not null) on that request.
Set the name-value pair of javax.faces.ViewState hidden field and the button.
Set the file to be uploaded.
You can use any HTTP client library to perform the task. The standard Java SE API offers java.net.URLConnection for this, which is pretty low level. To end up with less verbose code, you could use Apache HttpClient to do the HTTP requests and manage the cookies and Jsoup to extract data from the HTML.
Here's a kickoff example, assuming that the page has only one <form> (otherwise you need to include an unique identifier of that form in Jsoup's CSS selectors):
String url = "http://localhost:8088/playground/test.xhtml";
String viewStateName = "javax.faces.ViewState";
String submitButtonValue = "Upload"; // Value of upload submit button.
HttpClient httpClient = new DefaultHttpClient();
HttpContext httpContext = new BasicHttpContext();
httpContext.setAttribute(ClientContext.COOKIE_STORE, new BasicCookieStore());
HttpGet httpGet = new HttpGet(url);
HttpResponse getResponse = httpClient.execute(httpGet, httpContext);
Document document = Jsoup.parse(EntityUtils.toString(getResponse.getEntity()));
String viewStateValue = document.select("input[type=hidden][name=" + viewStateName + "]").val();
String uploadFieldName = document.select("input[type=file]").attr("name");
String submitButtonName = document.select("input[type=submit][value=" + submitButtonValue + "]").attr("name");
File file = new File("/path/to/file/you/want/to/upload.ext");
InputStream fileContent = new FileInputStream(file);
String fileContentType = "application/octet-stream"; // Or whatever specific.
String fileName = file.getName();
HttpPost httpPost = new HttpPost(url);
MultipartEntity entity = new MultipartEntity();
entity.addPart(uploadFieldName, new InputStreamBody(fileContent, fileContentType, fileName));
entity.addPart(viewStateName, new StringBody(viewStateValue));
entity.addPart(submitButtonName, new StringBody(submitButtonValue));
httpPost.setEntity(entity);
HttpResponse postResponse = httpClient.execute(httpPost, httpContext);
// ...

Try using HttpClient, here's an article that I think describes what you want, towards the bottom there's a section titled "Using HttpClient-Based FileUpload".
Hope this helps.

Probably that webpage just sends a POST request to the server with the contents of the form. You can easily send such a POST request yourself from Java, without using that page. For example this article shows an example of sending POST requests from Java

What you'll need to do is to examine the HTML on the page and work out what parameters are needed to post the form. It'll probably look something like this:
<form action="/RequestURL">
<input type=file name=file1>
<input type=textbox name=value1>
</form>
Based on that you can write some code to do a POST request to the url:
String data = URLEncoder.encode("value1", "UTF-8") + "=" + URLEncoder.encode("value1", "UTF-8");
data += "&" + URLEncoder.encode("file1", "UTF-8") + "=" + URLEncoder.encode(FileData, "UTF-8");
// Send data
URL url = new URL("http://servername.com/RequestURL");
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
wr.close();
Remember that the person who wrote the page might do some checks to make sure the POST request came from the same site. In that case you might be in trouble, and you might need to set the user agent correctly.

You could try to use HtmlUnit for this. It provides a very simply API for simulating browser actions. I already used this approach for similar requirements. It's very easy. You should give it a try.