We Keep Coding

HTTPGet unicode characters appearing in response String

I have a utility used for integrating data and have run into an issue when special characters are used such as "Ã". Below is the method in question where the issue comes in. The response is from an API and is in xml format.
protected String getStringHttpContent(URI url, Map<String,String> headerParameters) throws IOException
{
HttpGet request = new HttpGet(url);
for(String parameter : headerParameters.keySet())
request.setHeader(parameter, headerParameters.get(parameter));
CloseableHttpResponse response = getClient().execute(request);
dumpHeaders(response);
BufferedReader br = new BufferedReader(new InputStreamReader(response.getEntity().getContent(), "UTF-8"));
StringBuffer sb = new StringBuffer();
String output;
while ((output = br.readLine()) != null) {
sb.append(output);
}
response.close();
return sb.toString();
}
The result of njÃmientill in the response string is njämientill. I've tried changing the encoding, but result remains the same. Any advice would be appreciated.

Make sure that you are using UTF-8 encoding end-to-end (through the whole chain). This includes you web pages and user input if it comes from a html form (for example), setting UTF-8 on pages, web services (web.xml, sun-web.xml or so). Also Inbound HttpRequest should include the header attribute "charset", eg. "Content-Type: text/html; charset=utf-8 ". The way your configure server-side and client-side depends on the technologies you use (which I don't know).
EDIT: regarding your comment, even if you are the client you should set the content-type to define which type of content you expect from the server (as this one may be able to serve different contents at the same URL).
Please try configure your HttpGet with:
request.setHeader(HttpHeaders.CONTENT_TYPE, "application/xml; charset=utf-8");
or (if the server is quite old):
request.setHeader(HttpHeaders.CONTENT_TYPE, "text/xml; charset=utf-8");
Better, maybe specify the accept header together with the accepted charset:
request.setHeader("Accept-Charset", "utf-8");
request.setHeader("Accept", "application/xml");
If none of these works I suggest you show your Postman query here or do a Wireshark capture to see the actual request and response, plus also list the content of the headerParameters map. Otherwise we cannot help you more (as the rest of your code looks good, to my opinion).

Taking text from a response web page using Java

I am sending commands to a server using http, and I currently need to parse a response that the server sends back (I am sending the command via the command line, and the servers response appears in my browser).
There are a lot of resources such as this: Saving a web page to a file in Java, that clearly illustrate how to scrape a page such as cnn.com. However, since this is a response page that is only generated when the camera receives a specific command, my attempts to use the method described by Mike Deck (in the link above) have met with failure. (Specifically, when my program requests the page again the server returns a 401 error.)
The response from the server opens a new tab in my browser. Essentially, I need to know how to save the current web page using java, since reading in a file is probably the most simple way to approach this. Do any of you know how to do this?
TL;DR How do you save the current webpage to a webpage.html or webpage.txt file using java?
EDIT: I used Base64 from the Apache commons codec, which solved my 401 authentication issue. However, I am still getting a 400 error when I attempt to connect my InputStream (see below). Does this mean a connection isn't being established in the first place?
URL url = new URL ("http://"+ipAddress+"/axis-cgi/record/record.cgi?diskid=SD_DISK");
byte[] encodedBytes = Base64.encodeBase64("root:pass".getBytes());
String encoding = new String (encodedBytes);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setDoInput (true);
connection.setRequestProperty ("Authorization", "Basic " + encoding);
connection.connect();
InputStream content = (InputStream)connection.getInputStream();
BufferedReader in = new BufferedReader (new InputStreamReader (content));
String line;
while ((line = in.readLine()) != null) {
System.out.println(line);
}
EDIT 2: Changing the request to a GET resolved the issue.

So while scrutinizing my code above, I decided to change
connection.setRequestMethod("POST");
to
connection.setRequestMethod("GET");
This solved my problem. In hindsight, I think the server was not recognizing the HTTP because it is not set up to handle the various trappings that come along with post.

How to call a URL with params and get back response in servlet?

I have a situation where a intermediate servlet needs to be introduced which will handle requests from existing project and redirect the manipulated response to either existing project or the new one. This servlet will act as an interface to login into the new project from some other application.
So currently I use the following code to get back response in jsp as an xml.
var jqxhr =$.post("http://abhishek:15070/abc/login.action",
{ emailaddress: "ars#gmail.com",
projectid: "123" },
function(xml)
{
if($(xml).find('isSuccess').text()=="true")
{
sessiontoken=$(xml).find('sessiontoken').text();
setCookie("abcsessionid", sessiontoken , 1);
setCookie("abcusername",e_add,1);
}
}
)
.error(function() {
if(jqxhr.responseText == 'INVALID_SESSION') {
alert("Your Session has been timed out");
window.location.replace("http://abhishek:15070/abc/index.html");
}else {
alert( jqxhr.responseText);
}
});
xml content
<Response>
<sessiontoken>334465683124</sessiontoken>
<isSuccess>true</isSuccess>
</Response>
but now I want the same thing to be done using servlet, is it possible?
String emailid=(String) request.getParameter("emailaddress");
String projectid=(String) request.getParameter("projectid");
Update
I just came up with something.
Is it possible to return back a html page with form (from servlet), whose on body load it will submit a form and on submission of this form it will receive the response xml which will get processed.

Use java.net.URLConnection or Apache HttpComponents Client. Then, parse the returned HTTP response with a XML tool like as JAXB or something.
Kickoff example:
String emailaddress = request.getParameter("emailaddress");
String projectid = request.getParameter("projectid");
String charset = "UTF-8";
String query = String.format("emailaddress=%s&projectid=%s",
URLEncoder.encode(emailaddress, charset),
URLEncoder.encode(projectid, charset));
URLConnection connection = new URL("http://abhishek:15070/abc/login.action").openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Accept-Charset", charset);
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=" + charset);
try {
connection.getOutputStream().write(query.getBytes(charset));
}
finally {
connection.getOutputStream().close();
}
InputStream response = connection.getInputStream();
// ...
See also:
Using java.net.URLConnection to fire and handle HTTP requests
HttpClient tutorial and examples

Actually, what you probably want is not an intermediate servlet at all. What you probably want is called a servlet filter and writing one is not particularly hard. I've written one in the past and I just started on a new one yesterday.
An article like this one or this one lays out pretty simply how you can use a servlet filter to intercept calls to specific URLs and then redirect or reject from there. If the incoming URL matches the pattern for the filter, it will get a shot at the request and response and it can then make a choice whether or not to pass it on to the next filter in line.
I don't know if all third party security solutions do it like this, but at least CAS seemed to be implemented that way.

Better way to send lots of text to a server?

I have a java application that sends text to a sql database on a server. Currently my java application takes the text, puts it into the url, then sends it to a php page on the server that takes it with GET and puts it in the database. that works fine to an extent, the problem is, that i need to be able to send lots of text, and i keep getting 414, uri to long errors. is there a better way to do this?
ok, i tried what you said, and read the tutorial, but something is not working. here is my code that i tried
public void submitText(String urls,String data) throws IOException{
URL url = new URL(urls);
URLConnection con = url.openConnection();
con.setRequestProperty("Content-Type", "text/plain; charset=utf-8");
con.setDoOutput(true);
OutputStreamWriter out = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
out.write(data);
out.flush();
}
submitText(server + "publicPB.php", "param=" + text);
here is my php code
$param = $_POST['param'];
$sql = "UPDATE table SET cell='{$param}' WHERE 1";
mysql_query($sql);
...
im pretty sure its not a problem with the php as the php worked fine with GET, and thats all i change with it, my problem i think is that im not 100% sure how to send data to it with the java

Use a POST instead of a GET and send the text as the request body. You can only pass so much data to a URL. E.g.:
// Assuming 'input' is a String and contains your text
URL url = new URL("http://hostname/path");
URLConnection con = url.openConnection();
con.setRequestProperty("Content-Type", "text/plain; charset=utf-8");
con.setDoOutput(true);
OutputStreamWriter out = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
out.write(input);
out.close();
See Reading from and Writing to a URLConnection for more details.

Why don't you use POST to send data across to PHP page? GET does have a smaller limit of content.

Use POST requests, which do not have content length limits.

POST requests do not have length content limits and are much secure than GET requests ;)

If using SQL Server I would look into leveraging BCP. You can write the file and call BCP from within Java, and it will send the information directly to your database.

Writing post data from one java servlet to another

I am trying to write a servlet that will send a XML file (xml formatted string) to another servlet via a POST.
(Non essential xml generating code replaced with "Hello there")
StringBuilder sb= new StringBuilder();
sb.append("Hello there");
URL url = new URL("theservlet's URL");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Length", "" + sb.length());
OutputStreamWriter outputWriter = new OutputStreamWriter(connection.getOutputStream());
outputWriter.write(sb.toString());
outputWriter.flush();
outputWriter.close();
This is causing a server error, and the second servlet is never invoked.

This kind of thing is much easier using a library like HttpClient. There's even a post XML code example:
PostMethod post = new PostMethod(url);
RequestEntity entity = new FileRequestEntity(inputFile, "text/xml; charset=ISO-8859-1");
post.setRequestEntity(entity);
HttpClient httpclient = new HttpClient();
int result = httpclient.executeMethod(post);

I recommend using Apache HTTPClient instead, because it's a nicer API.
But to solve this current problem: try calling connection.setDoOutput(true); after you open the connection.
StringBuilder sb= new StringBuilder();
sb.append("Hello there");
URL url = new URL("theservlet's URL");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Length", "" + sb.length());
OutputStreamWriter outputWriter = new OutputStreamWriter(connection.getOutputStream());
outputWriter.write(sb.toString());
outputWriter.flush();
outputWriter.close();

The contents of an HTTP post upload stream and the mechanics of it don't seem to be what you are expecting them to be. You cannot just write a file as the post content, because POST has very specific RFC standards on how the data included in a POST request is supposed to be sent. It is not just the formatted of the content itself, but it is also the mechanic of how it is "written" to the outputstream. Alot of the time POST is now written in chunks. If you look at the source code of Apache's HTTPClient you will see how it writes the chunks.
There are quirks with the content length as result, because the content length is increased by a small number identifying the chunk and a random small sequence of characters that delimits each chunk as it is written over the stream. Look at some of the other methods described in newer Java versions of the HTTPURLConnection.
http://java.sun.com/javase/6/docs/api/java/net/HttpURLConnection.html#setChunkedStreamingMode(int)
If you don't know what you are doing and don't want to learn it, dealing with adding a dependency like Apache HTTPClient really does end up being much easier because it abstracts all the complexity and just works.

Don't forget to use:
connection.setDoOutput( true)
if you intend on sending output.