We Keep Coding

Sending special characters via URL Java

Say I have a string with "é" in it, and I want to send it over a URL to the next controller, the characters gets encoded to %C3%A9, and when it's received in the other controller, it gets decoded to "é".
My question is, how to encode the "é" over the URL so when it's received in the other controller it gets decoded to "é"? For now, I'm replacing them manually. I need a way to do it automatically and with any special character (éèà...)
Thank you.

Unfortunately there is no way to declare the encoding of URL data. The common encoding used to be ISO-8859-1 or Latin1, but nowadays, UTF-8 is often used in new developments. The reasons was that the servlet specification says that when the charset is not specified, ISO-8859-1 is implied, but HTML 4.0 recommends UTF-8 for URLs.
The problem is that the URL is composed of bytes and that the servlet container convert it to java characters before passing it to the application, so you must declare the used charset at the servlet container level. For compatibily reasons, the well known Tomcat before version 8.0 assumed by default a Latin1 charset for the URL. Since 8.0.0 the default now depends on the "strict servlet compliance" setting. It is ISO-8859-1 when true and UTF-8 when false
References:
What character set should I assume the encoded characters in a URL to be in?
Tomcat FAQ/CharacterEncoding
For your precise question, you have two ways to correctly process the é character:
leave your servlet container configuration unchanged and encode the URL in ISO-8859-1 (é will be %E9)
stick to UTF-8 in the URL but declare it to the servlet container

JSP doesn't display texts from DB in a language other than English

I have the following code in my JSP
... <%
out.println(request.getAttribute("textFromDB")); %> ...
When the JSP is called it just prints question marks (????..) instead of the actual text stored in a MySQL database which is not in English. What can I do to make it display the text correctly. I tried to change the charset and pageEncoding to UTF-8 but it didn't help.

Isn't encoding nice? Unfortunately it's hard to tell where it gets wrong: Your database might store in another character set than UTF-8. Or your database driver might be configured to work in another encoding. Or your server defaults to another one. Or your HTTP connection assumes different encoding and your charset change comes too late.
You'll have to go through all those layers - and keep in mind that everything might look fine and it's been the long-past write-operations to your database that already messed up the data beyond repair.

System.out.println(this.request.getHeader("Content-Encoding")); //check the content type
String data = new String(this.request.getParameter("data").getBytes("ISO-8859-1"), "UTF-8"); //this way it is encoded to byte and then to a string
if the above method didnt work you might wanna check with the database
if it encode characters to "UTF-8"
or
you can configure URIEncoding="UTF-8" in your tomcat setup and just (request.getAttribute("textFromDB")); do the rest.

Charset trouble (ø as 00F8)

I am getting a string from our database (third party tool) - and I have a trouble with one
name - sometimes it is right "Tarsøy", and all runs smoothly but sometimes it is "Tars00F8y".
And this ruins the process - I have tried to write some validator function via URLDecoder.decode(name, "UTF-8") that gets a string and return validated one but not succeed.
this is how I get a sting from our base:
Database.WIKI.get(index); // the index is the ID of the string
// this is no sql DB
now about "sometimes" - it means that this code just works different =) I think that is connected with inner DB exceptions or so. So I am trying to do something like validate(Database.WIKI.get(index))
May be I should try something like Encode String to UTF-8

In Java, JavaScript and (especially interesting) JSON there exists the notation \u00F8 for ø. I think this was sent to the database, maybe from a specific browser on a specific computer locale. \u disappeared and voilà. Maybe it is still as invisible control character in the string. That would be nice for repairs.
My guess is JSON data; however normally JSON libraries should parse u-escaped characters. That is weird.
Check what happens when storing "x\\u00FDx". Is the char length 6 or maybe 7 (lucky).
Some sanity checks: assuming you work in UTF-8, especially if the data arrive by HTML or JS:
Content-Type header text/html; charset=UTF-8
(Optional) meta tag with charset=UTF-8
<form action="..." accept-charset="UTF-8">
JSON: contentType: "application/json; charset=UTF-8"

Broken UTF-8 URI Encoding in JSPs

I got a strange issue with wrong URI Encoding and would appreciate any help!
The project uses JSPs, Servlets, Jquery, Tomcat 6.
Charset in the JSPs is set to UTF-8, all Tomcat connectors use URIEncoding=UTF-8 and I also use a character encoding filter as described here.
Also, I set the contentType in the meta Tag and my browser detects it correctly.
In Ajax calls with Jquery I use encodeURIComponent() on the terms I want to use as URL Parameters and then serialize the whole parameter set with $.param(). In the called servlet these parameters are decoded correctly with Java.net.URLDecoder.decode(term, "UTF-8").
In some places I generate URLs for href elements from a parameter map in the JSPs. Each parameter value is encoded with Java.net.URLEncoder.encode(value, "UTF-8") on JSP side but then decoding it the same way as before results in broken special characters. Instead, I have to encode it as "ISO-8859-2" in the JSP which is then decoded correctly as "UTF-8" in the servlet.
An example for clarifying:
The term "überfall" is URIEncoded via Javascript (%C3%BCberfall) and sent to the servlet for decoding and processing, which works. After passing it back to a JSP I would encode it as UTF-8 and build the URL which results for instance in:
Click here
However, clicking this link will send the parameter as "%C3%83%C2%BCberfall" to the servlet which decodes to "überfall". The same occurs when no encoding takes place.
When, using "ISO-8859-2" for encoding I get:
Click here
When clicking this link I can observe in Wireshark that %C3%BCberfall is sent as parameter which decodes again to "überfall"!
Can anyone tell me where I miss something?
EDIT:
While observing the Network Tab in Firebug I realized that by using
$.param({term : encodeURIComponent(term)});
the term is UTF-8 encoded twice, resulting in "%25C3%25BCberfall", i.e. the percent symbols are also percent-encoded. Analogously, it works for me if I call encode(term, "UTF-8") twice on each value from the parameter map.
Encoding once and not decoding the String results in "überfall" again.

What encoding is Java using internally? Did you start your application with
-Dfile.encoding=utf-8
Please clarify where the "parameter map in the JSPs" is defined. Does it come from some persistent datastorage or are the strings given in your code as literals?
Some thoughts on what is going on, which might help:
ü is what comes out when a UTF-8 encoded ü is read expecting ISO-8859-1, when each byte is decoded on its own. %C3%BC is the URI-encoded representationg of both UTF-8 bytes of a UTF-8 ü. I think this is what's happening:
%C3%BC gets wrongly decoded to → ü which gets encoded to → %C3%83%C2%BC which then gets decoded again to → ü so you end up with überfall.
So I guess, you use the wrong encoding for decoding a URI-encoded string. This might have something to do with the internal encoding used by Java/the JVM:
By default, the JRE 7 installer installs a European languages version if it recognizes that the host operating system only supports European languages.

I think I fixed the problem now definitely.
Following Jontro's comment I encoded all URL parameter values once and removed the manual servlet-side decoding.
Sending an ü should look like %C3%BC in Firebug's Network tab which gave me ü in the servlet.
Java was definitely set to "UTF-8" internal encoding with the -Dfile.encoding parameter.
I traced the problem to the request.getParameter() method like this. request.getQueryString was ok, but when extracting the actual parameters it fails:
request.getCharacterEncoding()) => UTF-8
request.getContentType() => null
request.getQueryString() => from=0&resultCount=10&sortAsc=true&searchType=quick&term=%C3%BC
request.getParameter("term") => ü
Charset.defaultCharset() => UTF-8
OutputStreamWriter.getEncoding() => UTF8
new String(request.getParameter("term").getBytes(), UTF-8) => ü
System.getProperty("file.encoding") => UTF-8
By looking into the sources of Tomcat and Coyote which implement request.getParameter() i found the problem: the URIEncoding from the connector was always null and in this case it defaults to org.apache.coyote.Constants.DEFAULT_CHARACTER_ENCODING which is "ISO-8859-1" like Wolfram said.
Long story short: my fault was editing the server.xml in Tomcat's conf directory which is only loaded ONCE into Eclipse when a new server is created in the servers view! After that, a separate server.xml in the Servers project has to be edited. After doing so, the connector setting is loaded correctly and everything works as it should.
Thanks for the comments! Hope this helps someone...

Java string encoding conversion within a webpage

I have a webpage that is encoded (through its header) as WIN-1255.
A Java program creates text string that are automatically embedded in the page. The problem is that the original strings are encoded in UTF-8, thus creating a Gibberish text field in the page.
Unfortunately, I can not change the page encoding - it's required by a customer propriety system.
Any ideas?
UPDATE:
The page I'm creating is an RSS feed that needs to be set to WIN-1255, showing information taken from another feed that is encoded in UTF-8.
SECOND UPDATE:
Thanks for all the responses. I've managed to convert th string, and yet, Gibberish. Problem was that XML encoding should be set in addition to the header encoding.
Adam

To the point, you need to set the encoding of the response writer. With only a response header you're basically only instructing the client application which encoding to use to interpret/display the page. This ain't going to work if the response itself is written with a different encoding.
The context where you have this problem is entirely unclear (please elaborate about it as well in future problems like this), so here are several solutions:
If it is JSP, you need to set the following in top of JSP to set the response encoding:
<%# page pageEncoding="WIN-1255" %>
If it is Servlet, you need to set the following before any first flush to set the response encoding:
response.setCharacterEncoding("WIN-1255");
Both by the way automagically implicitly set the Content-Type response header with a charset parameter to instruct the client to use the same encoding to interpret/display the page. Also see this article for more information.
If it is a homegrown application which relies on the basic java.net and/or java.io API's, then you need to write the characters through an OutputStreamWriter which is constructed using the constructor taking 2 arguments wherein you can specify the encoding:
Writer writer = new OutputStreamWriter(someOutputStream, "WIN-1255");

Assuming you have control of the original (properly represented) strings, and simply need to output them in win-1255:
import java.nio.charset.*;
import java.nio.*;
Charset win1255 = Charset.forName("windows-1255");
ByteBuffer bb = win1255.encode(someString);
byte[] ba = new byte[bb.limit()];
Then, simply write the contents of ba at the appropriate place.
EDIT: What you do with ba depends on your environment. For instance, if you're using servlets, you might do:
ServletOutputStream os = ...
os.write(ba);
We also should not overlook the possible approach of calling setContentType("text/html; charset=windows-1255") (setContentType), then using getWriter normally. You did not make completely clear if windows-1255 was being set in a meta tag or in the HTTP response header.
You clarified that you have a UTF-8 file that you need to decode. If you're not already decoding the UTF-8 strings properly, this should no big deal. Just look at InputStreamReader(someInputStream, Charset.forName("utf-8"))

What's embedding the data in the page? Either it should read it as text (in UTF-8) and then write it out again in the web page's encoding (Win-1255) or you should change the Java program to create the files (or whatever) in Win-1255 to start with.
If you can give more details about how the system works (what's generating the web page? How does it interact with the Java program?) then it will make things a lot clearer.

The page I'm creating is an RSS feed that needs to be set to WIN-1255, showing information taken from another feed that is encoded in UTF-8.
In this case, use a parser to load the UTF-8 XML. This should correctly decode the data to UTF-16 character data (Java Strings are always UTF-16). Your output mechanism should encode from UTF-16 to Windows-1255.

byte[] originalUtf8;//Here input
//utf-8 to java String:
String internal = new String(originalUtf8,Charset.forName("utf-8");
//java string to w1255 String
byte[] win1255 = internal.getBytes(Charset.forName("cp1255"));
//Here output