i have a text file with WINDOWS-1252 characters like ø and ß. the file is being uploaded via form submit to a servlet, where it's being parsed with opencsv and returned as a List object to a jsp page where it's displayed.
the utf-8 chars are displayed as ? and i'm trying to figure out where along the way the encoding might have gone wrong.
i've tried a bunch of stuff:
my page has the tag <%#page contentType="text/html" pageEncoding="WINDOWS-1252"%>
file input is encoded - new FileInputStream(file), "WINDOWS-1252")
every string is encoded - s = new String(s.getBytes("WINDOWS-1252"));
where else can the encoding fail? any ideas?
Some troubleshooting suggestions:
Debug print or otherwise examine the text as hex at various phases, and verify that encoding really is what you expect it to be.
Make sure there is no BOM (Byte Order Marker), and see this question and links in it if there is and you don't have an easy way to get rid of it: Reading UTF-8 - BOM marker
OK problem is fixed.
So the first problem was that it wasn't a utf-8 file at all but a WINDOWS-1252 one. i determined that using the juniversalchardet lib (very helpful and easy-to-use).
Then i had to make sure that i'm reading the file with the right charset by using a FileInputStream:
new FileInputStream(file), "WINDOWS-1252")
the i just had to make sure that i am displaying it with the right charset in the jsp file using the tag <%#page contentType="text/html" pageEncoding="WINDOWS-1252"%>
that's pretty much it-
(1) determine charset
(2) make sure you're reading the file right
(3) make sure you display it right
Related
I am having a program which supports internationalisation. I have entries where input is provided in Japanese characters. On exporting that entry in XML, using JAXB, Japanese characters looks fine in the file. Proper character is been exported in the XML file. I am facing issue when unmarshal that XML file to get back data as Java object. I am not get proper unmarshalled value of japanese character.
Here is my marshalling code:
OutputStreamWriter outputWriter = new OutputStreamWriter(new FileOutputStream(file), "UTF-8");
JAXB.marshal(xmlobj, outputWriter);
Unmarshalling code:
InputStreamReader inputReader = new InputStreamReader(xml, "UTF-8");
xmlobj = JAXB.unmarshal(inputReader, <JAVA_CLASS_TO_UNMARSHAL>);
The text I am marshalling-unmarshalling is: 説明_1
It displays correctly on fetching this record and display it to browser, but in case of JAXB unmarshalling incorrect value is displayed. After converting it to HTML compatible code I got value 説明_1, which is actually correct conversion of Japanese characters. And it should appear as proper character on the browser, but it does not do so. It displays as HTML codes 説明_1 to the browser.
Any guess where I am doing wrong?
If the HTML contains
<html>
<body>
説明_1<br>
</body>
</html>
and good browser like Firefox (I have 31.0) should display 説明_1. Can you add the HTML section to your question?
If your browser isn't fit to display these characters, you should see something like .
You report that you see 説明_1, which is possible if your HTML text contains
説明_1<br>
which would mean that the transformation to HTML hasn't worked correctly.
Once more: check your HTML code, and how it is produced from the XML.
Try using UTF-8 in your HTML Header. Note that just changing the charset in the header won't convert the content — you need to make sure that the content is actually UTF-8 as well.
<Meta http-equiv = "Content-Type" content = "text / html; charset = UTF-8" >
Comment specified by Wundwin Born has solved the issue. I forgot to unescape string.
Here is the code snippet.
org.apache.commons.lang.StringEscapeUtils.unescapeHtml(xmlString);
I have an application in java.
This application contains one text box and button.
Now I want to save Gujrati(other language)data in to database in click event of button.
How is it possible? Actually I done this think but my string return some other format.
So i don't know how to store gujrati data in to string ?
This works in Java as long as source code is in a Unicode-defined encoding, such as UTF-8 or UTF-16:
String ગઉજ = "ઋઊઘ";
That part solved, you need to specify where exactly your problem lies.
Java works in Unicode. Gujarati characters have unicode values as shown here
You can directly store them in a string. However if you can't directly take Gujrati input you can use the character class like this
int c = 0x0A82;
String s = Character.toString((char)c);
//s is ં
And so on
There are some changes that you should make in you website
In JSP file change
<%# page language="java" contentType="text/html; charset=UTF-16" pageEncoding="UTF-16"%>
and in .property file
save your text in uni-code format
Like this(here language of uni-code character is Gujarati
global.Name = \u0AA8\u0ABE\u0AAE
This will surely work as it worked for me in struts2
output will be :
નામ
I'm just using:
NumberFormat cfLocal = NumberFormat.getCurrencyInstance(Locale.JAPAN.toString());
And it works fine on most devices/browsers/currencies except in IE and Yen I'm getting a few extra characters - could it be a weird encoding being sent, or browser specific settings screwing up handling of the ¥ symbol?
The output looks like this:
ï¿¥15,180
Would appreciate any leads or tips.
Edit:
I am outputting the values with JSP. JSP file is defined with this preamble:
<?xml version="1.0" encoding="ISO-8859-1" ?>
<%# page language="java" contentType="text/html; charset=UTF-8"
pageEncoding="UTF-8"%>
I'm no encoding expert but your XML appears to say one thing and your content-type another - try setting both to UTF-8.
If your data is coming from outside the application (e.g. a database, file, etc.), what is the encoding of the source? For example, a MySQL database may have a different character set specified.
If you are using a web server like Apache, is that changing the encoding? For example you can have a httpd.conf directive to set the default character set:
AddDefaultCharset utf-8
It would be worth checking the HTTP Headers in the browser to see what is actually being sent to the browser, and work back from there.
EDIT
Thinking about it more, I'm not sure if the XML encoding is necessarily the problem. It would probably be best to check the headers first, and compare it to the the html being produced.
When I add a Filter to a particular JSP file, the Arabic characters in the output appears like ???, even when the page encoding is been set to UTF-8 by <% #page pageEncoding="UTF-8"%> and <% response.setCharacterEncoding("UTF-8");%>.
The strange thing is, before I added the Filter, the output of all Arabic pages appears with correct encoding. Can someone tell how this problem is caused and how I can solve it?
The filter is either directly or indirectly commiting the response and/or accessing the Writer or OutputStream of the HttpServletResponse which causes that the encoding cannot be changed anymore in the JSP. Fix the code in the filter accordingly. The filter should in any way not be writing anything to the response body. There the JSP (for HTML) or Servlet (for other content) is for.
By the way, you don't need to call <% response.setCharacterEncoding("UTF-8");%>. The <%#page pageEncoding="UTF-8"%> already implicitly does that.
On the web pages in our app, the trademark symbol (TM) is appearing as a questions mark. The registered trademark (R) works, though. We are displaying the value using the c:out tag in the JSP standard library. If I put ™ or ™ on the page to test this, those show up as they are supposed to.
<td><c:out value="${item.description}"/></td> <!-- does not work -->
<td>yada yada yada Spiros™ yada yada yada</td> <!-- works -->
To add to this, we're also using YUI, and before we display these pages, they show up in a YUI data table as the results of a query (the user clicks on a row to go to the page described above). The (TM) shows up properly in that table. That tells me that we are properly fetching the value from our database, and as well the server code generating the XML to send back to the YUI data table also works.
So why is the same String displayed properly in the YUI data table, but not in a normal JSP, unless we hardcode the symbol onto the page?
You probably have an encoding issue. If you do not have an explicit encoding in your JSP:
<%# page contentType="text/html; charset=UTF-8" pageEncoding="UTF-8" %>
then it's time to add one. Try UTF-8 and if that doesn't work try ISO-8859-1 ... or if you know the correct encoding, use that.
When a char appears as ? inside a browser (usually Firefox) it means that page encoding (as it's detected by the browser will not recognize the char. A good test would be to View->Character Encoding->UTF-8 in firefox. If the char appears correctly then it means that the (tm) char is encoded using UTF-8 standard. You have to instruct your page to set the response encoding header to UTF-8. This should work right now for you.
If that would not work you should first find out how is the character encoded (look at what encoding is read from the database for example) and try to set the page encoding header to that encoding.
The second format works because the (TM) char is encoded as a known html entity which the browser interprets regardless of the page encoding.