So basically I'm trying to send an email with Japanese characters, something like "𥹖𥹖𥹖" and then I got "???" what should I do to encode this? I have looked over a bunch of solutions but none of them have helped me solve this.
here's the method I've been trying to do the encode:
public String encoding(String str) throws UnsupportedEncodingException{
String Encoding = "Shift_JIS";
return this.changeCharset(str, Encoding);
}
public String changeCharset(String str, String newCharset) throws UnsupportedEncodingException {
if (str != null) {
byte[] jis = str.getBytes("Shift_JIS");
return new String(bs, newCharset);
}
return null;
}
You're making this too complicated...
First, make sure you have the Japanese text in a proper Java String object, using proper Unicode characters.
Then, set the content of the body part using this method:
htmlPart.setText(japaneseString, "Shift_JIS", "html");
I have a constraint: I cannot save some chars (like & and =) in a some special storage.
The problem is that I have strings (user input) that contain these not allowed special chars, which I'd like to save to that storage .
I'd like to convert such string to another string that wouldn't contain these special characters.
I'd like to still be able to convert back to the original string without creating ambiguity.
Any idea how to implement the de/convert? Thanks.
Convert the user input to Hex and save. And convert the hex value back to string. Use these methods.
public static String stringToHex(String arg) {
return String.format("%x", new BigInteger(1, arg.getBytes(Charset.forName("UTF-8"))));
}
public static String hexToString(String arg) {
byte[] bytes = DatatypeConverter.parseHexBinary(arg);
return new String(bytes, Charset.forName("UTF-8"));
}
Usage:
String h = stringToHex("Perera & Sons");
System.out.println(h);
System.out.println(hexToString(h));
OUTPUT
506572657261202620536f6e73
Perera & Sons
Already pointed out in the comments but URL Encoding looks like the way to go.
In Java done simply URLEncoder and URLDecoder
String encoded = URLEncoder.encode("My string &with& illegal = characters ", "UTF-8");
System.out.println("Encoded String:" + encoded);
String decoded = URLDecoder.decode(encoded, "UTF-8");
System.out.println("Decoded String:" + decoded);
URLEncoder
URLDecoder
I use the below function to retrieve the web service response:
private String getSoapResponse (String url, String host, String encoding, String soapAction, String soapRequest) throws MalformedURLException, IOException, Exception {
URL wsUrl = new URL(url);
URLConnection connection = wsUrl.openConnection();
HttpURLConnection httpConn = (HttpURLConnection)connection;
ByteArrayOutputStream bout = new ByteArrayOutputStream();
byte[] buffer = new byte[soapRequest.length()];
buffer = soapRequest.getBytes();
bout.write(buffer);
byte[] b = bout.toByteArray();
httpConn.setRequestMethod("POST");
httpConn.setRequestProperty("Host", host);
if (encoding == null || encoding == "")
encoding = UTF8;
httpConn.setRequestProperty("Content-Type", "text/xml; charset=" + encoding);
httpConn.setRequestProperty("Content-Length", String.valueOf(b.length));
httpConn.setRequestProperty("SOAPAction", soapAction);
httpConn.setDoOutput(true);
httpConn.setDoInput(true);
OutputStream out = httpConn.getOutputStream();
out.write(b);
out.close();
InputStreamReader is = new InputStreamReader(httpConn.getInputStream());
StringBuilder sb = new StringBuilder();
BufferedReader br = new BufferedReader(is);
String read = br.readLine();
while(read != null) {
sb.append(read);
read = br.readLine();
}
String response = decodeHtmlEntityCharacters(sb.toString());
return response = decodeHtmlEntityCharacters(response);
}
But my problem with this code is it returns lots of special characters and makes the structure of the XML invalid.
Example response:
<PLANT>A565</PLANT>
<PLANT>A567</PLANT>
<PLANT>A585</PLANT>
<PLANT>A921</PLANT>
<PLANT>A938</PLANT>
</PLANT_GROUP>
</KPI_PLANT_GROUP_KEYWORD>
<MSU_CUSTOMERS/>
</DU>
<DU>
So to solve this, I use the below method and pass the whole response to replace all the special characters with its corresponding punctuation.
private final static Hashtable htmlEntitiesTable = new Hashtable();
static {
htmlEntitiesTable.put("&","&");
htmlEntitiesTable.put(""","\"");
htmlEntitiesTable.put("<","<");
htmlEntitiesTable.put(">",">");
}
private String decodeHtmlEntityCharacters(String inputString) throws Exception {
Enumeration en = htmlEntitiesTable.keys();
while(en.hasMoreElements()){
String key = (String)en.nextElement();
String val = (String)htmlEntitiesTable.get(key);
inputString = inputString.replaceAll(key, val);
}
return inputString;
}
But another problem arised. If the response contains this segment <VALUE>< 0.5 </VALUE< and if this will be evaluated by the method, the output would be:
<VALUE>< 0.5</VALUE>
Which makes the structure of the XML invalid again.
The data is correct and valid "< 0.5" but having it within the VALUE elements causes issue on the structure of the XML.
Can you please help how to deal with this? Maybe the way I get or build the response can be improved. Is there any better way to call and get the response from web service?
How can I deal with elements containing "<" or ">"?
Do you know how to use a third-party open source library?
You should try using apache commons-lang:
StringEscapeUtils.unescapeXml(xml)
More detail is provided in the following stack overflow post:
how to unescape XML in java
Documentation:
http://commons.apache.org/proper/commons-lang/javadocs/api-release/index.html
http://commons.apache.org/proper/commons-lang/userguide.html#lang3.
You're using SOAP wrong.
In particular, you do not need the following line of code:
String response = decodeHtmlEntityCharacters(sb.toString());
Just return sb.toString(). And for $DEITY's sake, do not use string methods to parse the retrieved string, use an XML parser, or a full-blown SOAP stack...
Does the > or < character always appear at the beginning of a value? Then you could use regex to handle the cases in which the > or < are followed by a digit (or dot, for that matter).
Sample code, assuming the replacement strings used in it don't appear anywhere else in the XML:
private String decodeHtmlEntityCharacters(String inputString) throws Exception {
Enumeration en = htmlEntitiesTable.keys();
// Replaces > or < followed by dot or digit (while keeping the dot/digit)
inputString = inputString.replaceAll(">(\\.?\\d)", "Valuegreaterthan$1");
inputString = inputString.replaceAll("<(\\.?\\d)", "Valuelesserthan$1");
while(en.hasMoreElements()){
String key = (String)en.nextElement();
String val = (String)htmlEntitiesTable.get(key);
inputString = inputString.replaceAll(key, val);
}
inputString = inputString.replaceAll("Valuelesserthan", "<");
inputString = inputString.replaceAll("Valuegreaterthan", ">");
return inputString;
}
Note the most appropriate answer (and easier for everyone) would be to correctly encode the XML at the sender side (it would also render my solution non-working BTW).
It would be hard to cope with all the situations but you could cover the most common ones by adding a few more rules by assuming that any less than followed by a space is data, and a greater than that has a space in front of it is data and need to be encoded again.
private final static Hashtable htmlEntitiesTable = new Hashtable();
static {
htmlEntitiesTable.put("&","&");
htmlEntitiesTable.put(""","\"");
htmlEntitiesTable.put("<","<");
htmlEntitiesTable.put(">",">");
}
private String decodeHtmlEntityCharacters(String inputString) throws Exception {
Enumeration en = htmlEntitiesTable.keys();
while(en.hasMoreElements()){
String key = (String)en.nextElement();
String val = (String)htmlEntitiesTable.get(key);
inputString = inputString.replaceAll(key, val);
}
inputString = inputString.replaceAll("< ","< ");
inputString = inputString.replaceAll(" >"," >");
return inputString;
}
'>' is not escaped in XML. So you shouldn't have an issue with that. Regarding '<', here are the options I can think of.
Use CDATA in web response for text containing special characters.
Rewrite the text by reversing the order. For eg. if it is x < 2, change it to 2 > x. '>' is not escaped unless its a part of CDATA.
Use another attribute or element in the XML response to indicate '<' or '>'.
Use regular expression to find a sequence that starts with '<' and followed by a string, and followed by '<' of the closing tag. And replace it with some code or some value that you can interpret and replace later.
Also, you don't need to do this:
String response = decodeHtmlEntityCharacters(sb.toString());
You should be able to parse the XML after you take care of the '<' sign in text.
You can use this site for testing regular expressions.
Why not serialize your xml?, its much easier than what you are doing.
for an example:
var ser = new XmlSerializer(typeof(MyXMLObject));
using (var reader = XmlReader.Create("http.....xml"))
{
MyXMLObject _myobj = (response)ser.Deserialize(reader);
}
I am calling an API which returns a JSON response. While reading the JSON response there are some places where data has some special characters. I want to exclude these special characters while reading the response in an object. The JSON response looks like this:
{"data":[{"title":"PSY - GANGNAM STYLE (\uac15\ub0a8\uc2a4\ud0c0\uc77c) M\/V","content":All rights reserved."}]}
The Java code is this:
BufferedReader reader = new BufferedReader(new InputStreamReader(
is, "ISO-8859-1"), 8);
When I read the title key from the response, it gives me these special characters as well which I don't want. How do get rid of those? Do i need to specify some other encoding?
Data Source :http://pipes.yahoo.com/pipes/pipe.run?_id=920adeb2e95c15877e29dc678aa78dd7&_render=json&n=1
This isn't an encoding issue (like UTF-8), it's a JavaScript syntax issue. The "\uac15", for example, is JavaScript syntax that represents the Unicode character U+AC15, which is "강". Together, those escaped characters are the name of the song written in Hangul (Korean): "강남스타일".
It's normal and OK for your Java string to contain the backslash escape sequences. When you run that string though a JSON reader, you should get a JSON object containing the actual Hangul characters.
In response to your comment about getting wrong output from a JSON reader, that depends on what JSON library you're using (and how you're using it), which you didn't specify in the question. Here's an example that works for me using Jackson 2.1.0:
public final class JsonTest {
public static void main(final String[] args) {
final String json = "\"PSY - GANGNAM STYLE (\\uac15\\ub0a8\\uc2a4\\ud0c0\\uc77c) M\\/V\"";
System.out.println("JSON: " + json);
try {
// ObjectMapper is from Jackson 2.1 databind library.
final ObjectMapper mapper = new ObjectMapper();
final String decoded = mapper.readValue(json, String.class);
System.out.println("Decoded: " + decoded);
}
catch (final IOException e) {
e.printStackTrace();
}
}
}
I am expecting
System.out.println(java.net.URLEncoder.encode("Hello World", "UTF-8"));
to output:
Hello%20World
(20 is ASCII Hex code for space)
However, what I get is:
Hello+World
Am I using the wrong method? What is the correct method I should be using?
This behaves as expected. The URLEncoder implements the HTML Specifications for how to encode URLs in HTML forms.
From the javadocs:
This class contains static methods for
converting a String to the
application/x-www-form-urlencoded MIME
format.
and from the HTML Specification:
application/x-www-form-urlencoded
Forms submitted with this content type
must be encoded as follows:
Control names and values are escaped. Space characters are replaced
by `+'
You will have to replace it, e.g.:
System.out.println(java.net.URLEncoder.encode("Hello World", "UTF-8").replace("+", "%20"));
A space is encoded to %20 in URLs, and to + in forms submitted data (content type application/x-www-form-urlencoded). You need the former.
Using Guava:
dependencies {
compile 'com.google.guava:guava:23.0'
// or, for Android:
compile 'com.google.guava:guava:23.0-android'
}
You can use UrlEscapers:
String encodedString = UrlEscapers.urlFragmentEscaper().escape(inputString);
Don't use String.replace, this would only encode the space. Use a library instead.
This class perform application/x-www-form-urlencoded-type encoding rather than percent encoding, therefore replacing with + is a correct behaviour.
From javadoc:
When encoding a String, the following rules apply:
The alphanumeric characters "a" through "z", "A" through "Z" and "0" through "9" remain the same.
The special characters ".", "-", "*", and "_" remain the same.
The space character " " is converted into a plus sign "+".
All other characters are unsafe and are first converted into one or more bytes using some encoding scheme. Then each byte is represented by the 3-character string "%xy", where xy is the two-digit hexadecimal representation of the byte. The recommended encoding scheme to use is UTF-8. However, for compatibility reasons, if an encoding is not specified, then the default encoding of the platform is used.
Encode Query params
org.apache.commons.httpclient.util.URIUtil
URIUtil.encodeQuery(input);
OR if you want to escape chars within URI
public static String escapeURIPathParam(String input) {
StringBuilder resultStr = new StringBuilder();
for (char ch : input.toCharArray()) {
if (isUnsafe(ch)) {
resultStr.append('%');
resultStr.append(toHex(ch / 16));
resultStr.append(toHex(ch % 16));
} else{
resultStr.append(ch);
}
}
return resultStr.toString();
}
private static char toHex(int ch) {
return (char) (ch < 10 ? '0' + ch : 'A' + ch - 10);
}
private static boolean isUnsafe(char ch) {
if (ch > 128 || ch < 0)
return true;
return " %$&+,/:;=?#<>#%".indexOf(ch) >= 0;
}
Hello+World is how a browser will encode form data (application/x-www-form-urlencoded) for a GET request and this is the generally accepted form for the query part of a URI.
http://host/path/?message=Hello+World
If you sent this request to a Java servlet, the servlet would correctly decode the parameter value. Usually the only time there are issues here is if the encoding doesn't match.
Strictly speaking, there is no requirement in the HTTP or URI specs that the query part to be encoded using application/x-www-form-urlencoded key-value pairs; the query part just needs to be in the form the web server accepts. In practice, this is unlikely to be an issue.
It would generally be incorrect to use this encoding for other parts of the URI (the path for example). In that case, you should use the encoding scheme as described in RFC 3986.
http://host/Hello%20World
More here.
If you want to encode URI path components, you can also use standard JDK functions, e.g.
public static String encodeURLPathComponent(String path) {
try {
return new URI(null, null, path, null).toASCIIString();
} catch (URISyntaxException e) {
// do some error handling
}
return "";
}
The URI class can also be used to encode different parts of or whole URIs.
Just been struggling with this too on Android, managed to stumble upon Uri.encode(String, String) while specific to android (android.net.Uri) might be useful to some.
static String encode(String s, String allow)
https://developer.android.com/reference/android/net/Uri.html#encode(java.lang.String, java.lang.String)
The other answers either present a manual string replacement, URLEncoder which actually encodes for HTML format, Apache's abandoned URIUtil, or using Guava's UrlEscapers. The last one is fine, except it doesn't provide a decoder.
Apache Commons Lang provides the URLCodec, which encodes and decodes according to URL format rfc3986.
String encoded = new URLCodec().encode(str);
String decoded = new URLCodec().decode(str);
If you are already using Spring, you can also opt to use its UriUtils class as well.
Although quite old, nevertheless a quick response:
Spring provides UriUtils - with this you can specify how to encoded and which part is it related from an URI, e.g.
encodePathSegment
encodePort
encodeFragment
encodeUriVariables
....
I use them cause we already using Spring, i.e. no additonal library is required!
If you are using jetty then org.eclipse.jetty.util.URIUtil will solve the issue.
String encoded_string = URIUtil.encodePath(not_encoded_string).toString();
This worked for me
org.apache.catalina.util.URLEncoder ul = new org.apache.catalina.util.URLEncoder().encode("MY URL");
It's not one-liner, but you can use:
URL url = new URL("https://some-host.net/dav/files/selling_Rosetta Stone Case Study.png.aes");
URI uri = new URI(url.getProtocol(), url.getUserInfo(), url.getHost(), url.getPort(), url.getPath(), url.getQuery(), url.getRef());
System.out.println(uri.toString());
This will give you an output:
https://some-host.net/dav/files/selling_Rosetta%20Stone%20Case%20Study.png.aes
"+" is correct. If you really need %20, then replace the Plusses yourself afterwards.
Warning: This answer is heavily disputed (+8 vs. -6), so take this with a grain of salt.
I was already using Feign so UriUtils was available to me but Spring UrlUtils was not.
<!-- https://mvnrepository.com/artifact/io.github.openfeign/feign-core -->
<dependency>
<groupId>io.github.openfeign</groupId>
<artifactId>feign-core</artifactId>
<version>11.8</version>
</dependency>
My Feign test code:
import feign.template.UriUtils;
System.out.println(UriUtils.encode("Hello World"));
Outputs:
Hello%20World
As the class suggests, it encodes URIs and not URLs but the OP asked about URIs and not URLs.
System.out.println(UriUtils.encode("https://some-host.net/dav/files/selling_Rosetta Stone Case Study.png.aes"));
Outputs:
https%3A%2F%2Fsome-host.net%2Fdav%2Ffiles%2Fselling_Rosetta%20Stone%20Case%20Study.png.aes
Try below approach:
Add a new dependency
<!-- https://mvnrepository.com/artifact/org.apache.tomcat/tomcat-catalina -->
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>tomcat-catalina</artifactId>
<version>10.0.13</version>
</dependency>
Now do as follows:
String str = "Hello+World"; // For "Hello World", decoder is not required
// import java.net.URLDecoder;
String newURL = URLDecoder.decode(str, StandardCharsets.UTF_8);
// import org.apache.catalina.util.URLEncoder;
System.out.println(URLEncoder.DEFAULT.encode(newURL, StandardCharsets.UTF_8));
You'll get the output as:
Hello%20World
Check out the java.net.URI class.
USE MyUrlEncode.URLencoding(String url , String enc) to handle the problem
public class MyUrlEncode {
static BitSet dontNeedEncoding = null;
static final int caseDiff = ('a' - 'A');
static {
dontNeedEncoding = new BitSet(256);
int i;
for (i = 'a'; i <= 'z'; i++) {
dontNeedEncoding.set(i);
}
for (i = 'A'; i <= 'Z'; i++) {
dontNeedEncoding.set(i);
}
for (i = '0'; i <= '9'; i++) {
dontNeedEncoding.set(i);
}
dontNeedEncoding.set('-');
dontNeedEncoding.set('_');
dontNeedEncoding.set('.');
dontNeedEncoding.set('*');
dontNeedEncoding.set('&');
dontNeedEncoding.set('=');
}
public static String char2Unicode(char c) {
if(dontNeedEncoding.get(c)) {
return String.valueOf(c);
}
StringBuffer resultBuffer = new StringBuffer();
resultBuffer.append("%");
char ch = Character.forDigit((c >> 4) & 0xF, 16);
if (Character.isLetter(ch)) {
ch -= caseDiff;
}
resultBuffer.append(ch);
ch = Character.forDigit(c & 0xF, 16);
if (Character.isLetter(ch)) {
ch -= caseDiff;
}
resultBuffer.append(ch);
return resultBuffer.toString();
}
private static String URLEncoding(String url,String enc) throws UnsupportedEncodingException {
StringBuffer stringBuffer = new StringBuffer();
if(!dontNeedEncoding.get('/')) {
dontNeedEncoding.set('/');
}
if(!dontNeedEncoding.get(':')) {
dontNeedEncoding.set(':');
}
byte [] buff = url.getBytes(enc);
for (int i = 0; i < buff.length; i++) {
stringBuffer.append(char2Unicode((char)buff[i]));
}
return stringBuffer.toString();
}
private static String URIEncoding(String uri , String enc) throws UnsupportedEncodingException { //对请求参数进行编码
StringBuffer stringBuffer = new StringBuffer();
if(dontNeedEncoding.get('/')) {
dontNeedEncoding.clear('/');
}
if(dontNeedEncoding.get(':')) {
dontNeedEncoding.clear(':');
}
byte [] buff = uri.getBytes(enc);
for (int i = 0; i < buff.length; i++) {
stringBuffer.append(char2Unicode((char)buff[i]));
}
return stringBuffer.toString();
}
public static String URLencoding(String url , String enc) throws UnsupportedEncodingException {
int index = url.indexOf('?');
StringBuffer result = new StringBuffer();
if(index == -1) {
result.append(URLEncoding(url, enc));
}else {
result.append(URLEncoding(url.substring(0 , index),enc));
result.append("?");
result.append(URIEncoding(url.substring(index+1),enc));
}
return result.toString();
}
}
Am I using the wrong method? What is the correct method I should be using?
Yes, this method java.net.URLEncoder.encode wasn't made for converting " " to "20%" according to spec (source).
The space character " " is converted into a plus sign "+".
Even this is not the correct method, you can modify this to: System.out.println(java.net.URLEncoder.encode("Hello World", "UTF-8").replaceAll("\\+", "%20"));have a nice day =).
use character-set "ISO-8859-1" for URLEncoder