I have list of tweet as the input to the hdfs, and try to perform a map-reduce task. This is my mapper implementation:
#Override
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
try {
String[] fields = value.toString().split("\t");
StringBuilder sb = new StringBuilder();
for (int i = 1; i < fields.length; i++) {
if (i > 1) {
sb.append("\t");
}
sb.append(fields[i]);
}
tid.set(fields[0]);
content.set(sb.toString());
context.write(tid, content);
} catch(DecoderException e) {
e.printStackTrace();
}
}
As you can see, I tried to split the input by "\t", but the input (value.toString()) looks like this when I print it out:
2014\x091880284777\x09argento_un\x090\x090\x09RT #topmusic619: #RETWEET THIS!!!!!\x5CnFOLLOW ME &
; EVERYONE ELSE THAT RETWEETS THIS FOR 35+ FOLLOWERS\x5Cn#TeamFollowBack #Follow2BeFollowed #TajF\xE2\x80\xA6
here is another example:
2014\x0934447260\x09RBEKP\x090\x090\x09\xE2\x80\x9C#LENEsipper: Wild lmfaooo RT #Yerrp08: L**o some
n***a nutt up while gettin twerked
I noted that \x09 should be a tab character (ASCII 09 is tab), So I tried to use apache Hex:
String tmp = value.toString();
byte[] bytes = Hex.decodeHex(tmp.toCharArray());
But the decodeHex function returns null.
This is weird, since some of the characters are in hex while others are not. How can I decode them?
Edit:
Also note that besides tab, emojis are also encoded as hex values.
I am currently working on a application that reads a big text, and highlights a specific substring inside that text. And it kinda works...
But it seems that (and I have no clue why) the highligt keeps shifting every time it highlights the string.
public List<int[]> findString(String text) {
text = text.toLowerCase();
List<int[]> highlightPositions = new ArrayList<int[]>();
JTextPane pane = getTextPane();
String paneText = pane.getText().toLowerCase();
int end = 0;
while (paneText.indexOf(text, end) != -1) {
int start = paneText.indexOf(text, end);
end = start + text.length();
highlightPositions.add(new int[] {start, end});
}
try {
highlight(highlightPositions);
} catch (Exception ex) {
}
return null;
}
and this is the code that does the actual highlighting
public void highlight(List<int[]> highlightPositions) throws BadLocationException {
DefaultHighlighter.DefaultHighlightPainter highlightPainter = new DefaultHighlighter.DefaultHighlightPainter(Color.YELLOW);
JTextPane textPane = getTextPane();
for (int[] position : highlightPositions) {
System.out.println("Highlight: " + position[0] + " : " + position[1]);
textPane.getHighlighter().addHighlight(position[0], position[1], highlightPainter);
}
}
Does anyone know how to fix this?
EDIT:
Here is how it looks when I attempt to highlight the word "Device".
Highlighting output
String paneText = pane.getText().toLowerCase();
Don't use getText(). The getText() method will return the string with the end-of-line string for the platform which in the case of Windows is \r\n. However the Document only stores \n for the EOL string so you have a mismatch of offsets for every extra line in the Document.
The solution is to use:
String paneText = pane.getDocument().getText().toLowerCase();
See Text and New Lines for more complete information on the problem.
I want to generate a RTF-Document with my program. I use an RTFEditor which gives me the edited text in a rtfformat, but I also have TextFields and I should take Strings from the TextFields into the RTF Documeent. I have tried to edit the rtf document text, but i got this annoying encoding problem with ANSI and UTF-8 i think.
If my TextFields got strings with an "äöüß" ... and so on i get this encoding stuff like "ö"
My target is to get a good looking RTF-Document without encoding-issues.
What is the best way to handle this issue?
Here is my class which outputs the rtf document:
NOTE: Please focus into the important things in this code. I hope the german terms in my code dont confuse you :).
/**
* Handles the Saving of a Protokoll to a .rtf file.
* The .executeSaving() method executes the process of saving.
* #author me
*/
public class WordExport {
// other methods...
/**
* Executes the saving-progress by the given file.
* Structure of the rtf-File:
* -------------------------
* HeadString |
* -------------------------
* Content |
* .. |
* .. |
* .. |
* .. |
* -------------------------
* #param file points to the target where the .rtf should be saved.
* #param protokoll The protokoll which you want to save.
* #throws IOException Thrown when something is not okay with the IO.
*/
public void executeSaving(File file, Protokoll protokoll) throws IOException
{
//I want to insert my HeaderString into a specific position so i have to split etc...
String content = new String();
content = protokoll.getInhalt();
String[] split = content.split("}", 3);
for(int i = 0; i < split.length ; i++)
{
if(!split[i].contains("}"))
{
split[i] += "}";
}
}
String teilnehmer = new String();
for (Profil p : protokoll.getTeilnehmer())
{
teilnehmer += p.toString() + "\\par\n ";
}
String headString = new String();
//HERE IS THE IMPORTANT CODE --> here i put my strings bare into the rtf-format and i dont know how to handle the encoding.
headString = "\\f1\\fs44\\i0\\b0\\ul\\cf1\\ "+ protokoll.getTitel() +" \\par\n" +
"\\par\n" +
"\\fs24\\ul0 Raum: "+ protokoll.getRaum() +"\\par\n" +
"Zeitraum: "+ protokoll.getZeitraum() +"\\par\n" +
"Datum: "+ protokoll.getErstellungsdatum().toString() +"\\par\n" +
"\\par\n" +
"Teilnehmer: \\par\n" +
teilnehmer +
"\\ul0\\par ";
split[2] = headString+ split[2];
//Converts the String[] from above to the string, which you need to write into the .rtf file.
StringBuilder builder = new StringBuilder();
for(String s : split)
{
builder.append(s);
}
content = builder.toString();
if(!file.exists())
{
file.createNewFile();
}
FileWriter fw = new FileWriter(file.getAbsoluteFile());
BufferedWriter bw = new BufferedWriter(fw);
bw.write(content);
bw.close();
}
}
Use ISO-8859-1 encode. This encode format will support that characters too.
I hate character encodings... I believe the following blog might help you here
i will post the code here just in case that site has issues in the future though, NOTE I DID NOT WRITE THIS NOR DO I TAKE CREDIT.
// Create a hashtable to hold the characters to convert
Hashtable<String, String> replace = new Hashtable<String, String>();
// The String to convert
String str = "The Māori Macron";
// Values we'll use in the loop
int value;
String bit;
for (int i = 0; i < str.length(); i++) {
// Get the character value
bit = str.substring(i, i + 1);
value = str.codePointAt(i);
// If the character value is above the
// 7-bit range of RTF ASCII
if (value > 127) {
replace.put(bit, "\\\\u" + value + "\\\\' ");
}
}
// Now replace all the characters we found
Enumeration e = parameters.keys();
String key, value;
while (e.hasMoreElements()) {
// Get the key
key = (String)e.nextElement();
// Get the value
value = (String)parameters.get(key);
// Make the substitution
str = str.replaceAll(key, value);
}
I use the below function to retrieve the web service response:
private String getSoapResponse (String url, String host, String encoding, String soapAction, String soapRequest) throws MalformedURLException, IOException, Exception {
URL wsUrl = new URL(url);
URLConnection connection = wsUrl.openConnection();
HttpURLConnection httpConn = (HttpURLConnection)connection;
ByteArrayOutputStream bout = new ByteArrayOutputStream();
byte[] buffer = new byte[soapRequest.length()];
buffer = soapRequest.getBytes();
bout.write(buffer);
byte[] b = bout.toByteArray();
httpConn.setRequestMethod("POST");
httpConn.setRequestProperty("Host", host);
if (encoding == null || encoding == "")
encoding = UTF8;
httpConn.setRequestProperty("Content-Type", "text/xml; charset=" + encoding);
httpConn.setRequestProperty("Content-Length", String.valueOf(b.length));
httpConn.setRequestProperty("SOAPAction", soapAction);
httpConn.setDoOutput(true);
httpConn.setDoInput(true);
OutputStream out = httpConn.getOutputStream();
out.write(b);
out.close();
InputStreamReader is = new InputStreamReader(httpConn.getInputStream());
StringBuilder sb = new StringBuilder();
BufferedReader br = new BufferedReader(is);
String read = br.readLine();
while(read != null) {
sb.append(read);
read = br.readLine();
}
String response = decodeHtmlEntityCharacters(sb.toString());
return response = decodeHtmlEntityCharacters(response);
}
But my problem with this code is it returns lots of special characters and makes the structure of the XML invalid.
Example response:
<PLANT>A565</PLANT>
<PLANT>A567</PLANT>
<PLANT>A585</PLANT>
<PLANT>A921</PLANT>
<PLANT>A938</PLANT>
</PLANT_GROUP>
</KPI_PLANT_GROUP_KEYWORD>
<MSU_CUSTOMERS/>
</DU>
<DU>
So to solve this, I use the below method and pass the whole response to replace all the special characters with its corresponding punctuation.
private final static Hashtable htmlEntitiesTable = new Hashtable();
static {
htmlEntitiesTable.put("&","&");
htmlEntitiesTable.put(""","\"");
htmlEntitiesTable.put("<","<");
htmlEntitiesTable.put(">",">");
}
private String decodeHtmlEntityCharacters(String inputString) throws Exception {
Enumeration en = htmlEntitiesTable.keys();
while(en.hasMoreElements()){
String key = (String)en.nextElement();
String val = (String)htmlEntitiesTable.get(key);
inputString = inputString.replaceAll(key, val);
}
return inputString;
}
But another problem arised. If the response contains this segment <VALUE>< 0.5 </VALUE< and if this will be evaluated by the method, the output would be:
<VALUE>< 0.5</VALUE>
Which makes the structure of the XML invalid again.
The data is correct and valid "< 0.5" but having it within the VALUE elements causes issue on the structure of the XML.
Can you please help how to deal with this? Maybe the way I get or build the response can be improved. Is there any better way to call and get the response from web service?
How can I deal with elements containing "<" or ">"?
Do you know how to use a third-party open source library?
You should try using apache commons-lang:
StringEscapeUtils.unescapeXml(xml)
More detail is provided in the following stack overflow post:
how to unescape XML in java
Documentation:
http://commons.apache.org/proper/commons-lang/javadocs/api-release/index.html
http://commons.apache.org/proper/commons-lang/userguide.html#lang3.
You're using SOAP wrong.
In particular, you do not need the following line of code:
String response = decodeHtmlEntityCharacters(sb.toString());
Just return sb.toString(). And for $DEITY's sake, do not use string methods to parse the retrieved string, use an XML parser, or a full-blown SOAP stack...
Does the > or < character always appear at the beginning of a value? Then you could use regex to handle the cases in which the > or < are followed by a digit (or dot, for that matter).
Sample code, assuming the replacement strings used in it don't appear anywhere else in the XML:
private String decodeHtmlEntityCharacters(String inputString) throws Exception {
Enumeration en = htmlEntitiesTable.keys();
// Replaces > or < followed by dot or digit (while keeping the dot/digit)
inputString = inputString.replaceAll(">(\\.?\\d)", "Valuegreaterthan$1");
inputString = inputString.replaceAll("<(\\.?\\d)", "Valuelesserthan$1");
while(en.hasMoreElements()){
String key = (String)en.nextElement();
String val = (String)htmlEntitiesTable.get(key);
inputString = inputString.replaceAll(key, val);
}
inputString = inputString.replaceAll("Valuelesserthan", "<");
inputString = inputString.replaceAll("Valuegreaterthan", ">");
return inputString;
}
Note the most appropriate answer (and easier for everyone) would be to correctly encode the XML at the sender side (it would also render my solution non-working BTW).
It would be hard to cope with all the situations but you could cover the most common ones by adding a few more rules by assuming that any less than followed by a space is data, and a greater than that has a space in front of it is data and need to be encoded again.
private final static Hashtable htmlEntitiesTable = new Hashtable();
static {
htmlEntitiesTable.put("&","&");
htmlEntitiesTable.put(""","\"");
htmlEntitiesTable.put("<","<");
htmlEntitiesTable.put(">",">");
}
private String decodeHtmlEntityCharacters(String inputString) throws Exception {
Enumeration en = htmlEntitiesTable.keys();
while(en.hasMoreElements()){
String key = (String)en.nextElement();
String val = (String)htmlEntitiesTable.get(key);
inputString = inputString.replaceAll(key, val);
}
inputString = inputString.replaceAll("< ","< ");
inputString = inputString.replaceAll(" >"," >");
return inputString;
}
'>' is not escaped in XML. So you shouldn't have an issue with that. Regarding '<', here are the options I can think of.
Use CDATA in web response for text containing special characters.
Rewrite the text by reversing the order. For eg. if it is x < 2, change it to 2 > x. '>' is not escaped unless its a part of CDATA.
Use another attribute or element in the XML response to indicate '<' or '>'.
Use regular expression to find a sequence that starts with '<' and followed by a string, and followed by '<' of the closing tag. And replace it with some code or some value that you can interpret and replace later.
Also, you don't need to do this:
String response = decodeHtmlEntityCharacters(sb.toString());
You should be able to parse the XML after you take care of the '<' sign in text.
You can use this site for testing regular expressions.
Why not serialize your xml?, its much easier than what you are doing.
for an example:
var ser = new XmlSerializer(typeof(MyXMLObject));
using (var reader = XmlReader.Create("http.....xml"))
{
MyXMLObject _myobj = (response)ser.Deserialize(reader);
}
I am expecting
System.out.println(java.net.URLEncoder.encode("Hello World", "UTF-8"));
to output:
Hello%20World
(20 is ASCII Hex code for space)
However, what I get is:
Hello+World
Am I using the wrong method? What is the correct method I should be using?
This behaves as expected. The URLEncoder implements the HTML Specifications for how to encode URLs in HTML forms.
From the javadocs:
This class contains static methods for
converting a String to the
application/x-www-form-urlencoded MIME
format.
and from the HTML Specification:
application/x-www-form-urlencoded
Forms submitted with this content type
must be encoded as follows:
Control names and values are escaped. Space characters are replaced
by `+'
You will have to replace it, e.g.:
System.out.println(java.net.URLEncoder.encode("Hello World", "UTF-8").replace("+", "%20"));
A space is encoded to %20 in URLs, and to + in forms submitted data (content type application/x-www-form-urlencoded). You need the former.
Using Guava:
dependencies {
compile 'com.google.guava:guava:23.0'
// or, for Android:
compile 'com.google.guava:guava:23.0-android'
}
You can use UrlEscapers:
String encodedString = UrlEscapers.urlFragmentEscaper().escape(inputString);
Don't use String.replace, this would only encode the space. Use a library instead.
This class perform application/x-www-form-urlencoded-type encoding rather than percent encoding, therefore replacing with + is a correct behaviour.
From javadoc:
When encoding a String, the following rules apply:
The alphanumeric characters "a" through "z", "A" through "Z" and "0" through "9" remain the same.
The special characters ".", "-", "*", and "_" remain the same.
The space character " " is converted into a plus sign "+".
All other characters are unsafe and are first converted into one or more bytes using some encoding scheme. Then each byte is represented by the 3-character string "%xy", where xy is the two-digit hexadecimal representation of the byte. The recommended encoding scheme to use is UTF-8. However, for compatibility reasons, if an encoding is not specified, then the default encoding of the platform is used.
Encode Query params
org.apache.commons.httpclient.util.URIUtil
URIUtil.encodeQuery(input);
OR if you want to escape chars within URI
public static String escapeURIPathParam(String input) {
StringBuilder resultStr = new StringBuilder();
for (char ch : input.toCharArray()) {
if (isUnsafe(ch)) {
resultStr.append('%');
resultStr.append(toHex(ch / 16));
resultStr.append(toHex(ch % 16));
} else{
resultStr.append(ch);
}
}
return resultStr.toString();
}
private static char toHex(int ch) {
return (char) (ch < 10 ? '0' + ch : 'A' + ch - 10);
}
private static boolean isUnsafe(char ch) {
if (ch > 128 || ch < 0)
return true;
return " %$&+,/:;=?#<>#%".indexOf(ch) >= 0;
}
Hello+World is how a browser will encode form data (application/x-www-form-urlencoded) for a GET request and this is the generally accepted form for the query part of a URI.
http://host/path/?message=Hello+World
If you sent this request to a Java servlet, the servlet would correctly decode the parameter value. Usually the only time there are issues here is if the encoding doesn't match.
Strictly speaking, there is no requirement in the HTTP or URI specs that the query part to be encoded using application/x-www-form-urlencoded key-value pairs; the query part just needs to be in the form the web server accepts. In practice, this is unlikely to be an issue.
It would generally be incorrect to use this encoding for other parts of the URI (the path for example). In that case, you should use the encoding scheme as described in RFC 3986.
http://host/Hello%20World
More here.
If you want to encode URI path components, you can also use standard JDK functions, e.g.
public static String encodeURLPathComponent(String path) {
try {
return new URI(null, null, path, null).toASCIIString();
} catch (URISyntaxException e) {
// do some error handling
}
return "";
}
The URI class can also be used to encode different parts of or whole URIs.
Just been struggling with this too on Android, managed to stumble upon Uri.encode(String, String) while specific to android (android.net.Uri) might be useful to some.
static String encode(String s, String allow)
https://developer.android.com/reference/android/net/Uri.html#encode(java.lang.String, java.lang.String)
The other answers either present a manual string replacement, URLEncoder which actually encodes for HTML format, Apache's abandoned URIUtil, or using Guava's UrlEscapers. The last one is fine, except it doesn't provide a decoder.
Apache Commons Lang provides the URLCodec, which encodes and decodes according to URL format rfc3986.
String encoded = new URLCodec().encode(str);
String decoded = new URLCodec().decode(str);
If you are already using Spring, you can also opt to use its UriUtils class as well.
Although quite old, nevertheless a quick response:
Spring provides UriUtils - with this you can specify how to encoded and which part is it related from an URI, e.g.
encodePathSegment
encodePort
encodeFragment
encodeUriVariables
....
I use them cause we already using Spring, i.e. no additonal library is required!
If you are using jetty then org.eclipse.jetty.util.URIUtil will solve the issue.
String encoded_string = URIUtil.encodePath(not_encoded_string).toString();
This worked for me
org.apache.catalina.util.URLEncoder ul = new org.apache.catalina.util.URLEncoder().encode("MY URL");
It's not one-liner, but you can use:
URL url = new URL("https://some-host.net/dav/files/selling_Rosetta Stone Case Study.png.aes");
URI uri = new URI(url.getProtocol(), url.getUserInfo(), url.getHost(), url.getPort(), url.getPath(), url.getQuery(), url.getRef());
System.out.println(uri.toString());
This will give you an output:
https://some-host.net/dav/files/selling_Rosetta%20Stone%20Case%20Study.png.aes
"+" is correct. If you really need %20, then replace the Plusses yourself afterwards.
Warning: This answer is heavily disputed (+8 vs. -6), so take this with a grain of salt.
I was already using Feign so UriUtils was available to me but Spring UrlUtils was not.
<!-- https://mvnrepository.com/artifact/io.github.openfeign/feign-core -->
<dependency>
<groupId>io.github.openfeign</groupId>
<artifactId>feign-core</artifactId>
<version>11.8</version>
</dependency>
My Feign test code:
import feign.template.UriUtils;
System.out.println(UriUtils.encode("Hello World"));
Outputs:
Hello%20World
As the class suggests, it encodes URIs and not URLs but the OP asked about URIs and not URLs.
System.out.println(UriUtils.encode("https://some-host.net/dav/files/selling_Rosetta Stone Case Study.png.aes"));
Outputs:
https%3A%2F%2Fsome-host.net%2Fdav%2Ffiles%2Fselling_Rosetta%20Stone%20Case%20Study.png.aes
Try below approach:
Add a new dependency
<!-- https://mvnrepository.com/artifact/org.apache.tomcat/tomcat-catalina -->
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>tomcat-catalina</artifactId>
<version>10.0.13</version>
</dependency>
Now do as follows:
String str = "Hello+World"; // For "Hello World", decoder is not required
// import java.net.URLDecoder;
String newURL = URLDecoder.decode(str, StandardCharsets.UTF_8);
// import org.apache.catalina.util.URLEncoder;
System.out.println(URLEncoder.DEFAULT.encode(newURL, StandardCharsets.UTF_8));
You'll get the output as:
Hello%20World
Check out the java.net.URI class.
USE MyUrlEncode.URLencoding(String url , String enc) to handle the problem
public class MyUrlEncode {
static BitSet dontNeedEncoding = null;
static final int caseDiff = ('a' - 'A');
static {
dontNeedEncoding = new BitSet(256);
int i;
for (i = 'a'; i <= 'z'; i++) {
dontNeedEncoding.set(i);
}
for (i = 'A'; i <= 'Z'; i++) {
dontNeedEncoding.set(i);
}
for (i = '0'; i <= '9'; i++) {
dontNeedEncoding.set(i);
}
dontNeedEncoding.set('-');
dontNeedEncoding.set('_');
dontNeedEncoding.set('.');
dontNeedEncoding.set('*');
dontNeedEncoding.set('&');
dontNeedEncoding.set('=');
}
public static String char2Unicode(char c) {
if(dontNeedEncoding.get(c)) {
return String.valueOf(c);
}
StringBuffer resultBuffer = new StringBuffer();
resultBuffer.append("%");
char ch = Character.forDigit((c >> 4) & 0xF, 16);
if (Character.isLetter(ch)) {
ch -= caseDiff;
}
resultBuffer.append(ch);
ch = Character.forDigit(c & 0xF, 16);
if (Character.isLetter(ch)) {
ch -= caseDiff;
}
resultBuffer.append(ch);
return resultBuffer.toString();
}
private static String URLEncoding(String url,String enc) throws UnsupportedEncodingException {
StringBuffer stringBuffer = new StringBuffer();
if(!dontNeedEncoding.get('/')) {
dontNeedEncoding.set('/');
}
if(!dontNeedEncoding.get(':')) {
dontNeedEncoding.set(':');
}
byte [] buff = url.getBytes(enc);
for (int i = 0; i < buff.length; i++) {
stringBuffer.append(char2Unicode((char)buff[i]));
}
return stringBuffer.toString();
}
private static String URIEncoding(String uri , String enc) throws UnsupportedEncodingException { //对请求参数进行编码
StringBuffer stringBuffer = new StringBuffer();
if(dontNeedEncoding.get('/')) {
dontNeedEncoding.clear('/');
}
if(dontNeedEncoding.get(':')) {
dontNeedEncoding.clear(':');
}
byte [] buff = uri.getBytes(enc);
for (int i = 0; i < buff.length; i++) {
stringBuffer.append(char2Unicode((char)buff[i]));
}
return stringBuffer.toString();
}
public static String URLencoding(String url , String enc) throws UnsupportedEncodingException {
int index = url.indexOf('?');
StringBuffer result = new StringBuffer();
if(index == -1) {
result.append(URLEncoding(url, enc));
}else {
result.append(URLEncoding(url.substring(0 , index),enc));
result.append("?");
result.append(URIEncoding(url.substring(index+1),enc));
}
return result.toString();
}
}
Am I using the wrong method? What is the correct method I should be using?
Yes, this method java.net.URLEncoder.encode wasn't made for converting " " to "20%" according to spec (source).
The space character " " is converted into a plus sign "+".
Even this is not the correct method, you can modify this to: System.out.println(java.net.URLEncoder.encode("Hello World", "UTF-8").replaceAll("\\+", "%20"));have a nice day =).
use character-set "ISO-8859-1" for URLEncoder