We Keep Coding

How do I make the toString() of the JSONObject encode the UTF-8 characters to unicode like in json_encode of PHP?

In Java (using for Android), here are the code lines:
org.json.JSONObject array = new org.json.JSONObject("{\"value\":\"aész\"");
System.out.println("RES: " + array.toString());
The output I want:
RES: {"value":"a\u00e9sz"}
When it is actually:
RES: {"value":"aész"}
How do I make the JSONObject toString() method return the JSON String encoded with unicode values in the UTF-8 special characters, like the json_encode(array("value", "aész"));
Thanks in advance.

You need to build your own version of the org.json.JSONObject class if you want it to escape all non-ASCII characters.
The signature of the method you need to modify is
public static Writer quote(String string, Writer w) throws IOException
it is declared inside JSONObject.java. It is the method responsible of formatting all string values inside the produced json strings. it loops over all the characters of the source string and emits the corresponding output characters.
What you are looking for is in the "default" section of the switch statement.
the original code (at least in the sources I am watching right now) looks like this:
default:
if (c < ' ' || (c >= '\u0080' && c < '\u00a0')
|| (c >= '\u2000' && c < '\u2100')) {
w.write("\\u");
hhhh = Integer.toHexString(c);
w.write("0000", 0, 4 - hhhh.length());
w.write(hhhh);
} else {
w.write(c);
}
you need to change the "if" test to match all the characters you want to be escaped.
this does what you want:
default:
if (c < ' ' || c >= '\u0080') {
w.write("\\u");
hhhh = Integer.toHexString(c);
w.write("0000", 0, 4 - hhhh.length());
w.write(hhhh);
} else {
w.write(c);
}
Hope this helps.
P.S: I run into your question because I met your same problem: the json strings I am generating need to travel through a system that accepts only ascii characters and mangles any character >127.

It sounds more like jar issue. JsonObject is the class used across various open source libraries. Download this jar json-rpc-1.0.jar
Try this:
JSONObject json = new JSONObject();
json.put("value", "aész");
System.out.println(json.toString());
produces :
{"Name":"u00e9sz"}

I suppose what you want is ISO-8859-1 encoding. Here is a good answer on that subject:
Converting UTF-8 to ISO-8859-1 in Java - how to keep it as single byte

regular expression replace 2 characters with one

i would like to use a regular expression for the following problem:
SOME_RANDOM_TEXT
should be converted to:
someRandomText
so, the _(any char) should be replaced with just the letter in upper case. i found something like that, using the tool:
_\w and $&
how to get only the second letter from the replacement?? any advice? thanks.

It might be easier simply to String.split("_") and then rejoin, capitalising the first letter of each string in your collection.
Note that Apache Commons has lots of useful string-related stuff, including a join() method.

The problem is that the case conversion from lowercase to uppercase is not supported by Java.util.regex.Pattern
This means you will need to do the conversion programmatically as Brian suggested. See also this thread

You can also write a simple method to do this. It's more complicated but more optimized :
public static String toCamelCase(String value) {
value = value.toLowerCase();
byte[] source = value.getBytes();
int maxLen = source.length;
byte[] target = new byte[maxLen];
int targetIndex = 0;
for (int sourceIndex = 0; sourceIndex < maxLen; sourceIndex++) {
byte c = source[sourceIndex];
if (c == '_') {
if (sourceIndex < maxLen - 1)
source[sourceIndex + 1] = (byte) Character.toUpperCase(source[sourceIndex + 1]);
continue;
}
target[targetIndex++] = source[sourceIndex];
}
return new String(target, 0, targetIndex);
}
I like Apache commons libraries, but sometimes it's good to know how it works and be able to write some specific code for jobs like this.

Java Regex or XML parser?

I want to remove any tags such as
<p>hello <namespace:tag : a>hello</namespace:tag></p>
to become
<p> hello hello </p>
What is the best way to do this if it is regex for some reason this is now working can anyone help?
(<|</)[:]{1,2}[^</>]>
edit:
added

Definitely use an XML parser. Regex should not be used to parse *ML

You should not use regex for these purposes use a parser like lxml or BeautifulSoup
>>> import lxml.html as lxht
>>> myString = '<p>hello <namespace:tag : a>hello</namespace:tag></p>'
>>> lxht.fromstring(myString).text_content()
'hello hello'
Here is a reason why you should not parse html/xml with regex.

If you're just trying to pull the plain text out of some simple XML, the best (fastest, smallest memory footprint) would be to just run a for loop over the data:
PSEUDOCODE BELOW
bool inMarkup = false;
string text = "";
for each character in data // (dunno what you're reading from)
{
char c = current;
if( c == '<' ) inMarkup = true;
else if( c == '>') inMarkup = false;
else if( !inMarkup ) text += c;
}
Note: This will break if you encounter things like CDATA, JavaScript, or CSS in your parsing.
So, to sum up... if it's simple, do something like above and not a regular expression. If it isn't that simple, listen to the other guys an use an advanced parser.

This is a solution I personally used for a likewise problem in java. The library used for this is Jsoup : http://jsoup.org/.
In my particular case I had to unwrap tags that had an attribute with a particular value in them. You see that reflected in this code, it's not the exact solution to this problem but could put you on your way.
public static String unWrapTag(String html, String tagName, String attribute, String matchRegEx) {
Validate.notNull(html, "html must be non null");
Validate.isTrue(StringUtils.isNotBlank(tagName), "tagName must be non blank");
if (StringUtils.isNotBlank(attribute)) {
Validate.notNull(matchRegEx, "matchRegEx must be non null when an attribute is provided");
}
Document doc = Jsoup.parse(html);
OutputSettings outputSettings = doc.outputSettings();
outputSettings.prettyPrint(false);
Elements elements = doc.getElementsByTag(tagName);
for (Element element : elements) {
if(StringUtils.isBlank(attribute)){
element.unwrap();
}else{
String attr = element.attr(attribute);
if(!StringUtils.isBlank(attr)){
String newData = attr.replaceAll(matchRegEx, "");
if(StringUtils.isBlank(newData)){
element.unwrap();
}
}
}
}
return doc.html();
}

Can this regex be further optimized?

I wrote this regex to parse entries from srt files.
(?s)^\d++\s{1,2}(.{12}) --> (.{12})\s{1,2}(.+)\r?$
I don't know if it matters, but this is done using Scala programming language (Java Engine, but literal strings so that I don't have to double the backslashes).
The s{1,2} is used because some files will only have line breaks \n and others will have line breaks and carriage returns \n\r
The first (?s) enables DOTALL mode so that the third capturing group can also match line breaks.
My program basically breaks a srt file using \n\r?\n as a delimiter and use Scala nice pattern matching feature to read each entry for further processing:
val EntryRegex = """(?s)^\d++\s{1,2}(.{12}) --> (.{12})\s{1,2}(.+)\r?$""".r
def apply(string: String): Entry = string match {
case EntryRegex(start, end, text) => Entry(0, timeFormat.parse(start),
timeFormat.parse(end), text);
}
Sample entries:
One line:
1073
01:46:43,024 --> 01:46:45,015
I am your father.
Two Lines:
160
00:20:16,400 --> 00:20:19,312
<i>Help me, Obi-Wan Kenobi.
You're my only hope.</i>
The thing is, the profiler shows me that this parsing method is by far the most time consuming operation in my application (which does intensive time math and can even reencode the file several times faster than what it takes to read and parse the entries).
So any regex wizards can help me optimize it? Or maybe I should sacrifice regex / pattern matching succinctness and try an old school java.util.Scanner approach?
Cheers,

(?s)^\d++\s{1,2}(.{12}) --> (.{12})\s{1,2}(.+)\r?$
In Java, $ means the end of input or the beginning of a line-break immediately preceding the end of input. \z means unambiguously end of input, so if that is also the semantics in Scala, then \r?$ is redundant and $ would do just as well. If you really only want a CR at the end and not CRLF then \r?\z might be better.
The (?s) should also make (.+)\r? redundant since the + is greedy, the . should always expand to include the \r. If you do not want the \r included in that third capturing group, then make the match lazy : (.+?) instead of (.+).
Maybe
(?s)^\d++\s\s?(.{12}) --> (.{12})\s\s?(.+?)\r?\z
Other fine high-performance alternatives to regular expressions that will run inside a JVM &| CLR include JavaCC and ANTLR. For a Scala only solution, see http://jim-mcbeath.blogspot.com/2008/09/scala-parser-combinators.html

I'm not optimistic, but here are two things to try:
you could do is move the (?s) to just before you need it.
remove the \r?$ and use a greedy .++ for the text .+
^\d++\s{1,2}(.{12}) --> (.{12})\s{1,2}(?s)(.++)$
To really get good performance, I would refactor the code and regex to use findAllIn. The current code is doing a regex for every Entry in your file. I imagine the single findAllIn regex would perform better...But maybe not...

Check this out:
(?m)^\d++\r?+\n(.{12}) --> (.{12})\r?+\n(.++(?>\r?+\n.++)*+)$
This regex matches a complete .srt file entry in place. You don't have to split the contents up on line breaks first; that's a huge waste of resources.
The regex takes advantage of the fact that there's exactly one line separator (\n or \r\n) separating the lines within an entry (multiple line separators are used to separate entries from each other). Using \r?+\n instead of \s{1,2} means you can never accidentally match two line separators (\n\n) when you only wanted to match one.
This way, too, you don't have to rely on the . in (?s) mode. #Jacob was right about that: it's not really helping you, and it's killing your performance. But (?m) mode is helpful, for correctness as well as performance.
You mentioned java.util.Scanner; this regex would work very nicely with findWithinHorizon(0). But I'd be surprised if Scala doesn't offer a nice, idiomatic way to use it as well.

I wouldn't use java.util.Scanner or even strings. Everything you're doing will work perfectly on a byte stream as long as you can assume UTF-8 encoding of your files (or a lack of unicode). You should be able to speed things up by at least 5x.
Edit: this is just a lot of low-level fiddling of bytes and indices. Here's something based loosely on things I've done before, which seems about 2x-5x faster, depending on file size, caching, etc.. I'm not doing the date parsing here, just returning strings, and I'm assuming the files are small enough to fit in a single block of memory (i.e. <2G). This is being rather pedantically careful; if you know, for example, that the date string format is always okay, then the parsing can be faster yet (just count the characters after the first line of digits).
import java.io._
abstract class Entry {
def isDefined: Boolean
def date1: String
def date2: String
def text: String
}
case class ValidEntry(date1: String, date2: String, text: String) extends Entry {
def isDefined = true
}
object NoEntry extends Entry {
def isDefined = false
def date1 = ""
def date2 = ""
def text = ""
}
final class Seeker(f: File) {
private val buffer = {
val buf = new Array[Byte](f.length.toInt)
val fis = new FileInputStream(f)
fis.read(buf)
fis.close()
buf
}
private var i = 0
private var d1,d2 = 0
private var txt,n = 0
def isDig(b: Byte) = ('0':Byte) <= b && ('9':Byte) >= b
def nextNL() {
while (i < buffer.length && buffer(i) != '\n') i += 1
i += 1
if (i < buffer.length && buffer(i) == '\r') i += 1
}
def digits() = {
val zero = i
while (i < buffer.length && isDig(buffer(i))) i += 1
if (i==zero || i >= buffer.length || buffer(i) != '\n') {
nextNL()
false
}
else {
nextNL()
true
}
}
def dates(): Boolean = {
if (i+30 >= buffer.length) {
i = buffer.length
false
}
else {
d1 = i
while (i < d1+12 && buffer(i) != '\n') i += 1
if (i < d1+12 || buffer(i)!=' ' || buffer(i+1)!='-' || buffer(i+2)!='-' || buffer(i+3)!='>' || buffer(i+4)!=' ') {
nextNL()
false
}
else {
i += 5
d2 = i
while (i < d2+12 && buffer(i) != '\n') i += 1
if (i < d2+12 || buffer(i) != '\n') {
nextNL()
false
}
else {
nextNL()
true
}
}
}
}
def gatherText() {
txt = i
while (i < buffer.length && buffer(i) != '\n') {
i += 1
nextNL()
}
n = i-txt
nextNL()
}
def getNext: Entry = {
while (i < buffer.length) {
if (digits()) {
if (dates()) {
gatherText()
return ValidEntry(new String(buffer,d1,12), new String(buffer,d2,12), new String(buffer,txt,n))
}
}
}
return NoEntry
}
}
Now that you see that, aren't you glad that the regex solution was so quick to code?

How should I escape strings in JSON?

When creating JSON data manually, how should I escape string fields? Should I use something like Apache Commons Lang's StringEscapeUtilities.escapeHtml, StringEscapeUtilities.escapeXml, or should I use java.net.URLEncoder?
The problem is that when I use SEU.escapeHtml, it doesn't escape quotes and when I wrap the whole string in a pair of 's, a malformed JSON will be generated.

Ideally, find a JSON library in your language that you can feed some appropriate data structure to, and let it worry about how to escape things. It'll keep you much saner. If for whatever reason you don't have a library in your language, you don't want to use one (I wouldn't suggest this¹), or you're writing a JSON library, read on.
Escape it according to the RFC. JSON is pretty liberal: The only characters you must escape are \, ", and control codes (anything less than U+0020).
This structure of escaping is specific to JSON. You'll need a JSON specific function. All of the escapes can be written as \uXXXX where XXXX is the UTF-16 code unit¹ for that character. There are a few shortcuts, such as \\, which work as well. (And they result in a smaller and clearer output.)
For full details, see the RFC.
¹JSON's escaping is built on JS, so it uses \uXXXX, where XXXX is a UTF-16 code unit. For code points outside the BMP, this means encoding surrogate pairs, which can get a bit hairy. (Or, you can just output the character directly, since JSON's encoded for is Unicode text, and allows these particular characters.)

Extract From Jettison:
public static String quote(String string) {
if (string == null || string.length() == 0) {
return "\"\"";
}
char c = 0;
int i;
int len = string.length();
StringBuilder sb = new StringBuilder(len + 4);
String t;
sb.append('"');
for (i = 0; i < len; i += 1) {
c = string.charAt(i);
switch (c) {
case '\\':
case '"':
sb.append('\\');
sb.append(c);
break;
case '/':
// if (b == '<') {
sb.append('\\');
// }
sb.append(c);
break;
case '\b':
sb.append("\\b");
break;
case '\t':
sb.append("\\t");
break;
case '\n':
sb.append("\\n");
break;
case '\f':
sb.append("\\f");
break;
case '\r':
sb.append("\\r");
break;
default:
if (c < ' ') {
t = "000" + Integer.toHexString(c);
sb.append("\\u" + t.substring(t.length() - 4));
} else {
sb.append(c);
}
}
}
sb.append('"');
return sb.toString();
}

Try this org.codehaus.jettison.json.JSONObject.quote("your string").
Download it here: http://mvnrepository.com/artifact/org.codehaus.jettison/jettison

org.json.simple.JSONObject.escape() escapes quotes,\, /, \r, \n, \b, \f, \t and other control characters. It can be used to escape JavaScript codes.
import org.json.simple.JSONObject;
String test = JSONObject.escape("your string");

There is now a StringEscapeUtils#escapeJson(String) method in the Apache Commons Text library.
The methods of interest are as follows:
StringEscapeUtils#escapeJson(String)
StringEscapeUtils#unescapeJson(String)
This functionality was initially released as part of Apache Commons Lang version 3.2 but has since been deprecated and moved to Apache Commons Text. So if the method is marked as deprecated in your IDE, you're importing the implementation from the wrong library (both libraries use the same class name: StringEscapeUtils).
The implementation isn't pure Json. As per the Javadoc:
Escapes the characters in a String using Json String rules.
Escapes any values it finds into their Json String form. Deals
correctly with quotes and control-chars (tab, backslash, cr, ff, etc.)
So a tab becomes the characters '\' and 't'.
The only difference between Java strings and Json strings is that in
Json, forward-slash (/) is escaped.
See http://www.ietf.org/rfc/rfc4627.txt for further details.

org.json.JSONObject quote(String data) method does the job
import org.json.JSONObject;
String jsonEncodedString = JSONObject.quote(data);
Extract from the documentation:
Encodes data as a JSON string. This applies quotes and any necessary character escaping. [...] Null will be interpreted as an empty string

StringEscapeUtils.escapeJavaScript / StringEscapeUtils.escapeEcmaScript should do the trick too.

If you are using fastexml jackson, you can use the following:
com.fasterxml.jackson.core.io.JsonStringEncoder.getInstance().quoteAsString(input)
If you are using codehaus jackson, you can use the following:
org.codehaus.jackson.io.JsonStringEncoder.getInstance().quoteAsString(input)

Not sure what you mean by "creating json manually", but you can use something like gson (http://code.google.com/p/google-gson/), and that would transform your HashMap, Array, String, etc, to a JSON value. I recommend going with a framework for this.

I have not spent the time to make 100% certain, but it worked for my inputs enough to be accepted by online JSON validators:
org.apache.velocity.tools.generic.EscapeTool.EscapeTool().java("input")
although it does not look any better than org.codehaus.jettison.json.JSONObject.quote("your string")
I simply use velocity tools in my project already - my "manual JSON" building was within a velocity template

For those who came here looking for a command-line solution, like me, cURL's --data-urlencode works fine:
curl -G -v -s --data-urlencode 'query={"type" : "/music/artist"}' 'https://www.googleapis.com/freebase/v1/mqlread'
sends
GET /freebase/v1/mqlread?query=%7B%22type%22%20%3A%20%22%2Fmusic%2Fartist%22%7D HTTP/1.1
, for example. Larger JSON data can be put in a file and you'd use the # syntax to specify a file to slurp in the to-be-escaped data from. For example, if
$ cat 1.json 
{
  "type": "/music/artist",
  "name": "The Police",
  "album": []
}
you'd use
curl -G -v -s --data-urlencode query#1.json 'https://www.googleapis.com/freebase/v1/mqlread'
And now, this is also a tutorial on how to query Freebase from the command line :-)

Use EscapeUtils class in commons lang API.
EscapeUtils.escapeJavaScript("Your JSON string");

Consider Moshi's JsonWriter class. It has a wonderful API and it reduces copying to a minimum, everything can be nicely streamed to a filed, OutputStream, etc.
OutputStream os = ...;
JsonWriter json = new JsonWriter(Okio.buffer(Okio.sink(os)));
json.beginObject();
json.name("id").value(getId());
json.name("scores");
json.beginArray();
for (Double score : getScores()) {
json.value(score);
}
json.endArray();
json.endObject();
If you want the string in hand:
Buffer b = new Buffer(); // okio.Buffer
JsonWriter writer = new JsonWriter(b);
//...
String jsonString = b.readUtf8();

If you need to escape JSON inside JSON string, use org.json.JSONObject.quote("your json string that needs to be escaped") seem to work well

Apache commons-text now has a
StringEscapeUtils.escapeJson(String).

using the \uXXXX syntax can solve this problem, google UTF-16 with the name of the sign, you can find out XXXX, for example:utf-16 double quote

The methods here that show the actual implementation are all faulty.
I don't have Java code, but just for the record, you could easily convert this C#-code:
Courtesy of the mono-project #
https://github.com/mono/mono/blob/master/mcs/class/System.Web/System.Web/HttpUtility.cs
public static string JavaScriptStringEncode(string value, bool addDoubleQuotes)
{
if (string.IsNullOrEmpty(value))
return addDoubleQuotes ? "\"\"" : string.Empty;
int len = value.Length;
bool needEncode = false;
char c;
for (int i = 0; i < len; i++)
{
c = value[i];
if (c >= 0 && c <= 31 || c == 34 || c == 39 || c == 60 || c == 62 || c == 92)
{
needEncode = true;
break;
}
}
if (!needEncode)
return addDoubleQuotes ? "\"" + value + "\"" : value;
var sb = new System.Text.StringBuilder();
if (addDoubleQuotes)
sb.Append('"');
for (int i = 0; i < len; i++)
{
c = value[i];
if (c >= 0 && c <= 7 || c == 11 || c >= 14 && c <= 31 || c == 39 || c == 60 || c == 62)
sb.AppendFormat("\\u{0:x4}", (int)c);
else switch ((int)c)
{
case 8:
sb.Append("\\b");
break;
case 9:
sb.Append("\\t");
break;
case 10:
sb.Append("\\n");
break;
case 12:
sb.Append("\\f");
break;
case 13:
sb.Append("\\r");
break;
case 34:
sb.Append("\\\"");
break;
case 92:
sb.Append("\\\\");
break;
default:
sb.Append(c);
break;
}
}
if (addDoubleQuotes)
sb.Append('"');
return sb.ToString();
}
This can be compacted into
// https://github.com/mono/mono/blob/master/mcs/class/System.Json/System.Json/JsonValue.cs
public class SimpleJSON
{
private static bool NeedEscape(string src, int i)
{
char c = src[i];
return c < 32 || c == '"' || c == '\\'
// Broken lead surrogate
|| (c >= '\uD800' && c <= '\uDBFF' &&
(i == src.Length - 1 || src[i + 1] < '\uDC00' || src[i + 1] > '\uDFFF'))
// Broken tail surrogate
|| (c >= '\uDC00' && c <= '\uDFFF' &&
(i == 0 || src[i - 1] < '\uD800' || src[i - 1] > '\uDBFF'))
// To produce valid JavaScript
|| c == '\u2028' || c == '\u2029'
// Escape "</" for <script> tags
|| (c == '/' && i > 0 && src[i - 1] == '<');
}
public static string EscapeString(string src)
{
System.Text.StringBuilder sb = new System.Text.StringBuilder();
int start = 0;
for (int i = 0; i < src.Length; i++)
if (NeedEscape(src, i))
{
sb.Append(src, start, i - start);
switch (src[i])
{
case '\b': sb.Append("\\b"); break;
case '\f': sb.Append("\\f"); break;
case '\n': sb.Append("\\n"); break;
case '\r': sb.Append("\\r"); break;
case '\t': sb.Append("\\t"); break;
case '\"': sb.Append("\\\""); break;
case '\\': sb.Append("\\\\"); break;
case '/': sb.Append("\\/"); break;
default:
sb.Append("\\u");
sb.Append(((int)src[i]).ToString("x04"));
break;
}
start = i + 1;
}
sb.Append(src, start, src.Length - start);
return sb.ToString();
}
}

I think the best answer in 2017 is to use the javax.json APIs. Use javax.json.JsonBuilderFactory to create your json objects, then write the objects out using javax.json.JsonWriterFactory. Very nice builder/writer combination.