We Keep Coding

Camel-bindy : How to escape double quotes when marshalling CSV?

I am trying to escape double quotes inside a CSV field content. This is required because the double quote character is already used to enclose the fields.
But I don't see how to do that (escaping) with Camel/Camel-bindy.
What I want to produce (note how double quotes inside field content are escaped by doubling them):
"Some";"people";"Never ""finish""";"their"
What I'm actually producing (this won't parse as CSV):
"Some";"people";"Never "finish"";"their"
So,
Is there any option that I can add to tell Camel to escape quotes inside (all) CSV fields values?
Otherwise, is there an alternative solution that i could use to get the same result?
So far, following is the state of what i did :
My Camel version is 2.15.
I use a POJO which is later marshalled to CSV by bindy.
This is how the POJO looks like
#CsvRecord(separator = ";", crlf = "UNIX", generateHeaderColumns = true, quote = "\"", quoting = true)
public class MyCsvPOJO
{
#DataField(pos = 1)
private String prop1 = "";
// Some other properties + getters + setters
}
This is the camel route code that produces the CSV file (using Camel Java DSL) :
from("myRouteId")
.beanRef("myPojoProducerBean")
.marshal()
.bindy(BindyType.Csv, MyCsvPOJO.class)
.convertBodyTo(String.class, "UTF-8")
.to("/path/to/the/ouput-file.csv");
I considered using a custom crafted https://camel.apache.org/maven/camel-2.15.0/camel-csv/apidocs/org/apache/camel/dataformat/csv/CsvDataFormat.html that I could feed to .marshal(myCustomCsvDataFormat), but then I can no longer chain a call to bindy(...) because of incompatible return types.
So, at this point I'm stuck and any hint will be very much appreciated.
Thanks.

In version 2.19.0 was introduced quotingEscaped to #CsvRecord:
Indicate if the values must be escaped when quoting
Source (CAMEL-7519)
UPDATE:
I was able to get in output file your desired result ("Some";"people";"Never ""finish""";"their") using Camel CSV:
from("myRouteId")
.process(exchange -> {
ObjectMapper mapper = new ObjectMapper();
Map<String, String> map = mapper.convertValue(new SimplePojo(), Map.class);
exchange.getIn().setBody(map);
})
.marshal(new CsvDataFormat().setDelimiter(';').setQuoteMode(QuoteMode.ALL))
.to("file:out/?fileName=ouput-file.csv");
Where SimplePojo content is:
private String name = "Some";
private String people = "people";
private String never = "Never \"finish\"";
private String finish = "their"

JAXB: Get Tag as String

This question may have been answered before in some dark recess of the Interwebs, but I couldn't even figure out how to form a meaningful Google query to search for it.
So: Suppose I have a (simplified) XML document like so:
<root>
<tag1>Value</tag1>
<tag2>Word</tag2>
<tag3>
<something1>Foo</something1>
<something2>Bar</something2>
<something3>Baz</something3>
</tag3>
</root>
I know how to use JAXB to unmarshal this into a Java Object in the standard use cases.
What I don't know how to do is unmarshal tag3's contents wholesale into a String. By which I mean:
<something1>Foo</something1>
<something2>Bar</something2>
<something3>Baz</something3>
as a String, tags and all.

Use annotation #XmlAnyElement.
I've been looking for the same solution and I expected to find some annotation that prevents parsing dom and live it as it is, but did not find it.
Detail at:
Using JAXB to extract inner text of XML element
and
http://blog.bdoughan.com/2011/04/xmlanyelement-and-non-dom-properties.html
I added one cheking in method getElement(), otherwise we could get IndexOutOfBoundsException
if (xml.indexOf(START_TAG) < 0) {
return "";
}
For me it's quite strange behavior with this solution. method getElement() is called for every tag of your xml. The first call is for "Value", the second - "ValueWord", etc. It appends the next tag for previous
update:
I noticed that this approach works only for ONE occurence of tag that we want to parse to String. It's impossible to parse correctly the followint example:
<root>
<parent1>
<tag1>Value</tag1>
<tag2>Word</tag2>
<tag3>
<something1>Foo</something1>
<something2>Bar</something2>
<something3>Baz</something3>
</tag3>
</parent1>
<parent2>
<tag1>Value</tag1>
<tag2>Word</tag2>
<tag3>
<something1>TheSecondFoo</something1>
<something2>TheSecondBar</something2>
<something3>TheSecondBaz</something3>
</tag3>
</parent2>
"tag3" with parent tag "parent2" will contain parameters from the first tag (Foo, Bar, Baz) instead of (TheSecondFoo, TheSecondBar, TheSecondBaz)
Any suggestions are appreciated.
Thanks.

I have an utility method that might come in handy for you in that case. See if it helps. I made a sample code with your example:
public static void main(String[] args){
String text= "<root><tag1>Value</tag1><tag2>Word</tag2><tag3><something1>Foo</something1><something2>Bar</something2><something3>Baz</something3></tag3></root>";
System.out.println(extractTag(text, "<tag3>"));
}
public static String extractTag(String xml, String tag) {
String value = "";
String endTag = "</" + tag.substring(1);
Pattern p = Pattern.compile(tag + "(.*?)" + endTag);
Matcher m = p.matcher(xml);
if (m.find()) {
value = m.group(1);
}
return value;
}

How to convert arbitrary JSON into a usable structure in Java

I'm trying to use gson to convert this returned JSON into some kind of
data structure such that I can extract useful data.
For Example:
http://search.twitter.com/search.json?q=test&rpp=1
Returns:
{
"completed_in":0.028,
"max_id":196386333906837504,
"max_id_str":"196386333906837504",
"next_page":"?page=2&max_id=196386333906837504&q=test&rpp=1",
"page":1,
"query":"test",
"refresh_url":"?since_id=196386333906837504&q=test",
"results":[
{
"created_at":"Sat, 28 Apr 2012 23:52:05 +0000",
"from_user":"della_ky",
"from_user_id":525641596,
"from_user_id_str":"525641596",
"from_user_name":"kydella modeste",
"geo":null,
"id":196386333906837504,
"id_str":"196386333906837504",
"iso_language_code":"en",
"metadata":{
"result_type":"recent"
},
"profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/2159990525\/webcam-toy-photo3_20_2__normal.jpg",
"profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/2159990525\/webcam-toy-photo3_20_2__normal.jpg",
"source":"<a href="http:\/\/mobile.twitter.com" rel="nofollow">Mobile Web<\/a>",
"text":"RT #Y__U__NOOO: #SongsIKnowOffByHeart ALL SONGS I LISTEN TO. BRAIN, Y U NO REMEMBER TEST ANSWERS LIKE THAT?!?",
"to_user":null,
"to_user_id":null,
"to_user_id_str":null,
"to_user_name":null
}
],
"results_per_page":1,
"since_id":0,
"since_id_str":"0"
}
Ultimately, I would like to be able to output a list of tweets with the
name of the sender and the date/time of the tweet.
I have read through the gson documentation but it's going over my head
to be honest - lots of new concepts there for me.
Do I need to define a class which maps exactly to the structure of the
JSON in order to then populate an instance of that class? If so this
seems very inflexible/laborious. Ideally I'm looking for something
which will handle JSON in any form and give me a structure I can use
automatically...
Is anyone able to give me some pointers? Being new to this - the more
detailed and in words of the fewest syllables the better!
Update - Thanks to the responses I've already had on this I've had a go at putting a class together to capture the twitter JSON. However, since the JSON has an embedded ArrayList of Objects I'm struggling a bit... So far I have
public class tweetData {
private double completed_in;
private long max_id;
private long max_id_str;
private String next_page;
private int page;
private String query;
private String refresh_url;
private List<tweetDetails> tweets = new ArrayList<tweetDetails>();
}
and
public class tweetDetails {
private String created_at;
private String from_user;
private long from_user_id;
private long from_user_id_str;
private String from_user_name;
private String geo;
private long id;
private long id_str;
private String iso_language_code;
// "metadata":
// {
// "result_type":"recent"
// },
private String profile_image_url;
private String profile_image_url_https;
private String source;
private String text;
private String to_user;
private String to_user_id;
private String to_user_id_str;
private String to_user_name;
}
Which I'm instantiating with
URI uri = new URI("http", "search.twitter.com", "/search.json", "q="+ searchTerms + "&rrp=" + RRP, null);
URL twitterSearch = uri.toURL();
URLConnection yc = twitterSearch.openConnection();
JsonReader reader = new JsonReader(new InputStreamReader(yc.getInputStream()));
Gson gson = new Gson();
tweetData data = gson.fromJson(reader, tweetData.class);
System.out.println(data);
The basic name:values are being populated correctly but the ArrayList is not.
tweetData : 0.17196614959919140865196614959919140865?page=2&max_id=196614959919140865&q=test1test?since_id=196614959919140865&q=testSIZE 0[]
So, I'm still struggling a bit - any more tips hugely appreciated!
Tia,
Tom

Do I need to define a class which maps exactly to the structure of the JSON in order to then populate an instance of that class? If so this seems very inflexible/laborious.
Yes. GSON is a library that can be used to convert Java Objects into their JSON representation. It can also be used to convert a JSON string to an equivalent Java object. This is really powerful because you can automagically instantiate your Java objects from the JSON representation. Assuming your JSON doesn't change its structure, you only have to define the appropriate Java object representation once.
Ideally I'm looking for something which will handle JSON in any form and give me a structure I can use automatically...
However, if you don't want automagical serialisation/deserialisation, then try looking at a simpler library such as java.net/projects/jsonp.
You can extract stuff from it just by querying the keys:
final JSONObject json = new JSONObject(theJsonString);
final String id = json.getString("max_id");
final JSONArray results = json.getJSONArray("results");
final String user = results.getJSONObject(2).getString("from_user");

Gson actually does all the serialization for you. So yes, you would have to write the classes yourself. To you, this seams inflexible and laborious, but that's only because that library isn't made for what you're asking for (it doesn't parse 'arbitrary' JSON).
I would suggest at least considering writing the classes and using gson. The reason I say that is because either way your application's logic will have to expect a very specific format, and writing out that format in a Java class will make things tidier. Here's a nice guide that will help you get started that way.
If you want to simply decode the JSON without serializing it into a Java class (IMHO the only way to use 'arbitrary' JSON), you'll want to use another library. Try this one. It allows you to decode the JSON, and use it by getting values from it (as described in this question: Convert a JSON string to object in Java ME?).

There are some tools that do gson to schema mapping. You give some sample JSON responses, and the java classes to access them are created for you.
http://www.jsonschema2pojo.org/

Gson is a slick beast!
Or at least it became so over the years that have passed since the question had been asked.
You can pass it an Object.class as a second parameter to the fromJson() method and it will parse your Json into a reasonable structure of LinkedTreeMaps and ArrayLists.
Object result = (new Gson()).fromJson(jsonString, Object.class)
More than that, you can really do partial parsing and leave loose ends at any level of your object structure by defining a certain field as Object!
Gson will then parse Json into your structure and your field of type Object will contain the above mentioned structure of LinkedTreeMaps and ArrayLists.
E.g., you may define a class
Person {
String name;
Object details;
}
(Imagine, you care mostly about the person's name but may want the details also somewhere. To log them, for instance.)
Then you can pass the following Json to the fromJson(input, Person.class) method as a first parameter
{
"name": "Carlsson",
"details": {
"address": "Stockholm",
"phones": [
"work": "233-322-233-322",
"home": "none"
]
}
}
The result will have the name field filled with "Carlsson" string and details field will contain a LinkedTreeMap with keys "address" and "phones", etc.

known API to write Bean/ResultSet into CSV file

I would like to export a Java Bean or ResultSet(JDBC) into a CSV file through Reflection mechanism.
I have seen this api :
http://opencsv.sourceforge.net/apidocs/au/com/bytecode/opencsv/bean/BeanToCsv.html
but it's not released yet.
Also, it will be fine if we can set some filters to avoid to map some precised fields.
Do you know a known API which owns these features ?

Unless there are some ready-made API:s I would use
Apache commons http://commons.apache.org/lang/api-2.4/org/apache/commons/lang/builder/ReflectionToStringBuilder.html to get a String representation of an JavaBean. By setting your own ToStringStyle it would be possible to create a CSV style String. There are many possible settings for styling of the String, including excluding fields and so on.
And then of course writing it to a file.

You can just write out to a csv file as you would to a normal .txt file by using an outputstream or so.
If you need more advanced excel like stuff I recommend using Apache POI. It has always done the job nice & clean for me.

Adding to Kennets answer:
I implemented two classes: One for the header (if needed) and one for the body (actual data)
HEADER
The header style class needs to extend ToStringStyle
Invoke toString with a single element, e.g. ReflectionToStringBuilder.toString(firstElement, headerStyle)
Constructor:
this.setUseClassName(false);
this.setUseIdentityHashCode(false);
this.setContentStart("");
this.setUseFieldNames(true);
this.setFieldNameValueSeparator("");
this.setContentEnd("\n");
Override Method:
#Override
public void append(StringBuffer buffer, String fieldName, Object value, Boolean fullDetail) {
super.append(buffer, fieldName, "", fullDetail);
}
BODY
The body class needs to extend RecursiveToStringStyle
Invoke toString with an array, e.g. ReflectionToStringBuilder.toString(array, bodyStyle)
Constructor:
this.setUseClassName(false);
this.setUseIdentityHashCode(false);
this.setContentStart("");
this.setUseFieldNames(false);
this.setContentEnd("");
this.setNullText("n.a.");
this.setArrayStart("");
this.setArrayEnd("");
this.setArraySeparator("\n");
Override Method:
#Override
public void append(StringBuffer buffer, String fieldName, Object value, Boolean fullDetail) {
String csvField = Optional.ofNullable(value)
.map(Objects::toString)
.map(this::escapeLineBreak)
.map(this::escapeDoubleQuote)
.map(this::escapeField)
.orElse(null);
super.append(buffer, fieldName, csvField, fullDetail);
}
Formatting Methods:
private String escapeDoubleQuote(final String field) {
return field.replace("\"", "\"\"");
}
private String escapeLineBreak(final String field) {
return field.replaceAll("\\R", " ");
}
private String escapeField(final String field) {
return "\"" + field + "\"";
}

Can't parse JSON property "null"

I faced with one trouble when tried to parse JSON "null" property, please help me to understand what's the real problem. I had a following JSON:
{
"properties" : {
"null" : {
"value" : false
}
}
}
I used http://jsonlint.com to validate that this JSON is valid. I tried to parse it from java:
import net.sf.json.JSONObject;
import java.io.IOException;
public class Test {
public static void main(String[] args) throws IOException {
String st = "{" +
" 'properties' : {" +
" 'null' : {" +
" 'value' : false" +
" }" +
" }" +
"}";
JSONObject.fromObject(st);
}
}
But got the exception:
Exception in thread "main" java.lang.ClassCastException: JSON keys must be strings.
at net.sf.json.JSONObject._fromJSONObject(JSONObject.java:927)
at net.sf.json.JSONObject.fromObject(JSONObject.java:155)
at net.sf.json.JSONSerializer.toJSON(JSONSerializer.java:108)
at net.sf.json.AbstractJSON._processValue(AbstractJSON.java:238)
at net.sf.json.JSONObject._processValue(JSONObject.java:2655)
at net.sf.json.JSONObject.processValue(JSONObject.java:2721)
at net.sf.json.JSONObject.element(JSONObject.java:1786)
at net.sf.json.JSONObject._fromJSONTokener(JSONObject.java:1036)
at net.sf.json.JSONObject._fromString(JSONObject.java:1201)
at net.sf.json.JSONObject.fromObject(JSONObject.java:165)
at net.sf.json.JSONObject.fromObject(JSONObject.java:134)
I used json-lib-2.4-jdk15.jar from http://json-lib.sourceforge.net to parse it. Could anybody please clarify this? Why this library throws exception, but online validator said that it's valid JSON? It is a bug in the library or I made something wrong?

JSON-lib initially parses and populates a Java Map with the input JSON. Unfortunately, JSON-lib then checks whether every JSON object element name is a JSON null. It's null check is performed in the JSONNull.equals(Object) method. This method returns true for a "null" JSON string, which of course is not actually a JSON null value.
I recommend filing a bug with the JSON-lib project for this issue. The implementation of JSONNull.equals(Object) is flawed.
Unfortunately, it's not possible to handle this with a custom PropertyNameProcessor.
Options available for a more immediate solution include altering the JSON-lib code yourself, or switching libraries.
If you can switch libraries, I highly recommend Jackson. Following is an example of using it to deserialize the example JSON in the original question.
/*
{
"properties" : {
"null" : {
"value" : false
}
}
}
*/
String json = "{\"properties\":{\"null\":{\"value\":false}}}";
ObjectMapper mapper = new ObjectMapper();
Map<String, Object> map = mapper.readValue(json, Map.class);
System.out.println(map);
// output: {properties={null={value=false}}}
Map<String, Object> propertiesMap = (Map) map.get("properties");
System.out.println(propertiesMap);
// output: {null={value=false}}
Map<String, Object> nullMap = (Map) propertiesMap.get("null");
System.out.println(nullMap);
// output: {value=false}

The first JSON posted is valid JSON: the JSON in the Java, however, is not valid -- only " is valid for the [required] key quote. From json.org:
A string is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes....
However, that sounds like a bug, assuming it was not triggered by the invalid JSON fed to it (the library can do whatever it wants with invalid JSON)... one would have to look at the source (or bug reports / user experience) to say conclusively if this is indeed a "bug". I have added some suggestions of things to try below which may either show expected behavior or outline the cause/issue in further detail.
Consider this minimal test-case (with valid JSON):
String st = "{ \"null\": \"hello world!\" }";
This may also shed more light, depending on if the first item is "null" or null when extracted:
String st = "[ \"null\" ]";
Happy coding.

The gson library link is:
http://code.google.com/p/google-gson/
I normally usr gson to generate the josn string,so I found some example someone else posted in stackoverflow to parse json string with gson,see the link:
Converting JSON to Java

suggest you to use Gson,
and construct the json string using java Map and List,
then use Gson to output the Map or List object