I'm trying to parse a CSV file using jackson-dataformat-csv and I want to map the numeric column to the Number java type.
CsvSchema schema = CsvSchema.builder().setUseHeader(true)
.addColumn("firstName", CsvSchema.ColumnType.STRING)
.addColumn("lastName", CsvSchema.ColumnType.STRING)
.addColumn("age", CsvSchema.ColumnType.NUMBER)
.build();
CsvMapper csvMapper = new CsvMapper();
MappingIterator<Map<String, Object>> mappingIterator = csvMapper
.readerFor(Map.class)
.with(schema)
.readValues(is);
while (mappingIterator.hasNext()) {
Map<String, Object> entryMap = mappingIterator.next();
Number age = (Number) entryMap.get("age");
}
I'm expecting entryMap.get("age") should be a Number, but I get String instead.
My CSV file:
firstName,lastName,age
John,Doe,21
Error,Name,-10
I know that CsvSchema works fine with POJOs, but I need to process arbitrary CSV schemas, so I can't create a new java class for every case.
Any way to parse CSV into a typed Map or Array?
Right now it is not possible to configure Map deserialisation using CsvSchema. Process uses com.fasterxml.jackson.databind.deser.std.MapDeserializer which right now does not check schema. We could write custom Map deserialiser. There is a question on GitHub: CsvMapper does not respect CsvSchema.ColumnType when using #JsonAnySetter where cowtowncoder answered:
At this point schema type is not used much for anything, but I agree
it should.
EDIT
I decided to take a look closer what we can do with that fact that com.fasterxml.jackson.databind.deser.std.MapDeserializer is used behind the scene. Implementing custom Map deserialiser which will take care about types would be tricky to implement and register but we can use knowledge about ValueInstantiator. Let's define new Map type which knows what to do with ColumnType info:
class CsvMap extends HashMap<String, Object> {
private final CsvSchema schema;
private final NumberFormat numberFormat = NumberFormat.getInstance();
public CsvMap(CsvSchema schema) {
this.schema = schema;
}
#Override
public Object put(String key, Object value) {
value = convertIfNeeded(key, value);
return super.put(key, value);
}
private Object convertIfNeeded(String key, Object value) {
CsvSchema.Column column = schema.column(key);
if (column.getType() == CsvSchema.ColumnType.NUMBER) {
try {
return numberFormat.parse(value.toString());
} catch (ParseException e) {
// leave it as it is
}
}
return value;
}
}
For new type without no-arg constructor we should create new ValueInstantiator:
class CsvMapInstantiator extends ValueInstantiator.Base {
private final CsvSchema schema;
public CsvMapInstantiator(CsvSchema schema) {
super(CsvMap.class);
this.schema = schema;
}
#Override
public Object createUsingDefault(DeserializationContext ctxt) {
return new CsvMap(schema);
}
#Override
public boolean canCreateUsingDefault() {
return true;
}
}
Example usage:
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.MappingIterator;
import com.fasterxml.jackson.databind.ObjectReader;
import com.fasterxml.jackson.databind.deser.ValueInstantiator;
import com.fasterxml.jackson.databind.module.SimpleModule;
import com.fasterxml.jackson.dataformat.csv.CsvMapper;
import com.fasterxml.jackson.dataformat.csv.CsvSchema;
import java.io.File;
import java.io.IOException;
import java.text.NumberFormat;
import java.text.ParseException;
import java.util.HashMap;
public class CsvApp {
public static void main(String[] args) throws IOException {
File csvFile = new File("./resource/test.csv").getAbsoluteFile();
CsvSchema schema = CsvSchema.builder()
.addColumn("firstName", CsvSchema.ColumnType.STRING)
.addColumn("lastName", CsvSchema.ColumnType.STRING)
.addColumn("age", CsvSchema.ColumnType.NUMBER)
.build().withHeader();
// Create schema aware map module
SimpleModule csvMapModule = new SimpleModule();
csvMapModule.addValueInstantiator(CsvMap.class, new CsvMapInstantiator(schema));
// register map
CsvMapper csvMapper = new CsvMapper();
csvMapper.registerModule(csvMapModule);
// get reader for CsvMap + schema
ObjectReader objectReaderWithSchema = csvMapper
.readerWithSchemaFor(CsvMap.class)
.with(schema);
MappingIterator<CsvMap> mappingIterator = objectReaderWithSchema.readValues(csvFile);
while (mappingIterator.hasNext()) {
CsvMap entryMap = mappingIterator.next();
Number age = (Number) entryMap.get("age");
System.out.println(age + " (" + age.getClass() + ")");
}
}
}
Above code for below CSV payload:
firstName,lastName,age
John,Doe,21
Error,Name,-10.1
prints:
21 (class java.lang.Long)
-10.1 (class java.lang.Double)
It looks like a hack but I wanted to show this possibility.
You can use univocity-parsers for this sort of thing. It's faster and way more flexible:
CsvParserSettingssettings = new CsvParserSettings(); //configure the parser if needed
CsvParser parser = new CsvParser(settings);
for (Record record : parser.iterateRecords(is)) {
Short age = record.getShort("age");
}
To get a typed map, tell the parser what is the type of the columns you are working with:
parser.getRecordMetadata().setTypeOfColumns(Short.class, "age" /*, and other column names*/);
//to get 0 instead of nulls when the field is empty in the file:
parser.getRecordMetadata().setDefaultValueOfColumns("0", "age", /*, and other column names*/);
// then parse
for (Record record : parser.iterateRecords(is)) {
Map<String,Object> map = record.toFieldMap();
}
Hope this helps
Disclaimer: I'm the author of this library. It's open source and free (Apache 2.0 license)
Related
I have a JSON file(it contains an array of JSON objects.)
I am trying to read it object by object.
Each object I need to convert it to a LinkedHashMap<String,String> where both the key and value are strings. Note that even if the JSON objects contain a non-string value(Integer/Boolean), I want my LinkedHashMap to contain a string.
This is my JSON file (films.json):
[
{
"name": "Fight Club",
"year": 1999,
}
]
Now, this has 1 object. I want to convert it to a LinkedHashMap<String,String>.
So for the above example, my LinkedHashMap should contain(for the 1st JSON object) :
"name" : "Fight CLub"
"year" : "1999"
Notice how the year is String in the LinkedHashMap and not Integer.
This is what I tried.
Map<String, Object> myLinkedHashMap;
JsonParser jsonParser = new JsonFactory().createParser(new File("films.json"));
jsonParser = new JsonFactory().createParser(new File(filePath));
jsonParser.nextToken();
ObjectMapper mapper = new ObjectMapper();
mapper.registerModule(new JavaTimeModule());
mapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
while(jsonParser.nextToken() != JsonToken.END_ARRAY){
myLinkedHashMap = mapper.readValue(jsonParser, LinkedHashMap.class);
}
The variable myLinkedHashMap will contain a key/value pair for an object in my JSON file.
But the problem is that for 'year' of the JSON file, I am getting Integer in the LinkedHashMap as the JSON file also contains Integer.
Instead, I want the Integer as String in the LinkedHashMap.
Please help me get String in the LinkedHashMap instead of Integer.
Note: The solution should be generic to other data types also.
So if the JSON object contains boolean true, then my LinkedHashMap should contain "true".
You can construct map type using TypeFactory and constructMapType method to tell exactly what do you need from readValue method. See below example:
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonToken;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.type.MapType;
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import org.springframework.util.Assert;
import java.io.File;
import java.util.LinkedHashMap;
public class JsonMapApp {
public static void main(String[] args) throws Exception {
File jsonFile = new File("./resource/test.json").getAbsoluteFile();
ObjectMapper mapper = new ObjectMapper();
mapper.registerModule(new JavaTimeModule());
mapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
JsonParser jsonParser = mapper.getFactory().createParser(jsonFile);
jsonParser.nextToken();
MapType mapType = mapper.getTypeFactory().constructMapType(LinkedHashMap.class, String.class, String.class);
while (jsonParser.nextToken() != JsonToken.END_ARRAY) {
LinkedHashMap<String, String> map = mapper.readValue(jsonParser, mapType);
map.forEach((k, v) -> {
Assert.isInstanceOf(String.class, v);
System.out.println(k + " -> " + v + " (" + v.getClass().getName() + ")");
});
}
}
}
Above code prints:
name -> Fight Club (java.lang.String)
year -> 1999 (java.lang.String)
Change
Map<String, Object> myLinkedHashMap;
to
Map<String, String> myLinkedHashMap;
I'm creating a Spring application on backend and my main goal is to manage properties (add/update/delete) in *.properties file. I want to convert this file to JSON and then manipulate it from UI application.
Is there any possibility to convert structure like this:
a.x=1
a.y=2
b.z=3
To JSON like this:
{
"a": {
"x": 1,
"y": 2
},
"b": {
"z": 3
}
}
I found solution to use GSON library, but it creates for me flat structure, not hierarchical, code I used:
Properties props = new Properties();
try (FileInputStream in = new FileInputStream(classPathResource.getFile())) {
props.load(in);
}
String json = new GsonBuilder().enableComplexMapKeySerialization().create().toJson(props);
Is here someone who was facing same problem and maybe found a working project for this? Maybe GSON library can do that?
This solution does involve loads of work, but you will get what you want to achieve using the below code, basically, the idea is to split the key based on the single dot and then create a JsonObject if the same first key is found.
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.util.Iterator;
import java.util.Map.Entry;
import java.util.Properties;
import org.json.JSONObject;
import com.fasterxml.jackson.annotation.JsonIgnore;
public class SOTest {
public static void main(String[] args) throws IOException {
JSONObject jsonObject = new JSONObject();
FileReader fileReader = new FileReader(new File("C:\\Usrc\\main\\java\\Sample.properties"));
Properties properties = new Properties();
properties.load(fileReader);
Iterator<Entry<Object, Object>> iterator = properties.entrySet().iterator();
while (iterator.hasNext()) {
Entry<Object, Object> entry = iterator.next();
String value = (String) entry.getKey();
String[] values = value.split("\\.");
JSONObject opt = jsonObject.optJSONObject(values[0]);
if(opt!=null) {
opt.put(values[1],entry.getValue());
}else {
JSONObject object = new JSONObject();
object.put(values[1], entry.getValue());
jsonObject.put(values[0], object);
}
}
System.out.println(jsonObject.toString());
}
}
Output
{"a":{"x":"1","y":"3"},"b":{"z":"10"}}
I,m given a problem to convert JSON file to excel file, in short to convert JSON data to excel data. Tried mapping JSON keys and values but can't do it.
Tried mapping JSON keys and values but can't do it. I have already used apache POI api.
public class jsontoexcel {
public static void main(String[] args) throws IOException,JSONException {
jsontoexcel json4=new jsontoexcel();
JSONObject json=json4.ReadJson();
JSONArray array =new JSONArray();
JSONObject rowjson=json.getJSONArray("rows").getJSONObject(0);
XSSFWorkbook workbook=new XSSFWorkbook();
XSSFSheet sheet=workbook.createSheet("Company Details");
int len=rowjson.length();
String[] RowArr=new String[len];
Iterator<String> keys = rowjson.keys();
int i=0;
while(keys.hasNext())
{
RowArr[i]=keys.next();
System.out.print("key:"+keys);
i++;
}
List<String> slist= new ArrayList<String>();
slist=json.get(rowjson.toString(keys));
FileOutputStream out=new FileOutputStream(new File("C:\\code\\eclipse\\jsontoexcel\\src\\output.xlsx"));
createHeaderRow(sheet, RowArr);
workbook.write(out);
out.close();
// Map<String,Object> map=new Map<String,Object>();
}
public static void createHeaderRow(XSSFSheet sheet, String[] RowArr)
{
Row row=sheet.createRow(0);
for(int i=0;i<RowArr.length-1;i++)
{
Cell cellTitle=row.createCell(i+1);
String cellVal=RowArr[i];
System.out.print("Cell data" + cellVal);
}
}
}
I expect the output to be stored in an excel file. The headers are getting printed but not the values.
Do not generate Excel file until you really have to. In case, you want to generate data without any specific formatting, charts, macros, etc. just generate CSV file with pure data. To read JSON and generate CSV you can use Jackson library which supports these two data formats. Just assume your JSON looks like below:
{
"rows": [
{
"id": 1,
"name": "Vika",
"age": 27
},
{
"id": 2,
"name": "Mike",
"age": 28
}
]
}
You, need to create POJO model which fits to that structure, deserialise JSON to objects and serialise objects to CSV format. Example solution, could look like below:
import com.fasterxml.jackson.annotation.JsonPropertyOrder;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SequenceWriter;
import com.fasterxml.jackson.dataformat.csv.CsvMapper;
import com.fasterxml.jackson.dataformat.csv.CsvSchema;
import java.io.File;
import java.util.List;
public class JsonApp {
public static void main(String[] args) throws Exception {
File jsonFile = new File("./resource/test.json").getAbsoluteFile();
ObjectMapper jsonMapper = new ObjectMapper();
Response response = jsonMapper.readValue(jsonFile, Response.class);
CsvMapper csvMapper = new CsvMapper();
CsvSchema schema = csvMapper.schemaFor(Item.class).withHeader();
SequenceWriter sequenceWriter = csvMapper.writer(schema).writeValues(System.out);
sequenceWriter.writeAll(response.getRows());
}
}
class Response {
private List<Item> rows;
// getters, setters
}
#JsonPropertyOrder({"id", "name", "age"})
class Item {
private int id;
private String name;
private int age;
// getters, setters
}
Above code prints:
id,name,age
1,Vika,27
2,Mike,28
See also:
jackson-dataformats-text
Intro to the Jackson ObjectMapper
I am using Jackson objectMapper to read JSON to JsonNode, Then I am using xmlMapper to serialize this to XML.
I'd like to set an XML attribute value by parsing a JSON attribute with tag "#". Any help would be appreciated. Thanks.
EXAMPLE JSON:
{
"response" : {
"label" : {
"#data" : "someValue"
}
}
}
NEED TO MAP TO XML:
<response>
<label data="someValue" />
</response>
This is what I can get right now:
<response>
<label>
<#data>someValue</#data>
</label>
</response>
CODE:
JsonNode root = objectMapper.readTree(JSON);
xml = xmlMapper.writeValueAsString(root);
well, I found a solution that produces the required output. However, I am not sure if this is an optimal solution, in a sense that it requires custom inner classes to be able to specify the annotations that tell the mappers how to parse.
import java.util.*;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.*;
import com.fasterxml.jackson.databind.type.TypeFactory;
import com.fasterxml.jackson.dataformat.xml.XmlMapper;
import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlProperty;
public class JSONTest
{
#SuppressWarnings("unchecked")
public static void main(String[] args)
{
try {
String jsonInput = "{ \"response\" : { \"label\" : { \"#data\" : \"someValue\" } }}";
ObjectMapper om = new ObjectMapper();
TypeFactory tf = om.getTypeFactory();
JavaType mapType = tf.constructMapType(HashMap.class, String.class, response.class);
Map<String, response> map = (Map<String, response>)om.readValue(jsonInput, mapType);
XmlMapper xmlMapper = new XmlMapper();
String ss = xmlMapper.writeValueAsString(map.get("response"));
System.out.println(ss);
} catch (Exception e) {
e.printStackTrace();
}
}
public static class response {
public Label label;
}
public static class Label {
#JsonProperty("#data")
#JacksonXmlProperty(isAttribute = true)
public String data;
}
}
output:
<response><label data="someValue"/></response>
I am using following code:
CsvSchema bootstrap = CsvSchema.emptySchema().withHeader();
ObjectMapper mapper = new CsvMapper();
File csvFile = new File("input.csv"); // or from String, URL etc
Object user = mapper.reader(?).withSchema(bootstrap).readValue(new File("data.csv"));
mapper.writeValue(new File("data.json"), user);
It throws an error in my IDE saying cannot find symbol method withSchema(CsvSchema) but why? I have used the code from some examples.
I don't know what to write into mapper.reader() as I want to convert any CSV file.
How can I convert any CSV file to JSON and save it to the disk?
What to do next? The examples
I think, you should use MappingIterator to solve your problem. See below example:
import java.io.File;
import java.io.IOException;
import java.util.List;
import java.util.Map;
import com.fasterxml.jackson.databind.MappingIterator;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.csv.CsvMapper;
import com.fasterxml.jackson.dataformat.csv.CsvSchema;
public class JacksonProgram {
public static void main(String[] args) throws Exception {
File input = new File("/x/data.csv");
File output = new File("/x/data.json");
List<Map<?, ?>> data = readObjectsFromCsv(input);
writeAsJson(data, output);
}
public static List<Map<?, ?>> readObjectsFromCsv(File file) throws IOException {
CsvSchema bootstrap = CsvSchema.emptySchema().withHeader();
CsvMapper csvMapper = new CsvMapper();
try (MappingIterator<Map<?, ?>> mappingIterator = csvMapper.readerFor(Map.class).with(bootstrap).readValues(file)) {
return mappingIterator.readAll();
}
}
public static void writeAsJson(List<Map<?, ?>> data, File file) throws IOException {
ObjectMapper mapper = new ObjectMapper();
mapper.writeValue(file, data);
}
}
See this page: jackson-dataformat-csv for more information and examples.