We Keep Coding

OpenCSV - create row for every entry in list of maps

I am trying to implement an DMN (Decision Model and Notation) evaluation service, where the user can upload a csv file with test cases to be evaluated and receive results also as a csv file for every test cases in the input file.
Reading the input csv file and evaluating the test cases works without problems. But I have some issues in writing the results to a csv file using OpenCsv.
Here is the mapped bean, which should be converted to csv row:
#Data
#AllArgsConstructor
#NoArgsConstructor
public class DmnTestCaseResult {
private Map<String, Object> testInput;
private Map<String, Object> expectedOutput;
private List<Map<String, Object>> testOutput;
private String errorMessage;
}
As you can see here, the test case result can have in some situations multiple testOutputs, defined as a list of map.
What I want is to write for every map entry in the testOutput, a seperate row in the csv file. But with the code I wrote below, only the first entry of the testOutput is written as only one row in the csv file.
public String convertDmnRuleTestResultToCsv(DmnRuleTestResult result) {
List<DmnTestCaseResult> results = result.getTestCases();
try(StringWriter sw = new StringWriter(); CSVWriter writer = new CSVWriter(sw, CSVWriter.DEFAULT_SEPARATOR, CSVWriter.NO_QUOTE_CHARACTER, CSVWriter.NO_ESCAPE_CHARACTER, CSVWriter.DEFAULT_LINE_END)) {
StatefulBeanToCsv<DmnTestCaseResult> beanToCsv = new StatefulBeanToCsvBuilder<DmnTestCaseResult>(writer)
.withApplyQuotesToAll(false)
.build();
beanToCsv.write(results);
return sw.toString();
} catch(Exception ex){
throw new CsvParseException(ex.getMessage());
}
}
How can I tell the OpenCsv that it should create seperate row for each entry in the testOutputs ?
EDIT: Added more information
UI:
Resulted incorrect CSV:
Expected correct CSV:
As you can see from the screenshots, one input can have multiple test outputs. Therefore I want to create for every test output a seperate line in csv file.

As StatefulBeanToCsv does not seem to be capable to generating multiple lines for a single bean, I suggest implementing a custom mapping function. This also requires you to manually print the header line as well.
public static String convertDmnRuleTestResultToCsv(DmnRuleTestResult result) {
List<DmnTestCaseResult> results = result.getTestCases();
try (StringWriter sw = new StringWriter();
CSVWriter writer = new CSVWriter(sw, CSVWriter.DEFAULT_SEPARATOR,
CSVWriter.NO_QUOTE_CHARACTER, CSVWriter.NO_ESCAPE_CHARACTER,
CSVWriter.DEFAULT_LINE_END)) {
writeHeader(writer);
for (DmnTestCaseResult r : results) {
for (Map<String, Object> map : r.getTestOutput())
writer.writeNext(map(r, map));
}
return sw.toString();
} catch (Exception ex) {
throw new RuntimeException(ex.getMessage());
}
}
private static void writeHeader(CSVWriter writer) {
List<String> header = new ArrayList<>();
header.add("ERRORMESSAGE");
header.add("EXPECTEDOUTPUT");
header.add("INPUT");
header.add("OUTPUT");
writer.writeNext(header.toArray(new String[] {}));
}
private static String[] map(DmnTestCaseResult r, Map<String, Object> testOutput) {
// you can manually adjust formats here as well; entrySet() call can be left out, it does change the format. do what you like more
List<String> line = new ArrayList<>();
line.add(r.getErrorMessage());
line.add(r.getExpectedOutput().entrySet().toString());
line.add(r.getTestInput().entrySet().toString());
line.add(testOutput.entrySet().toString());
return line.toArray(new String[] {});
}
And this prints:
ERRORMESSAGE,EXPECTEDOUTPUT,INPUT,OUTPUT
errorMessage,[expectedOutput1=expectedOutput1, expectedOutput2=expectedOutput2],[input2=testInput2, input1=testInput1],[testOut2=testOut2, testOut=testOut1]
errorMessage,[expectedOutput1=expectedOutput1, expectedOutput2=expectedOutput2],[input2=testInput2, input1=testInput1],[testOut3=testOut3, testOut4=testOut4]

Using Jackson to convert CSV to JSON - How to remove newlines embedded in CSV column header

After some quick Googling, I found an easy way to read and parse a CSV file to JSON using the Jackson library. All well and good, except ... some of the CSV header column names have embedded newlines. The program handles it, but I'm left with JSON keys with newlines embedded within. I'd like to remove these (or replace them with a space).
Here is the simple program I found:
import java.io.File;
import java.util.List;
import java.util.Map;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.csv.CsvMapper;
import com.fasterxml.jackson.dataformat.csv.CsvSchema;
public class CSVToJSON {
public static void main(String[] args) throws Exception {
File input = new File("PDM_BOM.csv");
File output = new File("output.json");
CsvSchema csvSchema = CsvSchema.builder().setUseHeader(true).build();
CsvMapper csvMapper = new CsvMapper();
// Read data from CSV file
List<Object> readAll = csvMapper.readerFor(Map.class).with(csvSchema).readValues(input)
.readAll();
ObjectMapper mapper = new ObjectMapper();
// Write JSON formated data to output.json file
mapper.writerWithDefaultPrettyPrinter().writeValue(output, readAll);
// Write JSON formated data to stdout
System.out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(readAll));
}
}
So, as an example:
PARENT\nITEM\nNUMBER
Here's an example of what is produced:
"PARENT\nITEM\nNUMBER" : "208E8840040",
I need this to be:
"PARENT ITEM NUMBER" : "208E8840040",
Is there a configuration setting on the Jackson mapper that can handle this? Or, do I need to provide some sort of custom "handler" to the mapper?
Special cases
To add some complexity, there are cases where just replacing the newline with a space will not always yield what is needed.
Example 1:
Sometimes there is a column header like this:
QTY\nORDER/\nTRANSACTION
In this case, I need the newline removed and replaced with nothing, so that the result is:
QTY ORDER/TRANSACTION
, not
QTY ORDER/ TRANSACTION
Example 2:
Sometimes, for whatever reason, a column header has a space before the newline:
EFFECTIVE \nTHRU DATE
This needs to come out as:
EFFECTIVE THRU DATE
, not
EFFECTIVE THRU DATE
Any ideas on how to handle at least the main issue would be very much appreciated.

You can use the String replaceAll() method to replace all new lines with spaces.
String str = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(readAll);
str = str.trim().replaceAll("[\n\s]+", " ");

OK, came up with a solution. It's ugly, but it works. Basically, after the CsvMapper finishes, I go through the giant ugly collection that's produced and do a String.replaceAll (thanks to https://stackoverflow.com/users/4402505/prem-kurian-philip for that suggestion) to remove the unwanted characters and then rebuild the map.
In any case here's the new code:
public class CSVToJSON {
public static void main(String[] args) throws Exception {
File input = new File("PDM_BOM.csv");
File output = new File("output.json");
CsvSchema csvSchema = CsvSchema.builder().setUseHeader(true).build();
CsvMapper csvMapper = new CsvMapper();
// Read data from CSV file
List<Object> readData = csvMapper.readerFor(Map.class).with(csvSchema).readValues(input)
.readAll();
for (Object mapObj : readData) {
LinkedHashMap<String, String> map = (LinkedHashMap<String, String>) mapObj;
List<String> deleteList = new ArrayList<>();
LinkedHashMap<String, String> insertMap = new LinkedHashMap<>();
for (Object entObj : map.entrySet()) {
Entry<String, String> entry = (Entry<String, String>) entObj;
String oldKey = entry.getKey();
String newKey = oldKey.replaceAll("[\n\s]+", " ");
String value = entry.getValue();
deleteList.add(oldKey);
insertMap.put(newKey, value);
}
// Delete the old ...
for (String oldKey : deleteList) {
map.remove(oldKey);
}
// and bring in the new
map.putAll(insertMap);
}
ObjectMapper mapper = new ObjectMapper();
// Write JSON formated data to output.json file
mapper.writerWithDefaultPrettyPrinter().writeValue(output, readData);
// Write JSON formated data to stdout
System.out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(readAll));
}
}
It seems like there should be a better way to achieve this.

Apache Commons CSV Mapping not found

I am trying to read a CSV file with certain headers into a Java object using Apache Commons CSV. However, when I run the code, I get the following exeption:
Exception in thread "main" java.lang.IllegalArgumentException: Mapping for Color not found, expected one of [Color, Name, Price, House Cost, Rent, 1 House, 2 Houses, 3 Houses, 4 Houses, Hotel, Mortgage]
at org.apache.commons.csv.CSVRecord.get(CSVRecord.java:102)
at GameBoard.<init>(GameBoard.java:25)
at Game.main(Game.java:3)
Can someone explain where the exception is coming from? It appears to me that Apache Commons somehow is not matching my input to a column. Is there something wrong on my part or is something else broken? Here is my code snippet:
Reader in;
Iterable<CSVRecord> records = null;
try {
in = new FileReader(new File(Objects.requireNonNull(getClass().getClassLoader().getResource("Properties.csv")).getFile()));
records = CSVFormat.EXCEL.withFirstRecordAsHeader().parse(in);
} catch (IOException | NullPointerException e) {
e.printStackTrace();
System.exit(1);
}
for (CSVRecord record :
records) {
spaces.add(new Property(
record.get("Color"),
record.get("Name"),
Integer.parseInt(record.get("Price")),
And here are my csv headers (sorry, one was cut off but that's not the point):
Thanks!

I had the same probem which only occurs if you reference the first column, all other column names are working. The problem is, that the UTF-8 representation prepends the following characters "0xEF,0xBB,0xBF" (see Wikipedia page). This seems to be a known problem for commons-csv but since this is application specific, it won't be fixed (CSVFormat.EXCEL.parse should handle byte order marks).
However, there is a documented workaround for this:
http://commons.apache.org/proper/commons-csv/user-guide.html#Handling_Byte_Order_Marks

I got the same weird exception. It actually said "Expecting one of ..." and then listed the field it said it could not find - just like in your case.
The reason was that I had set the wrong CSVFormat:
CSVFormat csvFormat = CSVFormat.newFormat(';');
This meant that my code was trying to separate fields on semi-colons in a file that actually had comma separators.
Once I used the DEFAULT CSVFormat, everything started to work.
CSVFormat csvFormat = CSVFormat.DEFAULT;
So the answer is that probably you must set CSVFormat correctly for your file.

Moving to spring boot version 2.6.7 from 2.4.5 brought about this error.. I had to convert each csvRecord to a map before assigning it to my POJO as follows.
for (CSVRecord csvRecord : csvRecords) {
Map<String, String> csvMap = csvRecord.toMap();
Model newModel = new Model();
model.setSomething(csvMap.get("your_item"));
}

I also got the same exception by giving a different name of header in CSV file like xyz, or trying to get the value by calling csvRecord.get("x_z")
I resolved my problem changing the header name xyz.
try {
fileReader = new BufferedReader(new InputStreamReader(is, "UTF-8"));
csvParser = new CSVParser(fileReader,
CSVFormat.DEFAULT.withFirstRecordAsHeader().withIgnoreHeaderCase().withTrim());
Iterable<CSVRecord> csvRecords = csvParser.getRecords();
CSVFormat csvFormat = CSVFormat.DEFAULT;
for (CSVRecord csvRecord : csvRecords) {
} catch (Exception e) {
System.out.println("Reading CSV Error!");
e.printStackTrace();
} finally {
try {
fileReader.close();
csvParser.close();
} catch (IOException e) {
System.out.println("Closing fileReader/csvParser Error!");
e.printStackTrace();
}
}

JSON.parse() equivalent in mongo driver 3.x for Java

JSON.parse() from mongo (Java driver) returns either a BasicDBList or a BasicDBObject.
However, when migrating to mongo driver 3.x, what's the new parse method that returns either Document or List<Document>?
In the new driver, Document.parse() only parses an object, not an array (throws an exception when given an array).
What is the equivalent of JSON.parse() for Arrays with 3.x Java drivers ?

A simple trick to parse any JSON and to get either Document or List<Document>:
Document.parse("{\"json\":" + json + "}").get("json")

To parse JSON string data using the mongodb java driver 3.x:
Parse JSON document:
Use the Document.parse() static method to parse a single JSON document.
Document doc = Document.parse("{\"objA\":{\"foo\":1}}");
Parse JSON array:
Use an instance of BsonArrayCodec to decode a JsonReader.
For example:
final String JSON_DATA
= "[{\"objA\":{\"foo\":1}},"
+ "{\"objB\":{\"bar\":2}}]";
final CodecRegistry codecRegistry = CodecRegistries.fromProviders(asList(new ValueCodecProvider(),
new BsonValueCodecProvider(),
new DocumentCodecProvider()));
JsonReader reader = new JsonReader(JSON_DATA);
BsonArrayCodec arrayReader = new BsonArrayCodec(codecRegistry);
BsonArray docArray = arrayReader.decode(reader, DecoderContext.builder().build());
for (BsonValue doc : docArray.getValues()) {
System.out.println(doc);
}
ref: http://api.mongodb.org/java/3.2/org/bson/json/JsonReader.html,
http://api.mongodb.org/java/3.2/org/bson/codecs/BsonArrayCodec.html

Added cast to #Oleg Nitz answer, for completeness.
Object object = Document.parse("{\"json\":" + jsonData.getJson() + "}").get("json");
if (object instanceof ArrayList) {
documents = (ArrayList<Document>) object;
} else (object instanceof Document) {
document = (Document) object;
}

How about this:
Document doc = new Document("array", JSON.parse("[ 100, 500, 300, 200, 400 ]", new JSONCallback()));
System.out.println(doc.toJson()); //prints { "array" : [100, 500, 300, 200, 400] }

You're right that there's no easy equivalent.
If you use line-delimited JSON documents instead of a JSON array, it becomes fairly straightforward:
List<Document> getDocumentsFromLineDelimitedJson(final String lineDelimitedJson) {
BufferedReader stringReader = new BufferedReader(
new StringReader(lineDelimitedJson));
List<Document> documents = new ArrayList<>();
String json;
try {
while ((json = stringReader.readLine()) != null) {
documents.add(Document.parse(json));
}
} catch (IOException e) {
// ignore, can't happen with a StringReader
}
return documents;
}
For example, this call
System.out.println(getDocumentsFromLineDelimitedJson("{a : 1}\n{a : 2}\n{a : 3}"));
will print:
[Document{{a=1}}, Document{{a=2}}, Document{{a=3}}]

The easiest equivalent for me is to use any json library to convert the json to POJO. Below is an example using jackson:
String input = "[{\"objA\":{\"foo\":1}},{\"objB\":{\"bar\":2}}]";
ObjectMapper mapper = new ObjectMapper();
List<Document> output = (List<Document>) mapper.readValue(input, List.class)
.stream().map(listItem -> new Document((LinkedHashMap)listItem))
.collect(Collectors.toList());

Reading XML database column in java

I have DB2 database with XML column. I would like to read data from it and save each XML to separate file.
Here is a part of my code:
final List<Map<String, Object>> myList = dbcManager.createQuery(query).getResultList();
int i=0;
for (final Map<String, Object> element : myList) {
i++;
String filePath = "C://elements//elem_" + i + ".xml";
File file = new File(filePath);
if(!file.exists()){
file.createNewFile();
}
BufferedWriter out = new BufferedWriter(new FileWriter(filePath));
out.write(element.get("columnId"));
out.close();
}
Now, I have error in line out.write(element.get("columnId"));, because element.get("columnId") is an object type and it should be for example string.
And my question is: To which type should I convert (cast) element.get("columnId") to save it in xml file?

You should use the ResultSet.getSQLXML() method to read the XML column value, then use an appropriate method of the SQLXML class, e.g. getString() or getCharacterStream(). More info here.