I am using Jackson for JSON serialization. I am trying to convert a Java List (containing string values) to a JSON array. I tried the following approaches (issues given below for each)
1. write array elements using JsonGenerator's writeString
final JsonGenerator generator = factory.createGenerator(output, JsonEncoding.UTF8);
generator.writeStartArray();
for (String arg: argsList) {
generator.writeStartObject();
log.info("arg value is {}", arg);
generator.writeString(arg);
generator.writeEndObject();
}
generator.writeEndArray();
Exception
Can not write a string, expecting field name (context: Object)
I get the exception from "generator.writeString(arg)". I cannot use writeStringField.
object mapper
ObjectMapper mapper = new ObjectMapper();
mapper.writeValue(out, argsList);
final byte[] argsBytes = out.toByteArray();
generator.writeFieldName("args");
generator.writeObjectField("args", argsBytes)
This creates the array as a String and not an array within the JSON object (which is what I am trying to achieve). Any suggestions would be welcome.
End state (trying to achieve):
{
"args":["abc","def","ghi","jkl","mno"]
}
By starting/ending an object around each array entry what you are doing is trying to make invalid json:
{
"args":[{"abc"},{"def"},{"ghi"},{"jkl"},{"mno"}]
}
And the generator is rightly stopping you from doing this.
Just write the strings directly into the array:
final JsonGenerator generator = factory.createGenerator(output, JsonEncoding.UTF8);
generator.writeStartArray();
for (String arg: argsList) {
generator.writeString(arg);
}
generator.writeEndArray();
Related
I am trying to get a large json response using a recursive REST call like below:
private List<MyPojo> recursiveRestCallMethod(String folderId) throws IOException {
List<MyPojo> mypojoList = new ArrayList<>();
String hugeJson = webClient.get()
.uri("/my/rest/api/accepting/" + folderId
+ "/and/producing/huge/jsonresponse/for/all/files/recursively")
.retrieve().bodyToMono(String.class).block();
byte[] bytes = hugeJson.getBytes("UTF-8");
String json = new String(bytes, StandardCharsets.UTF_8);
ObjectMapper objectMapper = new ObjectMapper();
ObjectNode node = objectMapper.readValue(json, ObjectNode.class);
objectMapper.configure(
DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
if (node.get("list").get("entries").isArray()) {
for (JsonNode jsonNode : node.get("list").get("entries")) {
MyPojo pojo = new MyPojo();
JsonNode mainNode = jsonNode.get("entry");
if (mainNode.get("isFile").asBoolean()) {
JsonNode nameNode = mainNode.get("name");
pojo.setNodename(nameNode.toString());
// and 20 more fields
mypojoList.add(pojo);
}
if (mainNode.get("isFolder").asBoolean()) {
mypojoList.addAll(recursiveRestCallMethod(mainNode.get("id").toString().replaceAll("\"", "").trim()));
}
}
return mypojoList;
}
return null;
}
Now everytime the json returned has 4193150 characters and it throws exception - Unexpected end-of-input: expected close marker for Object as reported here and some other SO threads (obviously, the json is not complete and valid).
The incomplete json I am getting looks something like:
{"list":{"pagination":{"count":6097,"hasMoreItems":false,"totalItems":6097,"skipCount":0,"maxItems":10000},"entries":[{"entry":{"....
From above, as you can see I should get 6097 objects, but I am getting only 2024 entry array items. And after that json ends abruptly. i.e. invalid json string.
However, for smaller response, where I have 20/30 entry array items, it works as expected.
Note: I am using Spring-Boot 2.4.5 and hence Jackson 2.12.4
Question: Even though I am using .block(), why the response stops at 4193150 characters? What I am doing wrong here?
Not sure what was wrong using String but when I switched to DataBuffer, it worked fine.
Here is the snippet for what I used:
final Flux<DataBuffer> hugeJson = webClient.get()
.uri("/my/rest/api/accepting/" + folderId
+ "/and/producing/huge/jsonresponse/for/all/files/recursively")
.accept(MediaType.ALL)
.retrieve()
.bodyToFlux(DataBuffer.class);
I have converted Json Response to Jackson Object Mapper and CSV to CSV Mapper Object. How should I compare the two? Is this the correct approach?
public void test_CSVtoObject() throws KronosCoreCommonException, IOException {
File csvFile = new File("C:\\Users\\Desktop\\PayrollCSV.csv");
CsvMapper csvMapper = new CsvMapper();
CsvSchema csvSchema = csvMapper
.typedSchemaFor(PayrollExtractCSV.class)
.withHeader()
.withColumnSeparator(',')
.withComments();
MappingIterator<PayrollExtractCSV> mappingIter = csvMapper
.readerWithTypedSchemaFor(PayrollExtractCSV.class)
.with(csvSchema)
.readValues(csvFile);
List<PayrollExtractCSV> mapping = mappingIter.readAll();
}
public void test_JSONtoObject() throws KronosCoreCommonException, JsonMappingException, JsonProcessingException {
List<String> personNumberList = new ArrayList<>();
personNumberList.add("1619889490630");
APIResponse aggregationResponse = payRollStagingAPIObject.runAggregationAPI("T01", personNumberList);
ObjectMapper mapper = new ObjectMapper();
AggregationAPI[] aggregationAPI = mapper.readValue(aggregationResponse.getResponseAsString(), AggregationAPI[].class);
mapper.writeValueAsString(aggregationAPI);
}
Because of the input size, you should beware of collecting all of the data to memory, which might cause OutOfMemoryException.
For both sources, you should read the data line by line while discarding old data as soon as you are done processing it.
It seems like mappingIter.next() will do the trick, and you can use Jackson Streaming API for the JSON source.
You already have a PayrollExtractCSV class, so you can construct objects from both sources and compare them, one by one.
EDIT:
I am not sure how much the WHY matters here. There are lot of reasons I am doing what I am doing in a certain way not feasible to explain in one post.
But I will still give some context, the service that stores my json files gives them back as byte arrays, so i start with byte[] arrays, each byte array is a json structure.
From this json i need to convert to a matching JAVA POJO this is POJO serves as a validator and then the final step is using Freemarker generating HTML files out of this POJO.
My scenario is that i get potentially get multiple byte arrays. Each can be a maximum of 500mb. I need to combine them, convert to JSON string and then convert to JAVA POJO.
There are couple of problems i am trying to address:
I see the jsonData only having 1000 records in all.
How do I combine the two jsons into one single json to be then able to convert to JAVA POJO?
Here's my JSON structure:
{
"Header": {
"id": 123
},
"Data": [
{
"key": "123
}
]
}
Here's my current code stripped to bare which I use for BYTE[] -> JSON STRING - > POJO
byte[] dataFile = null;
for (String fileName : getFileNames()) {
try {
byte[] currentFile = null;
currentFile = SomeService.getByteFile(fileName);
dataFile = ArrayUtils.addAll(dataFile, currentFile);
}
String jsonData = new JSONObject(new String(dataFile)).toString();
ObjectMapper mapper = new ObjectMapper();
className classForMapping = mapper.readValue(jsonData, className.class));
this is the JSONArray:
String json = [{"1":"Kenny", "2":"Tom", "3":"Mosoti"}];
now i want to get a string array of values only eg Kenny, Tom, Mosoti
You may use any json parser to do that for instance Jackson.
It is not a good option to write your own parser as it is error-prone. Also don't re-invent the wheel.
Yeah, you should go with a mainstream library/framework like Jackson, you can read more about it here: http://wiki.fasterxml.com/JacksonInFiveMinutes
A basic example would be something like:
private static MailData unMarshallJson(String literalJson) throws JAXBException, IOException {
Data data = null;
ObjectMapper mapper = new ObjectMapper();
data = mapper.readValue(literalJson, Data.class);
return data;
}
Where the Data class is a class corresponding to the parsed data, read the link that I provided for more info.
I have been doing some work with apache CXF(version 2.2.2) JAX-RS. I am trying to introduce data validation layer in CXF request handler before business method be invoked. Fortunately :), I am encountering an issue on input parameter handling in request handler(DataValidationHandler). I can read the JSON Object manually by following code lines in request handler. But it's duplicated with JSONProvider registered in CXF framework. Because JSON object input stream only can be read once, otherwise we will meet exception "java.io.EOFException: No content to map to Object due to end of input". Moreover, duplicated JSON object deserializing will impacts performance. Following code is sample for your reference.
Read JSON Object from HTTP body manually:
OperationResourceInfo ori = paramMessage.getExchange().get(OperationResourceInfo.class);
MultivaluedMap<String, String> values = new MetadataMap<String, String>();
List<Object> objList = JAXRSUtils.processParameters(ori, values, paramMessage);
Register JSONProvider in CXF JAX-RS framework:
<bean id="JSONProvider" class="com.accela.govxml2.jaxrs.util.JSONProvider"></bean>
Read JSON Object to Java Object from input stream:
public Object readFrom(......){
ObjectMapper objectMapper = new ObjectMapper();
Object result = objectMapper.readValue(entityStream, TypeFactory.defaultInstance().constructType(genericType));
Return result;
}
I am dealing with Path Parameter manually by following code lines.
OperationResourceInfo ori = paramMessage.getExchange().get(OperationResourceInfo.class);
URITemplate t1 = ori.getClassResourceInfo().getURITemplate();
URITemplate t2 = ori.getURITemplate();
UriInfo uriInfo = new UriInfoImpl(paramMessage, null);
MultivaluedMap<String, String> map = new MetadataMap<String, String>();
t1.match(uriInfo.getPath(), map);
String str = map.get(URITemplate.FINAL_MATCH_GROUP).get(0);
t2.match(str, map);
String pathParameter= null;
if (map.containsKey("pathParam") && !ValidationUtil.isEmpty(map.get("pathParam")))
{
pathParameter= map.get("pathParam").get(0);
}
My questions are here:
How to deal with POST/PUT input parameter of http body in request handler in general?
How to avoid performance issue to read input parameter efficiently?
Is there any way to inject the validation (handler/interceptor) layer between parameter reading by CXF(JSONProvider) and business method invoking?
Is there any elegant way to deal with path parameter?
Thanks for your help. Any comments & suggestions will be appreciated.
Regards,
Dylan
I have found another way to inject DataValidation Interceptor into reading parameter phase. We can reuse deserialized input model from message content, which be deserialized by JSONProvider registered in framework. It can improve performance, because only deserialize input model once.
public class DataValidationInInterceptor extends AbstractPhaseInterceptor<Message>
{
public DataValidationInInterceptor()
{
super(Phase.READ);
}
#Override
public void handleMessage(Message message)
{
OperationResourceInfo ori = message.getExchange().get(OperationResourceInfo.class);
Method method = ori.getMethodToInvoke();
Class<?>[] types = method.getParameterTypes();
Type[] genericParameterTypes = method.getGenericParameterTypes();
for (int i = 0; i < types.length; i++)
{
Class<?> type = types[i];
List obj = (List) message.getContent(List.class);
System.out.println(obj);
System.out.println(type);
}
}
}
After researching, I can read the input stream twice based on the following question's answer ( Read stream twice ).
However, JSON object deserializing performance is still my concern. Who has better solution for it?
Intercept request and change message content from CoyoteInputStream to ByteArrayInputStream, so I can read the InputStream twice.
InputStream in = message.getContent(InputStream.class);
if (in != null)
{
ByteArrayOutputStream baos = new ByteArrayOutputStream();
IOUtils.copy(in, baos);
byte[] bytes = baos.toByteArray();
ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
message.setContent(InputStream.class, bais);
}
Reset ByteArrayInputStream before Read JSON Object to Java Object from input stream:
public Object readFrom(......){
ObjectMapper objectMapper = new ObjectMapper();
if (entityStream.markSupported())
{
entityStream.reset();
}
Object result = objectMapper.readValue(entityStream, TypeFactory.defaultInstance().constructType(genericType));
return result;
}