Entire large json response is not returned - java

I am trying to get a large json response using a recursive REST call like below:
private List<MyPojo> recursiveRestCallMethod(String folderId) throws IOException {
List<MyPojo> mypojoList = new ArrayList<>();
String hugeJson = webClient.get()
.uri("/my/rest/api/accepting/" + folderId
+ "/and/producing/huge/jsonresponse/for/all/files/recursively")
.retrieve().bodyToMono(String.class).block();
byte[] bytes = hugeJson.getBytes("UTF-8");
String json = new String(bytes, StandardCharsets.UTF_8);
ObjectMapper objectMapper = new ObjectMapper();
ObjectNode node = objectMapper.readValue(json, ObjectNode.class);
objectMapper.configure(
DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
if (node.get("list").get("entries").isArray()) {
for (JsonNode jsonNode : node.get("list").get("entries")) {
MyPojo pojo = new MyPojo();
JsonNode mainNode = jsonNode.get("entry");
if (mainNode.get("isFile").asBoolean()) {
JsonNode nameNode = mainNode.get("name");
pojo.setNodename(nameNode.toString());
// and 20 more fields
mypojoList.add(pojo);
}
if (mainNode.get("isFolder").asBoolean()) {
mypojoList.addAll(recursiveRestCallMethod(mainNode.get("id").toString().replaceAll("\"", "").trim()));
}
}
return mypojoList;
}
return null;
}
Now everytime the json returned has 4193150 characters and it throws exception - Unexpected end-of-input: expected close marker for Object as reported here and some other SO threads (obviously, the json is not complete and valid).
The incomplete json I am getting looks something like:
{"list":{"pagination":{"count":6097,"hasMoreItems":false,"totalItems":6097,"skipCount":0,"maxItems":10000},"entries":[{"entry":{"....
From above, as you can see I should get 6097 objects, but I am getting only 2024 entry array items. And after that json ends abruptly. i.e. invalid json string.
However, for smaller response, where I have 20/30 entry array items, it works as expected.
Note: I am using Spring-Boot 2.4.5 and hence Jackson 2.12.4
Question: Even though I am using .block(), why the response stops at 4193150 characters? What I am doing wrong here?

Not sure what was wrong using String but when I switched to DataBuffer, it worked fine.
Here is the snippet for what I used:
final Flux<DataBuffer> hugeJson = webClient.get()
.uri("/my/rest/api/accepting/" + folderId
+ "/and/producing/huge/jsonresponse/for/all/files/recursively")
.accept(MediaType.ALL)
.retrieve()
.bodyToFlux(DataBuffer.class);

Related

InputStream does not seem to read response objects in list

I am trying to read a response from an API endpoint. It works in Postman and appears to work when debugging my java code.
However it will not read any objects that are in the nested array.
I've followed this solution (in the comments someone lays out their solution to accessing objects in a nested array) yet that does not seem to have solved anything as the mapper consistently shows nothing inside the json list.
Here's my code:
try {
response = httpClient.execute(httpPost);
InputStream inputStream = response.getEntity().getContent();
final ObjectMapper mapper = new ObjectMapper();
final JsonNode node = mapper.readTree(inputStream);
final JsonNode value = node.get("value");
final JsonNode index = value.get(1);
final String status = index.get("status").asText();
return status;`
}
Here's the response that I'm trying to unpack. I'm trying to get the value of "status"
{
"value": [
{
"activityRunEnd": "2023-01-23T23:22:01.4234985Z",
},
{
"status": "Succeeded"
}
]
}
This is the response that I'm currently getting when I readTree(inputStream)
{"value":[]}
InputStream does not seem to read any values inside the list

Store request body as JSON into the database - Java

I want to store the request body into the database. I am storing the headers and request into the audit trail table.
For conversion of headers, I am creating a map and then storing the json into the database like this:
String method = httpServletRequest.getMethod();
Enumeration<String> values = httpServletRequest.getHeaderNames();
Map<String,String> headers = new HashMap<String, String>();
while (values.hasMoreElements()) {
String key = values.nextElement();
String innerValue = (String)httpServletRequest.getHeader(key);
headers.put(key,innerValue);
}
// converting object to Array
auditTrail.setHeaders(appUtility.objectToString(headers));
How can I store the request body as json into the database (ignore files for now)? The request body can be one object, array of objects or a combination.
Right now, I have written an ASPECT which will store all the incoming requests into the database. It's easy to store the json of the object when I know the Object but how can we make it generic?
If you know that content is JSON and want to read data from stream you can do it on this way:
String jsonBody = request.getReader().lines()
.collect(Collectors.joining(System.lineSeparator()));
If you need to validate is jsonBody really JSON you can use Jackson:
public static boolean isJSONValid(String jsonInString ) {
try {
final ObjectMapper mapper = new ObjectMapper();
mapper.readTree(jsonInString);
return true;
} catch (IOException e) {
return false;
}
}
If you have a body as the object you can use Jackson.
Example:
SomeObject body = ...; // Your body
ObjectMapper objectMapper = new ObjectMapper();
String bodyAsString = objectMapper.writeValueAsString(body); // To JSON
body = objectMapper.readValue(bodyAsString, SomeObject.class); // From JSON
Jakson with maven:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>

Java - Access JSON Element (llegalFormatConversionException: d !=java.lang.String)

I am trying to learn JAVA and build a plugin for Minecraft; I have been successfully able to get the JSON data from my api endpoint however, the issue I am facing right now is an llegalFormatConversionException: d !=java.lang.String which means that the format I am trying to make into string isn't equal to the type of string that it's looking for.
I am trying to access a JSON element from my endpoint called condition
{
"todaysdate": "2021-02-12",
"temperature": 25,
"description": [
"Overcast"
],
"condition": 122,
}
Coming from C#; I know there's a website called json2sharp where you can create a root class for the JSON. I'm not sure how it would be applied in Java but currently, my code looks like this.
private String fetchWeather() throws IOException, InvalidConfigurationException {
// Download
final URL url = new URL(API);
final URLConnection request = url.openConnection();
// Set HEADER
request.setRequestProperty("x-api-key", plugin.apiKey);
request.setConnectTimeout(5000);
request.setReadTimeout(5000);
request.connect();
// Convert to a JSON object to print data
JsonParser jp = new JsonParser(); //from gson
JsonElement root = jp.parse(new InputStreamReader((InputStream) request.getContent())); //Convert the input stream to a json element
JsonObject rootobj = root.getAsJsonObject();
//JsonElement code = rootobj.get("condition");
String condition_code = rootobj.get("condition").toString();
plugin.getLogger().fine(String.format(
"[%s] Weather is %d",
world.getName(), condition_code
));
return condition_code;
}
If I call
private JsonObject fetchWeather() throws IOException, InvalidConfigurationException {
// Download
final URL url = new URL(API);
final URLConnection request = url.openConnection();
// Set HEADER
request.setRequestProperty("x-api-key", plugin.apiKey);
request.setConnectTimeout(5000);
request.setReadTimeout(5000);
request.connect();
// Convert to a JSON object to print data
JsonParser jp = new JsonParser(); //from gson
JsonElement root = jp.parse(new InputStreamReader((InputStream) request.getContent())); //Convert the input stream to a json element
JsonObject rootobj = root.getAsJsonObject();
return rootobj;
}
state = fetchWeather();
plugin.getLogger().warning(state.toString());
I do get the actual JSON with all of the elements so I know the URL and accessing it is completely working, but I get an llegalexception for format if I try to call and print with logger the condition code.
So, if I change the return type in fetchWeather() to be the root json object and then try to print it with the state variable above it works; but if I try to return the condition json element and print it it gives me an iilegal exception.
Now before I posted this question I did read some other questions people had but I couldn't get a working solution from their suggested answers. So, I am hoping someone can point me out on what I'm doing wrong because I know I'm messing up somewhere with the variable format.
Thanks.
Line 37: state = fetchWeather();
Line 103:
plugin.getLogger().fine(String.format(
"[%s] Weather is %d",
world.getName(), bId
));
condition_code variable is String. You should use the %s format specifier.

write Java List to JSON Array via Jackson

I am using Jackson for JSON serialization. I am trying to convert a Java List (containing string values) to a JSON array. I tried the following approaches (issues given below for each)
1. write array elements using JsonGenerator's writeString
final JsonGenerator generator = factory.createGenerator(output, JsonEncoding.UTF8);
generator.writeStartArray();
for (String arg: argsList) {
generator.writeStartObject();
log.info("arg value is {}", arg);
generator.writeString(arg);
generator.writeEndObject();
}
generator.writeEndArray();
Exception
Can not write a string, expecting field name (context: Object)
I get the exception from "generator.writeString(arg)". I cannot use writeStringField.
object mapper
ObjectMapper mapper = new ObjectMapper();
mapper.writeValue(out, argsList);
final byte[] argsBytes = out.toByteArray();
generator.writeFieldName("args");
generator.writeObjectField("args", argsBytes)
This creates the array as a String and not an array within the JSON object (which is what I am trying to achieve). Any suggestions would be welcome.
End state (trying to achieve):
{
"args":["abc","def","ghi","jkl","mno"]
}
By starting/ending an object around each array entry what you are doing is trying to make invalid json:
{
"args":[{"abc"},{"def"},{"ghi"},{"jkl"},{"mno"}]
}
And the generator is rightly stopping you from doing this.
Just write the strings directly into the array:
final JsonGenerator generator = factory.createGenerator(output, JsonEncoding.UTF8);
generator.writeStartArray();
for (String arg: argsList) {
generator.writeString(arg);
}
generator.writeEndArray();

How to deal with input parameter in CXF request handler in general?

I have been doing some work with apache CXF(version 2.2.2) JAX-RS. I am trying to introduce data validation layer in CXF request handler before business method be invoked. Fortunately :), I am encountering an issue on input parameter handling in request handler(DataValidationHandler). I can read the JSON Object manually by following code lines in request handler. But it's duplicated with JSONProvider registered in CXF framework. Because JSON object input stream only can be read once, otherwise we will meet exception "java.io.EOFException: No content to map to Object due to end of input". Moreover, duplicated JSON object deserializing will impacts performance. Following code is sample for your reference.
Read JSON Object from HTTP body manually:
OperationResourceInfo ori = paramMessage.getExchange().get(OperationResourceInfo.class);
MultivaluedMap<String, String> values = new MetadataMap<String, String>();
List<Object> objList = JAXRSUtils.processParameters(ori, values, paramMessage);
Register JSONProvider in CXF JAX-RS framework:
<bean id="JSONProvider" class="com.accela.govxml2.jaxrs.util.JSONProvider"></bean>
Read JSON Object to Java Object from input stream:
public Object readFrom(......){
ObjectMapper objectMapper = new ObjectMapper();
Object result = objectMapper.readValue(entityStream, TypeFactory.defaultInstance().constructType(genericType));
Return result;
}
I am dealing with Path Parameter manually by following code lines.
OperationResourceInfo ori = paramMessage.getExchange().get(OperationResourceInfo.class);
URITemplate t1 = ori.getClassResourceInfo().getURITemplate();
URITemplate t2 = ori.getURITemplate();
UriInfo uriInfo = new UriInfoImpl(paramMessage, null);
MultivaluedMap<String, String> map = new MetadataMap<String, String>();
t1.match(uriInfo.getPath(), map);
String str = map.get(URITemplate.FINAL_MATCH_GROUP).get(0);
t2.match(str, map);
String pathParameter= null;
if (map.containsKey("pathParam") && !ValidationUtil.isEmpty(map.get("pathParam")))
{
pathParameter= map.get("pathParam").get(0);
}
My questions are here:
How to deal with POST/PUT input parameter of http body in request handler in general?
How to avoid performance issue to read input parameter efficiently?
Is there any way to inject the validation (handler/interceptor) layer between parameter reading by CXF(JSONProvider) and business method invoking?
Is there any elegant way to deal with path parameter?
Thanks for your help. Any comments & suggestions will be appreciated.
Regards,
Dylan
I have found another way to inject DataValidation Interceptor into reading parameter phase. We can reuse deserialized input model from message content, which be deserialized by JSONProvider registered in framework. It can improve performance, because only deserialize input model once.
public class DataValidationInInterceptor extends AbstractPhaseInterceptor<Message>
{
public DataValidationInInterceptor()
{
super(Phase.READ);
}
#Override
public void handleMessage(Message message)
{
OperationResourceInfo ori = message.getExchange().get(OperationResourceInfo.class);
Method method = ori.getMethodToInvoke();
Class<?>[] types = method.getParameterTypes();
Type[] genericParameterTypes = method.getGenericParameterTypes();
for (int i = 0; i < types.length; i++)
{
Class<?> type = types[i];
List obj = (List) message.getContent(List.class);
System.out.println(obj);
System.out.println(type);
}
}
}
After researching, I can read the input stream twice based on the following question's answer ( Read stream twice ).
However, JSON object deserializing performance is still my concern. Who has better solution for it?
Intercept request and change message content from CoyoteInputStream to ByteArrayInputStream, so I can read the InputStream twice.
InputStream in = message.getContent(InputStream.class);
if (in != null)
{
ByteArrayOutputStream baos = new ByteArrayOutputStream();
IOUtils.copy(in, baos);
byte[] bytes = baos.toByteArray();
ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
message.setContent(InputStream.class, bais);
}
Reset ByteArrayInputStream before Read JSON Object to Java Object from input stream:
public Object readFrom(......){
ObjectMapper objectMapper = new ObjectMapper();
if (entityStream.markSupported())
{
entityStream.reset();
}
Object result = objectMapper.readValue(entityStream, TypeFactory.defaultInstance().constructType(genericType));
return result;
}

Categories