JSON.parse() from mongo (Java driver) returns either a BasicDBList or a BasicDBObject.
However, when migrating to mongo driver 3.x, what's the new parse method that returns either Document or List<Document>?
In the new driver, Document.parse() only parses an object, not an array (throws an exception when given an array).
What is the equivalent of JSON.parse() for Arrays with 3.x Java drivers ?
A simple trick to parse any JSON and to get either Document or List<Document>:
Document.parse("{\"json\":" + json + "}").get("json")
To parse JSON string data using the mongodb java driver 3.x:
Parse JSON document:
Use the Document.parse() static method to parse a single JSON document.
Document doc = Document.parse("{\"objA\":{\"foo\":1}}");
Parse JSON array:
Use an instance of BsonArrayCodec to decode a JsonReader.
For example:
final String JSON_DATA
= "[{\"objA\":{\"foo\":1}},"
+ "{\"objB\":{\"bar\":2}}]";
final CodecRegistry codecRegistry = CodecRegistries.fromProviders(asList(new ValueCodecProvider(),
new BsonValueCodecProvider(),
new DocumentCodecProvider()));
JsonReader reader = new JsonReader(JSON_DATA);
BsonArrayCodec arrayReader = new BsonArrayCodec(codecRegistry);
BsonArray docArray = arrayReader.decode(reader, DecoderContext.builder().build());
for (BsonValue doc : docArray.getValues()) {
System.out.println(doc);
}
ref: http://api.mongodb.org/java/3.2/org/bson/json/JsonReader.html,
http://api.mongodb.org/java/3.2/org/bson/codecs/BsonArrayCodec.html
Added cast to #Oleg Nitz answer, for completeness.
Object object = Document.parse("{\"json\":" + jsonData.getJson() + "}").get("json");
if (object instanceof ArrayList) {
documents = (ArrayList<Document>) object;
} else (object instanceof Document) {
document = (Document) object;
}
How about this:
Document doc = new Document("array", JSON.parse("[ 100, 500, 300, 200, 400 ]", new JSONCallback()));
System.out.println(doc.toJson()); //prints { "array" : [100, 500, 300, 200, 400] }
You're right that there's no easy equivalent.
If you use line-delimited JSON documents instead of a JSON array, it becomes fairly straightforward:
List<Document> getDocumentsFromLineDelimitedJson(final String lineDelimitedJson) {
BufferedReader stringReader = new BufferedReader(
new StringReader(lineDelimitedJson));
List<Document> documents = new ArrayList<>();
String json;
try {
while ((json = stringReader.readLine()) != null) {
documents.add(Document.parse(json));
}
} catch (IOException e) {
// ignore, can't happen with a StringReader
}
return documents;
}
For example, this call
System.out.println(getDocumentsFromLineDelimitedJson("{a : 1}\n{a : 2}\n{a : 3}"));
will print:
[Document{{a=1}}, Document{{a=2}}, Document{{a=3}}]
The easiest equivalent for me is to use any json library to convert the json to POJO. Below is an example using jackson:
String input = "[{\"objA\":{\"foo\":1}},{\"objB\":{\"bar\":2}}]";
ObjectMapper mapper = new ObjectMapper();
List<Document> output = (List<Document>) mapper.readValue(input, List.class)
.stream().map(listItem -> new Document((LinkedHashMap)listItem))
.collect(Collectors.toList());
Related
I am trying to get a large json response using a recursive REST call like below:
private List<MyPojo> recursiveRestCallMethod(String folderId) throws IOException {
List<MyPojo> mypojoList = new ArrayList<>();
String hugeJson = webClient.get()
.uri("/my/rest/api/accepting/" + folderId
+ "/and/producing/huge/jsonresponse/for/all/files/recursively")
.retrieve().bodyToMono(String.class).block();
byte[] bytes = hugeJson.getBytes("UTF-8");
String json = new String(bytes, StandardCharsets.UTF_8);
ObjectMapper objectMapper = new ObjectMapper();
ObjectNode node = objectMapper.readValue(json, ObjectNode.class);
objectMapper.configure(
DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
if (node.get("list").get("entries").isArray()) {
for (JsonNode jsonNode : node.get("list").get("entries")) {
MyPojo pojo = new MyPojo();
JsonNode mainNode = jsonNode.get("entry");
if (mainNode.get("isFile").asBoolean()) {
JsonNode nameNode = mainNode.get("name");
pojo.setNodename(nameNode.toString());
// and 20 more fields
mypojoList.add(pojo);
}
if (mainNode.get("isFolder").asBoolean()) {
mypojoList.addAll(recursiveRestCallMethod(mainNode.get("id").toString().replaceAll("\"", "").trim()));
}
}
return mypojoList;
}
return null;
}
Now everytime the json returned has 4193150 characters and it throws exception - Unexpected end-of-input: expected close marker for Object as reported here and some other SO threads (obviously, the json is not complete and valid).
The incomplete json I am getting looks something like:
{"list":{"pagination":{"count":6097,"hasMoreItems":false,"totalItems":6097,"skipCount":0,"maxItems":10000},"entries":[{"entry":{"....
From above, as you can see I should get 6097 objects, but I am getting only 2024 entry array items. And after that json ends abruptly. i.e. invalid json string.
However, for smaller response, where I have 20/30 entry array items, it works as expected.
Note: I am using Spring-Boot 2.4.5 and hence Jackson 2.12.4
Question: Even though I am using .block(), why the response stops at 4193150 characters? What I am doing wrong here?
Not sure what was wrong using String but when I switched to DataBuffer, it worked fine.
Here is the snippet for what I used:
final Flux<DataBuffer> hugeJson = webClient.get()
.uri("/my/rest/api/accepting/" + folderId
+ "/and/producing/huge/jsonresponse/for/all/files/recursively")
.accept(MediaType.ALL)
.retrieve()
.bodyToFlux(DataBuffer.class);
I'm inclined to use CSVRecord because it can be used to map with a header and get the corresponding value. My application frequently uses CSVRecord class. However, I cannot instantiate the CSVRecord. I would prefer not to modify the source/create a new class since it already provides a parser that returns CSVRecord. I have got a list of strings (header as well as the values) that needed to be converted to the CSVRecord type. Is there a direct way that this can be done without going around with formatting and then parsing back? Like the one below:
private CSVRecord format(List<String> header, List<String> values)
{
CSVFormat csvFormat = CSVFormat.DEFAULT.withRecordSeparator(System.lineSeparator())
.withQuoteMode(QuoteMode.ALL);
CSVRecord csvRecord = null;
final StringWriter out = new StringWriter();
try (CSVPrinter csvPrinter = new CSVPrinter(out, csvFormat);)
{
csvPrinter.printRecord(values);
String value = out.toString().trim();
for (CSVRecord r : CSVParser.parse(value, csvFormat.withHeader(header.toArray(new String[header.size()]))))
csvRecord = r;
}
catch (IOException e)
{
logger.error("Unable to format the Iterable to CSVRecord. Header: [{}]; Values: [{}]", e,
String.join(", ", header), String.join(", ", values));
}
return csvRecord;
}
private void testMethod() throws Exception
{
List<String> header = Arrays.asList("header1", "header2", "header3");
List<String> record = Arrays.asList("val1", "val2", "val3");
CSVRecord csvRecord = format(header, record);
logger.info("{}", csvRecord.get("header2"));
}
You could pass the list as a string directly into the CSVParser instead of creating a writer.
CSVRecord csvr = CSVParser.parse(
values.stream().collect(Collectors.joining(","))
,csvFormat.withHeader(header.toArray(new String[header.size()])))
.getRecords().get(0);
The BeanIO and SimpleFlatMapper are way better at solving this problem. BeanIO uses a Map data structure and a config file to declare how the CSV file should be structured so it is very powerful. SimpleFlatMapper will take you POJO properties as the heading names by default and output the property values are column values.
BeanIO
http://beanio.org/2.1/docs/reference/index.html#CSVStreamFormat
SimpleFlatMapper
http://simpleflatmapper.org/
CsvParser
.mapTo(MyObject.class)
.stream(reader)
.forEach(System.out::println);
I am trying to write a test case where I want to stream json objects from a json file separated by new line into Java.
I want to stream one event object in Java and serialize it.
The json file is of the form:
{"event":[{"D49-64":0,"Bezeichnung":"A 41","D33-48":0}]}
{"event":[{"D49-64":1,"Bezeichnung":"A 41","D33-48":0}]}
Any suggestions to stream the objects in Java will be beneficial.
The blob that you have posted is not a valid JSONObject, but two individual objects.
To stream this, you would end up with something like the following:
String pathToFile = "/path/to/something.txt";
BufferedReader someReader = new BufferedReader( new FileReader( pathToFile ));
String someData;
while (( someData = someReader.readLine() ) != null ) {
JSONObject o = new JSONObject( someData );
doSomethingWith( o );
}
The library I generally use for JSON manipulation is org.json
I was solving the same problem: reading data from file which just has sequence of json objects in it. I am using com.fasterxml.jackson library for json manipulation. While it does not have direct methods for exactly this, the solution is still quite simple:
// InputStream in - input stream with your data
ObjectMapper mapper = new ObjectMapper();
JsonParser parser = mapper.getFactory().createParser(in);
ObjectNode nextObject;
do {
nextObject = mapper.readTree(parser); // returns null when end of stream is reached
// process your object here
} while(nextObject != null);
I am trying to convert a Json string into a generic Java Object, with an Avro Schema.
Below is my code.
String json = "{\"foo\": 30.1, \"bar\": 60.2}";
String schemaLines = "{\"type\":\"record\",\"name\":\"FooBar\",\"namespace\":\"com.foo.bar\",\"fields\":[{\"name\":\"foo\",\"type\":[\"null\",\"double\"],\"default\":null},{\"name\":\"bar\",\"type\":[\"null\",\"double\"],\"default\":null}]}";
InputStream input = new ByteArrayInputStream(json.getBytes());
DataInputStream din = new DataInputStream(input);
Schema schema = Schema.parse(schemaLines);
Decoder decoder = DecoderFactory.get().jsonDecoder(schema, din);
DatumReader<Object> reader = new GenericDatumReader<Object>(schema);
Object datum = reader.read(null, decoder);
I get "org.apache.avro.AvroTypeException: Expected start-union. Got VALUE_NUMBER_FLOAT" Exception.
The same code works, if I don't have unions in the schema.
Can someone please explain and give me a solution.
For anyone who uses Avro - 1.8.2, JsonDecoder is not directly instantiable outside the package org.apache.avro.io now. You can use DecoderFactory for it as shown in the following code:
String schemaStr = "<some json schema>";
String genericRecordStr = "<some json record>";
Schema.Parser schemaParser = new Schema.Parser();
Schema schema = schemaParser.parse(schemaStr);
DecoderFactory decoderFactory = new DecoderFactory();
Decoder decoder = decoderFactory.jsonDecoder(schema, genericRecordStr);
DatumReader<GenericData.Record> reader =
new GenericDatumReader<>(schema);
GenericRecord genericRecord = reader.read(null, decoder);
Thanks to Reza. I found this webpage.
It introduces how to convert a Json string into an avro object.
http://rezarahim.blogspot.com/2013/06/import-org_26.html
The key of his code is:
static byte[] fromJsonToAvro(String json, String schemastr) throws Exception {
InputStream input = new ByteArrayInputStream(json.getBytes());
DataInputStream din = new DataInputStream(input);
Schema schema = Schema.parse(schemastr);
Decoder decoder = DecoderFactory.get().jsonDecoder(schema, din);
DatumReader<Object> reader = new GenericDatumReader<Object>(schema);
Object datum = reader.read(null, decoder);
GenericDatumWriter<Object> w = new GenericDatumWriter<Object>(schema);
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
Encoder e = EncoderFactory.get().binaryEncoder(outputStream, null);
w.write(datum, e);
e.flush();
return outputStream.toByteArray();
}
String json = "{\"username\":\"miguno\",\"tweet\":\"Rock: Nerf paper, scissors is fine.\",\"timestamp\": 1366150681 }";
String schemastr ="{ \"type\" : \"record\", \"name\" : \"twitter_schema\", \"namespace\" : \"com.miguno.avro\", \"fields\" : [ { \"name\" : \"username\", \"type\" : \"string\", \"doc\" : \"Name of the user account on Twitter.com\" }, { \"name\" : \"tweet\", \"type\" : \"string\", \"doc\" : \"The content of the user's Twitter message\" }, { \"name\" : \"timestamp\", \"type\" : \"long\", \"doc\" : \"Unix epoch time in seconds\" } ], \"doc:\" : \"A basic schema for storing Twitter messages\" }";
byte[] avroByteArray = fromJsonToAvro(json,schemastr);
Schema schema = Schema.parse(schemastr);
DatumReader<Genericrecord> reader1 = new GenericDatumReader<Genericrecord>(schema);
Decoder decoder1 = DecoderFactory.get().binaryDecoder(avroByteArray, null);
GenericRecord result = reader1.read(null, decoder1);
With Avro 1.4.1, this works:
private static GenericData.Record parseJson(String json, String schema)
throws IOException {
Schema parsedSchema = Schema.parse(schema);
Decoder decoder = new JsonDecoder(parsedSchema, json);
DatumReader<GenericData.Record> reader =
new GenericDatumReader<>(parsedSchema);
return reader.read(null, decoder);
}
Might need some tweaks for later Avro versions.
As it was already mentioned here in the comments, JSON that is understood by AVRO libs is a bit different from a normal JSON object. Specifically, UNION type is wrapped into a nested object structure: "union_field": {"type": "value"}.
So if you want to convert "normal" JSON to AVRO you'll have to use 3rd-party library. For now at least.
https://github.com/allegro/json-avro-converter - Java project that claims to support unions, not sure about default values.
https://github.com/agolovenko/json-to-avro-converter - this is my project, although written in Scala, still usable from Java. Supports unions, default values, base64 binary data...
Your schema does not match the schema of the json string. You need to have a different schema that does not have a union in the place of the error but a decimal number. Such schema should then be used as a writer schema while you can freely use the other one as the reader schema.
Problem is not the code, but the wrong format of the json
String json = "{"foo": {"double": 30.1}, "bar": {"double": 60.2}}";
i need to access the yaml file data into the java file.
i used the YamlReader class and now the yaml file is loaded into the java class object.
now all the information is in object and i want to extract it from this object.How can i do this .
Can any one help me please i am stuck with this problem.
You can read into a Map:
YamlReader reader = new YamlReader(new FileReader("contact.yml"));
Object object = reader.read();
System.out.println(object);
Map map = (Map)object;
System.out.println(map.get("address"));
Source: http://code.google.com/p/yamlbeans/
Yaml yaml = new Yaml();
String document = "\n- Hesperiidae\n- Papilionidae\n- Apatelodidae\n- Epiplemidae";
List<String> list = (List<String>) yaml.load(document);
System.out.println(list);
Using snakeyaml, the code below didn't work for me. When the yaml object returned my object, it returned a LinkedHashMap type. So I cast my returned object into a LinkedHashMap.
public class YamlConfig {
Yaml yaml = new Yaml();
String path = "blah blah";
InputStream input = new FileInputStream(new File(this.path));
LinkedHashMap<String,String> map;
#SuppressWarnings("unchecked")
private void loadHashMap() throws IOException {
Object data = yaml.load(input);// load the yaml document into a java object
this.map = (LinkedHashMap<String,String>)data;
System.out.println(map.entrySet()); //use this to look at your hashmap keys
System.out.println(map);//see the entire map
}