I have a csv structure like this
and I also have one json response
[
{
"ID" : "1",
"Name" : "abc",
"Mobile" : "123456"
},
{
"ID" : "2",
"Name" : "cde",
"Mobile" : "123345"
}
]
I need the output like this
If your intention is to convert directly the JSON then that baeldung solution that you were given is good.
Otherwise, the way i see it and based on the info you're giving, you will need to have a representation of that JSON in a java object that will either represent some kind of request coming from somewhere or data you're getting from your database in order to be written on a csv.
Check out these, might be useful:
https://www.codejava.net/frameworks/spring-boot/csv-export-example
https://zetcode.com/springboot/csv/
Related
I'm building a Java Jersey API which uses MongoDb and MongoDb driver.
The resources should output JSON of the stored MongoDb document to be used in the frontend project using Svelte.
Due to the standard org.bson.Document.toJson() implementation the output of my documents look somehow like:
[{ "_id" : { "$oid" : "5e97f08f2175aa9174dbec0e" }, "hour" : 8, "minute" : 15, "enabled" : true, "duration" : 120 }
I would rather like it to be:
[{ "_id" : "5e97f08f2175aa9174dbec0e", "hour" : 8, "minute" : 15, "enabled" : true, "duration" : 120 }
That way it's easier to handle the id in the frontend. So how to get rid of the $oid object?
I already managed to get the format as I wish by using:
JsonWriterSettings settings = JsonWriterSettings.builder()
.outputMode(JsonMode.RELAXED)
.objectIdConverter((value, writer) -> writer.writeString(value.toHexString()))
.build();
System.out.println(doc.toJson(settings));
But how to register this setting object globally so that every doc.toJson() call will use it?
And what will happen if I send modified or new documents from the frontend to the API and do:
Document document = Document.parse(doc);
Is my modified _id field automatically converted again to an ObjectId? Or do I need a org.bson.codecs.Decoder or CodecRegistry? How would this be done?
$oid refers to ObjectId field type in bson spec. As far as I know, you need to manipulate your document to replace ObjectId for your _id into String.
String oidAsString = document.getObjectId("_id").toString();
document.put("_id", oidAsString);
I'd like to be able to query a JSON object and modify it in a way that is structure-agnostic and doesn't involve marshalling to POJOs. Here's an example to illustrate.
{
"user" : [ {
"username" : "foo",
"user-id" : "1234",
"name" : {
"first-name" : "John",
"last-name" : "Smith"
},
"ssn": "123-45-6789"
}, {
"username" : "bar",
"user-id" : "1235",
"name" : {
"first-name" : "Jane",
"last-name" : "Doe"
},
"ssn": "098-76-5432"
} ]
}
What I want to do is this:
Get the ssn nodes, retrieve their values, encrypt them, put the encrypted values back in the JSON, and return the whole thing. I've found two ways to do this so far:
Marshalling to a User POJO
Writing a method that iterates over the "user"s directly and modifies the ssn value.
The problem with either approach is that I have to generate "user" specific code. This is fine when I have only one data format, but I'm going to have dozens. I want to work more agnostically. Ideally I can do this with a line like this:
List<JsonNode> ssnNodes = jsonObj.match("$.user[*].ssn");
and then just iterate over the list -- just as I can with XML using XPath. This way I can maintain a list of json-paths to query with and that is as much as I need to know about the data.
Someone please tell me there is a way to do this in Java, but I haven't found a way so far. Thanks in advance!
I have an Apache Beam streaming job which reads data from Kafka and writes to ElasticSearch using ElasticSearchIO.
The issue I'm having is that messages in Kafka already have key field, and using ElasticSearchIO.Write.withIdFn() I'm mapping this field to document _id field in ElasticSearch.
Having a big volume of data I don't want the key field to be also written to ElasticSearch as part of _source.
Is there an option/workaround that would allow doing that?
Using the Ingest API and the remove processor you´ll be able to solve this pretty easy only using your elasticsearch cluster. You can also simulate ingest pipeline and the results.
I´ve prepared a example which will probably cover your case:
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description": "remove id form incoming docs",
"processors": [
{"remove": {
"field": "id",
"ignore_failure": true
}}
]
},
"docs": [
{"_source":{"id":"123546", "other_field":"other value"}}
]
}
You see, there is one test document containing a filed "id". This field is not present in the response/result anymore:
{
"docs" : [
{
"doc" : {
"_index" : "_index",
"_type" : "_type",
"_id" : "_id",
"_source" : {
"other_field" : "other value"
},
"_ingest" : {
"timestamp" : "2018-12-03T16:33:33.885909Z"
}
}
}
]
}
I've created a ticket in Apache Beam JIRA describing this issue.
For now the original issue can not be resolved as part of indexation process using Apache Beam API.
The workaround that Etienne Chauchot, one of the maintainers, proposed is to
have separate task which will clear indexed data afterwords.
See Remove a field from a Elasticsearch document for example.
For the future, if someone also would like to leverage such feature, you might want to follow the linked ticket.
I have document schema such as
{
"_id" : 18,
"name" : "Verdell Sowinski",
"scores" : [
{
"type" : "exam",
"score" : 62.12870233109035
},
{
"type" : "quiz",
"score" : 84.74586220889356
},
{
"type" : "homework",
"score" : 81.58947824932574
},
{
"type" : "homework",
"score" : 69.09840625499065
}
]
}
I have a solution using pull that copes with removing a single element at a time but saw
I want to get a general solution that would cope with irregular schema where there would be between one and many elements to the array and I would like to remove all elements based on a condition.
I'm using mongodb driver 3.2.2 and saw this pullByFilter which sounded good
Creates an update that removes from an array all elements that match the given filter.
I tried this
Bson filter = and(eq("type", "homework"), lt("score", highest));
Bson u = Updates.pullByFilter(filter);
UpdateResult ur = collection.updateOne(studentDoc, u);
Unsurprisingly, this did not have any effect since I wasn't specifying the array scores
I get an error
The positional operator did not find the match needed from the query. Unexpanded update: scores.$.type
when I change the filter to be
Bson filter = and(eq("scores.$.type", "homework"), lt("scores.$.score", highest));
Is there a one step solution to this problem?
There seems very little info on this particular method I can find. This question may relate to How to Update Multiple Array Elements in mongodb
After some more "thinking" (and a little trial and error), I found the correct Filters method to wrap my basic filter. I think I was focusing on array operators too much.
I'll not post it here in case of flaming.
Clue: think "matches..." (as in regex pattern matching) when dealing with Filters helper methods ;)
This question has been asked many times but I couldn't find the answer that fixes my issue.
I'm trying to convert nested JSON format to CSV format like this :
The JSON structure is arbitrary and could be anything, nested or not.
I'm not suppose to know it, it's a database answer and I need to export this JSON answer into CSV file.
Here is an example
Input :
{
"_id": 1,
"name": "Aurelia Menendez",
"scores": [
{
"type": "exam",
"score": 60.06045071030959
},
{
"type": "quiz",
"score": 52.79790691903873
},
{
"type": "homework",
"score": 71.76133439165544
}
]
}
The output I'm looking for :
_id,name,scores.type,scores.score,scores.type,scores.score,scores.type,scores.score
1,Aurelia Menendez,exam,60.06...,quiz,52.79...,homework,71.76..
This is an example, it could be any other JSON document.
The idea here is to use dot notation in the CSV column name.
I've already used CDL but the output is not what I want :
_id scores name
1 "[{score:60.06045071030959,type:exam},{score:52.79790691903873,type:quiz},{score:71.76133439165544,type:homework}]" Aurelia Menendez
So how can I convert nested JSON to CSV with dot notation and in a generic way ?
Edits
Deserialisation of the JSON with Jackson :
ObjectMapper mapper=new ObjectMapper();
JsonNode jsonNode=mapper.readValue(new File("C:\\...\\...\...\\test.json"), JsonNode.class);
Ismail
Like you said :
The JSON structure is arbitrary and could be anything, nested or not.
The JSON to CSV conversion can't be generalized as it varies from user to user and also depends specific requirements.
But still there's a library json2flat which tries to achieve it. But it may differ from user's requirement. Still it's worth a try.
For example for the JSON given above:
{
"_id": 1,
"name": "Aurelia Menendez",
"scores": [
{
"type": "exam",
"score": 60.06045071030959
},
{
"type": "quiz",
"score": 52.79790691903873
},
{
"type": "homework",
"score": 71.76133439165544
}
]
}
can be interpreted as follows :
/_id,/name,/scores/type,/scores/score
1,"Aurelia Menendez","exam",60.06045071030959
1,"Aurelia Menendez","quiz",52.79790691903873
1,"Aurelia Menendez","homework",71.76133439165544
Converting JSON to XLS/CSV in Java has what you are looking for.
Basically, you need to use org.json.CDL to convert from JSON to CSV format
Comments are not convenient place to post longs answers, so I post my answer here.
Analyze your JSON and all possible JSON structures you can get from your database. It should be a limited number of JSON forms.
As you have analyzed your JSON structure build a class/class hierarchy, that fully reflects this structure.
Use JSON serializer/deserializer library at your choice, to deserialize JSON to a java object.
Employ StringBuffer/StringBuilder classes, and iterate over your object information, and build comma delimited (or tab-delimited) strings.
Write strings you have built on the previous stage to the file.
That's it.