I want to modify a json content without converting it into a POJO. I am using GSON Library.
Following are the use case:
String jsonString = "[{\"key1\":\"Hello\",\"key2\":\"World\"},{\"key1\":\"Nice\",\"key2\":\"Town\"}]";
JsonElement jsonElement = gson.fromJson(jsonString, JsonElement.class);
Is there any way where I can set value of key1 to some value (let say "Test") in each array, without converting things into POJO
Here's the shortest I came up with.
JsonElement je = new Gson().fromJson(jsonString, JsonElement.class);
JsonObject jo = je.getAsJsonObject();
jo.add("key", value);
Once you have the JsonObject, gson has many methods to manipulate it.
You can always get a different type than JsonElement, or use JsonElement.getAsJsonObject to cast to an Object (if possible).
String jsonString = "[{\"key1\":\"Hello\",\"key2\":\"World\"}, ...]";
JsonArray jsonArray = gson.fromJson(jsonString, JsonElement.class).getAsJsonArray();
JsonObject firstObject = jsonArray.get(i).getAsJsonObject();
firstObject.addProperty("key1", "Test");
I was wrong earlier; there seems to be no JsonArray adapter; you'll have to get a JsonElement and use the casting tool.
One approach would be to just convert the JSON to a java.util.Map, modify the Map, and go from there (which may mean serializing the Map back to JSON).
This approach meets my preference to work with the right API for the right job, minimizing the use of tools like Gson to just handle serialization/deserialization (which is what I understand it was designed for). That is, to not use the Gson API as a replacement data structure.
GSON has two separate APIs (that can be combined): one is used for serialization and deserialization, and the other for streaming. If you want to process streams of JSON without memory overhead or using dynamic structures (rather than static POJOs) you can do something like:
create a JsonWriter (in my example I use StringWriter);
create a JsonReader;
make a loop that consumes events from the reader and feeds them to the writer, possibly making changes, additions, omissions etc.
The loop will consist of a single switch statement that must have a case all the possible events (10 of them). Even the simplest example must have all of them, so the code below looks rather verbose. But it is very easy to extend and further extensions will not make it much longer.
An example that appends "test": 1 pair to each object looks something like:
public class Whatever {
static void streamandmodify(JsonReader reader, JsonWriter writer) throws IOException {
while (true) {
JsonToken token = reader.peek();
switch (token) {
// most cases are just consume the event
// and pass an identical one to the writer
case BEGIN_ARRAY:
reader.beginArray();
writer.beginArray();
break;
case END_ARRAY:
reader.endArray();
writer.endArray();
break;
case BEGIN_OBJECT:
reader.beginObject();
writer.beginObject();
// this is where the change happens:
writer.name("test");
writer.value(1);
break;
case END_OBJECT:
reader.endObject();
writer.endObject();
break;
case NAME:
String name = reader.nextName();
writer.name(name);
break;
case STRING:
String s = reader.nextString();
writer.value(s);
break;
case NUMBER:
String n = reader.nextString();
writer.value(new BigDecimal(n));
break;
case BOOLEAN:
boolean b = reader.nextBoolean();
writer.value(b);
break;
case NULL:
reader.nextNull();
writer.nullValue();
break;
case END_DOCUMENT:
return;
}
}
}
public static void main(String[] args) throws IOException {
// just for test:
JsonReader jr = new JsonReader(new StringReader("{\"a\":1, \"b\":{\"c\":[1,2,3,{},{}]}}"));
StringWriter sw = new StringWriter();
JsonWriter jw = new JsonWriter(sw);
streamandmodify(jr, jw);
System.out.println(sw.getBuffer().toString());
}
}
The jsonString is a plain, ordinary Java String; so you can modify it whatever you like using the standards String functions of Java and replace the substring key1 with Test1:
jsonString = "[{\"key1\":\"Test\",\"key2\":\"World\"},{\"key1\":\"Nice\",\"key2\":\"Town\"}]";
Of course, String in Java are immutable so converting it first to a StringBuilder will possibly give you a better performance in term of memory usage.
Modify json with GSON JsonArray Java 8
Example of how to use GSON to modify a value within a JSON
import com.google.gson.Gson;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
public class ModifyJson {
public static void main(String[] args) {
String data = "[{\"ct_pk\":24,\"ct_name\":\"SISTEMA DE PRUEBAS\"},"
+ "{\"ct_pk\":1,\"ct_name\":\"CAPITAL FEDERAL\"}," +
"{\"ct_pk\":5,\"ct_name\":\"SISTEMA DE PRUEBAS DOS\"}]";
System.out.println("before................." + data);
JsonArray jsonArray = new Gson().fromJson(data, JsonElement.class).getAsJsonArray();
JsonArray jsonArray2 = new JsonArray();
for (JsonElement pa : jsonArray) {
JsonObject jsonObject2 = pa.getAsJsonObject();
String ct_name = jsonObject2.get("ct_name").getAsString();
if (ct_name.equals("SISTEMA DE PRUEBAS")) {
jsonObject2.addProperty("ct_name", "TODOS");
}
jsonArray2.add(jsonObject2);
}
System.out.println("after.................." +jsonArray2);
}
}
Related
My data is of newline delimited json form and looks like shown below. I am reading this type of data from a Kafka topic.
{"sender":"S1","senderHost":"ip-10-20-30-40","timestamp":"2018-08-13T16:17:12.874Z","topic":"test","messageType":"type_1","data":{"name":"John Doe", "id":"12DROIY321"}}
I want to build an apache Beam pipeline which reads this data from Kafka, parses this json format to give me an output as shown below:
S1,2018-08-13T16:17:12.874Z,type_1,12DROIY321
The output is basically a comma delimited string consisting of the sender, timestamp, messageType and id from within data.
My code so far is as below:
public class Pipeline1{
public static void main(String[] args){
PipelineOptions options = PipelineOptionsFactory.create();
// Create the Pipeline object with the options we defined above.
Pipeline p = Pipeline.create(options);
p.apply(KafkaIO.<Long, String>read()
.withBootstrapServers("localhost:9092")
.withTopic("test")
.withKeyDeserializer(LongDeserializer.class)
.withValueDeserializer(StringDeserializer.class)
.updateConsumerProperties(ImmutableMap.of("auto.offset.reset", (Object)"earliest"))
// We're writing to a file, which does not support unbounded data sources. This line makes it bounded to
// the first 35 records.
// In reality, we would likely be writing to a data source that supports unbounded data, such as BigQuery.
.withMaxNumRecords(35)
.withoutMetadata() // PCollection<KV<Long, String>>
)
.apply(Values.<String>create())
.apply(TextIO.write().to("test"));
p.run().waitUntilFinish();
}
}
I am unable to figure out how to parse the json to get the required csv format within the pipeline. Using the code above, I am able to write the same json lines into a file, and using the code below, i can parse the json, but can anyone please help me figure out how to accomplish this as an additional step with the beam pipeline logic?
JSONParser parser = new JSONParser();
Object obj = null;
try {
obj = parser.parse(strLine);
} catch (ParseException e) {
e.printStackTrace();
}
JSONObject jsonObject = (JSONObject) obj;
String sender = (String) jsonObject.get("sender");
String messageType = (String) jsonObject.get("messageType");
String timestamp = (String) jsonObject.get("timestamp");
System.out.println(sender+","+timestamp+","+messageType);
According to the documentation, you will need to write a transformation (or find one that matches your use case).
https://beam.apache.org/documentation/programming-guide/#composite-transforms
The documentation also provides an excellent example.
Example that should produce your output:
.apply(Values.<String>create())
.apply(
"JSONtoData", // the transform name
ParDo.of(new DoFn<String, String>() { // a DoFn as an anonymous inner class instance
#ProcessElement
public void processElement(#Element String word, OutputReceiver<String> out) {
JSONParser parser = new JSONParser();
Object obj = null;
try {
obj = parser.parse(strLine);
} catch (ParseException e) {
e.printStackTrace();
}
JSONObject jsonObject = (JSONObject) obj;
String sender = (String) jsonObject.get("sender");
String messageType = (String) jsonObject.get("messageType");
String timestamp = (String) jsonObject.get("timestamp");
out.output(sender+","+timestamp+","+messageType);
}
}));
To return CSV values, just change the generics to:
new DoFn<String, YourCSVClassHere>()
OutputReceiver<YourCSVClassHere> out
I didn't test this code, use at own risk.
I'm having a problem with Json file reading and writing. I want to append something into a json file but it doesn't work properly: it just put in a new jsonobject without the ',' to divide it from the previous one. I searched everywhere, on every site, but nothing that gave me an input on how to do it properly.
For example, I have a json file like this:
{
"Example":{
"Ok":"Ok1",
"Nice":"Nice1",
"Hi":"Hi1",
"Hello":"Hello1",
"Right":"Right1",
"Wow":"Wow1"
}
}
And I want to make it appear like this:
{
"Example":{
"Ok":"Ok1",
"Nice":"Nice1",
"Hi":"Hi1",
"Hello":"Hello1",
"Right":"Right1",
"Wow":"Wow1"
},
"Example1":{
"Ok":"Ok2",
"Nice":"Nice2",
"Hi":"Hi2",
"Hello":"Hello2",
"Right":"Right2",
"Wow":"Wow2"
}
}
So, I tried using this code:
Gson gson = new GsonBuilder().setPrettyPrinting().create();
JsonObject jsonObject = new JsonObject();
JsonObject dati = new JsonObject();
dati.addProperty("Cognome", StringUtils.capitalize((fields[0].getText())));
dati.addProperty("Nome", StringUtils.capitalize((fields[1].getText())));
dati.addProperty("Sesso", lblSesso.getText());
dati.addProperty("Luogo di nascita", StringUtils.capitalize((fields[2].getText())));
dati.addProperty("Provincia", lblProvincia.getText());
dati.addProperty("Data di nascita", fieldDDN.getText());
jsonObject.add(codfis, dati);
String json = gson.toJson(jsonObject);
try (BufferedReader br = new BufferedReader(new FileReader("CodFisCalcolati.json"));
BufferedWriter bw = new BufferedWriter(new FileWriter("CodFisCalcolati.json", true))) {
String jsonString = gson.fromJson(br, JsonElement.class).toString();
JsonElement jelement = new JsonParser().parse(jsonString);
JsonObject jobject = jelement.getAsJsonObject();
jobject.add(codfis, dati);
String resultingJson = gson.toJson(jelement);
bw.write(resultingJson);
bw.close();
} catch (IOException e1) { e1.printStackTrace(); }
But when I use it, it give me this output :
{
"Example":{
"Ok":"Ok1",
"Nice":"Nice1",
"Hi":"Hi1",
"Hello":"Hello1",
"Right":"Right1",
"Wow":"Wow1"
}
}{
"Example":{
"Ok":"Ok1",
"Nice":"Nice1",
"Hi":"Hi1",
"Hello":"Hello1",
"Right":"Right1",
"Wow":"Wow1"
},
"Example1":{
"Ok":"Ok2",
"Nice":"Nice2",
"Hi":"Hi2",
"Hello":"Hello2",
"Right":"Right2",
"Wow":"Wow2"
}
}
That's output, you see, it'wrong and i don't know how to make the code to give me a different output.
I'm using Gson 2.8.5 and I would rather not change to another library.
You change the question but now the answer to your new question is you use the same file to read and write. That's why you add the data inside ot the file. Change the name of the file that you write and see if you have problems
Please check if "br" is not null.
According to the specification of the method fromJson it returns:
an object of type T from the string. Returns null if json is null.
If this is the case than you call on the null toString() method and you get null pointer exception
I am a beginner with JAVA and are using the gson library to convert a JSON string something like this:
String json = "{\"Report Title\": \"Simple Embedded Report Example with Parameters\",\"Col Headers BG Color\": \"yellow\",\"Customer Names\":[\"American Souvenirs Inc\",\"Toys4GrownUps.com\",\"giftsbymail.co.uk\",\"BG&E Collectables\",\"Classic Gift Ideas, Inc\"]}";
Gson gson = new Gson();
jsonObject (Map) = gson.fromJson(json, Object.class);
But the problem is I need the "Customer Names" array to be returned as a string array and not an object array.
Can gson do this or would it have to be converted afterwards by somehow detecting the type (array) and then looping over each element converting it to a string array and replacing the object array ?
The added problem is that the JSON field names are not fixed, and there may be multiple arrays contained in the JSON string and all of them need converting.
you can use jsonarray to get specific field
you can find json api JSON
add json text into file or you can use buffer to pass to parameter
String json = "{\"customer_names\" : [...]}";
then
JSONParser parser = new JSONParser();
Object obj = parser.parse(new FileReader("test.json"));
JSONObject jsonObject = (JSONObject) obj;
JSONArray msg = (JSONArray) jsonObject.get("Customer Names");
Iterator<String> iterator = msg.iterator();
while (iterator.hasNext()) {
System.out.println(iterator.next());
}
or you can use GSON something like this
public class JsonPojo {
private String[] customer_names;
public String[] getCustomerNames(){ return this.customer_names;}
}
public class App{
public static main(String[] args) {
Gson gson = new Gson();
JsonPojo thing = gson.fromJson(json, JsonPojo.class);
if (thing.getCustomerNames()!= null)
{
// do what you want
}
}
}
I have a Array with some value
when i store that array i got result like this
[{"id":56678,"Name":"Rehman Agarwal"},{"id":66849,"Name":"Rasul Guha"}]
means in a single line.
but I just want to get output like
[{"id":56678,"Name":"Rehman Agarwal"},
{"id":66849,"Name":"Rasul Guha"}]
new line after JSON object ..
How to do it ?
EDIT : i use JSONObject && JSONArray for create json Object and array respectively
Using gson library you can pretty print your json strings like below -
public static String toPrettyFormat(String jsonString) {
JsonParser parser = new JsonParser();
JsonObject json = parser.parse(jsonString).getAsJsonObject();
Gson gson = new GsonBuilder().setPrettyPrinting().create();
String prettyJson = gson.toJson(json);
return prettyJson;
}
And call it like below -
public void testPrettyPrint() {
String compactJson = "{\"playerID\":1234,\"name\":\"Test\",\"itemList\":[{\"itemID\":1,\"name\":\"Axe\",\"atk\":12,\"def\":0},{\"itemID\":2,\"name\":\"Sword\",\"atk\":5,\"def\":5},{\"itemID\":3,\"name\":\"Shield\",\"atk\":0,\"def\":10}]}";
String prettyJson = toPrettyFormat(compactJson);
System.out.println("Compact:\n" + compactJson);
System.out.println("Pretty:\n" + prettyJson);
}
My Question is similar to what has been asked here .
few points :
I can not change the format. (No commas to be added etc)
This is basically a huge .txt file containing 1000's of Json objects.
My Json objects are HUGE.
This is what I am doing right now :
FileReader fileReader = new FileReader(fileName);
BufferedReader reader = new BufferedReader(fileReader);
String data = "";
while((data = reader.readLine()) != null){
ObjectMapper mapper = new ObjectMapper();
Map<String,String> map = mapper.readValue(data, Map.class);
}
Currently I am using Jackson and Ideally I would like to read one Json Object from the file at a time, Parse it and then move on to the next one. I need to count let say unique number of id's from these Json object and do more operations. It will be best to read them one by one.
Is jackson would be the best way going forward ?
This is a good example of parsing huge Json, But it deals with only one object per file. My file has huge Jsons (1000s of them).
Here is a Jackson example that works for me. I have thousands json objects (tokens) in a single json file. This code will iterate through the file read each token and print it's serial.
Required imports:
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.databind.ObjectMapper;
Using Jackson to read multiple json objects from FileInputStream:
try (FileInputStream fis = new FileInputStream("D:/temp/tokens.json")) {
JsonFactory jf = new JsonFactory();
JsonParser jp = jf.createParser(fis);
jp.setCodec(new ObjectMapper());
jp.nextToken();
while (jp.hasCurrentToken()) {
Token token = jp.readValueAs(Token.class);
jp.nextToken();
System.out.println("Token serial "+token.getSerialNumber());
}
}
Here is a more JAVA 8ish solution for your query, I always lean toward BufferedReader over InputStreams for any place where parsing is going to be done a lot of time.
ObjectMapper mapper = new ObjectMapper();
JsonFactory jsonFactory = new JsonFactory();
try(BufferedReader br = new BufferedReader(new FileReader("luser.txt"))) {
Iterator<luser> value = mapper.readValues( jsonFactory.createParser(br), luser.class);
value.forEachRemaining((u)->{System.out.println(u);});
}
The deserialization for each object happens as part of next(), in each iteration.
Here is how I used Gson's JSONReader API to handle similar requirement as above
public static List<YOURPOJO> readTraceLog(String filepath) throws IOException {
Gson gson = new Gson();
JsonReader jsonReader = new JsonReader(new FileReader(filepath));
// important as handles unwanted formatting stuffs such empty spaces
jsonReader.setLenient(true);
boolean start = true; // start of read
jsonReader.beginObject(); // first object begins
//List to hold object
List<YOURPOJO> completeList = new ArrayList<YOURPOJO>();
//YOURPOJO has two attributes one is ID and other is list of ANOTHERPOJO
while (jsonReader.hasNext()) {
if (!start) {
//to stop end of Document
if (jsonReader.peek().toString().matches("END_DOCUMENT")) {
break;
}
//continue reading object as the come -{
jsonReader.beginObject();
}
start = false;
YOURPOJO pojo = new YOURPOJO();
//read attribute id
String name = jsonReader.nextName();
pojo.setId(name);
//list to store ANOTHERPOJO objects
List<ANOTHERPOJO> tempList = new ArrayList<ANOTHERPOJO>();
//begin reading list - [
jsonReader.beginArray();
while (jsonReader.hasNext()) {
ANOTHERPOJO t = gson.fromJson(jsonReader, ANOTHERPOJO.class);
tempList.add(t);
}
//end reading list - ]
jsonReader.endArray();
//store data
pojo.setTraceDetails(tempList);
completeList.add(YOURPOJO);
//end of object - }
jsonReader.endObject();
}
jsonReader.close();
return completeList;
}