I have a JSON array that I'd like to map that looks like this:
{
"library": [
{
"key":"val"
},
{
"key":"val"
}
]
}
Is there a way to parse this using the object mapper starting at the array rather than at the root? I know you can do a manual node parse, but I would prefer not to do that if possible. any help with this would be greatly appreciated.
Jackson offers three principal ways to parse json: to a map, to an object, to a jackson node tree. None of these methods offer a way to start from anywhere other than the root. To start from somewhere other than the root, you need to parse your way to there from the root, which means you need to start parsing from the root! :)
That being said, if for example you use mapping to an object, it is very easy to get the array you need out of the object:
package test;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Map;
import org.codehaus.jackson.JsonParseException;
import org.codehaus.jackson.map.JsonMappingException;
import org.codehaus.jackson.map.ObjectMapper;
public class Test {
static String json = "{\"library\": [{\"key\":\"val\"},{\"key\":\"val\"}]}";
static class JsonClass {
private ArrayList<Map<?,?>> library;
public ArrayList<Map<?, ?>> getLibrary() {
return library;
}
public void setLibrary(ArrayList<Map<?, ?>> library) {
this.library = library;
}
}
public static void main(String[] args)
throws JsonParseException, JsonMappingException, IOException {
JsonClass parsed = new ObjectMapper().readValue(json, Test.JsonClass.class);
System.out.println(parsed.getLibrary());
}
}
Running this prints:
[{key=val}, {key=val}]
An alternative would be to use a streaming parser... it can pick any node, without bothering about understanding the whole structure. I believe Gson has that. But in your case it would probably be an overkill to use a streaming parser: it makes sense when the overall structure is complex, you need to process a big stream fast, and are interested in relatively small part of the data. These do not seem to apply to your scenario.
Related
I'm teaching myself Apache Beam, specifically for using in parsing JSON. I was able to create a simple example that parsed JSON to a POJO and POJO to CSV. It required that I use .setCoder()
for my simple POJO class.
pipeline
.apply("Read source JSON file.", TextIO.read().from(options.getInput()))
.apply("Parse to POJO matching schema", ParseJsons.of(Person.class))
.setCoder(SerializableCoder.of(Person.class))
.apply("Create comma delimited string", new PersonToCsvRow())
.apply("Write out to file", TextIO.write().to(options.getOutput())
.withoutSharding());
The problem
Now I am trying to skip the POJO step of parsing using some custom transforms. My pipeline looks like this:
pipeline
.apply("Read Json", TextIO.read().from("src/main/resources/family_tree.json"))
.apply("Traverse Json tree", new JSONTreeToPaths())
.apply("Format tree paths", new PathsToCSV())
.apply("Write to CSV", TextIO.write().to("src/main/resources/paths.csv")
.withoutSharding());
This pipeline is supposed to take a heavily nested JSON structure and print each individual path through the tree. I'm getting the same error I did in the POJO example above:
Exception in thread "main" java.lang.IllegalStateException: Unable to return a default Coder for Traverse Json tree/MapElements/Map/ParMultiDo(Anonymous).output [PCollection#331122245]. Correct one of the following root causes:
No Coder has been manually specified; you may do so using .setCoder().
What I tried
So I tried to add a coder in a few different ways:
.setCoder(SerializableCoder.of(List<String>.class))
Results in "Cannot select from parameterized type". I found another instance of this error generated by a different use case here, but the accepted answer seemed only be applicable to that use case.
So then I started perusing the Beam docs and found ListCoder.of() which has (literally) no description. But it looked promising, so I tried it:
.setCoder(ListCoder.of(SerializableCoder.of(String.class)))
But this takes me back to the initial error of not having manually set a coder.
The question
How do I satisfy this requirement to set a coder for a List<String> object?
Code
The transform that is causing the setCoder error is this one:
package transforms;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.values.PCollection;
import java.util.ArrayList;
import java.util.List;
public class JSONTreeToPaths extends PTransform<PCollection<String>, PCollection<List<String>>> {
public static class ExtractPathsFromTree extends SimpleFunction<JsonNode, List<String>> {
public List<String> apply(JsonNode root) {
List<String> pathContainer = new ArrayList<>();
getPaths(root, "", pathContainer);
return pathContainer;
}
}
public static class GetRootNode extends SimpleFunction<String, JsonNode> {
public JsonNode apply(String jsonString) {
try {
return getRoot(jsonString);
} catch (JsonProcessingException e) {
e.printStackTrace();
return null;
}
}
}
#Override
public PCollection<List<String>> expand(PCollection<String> input) {
return input
.apply(MapElements.via(new GetRootNode()))
.apply(MapElements.via(new ExtractPathsFromTree()));
}
private static JsonNode getRoot(String jsonString) throws JsonProcessingException {
ObjectMapper mapper = new ObjectMapper();
return mapper.readTree(jsonString);
}
private static void getPaths(JsonNode node, String currentPath, List<String> paths) {
//check if leaf:
if (node.path("children").isMissingNode()) {
currentPath += node.get("Id");
paths.add(currentPath);
System.out.println(currentPath);
return;
}
// recursively iterate over children
currentPath += (node.get("Id") + ",");
for (JsonNode child : node.get("children")) {
getPaths(child, currentPath, paths);
}
}
}
While the error message seems to imply that the list of strings is what needs encoding, it is actually the JsonNode. I just had to read a little further down in the error message, as the opening statement is a bit deceiving as to where the issue is:
Exception in thread "main" java.lang.IllegalStateException: Unable to return a default Coder for Traverse Json tree/MapElements/Map/ParMultiDo(Anonymous).output [PCollection#1324829744].
...
...
Inferring a Coder from the CoderRegistry failed: Unable to provide a Coder
for com.fasterxml.jackson.databind.JsonNode.
Building a Coder using a registered CoderProvider failed.
Once I discovered this, I solved the problem by extending Beam's CustomCoder class. This abstract class is nice because you only have to write the code to serialize and deserialize the object:
public class JsonNodeCoder extends CustomCoder<JsonNode> {
#Override
public void encode(JsonNode node, OutputStream outStream) throws IOException {
ObjectMapper mapper = new ObjectMapper();
String nodeString = mapper.writeValueAsString(node);
outStream.write(nodeString.getBytes());
}
#Override
public JsonNode decode(InputStream inStream) throws IOException {
byte[] bytes = IOUtils.toByteArray(inStream);
ObjectMapper mapper = new ObjectMapper();
String json = new String(bytes);
return mapper.readTree(json);
}
}
Hopes this helps some other Beam newbie out there.
I have a problem: I need to parse a JSON file in Java where each line represents a tweet and follows the standard JSON of Twitter. I do not need all the information, I attach two photos to show you which fields I need. I would do it without using any support library. Thank you!
This is what I did for now. I do not think it's the best way to do it, especially going ahead I'll be in trouble because the names of many fields repeat
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
public class TweetCorpus implements Iterable<Tweet>
{
private List<Tweet> tweets;
public static TweetCorpus parseFile(File file)
{
List<Tweet> tweets = new ArrayList<>();
try(BufferedReader br = Files.newBufferedReader(file.toPath()))
{
while(br.ready())
{
String tweet = br.readLine();
//System.out.println(tweet);
if(!tweet.isEmpty())
{
long l = Long.parseLong(tweet.substring(tweet.indexOf("\"id\":") + 5, tweet.indexOf(",\"id_str\":")));
String t = tweet.substring(tweet.indexOf(",\"text\":\"") + 9, tweet.indexOf(",\"source\":"));
tweets.add(new Tweet(l, t));
}
}
}
catch(IOException e)
{
e.printStackTrace();
}
return new TweetCorpus(tweets);
}
public int getTweetCount() { return tweets.size(); }
public TweetCorpus(List<Tweet> tweets)
{
this.tweets = tweets;
}
#Override
public Iterator<Tweet> iterator()
{
return tweets.iterator();
}
public static void main(String[] args)
{
TweetCorpus t = parseFile(new File("C:\\Users\\acer\\Desktop\\Moroder\\Uni\\1 Anno - 2 Semestre\\Metodologie Di Programmazione\\Progetto\\HM4Test\\tweetsCorpus.js"));
t.getTweetCount();
}
}
json media/retweet tweet
json "normal" tweet
You can use Gson or Jackson java library to parse json to Tweet object. Their are tools online which generates pojo from json, which you can use with jackson to parse your json string to object.
Once you have json values in an object, you can use getters/setters to extract/modify the values you are interested in from input json.
Well writing your own parser would be a reinventing the wheel kind of task. But if your need is to write your own parser, refer to jackson project on github for inspiration on design and maintenance.
This would help you in making a generic application.
Quick reference for jackson parser ,
https://dzone.com/articles/processing-json-with-jackson
Re-inventing a JSON parser using only readLine() is a really bad idea. If you don't have experience writing parsers by hand, you will end up with a lot of bad code that is really hard to understand. Just use a library. There are tons of good JSON libraries for Java.
Jackson
GSON
Boon
Example code:
static class User {
String id, name;
}
static class MyTweet {
String id, text;
User user;
}
// if the entire file is a JSON array:
void parse(Reader r) {
List<MyTweet> tweets = objectMapper.readValue(
r, new TypeReference<List<MyTweet>>(){});
}
// if each line is a single JSON object:
void parse(BufferedReader r) {
while (r.ready()) {
String line = r.readLine();
MyTweet tweet = objectMapper.readValue(line, MyTweet.class);
}
}
I am trying to convert XML String in the below program to JSON String.
I am able to convert it from file but not from the string.
Any idea about this?
package com.tda.topology;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import org.apache.camel.Exchange;
import org.apache.camel.dataformat.xmljson.XmlJsonDataFormat;
public class Demo2 {
public static void main(String[] args) throws Exception {
String xmlstring = "<soapenv:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:ser=\"http://services.web.post.list.com\"><soapenv:Header><authInfo xsi:type=\"soap:authentication\" xmlns:soap=\"http://list.com/services/SoapRequestProcessor\"><!--You may enter the following 2 items in any order--><username xsi:type=\"xsd:string\">user#email.com</username><password xsi:type=\"xsd:string\">password</password></authInfo></soapenv:Header></soapenv:Envelope>";
XmlJsonDataFormat xmlJsonDataFormat = new XmlJsonDataFormat();
xmlJsonDataFormat.setEncoding("UTF-8");
xmlJsonDataFormat.setForceTopLevelObject(true);
xmlJsonDataFormat.setTrimSpaces(true);
xmlJsonDataFormat.setRootName("newRoot");
xmlJsonDataFormat.setSkipNamespaces(true);
xmlJsonDataFormat.setRemoveNamespacePrefixes(true);
Exchange exchange;
//exchange.setIn(in);
InputStream stream = new ByteArrayInputStream(xmlstring.getBytes(StandardCharsets.UTF_8));
//xmlJsonDataFormat.getSerializer().readFromStream(stream).toString();
//xmlJsonDataFormat.marshal(exchange, graph, stream);
}
}
You need to call start on your xmlJsonDataFormat object and add xom jar to your class path (if it's not already there). This is what worked for me:
xmlJsonDataFormat.start();
String json = xmlJsonDataFormat.getSerializer().readFromStream(stream).toString();
I was able to work this out through looking in the source. getSerialiser was returning null and so I searched in xmlJsonDataFormat for where the serializer was initialised and it's done so by the doStart method which is called in the super class's start method.
Disclaimer: not sure you're supposed to use XmlJsonDataFormat like this, it's usually meant for use in a camel route: from("direct:marshal").marshal(xmlJsonFormat).to("mock:json"); but I don't know your specific use case.
I want to parse XML which looks like below:
<ROOT>
<ECHO>
<column1>NAME</column1>
<column2>LN</column2>
<column3>CD</column3>
<column4>DATA0</column4>
<column5>DATA1</column5>
<column6>DATA2</column6>
<column7>DATA3</column7>
</ECHO>
<ECHO1>
<column1>NAME</column1>
<column2>LN</column2>
<column3>CD</column3>
<column4>DATA0</column4>
</ECHO1>
</ROOT>
Here I want a Map to be returned and somehow the key should be a nd with respect to that all the other children should be there stored in a list
To be precise it should be something like this <key=echo ,List=ln ,cd,data0,data1,data2,data3>.
I tried exploring JAXB but I think that would not help in my case as the name of children is not same for each case so I cannot define static number of fields in any model and unmarshal the XML to model. I want to minimize the coding effort so SAX and DOM parser is never on my radar. Can anyone give any clue to do this without much coding?
Beggs basically asked if you have control over the XML format, that is, can it be slightly modified? You haven't answered that question.
Let's say that it can. The following is a slightly altered version of your original XML:
<ROOT>
<data name="ECHO">
<column>NAME</column>
<column>LN</column>
<column>CD</column>
<column>DATA0</column>
<column>DATA1</column>
<column>DATA2</column>
<column>DATA3</column>
</data>
<data name="ECHO1">
<column>NAME</column>
<column>LN</column>
<column>CD</column>
<column>DATA0</column>
</data>
</ROOT>
This can easily be parsed with JAXB. Create a class for the root of the document, I will call it Root. Create a class for the data element, I will call it Data.
Class Root:
import java.util.List;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;
#XmlRootElement(name="ROOT")
public class Root {
#XmlElement
public List<Data> data;
}
Class Data:
import java.util.List;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.XmlElement;
public class Data {
#XmlAttribute
public String name;
#XmlElement
public List<String> column;
}
And then a class with a main function that will load the XML file from a file called test.xml in the current directory. The data will then be transferred to a Map and the Map is printed to show its content.
import java.io.File;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.Unmarshaller;
public class Test {
public static void main(String[] args) throws Exception {
JAXBContext ctx = JAXBContext.newInstance(Root.class);
Unmarshaller um = ctx.createUnmarshaller();
Root root = (Root) um.unmarshal(new File("test.xml"));
// XML is now loaded. Turn it into a Map
Map<String,List<String>> map = new HashMap<String, List<String>>();
for (Data data : root.data) {
map.put(data.name, data.column);
}
System.out.println(map); // Easily show content
}
}
If you can't control the format of the XML data, this solution will not work.
While I do think the question is a bit strange, and I agree that rethinking the problem in a way such that SAX or DOM can be used is probably wise, you might be able to use a simple XmlPullParser to accomplish this.
Assuming that your XML is always structured according to your example w.r.t the depth of keys, values etc, and assuming that your desired output is a Map<String, List<String>>, you might be able to do something like:
int eventType;
String key = null;
List<String> value = null;
HashMap<String, List<String>> map = new HashMap<String, List<String>>();
while ((eventType = xpp.getEventType()) != XmlPullParser.END_DOCUMENT) {
switch (eventType) {
case XmlPullParser.START_TAG:
if (xpp.getDepth() == 2) {
key = xpp.getName();
value = new ArrayList<String>();
}
break;
case XmlPullParser.TEXT:
if (xpp.getDepth() == 3) {
value.add(xpp.getText());
}
break;
case XmlPullParser.END_TAG:
if (xpp.getDepth() == 2) {
map.put(key, value);
}
break;
}
}
I want to minimize the coding effort so SAX and DOM parser is never on my radar.
I think that you need to recalibrate your radar :-) I suspect that SAX or DOM or similar will be the simplest way to do this.
XML bindings rely on an XML schema of some kind to give them traction. It doesn't look like a schema is possible for your XML.
For what it is worth, you can probably code a DOM-based extractor for this use-case in 50 to 100 lines of Java code. That's not a huge coding effort, and probably less effort than scouring the internet for alternatives.
Google places API returns a JSON when it requested for a place under food category which includes the details of several places.
I want to create an object array where each object contains details of a specific place.
I have used GSON library for my implementation and it works fine for a dummy JSON object but not with the JSON result given from Google place API and 'JsonSyntaxException' is thrown.
I look for a solution for following matters..
1 How can I proceed with GSON and given JSON object to create my object array or
2 Is there any other way to accomplish my task (still using JSON result)
Thanks.
update
Class PlaceObject
import java.util.List;
public class PlaceObject {
private List<String> results;
}
Class JSONconverter
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import com.google.gson.Gson;
public class JSONconverter {
public static void main(String[] args) {
Gson gson = new Gson();
try {
BufferedReader br = new BufferedReader(
new FileReader("c:\\placeAPI.json"));
//convert the json string back to object
PlaceObject obj = gson.fromJson(br, PlaceObject.class);
//obj.results = null ,when debugged thats the problem
System.out.println("Result: " + obj);
} catch (IOException e) {
e.printStackTrace();
}
}
}
The link of JSON
http://www.mediafire.com/?8mmnuxuopimhdnz
I like working with gson
btw. there is another relevant thread
Jersey client's documentation proposes to use Jackson library (see wiki)
You can also take a look at Genson library http://code.google.com/p/genson/.
It provides an out of box integration with jersey. You only need to have the jar in your classpath.