On Play Framework's homepage they claim that "JSON is a first class citizen". I have yet to see the proof of that.
In my project I'm dealing with some pretty complex JSON structures. This is just a very simple example:
{
"key1": {
"subkey1": {
"k1": "value1"
"k2": [
"val1",
"val2"
"val3"
]
}
}
"key2": [
{
"j1": "v1",
"j2": "v2"
},
{
"j1": "x1",
"j2": "x2"
}
]
}
Now I understand that Play is using Jackson for parsing JSON. I use Jackson in my Java projects and I would do something simple like this:
ObjectMapper mapper = new ObjectMapper();
Map<String, Object> obj = mapper.readValue(jsonString, Map.class);
This would nicely parse my JSON into Map object which is what I want - Map of string and object pairs and would allow me easily to cast array to ArrayList.
The same example in Scala/Play would look like this:
val obj: JsValue = Json.parse(jsonString)
This instead gives me a proprietary JsObject type which is not really what I'm after.
My question is: can I parse JSON string in Scala/Play to Map instead of JsObject just as easily as I would do it in Java?
Side question: is there a reason why JsObject is used instead of Map in Scala/Play?
My stack: Play Framework 2.2.1 / Scala 2.10.3 / Java 8 64bit / Ubuntu 13.10 64bit
UPDATE: I can see that Travis' answer is upvoted, so I guess it makes sense to everybody, but I still fail to see how that can be applied to solve my problem. Say we have this example (jsonString):
[
{
"key1": "v1",
"key2": "v2"
},
{
"key1": "x1",
"key2": "x2"
}
]
Well, according to all the directions, I now should put in all that boilerplate that I otherwise don't understand the purpose of:
case class MyJson(key1: String, key2: String)
implicit val MyJsonReads = Json.reads[MyJson]
val result = Json.parse(jsonString).as[List[MyJson]]
Looks good to go, huh? But wait a minute, there comes another element into the array which totally ruins this approach:
[
{
"key1": "v1",
"key2": "v2"
},
{
"key1": "x1",
"key2": "x2"
},
{
"key1": "y1",
"key2": {
"subkey1": "subval1",
"subkey2": "subval2"
}
}
]
The third element no longer matches my defined case class - I'm at square one again. I am able to use such and much more complicated JSON structures in Java everyday, does Scala suggest that I should simplify my JSONs in order to fit it's "type safe" policy? Correct me if I'm wrong, but I though that language should serve the data, not the other way around?
UPDATE2: Solution is to use Jackson module for scala (example in my answer).
Scala in general discourages the use of downcasting, and Play Json is idiomatic in this respect. Downcasting is a problem because it makes it impossible for the compiler to help you track the possibility of invalid input or other errors. Once you've got a value of type Map[String, Any], you're on your own—the compiler is unable to help you keep track of what those Any values might be.
You have a couple of alternatives. The first is to use the path operators to navigate to a particular point in the tree where you know the type:
scala> val json = Json.parse(jsonString)
json: play.api.libs.json.JsValue = {"key1": ...
scala> val k1Value = (json \ "key1" \ "subkey1" \ "k1").validate[String]
k1Value: play.api.libs.json.JsResult[String] = JsSuccess(value1,)
This is similar to something like the following:
val json: Map[String, Any] = ???
val k1Value = json("key1")
.asInstanceOf[Map[String, Any]]("subkey1")
.asInstanceOf[Map[String, String]]("k1")
But the former approach has the advantage of failing in ways that are easier to reason about. Instead of a potentially difficult-to-interpret ClassCastException exception, we'd just get a nice JsError value.
Note that we can validate at a point higher in the tree if we know what kind of structure we expect:
scala> println((json \ "key2").validate[List[Map[String, String]]])
JsSuccess(List(Map(j1 -> v1, j2 -> v2), Map(j1 -> x1, j2 -> x2)),)
Both of these Play examples are built on the concept of type classes—and in particular on instances of the Read type class provided by Play. You can also provide your own type class instances for types that you've defined yourself. This would allow you to do something like the following:
val myObj = json.validate[MyObj].getOrElse(someDefaultValue)
val something = myObj.key1.subkey1.k2(2)
Or whatever. The Play documentation (linked above) provides a good introduction to how to go about this, and you can always ask follow-up questions here if you run into problems.
To address the update in your question, it's possible to change your model to accommodate the different possibilities for key2, and then define your own Reads instance:
case class MyJson(key1: String, key2: Either[String, Map[String, String]])
implicit val MyJsonReads: Reads[MyJson] = {
val key2Reads: Reads[Either[String, Map[String, String]]] =
(__ \ "key2").read[String].map(Left(_)) or
(__ \ "key2").read[Map[String, String]].map(Right(_))
((__ \ "key1").read[String] and key2Reads)(MyJson(_, _))
}
Which works like this:
scala> Json.parse(jsonString).as[List[MyJson]].foreach(println)
MyJson(v1,Left(v2))
MyJson(x1,Left(x2))
MyJson(y1,Right(Map(subkey1 -> subval1, subkey2 -> subval2)))
Yes, this is a little more verbose, but it's up-front verbosity that you pay for once (and that provides you with some nice guarantees), instead of a bunch of casts that can result in confusing runtime errors.
It's not for everyone, and it may not be to your taste—that's perfectly fine. You can use the path operators to handle cases like this, or even plain old Jackson. I'd encourage you to give the type class approach a chance, though—there's a steep-ish learning curve, but lots of people (including myself) very strongly prefer it.
I've chosen to use Jackson module for scala.
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper
val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
val obj = mapper.readValue[Map[String, Object]](jsonString)
For further reference and in the spirit of simplicity, you can always go for:
Json.parse(jsonString).as[Map[String, JsValue]]
However, this will throw an exception for JSON strings not corresponding to the format (but I assume that goes for the Jackson approach as well). The JsValue can now be processed further like:
jsValueWhichBetterBeAList.as[List[JsValue]]
I hope the difference between handling Objects and JsValues is not an issue for you (only because you were complaining about JsValues being proprietary). Obviously, this is a bit like dynamic programming in a typed language, which usually isn't the way to go (Travis' answer is usually the way to go), but sometimes that's nice to have I guess.
You can simply extract out the value of a Json and scala gives you the corresponding map.
Example:
var myJson = Json.obj(
"customerId" -> "xyz",
"addressId" -> "xyz",
"firstName" -> "xyz",
"lastName" -> "xyz",
"address" -> "xyz"
)
Suppose you have the Json of above type.
To convert it into map simply do:
var mapFromJson = myJson.value
This gives you a map of type : scala.collection.immutable.HashMap$HashTrieMap
Would recommend reading up on pattern matching and recursive ADTs in general to better understand of why Play Json treats JSON as a "first class citizen".
That being said, many Java-first APIs (like Google Java libraries) expect JSON deserialized as Map[String, Object]. While you can very simply create your own function that recursively generates this object with pattern matching, the simplest solution would probably be to use the following existing pattern:
import com.google.gson.Gson
import java.util.{Map => JMap, LinkedHashMap}
val gson = new Gson()
def decode(encoded: String): JMap[String, Object] =
gson.fromJson(encoded, (new LinkedHashMap[String, Object]()).getClass)
The LinkedHashMap is used if you would like to maintain key ordering at the time of deserialization (a HashMap can be used if ordering doesn't matter). Full example here.
Related
How to convert java.util.list[POJO] to Scala array[POJO]?
I tried list.toArray method but it gives array[object].
Can anyone help on this?
You have to create the target array first, and provide it as input for the toArray method:
list.toArray(Array.ofDim[POJO](list.size))
This API shifts all the problems with array instantiation from the toArray method to you, so it is your responsibility to make sure that POJO is either something concrete, or to provide a ClassTag.
You could also do the conversion in two steps, first using asScala from JavaConverters:
import scala.collection.JavaConverters._
and then invoking .toArray from the Scala API (unlike Java's API, it preserves the type):
list.asScala.toArray
Below code works for me:
applcation.conf
mydata.crypto {
ciphers = [
"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
"TLS_DHE_RSA_WITH_AES_256_GCM_SHA384"
"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
]
protocols = [
"TLSv1.2"
]
}
Sample code in scala
Reading as List:
val ciphersList = config.getStringList("mydata.crypto.ciphers")
val protocolsList = config.getStringList("mydata.crypto.protocols")
import scala.collection.JavaConverters._
val enableCiphersList = ciphersList.asScala.toArray
val enableProtocolsList = protocolsList.asScala.toArray
Now we can see "enableCiphersList" and "enableProtocolsList" are Array of Strings type.
I am using Jackson parser for parsing for Java object to JSON. I am forcefully adding JSON keys for some java objects, using the following code.
#JsonTypeInfo(use = JsonTypeInfo.Id.NAME, include = JsonTypeInfo.As.WRAPPER_OBJECT)
#JsonSubTypes({ #JsonSubTypes.Type(value = A.class, name = "a"),
#JsonSubTypes.Type(value = F.class, name = "f") })
I am having same set classes in Python also. I would like to do the same thing in python. But not sure about what are the alternatives for this Jackson annotation available in python.
My requirement is that I have to send a POST request to a REST API. I need to serialize the java object into JSON. But as my class structure is a little bit different I don't have all the JSON keys mentioned handy in java classes. To solve the issue what I am doing is that I am adding 'a' key in JSON when it finds 'A' object is passed from java. It is doing the same thing for 'F' object. So, I have achieved the solution the way I mentioned above. I want to achieve the same thing in Python.
Is there some JSON parser available what does the same thing as mentioned above, or I have to follow some different approach for that?
I would strongly recommend the pydantic library, which also provides the possibility to add validations of the data.
It is very easy and straightforward to use, also if you only use it for json marshalling/unmarshalling.
https://pypi.org/project/pydantic/
import json
from datetime import datetime
from pydantic import BaseModel
class MyInnerClass(BaseModel):
timestamp: datetime
some_string: str
class MyOuterClass(BaseModel):
inner_instance: MyInnerClass
other_string: str
# Class to json
test_class = MyOuterClass(inner_instance=MyInnerClass(timestamp=datetime(2022, 6, 22), some_string="some"),
other_string="other")
json_string = test_class.json()
# Json to Class
input_string = '{"inner_instance": {"timestamp": "2022-06-22T00:00:00", "some_string": "some"}, "other_string": "other"}'
loaded_class = MyOuterClass(**json.loads(input_string))
attrs + cattrs is very close for the task.
copy one cattr example here,
>>> import attr, cattr
>>>
>>> #attr.s(slots=True, frozen=True) # It works with normal classes too.
... class C:
... a = attr.ib()
... b = attr.ib()
...
>>> instance = C(1, 'a')
>>> cattr.unstructure(instance)
{'a': 1, 'b': 'a'}
>>> cattr.structure({'a': 1, 'b': 'a'}, C)
C(a=1, b='a')
but it's not as capable as Jackson, i've not yet been able to find a solution to map attribute between serialized json and deserialized python object.
I have the same problem and could not find anything suitable.
So I wrote pyson
https://tracinsy.ewi.tudelft.nl/pubtrac/Utilities/wiki/pyson
It's still under development and I will add new features along the way.
It's not ment to be a complete substitute for jackson as jackson is huge. I just implement what I need, in jackson style where possible.
I think the most similar option you will get in python ecosystem will be jsonpickle
although it's not as complete as complete Jackson.
python engineers and users chose a different respectable perspective and that is using schema-less approaches to problems so typing oriented serialization libraries like Jackson doesn't have a strong equivalent in Python.
You can use Jsonic library.
Jsonic is a lightweight utility for serializing/deserializing python objects to/from JSON.
Jsonic allows serializing/deserializing to/from specific Class instances.
It supports many of the functionalities Jackson supports in Java.
Example:
from jsonic import serialize, deserialize
class User(Serializable):
def __init__(self, user_id: str, birth_time: datetime):
super().__init__()
self.user_id = user_id
self.birth_time = birth_time
user = User('id1', datetime(2020,10,11))
obj = serialize(user) # {'user_id': 'id1', 'birth_time': {'datetime': '2020-10-11 00:00:00', '_serialized_type': 'datetime'}, '_serialized_type': 'User'}
# Here the magic happens
new_user : User = deserialize(obj) # new_user is a new instance of user with same attributes
Jsonic has some nifty features:
You can serialize objects of types that are not extending Serializable. 2. This can come handy when you need to serialize objects of third party library classes.
Support for custom serializers and deserializers
Serializing into JSON string or python dict
Transient class attributes
Supports both serialization of private fields or leave them out of the serialization process.
Here you can find some more advanced example:
from jsonic import Serializable, register_jsonic_type, serialize, deserialize
class UserCredentials:
"""
Represents class from some other module, which does not extend Serializable
We can register it using register_serializable_type function
"""
def __init__(self, token: str, exp: datetime):
self.token = token
self.expiration_time = exp
self.calculatedAttr = random.uniform(0, 1)
# Register UserCredentials which represents class from another module that does not implement Serializable
# exp __init__ parameter is mapped to expiration_time instace attribute
register_jsonic_type(UserCredentials, init_parameters_mapping={'exp': 'expiration_time'})
class User(Serializable):
transient_attributes = ['user_calculated_attr'] # user_calculated_attr won't be serialized and deserialzied
init_parameters_mapping = {'id': 'user_id'} # id __init__ parameter is mapped to user_id instace attribute
def __init__(self, user_id: str, birth_time: datetime, user_credentials: UserCredentials, *args):
super().__init__()
self.user_id = user_id
self.birth_time = birth_time
self.user_credentials = user_credentials
self.user_calculated_attr = 'user_calculated_attr'
user = User(user_id='user_1', birth_time=datetime(1995, 7, 5, 0),
user_credentials=UserCredentials(token='token', exp=datetime(2020, 11, 1, 0)))
# Here the magic happens
user_json_obj = serialize(user, string_output=True)
new_user = deserialize(user_json_obj, string_input=True, expected_type=User)
Full disclosure: I'm the creator of Jsonic, i recommend you read the Jsonic GitHub repository README to see if it fits your needs.
I have JSON that needs to be converted to a Java Object. The JSONs I need to handle can look like this:
{
"documents": [
{
"title": "Jobs",
"is-saved": false,
"abstract": "<span class=\"hit\">Jobs</span> may refer to:\n\n* Steve <span class=\"hit\">Jobs</span> (1955–2011), co-founder and former CEO of consumer electronics company...<br />",
"id": "Jobs",
"url": "http://en.wikipedia.org/wiki/Jobs"
}
],
"keywords_local": [
{
"keyword": "Jobs",
"interest": 1,
"angle": 0
}
],
"sessionid": "6cd6402e-1f67-45a8-b0fa-e79a5d0d50f4",
"q": "Jobs",
}
This JSON is returned when entering a search keyword on a searchengine, in this case "Jobs". I have not named these variables-to-be-created, this JSON was just "given" to me from a similar earlier app. So I'm obviously having trouble with variables is-saved and abstract. Abstract is a reserved keyword and everywhere I read a reserved keyword CANNOT be used as a variable name.
I do not have access to the previous app that I am sort of updating and I guess the point to that is that I need to figure it out by myself ;) But I am a bit of a stand still now, have no idea of how to move forward.
I'm a newbie, so do not give me hell if I'm asking a stupid question, it's my first time coding any sort of app! ;)
Thanks for any help!
If you use GSON for parsing you can name your members as you want and annotate them for mapping.
#SerializedName("abstract")
private String abstractText;
Another option you've got is to use Jackson, and use the #JsonProperty annotation..
#JsonProperty("abstract")
private String abstractText;
In fact, it depends on the tool you are using. With tools mapping directly to your custom POJO (like GSON, Jackson), you need to map your JSON field name with your Java correct and valid field name.
If you use a mors basic library such as JSON.org's, there is no need to do so because you parse it to specific object allowing you to handle it.
JSONObject obj = new JSONObject(" .... ");
JSONArray arr = obj.getJSONArray("documents");
String abstractValue = arr.getJSONObject(0).getString("abstract");
I am using goggle's search api to get topics id which is used to get JSON response from topic api.The returned response looks like this
{
"id":"/m/01d5g",
"property":{
"/amusement_parks/ride_theme/rides":{...},
"/award/ranked_item/appears_in_ranked_lists":{...},
"/book/book_character/appears_in_book":{
"valuetype":"object",
"values":[
{
"text":"Inferno",
"lang":"en",
"id":"/m/0g5qs3",
"creator":"/user/duck1123",
"timestamp":"2010-02-11T04:00:59.000Z"
},
{
"text":"Batman: Year One",
"lang":"en",
"id":"/m/0hzz_1h",
"creator":"/user/anasay",
"timestamp":"2012-01-25T11:05:03.000Z"
},
{
"text":"Batman: The Dark Knight Returns",
"lang":"en",
"id":"/m/0hzz_sb",
"creator":"/user/anasay",
"timestamp":"2012-01-25T11:22:17.001Z"
},
{
"text":"Batman: Son of the Demon",
"lang":"en",
"id":"/m/071l77",
"creator":"/user/wikimapper",
"timestamp":"2013-07-11T15:20:32.000Z"
},
{
"text":"Joker",
"lang":"en",
"id":"/m/04zxvhs",
"creator":"/user/wikimapper",
"timestamp":"2013-07-11T16:58:37.000Z"
},
{
"text":"Arkham Asylum: A Serious House on Serious Earth",
"lang":"en",
"id":"/m/0b7hyw",
"creator":"/user/wikimapper",
"timestamp":"2013-07-11T19:26:54.000Z"
}
],
"count":6.0
},
"/book/book_subject/works":{...},
"/comic_books/comic_book_character/cover_appearances":{...},
...
}
}
I want to decipher this so that i can get relevant information such as, "/book/book_character/appears_in_book" itself is a property for response and only required value that i want from it is "text" and "id" e.g. "text":"Batman: Year One" and "id":"/m/0hzz_1h".
Since the response does not have fixed properties, and which may varying according to response id. how can i covert this JSON response in java Class where i can store "/book/book_character/appears_in_book" as one serialized class and containing Collection of values such has id and text and appears_in_book as name variables for class.
I considered GSON to do this. since name of property is not constant i can not use it to covert JSON to Java Object. currently i am iterating over each property by hard coding and filling them in java variables.
If some one can provide efficient way to do so i will appreciate help.
You could do this dynamically using reflection in Java but this is an advanced feature of Java and it may make your code more complicated than it needs to be.
See: Dynamically create an object in java from a class name and set class fields by using a List with data
A simpler alternative would be to just parse the JSON into a bunch of nested Maps and Lists exactly as they're given in the JSON data.
See: How to parse JSON in Java
data: [
{
type: "earnings"
info: {
earnings: 45.6
dividends: 4052.94
gains: 0
expenses: 3935.24
shares_bought: 0
shares_bought_user_count: 0
shares_sold: 0
shares_sold_user_count: 0
}
created: "2011-07-04 11:46:17"
}
{
type: "mentions"
info: [
{
type_id: "twitter"
mentioner_ticker: "LOANS"
mentioner_full_name: "ERICK STROBEL"
}
]
created: "2011-06-10 23:03:02"
}
]
Here's my problem : like you can see the "info" is different in each of one, one is a json object, and one is a json array, i usually choose Gson to take the data, but with Gson we can't do this kind of thing . How can i make it work ?
If you want to use Gson, then to handle the issue where the same JSON element value is sometimes an array and sometimes an object, custom deserialization processing is necessary. I posted an example of this in the Parsing JSON with GSON, object sometimes contains list sometimes contains object post.
If the "info" element object has different elements based on type, and so you want polymorphic deserialization behavior to deserialize to the correct type of object, with Gson you'll also need to implement custom deserialization processing. How to do that has been covered in other StackOverflow.com posts. I posted a link to four different such questions and answers (some with code examples) in the Can I instantiate a superclass and have a particular subclass be instantiated based on the parameters supplied thread. In this thread, the particular structure of the JSON objects to deserialize varies from the examples I just linked, because the element to indicate the type is external of the object to be deserialized, but if you can understand the other examples, then handling the problem here should be easy.
Both key and value have to be within quotes, and you need to separate definitions with commas:
{
"key0": "value0",
"key1": "value1",
"key2": [ "value2_0", "value2_1" ]
}
That should do the trick!
The info object should be of the same type with every type.
So check the type first. Pseudocode:
if (data.get('type').equals("mentions") {
json_arr = data.get('info');
}
else if (data.get('type').equals("earnings") {
json_obj = data.get('info');
}
I'm not sure that helps, cause I'm not sure I understand the question.
Use simply org.json classes that are available in android: http://developer.android.com/reference/org/json/package-summary.html
You will get a dynamic structure that you will be able to traverse, without the limitations of strong typing.....
This is not a "usual" way of doing things in Java (where strong typing is default) but IMHO in many situations even in Java it is ok to do some dynamic processing. Flexibility is better but price to pay is lack of compile-time type verification... Which in many cases is ok.
If changing libraries is an option you could have a look at Jackson, its Simple Data Binding mode should allow you to deserialize an object like you describe about. A part of the doc that is probably quite important is this, your example would already need JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES to work...
Clarification for Bruce: true, in Jackson's Full Data Binding mode, but not in Simple Data Binding mode. This is simple data binding:
public static void main(String[] args) throws IOException {
File src = new File("test.json");
ObjectMapper mapper = new ObjectMapper();
mapper.configure(JsonParser.Feature. ALLOW_UNQUOTED_FIELD_NAMES, true);
mapper.configure(JsonParser.Feature.ALLOW_COMMENTS,true);
Object root = mapper.readValue(src, Object.class);
Map<?,?> rootAsMap = mapper.readValue(src, Map.class);
System.out.println(rootAsMap);
}
which with OP's sightly corrected sample JSON data gives:
{data=[{type=earnings, info={earnings=45.6, dividends=4052.94, gains=0,
expenses=3935.24, shares_bought=0, shares_bought_user_count=0, shares_sold=0,
shares_sold_user_count=0}, created=2011-07-04 11:46:17}, {type=mentions,
info=[{type_id=twitter, mentioner_ticker=LOANS, mentioner_full_name=ERICK STROBEL}],
created=2011-06-10 23:03:02}]}
OK, some hand-coding needed to wire up this Map to the original data, but quite often less is more and such mapping code, being dead simple has the advantage of being very easy to read/maintain later on.