I am trying to deserialize JSON of format
"{mapping:{MyType(type=\"A\", value=\"B\"):\"C\"}}"
into a class type
class MyMapping {
Map<MyType, String> mapping;
}
class MyType {
String type;
String value;
}
Since I have a map type, i added a KeyDeserializer to deserialize MyType(type=\"A\", value=\"B\")
But when i try to deserialize this i get an exception
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('(' (code 40)): was expecting a colon to separate field name and value
This is because it encounters ( in MyType(type=\"A\", value=\"B\") and it breaks there. If i put the whole key within double quotes \"MyType(type=\"A\", value=\"B\")\", it passes the the whole key to my deserializer.
Is there a way i can force it to escape the '(' char without having to put the whole key with quotes.
The string in your example is not valid JSON.
"{mapping:{MyType(type=\"A\", value=\"B\"):\"C\"}}"
You pointed out that the Jackson parser complains about the first (
If you want to use this particular string as a key you have to enclose it in quotes and escape internal quotes. Let's use a Javascript in a browser to do that, cause it's easy. From Chrome console...
> var c = {};
< undefined
> c['MyType(type="A", value="B")']='C';
< "C"
> JSON.stringify(c)
< "{"MyType(type=\"A\", value=\"B\")":"C"}"`
That's how Javascript escapes your key string as valid JSON.
KeyDeserializer
Based on Jackson 2.8 documentation for KeyDeserializer I would expect your KeyDeserializer to receive the whole key string. That's just how JSON works.
{"key-string":"value")
Of course "value" could be false, true, null or a number as well.
It's then up to you to parse the key-string and decide how to turn it into a key that your MyMapping can use.
Your Key String Appears Easy to Parse
Your KeyDeserializer can use a compiled regex pattern to get the two key properties as two regex 'groups'.
What if you don't want to parse the key string?
It sounds like you don't want to parse this key. There are several ways to avoid this. One key to this is to use a JSON format that makes it easy, such as ...
{"key":{"keyType":"A","keyValue":"B"}, "value":"C"}
Then you could use annotations to teach Jackson to work with your MyMapping class.
Such as (untested, just a sketch)
class MyMapping {
Map<MyType, String> mapping;
#JsonCreator
public MyMapping(Map<String,Object> input) {
Map<String,Object> key = (Map)input.get("key");
String value = (String)input.get("value");
MyType myKey = new MyType(key);
mapping = new HashMap<>();
mapping.put(myKey,value);
}
}
class MyType {
String type;
String value;
MyType (Map<String,Object> input) {
type = (String)input.getOrDefault("key","");
value = (String)input.getOrDefault("value","");
}
}
There's a very long list of problems with this example. You will be using the default object address based comparison in the Map, for example. I made this code short by ignoring likely ClassCastExceptions and other data problems. But the point here was not to write good code, or to try to understand the requirements, but simply to point out how you can get Jackson to parse the JSON for you.
There are many. other ways to achieve that. I just. picked the first one that came to mind.
Related
Even with the NUMBER shape why its printing string with double quotes?
#JsonFormat(shape = Shape.NUMBER)
private String count;
...
ObjectMapper mapper = new ObjectMapper();
Test test =new Test();
test.setCount("20");
String jsonString = mapper.writeValueAsString(test);
System.out.println(jsonString);
JSON result:
{
"count" : "20"
}
#JsonFormat annotation is an only suggestion and result depends from given custom serialiser which will be used to serialise given value. In your case, com.fasterxml.jackson.databind.ser.std.StringSerializer will be used to serialise count value. This serialiser does not implement any special behaviour. Just simple writing value as JSON String. From other side, com.fasterxml.jackson.databind.ser.std.NumberSerializer respects #JsonFormat annotation and if you set STRING shape it will produce JSON String instead of JSON Number.
So, you always have to check type serialiser implementation to answer questions like this.
I'm working on a configuration system. I'd like to be able to load config values from a JSON file and have them "automagically" convert to the Java type I need. I'm using Jackson for the JSON parsing. For primitive types like floats and strings, it's no big deal, but I'm running into a snag with enums.
Let's say I have the following enum:
public enum SystemMode
{
#JsonProperty("Mode1")
MODE1("Mode1"),
#JsonProperty("Mode2")
MODE2("Mode2"),
#JsonProperty("Mode3")
MODE3("Mode3");
private final String name;
private SystemMode(String name)
{
this.name = name;
}
#Override
#JsonValue
public String toString()
{
return this.name;
}
}
Now, let's say I want to represent a list of values of this enum for a given config variable using the following JSON representation:
{
"Project" : "TEST",
"System" : {
"ValidModes" : ["Mode1", "Mode2"]
}
}
And I'd like to be able to do something like the following:
ArrayList<SystemMode> validModes = (ArrayList<SystemMode>) configurator.getConfigValue("/System/ValidModes");
For reference, my configurator class's getConfigValue method is essentially a thin wrapper over the Jackson JSON parsing:
public Object getConfigValue(String JSON_String)
{
JsonNode node = JsonNodeFactory.instance.objectNode().at(JSON_String);
return objectMapper.convertValue(node, Object.class);
}
(The real method has some exception checking that has been omitted for clarity).
Now, when I call the above, Jackson correctly deduces that I want an ArrayList and fills it. However, instead of getting an ArrayList of SystemMode enums, I get an ArrayList of Strings and immediately throw an exception when I attempt to use the list. I have tried several different ways of representing the data to no avail. It seems no matter what I try, Jackson wants to return a list of strings instead of a list of enums.
So my question is this:
How can I make Jackson (version 2.9.4) JSON properly deserialize a list of enum values in a way that is compatible with my single "Object getConfigValue()" method?
The following will provide the correct binding for your enum.
public List<SystemMode> getConfigValue(String path)
{
JsonNode node = JsonNodeFactory.instance.objectNode().at(path);
return objectMapper.convertValue(node, new TypeReference<List<SystemMode>>(){});
}
The second option is to convert the list of String yourself, for example:
List<SystemMode> result = jsonResult.stream().map(SystemMode::valueOf).collect(Collectors.toList());
Third option:
public <T>List<T> getConfigValue(String path, Class<T> type)
{
JsonNode node = JsonNodeFactory.instance.objectNode().at(path);
CollectionType toType =
objectMapper.getTypeFactory().constructCollectionType(List.class, type);
return objectMapper.convertValue(node, toType);
}
I want to write a generic flink job in java, which can take any SQL-SELECT query, run it against a SQL-database and write it into a Elasticsearch index.
One of the problems I have to solve is creating a DataSource for a JDBC-Connection. I want to use the JDBCInputFormat. I followed the example in the documentation data source.
The problem is, the generic type DataSource type must be specified. And I can only use a Tuple type, because JDBCInputFormat generic type OUT extends Tuple. But I do not know at compile time which Tuple I will use.
Do I interpret something wrong?
Is there another jdbc InputFormat I can use?
Is there a way to specify Tuple as a generic type?
I use java 7 and apache-flink 0.10.2
I tried to use Tuple25 with only Strings in it, but I get an exception.
Here follows code and then the exception.
DataSource<StringsTuple25> database = flink.createInput(
JDBCInputFormat.buildJDBCInputFormat()//
.setDrivername(getDatabaseDriverName())//
.setDBUrl(getDatabaseUrl())//
.setUsername(getDatabaseUsername())//
.setPassword(getDatabasePassword())//
.setQuery(getQuery())//
.finish(),
StringsTuple25.typeInformation()
);
My StringTuple25 class
public class StringsTuple25 extends
Tuple25<String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String> {
private static final long serialVersionUID = 1L;
public static TypeInformation<?> typeInformation() {
TypeInformation<String>[] types = new TypeInformation[25];
Arrays.fill(types, STRING_TYPE_INFO);
return new TupleTypeInfo<>(Tuple25.class,types);
}
}
And I get this exception:
Caused by: java.io.IOException: Tuple size does not match columncount
at org.apache.flink.api.java.io.jdbc.JDBCInputFormat.extractTypes(JDBCInputFormat.java:180)
at org.apache.flink.api.java.io.jdbc.JDBCInputFormat.nextRecord(JDBCInputFormat.java:162)
at org.apache.flink.api.java.io.jdbc.JDBCInputFormat.nextRecord(JDBCInputFormat.java:51)
at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:169)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
at java.lang.Thread.run(Thread.java:745)
As the error indicate, the number of attributes on your used Tuple type must match the number of selected columns in your SQL query. Furthermore, the data types for each attribute must match.
For example if you SELECT id, name FROM ... with id is INTEGER and name is VARCHAR, you would specify use DataStream<Tuple2<Integer,String>> (or specialize your own class class MyResultType extends Tuple2<Integer,String> and DataStream<MyResultType>) and provide a corresponding TypeInformation.
You can also go with generic Tuple type. Your stream would be DataStream<Tuple> (without specifying the number or types of attributes). However, for the TypeInformation you need to know the number of attributes.
Tuple t = Tuple.getTupleClass(numberOfAttributes).newInstance();
for(int i = 0; i < numberOfAttributes; i++) {
t.setField("", i);
}
TypeInformation<Tuple> typeInfo = TypeExtractor.getForObject(t);
Thus, you need to infer the number of selected attributes from you given arguments that define your SQL query.
I have a POJO that is similar to:
public class MyGsonPojo {
#Expose
#SerializedName("value1")
private String valueOne;
#Expose
#SerializedName("value2")
private boolean valueTwo;
#Expose
#SerializedName("value3")
private int valueThree;
// Getters and other stuff here
}
The issue is that this object has to be serialized into a json body for a call
to the server. Some fields are optional for the request and if I even send it with default and null values, the API responds differently (Unfortunately changing the api is not an option).
So basically I need to exclude fields from serialization if any of them is set to a default value. For example if the field valueOne is null the resulting json should be:
{
"value2" : true,
"value3" : 2
}
Any idea how to make this a painless effort? I wouldn't want to build the json body manually.
Any help would be great. Thank you in advice.
Steps to follow:
Convert the JSON String into Map<String,Object> using Gson#fromJson()
Iterate the map and remove the entry from the map which are null
Form the JSON String back from the final map using Gson#toJson().
I have already posted the sample code in the same context here:
Remove empty collections from a JSON with Gson
Option 1) Use a TypeAdapter, see accepted answer here:
Option 2) If using Jackson instead of gson is a possibility, you can annotate/serialize on getters instead of on fields, and put your logic for returning
whatever you need for "default values" in your getters.
//won't get serialized because it's private
private String valueOne;
...
#JsonSerialize
String getValueOne(){
if (valueOne == null) return "true"
else...
}
You could also use a single #JsonInclude(Include.NON_NULL) or #JsonInclude(Include.NON_EMPTY) annotation at the top of your class to prevent any null or empty fields from being serialized.
Code shown below works well when JSON object contains jsonKey as it was passed to the method. I wonder ... if there is a way to get a value assigned to a case insensitive representation of a key?
Example:
public String getOutputEventDescription(JsonElement outputEvent) throws ParserException {
return retrieveString(outputEvent, DESCRIPTION);
}
Should work regardless whether DESCRIPTION is defined as "Description", "description" or "DeScRipTIOn"
protected String retrieveString(JsonElement e, String jsonKey) throws ParserException {
JsonElement value = e.getAsJsonObject().get(jsonKey);
if (value == null) {
throw new ParserException("Key not found: " + jsonKey);
}
if (value.getAsString().trim().isEmpty()) {
throw new ParserException("Key is empty: " + jsonKey);
}
return e.getAsJsonObject().get(jsonKey).getAsString();
}
Unfortunately, registering a FieldNamingStrategy with the GsonBuilder wouldn't do much good, as it translates only in the opposite-than-desired direction: from the Java field name to the JSON element name. It cannot be reasonably used for your purposes.
(In Detail:
The result of the translation request ends at FieldNamingStrategy.translateName(Field), where the translated name is used to get the associated JSON element from a JsonObject, which has a LinkedHashMap<String, JsonElement>, called members, mapping JSON element names to their associated values. The translated name is used as the parameter to the get(String) method of members, and Gson provides no mechanism for this final call to be made case insensitive.
The members map is populated with calls to JsonObject.add(String, JsonElement), made from Streams.parseRecursive(JsonReader), with the JSON element name retrieved from the JsonReader used as the key to 'members'. (JsonReader uses the characters exactly as they are in the JSON, with the exception where the escape character '\' is found.) Throughout this call stack, Gson provides no mechanism for the keys used to populate members to be altered, e.g., to be made all lower case or all upper case.
A FieldNamingPolicy works in the same way.)
A reasonable solution might be to simply use a custom deserializer, along the following lines.
input.json:
[
{"field":"one"},
{"Field":"two"},
{"FIELD":"three"},
{"fIElD":"four"}
]
Foo.java:
import java.io.FileReader;
import java.lang.reflect.Type;
import java.util.Map.Entry;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.JsonDeserializationContext;
import com.google.gson.JsonDeserializer;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParseException;
public class Foo
{
public static void main(String[] args) throws Exception
{
GsonBuilder gsonBuilder = new GsonBuilder();
gsonBuilder.registerTypeAdapter(MyClass.class, new MyTypeAdapter());
Gson gson = gsonBuilder.create();
MyClass[] myObjects = gson.fromJson(new FileReader("input.json"), MyClass[].class);
System.out.println(gson.toJson(myObjects));
}
}
class MyClass
{
String field;
}
class MyTypeAdapter implements JsonDeserializer<MyClass>
{
#Override
public MyClass deserialize(JsonElement json, Type myClassType, JsonDeserializationContext context)
throws JsonParseException
{
// json = {"field":"one"}
JsonObject originalJsonObject = json.getAsJsonObject();
JsonObject replacementJsonObject = new JsonObject();
for (Entry<String, JsonElement> elementEntry : originalJsonObject.entrySet())
{
String key = elementEntry.getKey();
JsonElement value = originalJsonObject.get(key);
key = key.toLowerCase();
replacementJsonObject.add(key, value);
}
return new Gson().fromJson(replacementJsonObject, MyClass.class);
}
}
Alternatively, you could first process the raw JSON to alter all of the element names to be the same case, all lower or all upper. Then, pass the altered JSON to Gson for deserialization. This would of course slow down JSON processing.
If you're able to change Gson code for your project, then probably the part to change for the most efficient result is the call to name = nextString((char) quote); in JsonReader. Since nextString(char) is also used to get the JSON element value, I'd probably just make a copy of it for getting the name, and then make small changes to force the element names to all lower or all upper case. Of course, this approach then locks your project to one release of Gson, else you'd need to repeat this change to upgrade to a newer Gson release.
With Jackson, the situation appears unfortunately similar. Translations with a PropertyNamingStrategy work in unfortunately much the same way: they translate from the Java field name to the JSON element name. None of the available JsonParser.Feature alterations would customize a JsonParser to force JSON element names to all upper or all lower case.
I faced the similar issue. I did this to get around the issue. (Replaced all the keys with their corresponding lowercase version and had all lower case fields in matching class). Hope this helps.
input = input.replaceAll("\\s","");
Matcher m = Pattern.compile("\"\\b\\w{1,}\\b\"\\s*:").matcher(input);
StringBuilder sanitizedJSON = new StringBuilder();
int last = 0;
while (m.find()) {
sanitizedJSON.append(input.substring(last, m.start()));
sanitizedJSON.append(m.group(0).toLowerCase());
last = m.end();
}
sanitizedJSON.append(input.substring(last));
input = sanitizedJSON.toString();
Unfortunately there doesn't seem to be a way in the current implementation to do this. If you look at the Gson source and more specifically at the JsonObject implementation you will see that the underlying data structure is a linked hash map. The get call simply invokes the get on the map, which in turn uses the hash code and equals method of your key to find the object you are looking for.
The only way around is to enforce some naming conventions for you keys. The easiest way would be to force all the keys to lowercase. If you need mixed case keys then you will have more difficulty and will need to write a more sophisticated algorithm for transforming the keys instead of simply calling jsonKey.toLowerCase().
I stumbled across this question when I ran into an issue where a different naming convention was being used at the two endpoints and subsequently discovered a less invasive solution.
Gson does support setting a naming convention that is used when mapping from the Java model names to the JSON names, both when when serializing and deserializing. Use the setFieldNamingPolicy method of the builder to change this behavior.
GsonBuilder gsonBuilder = new GsonBuilder();
gsonBuilder.setFieldNamingPolicy(FieldNamingPolicy.UPPER_CAMEL_CASE);
Gson gson = gsonBuilder.create();
See here for a nice article on the subject, including an overview of the different policies.
This isn't really a case insensitive solution, but it does provide a way to work around many of the situations where the case is not matching up.