I'm trying to serialize a protobuf message, which is represented as Java class, into JSON with Gson library and ProtoTypeAdapter
ProtoTypeAdapter adapter = ProtoTypeAdapter.newBuilder()
.setFieldNameSerializationFormat(CaseFormat.LOWER_UNDERSCORE, CaseFormat.LOWER_UNDERSCORE)
.build();
Gson gson = new GsonBuilder()
.registerTypeAdapter(SomeAutogeneratedClass.class, adapter)
.setFieldNamingPolicy(FieldNamingPolicy.LOWER_CASE_WITH_UNDERSCORES)
.disableHtmlEscaping()
.create();
suchMessage.getMyIntField() // which is 0
String serialized = gson.toJson(suchMessage)
But it seems that it does not serialize default values such as 0 for int field.
How can I include those fields with default value in JSON?
I modified this line as follows:
// final Map<FieldDescriptor, Object> fields = src.getAllFields(); // original line
// --- Include default value fields (See com.google.protobuf.util.JsonFormat) ---
final Map<FieldDescriptor, Object> fields = new TreeMap<>(src.getAllFields());
for (FieldDescriptor field : src.getDescriptorForType().getFields()) {
if (field.isOptional()) {
if (field.getJavaType() == FieldDescriptor.JavaType.MESSAGE
&& !src.hasField(field)) {
continue;
}
Descriptors.OneofDescriptor oneof = field.getContainingOneof();
if (oneof != null && !src.hasField(field)) {
continue;
}
}
if (!fields.containsKey(field)) {
fields.put(field, src.getField(field));
}
}
// ------
I've created a little script that uses reflection to fix this by changing the default value to something that can never be used.
public static void overrideDefaultValue(Descriptors.FieldDescriptor desc, Object newDefault) throws IllegalAccessException, NoSuchFieldException {
Field f = Descriptors.FieldDescriptor.class.getDeclaredField("defaultValue");
f.setAccessible(true);
f.set(desc, newDefault);
}
Let's say you're trying to serialize a value that represent an index. Then -1 is an impossible value. You can use it like this:
overrideDefaultValue(MyMessage.getDescriptor().findFieldByName("my_field"), -1);
Related
Is there a way to make Jackson interpret single JSON object as an array with one element and vice versa?
Example, I have 2 slightly different formats of JSON, I need both to map to same Java object:
Format A (JSON array with one element):
points : [ {
date : 2013-05-11
value : 123
}]
Format B (JSON object, yes I know it looks "wrong" but it's what I'm given):
points : {
date : 2013-05-11
value : 123
}
Target Java object that both of the above should convert to:
//Data.java
public List<Point> points;
//other members omitted
//Point.java
class Point {
public String date;
public int value;
}
Currently, only A will parse properly to Data. I want avoid directly tampering with the JSON itself. Is there some configuration in Jackson I can tamper with in order to make it accept B ?
Try with DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY - it should work for you.
Example:
final String json = "{\"date\" : \"2013-05-11\",\"value\" : 123}";
final ObjectMapper mapper = new ObjectMapper()
.enable(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY);
final List<Point> points = mapper.readValue(json,
new TypeReference<List<Point>>() {});
The Jackson 1.x-compatible version uses DeserializationConfig.Feature.ACCEPT_SINGLE_VALUE_AS_ARRAY. So the above answer changes to:
final String json = "{\"date\" : \"2013-05-11\",\"value\" : 123}";
final ObjectMapper mapper = new ObjectMapper()
.enable(DeserializationConfig.Feature.ACCEPT_SINGLE_VALUE_AS_ARRAY);
final List<Point> points = mapper.readValue(json,
new TypeReference<List<Point>>() {
});
System.out.println(points);
Can solve the above problem by this code is given below, this works
final ObjectMapper objectMapper = new
ObjectMapper().enable(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY);
objectMapper.enable(DeserializationFeature.ACCEPT_EMPTY_STRING_AS_NULL_OBJECT);
try {
String jsonInString = objectMapper.writeValueAsString(products.get(j));
InventoryParser inventoryParser = objectMapper.readValue(jsonInString,
InventoryParser.class);
System.out.println(inventoryParser.getId());
}
catch (IOException e)
{
e.printStackTrace();
}
"InventoryParser" is a POJO Class.
"products.get(j)" is JSON String.
I have an unbounded stream of complex objects that I want to load into BigQuery. The structure of these objects represents the schema of my destination table in BigQuery.
The problem is that since there are a lot of nested fields in the POJO, its an extremely tedious task to convert it to a TableSchema object and I'm looking for a quick/ automated way to convert my POJO to TableSchema object while writing to BigQuery.
I'm not very familiar with Apache Beam API, and any help will be appreciated.
In a pipeline, I load a list of schema from GCS. I keep them in string format because the TableSchema is not serializable. However, I load them to TableSchema for validate them.
Then I add them in string format to a map in the Option object.
String schema = new String(blob.getContent());
// Decorate list of fields for allowing a correct parsing
String targetSchema = "{\"fields\":" + schema + "}";
try {
//Preload schema to ensure validity, but then use string version
Transport.getJsonFactory().fromString(targetSchema, TableSchema.class);
String tableName = blob.getName().replace(SCHEMA_FILE_PREFIX, "").replace(SCHEMA_FILE_SUFFIX, "");
tableSchemaStringMap.put(tableName, targetSchema);
} catch (IOException e) {
logger.warn("impossible to read schema " + blob.getName() + " in bucket gs://" + options.getSchemaBucket());
}
I didn't find another solution when I developed this.
In my company I created kind of a ORM (we called OBQM) to do this. We are expecting to release it to the public. The code is quite big (specially because I created annotations and so on) but I can share with you some snippets for a quick schema generation:
public TableSchema generateTableSchema(#Nonnull final Class cls) {
final TableSchema tableSchema = new TableSchema();
tableSchema.setFields(generateFieldsSchema(cls));
return tableSchema;
}
public List<TableFieldSchema> generateFieldsSchema(#Nonnull final Class cls) {
final List<TableFieldSchema> schemaFields = new ArrayList<>();
final Field[] clsFields = cls.getFields();
for (final Field field : clsFields) {
schemaFields.add(fromFieldToSchemaField(field));
}
return schemaFields;
}
This code takes all the fields from the POJO class and creates a TableSchema object (the one that BigQueryIO uses in ApacheBeam). You can see a method that I created called fromFieldToSchemaField. This method identifies each field type and setup the field name, mode, description and type. In this case to keep it simple I'm going to focus on the type and name:
public static TableFieldSchema fromFieldToSchemaField(#Nonnull final Field field) {
return fromFieldToSchemaField(field, 0);
}
public static TableFieldSchema fromFieldToSchemaField(
#Nonnull final Field field,
final int iteration) {
final TableFieldSchema schemaField = new TableFieldSchema();
final Type customType = field.getGenericType().getTypeName()
schemaField.setName(field.getName());
schemaField.setMode("NULLABLE"); // You can add better logic here, we use annotations to override this value
schemaField.setType(getFieldTypeString(field));
schemaField.setDescription("Optional"); // Optional
if (iteration < MAX_RECURSION
&& (isStruct(schemaField.getType())
|| isRecord(schemaField.getType()))) {
final List<TableFieldSchema> schemaFields = new ArrayList<>();
final Field[] fields = getFieldsFromComplexObjectField(field);
for (final Field subField : fields) {
schemaFields.add(
fromFieldToSchemaField(
subField, iteration + 1));
}
schemaField.setFields(schemaFields.isEmpty() ? null : schemaFields);
}
return schemaField;
}
And now the method that returns the BigQuery field type.
public static String getFieldTypeString(#Nonnull final Field field) {
// On my side this code is much complex but this is a short version of that
final Class<?> cls = (Class<?>) field.getGenericType()
if (cls.isAssignableFrom(String.class)) {
return "STRING";
} else if (cls.isAssignableFrom(Integer.class) || cls.isAssignableFrom(Short.class)) {
return "INT64";
} else if (cls.isAssignableFrom(Double.class)) {
return "NUMERIC";
} else if (cls.isAssignableFrom(Float.class)) {
return "FLOAT64";
} else if (cls.isAssignableFrom(Boolean.class)) {
return "BOOLEAN";
} else if (cls.isAssignableFrom(Double.class)) {
return "BYTES";
} else if (cls.isAssignableFrom(Date.class)
|| cls.isAssignableFrom(DateTime.class)) {
return "TIMESTAMP";
} else {
return "STRUCT";
}
}
Keep in mind that I'm not showing how to identify primitive types or arrays. But this is a good start for your code :). Please let me know if you need any help.
If your using JSON for the message serialization in PubSub you can make use of one of the provided templates:
PubSub To BigQuery Template
The code for that template is here:
PubSubToBigQuery.java
Is there a way to make Jackson interpret single JSON object as an array with one element and vice versa?
Example, I have 2 slightly different formats of JSON, I need both to map to same Java object:
Format A (JSON array with one element):
points : [ {
date : 2013-05-11
value : 123
}]
Format B (JSON object, yes I know it looks "wrong" but it's what I'm given):
points : {
date : 2013-05-11
value : 123
}
Target Java object that both of the above should convert to:
//Data.java
public List<Point> points;
//other members omitted
//Point.java
class Point {
public String date;
public int value;
}
Currently, only A will parse properly to Data. I want avoid directly tampering with the JSON itself. Is there some configuration in Jackson I can tamper with in order to make it accept B ?
Try with DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY - it should work for you.
Example:
final String json = "{\"date\" : \"2013-05-11\",\"value\" : 123}";
final ObjectMapper mapper = new ObjectMapper()
.enable(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY);
final List<Point> points = mapper.readValue(json,
new TypeReference<List<Point>>() {});
The Jackson 1.x-compatible version uses DeserializationConfig.Feature.ACCEPT_SINGLE_VALUE_AS_ARRAY. So the above answer changes to:
final String json = "{\"date\" : \"2013-05-11\",\"value\" : 123}";
final ObjectMapper mapper = new ObjectMapper()
.enable(DeserializationConfig.Feature.ACCEPT_SINGLE_VALUE_AS_ARRAY);
final List<Point> points = mapper.readValue(json,
new TypeReference<List<Point>>() {
});
System.out.println(points);
Can solve the above problem by this code is given below, this works
final ObjectMapper objectMapper = new
ObjectMapper().enable(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY);
objectMapper.enable(DeserializationFeature.ACCEPT_EMPTY_STRING_AS_NULL_OBJECT);
try {
String jsonInString = objectMapper.writeValueAsString(products.get(j));
InventoryParser inventoryParser = objectMapper.readValue(jsonInString,
InventoryParser.class);
System.out.println(inventoryParser.getId());
}
catch (IOException e)
{
e.printStackTrace();
}
"InventoryParser" is a POJO Class.
"products.get(j)" is JSON String.
I have a protobuf message of the form
enum PolicyValidationType {
Number = 0;
}
message NumberPolicyValidation {
optional int64 maxValue = 1;
optional int64 minValue = 2;
}
message PolicyObject {
required string key = 1;
optional string value = 2;
optional string name = 3;
optional PolicyValidationType validationType = 4;
optional NumberPolicyValidation numberPolicyValidation = 5;
}
For example
policyObject {
key: "sessionIdleTimeoutInSecs"
value: "1800"
name: "Session Idle Timeout"
validationType: Number
numberPolicyValidation {
maxValue: 3600
minValue: 5
}
}
Can someone let me know how can I convert this to a Map like below:-
{validationType=Number, name=Session Idle Timeout, numberPolicyValidation={maxValue=3600.0, minValue=5.0}, value=1800, key=sessionIdleTimeoutInSecs}
One way I can think of is convert this to a json and then convert the json to map?
PolicyObject policyObject;
...
JsonFormat jsonFormat = new JsonFormat();
final String s = jsonFormat.printToString(policyObject);
Type objectMapType = new TypeToken<HashMap<String, Object>>() {}.getType();
Gson gson = new GsonBuilder().registerTypeAdapter(new TypeToken<HashMap<String,Object>>(){}.getType(), new PrimitiveDeserializer()).create();
Map<String, Object> mappedObject = gson.fromJson(s, objectMapType);
I think there must be some better way. Can someone suggest any better approach?
I created small dedicated class to generically convert any Google protocol buffer message into a Java Map.
public class ProtoUtil {
#NotNull
public Map<String, Object> protoToMap(Message proto) {
final Map<Descriptors.FieldDescriptor, Object> allFields = proto.getAllFields();
Map<String, Object> map = new LinkedHashMap<>();
for (Map.Entry<Descriptors.FieldDescriptor, Object> entry : allFields.entrySet()) {
final Descriptors.FieldDescriptor fieldDescriptor = entry.getKey();
final Object requestVal = entry.getValue();
final Object mapVal = convertVal(proto, fieldDescriptor, requestVal);
if (mapVal != null) {
final String fieldName = fieldDescriptor.getName();
map.put(fieldName, mapVal);
}
}
return map;
}
#Nullable
/*package*/ Object convertVal(#NotNull Message proto, #NotNull Descriptors.FieldDescriptor fieldDescriptor, #Nullable Object protoVal) {
Object result = null;
if (protoVal != null) {
if (fieldDescriptor.isRepeated()) {
if (proto.getRepeatedFieldCount(fieldDescriptor) > 0) {
final List originals = (List) protoVal;
final List copies = new ArrayList(originals.size());
for (Object original : originals) {
copies.add(convertAtomicVal(fieldDescriptor, original));
}
result = copies;
}
} else {
result = convertAtomicVal(fieldDescriptor, protoVal);
}
}
return result;
}
#Nullable
/*package*/ Object convertAtomicVal(#NotNull Descriptors.FieldDescriptor fieldDescriptor, #Nullable Object protoVal) {
Object result = null;
if (protoVal != null) {
switch (fieldDescriptor.getJavaType()) {
case INT:
case LONG:
case FLOAT:
case DOUBLE:
case BOOLEAN:
case STRING:
result = protoVal;
break;
case BYTE_STRING:
case ENUM:
result = protoVal.toString();
break;
case MESSAGE:
result = protoToMap((Message) protoVal);
break;
}
}
return result;
}
}
Hope that helps! Share and enjoy.
Be aware that both approaches described above (serialize/deserialize by tuk and custom converter by Zarnuk) will produce different outputs.
With the serialize/deserialize approach:
Field names in snake_case format will be automatically converted into camelCase. JsonFormat.printer() does this.
Numeric values will be converted to float. Gson does that for you.
Values of type Duration will be converted into strings with format durationInseconds + "s", i.e. "30s" for a duration of 30 seconds and "0.000500s" for a duration of 500,000 nanoseconds. JsonFormat.printer() does this.
With the custom converter approach:
Field names will remain as they are described on the proto file.
Integers and floats will keep their own type.
Values of type Duration will become objects with their corresponding fields.
To show the differences, here is a comparison of the outcomes of both approaches.
Original message (here is the proto file):
method_config {
name {
service: "helloworld.Greeter"
method: "SayHello"
}
retry_policy {
max_attempts: 5
initial_backoff {
nanos: 500000
}
max_backoff {
seconds: 30
}
backoff_multiplier: 2.0
retryable_status_codes: UNAVAILABLE
}
}
With the serialize/deserialize approach:
{
methodConfig=[ // field name was converted to cameCase
{
name=[
{
service=helloworld.Greeter,
method=SayHello
}
],
retryPolicy={
maxAttempts=5.0, // was integer originally
initialBackoff=0.000500s, // was Duration originally
maxBackoff=30s, // was Duration originally
backoffMultiplier=2.0,
retryableStatusCodes=[
UNAVAILABLE
]
}
}
]
}
With the custom converter approach:
{
method_config=[ // field names keep their snake_case format
{
name=[
{
service=helloworld.Greeter,
method=SayHello
}
],
retry_policy={
max_attempts=5, // Integers stay the same
initial_backoff={ // Duration values remains an object
nanos=500000
},
max_backoff={
seconds=30
},
backoff_multiplier=2.0,
retryable_status_codes=[
UNAVAILABLE
]
}
}
]
}
Bottom line
So which approach is better?
Well, it depends on what you are trying to do with the Map<String, ?>. In my case, I was configuring a grpc client to be retriable, which is done via ManagedChannelBuilder.defaultServiceConfig API. The API accepts a Map<String, ?> with this format.
After several trials and errors, I figured that the defaultServiceConfig API assumes you are using GSON, hence the serialize/deserialize approach worked for me.
One more advantage of the serialize/deserialize approach is that the Map<String, ?> can be easily converted back to the original protobuf value by serializing it back to json, then using the JsonFormat.parser() to obtain the protobuf object:
ServiceConfig original;
...
String asJson = JsonFormat.printer().print(original);
Map<String, ?> asMap = new Gson().fromJson(asJson, Map.class);
// Convert back to ServiceConfig
String backToJson = new Gson().toJson(asMap);
ServiceConfig.Builder builder = ServiceConfig.newBuilder();
JsonFormat.parser().merge(backToJson, builder);
ServiceConfig backToOriginal = builder.build();
... whereas the custom converter approach method doesn't have an easy way to convert back as you need to write a function to convert the map back to the original proto by navigating the tree.
I have a POJO class as:
public class D{
private JSONObject profileData;
public JSONObject getProfileData ()
{
return profileData;
}
public void setProfileData (JSONObject profileData)
{
this.profileData = profileData;
}
}
Now I populate this class like:
for (int i =0; i<identities.size();i++){
D d = new D();
d.setProfileData(profileData);
dList.add(d);
}
I create JSON object for profileData from GSON using a HashMap:
profileDataInJson = new JSONObject(gson.toJson(map1));
Where the signature of profileDataInJson is: JSONObject profileDataInJson = null;
Now the resultant JSON is like:
"profileData":{"map":{"ioCinema":"firstValue","ioSIMAvailable":"firstKey","Name":"onePair"}}
Wherein I get an unwanted object called map inserted in my main profileData object.
However when I print this inside the loop I get
{`"ioCinema":"firstValue","ioSIMAvailable":"firstKey","Name":"onePair"}`
Whish is exactly what I want inside profileData object, without nesting the map object.
How do I solve this?
"I am already aware that I can achieve this by converting the type of profileData in D class from JSONObject to String, which will induce escape characters - However I am looking for a generic solution"
EDIT:
map1 is constructed in two ways, depending on user input and both ways are as follows:
if (args.length >= 4 && args[1].equalsIgnoreCase("onePair")) {
map1 = new HashMap<>();
String key1 = args[2];
String value1 = args[3];
map1.put(key1, value1);
profileDataInJson = new JSONObject(gson.toJson(map1));
}
And:
if (args.length >= 1 && args[0].equalsIgnoreCase("update")) {
if (args.length >= 2)
profileData.setName(args[1] != null ? args[1] : "");
if (args.length >= 3)
profileData.setSIMAvailable(args[2] != null ? args[2] : "");
profileDataInJson = new JSONObject(profileData);
}
Signature: ProfileData profileData = new ProfileData();
The thing which puzzles me is when I try to traverse profileData and try to fetch the json object by name "map" I get a nullPointer exception
You don't need to use Gson to convert hashmap to a json object.
Simply use:
profileDataInJson = new JSONObject(map);
Add custom serializer to Gson, so that Gson serialize the org JSON as expected by you.
GsonBuilder gsonBuilder = new GsonBuilder();
gsonBuilder.registerTypeAdapter(JSONObject.class, new JsonSerializer<JSONObject>() {
#Override
public JsonElement serialize(final JSONObject src, final Type typeOfSrc,
final JsonSerializationContext context) {
return new JsonParser().parse(src.toString()).getAsJsonObject();
}
});
gsonBuilder.create().toJson(map1);
This will return {"ioCinema":"firstValue","ioSIMAvailable":"firstKey","Name":"onePair"}