We Keep Coding

Java: Protobuf byte to Json string to Pojo fast

I am receiving messages in protobuf format. I need to convert it to json format fast as all my business logic is written to handle json based POJO objects.
byte[] request = ..; // msg received
// convert to intermediate POJO
AdxOpenRtb.BidRequest bidRequestProto = AdxOpenRtb.BidRequest.parseFrom(request, reg);
// convert intermediate POJO to json string.
// THIS STEP IS VERY SLOW
Printer printer = JsonFormat.printer().printingEnumsAsInts().omittingInsignificantWhitespace();
String jsonBody = printer.print(bidRequestProto);
// convert json string to final POJO format
BidRequest bidRequest = super.parse(jsonBody.getBytes());
Proto object to json conversion step is very slow. Is there any faster approach for it?
can i reuse printer object? is it thread-safe?
Note: This POJO class (AdxOpenRtb.BidRequest & BidRequest) is very complex having many hierarchy and fields but contains similar data with slightly different fields name and data types.

I ran into some performance issues as well and ended up writing the QuickBuffers library. It generates dedicated JSON serialization methods (i.e. no reflection) and should give you a 10-30x speedup. It can be used side-by-side with Google's implementation. The code should look something like this:
// Initialization (objects can be reused if desired)
AdxOpenRtb.BidRequest bidRequestProto = AdxOpenRtb.BidRequest.newInstance();
ProtoSource protoSource = ProtoSource.newArraySource();
JsonSink jsonSink = JsonSink.newInstance().setWriteEnumsAsInts(true);
// Convert Protobuf to JSON
bidRequestProto.clearQuick() // or ::parseFrom if you want a new object
.mergeFrom(protoSource.setInput(request))
.writeTo(jsonSink.clear());
// Use the raw json bytes
RepeatedByte jsonBytes = jsonSink.getBytes();
JsonSinkBenchmark has some sample code for replacing the built-in JSON encoder with more battle-tested Gson/Jackson backends.
Edit: if you're doing this within a single process and are worried about performance, you're better off writing or generating code to convert the Java objects directly. JSON is not a very efficient format to go through.

I end up using MapStruct as suggested by some of you (#M.Deinum).
new code:
byte[] request = ..; // msg received
// convert to intermediate POJO
AdxOpenRtb.BidRequest bidRequestProto = AdxOpenRtb.BidRequest.parseFrom(request, reg);
// direct conversion from protobuf Pojo to my custom Pojo
BidRequest bidRequest = BidRequestMapper.INSTANCE.adxOpenRtbToBidRequest(bidRequestProto);
Code snippet of BidRequestMapper:
#Mapper(
collectionMappingStrategy = CollectionMappingStrategy.ADDER_PREFERRED, nullValueCheckStrategy = NullValueCheckStrategy.ALWAYS,
unmappedSourcePolicy = ReportingPolicy.WARN, unmappedTargetPolicy = ReportingPolicy.WARN)
#DecoratedWith(BidRequestMapperDecorator.class)
public abstract class BidRequestMapper {
public static final BidRequestMapper INSTANCE = Mappers.getMapper(BidRequestMapper.class);
#Mapping(source = "impList", target = "imp")
#Mapping(target = "impOverride", ignore = true)
#Mapping(target = "ext", ignore = true)
public abstract BidRequest adxOpenRtbToBidRequest(AdxOpenRtb.BidRequest adxOpenRtb);
...
...
}
// manage proto extensions
abstract class BidRequestMapperDecorator extends BidRequestMapper {
private final BidRequestMapper delegate;
BidRequestMapperDecorator(BidRequestMapper delegate) {
this.delegate = delegate;
}
#Override
public BidRequest adxOpenRtbToBidRequest(AdxOpenRtb.BidRequest bidRequestProto) {
// Covert protobuf msg to basic bid request object
BidRequest bidRequest = delegate.adxOpenRtbToBidRequest(bidRequestProto);
...
...
}
}
The new approach is 20-50x faster in my local test environment.
It's worth mentioning that MapStruct is an annotation processor which makes it much faster than other similar libraries which use reflection and it also has very good support for customization.

Renaming video files with a Java program

I have a folder containing many videos that i'd like to rename. I can't think of any convenient way of doing so. The naming convention is the following "SeasonX, EpisodeY: Episode name". This is going to be "SXEY:Name" for short.
An example: S01E01:JavaCode
That would be Season One, Episode One of Episode called JavaCode.
I wrote something that is able to change the file names, but I need different and unique file names for every episode because it's a TV show.
Here's the code:
import java.io.File;
import java.util.TreeMap;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class BatchFileRenamer {
public static void main(String[] args) {
// TODO Auto-generated method stub
File folder = new File("C:\\Users\\Tony\\Videos\\New folder");
TreeMap map = new TreeMap();
String name = "name";
File[] files = folder.listFiles();
Pattern p = Pattern.compile("\\..*");
for (int i = 0; i != files.length; i++) {
Matcher m = p.matcher(files[i].getName());
System.out.println(files[i].getName());
m.find();
files[i].renameTo(new File(folder.getAbsolutePath() + "\\" + name + " S01E" +
(i < 10 ? "1" : "") + i + m.group()));
}
}
}
I was thinking of creating an array containing the episode names but that's just as much work as manually renaming them in Windows. I guess if I had a txt file to download for all the TV shows with the names of the episodes in it it'd be useful.
Anyway, any suggestions would be greatly appreciated!

I think the best way to do this would be to use the Open Movie Database API. With this, you can get a REST response including a list of episodes for each season of a show. (Example request).
With this, you could use Gson or another parser to serialize the list of episodes:
Here is a Gist of some sample code. (There is probably a better getter method, but you get the point)
What the code does is it gets the information from the sample request above via the API, then it serializes it into a basic POJO from the Episodes.java class using Gson:
Gson gson = new Gson();
Episodes episodes = gson.fromJson(download, Episodes.class);
System.out.println(episodes);
You can then use this information to create the individual file names for the video files.

Deserializing SOAP response into generated classes C#

We are integrating with a third party and are having trouble deserializing the response we get from a request. From fiddler, I can see the response:
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body>
<ns2:updateItemsResponse xmlns:ns2="http://ThirdParty/dmn/items/1.1">
<fileId>bd6e7d83-8bfu-4573-fe32-c67c04355dd0</fileId>
<fileName>12345.MBG_Items.002.2016-10-26-16-23-35.xml</fileName>
</ns2:updateItemsResponse>
But we cannot deserialize it properly into an updateItemsResponse object. The object is created but the fileId and fileName are both null.
We generated the classes for this request and response and below is the class that was generated:
[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.CodeDom.Compiler.GeneratedCodeAttribute("System.ServiceModel", "4.0.0.0")]
[System.ServiceModel.MessageContractAttribute(WrapperName = "updateItemsResponse", WrapperNamespace = "http://ThirdParty/dmn/items/1.1", IsWrapped = true)]
public partial class updateItemsResponse
{
[System.ServiceModel.MessageBodyMemberAttribute(Namespace = "http://ThirdParty/dmn/items/1.1", Order = 0)]
public string fileName;
[System.ServiceModel.MessageBodyMemberAttribute(Namespace = "http://ThirdParty/dmn/items/1.1", Order = 1)]
public string fileId;
public updateItemsResponse()
{
}
public updateItemsResponse(string fileId, string fileName)
{
this.fileId= fileId;
this.fileName= fileName;
}
}
We know the web service was written in Java, and we had some issues generating the class files - the wsdl we were provided needed some jiggery pokery before we were able to create the class files using svcutil.exe. So I think there is just some tweak I need to make in the generated file.
I've done some searching around this and I am guessing the issue here is the namespaces but I cannot get this to work. I've tried changing the namespace, removing the namespace, changing the order for fileId and fileName - but none of these have made any difference.
Anyone able to see anything obvious I might be missing? It's driving me crazy!!!
EDIT: Have tried this changing the fileId and fileName to XmlElement too but still getting null responses.

Eureka!!!
So it was the attributes that were wrong - on fileId and fileName. Just needed to change them to
[System.ServiceModel.MessageBodyMemberAttribute(Name = "fileId", Namespace = "", Order = 0)]
public string fileId;
[System.ServiceModel.MessageBodyMemberAttribute(Name = "fileName", Namespace = "", Order = 1)]
public string fileName;
and I can now deserialize the response.

How to consume a H2OModel computed and saved in Python API

I've been reading the H2O documentation for a while, and I haven't found a clear example of how to load model trained and saved using the Python API. I was following the next example.
import h2o
from h2o.estimators.naive_bayes import H2ONaiveBayesEstimator
model = H2ONaiveBayesEstimator()
h2o_df = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/airlines/allyears2k_headers.zip")
model.train(y = "IsDepDelayed", x = ["Year", "Origin"],
training_frame = h2o_df,
family = "binomial",
lambda_search = True,
max_active_predictors = 10)
h2o.save_model(model, path=models)
But if you check the official documentation it states that you have to download the model as a POJO from the flow UI. Is it the only way? or, may I achieve the same result via python? Just for information, I show the doc's example below. I need some guidance.
import java.io.*;
import hex.genmodel.easy.RowData;
import hex.genmodel.easy.EasyPredictModelWrapper;
import hex.genmodel.easy.prediction.*;
public class main {
private static String modelClassName = "gbm_pojo_test";
public static void main(String[] args) throws Exception {
hex.genmodel.GenModel rawModel;
rawModel = (hex.genmodel.GenModel) Class.forName(modelClassName).newInstance();
EasyPredictModelWrapper model = new EasyPredictModelWrapper(rawModel);
//
// By default, unknown categorical levels throw PredictUnknownCategoricalLevelException.
// Optionally configure the wrapper to treat unknown categorical levels as N/A instead:
//
// EasyPredictModelWrapper model = new EasyPredictModelWrapper(
// new EasyPredictModelWrapper.Config()
// .setModel(rawModel)
// .setConvertUnknownCategoricalLevelsToNa(true));
RowData row = new RowData();
row.put("Year", "1987");
row.put("Month", "10");
row.put("DayofMonth", "14");
row.put("DayOfWeek", "3");
row.put("CRSDepTime", "730");
row.put("UniqueCarrier", "PS");
row.put("Origin", "SAN");
row.put("Dest", "SFO");
BinomialModelPrediction p = model.predictBinomial(row);
System.out.println("Label (aka prediction) is flight departure delayed: " + p.label);
System.out.print("Class probabilities: ");
for (int i = 0; i < p.classProbabilities.length; i++) {
if (i > 0) {
System.out.print(",");
}
System.out.print(p.classProbabilities[i]);
}
System.out.println("");
}
}

h2o.save_model will save the binary model to the provided file system, however, looking at the Java application above it seems you want to use model into a Java based scoring application.
Because of that you should be using h2o.download_pojo API to save the model to local file system along with genmodel jar file. The API is documented as below:
download_pojo(model, path=u'', get_jar=True)
Download the POJO for this model to the directory specified by the path; if the path is "", then dump to screen.
:param model: the model whose scoring POJO should be retrieved.
:param path: an absolute path to the directory where POJO should be saved.
:param get_jar: retrieve the h2o-genmodel.jar also.
Once you have download POJO, you can use the above sample application to perform the scoring and make sure the POJO class name and the "modelClassName" are same along with model type.

How to extract fields from message object content (Jade)

I have some problem with object type in JADE. In order to send object message to another agent I am using a method ACLMessage.setContentObject. However when I want to extract particular values/fields from the received message problem is arising.
Additionaly in sending agent I have created class to make possible sending message with object content:
class BusData implements java.io.Serializable
{
public Double current;
public Double power1;
public Double power2;
public Double voltage;
}
Receiving agent has following code (part):
data = MessageTemplate.MatchConversationId("measure");
ACLMessage data1 = receive(data); //receiving data with defined template
if (data1!=null) {
for (int i = 0; i < Agents.length; ++i) {
try {
Data1 received_data = (Data1)data1.getContentObject();
} catch (UnreadableException e) {
e.printStackTrace();}
//Serializable so = (Serializable)data1;
//System.out.println(Agents[i].getName()); //from whom received
if (data1!=null) {
System.out.println(getLocalName()+ " Info from " + data1.getSender().getName()); //from whom received
}
}
}
Should I add in receiving agent similar class, e.g. BusData1 with similar variables, like in sending agent in order to extract message content? I am quite new in Java so I am asking for understanding.
Every hint will be helpful.
Regards

Its too late for you but someone might be looking for it :)
In your receiving agent, instead of:
Data1 received_data = (Data1)data1.getContentObject();
Just do :
BusData received_data = (BusData)data1.getContentObject();
You can then access your received_data public variables (The cast allows the system to understand what is the serialized object's class to trigger the unserialization).
Tip : You should use variables names that correspond to their content.
data --> messageTemplate
data1 --> receivedMessage