I have a grpc client written in GO and a grpc server written in Java (both using the same proto files (syntax 2).
My grpc method takes a message that may contain extensions. I am able to construct a message containing desired extensions on the client and send it to the server. But when I try to read the message on the server, my extensions are available as unknown fields. (In other words, entity.hasExtension(extension) in Java returns false).
So my question is whether grpc allows extensions to be used in messages that are provided as method parameters. If not, is there a way to convert an unknown field to a field of specific type?
My proto file:
syntax = "proto2";
// proto file used as source for go client and java server as well
package my_services;
import "basic_types.proto";
// import "extension_types.proto";
// do not delete: options for generating java code
option java_multiple_files = true;
option java_package = "myservice.grpc";
option java_outer_classname = "MyServiceWrapper";
option objc_class_prefix = "Foo";
// Interface exposed by the server.
service DataService {
// Obtains all objects satisfying the request message
rpc MyMethod(DataRequest) returns (DataResponse) {}
}
message DataRequest {
optional IdDefinition id = 1;
repeated basic_types.Entity templates = 2;
}
message DataResponse {
repeated IdDefinition id = 1;
optional basic_types.DataResult result = 2;
}
message IdDefinition {
optional int32 myid = 1;
}
basic_types.Entity is a basic message containing extensions:
message Entity {
extensions 1 to max;
}
and may be extended e.g. like this:
extend basic_types.Entity {
optional Foo foo = 1000;
optional Bar bar = 1001;
}
Any help or hint would be much appreciated.
In java it is possible, but you need to set an extension registry with ProtoLiteUtils.setExtensionRegistry(). This is an experimental API, and there may be a different way in the future to do this, but for the time being it should be useable.
More generally, All message encodings are supported by gRPC. We natively support Proto3, but there are a lot of existing Proto2 users that use gRPC. Since gRPC is encoding agnostic, you can even use things like thrift or JSON if you really want to, though we don't automatically generate stubs for those.
Related
I am writing a program that works on two proto messages, I need to process the byte[] sent from different sources which sends either foo message or bar message. Since I cannot figure out which message it belongs to, I used Any Class (comes along with protobuf) to parse the byte array and
find which class it belongs to, but met with a compile time error. Is there any other method that I can use to detect if I add more proto message classes in future?
//Foo.proto
syntax = "proto3";
option java_outer_classname = "FooProto";
message Foo {
int32 a = 1;
}
and the second proto
//Bar.proto
syntax = "proto3";
option java_outer_classname = "BarProto";
message Bar {
int32 b = 1;
}
Code:
Any anyEvent = Any.parseFrom(protoBytes);
if (any.is(Foo.class)
{
Foo foo = any.unpack(Foo.class);
...
} else {
Bar bar = any.unpack(Bar.class);
...
}
Error in if statement while trying to invoke any.is() :
The method is(Class< T>) in the type Any is not applicable for the arguments (Class< Foo>)
Any doesn't mean "any"; it means "a type serialized via Any". If you didn't store it with Any: you can't decode it via Any.
The key point here is that protobuf does not include type metadata in a message payload. If you have a BLOB, you usually can't know what the message type is. Any solves that by encoding the message type in a wrapper message, but you won't have that here.
If your design is to have an API that accepts two different non-Any message types without prior knowledge of which it is: you probably have a bad design. Because that doesn't work with protobuf. On the wire, there is literally no difference between a Foo with a=42 and a Bar with b=42; the payloads are identical:
Foo with a=42 is the bytes 08 2A; Bar with b=42 is the bytes 08 2A. The 08 means "field 1, encoded as a varint", the 2A is a varint with raw value 42.
A better design might be a wrapper message specific to your scenario:
message RootMessage {
oneof test_oneof {
Foo foo = 1;
Bar bar = 2;
}
}
This adds a wrapper layer similar to how Any works, but much more efficiently - it just knows how to distinguish between your known types as an integer, rather than having to handle every possible type (as a rooted type name).
I use Apache Thrift protocol for tablet-server and interlanguage integration, and all is OK few years.
Integration is between languages (C#/C++/PC Java/Dalvik Java) and thrift is probably one of simplest and safest. So I want pack-repack sophisticated data structures (and changed over years) with Thrift library. Lets say in thrift terms kind of OfflineTransport or OfflineProtocol.
Scenario:
I want to make backup solution, for example during internet provider failure process data in offline mode: serialise, store, try to process in few ways. For example sent serialised data by normal email via poor backup connection etc.
Question is: where in Thrift philosophy is best extension point for me?
I understand, only part of online protocol is possible to backup offline, ie real time return of value is not possible, that is OK.
Look for serializer. There are misc. implementations but they all share the same common concept to use a buffer or file / stream as transport medium:
Writing data in C#
E.g. we plan to store the bits into a bytes[] buffer. So one could write:
var trans = new TMemoryBuffer();
var prot = new TCompactProtocol( trans);
var instance = GetMeSomeDataInstanceToSerialize();
instance.Write(prot);
Now we can get a hold of the data:
var data = trans.GetBuffer();
Reading data in C#
Reading works similar, except that you need to know from somewhere what root instance to construct:
var trans = new TMemoryBuffer( serializedBytes);
var prot = new TCompactProtocol( trans);
var instance = new MyCoolClass();
instance.Read(prot);
Additional Tweaks
One solution to the chicken-egg problem during load could be to use a union as an extra serialization container:
union GenericFileDataContainer {
1 : MyCoolClass coolclass;
2 : FooBar foobar
// more to come later
}
By always using this container as the root instance during serialization it is easy to add more classes w/o breaking compatibility and there is no need to know up front what exactly is in a file - you just read it and check what element is set in the union.
There is an RPC framework that uses the standard thrift Protocol named "thrifty", and it is the same effect as using thrift IDL to define the service, that is, thrify can be compatible with code that uses thrift IDL, which is very helpful for cross-platform. And has a ThriftSerializer class in it:
[ThriftStruct]
public class LogEntry
{
[ThriftConstructor]
public LogEntry([ThriftField(1)]String category, [ThriftField(2)]String message)
{
this.Category = category;
this.Message = message;
}
[ThriftField(1)]
public String Category { get; }
[ThriftField(2)]
public String Message { get; }
}
ThriftSerializer s = new ThriftSerializer(ThriftSerializer.SerializeProtocol.Binary);
byte[] s = s.Serialize<LogEntry>();
s.Deserialize<LogEntry>(s);
you can try it:https://github.com/endink/Thrifty
I'm currently working on an application built in Scala with Spray routing.
So for dealing with a JSON document sent over POST, it's pretty easy to access the variables within the body, as follows;
respondWithMediaType(`application/json`) {
entity(as[String]) { body =>
val msg = (parse(body) \ "msg").extract[String]
val url = (parse(body) \ "url").extractOpt[String]
However, I'm now trying to write an additional query with GET, and am having some issues accessing the parameters sent through with the query.
So, I'm opening with;
get {
respondWithMediaType(`application/json`) {
parameterSeq { params =>
var paramsList = params.toList
So, this works well enough in that I can access the GET params in a sequential order (just by accessing the index) - the problem is, unfortunately I don't think we can expect GET params to always be sent in the correct order.
The list itself prints out in the following format;
List((msg,this is a link to google), (url,http://google.com), (userid,13))
Is there any simple way to access these params? For example, something along the lines of;
var message = paramsList['msg']
println(message) //returns "this is a link to google"
Or am I going about this completely wrong?
Apologies if this is a stupid question - I've only switched over to Scala very recently, and am still getting both acquainted with that, and re-acquainted with Java.
What I usually do is use the parameters directive to parse the data out to a case class which contains all the relevant data:
case class MyParams(msg: String, url: String, userId: Int)
parameters(
"msg".as[String],
"url".as[String],
"userId".as[Int]
).as[MyParams] {
myParams =>
// Here you have the case class containing all the data, already parsed.
}
To build your routes you could use the parameters directives. I'm not sure if this is what you're looking for, anyway you could use them as:
get {
parameters('msg) { (msg) =>
complete(s"The message is '$msg'")
}
}
Spray directives can be easily composed so you can use combine them in any way you want.
I hope that helps you.
Recently I analyzed crash reports form my app and found several stack traces which points to okhttp
My app doesn't depend on okhttp explicitly.
AFAIK okhttp version depends on Android OS version, and okhttp library by itself placed on device
To help with troubleshooting I decided to log okhttp library version, and looks like I found several useful classes for this
com.squareup.okhttp.internal.Version
okhttp3.internal.Version
Just to make sure that I didn't mistake I took com.android.okhttp.internal.http.HttpURLConnectionImpl class form stack-trace and tried to Class.forName it - success
Also I noticed that com.squareup.okhttp transformed to com.android.okhttp looks like at build-time, so totally I tried such variants
Class.forName("com.android.okhttp.internal.Version") -> java.lang.ClassNotFoundException
Class.forName("com.squareup.okhttp.internal.Version") -> java.lang.ClassNotFoundException
Class.forName("okhttp3.internal.Version") -> java.lang.ClassNotFoundException
Class.forName("com.android.okhttp.internal.http.HttpURLConnectionImpl") -> success
Can anyone explain why? What I missed?
Update
I have pulled okhttp.jar from my device adb pull /system/framework/okhttp.jar but it contains MANIFEST.MF only
from 4.xx google is using okhttp part of squareup
/**
* This implementation uses HttpEngine to send requests and receive responses. This class may use
* multiple HttpEngines to follow redirects, authentication retries, etc. to retrieve the final
* response body.
*
* <h3>What does 'connected' mean?</h3> This class inherits a {#code connected} field from the
* superclass. That field is <strong>not</strong> used to indicate whether this URLConnection is
* currently connected. Instead, it indicates whether a connection has ever been attempted. Once a
* connection has been attempted, certain properties (request header fields, request method, etc.)
* are immutable.
*/
public class HttpURLConnectionImpl extends HttpURLConnection {
private String defaultUserAgent() {
String agent = System.getProperty("http.agent");
return agent != null ? Util.toHumanReadableAscii(agent) : Version.userAgent();
}
https://github.com/square/okhttp/blob/master/okhttp-urlconnection/src/main/java/okhttp3/internal/huc/HttpURLConnectionImpl.java
http://square.github.io/okhttp/
everything depends on device - what os version u using because api is evolving, u can use reflections but u need know what field is on specific api
see https://github.com/square/okhttp/blob/master/CHANGELOG.md
to compare diffrent api versions use: https://android.googlesource.com/platform/external/okhttp/
u can try at the beginning
System.getProperty("http.agent");
edit:
via reflections
HttpURLConnection connection = (HttpURLConnection) new URL("http://google.com")
.openConnection();
Method method = connection.getClass().getDeclaredMethod("defaultUserAgent");
method.setAccessible(true);
String okEngineVersion = (String) method.invoke(connection, new Object[]{});
same as
String okEngineVersion = System.getProperty("http.agent");
and if u want to bother:
every class is treated the same way - > as equals ( no versioning - u can only check magic minor major number - java compiler version from class)
manifest of /system/framework/okhttp.jar doesn't contain version properties
if u want okhttp.internal.Version class then:
File file = new File("/system/framework/okhttp.jar");
// using javaxt-core lib
Jar jar = new Jar(file);
jar.getVersion();
// load dex
DexFile dexfile = DexFile.loadDex(file.getAbsolutePath(),
File.createTempFile("opt", "dex", _context.getCacheDir()).getPath(), 0);
Enumeration<String> dexEntries = dexfile.entries();
ClassLoader systemClassLoader = DexClassLoader.getSystemClassLoader();
while (dexEntries.hasMoreElements()) {
String className = dexEntries.nextElement();
Class<?> aClass = systemClassLoader.loadClass(className);
}
conclusion: If you want to avoid crash of app from library changes delivery
own version of library and load classes on the fly or compile with apk
I need to show on my panel the working dir.
I use String value = System.getProperty("user.dir"). Afterwards i put this string on label but I receive this message on console:
The method getProperty(String, String) in the type System is not applicable for the arguments (String).
I use eclipse.
Issue
I am guessing you have not gone through GWT 101 - You cannot blindly use JAVA CODE on client side.
Explanation
You can find the list of classes and methods supported for GWT from JAVA.
https://developers.google.com/web-toolkit/doc/latest/RefJreEmulation
For System only the following are supported.
err, out,
System(),
arraycopy(Object, int, Object, int, int),
currentTimeMillis(),
gc(),
identityHashCode(Object),
setErr(PrintStream),
setOut(PrintStream)
Solution
In your case Execute System.getProperty("user.dir") in your server side code and access it using RPC or any other server side gwt communication technique.
System.getProperty("key") is not supported,
but System.getProperty("key", "default") IS supported, though it will only return the default value as there is not system properties per se.
If you need the working directory during gwt compile, you need to use a custom linker or generator, grab the system property at build time, and emit it as a public resource file.
For linkers, you have to export an external file that gwt can download and get the compile-time data you want. For generators, you just inject the string you want into compiled source.
Here's a slideshow on linkers that is actually very interesting.
http://dl.google.com/googleio/2010/gwt-gwt-linkers.pdf
If you don't want to use a linker and an extra http request, you can use a generator as well, which is likely much easier (and faster):
interface BuildData {
String workingDirectory();
}
BuildData data = GWT.create(BuildData.class);
data.workingDirectory();
Then, you need to make a generator:
public class BuildDataGenerator extends IncrementalGenerator {
#Override
public RebindResult generateIncrementally(TreeLogger logger,
GeneratorContext context, String typeName){
//generator boilerplate
PrintWriter printWriter = context.tryCreate(logger, "com.foo", "BuildDataImpl");
if (printWriter == null){
logger.log(Type.TRACE, "Already generated");
return new RebindResult(RebindMode.USE_PARTIAL_CACHED,"com.foo.BuildDataImpl");
}
SourceFileComposerFactory composer =
new SourceFileComposerFactory("com.foo", "BuildDataImpl");
//must implement interface we are generating to avoid class cast exception
composer.addImplementedInterface("com.foo.BuildData");
SourceWriter sw = composer.createSourceWriter(printWriter);
//write the generated class; the class definition is done for you
sw.println("public String workingDirectory(){");
sw.println("return \""+System.getProperty("user.dir")+"\";");
sw.println("}");
return new RebindResult(RebindMode.USE_ALL_NEW_WITH_NO_CACHING
,"com.foo.BuildDataImpl");
}
}
Finally, you need to tell gwt to use your generator on your interface:
<generate-with class="dev.com.foo.BuildDataGenerator">
<when-type-assignable class="com.foo.BuildData" />
</generate-with>