We Keep Coding

Trying to read a serialized Java object that I did not create

I'm trying to create a web GUI for a Minecraft game server I run. The data I'm trying to read is from CoreProtect, a logging plugin.
I'm mainly using PHP and then trying to write a small Java service that can convert the serialized data into a JSON string that I can then use - since I can't deserialize a Java object directly in PHP and it's only some meta data that's stored as a Java serialized object, the rest is normal non-blob columns.
I've identified that the CoreProtect plugin is using ObjectOutputStream to serialize the object and then writes it to a MySQL BLOB field, this is the code I've identified from CoreProtect that's handling this:
try {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(bos);
oos.writeObject(data);
oos.flush();
oos.close();
bos.close();
byte[] byte_data = bos.toByteArray();
preparedStmt.setInt(1, time);
preparedStmt.setObject(2, byte_data);
preparedStmt.executeUpdate();
preparedStmt.clearParameters();
} catch (Exception e) {
e.printStackTrace();
}
This is then outputting the bytes to the database. All of the rows in the database start with the same few characters (from what I've seen this should be Java's 'magic' header). However, when trying to use the below code to unserialize the data I receive an error stating that the header is corrupt 'invalid stream header: C2ACC3AD'
byte[] serializedData = ctx.bodyAsBytes();
ByteArrayInputStream bais = new ByteArrayInputStream(serializedData);
try {
ObjectInputStream ois = new ObjectInputStream(bais);
Object object = ois.readObject();
Gson gson = new Gson();
ctx.result(gson.toJson(object));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
I'm using Javalin as the web service and am just sending the raw output from the BLOB column to a post route, I'm then reading this with the bodyAsBytes method. I'm tried testing this by writing the BLOB column to a file and then copying the contents of the file into a test POST request using Postman. I've also tried using a PHP script to read directly from the DB and then send that as a POST request and I just get the same error.
I've looked into this and everything is pointing to corrupt data. However, the odd thing is when triggering a 'restore' via the CoreProtect plugin it correctly restores what it needs to, reading all of the relevant data from the database including this column. From what I've seen in CoreProtect's JAR it's just doing the same process with the InputStream method calls.
I'm not very familiar with Java and thought this would be a fairly simple process. Am I missing something here? I don't see anything in the CoreProtect plugin that may be overriding the stream header. It's unfortunately not open source so I'm having to use a Java decompiler to try and see how it's serializing the object so that I can then try and read it, I assume it's possible the decompiler is not reading how this is serialized/unserialized.
My other thought was maybe the 'magic' header changed between Java versions? Although I couldn't seem to confirm this online. The specific header I'm receiving I've also seen in some other similar posts, although those all lead to data corruption/using a different output stream.
I appreciate any help with this, it's not an essential feature but would be nice if I can read all of the data related to the logs generated by the server/plugin.
I understand the use case is niche, but hoping the issue/resolution is a general Java problem :).
Update ctx is an instance of Javelin's Context class. Since I'm trying to send a raw POST request to Java I needed some kind of web service and Javelin looked easy/lightweight for what I needed. On the PHP side I'm just reading the column from the database and then using Guzzle to send a raw body with the result to the Javelin service that's running.

Something, apparently ctx, is treating the binary data as a String. bodyAsBytes() is converting that String to bytes using the String’s getBytes() method, which immediately corrupts the data.
The first two bytes of a serialized Java object stream are always AC ED. getAsBytes() is treating these as characters, namely U+00AC NOT SIGN and U+00ED LATIN SMALL LETTER I WITH ACUTE. When encoded in UTF-8, these two characters produce the bytes C2 AC C3 AD, which is what you’re seeing.
Solution: Do not treat your data as a String under any circumstances. Binary data should only be treated as a byte stream. (I’m well aware that it was common to store bytes in a string in C, but Java is not C, and String and byte[] are distinct types.)
If you update your question and state what the type of ctx is, we may be able to suggest an exact course of action.

How do I know what the json for a protobuff should be formatted like?

I'm new to protobuf's, and I'd like to save some protobuf's to json format, and know what the full format for the protobuf is. I've tried just creating an empty instance of the protobuf, and saving it to json, but that only gives me an empty json object, {}.
If I fill in a value for a property, and serialize that, I get the property in the json, which is great, but I don't want to have to do this for all the properties of each protobuf I want to do this for.
Is there a way for me to see the full json format for a protobuf without supplying a value for every field?
Notes
I'm using Google's protobuf library in Java, and can serialize and deserialize my objects, I'm just not sure how to write the json for a particular object.
I've reviewed this stackoverflow question for info, but found nothing that helped.

Yes, the JSON formatting for proto3 is documented.
Alternatively, to see an example without changing the defaults, you could specify the includingDefaultValueFields when printing:
String json = JsonFormat.printer().includingDefaultValueFields().print(message);
(That should at least work for primitives; I suspect it will print null for nested messages if they haven't been initialized.)

Not really any different that what was done in this answer to this question, but here's how I wrapped this up for my purposes - your results may vary, lol! This allowed me to have messages loaded from a json file, and deserialized into requests for grpc methods.
import com.google.protobuf.InvalidProtocolBufferException;
import com.google.protobuf.MessageOrBuilder;
import com.google.protobuf.util.JsonFormat;
/**
* Convert gRPC message to Json string.
*
* #param messageOrBuilder the gRPC message
* #return a Json string
*/
public static String grpcMessageToJson(MessageOrBuilder messageOrBuilder) {
String result = "";
if (messageOrBuilder == null) {
return result;
}
try {
result = JsonFormat.printer().print(messageOrBuilder);
} catch (InvalidProtocolBufferException e) {
LOGGER.warn("Cannot serialize the gRPC message.", e);
}
return result;
}

How to convert protocol buffers binary to JSON using the descriptor in Java

I have a message with a field of the "Any" well known type which can hold a serialized protobuf message of any type.
I want to convert this field to its json representation.
I know the field names are required, and typically you would need the generated classes loaded in the app for this to work, but I am looking for a way to do it with the descriptors.
First, I parse the descriptors:
FileInputStream descriptorFile = new FileInputStream("/descriptor");
DescriptorProtos.FileDescriptorSet fdp = DescriptorProtos.FileDescriptorSet.parseFrom(descriptorFile);
Then, loop through the contained messages and find the correct one (using the "Any" type's URL, which contains the package and message name. I add this to a TypeRegistry which is used to format the JSON.
JsonFormat.TypeRegistry.Builder typeRegistryBuilder = JsonFormat.TypeRegistry.newBuilder();
String messageNameFromUrl = member.getAny().getTypeUrl().split("/")[1];
for (DescriptorProtos.FileDescriptorProto file : fdp.getFileList()) {
for (DescriptorProtos.DescriptorProto dp : file.getMessageTypeList()) {
if (messageNameFromUrl.equals(String.format("%s.%s", file.getPackage(), dp.getName()))) {
typeRegistryBuilder.add(dp.getDescriptorForType()); //Doesn't work.
typeRegistryBuilder.add(MyConcreteGeneratedClass.getDescriptor()); //Works
System.out.println(JsonFormat.printer().usingTypeRegistry(typeRegistryBuilder.build()).preservingProtoFieldNames().print(member.getAny()));
return;
}
}
}
The problem seems to be that parsing the descriptor gives me access to Descriptors.DescriptorProto objects, but I see no way to get the Descriptors.Descriptor object needed for the type registry. I can access the concrete class's descriptor with getDescriptor() and that works, but I am trying to format the JSON at runtime by accessing a pre-generated descriptor file from outside the app, and so I do not have that concrete class available to call getDescriptor()
What would be even better is if I could use the "Any" field's type URL to resolve the Type object and use that to generate the JSON, since it also appears to have the field numbers and names as required for this process.
Any help is appreciated, thanks!

If you convert a DescriptorProtos.FileDescriptorProto to Descriptors.FileDescriptor, the latter has a getMessageTypes() method that returns List<Descriptor>.
Following is a snippet of Kotlin code taken from an open-source library I'm developing called okgrpc. It's the first of its kind attempt to create a dynamic gRPC client/CLI in Java.
private fun DescriptorProtos.FileDescriptorProto.resolve(
index: Map<String, DescriptorProtos.FileDescriptorProto>,
cache: MutableMap<String, Descriptors.FileDescriptor>
): Descriptors.FileDescriptor {
if (cache.containsKey(this.name)) return cache[this.name]!!
return this.dependencyList
.map { (index[it] ?: error("Unknown dependency: $it")).resolve(index, cache) }
.let {
val fd = Descriptors.FileDescriptor.buildFrom(this, *it.toTypedArray())
cache[fd.name] = fd
fd
}
}

java android application cannot make http get

Can someone helps me to get json from the web.In the end of function jsonResponse is empty. I use this method to do it:
private String getJson() {
jsonResponsce = "";
AsyncTask.execute(new Runnable() {
#Override
public void run() {
try{
URL httpbinEndpoint = new URL(webPage);
HttpsURLConnection myConnection = (HttpsURLConnection) httpbinEndpoint.openConnection();
myConnection.setRequestMethod("GET");
// Enable writing
myConnection.setDoOutput(true);
String internetData = "";
// Write the data
myConnection.getOutputStream().write(internetData.getBytes());
jsonResponsce = internetData;
} catch (IOException e) {
e.printStackTrace();
}
}
});
return jsonResponsce;
}
I set an Internet permission to the manifest. I try go get Json from the next address: https://shori-dodjo-mobile-app.firebaseio.com/.json. Full code is placed here: https://github.com/GenkoKaradimov/Shori-Dodjo-Android-App/

You are executing the request asynchronously so the method starts the execution and then completes and therefore there is no result. The result will be there in a second but by that time the method getJson has already completed. You most probably need to put the code that uses the json at the end of the run method.
In addition your code for reading from the stream seems wrong. It should probably be something like
BufferedReader br = new BufferedReader(new InputStreamReader(myConnection.getInputStream()));
jsonResponsce = br.lines().collect(Collectors.joining("\n"));
(I haven't tested this)

There are multiple issues in your code:
First, AsyncTask means it's async(hronous), so you can't return the result right away. Instead, override AsyncTask's onPostExecute and do what you need to do with the data there. Here is the sample implementation.
Second, you're using getOutputStream, which is intended for writing to the connection, i.e. sending data to the server. In your case you need to getInputStream and read from it. Easiest way is to wrap it in a BufferedReader and read until it returns -1 (marking end of stream), and then convert to string.
There are a few quirks: You should handle, or at least recognize errors by checking HTTP status code, handle encodings (the convert-bytes-to-string part), and handle cases when response is compressed, e.g. using DEFLATE or gzip. I've implemented that in a pure Java way (reference code, warning: outdated docs), but I'd seriously recommend using one of the established libraries such as Retrofit or Volley.

Json objects usually get return as HashMaps.
So you might need something like, HashMap yourMap = new HashMap<~>();
then
yourMap.get("the objects name on the other side", the var its getting saved too.);
right now it looks like all you are trying to do is save the byte data, but this byte data needs to have a type. Hope this helps.

jInterface to create External Erlang Term

How can I format the the following erlang term:
{ atom, "message" }
In jInterface to an external format that I may call in an erlang shell
erlang:binary_to_term( Binary )
Example:
Note that since the tuple will be sent over the net, I finish by converting to byte[].
OtpErlangObject[] msg = new OtpErlangObject[2];
msg[0] = new OtpErlangAtom( "atom" );
msg[1] = new OtpErlangString( "message" );
OtpErlangTuple reply = new OtpErlangTuple(msg);
OtpOutputStream stream = new OtpOutputStream(reply);
stream.toByteArray() // byte[] which I send over net
The binary received by Erlang is:
B = <<104,2,100,0,4,97,116,111,109,107,0,7,109,101,115,115,97,103,101>>
Then in an erlang shell converting the received term to binary gives a badarg.
binary_to_term( B ).
** exception error: bad argument
in function binary_to_term/1
called as binary_to_term(<<104,2,107,0,4,97,116,111,109,107,0,7,109,
101,115,115,97,103,101>>)

binary_to_term( <<131,104,2,107,0,4,97,116,111,109,107,0,7,109,101,115,115,97,103,101>> ).
{"atom","message"}
It seems that the message is missing the 131 tag required by term_to_binary. As is evident from the Java output, this tag is not being added by jinterface encode. If I simply add 131 to the beginning of the binary it decodes correctly.
Now why is Java not adding it?
I will still accept answers as I have not officially answered my question ( in a supported way ie. not hacking with 131 )
Ref:
http://www.erlang.org/doc/apps/erts/erl_ext_dist.html

I haven't tested this, but if you're encoding {atom, "message"}, shouldn't you be sending over a tuple, not 2 objects one after the other? Try creating a Tuple object and adding atom and message as elements.