I'm trying to create a web GUI for a Minecraft game server I run. The data I'm trying to read is from CoreProtect, a logging plugin.
I'm mainly using PHP and then trying to write a small Java service that can convert the serialized data into a JSON string that I can then use - since I can't deserialize a Java object directly in PHP and it's only some meta data that's stored as a Java serialized object, the rest is normal non-blob columns.
I've identified that the CoreProtect plugin is using ObjectOutputStream to serialize the object and then writes it to a MySQL BLOB field, this is the code I've identified from CoreProtect that's handling this:
try {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(bos);
oos.writeObject(data);
oos.flush();
oos.close();
bos.close();
byte[] byte_data = bos.toByteArray();
preparedStmt.setInt(1, time);
preparedStmt.setObject(2, byte_data);
preparedStmt.executeUpdate();
preparedStmt.clearParameters();
} catch (Exception e) {
e.printStackTrace();
}
This is then outputting the bytes to the database. All of the rows in the database start with the same few characters (from what I've seen this should be Java's 'magic' header). However, when trying to use the below code to unserialize the data I receive an error stating that the header is corrupt 'invalid stream header: C2ACC3AD'
byte[] serializedData = ctx.bodyAsBytes();
ByteArrayInputStream bais = new ByteArrayInputStream(serializedData);
try {
ObjectInputStream ois = new ObjectInputStream(bais);
Object object = ois.readObject();
Gson gson = new Gson();
ctx.result(gson.toJson(object));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
I'm using Javalin as the web service and am just sending the raw output from the BLOB column to a post route, I'm then reading this with the bodyAsBytes method. I'm tried testing this by writing the BLOB column to a file and then copying the contents of the file into a test POST request using Postman. I've also tried using a PHP script to read directly from the DB and then send that as a POST request and I just get the same error.
I've looked into this and everything is pointing to corrupt data. However, the odd thing is when triggering a 'restore' via the CoreProtect plugin it correctly restores what it needs to, reading all of the relevant data from the database including this column. From what I've seen in CoreProtect's JAR it's just doing the same process with the InputStream method calls.
I'm not very familiar with Java and thought this would be a fairly simple process. Am I missing something here? I don't see anything in the CoreProtect plugin that may be overriding the stream header. It's unfortunately not open source so I'm having to use a Java decompiler to try and see how it's serializing the object so that I can then try and read it, I assume it's possible the decompiler is not reading how this is serialized/unserialized.
My other thought was maybe the 'magic' header changed between Java versions? Although I couldn't seem to confirm this online. The specific header I'm receiving I've also seen in some other similar posts, although those all lead to data corruption/using a different output stream.
I appreciate any help with this, it's not an essential feature but would be nice if I can read all of the data related to the logs generated by the server/plugin.
I understand the use case is niche, but hoping the issue/resolution is a general Java problem :).
Update ctx is an instance of Javelin's Context class. Since I'm trying to send a raw POST request to Java I needed some kind of web service and Javelin looked easy/lightweight for what I needed. On the PHP side I'm just reading the column from the database and then using Guzzle to send a raw body with the result to the Javelin service that's running.
Something, apparently ctx, is treating the binary data as a String. bodyAsBytes() is converting that String to bytes using the String’s getBytes() method, which immediately corrupts the data.
The first two bytes of a serialized Java object stream are always AC ED. getAsBytes() is treating these as characters, namely U+00AC NOT SIGN and U+00ED LATIN SMALL LETTER I WITH ACUTE. When encoded in UTF-8, these two characters produce the bytes C2 AC C3 AD, which is what you’re seeing.
Solution: Do not treat your data as a String under any circumstances. Binary data should only be treated as a byte stream. (I’m well aware that it was common to store bytes in a string in C, but Java is not C, and String and byte[] are distinct types.)
If you update your question and state what the type of ctx is, we may be able to suggest an exact course of action.
Related
The following code is used to serialize the data.
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
BinaryEncoder binaryEncoder =
EncoderFactory.get().binaryEncoder(byteArrayOutputStream, null);
DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<>(data.getSchema());
datumWriter.setSchema(data.getSchema());
datumWriter.write(data, binaryEncoder);
binaryEncoder.flush();
byteArrayOutputStream.close();
result = byteArrayOutputStream.toByteArray();
I used the following command
FileUtils.writeByteArrayToFile(new File("D:/sample.avro"), data);
to write avro byte array to a file. But when I try to read the same using
File file = new File("D:/sample.avro");
try {
dataFileReader = new DataFileReader(file, datumReader);
} catch (IOException exp) {
System.out.println(exp);
System.exit(1);
}
it throws exception
java.io.IOException: Not a data file.
at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:105)
at org.apache.avro.file.DataFileReader.<init>(DataFileReader.java:97)
at org.apache.avro.file.DataFileReader.<init>(DataFileReader.java:89)
What is the problem happening here. I refered two other similar stackoverflow questions this and this but haven't been of much help to me. Can someone help me understand this.
The actual data is encoded in the Avro binary format, but typically what's passed around is more than just the encoded data.
What most people think of an "avro file" is a format that includes the header (which has things like the writer schema) and then the actual data: https://avro.apache.org/docs/current/spec.html#Object+Container+Files. The first four bytes of an avro file should be b"Obj1" or 0x4F626A01. The error you are getting is because the binary you are trying to read as a data file doesn't start with the standard magic bytes.
Another standard format is the single object encoding: https://avro.apache.org/docs/current/spec.html#single_object_encoding. This type of binary format should start with 0xC301.
But if I had to guess, the binary you have could just be the raw serialized data without any sort of header information. Though it's hard to know for sure without knowing how the byte array that you have was created.
You'd need to utilize Avro to write the data as well as read it otherwise the schema isn't written (hence the "Not a data file" message). (see: https://cwiki.apache.org/confluence/display/AVRO/FAQ#FAQ-HowcanIserializedirectlyto/fromabytearray?)
If you're just looking to serialize an object, see: https://mkyong.com/java/how-to-read-and-write-java-object-to-a-file/
Can someone helps me to get json from the web.In the end of function jsonResponse is empty. I use this method to do it:
private String getJson() {
jsonResponsce = "";
AsyncTask.execute(new Runnable() {
#Override
public void run() {
try{
URL httpbinEndpoint = new URL(webPage);
HttpsURLConnection myConnection = (HttpsURLConnection) httpbinEndpoint.openConnection();
myConnection.setRequestMethod("GET");
// Enable writing
myConnection.setDoOutput(true);
String internetData = "";
// Write the data
myConnection.getOutputStream().write(internetData.getBytes());
jsonResponsce = internetData;
} catch (IOException e) {
e.printStackTrace();
}
}
});
return jsonResponsce;
}
I set an Internet permission to the manifest. I try go get Json from the next address: https://shori-dodjo-mobile-app.firebaseio.com/.json. Full code is placed here: https://github.com/GenkoKaradimov/Shori-Dodjo-Android-App/
You are executing the request asynchronously so the method starts the execution and then completes and therefore there is no result. The result will be there in a second but by that time the method getJson has already completed. You most probably need to put the code that uses the json at the end of the run method.
In addition your code for reading from the stream seems wrong. It should probably be something like
BufferedReader br = new BufferedReader(new InputStreamReader(myConnection.getInputStream()));
jsonResponsce = br.lines().collect(Collectors.joining("\n"));
(I haven't tested this)
There are multiple issues in your code:
First, AsyncTask means it's async(hronous), so you can't return the result right away. Instead, override AsyncTask's onPostExecute and do what you need to do with the data there. Here is the sample implementation.
Second, you're using getOutputStream, which is intended for writing to the connection, i.e. sending data to the server. In your case you need to getInputStream and read from it. Easiest way is to wrap it in a BufferedReader and read until it returns -1 (marking end of stream), and then convert to string.
There are a few quirks: You should handle, or at least recognize errors by checking HTTP status code, handle encodings (the convert-bytes-to-string part), and handle cases when response is compressed, e.g. using DEFLATE or gzip. I've implemented that in a pure Java way (reference code, warning: outdated docs), but I'd seriously recommend using one of the established libraries such as Retrofit or Volley.
Json objects usually get return as HashMaps.
So you might need something like, HashMap yourMap = new HashMap<~>();
then
yourMap.get("the objects name on the other side", the var its getting saved too.);
right now it looks like all you are trying to do is save the byte data, but this byte data needs to have a type. Hope this helps.
I'm trying to (HTTP) POST an image to a Nodejs server that is configured using Express. I have been able to accomplish this successfully using JSON, but unless I am mistaken, there is no way to obtain the image string without loading the entire request body into a new variable before parsing it as JSON. Since images are quite large and the image should already be stored in the request body anyway, is there a way to immediately pipe the image contents into fs.writeFile()? The content type for the request does not have to be JSON. I have tried using a querystring as well, but that was unsuccessful. The content type cannot be just an image though because I have to include a tag for the image too (in this case the user's email address).
Here is the code for when I attempted to use a query string. It is located in a post route method for the express app:
fs.writeFile('profiles/images/user.png', new Buffer(req.body.image, 'base64'),
function(error)
{
if (error)
res.end(error);
}
);
No error occurs, and the code creates the .png file, but the file is somehow corrupted and is larger than it should be.
All of this is actually for an Android app, so here is also the Java code that I am using to send the request:
URLConnection connection = new URL(UPLOAD_PICTURE_URL).openConnection();
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;");
connection.setDoOutput(true);
String image = Base64.encodeToString(
IOUtils.toByteArray(new FileInputStream(filePath)),
Base64.NO_WRAP
);
OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
out.write("email=" + email + "&image=" + image);
out.close();
Perhaps this belongs in another topic, but along the same lines, does anybody know a way to pipe the file input stream in the android code directly to the URLConnection's output stream with base64 encoding? I have tried writing the string literal (the out.write() line above ^) and then creating a Base64OutputStream to write the image before piping that stream into the URLConnection's outputstream, but calling req.body.image in the node app after doing that just returns undefined. And finally, does anybody know if IOUtils.toByteArray() (from Apache Commons), when used as the input argument for an input/output stream constructor, writes the entire byte array to memory anyway on the Android side? If so, is there a way of avoiding that too?
Thanks in advance.
I used already working code for save/load game for sending a player state via sockets. And I encountered a problem that game save is correct, but server is not receiving client's player state.
Here is the base code that is tested and working:
int retval = fc.showSaveDialog(givenComponent);
if (retval == JFileChooser.APPROVE_OPTION) {
File file = fc.getSelectedFile();
try {
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file), "UTF-8"));
XStream xs = new XStream();
GameSave gs = new GameSave();
ArrayList<PlayerSerialize> listps = new ArrayList<PlayerSerialize>();
for (Player tempplayer : Players.players) {
PlayerSerialize ps = new PlayerSerialize();
ps.getPlayerData(tempplayer);
listps.add(ps);
}
gs.playersSerialize = listps;
gs.gamedate = Dateutils.gamedate;
String s = xs.toXML(gs);
bw.write(s);
bw.close();
} catch (IOException ex) {
Logger.getLogger(DialogMainField.class.getName()).log(Level.SEVERE, null, ex);
}
}
Here is the client side code that is not sending anything to server:
XStream xs = new XStream();
GameSave gs = new GameSave();
ArrayList<PlayerSerialize> listps = new ArrayList<PlayerSerialize>();
PlayerSerialize ps = new PlayerSerialize();
ps.getPlayerData(Players.players.get(1));
listps.add(ps);
gs.playersSerialize = listps;
gs.gamedate = Dateutils.gamedate;
String s = xs.toXML(gs);
out.println("clientplayertoserver");
out.println(s);
Here is the server side just in case:
if (strIn.contains("clientplayertoserver")) {
strIn = in.readLine();
XStream xs = new XStream();
GameSave gs = (GameSave) xs.fromXML(strIn);
Players.players.get(1).getPlayerSerializeData(gs.playersSerialize.get(0));
}
I need some kind of clue because I'm stuck investigating the problem. Are there any XStream limitations? Or the error is in the working with sockets? The same code is working in one place and is not working in another - I greatly thank in advance for any help with this weird situation.
Well, you are doing two different things here:
1) Saving the data to a file, which is ok.
2) Sending data via a socket. You seem to assume that all your data (the XStream serialized object) is actually in one line. This will usually not be the case. Even if you configure XStream to serialize all data without identation, you still cannot be sure you won't have linebreaks in the serialized data (your variables).
So solve your issue, you should separate your concerns here.
1st serialize / deserialize your objects to String and back (that seems to be working for you.
2nd send this data to a medium, like a file (which you already have) or to a server.
For sending string data to a server, you'll need some kind of protocol. Either you can reuse an existing protocol, like HTTP (POST request to a server), Web Service, Rest Call or whatever else your server is running.
If you want to implement your own protocol (as you have tried above), you must ensure that the server knows what to expect and how to treat it properly. Usually you should split your request in a header and a payload section or something like that.
Include in your header what you want to do (e.g save player state) and the meta information of that (e.g how many bytes payload you are sending).
After the header, send the payload.
The server must now read the header 1st (like everything until the first newline), parse the header to understand what is going on (e.g save player state, 543 bytes data) and act on it (read the data, transform it to a string, deserialize the XStream object and store it in a local database or whatever the server should do with that).
So and after all this information, please adapt your question. As you have seen you do not really have a question about XStream, but about how to send some data from client to a custom server.
For my project I have to serialize and deserialize a random tree using Java and XStream. My teacher made the Tree/RandomTree algorithms, so I don't have to worry about that. What I don't know how to do is this: I am using FileInputStream to read/write the xml file that I serialized and deserialized, but when I deserialize, I do not know the method used to read the file. After I read the file I should be able to convert it from XML and then print it out as a string. Here's what I have so far. (I imported everything correctly, just didn't add it to my code segment).
FileInputStream fin;
try
{
// Open an input stream
fin = new FileInputStream ("/Users/Pat/programs/randomtree.xml");
//I don't know what to put below this, to read FileInpuStream object fin
String dexml = (String)xstream.fromXML(fin);
System.out.println(dexml);
// Close our input stream
fin.close();
System.out.println(dexml);
// Close our input stream
fin.close();
}
// Catches any error conditions
catch (IOException e)
{
System.err.println ("Unable to read from file");
System.exit(-1);
}
Edit: I figured it out; I don't think I have to print it as a string, I just needed to make a benchmarking framework to time it and such, but thanks again!
The xstream.fromXML() method will do the reading from the input stream for you. I think the problem is that you are casting the return value from xstream.fromXML(fin) into a String when it should be cast to the type of object you originally serialized (RandomTree I assume). So the code would look like this:
RandomTree tree = (RandomTree)xstream.fromXML(fin);
EDIT: after clarification in comments, the author's goal is to first read into a String so the XML contents can be printed before deserialization. With that goal in mind, I recommend taking a look at the IOUtils library mentioned in this thread
From what I understand from http://x-stream.github.io/tutorial.html (I've never worked with XStream before), you need to define your types first. Casting to String is definitely wrong, you probably want a customized type (depending on what's inside your random XML), then you need to map the XML tags to your members:
e.g.
xstream.alias("person", Person.class);
xstream.alias("phonenumber", PhoneNumber.class);
meaning that it maps the "person" tag inside your XML to your Person class.
To derserialize, you can do:
RandomTree myRandomTree = (RandomTree)xstream.fromXML( xml );
Also, you are closing your stream twice, and you probably want to do it in a finally block :)
edit: Having read your comment above...
Your task involves two steps:
Deserialization
Serialization
In order to serialize your object, you must deserialize it first from your input file.
To output your Object as String, simply do
String xml = xstream.toXML( myRandomTree );