I'm working on a network in which my python script will communicate with my java application. The python script is passing a DataPacket (just a packet that holds some strings and a little other data) to the java server for processing. I know how to pack the information into a byte array, but how do I unpack it to be used as strings? What I've got so far is I have to parse the arrays of data in the packet and send it in bits and pieces. Is this the only way to do this? Can I use ByteInputStream and if so how?
thanks
~Aedon
I'm not sure that what you're doing is quite right, in that you're fragmenting your strings into separate packets. This could cause problems with multibyte strings.
However, you may wish to check out ByteArrayOutputStream. You can write into this, then convert to a String using toString(enc), where enc is the encoding you've used in your Python to convert your strings into bytes in the first place.
Looking at your comment below, it appears you need some means to serialise in Python and deserialise in Java. Leaving aside solutions like XML serialisation, have you looked at possible solutions like Google Protocol Buffers ?
Related
I'm working on a proprietary TCP protocol. This protocol sends and receive messages with a specific sequence of bytes.
I should be complaiant to this protocol, and i cant change it.
So my input / output results are something like that :
\x01\x08\x00\x01\x00\x00\x01\xFF
\x01 - Message type
\x01 - Message type
\x00\x01 - Length
\x00\x00\x01 - Transaction
\xFF - Body
The sequence of field is important. And i want only the values of the fields in my serialization, and nothing about the structure of the class.
I'm working on a Java controller that use this protocol and I've thought to define the message structures in specific classes and serialize/deserialize them, but I was naive.
First of all I tried ObjectOutputStream, but it output the entire structure of the object, when I need only the values in a specific order.
Someone already faced this problem:
Java - Object to Fixed Byte Array
and solved it with a dedicated Marshaller.
But I was searching for a more flexible solution.
For text serialization and deserialization I've found:
http://jeyben.github.io/fixedformat4j/
that with annotation defines the schema of the line. But it outputs a String, not a byte[]. So 1 is output like "1" that is represented differently based on encoding, and often with more bytes.
What I was searching for is something that given the order of my class properties will convert each property in a bunch of bytes (based on the internal representation) and append them to a byte[].
Do you know some library used for that purpose?
Or a simple way to do that, without coding a serialization algorithm for each of my entities?
Serialization just isn't easy; it sounds from your question like you feel you can just invoke something and out rolls compact, simple, versionable, universal data you can then put on the wire. What you need to fix is to scratch the word 'just' from that sentence. You're going to have to invest some time and care.
As you figured out already, java's baked in serialization has a ton of downsides. Don't use that.
There are various serializers. The popular ones are things like GSON or Jackson, which lets you serialize java objects into JSON. This isn't particularly efficient, and is string based. This sounds like crucial downsides but they really aren't, see below.
You can also spend a little more time specifying the exact format and use protobuf which lets you write a quite lean and simple data protocol (and protobuf is available for many languages, if eventually you want to write an participant in this protocol in non-java later).
So, those are the good options: Go to JSON via Jackson or GSON, or, use protobuf.
But JSON is a string.
You can turn a string to bytes trivially using str.getBytes(StandardCharsets.UTF_8). This cannot fail due to charset encoding differences (as long as you also 'decode' in the same fashion: Turn the bytes into a string with new String(theBytes, StandardCharsets.UTF_8). UTF-8 is guaranteed to be available on all JVMs; if it is not there, your JVM is as broken as a JVM that is missing the String class - not something to worry about.
But JSON is inefficient.
Zip it up, of course. You can trivially wrap an InputStream and an OutputStream so that gzip compression is applied which is simple, available on just about every platform, and fast (it's not the most efficient cutting edge compression algorithm, but usually squeezing the last few bytes out is not worth it) - and zipped-up JSON can often be more efficient that carefully handrolled protobuf, even.
The one downside is that it's 'slow', but on modern hardware, note that the overhead of encrypting and decrypting this data (which you should obviously be doing!!) is usually multiple orders of magnitude more involved. A modern CPU is simply very, very fast - creating JSON and zipping it up is going to take 1% of CPU or less even if you are shipping the collected works of shakespeare every second.
If an arduino running on batteries needs to process this data, go with uncompressed, unencrypted protobuf-based data. If you are facebook and writing the whatsapp protocol, the IAAS creds saved by not having to unzip and decode JSON is tiny and pales in comparison to the creds you spend just running the servers, but at that scale its worth the development effort.
In just about every other case, just toss gzipped JSON on the line.
For a while, I've been working on a Java client that sends level information to save online. I have managed using printwriter, but it is really inefficient, and a simple 300KB level turns out to be 3MB after the transfer, and is rather slow.
I know people have used "file_get_contents("php://input")", such as in receive output from java DataOutputStream in a php page, but I am not sure how to receive specific data from:
//phpsend is the DataOutputStream using POST (java)
phpsend.writeUTF(username);
phpsend.writeUTF(verificationId);
phpsend.writeInt(levelsize);
phpsend.write(level); //level has been converted to a byte array
how would I read each separate write? I know Java had DataInputStream, which had all the corresponding read functions, but how would I do that in PHP? I've heard of "Sockets" and "SOAPClient", but I could not find any information that I could use
So I have found that
file_get_contents("php://input")
seems to do the what I want. Simply assign its result to a variable
$data = file_get_contents("php://input");
and the $data get all Java sent.
I want to send the ByteBuffer from FlatBuffers over network to an Android Application.
I tried using echo $builder->sizedByteArray, but then I'm wondering how to deserialize this String.
Note that this bytearray has to be sent in a way suited to binary transmission, not text.
As for deserialization with Java, have you looked here: https://google.github.io/flatbuffers/flatbuffers_guide_tutorial.html (make sure you select Java as your language) ? You can use ByteBuffer.wrap with a byte[] you received over the network.
Ok, just a simple question, I would like to send a object via java and obj-c. Is this possible to do so? Or I need to change the object to string or something first, and convert it back in to the receive side? Thanks.
It is possible to send a serialised Java object to objective C over a socket, but recovering it is difficult since you'd need to write a library to parse the binary data stream. It's possible someone has already written such a library.
It's easier to send objects encoded in JSON or XML, or with Google protocol buffers.
I would like to send a object via java and obj-c
This is the classic problem of communication between different systems/languages.
The solution has always been XML (usually SOAP which is not my taste) and nowadays the options #Joni mentions in his answer
Here's my scenario. I have an application written in C++ but not the complete source but the "meat" of it is there. I also have a compiled exe of this application. It communicates to a server somewhere here on our network. I am trying to replicate the C++ code in java, however it uses dwords and memory references, sizeof etc, all things that don't exist in java since it manages it's own memory. It builds this large complex message and then fires it over the network. So I am basically sniffing the traffic and inspecting the packet and trying to hardcode the data it's sending over to see if I can get a response from the server this way. However I can't seem to replicate the message perfectly. Some of it, such as the license code it sends is in "clear hex", that is, hex that translates into ascii, where-as some other portions of the data are not "clear hex" such as "aa" which does not translate into ascii (or at least a common character set?? if that makes any sense I'm not sure).
Ideally I'd like to not do it like this, but it's a stepping stone to see if can get the server to respond to me. One of the functions is for the application to register itself and that's the one I am trying to replicate.
Some of my assumptions above may be wrong, so I apologize in advance. Thanks for your time.
In Java, all "character" data is encoded as Unicode (and not ASCII). So when you talk to something outside, you need to map the internal strings to the outside world. There are several ways to do it:
Use a ByteArrayOutputStream. This is basically a growing buffer of bytes to which you can append. This allows you to build the message using bytes.
Use getBytes(encoding) where encoding is the encoding the other side understands. In your case, that would be "ASCII" for the text parts.
In your case, you probably need both. Create a byte buffer and then append strings and bytes to it and then send the final result (getByteArray()) via the socket API.