Here's my scenario. I have an application written in C++ but not the complete source but the "meat" of it is there. I also have a compiled exe of this application. It communicates to a server somewhere here on our network. I am trying to replicate the C++ code in java, however it uses dwords and memory references, sizeof etc, all things that don't exist in java since it manages it's own memory. It builds this large complex message and then fires it over the network. So I am basically sniffing the traffic and inspecting the packet and trying to hardcode the data it's sending over to see if I can get a response from the server this way. However I can't seem to replicate the message perfectly. Some of it, such as the license code it sends is in "clear hex", that is, hex that translates into ascii, where-as some other portions of the data are not "clear hex" such as "aa" which does not translate into ascii (or at least a common character set?? if that makes any sense I'm not sure).
Ideally I'd like to not do it like this, but it's a stepping stone to see if can get the server to respond to me. One of the functions is for the application to register itself and that's the one I am trying to replicate.
Some of my assumptions above may be wrong, so I apologize in advance. Thanks for your time.
In Java, all "character" data is encoded as Unicode (and not ASCII). So when you talk to something outside, you need to map the internal strings to the outside world. There are several ways to do it:
Use a ByteArrayOutputStream. This is basically a growing buffer of bytes to which you can append. This allows you to build the message using bytes.
Use getBytes(encoding) where encoding is the encoding the other side understands. In your case, that would be "ASCII" for the text parts.
In your case, you probably need both. Create a byte buffer and then append strings and bytes to it and then send the final result (getByteArray()) via the socket API.
Related
I'm working on a proprietary TCP protocol. This protocol sends and receive messages with a specific sequence of bytes.
I should be complaiant to this protocol, and i cant change it.
So my input / output results are something like that :
\x01\x08\x00\x01\x00\x00\x01\xFF
\x01 - Message type
\x01 - Message type
\x00\x01 - Length
\x00\x00\x01 - Transaction
\xFF - Body
The sequence of field is important. And i want only the values of the fields in my serialization, and nothing about the structure of the class.
I'm working on a Java controller that use this protocol and I've thought to define the message structures in specific classes and serialize/deserialize them, but I was naive.
First of all I tried ObjectOutputStream, but it output the entire structure of the object, when I need only the values in a specific order.
Someone already faced this problem:
Java - Object to Fixed Byte Array
and solved it with a dedicated Marshaller.
But I was searching for a more flexible solution.
For text serialization and deserialization I've found:
http://jeyben.github.io/fixedformat4j/
that with annotation defines the schema of the line. But it outputs a String, not a byte[]. So 1 is output like "1" that is represented differently based on encoding, and often with more bytes.
What I was searching for is something that given the order of my class properties will convert each property in a bunch of bytes (based on the internal representation) and append them to a byte[].
Do you know some library used for that purpose?
Or a simple way to do that, without coding a serialization algorithm for each of my entities?
Serialization just isn't easy; it sounds from your question like you feel you can just invoke something and out rolls compact, simple, versionable, universal data you can then put on the wire. What you need to fix is to scratch the word 'just' from that sentence. You're going to have to invest some time and care.
As you figured out already, java's baked in serialization has a ton of downsides. Don't use that.
There are various serializers. The popular ones are things like GSON or Jackson, which lets you serialize java objects into JSON. This isn't particularly efficient, and is string based. This sounds like crucial downsides but they really aren't, see below.
You can also spend a little more time specifying the exact format and use protobuf which lets you write a quite lean and simple data protocol (and protobuf is available for many languages, if eventually you want to write an participant in this protocol in non-java later).
So, those are the good options: Go to JSON via Jackson or GSON, or, use protobuf.
But JSON is a string.
You can turn a string to bytes trivially using str.getBytes(StandardCharsets.UTF_8). This cannot fail due to charset encoding differences (as long as you also 'decode' in the same fashion: Turn the bytes into a string with new String(theBytes, StandardCharsets.UTF_8). UTF-8 is guaranteed to be available on all JVMs; if it is not there, your JVM is as broken as a JVM that is missing the String class - not something to worry about.
But JSON is inefficient.
Zip it up, of course. You can trivially wrap an InputStream and an OutputStream so that gzip compression is applied which is simple, available on just about every platform, and fast (it's not the most efficient cutting edge compression algorithm, but usually squeezing the last few bytes out is not worth it) - and zipped-up JSON can often be more efficient that carefully handrolled protobuf, even.
The one downside is that it's 'slow', but on modern hardware, note that the overhead of encrypting and decrypting this data (which you should obviously be doing!!) is usually multiple orders of magnitude more involved. A modern CPU is simply very, very fast - creating JSON and zipping it up is going to take 1% of CPU or less even if you are shipping the collected works of shakespeare every second.
If an arduino running on batteries needs to process this data, go with uncompressed, unencrypted protobuf-based data. If you are facebook and writing the whatsapp protocol, the IAAS creds saved by not having to unzip and decode JSON is tiny and pales in comparison to the creds you spend just running the servers, but at that scale its worth the development effort.
In just about every other case, just toss gzipped JSON on the line.
Theory:
Let's say I have an application A, written in Java, that uses a TCP stream for client/server communication (it's on the client end in the relationship). Now, purely as an experiment, I am trying to create an application B, written in VB.NET, that would serve as a proxy for application A's network stream, allowing app B to read and write to the stream.
Is it, at all, possible to access such a network stream from another application, also taking the language boundary into account?
Your question is pretty vague, but if you're asking about the possibility of making a proxy server, then yes, it's possible. The language doesn't matter, but the interface does (the way that the content in the stream is encoded). For instance, Java typically serializes things into a stream using big endian (most significant byte of each byte sequence sent first), whereas .NET uses little endian (least significant byte of each byte sequence sent first). Again though, as long as you're aware of how the data is actually encoded into those streams, you can write a decent proxy server. If all that your proxy server will be doing is passing along data without caring what the data is, then you can just read a byte from one stream and write it to the other. But if you're actually reading values (integers, strings, pictures, etc.), then you will be dealing with the endianness issues, because Java and VB.NET's default stream readers will read and write integers differently, etc.
There will be some complications if you want to actually edit the data instead of simply passing it along. You'll have to deal with the client's and server's reactions to strange network behavior. For instance, if Client A is a video game, and Proxy B injects a message to the server to "join the game", then you'll have to deal with the fact that the server is going to send "ok, you've joined the game". When the client receives that message, it will most likely ignore it, because it had no knowledge that the proxy tried to join the game on its behalf, and will just assume the server made a mistake.
I have an JAX-RS web service that calls a db2 z/os database and returns about 240mb of data in a resultset. I am then creating an OutputStream to send this data to the client by looping through the resultset and adding a few XML tags for my output.
I am confused about what to use PrintWriter, BufferedWriter or OutputStreamWriter. I am looking for the fastest way to deliver the data. I also don't want the JVM to hold onto this data any longer than it needs to, so I don't use up it's memory.
Any help is appreciated.
You should use
BufferedWriter
Call .flush() frequently
Enable gzip for best compression
Start thinking about a different way of doing this. Can your data be paginated? Do you need all the data in one request.
If you are sending a large binary data, you probably don't want to use xml. When xml is used, binary data is usually represented using base64 which becomes larger than the original binary and uses quite a lot of CPU for the conversion into base64.
If I were you, I'd send the binary separate from the xml. If you are using WebService, MTOM attachment could help. Otherwise you could send the reference to the binary data in the xml, and let the app. download the binary data separately.
As for the fastest way to send binary, if you are using weblogic, just writing on the response's outputstram would be ok. That output stream is most probably buffered and whatever you do probably won't change the performance anyways.
Turning on gzip could also help depending on what you are sending (e.g. if you are sending jpeg (stuff that is already compressed) or something, it won't help a lot but if you are sending raw text then it can help a lot, etc.).
One solution (which might not work for you) is to spawn a job / thread that creates a file and then notifies the user when the file is ready to download, in this way you're not tied to the bandwidth of the client connection (and you can even compress the file properly, before the client downloads it)
Some Business Intelligence and data crunching applications do this, specially if the process takes some time to generate the data.
The output max speed will me limited by network bandwith and i am shure any Java OutputStream will be much more faster than you will notice the difference.
The choice depends on the data to send: is that text (lines) PrintWriter is easy, is that a byte array take OutputStream.
To hold not too much data in the buffers you should call flush() any x kb maybe.
You should never use PrintWriter to output data over a network. First of all, it creates platform-dependent line breaks. Second, it silently catches all I/O exceptions, which makes it hard for you to deal with those exceptions.
And if you're sending 240 MB as XML, then you're definitely doing something wrong. Before you start worrying about which stream class to use, try to reduce the amount of data.
EDIT:
The advice about PrintWriter (and PrintStream) came from a book by Elliotte Rusty Harold. I can't remember which one, but it was a few years ago. I think that ServletResponse.getWriter() was added to the API after that book was written - so it looks like Sun didn't follow Rusty's advice. I still think it was good advice - for the reasons stated above, and because it can tempt implementation authors to violate the API contract
in order to get predictable behavior.
I'm working on a network in which my python script will communicate with my java application. The python script is passing a DataPacket (just a packet that holds some strings and a little other data) to the java server for processing. I know how to pack the information into a byte array, but how do I unpack it to be used as strings? What I've got so far is I have to parse the arrays of data in the packet and send it in bits and pieces. Is this the only way to do this? Can I use ByteInputStream and if so how?
thanks
~Aedon
I'm not sure that what you're doing is quite right, in that you're fragmenting your strings into separate packets. This could cause problems with multibyte strings.
However, you may wish to check out ByteArrayOutputStream. You can write into this, then convert to a String using toString(enc), where enc is the encoding you've used in your Python to convert your strings into bytes in the first place.
Looking at your comment below, it appears you need some means to serialise in Python and deserialise in Java. Leaving aside solutions like XML serialisation, have you looked at possible solutions like Google Protocol Buffers ?
I'm working on an application with a javascript front end that can receive a bunch of octets from the server (for the time being I'm using php's chr() to simulate some data).
Trying to pass the data from javascript into an applet to be manipulated is proving difficult. Since the data can have nulls mid-string, it looks like it gets terminated at the first null going in. It also looks like once the binary data touches a javascript variable the encoding messes with some of the bytes (or maybe that's just a problem with how I'm displaying it)
Either way, what options do I have for taking a block of binary data, sent from a server and putting it into a Java applet to be manipulated. Is a conversion to base64 (or some other encoding) my only option if I want to maintain data integrity?
All this is new to me, so hopefully I got things across clearly.
Ah the bane of liveconnect.
Yes, you either need or use urlencode or base64 to get your data through.
Even when passing stuff from JavaScript to Flash (or back) you need to this, because the interface in between uses null terminated strings (which is just stupid, I know).
I think that architecture when your java script gets binary data and passes it to java applet is not optimal. Did you probably think to modify applet to make it to go directly to server and get the binary data? Without any java script?