We Keep Coding

Sending a complex object from Java client to C server via Socket

I want to send some complex objects from a Java client to C server via a TCP Socket.
How can I do that ?

Fundamentally the question is, "How to serialize/deserialize objects in a
language agnostic manner?" Specifically Java and C in your case. Since you'll
be sending this data over a network, it is also important to take care of network order/endianness issues.
I assume you have access to both the the client and the server. This means you
get to choose how to serialize the data. (If not, the answer is simple. Write
to the specs of what the other is expecting)
Personally, I would use Protocol Buffers.
There are Java bindings
and C bindings.
If you don't like Protocol Buffers, there are other options like:
JSON (already mentioned)
YAML
Apache Thrift
XDR
roll your own
...

Write the fields of the Java objects to a string (perhaps JSON), send them via TCP, and have the C program read the string and use it to initialize new C variables on the other end.

This question is pretty old, but just in case some one is still looking for a good solution, you can try out the protocol buffers implementation, as mentioned in the previous answer by #Adam Liss: (developers.google.com/protocol-buffers/)
In short, you define any complex message type as in your protocol implementation, and the tool generates C++/Java/Python code which can serialize and deserialize it.
For the same purpose using C code, a research project at the Technische Universität München (TUM) Germany have created a code generator in standard C, that can be used with embedded-C projects. This is fully compatible(with limitations due to C structs) with Google's protobuf implementation. This works better than the C Bindings because it does not need any library to be linked with.
I had issues in getting the C Bindings to work on the embedded systems I was working with, because it needs to be linked with the support library.
This saved my (painful) day with my embedded project - passing complex network data(request-responses) between an embedded system and Android app(Java)/Desktop app(C++/Qt).

Sending object via socket, but different language client, how to make the object serialize?

I have a Java Server that sending the java serializable object to my client, and receive java serializable object for execution. If my client is also java written, which is nice, that allow me to do communication within any problems.
But now, I would like to extend my programme to not only java client, the client may be written in C, objective C, python or php. So, I would like to do something to "convent" to client request to a java object, and send back to Server. The convent process, I can use the JSON to receive, and construct a Java object to the Server, but I also need a layer that convert back the Java object to JSON to the client.
My Question is except make a JSON-Java Translation layer, is there any other ways to do so? Also, we can afford to change some code in server side, but we must use Java as our primary language for that. Any suggestions? Thanks.

I use Netty API for designing my protocol and it is quite quick to do so if you can understand a NIO-like Byte and Buffer API.
It is design to work with a concept of Encoder and Decoder that could fit your need, there are a lot of default implementation of Encoder and Decoder for zipping, using ssl...
The problem you have seems to looks like this one:
JBoss Netty with JSON
I don't know JSON very well but most of the time is could also be quick and easy to design your own protocol.
Do you need a generic Serialization process for any kind of Object or do you simply need to serialize some String and primitive types (Integer, Short, Float..etc)?
In the case of simple objects it is easy and a lot faster to do the wrapper by yourself.
If objects are quite simple, and I would guess this is the case, your need it to design your own "protocol" specification meaning how to turn each Object into a sequence of primitive types, String and arrays. Than it should be quite easy to write both the Encoder and the Decoder in each language.
Good luck

There are other libraries designed for this, like protocol buffers and thrift.
http://thrift.apache.org/
http://code.google.com/p/protobuf/

Passing a Entity over a network?

I have been studying Java networking for a while.
I am using ObjectInputStream and ObjectOutputStream for the I/O between sockets.
Is this possible to transfer a Entity or Model from server to client and vise versa?
How can I implement this? Am I suppose to implement the Entity or Model to Serializable?
Your response is highly appreciated.

I am not sure what sort of special thing you mean to denote by capital-E Entity and capital-M Model; these terms don't have any fixed, privileged meaning in Java (although they might with respect to a certain API or framework.) In general, if by these you just mean some specific Java objects, then yes, you can send any sort of objects this way, and yes, they would be required to implement Serializable. The only limitations would be if these objects contained members whose values wouldn't make sense on the other end of the pipe -- like file paths, etc.
Note that if you send one object, you'll end up sending every other object it holds a non-transient reference to, as well.

First of all... why sending an object through I/O stream? What's wrong with XML?
However, you can always send/receive an object through I/O stream as long as the sender can serialize the object and the receiver can deserialize the object. Hope it helps

You definitely need to look at one of these two libraries
Google gson: http://code.google.com/p/google-gson/
Converts Java object to JSON and back. advantage is that the object can be consumed or generated by Javascript. I have also used this for Java-Java RPC, but it gives you flexibility if you want to target browsers later
Google protocol buffers: http://code.google.com/apis/protocolbuffers/
This is what google uses for RPC. Implementations for Java, C, Python. If you need performance and the smallest size, this is the one to go with (The trade off is you can't look at the data easily to debug problems, like you can with gson, which generates plaint text JSON).

Server-client communication packet content strategy

I'm trying to think in the best way on communication for the game I'm writing. The scenario is simple: tcp sockets and request for authentication, map updates, chat updates, etc. What I was thinking to use was set of classes, like User, Map, Creature, etc and have a Message class, which will have enum with message types and Object to store previously mentioned classes. After I will convert this with GSON to json and on other side I will decode it corresponding to the message type indicated by the element of enum. The problem is that I will pass sometimes too much unnecessary data and that's doesn't let me quiet plus the integration of new types of messages will not be very easy neither for me, nor for someone else who might use it. In the previous version I have used my own XML protocol which also doesn't let me very happy.
So what I'm asking is advice for me the better way for communication or maybe some improvement of my idea.
Thanks in advance,
Serhiy.

XML and JSOn are intended to make application integration simple, but still be human readable.
If you want a protocol tuned to your needs, I suggest you start by determining what information you want to send and how it would look. Document this before you even start implementing it. That way the data sent will suit your needs. (This is more work BTW which is why it is not done more often)

Deserialize in a different language

The log4j network adapter sends events as a serialised java object. I would like to be able to capture this object and deserialise it in a different language (python). Is this possible?
NOTE The network capturing is easy; its just a TCP socket and reading in a stream. The difficulty is the deserialising part

Generally, no.
The stream format for Java serialization is defined in this document, but you need access to the original class definitions (and a Java runtime to load them into) to turn the stream data back into something approaching the original objects. For example, classes may define writeObject() and readObject() methods to customise their own serialized form.
(edit: lubos hasko suggests having a little java program to deserialize the objects in front of Python, but the problem is that for this to work, your "little java program" needs to load the same versions of all the same classes that it might deserialize. Which is tricky if you're receiving log messages from one app, and really tricky if you're multiplexing more than one log stream. Either way, it's not going to be a little program any more. edit2: I could be wrong here, I don't know what gets serialized. If it's just log4j classes you should be fine. On the other hand, it's possible to log arbitrary exceptions, and if they get put in the stream as well my point stands.)
It would be much easier to customise the log4j network adapter and replace the raw serialization with some more easily-deserialized form (for example you could use XStream to turn the object into an XML representation)

Theoretically, it's possible. The Java Serialization, like pretty much everything in Javaland, is standardized. So, you could implement a deserializer according to that standard in Python. However, the Java Serialization format is not designed for cross-language use, the serialization format is closely tied to the way objects are represented inside the JVM. While implementing a JVM in Python is surely a fun exercise, it's probably not what you're looking for (-:
There are other (data) serialization formats that are specifically designed to be language agnostic. They usually work by stripping the data formats down to the bare minimum (number, string, sequence, dictionary and that's it) and thus requiring a bit of work on both ends to represent a rich object as a graph of dumb data structures (and vice versa).
Two examples are JSON (JavaScript Object Notation) and YAML (YAML Ain't Markup Language).
ASN.1 (Abstract Syntax Notation One) is another data serialization format. Instead of dumbing the format down to a point where it can be easily understood, ASN.1 is self-describing, meaning all the information needed to decode a stream is encoded within the stream itself.
And, of course, XML (eXtensible Markup Language), will work too, provided that it is not just used to provide textual representation of a "memory dump" of a Java object, but an actual abstract, language-agnostic encoding.
So, to make a long story short: your best bet is to either try to coerce log4j into logging in one of the above-mentioned formats, replace log4j with something that does that or try to somehow intercept the objects before they are sent over the wire and convert them before leaving Javaland.
Libraries that implement JSON, YAML, ASN.1 and XML are available for both Java and Python (and pretty much every programming language known to man).

I would recommend moving to a third-party format (by creating your own log4j adapters etc) that both languages understand and can easily marshal / unmarshal, e.g. XML.

In theory it's possible. Now how difficult in practice it might be depends on whether Java serialization format is documented or not. I guess, it's not. edit: oops, I was wrong, thanks Charles.
Anyway, this is what I suggest you to do
capture from log4j & deserialize Java object in your own little Java program.
now when you have the object again, serialize it using your own custom formatter.
Tip: Maybe you don't even have to write your own custom formatter. for example, JSON (scroll down for libs) has libraries for Python and Java, so you could in theory use Java library to serialize your objects and Python equivalent library to deserialize it
send output stream to your python application and deserialize it
Charles wrote:
the problem is that for this
to work, your "little java program"
needs to load the same versions of all
the same classes that it might
deserialize. Which is tricky if you're
receiving log messages from one app,
and really tricky if you're
multiplexing more than one log stream.
Either way, it's not going to be a
little program any more.
Can't you just simply reference Java log4j libraries in your own java process? I'm just giving general advice here that is applicable to any pair of languages (name of the question is pretty language agnostic so I just provided one of the generic solutions). Anyway, I'm not familiar with log4j and don't know whether you can "inject" your own serializer into it. If you can, then of course your suggestion is much better and cleaner.

Well I am not Python expert so I can't comment on how to solve your problem but if you have program in .NET you may use IKVM.NET to deserialize Java objects easily. I have experimented this by creating .NET Client for Log4J log messages written to Socket appender and it worked really well.
I am sorry, if this answer does not make sense here.

If you can have a JVM on the receiving side and the class definitions for the serialized data, and you only want to use Python and no other language, then you may use Jython:
you would deserialize what you received using the correct Java methods
and then you process what you get with you Python code