We Keep Coding

Sending objects between Python/Java server/client

I have a Python client and a Java server. I would like the client to send an object to the server. How to implement this?
How to also implement the other way around (Java client - Python server)?
Here's an attempt I made with Python server and Java client:
PYTHON SERVER SIDE
import pickle
import socket
from simple_message import SimpleMessage
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('', 9999))
s.listen(1)
while True:
print("Waiting for a message...")
conn, addr = s.accept()
data = conn.recv(4096)
incoming_message = pickle.loads(data)
conn.close() # Close connection, not needed anymore
print(SimpleMessage.get_payload(incoming_message))
The object it refers to (of class SimpleMessage) is defined as follows:
#!/usr/bin/env python
class SimpleMessage:
dest_address = str()
message_type = int()
payload = str()
def __init__(self, dest_address, message_type, payload):
self.dest_address = dest_address
self.message_type = message_type
self.payload = payload
def get_payload(self):
return self.payload
JAVA CLIENT SIDE
import java.io.IOException;
import java.io.ObjectOutputStream;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.Socket;
public class JavaClient {
public static void main(String[] args) throws IOException {
Socket sendingSocket = new Socket();
sendingSocket.connect(new InetSocketAddress("127.0.0.1", 9999));
OutputStream outputStream = sendingSocket.getOutputStream();
ObjectOutputStream objectOutputStream = new ObjectOutputStream(outputStream);
SimpleMessage message = new SimpleMessage("127.0.0.1", 1, "Test message!");
objectOutputStream.writeObject(message); // Write Message on socket
sendingSocket.close();
}
}
And the class SimpleMessage:
import java.io.Serializable;
public class SimpleMessage implements Serializable {
private String destAddress;
private Integer messageType;
private String payload;
public SimpleMessage(String destAddress, Integer messageType, String payload) {
this.destAddress = destAddress;
this.messageType = messageType;
this.payload = payload;
}
}
OUTPUTS
Here is the output I get on the Python server side:
Waiting for a message...
Traceback (most recent call last):
File "python_server.py", line 16, in <module>
incoming_message = pickle.loads(data)
_pickle.UnpicklingError: invalid load key, '\xac'.
And here's the output I get on the Java client side:
Exception in thread "main" java.net.SocketException: Broken pipe (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
at java.base/java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1883)
at java.base/java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1792)
at java.base/java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1287)
at java.base/java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1232)
at java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1428)
at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
at java.base/java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1583)
at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:352)

This is a specific instance of choosing a serialization format.
https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats
This is a potentially overbroad topic, so I will hold back from a long and possibly redundant answer considering all possible formats.
JSON is a good and currently fashionable format which will get you started, but also potentially work well more involved use cases in the future. There are well-known libraries in Python and Java.
No matter what you choose, if there is any use outside a small prototype or assignment, defining an explicit schema lays a good foundation for future work (eg using JSON schema for JSON).
pickle, as in your original example code, is a Python-specific serialization format. So unless you have some specific use for a tool like Jython in your Java world, it's not a great choice for communicating over a network with a service that can be written in another language.
You should also consider whether low level sockets are the best choice for your use case, or a higher level network library like HTTP may be a better fit.

What you are talking about here is also known as decoupling of elements in the systems.
This gives you a lot of flexibility to change languages and implementation in the system and this is also the method that is used when different backend services talk with each other in a micro-services architecture.
The common way to do it is to select a JSON protocol that the two sides transfer between each other: For example:
{
fName: "David",
lName: "Gold"
}
And then to make HTTP calls to GET or POST data between your two elements.
By doing so, you are free to change the implementation on each side (let's say you discover that you better write your client in JavaScript and the server in R).
As long as both sides keep working with the same protocol, they are agnostic to the implementation that the other side uses.

C# TcpClient to netty not always arriving

Me and a friend are working on a project which requires us to communicate between a C#.NET application and a Java application.
We're using the TcpClient and BinaryWriter classes on the .NET side of things to send and receive things. We're using code similar to this to send things:
byte[] content = //we're getting our content here
Writer.Write(new byte[9]); //this is the BinaryWriter with the NetworkStream of the TcpClient
Writer.Flush();
On the Java side of things, we're using netty to handle our networking. To receive the content we send from the .NET side, we add a ChannelInboundHandlerAdapter to the pipeline and use the channelRead method to read the content:
public void channelRead(ChannelHandlerContext ctx, Object received)
{
ByteBuf receivedByteBuf = (ByteBuf)received;
this.bytesRead = receivedByteBuf.readableBytes();
System.out.println("Received " + this.bytesRead + " bytes.");
final byte[] buffer = new byte[this.bytesRead];
receivedByteBuf.markReaderIndex();
receivedByteBuf.readBytes(buffer);
receivedByteBuf.resetReaderIndex();
}
Now the strange thing is, that when we try sending content, it doesn't always arrive in one piece. Sometimes we only receive all but some bytes we originally sent, which arrive in a new call of channelRead. In this example, only 6-8 bytes would arrive. This is very strange, as this only happens when using .NET. We tried sending content using python and everything worked fine and it arrived in one channelRead call.
import socket
import string, random
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1", 8888))
s.send(''.join(random.choice(string.lowercase) for x in range(500)))
s.close()
Unfortunately, the nature of our project prevents us from changing our Java networking library, so we're stuck with netty.
Did we miss some setting in netty or does this have to do with the nature of the .NET TCP libraries? We would appreciate any help we can get.

Passing parameter from Java to Python and vise versa [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I have a python code that waits for input parameter. This input is the output of another java code. Is there any way I can pass the parameters between the two codes like a bridge between the two?
Thanks in advance

The best way to do these kind of tasks is to use a message broker like RabbitMQ. It provides support for Java, Python, PHP, etc. You can send messages (a json message for example, or in any other format) between processes implemented with different languages. Here you can find tutorials implemented with different languages.
RabitMQ
RabbitMQ is a message broker. It sits between producers and consumers. Producers are components which produces messages and publish those messages to a queue in the RabbitMQ. RabbitMQ takes those messages and by the rules you defined routes and delivers those messages to the consumers. Consumers are task runners which wait to receive messages and run tasks. Here is a simple producer which connects to RabbitMQ, sends a single message, and exit.
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.Channel;
public class Send {
private final static String QUEUE_NAME = "hello";
public static void main(String[] argv) throws java.io.IOException {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
String message = "Hello World!";
channel.basicPublish("", QUEUE_NAME, null, message.getBytes());
System.out.println(" [x] Sent '" + message + "'");
channel.close();
connection.close();
}
}
And here we have a simple java consumer which we keep it running to listen for messages. You are not limited to java. You can use any language which there are RabbitMQ libraries for (Python, PHP, C#, JavaScript, etc):
import com.rabbitmq.client.*;
import java.io.IOException;
public class Recv {
private final static String QUEUE_NAME = "hello";
public static void main(String[] argv) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
System.out.println(" [*] Waiting for messages. To exit press CTRL+C");
Consumer consumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body)
throws IOException {
String message = new String(body, "UTF-8");
System.out.println(" [x] Received '" + message + "'");
}
};
channel.basicConsume(QUEUE_NAME, true, consumer);
}
}
To run it:
$ javac -cp rabbitmq-client.jar Send.java Recv.java
$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar Send
$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar Recv
Note: you'll need rabbitmq-client.jar and its dependencies on the classpath.

This kind of process interconnection have two main problems: serialization and transport.
When you passing data from Java process to Python application most likely you want to see a object of specific type rather than raw-bytes or text. There is number of frameworks that was designed to handle cross-language behavior. Take a look at Apache Avro, Protobuf or Apache Thrift. Each one have it's own pros and cons.
Second problem is transport. Like #Mustafa Shujaie already said message-oriented transport like RabitMQ is a good choice. But also have a look at REST services for Java and Python.
By the way, if you take byte-oriented serialization like Protobuf then plain TCP transport could be a good choice - data overhead would be significantly lover in comparison to any application-leveled protocol like HTTP. See this articles: python and java.

If you have deployed both programs on a single device you can consider using libraries that supports handling interprocess communication (IPC) such as ZeroMQ.
It has bindings for both:
Java bindings
Python bindings
I've successfully applied communication with 0MQ on embedded system with ARM processors. It acs as a communication bus between programs written in multiple languages (mostly in C and Python in my case).
While prototyping you can use human readable data format (like JSON). There were a lot of libraries that supports (de)serializing JSONs in Java (like GSON). Python has native modules for that.

Broadcast Message in Java

I need some mechanism that allows me to transfer some data from one java program to another within a same PC. I already investigated RMI but I want 1st app to broadcast some message for 2nd without request of 2nd application. In RMI only client can initiate a communication.
Raw sockets are also not desirable (very low level).
I need something like RMI with a different scheme of starting communication: 1 server broadcasts messages for several clients without requests from clients.
Could you please suggest me some libs/technologies (for desktop app)?

I suggest you to use java messaging service and one of it's implementations such as ApacheMQ.
A good starting point is here.

I would suggest using a database with a trigger and store procedure. Here is an example of calling java methods from the database. A message queue will work, but that is an overly complex solution.
Here's an excerpt from an example of how to create a procedure and call it via a trigger:
First, you add the following Java method to the class DBTrigger
CREATE OR REPLACE PROCEDURE add_emp (
emp_no NUMBER, emp_name VARCHAR2, dept_name VARCHAR2)
AS LANGUAGE JAVA
NAME 'DBTrigger.addEmp(int, java.lang.String, java.lang.String)';
Then, you create the INSTEAD OF trigger:
CREATE OR REPLACE TRIGGER emps_trig
INSTEAD OF INSERT ON emps
FOR EACH ROW
CALL add_emp(:new.empno, :new.ename, :new.dname);

Since you tagged this with CORBA, you could use the Event Service to broadcast a notice to all interested clients.

To enable the server to send packets of information to the client/clients. A datagram, by definition, is “an independent, self-contained message sent over the network whose arrival, arrival time, and content are not guaranteed”. Essentially, we are opening a DatagramSocket in order to send DatagramPacket messages to the client. We are using the datagram classes (instead of standard sockets) because they allow us to broadcast information to multiple clients, that are all connected to a MulticastSocket.
import java.io.IOException;
import java.net.DatagramPacket;
import java.net.DatagramSocket;
import java.net.InetAddress;
import java.net.UnknownHostException;
public class MulticastSocketServer {
final static String INET_ADDR = "224.0.0.3";
final static int PORT = 8888;
public static void main(String[] args) throws UnknownHostException, InterruptedException {
// Get the address that we are going to connect to.
InetAddress addr = InetAddress.getByName(INET_ADDR);
// Open a new DatagramSocket, which will be used to send the data.
try (DatagramSocket serverSocket = new DatagramSocket()) {
for (int i = 0; i < 100; i++) {
String msg = "Sent message no " + i;
// Create a packet that will contain the data
// (in the form of bytes) and send it.
DatagramPacket msgPacket = new DatagramPacket(msg.getBytes(),
msg.getBytes().length, addr, PORT);
serverSocket.send(msgPacket);
System.out.println("Server sent packet with msg: " + msg);
Thread.sleep(500);
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
One thing that we need to take into consideration here, is that there are specific addresses that allow us to use a MulticastSocket are limited, specifically in the range of 224.0.0.0 to 239.255.255.255. Some of them are reserved, like 224.0.0.0. The address that we are using, 224.0.0.3, can be used safely.
Regarding the client, we are going to move a little bit differently. We are going to create a client class, that will accept incoming messages from the server, and then we are going to duplicate this class. The point here is that by using the same code, we can connect to the server seamlessly, while having as many clients as we like.
import java.io.IOException;
import java.net.DatagramPacket;
import java.net.InetAddress;
import java.net.MulticastSocket;
import java.net.UnknownHostException;
public class MulticastSocketClient
{
final static String INET_ADDR = "224.0.0.3";
final static int PORT = 8888;
public static void main(String[] args) throws UnknownHostException {
// Get the address that we are going to connect to.
InetAddress address = InetAddress.getByName(INET_ADDR);
// Create a buffer of bytes, which will be used to store
// the incoming bytes containing the information from the server.
// Since the message is small here, 256 bytes should be enough.
byte[] buf = new byte[256];
// Create a new Multicast socket (that will allow other sockets/programs
// to join it as well.
try (MulticastSocket clientSocket = new MulticastSocket(PORT)){
//Joint the Multicast group.
clientSocket.joinGroup(address);
while (true) {
// Receive the information and print it.
DatagramPacket msgPacket = new DatagramPacket(buf, buf.length);
clientSocket.receive(msgPacket);
String msg = new String(buf, 0, buf.length);
System.out.println("Socket 1 received msg: " + msg);
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
First, we start the clients, which will keep waiting for incoming packets of information. As soon as we start the server, it will send the information packets and the clients will receive them and print the information on the screen

ZeroMQ could be what you are searching, but I have only used it in C, C++ and Python, so I am not totally sure of how well it is usable in Java. But they have implemented the Publisher-Subscriber pattern for their sockets and at least the static library of the version 3.2.2 under ubuntu 12.04 is stable and works very well.

As a "quick fix" write the output to file and have the second app read the file.
This is not elegant at all, but if you code to interfaces you can later replace the implementation with something better.

A good solution would be an implementation of the OMG Data Distribution Standard (DDS). With DDS there is just a global data space with a dynamic discovery protocol. See for example RTI DDS, there is a free community edition.

How to simply generate SSL certificate and ServerSocket form Java code

Is there simply way to dinamic generate untrusted ssl certificate without domain and applay it to server socket - all from code, no commandline or additional files?
Purpose is secure connection between two hosts witch know only IP and port to communicate each other - certificates generated randomly at server start and used as "untrusted", no domain so no verification (if I'm not wrong). I think this can be usefull in secureing data transfer between datacenters in third party apps.
This is working code for not encrypted simply client-server test.
package study.benchmark.utils;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
import javax.net.ServerSocketFactory;
import javax.net.SocketFactory;
import org.junit.Test;
public class DynamicSSLTest {
#Test
public void sslServerSocketTest() throws Exception {
System.out.println("ssl server test");
final int port = 8750;
// server
Thread th = new Thread() {
#Override
public void run() {
try {
//ServerSocketFactory factory = SSLServerSocketFactory.getDefault();
ServerSocketFactory factory = ServerSocketFactory.getDefault();
ServerSocket server = factory.createServerSocket(port);
Socket socket = server.accept();
OutputStream out = socket.getOutputStream();
out.write("some data".getBytes());
socket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
};
th.start();
//client
//SocketFactory factory = SSLSocketFactory.getDefault();
SocketFactory factory = SocketFactory.getDefault();
Socket socket = factory.createSocket("localhost", port);
InputStream is = socket.getInputStream();
StringBuffer sb = new StringBuffer();
int data;
while ((data = is.read()) >= 0) {
System.out.print((char) data);
}
System.out.println();
socket.close();
th.join();
}
}

You can generate a self-signed certificate dynamically using a library such as BouncyCastle (essentially, for the certificate to be self-signed, you using the same issuer DN as the subject DN and you sign with the private key corresponding to the certificate's public key). Then, you'll need to put it in a KeyStore (in memory at least, not necessarily on file) and build an SSLContext from it, so as to be able to build an SSLSocketFactory.
This can be useful for testing, but this will not make your application secure. Generally speaking, encryption without authentication of the remote party isn't secure. You can exchange information as "secretly" as you want with a remote party, but if you haven't verified its identity you're not really sure your secrets are given to the intended recipient.
If your certificate is generated dynamically, you would need to find a way for the client to know it's indeed the legitimate certificate, before making any calls to that server.
The general SSH approach (where one assumes few people actually check the fingerprint they get in the first connection -- some people actually do check out of band) is a compromise whereby clients tend to accept the key (more or less blindly) the first time but will be warned if it has changed. You could implement this sort of approach for handling X.509 certificate trust too, but if you re-generate a new self-signed certificate every time you restart your server, you're back to the initial problem.
You could address this problem by having some sort of online/dynamic CA, where the servers would request and be issued a certificate dynamically based on something they could prove to that CA dynamically (to prove they're one of your servers, perhaps based on some configuration parameter known by both), and then have the client trust that CA, but that's a more complex scenario.

Why? It's not secure, and who will trust it anyway? Why can't you create a server certificate per deployment offline?

For your stated purpose of securing communications between two hosts, both of which are under your control, SSH would be a better solution. You can generate a key pair shared only with your other machine. Google "java ssh" for a host of options.