We Keep Coding

Java client doesn't send whole message to python server?

I am writing a program in android studio that communicates with a python server. I tried to send a long message (mp3 file encoded in base64 - about 10k bytes). The problem is that when I check what I received in the server, I get a lot less than 10k bytes.
Anyone knows how to fix it?
Thanks in advance.
Recording message and putting it in base64:
// Record audio from user in mp3 format
MediaRecorder recorder = new MediaRecorder();
recorder.setAudioSource(MediaRecorder.AudioSource.MIC);
recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
recorder.setOutputFile(Environment.getExternalStorageDirectory()
.getAbsolutePath() + "/messageRecord.mp3");
recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AAC);
recorder.prepare();
recorder.start();
TimeUnit.SECONDS.sleep(5);
recorder.stop();
File file = new File(Environment.getExternalStorageDirectory()
.getAbsolutePath() + "/messageRecord.mp3");
int size = (int) file.length();
byte[] bytes = new byte[size];
BufferedInputStream buf = new BufferedInputStream(new FileInputStream(file));
buf.read(bytes, 0, bytes.length);
buf.close();
String content = Base64.encodeToString(bytes, Base64.DEFAULT);
// Prepare audio message request
JSONObject sendRecordReq = new JSONObject();
sendRecordReq.put("code", Codes.SPEECH_TO_TEXT_CODE);
sendRecordReq.put("src_phone", ChatScreen.this.srcPhone);
sendRecordReq.put("dst_phone", ChatScreen.this.dstPhone);
sendRecordReq.put("content", content);
// Send message request
ChatScreen.this.client.send(sendRecordReq);
How I send it:
//In class client
public void send(JSONObject request) {
this.outgoingMessages.addConversationFlow(request); //Send request
}
//In class OutgoingMessages
public void run() {
try {
while (true) {
while(!this.conversationFlow.isEmpty()) {
JSONObject msgToSend = this.conversationFlow.remove();
String strRequest = msgToSend.toString();
this.out.write(Integer.toString(strRequest.length()).getBytes()); //Sends message size
this.out.write(strRequest.getBytes()); //Sends message
}
}
}
catch (Exception e) {
System.out.println(e);
}
}
Server:
while True:
# Receiving data size from client
message_size = int(client_socket.recv(MAX_SIZE_LEN))
# Receiving data from the client
client_message = client_socket.recv(message_size)
print client_message
# Add message to messages queue with client's socket
MESSAGES_QUEUE.append((client_socket, client_message))
EDIT:
the "message_size" value is right (14806 - the size of the message that should b e received in the next line) but it still doesn't receive it all.
EDIT2:
I figured it out, ill post the solution in the answers

Your java code is sending data in a protocol that couldn't possibly work.
If the input JSON object is, say, the string 'a', then you'd send this:
3"a"
as in, 4 bytes: 51, 34, 97, 34.
The python side has no idea when the 'this is how long the data' is part ends.
I assume what you intended to send is something along the lines of:
00, 00, 00, 03, 34, 97, 34.
In other words: 4 bytes containing a network-endian sent integer value with the length, and then that many bytes.
Separately, don't call .getBytes(); when converting strings to bytes, you should always explicitly specify encoding. JSON is more or less by definition UTF-8, so, call .getBytes("UTF-8") instead.
NB: To replace your 'send the length' code, see Convert integer into byte array (Java)

So the problem wasn't with the length of the file or anything else. The problem was that the recv function in the python would get some of the bytes but the code continued for some reason, so it didn't get the whole message.
The solution is to add a function that doesn't continue till the whole length that is specified is received. I found the solution in the next post:
Python Socket Receive Large Amount of Data
I'll add the code as well:
Replace the recv function with recvall(socket, size). This is not my code, it was posted by Adam Rosenfield and edited by Hedde van der Heide.
def recvall(sock, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = sock.recv(n - len(data))
if not packet:
return None
data += packet
return data

Lag between DataOutputStream and DataInputStream using an android client and python server

I'm having an issue with my android client where I click a button to send data to a python server through a socket, which processes it, then sends it back upper cased. My android client's datainputstream always seems to be "one step behind": the first time I click the button the python server clearly receives it(as shown by the print function) but the TextView in my android doesn't show anything. When I click again with different data the python server receives the new data but my android's dis.readUTF() seems to read the old data.
Here is my android client onClick function:
public void onClick(View arg0) {
Thread t = new Thread(){
#Override
public void run() {
try {
Socket s = new Socket("192.168.4.1", 9999);
//OutputStreamWriter osw= new OutputStreamWriter(s.getOutputStream(), "UTF-8");
DataOutputStream dos = new DataOutputStream(s.getOutputStream());
//byte[] bufO=message.getText().toString().getBytes("UTF-8");
dos.writeUTF(message.getText().toString());
DataInputStream dis = new DataInputStream(s.getInputStream());
serverResponse= dis.readUTF();
s.close();
}
catch (IOException e) {
e.printStackTrace();
}
}
};
if(message.getText().toString().equals("")){
Toast.makeText(this, "Empty command", Toast.LENGTH_SHORT).show();
}
else{
message_received.setText("");
t.start();
Toast.makeText(this, "Command sent", Toast.LENGTH_SHORT).show();
message_received.append(serverResponse + "\n");
Log.d(TAG, serverResponse);
}
}
Here is my Python server:
import socketserver
import socket
class MyHandler(socketserver.BaseRequestHandler):
def handle(self):
self.sentence= self.request.recv(1024).strip()
print(self.sentence)
self.request.sendall(self.sentence.upper())
def main():
print ("Opening socket")
host='192.168.4.1'
port=9999
server1=socketserver.TCPServer((host,port),MyHandler)
print ("Running server")
server1.serve_forever()
main()
I've tried different things including adding a thread.sleep between the writeutf() and readutf() but nothing seems to fix this.

This is because readUTF (and therefore writeUTF) may not do what you think it does; from DataInput documentation, referenced by the DataInputStream:
First, two bytes are read and used to construct an unsigned 16-bit integer in exactly the manner of the readUnsignedShort method . This integer value is called the UTF length and specifies the number of additional bytes to be read.
In general it is a good idea to include the length of the string into the string encoding. So maybe you should alter the python server code to receive and send the length encoding as well.

Send Byte array then String then Byte over socket Java

I need to send a text message to server over Java socket and then to send a byte array and then a string etc...
What I've developed until now is working but the client manages to read only the first string that has been sent.
From the server side: I send byte array using BufferedOutputStream, and PrintWriter to send string.
The problem is that the client and the server are not synchronized, I mean the server send string then byte array then string without waiting for the client to consume each needed byte.
I mean the scenario is NOT like this:
Server Client
Send String read String
Send byte read byte
But it is like this:
Server Client
Send String
Send byte
Send String
Send byte
read String
read byte
read String
read byte
Something that could be useful is that I know exactly the size of each string and each byte array to be read.
Here is the methods used to send string and byte array respectively:
// Send String to Client
// --------------------------------------------------------------------
public void sendStringToClient (
String response,
PrintWriter output) {
try {
output.print(response);
output.flush();
} catch(Exception e) {
e.printStackTrace();
}
System.out.println("send Seeder String : " + response);
}
// Send Byte to Client
// --------------------------------------------------------------------
public void sendByteToClient (
byte[] response,
BufferedOutputStream output) {
try {
output.write(response, 0, response.length);
//System.out.println("send : " + response);
} catch (IOException e) {
e.printStackTrace();
}
}
Here is the methods used to read string and byte array respectively:
public byte[] readInByte(int size) {
byte[] command = new byte[size];
try {
this.inByte.read(command);
} catch (IOException e) {
e.printStackTrace();
}
return command;
}
public String readInString(int size) {
char[] c = new char[size];
try{
this.inString.read(c, 0, size);
} catch (IOException e) {
e.printStackTrace();
}
return String.valueOf(c);
}

Something that could be useful is that i know exactly the size of each string to be read and each byte array to be read.
Exactly. That's very common. Basically you length-prefix each message - and you might want to provide more header information than that (is it a string or a byte array message, for example).
You could represent the message length (always in bytes) either as a fixed number of bytes (e.g. 4, assuming you never need more than 4GB messages) or use a 7-bit encoded integer (where you send 7 bits of the length in each byte, and the top bit just indicates whether this is the last byte of the length or not).
Once you've got message lengths, you're basically set - you've effectively divided the stream of data into self-describing blocks. Job done.
(As an aside, I'd avoid using PrintWriter due to its exception-swallowing nature. You won't actually need the writer once you're doing this though, as you'll probably want to convert each String into a byte array anyway, to count its length in bytes before you send it. Remember to specify the encoding!)

I'd be really tempted to convert the data to JSON format and pipe it over http. You get a ton of benefits including ready made http servers and clients for just about every platform on earth along with the JSON interop not to mention all the built in error handling and recovery processing.
The drawbacks would be the additional overhead of http and the JSON encoding. You didn't mention if this was a UDP or TCP socket so that could be an additional drawback if you were trying to go connectionless.

Java sending byte[] through sockets...wrong length read

For my homework assignment, I have a network of Nodes that are passing messages to each other. Each Node is connected to a set amount of other Nodes (I'm using 4 for testing). Each Link has a weight, and all the Nodes have computed the shortest path for how they want their messages sent. Every Packet that is sent is composed of the message protocol (a hard-coded int), an integer that tells how many messages have passed through the sending Node, and the routing path for the Packet.
Every Node has a Thread for each of its Links. There is an active Socket in each Link. The Packets are sent by adding a 4-byte int to the beginning of the message telling the message's length.
Everything works fine until I stress the network. For my test, there are 10 Nodes, and I get 5 of them to send 10000 packets in a simple while() loop with no Thread.sleep(). Without exception, there is always an error at some point during execution at the if(a!=len) statement.
Please let me know if I can clarify anything. Thanks in advance! Here is the code (from the Link Thread; send() and forward() are called from the Node itself):
protected void listen(){
byte[] b;
int len;
try{
DataInputStream in = new DataInputStream(sock.getInputStream());
while(true){
len = in.readInt();
b = new byte[len];
int a = in.read(b,0,len);
if(a!=len){
System.out.println("ERROR: " + a + "!=" + len);
throw new SocketException(); //may have to fix...this will happen when message is corrupt/incomplete
}
Message m = new Message(b);
int p = m.getProtocol();
switch (p){
case CDNP.PACKET:
owner.incrementTracker();
System.out.print("\n# INCOMMING TRACKER: " + m.getTracker() + "\n>>> ");
owner.forward(m);
}
}
}catch (IOException e){
e.printStackTrace();
}
}
public void send(int tracker){
String[] message = { Conv.is(CDNP.PACKET), Conv.is(tracker), owner.getMST().toString() };
Message m = new Message(message);
forward(m);
}
public synchronized void forward(Message m){
try{
OutputStream out = sock.getOutputStream();
//convert length to byte array of length 4
ByteBuffer bb = ByteBuffer.allocate(4+m.getLength());
bb.putInt(m.getLength());
bb.put(m.getBytes());
out.write(bb.array());
out.flush();
}catch (UnknownHostException e){
System.out.println("ERROR: Could not send to Router at " + sock.getRemoteSocketAddress().toString());
return;
}catch (IOException e1){
}
}

int a = in.read(b,0,len);
if(a!=len){
That won't work. The InputStream may not read all the bytes you want, it may read only what is available right now, and return that much without blocking.
To quote the Javadocs (emphasis mine):
Reads up to len bytes of data from the input stream into an array of bytes. An attempt is made to read as many as len bytes, but a smaller number may be read, possibly zero. The number of bytes actually read is returned as an integer.
You need to continue reading in a loop until you have all the data you want (or the stream is finished).
Or, since you are using a DataInputStream, you can also use
in.readFully(b, 0, len);
which always reads exactly len bytes (blocking until those have arrived, throwing an exception when there is not enough data).

Android - Pass object via bluetooth

The Bluetooth chat example for Android is very useful to learn how to pass strings between phones - is it possible to use the same code to pass objects between phones? I have the same classes defined in both phones, I just want to pass the instance of one class from one phone to another. Is there any sample code available? I tried using serialization and replacing outputstream and inputstream in the chat example with objectoutputstream and objectinputstream but it didn't seem to work

The best way I found to handle this was the following:
I set up my objects as implementing Serializable that I wanted to send.
I set up the following code to manage the messages:
public byte[] serialize() throws IOException {
ByteArrayOutputStream b = new ByteArrayOutputStream();
ObjectOutputStream o = new ObjectOutputStream(b);
o.writeObject(this);
return b.toByteArray();
}
//AbstractMessage was actually the message type I used, but feel free to choose your own type
public static AbstractMessage deserialize(byte[] bytes) throws IOException, ClassNotFoundException {
ByteArrayInputStream b = new ByteArrayInputStream(bytes);
ObjectInputStream o = new ObjectInputStream(b);
return (AbstractMessage) o.readObject();
I changed the write statements to accept a Serializable, and then make the final write:
/**
* Write to the connected OutStream.
* #param buffer The bytes to write
*/
public void write(AbstractMessage buffer) {
try {
Log.v(TAG,"Writing \""+(buffer.serialize())+"\"");
mmOutStream.write(buffer.serialize());
// Share the sent message back to the UI Activity
mHandler.obtainMessage(AbstractMessageManager.MESSAGE_WRITE, -1, -1, buffer)
.sendToTarget();
} catch (IOException e) {
Log.e(TAG, "Exception during write", e);
}
}

The Bluetooth Chat example is a demonstration of using the Serial Port Profile (SPP) which is based upon RFCOMM. You can serially send across any data you like once the connection is established; you simply need to be able to represent your objects into a serial stream of bytes, i.e. serialize them.
Therefore the use of serialization would certainly be a way of getting your objects sent over the link. The Bluetooth API's send and receive functions deal with arrays of bytes, but you could easily adapt the Bluetooth Chat example to use streams, e.g. the send function would read bytes out of a stream and put them into an array buffer, then you send that buffer, etc. Then the application code would simply talk via input and output stream pipes - that's one way I've done it in the past.
So there's nothing wrong with your actual idea. The bigger problem is that the way you've implemented it is not right, and more problematic still is that the way you've asked your question is quite poor, too. You need to be more descriptive about exactly what didn't work, explain what debugging you've already tried, and post code samples and Logcat outputs so we can help you properly.
Finally, I did find what I think is a bug in the Bluetooth Chat code example: The data receive function passes a reference of the receive byte array to the ArrayList that's used to show each line of text received. This is alright when small amounts of slow text are being transmitted across, but when you try to send large amounts of data, you start to see the data being corrupted, presumably because the ArrayList adapter is still reading bytes out of that same array when the array is being filled with even newer data.

The answer is yes. A String is an Object. Remember? But how exactly to do it, I am still searching for a solution and that's what brought me here...

Trev16v,
First of all, thanks for your initial feedback.
In order to serialise my object, I used the classes serializeObject and deserializeObject from
http://www.jondev.net/articles/Android_Serialization_Example_(Java)
They seem to work well: if I serialise an object (created out of a class that implements Serializable) from a phone/activity and deserialize it from the same phone i manage to get an object out of the generated byte[].
I then tried to use the same code in the class BluetoothChatServices in the bluetooth chat example in oder to send the serialised object to the other phone (in that example there is
public ConnectedThread(BluetoothSocket socket) {
Log.d(TAG, "create ConnectedThread");
mmSocket = socket;
InputStream tmpIn = null;
OutputStream tmpOut = null;
// Get the BluetoothSocket input and output streams
try {
tmpIn = socket.getInputStream();
tmpOut = socket.getOutputStream();
} catch (IOException e) {
Log.e(TAG, "temp sockets not created", e);
}
mmInStream = tmpIn;
mmOutStream = tmpOut;
}
and the bytes are passed using
public void write(byte[] buffer) {
try {
mmOutStream.write(buffer);
// Share the sent message back to the UI Activity
mHandler.obtainMessage(BluetoothChat.MESSAGE_WRITE, -1, -1, buffer)
.sendToTarget();
} catch (IOException e) {
Log.e(TAG, "Exception during write", e);
}
}
and read using
public void run() {
Log.i(TAG, "BEGIN mConnectedThread");
byte[] buffer = new byte[10240];
int bytes;
// Keep listening to the InputStream while connected
while (true) {
try {
// Read from the InputStream
bytes = mmInStream.read(buffer);
// Send the obtained bytes to the UI Activity
mHandler.obtainMessage(BluetoothManageActivity.MESSAGE_READ, bytes, -1, buffer)
.sendToTarget();
} catch (IOException e) {
Log.e(TAG, "disconnected", e);
connectionLost();
break;
}
}
}
The problem with using BluetoothChatServices as it is is that the array of bytes received on the other phone is different from the one sent when it comes to serialised objects. For example, to give an idea element [0] of the seriealized object is =-84 when i send it, the one i receive from the other phone has element [0] to [4] =0, then [5]=4 and all the other elements are also not aligned. I tried in the methods write and run above to change Inputstream and Outputstream with ObjectInputStream and ObjectOutputstream but without success (if this was supposed to be the way to implement it, I can post the code I tried to use)
Again, thanks a lot for your help, I am new to all these concepts so if I am talking nonsense I will be also happy to be addressed to a tutorial
thanks

Facing same problem ... When i am sending a series of objects from one Android device, data sends properly ... But in receiving end all objects does not construct from received byte[].
Error occurs randomly for any received object but the same code works properly in Java ... I think the some bytes misses when transferring data from one device to another ...
Serializable object to byte[] and byte[] to object conversion can be done with the following code
public static byte[] toByteArray(Object obj)
{
byte[] bytes = null;
ObjectOutputStream oos = null;
try
{
oos = new ObjectOutputStream(new ByteArrayOutputStream());
oos.writeObject(obj);
oos.flush();
return bos.toByteArray();
}
catch(Exception e)
{
Log.e("Bluetooth", "Cast exception at sending end ...");
}
return bytes;
}
public static Object toObject(byte[] bytes)
{
Object obj = null;
ObjectInputStream ois = null;
try
{
ois = new ObjectInputStream(new ByteArrayInputStream(bytes));
return ois.readObject();
}
catch(Exception ex)
{
Log.e("Bluetooth", "Cast exception at receiving end ...");
}
return obj;
}

I actually found the problem - when the bytes are loaded using
try {
// Read from the InputStream
bytes = mmInStream.read(buffer);
they are actually loaded in 2 steps.. While debugging and stepping into the code I found that if first loads 990 bytes and then the remaining bytes.. so when i am back to the UI handler i see only the bytes loaded in the second step..
i wonder if there is a way to force to load all bytes at once