We Keep Coding

Java client doesn't send whole message to python server?

I am writing a program in android studio that communicates with a python server. I tried to send a long message (mp3 file encoded in base64 - about 10k bytes). The problem is that when I check what I received in the server, I get a lot less than 10k bytes.
Anyone knows how to fix it?
Thanks in advance.
Recording message and putting it in base64:
// Record audio from user in mp3 format
MediaRecorder recorder = new MediaRecorder();
recorder.setAudioSource(MediaRecorder.AudioSource.MIC);
recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
recorder.setOutputFile(Environment.getExternalStorageDirectory()
.getAbsolutePath() + "/messageRecord.mp3");
recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AAC);
recorder.prepare();
recorder.start();
TimeUnit.SECONDS.sleep(5);
recorder.stop();
File file = new File(Environment.getExternalStorageDirectory()
.getAbsolutePath() + "/messageRecord.mp3");
int size = (int) file.length();
byte[] bytes = new byte[size];
BufferedInputStream buf = new BufferedInputStream(new FileInputStream(file));
buf.read(bytes, 0, bytes.length);
buf.close();
String content = Base64.encodeToString(bytes, Base64.DEFAULT);
// Prepare audio message request
JSONObject sendRecordReq = new JSONObject();
sendRecordReq.put("code", Codes.SPEECH_TO_TEXT_CODE);
sendRecordReq.put("src_phone", ChatScreen.this.srcPhone);
sendRecordReq.put("dst_phone", ChatScreen.this.dstPhone);
sendRecordReq.put("content", content);
// Send message request
ChatScreen.this.client.send(sendRecordReq);
How I send it:
//In class client
public void send(JSONObject request) {
this.outgoingMessages.addConversationFlow(request); //Send request
}
//In class OutgoingMessages
public void run() {
try {
while (true) {
while(!this.conversationFlow.isEmpty()) {
JSONObject msgToSend = this.conversationFlow.remove();
String strRequest = msgToSend.toString();
this.out.write(Integer.toString(strRequest.length()).getBytes()); //Sends message size
this.out.write(strRequest.getBytes()); //Sends message
}
}
}
catch (Exception e) {
System.out.println(e);
}
}
Server:
while True:
# Receiving data size from client
message_size = int(client_socket.recv(MAX_SIZE_LEN))
# Receiving data from the client
client_message = client_socket.recv(message_size)
print client_message
# Add message to messages queue with client's socket
MESSAGES_QUEUE.append((client_socket, client_message))
EDIT:
the "message_size" value is right (14806 - the size of the message that should b e received in the next line) but it still doesn't receive it all.
EDIT2:
I figured it out, ill post the solution in the answers

Your java code is sending data in a protocol that couldn't possibly work.
If the input JSON object is, say, the string 'a', then you'd send this:
3"a"
as in, 4 bytes: 51, 34, 97, 34.
The python side has no idea when the 'this is how long the data' is part ends.
I assume what you intended to send is something along the lines of:
00, 00, 00, 03, 34, 97, 34.
In other words: 4 bytes containing a network-endian sent integer value with the length, and then that many bytes.
Separately, don't call .getBytes(); when converting strings to bytes, you should always explicitly specify encoding. JSON is more or less by definition UTF-8, so, call .getBytes("UTF-8") instead.
NB: To replace your 'send the length' code, see Convert integer into byte array (Java)

So the problem wasn't with the length of the file or anything else. The problem was that the recv function in the python would get some of the bytes but the code continued for some reason, so it didn't get the whole message.
The solution is to add a function that doesn't continue till the whole length that is specified is received. I found the solution in the next post:
Python Socket Receive Large Amount of Data
I'll add the code as well:
Replace the recv function with recvall(socket, size). This is not my code, it was posted by Adam Rosenfield and edited by Hedde van der Heide.
def recvall(sock, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = sock.recv(n - len(data))
if not packet:
return None
data += packet
return data

InputStream skip bytes

I'm communicating with peers using TCP Sockets and I see that when I read the inputStream for the first incoming message, all goes well. Then when I read the inputStream for the second incoming message, the inputStream skip the first n bytes (n is a different positive number in each run).
How do I know that inputStream skip n bytes? Using Wireshark, I can see that the second message received well but Java TCP socket still ignore the first n bytes.
Moreover, Wireshark itself show me something strange - by looking at the first message in Wireshark, it contains at the end: the start of the second message. And by looking at the second message in Wireshark, the start of the message appears here also.
I can't understand what is going on.
Technical details + Wireshark photos:
The first message I receive is a 'Handshake' message.
The second message I receive is different each time but most of the time it's 'extended' message.
I checked in my code and I only read the same InputStream in 2 places: When I'm waiting for 'Handshake' and when I'm waiting for the rest of the messages which is not equal to 'Handshake' message.
The first message I receive:
* Offset Size Name value
* 0 8-bit byte pstrLength
* 1 pstrlen-bit bytes pstr
* 1+pstrlen 64-bit byte reserved
* 9+pstrlen 20-bit String torrentInfoHash
* 29+pstrlen 20-bit String peerId
* 49+pstrlen
public HandShake(InputStream dataInputStream) throws IOException {
byte[] data = new byte[1];
dataInputStream.read(data);
byte pstrLength = ByteBuffer.wrap(data).get();
data = new byte[pstrLength + 48];// how much we need to read more.
dataInputStream.read(data);
ByteBuffer byteBuffer = ByteBuffer.allocate(1 + pstrLength + 48);
byteBuffer.put(pstrLength);
byteBuffer.put(data);
HandShake handShake = HandShake.createObjectFromPacket(byteBuffer.array());
Details: 13 until 45 is the content of the first message - Handshake. 00 until 3a is the first n bytes fo the second message which will appear also in here:
The second message I receive:
public static PeerMessage create(Peer from, Peer to, InputStream inputStream) throws IOException {
byte[] data = new byte[4];
boolean isPeerClosedConnection = (inputStream.read(data) == -1);
if (isPeerClosedConnection)
throw new IOException("the peer closed the socket:" + from.toString());
int lengthOfTheRest = ByteBuffer.wrap(data).getInt(); // how much do we need to read more
data = new byte[lengthOfTheRest];
isPeerClosedConnection = (inputStream.read(data) == -1);
if (isPeerClosedConnection)
throw new IOException("the peer closed the socket:" + from.toString());
ByteBuffer byteBuffer = ByteBuffer.allocate(4 + lengthOfTheRest);;
byteBuffer.putInt(lengthOfTheRest);
byteBuffer.put(data);
return create(from, to, byteBuffer.array()); // initialize message object from byte[]
}
Details: 00 until 3a is the first n bytes of the second message.
When I read the InputStream, I get the following bytes: from 6d to 65.
Why Wireshark shows the same data twice and why my InputStream skip the first n bytes of the second message?

You wrote:
I calculate how much to read and I use every byte.
You coded:
data = new byte[pstrLength + 48];// how much we need to read more.
dataInputStream.read(data);
This code does not conform with your description. The second read() is not guaranteed to fill the buffer. See the Javadoc. Change it to readFully().
NB There is another problem, in your isPeerConnected test. You are reading a byte of input and throwing it away. This will cause you to lose synchronization with the peer if it is still connected.

TCP sending/receiving missing bytes

I am using C# to create a server software for Windows and Java to create the client software.
It works fine most of the time, except for those few exceptions that I don't understand.
I am generally using .ReadLine() and .WriteLine() on both ends to communicate, unless I try to send binary data. That's when I write and read the bytes directly.
This is how the software is supposed work:
Client requests the binary data
Server responds with the length of the binary data as a string
Client receives the length and converts it into an integer and starts reading (length) bytes
Server starts writing (length) bytes
It works in most cases, but sometimes the client app doesn't receive the full data and blocks. The server always immediately flushes after writing data, so flushing is not the problem.
Furthermore I've noticed this usually happens with larger files, small files (up to ~1 MB) usually are not a problem.
NOTE It seems like the C# server does send the data completely, so the problem is most likely somewhere in the Java code.
EDIT - Here are some logs from the client side
Working download: pastebin.com/hFd5TvrF
Failing download: pastebin.com/Q3zFWRLB
It seems like the client is waiting for 2048 bytes at the end (as it should be, as length - processed = 2048 in this case), but for some reason the client blocks.
Any ideas what I'm doing wrong? Below are the source codes of both server and client:
C# Server:
public void Write(BinaryWriter str, byte[] data)
{
int BUFFER = 2048;
int PROCESSED = 0;
// WriteString sends the String using a StreamWriter (+ flushing)
WriteString(data.Length.ToString());
while (PROCESSED < data.Length)
{
if (PROCESSED + BUFFER > data.Length)
BUFFER = data.Length - PROCESSED;
str.Write(data, PROCESSED, BUFFER);
str.Flush();
PROCESSED += BUFFER;
}
}
Java Client:
public byte[] ReadBytes(int length){
byte[] buffer = new byte[length];
int PROCESSED = 0;
int READBUF = 2048;
TOTAL = length;
progress.setMax(TOTAL);
InputStream m;
try {
m = clientSocket.getInputStream();
while(PROCESSED < length){
if(PROCESSED + READBUF > length)
READBUF = length - PROCESSED;
try {
PROCESSED += m.read(buffer, PROCESSED, READBUF);
} catch (IOException e) {
}
XPROCESSED = PROCESSED;
}
} catch (IOException e1) {
// Removed because of sensitive data
}
return decryptData(buffer);
}

I've found a fix. As of now, the server sends the length and right after sends the byte array. For some reason this does not work.
So what I've changed is:
Send length and wait for the client to respond with "OK"
Start writing bytes
Not sure why, but it works. Ran it in a while(true) loop and it's been sending data 1000 times in 4 minutes straight and no problems, so I guess it's fixed.

AES/CTR/NoPadding last block is lost when sending encrypted data between PC and Android

I'm using AES/CTR/NoPadding algorithm to encrypt data sent using socket between PC and Android.
I wrote unit-test, it sends [1;512] bytes to Android device and receive back the same data - echo service. Received data must be equal to data that was sent.
Test client:
for (int n = 1; n <= 512; n++) {
... skip ...
try {
Object connection = socketFilter.openConnection(socket);
in = new CipherInputStream(socket.getInputStream(), encryptor);
out = new CipherOutputStream(socket.getOutputStream(), decryptor);
byte buf[] = new byte[n];
byte received[] = new byte[n];
TestUtils.numbers(buf);
out.write(buf, 0, buf.length);
socket.shutdownOutput();
int len = in.read(received, 0, received.length);
if (buf.length != len) {
System.err.println("Expected: " + buf.length + " but was: " + len);
}
}
finally {
... skip close streams ...
}
}
Echo server:
Socket clientSocket = socket.accept();
CipherInputStream in = new CipherInputStream(clientSocket.getInputStream(), decryptor);
CipherOutputStream out = new CipherOutputStream(clientSocket.getOutputStream(), encryptor);
try {
byte buf[] = new byte[512];
int len;
if ((len = in.read(buf)) > 0) {
out.write(buf, 0, len);
out.close();
}
}
finally {
in.close();
out.close();
}
I tested this code with localhost - all works fine.
When i testing it with Android device, the last block is lost if it's not full.
So, if it was 30 bytes, then only 16 bytes received.
Messages from the test:
... skip ...
Expected: 30 but was: 16
Expected: 31 but was: 16
Expected: 33 but was: 32
... skip ...
Expected: 207 but was: 192
Expected: 209 but was: 208
Expected: 210 but was: 208
... skip ...
What can be wrong?

It seems that the problem is caused by the fact that Android and Hotspot JVM uses different Cipher Provider.
Android uses one called Bouncy Castle which has a known 'bug' in AES/CTR mode. It will miss the last block when doing encryption/decryption. (see many other stackoverflow questions)
If you want only CTR mode. Known work around is that you implement it yourself on Android by generating blocks of keystream repeatedly "on the fly"(by encrypt byte array of 0's), and XOR them with your buffer.
Hope this helps

Have you completely flushed the encryption streams before you close them? AES processes data in block sized chunks, in CTR mode it is the keystream. If you don't fully flush the stream before closing it, on either encryption or decryption, you will probably lose the last block.
Similarly, you need to be sure that you have written/read everything from the transfer file streams between Android and PC. Your last pieces of data may have been sitting in a file transfer buffer somewhere waiting to be written when the buffer was closed.
Android does differ from Java, so I suspect that your error is probably on the Android side. Perhaps try Android -> Android as well as PC -> PC, just to make sure that everything on the Android side is fine.

How to read raw bytes from socket in Python?

I have an android java app sending bytes over a socket which is connected to a host machine running a server in Python. I need to receive these bytes as they were sent from the python socket. I see that in Python 'socket.recv' only returns a string. When I send an ASCII string from the java app, I am able to receive the data correctly in the python server, but when I send binary data using java byte, I see the data received is not same. I need to receive raw bytes in Python for my protocol to work correctly. Please point me in right direction.
Code snippet for Sending data on socket:
private void sendFrameMessage(byte[] data) {
byte[] lengthInfo = new byte[4];
Log.v(TAG, "sendFrameMessage");
for(int i=0; i<data.length; i++) {
Log.v(TAG, String.format("data[%d] = %d", i, data[i]));
}
try {
lengthInfo[0] = (byte) data.length;
lengthInfo[1] = (byte) (data.length >> 8);
lengthInfo[2] = (byte) (data.length >> 16);
lengthInfo[3] = (byte) (data.length >> 24);
DataOutputStream dos;
dos = new DataOutputStream(mSocket.getOutputStream());
dos.write(lengthInfo, 0, 4);
dos.write(data, 0, data.length);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
Python Code on receiver side
def recvFrameMessage(self, s):
recv_count = 4;
data = s.recv(recv_count)
if data == 0:
return None
total_rx = len(data)
lenInfo = data
while total_rx < recv_count:
data = s.recv(recv_count - total_rx)
if data == 0:
return None
total_rx += len(data)
lenInfo = lenInfo + data
recv_count = self.decodeFrameLen(lenInfo)
logger.info("length = %d" % recv_count)
data = s.recv(recv_count)
total_rx = len(data)
msg = data
while total_rx < recv_count:
data = s.recv(recv_count - total_rx)
if data == 0:
return None
total_rx += len(data)
msg = msg + data
logger.info("msg = " + msg)
for i in range(0, len(msg)-1):
logger.info("msg[%d] = %s" % (i, msg[i]))
return msg

#SteveP makes good points for binary data "with some structure", but if this is a plain stream of bytes, in Python 2 simply apply the ord() function to each "character" you get from the socket. For example, if the Java end sends a NUL byte, that will show up on the Python end as the character "\x00", and then:
>>> ord("\x00")
0
To convert a whole string s,
map(ord, s)
returns a list of the corresponding 8-bit unsigned integers.
I'm assuming Python 2 here.

Reading binary data is perfectly doable, but what if the binary representation from your android app is different than the byte representation on the Python server? From the Python documentation:
It is perfectly possible to send binary data over a socket. The major
problem is that not all machines use the same formats for binary data.
For example, a Motorola chip will represent a 16 bit integer with the
value 1 as the two hex bytes 00 01. Intel and DEC, however, are
byte-reversed - that same 1 is 01 00. Socket libraries have calls for
converting 16 and 32 bit integers - ntohl, htonl, ntohs, htons where
“n” means network and “h” means host, “s” means short and “l” means
long. Where network order is host order, these do nothing, but where
the machine is byte-reversed, these swap the bytes around
appropriately.
Without code and example input/output, this question is going to be really difficult to answer. I assume the issue is that the representation is different. The most likely issue is that Java uses big endian, whereas Python adheres to whatever machine you are running it off of. If your server uses little endian, then you need to account for that. See here for a more thorough explanation on endianness.