We Keep Coding

Java client doesn't send whole message to python server?

I am writing a program in android studio that communicates with a python server. I tried to send a long message (mp3 file encoded in base64 - about 10k bytes). The problem is that when I check what I received in the server, I get a lot less than 10k bytes.
Anyone knows how to fix it?
Thanks in advance.
Recording message and putting it in base64:
// Record audio from user in mp3 format
MediaRecorder recorder = new MediaRecorder();
recorder.setAudioSource(MediaRecorder.AudioSource.MIC);
recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
recorder.setOutputFile(Environment.getExternalStorageDirectory()
.getAbsolutePath() + "/messageRecord.mp3");
recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AAC);
recorder.prepare();
recorder.start();
TimeUnit.SECONDS.sleep(5);
recorder.stop();
File file = new File(Environment.getExternalStorageDirectory()
.getAbsolutePath() + "/messageRecord.mp3");
int size = (int) file.length();
byte[] bytes = new byte[size];
BufferedInputStream buf = new BufferedInputStream(new FileInputStream(file));
buf.read(bytes, 0, bytes.length);
buf.close();
String content = Base64.encodeToString(bytes, Base64.DEFAULT);
// Prepare audio message request
JSONObject sendRecordReq = new JSONObject();
sendRecordReq.put("code", Codes.SPEECH_TO_TEXT_CODE);
sendRecordReq.put("src_phone", ChatScreen.this.srcPhone);
sendRecordReq.put("dst_phone", ChatScreen.this.dstPhone);
sendRecordReq.put("content", content);
// Send message request
ChatScreen.this.client.send(sendRecordReq);
How I send it:
//In class client
public void send(JSONObject request) {
this.outgoingMessages.addConversationFlow(request); //Send request
}
//In class OutgoingMessages
public void run() {
try {
while (true) {
while(!this.conversationFlow.isEmpty()) {
JSONObject msgToSend = this.conversationFlow.remove();
String strRequest = msgToSend.toString();
this.out.write(Integer.toString(strRequest.length()).getBytes()); //Sends message size
this.out.write(strRequest.getBytes()); //Sends message
}
}
}
catch (Exception e) {
System.out.println(e);
}
}
Server:
while True:
# Receiving data size from client
message_size = int(client_socket.recv(MAX_SIZE_LEN))
# Receiving data from the client
client_message = client_socket.recv(message_size)
print client_message
# Add message to messages queue with client's socket
MESSAGES_QUEUE.append((client_socket, client_message))
EDIT:
the "message_size" value is right (14806 - the size of the message that should b e received in the next line) but it still doesn't receive it all.
EDIT2:
I figured it out, ill post the solution in the answers

Your java code is sending data in a protocol that couldn't possibly work.
If the input JSON object is, say, the string 'a', then you'd send this:
3"a"
as in, 4 bytes: 51, 34, 97, 34.
The python side has no idea when the 'this is how long the data' is part ends.
I assume what you intended to send is something along the lines of:
00, 00, 00, 03, 34, 97, 34.
In other words: 4 bytes containing a network-endian sent integer value with the length, and then that many bytes.
Separately, don't call .getBytes(); when converting strings to bytes, you should always explicitly specify encoding. JSON is more or less by definition UTF-8, so, call .getBytes("UTF-8") instead.
NB: To replace your 'send the length' code, see Convert integer into byte array (Java)

So the problem wasn't with the length of the file or anything else. The problem was that the recv function in the python would get some of the bytes but the code continued for some reason, so it didn't get the whole message.
The solution is to add a function that doesn't continue till the whole length that is specified is received. I found the solution in the next post:
Python Socket Receive Large Amount of Data
I'll add the code as well:
Replace the recv function with recvall(socket, size). This is not my code, it was posted by Adam Rosenfield and edited by Hedde van der Heide.
def recvall(sock, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = sock.recv(n - len(data))
if not packet:
return None
data += packet
return data

InputStream reads extra bytes which are not exist in source

I have the Java server that receives the RTMP packets that are sent from client app. The server reads the packet header using InputStream, recognizes how big the packet body is, then creates byte array with that size, and then reads that body from InputStream in that array.
The problem is: the received set of bytes are modified - there are neccessary bytes (that exist in source) standing with extra bytes that don't exist in the source packet (I watch the content of the source packet via WireShark and compare them with those bytes that I received on the server).
These extra bytes are 0xc6 bytes that meet periodically by the way...
It looks like this:
Source: ... 75 f1 f5 55 73 .... fc a9 47 14 ... 40 ca d5 75 ... fe 30 a7
Received: ... 75 f1 f5 55 73 c6 .... fc a9 47 14 c6 ... 40 ca d5 75 c6 ... fe 30 a7
... - means "some quantity of bytes here"
As a result, I can't receive neccessary data because it's stretched, it's bigger than it have to be, than the body size that I received from rtmp header. And most importantly, that modified data is not what I had to receive!
My questions are: how can it be fixed? What's wrong with InputStream? Why does it insert those 0xc6 bytes to the receiving array?
I understand that I can simply parse received array and exclude those extra bytes, but this is bad solution, since speed and performance are neccessary (and, in this case, it's not clear that it's an extra byte or byte from source, without the comparison of whole arrays) ...
enter code here
public static void getRtmpPacket(InputStream in) throws Exception {
byte[] rtmpHeader = new byte[8];
byte[] rtmpBody;
int bodySize = 0;
//reading rtmp header:
in.read(rtmpHeader);
//reading the body size. This method works fine
bodySize = Server.bigEndianBytesToInt(rtmpHeader, 4, 3);
rtmpBody = new byte[bodySize];
in.read(rtmpBody);
//printing received data:
System.out.println("Packet:");
System.out.println("Body size: " + bodySize);
System.out.print(bytesToString(rtmpHeader) + " ");
System.out.print(bytesToString(rtmpBody));
System.out.println();
}

According to the RTMP spec, it behaves normally. You need to "unchunk" the incoming data, so reading it all at once in a single read() will not work.
Something along these lines (pseudocode):
int remaining = payloadSize;
int totalRead = 0;
int totalReadForChunk = 0;
while (true) {
int num = read(buf, 0, min(remaining, chunkSize - totalReadForChunk))
if (num < 0) break; // i/o error
appendData(<buf>, 0, num)
totalReadForChunk += num
remaining -= num
if (remaining == 0) break; // end of payload
if (totalReadForChunk == chunkSize) {
totalReadForChunk = 0;
// read the chunk header (it's not neccessarily 0xc6)
int header = read()
if (header != currentStreamEmptyHeader) { // 0xc6
// ... parse the new rtmp message according to header value
// (usually invoke the upper-level message reading method "recursively")
}
}
}

Probably, you should see (and use) code of Red5 Media Server and other open-source solutions that implement RTMP protocol.

InputStream.read(byte[]) is only guarenteed to read one byte, and it return the length as an int of the actual length read.
in.read(rtmpHeader); // might read 1, 2, 3, .. 8 bytes.
//reading the body size. This method works fine
bodySize = Server.bigEndianBytesToInt(rtmpHeader, 4, 3);
rtmpBody = new byte[bodySize];
in.read(rtmpBody); // might read 1, 2, 3, ... bodySize bytes.
If you don't check the actual length, and assume the byte[] is full, you get whatever bytes where there before you called read().
What you intended is available using DataInputStream
DataInputStream dis = new DataInputStream(in);
int len = dis.readInt(); // read an int in big endian.
byte[]] bytes = new byte[len];
dis.readFully(bytes); // read the whole byte[] or throw an IOException.

The problem is resolved.
Those extra 0xc6 bytes were the chunking bytes of RTMP packet, which were not visible from the WireShark.
More than this, received header says the actual body size and WireShark "confirms" it, but in fact the body size will be bigger, and should be calculated.
https://www.wireshark.org/lists/wireshark-bugs/200801/msg00011.html
http://red5.osflash.narkive.com/LYumrzr4/rtmp-video-packets-and-streaming-thereof#post12

Chat application. What should be the end of message character

I am creating a chat application in Java. User can send multiple new lines in a single message. Previously, I was not allowing the user to send new lines. So it was easy to use new line character as End OF Message. But now I am allowing the user to send new lines in a message. What character/string should I use to mark the end of the message.

You can easily avoid end of message by adding extra 4 byte. First 4 byte represent length of your message. Then add full message.
Sample sender code:
BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(socket.getOutputStream());
String msg = "its a test message";
byte[] byteMsg = msg.getBytes();
int length = byteMsg.length;
byte[] lengthByte = ByteBuffer.allocate(4).putInt(length).array();
byte[] finalMsg = new byte[length+4];
System.arraycopy(lengthByte, 0, finalMsg, 0, 4);
System.arraycopy(byteMsg, 0, finalMsg, 4, length);
bufferedOutputStream.write(finalMsg);
When you read your message then read first 4 byte. Convert this 4 byte to integer. This is your incoming message length. Then parse those byte.

It's your application so you're free to use whatever you like, including EOF and NUL characters suggested by Marko and KDM.
Just make sure it's a character your users won't be using in their messages.

Java sending byte[] through sockets...wrong length read

For my homework assignment, I have a network of Nodes that are passing messages to each other. Each Node is connected to a set amount of other Nodes (I'm using 4 for testing). Each Link has a weight, and all the Nodes have computed the shortest path for how they want their messages sent. Every Packet that is sent is composed of the message protocol (a hard-coded int), an integer that tells how many messages have passed through the sending Node, and the routing path for the Packet.
Every Node has a Thread for each of its Links. There is an active Socket in each Link. The Packets are sent by adding a 4-byte int to the beginning of the message telling the message's length.
Everything works fine until I stress the network. For my test, there are 10 Nodes, and I get 5 of them to send 10000 packets in a simple while() loop with no Thread.sleep(). Without exception, there is always an error at some point during execution at the if(a!=len) statement.
Please let me know if I can clarify anything. Thanks in advance! Here is the code (from the Link Thread; send() and forward() are called from the Node itself):
protected void listen(){
byte[] b;
int len;
try{
DataInputStream in = new DataInputStream(sock.getInputStream());
while(true){
len = in.readInt();
b = new byte[len];
int a = in.read(b,0,len);
if(a!=len){
System.out.println("ERROR: " + a + "!=" + len);
throw new SocketException(); //may have to fix...this will happen when message is corrupt/incomplete
}
Message m = new Message(b);
int p = m.getProtocol();
switch (p){
case CDNP.PACKET:
owner.incrementTracker();
System.out.print("\n# INCOMMING TRACKER: " + m.getTracker() + "\n>>> ");
owner.forward(m);
}
}
}catch (IOException e){
e.printStackTrace();
}
}
public void send(int tracker){
String[] message = { Conv.is(CDNP.PACKET), Conv.is(tracker), owner.getMST().toString() };
Message m = new Message(message);
forward(m);
}
public synchronized void forward(Message m){
try{
OutputStream out = sock.getOutputStream();
//convert length to byte array of length 4
ByteBuffer bb = ByteBuffer.allocate(4+m.getLength());
bb.putInt(m.getLength());
bb.put(m.getBytes());
out.write(bb.array());
out.flush();
}catch (UnknownHostException e){
System.out.println("ERROR: Could not send to Router at " + sock.getRemoteSocketAddress().toString());
return;
}catch (IOException e1){
}
}

int a = in.read(b,0,len);
if(a!=len){
That won't work. The InputStream may not read all the bytes you want, it may read only what is available right now, and return that much without blocking.
To quote the Javadocs (emphasis mine):
Reads up to len bytes of data from the input stream into an array of bytes. An attempt is made to read as many as len bytes, but a smaller number may be read, possibly zero. The number of bytes actually read is returned as an integer.
You need to continue reading in a loop until you have all the data you want (or the stream is finished).
Or, since you are using a DataInputStream, you can also use
in.readFully(b, 0, len);
which always reads exactly len bytes (blocking until those have arrived, throwing an exception when there is not enough data).

DataInputStream's readFully query

I am using dataInputStream's readFully message to read a fixed length byte array as:
byte[] record = new byte[4660004];
in.readFully(record);
The problem here is that sometimes it takes more than 5 seconds to read these many bytes, which is equal to 20000 records. And I am receiving this data on socket. Client is sending data as byte array of 4660004 bytes. Is there a way to received this data faster as right now it takes about 5 minutes to 1 million such records.
EDIT:: complete data flow :
first I create the stream :
static DataInputStream dIn = null;
dIn = new DataInputStream(connection.getInputStream());
msgType = dIn.readByte();
int msgIntLen = dIn.readInt();
processBatch(msgIntType, msgIntLen, dIn, connector);
.
.
private static void processBatch(int msgIntType, int msgIntLen, DataInputStream in,
Connector connector) throws IOException {
int recordIntLen = in.readInt();
byte[] record = new byte[msgIntLen - 4];
in.readFully(record);
}
where should I include the Buffering if that wudf help ?

Comments are beginning to scroll, so moving to an answer.
Buffer your output on the client side by using a BufferedOutputStream. Make sure to call dlOut.flush() after writing the data, so that unsent bytes don't remain in the buffered output stream.
Buffer your input on the client side by using a BufferedInputStream.
Because you are just sending byte arrays, you probably don't need the DataInputStream/DataOuputStream, unless you are using them for an additional purpose. You could just be using BufferedInputStream/BufferedOutputStream.