I am creating a chat application in Java. User can send multiple new lines in a single message. Previously, I was not allowing the user to send new lines. So it was easy to use new line character as End OF Message. But now I am allowing the user to send new lines in a message. What character/string should I use to mark the end of the message.
You can easily avoid end of message by adding extra 4 byte. First 4 byte represent length of your message. Then add full message.
Sample sender code:
BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(socket.getOutputStream());
String msg = "its a test message";
byte[] byteMsg = msg.getBytes();
int length = byteMsg.length;
byte[] lengthByte = ByteBuffer.allocate(4).putInt(length).array();
byte[] finalMsg = new byte[length+4];
System.arraycopy(lengthByte, 0, finalMsg, 0, 4);
System.arraycopy(byteMsg, 0, finalMsg, 4, length);
bufferedOutputStream.write(finalMsg);
When you read your message then read first 4 byte. Convert this 4 byte to integer. This is your incoming message length. Then parse those byte.
It's your application so you're free to use whatever you like, including EOF and NUL characters suggested by Marko and KDM.
Just make sure it's a character your users won't be using in their messages.
Related
I am writing a program in android studio that communicates with a python server. I tried to send a long message (mp3 file encoded in base64 - about 10k bytes). The problem is that when I check what I received in the server, I get a lot less than 10k bytes.
Anyone knows how to fix it?
Thanks in advance.
Recording message and putting it in base64:
// Record audio from user in mp3 format
MediaRecorder recorder = new MediaRecorder();
recorder.setAudioSource(MediaRecorder.AudioSource.MIC);
recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
recorder.setOutputFile(Environment.getExternalStorageDirectory()
.getAbsolutePath() + "/messageRecord.mp3");
recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AAC);
recorder.prepare();
recorder.start();
TimeUnit.SECONDS.sleep(5);
recorder.stop();
File file = new File(Environment.getExternalStorageDirectory()
.getAbsolutePath() + "/messageRecord.mp3");
int size = (int) file.length();
byte[] bytes = new byte[size];
BufferedInputStream buf = new BufferedInputStream(new FileInputStream(file));
buf.read(bytes, 0, bytes.length);
buf.close();
String content = Base64.encodeToString(bytes, Base64.DEFAULT);
// Prepare audio message request
JSONObject sendRecordReq = new JSONObject();
sendRecordReq.put("code", Codes.SPEECH_TO_TEXT_CODE);
sendRecordReq.put("src_phone", ChatScreen.this.srcPhone);
sendRecordReq.put("dst_phone", ChatScreen.this.dstPhone);
sendRecordReq.put("content", content);
// Send message request
ChatScreen.this.client.send(sendRecordReq);
How I send it:
//In class client
public void send(JSONObject request) {
this.outgoingMessages.addConversationFlow(request); //Send request
}
//In class OutgoingMessages
public void run() {
try {
while (true) {
while(!this.conversationFlow.isEmpty()) {
JSONObject msgToSend = this.conversationFlow.remove();
String strRequest = msgToSend.toString();
this.out.write(Integer.toString(strRequest.length()).getBytes()); //Sends message size
this.out.write(strRequest.getBytes()); //Sends message
}
}
}
catch (Exception e) {
System.out.println(e);
}
}
Server:
while True:
# Receiving data size from client
message_size = int(client_socket.recv(MAX_SIZE_LEN))
# Receiving data from the client
client_message = client_socket.recv(message_size)
print client_message
# Add message to messages queue with client's socket
MESSAGES_QUEUE.append((client_socket, client_message))
EDIT:
the "message_size" value is right (14806 - the size of the message that should b e received in the next line) but it still doesn't receive it all.
EDIT2:
I figured it out, ill post the solution in the answers
Your java code is sending data in a protocol that couldn't possibly work.
If the input JSON object is, say, the string 'a', then you'd send this:
3"a"
as in, 4 bytes: 51, 34, 97, 34.
The python side has no idea when the 'this is how long the data' is part ends.
I assume what you intended to send is something along the lines of:
00, 00, 00, 03, 34, 97, 34.
In other words: 4 bytes containing a network-endian sent integer value with the length, and then that many bytes.
Separately, don't call .getBytes(); when converting strings to bytes, you should always explicitly specify encoding. JSON is more or less by definition UTF-8, so, call .getBytes("UTF-8") instead.
NB: To replace your 'send the length' code, see Convert integer into byte array (Java)
So the problem wasn't with the length of the file or anything else. The problem was that the recv function in the python would get some of the bytes but the code continued for some reason, so it didn't get the whole message.
The solution is to add a function that doesn't continue till the whole length that is specified is received. I found the solution in the next post:
Python Socket Receive Large Amount of Data
I'll add the code as well:
Replace the recv function with recvall(socket, size). This is not my code, it was posted by Adam Rosenfield and edited by Hedde van der Heide.
def recvall(sock, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = sock.recv(n - len(data))
if not packet:
return None
data += packet
return data
I'm communicating with peers using TCP Sockets and I see that when I read the inputStream for the first incoming message, all goes well. Then when I read the inputStream for the second incoming message, the inputStream skip the first n bytes (n is a different positive number in each run).
How do I know that inputStream skip n bytes? Using Wireshark, I can see that the second message received well but Java TCP socket still ignore the first n bytes.
Moreover, Wireshark itself show me something strange - by looking at the first message in Wireshark, it contains at the end: the start of the second message. And by looking at the second message in Wireshark, the start of the message appears here also.
I can't understand what is going on.
Technical details + Wireshark photos:
The first message I receive is a 'Handshake' message.
The second message I receive is different each time but most of the time it's 'extended' message.
I checked in my code and I only read the same InputStream in 2 places: When I'm waiting for 'Handshake' and when I'm waiting for the rest of the messages which is not equal to 'Handshake' message.
The first message I receive:
* Offset Size Name value
* 0 8-bit byte pstrLength
* 1 pstrlen-bit bytes pstr
* 1+pstrlen 64-bit byte reserved
* 9+pstrlen 20-bit String torrentInfoHash
* 29+pstrlen 20-bit String peerId
* 49+pstrlen
public HandShake(InputStream dataInputStream) throws IOException {
byte[] data = new byte[1];
dataInputStream.read(data);
byte pstrLength = ByteBuffer.wrap(data).get();
data = new byte[pstrLength + 48];// how much we need to read more.
dataInputStream.read(data);
ByteBuffer byteBuffer = ByteBuffer.allocate(1 + pstrLength + 48);
byteBuffer.put(pstrLength);
byteBuffer.put(data);
HandShake handShake = HandShake.createObjectFromPacket(byteBuffer.array());
Details: 13 until 45 is the content of the first message - Handshake. 00 until 3a is the first n bytes fo the second message which will appear also in here:
The second message I receive:
public static PeerMessage create(Peer from, Peer to, InputStream inputStream) throws IOException {
byte[] data = new byte[4];
boolean isPeerClosedConnection = (inputStream.read(data) == -1);
if (isPeerClosedConnection)
throw new IOException("the peer closed the socket:" + from.toString());
int lengthOfTheRest = ByteBuffer.wrap(data).getInt(); // how much do we need to read more
data = new byte[lengthOfTheRest];
isPeerClosedConnection = (inputStream.read(data) == -1);
if (isPeerClosedConnection)
throw new IOException("the peer closed the socket:" + from.toString());
ByteBuffer byteBuffer = ByteBuffer.allocate(4 + lengthOfTheRest);;
byteBuffer.putInt(lengthOfTheRest);
byteBuffer.put(data);
return create(from, to, byteBuffer.array()); // initialize message object from byte[]
}
Details: 00 until 3a is the first n bytes of the second message.
When I read the InputStream, I get the following bytes: from 6d to 65.
Why Wireshark shows the same data twice and why my InputStream skip the first n bytes of the second message?
You wrote:
I calculate how much to read and I use every byte.
You coded:
data = new byte[pstrLength + 48];// how much we need to read more.
dataInputStream.read(data);
This code does not conform with your description. The second read() is not guaranteed to fill the buffer. See the Javadoc. Change it to readFully().
NB There is another problem, in your isPeerConnected test. You are reading a byte of input and throwing it away. This will cause you to lose synchronization with the peer if it is still connected.
I'm trying to send progress values (actually hashmaps that contain progress values) from one device to another over bluetooth, so that both devices can display the same progress dialog value
So on the sender device I'm wrapping the value up in a hashmap (so I can differentiate between message types by checking the first element):
public static void sendProgress(Integer progress){
HashMap<String,String> saveProgress = new HashMap<>();
saveProgress.put("type", "saveProgress");
saveProgress.put("progressValue", Integer.toString(progress));
BTService.connectedThread.write(MainActivity.gson.toJson(saveProgress).getBytes());
}
On the receiving side I'm reading the message stream like this:
while (!this.isInterrupted()) {
try {
// Read from the InputStream
bytes = mmInStream.read(buffer);
// Send the obtained bytes to the UI activity
mHandler.obtainMessage(Constants.MESSAGE_READ, bytes, -1, buffer)
.sendToTarget();
} catch (IOException e) {
e.printStackTrace();
break;
}
}
The problem is that because its not actually sending an object, its just sending a stream of bytes, the receiving device doesn't know when one "object" ends and the next begins.
Is there a way to rewrite my read input stream code so that it can treat each individual hash map as objects?
I know that I can add some sort of character to the end of the json string before I convert it to bytes, and then check for the character on the receiving side, but I'm sending a lot of data all throughout my app and would rather not have to refactor everything
One thing I've tried is adding a terminator whenever I write on the sender device:
//Set the termination character (#)
byte[] stopBytes = "#".getBytes();
byte[] tempArray = new byte[bytes.length + stopBytes.length];
//Attach a terminator to the message
//copy bytes into start of destination (from pos 0, copy bytes.length bytes)
System.arraycopy(bytes, 0, tempArray, 0, bytes.length);
//copy stopBytes into end of destination (from pos bytes.length, copy stopBytes.length bytes)
System.arraycopy(stopBytes, 0, tempArray, bytes.length, stopBytes.length);
Log.d(TAG, new String(tempArray));
//Write the object with terminator attached
mmOutStream.write(tempArray);
In the case of sending across data like sensor data in a hashmap, I can confirm that it is sending something like this:
{"gravX":"0.39","accZ":"9.929751","magZ":"-27.125","accY":"0.60594","magY":"10.5","gravY":"0.7","accX":"0.44547364","magX":"23.3125","gravZ":"9.7699995","type":"accData"}#
Then on the receiving side I read each byte of the input stream and check for a terminator. If its found, then send the message, otherwise keep reading:
// Read from the InputStream
buffer[bytes] = (byte) mmInStream.read();
// Send the obtained bytes to the UI activity
if ((buffer[bytes] == '#')){
//Check for terminator (#) before sending the message/object
mHandler.obtainMessage(Constants.MESSAGE_READ, bytes, -1, buffer).sendToTarget();
bytes=0;
} else{
bytes++;
}
I get some errors with this method though, because in some cases the object I end up receiving is something like:
{"gravX":"0.39","accZ":"9.903406","magZ":"-26.3125","accY":"0.6394702","magY":"9.8125","gravY":"0.62","accX":"0.41912845","magX":"25.5625","gravZ":"9.78","type":"accData"}#}#ata"}
As you can see it isnt splitting by terminator correctly, and also adds some other junk to the end. Obviously this is a problem when I try to turn this back into a useable hashmap on the receiving side. Where do these extra characters come from?
I would like to get data from InputStream() as a String eg. Hi, Start, Stop, etc.
My Code fragment is
byte[] buffer = new byte[1024];
inputStream.read(buffer);
My data is Command Sent from Bluetooth.
The above code fragment only get (eg. 2 if I sent Hi, 5 if sent Start), I would like to get back Normal String as the same from the Sender Side.
I found only converting way from String to InputStream.
any suggestion , I would like to appreciate!
finally, i can solve this!
inputStream.read(buffer); only return the int num of how many bytes is in the buffer and the data from the socket is stored in buffer. So, from the buffer you can make String
eg. String result = new String (buffer);
The simplest way is via the Apache common-io library:
String input = IOUtils.toString(inputStream);
See the javadoc for this method for more.
I'm trying to write simple code for server/client.
The server (in java) wait for string from client:
ServerSocket ss = new ServerSocket(4001);
Socket s = ss.accept() ;
DataInputStream dataStreamIn = new DataInputStream(s.getInputStream()) ;
byte buffer[] = new byte[100];
dataStreamIn.read(buffer);
if((new String(buffer)).equals("1"))
System.out.print("yes");//never printed
the client (objective-c) send string to server:
uint8_t buffer[1000] ={0};
sprintf((char*)buffer,"%s", "1");
NSInteger wrote = [self.networkStreamOut
write:buffer
maxLength:(uint8_t)strlen((char *)buffer)];
The problem is the buffer on server is indeed "1" BUT when I'm trying to compare with .equals() it return false!
EDIT:
When I'm tring to add to server this line:
System.out.println(Integer.valueOf(new String(buffer))) ;
I'm getting this error:
Exception in thread "main" java.lang.NumberFormatException: For input string: "1
You should explicitly state what encoding you expect your incoming data to be in, on both the client and server side. This is especially important to do when communicating between different platforms/languages.
But, of course, that's not what your problem is - your string is getting created with a bunch of non-printing characters because you allocated it with the entire byte array:
byte[] buffer = new byte[100];
new String(buffer);
Presumably the buffer is not completely filled after you read data into it, and the defaulted zero values in it are getting converted in your string into non-printing characters. You can validate this by printing the length of the string you created, you will find it to be greater than 1.
You have two remedies - either trim() the String before comparing its value, or keep track of the number of actual bytes read and create a new byte array of the correct length (before creating your String).