We Keep Coding

Android UDP returning random characters

I was trying to make an app that receives telemetry of F1 2020 via UDP.
For some reason, no matter how I try to turn the received bytes into a string, it just returns random characters.
I'm sure its a noob mistake somewhere but I just can't figure out where
This is the class that receives and logs the packets:
class ClientListen implements Runnable {
private Thread t;
public void run() {
boolean run = true;
try {
DatagramSocket udpSocket = new DatagramSocket(20777);
byte[] buffer = new byte[2048];
DatagramPacket packet = new DatagramPacket(buffer, buffer.length);
while (run) {
udpSocket.receive(packet);
String received = new String(packet.getData(), 0, packet.getLength());
packet.setLength(buffer.length);
Log.i("Received data", received);
try{
Thread.sleep(50);
} catch (InterruptedException e) {}
}
udpSocket.close();
}catch (IOException e) {
Log.e("UDP client has IOException", "error: ", e);
run = false;
}
}
public void start() {
if (t == null) {
t = new Thread();
t.start();
}
}
}
The thread beggins when I click a button on the app, and I can see it logs data and stops when the game is paused, as it should
The data output is something like I/Received data: �����R<�<li�)C�������GndBƨS#3Z��N��9��y�}�Ժ�~��g�r��~�ө;��J�qӼ��?�RQ=�k�<h���p�<#84�C�������...
I thought it was beacause I was logging bytes and not a string, but I tried many different ways to turn in into a string and it always has this result.

For some reason, no matter how I try to turn the received bytes into a string, it just returns random characters.
When you do new String(bytes, ...) you are attempting to decode text encoded as bytes into Unicode characters ... using your platform's default character set:
If the data represented by the bytes is text encoded in a different character set, the characters will not be decoded correctly. You need to use the correct character set.
But if the data in the bytes is binary rather than text, it is unlikely that you will be able to turn it into anything other than "random characters" by decoding this way, no matter what character set you are using.
If is common for data sent in UDP packets to be binary. It looks like that is the case here.
I'm sure its a noob mistake somewhere but I just can't figure out where
I think that your mistake is assuming that all data is fundamentally text.
When there is a specification for the format of those UDP packets, you should start by reading it. The format will determine how you should decode the packets.
A Google search found this for me:
https://forums.codemasters.com/topic/50942-f1-2020-udp-specification/
If there was no specification, you would have to reverse engineer the format from example packets. That is tedious and time-consuming, and it isn't something we can teach you to do.

Java client doesn't send whole message to python server?

I am writing a program in android studio that communicates with a python server. I tried to send a long message (mp3 file encoded in base64 - about 10k bytes). The problem is that when I check what I received in the server, I get a lot less than 10k bytes.
Anyone knows how to fix it?
Thanks in advance.
Recording message and putting it in base64:
// Record audio from user in mp3 format
MediaRecorder recorder = new MediaRecorder();
recorder.setAudioSource(MediaRecorder.AudioSource.MIC);
recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
recorder.setOutputFile(Environment.getExternalStorageDirectory()
.getAbsolutePath() + "/messageRecord.mp3");
recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AAC);
recorder.prepare();
recorder.start();
TimeUnit.SECONDS.sleep(5);
recorder.stop();
File file = new File(Environment.getExternalStorageDirectory()
.getAbsolutePath() + "/messageRecord.mp3");
int size = (int) file.length();
byte[] bytes = new byte[size];
BufferedInputStream buf = new BufferedInputStream(new FileInputStream(file));
buf.read(bytes, 0, bytes.length);
buf.close();
String content = Base64.encodeToString(bytes, Base64.DEFAULT);
// Prepare audio message request
JSONObject sendRecordReq = new JSONObject();
sendRecordReq.put("code", Codes.SPEECH_TO_TEXT_CODE);
sendRecordReq.put("src_phone", ChatScreen.this.srcPhone);
sendRecordReq.put("dst_phone", ChatScreen.this.dstPhone);
sendRecordReq.put("content", content);
// Send message request
ChatScreen.this.client.send(sendRecordReq);
How I send it:
//In class client
public void send(JSONObject request) {
this.outgoingMessages.addConversationFlow(request); //Send request
}
//In class OutgoingMessages
public void run() {
try {
while (true) {
while(!this.conversationFlow.isEmpty()) {
JSONObject msgToSend = this.conversationFlow.remove();
String strRequest = msgToSend.toString();
this.out.write(Integer.toString(strRequest.length()).getBytes()); //Sends message size
this.out.write(strRequest.getBytes()); //Sends message
}
}
}
catch (Exception e) {
System.out.println(e);
}
}
Server:
while True:
# Receiving data size from client
message_size = int(client_socket.recv(MAX_SIZE_LEN))
# Receiving data from the client
client_message = client_socket.recv(message_size)
print client_message
# Add message to messages queue with client's socket
MESSAGES_QUEUE.append((client_socket, client_message))
EDIT:
the "message_size" value is right (14806 - the size of the message that should b e received in the next line) but it still doesn't receive it all.
EDIT2:
I figured it out, ill post the solution in the answers

Your java code is sending data in a protocol that couldn't possibly work.
If the input JSON object is, say, the string 'a', then you'd send this:
3"a"
as in, 4 bytes: 51, 34, 97, 34.
The python side has no idea when the 'this is how long the data' is part ends.
I assume what you intended to send is something along the lines of:
00, 00, 00, 03, 34, 97, 34.
In other words: 4 bytes containing a network-endian sent integer value with the length, and then that many bytes.
Separately, don't call .getBytes(); when converting strings to bytes, you should always explicitly specify encoding. JSON is more or less by definition UTF-8, so, call .getBytes("UTF-8") instead.
NB: To replace your 'send the length' code, see Convert integer into byte array (Java)

So the problem wasn't with the length of the file or anything else. The problem was that the recv function in the python would get some of the bytes but the code continued for some reason, so it didn't get the whole message.
The solution is to add a function that doesn't continue till the whole length that is specified is received. I found the solution in the next post:
Python Socket Receive Large Amount of Data
I'll add the code as well:
Replace the recv function with recvall(socket, size). This is not my code, it was posted by Adam Rosenfield and edited by Hedde van der Heide.
def recvall(sock, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = sock.recv(n - len(data))
if not packet:
return None
data += packet
return data

Chat application. What should be the end of message character

I am creating a chat application in Java. User can send multiple new lines in a single message. Previously, I was not allowing the user to send new lines. So it was easy to use new line character as End OF Message. But now I am allowing the user to send new lines in a message. What character/string should I use to mark the end of the message.

You can easily avoid end of message by adding extra 4 byte. First 4 byte represent length of your message. Then add full message.
Sample sender code:
BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(socket.getOutputStream());
String msg = "its a test message";
byte[] byteMsg = msg.getBytes();
int length = byteMsg.length;
byte[] lengthByte = ByteBuffer.allocate(4).putInt(length).array();
byte[] finalMsg = new byte[length+4];
System.arraycopy(lengthByte, 0, finalMsg, 0, 4);
System.arraycopy(byteMsg, 0, finalMsg, 4, length);
bufferedOutputStream.write(finalMsg);
When you read your message then read first 4 byte. Convert this 4 byte to integer. This is your incoming message length. Then parse those byte.

It's your application so you're free to use whatever you like, including EOF and NUL characters suggested by Marko and KDM.
Just make sure it's a character your users won't be using in their messages.

ServerSocket while not breaks on image data inputstream

I tried to send an image from One device to other Device using Bluetooth.For that I take Android Bluetooth chat application source code and it works fine when I send String.But If i send image as byte array the while loop not breaks or EOF not reached when read from Inputstream.
Model:1
It receives image properly.But here I need to pass resultByteArray length.But I dont know the length.How to know the length of byte array in inputstream? inputstream.available() returns 0.
while(true)
{
byte[] resultByteArray = new byte[150827];
DataInputStream dataInputStream = new DataInputStream(mmInStream);
dataInputStream.readFully(resultByteArray);
mHandler.obtainMessage(AppConstants.MESSAGE_READ, dataInputStream.available(),-1, resultByteArray).sendToTarget();
}
Model:2
In this code while loop not breaks,
ByteArrayOutputStream bao = new ByteArrayOutputStream();
byte[] resultByteArray = new byte[1024];
int bytesRead;
while ((bytesRead = mmInStream.read(resultByteArray)) != -1) {
Log.i("BTTest1", "bytesRead=>"+bytesRead);
bao.write(resultByteArray,0,bytesRead);
}
final byte[] data = bao.toByteArray();
Also tried byte[] resultByteArray = IOUtils.toByteArray(mmInStream);but it also not works.I followed Bluetooth chat sample.
How to solve this issue?

As noted in the comment, the server needs to put the length of image at front of the actual image data. And the length of the image length information should be fixed like 4 bytes.
Then in the while loop, you need to get 4 bytes first to figure out the length of the image. After that, read bytes of the exact length from the input stream. That is the actual image.
The while loop doesn't need to break during the connection is alive. Actually it needs to wait another image data in the same while loop. The InputStream.read() is a blocking function and the thread will be sleeping until it receives enough data from the input stream.
And then you can expect another 4 bytes right after the previous image data as a start of another image.
while(true) {
try {
// Get the length first
byte[] bytesLengthOfImage = new byte[4];
mmInStream.read(bytesLengthOfImage);
int lengthOfImage = 0;
{
ByteBuffer buffer = ByteBuffer.wrap(bytesLengthOfImage);
buffer.order(ByteOrder.BIG_ENDIAN); // Assume it is network byte order.
lengthOfImage = buffer.getInt();
}
byte[] actualImage = new byte[lengthOfImage]; // Mind the memory allocation.
mmInStream.read(actualImage);
mHandler.obtainMessage(AppConstants.MESSAGE_READ, lengthOfImage,-1, actualImage).sendToTarget();
} catch (Exception e) {
if(e instanceof IOException) {
// If the connection is closed, break the loop.
break;
}
else {
// Handle errors
break;
}
}
}
This is a kind of simplified communication protocol. There is an open source framework for easy protocol implementation, called NFCommunicator.
https://github.com/Neofect/NFCommunicator
It might be an over specificiation for a simple project, but is worth a look.

Send Byte array then String then Byte over socket Java

I need to send a text message to server over Java socket and then to send a byte array and then a string etc...
What I've developed until now is working but the client manages to read only the first string that has been sent.
From the server side: I send byte array using BufferedOutputStream, and PrintWriter to send string.
The problem is that the client and the server are not synchronized, I mean the server send string then byte array then string without waiting for the client to consume each needed byte.
I mean the scenario is NOT like this:
Server Client
Send String read String
Send byte read byte
But it is like this:
Server Client
Send String
Send byte
Send String
Send byte
read String
read byte
read String
read byte
Something that could be useful is that I know exactly the size of each string and each byte array to be read.
Here is the methods used to send string and byte array respectively:
// Send String to Client
// --------------------------------------------------------------------
public void sendStringToClient (
String response,
PrintWriter output) {
try {
output.print(response);
output.flush();
} catch(Exception e) {
e.printStackTrace();
}
System.out.println("send Seeder String : " + response);
}
// Send Byte to Client
// --------------------------------------------------------------------
public void sendByteToClient (
byte[] response,
BufferedOutputStream output) {
try {
output.write(response, 0, response.length);
//System.out.println("send : " + response);
} catch (IOException e) {
e.printStackTrace();
}
}
Here is the methods used to read string and byte array respectively:
public byte[] readInByte(int size) {
byte[] command = new byte[size];
try {
this.inByte.read(command);
} catch (IOException e) {
e.printStackTrace();
}
return command;
}
public String readInString(int size) {
char[] c = new char[size];
try{
this.inString.read(c, 0, size);
} catch (IOException e) {
e.printStackTrace();
}
return String.valueOf(c);
}

Something that could be useful is that i know exactly the size of each string to be read and each byte array to be read.
Exactly. That's very common. Basically you length-prefix each message - and you might want to provide more header information than that (is it a string or a byte array message, for example).
You could represent the message length (always in bytes) either as a fixed number of bytes (e.g. 4, assuming you never need more than 4GB messages) or use a 7-bit encoded integer (where you send 7 bits of the length in each byte, and the top bit just indicates whether this is the last byte of the length or not).
Once you've got message lengths, you're basically set - you've effectively divided the stream of data into self-describing blocks. Job done.
(As an aside, I'd avoid using PrintWriter due to its exception-swallowing nature. You won't actually need the writer once you're doing this though, as you'll probably want to convert each String into a byte array anyway, to count its length in bytes before you send it. Remember to specify the encoding!)

I'd be really tempted to convert the data to JSON format and pipe it over http. You get a ton of benefits including ready made http servers and clients for just about every platform on earth along with the JSON interop not to mention all the built in error handling and recovery processing.
The drawbacks would be the additional overhead of http and the JSON encoding. You didn't mention if this was a UDP or TCP socket so that could be an additional drawback if you were trying to go connectionless.