I am writing a program in android studio that communicates with a python server. I tried to send a long message (mp3 file encoded in base64 - about 10k bytes). The problem is that when I check what I received in the server, I get a lot less than 10k bytes.
Anyone knows how to fix it?
Thanks in advance.
Recording message and putting it in base64:
// Record audio from user in mp3 format
MediaRecorder recorder = new MediaRecorder();
recorder.setAudioSource(MediaRecorder.AudioSource.MIC);
recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
recorder.setOutputFile(Environment.getExternalStorageDirectory()
.getAbsolutePath() + "/messageRecord.mp3");
recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AAC);
recorder.prepare();
recorder.start();
TimeUnit.SECONDS.sleep(5);
recorder.stop();
File file = new File(Environment.getExternalStorageDirectory()
.getAbsolutePath() + "/messageRecord.mp3");
int size = (int) file.length();
byte[] bytes = new byte[size];
BufferedInputStream buf = new BufferedInputStream(new FileInputStream(file));
buf.read(bytes, 0, bytes.length);
buf.close();
String content = Base64.encodeToString(bytes, Base64.DEFAULT);
// Prepare audio message request
JSONObject sendRecordReq = new JSONObject();
sendRecordReq.put("code", Codes.SPEECH_TO_TEXT_CODE);
sendRecordReq.put("src_phone", ChatScreen.this.srcPhone);
sendRecordReq.put("dst_phone", ChatScreen.this.dstPhone);
sendRecordReq.put("content", content);
// Send message request
ChatScreen.this.client.send(sendRecordReq);
How I send it:
//In class client
public void send(JSONObject request) {
this.outgoingMessages.addConversationFlow(request); //Send request
}
//In class OutgoingMessages
public void run() {
try {
while (true) {
while(!this.conversationFlow.isEmpty()) {
JSONObject msgToSend = this.conversationFlow.remove();
String strRequest = msgToSend.toString();
this.out.write(Integer.toString(strRequest.length()).getBytes()); //Sends message size
this.out.write(strRequest.getBytes()); //Sends message
}
}
}
catch (Exception e) {
System.out.println(e);
}
}
Server:
while True:
# Receiving data size from client
message_size = int(client_socket.recv(MAX_SIZE_LEN))
# Receiving data from the client
client_message = client_socket.recv(message_size)
print client_message
# Add message to messages queue with client's socket
MESSAGES_QUEUE.append((client_socket, client_message))
EDIT:
the "message_size" value is right (14806 - the size of the message that should b e received in the next line) but it still doesn't receive it all.
EDIT2:
I figured it out, ill post the solution in the answers
Your java code is sending data in a protocol that couldn't possibly work.
If the input JSON object is, say, the string 'a', then you'd send this:
3"a"
as in, 4 bytes: 51, 34, 97, 34.
The python side has no idea when the 'this is how long the data' is part ends.
I assume what you intended to send is something along the lines of:
00, 00, 00, 03, 34, 97, 34.
In other words: 4 bytes containing a network-endian sent integer value with the length, and then that many bytes.
Separately, don't call .getBytes(); when converting strings to bytes, you should always explicitly specify encoding. JSON is more or less by definition UTF-8, so, call .getBytes("UTF-8") instead.
NB: To replace your 'send the length' code, see Convert integer into byte array (Java)
So the problem wasn't with the length of the file or anything else. The problem was that the recv function in the python would get some of the bytes but the code continued for some reason, so it didn't get the whole message.
The solution is to add a function that doesn't continue till the whole length that is specified is received. I found the solution in the next post:
Python Socket Receive Large Amount of Data
I'll add the code as well:
Replace the recv function with recvall(socket, size). This is not my code, it was posted by Adam Rosenfield and edited by Hedde van der Heide.
def recvall(sock, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = sock.recv(n - len(data))
if not packet:
return None
data += packet
return data
Related
I was trying to make an app that receives telemetry of F1 2020 via UDP.
For some reason, no matter how I try to turn the received bytes into a string, it just returns random characters.
I'm sure its a noob mistake somewhere but I just can't figure out where
This is the class that receives and logs the packets:
class ClientListen implements Runnable {
private Thread t;
public void run() {
boolean run = true;
try {
DatagramSocket udpSocket = new DatagramSocket(20777);
byte[] buffer = new byte[2048];
DatagramPacket packet = new DatagramPacket(buffer, buffer.length);
while (run) {
udpSocket.receive(packet);
String received = new String(packet.getData(), 0, packet.getLength());
packet.setLength(buffer.length);
Log.i("Received data", received);
try{
Thread.sleep(50);
} catch (InterruptedException e) {}
}
udpSocket.close();
}catch (IOException e) {
Log.e("UDP client has IOException", "error: ", e);
run = false;
}
}
public void start() {
if (t == null) {
t = new Thread();
t.start();
}
}
}
The thread beggins when I click a button on the app, and I can see it logs data and stops when the game is paused, as it should
The data output is something like I/Received data: �����R<�<li�)C�������GndBƨS#3Z��N��9��y�}�Ժ�~��g�r��~�ө;��J�qӼ��?�RQ=�k�<h���p�<#84�C�������...
I thought it was beacause I was logging bytes and not a string, but I tried many different ways to turn in into a string and it always has this result.
For some reason, no matter how I try to turn the received bytes into a string, it just returns random characters.
When you do new String(bytes, ...) you are attempting to decode text encoded as bytes into Unicode characters ... using your platform's default character set:
If the data represented by the bytes is text encoded in a different character set, the characters will not be decoded correctly. You need to use the correct character set.
But if the data in the bytes is binary rather than text, it is unlikely that you will be able to turn it into anything other than "random characters" by decoding this way, no matter what character set you are using.
If is common for data sent in UDP packets to be binary. It looks like that is the case here.
I'm sure its a noob mistake somewhere but I just can't figure out where
I think that your mistake is assuming that all data is fundamentally text.
When there is a specification for the format of those UDP packets, you should start by reading it. The format will determine how you should decode the packets.
A Google search found this for me:
https://forums.codemasters.com/topic/50942-f1-2020-udp-specification/
If there was no specification, you would have to reverse engineer the format from example packets. That is tedious and time-consuming, and it isn't something we can teach you to do.
I need to pass an Uint8List to a byte[] array in java over a TCPSocket. There is size mismatch at both ends. PS: I am a beginner in dart.
I have tried socket.add(buf), socket.write(buf), socket.writeAll(buf) but none of them worked
Code in Flutter side (TCP client)
void readVoiceData(Uint8List buf) {
print("Send data size:"+buf.lengthInBytes.toString());
socket.add(buf);
}
OUTPUT: Send data size: 1280
Code Snippet on java side (TCP server)
in = new DataInputStream(clientSocket.getInputStream());
Log.d(TAG, "****Opened InputStream*******");
while (!isInterrupted()) {
if (mTrack != null) {
try {
in.read(bytes);
Log.d(TAG, "Received data size"+bytes.length);
}
}
OUTPUT: Received data size: 1
I am sure the socket connection has been established correctly as I am able to send Strings and Integers flawlessly over them.
This happens because the read() method of the FilterInputStream (DataInput is his son) reads just the next avaliable byte.
Thou, if you want to get the total size of the InputStream, do something like:
size = in.available(); //returns an estimate of number of bytes avaliable on the stream
buf = new byte[size];
realLength= in.read(buf, 0, size);
This code snippet is taken from here.
I'm communicating with peers using TCP Sockets and I see that when I read the inputStream for the first incoming message, all goes well. Then when I read the inputStream for the second incoming message, the inputStream skip the first n bytes (n is a different positive number in each run).
How do I know that inputStream skip n bytes? Using Wireshark, I can see that the second message received well but Java TCP socket still ignore the first n bytes.
Moreover, Wireshark itself show me something strange - by looking at the first message in Wireshark, it contains at the end: the start of the second message. And by looking at the second message in Wireshark, the start of the message appears here also.
I can't understand what is going on.
Technical details + Wireshark photos:
The first message I receive is a 'Handshake' message.
The second message I receive is different each time but most of the time it's 'extended' message.
I checked in my code and I only read the same InputStream in 2 places: When I'm waiting for 'Handshake' and when I'm waiting for the rest of the messages which is not equal to 'Handshake' message.
The first message I receive:
* Offset Size Name value
* 0 8-bit byte pstrLength
* 1 pstrlen-bit bytes pstr
* 1+pstrlen 64-bit byte reserved
* 9+pstrlen 20-bit String torrentInfoHash
* 29+pstrlen 20-bit String peerId
* 49+pstrlen
public HandShake(InputStream dataInputStream) throws IOException {
byte[] data = new byte[1];
dataInputStream.read(data);
byte pstrLength = ByteBuffer.wrap(data).get();
data = new byte[pstrLength + 48];// how much we need to read more.
dataInputStream.read(data);
ByteBuffer byteBuffer = ByteBuffer.allocate(1 + pstrLength + 48);
byteBuffer.put(pstrLength);
byteBuffer.put(data);
HandShake handShake = HandShake.createObjectFromPacket(byteBuffer.array());
Details: 13 until 45 is the content of the first message - Handshake. 00 until 3a is the first n bytes fo the second message which will appear also in here:
The second message I receive:
public static PeerMessage create(Peer from, Peer to, InputStream inputStream) throws IOException {
byte[] data = new byte[4];
boolean isPeerClosedConnection = (inputStream.read(data) == -1);
if (isPeerClosedConnection)
throw new IOException("the peer closed the socket:" + from.toString());
int lengthOfTheRest = ByteBuffer.wrap(data).getInt(); // how much do we need to read more
data = new byte[lengthOfTheRest];
isPeerClosedConnection = (inputStream.read(data) == -1);
if (isPeerClosedConnection)
throw new IOException("the peer closed the socket:" + from.toString());
ByteBuffer byteBuffer = ByteBuffer.allocate(4 + lengthOfTheRest);;
byteBuffer.putInt(lengthOfTheRest);
byteBuffer.put(data);
return create(from, to, byteBuffer.array()); // initialize message object from byte[]
}
Details: 00 until 3a is the first n bytes of the second message.
When I read the InputStream, I get the following bytes: from 6d to 65.
Why Wireshark shows the same data twice and why my InputStream skip the first n bytes of the second message?
You wrote:
I calculate how much to read and I use every byte.
You coded:
data = new byte[pstrLength + 48];// how much we need to read more.
dataInputStream.read(data);
This code does not conform with your description. The second read() is not guaranteed to fill the buffer. See the Javadoc. Change it to readFully().
NB There is another problem, in your isPeerConnected test. You are reading a byte of input and throwing it away. This will cause you to lose synchronization with the peer if it is still connected.
I am using C# to create a server software for Windows and Java to create the client software.
It works fine most of the time, except for those few exceptions that I don't understand.
I am generally using .ReadLine() and .WriteLine() on both ends to communicate, unless I try to send binary data. That's when I write and read the bytes directly.
This is how the software is supposed work:
Client requests the binary data
Server responds with the length of the binary data as a string
Client receives the length and converts it into an integer and starts reading (length) bytes
Server starts writing (length) bytes
It works in most cases, but sometimes the client app doesn't receive the full data and blocks. The server always immediately flushes after writing data, so flushing is not the problem.
Furthermore I've noticed this usually happens with larger files, small files (up to ~1 MB) usually are not a problem.
NOTE It seems like the C# server does send the data completely, so the problem is most likely somewhere in the Java code.
EDIT - Here are some logs from the client side
Working download: pastebin.com/hFd5TvrF
Failing download: pastebin.com/Q3zFWRLB
It seems like the client is waiting for 2048 bytes at the end (as it should be, as length - processed = 2048 in this case), but for some reason the client blocks.
Any ideas what I'm doing wrong? Below are the source codes of both server and client:
C# Server:
public void Write(BinaryWriter str, byte[] data)
{
int BUFFER = 2048;
int PROCESSED = 0;
// WriteString sends the String using a StreamWriter (+ flushing)
WriteString(data.Length.ToString());
while (PROCESSED < data.Length)
{
if (PROCESSED + BUFFER > data.Length)
BUFFER = data.Length - PROCESSED;
str.Write(data, PROCESSED, BUFFER);
str.Flush();
PROCESSED += BUFFER;
}
}
Java Client:
public byte[] ReadBytes(int length){
byte[] buffer = new byte[length];
int PROCESSED = 0;
int READBUF = 2048;
TOTAL = length;
progress.setMax(TOTAL);
InputStream m;
try {
m = clientSocket.getInputStream();
while(PROCESSED < length){
if(PROCESSED + READBUF > length)
READBUF = length - PROCESSED;
try {
PROCESSED += m.read(buffer, PROCESSED, READBUF);
} catch (IOException e) {
}
XPROCESSED = PROCESSED;
}
} catch (IOException e1) {
// Removed because of sensitive data
}
return decryptData(buffer);
}
I've found a fix. As of now, the server sends the length and right after sends the byte array. For some reason this does not work.
So what I've changed is:
Send length and wait for the client to respond with "OK"
Start writing bytes
Not sure why, but it works. Ran it in a while(true) loop and it's been sending data 1000 times in 4 minutes straight and no problems, so I guess it's fixed.
I'm trying to send progress values (actually hashmaps that contain progress values) from one device to another over bluetooth, so that both devices can display the same progress dialog value
So on the sender device I'm wrapping the value up in a hashmap (so I can differentiate between message types by checking the first element):
public static void sendProgress(Integer progress){
HashMap<String,String> saveProgress = new HashMap<>();
saveProgress.put("type", "saveProgress");
saveProgress.put("progressValue", Integer.toString(progress));
BTService.connectedThread.write(MainActivity.gson.toJson(saveProgress).getBytes());
}
On the receiving side I'm reading the message stream like this:
while (!this.isInterrupted()) {
try {
// Read from the InputStream
bytes = mmInStream.read(buffer);
// Send the obtained bytes to the UI activity
mHandler.obtainMessage(Constants.MESSAGE_READ, bytes, -1, buffer)
.sendToTarget();
} catch (IOException e) {
e.printStackTrace();
break;
}
}
The problem is that because its not actually sending an object, its just sending a stream of bytes, the receiving device doesn't know when one "object" ends and the next begins.
Is there a way to rewrite my read input stream code so that it can treat each individual hash map as objects?
I know that I can add some sort of character to the end of the json string before I convert it to bytes, and then check for the character on the receiving side, but I'm sending a lot of data all throughout my app and would rather not have to refactor everything
One thing I've tried is adding a terminator whenever I write on the sender device:
//Set the termination character (#)
byte[] stopBytes = "#".getBytes();
byte[] tempArray = new byte[bytes.length + stopBytes.length];
//Attach a terminator to the message
//copy bytes into start of destination (from pos 0, copy bytes.length bytes)
System.arraycopy(bytes, 0, tempArray, 0, bytes.length);
//copy stopBytes into end of destination (from pos bytes.length, copy stopBytes.length bytes)
System.arraycopy(stopBytes, 0, tempArray, bytes.length, stopBytes.length);
Log.d(TAG, new String(tempArray));
//Write the object with terminator attached
mmOutStream.write(tempArray);
In the case of sending across data like sensor data in a hashmap, I can confirm that it is sending something like this:
{"gravX":"0.39","accZ":"9.929751","magZ":"-27.125","accY":"0.60594","magY":"10.5","gravY":"0.7","accX":"0.44547364","magX":"23.3125","gravZ":"9.7699995","type":"accData"}#
Then on the receiving side I read each byte of the input stream and check for a terminator. If its found, then send the message, otherwise keep reading:
// Read from the InputStream
buffer[bytes] = (byte) mmInStream.read();
// Send the obtained bytes to the UI activity
if ((buffer[bytes] == '#')){
//Check for terminator (#) before sending the message/object
mHandler.obtainMessage(Constants.MESSAGE_READ, bytes, -1, buffer).sendToTarget();
bytes=0;
} else{
bytes++;
}
I get some errors with this method though, because in some cases the object I end up receiving is something like:
{"gravX":"0.39","accZ":"9.903406","magZ":"-26.3125","accY":"0.6394702","magY":"9.8125","gravY":"0.62","accX":"0.41912845","magX":"25.5625","gravZ":"9.78","type":"accData"}#}#ata"}
As you can see it isnt splitting by terminator correctly, and also adds some other junk to the end. Obviously this is a problem when I try to turn this back into a useable hashmap on the receiving side. Where do these extra characters come from?