I need to send a text message to server over Java socket and then to send a byte array and then a string etc...
What I've developed until now is working but the client manages to read only the first string that has been sent.
From the server side: I send byte array using BufferedOutputStream, and PrintWriter to send string.
The problem is that the client and the server are not synchronized, I mean the server send string then byte array then string without waiting for the client to consume each needed byte.
I mean the scenario is NOT like this:
Server Client
Send String read String
Send byte read byte
But it is like this:
Server Client
Send String
Send byte
Send String
Send byte
read String
read byte
read String
read byte
Something that could be useful is that I know exactly the size of each string and each byte array to be read.
Here is the methods used to send string and byte array respectively:
// Send String to Client
// --------------------------------------------------------------------
public void sendStringToClient (
String response,
PrintWriter output) {
try {
output.print(response);
output.flush();
} catch(Exception e) {
e.printStackTrace();
}
System.out.println("send Seeder String : " + response);
}
// Send Byte to Client
// --------------------------------------------------------------------
public void sendByteToClient (
byte[] response,
BufferedOutputStream output) {
try {
output.write(response, 0, response.length);
//System.out.println("send : " + response);
} catch (IOException e) {
e.printStackTrace();
}
}
Here is the methods used to read string and byte array respectively:
public byte[] readInByte(int size) {
byte[] command = new byte[size];
try {
this.inByte.read(command);
} catch (IOException e) {
e.printStackTrace();
}
return command;
}
public String readInString(int size) {
char[] c = new char[size];
try{
this.inString.read(c, 0, size);
} catch (IOException e) {
e.printStackTrace();
}
return String.valueOf(c);
}
Something that could be useful is that i know exactly the size of each string to be read and each byte array to be read.
Exactly. That's very common. Basically you length-prefix each message - and you might want to provide more header information than that (is it a string or a byte array message, for example).
You could represent the message length (always in bytes) either as a fixed number of bytes (e.g. 4, assuming you never need more than 4GB messages) or use a 7-bit encoded integer (where you send 7 bits of the length in each byte, and the top bit just indicates whether this is the last byte of the length or not).
Once you've got message lengths, you're basically set - you've effectively divided the stream of data into self-describing blocks. Job done.
(As an aside, I'd avoid using PrintWriter due to its exception-swallowing nature. You won't actually need the writer once you're doing this though, as you'll probably want to convert each String into a byte array anyway, to count its length in bytes before you send it. Remember to specify the encoding!)
I'd be really tempted to convert the data to JSON format and pipe it over http. You get a ton of benefits including ready made http servers and clients for just about every platform on earth along with the JSON interop not to mention all the built in error handling and recovery processing.
The drawbacks would be the additional overhead of http and the JSON encoding. You didn't mention if this was a UDP or TCP socket so that could be an additional drawback if you were trying to go connectionless.
Related
I was trying to make an app that receives telemetry of F1 2020 via UDP.
For some reason, no matter how I try to turn the received bytes into a string, it just returns random characters.
I'm sure its a noob mistake somewhere but I just can't figure out where
This is the class that receives and logs the packets:
class ClientListen implements Runnable {
private Thread t;
public void run() {
boolean run = true;
try {
DatagramSocket udpSocket = new DatagramSocket(20777);
byte[] buffer = new byte[2048];
DatagramPacket packet = new DatagramPacket(buffer, buffer.length);
while (run) {
udpSocket.receive(packet);
String received = new String(packet.getData(), 0, packet.getLength());
packet.setLength(buffer.length);
Log.i("Received data", received);
try{
Thread.sleep(50);
} catch (InterruptedException e) {}
}
udpSocket.close();
}catch (IOException e) {
Log.e("UDP client has IOException", "error: ", e);
run = false;
}
}
public void start() {
if (t == null) {
t = new Thread();
t.start();
}
}
}
The thread beggins when I click a button on the app, and I can see it logs data and stops when the game is paused, as it should
The data output is something like I/Received data: �����R<�<li�)C�������GndBƨS#3Z��N��9��y�}�Ժ�~��g�r��~�ө;��J�qӼ��?�RQ=�k�<h���p�<#84�C�������...
I thought it was beacause I was logging bytes and not a string, but I tried many different ways to turn in into a string and it always has this result.
For some reason, no matter how I try to turn the received bytes into a string, it just returns random characters.
When you do new String(bytes, ...) you are attempting to decode text encoded as bytes into Unicode characters ... using your platform's default character set:
If the data represented by the bytes is text encoded in a different character set, the characters will not be decoded correctly. You need to use the correct character set.
But if the data in the bytes is binary rather than text, it is unlikely that you will be able to turn it into anything other than "random characters" by decoding this way, no matter what character set you are using.
If is common for data sent in UDP packets to be binary. It looks like that is the case here.
I'm sure its a noob mistake somewhere but I just can't figure out where
I think that your mistake is assuming that all data is fundamentally text.
When there is a specification for the format of those UDP packets, you should start by reading it. The format will determine how you should decode the packets.
A Google search found this for me:
https://forums.codemasters.com/topic/50942-f1-2020-udp-specification/
If there was no specification, you would have to reverse engineer the format from example packets. That is tedious and time-consuming, and it isn't something we can teach you to do.
I am writing a program in android studio that communicates with a python server. I tried to send a long message (mp3 file encoded in base64 - about 10k bytes). The problem is that when I check what I received in the server, I get a lot less than 10k bytes.
Anyone knows how to fix it?
Thanks in advance.
Recording message and putting it in base64:
// Record audio from user in mp3 format
MediaRecorder recorder = new MediaRecorder();
recorder.setAudioSource(MediaRecorder.AudioSource.MIC);
recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
recorder.setOutputFile(Environment.getExternalStorageDirectory()
.getAbsolutePath() + "/messageRecord.mp3");
recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AAC);
recorder.prepare();
recorder.start();
TimeUnit.SECONDS.sleep(5);
recorder.stop();
File file = new File(Environment.getExternalStorageDirectory()
.getAbsolutePath() + "/messageRecord.mp3");
int size = (int) file.length();
byte[] bytes = new byte[size];
BufferedInputStream buf = new BufferedInputStream(new FileInputStream(file));
buf.read(bytes, 0, bytes.length);
buf.close();
String content = Base64.encodeToString(bytes, Base64.DEFAULT);
// Prepare audio message request
JSONObject sendRecordReq = new JSONObject();
sendRecordReq.put("code", Codes.SPEECH_TO_TEXT_CODE);
sendRecordReq.put("src_phone", ChatScreen.this.srcPhone);
sendRecordReq.put("dst_phone", ChatScreen.this.dstPhone);
sendRecordReq.put("content", content);
// Send message request
ChatScreen.this.client.send(sendRecordReq);
How I send it:
//In class client
public void send(JSONObject request) {
this.outgoingMessages.addConversationFlow(request); //Send request
}
//In class OutgoingMessages
public void run() {
try {
while (true) {
while(!this.conversationFlow.isEmpty()) {
JSONObject msgToSend = this.conversationFlow.remove();
String strRequest = msgToSend.toString();
this.out.write(Integer.toString(strRequest.length()).getBytes()); //Sends message size
this.out.write(strRequest.getBytes()); //Sends message
}
}
}
catch (Exception e) {
System.out.println(e);
}
}
Server:
while True:
# Receiving data size from client
message_size = int(client_socket.recv(MAX_SIZE_LEN))
# Receiving data from the client
client_message = client_socket.recv(message_size)
print client_message
# Add message to messages queue with client's socket
MESSAGES_QUEUE.append((client_socket, client_message))
EDIT:
the "message_size" value is right (14806 - the size of the message that should b e received in the next line) but it still doesn't receive it all.
EDIT2:
I figured it out, ill post the solution in the answers
Your java code is sending data in a protocol that couldn't possibly work.
If the input JSON object is, say, the string 'a', then you'd send this:
3"a"
as in, 4 bytes: 51, 34, 97, 34.
The python side has no idea when the 'this is how long the data' is part ends.
I assume what you intended to send is something along the lines of:
00, 00, 00, 03, 34, 97, 34.
In other words: 4 bytes containing a network-endian sent integer value with the length, and then that many bytes.
Separately, don't call .getBytes(); when converting strings to bytes, you should always explicitly specify encoding. JSON is more or less by definition UTF-8, so, call .getBytes("UTF-8") instead.
NB: To replace your 'send the length' code, see Convert integer into byte array (Java)
So the problem wasn't with the length of the file or anything else. The problem was that the recv function in the python would get some of the bytes but the code continued for some reason, so it didn't get the whole message.
The solution is to add a function that doesn't continue till the whole length that is specified is received. I found the solution in the next post:
Python Socket Receive Large Amount of Data
I'll add the code as well:
Replace the recv function with recvall(socket, size). This is not my code, it was posted by Adam Rosenfield and edited by Hedde van der Heide.
def recvall(sock, n):
# Helper function to recv n bytes or return None if EOF is hit
data = b''
while len(data) < n:
packet = sock.recv(n - len(data))
if not packet:
return None
data += packet
return data
I am using C# to create a server software for Windows and Java to create the client software.
It works fine most of the time, except for those few exceptions that I don't understand.
I am generally using .ReadLine() and .WriteLine() on both ends to communicate, unless I try to send binary data. That's when I write and read the bytes directly.
This is how the software is supposed work:
Client requests the binary data
Server responds with the length of the binary data as a string
Client receives the length and converts it into an integer and starts reading (length) bytes
Server starts writing (length) bytes
It works in most cases, but sometimes the client app doesn't receive the full data and blocks. The server always immediately flushes after writing data, so flushing is not the problem.
Furthermore I've noticed this usually happens with larger files, small files (up to ~1 MB) usually are not a problem.
NOTE It seems like the C# server does send the data completely, so the problem is most likely somewhere in the Java code.
EDIT - Here are some logs from the client side
Working download: pastebin.com/hFd5TvrF
Failing download: pastebin.com/Q3zFWRLB
It seems like the client is waiting for 2048 bytes at the end (as it should be, as length - processed = 2048 in this case), but for some reason the client blocks.
Any ideas what I'm doing wrong? Below are the source codes of both server and client:
C# Server:
public void Write(BinaryWriter str, byte[] data)
{
int BUFFER = 2048;
int PROCESSED = 0;
// WriteString sends the String using a StreamWriter (+ flushing)
WriteString(data.Length.ToString());
while (PROCESSED < data.Length)
{
if (PROCESSED + BUFFER > data.Length)
BUFFER = data.Length - PROCESSED;
str.Write(data, PROCESSED, BUFFER);
str.Flush();
PROCESSED += BUFFER;
}
}
Java Client:
public byte[] ReadBytes(int length){
byte[] buffer = new byte[length];
int PROCESSED = 0;
int READBUF = 2048;
TOTAL = length;
progress.setMax(TOTAL);
InputStream m;
try {
m = clientSocket.getInputStream();
while(PROCESSED < length){
if(PROCESSED + READBUF > length)
READBUF = length - PROCESSED;
try {
PROCESSED += m.read(buffer, PROCESSED, READBUF);
} catch (IOException e) {
}
XPROCESSED = PROCESSED;
}
} catch (IOException e1) {
// Removed because of sensitive data
}
return decryptData(buffer);
}
I've found a fix. As of now, the server sends the length and right after sends the byte array. For some reason this does not work.
So what I've changed is:
Send length and wait for the client to respond with "OK"
Start writing bytes
Not sure why, but it works. Ran it in a while(true) loop and it's been sending data 1000 times in 4 minutes straight and no problems, so I guess it's fixed.
The Bluetooth chat example for Android is very useful to learn how to pass strings between phones - is it possible to use the same code to pass objects between phones? I have the same classes defined in both phones, I just want to pass the instance of one class from one phone to another. Is there any sample code available? I tried using serialization and replacing outputstream and inputstream in the chat example with objectoutputstream and objectinputstream but it didn't seem to work
The best way I found to handle this was the following:
I set up my objects as implementing Serializable that I wanted to send.
I set up the following code to manage the messages:
public byte[] serialize() throws IOException {
ByteArrayOutputStream b = new ByteArrayOutputStream();
ObjectOutputStream o = new ObjectOutputStream(b);
o.writeObject(this);
return b.toByteArray();
}
//AbstractMessage was actually the message type I used, but feel free to choose your own type
public static AbstractMessage deserialize(byte[] bytes) throws IOException, ClassNotFoundException {
ByteArrayInputStream b = new ByteArrayInputStream(bytes);
ObjectInputStream o = new ObjectInputStream(b);
return (AbstractMessage) o.readObject();
I changed the write statements to accept a Serializable, and then make the final write:
/**
* Write to the connected OutStream.
* #param buffer The bytes to write
*/
public void write(AbstractMessage buffer) {
try {
Log.v(TAG,"Writing \""+(buffer.serialize())+"\"");
mmOutStream.write(buffer.serialize());
// Share the sent message back to the UI Activity
mHandler.obtainMessage(AbstractMessageManager.MESSAGE_WRITE, -1, -1, buffer)
.sendToTarget();
} catch (IOException e) {
Log.e(TAG, "Exception during write", e);
}
}
The Bluetooth Chat example is a demonstration of using the Serial Port Profile (SPP) which is based upon RFCOMM. You can serially send across any data you like once the connection is established; you simply need to be able to represent your objects into a serial stream of bytes, i.e. serialize them.
Therefore the use of serialization would certainly be a way of getting your objects sent over the link. The Bluetooth API's send and receive functions deal with arrays of bytes, but you could easily adapt the Bluetooth Chat example to use streams, e.g. the send function would read bytes out of a stream and put them into an array buffer, then you send that buffer, etc. Then the application code would simply talk via input and output stream pipes - that's one way I've done it in the past.
So there's nothing wrong with your actual idea. The bigger problem is that the way you've implemented it is not right, and more problematic still is that the way you've asked your question is quite poor, too. You need to be more descriptive about exactly what didn't work, explain what debugging you've already tried, and post code samples and Logcat outputs so we can help you properly.
Finally, I did find what I think is a bug in the Bluetooth Chat code example: The data receive function passes a reference of the receive byte array to the ArrayList that's used to show each line of text received. This is alright when small amounts of slow text are being transmitted across, but when you try to send large amounts of data, you start to see the data being corrupted, presumably because the ArrayList adapter is still reading bytes out of that same array when the array is being filled with even newer data.
The answer is yes. A String is an Object. Remember? But how exactly to do it, I am still searching for a solution and that's what brought me here...
Trev16v,
First of all, thanks for your initial feedback.
In order to serialise my object, I used the classes serializeObject and deserializeObject from
http://www.jondev.net/articles/Android_Serialization_Example_(Java)
They seem to work well: if I serialise an object (created out of a class that implements Serializable) from a phone/activity and deserialize it from the same phone i manage to get an object out of the generated byte[].
I then tried to use the same code in the class BluetoothChatServices in the bluetooth chat example in oder to send the serialised object to the other phone (in that example there is
public ConnectedThread(BluetoothSocket socket) {
Log.d(TAG, "create ConnectedThread");
mmSocket = socket;
InputStream tmpIn = null;
OutputStream tmpOut = null;
// Get the BluetoothSocket input and output streams
try {
tmpIn = socket.getInputStream();
tmpOut = socket.getOutputStream();
} catch (IOException e) {
Log.e(TAG, "temp sockets not created", e);
}
mmInStream = tmpIn;
mmOutStream = tmpOut;
}
and the bytes are passed using
public void write(byte[] buffer) {
try {
mmOutStream.write(buffer);
// Share the sent message back to the UI Activity
mHandler.obtainMessage(BluetoothChat.MESSAGE_WRITE, -1, -1, buffer)
.sendToTarget();
} catch (IOException e) {
Log.e(TAG, "Exception during write", e);
}
}
and read using
public void run() {
Log.i(TAG, "BEGIN mConnectedThread");
byte[] buffer = new byte[10240];
int bytes;
// Keep listening to the InputStream while connected
while (true) {
try {
// Read from the InputStream
bytes = mmInStream.read(buffer);
// Send the obtained bytes to the UI Activity
mHandler.obtainMessage(BluetoothManageActivity.MESSAGE_READ, bytes, -1, buffer)
.sendToTarget();
} catch (IOException e) {
Log.e(TAG, "disconnected", e);
connectionLost();
break;
}
}
}
The problem with using BluetoothChatServices as it is is that the array of bytes received on the other phone is different from the one sent when it comes to serialised objects. For example, to give an idea element [0] of the seriealized object is =-84 when i send it, the one i receive from the other phone has element [0] to [4] =0, then [5]=4 and all the other elements are also not aligned. I tried in the methods write and run above to change Inputstream and Outputstream with ObjectInputStream and ObjectOutputstream but without success (if this was supposed to be the way to implement it, I can post the code I tried to use)
Again, thanks a lot for your help, I am new to all these concepts so if I am talking nonsense I will be also happy to be addressed to a tutorial
thanks
Facing same problem ... When i am sending a series of objects from one Android device, data sends properly ... But in receiving end all objects does not construct from received byte[].
Error occurs randomly for any received object but the same code works properly in Java ... I think the some bytes misses when transferring data from one device to another ...
Serializable object to byte[] and byte[] to object conversion can be done with the following code
public static byte[] toByteArray(Object obj)
{
byte[] bytes = null;
ObjectOutputStream oos = null;
try
{
oos = new ObjectOutputStream(new ByteArrayOutputStream());
oos.writeObject(obj);
oos.flush();
return bos.toByteArray();
}
catch(Exception e)
{
Log.e("Bluetooth", "Cast exception at sending end ...");
}
return bytes;
}
public static Object toObject(byte[] bytes)
{
Object obj = null;
ObjectInputStream ois = null;
try
{
ois = new ObjectInputStream(new ByteArrayInputStream(bytes));
return ois.readObject();
}
catch(Exception ex)
{
Log.e("Bluetooth", "Cast exception at receiving end ...");
}
return obj;
}
I actually found the problem - when the bytes are loaded using
try {
// Read from the InputStream
bytes = mmInStream.read(buffer);
they are actually loaded in 2 steps.. While debugging and stepping into the code I found that if first loads 990 bytes and then the remaining bytes.. so when i am back to the UI handler i see only the bytes loaded in the second step..
i wonder if there is a way to force to load all bytes at once
NOW here is the coding for j2me mobile for sending the string:
String s="hai";
try{
String url = "btspp://001F81000250:1;authenticate=false;encrypt=false;master=false";
StreamConnection stream = null;
InputStream in;
OutputStream out;
stream = (StreamConnection) Connector.open(url);
out=stream.openOutputStream();
String s=tf.getString();
byte size=(byte) s.length();
out.write(size);
out.write(s.getBytes());
out.flush();
out.close();
stream.close();
}
catch(Exception e){
}
NOW the coding for j2se for receiving the String :
StreamConnectionNotifier notifier=null;
try{
String url = "btspp://localhost:"+new UUID("1101", true).toString()+";name=PCServerCOMM;authenticate=false";
System.out.println(LocalDevice.getLocalDevice().getBluetoothAddress()+"\nCreate server by uri: " + url);
notifier= (StreamConnectionNotifier) Connector.open(url);
while(true){
System.out.println("waiting....");
StreamConnection con = notifier.acceptAndOpen();
System.out.println("Got connection..");
InputStream is=con.openInputStream();
//byte b[]=new byte[40];
/*
while(is.available()>0){
System.out.print((char)is.read());
}*/
//is.read(b, 0, 40);
int size=is.read();
byte b[]=new byte[size];
is.read(b, 0, size);
File f=new File("d://test.xml");
FileOutputStream fo=new FileOutputStream(f);
fo.write(b,0,b.length);
fo.close();
con.close();
System.out.println(new String (b));
}
//printing(f);
} catch(Exception e){
JOptionPane.showConfirmDialog(new JFrame(), e.getMessage());
}
I tried this coding for data transfer but it is not a successful one because when the string which we sent is too long then there is problem in receiving side. How can I solve this?
Is there any other way to transfer the data in rms to j2se, if so please help me.... please make your reply quick...
The way you are writing and reading here, only strings up to 255 characters in length, which additionally only take the same number of bytes in your default encoding, are written right.
On the writing side:
The statement byte size=(byte) s.length(); converts the length of the string in a byte, thus only takes the lower 8 bits of the length. So, only lengths up to 255 are written right.
Then you are converting the String to a byte array with s.getBytes() - this array could be longer (in bytes) than the original string in characters. This conversion uses the default encoding of your sending device.
On the reading side:
The statement int size=is.read(); reads the length written before, then you are creating a byte array.
is.read(b, 0, size); reads some bytes into this array - it does not necessarily fills the complete array.
Then you are converting your byte array (which may not even be filled completely) to a string, using the default encoding of the receiving device.
So, we have:
All strings longer than 255 characters are written wrongly.
If sending and receiving side are using different encodings, you may get a wrong output.
If the sending side uses an encoding like UTF-8 where some characters take more than one byte, the string is cut off at the end (if such characters occur).
How to solve this:
If you can use a DataInputStream and DataOutputStream on both sides (I don't know anything about J2ME), use them there, with their readUTF and writeUTF methods. They solve all your problems (if your strings take at most 65535 bytes in the modified UTF-8 encoding used here).
If not:
make a decision on how long the strings can be, and encode your length with the right number of bytes. 4 bytes are enough for every Java String.
measure the length after converting to a byte[], not before.
use a loop for reading into the array, to be sure to capture the whole string.
for the getBytes() and new String(...), use the variants which take an explicit encoding name and give them the same encoding (I recommend "UTF-8").