In my current project, I am trying to transmit a string from one computer to another, and after finding and learning from numerous examples I have managed to get a basic form of communication working.
The issue I am having is if one computer tries sending a message that is too long, it seems to get broken up into multiple parts (roughly 3700 characters), and my parsing method fails.
I am using a Selector to iterate through all of the channels. Here is the relevant code:
if(key.isReadable()) {
// Get the channel and read in the data
SocketChannel keyChannel = (SocketChannel)key.channel();
ByteBuffer buffer = buffers.get(keyChannel);
int length = 0;
try {
length = keyChannel.read(buffer);
} catch ( IOException ioe) {
closeChannel(keyChannel);
}
if(length > 0) {
buffer.flip();
// Gather the entire message before processing
while( buffer.remaining() > 0) {
byte[] data = new byte[buffer.remaining()];
buffer.get(data);
fireReceiveEvent(keyChannel, data);//Send the data for processing
}
buffer.compact();
} else if (length < 0) {
closeChannel(keyChannel);
}
}
How can I guarantee that the entire message (regardless of length) is read at once before passing it along?
After talking to numerous people that know more about this than I do. The issue turns out to be that with TCP it is impossible to know when an entire "message" has arrived because there is no such thing as a message since TCP works on a two-way byte stream. The solution is to create your own protocol and implements your own definition of "message".
For my project, every message either starts with a [ or { and ends with a ] or } depending on the starting character. I search through the received data, and if there is a complete message, I grab it and pass it along to the handler. Otherwise skip the channel, and wait for more to arrive.
Here is the final version of my code that handles the message receiving.
if(key.isReadable()) {
// Get the channel and read in the data
SocketChannel keyChannel = (SocketChannel)key.channel();
ByteBuffer buffer = buffers.get(keyChannel);
int length = 0;
try {
length = keyChannel.read(buffer);
} catch ( IOException ioe) {
key.cancel();
closeChannel(keyChannel);
}
if (length > 0) {
buffer.flip();
// Gather the entire message before processing
if (buffer.remaining() > 0) {
byte[] data = new byte[buffer.remaining()];
buffer.get(data);
buffer.rewind();
int index = 0;
int i = 0;
// Check for the beginning of a packet
//[ = 91
//] = 93
//{ = 123
//} = 125
if (data[0] == 91 || data[0] == 123) {
// The string we are looking for
byte targetByte = (byte) (data[0] + 2);
for (byte b : data) {
i += 1;
if (b == targetByte) {
index = i;
break;
}
}
if (index > 0) {
data = new byte[index];
buffer.get(data, 0, index);
fireReceiveEvent(keyChannel, data);
}
} else {
for (byte b : data) {
i += 1;
if (b == 91 || b == 123) {
index = i;
break;
}
}
if (index > 0) {
data = new byte[index];
buffer.get(data, 0, index); // Drain the data that we don't want
}
}
}
buffer.compact();
} else if (length < 0) {
key.cancel();
closeChannel(keyChannel);
}
}
Related
The problem is when i send up to 40 KB everything is okay when i send more sometime half of the data received some time nothing ,is there a limit of the networkstream.Read ,even though i cunked the data ,i can't determine if the problem form the java or the c# from the network stream or the Output stream
C# SERVER
private void ReadData(){
if (networkStream.DataAvailable)
{
int size = GetBufferSize();
Thread.Sleep(340);
byte[] myReadBuffer = new byte[size];
int numberOfBytesRead = 0;
while (true)
{
numberOfBytesRead = networkStream.Read(myReadBuffer, 0, myReadBuffer.Length);
if (numberOfBytesRead >= myReadBuffer.Length)
{
break;
}
}
string str = Encoding.UTF8.GetString(myReadBuffer, 0, size);
dynamic Message = Newtonsoft.Json.JsonConvert.DeserializeObject<dynamic>(str);
// Android Message , JSON String
if (OnAndroidMessage != null)
{
OnAndroidMessage(Message);
}
}
}
private int GetBufferSize()
{
byte[] myReadBuffer = new byte[4];
int numberOfBytesRead = 0;
do
{
numberOfBytesRead = networkStream.Read(myReadBuffer, 0, myReadBuffer.Length);
} while (networkStream.DataAvailable && numberOfBytesRead < myReadBuffer.Length);
if (numberOfBytesRead > 0)
{
// reverse the byte array.
if (BitConverter.IsLittleEndian)
{
Array.Reverse(myReadBuffer);
}
return BitConverter.ToInt32(myReadBuffer, 0);
}
else
{
return 0;
}
}
Java Client // i tested this also without cutting the data to smaller paces ,half of the data received not all of them
mBufferOut = socket.getOutputStream();
private void sendMessage(final String message) {
if (mBufferOut != null && message != null) {
try {
byte[] data = message.getBytes("UTF-8");
Log.d("_TAG", "Sending: " + message);
Log.d("_TAG", "Message length: " + Integer.toString(data.length));
mBufferOut.write(toByteArray(data.length));
mBufferOut.flush();
List<byte[]> divideArray = divideArray(data, 10000);
for (byte[] dataChunk : divideArray) {
Log.e("_TAG","CHUNK SIZE > " + Integer.toString(dataChunk.length));
mBufferOut.write(dataChunk, 0, dataChunk.length);
mBufferOut.flush();
}
} catch (Exception e1) {
e1.printStackTrace();
}
}
}
private List<byte[]> divideArray(byte[] source, int chunksize) {
List<byte[]> result = new ArrayList<byte[]>();
int start = 0;
while (start < source.length) {
int end = Math.min(source.length, start + chunksize);
result.add(Arrays.copyOfRange(source, start, end));
start += chunksize;
}
return result;
}
Any ideas ?
Solution from this post NetworkStream is reading data that should not be there
static void ReadExact(Stream stream, byte[] buffer, int offset, int count)
{
int read;
while(count > 0 && (read = stream.Read(buffer, offset, count)) > 0) {
offset += read;
count -= read;
}
if(count != 0) throw new EndOfStreamException();
}
the problem is the Read it takes size and want to get that size you need to give it chunks and check each chunk
And also read does not restart from where it stopped until it reads the amount is set to read meaning if i set to read 10 then if it not find the 10 then it will read what it find as example it reads 6 ,it will return 6 and when to loop another time ti read the rest it dose not start from 6 it start from 0 and read until 4 so you overwrite your data ,and if it read 10 from the first try then it set the read to finish so it dose not start from 0 ,it needs to read the amount the has been set to it to re set the read to new buffer location.
I am trying to skip a negative number of bytes with AudioInputStream
skip(long bytes) method .
The problem is trying to (let's say a small number of bytes...) :
int skipped = audioInputStream.skip(-bytes);
always returns 0 as described on this answer Java AudioInputStream skip with negative number of bytes always returns 0
I need to create an implementation which supports also negative number of bytes or something like backwards.
Here is the full code of the library on github .
What I do is recreating the line every time the user skips audio
which is extremely slow when i can of course do much better ... by just going backward or forward . Now it supports only forward ...
/**
* Skip bytes in the File input stream. It will skip N frames matching to bytes, so it will never skip given bytes len
*
* #param bytes
* the bytes
* #return value bigger than 0 for File and value = 0 for URL and InputStream
* #throws StreamPlayerException
* the stream player exception
*/
public long seek(long bytes) throws StreamPlayerException {
long totalSkipped = 0;
//If it is File
if (dataSource instanceof File) {
//Check if the requested bytes are more than totalBytes of Audio
long bytesLength = getTotalBytes();
System.out.println("Bytes: " + bytes + " BytesLength: " + bytesLength);
if ( ( bytesLength <= 0 ) || ( bytes >= bytesLength )) {
generateEvent(Status.EOM, getEncodedStreamPosition(), null);
return totalSkipped;
}
logger.info(() -> "Bytes to skip : " + bytes);
Status previousStatus = status;
status = Status.SEEKING;
try {
synchronized (audioLock) {
generateEvent(Status.SEEKING, AudioSystem.NOT_SPECIFIED, null);
initAudioInputStream();
if (audioInputStream != null) {
long skipped;
// Loop until bytes are really skipped.
while (totalSkipped < ( bytes )) { //totalSkipped < (bytes-SKIP_INACCURACY_SIZE)))
//System.out.println("Running");
skipped = audioInputStream.skip(bytes - totalSkipped);
if (skipped == 0)
break;
totalSkipped += skipped;
logger.info("Skipped : " + totalSkipped + "/" + bytes);
if (totalSkipped == -1)
throw new StreamPlayerException(StreamPlayerException.PlayerException.SKIP_NOT_SUPPORTED);
logger.info("Skeeping:" + totalSkipped);
}
}
}
generateEvent(Status.SEEKED, getEncodedStreamPosition(), null);
status = Status.OPENED;
if (previousStatus == Status.PLAYING)
play();
else if (previousStatus == Status.PAUSED) {
play();
pause();
}
} catch (IOException ex) {
logger.log(Level.WARNING, ex.getMessage(), ex);
}
}
return totalSkipped;
}
You can create your own buffer, It could be ByteArrayOutputStream but that is a bloated thing - always gives me Out of memory after a couple of minutes - or have your own Vector or other ArrayList.
I tried with a 10 min .wav file and it runs fine - as far as playing and adding the bytes to the buffer.
e.g.
Vector v=new Vector();
byte[] data=new byte[basicU];
while(true) {
k=audioInputStream.read(data, 0, data.length);
v.add(data);
if(k<0) break;
tot+=k;
}
--
Here is my method for playing a file with seeks. I have a thread for generating seek signals. The problem is complicated when we have multiple seeks. I use a variable K to check whether we need to add data to the buffer. I don't use skip but normal read; just don't play it in the line.
public void play() {
boolean seekingBack=false;
int i, j, k=0, seekPos=0, basicU=1024;
AudioFormat targetFormat=null;
int tot=0;
new Thread() {
public void run() {
while(true) {
numBytes=(Math.random()>0.5?1:-1)*500000;
try { Thread.sleep(5000); } catch (Exception e) {}
seekSignal=true;
}
}}.start();
try {
File fileIn=new File("........");
AudioInputStream audioInputStream=AudioSystem.getAudioInputStream(fileIn);
targetFormat=audioInputStream.getFormat();
DataLine.Info dinfo=new DataLine.Info(SourceDataLine.class, targetFormat);
SourceDataLine line=null;
line=(SourceDataLine)AudioSystem.getLine(dinfo);
if(line==null) return;
line.open(targetFormat);
line.start();
Vector v=new Vector();
byte[] data=new byte[basicU];
int K=0;
while(true) {
if(seekingBack) { // seeking backwards
K=seekPos;
k=data.length;
for(j=0; j<data.length; j++)
if(seekPos+j<v.size()) data[j]=((Byte)v.get(seekPos+j)).byteValue();
else { k=j; break; }
line.write(data, 0, k);
seekPos+=k;
K+=k;
if(seekPos>v.size()-1) seekingBack=false;
}
else { // normal playing
k=audioInputStream.read(data, 0, data.length);
if(k<0) break;
line.write(data, 0, k);
if(K>=v.size()) for(j=0; j<k; j++) v.add(data[j]);
K+=k;
}
if(seekSignal) { // received a seek signal
if(seekingBack) { // we are on a previous back seek - reading from the buffer
if(numBytes<0) {
seekPos+=numBytes;
if(seekPos<0) seekPos=0;
}
else { // depending on where the seek will go (in the buffer or actual audio stream)
if(numBytes+seekPos<v.size())
seekPos+=numBytes;
else { // actual stream
int rem=numBytes-(v.size()-seekPos);
K=v.size();
while(rem>0) {
k=audioInputStream.read(data, 0, data.length);
if(k<0) break;
if(K>=v.size()) for(j=0; j<k; j++) v.add(data[j]);
rem-=k;
K+=k;
}
}
}
}
else { // we are not processing a previous back seek
if(numBytes>=0) { // forward
while(numBytes>0) {
k=audioInputStream.read(data, 0, data.length);
if(k<0) break;
if(K>=v.size()) for(j=0; j<k; j++) v.add(data[j]);
numBytes-=k;
K+=k;
}
}
else { // backward
seekingBack=true; seekPos=v.size()+numBytes; if(seekPos<0) seekPos=0; }
}
seekSignal=false;
}
}
line.stop();
line.close();
}
catch(Exception ex) { ex.printStackTrace(); System.out.println("audio problem "+ex); }
}
Use your own buffer which holds a rolling window of history. I'd build a helper class that does this by allocating a List<byte[]> to manage history in blocks of for example 8192 bytes. Then you need some simple overflowing mechanism that throws out the oldest block, in combination with some pointer manipulation to keep track of where you actually are in the stream. Good luck!
My C++ program which is the client connects with the Java server . From the time of connection establishment the C++ client sends a block of data of size ~1MB to 3MB in a fixed frequency( say 10 sec).
My Java server opens a socket
Socket client = new ServerSocket(14001, 10).accept();//blocking
ReceiveThread st = new ReceiveThread(client);
and it receives the data from client as below.
private String getDataFromSocket(BufferedReader reader) throws IOException
{
int byteLimit = 1024*1024*2; //2 MB
String output = "";
char[] charArray = null;
int availableSize = is.available();
if(availableSize < 1) // If available size is 0 just return empty
{
return output;
}
while(availableSize > byteLimit) // Reads 2MB max if available size is more than 2 MB
{
charArray = new char[byteLimit];
reader.read(charArray,0,charArray.length);
output += new String(charArray);
availableSize = is.available();
}
charArray = new char[availableSize];
reader.read(charArray,0,charArray.length);
output = output +new String(charArray);
return output;
}
The above GetDataFromSocket keeps on checking for available data till the socket is closed gracefully.
the C++ connects with the Java server
void CreateSocket()
{
int err, nRet = 0;
sockfd = 0;
WORD wVersionRequested;
WSADATA wsaData;
//WSACleanup();// Is this needed?
wVersionRequested = MAKEWORD(1, 1);
while (1)
{
err = WSAStartup(wVersionRequested, &wsaData);
if (err != 0)
{
Sleep(50);
continue;
}
else
{
break;
}
}
while (1)
{
sockfd = socket(AF_INET, SOCK_STREAM, 0);
if (sockfd == -1 || sockfd == INVALID_SOCKET)
{
Sleep(50);
continue;
}
else
{
nRet = 1;
break;
}
}
}
void ConnectWithServer()
{
int nRet = 0;
char myname[256] = { 0 };
int wsaErr = 0, portNum = 0, retryCount=0;
struct hostent *h = NULL;
struct sockaddr_in server_addr;
gethostname(myname, 256);
portNum = 1401;
while (1)
{
if ((h = gethostbyname(myname)) != NULL)
{
memset(&server_addr, 0, sizeof(struct sockaddr_in));
memcpy((char *)&server_addr.sin_addr, h->h_addr, h->h_length);
server_addr.sin_family = AF_INET;
server_addr.sin_port = htons(portNum);
server_addr.sin_addr = *((struct in_addr*) h->h_addr);
}
if (0 == connect(sockfd, (struct sockaddr *)&server_addr, sizeof(server_addr)))
{
nRet = 1;
break;
}
else
{
}
Sleep(50);
}
}
The connection establishment to the server is done by the above two functions and it returns success. After these steps i am sending the data buffer to the Java server once in every 10 seconds.
while(index<retryCount)
{
string toSend = wstring_to_utf8(sRequestData);
nRet = send(sockfd, toSend.c_str(), toSend.length(), 0);
if (nRet == SOCKET_ERROR)
{
wsaErr = WSAGetLastError();
Sleep(DEFAULT);
index++;
}
else if(nRet == toSend.length())
{
break;
}
else
{
index = 0;
}
}
The problem here is, after some hours of send and receive from C++ to Java server , the send gets hanged for infinite time. The execution never comes out from the send() function. But after the hang if i abruptly close the Java server , then the send returns socket error and again works well for some hours and the hang still occurs.
As i mentioned i keep on sending data to server of size varied from 1 MB to 3 MB ten seconds once. What could be the issue here? How can i sort this out?
I am using a BluetoothSocket in Android (in spp mode). I send data like this:
Packet sent﹕ 0xAA 0x00 0x00 0x01 0x01 0x14 0x00 0x00 0xB6 0x34
and i get response:
Packet received﹕ 0xAA 0x01 0x00 0x01 0x81 0x14 0x00 0x00 0x8F 0x34
But when I try to get a large response, I get the following error:
09-25 11:13:26.583 6442-6495E/AndroidRuntime﹕ FATAL EXCEPTION: Thread-1258
Process: es.xxx.xxxx, PID: 6442
java.lang.ArrayIndexOutOfBoundsException: length=178; index=178
The error is in:
public void receive(int command, byte[] data) {
if (data.length != 0) {
int device = data[1];
int par = data[5];
short sizeData = (short)(((data[6]&0xFF) << 8) | ((data[7]&0xFF)));
byte[] datos = new byte[sizeData];
for (int i = 0; i < sizeData; i++) {
datos[i] = data[8 + i]; // Here ocurred the error
}
switch (command) {
case RETURN_PING:
break;
case RETURN_MOUNT:
...
}
My method in order to read input data from bluetooth is (I made manual timeout following a response in StackOverflow):
public byte[] read(){
try {
int timeout = 0;
int maxTimeout = 10; // leads to a timeout of 2 seconds
int available = 0;
while((available = in.available()) == 0 && timeout < maxTimeout){
timeout++;
Thread.sleep(50);
}
receive = new byte[available];
in.read(receive);
return receive.clone();
} catch (IOException e) {
e.printStackTrace();
if (socket != null){
close();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
return null;
}
So, my question is: It is possible when in.available != 0 the complete data has not been received yet? (in this way, method receive read bytes 6 and 7, where is store the packet lenght, but when this method try to iterate over all items it throw ArrayIndexOutOfBoundsException).
The major problem of your "read" is incorrect computing of how many bytes need to be read to get the whole package. There is a few main solutions how to pass and, then, read a data packet:
a. each packet has a header with its length specified
b. each packet has a predefined delimiter at its end, a kind of magic like "0x00" (this means you cannot use this byte in your data)
c. some other exotic ones
As I see, you use a. Then you may use something like this:
/**
* equals to java.io.DatainputStream.readFully
*/
public static void readFully(InputStream in, byte b[], int off, int len) throws IOException {
if (len < 0) {
throw new IndexOutOfBoundsException();
}
int n = 0;
while (n < len) {
final int count = in.read(b, off + n, len - n);
if (count < 0) {
throw new EOFException();
}
n += count;
}
}
public static int readByte (byte [] bytes, int offset) {
return ((int) bytes [offset]) & 0xFF;
}
public static short readShort(byte [] bytes, int offset) {
return (short)
(readByte(bytes, offset) << 8 |
readByte(bytes, offset + 1));
}
I see your header consists of 8 bytes. Then I'd suggest to do the following:
byte[] header = new byte[8];
readFully(in, header, 0, header.length);
int device = readByte(header, 1);
int par = readByte(header, 5);
int sizeData = readShort(header, 6);
byte[] data = new byte[sizeData];
readFully(in, data, 0, sizeData);
// now we have the whole data
After years of development I still have no a good idea what would we do with InputStream.available() :) To close a connection by data transmission timeout you could use
http://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSoTimeout(int)
or if not available, as in your case, a kind of timer
http://developer.android.com/reference/java/util/Timer.html
(update last receiving time after each call of readFully method and check the difference between current time and last receiving time by the timer)
It is possible when inputStream.available != 0 the complete data has not been received?
There is nothing in its Javadoc that says anything about 'complete data'. The Javadoc correctly states that it is a measure of how much data may be read without blocking.
It isn't:
a measure of the total length of the input stream
an indicator of message boundaries
an indicator of end of stream.
The Javadoc contains a specific warning about using its value to allocate a buffer ...
If you want a read timeout, use Socket.setSoTimeout().
I am writing a Java TCP client which sends chunks of data to a C server. The client-server worked very well on my development PC. This code upon deployment on a hardware board showed packet loss. I only have the logs with me and I know that the server did not receive all packets.
I do not have the hardware to test. Therefore, at the first level, I want to be very sure client code does send all the required data.
Here is my code(the client part in Java). How do I make sure this is done? Is there some resend commands with timings etc?
Socket mySocket = new Socket("10.0.0.2",2800);
OutputStream os = mySocket.getOutputStream();
System.out.println(" Sending 8 byte Header Msg with length of following data to Server");
os.write(hdr, 0, 8);
os.flush();
System.out.println(" Sending Data ");
start = 0;
for(int index=0; index < ((rbuffer.length/chucksize)+1); index++){
if(start + chucksize > rbuffer.length) {
System.arraycopy(rbuffer, start, val, 0, rbuffer.length - start);
} else {
System.arraycopy(rbuffer, start, val, 0, chucksize);
}
start += chucksize ;
os.write(val,0,chucksize);
os.flush();
}
Here is the C snippet which receives this data:
while ((bytes_received = recv(connected, rMsg, sizeof(rMsg),0)) > 0){
if (bytes_received > 0) // zero indicates end of transmission */
{
/* get length of message (2 bytes) */
tmpVal = 0;
tmpVal |= rMsg[idx++];
tmpVal = tmpVal << 8;
tmpVal |= rMsg[idx++];
msg_len = tmpVal;
len = msg_len;
//printf("msg_len = %d\n", len);
printf("length of following message from header message : %d\n", len);
char echoBuffer[RCVBUFSIZE] ;
memset(echoBuffer, 0, RCVBUFSIZE);
int recvMsgsize = 0;
plain=(char *)malloc(len+1);
if (!plain)
{
fprintf(stderr, "Memory error!");
}
for( i = RCVBUFSIZE; i < (len+RCVBUFSIZE); i=i+RCVBUFSIZE){
if(i>=len){
recvMsgSize = recv(connected, echoBuffer, (len - (i-RCVBUFSIZE)), 0);
memcpy(&plain[k], echoBuffer, recvMsgSize);
k = k+recvMsgSize;
}
else{
recvMsgSize = recv(connected, echoBuffer, RCVBUFSIZE, 0);
memcpy(&plain[k], echoBuffer, recvMsgSize);
k = k+recvMsgSize;
}
}
}//closing if
}//closing while
First of all there is no such thing as packet loss in TCP/IP. This protocol was designed to reliably send a stream of bytes in correct order. So the problem must be with your application or the other side.
I am not really in a mood to analyze this whole arraycopy() madness (C anyone?), but why aren't you just sending the whole rbuffer in one go through BufferedOutputStream?
OutputStream os = new BufferedOutputStream(mySocket.getOutputStream());
and then:
os.write(rbuffer);
Believe me, BufferedOutputStream is doing the exact same thing (collecting bytes into chunks and sending them in one go). Or maybe I am missing something?
I changed the C side program in the following way and it now works:
printf("length of following message from header message : %d\n", len);
plain=(char *)malloc(len+1);
if (!plain)
{
fprintf(stderr, "Memory error!");
}
memset(plain, 0, len+1);
int remain = len;
k= 0;
while (remain){
int toGet = remain > RCVBUFSIZE ? RCVBUFSIZE : remain;
remain -= toGet;
int recvd = 0;
while(recvd < toGet) {
if((recvMsgSize = recv(connected, echoBuffer, toGet-recvd, 0)) < 0){
printf("error receiving data\n");
}
memcpy(&plain[k], echoBuffer, recvMsgSize);
k += recvMsgSize;
printf("Total data accumulated after recv input %d\n", k);
recvd += recvMsgSize;
}
}