We Keep Coding

Java client to C# server TCP

The problem is when i send up to 40 KB everything is okay when i send more sometime half of the data received some time nothing ,is there a limit of the networkstream.Read ,even though i cunked the data ,i can't determine if the problem form the java or the c# from the network stream or the Output stream
C# SERVER
private void ReadData(){
if (networkStream.DataAvailable)
{
int size = GetBufferSize();
Thread.Sleep(340);
byte[] myReadBuffer = new byte[size];
int numberOfBytesRead = 0;
while (true)
{
numberOfBytesRead = networkStream.Read(myReadBuffer, 0, myReadBuffer.Length);
if (numberOfBytesRead >= myReadBuffer.Length)
{
break;
}
}
string str = Encoding.UTF8.GetString(myReadBuffer, 0, size);
dynamic Message = Newtonsoft.Json.JsonConvert.DeserializeObject<dynamic>(str);
// Android Message , JSON String
if (OnAndroidMessage != null)
{
OnAndroidMessage(Message);
}
}
}
private int GetBufferSize()
{
byte[] myReadBuffer = new byte[4];
int numberOfBytesRead = 0;
do
{
numberOfBytesRead = networkStream.Read(myReadBuffer, 0, myReadBuffer.Length);
} while (networkStream.DataAvailable && numberOfBytesRead < myReadBuffer.Length);
if (numberOfBytesRead > 0)
{
// reverse the byte array.
if (BitConverter.IsLittleEndian)
{
Array.Reverse(myReadBuffer);
}
return BitConverter.ToInt32(myReadBuffer, 0);
}
else
{
return 0;
}
}
Java Client // i tested this also without cutting the data to smaller paces ,half of the data received not all of them
mBufferOut = socket.getOutputStream();
private void sendMessage(final String message) {
if (mBufferOut != null && message != null) {
try {
byte[] data = message.getBytes("UTF-8");
Log.d("_TAG", "Sending: " + message);
Log.d("_TAG", "Message length: " + Integer.toString(data.length));
mBufferOut.write(toByteArray(data.length));
mBufferOut.flush();
List<byte[]> divideArray = divideArray(data, 10000);
for (byte[] dataChunk : divideArray) {
Log.e("_TAG","CHUNK SIZE > " + Integer.toString(dataChunk.length));
mBufferOut.write(dataChunk, 0, dataChunk.length);
mBufferOut.flush();
}
} catch (Exception e1) {
e1.printStackTrace();
}
}
}
private List<byte[]> divideArray(byte[] source, int chunksize) {
List<byte[]> result = new ArrayList<byte[]>();
int start = 0;
while (start < source.length) {
int end = Math.min(source.length, start + chunksize);
result.add(Arrays.copyOfRange(source, start, end));
start += chunksize;
}
return result;
}
Any ideas ?

Solution from this post NetworkStream is reading data that should not be there
static void ReadExact(Stream stream, byte[] buffer, int offset, int count)
{
int read;
while(count > 0 && (read = stream.Read(buffer, offset, count)) > 0) {
offset += read;
count -= read;
}
if(count != 0) throw new EndOfStreamException();
}
the problem is the Read it takes size and want to get that size you need to give it chunks and check each chunk
And also read does not restart from where it stopped until it reads the amount is set to read meaning if i set to read 10 then if it not find the 10 then it will read what it find as example it reads 6 ,it will return 6 and when to loop another time ti read the rest it dose not start from 6 it start from 0 and read until 4 so you overwrite your data ,and if it read 10 from the first try then it set the read to finish so it dose not start from 0 ,it needs to read the amount the has been set to it to re set the read to new buffer location.

Java SocketChannel Read Entire String

In my current project, I am trying to transmit a string from one computer to another, and after finding and learning from numerous examples I have managed to get a basic form of communication working.
The issue I am having is if one computer tries sending a message that is too long, it seems to get broken up into multiple parts (roughly 3700 characters), and my parsing method fails.
I am using a Selector to iterate through all of the channels. Here is the relevant code:
if(key.isReadable()) {
// Get the channel and read in the data
SocketChannel keyChannel = (SocketChannel)key.channel();
ByteBuffer buffer = buffers.get(keyChannel);
int length = 0;
try {
length = keyChannel.read(buffer);
} catch ( IOException ioe) {
closeChannel(keyChannel);
}
if(length > 0) {
buffer.flip();
// Gather the entire message before processing
while( buffer.remaining() > 0) {
byte[] data = new byte[buffer.remaining()];
buffer.get(data);
fireReceiveEvent(keyChannel, data);//Send the data for processing
}
buffer.compact();
} else if (length < 0) {
closeChannel(keyChannel);
}
}
How can I guarantee that the entire message (regardless of length) is read at once before passing it along?

After talking to numerous people that know more about this than I do. The issue turns out to be that with TCP it is impossible to know when an entire "message" has arrived because there is no such thing as a message since TCP works on a two-way byte stream. The solution is to create your own protocol and implements your own definition of "message".
For my project, every message either starts with a [ or { and ends with a ] or } depending on the starting character. I search through the received data, and if there is a complete message, I grab it and pass it along to the handler. Otherwise skip the channel, and wait for more to arrive.
Here is the final version of my code that handles the message receiving.
if(key.isReadable()) {
// Get the channel and read in the data
SocketChannel keyChannel = (SocketChannel)key.channel();
ByteBuffer buffer = buffers.get(keyChannel);
int length = 0;
try {
length = keyChannel.read(buffer);
} catch ( IOException ioe) {
key.cancel();
closeChannel(keyChannel);
}
if (length > 0) {
buffer.flip();
// Gather the entire message before processing
if (buffer.remaining() > 0) {
byte[] data = new byte[buffer.remaining()];
buffer.get(data);
buffer.rewind();
int index = 0;
int i = 0;
// Check for the beginning of a packet
//[ = 91
//] = 93
//{ = 123
//} = 125
if (data[0] == 91 || data[0] == 123) {
// The string we are looking for
byte targetByte = (byte) (data[0] + 2);
for (byte b : data) {
i += 1;
if (b == targetByte) {
index = i;
break;
}
}
if (index > 0) {
data = new byte[index];
buffer.get(data, 0, index);
fireReceiveEvent(keyChannel, data);
}
} else {
for (byte b : data) {
i += 1;
if (b == 91 || b == 123) {
index = i;
break;
}
}
if (index > 0) {
data = new byte[index];
buffer.get(data, 0, index); // Drain the data that we don't want
}
}
}
buffer.compact();
} else if (length < 0) {
key.cancel();
closeChannel(keyChannel);
}
}

Android Bluetooth Java, Constant bluetooth stream to fill up array and write to sdcard

I am writing an Android app that uses bluetooth to stream data (bytes) into an array in another class. The idea behind this is to record a set amount of data (i.e. buffer[60000]) and when it is full, write the buffer to a file and store it on the SD Card.
The issue i am having is everytime the call is made on the bluetooth class and it handles a method of HandleBluetooth.write(buffer, bytes); to send to the HandleBluetooth class my buffer index is getting reset each time that pass is made. The index for the buffer, and the buffer are both global variables which is confusing me as to why it is setting the index back to 0 every time HandleBluetooth.write(buffer, bytes); is called so i am never able to fill the buffer to write it to the file.
here is my code:
BluetoothSerialService.java
public void run() {
Log.i(TAG, "BEGIN mConnectedThread");
byte[] buffer = new byte[9999];
int bytes;
// Keep listening to the InputStream while connected
while (true) {
try {
// Read from the InputStream
bytes = mmInStream.read(buffer);
HandleBluetooth.write(buffer, bytes);
// Send the obtained bytes to the UI Activity
//mHandler.obtainMessage(BlueTerm.MESSAGE_READ, bytes, -1, buffer).sendToTarget();
String a = buffer.toString();
a = "";
} catch (IOException e) {
Log.e(TAG, "disconnected", e);
connectionLost();
break;
}
}
}
HandleBluetooth class:
public static void write(byte[] buffer, int length) {
byte[] readBuf = (byte[]) buffer;
data_length = length;
for (x = 0; x < data_length; x++) {
raw = UByte(readBuf[x]);
if (odd_even_flag == 0) {
buffer1[out_index] = raw;
// System.out.println("ch1: " + buffer1[out_index]);
odd_even_flag = 1;
} else {
buffer2[out_index] = raw;
// System.out.println("ch2: " + buffer2[out_index]);
odd_even_flag = 0;
}
if (x % 10000 == 0) {
Write2File(buffer1, buffer2);
}
out_index++;
if (x >= 60000) {
StopIncomingData();
break;
}
}
}
As you can see I have been trying to index the buffer using out_index which is a global class variable.
I feel like this is something dumb that i am over looking here. Please steer me in the right direction so i can understand why my index is being reset every time that method is called.

Streaming chunks of audio (mp3) using Java&JSP for real-time playback through servletOutputStream

I am trying to playback audio and keep it continuous and free from skips or blank spots. I have to first receive as bytes in chunks and convert this to mp3 to be streamed by the servletOutputStream. I only start playing once enough bytes have been collected by the consumer in an attempt to maintain a constant flow of audio. As you can see I have hard coded this buffer but would like it to work for any size of audio bytes. I was wondering if anyone had come across a similar problem and had any advice?
Thanks in advance. Any help would be greatly appreciated.
public class Consumer extends Thread {
private MonitorClass consBuf;
private InputStream mp3InputStream = null;
private OutputStream OutputStream = null;
public Consumer (MonitorClass buf, OutputStream servlet)
{
consBuf = buf;
OutputStream = servlet;
}
public void run()
{
byte[] data;
byte[] tempbuf;
int byteSize = 60720; //This should be dynamic
int byteIncrement = byteSize;
int dataPlayed = 0;
int start = 0;
int buffer = 0;
boolean delay = true;
AudioFormat generatedTTSAudioFormat = getGeneratedAudioFormat();
try
{
while(true)
{
try
{
data = consBuf.get(); //gets data from producer using a shared monitor class
if(data.length >= byteSize) //Buffer size hit, start playing
{
if(delay) //help with buffering
{
System.out.println("Pre-delay...");
consBuf.preDelay();
delay = false;
}
tempbuf = new byte[byteIncrement];
arraySwap(data, tempbuf, start, byteSize);
System.out.println("Section to play: " + start + ", " + byteSize);
mp3InputStream = FishUtils.convertToMP3( new ByteArrayInputStream(tempbuf), generatedTTSAudioFormat);
copyStream(mp3InputStream, OutputStream);
System.out.println("Data played: " + byteSize);
System.out.println("Data collected: " + consBuf.getDownloadedBytes() );
dataPlayed = byteSize;
start = byteSize;
byteSize += byteIncrement;
}
if( consBuf.getIsComplete() )
{
if (consBuf.checkAllPlayed(dataPlayed) > 0)
{
System.out.println("Producer finished, play remaining section...");
//mp3InputStream = convertToMP3(new ByteArrayInputStream(tempbuf), generatedTTSAudioFormat);
//copyStream(mp3InputStream, OutputStream);
}
System.out.println("Complete!");
break;
}
}
catch (Exception e)
{
System.out.println(e);
return;
}
}
}
finally
{
if (null != mp3InputStream)
{
try
{
mp3InputStream.skip(Long.MAX_VALUE);
}
catch (Exception e)
{}
}
closeStream(mp3InputStream);
closeStream(OutputStream);
}
}
}

Performance in reading and writing files, what the best?. Serialization X Java.nio

I have an object with 1 int and 4 doubles.
I compared the performance to write 5 million of these objects in a file using serialization and FileChannel object.
In the serialization used the following method to read and write the file.
public void print() throws IOException, ClassNotFoundException{
ObjectInputStream input = new ObjectInputStream(new FileInputStream(this.filePath) );
try {
while(true) {
this.sb = (Sbit) input.readObject();
//System.out.println(this.sb.toString());
}
}
catch ( EOFException eofException ) {
return;
}
catch (IOException ioException) {
System.exit( 1 );
}
finally {
if( input != null )
input.close();
}
}
public void build() throws IOException {
ObjectOutputStream output = new ObjectOutputStream( new FileOutputStream(this.filePath) );
try {
Random random = new Random();
for (int i = 0; i<5000000; i++) {
this.sb = new Sbit();
this.sb.setKey(i);
this.sb.setXMin( random.nextDouble() );
this.sb.setXMax( random.nextDouble() );
this.sb.setYMin( random.nextDouble() );
this.sb.setYMax( random.nextDouble() );
output.writeObject(this.sb);
}
}
catch (IOException ioException) {
System.exit( 1 );
}
finally {
try {
if( output != null)
output.close();
}
catch ( Exception exception ) {
exception.printStackTrace();
System.exit(1);
}
}
}
While using java.nio was:
public void print() throws IOException {
FileChannel file = new RandomAccessFile(this.filePath, "rw").getChannel();
ByteBuffer[] buffers = new ByteBuffer[5];
buffers[0] = ByteBuffer.allocate(4); // 4 bytes to int
buffers[1] = ByteBuffer.allocate(8); // 8 bytes to double
buffers[2] = ByteBuffer.allocate(8);
buffers[3] = ByteBuffer.allocate(8);
buffers[4] = ByteBuffer.allocate(8);
while (true) {
if(file.read(buffers[0]) == -1 ) // Read the int,
break; // if its EOF exit the loop
buffers[0].flip();
this.sb = new Sbit();
this.sb.setKey(buffers[0].getInt());
if(file.read(buffers[1]) == -1) { // Read the int primary value
assert false; // Should not get here!
break; // Exit loop on EOF
}
buffers[1].flip();
this.sb.setXMin( buffers[1].getDouble() );
if(file.read(buffers[2]) == -1) {
assert false;
break;
}
buffers[2].flip();
this.sb.setXMax( buffers[2].getDouble() );
if(file.read(buffers[3]) == -1) {
assert false;
break;
}
buffers[3].flip();
this.sb.setYMin( buffers[3].getDouble() );
if(file.read(buffers[4]) == -1) {
assert false;
break;
}
buffers[4].flip();
this.sb.setYMax( buffers[4].getDouble() );
for(int i = 0; i < 5; i++)
buffers[i].clear();
}
}
public void build() throws IOException {
FileChannel file = new RandomAccessFile(this.filePath, "rw").getChannel();
Random random = new Random();
for (int i = 0; i<5000000; i++) {
this.sb = new Sbit();
this.sb.setKey(i);
this.sb.setXMin( random.nextDouble() );
this.sb.setXMax( random.nextDouble() );
this.sb.setYMin( random.nextDouble() );
this.sb.setYMax( random.nextDouble() );
ByteBuffer[] buffers = new ByteBuffer[5];
buffers[0] = ByteBuffer.allocate(4); // 4 bytes to into
buffers[1] = ByteBuffer.allocate(8); // 8 bytes to double
buffers[2] = ByteBuffer.allocate(8);
buffers[3] = ByteBuffer.allocate(8);
buffers[4] = ByteBuffer.allocate(8);
buffers[0].putInt(this.sb.getKey()).flip();
buffers[1].putDouble(this.sb.getXMin()).flip();
buffers[2].putDouble(this.sb.getXMax()).flip();
buffers[3].putDouble(this.sb.getYMin()).flip();
buffers[4].putDouble(this.sb.getYMax()).flip();
try {
file.write(buffers);
}
catch (IOException e) {
e.printStackTrace(System.err);
System.exit(1);
}
for(int x = 0; x < 5; x++)
buffers[x].clear();
}
}
But I read a lot about on the java.nio and tried to use it precisely because it has better performance. But that's not what happened in my case.
To write the file were the following (java.nio):
file size: 175 MB
time in milliseconds: 57638
Using serialization:
file size: 200 MB
time in milliseconds: 34504
For the reading of this file, were as follows (java.nio):
time in milliseconds: 78172
Using serialization:
time in milliseconds: 35288
Am I doing something wrong in java.nio? I would like to write to the same binary files as done. There is another way to write file efficiently? actually serializing an object is the best way?
Thank you.

You are creating 25,000,000 ByteBuffer objects, with each ByteBuffer being at most 8 bytes. Thats very inefficient.
Create just one ByteBuffer by allocating it to 38 bytes outside the loop (before the for statement)
Inside the loop you can use the same ByteBuffer as follows:
buffer.clear();
buffer.putInt(this.sb.getKey());
buffer.putDouble(this.sb.getXMin());
buffer.putDouble(this.sb.getXMax());
buffer.putDouble(this.sb.getYMin());
buffer.putDouble(this.sb.getYMax());
buffer.flip();
try
{
file.write(buffer);
}
catch (IOException ex)
{
ex.printStackTrace();
//etc...
}
buffer.flip();
Try it out and let us know if you see any improvements.

Instead of using multiple ByteBuffers, declare a single byte buffer that is large enough to hold all of the data you want to put into it. Then put data into it just like you are now. When done, flip the buffer and write it out. When you are ready to read it back in, read the data from disk into the byte buffer, flip it, and then read the data out using getInt/getDouble.

I haven't tried to serialize stuff on my own, but have achieved good results with kryo. It is a lot faster than standard Java serialization.