I created a SocketChannel to a remote server to send and receive messages on Tomcat. To receive messages from a remote computer, I used a thread dedicated to task (only this thread will read from the socket, nothing else).
When some bytes are received at the SocketChannel (I keep polling the SocketChannel on non-blocking mode for new data), I first read 4 bytes to get the length of the next message, then allocate and read x bytes from the SocketChannel, which is then decoded and reconstructed into a message.
Below is my code for the receiving thread:
#Override
public void run() {
while (true) { //Don't exit thread
//Attempt to read the size of the incoming message
ByteBuffer buf = ByteBuffer.allocate(4);
int bytesread = 0;
try {
while (buf.remaining() > 0) {
bytesread = schannel.read(buf);
if (bytesread == -1) { //Socket was terminated
}
if (quitthread) break;
}
} catch (IOException ex) {
}
if (buf.remaining() == 0) {
//Read the header
byte[] header = buf.array();
int msgsize = (0xFF & (int)header[0]) + ((0xFF & (int)header[1]) << 8)
+ ((0xFF & (int)header[2]) << 16) + ((0xFF & (int)header[3]) << 24);
//Read the message coming from the pipeline
buf = ByteBuffer.allocate(msgsize);
try {
while (buf.remaining() > 0) {
bytesread = schannel.read(buf);
if (bytesread == -1) { //Socket was terminated
}
if (quitthread) break;
}
} catch (IOException ex) {
}
parent.recvMessage(buf.array());
}
if (quitthread) {
break;
}
}
}
The first bytes I received from the SocketChannel is fine, and I successfully decoded the message. However, the next time I read from the SocketChannel, the socket skipped ahead about 100 bytes, which caused the wrong bytes to be read and interpreted as length, causing everything to become corrupted.
What is wrong with the code? No other thread is reading from the SocketChannel.
Your parenthesis are off, the code is:
(0xFF & ((int)header[1] << 8))
which is always 0 (same with << 16 and << 24), my guess is you meant:
((0xFF & ((int)header[1])) << 8)
This would lead to reading not enough message bytes, also leading to a mismatch in synchronisation (as opposed to reading too many.)
Edit: now you fixed the above, I cannot see anything wrong. Could you tell us the relation between the length of the first message and the exact number of bytes that are eaten?
Based on the code shown, my only guess is that you edited some of the behaviour out of the sample shown which might influence the schannel, is the schannel referenced elsewhere?
If the line:
ByteBuffer buf = ByteBuffer.allocate(4);
would be outside of the while that would result in behaviour you describe, but in your sample code it isn't.
I presume when you say you're polling the socket in non-blocking mode you mean you're using a the "standard" Selector.select() approach?
When select returns and indicates that there's data available for reading from the socket you should only read the bytes that are available before re-entering the call to select(). If read() returns -1 it indicates that no more bytes are available for immediate reading in the buffer - It does not mean that the socket has been closed. Hence I suspect your of attempting to completely fill the buffer before returning is incorrect. Even if it does work your I/O thread will be constantly spinning whilst data arrives. In particular, it looks like you're simply ignoring a return value of -1.
Consider re-architecting your code to use a finite state machine approach. For example, I've implemented this in the past using a 3-state model: IDLE, READ_MESSAGE_LENGTH and READ_MESSAGE.
Related
I'm trying to work with JSSC.
I built my app according to this link:
https://code.google.com/p/java-simple-serial-connector/wiki/jSSC_examples
My event handler looks like:
static class SerialPortReader implements SerialPortEventListener {
public void serialEvent(SerialPortEvent event) {
if(event.isRXCHAR()){//If data is available
try {
byte buffer[] = serialPort.readBytes();
}
catch (SerialPortException ex) {
System.out.println(ex);
}
}
}
}
}
The problem is that I'm always not getting the incoming data in one piece. (I the message has a length of 100 bytes, Im getting 48 and 52 bytes in 2 separates calls)
- The other side send me messages in different lengths.
- In the ICD Im working with, there is a field which tell us the length of the message. (from byte #10 to byte #13)
- I cant read 14 bytes:
(serialPort.readBytes(14);,
parse the message length and read the rest of the message:
(serialPort.readBytes(messageLength-14);
But if I will do it, I will not have the message in once piece (I will have 2 separates byte[] and I need it in one piece (byte[]) without the work of copy function.
Is it possible ?
When working with Ethernet (SocketChannel) we can read data using ByteBuffer. But with JSSC we cant.
Is there a good alternative to JSSC ?
Thanks
You can't rely on any library to give you all the content you need at once because :
the library dont know how many data you need
the library will give you data as it comes and also depending on buffers, hardware, etc
You must develop your own business logic to handle your packets reception. It will of course depend on how your packets are defined : are they always the same length, are they separated with same ending character, etc.
Here is an example that should work with your system (note you should take this as a start, not a full solution, it doesn't include timeout for example) :
static class SerialPortReader implements SerialPortEventListener
{
private int m_nReceptionPosition = 0;
private boolean m_bReceptionActive = false;
private byte[] m_aReceptionBuffer = new byte[2048];
#Override
public void serialEvent(SerialPortEvent p_oEvent)
{
byte[] aReceiveBuffer = new byte[2048];
int nLength = 0;
int nByte = 0;
switch(p_oEvent.getEventType())
{
case SerialPortEvent.RXCHAR:
try
{
aReceiveBuffer = serialPort.readBytes();
for(nByte = 0;nByte < aReceiveBuffer.length;nByte++)
{
//System.out.print(String.format("%02X ",aReceiveBuffer[nByte]));
m_aReceptionBuffer[m_nReceptionPosition] = aReceiveBuffer[nByte];
// Buffer overflow protection
if(m_nReceptionPosition >= 2047)
{
// Reset for next packet
m_bReceptionActive = false;
m_nReceptionPosition = 0;
}
else if(m_bReceptionActive)
{
m_nReceptionPosition++;
// Receive at least the start of the packet including the length
if(m_nReceptionPosition >= 14)
{
nLength = (short)((short)m_aReceptionBuffer[10] & 0x000000FF);
nLength |= ((short)m_aReceptionBuffer[11] << 8) & 0x0000FF00;
nLength |= ((short)m_aReceptionBuffer[12] << 16) & 0x00FF0000;
nLength |= ((short)m_aReceptionBuffer[13] << 24) & 0xFF000000;
//nLength += ..; // Depending if the length in the packet include ALL bytes from the packet or only the content part
if(m_nReceptionPosition >= nLength)
{
// You received at least all the content
// Reset for next packet
m_bReceptionActive = false;
m_nReceptionPosition = 0;
}
}
}
// Start receiving only if this is a Start Of Header
else if(m_aReceptionBuffer[0] == '\0')
{
m_bReceptionActive = true;
m_nReceptionPosition = 1;
}
}
}
catch(Exception e)
{
e.printStackTrace();
}
break;
default:
break;
}
}
}
After writing data to serial port it need to be flushed. Check the timing and pay attention to the fact that read should occur only after other end has written. read size is just an indication to read system call and is not guaranteed. The data may have arrived and is buffered in serial port hardware buffer but may not have been transferred to operating system buffer hence not to application. Consider using scm library, it flushes data after each write http://www.embeddedunveiled.com/
Try this:
Write your data to the serial port (using serialPort.writeBytes()) and if you are expecting a response, use this:
byte[] getData() throws SerialPortException, IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] b;
try {
while ((b = serialPort.readBytes(1, 100)) != null) {
baos.write(b);
// System.out.println ("Wrote: " + b.length + " bytes");
}
// System.out.println("Returning: " + Arrays.toString(baos.toByteArray()));
} catch (SerialPortTimeoutException ex) {
; //don't want to catch it, it just means there is no more data to read
}
return baos.toByteArray();
}
Do what you want with the returned byte array; in my case I just display it for testing.
I found it works just fine if you read one byte at a time, using a 100ms timeout, and when it does time out, you've read all data in the buffer.
Source: trying to talk to an Epson serial printer using jssc and ESC/POS.
I have a server which reads from the SocketChannel like below:
private void readDataFromSocket(SocketChannel socketChannel) throws IOException {
BUFFER.clear();
int count = 0;
while ((count = socketChannel.read(BUFFER)) > 0) {
BUFFER.flip();
int limit = BUFFER.limit();
while (limit > 0) {
System.out.print((char) BUFFER.get());
limit--;
}
}
if (count < 0) {
System.out.println("closing the socket!!!");
socketChannel.close();
}
}
And below is the client where client writes to the SocketChannel:
private void write(String str, SocketChannel channel) throws IOException{
byte[] b = str.getBytes();
buffer.clear();
buffer.put(b);
buffer.flip();
while(buffer.hasRemaining()){
channel.write(buffer);
}
}
So my question:
when exactly in the server code the count value will be 0 ( while ((count = socketChannel.read(BUFFER)) > 0))?
is it possible that the count will be 0 if the server has read half of the message that client has sent i.e.
Suppose client wrote: stack overflow, is it possible that in the server count will be 0 after reading stack i.e. half of the message that the client has sent (think that the message can be of 1MB size)?
When using blocking mode, you will always get at least 1 byte. Note: you might only get 1 byte, it doesn't read "messages".
When using non-blocking mode, you will get 0 most of the time, in fact whenever there is not a packet waiting for you.
In TCP, data is sent in packets, not messages. This means if you send 1 MB, most likely it will be broken into packets of your MTU size e.g. ~1500 bytes. If you read this socket, you will most likely see blocks of this size or a multiple if multiple packets came in since the last read. You will never see part of a packet, unless you read less than the available data. e.g. if 1500 bytes is waiting and you read just 8 bytes, you get part of that packet.
I am using java comm library to try accomplish a simple read/write to a serial port. I am able to successfully write to the port, and catch the return input from the input stream, but when I read from the input stream I am only able to read 1 byte (when I know there should be 11 returned)
I can write to the port successfully using Putty and am receiving the correct return String there. I am pretty new to Java, buffers and serial i/o and think maybe there is some obvious syntax or understanding of how data is returned to the InputStream. Could someone help me? Thanks!
case SerialPortEvent.DATA_AVAILABLE:
System.out.println("Data available..");
byte[] readBuffer = new byte[11];
try {
System.out.println("We trying here.");
while (inputStream.available() > 0) {
int numBytes = inputStream.read(readBuffer, 1, 11);
System.out.println("Number of bytes read:" + numBytes);
}
System.out.println(new String(readBuffer));
} catch (IOException e) {System.out.println(e);}
break;
}
This code returns the following output:
Data available..
We trying here.
Number of bytes read:1
U
As the documentation states
Reads up to len bytes of data from the input stream into an array of bytes. An attempt is made to read as many as len bytes, but a smaller number may be read.
This behavior is perfectly legal. I would also expect that a SerialPortEvent.DATA_AVAILABLE does not guarantee that all data is available. It's potentially just 1 byte and you get that event 11 times.
Things you can try:
1) Keep reading until you have all your bytes. E.g. wrap your InputStream into a DataInputStream and use readFully, that's the simplest way around the behavior of the regular read method. This might fail if the InputStream does not provide any more bytes and signals end of stream.
DataInputStream din = new DataInputStream(in);
byte[] buffer = new byte[11];
din.readFully(buffer);
// either results in an exception or 11 bytes read
2) read them as they come and append them to some buffer. Once you have all of them take the context of the buffer as result.
private StringBuilder readBuffer = new StringBuilder();
public void handleDataAvailable(InputStream in) throws IOException {
int value;
// reading just one at a time
while ((value = in.read()) != -1) {
readBuffer.append((char) value);
}
}
Some notes:
inputStream.read(readBuffer, 1, 11)
Indices start at 0 and if you want to read 11 bytes into that buffer you have to specify
inputStream.read(readBuffer, 0, 11)
It would otherwise try to put the 11th byte at the 12th index which will not work.
I'm developing Server-Client application and I have a problem with waiting for input data on input stream.
I have thread dedicated to reading input data. Currently it uses while loop to hold until data is available. (N.B. protocol is as follow: send size of packet, say N, as int then send N bytes).
public void run(){
//some initialization
InputStream inStream = sock.getInputStream();
byte[] packetData;
//some more stuff
while(!interrupted){
while(inStream.available()==0);
packetData = new byte[inStream.read()];
while(inStream.available()<packetData.length);
inStream.read(packetData,0,packetData.length);
//send packet for procession in other thread
}
}
It works but blocking the thread by while loop is IMO a bad idea. I could use Thread.sleep(X) to prevent resources being continously consumed by the loop, but there surely must be a better way.
Also I can not rely on InputStream.read to block the thread as part of the data may be sent by the server with delays. I have tried but it always resulted in unexpected behaviour.
I'd appreciate any ideas :)
You can use DataInputStream.readFully()
DataInputStream in = new DataInputStream(sock.getInputStream());
//some more stuff
while(!interrupted) {
// readInt allows lengths of up to 2 GB instead of limited to 127 bytes.
byte[] packetData = new byte[in.readInt()];
in.readFully(packetData);
//send packet for procession in other thread
}
I prefer to use blocking NIO which supports re-usable buffers.
SocketChannel sc =
ByteBuffer bb = ByteBuffer.allocateDirect(1024 *1024); // off heap memory.
while(!Thread.currentThread.isInterrupted()) {
readLength(bb, 4);
int length = bb.getInt(0);
if (length > bb.capacity())
bb = ByteBuffer.allocateDirect(length);
readLength(bb, length);
bb.flip();
// process buffer.
}
static void readLength(ByteBuffer bb, int length) throws EOFException {
bb.clear();
bb.limit(length);
while(bb.remaining() > 0 && sc.read(bb) > 0);
if (bb.remaining() > 0) throw new EOFException();
}
As UmNyobe said, available() is meant to be used if you dont want to block as the default behaviour is blocking.
Just use the normal read to read whatever is available but only send packet for processing in other thread once you have packetData.length bytes in your buffer...
The problem I am having is that when I use an InputStream to read bytes, it blocks until the connection is finished. EG:
InputStream is = socket.getInputStream();
byte[] buffer = new byte[20000];
while (is.read(buffer) != -1) {
System.out.println("reading");
}
System.out.println("socket read");
"socket read" doesn't print out until the FYN packet is actually recieved, thus closing the connection. What is the proper way to receive all the bytes in without blocking and waiting for the connection to drop?
Take a look at java.nio which has non-blocking IO support.
Reading till you get -1 means that you want to read until EOS. If you don't want to read until EOS, don't loop till the -1: stop sooner. The question is 'when?'
If you want to read a complete 'message' and no more, you must send the message in such a way that the reader can find its end: for example, a type-length-value protocol, or more simply a size word before each message, or a self-describing protocol such as XML.
With traditional sockets the point is that usually you do want them to block: what you do when logically you don't want your program to block is you put your reading/writing code in another thread, so that the separate read/write thread blocks, but not your whole program.
Failing that, you can use the available() method to see if there is actually any input available before reading. But then you need to be careful not to sit in a loop burning CPU by constantly calling available().
Edit: if the problem is that you're happy to block until the bytes have arrived, but not until the connection has dropped (and that is what is happeningh), then you need to make the client at the other end call flush() on its output stream after it has sent the bytes.
Try this:
InputStream is = socket.getInputStream();
byte[] buffer = new byte[20000];
int bytesRead;
do {
System.out.println("reading");
bytesRead = is.read(buffer);
}
while (is.available() > 0 && bytesRead != -1);
System.out.println("socket read");
More info: https://docs.oracle.com/javase/1.5.0/docs/api/java/io/InputStream.html#available()
Example taken from exampledepot on java.nio
// Create a direct buffer to get bytes from socket.
// Direct buffers should be long-lived and be reused as much as possible.
ByteBuffer buf = ByteBuffer.allocateDirect(1024);
try {
// Clear the buffer and read bytes from socket
buf.clear();
int numBytesRead = socketChannel.read(buf);
if (numBytesRead == -1) {
// No more bytes can be read from the channel
socketChannel.close();
} else {
// To read the bytes, flip the buffer
buf.flip();
// Read the bytes from the buffer ...;
// see Getting Bytes from a ByteBuffer
}
} catch (IOException e) {
// Connection may have been closed
}
Be sure to understand buffer flipping because it causes a lot of headache. Basically, you have to reverse your buffer to read from it. If you are to reuse that buffer to have the socket to write in it, you have to flip it again. However clear() resets the buffer direction.
the code is probably not doing what you think it does.
read(buffer) returns the number of bytes it read, in other words: it is not guaranties to fill up your buffer anyway.
See DataInputStream.readFully() for code that fill up the entire array:
or you can use this functions (which are based on DataInputStream.readFully()) :
public final void readFully(InputStream in, byte b[]) throws IOException
{
readFully(in, b, 0, b.length);
}
public final void readFully(InputStream in, byte b[], int off, int len) throws IOException
{
if (len < 0) throw new IndexOutOfBoundsException();
int n = 0;
while (n < len)
{
int count = in.read(b, off + n, len - n);
if (count < 0) throw new EOFException();
n += count;
}
}
Your code would look like:
InputStream is = socket.getInputStream();
byte[] buffer = new byte[20000];
readFully(is, buffer);
System.out.println("socket read");