I want to create a TCP server that accept a certain custom protocol from IoT devices.
In netty, it gives a buffered bytes, and you use read method to get the some (4, 8, 16, n) bytes. Then the operation will return that bytes and moving the buffer index.
If the current buffered byte length is less than requested, then it will wait for the next stream, append it, and the operation begin again.
In asyncio(python) the read(n) have similar functionality.
In vert.x is there a similar method to read buffered bytes like that?? I know that you get Buffer object. But the buffer index will not automatically move it seems? And does it wait for the next bytes??
You need to use a record parser:
The record parser allows you to easily parse protocols which are
delimited by a sequence of bytes, or fixed size records.
To create a record parser for fixed-size records:
RecordParser.newFixed(4, h -> {
System.out.println(h.toString());
});
Record parsers can also be create to consume a ReadStream, like a TCP connection input.
Related
If I have two connections to a server which require multiple reads on a channel to complete a packet, how will I know which read goes with which packet?
For example two packets which are received as four interleaved buffers:
PacketA buffer part1
PacketB buffer part1
PacketA buffer part2
PacketB buffer part2
The first part MAY have a header but the second part could arrive as being split anywhere along the packet.
When receiving a partial packet, how do I know which buffer goes where?
I'm thinking about building a map to associate each channel object with its respective output buffer which will hold the reassembled packet. Is this the way it's supposed to be done?
Indeed, typical practice is to have separate buffers for each channel. You don't necessarily need a map. You could attach extra information to each SelectionKey. Every key can hold an object of your choice that your code can retrieve later. This is a convenient place to store a buffer, or a set of buffers.
For anyone interested, the answer to this questions is no, the socket wont read the buffer out of order.
Is it possible for the AsynchronousSocketChannel to read bytes out of order? Im strugling to debug where my issue starts, my protocol serializes objects up to 32k and writes them to the socket like this:
AsynchronousSocketChannel socket; ...
// serialize packet
ByteBuffer base; // serialized buffer (unknown size, size growns as needed with limit of 32k)
for (int j1 = 0; j1 < 6 && base.remaining() > 0; j1++) { // limit to 6 tries
socket.write(base).get(15, TimeUnit.SECONDS);
if (base.remaining() > 0) {
// aparently, if the write operation could write everything at once, we wouldnt have an issue
}
}
This write operation is not concurrent it is synchronized with locks. I use the standard read operation like this:
AsynchronousSocketChannel socket; ...
Future<Integer> reading = socket.read(getReadBuffer()); // the read buffer is 8k big
// consume the buffer also not concurrently
I can write up to 1000 packets per second with up to 1000 bytes each without issues, but eventually one or other client will break. If the packet is bigger, the frequency that it can handle without breaking will be lower, packets with 40.000 bytes will break if I write around 8 per second.
Example: I write 5 bytes (1,2,3,4,5), the buffer is big enough the write everything at once but the operation decides to stop with remaining bytes in the buffer (this should be the normal TCP behavior), so lets say the operation wrote 1,2,3, stopped and wrote the remain 4,5 (while buf.remain > 0 { write }), while reading, is most likely that I will read 4,5 first and 1,2,3 later, this should not happen.
While on localhost everything works fine, but just outside the same machine (still same network/routers) it wont work.
I do not flip the buffers to write/read. I can ensure its not an issue with the serialization and both the server and client are single-threaded. I'm forgetting to do something? Any suggestions on how to fix this?
It isn't clear why you're using asynchronous I/O if all you really want is synchronous I/O, which is what you are getting from that code. You'd be better off with an ordinary SocketChannel.
I do not flip the buffers to write/read.
You must. You must flip() before write(), and compact() afterwards, where 'afterwards' in this case means the same as above.
I am currently using java.net.Socket to send messages from the client and reading messages from the server. All my messages are fairly short so far, and I have never had any problems.
One of my friends noticed that I was not handling message fragmentation, where the data could come in pieces, and has advised that I should create a buffer to handle this. I insisted that TCP handles this for me, but I'm not 100% sure.
Who is right?
Also, I plan on creating a client in C as well in the future. Do Berkeley sockets handle message fragmentation?
Details: Currently, in Java, the server creates a socket and reads the first byte from the message with InputStream#read(). That first byte determines the length of the entire message, and creates a byte array of the appropriate length, and calls InputStream#read(byte[]) once and assumes that the entire message has been read.
If you are talking about WebSockets,you may be mixing different concepts.
One thing is TCP/IP message fragmentation.
Other thing is how buffering works. You read buffers of data, and you need a framing protocol that tells you when you have a complete "message" (or frame). Basically you:
Read buffer.
Has complete header? No-> Goto 1, Yes-> continue
Read until having all the bytes that the head indicates as message
length.
Has complete message? No-> Goto 3, Yes -> continue
Yield message.
Goto 1.
Other different thing is WebSocket message fragmentation. WebSocket has already a framing protocol and messages can be split in different data frames, and control frames can be interleaved with data frames: https://developer.mozilla.org/en-US/docs/WebSockets/Writing_WebSocket_servers#Message_Fragmentation
If you are writing a WebSocket client or server you have to be ready for this situation.
Expanding on what nos said, TCP will break up large messages into smaller chunks, if the message is large enough. Often, it isn't. Often, the data you write is already split into parts (by you), into meaningful chunks like discrete messages.
The stuff about the reads/writes taking different amounts of calls comes from how the data is written, how it travels over the wire, and how you read it.
If you write 2 bytes 100 times, and then 20 seconds later go to read, it will say there is 200 bytes to be read, which you can read all at once if you want. If you pass a massive 2mb buffer to be written (I dont even know if thats possible), it would take longer to write out, giving more of a chance to the reading program to get different read calls.
Details: Currently, in Java, the server creates a socket and reads the first byte from the message with InputStream#read(). That first byte determines the length of the entire message, and creates a byte array of the appropriate length, and calls InputStream#read(byte[]) once and assumes that the entire message has been read.
That won't work. Have a look at the contract for InputStream.read(byte[]). It isn't obliged to transfer more than one byte. The correct technique is to read the length byte and then use DataInputStream.readFully(), which has the obligation to fill the buffer.
I am trying to read some data from a network socket using the following code -
Socket s = new Socket(address, 502);
response = new byte[1024];
InputStream is = s.getInputStream();
int count = is.read(response, 0, 100);
The amount of data isn't large. It is 16 bytes in total. However the read() statement does not read all the data in one go. It reads only 8 bytes of data into my buffer.
I have to make multiple calls to read() like this in order to read the data -
Socket s = new Socket(address, 502);
response = new byte[1024];
InputStream is = s.getInputStream();
int count = is.read(response, 0, 100);
count += is.read(response, count, 100-count);
Why is this happening? Why does read() not read the entire stream in one go?
Please note that the data is not arriving gradually. If I wait for 2 seconds before reading the data by making a call to Thread.sleep(2000) the behavior remains the same.
Why does read() not read the entire stream in one go?
Because it isn't specified to do so. See the Javadoc. It blocks until at least one byte is available, then returns some number between 1 and the supplied length, inclusive.
That in turn is because the data doesn't necessarily arrive all in one go. You have no control over how TCP sends and receives data. You are obliged to just treat it as a byte stream.
I understand that it blocks until data arrives. "That in turn is because the data doesn't necessarily arrive all in one go." Why not is my question.
The data doesn't necessarily all arrive in one go because the network typically breaks it up into packets. IP is a packet switching protocol.
Does TCP transmit it blocks of 8 bytes?
Possibly, but probably not. The packet size depends on the network / networks that the data has traversed, but a typical internet packet size is around 1500 bytes.
If you are getting 8 bytes at a time, either your data is either coming through a network with an unusually small packet size, or (more likely) the sender is sending the data 8 bytes at a time. The second explanation more or less jives with what your other comments say.
And since I explicitly specify 100, a number much larger than the data in buffer shouldn't it attempt to read up till atleast 100 bytes?
Well no. It is not specified to work that way, and it doesn't work that way. You need to write your code according to what the spec says.
It is possible that this has something to do with the way the device is being "polled". But without looking at the specs for the device (or even knowing what it is exactly) this is only a guess.
Maybe the data is arriving gradually not because of your reading but because of the sender.
The sender should use a BufferedOutputStream (in the middle) to make big chunks before sending (and use flush only when it's needed).
I am currently sending large amounts of data over a Java socket, and I am using the Apache Commons IOUtils library to copy using the copyLarge method to send/receive the data. The problem is that the copyLarge reads until the the input stream returns -1. I have copied a snippet below
while (-1 != (n = input.read(buffer))) {
output.write(buffer, 0, n);
count += n;
}
This method will block until the socket is closed, but the problem is I want to reuse the socket to additional large data. The alternatives I see are to either reopen a new socket for each data being transferred, or to write my own read method that looks for an end of stream token (i.e. a newline character).
It has been a while since I've written low level socket code like this, but am I missing something here? Or is there an easier way to do this?
Do you know how much data you have before sending it over? If so, I'd basically length-prefix the message.
That's much easier to handle cleanly than using an end-of-stream token and having to worry about escaping, over-reading etc.
But yes, you'll need to do something, because TCP/IP is a stream-based protocol. Unless you have some indicator for the end of data somehow, you never know whether there might be some more to come Real Soon Now.