Before anything, allow me to show you my client code:
String command;
socket.setSoTimeout(5000);
while(true) {
try {
final byte[] targetArray = new byte[is.available()];
final int x = is.read(targetArray);
if (x == -1) {
System.out.println("end");
break;
}
if (x == 0) continue;
command = new String(targetArray);
handleCommand(command).start();
}catch (Exception e){
e.printStackTrace();
}
}
once connected to the actual socket, the client sends out some authentication data, but doesn't recieve any data, now, it waits to recieve data from the server, and when it does, it processes it fine etc, except, when I stop the server (literally shutting down the program), nothing happens, when in reality, im expecting it to send EOF (-1). Instead it just spams out 0 consistently.
According to available method's documentation (emphasis mine):
Returns an estimate of the number of bytes that can be read ...
and
It is never correct to use the return value of this method to allocate a buffer intended to hold all data in this stream ...
So according to the minimal code you posted, you shouldn't use available to allocate the buffer, because it may always return 0 which in turn makes the read operation always return 0. But according to your question you only see this behaviour when the sending program is closed which should mean that:
The available method correctly returns the buffered number of bytes in the stream, so this is why you receive the data. Even if you consume received data faster than the sending end sends (so some zeros may show up in available) you just continue and never handle that case, which makes some other loop in the future to capture the (eventually) received non-zero length data.
Once the sending end is closed, there will be a point after which it will always return 0 since it has no data. So you allocate zero length buffer, and the read method first checks that your buffer is zero length, rather than EOF.
So make sure the allocated buffer is of size at least 1.
Also, according to the Socket#getInputStream method's documentation:
If there are no bytes buffered on the socket, and the socket has not been closed using close, then available will return 0.
Related
public void createNewUser(String name, String passwort) {
try {
br = new BufferedReader(new FileReader("Data.txt"));
} catch (FileNotFoundException brCreateError) {
brCreateError.printStackTrace();
}
try {
br.mark(1);
System.out.println(br.readLine());
try {
if(br.readLine()==null) {
noUser=true;
}else {
noUser=false;
}
} catch (IOException e) {
e.printStackTrace();
}
br.reset();
} catch (IOException brMarkError) {
brMarkError.printStackTrace();
} ...
Why is the markedChar value changing to -2 after passing the if-statement?
Thx for every answer Nico.
public void mark(int readAheadLimit)
throws IOException
Marks the present position in the stream. Subsequent calls to reset()
will attempt to reposition the stream to this point.
...
Parameters:
readAheadLimit- Limit on the number of characters that may be read while still preserving the mark. An attempt to reset the stream
after reading characters up to this limit or beyond may fail. A limit
value larger than the size of the input buffer will cause a new buffer
to be allocated whose size is no smaller than limit. Therefore large
values should be used with care.
You set the readAheadLimit to 1 character then read an entire line. This invalidated the mark.
I had the same problem with that damn exception: Mark Invalid.
I came to this forum and I did not find something that would work so I had to figure out for myself what happens.
The BufferedReader creates a buffer (who would say it) from the moment I call the BufferedReader :: mark (int readAheadLimit) function, the size of the buffer IN CHARACTERS (NOT LINES) is given by readAheadLimit.
Once this limit is exceeded, it is not possible to go back with BufferedReader :: reset (), at the time of the mark because although the first data of the buffer could be recovered the later ones (those after the maximum of the buffer size and before the position current file) it would not be possible to retrieve them.
To avoid errors, the mark is invalidated by setting it to -2, and when the reset is called, the exception is produced: Mark Invalid. The issue is that it is NOT POSSIBLE with BufferedReader, to go back the pointer to read the file, it is simulated by storing the data in memory (the buffer).
So if you run into an Invalid Mark exception, you should make the buffer larger when calling the mark (int) method.
For this, we need to understand how the BufferedReader works...
BufferReader will read 8192 characters at one shot and store it in a character buffer - cb (default value can be changed).
readLine() - will try to fetch the data from the character buffer, if it needs more characters than what is available, it will do a fetch again and fill the next 8192 characters...
mark() - is used to make the current line, so that using reset() method we can go back to the previously marked line.
mark() method takes a parameter - readAheadLimit, which should be the maximum number of characters in that line.
As given in the Java DOCS - having a limit less than the line size may fail.
https://docs.oracle.com/javase/7/docs/api/java/io/BufferedReader.html#mark(int)
If we try to fill the character array again after the character buffer
array is exhausted and the number of pending characters is more than
the marked limit (readAheadLimit). It will mark the markedChar as
INVALIDATED. The next call to reset will lead to IOException - Mark
invalid
CONCLUSION:
Make sure that the readAheadLimit is the maximum number of characters that can be present in a line.
If the line is already filled in the character buffer, you will not get any error since the check is done while trying to fill the character buffer again.
That is the significance of may fail in Java Docs.
Right now, I'm trying to write a GUI based Java tic-tac-toe game that functions over a network connection. It essentially works at this point, however I have an intermittent error in which several chars sent over the network connection are lost during gameplay. One case looked like this, when println statements were added to message sends/reads:
Player 1:
Just sent ROW 14 COLUMN 11 GAMEOVER true
Player 2:
Just received ROW 14 COLUMN 11 GAMEOV
Im pretty sure the error is happening when I read over the network. The read takes place in its own thread, with a BufferedReader wrapped around the socket's InputStream, and looks like this:
try {
int input;
while((input = dataIn.read()) != -1 ){
char msgChar = (char)input;
String message = msgChar + "";
while(dataIn.ready()){
msgChar = (char)dataIn.read();
message+= msgChar;
}
System.out.println("Just received " + message);
this.processMessage(message);
}
this.sock.close();
}
My sendMessage method is pretty simple, (just a write over a DataOutputStream wrapped around the socket's outputstream) so I don't think the problem is happening there:
try {
dataOut.writeBytes(message);
System.out.println("Just sent " + message);
}
Any thoughts would be highly appreciated. Thanks!
As it turns out, the ready() method guaruntees only that the next read WON'T block. Consequently, !ready() does not guaruntee that the next read WILL block. Just that it could.
I believe that the problem here had to do with the TCP stack itself. Being stream-oriented, when bytes were written to the socket, TCP makes no guarantees as to the order or grouping of the bytes it sends. I suspect that the TCP stack was breaking up the sent string in a way that made sense to it, and that in the process, the ready() method must detect some sort of underlying break in the stream, and return false, in spite of the fact that more information is available.
I refactored the code to add a newline character to every message send, then simply performed a readLine() instead. This allowed my network protocol to be dependent on the newline character as a message delimiter, rather than the ready() method. I'm happy to say this fixed the problem.
Thanks for all your input!
Try flushing the OutputStream on the sender side. The last bytes might remain in some intenal buffers.
It is really important what types of streamed objects you use to operate with data. It seems to me that this troubleshooting is created by the fact that you use DataOutputStream for sending info, but something else for receiving. Try to send and receive info by DataOutputStream and DataInputStream respectively.
Matter fact, if you send something by calling dataOut.writeBoolean(b)
but trying to receive this thing by calling dataIn.readString(), you will eventually get nothing. DataInputStream and DataOutputStream are type-sensitive. Try to refactor your code keeping it in mind.
Moreover, some input streams return on invocation of read() a single byte. Here you try to convert this one single byte into char, while in java char by default consists of two bytes.
msgChar = (char)dataIn.read();
Check whether it is a reason of data loss.
I am trying to transfer a text file to another server using TCP and it is behaving differently than expected. The code sending the data is:
System.out.println("sending file name...");
String outputFileNameWithDelimiter = outputFileName + "\r\n"; //These 4 lines send the fileName with the delimiter
byte[] fileNameData = outputFileNameWithDelimiter.getBytes("US-ASCII");
outToCompression.write(fileNameData, 0, fileNameData.length);
outToCompression.flush();
System.out.println("sending content...");
System.out.println(new String(buffer, dataBegin, dataEnd-dataBegin));
outToCompression.write(buffer, dataBegin, dataEnd-dataBegin); //send the content
outToCompression.flush();
System.out.println("sending magic String...");
byte[] magicStringData = "--------MagicStringCSE283Miami".getBytes("US-ASCII"); //sends the magic string to tell Compression server the data being sent is done
outToCompression.write(magicStringData, 0, magicStringData.length);
outToCompression.flush();
Because this is TCP and you can't send discrete packets like in UDP, I expected all of the data to be in the input stream and I could just use delimiters to separate the file name, content, and ending string and then each in.read() would just give me the next subsequent amount of data.
Instead this is the data I am getting on each read:
On the first in.read() byteBuffer appears to only have "fileName\r\n".
On the second in.read() byteBuffer still has the same information.
On the third in.read() byteBuffer now holds the content I sent.
On the fourth in.read() byteBuffer holds the content I sent minus a few letters.
On the fifth in.read() I get the magicString + part of the message.
I am flushing on every send from the Webserver, but input streams don't seem to implement flushable.
Can anyone explain why this is happening?
EDIT:
This is how I am reading things in. Basically this in a loop then writing to a file.
in.read(byteBuffer, 0, BUFSIZE);
If your expectation is that read will fill the buffer, or receive exactly what was sent by a single write() by the peer, it is your expectation that is at fault here, not read(). it isn't specified to transfer more than one byte at a time, and there is no guarantee about preserving write boundaries.
It is quite impossible to write correct code without storing the result of read() into a variable.
When you read from an InputStream, you're giving it a byte array to write into (and optionally an offset and a maximum amount to read). InputStream makes no guarantees that the array will be filled with fresh data. The return value is the number of bytes that was actually read into it.
What's happening in your example is this:
The first TCP packet comes in with "fileName\r\n", gets written into your buffer, everything fine so far.
You call read() again, but the next packet hasn't arrived yet. read() will have returned 0, because it didn't want to block until data arrived. So the buffer still contains "fileName\r\n". Edit: as pointed out, read() always blocks until it reads at least one byte. Don't really know why the buffer didn't change then.
On the third read, the content has arrived.
The first bit of the content gets overwritten with the second part of the message, the last bit still contains part of the old message (I think that's what you meant).
etc., you get the idea
You need to check the return value, wait for the data to arrive, and only use as much of the buffer as was written by the last read().
This is a bit of an obscure problem that only seems to happen when I'm on certain computers.
I was having this issue today on our school's XP computers and I can't seem to replicate this on my home computer (W7).
Anyway, reading/writing to sockets in Java tends to be problematic whenever I use this code (where: int avail, InputStream socket, byte[] buffer, String output):
while( (avail = input.available()) > 0 )
{
read = input.read( buffer );
output += new String( buffer, 0, read );
}
It seems to make sense (reading all the data until no data is available to a temporary buffer, then to a string), but on our school computers (testing it using IE7), the whole thing somehow pauses. I'm thinking input.available() is causing it to somehow block because the thread just keeps running without ever reaching an endpoint... effectively just pausing somewhere.
OH, I forgot to mention: whenever I run this in debug mode and perform each line step-by-step, it works completely like it should... which just confuses me even more.
When I got home to replicate this issue, it works just fine (just using Firefox and IE8). I have no idea what would be a better alternative to this.
PS:
If the buffer is large enough and I just use:
read = input.read( buffer );
output += new String( buffer, 0, read );
It works just fine, but there's always a worry that the data sent will exceed the buffer size.
You're thinking about available() the wrong way. That method tells you approximately how many bytes can be read right now, without blocking. The commonly accepted idiom for what you're trying to do is
int length;
while ((length = in.read(buffer)) != -1) {
output += new String(buffer, 0, length);
}
or something along those lines (not compiled/tested).
Update: I think you misunderstand the concept of "end of the stream". "End of the stream" doesn't mean that all the data you want to read has been read. It means that there isn't, and won't ever be, anything else to read. For instance, it might mean that you were reading a file and have come to the end of it, or it might mean you were reading from an in-memory byte array and came to the end of that. Those are "end of streams".
In your question, you indicated, or at least implied, that you're reading from a Socket. Are you aware that you'll never get to the end of that stream until the associated Socket or the remote end of the connection is closed? Just because you received a bit of data from it doesn't make it the end of the stream.
Why not use a buffered reader? Something like:
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
String output = "";
try {
String readLine = null;
while ((readLine = reader.readLine()) != null) {
output += readLine + "\n";
}
} catch (IOException e) {
System.err.println("Error: " + e);
}
System.out.println("Read from Socket:" + output);
Your code is invalid. This is a misuse of available(). All it does is tell you how many bytes may be available for reading without blocking. It cannot be used to indicate how many bytes will ever be sent by the peer, and it has no necessary relationship with peer messages. There are no messages in TCP, only a byte stream. If you want to read to EOS, just remove the available() test and read until it returns -1. If you want to read a message, the peer will have to delimit it for you somehow, e.g. by an out-of-band terminator, a length word prefix, or a self-describing protocol such as Object Serialization or XML.
It 'works' in debug mode because you are radically changing the timing with breakpoints. This is further proof that what you are doing is incorrect.
Oh how I wish TCP was packet-based like UDP is! [see comments] But alas, that's not the case, so I'm trying to implement my own packet layer. Here's the chain of events so far (ignoring writing packets)
Oh, and my Packets are very simply structured: two unsigned bytes for length, and then byte[length] data. (I can't imagine if they were any more complex, I'd be up to my ears in if statements!)
Server is in an infinite loop, accepting connections and adding them to a list of Connections.
PacketGatherer (another thread) uses a Selector to figure out which Connection.SocketChannels are ready for reading.
It loops over the results and tells each Connection to read().
Each Connection has a partial IncomingPacket and a list of Packets which have been fully read and are waiting to be processed.
On read():
Tell the partial IncomingPacket to read more data. (IncomingPacket.readData below)
If it's done reading (IncomingPacket.complete()), make a Packet from it and stick the Packet into the list waiting to be processed and then replace it with a new IncomingPacket.
There are a couple problems with this. First, only one packet is being read at a time. If the IncomingPacket needs only one more byte, then only one byte is read this pass. This can of course be fixed with a loop but it starts to get sorta complicated and I wonder if there is a better overall way.
Second, the logic in IncomingPacket is a little bit crazy, to be able to read the two bytes for the length and then read the actual data. Here is the code, boiled down for quick & easy reading:
int readBytes; // number of total bytes read so far
byte length1, length2; // each byte in an unsigned short int (see getLength())
public int getLength() { // will be inaccurate if readBytes < 2
return (int)(length1 << 8 | length2);
}
public void readData(SocketChannel c) {
if (readBytes < 2) { // we don't yet know the length of the actual data
ByteBuffer lengthBuffer = ByteBuffer.allocate(2 - readBytes);
numBytesRead = c.read(lengthBuffer);
if(readBytes == 0) {
if(numBytesRead >= 1)
length1 = lengthBuffer.get();
if(numBytesRead == 2)
length2 = lengthBuffer.get();
} else if(readBytes == 1) {
if(numBytesRead == 1)
length2 = lengthBuffer.get();
}
readBytes += numBytesRead;
}
if(readBytes >= 2) { // then we know we have the entire length variable
// lazily-instantiate data buffers based on getLength()
// read into data buffers, increment readBytes
// (does not read more than the amount of this packet, so it does not
// need to handle overflow into the next packet's data)
}
}
public boolean complete() {
return (readBytes > 2 && readBytes == getLength()+2);
}
Basically I need feedback on my code and overall process. Please suggest any improvements. Even overhauling my entire system would be okay, if you have suggestions for how better to implement the whole thing. Book recommendations are welcome too; I love books. I just get the feeling that something isn't quite right.
Here's the general solution I came up with thanks to Juliano's answer: (feel free to comment if you have any questions)
public void fillWriteBuffer() {
while(!writePackets.isEmpty() && writeBuf.remaining() >= writePackets.peek().size()) {
Packet p = writePackets.poll();
assert p != null;
p.writeTo(writeBuf);
}
}
public void fillReadPackets() {
do {
if(readBuf.position() < 1+2) {
// haven't yet received the length
break;
}
short packetLength = readBuf.getShort(1);
if(readBuf.limit() >= 1+2 + packetLength) {
// we have a complete packet!
readBuf.flip();
byte packetType = readBuf.get();
packetLength = readBuf.getShort();
byte[] packetData = new byte[packetLength];
readBuf.get(packetData);
Packet p = new Packet(packetType, packetData);
readPackets.add(p);
readBuf.compact();
} else {
// not a complete packet
break;
}
} while(true);
}
Probably this is not the answer you are looking for, but someone would have to say it: You are probably overengineering the solution for a very simple problem.
You do not have packets before they arrive completely, not even IncomingPackets. You have just a stream of bytes without defined meaning. The usual, the simple solution is to keep the incoming data in a buffer (it can be a simple byte[] array, but a proper elastic and circular buffer is recommended if performance is an issue). After each read, you check the contents of the buffer to see if you can extract an entire packet from there. If you can, you construct your Packet, discard the correct number of bytes from the beginning of the buffer and repeat. If or when you cannot extract an entire packet, you keep those incoming bytes there until the next time you read something from the socket successfully.
While you are at it, if you are doing datagram-based communication over a stream channel, I would recommend you to include a magic number at the beginning of each "packet" so that you can test that both ends of the connection are still synchronized. They may get out of sync if for some reason (a bug) one of them reads or writes the wrong number of bytes to/from the stream.
Can't you just read whatever number of bytes that are ready to be read, and feed all incoming bytes into a packet parsing state machine? That would mean treating the incoming (TCP) data stream like any other incoming data stream (via serial line, or USB, a pipe, or whatever...)
So you would have some Selector determining from which connection(s) there are incoming bytes to be read, and how many. Then you would (for each connection) read the available bytes, and then feed those bytes into a (connection specific) state machine instance (the reading and feeding could be done from the same class, though). This packet parsing state machine class would then spit out finished packets from time to time, and hand those over to whoever will handle those complete and parsed packets.
For an example packet format like
2 magic header bytes to mark the start
2 bytes of payload size (n)
n bytes of payload data
2 bytes of checksum
the state machine would have states like (try an enum, Java has those now, I gather)
wait_for_magic_byte_0,
wait_for_magic_byte_1,
wait_for_length_byte_0,
wait_for_length_byte_1,
wait_for_payload_byte (with a payload_offset variable counting),
wait_for_chksum_byte_0,
wait_for_chksum_byte_1
and on each incoming byte you can switch the state accordingly. If the incoming byte does not properly advance the state machine, discard the byte by resetting the state machine to wait_for_magic_byte_0.
Ignoring client disconnects and server shutdown for now, here's more or less traditional structure of a socket server:
Selector, handles sockets:
polls open sockets
if it's the server socket, create new Connection object
for each active client socket find the Connection, call it with event (read or write)
Connection (one per socket), handles I/O on one socket:
Communicates to Protocol via two queues, input and output
keeps two buffers, one for reading, one for writing, and respective offsets
on read event: read all available input bytes, look for message boundaries, put whole messages onto Protocol input queue, call Protocol
on write event: write the buffer, or if it's empty, take message form output queue into buffer, start writing it
Protocol (one per connection), handles application protocol exchange on one connection:
take message from input queue, parse application portion of the message
do the server work (here's where the state machine is - some messages are appropriate in one state, while not in the other), generate response message, put it onto output queue
That's it. Everything could be in a single thread. The key here is separation of responsibilities.
Hope this helps.
I think you're approaching the issue from a slightly wrong direction. Instead of thinking of packets, think of a data structure. That's what you're sending. Effectively, yes, it's an application layer packet, but just think of it as a data object. Then, at the lowest level, write a routine which will read off the wire, and output data objects. That will give you the abstraction layer I think you're looking for.