We Keep Coding

How much data was transferred?

When transmitting a file through socket in blocking mode
bytesTransferred = fileIChannel.transferTo(0, fileIChannel.size(), socketChannel);
// or using a buffer
ByteBuffer byteBuffer = ByteBuffer.allocateDirect(1024*8);
while (fileIChannel.read(byteBuffer) != -1) {
byteBuffer.flip();
bytesTransferred += socketChannel.write(byteBuffer);
byteBuffer.clear();
In the event of a connection failure, I need to keep the number of bytes transferred.
I can do this by waiting for a response every from the server when it receives a particular number of bytes. Or, when the connection is restored, I send a request for the number of bytes received.
Which of the options will be more correct? How is this problem usually resolved?
And the second question. Is data integrity guaranteed when sending large data through a socket?

WIth this code you cannot tell. If you want to know that the peer application has received and processed sent data, the peer application has to tell you. TCP does buffering at both ends, so the API alone cannot tell you.
NB Your copy loop is wrong. It should be:
while ((fileIChannel.read(byteBuffer) != -1 && byteBuffer.position() > 0)
{
byteBuffer.flip();
bytesTransferred += socketChannel.write(byteBuffer);
byteBuffer.compact();
}
and there should also be an error test on the write. At present you are assuming that everything got written to the SocketChannel on every write, which isn't guaranteed in non-blocking mode.
The code using transferTo() is also wrong, as transferTo() isn't guaranteed to perform the entire transfer: that's why it returns a count. You have to loop.

Java networking input stream for client

I'm very new to networking in java and wanted to create a network chat program, I have found a few tutorials online and kind of drifted from that. I have the server of my program working and the only thing that is interfering is when I try to read the chat messages that the server sends over. The server does send the bytes of data over since the print message does work. So the problem is that the while loop never ends, what can this be a problem of?
public String[] updateChatDialog() throws IOException{
String returnString = "";
int accessed = -1;
while((accessed = in.read()) > 0){
System.out.println((char)accessed);
returnString = returnString + (char)accessed;
}
System.out.println(returnString);
return stringToTable(returnString);
}
Any tips on java networking would be helpful!
I do reset the BufferedInputStream every time the chats are rendered into a string, AKA the method above with the return in it.

The 'problem' is that your loop reads until end of stream, and the peer isn't closing the connection so EOS never arrives.
If you want to read messages you have to define them yourself. The easiest thing for you to do in this application is write and read lines, with e.g. PrintWriter.println() and BufferedReader.readLine().

difference between Java TCP Sockets and C TCP Sockets while trying to connect to JDBC

My problem is that C sockets look to act differently than Java sockets. I have a C proxy and I tested it between a workload generator (oltp benchmark client written in Java) and the JDBC connector of the Postgres DB.
This works great and forwards data from one to other, as it should. We need to make this proxy work in Java, so I used plain ServerSocket and Socket classes from java.net and I cannot make it work. The Postgres returns an authentication error message, assuming that the client did not send the correct password.
Here is how the authentication at the JDBC protocol works:
-client sends a requests to connect to a database specifying the database name and the username
-server responds back with a one time challenge message (13 byte message with random content)
-client concatenates this message with the user password and performs a md5 hash
-server compares the hash got from the client with the hash he computes
[This procedure is performed in order to avoid replay attacks (if client would send only the md5 hash of its password then an attacker could replay this message, pretending he is the client)]
So I inspected the packets with tcpdump and they look correct! The size is exactly as it should, so maybe the content is corrupted (??)
Sometimes though the DB server responds ok for the authentication (depending on the value of the challenge message)!! And then the oltp client sends a couple of queries, but it crashes in a while…
I guess that maybe it has to do with the encoding, so I tried with the encoding that C uses (US-ANSII), but still the same.
I send the data using fixed size character or byte arrays both in C and in Java!
I really don't have any more ideas, as I tried so many cases...
What is your guess of what would be the problem?
Here is a representative code that may help you have a more clear view:
byte [] msgBuf;
char [] msgBufChars;
while(fromInputReader.ready()){
msgBuf = new byte[1024];
msgBufChars = new char[1024];
// read data from one party
int read = fromInputReader.read(msgBufChars, 0, 1024);
System.out.println("Read returned : " + read);
for(int i=0; i<1024; i++)
msgBuf[i] = (byte) msgBufChars[i];
String messageRead = new String(msgBufChars);
String messageToWrite = new String(msgBuf);
System.out.println("message read : "+messageRead);
System.out.println("message to write : "+new String(messageToWrite));
// immediatelly write data to other party (write the amount of data we read (read value) )
// there is no write method that takes a char [] as a parameter, so pass a byte []
toDataOutputStream.write(msgBuf, 0, read);
toDataOutputStream.flush();
}
There are a couple of message exchanges in the beginning and then Postgres responds with an authentication failure message.
Thanks for your time!

What is your guess of what would be the problem?
It is nothing to do with C versus Java sockets. It is everything to do with bad Java code.
I can see some problems:
You are using a Reader in what should be a binary stream. This is going to result in the data being converted from bytes (from the JDBC client) to characters and then back to bytes. Depending on the character set used by the reader, this is likely to be destructive.
You should use plain, unadorned1 input streams for both reading and writing, and you should read / write to / from a preallocated byte[].
This is terrible:
for(int i=0; i<1024; i++)
msgBuf[i] = (byte) msgBufChars[i];
If the characters you read are not in the range 0 ... 255 you are mangling them when you stuff them into msgBuf.
You are assuming that you actually got 1024 characters.
You are using the ready() method to decide when to stop reading stuff. This is almost certainly wrong. Read the javadoc for that method (and think about it) and you should understand why it is wrong. (Hint: what happens if the proxy can read faster than the client can deliver?)
You should use a while(true), and then break out of the loop if read tells you it has reached the end of stream; i.e. if it returns -1 ...
1 - Just use the stream objects that the Socket API provides. DataXxxStream is unnecessary because the read and write methods are simply call-throughs. I wouldn't even use BufferedXxxStream wrappers in this case, because you are already doing your own buffering using the byte array.
Here's how I'd write that code:
byte [] buffer = new byte[1024]; // or bigger
while(true) {
int nosRead = inputStream.read(buffer);
if (nosRead < 0) {
break;
}
// Note that this is a bit dodgy, given that the data you are converting is
// binary. However, if the purpose is to see what embedded character data
// looks like, and if the proxy's charset matches the text charset used by
// the client-side JDBC driver for encoding data, this should achieve that.
System.out.println("Read returned : " + nosRead);
System.out.println("message read : " + new String(buffer, 0, nosRead));
outputStream.write(buffer, 0, nosRead);
outputStream.flush();
}

C sockets look to act differently than Java sockets.
Impossible. Java sockets are just a very thin layer over C sockets. You're on the wrong track with this line of thinking.
byte [] msgBuf;
char [] msgBufChars;
Why are you reading chars when you want to write bytes? Don't use Readers unless you know that the input is text.
And don't call ready(). There are very few correct uses, and this isn't one of them. Just block.

Problems writing a protocol on top of sockets in Java

I'm writing a protocol on top of sockets, so I've decided to implement headers then send the information. So there is one thread per connection on the server which sits there reading in headers, then delegates off to methods to read in the rest of the information when it arrives.
So essentially it looks like this:
while ((length = inStream.read(buffer)) != -1)
{
dispatch(buffer, length);
}
So the dispatch method then decrypts the headers and delegates the method depending what is found in the header. It looks similar to:
byte[] clearText = decrypt(message,length);
if (cleartext == foo) sendFooToSocket();
So then sendFooToSocket() would then sit there and read from the instream or send to the outstream.
This is where I seem to run into some problems, in the client I'm sending the header then flushing, then sending the rest of the data, but it appears it's all coming as one and not being split up into header then data. Also is there a best way to force out of the sendFooToSocket method?
public void sendFooToSocket()
{
byte[] buffer = new byte[1024];
int length = 0;
while ((length = inStream.read(buffer) >0)
{
message = decrypt(buffer, length);
}
}
I would assume flush would allow me to break out of this method as it closes then opens the stream?
So I have 2 problems, flush doesn't seem to be breaking up my messages and flush doesn't seem to be allowing to drop out of methods such as sendFooToSocket(), any suggestions?
For clarity sake, the client just does this:
byte[] header = "MESG".getBytes();
cipher = encrypt(header);
outStream.write(cipher,0,cipher.length);
outStream.flush();
byte[] message = "Hi server".getBytes();
cipher = encrypt(message);
outStream.write(cipher,0,cipher.length);
outStream.flush();
But this is received by the server as 1 message even though it's been flushed after every write. Sending just the header works, and we get stuck in the sendFooToSocket() method, but if I send the data after the flush it comes all at once.
The client uses OutputStream and InputStreams just from the socket.get. The client also uses OutputStream and InputStream. Not sure if this matters?

What you seem to want is "record boundaries". With streams in general there are no implicit record boundaries. If you want that kind of functionality you will need to implement it yourself, by buffering the input and looking for, say, newlines to indicate the end of a record.
Look at BufferedInputStream.

inStream.read() may not be returning on a message boundary. You can't assume that it'll return at any particular boundary (such as a blank line separating headers and content if that's how you're doing it.) You'll have to manually parse the content and ignore the fact that it could come from multiple read()s or maybe one read() contains both the headers and content.

Unless you actually want control at the level you have implemented, you could consider Object streams (see ObjectInputStream and ObjectOutputStream). Such streams will allow you to send Java Objects over sockets and read them at the other end with out having to deal with headers and boundaries etc. See ObjectOutputStream for more details, but it's pretty much:
Sender:
writeObject(objectX)
Receiver:
myCopyOfObjectx = readObject()
and you can send any objects you like (as long as they are Serializable).

What does 'end of stream' mean when working with sockets

When working with Sockets in Java, how can you tell whether the client has finished sending all (binary) data, before you could start processing them. Consider for example:
istream = new BufferedInputStream (socket.getInputStream());
ostream = new BufferedOutputStream(socket.getOutputStream());
byte[] buffer = new byte[BUFFER_SIZE];
int count;
while(istream.available() > 0 && (count = istream.read(buffer)) != -1)
{
// do something..
}
// assuming all input has been read
ostream.write(getResponse());
ostream.flush();
I've read similar posts on SO such as this, but couldn't find a conclusive answer. While my solution above works, my understanding is that you can never really tell if the client has finished sending all data. If for instance the client socket sends a few chunks of data and then blocks waiting for data from another data source before it could send more data, the code above may very well assume that the client has finished sending all data since istream.available() will return 0 for the current stream of bytes.

Yes, you're right - using available() like this is unreliable. Personally I very rarely use available(). If you want to read until you reach the end of the stream (as per the question title), keep calling read() until it returns -1. That's the easy bit. The hard bit is if you don't want the end of the stream, but the end of "what the server wants to send you at the moment."
As the others have said, if you need to have a conversation over a socket, you must make the protocol explain where the data finishes. Personally I prefer the "length prefix" solution to the "end of message token" solution where it's possible - it generally makes the reading code a lot simpler. However, it can make the writing code harder, as you need to work out the length before you send anything. This is a pain if you could be sending a lot of data.
Of course, you can mix and match solutions - in particular, if your protocol deals with both text and binary data, I would strongly recommend length-prefixing strings rather than null-terminating them (or anything similar). Decoding string data tends to be a lot easier if you can pass the decoder a complete array of bytes and just get a string back - you don't need to worry about reading to half way through a character, for example. You could use this as part of your protocol but still have overall "records" (or whatever you're transmitting) with an "end of data" record to let the reader process the data and respond.
Of course, all of this protocol design stuff is moot if you're not in control of the protocol :(

I think this is the task more of a protocol, assuming that you are the man who writes both the transmitting and receiving sides of application.
For example you could implement some simple logic protocol and divide you data into packets. Then divide packets into two parts: the head and the body. And then to say that your head consists of a predefined starting sequence and contains number of bytes in the body. Of forget about starting sequence and simpy transfer number of bytes in the bofy as a first byte of the packet.
Then you've could solve you problem.

As some ppl already said you can't avoid some kind of protocol for communication.
It should look like this for example:
On the server side you have:
void sendMSG(PrintWriter out){
try {
//just for example..
Process p = Runtime.getRuntime().exec("cmd /c dir C:");
BufferedReader br = new BufferedReader(new InputStreamReader(
p.getInputStream()));
//and then send all this crap to the client
String s = "";
while ((s = br.readLine()) != null) {
out.println("MSG");
out.println(s);
}
} catch (Exception e) {
System.out.println("Command incorrect!");
}
out.println("END");
}
//You are not supposed to close the stream or the socket, because you might want to send smth else later..
On the client side you have:
void recieveMSG(BufferedReader in) {
try {
while (in.readLine().equals("MSG")) {
System.out.println(in.readLine());
}
} catch (IOException e) {
System.out.println("Connection closed!");
}
}

as Nikita said this is more of task of protocol. Either you can go by header and body approach or you can send a special character or symbol for end of stream to break processing loop. Something like if you send say '[[END]]' on socket to denote end of stream.