I have created a basic client-server program that transfers file from server to the client. Since the file is about 500MB, the server transfers the file as chunks of bytes to the client over the DataOutputStream object. While this logic works okay when both client and server are running on the same computer, it doesn't work when the two programs are running on separate computers(Both computers are on the same network and I have disabled the firewall for both)
When running on separate computers the problem is that a few bytes get transferred
Server Logic:
byte byteArr[] = new byte[1024];
while((c=fileInputStream.read(byteArr, 0, 1024) != -1))
{
dataOutputStream.writeBoolean(true);
dataOutputStream.flush();
dataOutputStream.write(byteArr, 0, 1024);
dataOutputStream.flush();
}
/*When running on different computers, after a few hundred iterations
it just stops looping the following lines are never executed*/
dataOutputStream.writeBoolean(false);
System.out.println("Transfer complete");
Client Logic
byte byteArr[] = new byte[1024];
while(dataInputStream.readBoolean())
{
dataInputStream.read(byteArr, 0, 1024);
fileOutputStream.write(byteArr, 0, 1024);
}
A read(buf, 0, 1024) call is not guaranteed to read exactly 1024 bytes. This causes bugs in both pieces of code:
the server incorrectly assumes that each chunk read from the file is always exactly 1024 bytes long and sends the whole buffer to the client.
the client might not read the whole chunk in a single iteration. It will then treat the first byte of the remainder as a boolean and get out of sync with the server.
To resolve this, you could:
send the file size (if known) before sending the file,
then just keep reading until you've read that many bytes.
or send c (the chunk size) instead of a single boolean,
then use dataInputStream.readFully() to make sure that
many bytes will be read.
Related
I'm sending a string over the socket I previously sent a file to, but the recipient reads it as part of the file itself, is there a way to send a sort of EOF before sending the string?
To send the file I'm using
byte[] buffer = new byte[1024];
int count;
while ((count = fis.read(buffer)) >= 0) os.write(buffer, 0, count);
os.flush();
(and almost the same to receive it)
To send the string I'm using OutputStreamWriter
(Here you are my code: hatebin)
I've also read here that I should send a SOH character, but which one should I send and how?
Thanks in advance.
No there's no way to send an "eof" and then send something afterwards.
If you don't want to open a new connection, there are basically two ways to solve this.
You can modify the client so it recognizes some special byte sequence as a "delimiter", and stops writing to the file when it reads the delimiter from the socket. In this case you need to have some strategy to deal with the possibility that the file actually contains the delimiter.
You can send the size of the file in bytes before sending the file, and modify the client so it counts the number of bytes it reads from the socket. When the client has read enough, it should stop writing to the file.
When transmitting a file through socket in blocking mode
bytesTransferred = fileIChannel.transferTo(0, fileIChannel.size(), socketChannel);
// or using a buffer
ByteBuffer byteBuffer = ByteBuffer.allocateDirect(1024*8);
while (fileIChannel.read(byteBuffer) != -1) {
byteBuffer.flip();
bytesTransferred += socketChannel.write(byteBuffer);
byteBuffer.clear();
In the event of a connection failure, I need to keep the number of bytes transferred.
I can do this by waiting for a response every from the server when it receives a particular number of bytes. Or, when the connection is restored, I send a request for the number of bytes received.
Which of the options will be more correct? How is this problem usually resolved?
And the second question. Is data integrity guaranteed when sending large data through a socket?
WIth this code you cannot tell. If you want to know that the peer application has received and processed sent data, the peer application has to tell you. TCP does buffering at both ends, so the API alone cannot tell you.
NB Your copy loop is wrong. It should be:
while ((fileIChannel.read(byteBuffer) != -1 && byteBuffer.position() > 0)
{
byteBuffer.flip();
bytesTransferred += socketChannel.write(byteBuffer);
byteBuffer.compact();
}
and there should also be an error test on the write. At present you are assuming that everything got written to the SocketChannel on every write, which isn't guaranteed in non-blocking mode.
The code using transferTo() is also wrong, as transferTo() isn't guaranteed to perform the entire transfer: that's why it returns a count. You have to loop.
I'm working on a project which requires transmitting a MP4 video from a Java (Android) server to a C++ (Visual studio) client.
Everything works fine if I use a Java Client, but with the C++ client I receive a file 1 byte larger that the file sent. As a result, the file won't obviously open. I don't know if the problem is just with that extra bit or there is something else wrong.
Here's the relevant code of my C++ Client:
FILE *myFile=std::fopen("scan.mp4","wb");
char *recVideoBuffer=new char[videoSize];//videoSize previously received
int written;
totalBytesRcvd = 0;
while (totalBytesRcvd < videoSize)
{
if ((bytesRcvd = recv(sock, recVideoBuffer, videoSize, 0)) <= 0)
DieWithError("recv() failed or connection closed prematurely");
totalBytesRcvd += bytesRcvd; /* Keep tally of total bytes */
if ((written = fwrite(recVideoBuffer, 1, bytesRcvd, myFile)) != bytesRcvd)
DieWithError("fwrite() failed");
printf("Filesize: %d\nReceived: %d\Written %d\nTotal Received: %d\n\n", videoSize, bytesRcvd, written, totalBytesRcvd);
}
fclose(myFile);
closesocket(sock);
Java Server:
File source = new File (VIDEO_FILE);
byte byteArray[] = new byte [(int)source.length()];
BufferedInputStream bStream = new BufferedInputStream(new FileInputStream(source));
bStream.read(byteArray, 0, byteArray.length);
OutputStream outStream = s.getOutputStream();
outStream.write(byteArray, 0, byteArray.length);
outStream.flush();
outStream.close();
bStream.close();
s.close();
server.close();
Any ideas on what the problem(s) might be and how to fix them?
I've looked around and found some related questions but those were either unanswered or not relevant...
Thanks.
EDIT:
I tried sending a 'abcde' .asc file, and the client receives just 'abcd'. In this case both the files are 5 byte large though.
===
I looked into the byte values of the MP4 files.
The received one has 2 extra bytes, 0a, at the beginning, and is missing the last 2B. Manually changing these values makes the video play correctly.
Where does 0a come from? Obviously because of those 2B the last 2 are not written...
SOLVED
Quite a silly issue really... I would send the video size using println(), which recv() couldn't read entirely, apparently leaving those 0a bytes over the socket, ready to be received by the next recv() along with the file bytes.
That explains why everything worked with the Java client, as I would read the videoSize using readLine() .
How annoying having wasted a day on this, Lol.
I'm doing a program that sends large files through sockets client-server, that piece of code
while ((bytesRead = in.read(mybytearray, 0, mybytearray.length)) != -1) {
bos.write(mybytearray, 0, bytesRead);
}
All code can be found here
I have already downloaded data as where to pass this number was spooled file? For example there is a file size 35000 bytes transmitted 20000 is broken and how to make the program to start downloading from 20000 bytes to continue?
P.S. I`m very sorry, my English is bad
When you open the connection to the destination file just keep reading it until you reach 20000 bytes, discarding the data.
After you reach your "resume point" start appending the new data to the existing file.
Unfortunately you cannot "seek" a socket the same way you would in something like local file I/O.
My problem is that C sockets look to act differently than Java sockets. I have a C proxy and I tested it between a workload generator (oltp benchmark client written in Java) and the JDBC connector of the Postgres DB.
This works great and forwards data from one to other, as it should. We need to make this proxy work in Java, so I used plain ServerSocket and Socket classes from java.net and I cannot make it work. The Postgres returns an authentication error message, assuming that the client did not send the correct password.
Here is how the authentication at the JDBC protocol works:
-client sends a requests to connect to a database specifying the database name and the username
-server responds back with a one time challenge message (13 byte message with random content)
-client concatenates this message with the user password and performs a md5 hash
-server compares the hash got from the client with the hash he computes
[This procedure is performed in order to avoid replay attacks (if client would send only the md5 hash of its password then an attacker could replay this message, pretending he is the client)]
So I inspected the packets with tcpdump and they look correct! The size is exactly as it should, so maybe the content is corrupted (??)
Sometimes though the DB server responds ok for the authentication (depending on the value of the challenge message)!! And then the oltp client sends a couple of queries, but it crashes in a while…
I guess that maybe it has to do with the encoding, so I tried with the encoding that C uses (US-ANSII), but still the same.
I send the data using fixed size character or byte arrays both in C and in Java!
I really don't have any more ideas, as I tried so many cases...
What is your guess of what would be the problem?
Here is a representative code that may help you have a more clear view:
byte [] msgBuf;
char [] msgBufChars;
while(fromInputReader.ready()){
msgBuf = new byte[1024];
msgBufChars = new char[1024];
// read data from one party
int read = fromInputReader.read(msgBufChars, 0, 1024);
System.out.println("Read returned : " + read);
for(int i=0; i<1024; i++)
msgBuf[i] = (byte) msgBufChars[i];
String messageRead = new String(msgBufChars);
String messageToWrite = new String(msgBuf);
System.out.println("message read : "+messageRead);
System.out.println("message to write : "+new String(messageToWrite));
// immediatelly write data to other party (write the amount of data we read (read value) )
// there is no write method that takes a char [] as a parameter, so pass a byte []
toDataOutputStream.write(msgBuf, 0, read);
toDataOutputStream.flush();
}
There are a couple of message exchanges in the beginning and then Postgres responds with an authentication failure message.
Thanks for your time!
What is your guess of what would be the problem?
It is nothing to do with C versus Java sockets. It is everything to do with bad Java code.
I can see some problems:
You are using a Reader in what should be a binary stream. This is going to result in the data being converted from bytes (from the JDBC client) to characters and then back to bytes. Depending on the character set used by the reader, this is likely to be destructive.
You should use plain, unadorned1 input streams for both reading and writing, and you should read / write to / from a preallocated byte[].
This is terrible:
for(int i=0; i<1024; i++)
msgBuf[i] = (byte) msgBufChars[i];
If the characters you read are not in the range 0 ... 255 you are mangling them when you stuff them into msgBuf.
You are assuming that you actually got 1024 characters.
You are using the ready() method to decide when to stop reading stuff. This is almost certainly wrong. Read the javadoc for that method (and think about it) and you should understand why it is wrong. (Hint: what happens if the proxy can read faster than the client can deliver?)
You should use a while(true), and then break out of the loop if read tells you it has reached the end of stream; i.e. if it returns -1 ...
1 - Just use the stream objects that the Socket API provides. DataXxxStream is unnecessary because the read and write methods are simply call-throughs. I wouldn't even use BufferedXxxStream wrappers in this case, because you are already doing your own buffering using the byte array.
Here's how I'd write that code:
byte [] buffer = new byte[1024]; // or bigger
while(true) {
int nosRead = inputStream.read(buffer);
if (nosRead < 0) {
break;
}
// Note that this is a bit dodgy, given that the data you are converting is
// binary. However, if the purpose is to see what embedded character data
// looks like, and if the proxy's charset matches the text charset used by
// the client-side JDBC driver for encoding data, this should achieve that.
System.out.println("Read returned : " + nosRead);
System.out.println("message read : " + new String(buffer, 0, nosRead));
outputStream.write(buffer, 0, nosRead);
outputStream.flush();
}
C sockets look to act differently than Java sockets.
Impossible. Java sockets are just a very thin layer over C sockets. You're on the wrong track with this line of thinking.
byte [] msgBuf;
char [] msgBufChars;
Why are you reading chars when you want to write bytes? Don't use Readers unless you know that the input is text.
And don't call ready(). There are very few correct uses, and this isn't one of them. Just block.