I'm writing an application that needs to send a file over the network. I've only been taught how to use standard java.net and java.io classes so far (in my first year of college) so I have no experience with java.nio and netty and all those nice things. I've got a working server/client set up using Socket and ServerSocket classes along with BufferedInput/OutputStreams and BufferedFile streams, as follows:
The server:
public class FiletestServer {
static ServerSocket server;
static BufferedInputStream in;
static BufferedOutputStream out;
public static void main(String[] args) throws Exception {
server = new ServerSocket(12354);
System.out.println("Waiting for client...");
Socket s = server.accept();
in = new BufferedInputStream(s.getInputStream(), 8192);
out = new BufferedOutputStream(s.getOutputStream(), 8192);
File f = new File("test.avi");
BufferedInputStream fin = new BufferedInputStream(new FileInputStream(f), 8192);
System.out.println("Sending to client...");
byte[] b = new byte[8192];
while (fin.read(b) != -1) {
out.write(b);
}
fin.close();
out.close();
in.close();
s.close();
server.close();
System.out.println("done!");
}
}
And the client:
public class FiletestClient {
public static void main(String[] args) throws Exception {
System.out.println("Connecting to server...");
Socket s;
if (args.length < 1) {
s = new Socket("", 12354);
} else {
s = new Socket(args[0], 12354);
}
System.out.println("Connected.");
BufferedInputStream in = new BufferedInputStream(s.getInputStream(), 8192);
BufferedOutputStream out = new BufferedOutputStream(s.getOutputStream(), 8192);
File f = new File("test.avi");
System.out.println("Receiving...");
FileOutputStream fout = new FileOutputStream(f);
byte[] b = new byte[8192];
while (in.read(b) != -1) {
fout.write(b);
}
fout.close();
in.close();
out.close();
s.close();
System.out.println("Done!");
}
}
At first I was using no buffering, and writing each int from in.read(). That got me about 200kb/s transfer according to my network monitor gadget on windows 7. I then changed it as above but used 4096 byte buffers and got the same speed, but the file received was usually a couple kilobytes bigger than the source file, and that is what my problem is. I changed the buffer size to 8192 and I now get about 3.7-4.5mb/sec transfer over wireless to my laptop, which is plenty fast enough for now, but I still have the problem of the file getting slightly bigger (which would cause it to fail an md5/sha hash test) when it is received.
So my question is what is the proper way of buffering to get decent speeds and end up with exactly the same file on the other side? Getting it to go a bit faster would be nice too but I'm happy with the speed for now. I'm assuming a bigger buffer is better up to a point, I just need to find what that point is.
You are ignoring the size of data actually read.
while (in.read(b) != -1) {
fout.write(b);
}
will always write 8192 bytes even if only one byte is read. Instead I suggest using
for(int len; ((len = in.read(b)) > 0;)
fout.write(b, 0, len);
Your buffers are the same size as your byte[] so they are not really doing anything at the moment.
The MTU for most networks is around 1500 bytes and you get a performance improvement on slower networks (up to 1 GB) up to 2 KB. 8 KB as fine as well. Larger than that is unlikely to help.
If you actually want to make it 'so perfect', you should take a look at the try-catch-with-resources statement and the java.nio package (or any nio-derivated libraries).
Related
I recently tried to create a copy/paste app with Java using this answer and I didn't change much other than directories of the file. I created a regular text file with "test" written inside it. When I run the app it copies the file, but gives me an error about utf-8 encoding in success_test.txt and I'm not sure how to fix this or what's causing the problem. I'm running this on Ubuntu 18.04 in Intellij Ultimate 2019.2
here's the server and the client is pretty much the same as in the answer
Server:
public class Server extends Thread {
public static final int PORT = 3332;
public static final int BUFFER_SIZE = 626;
#Override
public void run() {
try {
ServerSocket serverSocket = new ServerSocket(PORT);
while (true) {
Socket s = serverSocket.accept();
saveFile(s);
}
} catch (Exception e) {
}
}
private void saveFile(Socket socket) throws Exception {
InputStream inputStream = socket.getInputStream();
FileOutputStream fileOutputStream = new FileOutputStream("/home/user/Documents/success_test.txt");;
byte[] byteArray = new byte[1024];
System.out.println("Reading file from server...");
BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(fileOutputStream);
int bytesRead;
while ((bytesRead = inputStream.read(byteArray)) != -1) {
bufferedOutputStream.write(byteArray);
}
bufferedOutputStream.close();
System.out.println("Writing file complete...");
}
public static void main(String[] args) {
new Server().start();
}
}
when I try to open success_test.txt this is what I see
https://imgur.com/a/3rpvkiJ
You are reading your data into a 1024 byte long array, then writing that to a file. This means your output file is padded to 1024 bytes with \00 which is the NULL character.
You have your bytesRead variable, so you should use it to only write that many bytes:
bufferedOutputStream.write(byteArray, 0, bytesRead);
As others have mentioned, you have a logic error: you’re ignoring bytesRead in your loop. However, you can eliminate that error, and the possibility of most other errors, by using Files.copy instead of writing your own save code:
try (InputStream inputStream = socket.getInputStream()) {
Files.copy(inputStream,
Paths.get("/home/user/Documents/success_test.txt"));
}
I wrote a piece of Java code to send PDF-turned postscript scripts to a network printer via Socket.
The files were printed in perfect shape but every job comes with one or 2 extra pages with texts like ps: stack underflow or error undefined offending command.
At beginning I thought something is wrong with the PDF2PS process so I tried 2 PS files from this PS Files. But the problem is still there.
I also verified the ps files with GhostView. Now I think there may be something wrong with the code. The code does not throw any exception.
The printer, Toshiba e-studion 5005AC, supports PS3 and PCL6.
File file = new File("/path/to/my.ps");
Socket socket = null;
DataOutputStream out = null;
FileInputStream inputStream = null;
try {
socket = new Socket(printerIP, printerPort);
out = new DataOutputStream(socket.getOutputStream());
DataInputStream input = new DataInputStream(socket.getInputStream());
inputStream = new FileInputStream(file);
byte[] buffer = new byte[8000];
while (inputStream.read(buffer) != -1) {
out.write(buffer);
}
out.flush();
} catch (IOException e) {
e.printStackTrace();
}
You are writing the whole buffer to the output stream regardless of how much actual content there is.
That means that when you write the buffer the last time it will most probably have a bunch of content from the previous iteration at the end of the buffer.
Example
e.g. imagine you have the following file and you use a buffer of size 10:
1234567890ABCDEF
After first inputStream.read() call it will return 10 and in the buffer you will have:
1234567890
After second inputStream.read() call it will return 6 and in the buffer you will have:
ABCDEF7890
After third inputStream.read() call it will return -1 and you will stop reading.
A printer socket will receive these data in the end:
1234567890ABCDEF7890
Here the last 7890 is an extra bit that the printer does not understand, but it can successfully interpret the first 1234567890ABCDEF.
Fix
You should consider the length returned by inputStream.read():
byte[] buffer = new byte[8000];
for (int length; (length = inputStream.read(buffer)) != -1; ){
out.write(buffer, 0, length);
}
Also consider using try-with-resources to avoid problems with unclosed streams.
This question already has answers here:
Can't get out the receive file loop in send file using socket java
(2 answers)
Closed 9 years ago.
I am developing a simple file tranfer app between client and server.
Here is my code:
My Client(Sender):
try{
File file_sender=XMLParser.register(peername, address, port, listfile);
int count;
byte[] buffer = new byte[1024];
int len=(int) file_sender.length();
byte[] mybytearray = new byte[len];
DataOutputStream output=new DataOutputStream(Info.connection.getOutputStream());
output.writeInt(len);
System.out.println(len);
BufferedInputStream bis=new BufferedInputStream(new FileInputStream(file_sender));
bis.read(mybytearray, 0, len);
output.write(mybytearray, 0, len);
bis.close();
output.close();
// Info.connection.close();
}catch(Exception ex){
System.out.println(ex.toString());
}
My server(Receiver):
public class IOThread extends Thread {
private Socket connection;
private DataInputStream input;
public IOThread(Socket connection) throws IOException{
this.connection=connection;
input=new DataInputStream(connection.getInputStream());
}
#Override
public void run(){
while(true){
try{
int filesize=12022386;
int bytesRead;
int currentTot = 0;
byte[] bytearray = new byte [filesize];
int len=input.readInt();
FileOutputStream fos = new FileOutputStream("data.xml");
BufferedOutputStream bos = new BufferedOutputStream(fos);
bytesRead=input.read(bytearray, 0,bytearray.length);
System.out.println(len);
currentTot=bytesRead;
do{
bytesRead=input.read(bytearray, currentTot, (bytearray.length-currentTot));
if(bytesRead>=0) currentTot+=bytesRead;
System.out.println("pos: "+currentTot);
} while(len!=currentTot);
bos.write(bytearray, 0, currentTot);
bos.close();
//connection.close();
System.out.println("Done");
}catch(EOFException ex){
System.out.println(ex.toString());
break;
}catch( Exception ex){}
}
}
I want to tranfer more than one file, so I don't want to close socket. So, I use variable "len" to check the file is tranfered completely.
If I close "output" after send file then server thows EOFException and the file is sent success.
When I don't close output, server can't receive file success. But Server doesn't throws EOFException.
Can you help me solve this problem?
Update: Here is my output on the console screen if I don't close variable "output" at client:
246
pos: 496
Thanks.
Sorry for my poor English
Two things:
First of all, you seem to be ignoring the file length in your receiving code? You have:
int filesize=12022386;
int bytesRead;
int currentTot = 0;
byte[] bytearray = new byte [filesize];
int len=input.readInt();
You are sizing bytearray to 12022386 bytes regardless of the value of len, and you are asking for that many bytes from the input stream.
Secondly, when you close the output stream, any cached/buffered data that hasn't been written yet is automatically flushed (i.e., in your case, sent to the server).
Flush the output stream explicitly when you're done sending the first file to force it to finish sending all of the data to the server. You can then continue using that output stream for other things.
Do output.flush() in your sending code when the transfer is complete.
The problem is that you read beyond the size of your file, i.e. instead of reading len bytes, you read bytearray.length bytes, which is larger.
Consequently, you read more than len bytes, so len!=currentTot is never satisfied, because, as you can see from your sample output, currentTot == 496 and len == 246.
Make the following changes in your while loop:
do{
bytesRead=input.read(bytearray, currentTot, (len - currentTot));
if(bytesRead>=0) currentTot += bytesRead;
System.out.println("pos: " + currentTot);
} while(len != currentTot);
Just to make sure you don't end up in an endless loop due to a similar error, you could use currentTot < len instead of len != currentTot as your condition.
Also, since you're already using DataInputStream, consider reading the socket contents using readFully. This blocks until a given number of bytes is read from the socket, essentially eliminating your need for the while loop. Read more here.
Throw all the loops away and use DataInputStream.readFully(); close the socket after sending the file; don't s end the file length; and use a normal size buffer, say 8192 bytes.
Your BufferedOutputStream filling seems incorrect. Below code must be inside the while loop.
bos.write(bytearray, 0, currentTot);
Try something like this instead:
BufferedOutputStream buffOut=new BufferedOutputStream(fos);
byte []arr = new byte [1024 * 1024];
int available = -1;
while((available = buffIn.read(arr)) != -1) {
buffOut.write(arr, 0, available);
}
buffOut.flush();
buffOut.close();
And test again.
Edit: Updated my answer with #Jason's correct comment.
I'm trying to running an external program from a Java program and I'm having trouble. Basically what I'd like to do would be this:
Runtime.getRuntime().exec("./extprogram <fileIn >fileOut");
However I've found that that doesn't work - Java apparentls needs to use a Process with input and output streams and other things which I'm not experienced with.
I've looked at a number of examples across the internet (many of which are from SO), and there doesn't seem to be a simple standard way of doing this, which for someone who doesn't fully understand what's going on, can be quite frustrating.
I'm also having trouble trying to build my own code off the examples of other people's code because generally it seems most other people 1. aren't interested in redirecting stdin, and 2. aren't necessarily redirecting stdout to a file, but instead to System.out.
So, would anyone be able to point me in the direction of any good simple code templates for calling external programs and redirecting stdin and stdout? Thanks.
You could try something like this:
ProcessBuilder pb = new ProcessBuilder();
pb.redirectInput(new FileInputStream(new File(infile));
pb.redirectOutput(new FileOutputStream(new File(outfile));
pb.command(cmd);
pb.start().waitFor();
If you must use Process, then something like this should work:
public static void pipeStream(InputStream input, OutputStream output)
throws IOException
{
byte buffer[] = new byte[1024];
int numRead = 0;
do
{
numRead = input.read(buffer);
output.write(buffer, 0, numRead);
} while (input.available() > 0);
output.flush();
}
public static void main(String[] argv)
{
FileInputStream fileIn = null;
FileOutputStream fileOut = null;
OutputStream procIn = null;
InputStream procOut = null;
try
{
fileIn = new FileInputStream("test.txt");
fileOut = new FileOutputStream("testOut.txt");
Process process = Runtime.getRuntime().exec ("/bin/cat");
procIn = process.getOutputStream();
procOut = process.getInputStream();
pipeStream(fileIn, procIn);
pipeStream(procOut, fileOut);
}
catch (IOException ioe)
{
System.out.println(ioe);
}
}
Note:
Be sure to close the streams
Change this to use buffered streams, I think the raw Input/OutputStreams implementation may copy a byte at a time.
The handling of the process will probably change depending on your specific process: cat is the simplest example with piped I/O.
have you tried System.setIn and System.setOut? has been around since JDK 1.0.
public class MyClass
{
System.setIn( new FileInputStream( "fileIn.txt" ) );
int oneByte = (char) System.in.read();
...
System.setOut( new FileOutputStream( "fileOut.txt" ) );
...
I am doing a program to saturate a link for performance testing in my networking lab, I tried different things, from changing Send and Receive buffers, creating a file and reading it, creating a long array and sending it through the socket all at once: OutputStream.write(byte[])
The array is 1000000 positions length, when I sniff the network traffic, according to the sniffer, the packets have "Data (1460 bytes)" which make me supose that I'm not sending byte by byte.
The bandwidth used is about 8% of the 100Mbps.
I post the relevant code as there is some interaction between client and server which I don't think is relevant:
Client:
int car=0;
do {
car=is.read();
//System.out.println(car);
contador++;
} while(car!=104);
Server:
byte dades[]=new byte[1000000];
FileInputStream fis=null;
try {
FileOutputStream fos = new FileOutputStream("1MB.txt");
fos.write(dades);
fos=null;
File f = new File("1MB.txt");
fis = new FileInputStream(f);
step=0;
correcte=true;
sck = srvSock.accept();
sck.setSendBufferSize(65535);
sck.setReceiveBufferSize(65535);
os = sck.getOutputStream();
is = sck.getInputStream();
}
...
BufferedInputStream bis = new BufferedInputStream(fis);
bis.read(dades);
for(int i=0;i<100;i++) {
os.write(dades);
}
In this case I put the last idea I had, to create a file with a million positions byte array and then read this file and write to the socket, before this idea I was sending the byte array.
Another thing which make me believe this is not a byte by byte sending is that in a quad core computer the client uses 25% CPU and uses around 8% of the bandwidth, and in an old computer which is single core (AMD Athlon) it uses 100% of the CPU and just 4% of the bandwidth. The server is not so CPU intensive.
Any ideas??? I feel a little lost right now...
Thanks!!!
Perhaps it's related to the fact that client reads data byte by byte, that can force flow control algorithm to limit transmission bandwidth:
int car=0;
do {
car=is.read();
//System.out.println(car);
contador++;
} while(car!=104);
Try to read data into array instead, or use BufferedInputStream:
byte[] buf = new byte[65536];
int size = 0;
boolean stop = false;
while (!stop && (size = is.read(buf)) != -1) {
for (int i = 0; i < size; i++) {
if (buf[i] == 104) {
stop = true;
break;
}
}
}