This question already has answers here:
Easy way to write contents of a Java InputStream to an OutputStream
(24 answers)
Closed 6 years ago.
I was to trying to find the best way to pipe the InputStream to OutputStream. I don't have an option to use any other libraries like Apache IO. Here is the snippet and output.
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.channels.FileChannel;
public class Pipe {
public static void main(String[] args) throws Exception {
for(PipeTestCase testCase : testCases) {
System.out.println(testCase.getApproach());
InputStream is = new FileInputStream("D:\\in\\lft_.txt");
OutputStream os = new FileOutputStream("D:\\in\\out.txt");
long start = System.currentTimeMillis();
testCase.pipe(is, os);
long end = System.currentTimeMillis();
System.out.println("Execution Time = " + (end - start) + " millis");
System.out.println("============================================");
is.close();
os.close();
}
}
private static PipeTestCase[] testCases = {
new PipeTestCase("Fixed Buffer Read") {
#Override
public void pipe(InputStream is, OutputStream os) throws IOException {
byte[] buffer = new byte[1024];
while(is.read(buffer) > -1) {
os.write(buffer);
}
}
},
new PipeTestCase("dynamic Buffer Read") {
#Override
public void pipe(InputStream is, OutputStream os) throws IOException {
byte[] buffer = new byte[is.available()];
while(is.read(buffer) > -1) {
os.write(buffer);
buffer = new byte[is.available() + 1];
}
}
},
new PipeTestCase("Byte Read") {
#Override
public void pipe(InputStream is, OutputStream os) throws IOException {
int c;
while((c = is.read()) > -1) {
os.write(c);
}
}
},
new PipeTestCase("NIO Read") {
#Override
public void pipe(InputStream is, OutputStream os) throws IOException {
FileChannel source = ((FileInputStream) is).getChannel();
FileChannel destnation = ((FileOutputStream) os).getChannel();
destnation.transferFrom(source, 0, source.size());
}
},
};
}
abstract class PipeTestCase {
private String approach;
public PipeTestCase( final String approach) {
this.approach = approach;
}
public String getApproach() {
return approach;
}
public abstract void pipe(InputStream is, OutputStream os) throws IOException;
}
Output (~4MB input file) :
Fixed Buffer Read
Execution Time = 71 millis
============================================
dynamic Buffer Read
Execution Time = 167 millis
============================================
Byte Read
Execution Time = 29124 millis
============================================
NIO Read
Execution Time = 125 millis
============================================
'Dynamic Buffer Read' uses available() method. But it is not reliable as per java docs
It is never correct to use the return value of this method to allocate
a buffer intended to hold all data in this stream.
'Byte Read' seems to be very slow.
So 'Fixed Buffer Read' is the best option for pipe? Any thoughts?
Java 9
Since Java 9 one can use this method from InputStream:
public long transferTo(OutputStream out) throws IOException
Pre Java 9
A one-liner from apache commons:
IOUtils.copy(inputStream, outputStream);
Documentation here. There are multiple copy methods with different parameters. It is also possible to specify the buffer size.
I came across this, and the final read can cause problems.
SUGGESTED CHANGE:
public void pipe(InputStream is, OutputStream os) throws IOException {
int n;
byte[] buffer = new byte[1024];
while((n = is.read(buffer)) > -1) {
os.write(buffer, 0, n); // Don't allow any extra bytes to creep in, final write
}
os.close ();
I also agree that 16384 is probably a better fixed buffer size than 1024.
IMHO...
I would say a fixed buffer size is the best/easiest to understand. However there are a few problems.
You're writing the entire buffer to the output stream each time. For the final block the read may have read < 1024 bytes so you need to take this into account when doing the write (basically only write number of bytes returned by read()
In the dynamic buffer case you use available(). This is not a terribly reliable API call. I'm not sure in this case inside a loop whether it will be ok, but I wouldn't be suprised if it was implemented sub-optimally in some implementations of InputStream.
The last case you are casting to FileInputStream. If you intend for this to be general purpose then you can't use this approach.
java.io contains PipedInputStream and PipedOutputStream
PipedInputStream input = new PipedInputStream();
PipedOutputStream output = new PipedOutputStream (input);
write to input and it will be visible in output as an Outputstream. Things can work the other way around as well
Related
I recently tried to create a copy/paste app with Java using this answer and I didn't change much other than directories of the file. I created a regular text file with "test" written inside it. When I run the app it copies the file, but gives me an error about utf-8 encoding in success_test.txt and I'm not sure how to fix this or what's causing the problem. I'm running this on Ubuntu 18.04 in Intellij Ultimate 2019.2
here's the server and the client is pretty much the same as in the answer
Server:
public class Server extends Thread {
public static final int PORT = 3332;
public static final int BUFFER_SIZE = 626;
#Override
public void run() {
try {
ServerSocket serverSocket = new ServerSocket(PORT);
while (true) {
Socket s = serverSocket.accept();
saveFile(s);
}
} catch (Exception e) {
}
}
private void saveFile(Socket socket) throws Exception {
InputStream inputStream = socket.getInputStream();
FileOutputStream fileOutputStream = new FileOutputStream("/home/user/Documents/success_test.txt");;
byte[] byteArray = new byte[1024];
System.out.println("Reading file from server...");
BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(fileOutputStream);
int bytesRead;
while ((bytesRead = inputStream.read(byteArray)) != -1) {
bufferedOutputStream.write(byteArray);
}
bufferedOutputStream.close();
System.out.println("Writing file complete...");
}
public static void main(String[] args) {
new Server().start();
}
}
when I try to open success_test.txt this is what I see
https://imgur.com/a/3rpvkiJ
You are reading your data into a 1024 byte long array, then writing that to a file. This means your output file is padded to 1024 bytes with \00 which is the NULL character.
You have your bytesRead variable, so you should use it to only write that many bytes:
bufferedOutputStream.write(byteArray, 0, bytesRead);
As others have mentioned, you have a logic error: you’re ignoring bytesRead in your loop. However, you can eliminate that error, and the possibility of most other errors, by using Files.copy instead of writing your own save code:
try (InputStream inputStream = socket.getInputStream()) {
Files.copy(inputStream,
Paths.get("/home/user/Documents/success_test.txt"));
}
Following the question I asked before: I am implementing an ByteArrayOutputStream having capacity restriction. My main limitation is an amount of available memory. So having such stream os:
When I write more than say 1MB to the output stream I need to "stop".
I prefer not throw exception but write the complete contents of os
output stream to the specified other output stream argument.
OutputStream out;
os.writeTo(out);
And after that continue the writings to os from its beginning
In order to prevent the situation described at 1. , I prefer to drain os,
as freuqntely as possible. I mean copy the data from it to out in chuncks
of 512KB
Is it feasible? If yes any advices how can it be done? Or may be there is a built in class which answers my requirements
Edit: The amount of bytes written to out is also limited. I can write there up to 1GB. If I have more, I need to create other output stream in order to drain from os there.
The proccess of writing to os. can be like that. 500MB was written there - I transfer it immidiately to out. After several seconds 700MB were written there - I need to drain only 500MB to out and other 200MB to other outputstream(out2), which I`ll need to create upon such situation
What you are describing is a BufferedOutputStream, which you can construct like that :
new BufferedOutputStream(out, 512000)
The first arg is the other outputstream you have and the second one is the size of the BufferedOutputStream internal buffer
EDIT:
ok, i did not fully understand your need at first. You will indeed need to extend OutputStream to achieve that. Here is a sample code :
Here is how to use the below code :
public static void main(String[] args) throws IOException {
AtomicLong idx = new AtomicLong(0);
try (
OutputStream out = new OutputStreamMultiVolume(10, () -> new FileOutputStream(getNextFilename(idx)));
) {
out.write("01234567890123456789012345678901234567890123456789".getBytes("UTF-8"));
}
}
private static File getNextFilename(AtomicLong idx) {
return new File("sample.file." + idx.incrementAndGet() + ".txt");
}
The first constructor arg of OutputStreamMultiVolume is the max size of a volume. If we reach this size, we will close the current outputStream, and call the OutputStreamSupplier to get the next one.
The example code here will write the String 01234567890123456789012345678901234567890123456789 (5 times 0123456789) to files named 'sample.file.idx.txt' where idx is increased each time we reach the outstream max size (so you'll get 5 files).
and the class intself :
public class OutputStreamMultiVolume extends OutputStream {
private final long maxBytePerVolume;
private long bytesInCurrentVolume = 0;
private OutputStream out;
private OutputStreamSupplier outputStreamSupplier;
static interface OutputStreamSupplier {
OutputStream get() throws IOException;
}
public OutputStreamMultiVolume(long maxBytePerOutput, OutputStreamSupplier outputStreamSupplier) throws IOException {
this.outputStreamSupplier = outputStreamSupplier;
this.maxBytePerVolume = maxBytePerOutput;
this.out = outputStreamSupplier.get();
}
#Override
public synchronized void write(byte[] bytes) throws IOException {
final int remainingBytesInVol = (int) (maxBytePerVolume - bytesInCurrentVolume);
if (remainingBytesInVol >= bytes.length) {
out.write(bytes);
bytesInCurrentVolume += bytes.length;
return;
}
out.write(bytes, 0, remainingBytesInVol);
switchOutput();
this.write(bytes, remainingBytesInVol, bytes.length - remainingBytesInVol);
}
#Override
public synchronized void write(int b) throws IOException {
if (bytesInCurrentVolume + 1 <= maxBytePerVolume) {
out.write(b);
bytesInCurrentVolume += 1;
return;
}
switchOutput();
out.write(b);
bytesInCurrentVolume += 1;
}
#Override
public synchronized void write(byte[] b, int off, int len) throws IOException {
final int remainingBytesInVol = (int) (maxBytePerVolume - bytesInCurrentVolume);
if (remainingBytesInVol >= len) {
out.write(b, off, len);
bytesInCurrentVolume += len;
return;
}
out.write(b, off, remainingBytesInVol);
switchOutput();
this.write(b, off + remainingBytesInVol, len - remainingBytesInVol);
bytesInCurrentVolume += len - remainingBytesInVol;
}
private void switchOutput() throws IOException {
out.flush();
out.close();
out = outputStreamSupplier.get();
bytesInCurrentVolume = 0;
}
#Override
public synchronized void close() throws IOException {
out.close();
}
#Override
public synchronized void flush() throws IOException {
out.flush();
}
}
I'm afraid that your original question was not fully explained, and so were not the answers you got.
You should not use nor extend BytArrayOutputStream for flushing, because its main feature is to "write data into a byte array": i.e.: all the data is in memory, so you can retrieve it at later through toByteArray.
If you want to flush your exceding data, you need a buffered aproach: It is enough with this construction:
OutputStream out=new FileOutputStream(...);
out=new BufferedOutputStream(out, 1024*1024);
In order to flush the data periodically, you can schedule a TimerTask to invoke flush:
Timer timer=new Timer(true);
TimerTask timerTask=new TimerTask(){
public void run()
{
try
{
out.flush();
}
catch (IOException e)
{
...
}
};
timer.schedule(timerTask, delay, period);
I guess you could try using a java.nio.ByteBuffer in combination with a java.nio.channel.Channels that has a method newChannel(OutputStream);
Like so:
ByteBuffer buffer = ByteBuffer.allocate(1024 * 1024);
//... use buffer
OutputStream out = ...
drainBuffer(buffer, out);
and
public void drainBuffer(ByteBuffer buffer, OutputStream stream) {
WritableByteChannel channel = Channels.newChannel(stream);
channel.write(buffer);
}
I was trying to read a file into an array by using FileInputStream, and an ~800KB file took about 3 seconds to read into memory. I then tried the same code except with the FileInputStream wrapped into a BufferedInputStream and it took about 76 milliseconds. Why is reading a file byte by byte done so much faster with a BufferedInputStream even though I'm still reading it byte by byte? Here's the code (the rest of the code is entirely irrelevant). Note that this is the "fast" code. You can just remove the BufferedInputStream if you want the "slow" code:
InputStream is = null;
try {
is = new BufferedInputStream(new FileInputStream(file));
int[] fileArr = new int[(int) file.length()];
for (int i = 0, temp = 0; (temp = is.read()) != -1; i++) {
fileArr[i] = temp;
}
BufferedInputStream is over 30 times faster. Far more than that. So, why is this, and is it possible to make this code more efficient (without using any external libraries)?
In FileInputStream, the method read() reads a single byte. From the source code:
/**
* Reads a byte of data from this input stream. This method blocks
* if no input is yet available.
*
* #return the next byte of data, or <code>-1</code> if the end of the
* file is reached.
* #exception IOException if an I/O error occurs.
*/
public native int read() throws IOException;
This is a native call to the OS which uses the disk to read the single byte. This is a heavy operation.
With a BufferedInputStream, the method delegates to an overloaded read() method that reads 8192 amount of bytes and buffers them until they are needed. It still returns only the single byte (but keeps the others in reserve). This way the BufferedInputStream makes less native calls to the OS to read from the file.
For example, your file is 32768 bytes long. To get all the bytes in memory with a FileInputStream, you will require 32768 native calls to the OS. With a BufferedInputStream, you will only require 4, regardless of the number of read() calls you will do (still 32768).
As to how to make it faster, you might want to consider Java 7's NIO FileChannel class, but I have no evidence to support this.
Note: if you used FileInputStream's read(byte[], int, int) method directly instead, with a byte[>8192] you wouldn't need a BufferedInputStream wrapping it.
A BufferedInputStream wrapped around a FileInputStream, will request data from the FileInputStream in big chunks (512 bytes or so by default, I think.) Thus if you read 1000 characters one at a time, the FileInputStream will only have to go to the disk twice. This will be much faster!
It is because of the cost of disk access. Lets assume you will have a file which size is 8kb. 8*1024 times access disk will be needed to read this file without BufferedInputStream.
At this point, BufferedStream comes to the scene and acts as a middle man between FileInputStream and the file to be read.
In one shot, will get chunks of bytes default is 8kb to memory and then FileInputStream will read bytes from this middle man.
This will decrease the time of the operation.
private void exercise1WithBufferedStream() {
long start= System.currentTimeMillis();
try (FileInputStream myFile = new FileInputStream("anyFile.txt")) {
BufferedInputStream bufferedInputStream = new BufferedInputStream(myFile);
boolean eof = false;
while (!eof) {
int inByteValue = bufferedInputStream.read();
if (inByteValue == -1) eof = true;
}
} catch (IOException e) {
System.out.println("Could not read the stream...");
e.printStackTrace();
}
System.out.println("time passed with buffered:" + (System.currentTimeMillis()-start));
}
private void exercise1() {
long start= System.currentTimeMillis();
try (FileInputStream myFile = new FileInputStream("anyFile.txt")) {
boolean eof = false;
while (!eof) {
int inByteValue = myFile.read();
if (inByteValue == -1) eof = true;
}
} catch (IOException e) {
System.out.println("Could not read the stream...");
e.printStackTrace();
}
System.out.println("time passed without buffered:" + (System.currentTimeMillis()-start));
}
I'm writing an application that needs to send a file over the network. I've only been taught how to use standard java.net and java.io classes so far (in my first year of college) so I have no experience with java.nio and netty and all those nice things. I've got a working server/client set up using Socket and ServerSocket classes along with BufferedInput/OutputStreams and BufferedFile streams, as follows:
The server:
public class FiletestServer {
static ServerSocket server;
static BufferedInputStream in;
static BufferedOutputStream out;
public static void main(String[] args) throws Exception {
server = new ServerSocket(12354);
System.out.println("Waiting for client...");
Socket s = server.accept();
in = new BufferedInputStream(s.getInputStream(), 8192);
out = new BufferedOutputStream(s.getOutputStream(), 8192);
File f = new File("test.avi");
BufferedInputStream fin = new BufferedInputStream(new FileInputStream(f), 8192);
System.out.println("Sending to client...");
byte[] b = new byte[8192];
while (fin.read(b) != -1) {
out.write(b);
}
fin.close();
out.close();
in.close();
s.close();
server.close();
System.out.println("done!");
}
}
And the client:
public class FiletestClient {
public static void main(String[] args) throws Exception {
System.out.println("Connecting to server...");
Socket s;
if (args.length < 1) {
s = new Socket("", 12354);
} else {
s = new Socket(args[0], 12354);
}
System.out.println("Connected.");
BufferedInputStream in = new BufferedInputStream(s.getInputStream(), 8192);
BufferedOutputStream out = new BufferedOutputStream(s.getOutputStream(), 8192);
File f = new File("test.avi");
System.out.println("Receiving...");
FileOutputStream fout = new FileOutputStream(f);
byte[] b = new byte[8192];
while (in.read(b) != -1) {
fout.write(b);
}
fout.close();
in.close();
out.close();
s.close();
System.out.println("Done!");
}
}
At first I was using no buffering, and writing each int from in.read(). That got me about 200kb/s transfer according to my network monitor gadget on windows 7. I then changed it as above but used 4096 byte buffers and got the same speed, but the file received was usually a couple kilobytes bigger than the source file, and that is what my problem is. I changed the buffer size to 8192 and I now get about 3.7-4.5mb/sec transfer over wireless to my laptop, which is plenty fast enough for now, but I still have the problem of the file getting slightly bigger (which would cause it to fail an md5/sha hash test) when it is received.
So my question is what is the proper way of buffering to get decent speeds and end up with exactly the same file on the other side? Getting it to go a bit faster would be nice too but I'm happy with the speed for now. I'm assuming a bigger buffer is better up to a point, I just need to find what that point is.
You are ignoring the size of data actually read.
while (in.read(b) != -1) {
fout.write(b);
}
will always write 8192 bytes even if only one byte is read. Instead I suggest using
for(int len; ((len = in.read(b)) > 0;)
fout.write(b, 0, len);
Your buffers are the same size as your byte[] so they are not really doing anything at the moment.
The MTU for most networks is around 1500 bytes and you get a performance improvement on slower networks (up to 1 GB) up to 2 KB. 8 KB as fine as well. Larger than that is unlikely to help.
If you actually want to make it 'so perfect', you should take a look at the try-catch-with-resources statement and the java.nio package (or any nio-derivated libraries).
This is my code, I'm using rxtx.
public void Send(byte[] bytDatos) throws IOException {
this.out.write(bytDatos);
}
public byte[] Read() throws IOException {
byte[] buffer = new byte[1024];
int len = 20;
while(in.available()!=0){
in.read(buffer);
}
System.out.print(new String(buffer, 0, len) + "\n");
return buffer;
}
the rest of code is just the same as this, i just changed 2 things.
InputStream in = serialPort.getInputStream();
OutputStream out = serialPort.getOutputStream();
They are global variables now and...
(new Thread(new SerialReader(in))).start();
(new Thread(new SerialWriter(out))).start();
not exist now...
I'm sending this (each second)
Send(("123456789").getBytes());
And this is what i got:
123456789123
456789
123456789
1234567891
23456789
can anybody help me?
EDIT
Later, i got the better way to solve it. Thanks, this was the Read Code
public byte[] Read(int intEspera) throws IOException {
try {
Thread.sleep(intEspera);
} catch (InterruptedException ex) {
Logger.getLogger(COM_ClComunica.class.getName()).log(Level.SEVERE, null, ex);
}//*/
byte[] buffer = new byte[528];
int len = 0;
while (in.available() > 0) {
len = in.available();
in.read(buffer,0,528);
}
return buffer;
}
It was imposible for me to erase that sleep but it is not a problem so, thanks veer
You should indeed note that InputStream.available is defined as follows...
Returns an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking by the next invocation of a method for this input stream. The next invocation might be the same thread or another thread. A single read or skip of this many bytes will not block, but may read or skip fewer bytes.
As you can see, this is not what you expected. Instead, you want to check for end-of-stream, which is indicated by InputStream.read() returning -1.
In addition, since you don't remember how much data you have already read in prior iterations of your read loop, you are potentially overwriting prior data in your buffer, which is again not something you likely intended.
What you appear to want is something as follows:
private static final int MESSAGE_SIZE = 20;
public byte[] read() throws IOException {
final byte[] buffer = new byte[MESSAGE_SIZE];
int total = 0;
int read = 0;
while (total < MESSAGE_SIZE
&& (read = in.read(buffer, total, MESSAGE_SIZE - total)) >= 0) {
total += read;
}
return buffer;
}
This should force it to read up to 20 bytes, less in the case of reaching the end of the stream.
Special thanks to EJP for reminding me to maintain the quality of my posts and make sure they're correct.
Get rid of the available() test. All it is doing is telling you whether there is data ready to be read without blocking. That isn't the same thing as telling you where an entire message ends. There are few correct uses for available(), and this isn't one of them.
And advance the buffer pointer when you read. You need to keep track of how many bytes you have read so far, and use that as the 2nd parameter to read(), with buffer.length as the third parameter.