I have a InputStream that I pass to a method to do some processing. I will use the same InputStream in other method, but after the first processing, the InputStream appears be closed inside the method.
How I can clone the InputStream to send to the method that closes him? There is another solution?
EDIT: the methods that closes the InputStream is an external method from a lib. I dont have control about closing or not.
private String getContent(HttpURLConnection con) {
InputStream content = null;
String charset = "";
try {
content = con.getInputStream();
CloseShieldInputStream csContent = new CloseShieldInputStream(content);
charset = getCharset(csContent);
return IOUtils.toString(content,charset);
} catch (Exception e) {
System.out.println("Error downloading page: " + e);
return null;
}
}
private String getCharset(InputStream content) {
try {
Source parser = new Source(content);
return parser.getEncoding();
} catch (Exception e) {
System.out.println("Error determining charset: " + e);
return "UTF-8";
}
}
If all you want to do is read the same information more than once, and the input data is small enough to fit into memory, you can copy the data from your InputStream to a ByteArrayOutputStream.
Then you can obtain the associated array of bytes and open as many "cloned" ByteArrayInputStreams as you like.
ByteArrayOutputStream baos = new ByteArrayOutputStream();
// Code simulating the copy
// You could alternatively use NIO
// And please, unlike me, do something about the Exceptions :D
byte[] buffer = new byte[1024];
int len;
while ((len = input.read(buffer)) > -1 ) {
baos.write(buffer, 0, len);
}
baos.flush();
// Open new InputStreams using recorded bytes
// Can be repeated as many times as you wish
InputStream is1 = new ByteArrayInputStream(baos.toByteArray());
InputStream is2 = new ByteArrayInputStream(baos.toByteArray());
But if you really need to keep the original stream open to receive new data, then you will need to track the external call to close(). You will need to prevent close() from being called somehow.
UPDATE (2019):
Since Java 9 the the middle bits can be replaced with InputStream.transferTo:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
input.transferTo(baos);
InputStream firstClone = new ByteArrayInputStream(baos.toByteArray());
InputStream secondClone = new ByteArrayInputStream(baos.toByteArray());
You want to use Apache's CloseShieldInputStream:
This is a wrapper that will prevent the stream from being closed. You'd do something like this.
InputStream is = null;
is = getStream(); //obtain the stream
CloseShieldInputStream csis = new CloseShieldInputStream(is);
// call the bad function that does things it shouldn't
badFunction(csis);
// happiness follows: do something with the original input stream
is.read();
You can't clone it, and how you are going to solve your problem depends on what the source of the data is.
One solution is to read all data from the InputStream into a byte array, and then create a ByteArrayInputStream around that byte array, and pass that input stream into your method.
Edit 1:
That is, if the other method also needs to read the same data. I.e you want to "reset" the stream.
If the data read from the stream is large, I would recommend using a TeeInputStream from Apache Commons IO. That way you can essentially replicate the input and pass a t'd pipe as your clone.
This might not work in all situations, but here is what I did: I extended the FilterInputStream class and do the required processing of the bytes as the external lib reads the data.
public class StreamBytesWithExtraProcessingInputStream extends FilterInputStream {
protected StreamBytesWithExtraProcessingInputStream(InputStream in) {
super(in);
}
#Override
public int read() throws IOException {
int readByte = super.read();
processByte(readByte);
return readByte;
}
#Override
public int read(byte[] buffer, int offset, int count) throws IOException {
int readBytes = super.read(buffer, offset, count);
processBytes(buffer, offset, readBytes);
return readBytes;
}
private void processBytes(byte[] buffer, int offset, int readBytes) {
for (int i = 0; i < readBytes; i++) {
processByte(buffer[i + offset]);
}
}
private void processByte(int readByte) {
// TODO do processing here
}
}
Then you simply pass an instance of StreamBytesWithExtraProcessingInputStream where you would have passed in the input stream. With the original input stream as constructor parameter.
It should be noted that this works byte for byte, so don't use this if high performance is a requirement.
UPD.
Check the comment before. It isn't exactly what was asked.
If you are using apache.commons you may copy streams using IOUtils .
You can use following code:
InputStream = IOUtils.toBufferedInputStream(toCopy);
Here is the full example suitable for your situation:
public void cloneStream() throws IOException{
InputStream toCopy=IOUtils.toInputStream("aaa");
InputStream dest= null;
dest=IOUtils.toBufferedInputStream(toCopy);
toCopy.close();
String result = new String(IOUtils.toByteArray(dest));
System.out.println(result);
}
This code requires some dependencies:
MAVEN
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.4</version>
</dependency>
GRADLE
'commons-io:commons-io:2.4'
Here is the DOC reference for this method:
Fetches entire contents of an InputStream and represent same data as
result InputStream. This method is useful where,
Source InputStream is slow. It has network resources associated, so we
cannot keep it open for long time. It has network timeout associated.
You can find more about IOUtils here:
http://commons.apache.org/proper/commons-io/javadocs/api-2.4/org/apache/commons/io/IOUtils.html#toBufferedInputStream(java.io.InputStream)
Below is the solution with Kotlin.
You can copy your InputStream into ByteArray
val inputStream = ...
val byteOutputStream = ByteArrayOutputStream()
inputStream.use { input ->
byteOutputStream.use { output ->
input.copyTo(output)
}
}
val byteInputStream = ByteArrayInputStream(byteOutputStream.toByteArray())
If you need to read the byteInputStream multiple times, call byteInputStream.reset() before reading again.
https://code.luasoftware.com/tutorials/kotlin/how-to-clone-inputstream/
Cloning an input stream might not be a good idea, because this requires deep knowledge about the details of the input stream being cloned. A workaround for this is to create a new input stream that reads from the same source again.
So using some Java 8 features this would look like this:
public class Foo {
private Supplier<InputStream> inputStreamSupplier;
public void bar() {
procesDataThisWay(inputStreamSupplier.get());
procesDataTheOtherWay(inputStreamSupplier.get());
}
private void procesDataThisWay(InputStream) {
// ...
}
private void procesDataTheOtherWay(InputStream) {
// ...
}
}
This method has the positive effect that it will reuse code that is already in place - the creation of the input stream encapsulated in inputStreamSupplier. And there is no need to maintain a second code path for the cloning of the stream.
On the other hand, if reading from the stream is expensive (because a it's done over a low bandwith connection), then this method will double the costs. This could be circumvented by using a specific supplier that will store the stream content locally first and provide an InputStream for that now local resource.
The class below should do the trick. Just create an instance, call the "multiply" method, and provide the source input stream and the amount of duplicates you need.
Important: you must consume all cloned streams simultaneously in separate threads.
package foo.bar;
import java.io.IOException;
import java.io.InputStream;
import java.io.PipedInputStream;
import java.io.PipedOutputStream;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class InputStreamMultiplier {
protected static final int BUFFER_SIZE = 1024;
private ExecutorService executorService = Executors.newCachedThreadPool();
public InputStream[] multiply(final InputStream source, int count) throws IOException {
PipedInputStream[] ins = new PipedInputStream[count];
final PipedOutputStream[] outs = new PipedOutputStream[count];
for (int i = 0; i < count; i++)
{
ins[i] = new PipedInputStream();
outs[i] = new PipedOutputStream(ins[i]);
}
executorService.execute(new Runnable() {
public void run() {
try {
copy(source, outs);
} catch (IOException e) {
e.printStackTrace();
}
}
});
return ins;
}
protected void copy(final InputStream source, final PipedOutputStream[] outs) throws IOException {
byte[] buffer = new byte[BUFFER_SIZE];
int n = 0;
try {
while (-1 != (n = source.read(buffer))) {
//write each chunk to all output streams
for (PipedOutputStream out : outs) {
out.write(buffer, 0, n);
}
}
} finally {
//close all output streams
for (PipedOutputStream out : outs) {
try {
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
Enhancing the #Anthony Accioly with the example.
InputStream: Clones the bytes-Stream and provides number of copies as a List Collection.
public static List<InputStream> multiplyBytes(InputStream input, int cloneCount) throws IOException {
List<InputStream> copies = new ArrayList<InputStream>();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
copy(input, baos);
for (int i = 0; i < cloneCount; i++) {
copies.add(new ByteArrayInputStream(baos.toByteArray()));
}
return copies;
}
// IOException - If reading the Reader or Writing into the Writer goes wrong.
public static void copy(Reader in, Writer out) throws IOException {
try {
char[] buffer = new char[1024];
int nrOfBytes = -1;
while ((nrOfBytes = in.read(buffer)) != -1) {
out.write(buffer, 0, nrOfBytes);
}
out.flush();
} finally {
close(in);
close(out);
}
}
Reader: Clones the chars-Stream and provides number of copies as a List Collection.
public static List<Reader> multiplyChars(Reader reader, int cloneCOunt) throws IOException {
List<Reader> copies = new ArrayList<Reader>();
BufferedReader bufferedInput = new BufferedReader(reader);
StringBuffer buffer = new StringBuffer();
String delimiter = System.getProperty("line.separator");
String line;
while ((line = bufferedInput.readLine()) != null) {
if (!buffer.toString().equals(""))
buffer.append(delimiter);
buffer.append(line);
}
close(bufferedInput);
for (int i = 0; i < cloneCOunt; i++) {
copies.add(new StringReader(buffer.toString()));
}
return copies;
}
public static void copy(InputStream in, OutputStream out) throws IOException {
try {
byte[] buffer = new byte[1024];
int nrOfBytes = -1;
while ((nrOfBytes = in.read(buffer)) != -1) {
out.write(buffer, 0, nrOfBytes);
}
out.flush();
} finally {
close(in);
close(out);
}
}
Full Example:
public class SampleTest {
public static void main(String[] args) throws IOException {
String filePath = "C:/Yash/StackoverflowSSL.cer";
InputStream fileStream = new FileInputStream(new File(filePath) );
List<InputStream> bytesCopy = multiplyBytes(fileStream, 3);
for (Iterator<InputStream> iterator = bytesCopy.iterator(); iterator.hasNext();) {
InputStream inputStream = (InputStream) iterator.next();
System.out.println("Byte Stream:"+ inputStream.available()); // Byte Stream:1784
}
printInputStream(bytesCopy.get(0));
//java.sql.Clob clob = ((Clob) getValue(sql)); - clob.getCharacterStream();
Reader stringReader = new StringReader("StringReader that reads Characters from the specified string.");
List<Reader> charsCopy = multiplyChars(stringReader, 3);
for (Iterator<Reader> iterator = charsCopy.iterator(); iterator.hasNext();) {
Reader reader = (Reader) iterator.next();
System.out.println("Chars Stream:"+reader.read()); // Chars Stream:83
}
printReader(charsCopy.get(0));
}
// Reader, InputStream - Prints the contents of the reader to System.out.
public static void printReader(Reader reader) throws IOException {
BufferedReader br = new BufferedReader(reader);
String s;
while ((s = br.readLine()) != null) {
System.out.println(s);
}
}
public static void printInputStream(InputStream inputStream) throws IOException {
printReader(new InputStreamReader(inputStream));
}
// Closes an opened resource, catching any exceptions.
public static void close(Closeable resource) {
if (resource != null) {
try {
resource.close();
} catch (IOException e) {
System.err.println(e);
}
}
}
}
Related
I want to show the percentage while copying file by using binary stream but I don't know the way, that How to do it?
Below is my code.
public static void binaryStream() throws IOException {
try {
FileInputStream inputStream = new FileInputStream(new File("Untitled.png"));
FileOutputStream outputStream = new FileOutputStream(new File("Untitled-copied.png"));
int data;
while ((data = inputStream.read()) >= 0) {
outputStream.write(data);
}
outputStream.write(data);
inputStream.close();
outputStream.close();
} catch (FileNotFoundException e) {
System.out.println("Error");
} catch (IOException e) {
System.out.println("Error");
}
}
Example of how to do it like other people mentioned in the comments.
import java.io.*;
public class BinaryStream {
public static void binaryStream(String file1, String file2) throws Exception
{
File sourceFile = new File(file1);
try(
FileInputStream inputStream = new FileInputStream(sourceFile);
FileOutputStream outputStream = new FileOutputStream(new File(file2))
) {
long lenOfFile = sourceFile.length();
long currentBytesWritten = 0;
int data;
while ((data = inputStream.read()) != -1) {
outputStream.write(data);
currentBytesWritten += 1;
System.out.printf("%2.2f%%%n",
100*((double)currentBytesWritten)/((double)lenOfFile));
}
}
}
public static void main(String args[]) throws Exception {
binaryStream("Untitled.png", "Untitled-copied.png");
}
}
Note that I've made some changes:
Removed the extra outputStream.write() call you had that was writing extra content incorrectly
Using try-with-resources idiom to close the streams you open even on exceptions
Throw the exceptions instead of catching, as you shouldn't catch them if you can't handle them
Compare to -1, as that is the documented value for end of file (end of stream)
Output is like this on my computer:
0,06%
// removed data
99,89%
99,94%
100,00%
Note also that this code will print something after each byte written, so it is highly inefficient. You might want to do that less often. On that note, you're reading and writing one byte at a time, which is also very inefficient - you might want to use read(byte[]) instead, reading in chunks. Example of that, using 256 byte array:
import java.io.*;
public class BinaryStream {
public static void binaryStream(String file1, String file2) throws Exception {
File sourceFile = new File(file1);
try(
FileInputStream inputStream = new FileInputStream(sourceFile);
FileOutputStream outputStream = new FileOutputStream(new File(file2))
) {
long lenOfFile = sourceFile.length();
long bytesWritten = 0;
int amountOfBytesRead;
byte[] bytes = new byte[256];
while ((amountOfBytesRead = inputStream.read(bytes)) != -1) {
outputStream.write(bytes, 0, amountOfBytesRead);
bytesWritten += amountOfBytesRead;
System.out.printf("%2.2f%%%n",
100*((double)bytesWritten)/((double)lenOfFile));
}
}
}
public static void main(String args[]) throws Exception {
binaryStream("Untitled.png", "Untitled-copied.png");
}
}
Output on my computer:
14,69%
29,37%
44,06%
58,75%
73,44%
88,12%
100,00%
Note that in the first example, return value of .read() is actually the byte that was read, whereas in the second example, return value of .read() is the amount of bytes read and the actual bytes go into the byte array.
I am messing about with some code and was wondering is there a way to order the output in an ascending/descending order using the fileOutputStream?
code:
public static void main(String[] args) throws IOException
{
String directory = "C:\\Users\\xxxx\\Desktop\\Files\\ex1.txt";
String output = "C:\\Users\\xxxxx\\Desktop\\Files\\ex1_temp.txt";
BufferedInputStream readFile = null;
BufferedOutputStream writeFile = null;
try {
readFile = new BufferedInputStream(new FileInputStream(directory));
writeFile = new BufferedOutputStream(new FileOutputStream(output));
int data;
while ((data = readFile.read()) != -1) {
//System.out.println(data);
//Collections.sort(data);
writeFile.write(data);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
if (readFile != null)
readFile.close();
if (writeFile != null)
writeFile.close();
}
}
Generally, you need to have the data in memory to sort them, so you can't use streams well for that.
If you need to sort large data, you can use External sorting. While implementing such algorithm, you'll probably end up using streams (to read the original file in smaller chunks etc.), but streams alone won't help you here, they're merely part of the solution.
Here is my source code. I got the reading part but need a simple logic for my writing part which I"m not getting. Here in my current logic, data gets overwritten and I'm always able to see the last block of read data in my written file.
import java.io.File;
import java.io.FileInputStream;
java.io.FileNotFoundException;
java.io.FileOutputStream;
import java.io.IOException;
public class LoadTest
{
public void readFiles(File file) throws FileNotFoundException
{
int fsize = (int) file.length();
int part = (fsize/4)+(fsize%4);
byte[] block = new byte[part];
FileInputStream fin = new FileInputStream(file);
try
{
int val=-1;
do
{
int bytesread =0;
while(bytesread<part)
{
val = fin.read(block, bytesread, part-bytesread);
if (val<0)
break;
bytesread += val;
}
writeFiles(block,bytesread);
}
while(val>=0);
fin.close();
}
catch(IOException e)
{
e.printStackTrace();
}
}
public void writeFiles(byte[] block, int len) throws IOException
{
int byteswritten = 0;
FileOutputStream fout = new FileOutputStream("input.csv");
fout.write(block, byteswritten, len+byteswritten);
byteswritten +=len;
fout.close();
}
public static void main(String[] args) throws FileNotFoundException
{
LoadTest testobj = new LoadTest();
String folder = "/Users/NiranjanSubramanian/Desktop/TestFiles";
File dir = new File(folder);
File[] files = dir.listFiles();
System.out.println("Started");
for( File entry: files)
{
testobj.readFiles(entry);
}
System.out.println("Ended");
}
}
See my comments for how to solve the issue in a simple manner. However, let me suggest to you a simple alternative to do what you're asking.
final Path dir = Paths.get("/Users/NiranjanSubramanian/Desktop/TestFiles");
try (final OutputStream out = Files.newOutputStream(Paths.get("input.csv"))) {
for (final Path file : Files.newDirectoryStream(dir)) {
Files.copy(file, out);
}
}
This relies on Java 7's new file API but is (at least in my opinion) a far cleaner solution.
There are some libs that handle it for you, or even NIO, but the simplest way to do it is the following:
int read = 0;
byte[] buff = new byte[1024];
FileInputStream fis = new FileInputStream(yourInputFile);
FileOutputStream fos = new FileOutputStream(yourOutputFile);
while((read = fis.read(buff)) >= 0){
fos.write(buff, 0, read);
}
fos.flush();
fos.close();
fis.close();
Open the file in append mode.. your code will override since the default is overwrite, not append.
To append you need to pass the append parameter as true.
change
FileOutputStream fout = new FileOutputStream("input.csv");
to
FileOutputStream fout = new FileOutputStream("input.csv", true);
How to copy streams in Java:
byte[] buffer = new byte[8192]; // or whatever you like really
int count;
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Note that you don't need the entire input in memory before you start writing, so you are saving both time and space; and handling partial reads including the probable final one is trivially easy.
In Java, is there a way of reading a file (text file) in a way that it would only read one character at a time, rather than String by String. This is for the purpose of an extremely basic lexical analyzer, so you can understand why I'd want such a method. Thank you.
Here's a sample code for reading / writing one character at a time
public class CopyCharacters {
public static void main(String[] args) throws IOException {
FileReader inputStream = null;
FileWriter outputStream = null;
try {
inputStream = new FileReader("xanadu.txt");
outputStream = new FileWriter("characteroutput.txt");
int c;
while ((c = inputStream.read()) != -1) {
outputStream.write(c);
}
} finally {
if (inputStream != null) {
inputStream.close();
}
if (outputStream != null) {
outputStream.close();
}
}
}
}
Note, this answer was updated to copy the sample code from the Ref link, but I see this is essentially the same answer given below.
ref:
http://download.oracle.com/javase/tutorial/essential/io/charstreams.html
You can use the read method from the InputStreamReader class which reads one character from the stream and returns -1 when it reaches the end of the stream
public static void processFile(File file) throws IOException {
try (InputStream in = new FileInputStream(file);
Reader reader = new InputStreamReader(in)) {
int c;
while ((c = reader.read()) != -1) {
processChar((char) c); // this method will do whatever you want
}
}
}
You can read the whole file (if it is not much big) in the memory as string, and iterate on the string character by character
There are several possible solutions. Generally you can use any Reader from java.io package for reading characters, e.g.:
// Read from file
BufferedReader reader = new BufferedReader(new FileReader("file.txt"));
// Read from sting
BufferedReader reader = new BufferedReader(new StringReader("Some text"));
Is there a way to read a ByteBuffer with a BufferedReader without having to turn it into a String first? I want to read through a fairly large ByteBuffer as lines of text and for performance reasons I want to avoid writing it to the disk. Calling toString on the ByteBuffer doesn't work because the resulting String is too large (it throws java.lang.OutOfMemoryError: Java heap space). I would have thought there would be something in the API to wrap a ByteBuffer in a suitable reader, but I can't seem to find anything suitable.
Here's an abbreviated code sample the illustrates what I am doing):
// input stream is from Process getInputStream()
public String read(InputStream istream)
{
ReadableByteChannel source = Channels.newChannel(istream);
ByteArrayOutputStream ostream = new ByteArrayOutputStream(bufferSize);
WritableByteChannel destination = Channels.newChannel(ostream);
ByteBuffer buffer = ByteBuffer.allocateDirect(writeBufferSize);
while (source.read(buffer) != -1)
{
buffer.flip();
while (buffer.hasRemaining())
{
destination.write(buffer);
}
buffer.clear();
}
// this data can be up to 150 MB.. won't fit in a String.
result = ostream.toString();
source.close();
destination.close();
return result;
}
// after the process is run, we call this method with the String
public void readLines(String text)
{
BufferedReader reader = new BufferedReader(new StringReader(text));
String line;
while ((line = reader.readLine()) != null)
{
// do stuff with line
}
}
It's not clear why you're using a byte buffer to start with. If you've got an InputStream and you want to read lines for it, why don't you just use an InputStreamReader wrapped in a BufferedReader? What's the benefit in getting NIO involved?
Calling toString() on a ByteArrayOutputStream sounds like a bad idea to me even if you had the space for it: better to get it as a byte array and wrap it in a ByteArrayInputStream and then an InputStreamReader, if you really have to have a ByteArrayOutputStream. If you really want to call toString(), at least use the overload which takes the name of the character encoding to use - otherwise it'll use the system default, which probably isn't what you want.
EDIT: Okay, so you really want to use NIO. You're still writing to a ByteArrayOutputStream eventually, so you'll end up with a BAOS with the data in it. If you want to avoid making a copy of that data, you'll need to derive from ByteArrayOutputStream, for instance like this:
public class ReadableByteArrayOutputStream extends ByteArrayOutputStream
{
/**
* Converts the data in the current stream into a ByteArrayInputStream.
* The resulting stream wraps the existing byte array directly;
* further writes to this output stream will result in unpredictable
* behavior.
*/
public InputStream toInputStream()
{
return new ByteArrayInputStream(array, 0, count);
}
}
Then you can create the input stream, wrap it in an InputStreamReader, wrap that in a BufferedReader, and you're away.
You can use NIO, but there's no real need here. As Jon Skeet suggested:
public byte[] read(InputStream istream)
{
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[1024]; // Experiment with this value
int bytesRead;
while ((bytesRead = istream.read(buffer)) != -1)
{
baos.write(buffer, 0, bytesRead);
}
return baos.toByteArray();
}
// after the process is run, we call this method with the String
public void readLines(byte[] data)
{
BufferedReader reader = new BufferedReader(new InputStreamReader(new ByteArrayInputStream(data)));
String line;
while ((line = reader.readLine()) != null)
{
// do stuff with line
}
}
This is a sample:
public class ByteBufferBackedInputStream extends InputStream {
ByteBuffer buf;
public ByteBufferBackedInputStream(ByteBuffer buf) {
this.buf = buf;
}
public synchronized int read() throws IOException {
if (!buf.hasRemaining()) {
return -1;
}
return buf.get() & 0xFF;
}
#Override
public int available() throws IOException {
return buf.remaining();
}
public synchronized int read(byte[] bytes, int off, int len) throws IOException {
if (!buf.hasRemaining()) {
return -1;
}
len = Math.min(len, buf.remaining());
buf.get(bytes, off, len);
return len;
}
}
And you can use it like this:
String text = "this is text"; // It can be Unicode text
ByteBuffer buffer = ByteBuffer.wrap(text.getBytes("UTF-8"));
InputStream is = new ByteBufferBackedInputStream(buffer);
InputStreamReader r = new InputStreamReader(is, "UTF-8");
BufferedReader br = new BufferedReader(r);