When I start with an unbounded Stream and use the limit() method to put a bound on it, how can I clean up the resources that were used by the Stream once the limit is reached? For example, if I'm trying to do what the Files.lines method does, but with floats instead of strings, I would write a function that looks something like this:
public static Stream<Float> floats(File f) throws FileNotFoundException {
FileInputStream fis = new FileInputStream(f);
return Stream.generate(() -> {
byte[] buff = new byte[4];
try { fis.read(buff); }
catch (Exception e) { e.printStackTrace(); }
return ByteBuffer.wrap(buff).getFloat(0);
}).limit(f.length()/4);
}
This should stream a large binary file full of floats until I reach the end of the file. I divide the length of the file in bytes by the four bytes that make up a float. However, I'd like to find some way to execute fis.close() once this limit is reached. Does the Streams API have any way of letting you do this?
I came up with another possibility. It's kind of a hack, but it seems to work. I just run the Stream through a filter that always returns true, but in the process of doing so, it checks for EOF and closes the input stream if it's reached. I was hoping there was a cleaner way to do this built into the API.
Stream<Float> answer;
FileInputStream fis = new FileInputStream(f);
answer = Stream
.generate(() -> {
byte[] buff = new byte[4];
try { fis.read(buff); }
catch (Exception e) { e.printStackTrace(); }
return ByteBuffer.wrap(buff).getFloat(0);})
.filter(x -> {
try { if (fis.available() == 0) fis.close(); }
catch (Exception e) { e.printStackTrace(); }
return true;})
.limit(f.length()/4);
return answer;
I’d use an entirely different approach:
public static Stream<Float> floats(File f) throws FileNotFoundException {
try(FileChannel fch = FileChannel.open(f.toPath(), StandardOpenOption.READ)) {
ByteBuffer bb = fch.map(FileChannel.MapMode.READ_ONLY, 0, fch.size());
FloatBuffer fb = bb.asFloatBuffer();
return IntStream.range(0, fb.remaining()).mapToObj(fb::get);
}
catch(IOException ex) { // or consider to declare "throws IOException" instead
FileNotFoundException fnfe = new FileNotFoundException(ex.getMessage());
fnfe.initCause(ex);
throw fnfe;
}
}
with this, the FileChannel has been closed already when the method returns. It’s also potentially more efficient. Note that for a lot of operations, it might be more efficient to process the data as DoubleStream, even when the source values are floats and only narrow the type of the final result back to float, if necessary:
public static DoubleStream floatsAsDouble(File f) throws FileNotFoundException {
try(FileChannel fch = FileChannel.open(f.toPath(), StandardOpenOption.READ)) {
ByteBuffer bb = fch.map(FileChannel.MapMode.READ_ONLY, 0, fch.size());
FloatBuffer fb = bb.asFloatBuffer();
return IntStream.range(0, fb.remaining()).mapToDouble(fb::get);
}
catch(IOException ex) { // or consider to declare throwsIOException instead
FileNotFoundException fnfe = new FileNotFoundException(ex.getMessage());
fnfe.initCause(ex);
throw fnfe;
}
}
…
float sum=(float)floatsAsDouble(new File("path")).sum();
Also, you may consider to directly work with Paths instead of File.
Related
I need to read a binary file and save each byte into a byte array. I've read other stackoverflow posts on this topic, but cannot figure out why mine does not work. Here is what I have:
String fileOne = "file1.bin";
byte[] byteArray = new byte[1000];
try{
FileInputStream fileIS = new FileInputStream(fileOne);
ObjectInputStream is = new ObjectInputStream(fileIS);
is.read(byteArray);
is.close();
for(int i =0; i < byteArray.length; i++){
System.out.println(byteArray[i]);
}
}
catch (FileNotFoundException e){
e.toString();
System.exit(0);
}
catch (IOException io){
io.toString();
System.exit(0);
}
Here's a way to read the contents of a file into a byte array. FileInputStream is all you need – leave ObjectInputStream out of it (unless you are explicitly dealing with data that was created from an ObjectOutputStream, but that doesn't seem to be the case since you are calling println() on each byte).
public static void main(String[] args) {
String filename = "file1.bin";
try (FileInputStream fis = new FileInputStream(filename)) {
byte[] bytes = fis.readAllBytes();
for (byte b : bytes) {
System.out.print(b);
}
} catch (Exception e) {
e.printStackTrace();
}
}
A few things here:
omit using ObjectInputStream – not needed for reading byte data, and won't work unless the data was created by the corresponding output stream. From the Javadoc: "An ObjectInputStream deserializes primitive data and objects previously written using an ObjectOutputStream. "
use try with resources – it will close the associated stream for you
catch Exception – in the code you posted, you will only see info if FileNotFoundException or IOException is thrown. For anything else, your code doesn't handle them or print out any info.
I have a file with some info how can I read all info?
Name names;
try (FileInputStream fileInputStream = new FileInputStream(file)) {
ObjectInputStream objectInputStream = new ObjectInputStream(fileInputStream);
names = (Name) objectInputStream.readObject();
} catch (IOException | ClassNotFoundException e) {
e.printStackTrace();
}
You have several solution, all depending on the input:
You can iterate until the stream is fully consumed: I think that is the worse solution out of those I provide you. It is worse because you are checking if EOF was reached, whilst you should know when you're done (eg: your file format is wrong).
Set<Name> result = new HashSet<>();
try {
for (;;) {
result.add((Name)objectInputStream.readObject());
}
} catch (EOFException e) {
// End of stream
}
return result;
When producing the input, serialize a collection and invoke readObject() on it. Serialization should be able to read the collection, as long as each object implements Serializable.
static void write(Path path, Set<Name> names) throws IOException {
try (OutputStream os = Files.newOutputStream(path);
ObjectOutputStream oos = new ObjectOutputStream(os)) {
oos.writeObject(names);
}
}
static Set<Name> read(Path path) throws IOException {
try (InputStream is = Files.newInputStream(path);
ObjectInputStream ois = new ObjectInputStream(is)) {
// WARN Files.newInputStream is not buffered; ObjectInputStream might
// be buffered (I don't remember).
return (Set<Name>) ois.readObject();
}
}
When producing the input, you can add a int indicating the number of object to read, and iterate over it: this is useful in case where you don't really care of the collection (HashSet). The resulting file will be smaller (because you won't have the HashSet metadata).
int result = objectInputStream.readInt();
Name[] names = new Name[result]; // do some check on result!
for (int i = 0; i < result; ++i) {
names[i] = (Name) objectInputStream.readObject();
}
Also, Set are good, but since they remove duplicate using hashCode()/equals() you may get less object if your definition of equals/hashCode changed after the fact (example: your Name was case sensitive and now it is not, eg: new Name("AA").equals(new Name("aa"))).
I am messing about with some code and was wondering is there a way to order the output in an ascending/descending order using the fileOutputStream?
code:
public static void main(String[] args) throws IOException
{
String directory = "C:\\Users\\xxxx\\Desktop\\Files\\ex1.txt";
String output = "C:\\Users\\xxxxx\\Desktop\\Files\\ex1_temp.txt";
BufferedInputStream readFile = null;
BufferedOutputStream writeFile = null;
try {
readFile = new BufferedInputStream(new FileInputStream(directory));
writeFile = new BufferedOutputStream(new FileOutputStream(output));
int data;
while ((data = readFile.read()) != -1) {
//System.out.println(data);
//Collections.sort(data);
writeFile.write(data);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
if (readFile != null)
readFile.close();
if (writeFile != null)
writeFile.close();
}
}
Generally, you need to have the data in memory to sort them, so you can't use streams well for that.
If you need to sort large data, you can use External sorting. While implementing such algorithm, you'll probably end up using streams (to read the original file in smaller chunks etc.), but streams alone won't help you here, they're merely part of the solution.
I am trying merge n pieces of file become single file. But I got strange behavior on my function. The function are called for x times in n seconds. Let say I have 100 files which I will merge, every second I call 5 files and merger it. and in the next second the amount is double to be 10, but from 1-5 is the same file as before the rest is new file. It work normal but in some point, its give zero byte or sometime give the right size.
Could you help me spot the mistake in my function below?
public void mergeFile(list<String> fileList, int x) {
int count = 0;
BufferedOutputStream out = null;
try {
out = new BufferedOutputStream(new FileOutputStream("Test.doc"));
for (String file : fileList) {
InputStream in = new BufferedInputStream(new FileInputStream(file));
byte[] buff = new byte[1024];
in.read(buff);
out.write(buff);
in.close();
count++;
if (count == x) {
break;
}
}
out.flush();
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
*sorry for my English
in.read(buff);
Check the Javadoc. That method isn't guaranteed to fill the buffer. It returns a value which tells you how many bytes it read. You're supposed to use that, and in this situation you are supposed to use it when deciding how many bytes, if any, to write.
You do not read the full file, you read from each file only up to 1024 bytes. You need to loop the read as long as it returns data (or use something like Files.copy().
BTW: you dont need a BufferedOutputStream if you copy with large buffers.
public void mergeFile(list<String> fileList, int x) throws IOException {
try (OutputStream out = new FileOutputStream("Test.doc");) {
int count=0;
for (String file : fileList) {
Files.copy(new File(file).toPath(), out);
count++;
if (count == x) {
break;
}
}
}
}
You also do not need to flush() if you close. I am using "try-with-resource" here, so I dont need to close it explicitely. It is best to propagate the exceptions.
I have a method that reads text from a file; decompression may be required, depending on an input parameter:
public static String readText(File inFile, boolean compressed) {
InputStream in = null;
InputStreamReader isr = null;
StringBuilder sb = new StringBuilder();//constant resizing is costly, so set the STRING_SIZE
try {
in = new FileInputStream(inFile);
if (compressed) {
in = new GZIPInputStream(in);
}
isr = new InputStreamReader(in);
int length = 0;
char[] cbuf = new char[8 * 1024];
while ((length = isr.read(cbuf)) != -1) {
sb.append(cbuf, 0, length);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
in.close();
} catch (Exception e1) {
e1.printStackTrace();
}
}
return sb.toString();
}
It was suggested that I use InputStream like this so it is easier to write, and so that in the end I only have to close one thing. I am still a bit worried this might cause a memory leak. So my question is: does anyone knows if the code above is OK? Or do I have to get back to a dozen of streams and close them one by one in a finally block?
Thanks a lot.
Yes, closing the outermost stream/reader is sufficient.
However, your code has another potential bug: new InputStreamReader(in) will use the platform default encoding, which depends on the OS region/language settings. You should specify the encoding of the text file and use it explicitly in the constructor.
Here's one point to add: see if 'in' is null before calling 'in.close()' as the exception could happen without the first assignment succeeding.
Also, it's good form to only catch possible exceptions (e.g. IOException). That way if you add more code and the IDE tells you that a new exception type isn't handled you can add the proper specific code rather than never hearing about it because the catch (Exception ) which was originally for IOException is also (mishandling?) every other type.
Here's the clean Java 7 way which works for anything that implements AutoCloseable/Closeable:
try (InputStream in = compressed ?
new GZIPInputStream(new FileInputStream(inFile))
: new FileInputStream(inFile);
InputStreamReader isr = new InputStreamReader(in))
{
int length = 0;
char[] cbuf = new char[8 * 1024];
while ((length = isr.read(cbuf)) != -1) {
sb.append(cbuf, 0, length);
}
}
catch (Exception e) {
e.printStackTrace();
}
If you're wondering what happens if there's an exception while closing the resource, read about getSuppressedExceptions() which was also added.