I have an OutputStream that I needed to read from, and so I used the following (Groovy) code to get an InputStream reference to the data:
PipedInputStream inputStream = new PipedInputStream()
PipedOutputStream outputStream = new PipedOutputStream(inputStream)
new Thread(
new Runnable() {
public void run() {
// Some API method
putDataInOutputStream(outputStream)
outputStream.close()
}
}
).start()
handler.process(inputStream)
In this case, handler is some class that implements an interface which has this method:
public void process(InputStream stream);
The problem that came up in our new requirements was that there was some pre-processing on the stream, and therefore I need to read the stream at least twice in the handler.process() method. Here's some example code from one implementation:
public void process(InputStream stream) {
def bufferedStream = new BufferedInputStream(stream, 30 * 1048576) // 30 MB
bufferedStream.mark(Integer.MAX_VALUE)
parseMetadata(bufferedStream)
bufferedStream.reset()
doTheThingYouDo(bufferedStream)
}
I know that for some input I am not hitting the 30 MB limit or the Integer.MAX_VALUE buffer size. However, I'm always getting the following exception:
java.io.IOException: Stream closed
Is this even possible? I think the problem is the thread closing on the PipedOutputStream, but I don't know how to prevent that or if perhaps I'm creating more problems by being a novice at Java Stream IO.
My best guess is that your parseMetadata somehow closed the stream. I've tried your scenario, and it works fine for me. In general, closing the OutputStream before your handler is done reading is not the problem, that's exactly what the piped streams are for.
Besides, given your situation, I would leave out the piping, and the additional thread. If you don't mind having your entire stream in memory, you can do something like
ByteArrayOutputStream out = new ByteArrayOutputStream();
fillTheOutput(out);
ByteArrayInputStream in = new ByteArrayInputStream(out.toByteArray());
pass1(in);
in.reset();
pass2(in);
If you do mind having everything in memory, you're in trouble anyway, since your BufferedInputStream does roughly the same thing.
edit: Note that you can easily build a new ByteArrayInputStream based on the byte array, which is something you cannot do with regular streams.
Related
public static void main(String[] args) throws Exception
{
// Server sends 3 numbers to the client
ByteArrayOutputStream bos = new ByteArrayOutputStream();
bos.write(1000);
bos.write(2000);
bos.write(3000);
// Client receive the bytes
final byte[] bytes = bos.toByteArray();
ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
System.out.println(bis.read());
System.out.println(bis.read());
System.out.println(bis.read());
}
The code above is breaking because bis.read() returns an int in the range 0 to 255
How can I receive those numbers properly? Should I use a delimiter and keep reading the stream until I find it? If so, what if I'm sending multiple files, I think if the delimiter as a single byte it could matched somewhere in the file and also break.
Use decorators for your streams!
All you have to do is to wrap your Output- and InputStream by java.io.ObjectOutputStream / and java.io.ObjectInputStream. These classes support writing and reading ints (a 4-byte value) with a single method call to writeInt/readInt.
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream os = new ObjectOutputStream(bos);
os.writeInt(1000);
os.writeInt(2000);
os.writeInt(3000);
os.close();
// Client receive the bytes
final byte[] bytes = bos.toByteArray();
ObjectInputStream is = new ObjectInputStream(new ByteArrayInputStream(bytes));
System.out.println(is.readInt());
System.out.println(is.readInt());
System.out.println(is.readInt());
Don't forget to close the streams. Use try/finally or try-with-resources.
Byte stream is a stream of bytes. So if you're reading stream and want to differentiate between different parts of the stream then you should "create" some sort of protocol.
Here are some ideas that can be relevant:
Use delimiter as you've stated by yourself, If you're concerned about the length - do not one byte length, but something more unique - something that you're sure you won't see in the parts themselves.
At the beginning of the part allocate N bytes (2-4 or maybe more, depending on data) and write the size of the part that will follow.
So that when you create the stream (writer), before actually streaming the "part" - calculate its size and encode it. This is a protocol between reader and writer.
When you read - read the size (=N bytes for example), and then read N bytes. Now you know that the part is ended, and the next part (again, size + content) will follow
Can you try ByteBuffer class?
ByteStream is just a stream of bytes. It doesn't understand integer which actually needs more than one byte. If you print bytes.length it will return you 3. Surely you need more bytes than that. Allocate 4 bytes before you write an integer and then write to it. Check out this class above. Hope that helps!
I have a piece of code which uses the deflate algorithm to compress a file:
public static File compressOld(File rawFile) throws IOException
{
File compressed = new File(rawFile.getCanonicalPath().split("\\.")[0]
+ "_compressed." + rawFile.getName().split("\\.")[1]);
InputStream inputStream = new FileInputStream(rawFile);
OutputStream compressedWriter = new DeflaterOutputStream(new FileOutputStream(compressed));
byte[] buffer = new byte[1000];
int length;
while ((length = inputStream.read(buffer)) > 0)
{
compressedWriter.write(buffer, 0, length);
}
inputStream.close();
compressedWriter.close();
return compressed;
}
However, I'm not happy with the OutputStream copying loop since it's the "outdated" way of writing to streams. Instead, I want to use a Java 7 API method such as Files.copy:
public static File compressNew(File rawFile) throws IOException
{
File compressed = new File(rawFile.getCanonicalPath().split("\\.")[0]
+ "_compressed." + rawFile.getName().split("\\.")[1]);
OutputStream compressedWriter = new DeflaterOutputStream(new FileOutputStream(compressed));
Files.copy(compressed.toPath(), compressedWriter);
compressedWriter.close();
return compressed;
}
The latter method however does not work correctly, the compressed file is messed up and only a few bytes are copied. How come?
I see mainly two problems.
You copy from the target instead of the source. I think the copying has to be changed to Files.copy(rawFile.toPath(), compressedWriter);.
The Javadoc of copy says: "Note that if the given output stream is Flushable then its flush method may need to invoked after this method completes so as to flush any buffered output." So, you have to call the flush-method of the OutputStream after copy.
Additionally there is one more point. The Javadoc of copy says:
It is strongly recommended that the output stream be promptly closed if an I/O error occurs.
You can close the OutputStream in a finally-block to make sure it happens in case of an error. Another possibility is to use try with resources that was introduced in Java 7.
//Reading a image file from #drawable res folder and writing to a file on external sd card
//below one works no doubt but I want to imrpove it:
OutputStream os = new FileOutputStream(file); //File file.........
InputStream is =getResources().openRawResource(R.drawable.an_image);
byte[] b = new byte[is.available()];
is.read(b);
os.write(b);
is.close();
os.close();
In above code I am using basic io classes to read and write. My question is what can I do in order to able to use wrapper classes like say DataInputStream/ BufferedReaderd or PrintStream / BufferedWriter /PrintWriter.
As openRawResources(int id ) returns InputStream ;
to read a file from res I either need to typecast like this:
DataInputStream is = (DataInputStream) getResources().openRawResource(R.drawble.an_image));
or I can link the stream directly like this:
DataInputStream is = new DataInputStream(getResources().openRawResource(R.drawable.greenball));
and then I may do this to write it to a file on sd card:
PrintStream ps =new PrintStream (new FileOutputStream(file));
while(s=is.readLine()!=null){
ps.print(s);
}
So is that correct approach ? which one is better? Is there a better way?better practice..convention?
Thanks!!!
If openRawResource() is documented to return an InputStream then you cannot rely on that result to be any more specific kind of InputStream, and in particular, you cannot rely on it to be a DataInputStream. Casting does not change that; it just gives you the chance to experience interesting and exciting exceptions. If you want a DataInputStream wrapping the the result of openRawResource() then you must obtain it via the DataInputStream constructor. Similarly for any other wrapper stream.
HOWEVER, do note that DataInputStream likely is not the class you want. It is appropriate for reading back data that were originally written via a DataOutputStream, but it is inappropriate (or at least offers no advantages over any other InputStream) for reading general data.
Furthermore, your use of InputStream.available() is incorrect. That method returns the number of bytes that can currently be read from the stream without blocking, which has only a weak relationship with the total number of bytes that could be read from the stream before it is exhausted (if indeed it ever is).
Moreover, your code is also on shaky ground where it assumes that InputStream.read(byte[]) will read enough bytes to fill the array. It probably will, since that many bytes were reported available, but that's not guaranteed. To copy from one stream to another, you should instead use code along these lines:
private final static int BUFFER_SIZE = 2048;
void copyStream(InputStream in, OutputStream out) throws IOException {
byte[] buffer = new byte[BUFFER_SIZE];
int nread;
while ( (nread = in.read(buffer) != 0 ) do {
out.write(buffer, 0, nread);
}
}
Desclaimer My question is different from two following links
Question 1
Question 2
public class AppendableObjectOutputStream extends ObjectOutputStream {
public AppendableObjectOutputStream(OutputStream out) throws IOException {
super(out);
}
#Override
protected void writeStreamHeader() throws IOException {}
}
The problem with above solutions is that they do not support writing multiple objects to appendable stream w/o closing the stream.
If I open appendable stream, write multiple objects - then at time of reading I can read only first object properly and on trying to read second object, I get EOF exception.
If I proceed the way like write on object to appendable stream, close stream. Then again open stream, write another object close and so on. This way I am able to read multiple objects properly.
fileOutputStream = new FileOutputStream("abc.dat",true);
outputBuffer = new BufferedOutputStream(fileOutputStream);
objectStream = new AppendableObjectOutputStream(outputBuffer);
BucketUpdate b1 = new BucketUpdate("getAllProducts1",null,"1",null);
BucketUpdate b2 = new BucketUpdate("getAllProducts2",null,"2",null);
BucketUpdate b3 = new BucketUpdate("getAllProducts3",null,"3",null);
objectStream.writeObject(b1);
objectStream.writeObject(b2);
objectStream.writeObject(b3);
objectStream.close();
Calling ObjectOutputStream.reset() after writing each object will fix this.
If you check question you mentioned, you will see that you have to use AppendableObjectOutputStream only to append objects to file, if file already contains some objects. For empty file you have to use ordinary ObjectOutputStream because the header should be written to the beginning in this case.
InputStream data = realResponse.getEntity().getContent();
byte[] preview = new byte[100];
data.read(preview, 0, 100);
// Now I want to refer to the InputStream later on, but I want it from the beginning of the stream, not 100 bytes in. I tried mark() it at 100, and then reset() after I read the first 100 bytes, but that doesn't work either.
Any ideas? Probably a stupid mistake..just not seeing it.
When you use mark() of the java.io.InputStream object you should check with the markSupported() method if your InputStream actually support using mark. According to the API the InputStream class doesn't, but the java.io.BufferedInputStream class does. Maybe you should embed your stream inside a BufferedInputStream object like:
InputStream data = new BufferedInputStream(realResponse.getEntity().getContent());
// data.markSupported() should return "true" now
data.mark(some_size);
// work with "data" now
...
data.reset();
If the InputStream supports mark (you can check with the markSupported() method), then the following should work:
InputStream data = realResponse.getEntity().getContent();
byte[] preview = new byte[100];
data.mark(100);
data.read(preview, 0, 100);
data.reset();
However, be aware that data.read(preview, 0, 100) is not guaranteed to read 100 bytes in one go, it may read less.