We Keep Coding

How to control the buffer size while reading from a inputstream

I am reading from an input stream from
FileItemIterator iter = upload.getItemIterator(request);
while (iter.hasNext()) {
FileItemStream item = iter.next();
String name = item.getFieldName();
stream = item.openStream();
}
I am trying to upload this content to AWS s3. But the problem is the number of uploadParts that it created for the file which is exceeding 10000. After which AWS s3 is throwing an exception.
I am not able to control the buffer size of the stream even though I explicitly mention the buffer size. Need a solution to control the number of disc reads of the stream that i use
stream = item.openStream()
Even when I use varying buffer size.
public String addFile(InputStream stream, FileOutputStream fout) throws IOException {
byte[] buffer = new byte[<buffer size>];
int count;
int visit=1;
try {
while ((count = stream.read(buffer)) != -1) {
if (fout != null) {
fout.write(buffer, 0, count);
}
visit++;
}
} finally {
if (fout != null) {
fout.flush();
fout.close();
}
}
System.out.println(visit);
return "";
}
The visit number remains constant even when I vary the buffer size. Need help in resolving this.

MediaCodecBufferedOutput index is negative for android MediaCodec

Following is a code snippet which is supposed to read all images from image folder and encode them to a h.262 video and store in the sdcard. I followed the android documentation (get buffer, fill buffer, queue buffer for encoding, dequeue output buffer and then write to file). Problem is when I dequeue output buffer I get negative index whereas its supposed to return the index of output data. The output file is 0 Bytes and nothing is written into it.
I am pretty much very new in mediaCodec. Any suggestion would be appreciated.
MediaCodec mediaCodec=null;
byte[] input = new byte[2000];
BufferedOutputStream outputStream = null;
try {
//TODO
//adjust parameters by consulting with hari sir
mediaCodec = MediaCodec.createEncoderByType("video/avc");
MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/avc", 320, 240);
mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, 700000);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 10);
//not all phones support given color format, if color format is not supported app will crash with mediaCodec exception
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar);
mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5);
mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
mediaCodec.start();
//after the mediaCodec is started we don't have ownership of input or output buffers
Log.i("Codecinfo",""+mediaCodec.getCodecInfo());
Log.i("Codecname",""+mediaCodec.getName());
}
catch (Exception e)
{
Log.e("ExceptionMediaCodec","Some exception in media codec");
}
//reached here
System.out.println("mediacodec info="+mediaCodec.getCodecInfo());
try {
File ff = new File(Environment.getExternalStorageDirectory(), "Download/video_encoded.264");
if (!ff.exists()) ff.createNewFile();
System.out.println("H.264 output file initialized");
outputStream = new BufferedOutputStream(new FileOutputStream(ff));
Log.i("H264 avc Encoder", "outputStream initialized");
} catch (Exception e){
e.printStackTrace();
}
String path = Environment.getExternalStorageDirectory().toString()+"/images";
File f = new File(path);
Log.i("ExternalFileInfo",path.toString());
//read image files onto an array
File[] files = f.listFiles();
System.out.println(files.getClass().getName());
int NUM_IMAGES = files.length;
String[] images = new String[NUM_IMAGES];
for (int i=0;i<NUM_IMAGES;i++)
images[i]=files[i].getName();
for (String eachimage: images) {
System.out.println(eachimage);
byte[] eachByte = eachimage.getBytes();
input = eachByte; //demo
System.out.println("input byte initialized"+input.toString());
try {
System.out.println("Following is the content of byte array input");
System.out.write(input);
}catch (Exception e)
{
e.printStackTrace();
}
}
//reached here
System.out.println("image byte size="+input.length);
//all images converted to bytearray
ByteBuffer[] outputBuffers = mediaCodec.getOutputBuffers();
//System.out.println("inputBuffers="+(inputBuffers));
//System.out.println("outputBuffers="+(outputBuffers));
//reached here
//returns the index of input buffer to be filled for encoding
int inputBufferIndex = mediaCodec.dequeueInputBuffer(-1); //-1 => wait indefinitely
System.out.println("inputBufferedIndex="+inputBufferIndex); //0
if (inputBufferIndex >= 0) {
ByteBuffer[] inputBuffers = mediaCodec.getInputBuffers();
ByteBuffer inputBuffer = inputBuffers[inputBufferIndex];
inputBuffer.clear();
System.out.println("input byte placed in input buffer");
inputBuffer.put(input);
System.out.println("inputBuffer after filling up" + inputBuffer);
mediaCodec.queueInputBuffer(inputBufferIndex, 0, input.length, System.nanoTime(), 0); //send each request with different timestamp
System.out.println("mediacodec input queued");
}
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
System.out.println("buffer info="+bufferInfo);
int outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo, -1); //-ve value for indefinite waiting
//reached here
System.out.println("buffer info meta data=" + bufferInfo);
System.out.println("outputBufferedIndex=" + outputBufferIndex);
try {
while (outputBufferIndex >= 0) {
ByteBuffer outputBuffer = outputBuffers[outputBufferIndex];
byte[] outData = new byte[bufferInfo.size];
outputBuffer.get(outData);
outputStream.write(outData, 0, outData.length);
outputStream.flush();
Log.i("AvcEncoder", outData.length + " bytes written");
mediaCodec.releaseOutputBuffer(outputBufferIndex, false);
outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo, -1);
}
} catch (Throwable t) {
t.printStackTrace();
}
try {
mediaCodec.stop();
mediaCodec.release();
outputStream.flush();
outputStream.close();
} catch (Exception e){
e.printStackTrace();
}
System.out.println("Mediacodec="+mediaCodec);

Please see the documentation for the MediaCodec.dequeueOutputBuffer() method, which says:
Returns the index of an output buffer that has been successfully decoded or one of the INFO_* constants.
The negative values are the INFO_* constants, which may be one of the following:
INFO_OUTPUT_BUFFERS_CHANGED
INFO_OUTPUT_FORMAT_CHANGED
INFO_TRY_AGAIN_LATER
The last one is not too probable because you're waiting indefinitely though.
Additionally: You can't always rely on waiting for one output buffer after you've given one single buffer as input. You need to feed input buffers as long as the encoder has got free input buffers, and consume whatever output buffers it gives you.
The last few output buffers might be output only once you signal that you won't be submitting any more input buffers, by setting the flag BUFFER_FLAG_END_OF_STREAM.

Java: How to count read bytes from InputStream without allocating the full memory before

I have a Java-backend where user can upload files to it. I want to limit these uploaded files to a max size and want to check the amount of uploaded bytes while the upload happens and break the transmission as soon as the limit is reached.
Currently I am using InputStream.available() before allocation for determination of estimated size, but that seems to be seen as unreliable.
Any suggestions?

You can use Guava's CountingInputstream or Apache IO's CountingInputStream when you want to know how many bytes have been read.
On the other hand when you want to stop the upload immediatly when reaching some limit then just count while reading chunks of bytes and close the stream when the limit has been exceeded.

You don't have to 'allocat[e] the full memory before'. Just use a normally sized buffer, say 8k, and perform the normal copy loop, tallying the total transferred. If it exceeds the quota, stop, and destroy the output file.

int count = 1;
InputStream stream;
if (stream.available() < 3) {
count++;
}
Result:
[0][1]{2][3]
1 1 1 1

If you're using a servlet and a multipart request you can do this:
public void doPost( final HttpServletRequest request, final HttpServletResponse response )
throws ServletException, IOException {
String contentLength = request.getHeader("Content-Length");
if (contentLength != null && maxRequestSize > 0 &&
Integer.parseInt(contentLength) > maxRequestSize) {
throw new MyFileUploadException("Multipart request is larger than allowed size");
}
}

My solution looks like this:
public static final byte[] readBytes (InputStream in, int maxBytes)
throws IOException {
byte[] result = new byte[maxBytes];
int bytesRead = in.read (result);
if (bytesRead > maxBytes) {
throw new IOException ("Reached max bytes (" + maxBytes + ")");
}
if (bytesRead < 0) {
result = new byte[0];
}
else {
byte[] tmp = new byte[bytesRead];
System.arraycopy (result, 0, tmp, 0, bytesRead);
result = tmp;
}
return result;
}
EDIT:
New variant
public static final byte[] readBytes (InputStream in, int bufferSize, int maxBytes)
throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buffer = new byte[bufferSize];
int bytesRead = in.read (buffer);
out.write (buffer, 0, bytesRead);
while (bytesRead >= 0) {
if (maxBytes > 0 && out.size() > maxBytes) {
String message = "Reached max bytes (" + maxBytes + ")";
log.trace (message);
throw new IOException (message);
}
bytesRead = in.read (buffer);
if (bytesRead < 0)
break;
out.write (buffer, 0, bytesRead);
}
return out.toByteArray();
}

All method implementations of read return the number of bytes read. So you can initiate a counter and increment it appropriately with each read to see how many bytes you've reads so far. Method available() allows you to see how many bytes are available for reading at the buffer at the moment and it has no relation to the total size of the file. this method could be very useful though to optimize your reading so each time you can request to read the chunk that is readily available and avoid blocking. Also in your case you can predict before reading if the amount of bytes that you will have after the upcoming reading will exceed your limit and thus you can cancel it even before you read the next chunk

Why does getResourceAsStream() and reading file with FileInputStream return arrays of different length?

I want to read files as byte arrays and realised that amount of read bytes varies depending on the used method. Here the relevant code:
public byte[] readResource() {
try (InputStream is = getClass().getClassLoader().getResourceAsStream(FILE_NAME)) {
int available = is.available();
byte[] result = new byte[available];
is.read(result, 0, available);
return result;
} catch (Exception e) {
log.error("Failed to load resource '{}'", FILE_NAME, e);
}
return new byte[0];
}
public byte[] readFile() {
File file = new File(FILE_PATH + FILE_NAME);
try (InputStream is = new FileInputStream(file)) {
int available = is.available();
byte[] result = new byte[available];
is.read(result, 0, available);
return result;
} catch (Exception e) {
log.error("Failed to load file '{}'", FILE_NAME, e);
}
return new byte[0];
}
Calling File.length() and reading with the FileInputStream returns the correct length of 21566 bytes for the given test file, though reading the file as a resources returns 21622 bytes.
Does anyone know why I get different results and how to fix it so that readResource() returns the correct result?

Why does getResourceAsStream() and reading file with FileInputStream return arrays of different length?
Because you're misusing the available() method in a way that is specifically warned against in the Javadoc:
"It is never correct to use the return value of this method to allocate a buffer intended to hold all data in this stream."
and
Does anyone know why I get different results and how to fix it so that readResource() returns the correct result?
Read in a loop until end of stream.

According to the the API docs of InputStream, InputStream.available() does not return the size of the resource - it returns
an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking
To get the size of a resource from a stream, you need to fully read the stream, and count the bytes read.
To read the stream and return the contents as a byte array, you could do something like this:
try ( InputStream is = getClass().getClassLoader().getResourceAsStream(FILE_NAME);
ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
byte[] buffer = new byte[4096];
int bytesRead = 0;
while ((bytesRead = is.read(buffer)) != -1) {
bos.write(buffer, 0, bytesRead);
}
return bos.toByteArray();
}

Error while sending large files through socket

I'm trying to send large files via socket. The program works fine for small files (such as html pages or pdf), but when i send files over 3/4 mb the output is always corrupted (viewing it with a text editor i noticed that the last few lines are always missing).
Here's the code of the server:
BufferedInputStream in = null;
FileOutputStream fout = null;
try {
server = new ServerSocket(port);
sock = server.accept();
in = new BufferedInputStream(sock.getInputStream());
setPerc(0);
received = 0;
int incByte = -1;
fout = new FileOutputStream(path+name, true);
long size = length;
do{
int buffSize;
if(size >= 4096){
buffSize = 4096;
}else{
buffSize = 1;
}
byte[] o = new byte[buffSize];
incByte = in.read(o, 0, buffSize);
fout.write(o);
received+=buffSize;
setPerc(calcPerc(received, length));
size -= buffSize;
//d("BYTE LETTI => "+incByte);
}while(size > 0);
server.close();
} catch (IOException e) {
e("Errore nella ricezione file: "+e);
}finally{
try {
fout.flush();
fout.close();
in.close();
} catch (IOException e) {
e("ERRORE INCOMINGFILE");
}
}
pr.release(port);
And here's the code of the client:
FileInputStream fin = null;
BufferedOutputStream out = null;
try {
sock = new Socket(host, port);
fin = new FileInputStream(file);
out = new BufferedOutputStream(sock.getOutputStream());
long size = file.length();
int read = -1;
do{
int buffSize = 0;
if(size >= 4096){
buffSize = 4096;
}else{
buffSize = (int)size;
}
byte[] o = new byte[buffSize];
for(int i = 0; i<o.length;i++){
o[i] = (byte)0;
}
read = fin.read(o, 0, buffSize);
out.write(o);
size -= buffSize;
//d("BYTE LETTI DAL FILE => "+read);
}while(size > 0);
} catch (UnknownHostException e) {
} catch (IOException e) {
d("ERRORE NELL'INVIO DEL FILE: "+e);
e.printStackTrace();
}finally{
try {
out.flush();
out.close();
fin.close();
} catch (IOException e) {
d("Errore nella chiusura dei socket invio");
}
}
i think it's something related with the buffer size, but i can't figure out what's wrong here.

This is incorrect:
byte[] o = new byte[buffSize];
incByte = in.read(o, 0, buffSize);
fout.write(o);
You are reading up to buffSize bytes and then writing exactly buffSize bytes.
You are doing the same thing at the other end as well.
You may be able to get away with this when reading from a file1, but when you read from a socket then a read is liable to give you a partially filled buffer, especially if the writing end can't always keep ahead of the reading end 'cos you are hammering the network with a large transfer.
The right way to do it is:
incByte = in.read(o, 0, buffSize);
fout.write(o, 0, incByte);
1 - It has been observed that when you read from a local file, a read call will typically give you all of the bytes that you requested (subject to the file size, etc). So, if you set buffSize to the length of the file, this code would probably work when reading from a local file. But doing this is a bad idea, because you are relying behaviour that is not guaranteed by either Java or a typical operating system.

You might have a problem e.g. here.
read = fin.read(o, 0, buffSize);
out.write(o);
Here read gives you the count of bytes you've actually just read.
On the next line you should write out only as many bytes as you've read.
In other words, you cannot expect the size of the file
you're reading to be multiple of your buffer size.
Review your server code too for the same issue.

The correct way to copy streams in Java is as follows:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
where count is an int, and buffer is a byte[] array of length > 0, typically 8k. You don't need to allocate byte arrays inside the loop, and you don't need a byte array of a specific size. Specifically, it's a complete waste of space to allocate a buffer as large as the file; it only works up to files of Integer.MAX_VALUE bytes, and it doesn't scale.
You do need to save the count returned by 'read()' and use it in the 'write()' method as shown above.