We Keep Coding

Read the newly appended file content to an InputStream in Java

I have a writer program that writes a huge serialized java object (at the scale of 1GB) into a binary file on local disk at a specific speed. Actually, the writer program (implemented in C language) is a network receiver that receives the bytes of the serialized object from a remote server. The implementation of the writer is fixed.
Now, I want to implement a Java reader program that reads the file and deserializes it to a Java object. Since the file could be very large, it is beneficial to reduce the latency of deserializing the object. Particularly, I want the Java reader starts to read/deserialize the object once the first byte of the object has been written to the disk file so that the reader can start to deserialize the object even before the entire serialized object has been written to the file. The reader knows the size of the file ahead of time (before the first byte is written to the file).
I think what I need is something like a blocking file InputStream that will be blocked when it reaches the EndOfFile but it has not read the expected number of bytes (the size of the file will be). Thus, whenever new bytes have been written to the file, the reader's InputStream could keep reading the new content. However, FileInputStream in Java does not support this feature.
Probably, I also need a file listener that monitoring the changes made to the file to achieve this feature.
I am wondering if there is any existing solution/library/package can achieve this function. Probably the question may be similar to some of the questions in monitoring the log files.
The flow of the bytes is like this:
FileInputStream -> SequenceInputStream -> BufferedInputStream -> JavaSerializer

You need two threads: Thread1 to download from the server and write to a File, and Thread2 to read the File as it becomes available.
Both threads should share a single RandomAccessFile, so access to the OS file can be synchronized correctly. You could use a wrapper class like this:
public class ReadWriteFile {
ReadWriteFile(File f, long size) throws IOException {
_raf = new RandomAccessFile(f, "rw");
_size = size;
_writer = new OutputStream() {
#Override
public void write(int b) throws IOException {
write(new byte[] {
(byte)b
});
}
#Override
public void write(byte[] b, int off, int len) throws IOException {
if (len < 0)
throw new IllegalArgumentException();
synchronized (_raf) {
_raf.seek(_nw);
_raf.write(b, off, len);
_nw += len;
_raf.notify();
}
}
};
}
void close() throws IOException {
_raf.close();
}
InputStream reader() {
return new InputStream() {
#Override
public int read() throws IOException {
if (_pos >= _size)
return -1;
byte[] b = new byte[1];
if (read(b, 0, 1) != 1)
throw new IOException();
return b[0] & 255;
}
#Override
public int read(byte[] buff, int off, int len) throws IOException {
synchronized (_raf) {
while (true) {
if (_pos >= _size)
return -1;
if (_pos >= _nw) {
try {
_raf.wait();
continue;
} catch (InterruptedException ex) {
throw new IOException(ex);
}
}
_raf.seek(_pos);
len = (int)Math.min(len, _nw - _pos);
int nr = _raf.read(buff, off, len);
_pos += Math.max(0, nr);
return nr;
}
}
}
private long _pos;
};
}
OutputStream writer() {
return _writer;
}
private final RandomAccessFile _raf;
private final long _size;
private final OutputStream _writer;
private long _nw;
}
The following code shows how to use ReadWriteFile from two threads:
public static void main(String[] args) throws Exception {
File f = new File("test.bin");
final long size = 1024;
final ReadWriteFile rwf = new ReadWriteFile(f, size);
Thread t1 = new Thread("Writer") {
public void run() {
try {
OutputStream w = new BufferedOutputStream(rwf.writer(), 16);
for (int i = 0; i < size; i++) {
w.write(i);
sleep(1);
}
System.out.println("Write done");
w.close();
} catch (Exception ex) {
ex.printStackTrace();
}
}
};
Thread t2 = new Thread("Reader") {
public void run() {
try {
InputStream r = new BufferedInputStream(rwf.reader(), 13);
for (int i = 0; i < size; i++) {
int b = r.read();
assert (b == (i & 255));
}
int eof = r.read();
assert (eof == -1);
r.close();
System.out.println("Read done");
} catch (IOException ex) {
ex.printStackTrace();
}
}
};
t1.start();
t2.start();
t1.join();
t2.join();
rwf.close();
}

copying files in android with input/output stream: the good and the bad way

i have made this two routines to copy files using inputstream and outpustream.
they are quite the same however the second one rise ArrayIndexOutOfBoundsException while the first one works flawlessly and i don't know why:
public void CopyStream(long size, InputStream is, OutputStream os) {
final int buffer_size = 4096;
byte[] bytes = new byte[buffer_size];
try {
int count,prog=0;
while ((count = is.read(bytes)) != -1) {
os.write(bytes, 0, count); //write buffer
prog = prog + count;
publishProgress(((long) prog) * 100 / size);
}
os.flush();
is.close();
os.close();
} catch (Exception ex) {
Log.e(TAG,"CS "+ex);
}
}
as you may guess the routine is called inside an AsyncTask, therefore the publishProgresss
public void CopyStream(long size, InputStream is, OutputStream os) {
final int buffer_size = 4096;
try {
byte[] bytes = new byte[buffer_size];
for (int count=0,prog=0;count!=-1;) {
count = is.read(bytes);
os.write(bytes, 0, count);
prog=prog+count;
publishProgress(((long) prog)*100/size);
}
os.flush();
is.close();
os.close();
} catch (Exception ex) {
Log.e(TAG,"CS "+ex);
}
}
Does anyone know why the while works but the for no ? what am i missing?

The problem lies in your for loop checking the condition after the first run through. Basically the error occurs when it has read fine the last loop but on the next loop the is.read call returns -1. Afterwards you try to call os.write(bytes,0,-1); -1 is an invalid index. The solution would be:
public void CopyStream(long size, InputStream is, OutputStream os) {
final int buffer_size = 4096;
try {
byte[] bytes = new byte[buffer_size];
for (int count=0,prog=0;count!=-1;) {
count = is.read(bytes);
if(count != -1) {
os.write(bytes, 0, count);
prog=prog+count;
publishProgress(((long) prog)*100/size);
}
}
os.flush();
is.close();
os.close();
} catch (Exception ex) {
Log.e(TAG,"CS "+ex);
}
}
But it is much more readable as the while loop so I would stick with that. For loops should be used either when you know the quantity of times to loop or as a for each where you loop through each individual item of a collection.

For loop stop condition is checked before calling is.read(). This allow situation when you try to read bytes, get result in -1 value and try to continue executing for loop code. While stops immediately after is.read() returns -1
Try following:
int count = is.read(bytes);
for (prog=0;count!=-1;) {
os.write(bytes, 0, count);
prog=prog+count;
publishProgress(((long) prog)*100/size);
count = is.read(bytes);
}

private static final int BASE_BUFFER_SIZE = 4096;
public static void copyFile(InputStream inputStream, OutputStream outputStream)
throws IOException {
byte[] bytes = new byte[BASE_BUFFER_SIZE];
int count;
while ((count = inputStream.read(bytes)) != -1){
outputStream.write(bytes, 0, count);
}
close(inputStream);
close(outputStream);
}
public static void close(#Nullable OutputStream stream) {
if (stream != null) {
try {
stream.flush();
} catch (IOException ignored) {
}
try {
stream.close();
} catch (IOException ignored) {
}
}
}
public static void close(#Nullable InputStream stream) {
if (stream != null) {
try {
stream.close();
} catch (IOException ignored) {
}
}
}

Java program slow down by printing line

Hi I have a script that downloads are file from the web and while doing so prints out the progress. The problem is that the line that prints out the progress slows the program down alot, is there any way to stop this?
import java.io.FileOutputStream;
import java.io.InputStream;
import java.net.URL;
public class download {
public static void main(String[] args) {
try{
URL u = new URL("http://upload.wikimedia.org/wikipedia/commons/1/16/Appearance_of_sky_for_weather_forecast,_Dhaka,_Bangladesh.JPG");
FileOutputStream fos = new FileOutputStream("C://Users/xxx/Desktop/test.jpg");
InputStream is = u.openStream();
long size = u.openConnection().getContentLengthLong();
int data;
long done = 0;
while((data = is.read())!=-1){
double progress = (double) (done)/(double)(size)*100;
System.out.println(progress); // if we dont do this then then it completes fast
fos.write(data);
done++;
}
fos.close();
}catch(Exception e){
e.printStackTrace();
}
}
}

First of all, every I/O operation takes a high cost. Now, you're printing a message for every byte read! (noted in InputStream#read).
If you want/need to print the progress, do it for a bunch of KBs read, usually every 4 KBs. You can do this by using a byte[] buffer to read and write the data from the streams.
BufferedInputStream input = null;
BufferedOutStream output = null;
final int DEFAULT_BUFFER_SIZE = 4 * 1024;
try {
input = new BufferedInputStream(is, DEFAULT_BUFFER_SIZE);
output = new BufferedOutputStream(fos, DEFAULT_BUFFER_SIZE);
byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
int length;
while ((length = input.read(buffer)) > 0) {
output.write(buffer, 0, length);
done += length;
double progress = (double) (done)/(double)(size)*100
System.out.println(progress);
}
} catch (IOException e) {
//log your exceptions...
} finally {
closeResource(output);
closeResource(input);
}
And have this closeResource method:
public void closeResource(Closeable resource) {
if (resource != null) {
try {
resource.close();
} catch (IOException e) {
logger.error("Error while closing the resource.", e);
}
}
}

Try only printing out every xth loop.
if(done % 10 == 0) System.out.println(progress);

You can print the line only if (done % 100 == 0) let's say.
Also, you can use buffered way of reading, that would speed the program up.

Suggestion: don't print the progress with every iteration of the loop. Use a counter, decide on a reasonable frequency, a number to mod the counter by, and print the progress at that selected frequency.

Streaming chunks of audio (mp3) using Java&JSP for real-time playback through servletOutputStream

I am trying to playback audio and keep it continuous and free from skips or blank spots. I have to first receive as bytes in chunks and convert this to mp3 to be streamed by the servletOutputStream. I only start playing once enough bytes have been collected by the consumer in an attempt to maintain a constant flow of audio. As you can see I have hard coded this buffer but would like it to work for any size of audio bytes. I was wondering if anyone had come across a similar problem and had any advice?
Thanks in advance. Any help would be greatly appreciated.
public class Consumer extends Thread {
private MonitorClass consBuf;
private InputStream mp3InputStream = null;
private OutputStream OutputStream = null;
public Consumer (MonitorClass buf, OutputStream servlet)
{
consBuf = buf;
OutputStream = servlet;
}
public void run()
{
byte[] data;
byte[] tempbuf;
int byteSize = 60720; //This should be dynamic
int byteIncrement = byteSize;
int dataPlayed = 0;
int start = 0;
int buffer = 0;
boolean delay = true;
AudioFormat generatedTTSAudioFormat = getGeneratedAudioFormat();
try
{
while(true)
{
try
{
data = consBuf.get(); //gets data from producer using a shared monitor class
if(data.length >= byteSize) //Buffer size hit, start playing
{
if(delay) //help with buffering
{
System.out.println("Pre-delay...");
consBuf.preDelay();
delay = false;
}
tempbuf = new byte[byteIncrement];
arraySwap(data, tempbuf, start, byteSize);
System.out.println("Section to play: " + start + ", " + byteSize);
mp3InputStream = FishUtils.convertToMP3( new ByteArrayInputStream(tempbuf), generatedTTSAudioFormat);
copyStream(mp3InputStream, OutputStream);
System.out.println("Data played: " + byteSize);
System.out.println("Data collected: " + consBuf.getDownloadedBytes() );
dataPlayed = byteSize;
start = byteSize;
byteSize += byteIncrement;
}
if( consBuf.getIsComplete() )
{
if (consBuf.checkAllPlayed(dataPlayed) > 0)
{
System.out.println("Producer finished, play remaining section...");
//mp3InputStream = convertToMP3(new ByteArrayInputStream(tempbuf), generatedTTSAudioFormat);
//copyStream(mp3InputStream, OutputStream);
}
System.out.println("Complete!");
break;
}
}
catch (Exception e)
{
System.out.println(e);
return;
}
}
}
finally
{
if (null != mp3InputStream)
{
try
{
mp3InputStream.skip(Long.MAX_VALUE);
}
catch (Exception e)
{}
}
closeStream(mp3InputStream);
closeStream(OutputStream);
}
}
}

how to add ProgressMonitorInputStream to ftp upload?

Can anybody see what is wrong with this code. it does not show up progress-bar but uploades all the files.
I did checkout sun tutorial and swingworkers also but i couldn't fix it yet.
private static boolean putFile(String m_sLocalFile, FtpClient m_client) {
boolean success = false;
int BUFFER_SIZE = 10240;
if (m_sLocalFile.length() == 0) {
System.out.println("Please enter file name");
}
byte[] buffer = new byte[BUFFER_SIZE];
try {
File f = new File(m_sLocalFile);
int size = (int) f.length();
System.out.println("File " + m_sLocalFile + ": " + size + " bytes");
System.out.println(size);
FileInputStream in = new FileInputStream(m_sLocalFile);
//test
InputStream inputStream = new BufferedInputStream(
new ProgressMonitorInputStream(null,"Uploading " + f.getName(),in));
//test
OutputStream out = m_client.put(f.getName());
int counter = 0;
while (true) {
int bytes = inputStream.read(buffer); //in
if (bytes < 0)
break;
out.write(buffer, 0, bytes);
counter += bytes;
System.out.println(counter);
}
out.close();
in.close();
inputStream.close();
success =true;
} catch (Exception ex) {
System.out.println("Error: " + ex.toString());
}
return true;
}

I think your code is fine.
Maybe the task isn't taking long enough for the progress bar to be needed?
Here's a modified version of your code which reads from a local file and writes to another local file.
I have also added a delay to the write so that it gives the progress bar time to kick in.
This works fine on my system with a sample 12MB PDF file, and shows the progress bar.
If you have a smaller file then just increase the sleep from 5 milliseconds to 100 or something - you would need to experiment.
And I didn't even know that the ProgressMonitorInputStream class existed, so I've learnt something myself ;].
/**
* main
*/
public static void main(String[] args) {
try {
System.out.println("start");
final String inf = "d:/testfile.pdf";
final String outf = "d:/testfile.tmp.pdf";
final FileOutputStream out = new FileOutputStream(outf) {
#Override
public void write(byte[] b, int off, int len) throws IOException {
super.write(b, off, len);
try {
// We delay the write by a few millis to give the progress bar time to kick in
Thread.sleep(5);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
};
putFile(inf, out);
System.out.println("end");
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
private static boolean putFile(String m_sLocalFile, OutputStream out /*FtpClient m_client*/) {
boolean success = false;
int BUFFER_SIZE = 10240;
if (m_sLocalFile.length() == 0) {
System.out.println("Please enter file name");
}
byte[] buffer = new byte[BUFFER_SIZE];
try {
File f = new File(m_sLocalFile);
int size = (int) f.length();
System.out.println("File " + m_sLocalFile + ": " + size + " bytes");
System.out.println(size);
FileInputStream in = new FileInputStream(m_sLocalFile);
//test
InputStream inputStream = new BufferedInputStream(
new ProgressMonitorInputStream(null,"Uploading " + f.getName(),in));
//test
//OutputStream out = m_client.put(f.getName());
int counter = 0;
while (true) {
int bytes = inputStream.read(buffer); //in
if (bytes < 0)
break;
out.write(buffer, 0, bytes);
counter += bytes;
System.out.println(counter);
}
out.close();
in.close();
inputStream.close();
success =true;
} catch (Exception ex) {
System.out.println("Error: " + ex.toString());
}
return true;
}