Hi I have a script that downloads are file from the web and while doing so prints out the progress. The problem is that the line that prints out the progress slows the program down alot, is there any way to stop this?
import java.io.FileOutputStream;
import java.io.InputStream;
import java.net.URL;
public class download {
public static void main(String[] args) {
try{
URL u = new URL("http://upload.wikimedia.org/wikipedia/commons/1/16/Appearance_of_sky_for_weather_forecast,_Dhaka,_Bangladesh.JPG");
FileOutputStream fos = new FileOutputStream("C://Users/xxx/Desktop/test.jpg");
InputStream is = u.openStream();
long size = u.openConnection().getContentLengthLong();
int data;
long done = 0;
while((data = is.read())!=-1){
double progress = (double) (done)/(double)(size)*100;
System.out.println(progress); // if we dont do this then then it completes fast
fos.write(data);
done++;
}
fos.close();
}catch(Exception e){
e.printStackTrace();
}
}
}
First of all, every I/O operation takes a high cost. Now, you're printing a message for every byte read! (noted in InputStream#read).
If you want/need to print the progress, do it for a bunch of KBs read, usually every 4 KBs. You can do this by using a byte[] buffer to read and write the data from the streams.
BufferedInputStream input = null;
BufferedOutStream output = null;
final int DEFAULT_BUFFER_SIZE = 4 * 1024;
try {
input = new BufferedInputStream(is, DEFAULT_BUFFER_SIZE);
output = new BufferedOutputStream(fos, DEFAULT_BUFFER_SIZE);
byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
int length;
while ((length = input.read(buffer)) > 0) {
output.write(buffer, 0, length);
done += length;
double progress = (double) (done)/(double)(size)*100
System.out.println(progress);
}
} catch (IOException e) {
//log your exceptions...
} finally {
closeResource(output);
closeResource(input);
}
And have this closeResource method:
public void closeResource(Closeable resource) {
if (resource != null) {
try {
resource.close();
} catch (IOException e) {
logger.error("Error while closing the resource.", e);
}
}
}
Try only printing out every xth loop.
if(done % 10 == 0) System.out.println(progress);
You can print the line only if (done % 100 == 0) let's say.
Also, you can use buffered way of reading, that would speed the program up.
Suggestion: don't print the progress with every iteration of the loop. Use a counter, decide on a reasonable frequency, a number to mod the counter by, and print the progress at that selected frequency.
Related
What can I use to read log file in real time in Java 8?
I read blogs to understand BufferedReader is a good option for reading fine.
I tried below:
BufferedReader reader = new
BufferedReader(new
InputStreamReader(inputStream));
String line;
while(true) {
line = reader.readLine(); // blocks until next line
available
// do whatever You want with line
}
However it keeps printing null irrespective of file is updated or not. Any idea what can be going wrong.
Any other options?
Details are as below :
I am trying to create an utility in Java 8 or above, where I need to read log file of an application at real time (as live transactions are occurring and getting printed in logs).
I can access log file as I am on sme server, so that is not an issue.
So some of the specifics are below
-> I don't want to poll the log files for Changes, I want to keep it the bridge open to read log file in "while true" loop. So ideally i want to block my reader if there are no new lines getting printed.
-> I don't want to store the entire content of the file in memory at all time as I want it to be memory efficient.
-> my code will run as a separate application to read log file of another application.
-> only job of my code is to read log, match against a pattern, if matched then send a message with log content.
Kindly let me know if any detail is ambiguous.
Any help is appericiated, thanks.
For this to work, your inputStream must block until new data becomes available, which a standard FileInputStream does not when reaching the end-of-file.
I suppose, you initialize inputStream to just new FileInputStream("my-logfile.log");. This stream will only read to the current end of the log file and signal the "end of file" condition to the BufferedReader. This in turn will signal "end of file" by returning null from readLine().
Have a look at the utility org.apache.commons.io.input.Tailer. This allows to write programs like the Unix utility tail -f.
To make your code work, you would have to use an "infinite" input stream that could be realized using a RandomAccessFile as in the following example:
package test;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.PrintWriter;
import java.io.RandomAccessFile;
import java.nio.file.Files;
import java.nio.file.StandardOpenOption;
public class TestRead {
public static void main(String[] args) throws IOException, InterruptedException {
File logFile = new File("my-log.log");
// Make sure to start form a defined condition.
logFile.delete();
try (OutputStream out = Files.newOutputStream(logFile.toPath(), StandardOpenOption.CREATE)) {
// Just create an empty file to append later on.
}
Thread analyzer = Thread.currentThread();
// Simulate log file writing.
new Thread() {
#Override
public void run() {
try {
for (int n = 0; n < 16; n++) {
try (OutputStream out = Files.newOutputStream(logFile.toPath(), StandardOpenOption.APPEND)) {
PrintWriter printer = new PrintWriter(out);
String line = "Line " + n;
printer.println(line);
printer.flush();
System.out.println("wrote: " + line);
}
Thread.sleep(1000);
}
} catch (Exception ex) {
ex.printStackTrace();
} finally {
analyzer.interrupt();
}
}
}.start();
// The original code reading the log file.
try (InputStream inputStream = new InfiniteInputStream(logFile);) {
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream), 8);
String line;
while (true) {
line = reader.readLine();
if (line == null) {
System.out.println("End-of-file.");
break;
}
System.out.println("read: " + line);
}
}
}
public static class InfiniteInputStream extends InputStream {
private final RandomAccessFile _in;
public InfiniteInputStream(File file) throws IOException {
_in = new RandomAccessFile(file, "r");
}
#Override
public int read(byte[] b, int off, int len) throws IOException {
if (b == null) {
throw new NullPointerException();
} else if (off < 0 || len < 0 || len > b.length - off) {
throw new IndexOutOfBoundsException();
} else if (len == 0) {
return 0;
}
int c = read();
if (c == -1) {
return -1;
}
b[off] = (byte)c;
int i = 1;
try {
for (; i < len ; i++) {
c = readDirect();
if (c == -1) {
break;
}
b[off + i] = (byte)c;
}
} catch (IOException ee) {
}
return i;
}
#Override
public int read() throws IOException {
int result;
while ((result = readDirect()) < 0) {
// Poll until more data becomes available.
try {
Thread.sleep(500);
} catch (InterruptedException ex) {
return -1;
}
}
return result;
}
private int readDirect() throws IOException {
return _in.read();
}
}
}
I am trying to playback audio and keep it continuous and free from skips or blank spots. I have to first receive as bytes in chunks and convert this to mp3 to be streamed by the servletOutputStream. I only start playing once enough bytes have been collected by the consumer in an attempt to maintain a constant flow of audio. As you can see I have hard coded this buffer but would like it to work for any size of audio bytes. I was wondering if anyone had come across a similar problem and had any advice?
Thanks in advance. Any help would be greatly appreciated.
public class Consumer extends Thread {
private MonitorClass consBuf;
private InputStream mp3InputStream = null;
private OutputStream OutputStream = null;
public Consumer (MonitorClass buf, OutputStream servlet)
{
consBuf = buf;
OutputStream = servlet;
}
public void run()
{
byte[] data;
byte[] tempbuf;
int byteSize = 60720; //This should be dynamic
int byteIncrement = byteSize;
int dataPlayed = 0;
int start = 0;
int buffer = 0;
boolean delay = true;
AudioFormat generatedTTSAudioFormat = getGeneratedAudioFormat();
try
{
while(true)
{
try
{
data = consBuf.get(); //gets data from producer using a shared monitor class
if(data.length >= byteSize) //Buffer size hit, start playing
{
if(delay) //help with buffering
{
System.out.println("Pre-delay...");
consBuf.preDelay();
delay = false;
}
tempbuf = new byte[byteIncrement];
arraySwap(data, tempbuf, start, byteSize);
System.out.println("Section to play: " + start + ", " + byteSize);
mp3InputStream = FishUtils.convertToMP3( new ByteArrayInputStream(tempbuf), generatedTTSAudioFormat);
copyStream(mp3InputStream, OutputStream);
System.out.println("Data played: " + byteSize);
System.out.println("Data collected: " + consBuf.getDownloadedBytes() );
dataPlayed = byteSize;
start = byteSize;
byteSize += byteIncrement;
}
if( consBuf.getIsComplete() )
{
if (consBuf.checkAllPlayed(dataPlayed) > 0)
{
System.out.println("Producer finished, play remaining section...");
//mp3InputStream = convertToMP3(new ByteArrayInputStream(tempbuf), generatedTTSAudioFormat);
//copyStream(mp3InputStream, OutputStream);
}
System.out.println("Complete!");
break;
}
}
catch (Exception e)
{
System.out.println(e);
return;
}
}
}
finally
{
if (null != mp3InputStream)
{
try
{
mp3InputStream.skip(Long.MAX_VALUE);
}
catch (Exception e)
{}
}
closeStream(mp3InputStream);
closeStream(OutputStream);
}
}
}
I have an object with 1 int and 4 doubles.
I compared the performance to write 5 million of these objects in a file using serialization and FileChannel object.
In the serialization used the following method to read and write the file.
public void print() throws IOException, ClassNotFoundException{
ObjectInputStream input = new ObjectInputStream(new FileInputStream(this.filePath) );
try {
while(true) {
this.sb = (Sbit) input.readObject();
//System.out.println(this.sb.toString());
}
}
catch ( EOFException eofException ) {
return;
}
catch (IOException ioException) {
System.exit( 1 );
}
finally {
if( input != null )
input.close();
}
}
public void build() throws IOException {
ObjectOutputStream output = new ObjectOutputStream( new FileOutputStream(this.filePath) );
try {
Random random = new Random();
for (int i = 0; i<5000000; i++) {
this.sb = new Sbit();
this.sb.setKey(i);
this.sb.setXMin( random.nextDouble() );
this.sb.setXMax( random.nextDouble() );
this.sb.setYMin( random.nextDouble() );
this.sb.setYMax( random.nextDouble() );
output.writeObject(this.sb);
}
}
catch (IOException ioException) {
System.exit( 1 );
}
finally {
try {
if( output != null)
output.close();
}
catch ( Exception exception ) {
exception.printStackTrace();
System.exit(1);
}
}
}
While using java.nio was:
public void print() throws IOException {
FileChannel file = new RandomAccessFile(this.filePath, "rw").getChannel();
ByteBuffer[] buffers = new ByteBuffer[5];
buffers[0] = ByteBuffer.allocate(4); // 4 bytes to int
buffers[1] = ByteBuffer.allocate(8); // 8 bytes to double
buffers[2] = ByteBuffer.allocate(8);
buffers[3] = ByteBuffer.allocate(8);
buffers[4] = ByteBuffer.allocate(8);
while (true) {
if(file.read(buffers[0]) == -1 ) // Read the int,
break; // if its EOF exit the loop
buffers[0].flip();
this.sb = new Sbit();
this.sb.setKey(buffers[0].getInt());
if(file.read(buffers[1]) == -1) { // Read the int primary value
assert false; // Should not get here!
break; // Exit loop on EOF
}
buffers[1].flip();
this.sb.setXMin( buffers[1].getDouble() );
if(file.read(buffers[2]) == -1) {
assert false;
break;
}
buffers[2].flip();
this.sb.setXMax( buffers[2].getDouble() );
if(file.read(buffers[3]) == -1) {
assert false;
break;
}
buffers[3].flip();
this.sb.setYMin( buffers[3].getDouble() );
if(file.read(buffers[4]) == -1) {
assert false;
break;
}
buffers[4].flip();
this.sb.setYMax( buffers[4].getDouble() );
for(int i = 0; i < 5; i++)
buffers[i].clear();
}
}
public void build() throws IOException {
FileChannel file = new RandomAccessFile(this.filePath, "rw").getChannel();
Random random = new Random();
for (int i = 0; i<5000000; i++) {
this.sb = new Sbit();
this.sb.setKey(i);
this.sb.setXMin( random.nextDouble() );
this.sb.setXMax( random.nextDouble() );
this.sb.setYMin( random.nextDouble() );
this.sb.setYMax( random.nextDouble() );
ByteBuffer[] buffers = new ByteBuffer[5];
buffers[0] = ByteBuffer.allocate(4); // 4 bytes to into
buffers[1] = ByteBuffer.allocate(8); // 8 bytes to double
buffers[2] = ByteBuffer.allocate(8);
buffers[3] = ByteBuffer.allocate(8);
buffers[4] = ByteBuffer.allocate(8);
buffers[0].putInt(this.sb.getKey()).flip();
buffers[1].putDouble(this.sb.getXMin()).flip();
buffers[2].putDouble(this.sb.getXMax()).flip();
buffers[3].putDouble(this.sb.getYMin()).flip();
buffers[4].putDouble(this.sb.getYMax()).flip();
try {
file.write(buffers);
}
catch (IOException e) {
e.printStackTrace(System.err);
System.exit(1);
}
for(int x = 0; x < 5; x++)
buffers[x].clear();
}
}
But I read a lot about on the java.nio and tried to use it precisely because it has better performance. But that's not what happened in my case.
To write the file were the following (java.nio):
file size: 175 MB
time in milliseconds: 57638
Using serialization:
file size: 200 MB
time in milliseconds: 34504
For the reading of this file, were as follows (java.nio):
time in milliseconds: 78172
Using serialization:
time in milliseconds: 35288
Am I doing something wrong in java.nio? I would like to write to the same binary files as done. There is another way to write file efficiently? actually serializing an object is the best way?
Thank you.
You are creating 25,000,000 ByteBuffer objects, with each ByteBuffer being at most 8 bytes. Thats very inefficient.
Create just one ByteBuffer by allocating it to 38 bytes outside the loop (before the for statement)
Inside the loop you can use the same ByteBuffer as follows:
buffer.clear();
buffer.putInt(this.sb.getKey());
buffer.putDouble(this.sb.getXMin());
buffer.putDouble(this.sb.getXMax());
buffer.putDouble(this.sb.getYMin());
buffer.putDouble(this.sb.getYMax());
buffer.flip();
try
{
file.write(buffer);
}
catch (IOException ex)
{
ex.printStackTrace();
//etc...
}
buffer.flip();
Try it out and let us know if you see any improvements.
Instead of using multiple ByteBuffers, declare a single byte buffer that is large enough to hold all of the data you want to put into it. Then put data into it just like you are now. When done, flip the buffer and write it out. When you are ready to read it back in, read the data from disk into the byte buffer, flip it, and then read the data out using getInt/getDouble.
I haven't tried to serialize stuff on my own, but have achieved good results with kryo. It is a lot faster than standard Java serialization.
I'm running some unsecure code which I have set its stdout and stderr streams to FileStreams wrapped in PrintStreams. (Standard output/error MUST be redirected.)
Is there any way to configure those redirected FileStreams/PrintStreams to set a maximum of say 10 MB written, so that, for example,
while (true) System.out.write("lots of bytes");
doesn't write excessive amounts of data to the server's disk.
The code does have a time limit of 15s, but I'd like a separate guard here.
One way to do it is to define a FilterOutputStream that you wrap the file stream in, which keeps an internal counter that it increments on every write, and after reaching a set threshold, starts throwing Exceptions or simply ignores the writes.
Something along the lines of:
import java.io.*;
class LimitOutputStream extends FilterOutputStream{
private long limit;
public LimitOutputStream(OutputStream out,long limit){
super(out);
this.limit = limit;
}
public void write(byte[]b) throws IOException{
long left = Math.min(b.length,limit);
if (left<=0)
return;
limit-=left;
out.write(b, 0, (int)left);
}
public void write(int b) throws IOException{
if (limit<=0)
return;
limit--;
out.write(b);
}
public void write(byte[]b,int off, int len) throws IOException{
long left = Math.min(len,limit);
if (left<=0)
return;
limit-=left;
out.write(b,off,(int)left);
}
}
I had similar task but reading InputStreams from a DB and made a small method.
Don't want to be the Captain Obvious but it also can be used with inpustreams like FileInputStream too :)
public static void writeBytes2File(InputStream is, String name,long limit) {
byte buf[] = new byte[8192];
int len = 0;
long size = 0;
FileOutputStream fos = null;
try {
fos = new FileOutputStream(name);
while ((len = is.read(buf)) != -1) {
fos.write(buf, 0, len);
size += len;
if (size > limit*1024*1024) {
System.out.println("The file size exceeded " + size + " Bytes ");
break;
}
}
System.out.println("File written: " +name);
}
catch (FileNotFoundException fnone) {
fnone.printStackTrace();
}
catch (IOException ioe) {
ioe.printStackTrace();
}
finally {
try {
if(is!=null){is.close();}
if (fos != null) {fos.flush();fos.close();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
hope somebody might find it useful.
I'm working on downloading a file on a software, this is what i got, it sucesfully download, and also i can get progress, but still 1 thing left that I dont know how to do. Measure download speed. I would appreciate your help. Thanks.
This is the current download method code
public void run()
{
OutputStream out = null;
URLConnection conn = null;
InputStream in = null;
try
{
URL url1 = new URL(url);
out = new BufferedOutputStream(
new FileOutputStream(sysDir+"\\"+where));
conn = url1.openConnection();
in = conn.getInputStream();
byte[] buffer = new byte[1024];
int numRead;
long numWritten = 0;
double progress1;
while ((numRead = in.read(buffer)) != -1)
{
out.write(buffer, 0, numRead);
numWritten += numRead;
this.speed= (int) (((double)
buffer.length)/8);
progress1 = (double) numWritten;
this.progress=(int) progress1;
}
}
catch (Exception ex)
{
echo("Unknown Error: " + ex);
}
finally
{
try
{
if (in != null)
{
in.close();
}
if (out != null)
{
out.close();
}
}
catch (IOException ex)
{
echo("Unknown Error: " + ex);
}
}
}
The same way you would measure anything.
System.nanoTime() returns a Long you can use to measure how long something takes:
Long start = System.nanoTime();
// do your read
Long end = System.nanoTime();
Now you have the number of nanoseconds it took to read X bytes. Do the math and you have your download rate.
More than likely you're looking for bytes per second. Keep track of the total number of bytes you've read, checking to see if one second has elapsed. Once one second has gone by figure out the rate based on how many bytes you've read in that amount of time. Reset the total, repeat.
here is my implementation
while (mStatus == DownloadStatus.DOWNLOADING) {
/*
* Size buffer according to how much of the file is left to
* download.
*/
byte buffer[];
// handled resume case.
if ((mSize < mDownloaded ? mSize : mSize - mDownloaded <= 0 ? mSize : mSize - mDownloaded) > MAX_BUFFER_SIZE) {
buffer = new byte[MAX_BUFFER_SIZE];
} else {
buffer = new byte[(int) (mSize - mDownloaded)];
}
// Read from server into buffer.
int read = stream.read(buffer);
if (read == -1)
break;// EOF, break while loop
// Write buffer to file.
file.write(buffer, 0, read);
mDownloaded += read;
double speedInKBps = 0.0D;
try {
long timeInSecs = (System.currentTimeMillis() - startTime) / 1000; //converting millis to seconds as 1000m in 1 second
speedInKBps = (mDownloaded / timeInSecs) / 1024D;
} catch (ArithmeticException ae) {
}
this.mListener.publishProgress(this.getProgress(), this.getTotalSize(), speedInKBps);
}
I can give you a general idea. Start a timer at the beginning of the download. Now, multiply the (percentage downloaded) by the download size, and divide it by the time elapsed. That gives you average download time. Hope I get you on the right track!
You can use System.nanoTime(); as suggested by Brian.
Put long startTime = System.nanoTime(); outside your while loop. and
long estimatedTime = System.nanoTime() - startTime; will give you the elapsed time within your loop.