We Keep Coding

Using RandomAccessFile along with BufferedReader to speed up file read

I have to :-
Read large text file line by line.
Note down file pointer position after every line read.
Stop the file read if running time is greater than 30 seconds.
Resume from last noted file pointer in a new process.
What I am doing :
Using RandomAccessFile.getFilePointer() to note the file pointer.
Wrap RandomAccessFile into another BufferedReader to speed up file read process as per this answer.
When time is greater than 30 seconds, I stop reading the file. Restarting the process with new RandomAccessFile and using RandomAccessFile.seek method to move file pointer to where I left.
Problem:
As I am reading through BufferedReader wrapped around RandomAccessFile, it seems file pointer is moving far ahead in a single call to BufferedReader.readLine(). However, if I use RandomAccessFile.readLine() directely, file pointer is moving properly step by step in forward direction.
Using BufferedReader as a wrapper :
RandomAccessFile randomAccessFile = new RandomAccessFile("mybigfile.txt", "r");
BufferedReader brRafReader = new BufferedReader(new FileReader(randomAccessFile.getFD()));
while((line = brRafReader.readLine()) != null) {
System.out.println(line+", Position : "+randomAccessFile.getFilePointer());
}
Output:
Line goes here, Position : 13040
Line goes here, Position : 13040
Line goes here, Position : 13040
Line goes here, Position : 13040
Using Direct RandomAccessFile.readLine
RandomAccessFile randomAccessFile = new RandomAccessFile("mybigfile.txt", "r");
while((line = randomAccessFile.readLine()) != null) {
System.out.println(line+", Position : "+randomAccessFile.getFilePointer());
}
Output: (This is as expected. File pointer moving properly with each call to readline)
Line goes here, Position : 11011
Line goes here, Position : 11089
Line goes here, Position : 12090
Line goes here, Position : 13040
Could anyone tell, what wrong am I doing here ? Is there any way I can speed up reading process using RandomAccessFile ?

The reason for the observed behavior is that, as the name suggests, the BufferedReader is buffered. It reads a larger chunk of data at once (into a buffer), and returns only the relevant parts of the buffer contents - namely, the part up to the next \n line separator.
I think there are, broadly speaking, two possible approaches:
You could implement your own buffering logic.
Using some ugly reflection hack to obtain the required buffer offset
For 1., you would no longer use RandomAccessFile#readLine. Instead, you'd do your own buffering via
byte buffer[] = new byte[8192];
...
// In a loop:
int read = randomAccessFile.read(buffer);
// Figure out where a line break `\n` appears in the buffer,
// return the resulting lines, and take the position of the `\n`
// into account when storing the "file pointer"
As the vague comment indicates: This may be cumbersome and fiddly. You'd basically re-implement what the readLine method does in the BufferedReader class. And at this point, I don't even want to mention the headaches that different line separators or character sets could cause.
For 2., you could simply access the field of the BufferedReader that stores the buffer offset. This is implemented in the example below. Of course, this is a somewhat crude solution, but mentioned and shown here as a simple alternative, depending on how "sustainable" the solution should be and how much effort you are willing to invest.
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.RandomAccessFile;
import java.lang.reflect.Field;
import java.util.ArrayList;
import java.util.List;
public class LargeFileRead {
public static void main(String[] args) throws Exception {
String fileName = "myBigFile.txt";
long before = System.nanoTime();
List<String> result = readBuffered(fileName);
//List<String> result = readDefault(fileName);
long after = System.nanoTime();
double ms = (after - before) / 1e6;
System.out.println("Reading took " + ms + "ms "
+ "for " + result.size() + " lines");
}
private static List<String> readBuffered(String fileName) throws Exception {
List<String> lines = new ArrayList<String>();
RandomAccessFile randomAccessFile = new RandomAccessFile(fileName, "r");
BufferedReader brRafReader = new BufferedReader(
new FileReader(randomAccessFile.getFD()));
String line = null;
long currentOffset = 0;
long previousOffset = -1;
while ((line = brRafReader.readLine()) != null) {
long fileOffset = randomAccessFile.getFilePointer();
if (fileOffset != previousOffset) {
if (previousOffset != -1) {
currentOffset = previousOffset;
}
previousOffset = fileOffset;
}
int bufferOffset = getOffset(brRafReader);
long realPosition = currentOffset + bufferOffset;
System.out.println("Position : " + realPosition
+ " with FP " + randomAccessFile.getFilePointer()
+ " and offset " + bufferOffset);
lines.add(line);
}
return lines;
}
private static int getOffset(BufferedReader bufferedReader) throws Exception {
Field field = BufferedReader.class.getDeclaredField("nextChar");
int result = 0;
try {
field.setAccessible(true);
result = (Integer) field.get(bufferedReader);
} finally {
field.setAccessible(false);
}
return result;
}
private static List<String> readDefault(String fileName) throws Exception {
List<String> lines = new ArrayList<String>();
RandomAccessFile randomAccessFile = new RandomAccessFile(fileName, "r");
String line = null;
while ((line = randomAccessFile.readLine()) != null) {
System.out.println("Position : " + randomAccessFile.getFilePointer());
lines.add(line);
}
return lines;
}
}
(Note: The offsets may still appear to be off by 1, but this is due to the line separator not being taken into account in the position. This could be adjusted if necessary)
NOTE: This is only a sketch. The RandomAccessFile objects should be closed properly when reading is finished, but that depends on how the reading is supposed to be interrupted when the time limit is exceeded, as described in the question

BufferedReader reads a block of data from the file, 8 KB by default. Finding line breaks on order to return the next line is done in the buffer.
I guess, this is why you see a huge increment in the physical file position.
RandomAccessFile will not be using a buffer when reading the next line. It will read byte after byte. That's really slow.
How is performance when you just use a BufferedReader and remember the line you need to continue from?

Java LineNumberReader reset to beginning

I want to read an InputStream in two passes, line by line. I use the following code for the first pass:
LineNumberReader reader = new LineNumberReader(new InputStreamReader(inputStream));
String line;
String eventId = null;
Set<Integer> artistIds = new HashSet<Integer>();
while((line = reader.readLine())!=null) {
// process first pass
}
// how do I reset reader so that I can read from the beginning again?
There is a reset() method available but it resets to the last mark in the file. I don't quite understand what that means. Can I use mark and reset to achieve the reset to beginning behavior? Something like
LineNumberReader reader = new LineNumberReader(new InputStreamReader(inputStream));
reader.mark(0); // mark at the 0th position
// process first pass: repeated calls to readline() until EOF
reader.reset(); // reset to 0th position??
// process second pass
While testing at my local machine, I was reader.close()-ing before the second pass and it worked. However, when I do this in HDFS, reader.close() probably closes the HDFS InputStream too and I get a java.io.IOException: Stream closed exception.

Mark and reset work, but don't call .mark(0) that sets the read ahead limit to 0 which means .reset() won't work reliably if you read more than 0 bytes.
EDIT: .mark() marks the current location in the stream. Unlike C++ where you can .seek() the beginning or end of a file and offsets, Java streams only allow you to mark a current location and then go back to it with .reset(). This can go "back to the beginning" but only if it was marked before processing started.
Try this:
import java.io.*;
public class StreamTwice
{
public static void printLines(LineNumberReader r) throws IOException
{
String line;
while( (line = r.readLine()) != null )
System.out.println(line);
System.out.println();
}
public static void main(String []args) throws Exception
{
ByteArrayInputStream s = new ByteArrayInputStream(
"one\ntwo\nthree".getBytes()
);
LineNumberReader r = new LineNumberReader(new InputStreamReader(s));
r.mark(5000); // more than the number of bytes being read.
// this is the read ahead limit.
printLines(r);
r.reset(); // go back to where mark was called.
printLines(r);
}
}
Try to make sure you don't read more bytes than the read ahead limit you set in .mark() before calling .reset().
P.S. - Not all streams (or readers) support .mark(), which you can check with .markSupported().

Reset() resets the line reader to the most recent mark (which is the last if you are going sequentially.) What you need to do is manually change the line by calling " reader.SetLineNumber(0); " the parameter indicates the line number that you want to go to.

Trying to use BufferedReader to read input from System.in and put it into an array, however doesn't seem to want to give me the array

As stated in the title, I am using a BufferedReader that reads from System.in and puts the values into an array of Strings. When I call the method to do this, I try to check the length of the array, but nothing gets printed out when I do so.
String s = "";
BufferedReader io = new BufferedReader
(new InputStreamReader(System.in)); //create a new BufferedReader using system input
String line; //will be changed to the next line in the input
try //required or we cant compile it because exceptions
{
s = io.readLine() + "_"; //get the first integer value, which is how many
// pieces of info we have to process
while((line = io.readLine()) != null) //read until the end of the file
{
s += io.readLine() + "_"; //add it to the ArrayList (may not be needed)
//System.out.println("Added line to list");
//System.out.println(line);
}
io.close(); //done with input, close the stream
}
catch (IOException e)
{
e.printStackTrace();
}
return s.split("_");
I then call the method containing this code, but literally nothing is printed out:
String[] input = readInput();
System.out.println(input.length);
Any suggestions?

You're calling readLine() twice inside your loop and throwing every odd result away. You need to assign the result obtained in the while condition and use that inside the loop, instead of calling it again.

Stop BufferedReader if the same text is read

I have a buffer reader working to read a text file then save the read data as a string. The buffer reader is in a continuous loop reading the text file every second. I would like to do a Thread.sleep if the buffer reader reads the same text two times in a row.
For example if the buffer reader reads "foo" then when it is ran again in the next second if it reads "foo" again then the thread sleeps for a certain amount of time.
I have the buffer reader within a try method but I do not know how to get it to do the above.
Could anyone help with this?
Thanks

Keep track of the previous line read in in a String variable, which you can initialize as an empty string. When you read in each line, check whether it's equals() to the previously stored line in that variable. If so, do what you want to do, like Thread.sleep(). Otherwise save the current line into that variable. In code:
BufferedReader reader = new BufferedReader(new FileReader("foo.txt"));
String foo = "";
String old = "";
while((foo = reader.readLine()) != null) {
if(foo.equals(old)) {
System.out.println("Sleeping...");
Thread.sleep(1000);
} else {
old = foo;
}
}

How can I read text appended to the end of a file I already read?

I want to read a text file in Java. After I finish, some text will be appended by another application, and then I want to read that. Lets say there are ten lines. When the other app appends one more line, I dont want to read the whole file again; just the new line. How can I do this?

Something like this could work:
BufferedReader reader = .. // create a reader on the input file without locking it
while(otherAppWritesToFile) {
String line = reader.readLine();
while(line != null) {
processLine(line);
line = reader.readLine();
}
Thread.sleep(100);
}
Exception handling has been left out for the sake of simplicity.

Once you get an EOF indication, wait a little bit and then try reading again.
Edit: Here is teh codez to support this solution. You can try it and then change the control flow mechanisms as needed.
public static void main(final String[] args) throws IOException {
final Scanner keyboard = new Scanner(System.in);
final BufferedReader input = new BufferedReader(new FileReader("input.txt"));
boolean cont = true;
while (cont) {
String line = input.readLine();
while (line != null) {
System.out.println(line);
line = input.readLine();
}
System.out.println("EOF reached, add more input and type 'y' to continue.");
final String in = keyboard.nextLine();
cont = in.equalsIgnoreCase("y");
}
}
EDIT: Thanks for adding some code Tim. Personally, I would just do a sleep instead of waiting for user input. That would more closely match the users' requirements.

You could try using a RandomAccessFile.
Open the file and then invoke the length() to get the length of the file. Then you can use the readLine() method to get your data. Then the next time you open the file you can use the seek() method to position yourself to the previous end of the file. Then read the lines and save the new length of the file.