We Keep Coding

breaking the lines while chunking into multiple files

I am dividing my file into chunks but only problem i am facing is,
i have .srt file, but while doing chunks, it's cutting the characters i.e in first .srt file it's like 00:26:20,230 --> . in next file it continuing the next time stamp 00:27:40,343.
I need to check the timestamp to be complete and then next full subtitle sentence too. i.e if it's cutting the subtitle timesstamp or dialogue in in file, that tect to be append to next file. Please suggest me how can i achieve.
I am trying like below,
String FilePath = "/Users/meh/Desktop/escapeplan.srt";
FileInputStream fin = new FileInputStream(FilePath);
System.out.println("size: " +fin.getChannel().size());
long abc = 0l;
abc = (fin.getChannel().size())/3;
System.out.println("6: " +abc);
System.out.println("abc: " +abc);
//FilePath = args[1];
File filename = new File(FilePath);
long splitFileSize = 0,bytefileSize=0;
if (filename.exists()) {
try {
//bytefileSize = Long.parseLong(args[2]);
splitFileSize = abc;
Splitme spObj = new Splitme();
spObj.split(FilePath, (long) splitFileSize);
spObj = null;
} catch (Exception e) {
e.printStackTrace();
}
} else {
System.out.println("File Not Found....");
}
public void split(String FilePath, long splitlen) {
long leninfile = 0, leng = 0;
int count = 1, data;
try {
File filename = new File(FilePath);
InputStream infile = new BufferedInputStream(new FileInputStream(filename));
data = infile.read();
System.out.println("data");
System.out.println(data);
while (data != -1) {
filename = new File("/Users/meh/Documents/srt" + count + ".srt");
//RandomAccessFile outfile = new RandomAccessFile(filename, "rw");
OutputStream outfile = new BufferedOutputStream(new FileOutputStream(filename));
while (data != -1 && leng < splitlen) {
outfile.write(data);
leng++;
data = infile.read();
}
leninfile += leng;
leng = 0;
outfile.close();
changeTimeStamp(filename, count);
count++;
}
} catch (Exception e) {
e.printStackTrace();
}
}
i am trying to check the time stamp is in correct format or not. Then i need to check next line to be a dialogue and then the next line to be empty line. then it can stop chunk or else it should append the text from the previous chunk to next chunk file in the beginning of line . so that it may get in correct format.
I tried checking the format like,
while ((strLine = br.readLine()) != null) {
String[] atoms = strLine.split(" --> ");
if (atoms.length == 1) {
out.write(strLine + "\n");
} else {
String startTS = atoms[0];
String endTS = atoms[1];
System.out.print("sri atmos start" + startTS);
System.out.print("sri atmos end" + endTS);
SimpleDateFormat sdf = new SimpleDateFormat("HH:mm:ss,SSS");
sdf.setLenient(false);
try
{
sdf.parse(startTS);
sdf.parse(endTS);
System.out.println("Valid time");
System.out.println("File path" + srcFileNm);
}
catch(Exception e) {
System.out.println("Invalid time");
System.out.println("Exception start" + startTS);
System.out.println("Exception end" + endTS);
}
}
some screens of my output chunks,
Help me how can i make this possible.

I think you should change approach, and fully use basic I/O methods. I tried to encapsulate logic in a small class, that produces a triple with id, msecs and a list of subtitles (if I'm not wrong, you can have more than a line). Then I leaved the remainder externally. Chunker is a class that reads a triple (class Three) from file, so that you can manage it and write it somewhere.
This is just a "quick&dirty" idea that you can refine, but it should work.
package org.norsam.stackoverflow;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
public class Chunker
{
BufferedReader r;
int chunk = 0;
File dir;
public Chunker(File dir, String filename) throws IOException
{
File f = new File(dir, filename);
this.dir = dir;
this.r = new BufferedReader(new FileReader(f));
}
public Three readThree() throws IOException
{
Integer id = Integer.parseInt(r.readLine());
String msecs = r.readLine();
String s = null;
List<String> srt = new ArrayList<>();
while (!(s = r.readLine().trim()).isEmpty()) {
srt.add(s);
}
return new Three(id, msecs, srt);
}
class Three
{
Integer id;
String msecs;
List<String> srts;
Three(Integer id, String msecs, List<String> srts)
{
this.id = id;
this.msecs = msecs;
this.srts = srts;
}
Three doSomething() {
// here you can do something with your data,
// e.g. split msecs on "-->" and check times
return this;
}
void write(BufferedWriter r) throws IOException
{
r.write(id);
r.newLine();
r.write(msecs);
r.newLine();
for (String s : srts) {
r.write(s);
r.newLine();
}
r.newLine();
}
}
public static void main(String[] args) throws IOException
{
String baseDir = "/dir/where/resides/srt";
String filename = "filename.srt";
int elemPerChunk = 50;
int fileNum = 0;
File dir = new File(baseDir);
Chunker chunker = new Chunker(dir, filename);
boolean completed = false;
while (!completed) {
int srtCount = 0;
File f = new File(baseDir, "ch." + (fileNum++) + "." + filename);
BufferedWriter w = new BufferedWriter(new FileWriter(f));
try {
while (srtCount++ < elemPerChunk) {
chunker.readThree().doSomething().write(w);
}
} catch (NullPointerException e) {
completed = true;
}
w.close();
}
}
}

Make faster a read from file

I'm going to pass my data from MongoDB to Neo4j.
So, I exported my MongoDB documents in .csv. As you can read here I have a problem with the array uniform.
So I wrote a java program to fix this problem.
Here is the .csv exported from MongoDB (note the different about uniform array):
_id,official_name,common_name,country,started_by.day,started_by.month,started_by.year,championship,stadium.name,stadium.capacity,palmares.first_prize,palmares.second_prize,palmares.third_prize,palmares.fourth_prize,average_age,squad_value,foreigners,uniform
0,yaDIXxLAOV,WWYWLqPcYM,QsVwiNmeGl,7,9,1479,oYKGgstIMv,qskcxizCkd,8560,10,25,9,29,16,58,6,"[""first_colour"",""second_colour"",""third_colour""]"
Here is how it must be to import in Neo4j:
_id,official_name,common_name,country,started_by.day,started_by.month,started_by.year,championship,stadium.name,stadium.capacity,palmares.first_prize,palmares.second_prize,palmares.third_prize,palmares.fourth_prize,average_age,squad_value,foreigners,uniform.0,uniform.1,uniform.2
0,yaDIXxLAOV,WWYWLqPcYM,QsVwiNmeGl,7,9,1479,oYKGgstIMv,qskcxizCkd,8560,10,25,9,29,16,58,6,first_colour,second_colour,third_colour
My code works, but I have to convert 500k line of the .csv file and the program it is too much slow(it's still working after 20 minutes :/):
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.PrintWriter;
public class ConvertireCSV {
public static void main(String[] args) throws IOException {
FileReader f;
f=new FileReader("output.csv");
BufferedReader b;
b=new BufferedReader(f);
String firstLine= b.readLine();
int uniform = firstLine.indexOf("uniform");
firstLine=firstLine.substring(0, uniform);
firstLine = firstLine + "uniform.0,uniform.1,uniform.2\n";
String line="";
String csv="";
while(true) {
line=b.readLine();
if(line==null)
break;
int u = line.indexOf("\"[");
line=line.substring(0, u);
line=line + "first_colour,second_colour,third_colour \n";
csv=csv+line;
}
File file = new File("outputForNeo4j.csv");
if(file.createNewFile()) {
PrintWriter pw = new PrintWriter(file);
pw.println(firstLine + csv);
System.out.println("New file \"outputForNeo4j.csv\" created.");
pw.flush();
pw.close();
}
}
}
How can I make it faster?

Okay some basic ways to improve your code:
Make sure that your variables got the minimal scope required. If you don't need line outside your loop, don't declare it outside your loop.
Concatenation of simple strings is in general slow. Use a StringBuilder to speed things to there.
Why are you buffering the string anyway? Seems like a waste of memory. Just open the output stream to your target file and write the lines to the new file as you process them.
Examples:
I don't think you need a example on the first point.
For the second things could look like this:
...
StringBuilder csv = new StringBuilder();
while(true) {
...
csv.append(line);
}
...
if(file.createNewFile()) {
...
pw.println(firstLine + csv.toString());
...
}
For the third point the rewriting would be a little more extensive:
public static void main(String[] args) throws IOException {
FileReader f;
f=new FileReader("output.csv");
BufferedReader b;
b=new BufferedReader(f);
String firstLine= b.readLine();
int uniform = firstLine.indexOf("uniform");
firstLine=firstLine.substring(0, uniform);
firstLine = firstLine + "uniform.0,uniform.1,uniform.2\n";
File file = new File("outputForNeo4j.csv");
if(!file.createNewFile()) {
// all work would be for nothing! Bailing out.
return;
}
PrintWriter pw = new PrintWriter(file);
pw.print(firstLine);
while(true) {
String line=b.readLine();
if(line==null)
break;
int u = line.indexOf("\"[");
line=line.substring(0, u);
line=line + "first_colour,second_colour,third_colour \n";
pw.print(line);
}
System.out.println("New file \"outputForNeo4j.csv\" created.");
pw.flush();
pw.close();
b.close()
}

csv=csv+line;
string concatenation is expensive operation. I would suggest using bufferedWriter.
something like this:
FileReader f;
f=new FileReader("output.csv");
BufferedReader b;
BufferedWriter out;
b=new BufferedReader(f);
try{
out = new BufferedWriter(new FileWriter("outputForNeo4j.csv"));
} catch(Exception e){
//cannot create file
}
System.out.println("New file \"outputForNeo4j.csv\" created.");
String firstLine= b.readLine();
int uniform = firstLine.indexOf("uniform");
firstLine=firstLine.substring(0, uniform);
firstLine = firstLine + "uniform.0,uniform.1,uniform.2\n";
String line="";
String csv="";
out.write(firstLine);
while(true) {
line=b.readLine();
if(line==null)
break;
int u = line.indexOf("\"[");
line=line.substring(0, u);
line=line + "first_colour,second_colour,third_colour \n";
out.write(line);
}
out.flush();
}

Results :
test0 : Runs: 241 iterations ,avarage milis = 246
test1 : Runs: 249 iterations ,avarage milis = 118
test2 : Runs: 269 iterations ,avarage milis = 5
test3 : Runs: 241 iterations ,avarage milis = 2
import java.io.*;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.Random;
public class Tester {
private static final String filePath = "c:\\bigFile.txt";
//private static final String filePath = "c:\\bigfileNewLine.txt";
private static final int numOfMethods = 4;
private static final int numOfIter = 1000;
public Tester() throws NoSuchMethodException {
System.out.println("Tester.Tester");
int[] milisArr = new int [numOfMethods];
int[] actualRun = new int [numOfMethods];
Random rnd = new Random(System.currentTimeMillis());
Long startMs = 0l, endMs = 0l;
Method[] method = new Method[numOfMethods];
for (int i = 0; i < numOfMethods; i++)
method[i] = this.getClass().getMethod("test" + i);
int testCount = 0;
while (testCount++ < numOfIter) {
int testMethod = rnd.nextInt(numOfMethods);
Method m = method[testMethod];
try {
System.gc();
startMs = System.currentTimeMillis();
String retval = (String) m.invoke(null);
endMs = System.currentTimeMillis();
} catch (IllegalAccessException e) {
e.printStackTrace(); //To change body of catch statement use File | Settings | File Templates.
} catch (InvocationTargetException e) {
e.printStackTrace(); //To change body of catch statement use File | Settings | File Templates.
}
milisArr[testMethod] += (endMs - startMs);
actualRun[testMethod]++;
System.out.println("Test name: " + m.getName() + " testCount=" + testCount + " Of " + numOfIter + " iteration, Total time :" + (endMs - startMs) / 1000.0 + " seconds");
}
System.out.println("Test Summery :");
for (int i = 0; i < numOfMethods; i++)
System.out.println("test" + i + " : Runs: " + actualRun[i] + " iterations ,avarage milis = " + milisArr[i]/numOfIter);
}
public static String test0() throws IOException {
InputStream file = getInputStream();
StringBuffer textBuffer = new StringBuffer();
int c;
while ((c = file.read()) != -1)
textBuffer.append((char) c);
file.close();
return textBuffer.toString();
}
public static String test1() throws IOException {
Reader reader = new FileReader(new File(filePath));
BufferedReader br = new BufferedReader(reader);
String line = br.readLine();
String result = line;
while (line != null) {
line = br.readLine();
if (line == null) {
} else {
result = result + "\n" + line;
}
}
br.close();
reader.close();
return result;
}
public static String test2() throws IOException {
byte[] buf = new byte[1024];
int l;
InputStream is = getInputStream();
StringBuffer tmpBuf = new StringBuffer();
while ((l = is.read(buf)) != -1) {
tmpBuf.append(new String(buf, 0, l));
}
is.close();
return tmpBuf.toString();
}
public static String test3() throws IOException {
File source = new File(filePath);
final DataInputStream dis = new DataInputStream(new BufferedInputStream(new FileInputStream(source)));
final byte[] buffer = new byte[(int) source.length()];
dis.readFully(buffer);
dis.close();
return new String(buffer, "UTF-8");
}
private static InputStream getInputStream() {
try {
return new FileInputStream(filePath);
} catch (FileNotFoundException e) {
e.printStackTrace();
return null;
}
}
public static void main(String[] args) {
try {
new Tester();
} catch (NoSuchMethodException e) {
e.printStackTrace(); //To change body of catch statement use File | Settings | File Templates.
}
}
}

Splitting and Merging large files (size in GB) in Java

Suppose,
I am splitting 2590400 KB (approx 2.5 GB) file in 30 parts.
It will produce 30 files with size of 86347 KB.
Which seems correct, 2590400/30 = 86346.66666667
Now if I merge all the parts (30) again it is producing the file of 3453873 KB file, which should be 2590410 KB.
Can anyone help me why this difference is there? I am using below code for merge and split files.
SplitFile.java
import java.io.BufferedOutputStream;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.RandomAccessFile;
/**
* #author vishal.zanzrukia
*
*/
public class SplitFile {
public static final String INPUT_FILE = "D:\\me\\projects\\input\\file\\path.txt";
public static final int NUMBER_OF_OUTPUT_FILES = 30;
public static final String FILE_SUFFIX = ".txt";
/**
* split file
*
* #throws Exception
*/
static void splitFile() throws Exception{
File inputFile = new File(INPUT_FILE + "_Splits");
inputFile.mkdir();
RandomAccessFile raf = new RandomAccessFile(INPUT_FILE, "r");
long sourceSize = raf.length();
long bytesPerSplit = sourceSize / NUMBER_OF_OUTPUT_FILES;
long remainingBytes = sourceSize % NUMBER_OF_OUTPUT_FILES;
int maxReadBufferSize = 8 * 1024; // 8KB
for (int destIx = 1; destIx <= NUMBER_OF_OUTPUT_FILES; destIx++) {
BufferedOutputStream bw = new BufferedOutputStream(new FileOutputStream(INPUT_FILE + "_Splits\\split." + destIx + FILE_SUFFIX));
if (bytesPerSplit > maxReadBufferSize) {
long numReads = bytesPerSplit / maxReadBufferSize;
long numRemainingRead = bytesPerSplit % maxReadBufferSize;
for (int i = 0; i < numReads; i++) {
readWrite(raf, bw, maxReadBufferSize);
}
if (numRemainingRead > 0) {
readWrite(raf, bw, numRemainingRead);
}
} else {
readWrite(raf, bw, bytesPerSplit);
}
bw.close();
}
if (remainingBytes > 0) {
BufferedOutputStream bw = new BufferedOutputStream(new FileOutputStream("split." + NUMBER_OF_OUTPUT_FILES + 1));
readWrite(raf, bw, remainingBytes);
bw.close();
}
raf.close();
}
/**
* join file
*
* #throws Exception
*/
static void joinFiles() throws Exception{
int maxReadBufferSize = 8 * 1024;
BufferedOutputStream bw = new BufferedOutputStream(new FileOutputStream(INPUT_FILE + "_Splits\\fullJoin" + FILE_SUFFIX));
File inputFileDir = new File(INPUT_FILE + "_Splits");
RandomAccessFile raf = null;
if(inputFileDir.isDirectory()){
for(File file : inputFileDir.listFiles()){
raf = new RandomAccessFile(file, "r");
long numReads = raf.length() / maxReadBufferSize;
long numRemainingRead = raf.length() % maxReadBufferSize;
for (int i = 0; i < numReads; i++) {
readWrite(raf, bw, maxReadBufferSize);
}
if (numRemainingRead > 0) {
readWrite(raf, bw, numRemainingRead);
}
raf.close();
}
}
bw.close();
}
public static void mergeFiles() {
File[] files = new File[NUMBER_OF_OUTPUT_FILES];
for(int i=1;i<=NUMBER_OF_OUTPUT_FILES;i++){
files[i-1] = new File(INPUT_FILE + "_Splits\\split."+i+FILE_SUFFIX);
}
String mergedFilePath = INPUT_FILE + "_Splits\\fullJoin" + FILE_SUFFIX;
File mergedFile = new File(mergedFilePath);
mergeFiles(files, mergedFile);
}
public static void mergeFiles(File[] files, File mergedFile) {
FileWriter fstream = null;
BufferedWriter out = null;
try {
fstream = new FileWriter(mergedFile, true);
out = new BufferedWriter(fstream);
} catch (IOException e1) {
e1.printStackTrace();
}
for (File f : files) {
System.out.println("merging: " + f.getName());
FileInputStream fis;
try {
fis = new FileInputStream(f);
BufferedReader in = new BufferedReader(new InputStreamReader(fis));
String aLine;
while ((aLine = in.readLine()) != null) {
out.write(aLine);
out.newLine();
}
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
try {
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
public static void main(String[] args) throws Exception {
// splitFile();
mergeFiles();
}
static void readWrite(RandomAccessFile raf, BufferedOutputStream bw, long numBytes) throws IOException {
byte[] buf = new byte[(int) numBytes];
int val = raf.read(buf);
if (val != -1) {
bw.write(buf);
}
}
}

Use your joinFiles method: don't try to read a file by line-by-line using a Reader if you want to keep it exactly like it was, because line endings may differ by platform.
Instead read them as a binary file using an InputStream or RandomAccessFile and write using an OutputStream.
The only problem in your joinFiles method is that it used File.listFiles() which makes no guarantees about the order in which the files are returned.
I combined your mergeFiles() code with joinFiles() to make this work (remember to invoke joinFiles() instead of mergeFiles() from your main method)
static void joinFiles(File[] files) throws Exception {
int maxReadBufferSize = 8 * 1024;
BufferedOutputStream bw = new BufferedOutputStream(new FileOutputStream(INPUT_FILE + "_Splits\\fullJoin"
+ FILE_SUFFIX));
RandomAccessFile raf = null;
for (File file : files) {
raf = new RandomAccessFile(file, "r");
long numReads = raf.length() / maxReadBufferSize;
long numRemainingRead = raf.length() % maxReadBufferSize;
for (int i = 0; i < numReads; i++) {
readWrite(raf, bw, maxReadBufferSize);
}
if (numRemainingRead > 0) {
readWrite(raf, bw, numRemainingRead);
}
raf.close();
}
bw.close();
}
public static void joinFiles() throws Exception {
File[] files = new File[NUMBER_OF_OUTPUT_FILES];
for (int i = 1; i <= NUMBER_OF_OUTPUT_FILES; i++) {
files[i - 1] = new File(INPUT_FILE + "_Splits\\split." + i + FILE_SUFFIX);
}
joinFiles(files);
}

The problem is the very last line of code:
static void readWrite(RandomAccessFile raf, BufferedOutputStream bw, long numBytes) throws IOException {
byte[] buf = new byte[(int) numBytes];
int val = raf.read(buf);
if (val != -1) {
bw.write(buf);
}
}
When you write, you write back numBytes of data, but the read function has usefully returned:
the total number of bytes read into the buffer, or -1 if there is no more data because the end of this file has been reached.
Therefore, your fix is to use a different write:
bw.write(buf, 0 val);

Byte Array not printing to file correctly

I'm creating a program that aims to take in n number of splits and split a file into that amount of sub-files. In my SplitFile.java, I'm reading a file, passing an array of substrings that show the text that's supposed to go in each split file. I then convert the string into a byte array and write the byte array to the split file, but each file I'm creating is outputting something just slightly different.
SplitFile.java
import java.io.*;
import java.nio.charset.Charset;
import java.util.ArrayList;
public class SplitFile
{
int numberOfSplits = 1;
File file;
String[] parts = new String[5];
public SplitFile(File file,int numberOfSplits)
{
this.file = file;
this.numberOfSplits = numberOfSplits;
}
public void FileSplitter() throws IOException
{
FileInputStream fis = new FileInputStream(file);
String fileText = readFile();
if(numberOfSplits == 2)
{
int mid = fileText.length() / 2;
parts[0] = fileText.substring(0, mid);
parts[1] = fileText.substring(mid);
}
else if(numberOfSplits == 3)
{
int third = fileText.length() / 3;
int secondThird = third + third;
parts[0] = fileText.substring(0, third);
parts[1] = fileText.substring(third, secondThird);
parts[2] = fileText.substring(secondThird);
}
for(int i = 1; i <= numberOfSplits; i++)
{
BufferedInputStream bis = new BufferedInputStream(new fileInputStream(file));
FileOutputStream out;
String name = file.getName();
byte[] b = parts[i - 1].getBytes(Charset.forName("UTF-8"));
int temp = 0;
while((temp = bis.read(b)) > 0);
{
File newFile = new File(name + " " + i + ".txt");
newFile.createNewFile();
out = new FileOutputStream(newFile);
out.write(b, 0, temp); // Writes to the file
out.close();
temp = 0;
}
}
}
public String readFile() throws IOException
{
BufferedReader br = new BufferedReader(new FileReader(file));
try
{
StringBuilder sb = new StringBuilder();
String line = br.readLine();
while (line != null)
{
sb.append(line);
sb.append("\n");
line = br.readLine();
}
return sb.toString();
}
finally
{
br.close();
}
}
}
If I pass in 2 as the amount of splits I want, it is not splitting it right at the middle, with file 1 being the first half and file 2 being the second half, and instead giving the end of the text file for both files. My problem seems to be here:
while((temp = bis.read(b)) > 0);
{
File newFile = new File(name + " " + i + ".txt");
newFile.createNewFile();
out = new FileOutputStream(newFile);
out.write(b, 0, temp); // Writes to the file
out.close();
temp = 0;
}
An example file I'll use on here is this file:
myFile.txt
abcdefghijklmnopqrstuvwxyz
It splits into two files that go as follows:
myFile.txt 1
nopqrstuvqxyz
myFile.txt 2
opqrstuvqxyz
Any idea on what the problem is?

In your code, you define File newFile = new File(name + " " + i + ".txt"); and out = new FileOutputStream(newFile); in whilte loop, it's not correct.
while((temp = bis.read(b)) > 0); not semicolon here =.="
Many mistake in your code
I will change your code like:
File newFile = new File(name + " " + i + ".txt");
newFile.createNewFile();
out = new FileOutputStream(newFile);
out.write(b); // Writes to the file
out.flush();
out.close();
If you need your code run as you want, here you are
for (int i = 1; i <= numberOfSplits; i++) {
String name = file.getName();
byte[] b = parts[i - 1].getBytes(Charset.forName("UTF-8"));
ByteArrayInputStream bis = new ByteArrayInputStream(b);
int temp = 0;
File newFile = new File(name + " " + i + ".txt");
newFile.createNewFile();
FileOutputStream out = new FileOutputStream(newFile);
while ((temp = bis.read()) > 0)
{
out.write(temp); // Writes to the file
}
out.flush();
out.close();
}

Read large file error "outofmemoryerror"(java)

sorry for my english. I want to read a large file, but when I read error occurs outOfMemoryError. I do not understand how to work with memory in the application. The following code does not work:
try {
StringBuilder fileData = new StringBuilder(1000);
BufferedReader reader = new BufferedReader(new FileReader(file));
char[] buf = new char[8192];
int bytesread = 0,
bytesBuffered = 0;
while( (bytesread = reader.read( buf )) > -1 ) {
String readData = String.valueOf(buf, 0, bytesread);
bytesBuffered += bytesread;
fileData.append(readData); //this is error
if (bytesBuffered > 1024 * 1024) {
bytesBuffered = 0;
}
}
System.out.println(fileData.toString().toCharArray());
} finally {
}

You need pre allocate a large buffer to avoid reallocate.
File file = ...;
StringBuilder fileData = new StringBuilder(file.size());
And running with large heap size:
java -Xmx2G
==== update
A while loop using buffer doesn't need too memory to run. Treat input like a stream, match your search string with the stream. It's a really simple state machine. If you need search multiple words, you can find a TrieTree implementation(support stream) for that.
// the match state model
...xxxxxxabxxxxxaxxxxxabcdexxxx...
ab a abcd
File file = new File("path_to_your_file");
String yourSearchWord = "abcd";
int matchIndex = 0;
boolean matchPrefix = false;
try (BufferedReader reader = new BufferedReader(new FileReader(file))) {
int chr;
while ((chr = reader.read()) != -1) {
if (matchPrefix == false) {
char searchChar = yourSearchWord.charAt(0);
if (chr == searchChar) {
matchPrefix = true;
matchIndex = 0;
}
} else {
char searchChar = yourSearchWord.charAt(++matchIndex);
if (chr == searchChar) {
if (matchIndex == yourSearchWord.length() - 1) {
// match!!
System.out.println("match: " + matchIndex);
matchPrefix = false;
matchIndex = 0;
}
} else {
matchPrefix = false;
matchIndex = 0;
}
}
}
}

Try this. This might be helpful :-
try{
BufferedReader reader = new BufferedReader(new FileReader(file));
String txt = "";
while( (txt = reader.read()) != null){
System.out.println(txt);
}
}catch(Exception e){
System.out.println("Error : "+e.getMessage());
}

You should not hold such big files in memory, because you run out of it, as you see. Since you use Java 7, you need to read the file manually as stream and check the content on the fly. Otherwise you could use the stream API of Java 8. This is just an example. It works, but keep in mind, that the position of the found word could vary due to encoding issues, so this is no production code:
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
public class FileReader
{
private static String wordToFind = "SEARCHED_WORD";
private static File file = new File("YOUR_FILE");
private static int currentMatchingPosition;
private static int foundAtPosition = -1;
private static int charsRead;
public static void main(String[] args) throws IOException
{
try (FileInputStream fis = new FileInputStream(file))
{
System.out.println("Total size to read (in bytes) : " + fis.available());
int c;
while ((c = fis.read()) != -1)
{
charsRead++;
checkContent(c);
}
if (foundAtPosition > -1)
{
System.out.println("Found word at position: " + (foundAtPosition - wordToFind.length()));
}
else
{
System.out.println("Didnt't find the word!");
}
}
catch (IOException e)
{
e.printStackTrace();
}
}
private static void checkContent(int c)
{
if (currentMatchingPosition >= wordToFind.length())
{
//already found....
return;
}
if (wordToFind.charAt(currentMatchingPosition) == (char)c)
{
foundAtPosition = charsRead;
currentMatchingPosition++;
}
else
{
currentMatchingPosition = 0;
foundAtPosition = -1;
}
}
}