reading file java.lang.OutOfMemoryError - java

Trying to find a word in a large file. File is read line by line. When reading the way redLine exception is thrown. Are there any way around this? You can read it on the floor as a string?
for(String line; (line = fileOut.readLine()) != null; ){
if(line.contains(commandString))
System.out.println(count + ": " + line);
count++;
}
java.lang.OutOfMemoryError:
UDP:
this is all my bad code:
static String file = "files.txt";
static String commandString = "first";
static int count = 1;
public static void main(String[] args) throws IOException
{
try(BufferedReader fileOut = new BufferedReader(new InputStreamReader(new FileInputStream(file), "Cp1251")) ){
for(String line; (line = fileOut.readLine()) != null; ){
if(line.contains(commandString))
System.out.println(count + ": " + line);
count++;
}
System.out.println("before wr close :" + Runtime.getRuntime().freeMemory());
fileOut.close();
}catch(Exception e) {
System.out.println(e);
}
}

Searching for a word, you can read the file bytewise without holding more than a single byte of the file in memory.
Read byte by byte and every time, a byte is equal to the first byte of the searched word, start a second loop and read the following bytes and check if the next byte is equal to the next byte in the word and so on.
To give you an example, I have modified an sample to your needs.
I've omitted on the output of the file, because I don't know, if you want to output all lines or only those which contains your keyword and the latter might be as problematic as reading the code line by line.
static String fileName = "files.txt";
static byte[] searchString = { 'f', 'i', 'r', 's', 't' };
static int count = 0;
static long position = 1;
public static void main(String[] args) throws IOException {
try (FileInputStream file = new FileInputStream(fileName)) {
byte read[] = new byte[1];
outerLoop: while (-1 < file.read(read, 0, 1)) {
position++;
if (read[0] == searchString[0]) {
int matches = 1;
for (int i = 1; i < searchString.length; i++) {
if (-1 > file.read(read, 0, 1)) {
break outerLoop;
}
position++;
if (read[0] == searchString[i]) {
matches++;
} else {
break;
}
}
if (matches == searchString.length) {
System.out.println((++count)+". found at position "+ (position-matches));
}
}
}
file.close();
} catch (Exception e) {
e.printStackTrace();
}
}

Related

Manipulate this code so that it counts the # of digits in a file

I need to manipulate this code so that it will read the # of digits from a file.
I am honestly stumped on this one for some reason. Do i need to tokenize it first?
Thanks!
import java.io.*;
import java.util.*;
public class CountLetters {
public static void main(String args[]) {
if (args.length != 1) {
System.err.println("Synopsis: Java CountLetters inputFileName");
System.exit(1);
}
String line = null;
int numCount = 0;
try {
FileReader f = new FileReader(args[0]);
BufferedReader in = new BufferedReader(f);
while ((line = in.readLine()) != null) {
for (int k = 0; k < line.length(); ++k)
if (line.charAt(k) >= 0 && line.charAt(k) <= 9)
++numCount;
}
in.close();
f.close();
} catch (Exception e) {
e.printStackTrace();
}
System.out.println(numCount + " numbers in this file.");
} // main
} // CountNumbers
Use '' to indicate a char constant (you are comparing chars to ints), also I would suggest you use try-with-resources Statement to avoid explicit close calls and please avoid using one line loops without braces (unless you are using lambdas). Like
public static void main(String args[]) {
if (args.length != 1) {
System.err.println("Synopsis: Java CountLetters inputFileName");
System.exit(1);
}
String line = null;
int numCount = 0;
try (BufferedReader in = new BufferedReader(new FileReader(args[0]))) {
while ((line = in.readLine()) != null) {
for (int k = 0; k < line.length(); ++k) {
if ((line.charAt(k) >= '0' && line.charAt(k) <= '9')) {
++numCount;
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
System.out.println(numCount + " numbers in this file.");
} // main
Also, you could use a regular expression to remove all non-digits (\\D) and add the length of the resulting String (which is all-digits). Like,
while ((line = in.readLine()) != null) {
numCount += line.replaceAll("\\D", "").length();
}
Use if(Charachter.isDigit(char)) replace char with each char, this will count each number, and I believe arabic numbers as well.

Storing an array of strings without initializing the size

Background: This program reads in a text file and replaces a word in the file with user input.
Problem: I am trying to read in a line of text from a text file and store the words into an array.
Right now the array size is hard-coded with an number of indexes for test purposes, but I want to make the array capable of reading in a text file of any size instead.
Here is my code.
public class FTR {
public static Scanner input = new Scanner(System.in);
public static Scanner input2 = new Scanner(System.in);
public static String fileName = "C:\\Users\\...";
public static String userInput, userInput2;
public static StringTokenizer line;
public static String array_of_words[] = new String[19]; //hard-coded
/* main */
public static void main(String[] args) {
readFile(fileName);
wordSearch(fileName);
replace(fileName);
}//main
/*
* method: readFile
*/
public static void readFile(String fileName) {
try {
FileReader file = new FileReader(fileName);
BufferedReader read = new BufferedReader(file);
String line_of_text = read.readLine();
while (line_of_text != null) {
System.out.println(line_of_text);
line_of_text = read.readLine();
}
} catch (Exception e) {
System.out.println("Unable to read file: " + fileName);
System.exit(0);
}
System.out.println("**************************************************");
}
/*
* method: wordSearch
*/
public static void wordSearch(String fileName) {
int amount = 0;
System.out.println("What word do you want to find?");
userInput = input.nextLine();
try {
FileReader file = new FileReader(fileName);
BufferedReader read = new BufferedReader(file);
String line_of_text = read.readLine();
while (line_of_text != null) { //there is a line to read
System.out.println(line_of_text);
line = new StringTokenizer(line_of_text); //tokenize the line into words
while (line.hasMoreTokens()) { //check if line has more words
String word = line.nextToken(); //get the word
if (userInput.equalsIgnoreCase(word)) {
amount += 1; //count the word
}
}
line_of_text = read.readLine(); //read the next line
}
} catch (Exception e) {
System.out.println("Unable to read file: " + fileName);
System.exit(0);
}
if (amount == 0) { //if userInput was not found in the file
System.out.println("'" + userInput + "'" + " was not found.");
System.exit(0);
}
System.out.println("Search for word: " + userInput);
System.out.println("Found: " + amount);
}//wordSearch
/*
* method: replace
*/
public static void replace(String fileName) {
int amount = 0;
int i = 0;
System.out.println("What word do you want to replace?");
userInput2 = input2.nextLine();
System.out.println("Replace all " + "'" + userInput2 + "'" + " with " + "'" + userInput + "'");
try {
FileReader file = new FileReader(fileName);
BufferedReader read = new BufferedReader(file);
String line_of_text = read.readLine();
while (line_of_text != null) { //there is a line to read
line = new StringTokenizer(line_of_text); //tokenize the line into words
while (line.hasMoreTokens()) { //check if line has more words
String word = line.nextToken(); //get the word
if (userInput2.equalsIgnoreCase(word)) {
amount += 1; //count the word
word = userInput;
}
array_of_words[i] = word; //add word to index in array
System.out.println("WORD: " + word + " was stored in array[" + i + "]");
i++; //increment array index
}
//THIS IS WHERE THE PRINTING HAPPENS
System.out.println("ARRAY ELEMENTS: " + Arrays.toString(array_of_words));
line_of_text = read.readLine(); //read the next line
}
BufferedWriter outputWriter = null;
outputWriter = new BufferedWriter(new FileWriter("C:\\Users\\..."));
for (i = 0; i < array_of_words.length; i++) { //go through the array
outputWriter.write(array_of_words[i] + " "); //write word from array to file
}
outputWriter.flush();
outputWriter.close();
} catch (Exception e) {
System.out.println("Unable to read file: " + fileName);
System.exit(0);
}
if (amount == 0) { //if userInput was not found in the file
System.out.println("'" + userInput2 + "'" + " was not found.");
System.exit(0);
}
}//replace
}//FTR
You can use java.util.ArrayList (which dynamically grows unlike an array with fixed size) to store the string objects (test file lines) by replacing your array with the below code:
public static List<String> array_of_words = new java.util.ArrayList<>();
You need to use add(string) to add a line (string) and get(index) to retrieve the line (string)
Please refer the below link for more details:
http://docs.oracle.com/javase/8/docs/api/java/util/ArrayList.html
You may want to give a try to ArrayList.
In Java normal arrays cannot be initialized without giving initial size and they cannot be expanded during run time. Whereas ArrayLists have resizable-array implementation of the List interface.ArrayList also comes with number of useful builtin functions such as
Size()
isEmpty()
contains()
clone()
and others. On top of these you can always convert your ArrayList to simple array using ArrayList function toArray(). Hope this answers your question. I'll prepare some code and share with you to further explain things you can achieve using List interface.
Use not native [] arrays but any kind of java collections
List<String> fileContent = Files.readAllLines(Paths.get(fileName));
fileContent.stream().forEach(System.out::println);
long amount = fileContent.stream()
.flatMap(line -> Arrays.stream(line.split(" +")))
.filter(word -> word.equalsIgnoreCase(userInput))
.count();
List<String> words = fileContent.stream()
.flatMap(line -> Arrays.stream(line.split(" +")))
.filter(word -> word.length() > 0)
.map(word -> word.equalsIgnoreCase(userInput) ? userInput2 : word)
.collect(Collectors.toList());
Files.write(Paths.get(fileName), String.join(" ", words).getBytes());
of course you can works with such lists more traditionally, with loops
for(String line: fileContent) {
...
}
or even
for (int i = 0; i < fileContent.size(); ++i) {
String line = fileContent.get(i);
...
}
i just like streams :)

Made a wc program in Java, but it will not count end of line character

So I have pretty much completed (I think) my wc program in Java, that takes a filename from a user input (even multiple), and counts the lines, words, bytes (number of characters) from the file. There were 2 files provided for testing purposes, and they are in a .dat format, being readable from dos/linux command lines. Everything is working properly except for the count when there are \n or \r\n characters at the end of line. It will not count these. Please help?
import java.io.*;
import java.util.regex.Pattern;
public class Prog03 {
private static int totalWords = 0, currentWords = 0;
private static int totalLines =0, currentLines = 0;
private static int totalBytes = 0, currentBytes = 0;
public static void main(String[] args) {
System.out.println("This program determines the quantity of lines, words, and bytes\n" +
"in a file or files that you specify.\n" +
"\nPlease enter one or more file names, comma-separated: ");
getFileName();
System.out.println();
} // End of main method.
public static void countSingle (String fileName, BufferedReader in) {
try {
String line;
String[] words;
//int totalWords = 0;
int totalWords1 = 0;
int lines = 0;
int chars = 0;
while ((line = in.readLine()) != null) {
lines++;
currentLines = lines;
chars += line.length();
currentBytes = chars;
words = line.split(" ");
totalWords1 += countWords(line);
currentWords = totalWords1;
} // End of while loop.
System.out.println(currentLines + "\t\t" + currentWords + "\t\t" + currentBytes + "\t\t"
+ fileName);
} catch (Exception ex) {
ex.printStackTrace();
}
}
public static void countMultiple(String fileName, BufferedReader in) {
try {
String line;
String[] words;
int totalWords1 = 0;
int lines = 0;
int chars = 0;
while ((line = in.readLine()) != null) {
lines++;
currentLines = lines;
chars += line.length();
currentBytes = chars;
words = line.split(" ");
totalWords1 += countWords(line);
currentWords = totalWords1;
} // End of while loop.
totalLines += currentLines;
totalBytes += currentBytes;
totalWords += totalWords1;
} catch (Exception ex) {
ex.printStackTrace();
}
} // End of method count().
private static long countWords(String line) {
long numWords = 0;
int index = 0;
boolean prevWhitespace = true;
while (index < line.length()) {
char c = line.charAt(index++);
boolean currWhitespace = Character.isWhitespace(c);
if (prevWhitespace && !currWhitespace) {
numWords++;
}
prevWhitespace = currWhitespace;
}
return numWords;
} // End of method countWords().
private static void getFileName() {
BufferedReader in ;
try {
in = new BufferedReader(new InputStreamReader(System.in));
String fileName = in.readLine();
String [] files = fileName.split(", ");
System.out.println("Lines\t\tWords\t\tBytes" +
"\n--------\t--------\t--------");
for (int i = 0; i < files.length; i++) {
FileReader fileReader = new FileReader(files[i]);
in = new BufferedReader(fileReader);
if (files.length == 1) {
countSingle(files[0], in);
in.close();
}
else {
countMultiple(files[i], in);
System.out.println(currentLines + "\t\t" +
currentWords + "\t\t" + currentBytes + "\t\t"
+ files[i]);
in.close();
}
}
if (files.length > 1) {
System.out.println("----------------------------------------" +
"\n" + totalLines + "\t\t" + totalWords + "\t\t" + totalBytes + "\t\tTotals");
}
}
catch (FileNotFoundException ioe) {
System.out.println("The specified file was not found. Please recheck "
+ "the spelling and try again.");
ioe.printStackTrace();
}
catch (IOException ioe) {
ioe.printStackTrace();
}
}
} // End of class
that is the entire program, if anyone helping should need to see anything, however this is where I count the length of each string in a line (and I assumed that the eol characters would be part of this count, but they aren't.)
public static void countMultiple(String fileName, BufferedReader in) {
try {
String line;
String[] words;
int totalWords1 = 0;
int lines = 0;
int chars = 0;
while ((line = in.readLine()) != null) {
lines++;
currentLines = lines;
**chars += line.length();**
currentBytes = chars;
words = line.split(" ");
totalWords1 += countWords(line);
currentWords = totalWords1;
} // End of while loop.
totalLines += currentLines;
totalBytes += currentBytes;
totalWords += totalWords1;
} catch (Exception ex) {
ex.printStackTrace();
}
}
BufferedReader always ignores new line or line break character. There is no way to do this using readLine().
You can use read() method instead. But in that case you have to read each character individually.
just a comment, to split a line to words, it is not enough to split based on single space: line.split(" "); you will miss if there are multiple spaces or tabs between words. better to do split on any whitespace char line.split("\\s+");

swap first and last column of a 2D array in a text file in java

Input file-
5
1 2 3 4 5
2 3 4 5 6
3 2 1 5 8
My work is i should read this input file my.txt and swap its first and last column and output it to another file comp.txt but I am getting a blank file comp.txt
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.PrintStream;
import java.io.*;
// to swap first and last column of 2D array
public class Swap
{
private static BufferedReader in = null;
private static int row = 0;
private static int column = 0;
private static int[][] matrix = null;
public static void main(String[] args) throws Exception
{
try
{
// String filepath = args[0];
int lineNum = 0;
int row = 0;
in = new BufferedReader(new FileReader("my.txt"));
String line = null;
while ((line = in.readLine()) != null)
{
lineNum++;
if (lineNum == 1)
{
column = Integer.parseInt(line);
}
else
{
String[] tokens = line.split(",");
for (int j = 0; j < tokens.length; j++)
{
if (j == 0) matrix[row][0] = Integer.parseInt(tokens[column]);
else matrix[row][j] = Integer.parseInt(tokens[j]);
}
row++;
}
}
}
catch (Exception ex)
{
System.out.println("The code throws an exception");
System.out.println(ex.getMessage());
}
finally
{
if (in != null) in.close();
}
try
{
PrintStream output = new PrintStream(new File("comp.txt"));
for (int i = 0; i < row; i++)
{
for (int j = 0; j < column; j++)
{
output.println(matrix[i][j] + " ");
}
}
output.close();
}
catch (FileNotFoundException e)
{
e.printStackTrace();
}
}
}
Here is the output I am getting at console-
The code throws an exception
null
Apart from this I am getting a blank comp.txt file
After I replaced your catch block with
catch (Exception ex)
{
System.out.println("The code throws an exception");
System.out.println(ex.getMessage());
ex.printStackTrace();
}
I could tell that the error is
java.lang.NullPointerException
at test.example.code.Swap.main(Swap.java:37)
The code throws an exception
null
Where the line is
if (j == 0) matrix[row][0] = Integer.parseInt(tokens[column]);
Because you are trying to access the 5th column of tokens but it's actually a 5-length array and indices in Java and most C-based languages in general are zero-indiced aka the valid indices are from 0 to 4.
Also, I'm not entirely sure what you are trying to do with this line, so I won't fix it, but that's the error.
EDIT:
Now this is not exactly a legitimate thing to do, but I solved it as I would have done it myself, and I will explain it afterwards. If you mustn't read the code because your "mentor" would be against it, then just read the final lines of this post and solve it on your own.
// to swap first and last column of 2D array
public class Swap
{
private static BufferedReader in = null;
private static int column = 0;
private static List<int[]> matrix = null;
public static void main(String[] args) throws Exception
{
try
{
// String filepath = args[0];
in = new BufferedReader(new FileReader("my.txt"));
String line = null;
matrix = new ArrayList<int[]>(); //array length is variable length
//so using a 2D array without knowing the row size is not right
boolean isFirstLine = true;
while ((line = in.readLine()) != null)
{
if (isFirstLine)
{
column = Integer.parseInt(line); //first line determines column length
isFirstLine = false;
}
else
{
String[] tokens = line.split(" "); // there are no commas in input file so split on spaces instead
matrix.add(new int[column]);
for (int i = 0; i < tokens.length; i++)
{
matrix.get(matrix.size() - 1)[i] = Integer.parseInt(tokens[i]);
//store the lines of the currently read line in the latest added array of the list
}
}
}
}
catch (Exception ex)
{
System.out.println("The code throws an exception");
ex.printStackTrace(); //changed exception write out for better info
}
finally
{
if (in != null)
in.close();
}
// swapping the elements of first and last in each int array of list
for(int[] intLines : matrix)
{
int temp = intLines[0];
intLines[0] = intLines[intLines.length-1];
intLines[intLines.length-1] = temp;
}
BufferedWriter output = null;
try
{
output = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(new File(
"comp.txt")))); //i just prefer bufferedwriters don't mind me
output.write("" + column); //write the first line that is the array length into file
output.newLine();
for (int[] intLines : matrix) //write out each line into file
{
for (int i = 0; i < intLines.length; i++)
{
output.write("" + intLines[i]);
if (i < (intLines.length - 1))
{
output.write(" ");
}
}
output.newLine();
}
}
catch (FileNotFoundException e)
{
e.printStackTrace();
}
finally
{
if (output != null)
{
try
{
output.close();
}
catch (IOException e)
{
}
}
}
}
}
So basically I changed the 2D array to a list of arrays, read in the file, fixed that you were splitting along commas instead of spaces, and actually did the swap. You don't necessarily have to take this approach, especially if that is against the rules. Don't even read the code if you mustn't.
1) Change
System.out.println("The code throws an exception");
System.out.println(ex.getMessage());
To get detailed description for cause of exception.
System.out.println("The code throws an exception");
ex.printStackTrace();
2) From code & exception message it seems like there is a NullPointerException as ur array matrix is initialized to null and not to array of matching size.
So solution to the exception is[Assuming your core logic is fine] , before assigning any values to matrix array, initialize with relevant size i.e with max rows & columns

Splitting a Byte Array on A Particular Byte

I am trying to read an old .dat file byte by byte, and have run into an issue: a record is terminated by \n (newline). I'd like to read in the whole byte array, then split it on the character.
I can do this by reading the whole byte array from the file, creating a String with the contents of the byte array, then calling String.split(), but find this to be inefficient. I'd rather split the byte array directly if possible.
Can anyone assist?
Update: Code was requested.
public class NgcReader {
public static void main(String[] args) {
String location;
if (System.getProperty("os.name").contains("Windows")) {
location = "F:\\Programming\\Projects\\readngc\\src\\main\\java\\ngcreader\\catalog.dat";
} else {
location = "/media/My Passport/Programming/Projects/readngc/src/main/java/ngcreader/catalog.dat";
}
File file = new File(location);
InputStream is = null;
try {
is = new FileInputStream(file);
} catch (FileNotFoundException e) {
System.out.println("It didn't work!");
System.exit(0);
}
byte[] fileByteArray = new byte[(int) file.length() - 1];
try {
is.read(fileByteArray);
is.close();
} catch (IOException e) {
System.out.println("IOException!");
System.exit(0);
}
// I do NOT like this. I'd rather split the byte array on the \n character
String bigString = new String(fileByteArray);
List<String> stringList = Arrays.asList(bigString.split("\\n"));
for (String record : stringList) {
System.out.print("Catalog number: " + record.substring(1, 6));
System.out.print(" Catalog type: " + record.substring(7, 9));
System.out.print(" Right Ascension: " + record.substring(10, 12) + "h " + record.substring(13, 17) + "min");
System.out.print(" Declination: " + record.substring(18, 21) + " " + record.substring(22, 24));
if (record.length() > 50) {
System.out.print(" Magnitude: " + record.substring(47, 51));
}
if (record.length() > 93) {
System.out.print(" Original Notes: " + record.substring(54,93));
}
if (record.length() > 150) {
System.out.print(" Palomar Notes: " + record.substring(95,150));
}
if (record.length() > 151) {
System.out.print(" Notes: " + record.substring(152));
}
System.out.println();
}
}
Another Update: Here's a README with a description of the file I'm processing:
http://cdsarc.u-strasbg.fr/viz-bin/Cat?VII/1B
It sounds like this might actually just be a text file to start with, in which case:
InputStream stream = new FileInputStream(location);
try {
BufferedReader reader = new BufferedReader(new InputStreamReader(stream,
"ASCII"));
String line;
while ((line = reader.readLine()) != null) {
// Handle the line, ideally in a separate method
}
} finally {
stream.close();
}
This way you never need to have more than a single line of the file in memory at a time.
if you're set on using byte arrays...
byte[] buff = new byte[1024];//smaller buffer
try {
int ind=0,from=0,read;
while((read=is.read(buff,ind,buff.length-ind))!=-1){
for(int i=ind;i<ind+read;i++){
if(buff[i]=='\n'){
string record = new String(buff,from,i+1);
//handle
from=i+1;
}
}
System.arraycopy(buff,from,buff,0,buff.length-from);
ind=ind+read-from;
from=0;
}
} catch (IOException e) {
System.out.println("IOException!");
//System.exit(0);
throw RunTimeException(e);//cleaner way to die
} finally{
is.close();
}
this also avoids loading in the entire file and it puts the close inside a finally

Categories