Slow CSV row parsing and splitting - java

I'm trying to parse a csv of over 100,000 lines and the performance problems don't even let me get to the end of the file before hitting "Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded"
Is there something wrong, or any way I can improve?
public static List<String[]> readCSV(String filePath) throws IOException{
List<String[]> csvLine= new ArrayList<String[]>();
CSVReader reader = new CSVReader(new FileReader(filePath), '\n');
String[] row;
while((row = reader.readNext()) != null){
csvLine.add(removeWhiteSpace(row[0].toString().split(",")));
}
reader.close();
return csvLine;
}
private static String[] removeWhiteSpace(String[] split) {
for(int index =0; index < split.length;index++){
split[index] = split[index].trim();
}
return split;
}

First you are running out of memory because all rows are being added to a list.
Second you are using String.split() which is extremely slow.
Third never try processing CSV by writing your own parsing code as there are many edge cases around this format (need to handle escape of delimiter, quotes, etc).
The solution is to use a library for that, such as univocity-parsers. You should be able to read 1 million rows in less than a second.
To parse, just do this:
public static IterableResult<String[], ParsingContext> readCSV(String filePath) {
File file = new File(filePath);
//configure the parser here. By default all values are trimmed
CsvParserSettings parserSettings = new CsvParserSettings();
//create the parser
CsvParser parser = new CsvParser(parserSettings);
//create an iterable over rows. This will not load everything into memory.
IterableResult<String[], ParsingContext> rows = parser.iterate(file);
return rows;
}
Now you can use your method like this:
public static void main(String... args) {
IterableResult<String[], ParsingContext> rows = readCSV("c:/path/to/input.csv");
try {
for (String[] row : rows) {
//process the rows however you want
}
} finally {
//the parser closes itself but in case any errors processing the rows (outside of the control of the iterator), close the parser.
rows.getContext().stop();
}
}
This is just an example of how you can use the parser, but there are many different ways to use it.
Now for writing, you can do this:
public static void main(String... args) {
//this is your output file
File output = new File("c:/path/to/output.csv");
//configure the writer if you need to
CsvWriterSettings settings = new CsvWriterSettings();
//create the writer. Here we write to a file
CsvWriter writer = new CsvWriter(output, settings);
//get the row iterator
IterableResult<String[], ParsingContext> rows = readCSV("c:/temp");
try {
//do whatever you need to the rows here
for (String[] row : rows) {
//then write it each one to the output.
writer.writeRow(row);
}
} finally {
//cleanup
rows.getContext().stop();
writer.close();
}
}
If all you want is to read the data, modify it and write it back to another file, you can just do this:
public static void main(String... args) throws IOException {
CsvParserSettings parserSettings = new CsvParserSettings();
parserSettings.setProcessor(new AbstractRowProcessor() {
#Override
public void rowProcessed(String[] row, ParsingContext context) {
//modify the row data here.
}
});
CsvWriterSettings writerSettings = new CsvWriterSettings();
CsvRoutines routines = new CsvRoutines(parserSettings, writerSettings);
FileReader input = new FileReader("c:/path/to/input.csv");
FileWriter output = new FileWriter("c:/path/to/output.csv");
routines.parseAndWrite(input, output);
}
Hope this helps.
Disclaimer: I'm the author of this libary. It's open source and free (Apache 2.0 license).

Is a design error try to put such a large file in memory.
Depending of what you want to do, you should either write a new file processed, or put the lines into a dba.
This implements the first:
FileInputStream inputStream = null;
Scanner sc = null;
try {
inputStream = new FileInputStream(path);
sc = new Scanner(inputStream, "UTF-8");
while (sc.hasNextLine()) {
String line = sc.nextLine();
// System.out.println(line);
}
// note that Scanner suppresses exceptions
if (sc.ioException() != null) {
throw sc.ioException();
}
} finally {
if (inputStream != null) {
inputStream.close();
}
if (sc != null) {
sc.close();
}
}

Related

How to delete a line of string in a text file - Java [duplicate]

I'm looking for a small code snippet that will find a line in file and remove that line (not content but line) but could not find. So for example I have in a file following:
myFile.txt:
aaa
bbb
ccc
ddd
Need to have a function like this: public void removeLine(String lineContent), and if I pass
removeLine("bbb"), I get file like this:
myFile.txt:
aaa
ccc
ddd
This solution may not be optimal or pretty, but it works. It reads in an input file line by line, writing each line out to a temporary output file. Whenever it encounters a line that matches what you are looking for, it skips writing that one out. It then renames the output file. I have omitted error handling, closing of readers/writers, etc. from the example. I also assume there is no leading or trailing whitespace in the line you are looking for. Change the code around trim() as needed so you can find a match.
File inputFile = new File("myFile.txt");
File tempFile = new File("myTempFile.txt");
BufferedReader reader = new BufferedReader(new FileReader(inputFile));
BufferedWriter writer = new BufferedWriter(new FileWriter(tempFile));
String lineToRemove = "bbb";
String currentLine;
while((currentLine = reader.readLine()) != null) {
// trim newline when comparing with lineToRemove
String trimmedLine = currentLine.trim();
if(trimmedLine.equals(lineToRemove)) continue;
writer.write(currentLine + System.getProperty("line.separator"));
}
writer.close();
reader.close();
boolean successful = tempFile.renameTo(inputFile);
public void removeLineFromFile(String file, String lineToRemove) {
try {
File inFile = new File(file);
if (!inFile.isFile()) {
System.out.println("Parameter is not an existing file");
return;
}
//Construct the new file that will later be renamed to the original filename.
File tempFile = new File(inFile.getAbsolutePath() + ".tmp");
BufferedReader br = new BufferedReader(new FileReader(file));
PrintWriter pw = new PrintWriter(new FileWriter(tempFile));
String line = null;
//Read from the original file and write to the new
//unless content matches data to be removed.
while ((line = br.readLine()) != null) {
if (!line.trim().equals(lineToRemove)) {
pw.println(line);
pw.flush();
}
}
pw.close();
br.close();
//Delete the original file
if (!inFile.delete()) {
System.out.println("Could not delete file");
return;
}
//Rename the new file to the filename the original file had.
if (!tempFile.renameTo(inFile))
System.out.println("Could not rename file");
}
catch (FileNotFoundException ex) {
ex.printStackTrace();
}
catch (IOException ex) {
ex.printStackTrace();
}
}
This I have found on the internet.
You want to do something like the following:
Open the old file for reading
Open a new (temporary) file for writing
Iterate over the lines in the old file (probably using a BufferedReader)
For each line, check if it matches what you are supposed to remove
If it matches, do nothing
If it doesn't match, write it to the temporary file
When done, close both files
Delete the old file
Rename the temporary file to the name of the original file
(I won't write the actual code, since this looks like homework, but feel free to post other questions on specific bits that you have trouble with)
So, whenever I hear someone mention that they want to filter out text, I immediately think to go to Streams (mainly because there is a method called filter which filters exactly as you need it to). Another answer mentions using Streams with the Apache commons-io library, but I thought it would be worthwhile to show how this can be done in standard Java 8. Here is the simplest form:
public void removeLine(String lineContent) throws IOException
{
File file = new File("myFile.txt");
List<String> out = Files.lines(file.toPath())
.filter(line -> !line.contains(lineContent))
.collect(Collectors.toList());
Files.write(file.toPath(), out, StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING);
}
I think there isn't too much to explain there, basically Files.lines gets a Stream<String> of the lines of the file, filter takes out the lines we don't want, then collect puts all of the lines of the new file into a List. We then write the list over top of the existing file with Files.write, using the additional option TRUNCATE so the old contents of the file are replaced.
Of course, this approach has the downside of loading every line into memory as they all get stored into a List before being written back out. If we wanted to simply modify without storing, we would need to use some form of OutputStream to write each new line to a file as it passes through the stream, like this:
public void removeLine(String lineContent) throws IOException
{
File file = new File("myFile.txt");
File temp = new File("_temp_");
PrintWriter out = new PrintWriter(new FileWriter(temp));
Files.lines(file.toPath())
.filter(line -> !line.contains(lineContent))
.forEach(out::println);
out.flush();
out.close();
temp.renameTo(file);
}
Not much has been changed in this example. Basically, instead of using collect to gather the file contents into memory, we use forEach so that each line that makes it through the filter gets sent to the PrintWriter to be written out to the file immediately and not stored. We have to save it to a temporary file, because we can't overwrite the existing file at the same time as we are still reading from it, so then at the end, we rename the temp file to replace the existing file.
Using apache commons-io and Java 8 you can use
List<String> lines = FileUtils.readLines(file);
List<String> updatedLines = lines.stream().filter(s -> !s.contains(searchString)).collect(Collectors.toList());
FileUtils.writeLines(file, updatedLines, false);
public static void deleteLine() throws IOException {
RandomAccessFile file = new RandomAccessFile("me.txt", "rw");
String delete;
String task="";
byte []tasking;
while ((delete = file.readLine()) != null) {
if (delete.startsWith("BAD")) {
continue;
}
task+=delete+"\n";
}
System.out.println(task);
BufferedWriter writer = new BufferedWriter(new FileWriter("me.txt"));
writer.write(task);
file.close();
writer.close();
}
Here you go. This solution uses a DataInputStream to scan for the position of the string you want replaced and uses a FileChannel to replace the text at that exact position. It only replaces the first occurrence of the string that it finds. This solution doesn't store a copy of the entire file somewhere, (either the RAM or a temp file), it just edits the portion of the file that it finds.
public static long scanForString(String text, File file) throws IOException {
if (text.isEmpty())
return file.exists() ? 0 : -1;
// First of all, get a byte array off of this string:
byte[] bytes = text.getBytes(/* StandardCharsets.your_charset */);
// Next, search the file for the byte array.
try (DataInputStream dis = new DataInputStream(new FileInputStream(file))) {
List<Integer> matches = new LinkedList<>();
for (long pos = 0; pos < file.length(); pos++) {
byte bite = dis.readByte();
for (int i = 0; i < matches.size(); i++) {
Integer m = matches.get(i);
if (bytes[m] != bite)
matches.remove(i--);
else if (++m == bytes.length)
return pos - m + 1;
else
matches.set(i, m);
}
if (bytes[0] == bite)
matches.add(1);
}
}
return -1;
}
public static void replaceText(String text, String replacement, File file) throws IOException {
// Open a FileChannel with writing ability. You don't really need the read
// ability for this specific case, but there it is in case you need it for
// something else.
try (FileChannel channel = FileChannel.open(file.toPath(), StandardOpenOption.WRITE, StandardOpenOption.READ)) {
long scanForString = scanForString(text, file);
if (scanForString == -1) {
System.out.println("String not found.");
return;
}
channel.position(scanForString);
channel.write(ByteBuffer.wrap(replacement.getBytes(/* StandardCharsets.your_charset */)));
}
}
Example
Input: ABCDEFGHIJKLMNOPQRSTUVWXYZ
Method Call:
replaceText("QRS", "000", new File("path/to/file");
Resulting File: ABCDEFGHIJKLMNOP000TUVWXYZ
Here is the complete Class. In the below file "somelocation" refers to the actual path of the file.
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
public class FileProcess
{
public static void main(String[] args) throws IOException
{
File inputFile = new File("C://somelocation//Demographics.txt");
File tempFile = new File("C://somelocation//Demographics_report.txt");
BufferedReader reader = new BufferedReader(new FileReader(inputFile));
BufferedWriter writer = new BufferedWriter(new FileWriter(tempFile));
String currentLine;
while((currentLine = reader.readLine()) != null) {
if(null!=currentLine && !currentLine.equalsIgnoreCase("BBB")){
writer.write(currentLine + System.getProperty("line.separator"));
}
}
writer.close();
reader.close();
boolean successful = tempFile.renameTo(inputFile);
System.out.println(successful);
}
}
This solution reads in an input file line by line, writing each line out to a StringBuilder variable. Whenever it encounters a line that matches what you are looking for, it skips writing that one out. Then it deletes file content and put the StringBuilder variable content.
public void removeLineFromFile(String lineToRemove, File f) throws FileNotFoundException, IOException{
//Reading File Content and storing it to a StringBuilder variable ( skips lineToRemove)
StringBuilder sb = new StringBuilder();
try (Scanner sc = new Scanner(f)) {
String currentLine;
while(sc.hasNext()){
currentLine = sc.nextLine();
if(currentLine.equals(lineToRemove)){
continue; //skips lineToRemove
}
sb.append(currentLine).append("\n");
}
}
//Delete File Content
PrintWriter pw = new PrintWriter(f);
pw.close();
BufferedWriter writer = new BufferedWriter(new FileWriter(f, true));
writer.append(sb.toString());
writer.close();
}
Super simple method using maven/gradle+groovy.
public void deleteConfig(String text) {
File config = new File("/the/path/config.txt")
def lines = config.readLines()
lines.remove(text);
config.write("")
lines.each {line -> {
config.append(line+"\n")
}}
}
public static void deleteLine(String line, String filePath) {
File file = new File(filePath);
File file2 = new File(file.getParent() + "\\temp" + file.getName());
PrintWriter pw = null;
Scanner read = null;
FileInputStream fis = null;
FileOutputStream fos = null;
FileChannel src = null;
FileChannel dest = null;
try {
pw = new PrintWriter(file2);
read = new Scanner(file);
while (read.hasNextLine()) {
String currline = read.nextLine();
if (line.equalsIgnoreCase(currline)) {
continue;
} else {
pw.println(currline);
}
}
pw.flush();
fis = new FileInputStream(file2);
src = fis.getChannel();
fos = new FileOutputStream(file);
dest = fos.getChannel();
dest.transferFrom(src, 0, src.size());
} catch (IOException e) {
e.printStackTrace();
} finally {
pw.close();
read.close();
try {
fis.close();
fos.close();
src.close();
dest.close();
} catch (IOException e) {
e.printStackTrace();
}
if (file2.delete()) {
System.out.println("File is deleted");
} else {
System.out.println("Error occured! File: " + file2.getName() + " is not deleted!");
}
}
}
package com.ncs.cache;
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.File;
import java.io.FileWriter;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.PrintWriter;
public class FileUtil {
public void removeLineFromFile(String file, String lineToRemove) {
try {
File inFile = new File(file);
if (!inFile.isFile()) {
System.out.println("Parameter is not an existing file");
return;
}
// Construct the new file that will later be renamed to the original
// filename.
File tempFile = new File(inFile.getAbsolutePath() + ".tmp");
BufferedReader br = new BufferedReader(new FileReader(file));
PrintWriter pw = new PrintWriter(new FileWriter(tempFile));
String line = null;
// Read from the original file and write to the new
// unless content matches data to be removed.
while ((line = br.readLine()) != null) {
if (!line.trim().equals(lineToRemove)) {
pw.println(line);
pw.flush();
}
}
pw.close();
br.close();
// Delete the original file
if (!inFile.delete()) {
System.out.println("Could not delete file");
return;
}
// Rename the new file to the filename the original file had.
if (!tempFile.renameTo(inFile))
System.out.println("Could not rename file");
} catch (FileNotFoundException ex) {
ex.printStackTrace();
} catch (IOException ex) {
ex.printStackTrace();
}
}
public static void main(String[] args) {
FileUtil util = new FileUtil();
util.removeLineFromFile("test.txt", "bbbbb");
}
}
src : http://www.javadb.com/remove-a-line-from-a-text-file/
This solution requires the Apache Commons IO library to be added to the build path. It works by reading the entire file and writing each line back but only if the search term is not contained.
public static void removeLineFromFile(File targetFile, String searchTerm)
throws IOException
{
StringBuffer fileContents = new StringBuffer(
FileUtils.readFileToString(targetFile));
String[] fileContentLines = fileContents.toString().split(
System.lineSeparator());
emptyFile(targetFile);
fileContents = new StringBuffer();
for (int fileContentLinesIndex = 0; fileContentLinesIndex < fileContentLines.length; fileContentLinesIndex++)
{
if (fileContentLines[fileContentLinesIndex].contains(searchTerm))
{
continue;
}
fileContents.append(fileContentLines[fileContentLinesIndex] + System.lineSeparator());
}
FileUtils.writeStringToFile(targetFile, fileContents.toString().trim());
}
private static void emptyFile(File targetFile) throws FileNotFoundException,
IOException
{
RandomAccessFile randomAccessFile = new RandomAccessFile(targetFile, "rw");
randomAccessFile.setLength(0);
randomAccessFile.close();
}
I refactored the solution that Narek had to create (according to me) a slightly more efficient and easy to understand code. I used embedded Automatic Resource Management, a recent feature in Java and used a Scanner class which according to me is more easier to understand and use.
Here is the code with edited Comments:
public class RemoveLineInFile {
private static File file;
public static void main(String[] args) {
//create a new File
file = new File("hello.txt");
//takes in String that you want to get rid off
removeLineFromFile("Hello");
}
public static void removeLineFromFile(String lineToRemove) {
//if file does not exist, a file is created
if (!file.exists()) {
try {
file.createNewFile();
} catch (IOException e) {
System.out.println("File "+file.getName()+" not created successfully");
}
}
// Construct the new temporary file that will later be renamed to the original
// filename.
File tempFile = new File(file.getAbsolutePath() + ".tmp");
//Two Embedded Automatic Resource Managers used
// to effectivey handle IO Responses
try(Scanner scanner = new Scanner(file)) {
try (PrintWriter pw = new PrintWriter(new FileWriter(tempFile))) {
//a declaration of a String Line Which Will Be assigned Later
String line;
// Read from the original file and write to the new
// unless content matches data to be removed.
while (scanner.hasNextLine()) {
line = scanner.nextLine();
if (!line.trim().equals(lineToRemove)) {
pw.println(line);
pw.flush();
}
}
// Delete the original file
if (!file.delete()) {
System.out.println("Could not delete file");
return;
}
// Rename the new file to the filename the original file had.
if (!tempFile.renameTo(file))
System.out.println("Could not rename file");
}
}
catch (IOException e)
{
System.out.println("IO Exception Occurred");
}
}
}
Try this:
public static void main(String[] args) throws IOException {
File file = new File("file.csv");
CSVReader csvFileReader = new CSVReader(new FileReader(file));
List<String[]> list = csvFileReader.readAll();
for (int i = 0; i < list.size(); i++) {
String[] filter = list.get(i);
if (filter[0].equalsIgnoreCase("bbb")) {
list.remove(i);
}
}
csvFileReader.close();
CSVWriter csvOutput = new CSVWriter(new FileWriter(file));
csvOutput.writeAll(list);
csvOutput.flush();
csvOutput.close();
}
Old question, but an easy way is to:
Iterate through file, adding each line to an new array list
iterate through the array, find matching String, then call the remove method.
iterate through array again, printing each line to the file, boolean for append should be false, which basically replaces the file
This solution uses a RandomAccessFile to only cache the portion of the file subsequent to the string to remove. It scans until it finds the String you want to remove. Then it copies all of the data after the found string, then writes it over the found string, and everything after. Last, it truncates the file size to remove the excess data.
public static long scanForString(String text, File file) throws IOException {
if (text.isEmpty())
return file.exists() ? 0 : -1;
// First of all, get a byte array off of this string:
byte[] bytes = text.getBytes(/* StandardCharsets.your_charset */);
// Next, search the file for the byte array.
try (DataInputStream dis = new DataInputStream(new FileInputStream(file))) {
List<Integer> matches = new LinkedList<>();
for (long pos = 0; pos < file.length(); pos++) {
byte bite = dis.readByte();
for (int i = 0; i < matches.size(); i++) {
Integer m = matches.get(i);
if (bytes[m] != bite)
matches.remove(i--);
else if (++m == bytes.length)
return pos - m + 1;
else
matches.set(i, m);
}
if (bytes[0] == bite)
matches.add(1);
}
}
return -1;
}
public static void remove(String text, File file) throws IOException {
try (RandomAccessFile rafile = new RandomAccessFile(file, "rw");) {
long scanForString = scanForString(text, file);
if (scanForString == -1) {
System.out.println("String not found.");
return;
}
long remainderStartPos = scanForString + text.getBytes().length;
rafile.seek(remainderStartPos);
int remainderSize = (int) (rafile.length() - rafile.getFilePointer());
byte[] bytes = new byte[remainderSize];
rafile.read(bytes);
rafile.seek(scanForString);
rafile.write(bytes);
rafile.setLength(rafile.length() - (text.length()));
}
}
Usage:
File Contents: ABCDEFGHIJKLMNOPQRSTUVWXYZ
Method Call: remove("ABC", new File("Drive:/Path/File.extension"));
Resulting Contents: DEFGHIJKLMNOPQRSTUVWXYZ
This solution could easily be modified to remove with a certain, specifiable cacheSize, if memory is a concern. This would just involve iterating over the rest of the file to continually replace portions of size, cacheSize. Regardless, this solution is generally much better than caching an entire file in memory, or copying it to a temporary directory, etc.

How to prevent CSVPrinter from rewriting data again

I'm using this method for writing data into csv but the problem is that it is rewriting data again below the old data. How can I prevent it from doing this ? I tried to set the FileWriter writer = new FileWriter(answerFile, false); but then its only writing the last array in the csv file.
I have this code:
public static void writeCsv(List<String> myList) throws IOException {
FileWriter writer = new FileWriter(answerFile, true);
CSVPrinter csvPrinter = new CSVPrinter(writer, CSVFormat.DEFAULT);
List<String[]> myListSplitted = myList.stream().map(row -> row.split(",")).collect(Collectors.toList());
csvPrinter.printRecords(myListSplitted);
csvPrinter.flush();
csvPrinter.close();
}
This the method in which I'm calling this method:
public static void appendAnswers() throws IOException {
try (BufferedReader br = new BufferedReader(new FileReader(questionFile))) {
String csvRow;
int counter = 0;
String[] csvArr;
String data;
br.readLine();
List<String> myList = new ArrayList<>();
while ((csvRow = br.readLine()) != null) {
csvArr = csvRow.split(",");
csvArr = Arrays.copyOf(csvArr, csvArr.length + 1);
csvArr[csvArr.length - 1] = answers.get(counter);
data = Arrays.toString(csvArr).replace("[", "").replace("]",
"").trim();
counter++;
myList = new ArrayList<String>(Arrays.asList(data.split("\n")));
}
writeCsv(myList);
}
From the manual page :
Prints values in a CSV format.
Values can be appended to the output by calling the print(Object) method.
Basically, CSVPrinter emulates a printer. A printer appends new lines to whatever has already been printed.
If you want to overwrite, use FileWriter more directly rather than a printer emulator.

Read Tab-Separated-Columns into Lists - Java

Tab-Separated File:
2019-06-06 10:00:00 1.0
2019-06-06 11:00:00 2.0
I'd like to iterate over the file once and add the value of each column to a list.
My working approach would be:
import java.util.*;
import java.io.*;
public class Program {
public static void main(String[] args)
{
ArrayList<Double> List_1 = new ArrayList<Double>();
ArrayList<Double> List_2 = new ArrayList<Double>();
String[] values = null;
String fileName = "File.txt";
File file = new File(fileName);
try
{
Scanner inputStream = new Scanner(file);
while (inputStream.hasNextLine()){
try {
String data = inputStream.nextLine();
values = data.split("\\t");
if (values[1] != null && !values[1].isEmpty() == true) {
double val_1 = Double.parseDouble(values[1]);
List_1.add(val_1);
}
if (values[2] != null && !values[2].isEmpty() == true) {
double val_2 = Double.parseDouble(values[2]);
List_2.add(val_2);
}
}
catch (ArrayIndexOutOfBoundsException exception){
}
}
inputStream.close();
}
catch (FileNotFoundException e) {
e.printStackTrace();
}
System.out.println(List_1);
System.out.println(List_2);
}
}
I get:
[1.0]
[2.0]
It doesn't work without the checks for null, ìsEmpty and the ArrayIndexOutOfBoundsException.
I would appreciate any hints on how to save a few lines while keeping the scanner approach.
One option is to create a Map of Lists using column number as a key. This approach gives you "unlimited" number of columns and exactly the same output than one in the question.
public class Program {
public static void main(String[] args) throws Exception
{
Map<Integer, List<Double>> listMap = new TreeMap<Integer, List<Double>>();
String[] values = null;
String fileName = "File.csv";
File file = new File(fileName);
Scanner inputStream = new Scanner(file);
while (inputStream.hasNextLine()){
String data = inputStream.nextLine();
values = data.split("\\t");
for (int column = 1; column < values.length; column++) {
List<Double> list = listMap.get(column);
if (list == null) {
listMap.put(column, list = new ArrayList<Double>());
}
if (!values[column].isEmpty()) {
list.add(Double.parseDouble(values[column]));
}
}
}
inputStream.close();
for(List<Double> list : listMap.values()) {
System.out.println(list);
}
}
}
You can clean up your code some by using try-with resources to open and close the Scanner for you:
try (Scanner inputStream = new Scanner(file))
{
//your code...
}
This is useful because the inputStream will be closed automatically once the try block is left and you will not need to close it manually with inputStream.close();.
Additionally if you really want to "save lines" you can also combine these steps:
double val_2 = Double.parseDouble(values[2]);
List_2.add(val_2);
Into a single step each, since you do not actually use the val_2 anywhere else:
List_2.add(Double.parseDouble(values[2]));
Finally you are also using !values[1].isEmpty() == true which is comparing a boolean value to true. This is typically bad practice and you can reduce it to !values[1].isEmpty() instead which will have the same functionality. Try not to use == with booleans as there is no need.
you can do it like below:
BufferedReader bfr = Files.newBufferedReader(Paths.get("inputFileDir.tsv"));
String line = null;
List<List<String>> listOfLists = new ArrayList<>(100);
while((line = bfr.readLine()) != null) {
String[] cols = line.split("\\t");
List<String> outputList = new ArrayList<>(cols);
//at this line your expected list of cols of each line is ready to use.
listOfLists.add(outputList);
}
As a matter of fact, it is a simple code in java. But because it seems that you are a beginner in java and code like a python programmer, I decided to write a sample code to let you have a good start point. good luck

Java Reading in text file and outputting it to new file with removed duplicates

I have a text file with an integer on each line, ordered from least to greatest, and I want to put them in a new text file with any duplicate numbers removed.
I've managed to read in the text file and print the numbers on the screen, but I'm unsure on how to actually write them in a new file, with duplicates removed?
public static void main(String[] args)
{
try
{
FileReader fr = new FileReader("sample.txt");
BufferedReader br = new BufferedReader(fr);
String str;
while ((str = br.readLine()) != null) {
out.println(str + "\n");
}
br.close();
}
catch (IOException e) {
out.println("File not found");
}
}
When reading the file, you could add the numbers to a Set, which is a data structure that doesn't allow duplicate values (just Google for "java collections" for more details)
Then you iterate through this Set, writing the numbers to a FileOutputStream (google for "java io" for more details)
Instead of printing each of the numbers, add them to an Array. After you've added all the integers, you can cycle through the array to remove duplicates (sample code for this can be found fairly easily).
Once you have an array, use BufferedWriter to write to an output file. Example code for how to do this can be found here: https://www.mkyong.com/java/how-to-write-to-file-in-java-bufferedwriter-example/
Alternatively, use a Set, and BufferedWriter should still work in the same way.
assuming the input file is already ordered:
public class Question42475459 {
public static void main(final String[] args) throws IOException {
final String inFile = "sample.txt";
try (final Scanner scanner = new Scanner(new BufferedInputStream(new FileInputStream("")), "UTF-8");
BufferedWriter writer = new BufferedWriter(new FileWriter(inFile + ".out", false))) {
String lastLine = null;
while (scanner.hasNext()) {
final String line = scanner.next();
if (!line.equals(lastLine)) {
writer.write(line);
writer.newLine();
lastLine = line;
}
}
}
}
}

How do I read from a File to an array

I am trying to read from a file to an array. I tried two different styles and both aren't working. Below are the two styles.
Style 1
public class FileRead {
int i;
String a[] = new String[2];
public void read() throws FileNotFoundException {
//Z means: "The end of the input but for the final terminator, if any"
a[i] = new Scanner(new File("C:\\Users\\nnanna\\Documents\\login.txt")).useDelimiter("\\n").next();
for(i=0; i<=a.length; i++){
System.out.println("" + a[i]);
}
}
public static void main(String args[]) throws FileNotFoundException{
new FileRead().read();
}
}
Style 2
public class FileReadExample {
private int j = 0;
String path = null;
public void fileRead(File file){
StringBuilder attachPhoneNumber = new StringBuilder();
try{
FileReader read = new FileReader(file);
BufferedReader bufferedReader = new BufferedReader(read);
while((path = bufferedReader.readLine()) != null){
String a[] = new String[3];
a[j] = path;
j++;
System.out.println(path);
System.out.println(a[j]);
}
bufferedReader.close();
}catch(IOException exception){
exception.printStackTrace();
}
}
I need it to read each line of string and store each line in an array. But neither works. How do I go about it?
Do yourself a favor and use a library that provides this functionality for you, e.g.
Guava:
// one String per File
String data = Files.toString(file, Charsets.UTF_8);
// or one String per Line
List<String> data = Files.readLines(file, Charsets.UTF_8);
Commons / IO:
// one String per File
String data = FileUtils.readFileToString(file, "UTF-8");
// or one String per Line
List<String> data = FileUtils.readLines(file, "UTF-8");
It's not really clear exactly what you're trying to do (partly with quite a lot of code commented out, leaving other code which won't even compile), but I'd recommend you look at using Guava:
List<String> lines = Files.readLines(file, Charsets.UTF_8);
That way you don't need to mess around with the file handling yourself at all.

Categories