I'm using this method for writing data into csv but the problem is that it is rewriting data again below the old data. How can I prevent it from doing this ? I tried to set the FileWriter writer = new FileWriter(answerFile, false); but then its only writing the last array in the csv file.
I have this code:
public static void writeCsv(List<String> myList) throws IOException {
FileWriter writer = new FileWriter(answerFile, true);
CSVPrinter csvPrinter = new CSVPrinter(writer, CSVFormat.DEFAULT);
List<String[]> myListSplitted = myList.stream().map(row -> row.split(",")).collect(Collectors.toList());
csvPrinter.printRecords(myListSplitted);
csvPrinter.flush();
csvPrinter.close();
}
This the method in which I'm calling this method:
public static void appendAnswers() throws IOException {
try (BufferedReader br = new BufferedReader(new FileReader(questionFile))) {
String csvRow;
int counter = 0;
String[] csvArr;
String data;
br.readLine();
List<String> myList = new ArrayList<>();
while ((csvRow = br.readLine()) != null) {
csvArr = csvRow.split(",");
csvArr = Arrays.copyOf(csvArr, csvArr.length + 1);
csvArr[csvArr.length - 1] = answers.get(counter);
data = Arrays.toString(csvArr).replace("[", "").replace("]",
"").trim();
counter++;
myList = new ArrayList<String>(Arrays.asList(data.split("\n")));
}
writeCsv(myList);
}
From the manual page :
Prints values in a CSV format.
Values can be appended to the output by calling the print(Object) method.
Basically, CSVPrinter emulates a printer. A printer appends new lines to whatever has already been printed.
If you want to overwrite, use FileWriter more directly rather than a printer emulator.
Related
I'm trying to parse a csv of over 100,000 lines and the performance problems don't even let me get to the end of the file before hitting "Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded"
Is there something wrong, or any way I can improve?
public static List<String[]> readCSV(String filePath) throws IOException{
List<String[]> csvLine= new ArrayList<String[]>();
CSVReader reader = new CSVReader(new FileReader(filePath), '\n');
String[] row;
while((row = reader.readNext()) != null){
csvLine.add(removeWhiteSpace(row[0].toString().split(",")));
}
reader.close();
return csvLine;
}
private static String[] removeWhiteSpace(String[] split) {
for(int index =0; index < split.length;index++){
split[index] = split[index].trim();
}
return split;
}
First you are running out of memory because all rows are being added to a list.
Second you are using String.split() which is extremely slow.
Third never try processing CSV by writing your own parsing code as there are many edge cases around this format (need to handle escape of delimiter, quotes, etc).
The solution is to use a library for that, such as univocity-parsers. You should be able to read 1 million rows in less than a second.
To parse, just do this:
public static IterableResult<String[], ParsingContext> readCSV(String filePath) {
File file = new File(filePath);
//configure the parser here. By default all values are trimmed
CsvParserSettings parserSettings = new CsvParserSettings();
//create the parser
CsvParser parser = new CsvParser(parserSettings);
//create an iterable over rows. This will not load everything into memory.
IterableResult<String[], ParsingContext> rows = parser.iterate(file);
return rows;
}
Now you can use your method like this:
public static void main(String... args) {
IterableResult<String[], ParsingContext> rows = readCSV("c:/path/to/input.csv");
try {
for (String[] row : rows) {
//process the rows however you want
}
} finally {
//the parser closes itself but in case any errors processing the rows (outside of the control of the iterator), close the parser.
rows.getContext().stop();
}
}
This is just an example of how you can use the parser, but there are many different ways to use it.
Now for writing, you can do this:
public static void main(String... args) {
//this is your output file
File output = new File("c:/path/to/output.csv");
//configure the writer if you need to
CsvWriterSettings settings = new CsvWriterSettings();
//create the writer. Here we write to a file
CsvWriter writer = new CsvWriter(output, settings);
//get the row iterator
IterableResult<String[], ParsingContext> rows = readCSV("c:/temp");
try {
//do whatever you need to the rows here
for (String[] row : rows) {
//then write it each one to the output.
writer.writeRow(row);
}
} finally {
//cleanup
rows.getContext().stop();
writer.close();
}
}
If all you want is to read the data, modify it and write it back to another file, you can just do this:
public static void main(String... args) throws IOException {
CsvParserSettings parserSettings = new CsvParserSettings();
parserSettings.setProcessor(new AbstractRowProcessor() {
#Override
public void rowProcessed(String[] row, ParsingContext context) {
//modify the row data here.
}
});
CsvWriterSettings writerSettings = new CsvWriterSettings();
CsvRoutines routines = new CsvRoutines(parserSettings, writerSettings);
FileReader input = new FileReader("c:/path/to/input.csv");
FileWriter output = new FileWriter("c:/path/to/output.csv");
routines.parseAndWrite(input, output);
}
Hope this helps.
Disclaimer: I'm the author of this libary. It's open source and free (Apache 2.0 license).
Is a design error try to put such a large file in memory.
Depending of what you want to do, you should either write a new file processed, or put the lines into a dba.
This implements the first:
FileInputStream inputStream = null;
Scanner sc = null;
try {
inputStream = new FileInputStream(path);
sc = new Scanner(inputStream, "UTF-8");
while (sc.hasNextLine()) {
String line = sc.nextLine();
// System.out.println(line);
}
// note that Scanner suppresses exceptions
if (sc.ioException() != null) {
throw sc.ioException();
}
} finally {
if (inputStream != null) {
inputStream.close();
}
if (sc != null) {
sc.close();
}
}
I need help trying to read two files that have the census from 2010 and 2000. I have to read both files and then find out the population growth between those two files. I keep getting null for ever single state. I know that I have null for inLine1 and inLine2.
The file looks like this
Alabama,4779736
Alaska,710231
Arizona,6392017
Arkansas,2915918
Code:
import java.util.ArrayList;
import java.util.Scanner;
import java.io.*;
public class pa10
{
public static void main(String[] args, char[] inLine2, char[] inLine1)
throws java.io.IOException
{
String fileName1 = "Census2000growth.txt";
String fileName2 = "Census2010growth.txt";
int i;
File f = new File("Census2010growth.txt");
if(!f.exists()) {
System.out.println( "file does not exist ");
}
Scanner infile = new Scanner(f);
infile.useDelimiter ("[\t|,|\n|\r]+"); //create a delimiter
final int MAX = 51;
int [] myarray = new int [MAX];
String[] statearray = new String[MAX];
int fillsize;
// set up input stream1
FileReader fr1 = new
FileReader(fileName1);
// buffer the input stream
BufferedReader br1 =
new BufferedReader(fr1);
// set up input stream2
FileReader fr2 = new
FileReader(fileName2);
// buffer the input stream
BufferedReader br2 =
new BufferedReader(fr2);
// read and display1
String buffer1 = "";
ArrayList<String> firstFile1 = new ArrayList<String>();
while ((buffer1 = br1.readLine()) != null) {
firstFile1.add(buffer1);
System.out.println(inLine1); // display the line
}
br1.close();
//Now read the second file or make for this separate method
// read and display2
String buffer2 = "";
ArrayList<String> firstFile2 = new ArrayList<String>();
while ((buffer2 = br2.readLine()) != null) {
firstFile2.add(buffer2);
System.out.println(inLine2); // display the line
}
br2.close();
//Read all the lines in array or list
//After that you can calculate them.
}
}
Read the BufferedReader documentation. Your file isn't formatted with the types of line separators it is expecting. I suggest using a Scanner and setting the line separator to the appropriate pattern, or using String.split
You have two different variables, buffer1 and inline1. Since you never set the value of inline1, it will always be null.
This question already has answers here:
Java: Reading a file into an array
(5 answers)
Closed 1 year ago.
I have a text file like this :
abc def jhi
klm nop qrs
tuv wxy zzz
I want to have a string array like :
String[] arr = {"abc def jhi","klm nop qrs","tuv wxy zzz"}
I've tried :
try
{
FileInputStream fstream_school = new FileInputStream("text1.txt");
DataInputStream data_input = new DataInputStream(fstream_school);
BufferedReader buffer = new BufferedReader(new InputStreamReader(data_input));
String str_line;
while ((str_line = buffer.readLine()) != null)
{
str_line = str_line.trim();
if ((str_line.length()!=0))
{
String[] itemsSchool = str_line.split("\t");
}
}
}
catch (Exception e)
{
// Catch exception if any
System.err.println("Error: " + e.getMessage());
}
Anyone help me please....
All answer would be appreciated...
If you use Java 7 it can be done in two lines thanks to the Files#readAllLines method:
List<String> lines = Files.readAllLines(yourFile, charset);
String[] arr = lines.toArray(new String[lines.size()]);
Use a BufferedReader to read the file, read each line using readLine as strings, and put them in an ArrayList on which you call toArray at end of loop.
Based on your input you are almost there. You missed the point in your loop where to keep each line read from the file. As you don't a priori know the total lines in the file, use a collection (dynamically allocated size) to get all the contents and then convert it to an array of String (as this is your desired output).
Something like this:
String[] arr= null;
List<String> itemsSchool = new ArrayList<String>();
try
{
FileInputStream fstream_school = new FileInputStream("text1.txt");
DataInputStream data_input = new DataInputStream(fstream_school);
BufferedReader buffer = new BufferedReader(new InputStreamReader(data_input));
String str_line;
while ((str_line = buffer.readLine()) != null)
{
str_line = str_line.trim();
if ((str_line.length()!=0))
{
itemsSchool.add(str_line);
}
}
arr = (String[])itemsSchool.toArray(new String[itemsSchool.size()]);
}
Then the output (arr) would be:
{"abc def jhi","klm nop qrs","tuv wxy zzz"}
This is not the optimal solution. Other more clever answers have already be given. This is only a solution for your current approach.
This is my code to generate random emails creating an array from a text file.
import java.io.*;
public class Generator {
public static void main(String[]args){
try {
long start = System.currentTimeMillis();
String[] firstNames = new String[4945];
String[] lastNames = new String[88799];
String[] emailProvider ={"google.com","yahoo.com","hotmail.com","onet.pl","outlook.com","aol.mail","proton.mail","icloud.com"};
String firstName;
String lastName;
int counter0 = 0;
int counter1 = 0;
int generate = 1000000;//number of emails to generate
BufferedReader firstReader = new BufferedReader(new FileReader("firstNames.txt"));
BufferedReader lastReader = new BufferedReader(new FileReader("lastNames.txt"));
PrintWriter write = new PrintWriter(new FileWriter("emails.txt", false));
while ((firstName = firstReader.readLine()) != null) {
firstName = firstName.toLowerCase();
firstNames[counter0] = firstName;
counter0++;
}
while((lastName= lastReader.readLine()) !=null){
lastName = lastName.toLowerCase();
lastNames[counter1]=lastName;
counter1++;
}
for(int i=0;i<generate;i++) {
write.println(firstNames[(int)(Math.random()*4945)]
+'.'+lastNames[(int)(Math.random()*88799)]+'#'+emailProvider[(int)(Math.random()*emailProvider.length)]);
}
write.close();
long end = System.currentTimeMillis();
long time = end-start;
System.out.println("it took "+time+"ms to generate "+generate+" unique emails");
}
catch(IOException ex){
System.out.println("Wrong input");
}
}
}
You can read file line by line using some input stream or scanner and than store that line in String Array.. A sample code will be..
File file = new File("data.txt");
try {
//
// Create a new Scanner object which will read the data
// from the file passed in. To check if there are more
// line to read from it we check by calling the
// scanner.hasNextLine() method. We then read line one
// by one till all line is read.
//
Scanner scanner = new Scanner(file);
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
//store this line to string [] here
System.out.println(line);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
}
Scanner scanner = new Scanner(InputStream);//Get File Input stream here
StringBuilder builder = new StringBuilder();
while (scanner.hasNextLine()) {
builder.append(scanner.nextLine());
builder.append(" ");//Additional empty space needs to be added
}
String strings[] = builder.toString().split(" ");
System.out.println(Arrays.toString(strings));
Output :
[abc, def, jhi, klm, nop, qrs, tuv, wxy, zzz]
You can read more about scanner here
You can use the readLine function to read the lines in a file and add it to the array.
Example :
File file = new File("abc.txt");
FileInputStream fin = new FileInputStream(file);
BufferedReader reader = new BufferedReader(fin);
List<String> list = new ArrayList<String>();
while((String str = reader.readLine())!=null){
list.add(str);
}
//convert the list to String array
String[] strArr = Arrays.toArray(list);
The above array contains your required output.
I'm working on a csv parser, I want to read headers and the rest of the csv file separately.
Here is my code to read csv.
The current code reads everything in the csv file, but I need to read headers separate.
please help me regarding this.
public class csv {
private void csvRead(File file)
{
try
{
BufferedReader br = new BufferedReader( new FileReader(file));
String strLine = "";
StringTokenizer st = null;
File cfile=new File("csv.txt");
BufferedWriter writer = new BufferedWriter(new FileWriter(cfile));
int tokenNumber = 0;
while( (strLine = br.readLine()) != null)
{
st = new StringTokenizer(strLine, ",");
while(st.hasMoreTokens())
{
tokenNumber++;
writer.write(tokenNumber+" "+ st.nextToken());
writer.newLine();
}
tokenNumber = 0;
writer.flush();
}
}
catch(Exception e)
{
e.getMessage();
}
}
We have withHeader() method available in CSVFormat. If you use this option then you will be able to read the file using headers.
CSVFormat format = CSVFormat.newFormat(',').withHeader();
Map<String, Integer> headerMap = dataCSVParser.getHeaderMap();
will give you all headers.
public class CSVFileReaderEx {
public static void main(String[] args){
readFile();
}
public static void readFile(){
List<Map<String, String>> csvInputList = new CopyOnWriteArrayList<>();
List<Map<String, Integer>> headerList = new CopyOnWriteArrayList<>();
String fileName = "C:/test.csv";
CSVFormat format = CSVFormat.newFormat(',').withHeader();
try (BufferedReader inputReader = new BufferedReader(new FileReader(new File(fileName)));
CSVParser dataCSVParser = new CSVParser(inputReader, format); ) {
List<CSVRecord> csvRecords = dataCSVParser.getRecords();
Map<String, Integer> headerMap = dataCSVParser.getHeaderMap();
headerList.add(headerMap);
headerList.forEach(System.out::println);
for(CSVRecord record : csvRecords){
Map<String, String> inputMap = new LinkedHashMap<>();
for(Map.Entry<String, Integer> header : headerMap.entrySet()){
inputMap.put(header.getKey(), record.get(header.getValue()));
}
if (!inputMap.isEmpty()) {
csvInputList.add(inputMap);
}
}
csvInputList.forEach(System.out::println);
} catch (Exception e) {
System.out.println(e);
}
}
}
Please consider the use of Commons CSV. This library is written according RFC 4180 - Common Format and MIME Type for Comma-Separated Values (CSV) Files. What is compatible to read such lines:
"aa,a","b""bb","ccc"
And the use is quite simple, there is just 3 classes, and a small sample according documentation:
Parsing of a csv-string having tabs as separators, '"' as an optional
value encapsulator, and comments starting with '#':
CSVFormat format = new CSVFormat('\t', '"', '#');
Reader in = new StringReader("a\tb\nc\td");
String[][] records = new CSVParser(in, format).getRecords();
And additionally you get this parsers already available as constants:
DEFAULT - Standard comma separated format as defined by RFC 4180.
EXCEL - Excel file format (using a comma as the value delimiter).
MYSQL - Default MySQL format used by the SELECT INTO OUTFILE and LOAD DATA INFILE operations.
TDF - Tabulation delimited format.
Have you considered OpenCSV?
Previous question here...
CSV API for Java
Looks like you can split out the header quite easily...
String fileName = "data.csv";
CSVReader reader = new CSVReader(new FileReader(fileName ));
// if the first line is the header
String[] header = reader.readNext();
// iterate over reader.readNext until it returns null
String[] line = reader.readNext();
Your code here, being
while( (strLine = br.readLine()) != null)
{
//reads everything in your csv
}
will print all of your CSV content.
For example, the following fetches your header:
Reader in = ...;
CSVFormat.EXCEL.withHeader("Col1", "Col2", "Col3").parse(in);
As suggested, life could be easier using the predefined CSVFormat from the apache commons library. Link here (https://commons.apache.org/proper/commons-csv/user-guide.html).
Cheers.
I am trying to read from a file to an array. I tried two different styles and both aren't working. Below are the two styles.
Style 1
public class FileRead {
int i;
String a[] = new String[2];
public void read() throws FileNotFoundException {
//Z means: "The end of the input but for the final terminator, if any"
a[i] = new Scanner(new File("C:\\Users\\nnanna\\Documents\\login.txt")).useDelimiter("\\n").next();
for(i=0; i<=a.length; i++){
System.out.println("" + a[i]);
}
}
public static void main(String args[]) throws FileNotFoundException{
new FileRead().read();
}
}
Style 2
public class FileReadExample {
private int j = 0;
String path = null;
public void fileRead(File file){
StringBuilder attachPhoneNumber = new StringBuilder();
try{
FileReader read = new FileReader(file);
BufferedReader bufferedReader = new BufferedReader(read);
while((path = bufferedReader.readLine()) != null){
String a[] = new String[3];
a[j] = path;
j++;
System.out.println(path);
System.out.println(a[j]);
}
bufferedReader.close();
}catch(IOException exception){
exception.printStackTrace();
}
}
I need it to read each line of string and store each line in an array. But neither works. How do I go about it?
Do yourself a favor and use a library that provides this functionality for you, e.g.
Guava:
// one String per File
String data = Files.toString(file, Charsets.UTF_8);
// or one String per Line
List<String> data = Files.readLines(file, Charsets.UTF_8);
Commons / IO:
// one String per File
String data = FileUtils.readFileToString(file, "UTF-8");
// or one String per Line
List<String> data = FileUtils.readLines(file, "UTF-8");
It's not really clear exactly what you're trying to do (partly with quite a lot of code commented out, leaving other code which won't even compile), but I'd recommend you look at using Guava:
List<String> lines = Files.readLines(file, Charsets.UTF_8);
That way you don't need to mess around with the file handling yourself at all.