I am new to Java and I got a situation to which I'm clueless about. I need to take all the csv files from a folder read them one by one, validate them for eg. There's data like name, age, email etc. So the name should have only letters, age should be numeric and email should be in valid email format. The file which has invalid data in any of the row, there shouldn't be any further processing of that particular csv file and it should be moved to another folder which will have erroneous csv files and the program will move onto next csv in the folder until all of them gets checked, validated and moved.
I don't know how to begin with this. Please help me out guys.
OK, let's separate this question into following four smaller topics:
Java program to read a folder, result is a list of files
Java program to read a file, result is a list of lines
Java program to parse a line, get a list of columns
For name, age, email, validate the data
Step 1: Java program to read a folder, result is a list of files
Assuming you have below in your top of java program
import java.io.*;
import java.util.*;
Below code should get a list of file in a folder
File f = new File(folder);
File[] fileList = f.listFiles();
Step 2: Java program to read a file, result is a list of lines
String line;
BufferedReader br = new BufferedReader(new FileReader(path));
while ((line = br.readLine()) != null) {
String l = line.trim(); // Remove end of line. You can print line here.
}
br.close();
Step 3: Java program to parse a line, get a list of columns
String[] columns = l.split(","); // separate line by comma
for( int i=0; i<columns.length; i++ )
{
System.out.println(columns[i].trim());// remove space after comma
}
Step 4: Validate e.g. age
Age has to be integer so parse it as integer
int age = Integer.parseInt(columns[3].trim());//assuming age at column #3
See another answer come out. That answer doesn't have folder to file loop.
Hope this helps.
Firstly, save your file to .csv format1. This works on excel sheets. Then call this function in main() by this code, you will read the .csv file row-wise, each cell at a time .
Try this out:
public List<HashMap<String, Object>> convertCSVRecordToList() {
String csvFile = "your_file_name.csv";
BufferedReader br = null;
String line = "";
String cvsSplitBy = ",";
HashMap<String, Object> Map = new HashMap<String, Object>();
List<HashMap<String, Object>> MapList = new ArrayList<HashMap<String, Object>>();
try {
br = new BufferedReader(new FileReader(csvFile));
while ((line = br.readLine()) != null) {
System.out.println(line);
String[] data = line.split(cvsSplitBy);
Map.put("filed_name", data[3]);
Map.put("field_name", data[0]);
Map.put("field_name", data[2]);
Map.put("fiels_name", data[1]);
MapList.add(Map);
Map = new HashMap<String, Object>();
}
} catch (FileNotFoundException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
return MapList;
}
Related
I have 2-3 .csv files with fields like Date, Amount, Transaction Description etc and all the csv files contains these fields but in shuffled order. I want a output file to with a standard order (like if I input the sample .csv file, then I can get the things in order in output file).
I tried to do it for one file by taking substrings from the .csv file (at that time I didn't know that other files have shuffled order fields).
I am kind of new, tell me if I am asking question in a good format!
Can I put a link for the sample input and output .csv file for the reference?
--> https://drive.google.com/drive/folders/1-NZi5OTMTbOWXAfCTsc-ahNYm1N5DG2g (just because it would be very hard to explain that how file looks like)
What I have done?
I have just tried to extract data from the fields using the BufferReader using split but it can only work for one type of file, I cant have a standard format using this!
Sorry for posting such a long code but what I have done is selected field from the file and copied them into output file corresponding to the standard fields in the output file.
Suggest me if there is any other method with which I can proceed.
File file = new File("C:\\Users\\R\\Desktop\\CSVDemo.csv");
try {
// create FileWriter object with file as parameter
FileWriter outputfile = new FileWriter(file);
CSVWriter writer = new CSVWriter(outputfile, ',',
CSVWriter.NO_QUOTE_CHARACTER,
CSVWriter.DEFAULT_ESCAPE_CHARACTER,
CSVWriter.DEFAULT_LINE_END);
// create a List which contains String array
String[] header = { "Date", "Transaction Description", "Debit","Credit","Currency","CardName","Transaction","Location" };
writer.writeNext(header);
String splitBy = ",";
BufferedReader br = new BufferedReader(new FileReader("G:\\US\\HDFC-Input-Case1.csv"));
String line;
String transaction = "",name = "";
while ((line = br.readLine()) != null) {
// use comma as separator
String[] cols = line.split(splitBy);
if(cols.length == 2 && cols[1].equals("Domestic Transactions")) {
transaction = "Domestic";
continue;
}
else if(cols.length == 2 && cols[1].equals("International Transactions")) {
transaction = "International";
continue;
}
else if(cols.length == 2) {
name = cols[1];
continue;
}
else if(cols.length<1){
continue;
}
else if(cols.length>2) {
if(cols[0].contains("Date")){
continue;
}
String[] data1 = new String[header.length];
data1[0] = cols[0];
String curr ;
if(cols[1].substring(cols[1].length()-3).equals("USD") || cols[1].substring(cols[1].length()-3).equals("EUR")) {
data1[4] = cols[1].substring(cols[1].length()-3);
curr = cols[1].substring(0,cols[1].length()-4);
data1[1] = curr;
}
else {
data1[4] = "INR";
data1[1] = cols[1];
}
if(cols[2].contains("cr")){
data1[3] = cols[2].substring(0,cols[2].length()-2);
data1[2] = "0";
}
else {
data1[2] = cols[2];
data1[3] = "0";
}
data1[5] = name;
data1[6] = transaction;
writer.writeNext(data1);
}
System.out.println();
}
// closing writer connection
writer.close();
}
You can read the header of your input csv files first and find the indexes of required field in this given csv file.
Once you have required indexes for every header, read those fields using indexes in the standard order you want for your output csv file.
sample codes:
`CSVReader reader = new CSVReader(new FileReader(fileName ));
String[] header = reader.readNext();
List<String> list= Arrays.asList(header);
int indexOfFieldTransaction=list.indexOf("transaction");`
Now make a List and insert the field in order you want to write in output file.you will get -1 if the field you are trying to get index of is not present in the input file.
I am a rookie in java coding area,
I am trying to parse a csv format file, split with only commas(,)
this file includes users' account names and true names,
for example, it looks like: tom123,Tom Halland,kelly02,Kelly Chen,..and so on,
I want to parse these user's data into something like arrays using java,
so I can reuse this array, then to generate ldif format file and import it into LDAP server to create accounts automatically,
is there any easier way to do it? or any technical advises for me?
Thanks a lot!
I am sharing code snippet from one of my previous assignments. Basically code does following tasks:
1) Read CSV file line by line
2) Split each line into token with help of predefined characher(in case of csv it's ',')
3) Generate a string form of record in desired ldif format
4) write record to output file
String inputCSVFile = "/input_folder_path/sample.csv";
BufferedReader bufferedReader = null;
String lines = "";
String splitChar = ",";
String[] columns;
int count = 0;
try {
PrintStream printStream = new PrintStream(new FileOutputStream("/output_folder_path/e.ldif"));// Step 1
bufferedReader = new BufferedReader(new FileReader(inputCSVFile));
while ((lines = bufferedReader.readLine()) != null) {
columns = lines.split(splitChar);// Step 2
if (count > 0) {// Step 3 ,4
printStream.println("dn: cn="+columns[1]+", ou="+columns[2]+", o=Data"
+ "\ngivenName: "+columns[0]
+ "\nsn: "+columns[3]
+ "\n"
);
}
count++;
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (bufferedReader != null) {
try {
bufferedReader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
I am trying to read in a text file and then manipulate a little and update the records into a new text file.
Here is what I have so far:
ArrayList<String> linesList = new ArrayList<>();
BufferedReader br;
String empid, email;
String[] data;
try {
String line;
br = new BufferedReader(new FileReader("file.txt"));
while ((line = br.readLine()) !=null) {
linesList.add(line);
}
br.close();
}
catch (IOException e) { e.printStackTrace(); }
for (int i = 0; i < linesList.size(); i++) {
data = linesList.get(i).split(",");
empid = data[0];
ccode = data[3];
}
File tempFile = new File("File2.txt");
BufferedWriter bw = new BufferedWriter(new FileWriter(tempFile));
for (int i = 0; i < linesList.size(); i++) {
if(i==0){
bw.write(linesList.get(i));
bw.newLine();
}
else{
data = linesList.get(i).split(",");
String empid1 = data[0];
if(data[13].equals("IND")) {
String replace = data[3].replaceAll("IND", "IN");
ccode1 = replace;
System.out.println(ccode1);
}
else if(data[13].equals("USA")) {
String replace = data[3].replaceAll("USA", "US");
ccode1 = replace;
}
else {
ccode1 = replace; //This does not work as replace is not defined here, but how can I get it to work here.
}
String newData=empid1+","+ccode1;
bw.write(newData);
bw.newLine();
}
}
Here is what is inside the text file:
EID,First,Last,Country
1,John,Smith,USA
2,Jane,Smith,IND
3,John,Adams,USA
So, what I need help with is editing the three letter country code and replacing it with a 2 letter country code. For example: USA would become US, and IND would become IN. I am able to read in the country code, but am having trouble in changing the value and then replacing the changed value back into a different text file. Any help is appreciated. Thanks in advance.
Open file in text editor, Search and Replace, ,USA with ,US, ,IND with ,IN and so on.
As such, to automate it, on the same while loop you read a line do:
//while(read){ line.replaceAll(",USA",",US");
That will be the easiest way to complete your objective.
To save, open a BufferedWriter bw; just like you opened a reader and use bw.write(). You would probably prefer to open both at the same time, the reader on your source file, and the writer on a new file, with _out suffix. That way you dont need to keep the file data in memory, you can read and write as you loop.
For harder ways, read the csv specs: https://www.rfc-editor.org/rfc/rfc4180#section-2
Notice that you have to account for the possibility of fields being enclosed in quotes, like: "1","John","Smith","USA", which means you also have to replace ,\"USA with ,\"US.
The delimiter may or may not be a comma, you have to make sure yur input will always use the same delimiter, or that you can detect and switch at runtime.
You have to account for the case where a delimiter may be part of a field, or where quotes are part of a field.
Now you know/can solve these issues you can, instead of using replace, parse the lines character by character using while( (/*int*/ c = br.read()) != -1), and do this replacement manually with an if gate.
/*while(read)*/
if( c == delimiter ){
if not field, start next field, else add to field value
} else if( c == quote ){
if field value empty, ignore and expect closing quote, else if quote escape not marked, mark it, else, add quote to field value
}
(...)
} else if( c == 13 or c == 10 ){
finished line, check last field of row read and replace data
}
To make it better/harder, define a parsing state machine, put the states into an Enum, and write the if gates with them in mind (this will make your code be more like a compiler parser).
You can find parsing code at different stages here: https://www.mkyong.com/java/how-to-read-and-parse-csv-file-in-java/
You need to change a little bit in your concept. If you want to edit a file then,
create a new file and write content in new file and delete old file and rename new file
with old name.
ArrayList<String> linesList = new ArrayList<>();
BufferedReader br;
String[] data;
File original=new File("D:\\abc\\file.txt");
try {
String line;
br = new BufferedReader(new FileReader(original));
while ((line = br.readLine()) !=null) {
linesList.add(line);
}
br.close();
}
catch (IOException e) { e.printStackTrace(); }
File tempFile = new File("D:\\abc\\tempfile.txt");
BufferedWriter bw = new BufferedWriter(new FileWriter(tempFile));
for (int i = 0; i < linesList.size(); i++) {
if(i==0){
bw.write(linesList.get(i));
bw.newLine();
}
else{
data = linesList.get(i).split(",");
String empid = data[0];
String name=data[1];
String lname=data[2];
String ccode = data[3].substring(0, 2);
String newData=empid+","+name+","+lname+","+ccode+"\n";
bw.write(newData);
bw.newLine();
}
}
bw.close();
if (!original.delete()) {
System.out.println("Could not delete file");
return;
}
// Rename the new file to the filename the original file had.
if (!tempFile.renameTo(original))
System.out.println("Could not rename file");
My guess is, the code I've written doesn't work with .CSV files, but only .txt.
The purpose of my code is to take the user input from field1, and check against my .CSV file to see if there is an instance of the user input located within the file. If there is, then it will be replaced by the user input from field2.
This works with my .txt file, but not with my .CSV file.
Here's the code that is activated at the push of a button (save button):
try{
// Input the file location into Path variable 'p'
//Cannot write to CSV files
//Path p = Paths.get("C:\\Users\\myname\\Documents\\Stock Take Program\\tiger.csv");
Path p = Paths.get("C:\\Users\\myname\\Desktop\\test.txt");
//Read the whole file to a ArrayList
List<String> fileContent = new ArrayList<>(Files.readAllLines(p));
//Converting user input from editSerialField to a string
String strSerial = editSerialField.getText();
//Converting user input from editLocationField to a string
String strLocation = editLocationField.getText();
//This structure looks for a duplicate in the text file, if so, replaces it with the user input from editLocationField.
for (int i = 0; i < fileContent.size(); i++)
{
if (fileContent.get(i).equals(strSerial))
{
fileContent.set(i, strLocation);
}
break;
}
// write the new String with the replaced line OVER the same file
Files.write(p, fileContent);
}catch(IOException e)
{
e.printStackTrace();
}
My question is, how can I update my code to work with updating and replacing the contents of a .CSV file with the user input, the same way as it works for my .txt files.
When writing to a text file, it replaces only the first line, but when writing to a .CSV file, it does not replace anything.
Is there anyway I should be writing my code differently to replace text within a .CSV file.
Any help is greatly appreciated, thanks.
I'm an idiot. My '.CSV' file is actually titled tiger.csv as a text file. I've now saved an actual CSV version and it is now working.
Haha, thanks for the help guys.
Probably should be in another question, but the problem relating to the change only working on the first line is due to the break being called and shortcutting the loop on the first way round. Put it within the if block.
for (int i = 0; i < fileContent.size(); i++)
{
if (fileContent.get(i).equals(strSerial))
{
fileContent.set(i, strLocation);
break;
}
}
Or leave it off completely if you want it to be able to update multiple lines.
HTH,
public static void main(String[] args) throws FileNotFoundException {
String UserEntredValu="Karnataka";
String csvFile = "C:/Users/GOOGLE/Desktop/sample/temp.csv";
String line = "";
String cvsSplitBy = ",";
PrintWriter pw = null;
pw = new PrintWriter(new File(csvFile));
try (BufferedReader br = new BufferedReader(new FileReader(csvFile))) {
while ((line = br.readLine()) != null) {
// use comma as separator
String[] country = line.split(cvsSplitBy);
for( int i = 0; i < country.length - 1; i++)
{
String element = country[i];
if(element.contains(UserEntredValu)){
String newEle=element.replace(element, "NEW INDIA");
pw.write(newEle);
System.out.println("done!");
pw.close();
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
I have a csv file that currently has 20 lines of data.
The data contains employee info and is in the following format:
first name, last name, Employee ID
So one line would like this: Emma, Nolan, 2
I know how to write to the file in java and have all 20 lines print to the console, but what I'm not sure how to do is how to get Java to print one specific line to the console.
I also want to take the last employee id number in the last entry and have java add 1 to it one I add new employees. I thinking this needs to be done with a counter just not sure how.
You can do something like this:
BufferedReader reader = new BufferedReader(new FileReader(<<your file>>));
List<String> lines = new ArrayList<>();
String line = null;
while ((line = reader.readLine()) != null) {
lines.add(line);
}
System.out.println(lines.get(0));
With BufferedReader you are able to read lines directly. This example reads the file line by line and stores the lines in an array list. You can access the lines after that by using lines.get(lineNumber).
You can read text from a file one line at a time and then do whatever you want to with that line, print it, compare it, etc...
// Construct a BufferedReader object from the input file
BufferedReader r = new BufferedReader(new FileReader("employeeData.txt"));
int i = 1;
try {
// "Prime" the while loop
String line = r.readLine();
while (line != null) {
// Print a single line of input file to console
System.out.print("Line "+i+": "+line);
// Prepare for next loop iteration
line = r.readLine();
i++;
}
} finally {
// Free up file descriptor resources
r.close();
}
// Remember the next available employee number in a one-up scheme
int nextEmployeeId = i;
BufferedReader reader =new BufferedReader(new FileReader("yourfile.csv"));
String line = "";
while((line=reader.readLine())!=null){
String [] employee =line.trim().split(",");
// if you want to check either it contains some name
//index 0 is first name, index 1 is last name, index 2 is ID
}
Alternatively, If you want more control over read CSV files then u can think about CsvBeanReader that will give you more access over files contents..
Here is an algorithm which I use for reading csv files. The most effective way is to read all the data in the csv file into a 2D array first. It just makes it a lot more flexible to manipulate the data.
That way you can specify which line of the file to print to the console by specifying it in the index of the array and using a for. I.e: System.out.println(employee_Data[1][y]); for record 1. y is the index variable for fields. You would need to use a For Loop of course, to print every element for each line.
By the way, if you want to use the employee data in a larger program, in which it may for example store the data in a database or write to another file, I'd recommend encapsulating this entire code block into a function named Read_CSV_File(), which will return a 2D String array.
My Code
// The return type of this function is a String.
// The CSVFile_path can be for example "employeeData.csv".
public static String[][] Read_CSV_File(String CSVFile_path){
String employee_Data[][];
int x;
int y;
int noofFields;
try{
String line;
BufferedReader in = new BufferedReader(new FileReader(CSVFile_path));
// reading files in specified directory
// This assigns the data to the 2D array
// The program keeps looping through until the line read in by the console contains no data in it i.e. the end of the file.
while ( (( line = in.readLine()) != null ){
String[] current_Record = line.split(",");
if(x == 0) {
// Counts the number of fields in the csv file.
noofFields = current_Record.length();
}
for (String str : values) {
employee_Data[x][y] = str;
System.out.print(", "+employee_Data[x][y]);
// The field index variable, y is incremented in every loop.
y = y + 1;
}
// The record index variable, x is incremented in every loop.
x = x + 1;
}
// This frees up the BufferedReader file descriptor resources
in.close();
/* If an error occurs, it is caught by the catch statement and an error message
* is generated and displayed to the user.
*/
}catch( IOException ioException ) {
System.out.println("Exception: "+ioException);
}
// This prints to console the specific line of your choice
System.out.println(("Employee 1:);
for(y = 0; y < noofFields ; y++){
// Prints out all fields of record 1
System.out.print(employee_Data[1][y]+", ");
}
return employee_Data;
}
For reading large file,
log.debug("****************Start Reading CSV File*******");
copyFile(inputCSVFile);
StringBuilder stringBuilder = new StringBuilder();
String line= "";
BufferedReader brOldFile = null;
try {
String inputfile = inputCSVFile;
log.info("inputfile:" + inputfile);
brOldFile = new BufferedReader(new FileReader(inputfile));
while ((line = brOldFile.readLine()) != null) {
//line = replaceSpecialChar(line);
/*do your stuff here*/
stringBuilder.append(line);
stringBuilder.append("\n");
}
log.debug("****************End reading CSV File**************");
} catch (Exception e) {
log.error(" exception in readStaffInfoCSVFile ", e);
}finally {
if(null != brOldFile) {
try {
brOldFile.close();
} catch (IOException e) {
}
}
}
return stringBuilder.toString();