We Keep Coding

How to use Commons CSV remove duplicate in csv file using Java?

I have a csv file. It contains several duplicate columns. I am trying to remove these duplicates using Java. I found Apache Common csv library, some people use it to remove duplicate rows. How can I use it to remove or skip duplicate columns?
For example: my csv header is:
ID Name Email Email
So far my code is:
Reader reader = Files.newBufferedReader(Paths.get("user.csv"));
// read csv file
Iterable<CSVRecord> records = CSVFormat.DEFAULT.withFirstRecordAsHeader()
.withIgnoreHeaderCase()
.withTrim()
.parse(reader);
for (CSVRecord record : records) {
System.out.println("Record #: " + record.getRecordNumber());
System.out.println("ID: " + record.get("ID"));
System.out.println("Name: " + record.get("Name"));
System.out.println("Email: " + record.get("Email"));
}
// close the reader
reader.close();

Your code is close to what you need - you just need to use CSVPrinter to write out your data to a new file.
import java.io.IOException;
import java.io.Reader;
import java.io.Writer;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.nio.file.StandardOpenOption;
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVPrinter;
import org.apache.commons.csv.CSVRecord;
public class App {
public static void main(String[] args) throws IOException {
try (final Reader reader = Files.newBufferedReader(Paths.get("source.csv"),
StandardCharsets.UTF_8)) {
final Writer writer = Files.newBufferedWriter(Paths.get("target.csv"),
StandardCharsets.UTF_8,
StandardOpenOption.CREATE); // overwrites existing output file
try (final CSVPrinter printer = CSVFormat.DEFAULT
.withHeader("ID", "Name", "Email")
.print(writer)) {
// read each input file record:
Iterable<CSVRecord> records = CSVFormat.DEFAULT
.withFirstRecordAsHeader()
.withIgnoreHeaderCase()
.withTrim()
.parse(reader);
// write each output file record
for (CSVRecord record : records) {
printer.print(record.get("ID"));
printer.print(record.get("Name"));
printer.print(record.get("Email"));
printer.println();
}
}
}
}
}
This transforms the following source file:
ID,Name,Email,Email
1,Albert,foo#bar.com,foo#bar.com
2,Brian,baz#bat.com,baz#bat.com
To this target file:
ID,Name,Email
1,Albert,foo#bar.com
2,Brian,baz#bat.com
Some points to note:
I was wrong in my comment. You do not need to use column indexes - you can use headings (as I do above) in your specific case.
Whenever reading and writing a file, it is recommended to provide the character encoding. In my case, I use UTF-8. (This assumes the original file was created as a URF-8 file, of course - or is compatible with that encoding.)
When opening the reader and the writer I use "try-with-resources" statements. These mean I do not have to explicitly close the file resources - Java takes care of that for me.

How to store multiple columns of a csv dataset in a single variable in java so that the variable can be used as input feature for ml model

I have done this in python. Here is my python code:
Here X is the input variable in which I stored all the
input columns of csv file and y is the target variable.
dataset=pandas.read_csv("newone.csv")
features = [0,1,4,5,6,7]
X =dataset.iloc[:,features]
y =dataset.iloc[:,2]
How can I do this in java?
Here is my java code in which I read the csv file but
I am able to store only one column value of the csv in a variable.
public static void main(String[] args) throws IOException {
BufferedReader reader = Files.newBufferedReader(Paths.get("C:/Users/N/Desktop/newone.csv"));
CSVParser csvParser = new CSVParser(reader,
CSVFormat.DEFAULT.withHeader("Enounter", "Relation", "Event", "Tag","Encounter_no", "Diagonosis", "User_Id", "Client_Id").withIgnoreHeaderCase().withTrim());
for (CSVRecord csvRecord : csvParser) {
encntr=csvRecord.get("Encounter");
}
}
----------

Consider using MyKong's code on how to read CSV files in Java. Something like the following snippet below (heavily borrowed from MyKong's better version):
br = new BufferedReader(new FileReader(csvFile));
while ((line = br.readLine()) != null) {
String[] valuesFromLine = line.split(cvsSplitBy);
String secondValue = valuesFromLine[1];
// do something with second value
}

This depends entirely on what the relationship between your columns is like. It is impossible to answer this question in a general manner as this changes from dataset to dataset and even from algorithm to algorithm, but here are a few approaches you might like to try:
Use Principal Component Analysis to identify if there any variables in your desired tuple of columns you can omit because they contribute very little to the row class variable.
Use Feature Hashing to reduce the dimensionality of your dataset by bundling together related properties (this does not work as a blanket solution - indeed, nothing in ML ever does. Try it before you commit to it).
If the columns you'd like to unite are numerical, you may want to think of an algorithm to join them in a unique way, or a way which makes sense. If they are categorical, a sparse one-hot bit vector may help you.

Download these jar https://mvnrepository.com/artifact/org.apache.poi/poi-ooxml/3.15 and https://mvnrepository.com/artifact/org.apache.poi/poi/3.15
and add them to your build path
import java.io.FileInputStream;
import java.io.IOException;
import org.apache.poi.ss.usermodel.Cell;
import org.apache.poi.ss.usermodel.Row;
import org.apache.poi.xssf.usermodel.XSSFSheet;
import org.apache.poi.xssf.usermodel.XSSFWorkbook;
public class ExcelReader {
public static void main(String[] args) throws IOException
{
//specify your file path
FileInputStream file = new FileInputStream("D:\\test.xlsx");
XSSFWorkbook workbook = new XSSFWorkbook(file);
//to fetch sheet
XSSFSheet sheet = workbook.getSheetAt(0);
// for iterating through rows
for(int c=0;c<=sheet.getLastRowNum();c++)
{
// for iterating through columns
Row rows = sheet.getRow(c);
for(int b=0;b<=rows.getLastCellNum();b++)
{
Cell cells=rows.getCell(b);
//to read cell value as string
String comp=cells.getStringCellValue();
}
}
}
}

Deleting Duplicates in a List

I am managing an email listserv for my club. I need to sort the email addresses from a txt file, delete the duplicates, and then ouput them so that I may easily organize them. Most of my code is spliced together from various sources online since I have very minimal programming skills right now. This is what I have so far:
import java.io.IOException;
import java.nio.charset.Charset;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.*;
public class ReadAllLines {
/**
* #param args the command line arguments
*/
public static void main(String[] args) {
// TODO code application logic here
Path textFile_path = Paths.get("V:/", "Entrepreneurship Club Email List.txt");
Charset charset = Charset.forName("ISO-8859-1");
try {
//Reads txt file
List<String> lines = Files.readAllLines(textFile_path, charset);
//Sorts txt file alphabetically
Collections.sort(lines);
//prints txt file
for (String line : lines) {
System.out.println(line);
}
//File Not Found Error
} catch (IOException e) {
System.out.println(e);
}
}
}

I need to sort the email addresses from a txt file, delete the duplicates, and then ouput them so that I may easily organize them.
In that case, use a SortedSet
Set<String> set = new TreeSet<>(Files.readAllLines(textFile_path, charset));
set will be sorted with duplicates ignored.
In Java 8, you can do
Set<String> set = Files.lines(textFile_path, charset)
.sorted()
.collect(toSet());

Java OpenCSV How to edit specific cells from csv file

I have a CSV file which looks like this:
http://gyazo.com/5dcfb8eca4e133cbeac87f514099e320.png
I need to figure out how I can read specific cells and update them in the file.
This is the code I am using:
import java.util.List;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import com.opencsv.*;
public class ReadCSV {
private static final char SEPARATOR = ';';
public static void updateCSV(String input, String output, String replace, int row, int col) throws IOException {
CSVReader reader = new CSVReader(new FileReader(input),SEPARATOR);
List<String[]> csvBody = reader.readAll();
csvBody.get(row)[col]=replace;
reader.close();
CSVWriter writer = new CSVWriter(new FileWriter(output),SEPARATOR,' ');
writer.writeAll(csvBody);
writer.flush();
writer.close();
}
public static void main(String[] args) throws IOException {
String source = "townhall_levels.csv";
String destiantion="output.csv";
ReadCSV.updateCSV(source, destiantion, "lol", 1, 1);
}
}
In this code I am just trying to change A1 to "lol" as an example test to see if it works but I get the following error:
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
at ReadCSV.updateCSV(ReadCSV.java:16)
at ReadCSV.main(ReadCSV.java:30)
How should I go about achieving my goal and fixing the error?
CSV File: www.forumalliance.net/townhall_levels.csv

You're using ;as the separator to parse the file. Your file uses ,. Also, using a space as the quote char doesn't make much sense. You should use " instead, since that's also what your file uses.

The first values you're passing to row and col are 1 and 1. However, these need to start at 0.

Convert .csv to .xls in Java

Does anyone here know of any quick, clean way to convert csv files to xls or xlsx files in java?
I have something to manage csv files already in place and I need the extra compatibility for other programs.
Sample code in addition to package names is always well appreciated.
Many thanks,
Justian
Here's my code thus far. I need to remove the returns ("\n") from the lines. Some of my cells contain multiple lines of information (a list), so I can use "\n" in csv to indicate multiple lines within a cell, but xls treats these as if I mean to put them on a new line.
The code is modified from the internet and a little messy at the moment. You might notice some deprecated methods, as it was written in 2004, and be sure to ignore the terrible return statements. I'm just using S.o.p at the moment for testing and I'll clean that up later.
package jab.jm.io;
import java.io.DataInputStream;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.ArrayList;
import org.apache.poi.hssf.usermodel.HSSFCell;
import org.apache.poi.hssf.usermodel.HSSFRow;
import org.apache.poi.hssf.usermodel.HSSFSheet;
import org.apache.poi.hssf.usermodel.HSSFWorkbook;
public class FileConverter {
public static String ConvertCSVToXLS(String file) throws IOException {
if (file.indexOf(".csv") < 0)
return "Error converting file: .csv file not given.";
String name = FileManager.getFileNameFromPath(file, false);
ArrayList<ArrayList<String>> arList = new ArrayList<ArrayList<String>>();
ArrayList<String> al = null;
String thisLine;
DataInputStream myInput = new DataInputStream(new FileInputStream(file));
while ((thisLine = myInput.readLine()) != null) {
al = new ArrayList<String>();
String strar[] = thisLine.split(",");
for (int j = 0; j < strar.length; j++) {
// My Attempt (BELOW)
String edit = strar[j].replace('\n', ' ');
al.add(edit);
}
arList.add(al);
System.out.println();
}
try {
HSSFWorkbook hwb = new HSSFWorkbook();
HSSFSheet sheet = hwb.createSheet("new sheet");
for (int k = 0; k < arList.size(); k++) {
ArrayList<String> ardata = (ArrayList<String>) arList.get(k);
HSSFRow row = sheet.createRow((short) 0 + k);
for (int p = 0; p < ardata.size(); p++) {
System.out.print(ardata.get(p));
HSSFCell cell = row.createCell((short) p);
cell.setCellValue(ardata.get(p).toString());
}
}
FileOutputStream fileOut = new FileOutputStream(
FileManager.getCleanPath() + "/converted files/" + name
+ ".xls");
hwb.write(fileOut);
fileOut.close();
System.out.println(name + ".xls has been generated");
} catch (Exception ex) {
}
return "";
}
}

Don't know if you know this already, but:
Excel (if that's your real target) is easily able to read .csv files directly, so any conversion you'd do would only be a courtesy to your less "gifted" users.
CSV is a lowest-common-denominator format. It's unlikely for any converter to add information to that found in a .csv file that will make it more useful. In other words, CSV is a "dumb" format and converting it to .xls will (probably) increase file size but not make the format any smarter.
Curtis' suggestion of POI is the first thing that would come to my mind too.
If you're doing this conversion on a Windows machine, another alternative could be Jacob, a Java-COM bridge that would allow you to effectively remote control Excel from a Java program so as to do things like open a file and save in a different format, perhaps even applying some formatting changes or such.
Finally, I've also had some success doing SQL INSERTs (via JDBC) into an Excel worksheet accessed via the JDBC-ODBC bridge. i.e. ODBC can make an Excel file look like a database. It's not very flexible though, you can't ask the DB to create arbitrarily named .XLS files.
EDIT:
It looks to me like readLine() is already not giving you whole lines. How is it to know that carriage return is not a line terminator? You should be able to verify this with debug print statements right after the readLine().
If this is indeed so, it would suck because the way forward would be for you to
either recognize incomplete lines and paste them together after the fact,
or write your own substitute for readLine(). A simple approach would be to read character by character, replacing CRs within a CSV string and accumulating text in a StringBuilder until you feel you have a complete line.
Both alternatives are work you probably weren't looking forward to.

If you want to read or write XLS or XLSX files in Java, Apache POI is a good bet: http://poi.apache.org/

Copy paste the below program,I ran the program and it is working fine,Let me know if you have any concerns on this program.(You need Apache POI Jar to run this program)
import java.io.DataInputStream;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.ArrayList;
import org.apache.poi.hssf.usermodel.HSSFCell;
import org.apache.poi.hssf.usermodel.HSSFRow;
import org.apache.poi.hssf.usermodel.HSSFSheet;
import org.apache.poi.hssf.usermodel.HSSFWorkbook;
import org.apache.poi.ss.usermodel.Cell;
public class CSVToExcelConverter {
public static void main(String args[]) throws IOException
{
ArrayList arList=null;
ArrayList al=null;
String fName = "test.csv";
String thisLine;
int count=0;
FileInputStream fis = new FileInputStream(fName);
DataInputStream myInput = new DataInputStream(fis);
int i=0;
arList = new ArrayList();
while ((thisLine = myInput.readLine()) != null)
{
al = new ArrayList();
String strar[] = thisLine.split(",");
for(int j=0;j<strar.length;j++)
{
al.add(strar[j]);
}
arList.add(al);
System.out.println();
i++;
}
try
{
HSSFWorkbook hwb = new HSSFWorkbook();
HSSFSheet sheet = hwb.createSheet("new sheet");
for(int k=0;k<arList.size();k++)
{
ArrayList ardata = (ArrayList)arList.get(k);
HSSFRow row = sheet.createRow((short) 0+k);
for(int p=0;p<ardata.size();p++)
{
HSSFCell cell = row.createCell((short) p);
String data = ardata.get(p).toString();
if(data.startsWith("=")){
cell.setCellType(Cell.CELL_TYPE_STRING);
data=data.replaceAll("\"", "");
data=data.replaceAll("=", "");
cell.setCellValue(data);
}else if(data.startsWith("\"")){
data=data.replaceAll("\"", "");
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellValue(data);
}else{
data=data.replaceAll("\"", "");
cell.setCellType(Cell.CELL_TYPE_NUMERIC);
cell.setCellValue(data);
}
//*/
// cell.setCellValue(ardata.get(p).toString());
}
System.out.println();
}
FileOutputStream fileOut = new FileOutputStream("test.xls");
hwb.write(fileOut);
fileOut.close();
System.out.println("Your excel file has been generated");
} catch ( Exception ex ) {
ex.printStackTrace();
} //main method ends
}
}

The tools in Excel are not adequate for what the OP wants to do. He's on the right track there. Excel cannot import multiple CSV files into different worksheets in the same file, which is why you'd want to do it in code. My suggestion is to use OpenCSV to read the CSV, as it can automatically correct for newlines in data and missing columns, and it's free and open source. It's actually very, very robust and can handle all sorts of different non-standard CSV files.

You wrote:
I have something to manage csv files
already in place and I need the extra
compatibility for other programs.
What are those other programs? Are they required to access your data through Excel files, or could they work with an JDBC or ODBC connection to a database? Using a database as the central location, you could extract the data into CSV files or other formats as needed.

I created a small software called csv2xls. It needs Java.