I have a CSV file which will have delimiter or unclosed quotes inside a quotes, How do i make CSVReader ignore the quotes and delimiters inside quotes.
For example:
123|Bhajji|Maga|39|"I said Hey|" I am "5|'10."|"I a do "you"|get that"
This is the content of file.
The below program to read the csv file.
#Test
public void readFromCsv() throws IOException {
FileInputStream fis = new FileInputStream(
"/home/netspurt/awesomefile.csv");
InputStreamReader isr = new InputStreamReader(fis, "UTF-8");
CSVReader reader = new CSVReader(isr, '|', '\"');
for (String[] row; (row = reader.readNext()) != null;) {
System.out.println(Arrays.toString(row));
}
reader.close();
isr.close();
fis.close();
}
I get the o/p something like this.
[123, Bhajji, Maga, 39, I said Hey| I am "5|'10., I am an idiot do "you|get that]
what happened to quote after you
Edit:
The Opencsv dependency
com.opencsv
opencsv
3.4
from the source code of com.opencsv:opencsv:
/**
* Constructs CSVReader.
*
* #param reader the reader to an underlying CSV source.
* #param separator the delimiter to use for separating entries
* #param quotechar the character to use for quoted elements
* #param escape the character to use for escaping a separator or quote
*/
public CSVReader(Reader reader, char separator,
char quotechar, char escape) {
this(reader, separator, quotechar, escape, DEFAULT_SKIP_LINES, CSVParser.DEFAULT_STRICT_QUOTES);
}
see http://sourceforge.net/p/opencsv/source/ci/master/tree/src/main/java/com/opencsv/CSVReader.java
There is a constructor with an additional parameter escape which allows to escape separators and quotes (as per the javadoc).
As the CSV format specifies the quotes(") if its inside a field we need to precede it by another quote("). So this solved my problem.
123|Bhajji|Maga|39|"I said Hey|"" I am ""5|'10."|"I a do ""you""|get that"
Refrence: https://www.ietf.org/rfc/rfc4180.txt
Sorry but I don't have enough rep to add a comment so I will have to add an answer.
For your original question of what happened to the quote after the you the answer is the same as what happened to the quote before the I.
For CSV data the quote immediately before and after the separator is the start and end of the field data and is thus removed. That is why those two quotes are missing.
You need to escape out the quotes that are part of the field. The default escape character is the \
Taking a guess as to which quotes you want to escape the string should look like
123|Bhajji|Maga|39|"I said \"Hey I am \"5'10. Do \"you\" get that?\""
Related
I have a CSV and I want to check if it has all the data it should have. But it looks like ZWNBSP appears at the beginning of the 1st column name in the 1st string.
My simplified code is
#Test
void parseCsvTest() throws Exception {
Configuration.holdBrowserOpen = true;
ClassLoader classLoader = getClass().getClassLoader();
try (
InputStream inputStream = classLoader.getResourceAsStream("files/csv_example.csv");
CSVReader reader = new CSVReader(new InputStreamReader(inputStream))
) {
List<String[]> content = reader.readAll();
var csvStrings0line = content.get(0);
var csv1stElement = csvStrings0line[0];
var csv1stElementShouldBe = "Timestamp";
assertEquals(csv1stElementShouldBe,csv1stElement);
My CSV contains
"Timestamp","Source","EventName","CountryId","Platform","AppVersion","DeviceType","OsVersion"
"2022-05-02T14:56:59.536987Z","courierapp","order_delivered_sent","643","ios","3.11.0","iPhone 11","15.4.1"
"2022-05-02T14:57:35.849328Z","courierapp","order_delivered_sent","643","ios","3.11.0","iPhone 8","15.3.1"
My test fails with
expected: <Timestamp> but was: <Timestamp>
Expected :Timestamp
Actual :Timestamp
<Click to see difference>
Clicking on the see difference shows that there is a ZWNBSP at the beginning of the Actual text.
Copypasting my text to the online tool for displaying non-printable unicode characters https://www.soscisurvey.de/tools/view-chars.php shows only CR LF at the ends of the lines, no ZWNBSPs.
But where does it come from?
It's a BOM character. You may remove it yourself or use several other solutions (see https://stackoverflow.com/a/4897993/1420794 for instance)
That is the Unicode zero-width no-break space character. When used at the beginning of Unicode encoded text files, it serves as a 'byte-order-mark' . You read it to determine the encoding of the text file, then you can safely discard it if you want. The best thing you can do is spread awareness.
I am trying to parse a CSV file as below
String NEW_LINE_SEPARATOR = "\r\n";
CSVFormat csvFileFormat = CSVFormat.DEFAULT.withRecordSeparator(NEW_LINE_SEPARATOR);
FileReader fr = new FileReader("201404051539.csv");
CSVParser csvParser = csvFileFormat.withHeader().parse(fr);
List<CSVRecord> recordsList = csvParser.getRecords();
Now the file got normal lines ending with CRLF characters however for few lines there is additional LF character appearing in middle.
i.e.
a,b,c,dCRLF --line1
e,fLF,g,h,iCRLF --line2
Due to this, the parse operation creates three records whereas actually they are only two.
Is there a way I can get the LF character appearing in middle of second line not treated as line break and get two records only upon parsing?
Thanks
I think uniVocity-parsers is the only parser you will find that will work with line endings as you expect.
The equivalent code using univocity-parsers will be:
CsvParserSettings settings = new CsvParserSettings(); //many options here, check the tutorial
settings.getFormat().setLineSeparator("\r\n");
settings.getFormat().setNormalizedNewline('\u0001'); //uses a special character to represent a new record instead of \n.
settings.setNormalizeLineEndingsWithinQuotes(false); //does not replace \r\n by the normalized new line when reading quoted values.
settings.setHeaderExtractionEnabled(true); //extract headers from file
settings.trimValues(false); //does not remove whitespaces around values
CsvParser parser = new CsvParser(settings);
List<Record> recordsList = parser.parseAllRecords(new File("201404051539.csv"));
If you define a line separator to be \r\n then this is the ONLY sequence of characters that should identify a new record (when outside quotes). All values can have either \r or \n without being enclosed in quotes because that's NOT the line separator sequence.
When parsing the input sample you gave:
String input = "a,b,c,d\r\ne,f\n,g,h,i\r\n";
parser.parseAll(new StringReader(input));
The result will be:
LINE1 = [a, b, c, d]
LINE2 = [e, f
, g, h, i]
Disclosure: I'm the author of this library. It's open-source and free (Apache 2.0 license)
I have following csv file,
"id","Description","vale"
1,New"Account","val1"
I am unable to read the above csv file with opencsv jar. It cannot read New"Account, since the double quotes inside data. My csv reader constructor is following,
csvReader = new CSVReader(new FileReader(currentFile), ',', '\"', '\0');
This is invalid csv:
1,New"Account","val1"
should be:
1,"New""Account","val1" -> if you want 1 New"Account val1
or
1,"New""Account""","val1" -> if you want 1 New"Account" val1
Quotes inside (quoted) fields, must be escaped with another quote.
While you could change your code to read the malformed csv correctly, the csv data should be fixed in the first place, because you might get some more erros with larger csv-files or updates of that data.
Usually, quotes are used when there is a seperator or another quote inside the field. So if you would ignore the quotes and only split on the seperator, there will be problems if there is a seperator inside a field in future updates of the data - for example:
1,"John, Doe",123
That is as designed. Your constructor specifies a quote character as "\"" so OpenCSV will treat that character as a quote character, i.e. when it reads a quote it will ignore all commas until a matching quote is found.
To get around this you could use a FilterReader.
Reader reader = new FilterReader(fileReader) {
private int filter(int ch) {
return ch == '"'?' ':ch;
}
#Override
public int read(char[] cbuf, int off, int len) throws IOException {
int red = super.read(cbuf, off, len);
for ( int i = off; i < off + red; i++) {
cbuf[i] = (char)filter(cbuf[i]);
}
return red;
}
#Override
public int read() throws IOException {
return filter(super.read());
}
};
Using jcsv I'm trying to parse a CSV to a specified type. When I parse it, it says length of the data param is 1. This is incorrect. I tried removing line breaks, but it still says 1. Am I just missing something in plain sight?
This is my input string csvString variable
"Symbol","Last","Chg(%)","Vol",
INTC,23.90,1.06,28419200,
GE,26.83,0.19,22707700,
PFE,31.88,-0.03,17036200,
MRK,49.83,0.50,11565500,
T,35.41,0.37,11471300,
This is the Parser
public class BuySignalParser implements CSVEntryParser<BuySignal> {
#Override
public BuySignal parseEntry(String... data) {
// console says "Length 1"
System.out.println("Length " + data.length);
if (data.length != 4) {
throw new IllegalArgumentException("data is not a valid BuySignal record");
}
String symbol = data[0];
double last = Double.parseDouble(data[1]);
double change = Double.parseDouble(data[2]);
double volume = Double.parseDouble(data[3]);
return new BuySignal(symbol, last, change, volume);
}
}
And this is where I use the parser (right from the example)
CSVReader<BuySignal> cReader = new CSVReaderBuilder<BuySignal>(new StringReader( csvString)).entryParser(new BuySignalParser()).build();
List<BuySignal> signals = cReader.readAll();
jcsv allows different delimiter characters. The default is semicolon. Use CSVStrategy.UK_DEFAULT to get to use commas.
Also, you have four commas, and that usually indicates five values. You might want to remove the delimiters off the end.
I don't know how to make jcsv ignore the first line
I typically use CSVHelper to parse CSV files, and while jcsv seems pretty good, here is how you would do it with CVSHelper:
Reader reader = new InputStreamReader(new FileInputStream("persons.csv"), "UTF-8");
//bring in the first line with the headers if you want them
List<String> firstRow = CSVHelper.parseLine(reader);
List<String> dataRow = CSVHelper.parseLine(reader);
while (dataRow!=null) {
...put your code here to construct your objects from the strings
dataRow = CSVHelper.parseLine(reader);
}
You shouldn't have commas at the end of lines. Generally there are cell delimiters (commas) and line delimiters (newlines). By placing commas at the end of the line it looks like the entire file is one long line.
I am writing a Java app to export data from Oracle to csv file
Unfortunately the content of data may quite tricky. Still comma is the deliminator, but some data on a row could be like this:
| ID | FN | LN | AGE | COMMENT |
|----------------------------------------------------------------|
| 123 | John | Smith | 39 | I said "Hey, I am 5'10"." |
|----------------------------------------------------------------|
so this is one of the string on the comment column:
I said "Hey, I am 5'10"."
No kidding, I need to show above comment without compromise in excel or open office from a CSV file generated by Java, and of course cannot mess up other regular escaping situation(i.e. regular double quotes, and regular comma within a tuple). I know regular expression is powerful but how can we achieve the goal with such complicated situation?
There are several libraries. Here are two examples:
❐ Apache Commons Lang
Apache Commons Lang includes a special class to escape or unescape strings (CSV, EcmaScript, HTML, Java, Json, XML): org.apache.commons.lang3.StringEscapeUtils.
Escape to CSV
String escaped = StringEscapeUtils
.escapeCsv("I said \"Hey, I am 5'10\".\""); // I said "Hey, I am 5'10"."
System.out.println(escaped); // "I said ""Hey, I am 5'10""."""
Unescape from CSV
String unescaped = StringEscapeUtils
.unescapeCsv("\"I said \"\"Hey, I am 5'10\"\".\"\"\""); // "I said ""Hey, I am 5'10""."""
System.out.println(unescaped); // I said "Hey, I am 5'10"."
* You can download it from here.
❐ OpenCSV
If you use OpenCSV, you will not need to worry about escape or unescape, only for write or read the content.
Writing file:
FileOutputStream fos = new FileOutputStream("awesomefile.csv");
OutputStreamWriter osw = new OutputStreamWriter(fos, "UTF-8");
CSVWriter writer = new CSVWriter(osw);
...
String[] row = {
"123",
"John",
"Smith",
"39",
"I said \"Hey, I am 5'10\".\""
};
writer.writeNext(row);
...
writer.close();
osw.close();
os.close();
Reading file:
FileInputStream fis = new FileInputStream("awesomefile.csv");
InputStreamReader isr = new InputStreamReader(fis, "UTF-8");
CSVReader reader = new CSVReader(isr);
for (String[] row; (row = reader.readNext()) != null;) {
System.out.println(Arrays.toString(row));
}
reader.close();
isr.close();
fis.close();
* You can download it from here.
Excel has to be able to handle the exact same situation.
Put those things into Excel, save them as CSV, and examine the file with a text editor. Then you'll know the rules Excel is applying to these situations.
Make Java produce the same output.
The formats used by Excel are published, by the way...
****Edit 1:**** Here's what Excel does
****Edit 2:**** Note that php's fputcsv does the same exact thing as excel if you use " as the enclosure.
rdeslonde#mydomain.com
Richard
"This is what I think"
gets transformed into this:
Email,Fname,Quoted
rdeslonde#mydomain.com,Richard,"""This is what I think"""
Thanks to both Tony and Paul for the quick feedback, its very helpful. I actually figure out a solution through POJO. Here it is:
if (cell_value.indexOf("\"") != -1 || cell_value.indexOf(",") != -1) {
cell_value = cell_value.replaceAll("\"", "\"\"");
row.append("\"");
row.append(cell_value);
row.append("\"");
} else {
row.append(cell_value);
}
in short if there is special character like comma or double quote within the string in side the cell, then first escape the double quote("\"") by adding additional double quote (like "\"\""), then put the whole thing into a double quote (like "\""+theWholeThing+"\"" )
You could also look at how Python writes Excel-compatible csv files.
I believe the default for Excel is to double-up for literal quote characters - that is, literal quotes " are written as "".
If you're using CSVWriter. Check that you don't have the option
.withQuotechar(CSVWriter.NO_QUOTE_CHARACTER)
When I removed it the comma was showing as expected and not treating it as new column
"cell one","cell "" two","cell "" ,three"
Save this to csv file and see the results, so double quote is used to escape itself
Important Note
"cell one","cell "" two", "cell "" ,three"
will give you a different result because there is a space after the comma, and that will be treated as "
String stringWithQuates = "\""+ "your,comma,separated,string" + "\"";
this will retain the comma in CSV file
In openCSV, use below method to create csvWriter obj,
CSVWriter csvWriter = new CSVWriter(writer, CSVWriter.DEFAULT_SEPARATOR, CSVWriter.DEFAULT_ESCAPE_CHARACTER, CSVWriter.DEFAULT_LINE_END, CSVWriter.DEFAULT_QUOTE_CHARACTER);
In this, DEFAULT_QUOTE_CHARACTER is very important.
It will work perfectly, If you want to insert any ',' or '"' in csv file.