We Keep Coding

Java: Using InputStream and Apache Commons CSV without line numbers

This is probably very simple, but I have not been able to find an option to do this. I'm trying to Apache Commons CSV to read a file for later validations. The CSV in question is submitted as an Input Stream, which seems to add an additional column to the file when it reads it, containing the line numbers. I would like to be able to ignore it, if possible, as the header row does not contain a number, which causes an error. Is there an option already in InputStream to do this, or will I have to set up some kind of post processing?
The code I'm using is as follows:
public String validateFile(InputStream filePath) throws Exception{
System.out.println("Sending file to reader");
System.out.println(filePath);
InputStreamReader in = new InputStreamReader(filePath);
//CSVFormat parse needs a reader object
System.out.println("sending reader to CSV parse");
for (CSVRecord record : CSVFormat.DEFAULT.withHeader().parse(in)) {
for (String field : record) {
System.out.print("\"" + field + "\", ");
}
System.out.println();
}
return null;
}
When using withHeader(), I end up with the following error:
java.lang.IllegalArgumentException: A header name is missing in [, Employee_ID, Department, Email]
and I can't simply skip it, as I will need to do some validations on the header row.
Also, here is an example CSV file:
"Employee_ID", "Department", "Email"
"0123456","Department of Hello World","John.Doe#gmail.com"
EDIT: Also, The end goal is to validate the following:
That there are columns called "Employee_ID", "Department", and "Email". For this, I think I'll need to remove .withHeader().
Each line is comma delimited.
There are no empty cells values

Newer versions of Commons-CSV have trouble with empty headers.
Maybe that's the case here as well?
You just mentioned "no empty cell values" not sure if this included headers as well...
Also see: https://issues.apache.org/jira/browse/CSV-257
Setting .setAllowMissingColumnNames(true) did the trick for me.
final CSVFormat csvFormat = CSVFormat.Builder.create()
.setHeader(HEADERS)
.setAllowMissingColumnNames(true)
.build();
final Iterable<CSVRecord> records = csvFormat.parse(reader);

Univocity - Detect missing column when parsing CSV

I'm using Univocity library to parse CSV and it works perfectly, but I need a way to detect if the file being parsed has less columns than required
For example, if I'm expecting a 3 columns file, with columns mapped to [H1,H2,H3] then I received a file (which has no headers) that looks like
V1_H1,V1_H2
V2_H1,V2_H2
When using
record.getString("H3");
this would return null, instead, I need this file to either fail to be parsed or I can check if it misses a column and stop processing it
Is there any way to achieve this?

So since my main issue here is to make sure that the headers count is the same as the number of columns provided in the CSV file, and since I'm using an iterator to iterate over records, I've added a check like:
CsvParser parser = new CsvParser(settings);
ResultIterator<Record, ParsingContext> iterator = parser.iterateRecords(inputStream).iterator();
if(iterator.getContext().parsedHeaders().length != settings.getHeaders().length){
throw new Exception("Invalid file");
}
It's working for me, not sure if there is a better way to do it.

I've watched Univocity documentation and I've found here that there is a way to add annotations to the destination objects you are going to generate from the CSV input
#Parsed
#Validate
public String notNulNotBlank; //This should fail if the field is null or blank
#Parsed
#Validate(nullable = true)
public String nullButNotBlank;
#Parsed
#Validate(allowBlanks = true)
public String notNullButBlank;
This will also help you to use the objects instead of having to work with fields.
Hope that helps :-)

How to use cell values of .csv file in jmeter?

I have a .csv file which contains the multiple entries of employee names.Now i want to read that .csv file in my jmx script in jmeter. It should be in a way that each thread of my jmx script reads the different value from the .csv file. And then i want to use those cell values in the HTTP Request.
I am using Bean Shell pre processor for it. But it is of no help for me.The below is code which i have written for it
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
CSVReader reader = new CSVReader(new FileReader(C:/Users/manoj/Downloads/View_All_names.csv));
String[] row;
String name;
while ((row = reader.readNext()) != null)
{
name = row[1];
}
The value "name" i want to use it in my HTTP Request then.DO i have to mention this attribute "name" in Test Plan??
Any help is appreciated.

First of all, are you aware of __CSVRead() function? You should be able to use it directly in the place where you need the value from CSV file without having to write a single line of code.
Just in case you still want Beanshell for any reason.
You need to surround the path to CSV file with quotation marks
You need to write the resulting value into JMeter Variables for later reuse.
Where did you get the above code? If CSVReader stands for com.helger.commons.csv.CSVReader from ph-commons-6.2.4.jar - you need to use it a little bit differently, like use List instead of array of strings to wit:
import com.helger.commons.csv.CSVReader;
CSVReader reader = new CSVReader(new FileReader("C:/Users/manoj/Downloads/View_All_names.csv"));
List row;
String name;
int counter = 1;
while ((row = reader.readNext()) != null) {
name = row.get(1).toString();
vars.put("name_" + counter, name);
counter++;
}
After Beanshell PreProcessor finishes its work you will be able to access variables values like:
${name_1}
${name_2}
${name_3}
etc.
vars stands for an instance of JMeterVariables class which provides read/write access to the JMeter Variables in scope. See How to Use BeanShell: JMeter's Favorite Built-in Component article for more information on Beanshell scripting in JMeter tests.

All you need is to put one CSV Data Set Config component, and it will read entries for you. Each line for each thread (user). In the component you need to specify variable name which you will later use in your test plan, as well as the path to your .csv file.
If you put variable name like "name" (w/out quotes) you will use ${name} in requests/samplers where you want to use it.

A simple way to capture the column values from excel in jMeter
Just insert data in excel column wise i.e. horizontally insert all the data and use
${__CSVRead(filePath,ColNum)}

CSV Generation, Date and Long number formats

I am generating a CSV file on the fly/runtime with JSF and the code is as follows
FacesContext context = FacesContext.getCurrentInstance();
HttpServletResponse response = HttpServletResponse)context.getExternalContext().getResponse();
int read = 0;
byte[] bytes = new byte[1024];
response.setContentType("text/csv");
response.setHeader("Content-Disposition", "attachment;filename=\"" + fileName + "\"");
ServletOutputStream os = null;
StringBuffer stringBuffer1 = new StringBuffer("");
stringBuffer1.append("Disconnect Time");
stringBuffer1.append(',');
stringBuffer1.append("Calling Number");
stringBuffer1.append("01/06/2010 01:00:35 AM");
stringBuffer1.append(", ");
stringBuffer1.append("447744369900");
ByteArrayInputStream bis1;
try {
bis1 = new ByteArrayInputStream(stringBuffer1.toString().getBytes("UTF-8"));
os = response.getOutputStream();
while ((read = bis1.read(bytes)) != -1) {
os.write(bytes, 0, read);
}
os.flush();
os.close();
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
FacesContext.getCurrentInstance().responseComplete();
and below is the content of the file when opened
Disconnect Time Calling Number
1/6/2010 1:00 4.47744E+11
The Actual expected result will be complete date format w.r.t AM/PM and number in complete length.
I have already tried example of double quotes as given Excel CSV - Number cell format and some other of adding blank space but they did not worked. Also the user will be performing arithmetic operation on the number columns.
Thanks in advance.

I'm no JSF expert but this seems entirely related to how Excel reads this information.
I tried playing a bit with the data, but could not make it do what you want it to do in CSV.
I created a sample CSV file with this content (using Nopepad++):
Disconnect Time, Calling Number
01/06/2010 01:00:35 AM,447744369900
And when I viewed it using Excel it gave me the same output you got - so the question you are facing is related to showing CSV in Excel - and not JSF.

You've mentioned that you're generating the file dynamically. If you are exporting data from a dataTable you might want to consider using the p:dataExporter component from PrimeFaces (you'll have to use a p:dataTable though). It may save you programatically creating the excel file if it's already in the correct structure in your dataTable.
Then it's as easy as attaching p:dataExporter to a link, and pointing it to your dataTable (tbl in this case).
<h:commandLink>
<p:graphicImage value="/images/excel.png" />
<p:dataExporter type="xls" target="tbl" fileName="cars" />
</h:commandLink>
It also supports CSV and PDF by the way.

RonK is right. There is no way you can tell Excel how to format the data when using a CSV file. Excel just uses a best guess as to what the data in the CSV columns are and then applies the default format for that type of data.
It is very likely that Excel still has the correct data (i.e. still has the seconds of your time column) and it is just an issue with formatting in Excel.
Have you tried applying a different format in Excel?
You will either have to tell the users how to format the data correctly in Excel or you have to create an Excel file directly (e.g. using Apache POI) and applying the correct format that way.

Sorry, looks like a bug ("feature", "deficit") in Excel.
It assumes anything that looks like a number is a number (and loses precision when reading it).
The simplest way probably is to ensure that the phone numbers do not look like numbers.
For example by including ( and similar characters.
Remeber that in Excel you are sometimes expected to write '=123+456 if you don't want it to compute the equation. Try this, too.
Date is probably just formatting, try formatting the column to include seconds.

To show correct format for dates and numbers in CSV I used =("447744369900") approach. The disadvantage is this that we cannot perform arithmetic operations on the cell. Now as Discussed with RonK and Turismo I am now shifted to Apache Excel library that allows you to define the format of the Excel file and create a XLS file instead of CSV. It is little slower than CSV but fulfills my requirements. See API to write huge excel files using java
Thanks again RonK and Turismo

After you open the CSV in excel, you can change the format of the "calling number" cells as Number. Select column -> Format Cells -> Change to Number

Load txt's file into Java application and save it to XML's file

I read the next answer about load file into java application.
I need to write a program that load .txt, which contains a list of records. After I parse it, I need to match the records (with conditions that I will check), and save the result to XML's file.
I am stuck on this issue, and I will happy for answer to next questions:
How I load the .txt file into Java?
After I load the file, how I can acsses to the information into it? for example, How I can asked if the first line of one of the records is equal to "1";
How I export the result to XML's file.

one: you need a sample-code for reading a file line by line
two: the split-method of a string might be helpful. For instance getting the number of the first element if information is seperated by a space
String myLine;
String[] components = myLine.split(" ");
if(components != null && components.length >= 1) {
int num = Integer.parseInt(components[0]);
....
}
three: you can just write it like any text-file, or use any XML-Writer you want

Basic I/O
Integer.parseInt(1stLine)
There are a plethora of choices.
Create POJO's to represent the records and write them using XMLEncoder
SAX
DOM..