We Keep Coding

How to speed up the formation of excel file?

I have such a code that generates using the POI library. The list contains 250,000 lines. The formation of an Excel file takes 30-40 minutes. Reading about POI, I realized that this is not normal. How to speed up the process?
private void create(List<User> list) throws IOException {
XSSFWorkbook workbook = new XSSFWorkbook();
XSSFSheet sheet = workbook.createSheet("Statistic");
var rowCount = 0;
for (int i = 0; i < list.size(); i++) {
sheet.autoSizeColumn(i);
XSSFRow row = sheet.createRow(++rowCount);
XSSFCell loginCell = row.createCell(0);
loginCell.setCellValue(list.get(i).getLogin());
//...
XSSFCell amountSpecCell = row.createCell(9);
amountSpecCell.setCellValue(list.get(i).getLevel());
}
try {
FileOutputStream fileOutputStream = new FileOutputStream("myfilenew.xlsx");
workbook.write(fileOutputStream);
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
workbook.close();
}
}

Read Excel horizontally with TestNG

I can read an Excel sheet vertically and the pass the values to TestNG dataprovide.
This makes TestNG executes each row.
public Object[][] getData(ITestContext context) throws IOException {
Object[][] obj = null;
try {
File file = new File(context.getCurrentXmlTest().getParameter("resource"));
FileInputStream fis = new FileInputStream(file);
XSSFWorkbook wb = new XSSFWorkbook(fis);
XSSFSheet sheet = wb.getSheetAt(0);
wb.close();
int rowCount= sheet.getLastRowNum();
int colCount = sheet.getRow(0).getLastCellNum();
obj = new Object[rowCount][1];
DataFormatter df = new DataFormatter();
for (int i = 0; i < rowCount; i++) {
Map<String, String> datamap= new HashMap<String, String>();
for (int j = 0; j < colCount; j++) {
String a = df.formatCellValue(sheet.getRow(0).getCell(j));
String b = df.formatCellValue(sheet.getRow(i+1).getCell(j));
if(!b.equals(""))
datamap.put(a, b);
if(b.equals("[PWD]")) {
b=PropertiesFile.getProperty("psw");
byte[] byteArray = Base64.decodeBase64(b.getBytes());
String decodedString = new String(byteArray);
b=decodedString;
datamap.put(a, b);
}
}
obj[i][0] =datamap;
}
}catch(Exception e){
e.printStackTrace();
}
return obj;
}
I'd like to read it horizontally, so that TestNG executes each column (the first column would be the keys column), how can i do?
Thanks

You can transpose your sheet and work with former columns which are now the rows: How to transpose sheet with POI SS/XSSF?
Or just change your loop nesting so that you go by columns in the outer loops. Not by rows.

I think i solved it
public Object[][] getData(ITestContext context) throws IOException {
Object[][] obj = null;
try {
File file = new File(context.getCurrentXmlTest().getParameter("resource"));
FileInputStream fis = new FileInputStream(file);
XSSFWorkbook wb = new XSSFWorkbook(fis);
XSSFSheet sheet = wb.getSheetAt(0);
wb.close();
int rowCount= sheet.getLastRowNum();
int colCount = sheet.getRow(0).getLastCellNum();
obj = new Object[colCount-1][1];
DataFormatter df = new DataFormatter();
for (int i = 0; i < colCount-1; i++) {
Map<String, String> datamap= new HashMap<String, String>();
for (int j = 0; j < rowCount+1; j++) {
String a = df.formatCellValue(sheet.getRow(j).getCell(0));
String b = df.formatCellValue(sheet.getRow(j).getCell(i+1));
System.out.println(a+ " "+b);
if(!b.equals(""))
datamap.put(a, b);
if(b.equals("[PWD]")) {
b=PropertiesFile.getProperty("psw");
byte[] byteArray = Base64.decodeBase64(b.getBytes());
String decodedString = new String(byteArray);
b=decodedString;
datamap.put(a, b);
}
}
obj[i][0] =datamap;
}
}catch(Exception e){
e.printStackTrace();
}
return obj;
}

I have written a program in java to read data from MS excel sheet but it is showing that (The system cannot find the file specified)

public class Excel {
public static void main(String[] args) throws IOException, FileNotFoundException {
try {
InputStream input = new BufferedInputStream(new FileInputStream("D:/one"));
POIFSFileSystem fs = new POIFSFileSystem(input);
HSSFWorkbook wb = new HSSFWorkbook(fs);
HSSFSheet sheet = wb.getSheetAt(0);
Iterator rows = sheet.rowIterator();
while (rows.hasNext()) {
HSSFRow next = (HSSFRow) rows.next();
System.out.println("\n");
Iterator cells = next.cellIterator();
while (cells.hasNext()) {
HSSFCell next2 = (HSSFCell) cells.next();
if (HSSFCell.CELL_TYPE_NUMERIC == next2.getCellType()) {
System.out.println(next2.getNumericCellValue() + "");
} else if (HSSFCell.CELL_TYPE_STRING == next2.getCellType()) {
System.out.println(next2.getStringCellValue());
} else if (HSSFCell.CELL_TYPE_BOOLEAN == next2.getCellType()) {
System.out.println(next2.getBooleanCellValue() + "");
} else if (HSSFCell.CELL_TYPE_BLANK == next2.getCellType()) {
System.out.println("BLANK ");
} else {
System.out.println("unknown cell type");
}
}
}
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}

you have not given the file extension in your code for your file "D:/one" is it an xls ot xlsx or csv .

This line:
InputStream input = new BufferedInputStream(new FileInputStream("D:/one"));
...should be something like:
InputStream input = new BufferedInputStream(new FileInputStream("D:/folder/filename.xls"));
...depending on your file location and extension of course.
As an aside, I highly recommend JExcelAPI and this tutorial by Lars Vogel.

How continue in the program after finding all the .xls files in a specific directory?

I am making a program where I would read data from multiple excel files and store the data in tables. I have managed to make this program and works fine when the user give the full path of the file. What I am trying to do now is the user would give the directory of where all the excel files are, automatically find all the .xls files and do the rest program for each of them (read the data, create the table and insert the data in it).
The code for the user to give the path and print all the .xls files is:
String dirpath = "";
Scanner scanner1 = new Scanner(System.in);
while (true) {
System.out.println("Please give the directory:");
dirpath = scanner1.nextLine();
File fl = new File(dirpath);
if (fl.canRead()) break;
System.out.println("Error:Directory does not exists");
}
try{
String files;
File folder = new File(dirpath);
File[] listOfFiles = folder.listFiles();
for (int i = 0; i < listOfFiles.length; i++) {
if (listOfFiles[i].isFile()) {
files = listOfFiles[i].getName();
if (files.endsWith(".xls") || files.endsWith(".XLS")) {
System.out.println(files);
}
}
}
}catch (Exception e){
System.out.println();
}
How I would get each one of these file to continue in the rest program?
What I am doing next is described in the code below:
List sheetData = new ArrayList();
FileInputStream fis = null;
try {
fis = new FileInputStream(strfullPath);
HSSFWorkbook workbook = new HSSFWorkbook(fis);
HSSFSheet sheet = workbook.getSheetAt(0);
Iterator rows = sheet.rowIterator();
while (rows.hasNext()) {
HSSFRow row = (HSSFRow) rows.next();
Iterator cells = row.cellIterator();
List data = new ArrayList();
while (cells.hasNext()) {
HSSFCell cell = (HSSFCell) cells.next();
data.add(cell);
}
sheetData.add(data);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
if (fis != null) {
fis.close();
}
}
After this code I am creating the table and after that I am filling it with the data.

Wrap the code into a method and instead of
System.out.println(files);
call this method and pass listOfFiles[i].

You can add the the XLS files to a list of files then iterate it for the xls file processing code.
//Delcare
List<File> xlsFiles = new ArrayList<File>();
//Add the below code in the if loop
xlsFiles.add(listOfFiles[i])
for(File xlsFile : xlsFiles){
List sheetData = new ArrayList();
FileInputStream fis = null;
try {
fis = new FileInputStream(xlsFile);
}
Regards,
Dinesh.R

You can read data from excel file and it will usable for further execution process.
Please find following example for the same.
private void getSpecificExcelWSheetDataList(String absolutePath) {
LogLoader.serverLog.trace("In getSpecificExcelWSheetDataList() [File Location : " + absolutePath + "]");
try {
File file = new File(absolutePath);
if (!file.isAbsolute()) {
LogLoader.serverLog.error("Source file does not exist");
} else if (!file.canRead()) {
LogLoader.serverLog.error("Source file doesn't have permission for read operation");
} else {
FileInputStream ExcelFile = new FileInputStream(absolutePath);
excelWBook = new XSSFWorkbook(ExcelFile);
Iterator<Sheet> sheetIterator = excelWBook.sheetIterator();
while (sheetIterator.hasNext()) {
XSSFSheet excelWSheet = (XSSFSheet) sheetIterator.next();
Iterator<Row> rowIterator = excelWSheet.rowIterator();
while (rowIterator.hasNext()) {
Row row = rowIterator.next();
if (row.getRowNum() != 0) {
this.serviceProcessData = new ServiceProcessData();
Iterator<Cell> cellIterator = row.cellIterator();
while (cellIterator.hasNext()) {
Cell cell = cellIterator.next();
int columnIndex = cell.getColumnIndex();
String cellValue = dataFormatter.formatCellValue(cell).trim();
System.out.println("Row Number :"+row.getRowNum()
+"Column Number :"+columnIndex
+"Value : "+cellValue);
}
}
}
}
}
} catch (Exception e) {
LogLoader.errorLog.error(e.getMessage(), e);
LogLoader.serverLog.error(e.getMessage());
}
LogLoader.serverLog.trace("Out getSpecificExcelWSheetDataList()");
}

How To handle Null Row using Apache POI?

I am using Apache POI to read xlsx file, it works well. I have question to you when row is found null, how I'm able to handle it? My file contain 500 row, but it show 105667 row, rest of row found null.
used code:
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.text.SimpleDateFormat;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.apache.poi.openxml4j.exceptions.InvalidFormatException;
import org.apache.poi.ss.usermodel.Cell;
import org.apache.poi.ss.usermodel.DateUtil;
import org.apache.poi.ss.usermodel.Row;
import org.apache.poi.ss.usermodel.Sheet;
import org.apache.poi.ss.usermodel.Workbook;
import org.apache.poi.ss.usermodel.WorkbookFactory;
import org.apache.poi.xssf.usermodel.XSSFRow;
import org.apache.poi.xssf.usermodel.XSSFSheet;
import org.apache.poi.xssf.usermodel.XSSFWorkbook;
/**
*
* #author SAMEEK
*/
public class readXLSXFile {
public int getNumberOfColumn(String fileName, int sheetIndex) throws FileNotFoundException, IOException {
File inputFile = null;
FileInputStream fis = null;
XSSFWorkbook workbook = null;
XSSFSheet sheet = null;
XSSFRow row = null;
int lastRowNum = 0;
int lastCellNum = 0;
// Open the workbook
inputFile = new File(fileName);
fis = new FileInputStream(inputFile);
workbook = new XSSFWorkbook(fis);
sheet = workbook.getSheetAt(sheetIndex);
lastRowNum = sheet.getLastRowNum();
for (int i = 0; i < lastRowNum; i++) {
row = sheet.getRow(i);
if (row != null) {
if (row.getLastCellNum() > lastCellNum) {
lastCellNum = row.getLastCellNum();
}
}
}
return lastCellNum;
}
public int getNumberOfRow(String fileName, int sheetIndex) throws FileNotFoundException, IOException {
File inputFile = null;
FileInputStream fis = null;
XSSFWorkbook workbook = null;
XSSFSheet sheet = null;
int lastRowNum = 0;
// Open the workbook
inputFile = new File(fileName);
fis = new FileInputStream(inputFile);
workbook = new XSSFWorkbook(fis);
sheet = workbook.getSheetAt(sheetIndex);
lastRowNum = sheet.getLastRowNum();
return lastRowNum;
}
public String[] getSheetName(String fileName) throws FileNotFoundException, IOException {
int totalsheet = 0;
int i = 0;
String[] sheetName = null;
File inputFile = null;
FileInputStream fis = null;
XSSFWorkbook workbook = null;
// Open the workbook
inputFile = new File(fileName);
fis = new FileInputStream(inputFile);
workbook = new XSSFWorkbook(fis);
totalsheet = workbook.getNumberOfSheets();
sheetName = new String[totalsheet];
while (i < totalsheet) {
sheetName[i] = workbook.getSheetName(i);
i++;
}
return sheetName;
}
public int getNumberOfSheet(String fileName) throws FileNotFoundException, IOException {
int totalsheet = 0;
File inputFile = null;
FileInputStream fis = null;
XSSFWorkbook workbook = null;
XSSFSheet sheet = null;
int lastRowNum = 0;
// Open the workbook
inputFile = new File(fileName);
fis = new FileInputStream(inputFile);
workbook = new XSSFWorkbook(fis);
totalsheet = workbook.getNumberOfSheets();
return totalsheet;
}
public String[][] getSheetData(String fileName, int sheetIndex) throws FileNotFoundException, IOException, InvalidFormatException {
String[][] data = null;
int i = 0;
int j = 0;Cell cell=null;
long emptyrowcount = 0;
InputStream inputStream = new FileInputStream(
fileName);
// Create a workbook object.
Workbook wb = WorkbookFactory.create(inputStream);
wb.setMissingCellPolicy(Row.CREATE_NULL_AS_BLANK);
Sheet sheet = wb.getSheetAt(sheetIndex);
// Iterate over all the row and cells
int noOfColumns = getNumberOfColumn(fileName, sheetIndex);
System.out.println("noOfColumns::" + noOfColumns);
int noOfRows = getNumberOfRow(fileName, sheetIndex) + 1;
System.out.println("noOfRows::" + noOfRows);
data = new String[noOfRows][noOfColumns];
for (int k = 0; k < noOfRows; k++) {
Row row = sheet.getRow(k);
if (row == null) {
} else {
j = 0;
for (int l = 0; l < noOfColumns; l++) {
// Cell cell = cit.next();
cell = row.getCell(j);
if (cell.getCellType() == cell.CELL_TYPE_BLANK) {
cell = row.getCell(j, Row.CREATE_NULL_AS_BLANK);
}
data[i][j] = getCellValueAsString(cell);
j++;
}
i++;
}
}
return data;
}
/**
* This method for the type of data in the cell, extracts the data and
* returns it as a string.
*/
public static String getCellValueAsString(Cell cell) {
String strCellValue = null;
if (cell != null) {
switch (cell.getCellType()) {
case Cell.CELL_TYPE_STRING:
strCellValue = cell.toString();
break;
case Cell.CELL_TYPE_NUMERIC:
if (DateUtil.isCellDateFormatted(cell)) {
SimpleDateFormat dateFormat = new SimpleDateFormat(
"dd/MM/yyyy");
strCellValue = dateFormat.format(cell.getDateCellValue());
} else {
Double value = cell.getNumericCellValue();
Long longValue = value.longValue();
strCellValue = new String(longValue.toString());
}
break;
case Cell.CELL_TYPE_BOOLEAN:
strCellValue = new String(new Boolean(
cell.getBooleanCellValue()).toString());
break;
case Cell.CELL_TYPE_BLANK:
strCellValue = "";
break;
}
}
return strCellValue;
}
public static void main(String s[]) {
try {
readXLSXFile readXLSxFile = new readXLSXFile();
String[][] sheetData = readXLSxFile.getSheetData("F:/work.xlsx", 0);
int columnLength = 0;
columnLength = readXLSxFile.getNumberOfColumn("F:/work.xlsx", 0);
int rowLength = 0;
rowLength = readXLSxFile.getNumberOfRow("F:/work.xlsx", 0);
int h = 0;
int j = 0;
while (j < rowLength) {
h = 0;
while (h < columnLength) {
System.out.print("\t " + sheetData[j][h]);
h++;
}
System.out.println("");
j++;
}
} catch (InvalidFormatException ex) {
Logger.getLogger(readXLSFile.class.getName()).log(Level.SEVERE, null, ex);
} catch (FileNotFoundException ex) {
Logger.getLogger(readXLSFile.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException ex) {
Logger.getLogger(readXLSFile.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
Please help me how to handle null row in excel sheet?

If you fetch a row, and get back null, then that means there is no data stored in the file for that row - it's completely blank.
POI by default gives you what's in the file. With Cells, you can set a MissingCellPolicy to control how missing and blank cells are handled. There's some examples of using this in the Apache POI docs. With rows, they're either there or not, so you need to check for nulls when fetching a row.

In case your .xlsx file contains any of the formatting for the blank cells, the poi reading is not treating it as null, however if you want to print it's value, it will give NullPointerException. To understand it I have created a sheet and mark the first columns boundary with to "All Border" for 10 rows, but not given any value to it. now applying following piece of code is showing output sheet.lastRowNum() as 10, while the RowCountWithNullValue is 990, and RowCountWithoutNullValue is 10. However the sheet is completely blank. If you uncomment the print statement, it will show NullPointerException.
public class Rough {
public static void main(String args[]) throws IOException{
public static void main(String args[]) throws IOException{
FileInputStream fin = new FileInputStream(AddressOfxlsxFile);
XSSFWorkbook wb = new XSSFWorkbook(fin);
XSSFSheet sheet = wb.getSheetAt(1);
int RowCountWithNullValue=0, RowCountWithoutNullValue=0;
for (int i=0;i<1000;i++){
if (sheet.getRow(i)==null)
RowCountWithNullValue++;
else{
RowCountWithoutNullValue++;
// System.out.println(sheet.getRow(0).getCell(0));
}
}
System.out.println(sheet.getLastRowNum());
System.out.println(RowCountWithNullValue+","+RowCountWithoutNullValue);
}
}
I am not sure if the same is happening on your end or not, but if you are saying your file contain 500 row, but it show 105667 row, this may be one of the cause.