How do I speed up the dataset reading process in java - java

I am working on a java project and I have to read the dataset from excel sheets to use them later in the project, so I have a separated class for the reading methods, its working well but it takes a long time (maybe 10 or 11 mins)
public class readExcel {
String[] excelSheets = {"f1.xls","f2.xls","f3.xls","f4.xls","f5.xls","f6.xls","f7.xls","f8.xls","f9.xls"};
//Reading All The Excel Sheets
double[][][][] arraysSigmaMatrices=new double[9][30][13][13];
double[][][][] arrayDeterminantSigmaMatrices=new double[9][30][1][1];
double[][][][] arrayInverseSigmaMatrices=new double[9][30][13][13];
double[][][][] arraysSigmaDiagonalMatrices=new double[9][30][1][13];
double[][][] arraysMuMatrices=new double[9][30][13];
double[][][] arraysComponentProportionalMatrices=new double[9][1][30];
public void readExcelsheets() throws FileNotFoundException, IOException {
System.out.println("Wait to Read The Files...");
arraysSigmaMatrices();
arrayDeterminantSigmaMatrices();
arrayinverseSigmaMatrices();
arraysSigmaDiagonalMatrices();
arraysMuMatrices();
arraysComponentProportionalMatrices();
System.out.println("Done");
}
public void arraysSigmaMatrices() throws FileNotFoundException, IOException {
for(int catrgory = 0; catrgory < excelSheets.length; catrgory++) {
for(int ngauss = 0; ngauss < 30; ngauss++){
for(int row= 0; row < 13; row++) {
for(int column= 0; column < 13; column++) {
HSSFWorkbook workbook=new HSSFWorkbook(new FileInputStream(excelSheets[catrgory]));//to be able to create everything in the excel sheet
String sheetname="Sigma"+(String.valueOf(ngauss+1));//adding the index to the sheet name
HSSFSheet sheet=workbook.getSheet(sheetname);//getting the sheet
HSSFRow rows=sheet.getRow(row);
arraysSigmaMatrices[catrgory][ngauss][row][column]=rows.getCell(column).getNumericCellValue();
}
}
}
}
}
readExcel is the separated class for getting the data once the program runs and save them in the variable arrays to use them later,"arraysSigmaMatrices()"is one of the methods to get the data.
my question now is there anyway to make the process much faster?
and my second question,what is the speed of running threads in java related to?
and of course if you see something in the code can be done in a better way feel free to let me know
thanks

You should change your method like this.
public void arraysSigmaMatrices() throws FileNotFoundException, IOException {
for(int catrgory = 0; catrgory < excelSheets.length; catrgory++) {
try (FileInputStream input = new FileInputStream(excelSheets[catrgory])) {
HSSFWorkbook workbook=new HSSFWorkbook(input);//to be able to create everything in the excel sheet
}
for(int ngauss = 0; ngauss < 30; ngauss++){
String sheetname="Sigma"+(String.valueOf(ngauss+1));//adding the index to the sheet name
HSSFSheet sheet=workbook.getSheet(sheetname);//getting the sheet
for(int row= 0; row < 13; row++) {
HSSFRow rows=sheet.getRow(row);
for(int column= 0; column < 13; column++) {
arraysSigmaMatrices[catrgory][ngauss][row][column]=rows.getCell(column).getNumericCellValue();
}
}
}
}
}
This will reduce the number of file reads, and also save resources.

Related

How to fetch single row from excel sheet and pass into testng data provider using java

I have a scenario to fetch entire 3rd row data from excel sheet and pass to data provider. I can fetch all the row available in excel sheet (Please find the code below). But, I want to fetch nth row of data and pass to data provider annotation. Thanks in advance
Utility class:
public static Object[][] getTestData(String sheetName) {
Object[][] data = null;
DataFormatter fmt = new DataFormatter();
try (Workbook workbook = new XSSFWorkbook(
new FileInputStream("path"))) {
Sheet sheet = workbook.getSheet(sheetName);
data = new Object[sheet.getLastRowNum()][sheet.getRow(0).getLastCellNum()];
for (int i = 0; i < sheet.getLastRowNum(); i++) {
for (int k = 0; k < sheet.getRow(0).getLastCellNum(); k++) {
data[i][k] = fmt.formatCellValue(sheet.getRow(i + 1).getCell(k));
}
}
} catch (IOException e) {
e.printStackTrace();
}
return data;
}
I am preparing utlity class for fetching single row of data and passing into data provider annotation in testng
Just return the 3rd row after reading the data
for (int i = 0; i < sheet.getLastRowNum(); i++) {
for (int k = 0; k < sheet.getRow(0).getLastCellNum(); k++) {
data[i][k] = fmt.formatCellValue(sheet.getRow(i + 1).getCell(k));
}
}
if (i>2)
for k...
return data[2,k]
or break your loop after the 3rd row has been read if you don't need to fetch all rows.

Apache POI recording only 1 row in the spreadsheet

I am using selenium and java to scrape data on a specific site, but with the code below I can write only 1 data in the spreadsheet, it is not recording all the data in sequence as it should be.
I can't structure the loop correctly.
public void gravarDados() throws IOException {
int i = 0;
File localPlanilha = new File("tools/resultado_da_pesquisa.xlsx");
FileInputStream planilhaExistente = new FileInputStream(localPlanilha);
XSSFWorkbook plan = new XSSFWorkbook(planilhaExistente);
XSSFSheet sheetExistente = plan.getSheetAt(0);
for (int i = 0; i < inicio; i++) {
// Writing data
sheetExistente.getRow(2).createCell(5).setCellValue(TitulosHoteis.get(i).getText());
FileOutputStream fechandoArquivo = new FileOutputStream(localPlanilha);
plan.write(fechandoArquivo);
}
}
Currently you are getting only the 0th element.
You need to iterate the below with a for loop
TitulosHoteis.get(i).getText());
to write the result to rows and columns.
Please modify it as below
for (int i = 0; i < inicio; i++) {
// Writing data
sheetExistente.getRow(i+2).createCell(5).setCellValue(TitulosHoteis.get(i).getText());
}
FileOutputStream fechandoArquivo = new FileOutputStream(localPlanilha);
plan.write(fechandoArquivo);
As mentioned before you're not iterating over the rows as the row number stays the same in your code, however there is also another problem with your code. You need to check if a row exists and if it doesn't create it before you can set a cell value of that row.
It should look something like this:
for (int i = 0; i < inicio; i++) {
Row row = sheetExistente.getRow(2+i);
if (row == null) sheetExistente.createRow(2+i);
sheetExistente.getRow(2 + i).createCell(5).setCellValue(TitulosHoteis.get(i).getText());
}

Selenium/Java - Writing Data to Excel

I am new to Selenium and I am trying to write a code to write data into excel. This code is working. However, it just write on the second row. When I change the value of "String FieldName1 to 3" it doesn't write on the next row. I change the value of header = spreadsheet.createRow(0); to (1) it did write to next row, but it is a pain to do it every time I run my test. What I need is to write the data to the next ROW everytime I run it and change the value of "String FieldName1 to 3". THANKS IN ADVANCE!
Selenium > Java > Maven
public class testBed2 {
#Test
public void writeExcel() throws IOException{
FileInputStream fis = new FileInputStream("C:\\Users\\ExportExcel.xlsx");
XSSFWorkbook workbook = new XSSFWorkbook();
XSSFSheet spreadsheet = workbook.createSheet( "TestData");
XSSFRow header;
header = spreadsheet.createRow(0);
header.createCell(0).setCellValue("FieldName1");
header.createCell(1).setCellValue("FieldName2");
header.createCell(2).setCellValue("FieldName3");
int rowNumber = 1;
Row row = spreadsheet.getRow(0);
//Column Count
int colCount = row.getLastCellNum();
for (int j = 0; j < colCount; j++) {
System.out.println("Col Count : " + j);
//Row Count
int rowCount = spreadsheet.getLastRowNum() + 1;
for (int i = 0; i < rowCount; i++) {
XSSFRow currentRow = spreadsheet.createRow(rowNumber);
System.out.println("Row Count : " + i);
String FieldName1 = "NAME1";
String FieldName2 = "NAME2";
String FieldName3 = "NAME3";
currentRow.createCell(0).setCellValue(FieldName1);
currentRow.createCell(1).setCellValue(FieldName2);
currentRow.createCell(2).setCellValue(FieldName3);
FileOutputStream fos = new FileOutputStream("C:\\Users\\ExportExcel.xlsx");
workbook.write(fos);
fos.close();
}
}
}
}
I think the reason why your code write just second row is, your close code is in the For loop. If you write close code at the end of {} you will get right answer. :)

how can we create excel sheets where each sheet contains data from a given number of text files in the form of rows and columns

in my program first i have written a function to count sheets to be created according to the number of text files data to be inserted in the excel file then a function which counts rows in each text file and finally a function
that counts number of columns by using StringTokenizer method countTokens
then i have passed these values in the below method....but the code is not working properly as the number of sheets it creates is less than the number of text files and data in the sheets is not inserted appropriately
void store(int sheetnum, int rows, String filename, int columns) {
String datafile = filename;
FileReader fr = new FileReader(datafile);
BufferedReader in = new BufferedReader(fr);
String data = in.readLine();
for (int i = 0; i < sheetnum; i++) {
while (data != null) {
HSSFSheet sh = HSSFWorkbook.createSheet(i);
for (int j = 0; j < rows; j++) {
HSSFRow row = sh.createRow(j);
for (int k = 0; k < columns; k++) {
// createcell
// setcellvalue
}
data = in.readLine();
}
}
}
}

How to remove the index out of bound error in my program?

I have tried various ways of removing the index out of bound error by changing the upper limits of the array.But the error persists. Where am I going wrong?
Screenshot of my excel sheet
My program reads values(all rows) in the first column of excel sheet and finds the maximum value. Then based on the maximum value,criteria are formulated and the values are classified as Low,Medium,High and written back into a new excel sheet.
import java.io.FileInputStream;
import java.io.IOException;
import jxl.Cell;
import jxl.Sheet;
import jxl.Workbook;
import jxl.read.biff.BiffException;
import java.io.*;
import java.util.*;
import jxl.write.WritableSheet;
import jxl.write.WritableWorkbook;
import jxl.write.Label;
import jxl.write.WriteException;
public class Bus3{
List<String> numbusarray = new ArrayList<String>();
List<String> numcommutersarray = new ArrayList<String>();
List<String> numcommercialarray = new ArrayList<String>();
static WritableWorkbook workbook;
static WritableSheet wSheet;
public void readExcel() throws BiffException, IOException, WriteException//method to read contents form excel
{
String FilePath = "Bus1.xls";
Scanner sc = new Scanner(System.in);
int max=0;
FileInputStream fs = new FileInputStream(FilePath);
Workbook wb = Workbook.getWorkbook(fs);
Sheet sh = wb.getSheet("Bus1");// TO get the access to the sheet
int totalNoOfRows = sh.getRows();// To get the number of rows present in sheet
int totalNoOfCols = sh.getColumns();// To get the number of columns present in sheet
System.out.println(totalNoOfRows);
//adding excel contents from every column to arraylist
for (int row = 1; row <totalNoOfRows; row++)
{
numbusarray.add(sh.getCell(2, row).getContents());
}
for (int row = 1; row <totalNoOfRows; row++)
{
numcommutersarray.add(sh.getCell(3, row).getContents());
}
for (int row = 1; row <totalNoOfRows; row++)
{
numcommercialarray.add(sh.getCell(4, row).getContents());
}
//to find maximum of numbusarray
max=Integer.parseInt(numbusarray.get(0));
for (int row = 1; row < totalNoOfRows-1; row++)
{
if(!(numbusarray.get(row)).isEmpty())
{
int intNumber=Integer.parseInt(numbusarray.get(row));
if(intNumber>max)
{
max=intNumber;
//System.out.println(max);
}
}
}
System.out.println(max);
WritableWorkbook workbook = Workbook.createWorkbook(new File("sampletestfile.xls"));
WritableSheet wSheet = workbook.getSheet(0);
int increment=max/3;
int a=increment;
int b=a+increment;
int c=b+increment;
for (int row = 0; row < totalNoOfRows-1; row++)
{
if(!(numbusarray.get(row)).isEmpty())
{
int compare=Integer.parseInt(numbusarray.get(row));
if(compare<=a)
{Label label= new Label(0, row, "Low");//column,row,strngdata
wSheet.addCell(label);}
else if((compare>a)&&(compare<=b))
{Label label= new Label(0, row, "Medium");//column,row,strngdata
wSheet.addCell(label);}
else
{Label label= new Label(0, row, "High");//column,row,strngdata
wSheet.addCell(label);}
}
}
/*Iterator itr=numbusarray.iterator(); //to print arraylist demo
while(itr.hasNext()){
System.out.println(itr.next());
}*/
}//end of method to read contents from excel
//to close file
public static void closeFile()
{
try {
// Closing the writable work book
workbook.write();
workbook.close();
// Closing the original work book
} catch (Exception e)
{
e.printStackTrace();
}
}
public static void main(String args[]) throws BiffException, IOException, WriteException //main class
{
Bus3 DT = new Bus3();
DT.readExcel();
Bus3.closeFile();
}//end of main class
}
It is because your sh Sheet.class object doesn't have cells with column = 4.
This should fix it:
for (int row = 1; row < totalNoOfRows; row++) {
numbusarray.add(sh.getCell(1, row).getContents());
}
for (int row = 1; row < totalNoOfRows; row++) {
numcommutersarray.add(sh.getCell(2, row).getContents());
}
for (int row = 1; row < totalNoOfRows; row++) {
numcommercialarray.add(sh.getCell(3, row).getContents());
}
LAST EDIT:
for (int row = 1; row < totalNoOfRows; row++) {
numbusarray.add(sh.getCell(1, row).getContents());
}
for (int row = 1; row < totalNoOfRows; row++) {
numcommutersarray.add(sh.getCell(2, row).getContents());
}
for (int row = 1; row < totalNoOfRows; row++) {
numcommercialarray.add(sh.getCell(3, row).getContents());
}
// to find maximum of numbusarray
max = 0;
for (int row = 1; row < totalNoOfRows; row++) {
if (!(numbusarray.get(row - 1)).isEmpty()) {
int intNumber = Integer.parseInt(numbusarray.get(row - 1));
if (intNumber > max) {
max = intNumber;
System.out.println("max: " + max);
}
}
}
System.out.println(max);
workbook = Workbook.createWorkbook(new File("sampletestfile.xls"));
WritableSheet wSheet = workbook.createSheet("name", 0);
It does not look like a very complex problem.
Index out of bounds means that you are trying to access a position in the array that does not exists.
Watch your numbusarray variable, probably row is being set to an invalid index.
Good afternoon, for me what worked was to create a new xls file, and copy the data from the old to the new one. and the errors stopped.

Categories