We Keep Coding

Java Out of memory error for reading from and writing to an xlsx

I need to read several xlsx files looking for data specific to an employee and simultaneously create another xlsx file (if I find data in any of the file)with file name as employee Id appended to the name I found the data in. Eg. there is an employee with emp id 1 and there are severaal xlsx files such as A,B, C... so on; I need to look for data relating to emp id 1 in each file and for the files I get a hit I need to create a file named 1_A.xlsx.
Now although I have built the logic and am using Apache POI APIs for reading and writing, my code is throwing Out Of Memory error after creating just the first file with the data. And is unable to read the rest of the files.
I have tried using SXSSF instead of XSSF but same OOM happens.
Increasing the heap space is not an option for me.
Please help here...Thanks in advance.
Here is a piece of code :
//Reader:
Row row = null;
List<Row> listOfRecords = new ArrayList<Row>();
try {
FileInputStream fis = new FileInputStream(metaDataFile);
new InputStreamReader(fis, "ISO-8859-1");
XSSFWorkbook wb = new XSSFWorkbook(fis);
XSSFSheet sheet = wb.getSheetAt(0);
Iterator<Row> rowIterator = sheet.iterator();
while (rowIterator.hasNext()) {
row = rowIterator.next();
if (!isEmptyRow(row)) {
listOfRecords.add(row);
}
}
wb.close();
fis.close();
//Writer
LOGGER.info("in createWorkbook " );
Workbook empWorkbook = new SXSSFWorkbook(200);
Sheet empSheet = empWorkbook.createSheet("Itype Sheet For Emp_"
+ personnelNumber);
int rowNum = listOfRecords.size();
System.out.println("Creating excel");
Cell c = null;
for (int i = 0; i < rowNum; i++) {
Row record = listOfRecords.get(i);
Row empRow = empSheet.createRow(i++);
if (!isEmptyRow(record)) {
int colNum = record.getLastCellNum() + 1;
for (int j = 0; j < colNum; j++) {
Cell newCell = empRow.createCell(j);
System.out.println("cellVal:"
+ String.valueOf(record.getCell(j)));
newCell.setCellValue(String.valueOf(record.getCell(j)));
}
}
}
The writer method is called from within the reader.

Reading of multiple xlsx files is indeed tricky business butI finally solved it.
I had to break down my code several folds to realise that the OOM error was due to the fact that after reading 3 files no more memory was left to process the rest of the files.
xlsx files are compressed xml files. So when we try to read them using XSSF or SXSSF APIs it loads the entire DOM to the memory thereafter choking it.
I found an excellent solution here :
[https://github.com/monitorjbl/excel-streaming-reader]
Hope this will help others who come here facing the same issue.

Apache Poi, how to get the linked workbook?

I try to analyze excel files with links to other files and I like to know the file name and path. For that I'm using apache poi 3.14.
I figured it out for Ref3DPtg objects but for Ref3DPxg I don't know how to do it. I only get access to the cell address and the sheet name.
Does anyone know how to do it?
Code:
...
if(ptg instanceof Ref3DPxg){
cellAddress = ptg.format2DRefAsString();
sheetName = ptg.getSheetName();
workbookName = ???;
} else if(ptg instanceof Ref3DPtg) {
// by Ref3DPtg is no problem
}

Because of the way that the XLSX file format stores external references, which isn't actually =[Other.xlsx]Sheet1!A1 but actually =[23]Sheet1!A1, it's a two step process. First, get the external workbook number from the Pxg. Next, from Workbook get the ExternalLinks table for that workbook number, noting the off-by-one. (External Workbook 0 is actually the current workbook, so External Workbook 1 corresponds to External Link 0). Finally, fetch the filename for that link
So, your code should be something like:
if(ptg instanceof Ref3DPxg){
Ref3DPxg pxg = (Ref3DPxg)ptg;
int extWB = pxg.getExternalWorkbookNumber();
int extLink = extWB-1;
ExternalLinksTable links = wb.getExternalLinksTable().get(extLink);
String filename = links.getLinkedFileName();
}

POI - Excel remains held by some process and cannot be edited

I am using POI to read,edit and write excel files.
My process flow is like write an excel, read it and again write it.
Then if I try to edit the file using normal desktop excel application while my Java app is still running, the excel cannot be saved, it says some process is holding the excel,
I am properly closing all file handles.
Please help and tell me how to fix this issue.
SXSSFWorkbook wb = new SXSSFWorkbook(WINDOW_SIZE);
Sheet sheet = getCurrentSheet();//stores the current sheet in a instance variable
for (int rowNum = 0; rowNum < data.size(); rowNum++) {
if (rowNum > RECORDS_PER_SHEET) {
if (rowNum % (RECORDS_PER_SHEET * numberOfSheets) == 1) {
numberOfSheets++;
setCurrentSheet(wb.getXSSFWorkbook().createSheet());
sheet = getCurrentSheet();
}
}
final Row row = sheet.createRow(effectiveRowCnt);
for (int columnCount = 0; columnCount < data.get(rowNum).size(); columnCount++) {
final Object value = data.get(rowNum).get(columnCount);
final Cell cell = row.createCell(columnCount);
//This method creates the row and cell at the given loc and adds value
createContent(value, cell, effectiveRowCnt, columnCount, false, false);
}
}
public void closeFile(boolean toOpen) {
FileOutputStream out = null;
try {
out = new FileOutputStream(getFileName());
wb.write(out);
}
finally {
try {
if (out != null) {
out.close();
out = null;
if(toOpen){
// Open the file for user with default program
final Desktop dt = Desktop.getDesktop();
dt.open(new File(getFileName()));
}
}
}
}
}

The code looks correct. After out.close();, there shouldn't be any locks left.
Things that could still happen:
You have another Java process (for example hanging in a debugger). Your new process tries to write the file, fails (because of process 1) and in the finally, it tries to open Excel which sees the same problem. Make sure you log all exceptions that happen in wb.write(out);
Note: The code above looks correct in this respect, since it only starts Excel when out != null and that should only be the case when Java could open the file.
Maybe the file wasn't written completely (i.e. there was an exception during write()). Excel tries to open the corrupt file and gives you the wrong error message.
Use a tool like Process Explorer to find out which process keeps a lock on a file.

I tried all the options. After looking thoroughly, it seems the problem is the Event User model.
I am using the below code for reading the data:
final OPCPackage pkg = OPCPackage.open(getFileName());
final XSSFReader r = new XSSFReader(pkg);
final SharedStringsTable sst = r.getSharedStringsTable();
final XMLReader parser = fetchSheetParser(sst);
final Iterator<InputStream> sheets = r.getSheetsData();
while (sheets.hasNext()) {
final InputStream sheet = sheets.next();
final InputSource sheetSource = new InputSource(sheet);
parser.parse(sheetSource);
sheet.close();
}
I assume the excel is for some reason held by this process for some time. If I remove this code and use the below code:
final File file = new File(getFileName());
final FileInputStream fis = new FileInputStream(file);
final XSSFWorkbook xWb = new XSSFWorkbook(fis);
The process works fine an the excel does not remain locked.

I figured it out actually.
A very simple line was required but some some reason it was not explained in the New Halloween Document (http://poi.apache.org/spreadsheet/how-to.html#sxssf)
I checked the Busy Developer's Guide and got the solution.
I needed to add
pkg.close(); //To close the OPCPackage
This added my code works fine with any number of reads and writes on the same excel file.

How to update cell reference values using Apache POI

I am using Apache POI to create new XSSFWorkbook from an existing one, after updating some values. Suppose I have two worksheets (Lets say: worksheet A & B) in my existing workbook. Worksheet B has some cell reference from Worksheet A. IF i modify those cell values of worksheet A and save them as a new workbook, corresponding cell values of worksheet B should be updated too. But it doesn't. How can i update them programmatically? . Thank you.
My code:
public void createExcel(ClientData cd) throws FileNotFoundException, IOException, InvalidFormatException{
// create a new file
double[] dataHolder1= cd.getFinalData1(), param1 = cd.getRecord1Param();
double[] dataHolder2 = cd.getFinalData2(), param2 = cd.getRecord2Param();
double[] ncv = cd.getNcv();
String[] pname = cd.getName();
Workbook workbook = new XSSFWorkbook(OPCPackage.open(new FileInputStream("template/mncv.xlsx"))); // or sample.xls
//CreationHelper createHelper = workbook.getCreationHelper();
Sheet s=workbook.getSheetAt(0);
int counter = dataHolder1.length + param1.length +param2.length+dataHolder2.length;//+ param1.length + param2.length;
// r = s.getRow(0);
// r.getCell(0).setCellValue("Param1");
// r.getCell(1).setCellValue("Record1");
// r.getCell(2).setCellValue("Param2");
// r.getCell(3).setCellValue("Record2");
int i;
for(i=0;i<counter;i++){
if(i<param1.length){
for(int j=0;j<param1.length;j++){
r = s.getRow(i);
r.getCell(0).setCellValue(param1[j]);
i++;
}
}else if(i<dataHolder1.length+param1.length && i>=param1.length){
for(int j=0;j<dataHolder1.length;j++){
r = s.getRow(i);
r.getCell(0).setCellValue(dataHolder1[j]);
i++;
}
}else if(i<dataHolder1.length+param1.length+param2.length && i>=dataHolder1.length+param1.length){
for(int j=0;j<param2.length;j++){
r = s.getRow(i);
r.getCell(0).setCellValue(param2[j]);
i++;
}
}else{
for(int j=0;j<dataHolder2.length;j++){
r = s.getRow(i);
r.getCell(0).setCellValue(dataHolder2[j]);
i++;
}
}
// if(i<=param1.length){
// r.getCell(0).setCellValue(param1[i-1]);
// r.getCell(2).setCellValue(param2[i-1]);
//
// }
// r.getCell(0).setCellValue(param1[i]);
//r.getCell(3).setCellValue(dataHolder2[i-1]);
i--;
}
for(int k=0;k<ncv.length;k++){
r = s.getRow(i);
r.getCell(0).setCellValue(ncv[k]);
i++;
}
s = workbook.getSheetAt(1);
s.getRow(2).getCell(5).setCellValue(pname[0]+" "+pname[1]+" "+pname[2]);
s.getRow(3).getCell(5).setCellValue(cd.getAge());
s.getRow(4).getCell(5).setCellValue(cd.getGender());
try (FileOutputStream out = new FileOutputStream("workbook.xlsx")) {
//WorkbookEvaluator we = new WorkbookEvaluator(workbook);
workbook.write(out);
out.close();
XSSFFormulaEvaluator.evaluateAllFormulaCells((XSSFWorkbook) workbook);
}catch(Exception e){
System.out.println(e);
}

The Excel file format caches the result of formula evaluation, to make opening the file quicker. This means that when you're done making changes to your file, you'll need to evaluate all of the formula cells to updated their cached value. (Otherwise, when you load the file in Excel, for almost all cases it'll still show the old value until you go into that cell)
Luckily, Apache POI provides code to do that, see the Formula Evaluation documentation for details. (You can choose to only recalculate certain formulas, if you know just those cells have changed, or do everything)

For any cell, say "B5", at runtime,
cell.getReference();
will give you cell reference (like in example... it will return you "B5")
cell.getReference().toString().charAt(0);
will give you the Column Reference (will give you "B" if the current cell is B5). Now
cell.getRowIndex();
OR
cell.getReference().toString().charAt(1);
will give you Row Index. Now you have the reference of the target cell. just replace these character with the references you have already created. This will update the cell references.

The following solution worked for me
wb.setForceFormulaRecalculation(true);
// replace "wb" with your HSSFWorkbook/XSSFWorkbook object

How to read and write values in excel using java?

Sample Excel --> Sorry I'm not allowed to attached a image..
TC No. | Title | Result
1 | State and Vin | Failed
2 | State and Reg Code | Passed
3 | Booking a Test Drive | Passed
public class sampleTest{
public static void main(String[] args) throws Exception {
int iTest = 2, iTest2 = 3;
if (iTest == iTest2){
//It will pass a value into the excel (e.g. "Passed")
}
else{
//It will pass a value into the excel (e.g. "Failed")
}
}
My program's goal is to generate a report by getting the Passed and Failed results from my tests. My main problem here is on how to read the results from the excel and place the value "Passed" or "Failed" under Result column.

Download the apache poi jar from here
Go through these examples which demonstrates how to read/write xls data from a java program
Sample help code:
public static void main(String[] args) throws IOException {
Workbook wb = new HSSFWorkbook();
Sheet sheet = wb.createSheet("sheet");
Row row = sheet.createRow((short) 0);
row.createCell(0).setCellValue(1.2);
row.createCell(1).setCellValue(wb.getCreationHelper().createRichTextString("This is a string"));
row.createCell(2).setCellValue(true);
FileOutputStream fileOut = new FileOutputStream("workbook.xls");
wb.write(fileOut);
fileOut.close();
}
This might help you to get started.
The whole flow is:
Create a workbook => The main xls file
Then create a sheet
Then create a row.
For each row create as many cells as you want and fill the cells with different values
Write the workbook like a file.
There can be multiple type of cells see this for more info.
To know how to read an excel file:
InputStream myxls = new FileInputStream("workbook.xls");
wb = new HSSFWorkbook(myxls);
sheet = wb.getSheetAt(0); // first sheet
row = sheet.getRow(0); // third row
HSSFCell cell = (HSSFCell) row.getCell((short)1); // fourth cell
if (cell.getCellType() == HSSFCell.CELL_TYPE_STRING) {
System.out.println("The Cell was a String with value \" " + cell.getStringCellValue()+" \" ");
} else if (cell.getCellType() == HSSFCell.CELL_TYPE_NUMERIC) {
System.out.println("The cell was a number " + cell.getNumericCellValue());
} else {
System.out.println("The cell was nothing we're interested in");
}
For more info see this

Via library that will be your interface to Excel document. One option is Apache POI. Excel example code can be found from here.
Other option is Java Excel API.