We Keep Coding

Java Out of memory error for reading from and writing to an xlsx

I need to read several xlsx files looking for data specific to an employee and simultaneously create another xlsx file (if I find data in any of the file)with file name as employee Id appended to the name I found the data in. Eg. there is an employee with emp id 1 and there are severaal xlsx files such as A,B, C... so on; I need to look for data relating to emp id 1 in each file and for the files I get a hit I need to create a file named 1_A.xlsx.
Now although I have built the logic and am using Apache POI APIs for reading and writing, my code is throwing Out Of Memory error after creating just the first file with the data. And is unable to read the rest of the files.
I have tried using SXSSF instead of XSSF but same OOM happens.
Increasing the heap space is not an option for me.
Please help here...Thanks in advance.
Here is a piece of code :
//Reader:
Row row = null;
List<Row> listOfRecords = new ArrayList<Row>();
try {
FileInputStream fis = new FileInputStream(metaDataFile);
new InputStreamReader(fis, "ISO-8859-1");
XSSFWorkbook wb = new XSSFWorkbook(fis);
XSSFSheet sheet = wb.getSheetAt(0);
Iterator<Row> rowIterator = sheet.iterator();
while (rowIterator.hasNext()) {
row = rowIterator.next();
if (!isEmptyRow(row)) {
listOfRecords.add(row);
}
}
wb.close();
fis.close();
//Writer
LOGGER.info("in createWorkbook " );
Workbook empWorkbook = new SXSSFWorkbook(200);
Sheet empSheet = empWorkbook.createSheet("Itype Sheet For Emp_"
+ personnelNumber);
int rowNum = listOfRecords.size();
System.out.println("Creating excel");
Cell c = null;
for (int i = 0; i < rowNum; i++) {
Row record = listOfRecords.get(i);
Row empRow = empSheet.createRow(i++);
if (!isEmptyRow(record)) {
int colNum = record.getLastCellNum() + 1;
for (int j = 0; j < colNum; j++) {
Cell newCell = empRow.createCell(j);
System.out.println("cellVal:"
+ String.valueOf(record.getCell(j)));
newCell.setCellValue(String.valueOf(record.getCell(j)));
}
}
}
The writer method is called from within the reader.

Reading of multiple xlsx files is indeed tricky business butI finally solved it.
I had to break down my code several folds to realise that the OOM error was due to the fact that after reading 3 files no more memory was left to process the rest of the files.
xlsx files are compressed xml files. So when we try to read them using XSSF or SXSSF APIs it loads the entire DOM to the memory thereafter choking it.
I found an excellent solution here :
[https://github.com/monitorjbl/excel-streaming-reader]
Hope this will help others who come here facing the same issue.

HWPF-POI:The table insert into doc with poi hwpf is not visible

I want to insert a table at a specific position with poi, the table is generated, but I find this table is not visible.
The generated table in doc is visible when previewing or editing this doc with macOS and its text tool, POI can read the table and content, too. I plan to upload 4 pictures to display the process, but I can only post 2 images, sorry for that.
#Test
public void exportDoc() throws Exception {
FileInputStream readFile = new FileInputStream(new File(readDoc));
FileOutputStream replaceFile = new FileOutputStream(new File(replaceDoc));
HWPFDocument document = new HWPFDocument(readFile);
Table table = WordUtil.insertNewTable(document,"${table}");
insertTableInDoc(table);
document.write(replaceFile);
readFile.close();
replaceFile.close();
}
private Table insertNewTable(HWPFDocument doc, String sourceValue) {
Range range = doc.getRange();
Table table = null;
for (int i = 0; i < range.numSections(); ++i) {
Section s = range.getSection(i);
for (int x = 0; x < s.numParagraphs(); x++) {
Paragraph p = s.getParagraph(x);
if (p.text().contains(sourceValue)) {
//remove target text
range.replaceText(sourceValue, "");
table = p.insertTableBefore((short) 3, 3);
return table;
}
}
}
return table;
}
private void insertTableInDoc(Table table) {
int count = 1;
for (int rowNum = 0; rowNum < table.numRows(); rowNum++) {
TableRow tableRow = table.getRow(rowNum);
for (int colNum = 0; colNum < tableRow.numCells(); colNum++) {
TableCell cell = tableRow.getCell(colNum);
Paragraph paragraph = cell.getParagraph(0);
CharacterRun characterRun = paragraph.getCharacterRun(0);
characterRun.insertBefore("number: " + count++);
}
}
}
PS:
I am sure this is not microsoft for mac 's problem, the generate table in doc at windows platform is not visible, too.
(First time to ask question, if anything wrong or my expression is not clear, please let me know and I will modify it without delay. Thanks)

With the current state of the HWPF project, you likely are out of luck when trying to insert content into a .doc file. Your best bet is to use a different format (docx).
I did not look at HWPF for the past year, so I may be wrong here regarding the current state of HWPF:
Some years ago I was developing a custom HWPF library for a client. The major goal for that custom library was the ability to modify .doc files and that Word can process the modified files correctly. Hence I know in how many levels modifying a .doc file can fail in the end. The public HWPF library is not able to handle many aspects of the .doc file format when it comes to modification (textboxes, two-byte character ranges, shape files, nested tables, ... to name a few).
To handle modification correctly, all the "features" of the specific .doc file must be supported by the library. So when there are shapes in the .doc file, HWPF must adjust the position tables of the shapes even when a simple text snippet is inserted and the shapes are not touched. If shapes are not handled, Word will crash when opening the output file.
So if you can, use docx or rtf. If it is an option, you might try one of the commercial libraries which are able to handle .doc files.

Writing 1 million records to Excel file with 300 columns

I am using Apache POI Streaming API- SXSSFWorkbook to write data to excel file.
But Excel file is getting corrupted for more than 100 000 records with 300 columns generally if the size is greater then 100Mb. Is there any way for writing huge data to excel file.
class Test1 {
public static void main(String[] args) throws FileNotFoundException, {
SXSSFWorkbook workbook = new SXSSFWorkbook(100);
workbook.setCompressTempFiles(true);
Sheet sheet = null;
Row row = null;
Cell cell = null;
sheet = workbook.createSheet("Demo1");
FileOutputStream outStream = new FileOutputStream("D:\\Test1.xlsx");
try {
for (int i = 0; i < 100000; i++) {
row = sheet.createRow(i);
for (int j = 0; j < 300; j++) {
cell = row.createCell(j);
cell.setCellValue(" row : "+i +" col: "+ j);
}
}
workbook.write(outStream);
} catch (Exception exception) {
exception.printStackTrace();
} finally {
workbook.dispose();
try {
outStream.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Edit 1:
What I found is, it is not the problem with Apache POI Streaming Api.It is generating the file with 1 Million records. But Excel is not loading that file. It is giving 'There isn't enough memory to complete this action' error.
I am using Excel 2013 32 bit version which can use only up till 2GB of memory. The excel file I created with 100k records and 300 columns has file size 108MB. When I try to open this file in Excel it is taking up whole lot of System memory. As soon as the memory consumption reaches 1.7 MB Excel is giving an error.
What is the minimum configuration to load 1million rows generated using Apache Streaming API? Any help would be appreciated.
Thanks.
Edit 2:
If I open Excel file generated using Apache Streaming Api in zip format(by renaming .xlsx to .zip),size of the xml file in xl->worksheets folder is around 2GB for 100k records and 300 columns. Is there any way to reduce the size of this xml file.

Never had tried generating more than about 100 to 120 columns myself. But a restriction at 255 columns max is not really surprising (had been this way with older Excel formats). Your observation that 100k rows with 200 columns is working fine, while 100k rows with 300 columns is failing, is a strong indicator of such a restriction.
Then you should be able to generate the 1 million (exactly 1048576) rows sheet with up to 255 columns.,For any extra rows and extra columns you will need to create extra sheets.
So, with your 300 columns target, you would generate sheet1 with the first 255 columns (or some less if there is reasonable logical grouping) and sheet2 with the other columns.
For more rows repeat the 2 sheets approach with a new pair of sheets until all rows have been generated.
BTW,
did you recognize that with using SXSSFWorkbook a rowAccessWindowSize of "1" is giving best performance?

ColdFusion 8 Reading large Excel files

I have been able to create my application so it can take Excel files with multiple sheets and put them into the database. However I'm not sure how I do this with large files 30mb+ I tried looking into bufferedReader but I'm not sure this is what I want since it reads a portion at a time and I am using Apache POI to read in multiple Excel sheets.
Currently if I send in a file to big I'm getting Java heap space errors, which is why I'm feeling the file is to large. During testing it worked fine, and then I used the 32mb file, and it failed.
Below is the code, I appreciate any help in advance.
also myfile is an argument
ci_model = createObject("component", "com.data_feeds.models.ci.model_item_setup");
FileIn = createObject("java","java.io.FileInputStream").init(javacast("string", myfile));
wb = createObject("java","org.apache.poi.xssf.usermodel.XSSFWorkbook").init(FileIn);
workbook = createObject("java","org.apache.poi.xssf.streaming.SXSSFWorkbook").init(wb);
sheetCount = workbook.getNumberOfSheets();
for(sheetIndex = 0; sheetIndex LT sheetCount; sheetIndex = sheetIndex + 1)
{
recordIndex = 1;
sheet = workbook.getSheetAt(sheetIndex);
rowHeader = sheet.getFirstRowNum();
rowCount = sheet.getLastRowNum();
for (rowIndex = 1; rowIndex LTE rowCount; rowIndex = rowIndex + 1)
{
row = sheet.getRow(rowIndex);
columnCount = row.getLastCellNum();
for(columnIndex = 0; columnIndex LT columnCount; columnIndex = columnIndex +1)
{
record[recordIndex][columnIndex+1] = row.getCell(columnIndex);
if((columnIndex+1)==4)
{
record[recordIndex][columnIndex+1] = ci_model.boolean_number(row.getCell(columnIndex).toString());
}
}
recordIndex = recordIndex + 1;
if(recordIndex == 500)
{
this.insert_record(record);
arrayClear(record);
request.help.debug.collect_garbage();
recordIndex = 1;
}
}
this.insert_record(record);
arrayClear(record);
}
FileIn.close();
I am using ColdFusion 8.

Your best bet is to import the data in a temporary table. Then import the data from the temporary table to the primary table using ColdFusion. This way you could use ColdFusion to simply import the data in chunks and not all at once.. To import into SQL Server all you need to do is use the import tool and for mySQL there are several ways you can. I personally just import using a csv file. I know this process is a bit less direct but it will put less strain on your ColdFusion process and more on your SQL server.
Another suggestion is to also make sure your queries are not timing out or your page. Adding ?requestTimeout=1000 after your URL could also stop your process from timing out. I understand that it's a java error but if your process is lasting too long it could be timing out.

FileOutputStream() not working?

I have this app. I'm working on. It allows users to view JTable instancess and at the same time have the option to save it in excel format or print it. It asks the user for the file name and location where to place the file then saves it in that location. The app is ran in the local server (tomcat), and it is used by other users in different workstations. I'm using POI API in converting the table into an Excel file and it works well.
The problem now is that FileOutputStream() wont create any file at all for other workstations. It only creates/saves a file in a single workstation, the workstation where I am currently developing the app. I've searched long enough to find a solution, but I haven't found any yet so I'm trying to ask you guys for help. I'll be including the codes.
public class ReportToExcel
{
//file returned comes from the user from antoher function.
public void exportTable(JTable table, File file) throws IOException {
HSSFWorkbook report = new HSSFWorkbook();
HSSFSheet report_sheet = report.createSheet("report_sheet");
HSSFRow rowHead = report_sheet.createRow((int)0);
TableModel model = table.getModel();
FileOutputStream reportOut = new FileOutputStream(file.getAbsolutePath());
System.out.println("table to excel "+table.getName());
for(int x=0; x < model.getColumnCount(); x++) {
rowHead.createCell((int)x).setCellValue(model.getColumnName(x));
report_sheet.autoSizeColumn((int)x);
System.out.println(x+". "+model.getColumnName(x));
}
for(int i=1; i< model.getRowCount(); i++) {
HSSFRow row = report_sheet.createRow((int)i);
for(int j=0; j < model.getColumnCount(); j++) {
row.createCell((int)j).setCellValue(model.getValueAt(i,j).toString());
System.out.println(i+". "+model.getValueAt(i,j).toString());
}
}
report.write(reportOut);
reportOut.close();
System.out.println("write out to: " + file.getAbsolutePath());
}
}

Already found the answer. For NoClassDefFound error, the workstations need to have priveleges in accessing the classpath of the POI api which i definiteley overlooked.

I think a key statement from you is that the code only works on your workstation. Many times the issue is related to file/directory permissions. Normal users do have less permission than a developer.
Also helps if you tell us the output to file.getAbsolutePath() in case the path is suspicious.
The path should be a safe place for your users to create/modify files.

Have you tried saving the output to the server, rather than the local machine for each workstation?