We Keep Coding

How to use LightCells Api to read and write large excel files in Aspose.Cells

I want to make use of the LightCells API within Aspose.Cells to help optimize my read and write processes while also reducing the memory utilization while processing the excel file.
I have queries regarding the implementation of this API. How can I use light cell APIs - LightCellsDataProvider and LightCellsDataHandler - to efficiently read and write Excel through less memory utilization. I specifically want to load the data of the cells in memory in chunks (for e.g. processing only the first 100 rows and then the next 100 and so on) so that less memory will be consumed in the entire process.
Want to implement below code but using lightCell api in aspose-
public static void createTemplate( String originalFilePath, List<String> originalFileSheetNameList, String templateFilePath ) throws Exception {
LoadOptions opts = new LoadOptions();
opts.setMemorySetting(MemorySetting.MEMORY_PREFERENCE);
Workbook wb = new Workbook(originalFilePath, opts);
WorksheetCollection wsCollections = wb.getWorksheets();
for(String sheetName: originalFileSheetNameList) {
Worksheet sheet = wsCollections.get(sheetName);
sheet.getCells().deleteRows(1, sheet.getCells().getMaxDataRow());
}
FileOutputStream outputStream = new FileOutputStream(templateFilePath);
wb.save(outputStream, SaveFormat.XLSX);
outputStream.close();
}
Below code is the Aspose.Cells light cell api implementation.
Using LightCells how can I implement above code -
public static void createTemplate(String originalFilePath, List<String> originalFileSheetName, String templateFilePath ) throws Exception {
LoadOptions opts = new LoadOptions();
LightCellsDataHandlerVisitCells v = new LightCellsDataHandlerVisitCells();
opts.setLightCellsDataHandler((LightCellsDataHandler) v);
Workbook wb = new Workbook(originalFilePath, opts);
int sheetCount = wb.getWorksheets().getCount();
v.startSheet(wb.getWorksheets().get("System_Data"));
System.out.println("Total sheets: " + sheetCount + ", cells: " + v.cellCount + ", strings: " + v.stringCount
+ ", formulas: " + v.formulaCount);
}
For reading -
import com.aspose.cells.Cell;
import com.aspose.cells.CellValueType;
import com.aspose.cells.LightCellsDataHandler;
import com.aspose.cells.Row;
import com.aspose.cells.Worksheet;
public class LightCellsDataHandlerVisitCells implements LightCellsDataHandler {
public int cellCount;
public int formulaCount;
public int stringCount;
public LightCellsDataHandlerVisitCells() {
this.cellCount = 0;
this.formulaCount = 0;
this.stringCount = 0;
}
public int cellCount() {
return cellCount;
}
public int formulaCount() {
return formulaCount;
}
public int stringCount() {
return stringCount;
}
public boolean startSheet(Worksheet sheet) {
System.out.println("Processing sheet[" + sheet.getName() + "]");
System.out.println("Processing sheet rows[" + sheet.getCells().getMaxDataRow() + "]");
return true;
}
public boolean startRow(int rowIndex) {
return true;
}
public boolean processRow(Row row) {
return true;
}
public boolean startCell(int column) {
return true;
}
public boolean processCell(Cell cell) {
this.cellCount = this.cellCount + 1;
cell.getRow();
if (cell.isFormula()) {
this.formulaCount = this.formulaCount + 1;
} else if (cell.getType() == CellValueType.IS_STRING) {
this.stringCount = this.stringCount + 1;
}
return false;
}
}
getting print -1 on System.out.println("Processing sheet rows[" + sheet.getCells().getMaxDataRow() + "]");
For writing -
public class LightCellsDataProviderDemo implements LightCellsDataProvider {
private final int sheetCount;
private final int maxRowIndex;
private final int maxColIndex;
private int rowIndex;
private int colIndex;
private final Style style1;
private final Style style2;
public LightCellsDataProviderDemo(Workbook wb, int sheetCount, int rowCount, int colCount) {
// set the variables/objects
this.sheetCount = sheetCount;
this.maxRowIndex = rowCount - 1;
this.maxColIndex = colCount - 1;
// add new style object with specific formattings
style1 = wb.createStyle();
Font font = style1.getFont();
font.setName("MS Sans Serif");
font.setSize(10);
font.setBold(true);
font.setItalic(true);
font.setUnderline(FontUnderlineType.SINGLE);
font.setColor(Color.fromArgb(0xffff0000));
style1.setHorizontalAlignment(TextAlignmentType.CENTER);
// create another style
style2 = wb.createStyle();
style2.setCustom("#,##0.00");
font = style2.getFont();
font.setName("Copperplate Gothic Bold");
font.setSize(8);
style2.setPattern(BackgroundType.SOLID);
style2.setForegroundColor(Color.fromArgb(0xff0000ff));
style2.setBorder(BorderType.TOP_BORDER, CellBorderType.THICK, Color.getBlack());
style2.setVerticalAlignment(TextAlignmentType.CENTER);
}
public boolean isGatherString() {
return false;
}
public int nextCell() {
if (colIndex < maxColIndex) {
colIndex++;
return colIndex;
}
return -1;
}
public int nextRow() {
if (rowIndex < maxRowIndex) {
rowIndex++;
colIndex = -1; // reset column index
if (rowIndex % 1000 == 0) {
System.out.println("Row " + rowIndex);
}
return rowIndex;
}
return -1;
}
public void startCell(Cell cell) {
if (rowIndex % 50 == 0 && (colIndex == 0 || colIndex == 3)) {
// do not change the content of hyperlink.
return;
}
if (colIndex < 10) {
cell.putValue("test_" + rowIndex + "_" + colIndex);
cell.setStyle(style1);
} else {
if (colIndex == 19) {
cell.setFormula("=Rand() + test!L1");
} else {
cell.putValue(rowIndex * colIndex);
}
cell.setStyle(style2);
}
}
public void startRow(Row row) {
row.setHeight(25);
}
public boolean startSheet(int sheetIndex) {
if (sheetIndex < sheetCount) {
// reset row/column index
rowIndex = -1;
colIndex = -1;
return true;
}
return false;
}`
Source - https://forum.aspose.com/t/read-write-huge-excel-files-1m-rows/38441/8
https://docs.aspose.com/cells/java/using-lightcells-api/

You need to understand the object model, internal structure and working of the light cells APIs first. The LightCells APIs (provided by Aspose.Cells) are mainly designed to manipulate cell data one by one without building a complete data model block (using the Cell collection, etc.) into memory. It works in an event-driven mode. To save workbooks, it will provide the cell content cell by cell when saving, and the component saves it to the output file directly. Similarly when reading template files, the component parses every cell and provides their value one by one.
In short, when using light cells APIs, one Cell object is processed and then discarded, the Workbook object does not hold the collection at all in the memory. This model would surely save lots of memory when handling large Excel file with data.
Please consider and note:
For your needs, you may choose XLSX file format instead of XLS, because XLSX is an Open Office XML-based spreadsheet format. Generally, using LightCells API to save XLSX file may save 50% or more memory than using the common way, saving XLS may save about 20-40% memory.
I specifically want to load the data of the cells in memory in chunks (for e.g. processing only the first 100 rows and then the next 100 and so on) so that less memory will be consumed in the entire process.
As I mentioned above, you do not need to do this as in LightCells APIs, data is processed cell by cell (without building the complete data model for the whole spreadsheet) which is more performance oriented approach and will consume less memory and works efficiently. In light weight mode, one cell is processed and then discarded immediately and then it navigates to next cell and so on.
Want to implement below code but using lightCell api in aspose-
public static void createTemplate( String originalFilePath, List originalFileSheetNameList, String templateFilePath ) throws Exception {
LoadOptions opts = new LoadOptions();
opts.setMemorySetting(MemorySetting.MEMORY_PREFERENCE);
Workbook wb = new Workbook(originalFilePath, opts);
In LightCells mode, importing additional data, deleting rows, columns or range of cells may not work (considering the architecture and nature of the LightCells APIs), so you have to do such things outside of the implementation, i.e., perform the operations like removing rows/cols or cells in the class that implements the light cells interface.
Should you have further queries, doubts or issues working with light cells APIs, you may also post in the dedicated forums.
PS. I am working as Support developer/ Evangelist at Aspose.

Apache POI: Remove Chart from Word Template file entirely

I am using a Word template with Excel graphs which I want to programmatically manipulate with the Java Apache POI library. For this I also need to be able to conditionally delete a Chart which is stored in this template.
Based on Axel Richters post (Removing chart from PowerPoint slide with Apache POI) I think I am almost there, but when I want to open the updated Word file it gives an error about unreadable content. This is what I have thus far:
PackagePart packagePartChart = xWPFChart.getPackagePart();
PackagePart packagePartWordDoc = xWPFDocument.getPackagePart();
OPCPackage packageWordDoc = packagePartWordDoc.getPackage();
// iterate over all relations the chart has and remove them
for (PackageRelationship chartrelship : packagePartChart.getRelationships()) {
String partname = chartrelship.getTargetURI().toString();
PackagePart part = packageWordDoc.getPartsByName(Pattern.compile(partname)).get(0);
packageWordDoc.removePart(part);
packagePartChart.removeRelationship(chartrelship.getId());
}
// now remove the chart itself from the word doc
Method removeRelation = POIXMLDocumentPart.class.getDeclaredMethod("removeRelation", POIXMLDocumentPart.class);
removeRelation.setAccessible(true);
removeRelation.invoke(xWPFDocument, xWPFChart);
If I unzip the Word file I correctly see that:
the relation between the WordDoc and the Chart are deleted in '\word\ _rels\document.xml.rels'
the chart itself is deleted in folder '\word\charts'
the relations between the documents supporting the Chart itself are deleted in folder '\word\charts\' _rels
the related chart items themselves are deleted:
StyleN / ColorsN in folder '\word\charts' and
Microsoft_Excel_WorksheetN in folder '\word\embeddings'
Anybody any idea on what could be going wrong here?

Indeed the finding of the right paragraph holding the chart was a challenge. In the end, for simplicity sake, I added a one row/one column placeholder table with one cell with a text, let's say 'targetWordString' in it, DIRECTLY before the chart. With the below function I am location the BodyElementID of this table:
private Integer iBodyElementIterator (XWPFDocument wordDoc,String targetWordString) {
Iterator<IBodyElement> iter = wordDoc.getBodyElementsIterator();
Integer bodyElementID = null;
while (iter.hasNext()) {
IBodyElement elem = iter.next();
bodyElementID = wordDoc.getBodyElements().indexOf(elem);
if (elem instanceof XWPFParagraph) {
XWPFParagraph paragraph = (XWPFParagraph) elem;
for (XWPFRun runText : paragraph.getRuns()) {
String text = runText.getText(0);
Core.getLogger("WordExporter").trace("Body Element ID: " + bodyElementID + " Text: " + text);
if (text != null && text.equals(targetWordString)) {
break;
}
}
} else if (elem instanceof XWPFTable) {
if (((XWPFTable) elem).getRow(0) != null && ((XWPFTable) elem).getRow(0).getCell(0) != null) {
// the first cell holds the name via the template
String tableTitle = ((XWPFTable) elem).getRow(0).getCell(0).getText();
if (tableTitle.equals(targetWordString)) {
break;
}
Core.getLogger("WordExporter").trace("Body Element ID: " + bodyElementID + " Text: " + tableTitle);
} else {
Core.getLogger("WordExporter").trace("Body Element ID: " + bodyElementID + " Table removed!");
}
}
else {
Core.getLogger("WordExporter").trace("Body Element ID: " + bodyElementID + " Text: ?");
}
}
return bodyElementID;
}
In the main part of the code I am calling this function to locate the table, then first delete the chart (ID +1) and then the table (ID)
int elementIDToBeRemoved = iBodyElementIterator(xWPFWordDoc,targetWordString);
xWPFWordDoc.removeBodyElement(elementIDToBeRemoved + 1);
xWPFWordDoc.removeBodyElement(elementIDToBeRemoved);
It needs to be in this order, because the ID's are reordered once you delete a number in between, hence deleting the table first, would mean the chart would get that ID in principle.

I did find a more structural solution for it without the use of dummy tables for the positioning. The solution is build out into 2 parts:
Determine relationship id of chart in word document
Remove chart based on relationship id and remove paragraph + runs
Determine relationship id of chart in word document
First I have determined with a boolean variable 'chartUsed' whether the chart needs to be deleted.
if (!chartUsed) {
String chartPartName = xWPFChart.getPackagePart().getPartName().getName();
String ridChartToBeRemoved = null;
// iterate over all relations the chart has and remove them
for (RelationPart relation : wordDoc.getRelationParts()) {
PackageRelationship relShip = relation.getRelationship();
if (relShip.getTargetURI().toString().equals(chartPartName)) {
ridChartToBeRemoved = relShip.getId();
Core.getLogger(logNode).info("Chart with title " + chartTitle + " has relationship id " + relShip.getId());
break;
}
}
removeChartByRelId(wordDoc, ridChartToBeRemoved);
}
(Core.getLogger is an internal API from the Mendix low code platform I use).
At the end I am calling the function removeChartByRelId which will remove the chart if the relation is found within a run of a paragraph:
Remove chart based on relationship id and remove paragraph + runs
private void removeChartByRelId (XWPFDocument wordDoc, String chartRelId) {
Iterator<IBodyElement> iter = wordDoc.getBodyElementsIterator();
Integer bodyElementID = null;
while (iter.hasNext()) {
IBodyElement elem = iter.next();
bodyElementID = wordDoc.getBodyElements().indexOf(elem);
if (elem instanceof XWPFParagraph) {
Core.getLogger(logNode).trace("XWPFParagraph Body Element ID: " + bodyElementID);
XWPFParagraph paragraph = (XWPFParagraph) elem;
boolean removeParagraph = false;
for (XWPFRun run : paragraph.getRuns()) {
if (run.getCTR().getDrawingList().size()>0) {
String drawingXMLText = run.getCTR().getDrawingList().get(0).xmlText();
boolean isChart = drawingXMLText.contains("http://schemas.openxmlformats.org/drawingml/2006/chart");
String graphicDataXMLText = run.getCTR().getDrawingList().get(0).getInlineArray(0).getGraphic().getGraphicData().xmlText();
boolean contains = graphicDataXMLText.contains(chartRelId);
if (isChart && contains) {
Core.getLogger(logNode).info("Removing chart with relId " + chartRelId + " as it isnt used in data");
run.getCTR().getDrawingList().clear();
removeParagraph = true;
}
}
}
if (removeParagraph) {
for (int i = 0; i < paragraph.getRuns().size(); i++){
paragraph.removeRun(i);
}
}
}
}
}

Retrieving and locating cell value in Excel

Hopefully i will explain clear enough, I have a List of objects ( each object have 3 properties ) those objects are printed to xls file, I am trying to locate the cell where the value were assigned first and the last in order to get a references and setFormula to get Sum of all of the middle values
public String getCellReference(int rowPosition, int cellPosition) {
CellReference reference = new CellReference(rowPosition, cellPosition);
return reference.formatAsString();
}
public String getCellReference(int rowPosition, int cellPosition) {
CellReference reference = new CellReference(rowPosition, cellPosition);
return reference.formatAsString();
}
public String getSumFormula(int rowPosition, int cellPosition, int rowPosition2, int cellPosition2) {
String startCell = getCellReference(rowPosition, cellPosition);
String finishCell = getCellReference(rowPosition2, cellPosition2);
return "SUM(" + startCell + ":" + finishCell + ")";
}
To locate first value and last its not a problem but its seems to be a problem for me to locate a cell indexes
public Cell locateCell(List<Invoice> invoices){
BigDecimal firstGeneralTotal = invoices.get(0).generalTotal();
BigDecimal lastGeneralTotal = invoices.get(invoices.size()-1).generalTotal();
}

Basic access using POI would be like this:
XSSFWorkbook wb = new XSSFWorkbook( … );
XSSFSheet sheet = workbook1.getSheetAt(0);
XSSFRow row = sheet.getRow(0);
XSSFCell cell = row.getCell(0);
String value = cell.getStringCellValue();
Instead of using getStringCellValue you can also use one of those methods:
cell.getCellFormula()
cell.getNumericCellValue()
cell.getBooleanCellValue()
cell.getErrorCellValue()
Accordingly, you can do cell.setCellFormula(String formula) to set a cell.
If you want to add a formular to the end of the table containing the sum, you could use the cell references if you like. In this case, you'd need to find the last row in the excel sheet:
sheet.getPhysicalNumberOfRows()
Then you could add a new row to the end of your sheet:
sheet.createRow(sheet.getPhysicalNumberOfRows()+1);
and put your sum in according cell.
Btw, I am not sure if I would use CellReferences, since they can contain references to other sheets as well. If I work only with a single sheet, I'd try to use the index numbers and translate them accordingly to A1:A200 etc.
Does this help you? If I got your question wrong, let me know in the comments, I may update my answer if possible.

public String getCellReference(int rowPosition, int cellPosition) {
CellReference reference = new CellReference(rowPosition, cellPosition);
return reference.formatAsString();
}
public String getCoefficientFormula(int startRow, int firstCell, int endRow, int lastCell) {
String firstReference = getCellReference(startRow, firstCell);
String lastReference = getCellReference(endRow, lastCell);
return firstReference + "/" + lastReference;
}
public String getSumFormula(int startRow, int endRow, int start, int finish) {
String startCell = getCellReference(startRow,start);
String finishCell = getCellReference(endRow,finish);
return "SUM(" + startCell + ":" + finishCell + ")";
}

Spreadsheet API : Get cell entry from specific cell address

how can I get a CellEntry given a specific cell address e.g in A1 notation. For example,
String address = "A1";
CellEntry cellEntry = getCellEntry(address);
public CellEntry getCellEntry(String address){
//- What can I do inside here
}
Previously, I fetch everything using CellFeed and iterating using cellFeed.getEntries to get each CellEntry and filter out based on cell address. But I think it will have a performance issues since my spreadsheet is large. Any idea ? I'm using Java.

I think I got two approaches here :
Method 1
Using CellQuery
public CellEntry getCellEntry(URL cellFeedUrl, Integer colNum, Integer rowNum){
CellQuery cellQuery = new CellQuery(cellFeedUrl);
cellQuery.setMaximumCol(colNum);
cellQuery.setMinimumCol(colNum);
cellQuery.setMaximumRow(rowNum);
cellQuery.setMinimumRow(rowNum);
CellFeed cellFeed = service.getFeed(cellQuery, CellFeed.class)
return cellFeed.getEntries().get(0);
}
Note : The setMaximumCol and setMinimumCol must be the same to get the specified column. Similar with the setMaximumRow and setMinimumRow.
Method 2
Using CellFeedURL
public CellEntry getCellEntry(WorksheetEntry worksheet, Integer colNum, Integer rowNum){
if (colNum == null || rowNum == null){
//- do something
}
else {
String row = "?min-row=" + rowNum + "&max-row=" + rowNum;
String col = "&min-col=" + colNum + "&max-col=" + colNum;
URL cellFeedUrl = new URI(worksheet.getCellFeedUrl().toString()
+ row + col).toURL();
CellFeed cellFeed = service.getFeed(cellFeedUrl, CellFeed.class);
return cellFeed.getEntries().get(0);
}
}
Note : Based on Google Spreadsheet API : Changing content of a cell
Personally I think the second method is better.

Retrieve values from excel using poi

I am trying to get the column values for a specific row in a excel using poi methods.
I am able to get the values but the problem is I want the values only from second column.
public static ArrayList<String> GetBusinessComponentList() throws IOException{
String Tcname = "TC02_AggregateAutoByPassRO_CT";
ArrayList<String> arrayListBusinessFlow ;
arrayListBusinessFlow = new ArrayList<String>();
FileInputStream fileInput = new FileInputStream(oFile);
wb = new HSSFWorkbook(fileInput);
sheet = wb.getSheet("Business Flow");
int rownr = findRow(sheet, Tcname);
row = sheet.getRow(rownr);
for (Cell cell : row) {
String arr = cell.getStringCellValue();
arrayListBusinessFlow.add(arr);
}
return arrayListBusinessFlow;
}
private static int findRow(HSSFSheet sheet, String cellContent){
for (Row row : sheet) {
for (Cell cell : row) {
if (cell.getCellType() == Cell.CELL_TYPE_STRING) {
if (cell.getRichStringCellValue().getString().trim().equals(cellContent)) {
return row.getRowNum();
}
}
}
}
return 0;
}
}
OUTPUT:
[TC02_AggregateAutoByPassRO_CT,
StrategicUINewBusiness.Login,
StrategicUINewBusiness.CustomerSearch,
StrategicUINewBusiness.NamedInsured,
StrategicUINewBusiness.InsuranceScoreByPass,
StrategicUINewBusiness.VehiclePage,
StrategicUINewBusiness.DriverPage,
StrategicUINewBusiness.ViolationPage,
StrategicUINewBusiness.UnderwritingPage,
StrategicUINewBusiness.CoveragePage,
StrategicUINewBusiness.Portfolio,
StrategicUINewBusiness.BillingPage,
StrategicUINewBusiness.FinalSalePage,
StrategicUINewBusiness.PolicyConfirmation, , , ]
But I do not want my test case name when I am getting.
Please help me what changes i needed to do. thanks!

Currently, the code you're using to iterate over cells only returns cells with content or styling, and skips totally empty ones. You need to change to one of the other ways of iterating over cells, so you can control it to read from the second column onwards.
If you look at the Apache POI Documentation on iterating over rows and cells, you'll see a lot more details on the two main ways to iterate.
For your case, you'll want something like:
// We want to read from the 2nd column onwards, zero based
int firstColumn = 1;
// Always fetch at least 4 columns
int MY_MINIMUM_COLUMN_COUNT = 5;
// Work out the last column to go to
int lastColumn = Math.max(r.getLastCellNum(), MY_MINIMUM_COLUMN_COUNT);
// To format cells into strings
DataFormatter df = new DataFormatter();
// Iterate over the cells
for (int cn = firstColumn; cn < lastColumn; cn++) {
Cell c = r.getCell(cn, Row.RETURN_BLANK_AS_NULL);
if (c == null) {
// The spreadsheet is empty in this cell
} else {
// Do something useful with the cell's contents
// eg get the cells value as a string
String cellAsString = df.formatCellValue(c);
}
}

Use Cell cell=row.getCell(1); and also you can use sheet.getLastRowNum() to get the number last row on the sheet.
for (int i=0;i<=row.getLastCellNum();i++) {
if (i!=1){
//your stuff
}
}