We Keep Coding

Appending data to a file in HDFS using java failed , getting error

i create a new csv file in hdfs using java , and i am trying to append the data to that csv file , but failed to append with error
Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[192.168.1.25:9866,DS-b6d8a63b-357d-4d39-9f27-1ab76b8b6ccc,DISK]], original=[Dat
below is the code
csv file created and uplaoded to HDFS from java code , but not able append data to the existing file . but a newly uploaded csv from ui interface was able to appended data with java code , please help to resolve this issue.
private void appendFileToFile (String fileName) throws Exception{
long testTime1 = System.currentTimeMillis();
String hdfsHostDetails = new String("hdfs://192.168.1.25:9000");
Configuration conf = new Configuration();
conf.setBoolean("dfs.support.append", true);
FileSystem fs = FileSystem.get(URI.create(hdfsHostDetails), conf);
String dirpath = new String(hdfsHostDetails);
String targetfilepath = new String(dirpath+"/"+fileName);
int count = 0;
while (count < 2) {
int offset = 0;
int limit = 10000;
IgniteTable table = new IgniteTable(ignite, "nok_customer_demand");
String query = "SELECT * FROM nok_customer_demand OFFSET "+ offset +" ROWS FETCH NEXT "+ limit +" ROWS ONLY";
List<List<?>> lists = table._select(query);
List<String[]> rows = new ArrayList();
System.out.println(":::::::::::::::::: Data Ready for iteration ::::::::::::::"+ count);
// create a new file on each iteration
File file = new File("/home/tejatest1"+count+".csv");
FileWriter outputfile = new FileWriter(file);
CSVWriter writer = new CSVWriter(outputfile);
for (List eachlist : lists) {
String[] eachRowAsString = new String[eachlist.size()];
;
int i = 0;
for (Object eachcol : eachlist) {
eachRowAsString[i] = String.valueOf(eachcol);
rows.add(eachRowAsString);
i++;
}
writer.writeNext(eachRowAsString);
}
// on each iteration append the data in the file to hdfs
InputStream in = new BufferedInputStream(new FileInputStream(file));
FSDataOutputStream out =null;
if(!fs.exists(new Path(targetfilepath))) {
out = fs.create(new Path(targetfilepath));
} else{
out = fs.append(new Path(targetfilepath));
}
IOUtils.copyBytes(in, out, 4096, true);
writer.close();
out.close();
outputfile.close();
lists.clear();
in.close();
file.delete();
count++;
}
long testTime2 = System.currentTimeMillis();
System.out.println("-----total time taken for data fetch for all records in table using limit and offset:-------" + (testTime2 - testTime1) + " ms");
fs.close();
}

i resolve this issue with the below configuration
Configuration conf = new Configuration();
conf.set("fs.defaultFS",hdfsHostDetails);
conf.setInt("dfs.replication",1);
conf.setBoolean("dfs.client.block.write.replace-datanode-on-failure.enable",false);
conf.setBoolean("dfs.support.append", true);
FileSystem fs = FileSystem.get(URI.create(hdfsHostDetails), conf);

Create one Zip file using a set of pdf files

My app is a tender document system where each tender number has one or more pdf files attached.
application is done in java ee using struts and mysql.
in a database table the paths of each related pdf file for a tender number is stores.
I want to get all the pdf files and create a single ZIP file for each tender number so that user can download that zip file and have all the related documents in a single click.
I tried Google and found something called ZipOutputStream but i cannot understand how to use this in my application.

You're almost there... This is a small example of how to use ZipOutputStream... let's asume that you have a JAVA helper H that returns database records with pdf file paths (and related info):
FileOutputStream zipFile = new FileOutputStream(new File("xxx.zip"));
ZipOutputStream output = new ZipOutputStream(zipFile);
for (Record r : h.getPdfRecords()) {
ZipEntry zipEntry = new ZipEntry(r.getPdfName());
output.putNextEntry(zipEntry);
FileInputStream pdfFile = new FileInputStream(new File(r.getPath()));
IOUtils.copy(pdfFile, output); // this method belongs to apache IO Commons lib!
pdfFile.close();
output.closeEntry();
}
output.finish();
output.close();

Checkout this code, here you can easily create a zip file directory:
public class CreateZipFileDirectory {
public static void main(String args[])
{
try
{
String zipFile = "C:/FileIO/zipdemo.zip";
String sourceDirectory = "C:/examples";
//create byte buffer
byte[] buffer = new byte[1024];
FileOutputStream fout = new FileOutputStream(zipFile);
ZipOutputStream zout = new ZipOutputStream(fout);
File dir = new File(sourceDirectory);
if(!dir.isDirectory())
{
System.out.println(sourceDirectory + " is not a directory");
}
else
{
File[] files = dir.listFiles();
for(int i=0; i < files.length ; i++)
{
System.out.println("Adding " + files[i].getName());
FileInputStream fin = new FileInputStream(files[i]);
zout.putNextEntry(new ZipEntry(files[i].getName()));
int length;
while((length = fin.read(buffer)) > 0)
{
zout.write(buffer, 0, length);
}
zout.closeEntry();
fin.close();
}
}
zout.close();
System.out.println("Zip file has been created!");
}
catch(IOException ioe)
{
System.out.println("IOException :" + ioe);
}
}
}

Is it possible to write in cell like POI using POITransformer?

This is what I need to do.
1) Accept an xlsx/xls file from client.
2) Backend will receive it in the form of multipart file
3) The file will be processed and if the format of the data is invalid, that same file will be updated and the error message will be written in the side of the input of the client.
4) this modified file will be sent back to the user.
But after several attempts, I could not make my code work.
def generateErrorReport(ServletResponse response, Map messageCollections, MultipartFile file, String ext){
FileInputStream fileIn = file.getInputStream()
Workbook workbook = (ext.equalsIgnoreCase("xls")) ? new HSSFWorkbook(fileIn) : new XSSFWorkbook(fileIn)
workbook = this.getWorkbook((MultipartFile) file, ext.equalsIgnoreCase("xls"));
try {
Sheet sheet = workbook.getSheetAt(0)
Long lastCellNum = sheet.getRow(0).getLastCellNum();
for(int i=1; i<sheet.getLastRowNum(); i++){
if(messageCollections[i]!=null && messageCollections[i]!=[] ) {
Cell cell = sheet.getRow(i).getCell(lastCellNum + 1)
cell.setCellValue(messageCollections[i]);
}
}
fileIn.close()
FileOutputStream fileOut = new FileOutputStream((File) file)
workbook.write(fileOut);
response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")
response.setHeader("Content-Disposition", "Attachment;Filename=error.xlsx")
response.outputStream << fileOut
response.outputStream.flush()
fileOut.close()
}catch(Exception ex){
println ex
}
}
Now, I think a found way to do it, that is to use POITransformer. The problem is it is used when you have a template. My idea was to use the file sent by the client as template and then just simply write the error message beside the client's input. However I couldnt find a way to write it because I couldn't find a setCellData or any method same as that.
The question is, is it possible to write to a desired cell data using POITransformer? If yes, how will I do that?
So far this is what i've done. But it still not writing. Could you tell me whats wrong?
private void bla(ServletResponse response, Map messageCollections, MultipartFile file, String ext){
InputStream is = file.getInputStream();
OutputStream os = response.outputStream;
String fileName = "error";
response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
response.setHeader("Content-Disposition", "Attachment;Filename=${fileName}");
try {
PoiTransformer transformer = PoiTransformer.createTransformer(is, os);
org.apache.poi.ss.usermodel.Workbook workbook = transformer.getWorkbook()
Sheet sheet = workbook.getSheetAt(workbook.getActiveSheetIndex())
int lastColNum = sheet.getRow(0).getLastCellNum()
Cell cellData;
(0..sheet.getLastRowNum()) {
if (messageCollections[it]!=null && messageCollections[it]!=[]) {
cellData = sheet.getRow(it).getCell(lastColNum+1);
cellData.setCellValue(messageCollections[it].toString())
}
}
transformer.write();
} catch (IOException ex) {
println ex
// Logger.getLogger(ExcelFileHandler.class.getName()).log(Level.SEVERE, null, ex);
} finally {
closeStream(is);
closeStream(os);
}
}

Hopefully, this answer would be helpful to others.
InputStream is = file.getInputStream();
OutputStream os = response.outputStream;
String fileName = "desiredFilename." + ext
response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
response.setHeader("Content-Disposition", "Attachment;Filename=${fileName}");
try {
PoiTransformer transformer = PoiTransformer.createTransformer(is, os);
org.apache.poi.ss.usermodel.Workbook workbook = transformer.getWorkbook()
Sheet sheet = workbook.getSheetAt(workbook.getActiveSheetIndex())
int lastColNum = sheet.getRow(0).getLastCellNum()
Cell cell;
cell = sheet.getRow(0).getCell(lastColNum);
if(cell==null){
cell = sheet.getRow(0).createCell(lastColNum);
}
cell.setCellType(1)
cell.setCellValue("Message")
cell.setCellStyle(getStyle(workbook, 2))
for(int it=1; it<sheet.getLastRowNum(); it++) {
if (message.get(new Long(it))!=null && message.get(new Long(it))!=[]) {
cell = sheet.getRow(it).getCell(lastColNum);
if(cell==null){
cell = sheet.getRow(it).createCell(lastColNum);
}
cell.setCellType(1)
cell.setCellValue(message.get(new Long(it)).join(', '))
cell.setCellStyle(getStyle(workbook, 1))
}
}
sheet.autoSizeColumn(lastColNum);
transformer.write();

Error convert docx to pdf in java

Good afternoon all,
Come to my case, I'm generating a docx document Junction 2 other docx, I'm doing a merge.
public static void main(String[] args) throws Exception {
InputStream in1 = new FileInputStream(new File("C:\\Clientes\\Constremac\\Repositorio_DOCS\\UPLOAD\\LAYOUT_PAGINA_VERSAO_FINAL.docx"));
InputStream in2 = new FileInputStream(new File("C:\\Clientes\\Constremac\\Repositorio_DOCS\\UPLOAD\\modeloContratoSocial.docx"));
OutputStream out = new FileOutputStream(new File("C:\\Clientes\\Constremac\\Repositorio_DOCS\\UPLOAD\\modeloContratoSocialMerge.docx"));
mergeDocx(in1,in2,out);
}
public static void mergeDocx(InputStream s1, InputStream s2, OutputStream os) throws Exception {
WordprocessingMLPackage target = WordprocessingMLPackage.load(s1);
insertDocx(target.getMainDocumentPart(), IOUtils.toByteArray(s2));
SaveToZipFile saver = new SaveToZipFile(target);
saver.save(os);
}
private static void insertDocx(MainDocumentPart main, byte[] bytes) throws Exception {
AlternativeFormatInputPart afiPart = new AlternativeFormatInputPart(new PartName("/part" + (chunk++) + ".docx"));
afiPart.setContentType(new ContentType(CONTENT_TYPE));
afiPart.setBinaryData(bytes);
Relationship altChunkRel = main.addTargetPart(afiPart);
//convertAltChunks()
CTAltChunk chunk = Context.getWmlObjectFactory().createCTAltChunk();
chunk.setId(altChunkRel.getId());
main.addObject(chunk);
}
My final document (docx) is ok, I can open it normally. The problem occurs when I will convert this generated file to PDF, the following error appears: NOT IMPLEMENTED: support for w: altChunk -.
public boolean createPDF(String nomeArquivo) {
try {
long start = System.currentTimeMillis();
Configuration confg = new Configuration();
System.out.println(Configuration.repositorioUpload + nomeArquivo + ".docx");
InputStream is = new FileInputStream(new File(Configuration.repositorioUpload + nomeArquivo + ".docx"));
WordprocessingMLPackage wordMLPackage = WordprocessingMLPackage.load(is);
PdfSettings pdfSettings = new PdfSettings();
OutputStream out = new FileOutputStream(new File(Configuration.repositorioUpload + nomeArquivo + ".pdf"));
PdfConversion converter = new Conversion(wordMLPackage);
converter.output(out, pdfSettings);
System.err.println("Generate " + Configuration.repositorioUpload + nomeArquivo + ".pdf" + " with " + (
System.currentTimeMillis() - start) + "ms");
}
catch (Throwable e) {
e.printStackTrace();
}
return false;
}
I'm sending the java code i use, for a while I'm trying to generate this pdf, if anyone able to help me I am grateful.
Thank you all.
Hugs!
I found a way to use AltChunck, but even beyond not run correctly merge the images footer and header when exported to PDF does not appear.
public static void main(String[] args) throws Exception {
boolean ADD_TO_HEADER = true;
HeaderPart hp = null;
String inputfilepath = "C:\\Clientes\\Constremac\\Repositorio_DOCS\\UPLOAD\\default_template.xml";
String chunkPath = "C:\\Clientes\\Constremac\\Repositorio_DOCS\\UPLOAD\\sample.docx";
boolean save = true;
String outputfilepath = "C:\\Clientes\\Constremac\\Repositorio_DOCS\\UPLOAD\\altChunk_out.docx";
// Open a document from the file system
// 1. Load the Package
WordprocessingMLPackage wordMLPackage = WordprocessingMLPackage.load(new java.io.File(inputfilepath));
//proce
MainDocumentPart main = wordMLPackage.getMainDocumentPart();
if (ADD_TO_HEADER) {
hp = wordMLPackage.getDocumentModel().getSections().get(0).getHeaderFooterPolicy().getDefaultHeader();
}
AlternativeFormatInputPart afiPart = new AlternativeFormatInputPart(new PartName("/chunk.docx"));
afiPart.setBinaryData(new FileInputStream(chunkPath));
afiPart.setContentType(new ContentType("application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml")); //docx
//afiPart.setContentType(new ContentType("application/xhtml+xml")); //xhtml
Relationship altChunkRel = null;
if (ADD_TO_HEADER) {
altChunkRel = hp.addTargetPart(afiPart);
} else {
altChunkRel = main.addTargetPart(afiPart);
}
CTAltChunk ac = Context.getWmlObjectFactory().createCTAltChunk();
ac.setId(altChunkRel.getId());
if (ADD_TO_HEADER) {
hp.getJaxbElement().getEGBlockLevelElts().add(ac);
} else {
main.addObject(ac);
}
// Save it
if (save) {
SaveToZipFile saver = new SaveToZipFile(wordMLPackage);
saver.save(outputfilepath);
System.out.println("Saved " + outputfilepath);
}
}
What am I doing wrong?

An altChunk is not "real" docx content.
Before it can be outputted in PDF, it needs to be replaced with normal WordML paragraphs, tables etc.
You can try doing this yourself, which is easy enough if the content does not include any relationships (images, hyperlinks etc), or conflicting styles or numbering. Please see further http://www.docx4java.org/blog/2010/11/merging-word-documents/ .. or my company's website plutext.com

This can be solved
An altChunk is not "real" docx content.
using java we can convert altchunk to original content word tags,
convert the document.xml inside docx
Docx4jProperties.setProperty(“docx4j.Convert.Out.HTML.OutputMethodXML”,
true);
Docx4J.toHTML(htmlSettings, os, Docx4J.FLAG_EXPORT_PREFER_XSL);
Open the link for complete code.
[Convert AltChunk To Original Content][1]
https://kishankichi.wordpress.com/2016/05/26/convert-altchunk-to-original-content-or-convert-to-real-docx-format-using-java
https://kishankichi.wordpress.com/2016/05/26/convert-altchunk-to-original-content-or-convert-to-real-docx-format-using-java/
Note:
Kindly ignore &nbsp and other such tags in your html content.
I have checked only for &nbsp.
Thanks for the replay...

How to extract Lotus Notes database icon?

I have tried to extract Lotus Notes database icon by using DXL Exporter but it is not success. Result file is corrupt and can not be opened by image viewer.
How can I extract Lotus Notes database icon by using java?
private String extractDatabaseIcon() {
String tag = "";
String idfile = "";
String password = "";
String dbfile = "";
NotesThread.sinitThread();
Session s = NotesFactory.createSessionWithFullAccess();
s.createRegistration().switchToID(idfile, password);
Database d = s.getDatabase("", dbfile);
NoteCollection nc = d.createNoteCollection(false);
nc.setSelectIcon(true);
nc.buildCollection();
String noteId = nc.getFirstNoteID();
int counter = 0;
while (noteId != null) {
counter++;
try {
Document doc = d.getDocumentByID(noteId);
DxlExporter dxl = s.createDxlExporter();
String xml = dxl.exportDxl(doc);
xml = xml.substring(xml.indexOf("<note "));
org.jsoup.nodes.Document jdoc = Jsoup.parse(xml);
Element ele = jdoc.select("rawitemdata").first();
String raw = ele.text().trim();
String temp = System.getProperty("java.io.tmpdir") + UUID.randomUUID().toString() + "\\";
File file = new File(temp);
file.mkdir();
String filename = temp + UUID.randomUUID().toString().replaceAll("-", "") + ".gif";
byte[] buffer = decode(raw.getBytes());
FileOutputStream fos = new FileOutputStream(filename);
fos.write(buffer);
fos.close();
tag = filename;
} catch (Exception e) {
logger.error("", e);
}
if (counter >= nc.getCount()) {
noteId = null;
} else {
noteId = nc.getNextNoteID(noteId);
}
}
return tag;
}
private byte[] decode(byte[] b) throws Exception {
ByteArrayInputStream bais = new ByteArrayInputStream(b);
InputStream b64is = MimeUtility.decode(bais, "base64");
byte[] tmp = new byte[b.length];
int n = b64is.read(tmp);
byte[] res = new byte[n];
System.arraycopy(tmp, 0, res, 0, n);
return res;
}

It is not even a bitmap, it is an icon. The format you can find here:
http://www.daubnet.com/formats/ICO.html
I managed to do this, a long time ago, in LotusScript. My code was based on an earlier version of this page:
http://www2.tcl.tk/11202
For the icon itself, you only have to open one document:
NotesDocument doc = db.getDocumentByID("FFFF8010")
exporter = session.createDXLExporter
exporter.setConvertNotesBitmapsToGIF(false)
outputXML = exporter.export(doc)
and then parse the XML to find the rawitemdata from the IconBitmap item, as you did in your original code.

I'm not sure what the format is. As far as I know' it's a 16 color bitmap, but not in standard BMP file format. And it's definitely not GIF format, but you can tell the DXLExporter to convert it. The default is to leave it native, so you need to add this to your code before you export:
dxl.setConvertNotesBitmapsToGIF(true);