I have some Java code that does this (on a Windows 10 laptop with Windows Defender on):
unzip a .zip file into a folder (using a zip stream)
immediately list the folder's contents (using File#list)
There are ten files in the zip, from 100kB to 40MB. Normally only the first two files are listed -- the other 8 are silently lost. I know they actually make it to the directory because I can see them when I navigate to the directory myself.
I know there are better ways to implement this code, but I'm curious: is this expected? Is "write file into folder" and "file is listed when you list folder contents" not atomic on Windows? Is it a quirk of the underlying file system? Does Windows Defender make the file invisible for some period of time after it's written?
I am not facing the issue you mentioned and my code is able to list all the files correctly:
package com.test;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
public class ZipTest {
private static final int BUFFER_SIZE = 4096;
private static String INPUT_LOCATION = "C:/temp/zip/test.zip";
private static String OUTPUT_LOCATION = "C:/temp/unzip";
public static void main(String[] args) throws IOException {
unzip(INPUT_LOCATION, OUTPUT_LOCATION);
for (String s : new File(OUTPUT_LOCATION).list()) {
System.out.println(s);
}
}
public static void unzip(String zipFilePath, String destDirectory) throws IOException {
File destDir = new File(destDirectory);
if (!destDir.exists()) {
destDir.mkdir();
}
ZipInputStream zipIn = new ZipInputStream(new FileInputStream(zipFilePath));
ZipEntry entry = zipIn.getNextEntry();
while (entry != null) {
String filePath = destDirectory + File.separator + entry.getName();
if (!entry.isDirectory()) {
extractFile(zipIn, filePath);
} else {
File dir = new File(filePath);
dir.mkdir();
}
zipIn.closeEntry();
entry = zipIn.getNextEntry();
}
zipIn.close();
}
private static void extractFile(ZipInputStream zipIn, String filePath) throws IOException {
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(filePath));
byte[] bytesIn = new byte[BUFFER_SIZE];
int read = 0;
while ((read = zipIn.read(bytesIn)) != -1) {
bos.write(bytesIn, 0, read);
}
bos.close();
}
}
I am having issue with extracting files into subfolders. The zip files I need to unzip have multiple folders, and I am trying to keep the directory structure when unzipping.
package components;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
/**
* This utility extracts files and directories of a standard zip file to
* a destination directory.
*
*
*/
public class UnzipUtility {
/**
* Size of the buffer to read/write data
*/
private static final int BUFFER_SIZE = 4096;
/**
* Extracts a zip file specified by the zipFilePath to a directory specified by
* destDirectory (will be created if does not exists)
* #param zipFilePath
* #param destDirectory
* #throws IOException
*/
public void unzip(String zipFilePath, String destDirectory) throws IOException {
File destDir = new File(destDirectory);
if (!destDir.exists()) {
destDir.mkdir();
}
ZipInputStream zipIn = new ZipInputStream(new FileInputStream(zipFilePath));
ZipEntry entry = zipIn.getNextEntry();
// iterates over entries in the zip file
while (entry != null) {
String filePath = destDirectory + File.separator + entry.getName();
if (!entry.isDirectory()) {
// if the entry is a file, extracts it
extractFile(zipIn, filePath);
System.out.println(filePath);
} else {
// if the entry is a directory, make the directory
File dir = new File(filePath);
System.out.println(filePath);
dir.mkdir();
}
zipIn.closeEntry();
entry = zipIn.getNextEntry();
}
zipIn.close();
}
/**
* Extracts a zip entry (file entry)
* #param zipIn
* #param filePath
* #throws IOException
*/
private void extractFile(ZipInputStream zipIn, String filePath) throws IOException {
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(filePath));
byte[] bytesIn = new byte[BUFFER_SIZE];
int read = 0;
while ((read = zipIn.read(bytesIn)) != -1) {
bos.write(bytesIn, 0, read);
}
bos.close();
}
}
The output I get from the code is below. The issue is that it the file DoD-DISA-logos-as-JPEG.jpg is part of a child folder to subfolder. So path should be folder/subfolder/childfolder/DoD-DISA-logos-as-JPEG.jpg. It is not including that childfolder so it gets IO exception I believe.
java.io.FileNotFoundException: /Users/user/Desktop/folder/subfolder/DoD-DISA-logos-as-JPEG.jpg (No such file or directory)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:101)
at components.UnzipUtility.extractFile(UnzipUtility.java:62)
at components.UnzipUtility.unzip(UnzipUtility.java:42)
at components.UnzipUtilityTest.main(UnzipUtilityTest.java:9)
You should not rely on entry.isDirectory() or on the fact that the ZipEntry iterator will list all the folders hierarchies.
A zip files structure may contain only a ZipEntry file wich contains in its name a subfolders hierarchy. Eg. foo/bar/file.txt will be a ZipEntry but not foo/ nor foo/bar/
Thus for :
a file, you should get the parent directory and create the whole hierarchy if it does not exists with File#mkdirs
a directory, you should create the whole hierarchy if it does not exists with File#mkdirs
Your code should be transformed to something like that:
if (!entry.isDirectory()) {
// if the entry is a file, extracts it
new File(filePath).getParentFile().mkdirs();
extractFile(zipIn, filePath);
System.out.println(filePath);
} else {
// if the entry is a directory, make the directory
File dir = new File(filePath);
System.out.println(filePath);
dir.mkdirs();
}
I recommend to use an external library to zip/unzip like zt-zip.
I am trying to export whole repository in zip format .I am not able to find anything related to exporting a folder or repository in svnkit.
I have never tried zipping the whole repository using org.tmatesoft.svnkit. But I have an utility that zips files from a local folder. So if you copy the remote repository locally (into a temp directory), you should be able to zip it using the following class. It needs 2 values, starting folder path and the full, zip file name.
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
public class ZipFolder {
private String startingFolderPath;
private String outputZipFilePath;
private List<String> fileList;
/**
* Zip files
*/
public void zipFiles(){
byte[] buffer = new byte[1024];
try{
if (fileList == null) {
fileList = new ArrayList<String>();
generateFileList(new File(startingFolderPath));
}
System.out.println("File Count:" + fileList.size());
FileOutputStream fos = new FileOutputStream(outputZipFilePath);
ZipOutputStream zos = new ZipOutputStream(fos);
System.out.println("Output to Zip : " + outputZipFilePath);
for(String file : fileList){
System.out.println("File Included : " + file);
ZipEntry ze= new ZipEntry(file);
zos.putNextEntry(ze);
FileInputStream in =
new FileInputStream(startingFolderPath + File.separator + file);
int len;
while ((len = in.read(buffer)) > 0) {
zos.write(buffer, 0, len);
}
in.close();
}
zos.closeEntry();
//remember close it
zos.close();
System.out.println("Done");
}catch(IOException ex){
ex.printStackTrace();
}
}
/**
* Traverse a directory and get all files,
* and add the file into fileList
* #param node file or directory
*/
public void generateFileList(File node) throws IOException {
//add file only
if(node.isFile()){
fileList.add(generateZipEntry(node.getAbsoluteFile().toString()));
}
if(node.isDirectory()){
String[] subNote = node.list();
for(String filename : subNote){
generateFileList(new File(node, filename));
}
}
}
/**
* Format the file path for zip
* #param file file path
* #return Formatted file path
*/
private String generateZipEntry(String file){
return file.substring(startingFolderPath.length()+1, file.length());
}
public void setStartingFolderPath(String startingFolderPath) {
this.startingFolderPath = startingFolderPath;
}
public void setOutputZipFilePath(String outputZipFilePath) {
this.outputZipFilePath = outputZipFilePath;
}
}
I'm working on an application that need to create a tar archive in order to calculate his hash.
But I encounter some problems :
the tar is not the same in different machine, then the hash calculated is different
I'm not able to add directories properly
If I add an zip file, at the end, in the tar, I have the content off my zip file :/
I have read different post in SO and the dedicated tutorial on apache, and also the source test code of the apache commons compress jar, but I don't get the right solution.
Are there anybody that can find where my code is not correct ?
public static File createTarFile(File[] files, File repository) {
File tarFile = new File(TEMP_DIR + File.separator + repository.getName() + Constants.TAR_EXTENSION);
if (tarFile.exists()) {
tarFile.delete();
}
try {
OutputStream out = new FileOutputStream(tarFile);
TarArchiveOutputStream aos = (TarArchiveOutputStream) new ArchiveStreamFactory().createArchiveOutputStream("tar", out);
for(File file : files){
Utilities.addFileToTar(aos, file, "");
}
aos.finish();
aos.close();
out.close();
} catch (Exception e) {
e.printStackTrace();
}
return tarFile;
}
private static void addFileToTar(TarArchiveOutputStream tOut, File file, String base) throws IOException {
TarArchiveEntry entry = new TarArchiveEntry(file, base + file.getName());
entry.setModTime(0);
entry.setSize(file.length());
entry.setUserId(0);
entry.setGroupId(0);
entry.setUserName("avalon");
entry.setGroupName("excalibur");
entry.setMode(0100000);
entry.setSize(file.length());
tOut.putArchiveEntry(entry);
if (file.isFile()) {
IOUtils.copy(new FileInputStream(file), tOut);
tOut.closeArchiveEntry();
} else {
tOut.closeArchiveEntry();
File[] children = file.listFiles();
if (children != null) {
for (File child : children) {
addFileToTar(tOut, child, file.getName());
}
}
}
}
Thank you.
I finally found the solution after reading the post of caarlos0 : Encoding problem when compressing files with Apache Commons Compression on Linux
Using the apache-commons-1.8.jar library, I have made a tool class that can do the job :
You can find this code here : GitHub repository of the library MakeTar
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
import org.apache.commons.compress.utils.IOUtils;
/**
* The Class TarArchive.
*/
public class TarArchive {
/**
* Creates the tar of files.
*
* #param files the files
* #param tarPath the tar path
* #throws IOException Signals that an I/O exception has occurred.
*/
public static void createTarOfFiles(String[] files, String tarPath) throws IOException
{
FileOutputStream fOut = null;
BufferedOutputStream bOut = null;
TarArchiveOutputStream tOut = null;
Arrays.sort(files);
try
{
fOut = new FileOutputStream(new File(tarPath));
bOut = new BufferedOutputStream(fOut);
tOut = new TarArchiveOutputStream(bOut);
for (String file : files) {
addFileToTar(tOut, file, "");
}
}
finally
{
tOut.finish();
tOut.close();
bOut.close();
fOut.close();
}
}
/**
* Creates the tar of directory.
*
* #param directoryPath the directory path
* #param tarPath the tar path
* #throws IOException Signals that an I/O exception has occurred.
*/
public static void createTarOfDirectory(String directoryPath, String tarPath) throws IOException
{
FileOutputStream fOut = null;
BufferedOutputStream bOut = null;
TarArchiveOutputStream tOut = null;
try
{
fOut = new FileOutputStream(new File(tarPath));
bOut = new BufferedOutputStream(fOut);
tOut = new TarArchiveOutputStream(bOut);
addFileToTar(tOut, directoryPath, "");
}
finally
{
tOut.finish();
tOut.close();
bOut.close();
fOut.close();
}
}
/**
* Adds the file to tar.
*
* #param tOut the t out
* #param path the path
* #param base the base
* #throws IOException Signals that an I/O exception has occurred.
*/
private static void addFileToTar(TarArchiveOutputStream tOut, String path, String base) throws IOException
{
File f = new File(path);
String entryName = base + f.getName();
TarArchiveEntry tarEntry = new TarArchiveEntry(f, entryName);
tOut.setLongFileMode(TarArchiveOutputStream.LONGFILE_GNU);
if(f.isFile())
{
tarEntry.setModTime(0);
tOut.putArchiveEntry(tarEntry);
IOUtils.copy(new FileInputStream(f), tOut);
tOut.closeArchiveEntry();
}
else
{
File[] children = f.listFiles();
Arrays.sort(children);
if(children != null)
{
for(File child : children)
{
addFileToTar(tOut, child.getAbsolutePath(), entryName + "/");
}
}
}
}
}
Thanks for read me.
EDIT : Little correction, I have add the sort of the arrays.
EDIT 2 : I have corrected the code in order to have the same archive on all machine. The hash calculated on the archive is the same everywhere.
I need to open a .doc/.dot/.docx/.dotx (I'm not picky, I just want it to work) document,
parse it for placeholders (or something similar),
put my own data,
and then return generated .doc/.docx/.dotx/.pdf document.
And on top of all that, I need the tools to accomplish that to be free.
I've searched around for something that would suit my needs, but I can't find anything.
Tools like Docmosis, Javadocx, Aspose etc. are commercial.
From what I've read, Apache POI is nowhere near successfully implementing this (they currently don't have any official developer working on Word part of framework).
The only thing that looks that could do the trick is OpenOffice UNO API.
But that is a pretty big byte for someone that has never used this API (like me).
So if I am going to jump into this, I need to make sure that I am on the right path.
Can someone give me some advice on this?
I know it's been a long time since I've posted this question, and I said that I would post my solution when I'm finished.
So here it is.
I hope that it will help someone someday.
This is a full working class, and all you have to do is put it in your application, and place TEMPLATE_DIRECTORY_ROOT directory with .docx templates in your root directory.
Usage is very simple.
You put placeholders (key) in your .docx file, and then pass file name and Map containing corresponding key-value pairs for that file.
Enjoy!
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.BufferedReader;
import java.io.Closeable;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.URI;
import java.util.Deque;
import java.util.Enumeration;
import java.util.HashMap;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.Map;
import java.util.UUID;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipOutputStream;
import javax.faces.context.ExternalContext;
import javax.faces.context.FacesContext;
import javax.servlet.http.HttpServletResponse;
public class DocxManipulator {
private static final String MAIN_DOCUMENT_PATH = "word/document.xml";
private static final String TEMPLATE_DIRECTORY_ROOT = "TEMPLATES_DIRECTORY/";
/* PUBLIC METHODS */
/**
* Generates .docx document from given template and the substitution data
*
* #param templateName
* Template data
* #param substitutionData
* Hash map with the set of key-value pairs that represent
* substitution data
* #return
*/
public static Boolean generateAndSendDocx(String templateName, Map<String,String> substitutionData) {
String templateLocation = TEMPLATE_DIRECTORY_ROOT + templateName;
String userTempDir = UUID.randomUUID().toString();
userTempDir = TEMPLATE_DIRECTORY_ROOT + userTempDir + "/";
try {
// Unzip .docx file
unzip(new File(templateLocation), new File(userTempDir));
// Change data
changeData(new File(userTempDir + MAIN_DOCUMENT_PATH), substitutionData);
// Rezip .docx file
zip(new File(userTempDir), new File(userTempDir + templateName));
// Send HTTP response
sendDOCXResponse(new File(userTempDir + templateName), templateName);
// Clean temp data
deleteTempData(new File(userTempDir));
}
catch (IOException ioe) {
System.out.println(ioe.getMessage());
return false;
}
return true;
}
/* PRIVATE METHODS */
/**
* Unzipps specified ZIP file to specified directory
*
* #param zipfile
* Source ZIP file
* #param directory
* Destination directory
* #throws IOException
*/
private static void unzip(File zipfile, File directory) throws IOException {
ZipFile zfile = new ZipFile(zipfile);
Enumeration<? extends ZipEntry> entries = zfile.entries();
while (entries.hasMoreElements()) {
ZipEntry entry = entries.nextElement();
File file = new File(directory, entry.getName());
if (entry.isDirectory()) {
file.mkdirs();
}
else {
file.getParentFile().mkdirs();
InputStream in = zfile.getInputStream(entry);
try {
copy(in, file);
}
finally {
in.close();
}
}
}
}
/**
* Substitutes keys found in target file with corresponding data
*
* #param targetFile
* Target file
* #param substitutionData
* Map of key-value pairs of data
* #throws IOException
*/
#SuppressWarnings({ "unchecked", "rawtypes" })
private static void changeData(File targetFile, Map<String,String> substitutionData) throws IOException{
BufferedReader br = null;
String docxTemplate = "";
try {
br = new BufferedReader(new InputStreamReader(new FileInputStream(targetFile), "UTF-8"));
String temp;
while( (temp = br.readLine()) != null)
docxTemplate = docxTemplate + temp;
br.close();
targetFile.delete();
}
catch (IOException e) {
br.close();
throw e;
}
Iterator substitutionDataIterator = substitutionData.entrySet().iterator();
while(substitutionDataIterator.hasNext()){
Map.Entry<String,String> pair = (Map.Entry<String,String>)substitutionDataIterator.next();
if(docxTemplate.contains(pair.getKey())){
if(pair.getValue() != null)
docxTemplate = docxTemplate.replace(pair.getKey(), pair.getValue());
else
docxTemplate = docxTemplate.replace(pair.getKey(), "NEDOSTAJE");
}
}
FileOutputStream fos = null;
try{
fos = new FileOutputStream(targetFile);
fos.write(docxTemplate.getBytes("UTF-8"));
fos.close();
}
catch (IOException e) {
fos.close();
throw e;
}
}
/**
* Zipps specified directory and all its subdirectories
*
* #param directory
* Specified directory
* #param zipfile
* Output ZIP file name
* #throws IOException
*/
private static void zip(File directory, File zipfile) throws IOException {
URI base = directory.toURI();
Deque<File> queue = new LinkedList<File>();
queue.push(directory);
OutputStream out = new FileOutputStream(zipfile);
Closeable res = out;
try {
ZipOutputStream zout = new ZipOutputStream(out);
res = zout;
while (!queue.isEmpty()) {
directory = queue.pop();
for (File kid : directory.listFiles()) {
String name = base.relativize(kid.toURI()).getPath();
if (kid.isDirectory()) {
queue.push(kid);
name = name.endsWith("/") ? name : name + "/";
zout.putNextEntry(new ZipEntry(name));
}
else {
if(kid.getName().contains(".docx"))
continue;
zout.putNextEntry(new ZipEntry(name));
copy(kid, zout);
zout.closeEntry();
}
}
}
}
finally {
res.close();
}
}
/**
* Sends HTTP Response containing .docx file to Client
*
* #param generatedFile
* Path to generated .docx file
* #param fileName
* File name of generated file that is being presented to user
* #throws IOException
*/
private static void sendDOCXResponse(File generatedFile, String fileName) throws IOException {
FacesContext facesContext = FacesContext.getCurrentInstance();
ExternalContext externalContext = facesContext.getExternalContext();
HttpServletResponse response = (HttpServletResponse) externalContext
.getResponse();
BufferedInputStream input = null;
BufferedOutputStream output = null;
response.reset();
response.setHeader("Content-Type", "application/msword");
response.setHeader("Content-Disposition", "attachment; filename=\"" + fileName + "\"");
response.setHeader("Content-Length",String.valueOf(generatedFile.length()));
input = new BufferedInputStream(new FileInputStream(generatedFile), 10240);
output = new BufferedOutputStream(response.getOutputStream(), 10240);
byte[] buffer = new byte[10240];
for (int length; (length = input.read(buffer)) > 0;) {
output.write(buffer, 0, length);
}
output.flush();
input.close();
output.close();
// Inform JSF not to proceed with rest of life cycle
facesContext.responseComplete();
}
/**
* Deletes directory and all its subdirectories
*
* #param file
* Specified directory
* #throws IOException
*/
public static void deleteTempData(File file) throws IOException {
if (file.isDirectory()) {
// directory is empty, then delete it
if (file.list().length == 0)
file.delete();
else {
// list all the directory contents
String files[] = file.list();
for (String temp : files) {
// construct the file structure
File fileDelete = new File(file, temp);
// recursive delete
deleteTempData(fileDelete);
}
// check the directory again, if empty then delete it
if (file.list().length == 0)
file.delete();
}
} else {
// if file, then delete it
file.delete();
}
}
private static void copy(InputStream in, OutputStream out) throws IOException {
byte[] buffer = new byte[1024];
while (true) {
int readCount = in.read(buffer);
if (readCount < 0) {
break;
}
out.write(buffer, 0, readCount);
}
}
private static void copy(File file, OutputStream out) throws IOException {
InputStream in = new FileInputStream(file);
try {
copy(in, out);
} finally {
in.close();
}
}
private static void copy(InputStream in, File file) throws IOException {
OutputStream out = new FileOutputStream(file);
try {
copy(in, out);
} finally {
out.close();
}
}
}
Since a docx file is merely a zip-archive of xml files (plus any binary files for embedded objects such as images), we met that requirement by unpacking the zip file, feeding the document.xml to a template engine (we used freemarker) that does the merging for us, and then zipping the output document to get the new docx file.
The template document then is simply an ordinary docx with embedded freemarker expressions / directives, and can be edited in Word.
Since (un)zipping can be done with the JDK, and Freemarker is open source, you don't incur any licence fees, not even for word itself.
The limitation is that this approach can only emit docx or rtf files, and the output document will have the same filetype as the template. If you need to convert the document to another format (such as pdf) you'll have to solve that problem separately.
I ended up relying on Apache Poi 3.12 and processing paragraphs (separately extracting paragraphs also from tables, headers/footers, and footnotes, as such paragraphs aren't returned by XWPFDocument.getParagraphs() ).
The processing code (~100 lines) and unit tests are here on github.
I've been in more or less the same situation as you, I had to modify a whole bunch of MS Word merge templates at once. After having googled a lot to try to find a Java solution I finally installed Visual Studio 2010 Express which is free and did the job in C#.
I have recently dealt with similar problem:
"A tool which accepts a template '.docx' file, processes the file by evaluation of passed parameter context and outputs a '.docx' file as the result of the process."
finally god brought us scriptlet4dox :).
the key features for this product is:
1. groovy code injection as scripts in template file (parameter injection, etc.)
2. loop over collection items in table
and so many other features.
but as I checked the last commit on the project is performed about a year ago, so there is a probability that the project is not supported for new features and new bug-fixes. this is your choice to use it or not.