We Keep Coding

Apache commons compress 7z file size way bigger than p7zip compression

When I zip 500mb of html files, p7zip does it in a couple of seconds and the filesize is 7mb (Without any custom settings, just 7z a filename.7z /folder).
Thus I expected apache commons compress to also compress, using 7z, to a comparable size. This is however, not the case. Even though I enabled the max presets for apache commons compress 7z. The resulting file size is also huge, close to 100mb.
Do I do something wrong or do I need to tune my presets? I have read the apache commons compress wiki but have not found my answers.
Relevant code for the java implementation :
public static Path compress(String name, List<Path> files) throws IOException {
try (SevenZOutputFile out = new SevenZOutputFile(new File(name))) {
List<SevenZMethodConfiguration> methods = new ArrayList<>();
LZMA2Options lzma2Options = new LZMA2Options();
lzma2Options.setPreset(LZMA2Options.PRESET_MAX);
SevenZMethodConfiguration lzmaConfig =
new SevenZMethodConfiguration(SevenZMethod.LZMA, lzma2Options);
methods.add(lzmaConfig);
out.setContentMethods(methods);
for (Path file : files) {
addToArchiveCompression(out, file, ".");
}
}
return Paths.get(name);
}
private static void addToArchiveCompression(SevenZOutputFile out, Path file,
String dir) throws IOException {
String name = dir + File.separator + file.getFileName();
if (Files.isRegularFile(file)) {
SevenZArchiveEntry entry = out.createArchiveEntry(file.toFile(), name);
out.putArchiveEntry(entry);
FileInputStream in = new FileInputStream(file.toFile());
byte[] b = new byte[1024];
int count = 0;
while ((count = in.read(b)) > 0) {
out.write(b, 0, count);
}
out.closeArchiveEntry();
} else if (Files.isDirectory(file)) {
File[] children = file.toFile().listFiles();
if (children != null) {
for (File child : children) {
addToArchiveCompression(out, Paths.get(child.toURI()), name);
}
}
} else {
System.out.println(file.getFileName() + " is not supported");
}
}

Could you please try to remove these lines:
List<SevenZMethodConfiguration> methods = new ArrayList<>();
LZMA2Options lzma2Options = new LZMA2Options();
lzma2Options.setPreset(LZMA2Options.PRESET_MAX);
SevenZMethodConfiguration lzmaConfig =
new SevenZMethodConfiguration(SevenZMethod.LZMA, lzma2Options);
methods.add(lzmaConfig);
out.setContentMethods(methods);

Download attachments with same name without overwriting in Java

As per my requirement,i need to download a file from mail inbox into a specified directory,later after some time if same comes in , i need to save the same file into the same directory but with different name,here previous file should not be overridden means files must be saved in the same directory with same names(here i have one assumption,that , for example if my file is abc.txt, after modifications if i download the modified file it can be saved as abc(1).txt ). how can i resolve my issue? can anybody assist me to come out from this issue in JAVA. Below is my code but it is overwriting same file.
if (contentType.contains("multipart")) {
// this message may contain attachment
Multipart multiPart = (Multipart) message.getContent();
for (int i = 0; i < multiPart.getCount(); i++) {
MimeBodyPart part = (MimeBodyPart) multiPart.getBodyPart(i);
if (Part.ATTACHMENT.equalsIgnoreCase(part.getDisposition())) {
// save an attachment from a MimeBodyPart to a file
String destFilePath = "F:/unprocessed/"+part.getFileName();
InputStream input = part.getInputStream();
BufferedInputStream in = null;
in = new BufferedInputStream(input);
FileOutputStream output = new FileOutputStream(destFilePath);
byte[] buffer = new byte[4096];
int byteRead;
while ((byteRead = input.read(buffer)) != -1) {
output.write(buffer, 0, byteRead);
}
System.out.println("FileOutPutStream is Being Closed");
output.close();
}
}
}

As said before, you need to check the existing files. Here's one way of doing that:
public String getUniqueFileName(String input) {
String base = "F:/unprocessed/";
String filename = base+input;
File file = new File(filename);
int version = 0;
while (file.exists()) {
version++;
String filenamebase = filename.substring(0, filename.lastIndexOf('.'));
String extension = filename.substring(filename.lastIndexOf('.'));
file = new File(filenamebase+"("+ version+")"+extension);
}
return file.getAbsolutePath();
}
Then change the assignment of destFilePath to a call this method:
String destFilePath = getUniqueFileName(part.getFileName());

Split video into smaller timed segments in Java

I am looking to take a large video files (3hours+) and pass in segments of the video that I would like to split.
For example, I could pass in a 3 hour video - and want from 00:10 to 00:11 to be split into a separate file.
I currently have the below code - that takes my video and splits it into a split no. of segments, but how would I go about splitting the video by time instead?
Code:
try {
File file = new File("//Users//robeves//Desktop//Videos to split//TestVideo.mp4");//File read from Source folder to Split.
if (file.exists()) {
String videoFileName = file.getName().substring(0, file.getName().lastIndexOf(".")); // Name of the videoFile without extension
File splitFile = new File("//Users//robeves//Desktop//Videos to split//Converted//"+ videoFileName);//Destination folder to save.
if (!splitFile.exists()) {
splitFile.mkdirs();
System.out.println("Directory Created -> "+ splitFile.getAbsolutePath());
}
int i = 01;// Files count starts from 1
InputStream inputStream = new FileInputStream(file);
String videoFile = splitFile.getAbsolutePath() +"/"+ String.format("%02d", i) +"_"+ file.getName();// Location to save the files which are Split from the original file.
OutputStream outputStream = new FileOutputStream(videoFile);
System.out.println("File Created Location: "+ videoFile);
int totalPartsToSplit = 20;// Total files to split.
int splitSize = inputStream.available() / totalPartsToSplit;
int streamSize = 0;
int read = 0;
while ((read = inputStream.read()) != -1) {
if (splitSize == streamSize) {
if (i != totalPartsToSplit) {
i++;
String fileCount = String.format("%02d", i); // output will be 1 is 01, 2 is 02
videoFile = splitFile.getAbsolutePath() +"/"+ fileCount +"_"+ file.getName();
outputStream = new FileOutputStream(videoFile);
System.out.println("File Created Location: "+ videoFile);
streamSize = 0;
}
}
outputStream.write(read);
streamSize++;
}
inputStream.close();
outputStream.close();
System.out.println("Total files Split ->"+ totalPartsToSplit);
} else {
System.err.println(file.getAbsolutePath() +" File Not Found.");
}
} catch (Exception e) {
e.printStackTrace();
}
}

If you do want to be able to play the segments individually, then the above code may not work if it is just splitting the file at arbitrary points, as many video formats need to finish on a good 'boundary' to enable proper playback.
As Binkan suggests, using a video library like ffmpeg, either in cmd line, wrapped cmd line or by using its associated C libraries, will allow you safely split a video in most common formats.
For example the following ffmpeg cmd line will create a segment from an mp4 video:
ffmpeg -i inputVideo.mp4 -ss 00:00:00 -t 00:00:10 -c copy outputVideoSegment.mp4
The following code uses this utility in a 'wrapper' to segment a video file into chunks:
int chunkSize = videoDurationSecs/(numberOfChunks + 1);
int startSecs = 0;
for (int i=0; i<numberOfChunks; i++) {
//Create video chunk
String startTime = convertSecsToTimeString(startSecs);
int endSecs = startSecs + ((i+1)*chunkSize);
if (endSecs > videoDurationSecs) {
//make sure rounding does not mean we go beyond end of video
endSecs = videoDurationSecs;
}
String endTime = convertSecsToTimeString(endSecs);
//Call ffmpeg to create this chunk of the video using a ffmpeg wrapper
String argv[] = {"ffmpeg", "-i", videoPath,
"-ss",startTime, "-t", endTime,
"-c","copy", segmentVideoPath[i]};
int ffmpegWrapperReturnCode = ffmpegWrapper(argv);
}
String convertSecsToTimeString(int timeSeconds) {
//Convert number of seconds into hours:mins:seconds string
int hours = timeSeconds / 3600;
int mins = (timeSeconds % 3600) / 60;
int secs = timeSeconds % 60;
String timeString = String.format("%02d:%02d:%02d", hours, mins, secs);
return timeString;
}
Examples of wrappers are here, but you can also use the ffmpeg libraries directly if you would rather avoid the wrapper approach (which does have the disadvantage that ffmpeg cmd line is not really intended to be wrapped in this way):
http://www.sauronsoftware.it/projects/jave/
https://github.com/jhotovy/android-ffmpeg (Android based)

Java Code to Zip/FTP Directory in a UNIX Server

I want to write a couple of methods in a Java package, which would be deployed in a UNIX Server.
As of now, my code was for Windows Server, for which I used the following code to zip Directory.
public static final void zipDirectory(File fBatchDirectory, String batchName, String ondemandDocExtension) throws IOException
{
//Set zip file name
File zip = new File(fBatchDirectory + "\\" + StringUtils.replace(batchName,".ind", "") + ".zip");
//filter file
FileFilter filter = new FileFilter(ondemandDocExtension);
File[] files = fBatchDirectory.listFiles(filter);
if(files.length > 0)
{
ZipOutputStream zos = new ZipOutputStream( new FileOutputStream(zip) );
zip(files, fBatchDirectory, zos , ondemandDocExtension);
zos.close();
}
}
private static final void zip(File[] files, File base,ZipOutputStream zos , String docExtension) throws IOException
{
byte[] buffer = new byte[8192];
int read = 0;
for (int i = 0, n = files.length; i < n; i++)
{
//Add to zip only if its file
if (files[i].isFile())
{
FileInputStream in = new FileInputStream(files[i]);
ZipEntry entry = new ZipEntry(files[i].getPath().substring(base.getPath().length() + 1));
zos.putNextEntry(entry);
while (-1 != (read = in.read(buffer)))
{
zos.write(buffer, 0, read);
}
in.close();
}
}
}
I am confused as to how to replicate the same functionality to zip Directory in Java, for UNIX?
And then I want to FTP the files from a UNIX Serve to another UNIX Server.
Any pointers would be greatly appreciated.

At a first glance, the only problem I see is at this line:
File zip = new File(fBatchDirectory + "\\" + StringUtils.replace(batchName,".ind", "") + ".zip");
Because you are explicitly using the double backslash (\\) in your filename. If you change that for File.separator your code should work for both operating systems:
File zip = new File(fBatchDirectory + File.separator + StringUtils.replace(batchName,".ind", "") + ".zip");
For the FTP part of it, you can get down and dirty and use an FTP client or use a more high level library like Apache Commons VFS which, by the way, inspired the new IO FileSystem API in Java 7, but I don't now about any library implementing the FTP protocol with the new API at the moment.

how to create java zip archives with a max file size limit

I need to write an algorithm in java (for an android app) to read a folder containing more folders and each of those containing images and audio files so the structure is this: mainDir/categorySubfolder/myFile1.jpg
My problem is that I need to limit the size of the archive to 16mb and at runtime, create as many archives as needed to contain all my files from my main mainDir folder.
I tried several examples from the net and I read the java documentation but I can't manage to understand and put it all together the way I need it. Has someone done this before or has a link or an example for me?
I resolved the reading of the files with a recursive method but I can't write the logic for the zip creation.
I'm open for suggestions or better a working example.

zip4j is a great library that can create multi-part zip files.
net.lingala.zip4j.core.ZipFile zipFile = new ZipFile("out.zip");
ZipParameters parameters = new ZipParameters();
parameters.setCompressionMethod(Zip4jConstants.COMP_DEFLATE);
parameters.setCompressionLevel(Zip4jConstants.DEFLATE_LEVEL_NORMAL);
zipFile.createZipFileFromFolder("path/to/source/dir", parameters, true, maximum size);
You can find more examples on their web-site.

I am using below code/class to split and zip a large amount/size of files.
I have tested this class on below
number of uncompressed files : 116
total size (uncompressed) : 29.1 GB
ZIP file size limit (each) : 3 GB [MAX_ZIP_SIZE]
total size (compressed) : 7.85 GB
number of ZIP file (splited as MAX_ZIP_SIZE): 3
you have to change the value of MAX_ZIP_SIZE to 16(MB)10241024=16777216-22(zip header size)=16777194.
In my code, MAX_ZIP_SIZE set to 3 GB (ZIP has limitation of 4GB on various things).
final long MAX_ZIP_SIZE = 3221225472L; //3 GB
package com.company;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
public class QdeZip {
public static String createZIP(String directoryPath, String zipFileName, String filesToZip) {
try {
final int BUFFER = 104857600; // 100MB
final long MAX_ZIP_SIZE = 3221225472L; //3 GB
long currentSize = 0;
int zipSplitCount = 0;
String files[] = filesToZip.split(",");
if (!directoryPath.endsWith("/")) {
directoryPath = directoryPath + "/";
}
byte fileRAW[] = new byte[BUFFER];
ZipOutputStream zipOut = new ZipOutputStream(new FileOutputStream(directoryPath + zipFileName.toUpperCase()));
ZipEntry zipEntry;
FileInputStream entryFile;
for (String aFile : files) {
zipEntry = new ZipEntry(aFile);
if (currentSize >= MAX_ZIP_SIZE) {
zipSplitCount++;
//zipOut.closeEntry();
zipOut.close();
zipOut = new ZipOutputStream(new FileOutputStream(directoryPath + zipFileName.toLowerCase().replace(".zip", "_" + zipSplitCount + ".zip").toUpperCase()));
currentSize = 0;
}
zipOut.putNextEntry(zipEntry);
entryFile = new FileInputStream(directoryPath + aFile);
int count;
while ((count = entryFile.read(fileRAW, 0, BUFFER)) != -1) {
zipOut.write(fileRAW, 0, count);
//System.out.println("number of Bytes read = " + count);
}
entryFile.close();
zipOut.closeEntry();
currentSize += zipEntry.getCompressedSize();
}
zipOut.close();
//System.out.println(directory + " -" + zipFileName + " -Number of Files = " + files.length);
} catch (FileNotFoundException e) {
return "FileNotFoundException = " + e.getMessage();
} catch (IOException e) {
return "IOException = " + e.getMessage();
} catch (Exception e) {
return "Exception = " + e.getMessage();
}
return "1";
}
}
I have returned all Exception Messages as String to work with it. this
my own case related to project.

As far as I can see How to split a huge zip file into multiple volumes? just suggests keeping track of the archive size so far, and when it approaches some arbitrary value (which should be lower than your max), it'll decide to start a new file. So for a 16MB limit, you could set the value to 10MB and start a new zip whenever this is reached, but if you reach 9MB and your next file zips down to 8MB, you'll end up with a zip bigger than your limit.
The code given in that post didn't seem to work for me because 1) it got the size before the ZipEntry was created, so it was always 0 and 2) it didn't write out any zip :-) If I've got that wrong - let me know.
The following works for me. For simplicity, I've taken it out of the Wrapper and just have it all in main(String args[]). There are many, many ways this code could be improved :-)
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.Deflater;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
public class ChunkedZipTwo {
static final long MAX_LIMIT = 10 * 1000 * 1024; //10MB limit - hopefully this
public static void main(String[] args) throws Exception {
String[] files = {"file1", "file2", "file3"};
int i = 0;
boolean needNewFile = false;
long overallSize = 0;
ZipOutputStream out = getOutputStream(i);
byte[] buffer = new byte[1024];
for (String thisFileName : files) {
if (overallSize > MAX_LIMIT) {
out.close();
i++;
out = getOutputStream(i);
overallSize = 0;
}
FileInputStream in = new FileInputStream(thisFileName);
ZipEntry ze = new ZipEntry(thisFileName);
out.putNextEntry(ze);
int len;
while ((len = in.read(buffer)) > 0) {
out.write(buffer, 0, len);
}
out.closeEntry();
in.close();
overallSize += ze.getCompressedSize();
}
out.close();
}
public static ZipOutputStream getOutputStream(int i) throws IOException {
ZipOutputStream out = new ZipOutputStream(new FileOutputStream("bigfile" + i + ".zip"));
out.setLevel(Deflater.DEFAULT_COMPRESSION);
return out;
}
}