We Keep Coding

Combining methods to TAR + LZ4 in a single run in Java

I have two methods that separately TAR a set of files and then another that will compress it with LZ4. They both work fine, but I'm wondering if combining them together would be more efficient or save some time? Also I'm not really sure how I would combine them. Any suggestions would be useful. As you can see in my code below, I'm trying to have fine access to the data so I can give a good % complete to the user.
public static boolean createTarFile(List<Path> paths, Path output, int machineId)
{
boolean success = false;
boolean failure = false;
try (OutputStream fOut = Files.newOutputStream(output, StandardOpenOption.APPEND);
BufferedOutputStream buffOut = new BufferedOutputStream(fOut);
TarArchiveOutputStream tOut = new TarArchiveOutputStream(buffOut))
{
float index = 1;
for (Path path : paths)
{
TarArchiveEntry tarEntry = new TarArchiveEntry(
path.toFile(),
path.getFileName().toString());
tarEntry.setSize(path.toFile().length());
tOut.putArchiveEntry(tarEntry);
// copy file to TarArchiveOutputStream
Files.copy(path, tOut);
tOut.closeArchiveEntry();
tarPercentComplete = (index / (float) paths.size()) * 100;
index++;
if (abort)
{
break;
}
}
tOut.finish();
}
catch (Exception e)
{
LOG.error("Tarring file failed ", e);
failure = true;
}
}
return success;
}
/**
* Zip a tar file using LZ4
*
* #param fileToZip
* #param outputFileName
* #return
*/
public boolean zipFile(File fileToZip, File outputFileName)
{
boolean success = false;
boolean failure = false;
try (FileOutputStream fos = new FileOutputStream(outputFileName);
LZ4FrameOutputStream lz4fos = new LZ4FrameOutputStream(fos);)
{
try (FileInputStream fis = new FileInputStream(fileToZip))
{
byte[] buf = new byte[bufferSizeZip];
int length;
long count = 0;
while ((length = fis.read(buf)) > 0)
{
lz4fos.write(buf, 0, length);
if (count % 50 == 0)
{
zipPercentComplete = ((bufferSizeZip * count) / (float) fileToZip.length()) * 100;
}
count++;
if (abort)
{
break;
}
}
}
}
catch (Exception e)
{
LOG.error("Zipping file failed ", e);
failure = true;
}
}
return success;
}

Just chain them.
try (OutputStream fOut = Files.newOutputStream(output, StandardOpenOption.APPEND);
BufferedOutputStream buffOut = new BufferedOutputStream(fOut);
LZ4FrameOutputStream lz4fos = new LZ4FrameOutputStream(buffOut);)
TarArchiveOutputStream tOut = new TarArchiveOutputStream(lz4fos)) {
No need to name them all.
try (TarArchiveOutputStream tOut = new TarArchiveOutputStream(
new LZ4FrameOutputStream(
new BufferedOutputStream(
Files.newOutputStream(output, StandardOpenOption.APPEND))))) {

Combining tar + lz4 in a single passs will definitely be more efficient,
because it will avoid moving data to/from storage
for a content which is inherently temporary.
On a shell (command line), one would do something like that :
tar cvf - DIR | lz4 > DIR.tar.lz4
which uses stdout / stdin as the intermediate interface (instead of storage file I/O).
However, as you are not using shell in your code,
prefer following suggestions from #Andreas for a Java example.
The main idea is the same, but the implementation is definitely different.

Counting the number of attachments on a PDF using iText

I am trying to count the number of attachments on a PDF to verify our attachment code. The code I have works most of the time but recently it started failing when the number of attachments went up as well as the size of the attachments. Example: I have a PDF with 700 attachments which total 1.6 gb. And another with 65 attachments of around 10mb. The 65 count was done incrementally. We had built it up file by file. At 64 files (about 9.8mb) the routine counted fine. Add file 65 (about .5mb) and the routine failed.
This is on itextpdf-5.5.9.jar under jre1.8.0_162
We are still testing different combinations of file numbers and size to see where it breaks.
private static String CountFiles() throws IOException, DocumentException {
Boolean errorFound = new Boolean(true);
PdfDictionary root;
PdfDictionary names;
PdfDictionary embeddedFiles;
PdfReader reader = null;
String theResult = "unknown";
try {
if (!theBaseFile.toLowerCase().endsWith(".pdf"))
theResult = "file not PDF";
else {
reader = new PdfReader(theBaseFile);
root = reader.getCatalog();
names = root.getAsDict(PdfName.NAMES);
if (names == null)
theResult = "0";
else {
embeddedFiles = names.getAsDict(PdfName.EMBEDDEDFILES);
PdfArray namesArray = embeddedFiles.getAsArray(PdfName.NAMES);
theResult = String.format("%d", namesArray.size() / 2);
}
reader.close();
errorFound = false;
}
}
catch (Exception e) {
theResult = "unknown";
}
finally {
if (reader != null)
reader.close();
}
if (errorFound)
sendError(theResult);
return theResult;
}
private static String AttachFileInDir() throws IOException, DocumentException {
String theResult = "unknown";
String outputFile = theBaseFile.replaceFirst("(?i).pdf$", ".attach.pdf");
int maxFiles = 1000;
int fileCount = 1;
PdfReader reader = null;
PdfStamper stamper = null;
try {
if (!theBaseFile.toLowerCase().endsWith(".pdf"))
theResult = "basefile not PDF";
else if (theFileDir.length() == 0)
theResult = "no attach directory";
else if (!Files.isDirectory(Paths.get(theFileDir)))
theResult = "invalid attach directory";
else {
reader = new PdfReader(theBaseFile);
stamper = new PdfStamper(reader, new FileOutputStream(outputFile));
stamper.getWriter().setPdfVersion(PdfWriter.VERSION_1_7);
Path dir = FileSystems.getDefault().getPath(theFileDir);
DirectoryStream<Path> stream = Files.newDirectoryStream(dir);
for (Path path : stream) {
stamper.addFileAttachment(null, null, path.toFile().toString(), path.toFile().getName());
if (++fileCount > maxFiles) {
theResult = "maxfiles exceeded";
break;
}
}
stream.close();
stamper.close();
reader.close();
theResult = "SUCCESS";
}
}
catch (Exception e) {
theResult = "unknown";
}
finally {
if (stamper != null)
stamper.close();
if (reader != null)
reader.close();
}
if (theResult != "SUCCESS")
sendError(theResult);
return theResult;
}
I expect a simple count of attachments back. What seems to be happening is the namesArray is coming back null. The result stays "unknown". I suspect the namesArray is trying to hold all the files and choking on the size.
Note: The files are being attached using the AttachFileInDir procedure. Dump all the files in a directory and run the AttachFileInDir. And yes the error trapping in AttachFileInDir needs work.
Any help would be appreciated or another method welcome

I finally got it. Turns out each KID is a dictionary of NAMES….
Each NAMES hold 64 file references. At 65 files and up it made a KIDS dictionary array of names. So 279 files = ( 8*64 +46 )/2 (9 total KIDS array elements).
One thing that I had to compensate for. If one deletes all the attachments off a pdf it leaves artifacts behind as opposed to a PDF that never had an attachment
private static String CountFiles() throws IOException, DocumentException {
Boolean errorFound = new Boolean(true);
int totalFiles = 0;
PdfArray filesArray;
PdfDictionary root;
PdfDictionary names;
PdfDictionary embeddedFiles;
PdfReader reader = null;
String theResult = "unknown";
try {
if (!theBaseFile.toLowerCase().endsWith(".pdf"))
theResult = "file not PDF";
else {
reader = new PdfReader(theBaseFile);
root = reader.getCatalog();
names = root.getAsDict(PdfName.NAMES);
if (names == null){
theResult = "0";
errorFound = false;
}
else {
embeddedFiles = names.getAsDict(PdfName.EMBEDDEDFILES);
filesArray = embeddedFiles.getAsArray(PdfName.NAMES);
if (filesArray != null)
totalFiles = filesArray.size();
else {
filesArray = embeddedFiles.getAsArray(PdfName.KIDS);
if (filesArray != null){
for (int i = 0; i < filesArray.size(); i++)
totalFiles += filesArray.getAsDict(i).getAsArray(PdfName.NAMES).size();
}
}
theResult = String.format("%d", totalFiles / 2);
reader.close();
errorFound = false;
}
}
}
catch (Exception e) {
theResult = "unknown" + e.getMessage();
}
finally {
if (reader != null)
reader.close();
}
if (errorFound)
sendError(theResult);
return theResult;
}

How do I concatenate sequential files in order with Java?

I have a directory that contains sequentially numbered log files and some Excel spreadsheets used for analysis. The log file are ALWAYS sequentially numbered beginning at zero, but the number of them can vary. I am trying to concatenate the log files, in the order they were created into a single text file which will be a concatenation of all the log files.
For instance, with log files foo0.log, foo1.log, foo2.log would be output to concatenatedfoo.log by appending foo1 after foo0, and foo2 after foo1.
I need to count all the files in the given directory with the extension of *.log, using the count to drive a for-loop that also generates the file name for concatenation. I'm having a hard time finding a way to count the files using a filter...none of the Java Turtorials on file operations seem to fit the situation, but I'm sure I'm missing something. Does this approach make sense? or is there an easier way?
int numDocs = [number of *.log docs in directory];
//
for (int i = 0; i <= numberOfFiles; i++) {
fileNumber = Integer.toString(i);
try
{
FileInputStream inputStream = new FileInputStream("\\\\Path\\to\\file\\foo" + fileNumber + ".log");
BufferedReader br = new BufferedReader(new InputStreamReader(inputStream));
try
{
BufferedWriter metadataOutputData = new BufferedWriter(new FileWriter("\\\\Path\\to\\file\\fooconcat.log").append());
metadataOutputData.close();
}
//
catch (IOException e) // catch IO exception writing final output
{
System.err.println("Exception: ");
System.out.println("Exception: "+ e.getMessage().getClass().getName());
e.printStackTrace();
}
catch (Exception e) // catch IO exception reading input file
{
System.err.println("Exception: ");
System.out.println("Exception: "+ e.getMessage().getClass().getName());
e.printStackTrace();
}
}

how about
public static void main(String[] args){
final int BUFFERSIZE = 1024 << 8;
File baseDir = new File("C:\\path\\logs\\");
// Get the simple names of the files ("foo.log" not "/path/logs/foo.log")
String[] fileNames = baseDir.list(new FilenameFilter() {
#Override
public boolean accept(File dir, String name) {
return name.endsWith(".log");
}
});
// Sort the names
Arrays.sort(fileNames);
// Create the output file
File output = new File(baseDir.getAbsolutePath() + File.separatorChar + "MERGED.log");
try{
BufferedOutputStream out = new BufferedOutputStream(new FileOutputStream(output), BUFFERSIZE);
byte[] bytes = new byte[BUFFERSIZE];
int bytesRead;
final byte[] newLine = "\n".getBytes(); // use to separate contents
for(String s : fileNames){
// get the full path to read from
String fullName = baseDir.getAbsolutePath() + File.separatorChar + s;
BufferedInputStream in = new BufferedInputStream(new FileInputStream(fullName),BUFFERSIZE);
while((bytesRead = in.read(bytes,0,bytes.length)) != -1){
out.write(bytes, 0, bytesRead);
}
// close input file and ignore any issue with closing it
try{in.close();}catch(IOException e){}
out.write(newLine); // seperation
}
out.close();
}catch(Exception e){
throw new RuntimeException(e);
}
}
This code DOES assume that the "sequential naming" would be zero padded such that they will lexigraphically (?? sp) sort correctly. i.e. The files would be
0001.log (or blah0001.log, or 0001blah.log etc)
0002.log
....
0010.log
and not
1.log
2.log
...
10.log
The latter pattern will not sort correctly with the code I have given.

Here's some code for you.
File dir = new File("C:/My Documents/logs");
File outputFile = new File("C:/My Documents/concatenated.log");
Find the ".log" files:
File[] files = dir.listFiles(new FilenameFilter() {
#Override
public boolean accept(File file, String name) {
return name.endsWith(".log") && file.isFile();
}
});
Sort them into the appropriate order:
Arrays.sort(files, new Comparator<File>() {
#Override
public int compare(File file1, File file2) {
return numberOf(file1).compareTo(numberOf(file2));
}
private Integer numberOf(File file) {
return Integer.parseInt(file.getName().replaceAll("[^0-9]", ""));
}
});
Concatenate them:
byte[] buffer = new byte[8192];
OutputStream out = new BufferedOutputStream(new FileOutputStream(outputFile));
try {
for (File file : files) {
InputStream in = new FileInputStream(file);
try {
int charCount;
while ((charCount = in.read(buffer)) >= 0) {
out.write(buffer, 0, charCount);
}
} finally {
in.close();
}
}
} finally {
out.flush();
out.close();
}

By having the log folder as a File object, you can code like this
for (File logFile : logFolder.listFiles()){
if (logFile.getAbsolutePath().endsWith(".log")){
numDocs++;
}
}
to find the number of log files.

I would;
open the output file once. Just use a PrintWriter.
in a loop ...
create a File for each possible file
if it doesn't exist break the loop.
Using a BufferedReader
to read the lines of the file with readLine()
write each line to the output file.
You should be able to do this with about 12 lines of code. I would pass the IOExceptions to the caller.

You can use SequenceInputStream for concatenation of FileInputStreams.
To see all log files File.listFiles(FileFilter) can be used.
It will give you unsorted array with files. To sort files in right order, use Arrays.sort.
Code example:
static File[] logs(String dir) {
File root = new File(dir);
return root.listFiles(new FileFilter() {
#Override
public boolean accept(File pathname) {
return pathname.isFile() && pathname.getName().endsWith(".log");
}
});
}
static String cat(final File[] files) throws IOException {
Enumeration<InputStream> e = new Enumeration<InputStream>() {
int index;
#Override
public boolean hasMoreElements() {
return index < files.length;
}
#Override
public InputStream nextElement() {
index++;
try {
return new FileInputStream(files[index - 1]);
} catch (FileNotFoundException ex) {
throw new RuntimeException("File not available!", ex);
}
}
};
SequenceInputStream input = new SequenceInputStream(e);
StringBuilder sb = new StringBuilder();
int c;
while ((c = input.read()) != -1) {
sb.append((char) c);
}
return sb.toString();
}
public static void main(String[] args) throws IOException {
String dir = "<path-to-dir-with-logs>";
File[] logs = logs(dir);
for (File f : logs) {
System.out.println(f.getAbsolutePath());
}
System.out.println();
System.out.println(cat(logs));
}

Loop only deletes first file

I'm having a problem getting all the selected files deleted. What I'm trying to do is after clicking "Add" whatever files are selected are moved to a new folder and are deleted in their previous folder. One file works fine. It deletes and moves the file. But more than one and only the first gets deleted. My loop is recognizing each file just not deleting them. I'm posting the actionevent. If more code is needed let me know. I've indicated where the problem is, so I think, so you don't have to search the code.
public void actionPerformed(ActionEvent e) {
int returnValue = 0;
int option = 0;
File[] selectedFiles = new File[0];
if (e.getActionCommand().equals("CLOSE")) System.exit(0);
else if (e.getActionCommand().equals("ADD")) {
JFileChooser chooser = new JFileChooser();
chooser.setMultiSelectionEnabled(true);
returnValue = chooser.showOpenDialog(this);
if (returnValue == JFileChooser.APPROVE_OPTION) {
File[] file = chooser.getSelectedFiles();
try {
FileInputStream fstream = null;
FileOutputStream ostream = null;
for (int i = 0; i < file.length; i++) {
fstream = new FileInputStream(file[i]);
ostream = new
FileOutputStream(file[i].getName());
Path path = Paths.get(file[i].getPath());
byte[] fileArray;
fileArray = Files.readAllBytes(path);
listModel.add(0, file[i].getName());
selectedFilesList.setModel(listModel);
//ostream.write(fileArray, 0, fileArray.length);
}
fstream.close();
//ostream.close();
try {
for(int i = 0; i < file.length; i++) {
//**----------------------->>>PROBLEM**
Files.delete(Paths.get(file[i].getPath()));
System.out.println(file[i].getName());
}
} catch (NoSuchFileException x) {}
System.err.format("%s: no such" + " file or directory%n")
} catch (DirectoryNotEmptyException x) {
System.err.format("%s not empty%n");
} catch (IOException x) {
// File permission problems are caught here.
System.err.println(x);
} catch (Exception err) {
}
}

This could be caused by you failing to close your file streams in the first loop.
for (int i = 0; i < file.length; i++) {
fstream = new FileInputStream(file[i]);
ostream = new
FileOutputStream(file[i].getName()); // This is never closed
Path path = Paths.get(file[i].getPath());
byte[] fileArray;
fileArray = Files.readAllBytes(path);
listModel.add(0, file[i].getName());
selectedFilesList.setModel(listModel);
//ostream.write(fileArray, 0, fileArray.length);
}
fstream.close(); // Only the last input stream is closed
should be more like
for (int i = 0; i < file.length; i++) {
try {
fstream = new FileInputStream(file[i]);
ostream = new
FileOutputStream(file[i].getName());
Path path = Paths.get(file[i].getPath());
byte[] fileArray;
fileArray = Files.readAllBytes(path);
listModel.add(0, file[i].getName());
selectedFilesList.setModel(listModel);
//ostream.write(fileArray, 0, fileArray.length);
} finally {
fstream.close();
ostream.close();
}
}
Closing the same number of files you open.
This could be causing your problem by holding a lock on all but one of your files which would prevent deletion.
Also your catch exception block (last statement) does nothing with the error.

Don't move files like that!
If you are on Java 7, have at look at this page instead.
For older versions, use oldFile.renameTo(newFile).
EDIT: To understand why your code is not working, use a debugger. I would think your deletion loop is left because of an exception.

Trouble with java snippets

public class GenericWorldLoader implements WorldLoader {
#Override
public LoginResult checkLogin(PlayerDetails pd) {
Player player = null;
int code = 2;
File f = new File("data/savedGames/" + NameUtils.formatNameForProtocol(pd.getName()) + ".dat.gz");
if(f.exists()) {
try {
InputStream is = new GZIPInputStream(new FileInputStream(f));
String name = Streams.readRS2String(is);
String pass = Streams.readRS2String(is);
if(!name.equals(NameUtils.formatName(pd.getName()))) {
code = 3;
}
if(!pass.equals(pd.getPassword())) {
code = 3;
}
} catch(IOException ex) {
code = 11;
}
}
if(code == 2) {
player = new Player(pd);
}
return new LoginResult(code, player);
}
#Override
public boolean savePlayer(Player player) {
try {
OutputStream os = new GZIPOutputStream(new FileOutputStream("data/savedGames/" + NameUtils.formatNameForProtocol(player.getName()) + ".dat.gz"));
IoBuffer buf = IoBuffer.allocate(1024);
buf.setAutoExpand(true);
player.serialize(buf);
buf.flip();
byte[] data = new byte[buf.limit()];
buf.get(data);
os.write(data);
os.flush();
os.close();
return true;
} catch(IOException ex) {
return false;
}
}
#Override
public boolean loadPlayer(Player player) {
try {
File f = new File("data/savedGames/" + NameUtils.formatNameForProtocol(player.getName()) + ".dat.gz");
InputStream is = new GZIPInputStream(new FileInputStream(f));
IoBuffer buf = IoBuffer.allocate(1024);
buf.setAutoExpand(true);
while(true) {
byte[] temp = new byte[1024];
int read = is.read(temp, 0, temp.length);
if(read == -1) {
break;
} else {
buf.put(temp, 0, read);
}
}
buf.flip();
player.deserialize(buf);
return true;
} catch(IOException ex) {
return false;
}
}
}
Yeah so... My problem is that this seems to save 'something' in really complex and hard to read way(binary) and I'd rather have it as an .txt, in easily readable format. how to convert?
EDIT: I'm not using Apache Mina, so what should I replace
IoBuffer buf = IoBuffer.allocate(1024);
buf.setAutoExpand(true);"
with?

checkLogin() obviously checks whether the specified login has matching data present and whether the password is correct.
savePlayer() method saves the player.
loadPlayer() loads it again.
The data format used is gzip (wiki) and it is written as a stream of serialized data. If you want to make it more readable, you might want to overload (or just use it, if it is good) toString() method of Player class and to write player.toString() into a new text file using e.g. BufferedWriter wrapped around a File Writer:
String playerName = NameUtils.formatNameForProtocol(player.getName());
BufferedWriter writer = new BufferedWriter(new FileWriter(playerName + ".txt"));
writer.write(player.toString());
writer.close();