We Keep Coding

Find MD5 hash of files inside a tar.gz file in java without extracting it

I have a huge tar.gz file with lots of images in it. I need to find the md5 hash of each images. I am not able to find hash of images inside the tar file but same code works for normal folders and images. Is there any way to find hash without extracting the tar?
public static String digestAndBuildImageEntry(Path filePath) throws NoSuchAlgorithmException {
try (InputStream is = Files.newInputStream(filePath);
BufferedInputStream buffered = new BufferedInputStream(is)) {
byte[] data = Files.readAllBytes(filePath);
byte[] hashByte = MessageDigest.getInstance("MD5").digest(data);
String hash = hashByte.toString();
return hash;
} catch (Exception ex) {
return null;
}
}
I get below exception when i run this code
Caused by: java.nio.file.FileSystemException: /Users/myuser/old/file.tar.gz/1.jpg: Not a directory
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at java.nio.file.Files.readAttributes(Files.java:1737)
at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
at java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:72)
at java.nio.file.Files.walk(Files.java:3574)
at java.nio.file.Files.walk(Files.java:3625)
at com.example.demo.ImageDeduplication.listFiles(ImageDeduplication.java:78)
at com.example.demo.SparkSQL.lambda$1(SparkSQL.java:82)
at org.apache.spark.sql.UDFRegistration.$anonfun$register$352(UDFRegistration.scala:775)
... 17 more
Below Path variables worked
/Users/myuser/old/1.jpg - worked
/Users/myuser/old/ - able to iterate and get all file inside the folder
/Users/myuser/old/file.tar.gz - gives the hash of the entire tar file
Not working for
/Users/myuser/old/file.tar.gz/1.jpg - says not a directory

Apache Commons Compress has classes that can stream tar.gz format.
From examples and docs it would be something like this:
try (InputStream fi = Files.newInputStream(Paths.get("my.tar.gz"));
InputStream bi = new BufferedInputStream(fi);
InputStream gzi = new GzipCompressorInputStream(bi);
TarArchiveInputStream tarInput = new TarArchiveInputStream(gzi)) {
TarArchiveEntry entry = tarInput.getNextTarEntry();
// here you can read specific file's content and do md5 computation
byte[] content = new byte[entry.getSize()];
tarInput.read(content, offset, content.length - offset);
}
Another option to quickly access files inside of tar.gz is to mount it as virtual file system by commons-vfs

Latest version of common compress has TarFile class which provides random access to the files and inputstream. We can get the TarArchiveEntry of each files as a list and get the corresponding inputstream from the exposed method in TarFile class. Below code worked for me.
public static Map<String,String> getMD5HashMap(String path) throws Exception {
Map<String,String> map = new ConcurrentHashMap<>();
FileInputStream in = new FileInputStream(path);
GzipCompressorInputStream gzIn = new GzipCompressorInputStream(in);
byte[] bytes = IOUtils.toByteArray(gzIn);
TarFile tarFile = new TarFile(bytes);
for(TarArchiveEntry tarArchiveEntry:tarFile.getEntries()) {
if(tarArchiveEntry.isFile()) {
try(InputStream is = tarFile.getInputStream(tarArchiveEntry);
BufferedInputStream buffered = new BufferedInputStream(is)){
String hash = DigestUtils.md5Hex(buffered);
map.put(tarArchiveEntry.getName(), hash);
System.out.println(hash);
}
}
}
return map;
}

org.tukaani.xz.CorruptedInputException: Compressed data is corrupt while extracting 7z archive File in java

Getting org.tukaani.xz.CorruptedInputException: Compressed data is corrupt error while extracting 7z file. i am using common-compress-1.21.jar and xz-1.9.jar.here is my code
File f = new File("/home/local/Documents/7zFile/7zFile.7z");
InputStream inputStream;
ArchiveEntry entry = null;
try {
inputStream = new FileInputStream(f);
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
if (!inputStream.markSupported()) {
inputStream = new BufferedInputStream(inputStream);
}
inputStream.mark(Integer.MAX_VALUE);
byte[] inputData = IOUtils.toByteArray(inputStream);
inputStream.reset();
SeekableInMemoryByteChannel channel = new
SeekableInMemoryByteChannel(inputData);
sevenZFile = new SevenZFile(channel);
while ((sevenZFile!=null && (entry = sevenZFile.getNextEntry()) != null)) {
System.out.print(entry.getName());
}
stackTrace of error
org.tukaani.xz.CorruptedInputException: Compressed data is corrupt
at org.tukaani.xz.LZMAInputStream.read(Unknown Source)
at org.apache.commons.compress.utils.ChecksumVerifyingInputStream.read(ChecksumVerifyingInputStream.java:88)
at org.apache.commons.compress.utils.IOUtils.copyRange(IOUtils.java:330)
at org.apache.commons.compress.utils.IOUtils.copyRange(IOUtils.java:301)
at org.apache.commons.compress.utils.IOUtils.readRange(IOUtils.java:350)
at org.apache.commons.compress.archivers.sevenz.SevenZFile.readEncodedHeader(SevenZFile.java:706)
at org.apache.commons.compress.archivers.sevenz.SevenZFile.initializeArchive(SevenZFile.java:544)
at org.apache.commons.compress.archivers.sevenz.SevenZFile.readHeaders(SevenZFile.java:474)
at org.apache.commons.compress.archivers.sevenz.SevenZFile.<init>(SevenZFile.java:343)
at org.apache.commons.compress.archivers.sevenz.SevenZFile.<init>(SevenZFile.java:255)
at org.apache.commons.compress.archivers.sevenz.SevenZFile.<init>(SevenZFile.java:183)
at org.apache.commons.compress.archivers.sevenz.SevenZFile.<init>(SevenZFile.java:167)
in this line sevenZFile = new SevenZFile(channel); its throwing error.
i have created 7z archive file in ubuntu OS for this file its throwing exception, the 7z archive which i created online website its working fine. is there any issue with ubuntu? and how to fix this?
Thanks in advance.

The LZMA data has an end of stream marker. Such .7z files are valid but quite uncommon. XZ for Java 1.9 added a feature specifically. please find this link for better understanding. https://issues.apache.org/jira/browse/COMPRESS-591
https://sourceforge.net/p/lzmautils/discussion/708858/thread/822d80d5ea/

I want to TAR a multiple files on JBOSS server in Eclipse using java

I have tried multiple examples... But non will work for me
Make tar file by Java
If I use above code, After reading the line TarOutputStream out = new TarOutputStream( new BufferedOutputStream( dest ) ); it just come out of the method(i.e.After reading that line... it won't executed the code which are after to that. The Sequence directly gone to the finally block where that method called).
For another example, I used apache common compress Jar. In that case it won't execute the java class itself it exit from the FormBean class.
The files are fetched from the database and it will be in a CSV format and all the files are placed in an Array List.
fileNames = "Security.smod."+SYSDATE;
strSavePath = D\:\\CMSREPORT\\reportZip\\
tarFunction(fileList, strSavePath, fileNames);
//Example function 1
private String tarFunction(ArrayList fileList, String strSavePath, String outFileName) {
int fileCountSize = fileList.size();
// Output file stream
FileOutputStream dest = new FileOutputStream( strSavePath );
// Create a TarOutputStream
TarOutputStream out = new TarOutputStream( new BufferedOutputStream( dest ) );
// Files to tar
File[] filesToTar=new File[3];
for(int i=0; i<fileCountSize; i++)
{
filesToTar[i]=new File((String) fileList.get(i);
}
for(File f:filesToTar){
out.putNextEntry(new TarEntry(f, f.getName()));
BufferedInputStream origin = new BufferedInputStream(new FileInputStream( f ));
int count;
byte data[] = new byte[2048];
while((count = origin.read(data)) != -1) {
out.write(data, 0, count);
}
out.flush();
origin.close();
}
}
For Example 2, I used apache.common.compress.1.2 as a jar.
In that case, It won't execute the java class, it exited from the FormBean class.
//Example function 2
private void tarFunc(){
try{
// Files to tar
File resource1 = new File((String) fileList.get(0));
File resource2 = new File((String) fileList.get(1));
File resource3 = new File((String) fileList.get(2));
// Output Stream - that will hold the physical TAR file
OutputStream tar_output = new FileOutputStream(new File(fileNames+".tar"));
// Create Archive Output Stream that attaches File Output Stream / and specifies TAR as type of compression
ArchiveOutputStream my_tar_ball = new ArchiveStreamFactory().createArchiveOutputStream(ArchiveStreamFactory.TAR, tar_output);
// Create Archieve entry - write header information
TarArchiveEntry tar_file = new TarArchiveEntry(resource1);
// length of the TAR file needs to be set using setSize method
tar_file.setSize(resource1.length());
my_tar_ball.putArchiveEntry(tar_file);
IOUtils.copy(new FileInputStream(resource1), my_tar_ball);
// Close Archieve entry, write trailer information
my_tar_ball.closeArchiveEntry();
// Repeat steps for the next file that needs to be added to the TAR
tar_file = new TarArchiveEntry(resource2);
tar_file.setSize(resource2.length());
my_tar_ball.putArchiveEntry(tar_file);
IOUtils.copy(new FileInputStream(resource2), my_tar_ball);
// Close Archieve entry, write trailer information
my_tar_ball.closeArchiveEntry();
tar_file = new TarArchiveEntry(resource3);
tar_file.setSize(resource3.length());
my_tar_ball.putArchiveEntry(tar_file);
IOUtils.copy(new FileInputStream(resource3), my_tar_ball);
// Close Archieve entry, write trailer information
my_tar_ball.closeArchiveEntry();
my_tar_ball.finish();
// Close output stream, our files are zipped
tar_output.close();
}catch(Exception e){
e.printStackTrace();
}
}
Anyone please help to solve this issue. Thank you:)

Saving files received through DatagramPackets in Java

I am trying to send Files in fragments using DatagramPackets in Java (part of an assignemt.) When I am trying to save the incoming File I get access denied error, but I believe that it is not a permissions issue.
Here is the brunt of it:
I let the user sending the file to choose it using FileChooser. And create a new Message object.
//....
File f = content.showFileChooser();
byte type = Byte.parseByte("4");
Message m;
try {
if (mode == 1){
m = new Message(f, content.getServerData().getFragmentSize(), (short) (sentMessages.size()+1), type);
serverThread.send(m);
}
//...
During Message creation the file gets split up into byte arrays, where the size of each array is predetermined by the user. The code is quite lengthy so I am not going to post the chopping process, but this is how I convert the File object into a big byte[] which then gets chopped up
Path p = Paths.get(file.getAbsolutePath());
this.rawData = Files.readAllBytes(p);
After the Message is created and chopped up into byte arrays I send them using DatagramPackets. The other side then uses those to create a new Message object. Once all fragments arrive rawData is extracted from the Message object again. The problem believe lies here:
Message m = receivedMessages.get(msgIndex-1);
byte[] fileData = m.getFile();
if (fileData != null){
System.out.println("All file fragments received.");
content.append("Received a file in" + m.getFragmentCount()+" fragments. Choose directory. " ,1);
//I BELIEVE THIS TO BE THE CRITICAL POINT
String filePath = content.chooseDirectory();
if (filePath == null)
return;
FileOutputStream fos;
try {
fos = new FileOutputStream(filePath);
fos.write(fileData);
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Once all fragments arrive I let the user select a directory using FileChooser with DIRECTORY_ONLY choice mode. As I understand, FileOutputStream requires a full path for the new File. Do I have to send the file name and extension separately or can it be extracted from the received File data?

You are writing directory path to filePath, then try to open that directory with FileOutputStream. No wonder that doesn't work, you have to specify the filename too.
String filename = "myfile.txt"; //Or receive and set this some other way
File outFile = new File(filePath, filename);
fos = new FileOutputStream(outFile);
I don't see you sending/receiving the filename anywhere, though. You'll need to either make it constant or transfer it along with file contents.

How to determine the compression method of a zip file

From a third party I am retrieving .zip files. I want to unzip these to another folder. To this end I found a method that does exactly that, see code below. It iterates through all files and unzips them to another folder. However, when observing the corresponding compression method I found out that this changes for some files. And for some files it states: "invalid compression method", after which it aborts further unzipping of the zip file.
As the compression method seems to change, I suspect I need to set the compression method to the correct one (however that might be a wrong assumption). So rises my question: how to determine the compression method needed?
The code I am using:
public void unZipIt(String zipFile, String outputFolder){
//create output directory is not exists
File folder = new File(OUTPUT_FOLDER);
if(!folder.exists()){
folder.mkdir();
}
FileInputStream fis = null;
ZipInputStream zipIs = null;
ZipEntry zEntry = null;
try
{
fis = new FileInputStream(zipFile);
zipIs = new ZipInputStream(new BufferedInputStream(fis));
while((zEntry = zipIs.getNextEntry()) != null){
System.out.println(zEntry.getMethod());
try{
byte[] tmp = new byte[4*1024];
FileOutputStream fos = null;
String opFilePath = OUTPUT_FOLDER + "\\" + zEntry.getName();
System.out.println("Extracting file to "+opFilePath);
fos = new FileOutputStream(opFilePath);
int size = 0;
while((size = zipIs.read(tmp)) != -1){
fos.write(tmp, 0 , size);
}
fos.flush();
fos.close();
} catch(IOException e){
System.out.println(e.getMessage());
}
}
zipIs.close();
} catch (FileNotFoundException e) {
System.out.println(e.getMessage());
}
catch(IOException ex){
System.out.println(ex.getMessage());
}
}
Currently I am retrieving the following output:
8
Extracting file to C:\Users\nlmeibe2\Documents\Projects\Output_test\SOPHIS_cptyrisk_tradedata_1192_20140616.csv
8
Extracting file to C:\Users\nlmeibe2\Documents\Projects\Output_test\SOPHIS_cptyrisk_underlying_1192_20140616.csv
0
Extracting file to C:\Users\nlmeibe2\Documents\Projects\Output_test\10052013/
12
Extracting file to C:\Users\nlmeibe2\Documents\Projects\Output_test\MRM_Daily_Position_Report_Package_Level_Underlying_View_EQB_v2_COBDATE_2014-06-16_RUNDATETIME_2014-06-17-04h15.csv
invalid compression method
invalid compression method

Since you only print the exception message and not the stack trace (with line numbers), it is impossible to know exactly where the exception is thrown, but I suppose it is not thrown until you actually try to read from the ZipEntry.
If the numbers in your output is the ZIP method, the last entry you encounter is compressed with method 12 (bzip2), which is not supported by the Java ZIP implementation. PKWare (the maintainers of the ZIP format) regularly add new compression methods to the ZIP specification and there are currently some 12-15 (not sure about the exact number) compression methods specified. Java only supports the methods 0 (stored) and 8 (deflated) and will throw an exception with the message "invalid compression method" if you try to decompress a ZIP file using an unsupported compression method.
Both WinZip and the ZIP functions in Windows may use compression methods not supported by the Java API.

Use zEntry.getMethod() to get the compression method
Returns the compression method of the entry, or -1 if not specified.
It will return an int which will be
public static final int STORED
public static final int DEFLATED
or -1 if it don't know the method.
Docs.