I use the jar command to build jar files. While trying to cache the jar files using md5 signatures, I found that jars built from the exact same sources had different md5 signatures.
Upon closer inspection, I found that every time the jar was created the contents were exactly the same (diff -qr was empty). It turns out that the timestamp of creation is encoded in the jar file which throws off the md5 signature. Other people have discovered the same here.
There is even a blog post on how to create jar files identically each time with maven. However, I want a simple solution using the command line using readily available commands such as jar and zip (may have to do this on a server without install permissions), possibly leading to the same "functional" jar as I'm currently getting using jar command.
EDIT: For my purpose, it also suffices to quickly find the md5 so that it is the same across builds, even if the jars are not identical. The only way I found so far is to extract the files in the jar and to md5 all component files. But I'm afraid that is slow for bigger jars and is going to defeat the purpose of caching them to avoid building them in the first place. Is there a better and faster solution?
The main issue is jar command always create META-INF\MANIFEST.MF with current time.
The file time is saved in zip entry header. This is why MD5 value is different even all file content in jar remain the same:
the different zip entry headers produce different zip file.
For jar command, the only solutionis is option -M: not to create a manifest file for the entries.
Jar command always create META-INF\MANIFEST.MF with current time. Zip stores files with timestamp and file attributes due to which sha256 or MD5 will be different for two artifacts.
We need to make sure that created, last modified, accessed timestamp and file attributes are always same of all files which are required to create jar or zip.
I have created below script which can take a jar or zip file and make it deterministic by making timestamp constant and setting the right compression level and offset.
#!/bin/bash
usage() {
echo "Usage : ./createDeterministicArtifact.sh <zip/jar file name>"
exit 1
}
info() {
echo "$1"
}
strip_artifact() {
if [ -z ${file} ]; then
usage
fi
if [ -f ${file} -a -s ${file} ]; then
mkdir -p ${file}.tmp
unzip -oq -d ${file}.tmp ${file}
find ${file}.tmp -follow -exec touch -a -m -t 201912010000.00 {} \+
if [ "$UNAME" == "Linux" ] ; then
find ${file}.tmp -follow -exec chattr -a {} \+
elif [[ "$UNAME" == CYGWIN* || "$UNAME" == MINGW* ]] ; then
find ${file}.tmp -follow -exec attrib -A {} \+
fi
cd ${file}.tmp
zip -rq -D -X -9 -A --compression-method deflate ../${file}.new .
cd -
rm -rf ${file}.tmp
info "Recreated deterministic artifact: ${file}.new"
else
info "Input file is empty. Please validate the file and try again"
fi
}
file=$1
Related
I have folders with lots (20) of jar files. Is there a way to extract all those jars in one command in the terminal instead doing it one by one?
I'm using a MAC.
A simple solution.- Get all the jars and extract it
find ./ -name "*.jar" -exec jar -xf {} \;
You can use this from the folder in which all your jars are -
jar {ctxu}[vfm0Mi] [jar-file] [manifest-file] [-C dir]
Options:
-c create new archive
-t list table of contents for archive
-x extract named (or all) files from archive
-u update existing archive
-v generate verbose output on standard output
-f specify archive file name
-m include manifest information from specified manifest file
-0 store only; use no ZIP compression
-M do not create a manifest file for the entries
-i generate index information for the specified jar files
-C change to the specified directory and include the following file
refer the documentation for explaination - http://docs.oracle.com/javase/tutorial/deployment/jar/unpack.html
you may user *.jar in place of jar-file to extract all
This should extract in same location where the jars are present.
jar xvf *.jar
I want to compile a basic java project, https://sourceforge.net/projects/pdfformfiller2 its installation instructions are quite short:
Make sure that iText library, itext-xtra-5.x.0.jar and itextpdf-5.x.0.jar, are accessible to JAVA,
e.g. these are placed in the "lib" subfolder of the current folder.
Get latest ones from: https://sourceforge.net/projects/itext/files/iText/
Compile PdfFormFiller.java
Then from the command line you give command (to see usage help):
java -jar pdfformfiller.jar
I never compiled jars before, and I'm having hard time trying to compile PdfFormFiller correctly. Here's where I've get:
wget -O pdfformfiller.zip https://sourceforge.net/projects/pdfformfiller2/files/latest/download
# author mentions 5.2.0, which is not available anymore, so we go for the latest 5.x:
wget http://kent.dl.sourceforge.net/project/itext/5.5.10/itext5-5.5.10.zip
unzip pdfformfiller.zip
unzip itext5-5.5.10.zip -d pdfformfiller/lib
cd pdfformfiller
javac -cp "lib/*" PdfFormFiller.java
mkdir META-INF
echo -e 'Manifest-Version: 1.0\nClass-Path: pdfformfiller.jar\nMain-Class: PdfFormFiller' > META-INF/MANIFEST.MF
jar -cvfm pdfformfiller.jar META-INF/MANIFEST.MF lib PdfFormFiller.class
Which succeeds without an error, but still doesn't run:
$ java -jar pdfformfiller.jar
Error: Could not find or load main class PdfFormFiller
I guess I'm missing something trivial?
Edit
Complete automation:
iText5=5.5.10
wget -O pdfformfiller.zip https://sourceforge.net/projects/pdfformfiller2/files/latest/download
wget http://kent.dl.sourceforge.net/project/itext/${iText5}/itext5-${iText5}.zip
unzip pdfformfiller.zip
unzip itext5-${iText5}.zip -d pdfformfiller/lib
cd pdfformfiller
mkdir classes
javac -cp "lib/*" -d ./classes/ PdfFormFiller.java
mkdir META-INF
echo 'Manifest-Version: 1.0' > META-INF/MANIFEST.MF
echo "Class-Path: ./lib/itextpdf-${iText5}.jar ./lib/itext-xtra-${iText5}.jar ./lib/itext-pdfa-${iText5}.jar" >> META-INF/MANIFEST.MF
echo 'Main-Class: PdfFormFiller.PdfFormFiller' >> META-INF/MANIFEST.MF
jar -cvfm pdfformfiller.jar ./META-INF/MANIFEST.MF ./lib -C ./classes/ PdfFormFiller
Edit 2
It seems to be the only way to fill pdf form from CLI reliably:
# list fields in a file:
$ java -jar pdfformfiller.jar input.pdf -l
myfield
# prepare field data:
$ echo 'myfield αβγ' > fields
# specify font, fill the fields, flatten the form:
$ java -jar pdfformfiller.jar input.pdf -f fields -font Times_New_Roman.ttf -flatten output.pdf
Works like a charm!
Here are the steps I followed to get it working.
First of all, just for the sake of clarity, let's create a dedicated folder for your compiled classes. It's not mandatory, but just an example of good development practice. I'm omitting the steps of creating folders, changing dirs etc. because it's quite obvious. All commands are run from the project's root directory
javac -cp "lib/*" -d ./classes/ PdfFormFiller.java
Fixing the two main things that were missed:
a) the reference for required lib folder and
b) package name:
echo -e 'Manifest-Version: 1.0\nClass-Path: ./lib/itextpdf-5.5.4.jar ./lib/itext-xtra-5.5.4.jar ./lib/itext-pdfa-5.5.4.jar\nMain-Class: PdfFormFiller.PdfFormFiller' > META-INF/MANIFEST.MF
Assembling jar (please note that additional option: -C is being used here):
jar -cvfm pdfformfiller.jar ./META-INF/MANIFEST.MF ./lib -C ./classes/ PdfFormFiller
This is the final output from executing the resulting jar file:
$ java -jar pdfformfiller.jar
USAGE: pdfformfiller document.pdf [ -l ] [ -v ] [ -f fields_filename ] [ -font font_file ] [ -flatten] [ output.pdf ]
document.pdf - name of source pdf file (required).
-l - only list available fields in document.pdf.
-v - verbose. Use to debug the fields_filename file.
-f fields_filename - name of file with the list of fields values to apply to document.pdf.
if ommited, stdin is used.
-font font_file - font to use. Needed UTF-8 support, e.g. cyrillic and non-latin alphabets.
-flatten - Flatten pdf forms (convert them to text disabling editing in PDF Reader).
output.pdf - name of output file. If omitted, the output if sent to stdout.
fields_filename file can be in UTF-8 as is of the following format:
On each line, one entry consists of 'field name' followed by value of that field without any quotes.
Any number of whitespaces allowed before 'field name', and one space separates 'field name' and its value.
In value, newline characters should be encoded as "\n",
'U+2029 utf-8 E280A9 : PARAGRAPH SEPARATOR PS' should be encoded as "\p",
and '\' characters should be escaped as "\\".
For checkboxes, values are 'Yes'/'Off'.
Based on the Belgian iText library v. 5.2.0, http://www.itextpdf.com/
I have 2 files, "launcher.jar" and "camStudio.jar" that I need merged. I decided to try to do this using batch with the code:
copy /b launcher.jar + camStudio.jar file.jar
However, the resulting "file.jar" only contains the contents of "camStudio.jar". How can I prevent the files in "launcher.jar" from being deleted?
Combining the contents of two .jar files is a little more complicated than just calling copy from the command line. Rather than a normal directory, .jar files are a type of compressed file, so you need special utilities to manipulate them. Fortunately these tools come with the standard JKD.
The JDK comes with the utility jar that is unsurprisingly used for manipulating .jar files. It's usage is described as this:
Usage: jar {ctxui}[vfmn0Me] [jar-file] [manifest-file] [entry-point] [-C dir] files ...
Options:
-c create new archive
-t list table of contents for archive
-x extract named (or all) files from archive
-u update existing archive
-v generate verbose output on standard output
-f specify archive file name
-m include manifest information from specified manifest file
-n perform Pack200 normalization after creating a new archive
-e specify application entry point for stand-alone application
bundled into an executable jar file
-0 store only; use no ZIP compression
-M do not create a manifest file for the entries
-i generate index information for the specified jar files
-C change to the specified directory and include the following file
If any file is a directory then it is processed recursively.
The manifest file name, the archive file name and the entry point name are
specified in the same order as the 'm', 'f' and 'e' flags.
Example 1: to archive two class files into an archive called classes.jar:
jar cvf classes.jar Foo.class Bar.class
Example 2: use an existing manifest file 'mymanifest' and archive all the
files in the foo/ directory into 'classes.jar':
jar cvfm classes.jar mymanifest -C foo/ .
Relevant commands for combining two .jar files are x and c. Even with this, combining the .jar files takes more than a line or two, so I put together this .bat files to automate it.
:: Pass one or more .jar files as command line arguments
:: Combine_Jar [file1] [file2 ...]
:: Combine_Jar Test.jar
:: Combine_Jar Test.jar Test2.jar Test3.jar
#echo off & setlocal enabledelayedexpansion
set "jarDir=%cd%"
set "newJar="
set "folders="
pushd %temp%
for %%a in (%*) do (
call :extract %%a
set "newJar=!newJar!_%%~na_"
)
set "tempDirs=!newJar:_=^"!"
set "tempDirs=%tempDirs:^"^"=^" ^"%"
set "newJar=!newJar:~1,-1!.jar"
set "newJar=!newJar:__=_!"
if exist "!newJar!" del /Q "!newJar!"
jar cf "!newJar!" %tempDirs%
for %%a in (%*) do call rd /s /q "%%~na"
move /Y "!newJar!" "%jarDir%" > nul
popd
exit /B
:extract
set "tempDir=%~n1"
if exist "%tempDir%" (
rd /s /q "%tempDir%"
)
md "%tempDir%"
pushd "%tempDir%"
jar xf "%jarDir%\%~1"
popd
exit /B
It will all jar files passed as arguments into a single jar files.
I need to make a makefile for a Java project.
My project is basic. A package which contains my main file and some others packages.
Can someone help me to make a proper makefile for that kind of project ?
Sorry I have not tested so it is likely to fail...
Assuming that what you need is to generate a executable jar file the following should work but I have not tested it.
The Makefile below assumes that your sources are located under ./src and that you are happy to use ./build for intermediary files (which is totally obliterated by the clean target so be careful).
Then 'make' or 'make jar' should generate the jar file.
NAME=MyProject
MAIN_CLASS=MyMainClass
SRC_DIR=./src
CLS_DIR=./build/classes
MANIFEST_FILE=./build/META-INF.MF
JAR_FILE=./$(NAME).jar
SRC_FILES=$(shell find $(SRC_DIR) -iname "*.java")
CLS_FILES=$(patsubst $(SRC_DIR)/%,$(CLS_DIR)/%,$(patsubst %.java,%.class,$(SRC_FILES)))
.PHONY: jar run clean mrproper
jar : $(JAR_FILE)
run : $(JAR_FILE)
java -jar $(JAR_FILE)
$(JAR_FILE) : $(MANIFEST_FILE) $(CLS_FILES)
jar cmf $< $# $(CLS_DIR)
$(MANIFEST_FILE) :
mkdir -p $(dir $#)
echo Main-Class: $(MAIN_CLASS) > $#
$(CLS_DIR) :
mkdir -p $(CLS_DIR)
$(CLS_DIR)/%.class : $(SRC_DIR)/%.java $(CLS_DIR)
javac -d $(CLS_DIR) -sourcepath $(SRC_DIR) $<
clean :
rm -Rf ./build
mrproper : clean
rm -f $(JAR_FILE)
Each time you execute Make it performs a find operation to get the list of source files, that might cause some delay depending on how many files and how fast is the file-system.... but you said is is a small project so it should not be an issue.
The double patsubst might well be compressed into a single one but I did it in two steps just in case.
Also notice that this solution compiles each Java class separately. This can be quite costly and it might be advisable to have another target to compile all at once ideally by creating a file that contains the name of all src java files and passing it to javac.
How do I compare two .jar files?
Both of them have compiled .class files.
I want the difference in terms of method changes, etc.
JAPICC, sample usage:
japi-compliance-checker OLD.jar NEW.jar
Sample reports for log4j: http://abi-laboratory.pro/java/tracker/timeline/log4j/
PkgDiff, sample usage:
pkgdiff OLD.jar NEW.jar
See sample report for args4j.
Clirr, sample usage:
java -jar clirr-core-0.6-uber.jar -o OLD.jar -n NEW.jar
If you select two files in IntellijIdea and press Ctrl + Dthen it will show you the diff. I use Ultimate and don't know if it will work with Community edition.
Rename .jar to .zip
Extract
Decompile class files with jad
Recursive diff
Extract each jar to it's own directory using the jar command with parameters xvf. i.e. jar xvf myjar.jar for each jar.
Then, use the UNIX command diff to compare the two directories. This will show the differences in the directories. You can use diff -r dir1 dir2 two recurse and show the differences in text files in each directory(.xml, .properties, etc).
This will also show if binary class files differ. To actually compare the class files you will have to decompile them as noted by others.
Create a folder and create another 2 folders inside it like old and new. add relevant jar files to the folders. then open the first folder using IntelliJ. after that click whatever 2 files do you want to compare and right-click and click compare archives.
I use to ZipDiff lib (have both Java and ant API).
Here is my script to do the process described by sje397:
#!/bin/sh
# Needed if running on Windows
FIND="/usr/bin/find"
DIFF="diff -r"
# Extract the jar (war or ear)
JAR_FILE1=$1
JAR_FILE2=$2
JAR_DIR=${PWD} # to assign to a variable
TEMP_DIR=$(mktemp -d)
echo "Extracting jars in $TEMP_DIR"
EXT_DIR1="${TEMP_DIR}/${JAR_FILE1%.*}"
EXT_DIR2="${TEMP_DIR}/${JAR_FILE2%.*}"
mkdir ${EXT_DIR1}
cd ${EXT_DIR1}
jar xf ${JAR_DIR}/${JAR_FILE1}
jad -d . -o -t2 -safe -space -b -ff -s java -r **/*.class
cd ..
mkdir ${EXT_DIR2}
cd ${EXT_DIR2}
jar xf ${JAR_DIR}/${JAR_FILE2}
jad -d . -o -t2 -safe -space -b -ff -s java -r **/*.class
cd ..
# remove class files so the diff is clean
${FIND} ${TEMP_DIR} -name '*.class' | xargs rm
# diff recursively
${DIFF} ${EXT_DIR1} ${EXT_DIR2}
I can run it on Windows using GIT for Windows. Just open a command prompt. Run bash and then execute the script from there.
Use Java Decompiler to turn the jar file into source code file, and then use WinMerge to perform comparison.
You should consult the copyright holder of the source code, to see whether it is OK to do so.
In Linux/CygWin a handy script I use at times is:
#Extract the jar (war or ear)
cd dir1
jar xvf jar-file1
for i in `ls *.class`
do
javap $i > ${i}.txt #list the functions/variables etc
done
cd dir2
jar xvf jar-file2
for i in `ls *.class`
do
javap $i > ${i}.txt #list the functions/variables etc
done
diff -r dir1 dir2 #diff recursively
If you are using IntelliJ IDEA or Android Studio, add your jar files to a project under the libs folder.
Then select the both jar files, right click then select "Compare Archives"
use java decompiler and decompile all the .class files and save all files as project structure .
then use meld diff viewer and compare as folders ..
Here's an aparently free tool http://www.extradata.com/products/jarc/
Please try http://www.osjava.org/jardiff/ - tool is old and the dependency list is large. From the docs, it looks like worth trying.
This application may be what you need, works great and display a simple GUI showing differences. Try Jarcomp