I'm writing a custom tool to build jar files from our build tree. We like to build "minimal" jar files that contain only the .class files actually referenced from the "root" class (where main() lives) -- and so on, recursively following dependencies.
Historically we have done this by getting javac to follow source dependencies, but that means recompiling common files many many times. (We build 60 or 70 distinct application jars from a single source tree.) I'm writing a new build system that compiles each source file only once, but that means we need to follow dependencies by parsing .class files.
The good news is, I've got working code that does what I want. But I need to be absolutely sure I didn't screw it up, i.e. I want to ensure that I'm building internally consistent jar files, where "consistent" means that all unresolved references can be resolved with one of our known third-party jars.
So ideally I want a MagicTool that I can run like
MagicTool \
--classpath commons-lang.jar:commons-collections.jar:[...etc...] \
myapp.jar
that will examine every unresolved reference inside myapp.jar and make sure that it can be resolved by one of the third-party jars passed to --classpath. If not, barf.
Better hope nothing in that path includes any sort of reflection, forName, etc.
I use depfind (if I'm not using my own Java spelunking tools) which may or may not provide output in a way that's helpful for you. jdepend is another option, although I've never used it for anything other than package-level dependencies.
Other tools like ProGuard will strip out unused classes (amongst other things), with the same reflective caveats.
I'm very wary of trying too hard to create minimal jar files; there's a point of diminishing returns/increased risk.
I am pretty sure that ProGuard is your magic tool, even if I don't now the exact call syntax.
At one stage some years ago, trying to index a JAR file with jar -i made it throw an exception if there were unresolved dependencies. I can't quickly test the current state of things.
Related
I have an older Jar file which I have to modify in order to get it to work.
I have already decompiled it with jd-gui and changed the parts that needed change. I would now like to know what I have to do in order to get a jar back?
I guess that I have to add the libraries, but there is no extra lib directory.
How do I put everything together?
#Stephen C
"Therefore, when you make your changes and recompile, the resulting ".class" file could have unexpected "
How exactly is this relevant? I only want the function of the Jar file and not a 1:1 copy of it.
The Jar file is unsigned. What exactly would be the reason to sign a Jar file anyway? If I start it from another program?
If you are missing dependencies after decompiling the entire jar, that could mean the jar did not include them. When you create a jar, you can choose not to include dependencies in it, but this assumes that they will be available on the classpath at runtime. Check if there are any relationships with other jar files on the classpath when you run the original jar. See this answer for details on how to achieve this. Once you have identified all the dependencies, you can easily compile the jar from an IDE such as IntelliJ/Eclipse or from command line.
As a side note, if the changes you would like to make to the jar are minor or isolated I recommend editing the bytecode (much easier for small edits). I prefer this bytecode editor.
If decompilation fails on some parts of the code, the functionality of the jar will not be restored upon recompilation. In this case, instead of going through all available decompilers and hoping that you can decompile that code, I suggest you identify the set of classes which you are trying to edit, modify the rest of the classes such that they compile and do not have any side effects on the classes which are of interest to you (do not delete anything that is referenced by these) and compile the jar (even if this isn't the jar you want). You can then extract only the class files which you wanted to modify from the jar, and overwrite them in the original jar. This will only work if your changes didn't have any side effects.
If you are asking how to create a JAR:
You can do it using a build tool such as Maven, Gradle, Ant and so on.
You can do it using a Java IDE
You can do it using the command line jar tool; see How to create a JAR file from the official Oracle Java tutorials.
But it there is no guarantee that what you are doing will actually work. Here are a couple of the problems.
If the original JAR file was signed, you won't be able to re-sign the new JAR ... unless you have the private key that was used when signing.
Decompilation is often inaccurate. There is no guarantee that the Java code that it produces is a correct representation of the original class. Therefore, when you make your changes and recompile, the resulting ".class" file could have unexpected / unwanted differences relative to the original.
I guess that I have to add the libraries, but there is no extra lib directory.
I'm not sure what you mean by that. The (new) dependencies of the JAR don't necessarily need to be in the JAR itself. It depends on what kind of JAR it is; e.g. is it a so-called "executable" JAR that you launch using java -jar my.jar ....
Use jar plugins for gradle or maven for example, or build it with intelliJ, here's an answer in another post: How to build jars from IntelliJ properly?
Since I had trouble getting the .jar file completely recompiled and working with my application, I wanted to share here the steps that were missing in the previous answers.
In my specific case, to make things as easy as possible, I wanted to only modify one .java file in the result of the decompilation and update it directly in the .jar file.
I started from the answer from Paulo Ebermann on a similar question, and I ended up with:
javac -source 8 -target 8 -d classdir -cp \
"\
path/to/dep1.jar; \
path/to/dep2.jar; \
path/to/myoriginal.jar \
" path/to/my/java/class.java
cd classdir
jar -uf path/to/myoriginal.jar path/to/my/java/class.class
Notes:
the path/to/myoriginal.jar which is in the path of the dependencies, this forces the java compiler to look for missing classses in the original .jar file rather than in the local decompiled code (which would require recompilation and more dependencies) and therefore I have intentionally left out the . path in the list of class paths
the -source 8 -target 8 which were required to force the class to compile for the appropriate version of the runtime machine.
the -uf which indicates that the jar tool should only update the file with the provided class
After developing a java app in Eclipse, I would like to deploy it by packaging it into a runnable .jar with only the minimum necessary by the main method and its dependencies.
I have several packages in my workspace I work with too, but do not need to be in the resulting .jar file.
When I try to export, it clearly says that the required libs will be there, but also the other independent packages are inside too. (With the Export option happen exactly the same)
I choose to export only the Main class of the com.project... package, but also the test.project... has been packaged.
In the project I have both com... and test... packages obviously.
How could I force it to truly package only the required ones?
Thanks in advance.
TEMPORARY ANSWER (2019-07-03):
Seems that, for now, there is no way to achieve this automatically, thus the answer from #arnonuem seems a good workaround.
If better news, please feel free to improve this thread.
Thank you all.
I would create an ANT file for this specific task. There you can freely customize which packages should be compiled into the jar and which not.
Please inspire yourself reading this example.
For a general overview what i am talking about you could take a quick look into this.
https://howtodoinjava.com/ant/ant-build-jar-file-example/
Please focus on
<javac srcdir="${src.dir}" destdir="${build.dir}" classpathref="classpath"
includes="src/path/to/TheClassToBeIncluded.java" />
For more detailed information on how to include or exclude files or packages you should refer to the documentation over here:
https://ant.apache.org/manual/Tasks/javac.html
At the bottom of this page there is a list with valid build parameters.
First of all, we have to distinguish build and export.
Build transfers .java source code into .class byte code and mostly copies other resources. While doing that it usually merges all source folders into a single bin folder.
Export runs a build (or relies on Build Automatically) and then modifies the resulting .class files and other resources. Usually it packages them into one or more .jar files.
Therefore, our solution includes two steps:
build everything that is unwanted for export into a separate folder (or more)
export from a specific folder (or more)
For step 1, refer to this answer on Stack Overflow.
In a nutshell: Go to Project Properties > Java Build Path > Source tab > Allow output folder for source folders. This enables you to configure a specific output folder for each source folder in the centered viewer.
For step 2, we need to understand that eclipse's Runnable Jar File Export relies on a Launch Configuration.
So before exporting, go to Run > Run Configurations..., select the Classpath tab, remove the default User Entries and hit Advanced.... Now you can Add Folders containing your built classes.
You might want to use separate Run Configurations for internal testing and exporting.
I have struggled with this problem on and off for years, supposing it was just me who was failing to find the right solution. Possible solutions always seem to involve detailed manual configuration e.g. configuration of the build path, or selection of the folders from which classes are exported into a jar, or learning ant, but which still requires manual configuration. The problem is that the inter-dependencies between classes (and packages) are complex - imagine drawing a network diagram from import statements. Manual configuration is time-consuming, error-prone and, I think, infeasible except in simple cases. I am a bit stunned. If there is no automatic solution for selecting necessary classes, I suppose people are regularly exporting their entire code base and that the world is full of bloated jars ... (and, incidentally, without obfuscation, the entire source code base is thereby made available through reverse engineering).
I'm not sure why I can't process this right now, but I have four packages in one source folder:
./src/common
./src/server
./src/client
./src/unittest
Common uses no files from any of the others, but server and client use files from the common package. In each of the files in those packages I have import common.*. But when I run the compiler with javac ./src/server/*.java it can't find the common package.
The only info I can seem to find is for tree structured package hierarchies, but how do I reference something at the same folder height as me? Do I need to nest common inside both server and client? That seems likely to generate a lot of redundant code.
I'm sure this is probably a question someone has asked before, so I apologize, but I cannot find it anywhere.
If you want it to find those files automatically without having to specify them, you'll need to be in the package root, so run this from within src:
javac server/*.java
That would then work.
However, personally I would always specify all the code you want to build, rather than letting javac pick up extra bits. If you want to build the common code first, I'd do so and then add a classpath entry so that javac will pick up the compiled classes rather than the source code.
(Alternatively, I'd spend most of the time in an IDE where it's largely automatic, and then use a fuller build system such as Ant to do the actual building.)
In one project (http://jodd.org) we have some classses generated by python scripts. Those are some big classes that has utility methods with similar behavior for all primitives and Object; we just wanted to simplify life and avoid potential errors made by repeating similar code.
Also note that this scripts are invoked manually, so we do NOT need any automation process etc.
We are mavenizing the project, and these scripts fall in the src/main/resources folder. Therefore, they become part of distributed jar with other resources, which is what we want to avoid.
What would be the standard maven location for these files; so they not become part of the distro jar? Is it src/main/scripts or src/main/python, or we should simply put them outside, in e.g. etc folder?
I think a good place to put python scripts is in src/main/python as you described. Doing so, they won't be copied to the final jar. I use similar approach with some native codes written in C in my projects.
How do I clean up stale .class files out of ${workdir} given set of existing .java files in ${srcdir}? By stale I mean .class files that were generated from now removed .java files. I have tried coming up with something using Ant mappers and filesets etc. but I failed. Removing all .class files older than their respective source .java files would be acceptable, too.
I'm pretty sure there's an ant task to kill .classes older than the .java...
Depend sounds close, and may actually do what you want, but this isn't its intended purpose. Given developmentalinsanity's answer however, this may be the only thing that will Actually Work.
The problem is determining whether a class file without an obviously corresponding source file is really stale.
Try this in a single file (A.java)
public class A{}
class B{}
This will result in both A.class and B.class. So, B.class would seem stale because of the missing java file. You'd probably get similar issues with any inner classes.
Safest bet if you want to make sure there's no old class files lying around would be just to delete them all.
As it’s just not possible to detect what’s stale and what’s not, most builds have a clean target (that’s also part of cleanbuild). The clean target, just removes all files from you’re build directory. This directory normally is unversioned (svn:ignore).
Not all files in you’re build will be the result of the compiler, for example .property files, these files can be stored in an alternative directory that will be copies in to the build directory. For example in a web application build you can store those files in /web/WEB-INF/classes and let Ant copy them into the build directory.