Is there any posibility to cache java compilation like ccache does for C or C++ compilation?
I have a lot code to compile, and I compile the same code many times, so it took a lot of time. So I thought to cached it to speed up.
But the main problem is that, the whole source code has many files in many directories, and there are so many java files. Compilation is done by the ant scripts which are invoked by the make script. It has to be compiled by the Sun's JDK. And what is more... between compilations I have to clean all created *class files.
So I cannot use any of IDE's features.
Eclipse does incremental compile of classes (I guess others do this as well). Only the portion of your class that changes will be compiled. Everything else will not be changed. It works reasonably well in real world projects.
Build tools like Maven and Ant can be configured to compile only changed java files. These do not track dependencies in some scenarios they miss needed compiles. (For example change the signature in a super class.)
The best way would be to factor the shared classes into utility jars that you would simply include in your compile classpath, unless there is really a specific (and good) reason that prevents you from doing so (code duplication is evil and will lead you to doom :)).
There's no caching functionality in Sun's JDK itself, as far as I am aware.
Most Java IDEs however will cache compilation, so if you need "compilation on demand", that should do.
Eclipse and Netbeans definitely have this. They will only recompile changed classes (and their dependencies, which they figure out automatically). This is fairly fast for most changes.
Related
The feature build automatically under the eclipse is much faster than the ./gradlew build.
My findings after some research is that it compiles and builds only the changed file and replaces it in build folder.
So why can't ./gradlew build command compiles and builds files that have changed and replace it in build folder and make the whole building process faster.
I have recently started using build automatically feature with hotswap agent + DCEVM.
Why can't gradlew build command compile and build only the things that have changed and make the process faster?
There's no dependable way how to determine what needs to be recompiled. For example, compile-time constants get inlined and there's no trace of where they come from in the class files (it can be found in the source files, that implies parsing them and losing time; it can be stored in some auxiliary files and some tools do it).
See the "Limitations" section of this for details.
The reason maybe is that they don't go through configure step of gradle.
Sure, but the configure step doesn't usually take that long.
Eclipse knows which files have changed
Good point (in a comment by holwgler).
Some time ago I spent some time trying to make my gradle compilation faster and I gave up. Eclipse is damn fast for many reasons:
incremental compilation
multithreading using all cores
knowing all changed files
having the whole compiler code optimized by the JIT
probably caching file dependencies
ugly highly optimized code
My "solution" is ignoring the problem. I do everything in Eclipse, except for integration tests (which take way longer than the compilation) and production builds (which are rare enough so I don't care).
You may want to read these performance tips.
To find out where the time gets spent, use
./gradlew clean; ./gradlew --profile jar
For me, 90% of the time is just :compileJava.
IDEs offer a lot of cool features like jump to declaration and syntax highlighting. Just out of curiosity, in order to provide these features, does my IDE (IntelliJ) first have to compile my source code? Like I noticed that whenever I open a new project in my IDE, there is a lag, and it generates a "target" folder. Does the IDE compile any of my source code without me explicitly running the code or telling it to compile?
There's several things going on here. When you import an sbt project, IntelliJ runs sbt to extract the project structure. This takes a while, and sbt itself creates the target folder.
Most IDE features, such as jumping to declaration don't require compilation. IntelliJ instead parses the code into it's own internal syntax model which allows for indexing, error highlighting and so on. This model allows many refactorings and analysis even on code that has compile errors. Indexing typically happens after the initial import in a background process.
This largely depends on the IDE.
IntelliJ makes use of an indexing process in which it takes your entire project and indexes it in its own internal structures known as Psi* classes. These classes then are referenced by the IDE to provide static analysis and determine code flow.
I'm not as familiar with Eclipse, but the Abstract Syntax Tree exists and is likely analogous to IntelliJ's internal structure.
IntelliJ doesn't make an effort to compile your code on your behalf unless you let it, but in order for its static analysis to actually work, your code must be compilable. If it isn't, you won't get static analysis; you'll get red squiggly lines instead.
I have 3 projects that are used as libraries within a 4th (main project).
The 3 projects are complied within each other as follows (build.gradle):
Library Project:
Project A
compile project(":projectA")
compile project(":projectB")
Project B
compile project(':projectC')
Main Project:
compile(name: 'projectA', ext: 'aar')
compile(name: 'projectB', ext: 'aar')
compile(name: 'projectC', ext: 'aar')
I would like to do something to the "Library Project", so that from within the Main Project, if I click on any class from within the Library project, I should either not be able to see the code, or it should be encrypted.
So for example if there is InterfaceA in ProjectA, and the main activity of the Main Project implements that interface, if I "Ctrl-Click" into the interface, the result should be similar to what I specified above.
I understand Proguard does something similar, but that is only if you are building a release .apk, I need the same result for compiled libraries.
Many projects use ProGuard to achieve this protection.
You can use Gradle to build library components for Android
Libraries, just like apps, can be build in development or release build types
Proguard can be configured to run on a component (app or library), but only in the release build type. See here: https://sites.google.com/a/android.com/tools/tech-docs/new-build-system/user-guide#TOC-Running-ProGuard
If the component is minified (highly advised), then you need to tell Progaurd what the "root" classes are, otherwise it will minify the library to literally nothing. This can be achieved by adding a rule to the configuration file:
-keep class your.package.name {public *;}
A more extensive example is here: http://proguard.sourceforge.net/manual/examples.html#library
However there are some limitations:
ProGuard's main use is is removing as much debug information, line numbers and names as possible from the bytecode without changing what the bytecode actually does. It replaces the names of members and arguments, non-public classes with meaningless ones, for example vehicleLicensePlate might become _a. As any code maintainer will relate, bad member and variable names make maintenance really hard.
ProGuard can (slightly) modify bytecode by optimising as much as possible (computing constants defined as expressions, playing around with inlining, etc. The optimisations are listed here: http://proguard.sourceforge.net/FAQ.html#optimization)
ProGuard does not encrypt the bytecode - the JVM needs to see the actual bytecode otherwise it could not run the program.
So, obfuscation only makes it harder to reverse-engineer and understand a library, it cannot make this task impossible.
One last pointer: ProGuard dumps a file containing a list of what it has changed, in particular the line numbers. When you get stack traces back from your customers (or through online tools like Crashlytics) you can revert the obfuscation so you can debug. In any release-build process, you need to find a way to save this file.
This file is also needed when you make incremental releases of your library so the obfuscation is consistent to the previously released version. If you don't, the customer cannot drop-in replace your library and will have to do a complete rebuild (and link) of their app.
While ProGuard is a free-n-easy option which just works, there are other free and paid-for obfuscators. Some offer a few more features, but they are fundamentally the same, and the compatibility of ProGuard with IDEs, tools and services is excellent.
You could set all the methods you don't want to be public to default, so they can't be used outside of the original project. And also, you should separate the libraries from the app project, compile them, and use them as external dependencies. If you don't want the source code of the library published, just don't add it to the compilation options. If somebody else than you needs to use your library, publish it using bintray, or just add the compiled aar/jar files to the app project.
Here's a guide for the whole process: https://inthecheesefactory.com/blog/how-to-upload-library-to-jcenter-maven-central-as-dependency/en
Alternatively, you can build library projects using maven (I find it a lot easier than using gradle), take a look here for an example: https://github.com/simpligility/android-maven-plugin/tree/master/src/test/projects/libraryprojects
and a concrete project:
https://github.com/fcopardo/BaseViews-Android
2 steps:
add your libraries to local maven repo
use maven dependences instead of project dependences.
You cannot do that. A compiled library has .class files (bytecode), which can be de-compiled and viewed using various de-compilers like JD-GUI etc. Android Studio has a built-in de-compiler which makes it more easy for someone to just ctrl-click and view the .class file. The best option you have is to obfuscate your code. Here are some obfuscators you can use. But always keep in mind that it's never impossible to reverse-engineer something. Everything is hackable.
If I would like to write a program to automatically exclude not-used jar files from a maven project (regardless of what programming language I will use), how should I get start?
Currently, I have a stupid idea. Except the declared jar files, every time excluding a node of the dependency tree top down, then compile and run the project to see whether this jar file could be excluded.
I cannot find any other better ideas and I need help. Please share me with whatever come into your minds. If you know somebody else has done this before, please post the link to the source code or share with me its ideas. Thanks!!!!
By the way, what if I want to write a program automatically solving the dependency conflict problems? Is this possible? I know Such problems have bothered java developers a lot.
The idea of compiling for every removed dependency is good, but don't forget to launch the program also (some dependencies are required at runtime only).
I had to do that a few times, but I did manually (not automatically with a program), i removed all dependencies and added them one by one until code compiles, and until the program run OK.
The problem with approaches like this is the fact that even though everything might compile correctly, you are missing resources during runtime.
That might be the case, if you load classes and stuff with help of the Classloader or if you are using reflection to access classes and methods.
This gets even worse if you look at more dynamic languages aside of Java: the compiler won't help you that much anymore
I'm currently passing a very large classpath to javac to compile a java project.
I know that a number of those jar files aren't needed.
Is there a simple way of finding out which files aren't needed?
You need the Class Dependency Analyzer tool. To quote the introduction:
The purpose of this tool is to analyze Java™ class files in order to learn more about the dependencies between those classes.
True, it won't catch runtime dependencies - but short of running an exhaustive 100% coverage test suite you can never be sure you've caught all runtime dependencies.
If you expect runtime dependencies you should use the CDA as a first-pass, then do exhaustive testing of the resultant app to ensure that there are no jar files which were only referenced through runtime dependencies.
I guess that "remove them one by one and check if the application still compiles and works" is not the expected answer :)
(EDIT: While the approach suggested above can be a bit automated, it remains somehow painfull and there must be an alternative, at least for compile-time dependencies. After some googling, I found Jar Analyzer which seems to be a nice tool for this work as explained in this blog post:
Jar Analyzer scans for compile dependencies, meaning that it can create a tree of which JAR-files are needed to compile which JAR-files that are needed to compile these JAR-files, and so on. You get a very nice report/graph which shows you all the JAR-files and why they are in there.
You can also see the JAR-files that don't have any connection to your code, remove them and their children. What I found in our libs folder was that about 20% of the 150 JAR files in our libs folder were unused at compile time, and these were potential JARs to be removed.
The big aber is that you don't get any hint on which JAR-files are used only at runtime by means of discovery and reflection. And this is where the real work begins.
The only way to find out whether a JAR file is used at runtime is basically to take it out, start up your application and test every functionality. If you have an application of moderate size, performing a 100% regression test takes many hours. So in practice, I ended up doing alot of guessing, quick and dirty testing, and asking around to find out which of the runtime dependencies were actually in use.
It seems pretty easy to use: download, unzip and run the tool on a directory containing all jars. Or use the provided Ant task.)
You also have loosejar.jar which enables you to find the real jar dependencies of your project at runtime !
The compiler has a -verbose option, and it is quite verbose. It informs of each class that gets loaded and where it is loaded from!
While it's not quite user-friendly and it doesn't provide high-level analysis, I found it very useful for debugging classpath conflicts. This tells you the jars that get used (with help of grep), not the ones that don't get used.
The latest build of eclipse will warn you about unused imports in your source code
Check Classpath Helper