We Keep Coding

Intellij build times are too long. Gradle not helping

I am frustrated by the long build time that IDEA needs before starting my application. I do use DCEVM to enjoy changing code on the fly better than plain hot-swap does, but this doesn't work if I really do need to start my applications multiple times.
IDEA is supposed to be able to have incremental compilation, but I'm not seeing it. I see no options to enable this. Start or starting a debug of the application, takes nearly half a minute. Must it take so long? In theory, IDEA has all the class files already generated, except for the source files that were just edited. IDEA ought to be able to fill in the few blanks, and get the application running almost instantly. What's the hold up, why does it insist it must re-compile every single source file again?
ps. I'm using the Eclipse compiler in IDEA, so that it tolerates some source files not being compilable. I have to do that, because it is one giant tree of java files, where I can not control every one.
Then I saw that perhaps Gradle does have incremental compilation. So I thought to re-create my project, this time with Gradle. I am having a very difficult time figuring out how it works, how it integrates, what tasks are moved from IDEA to Gradle (if any). Even creating a simple skeleton test application with Gradle in IDEA is very frustrating. There are all sorts of fuzzy bits all over the place that leaves you guessing. The tutorials on Jetbrains are almost useless. I can't find one single clear to the point that just explains it in a nutshell how it works. I see some demos that says, click here, and here, and there, but the results on my end aren't the same as what I see outlined.
For instance, Gradle complains that it can't find the junit jar. Ok, so I go into the project settings, and attempt to add the junit jar to the classpath. Under Libraries, I then see an entry to "Gradle: junit:junit:4.11". But there is a second one: "junit-4.11". Why two? Can I delete one? Why is the one so verbal, colon this colon that, etc.
So I add the junit jar files I have to both.
But then in the Gradle Projects window, where I see the hierarchical breakdown of the gradle project setup, verification->test. This is then supposed to run the test class I did manage to create.
FAILURE: Build failed with an exception.
* What went wrong:
Could not resolve all dependencies for configuration ':testCompileClasspath'.
> Could not resolve junit:junit:4.11.
Required by:
xpert_client4:xpert_client4:1.0-SNAPSHOT
> No cached version of junit:junit:4.11 available for offline mode.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
"Could not resolve junit:junit:4.11".
Gradle is extremely frustrating. If all I have to have, is to start a main() in a large project, why would I go through this trouble of using this gradle system that offers things I don't need? Why is it so convoluted, fuzzy, and confusing?
Same goes for an Android project. IDEA can compile and run an Android app without Gradle just fine. Simply avoid anything gradle when setting up, and it just works.
Plus there are all the things that one has to do to speed up the supposedly slow Gradle system. There is the daemon thing. Parallelize, and apparently you can set it in two ways (why). Configure on demand. Globalize. Minimize. minSdkVersion. Offline (oh no, not that Maven thing that pulls down jar files from the internet over and over). jCenter. Profile.... it goes on and on. All those workarounds, hacks, adjustments, tweaks.
Is there anything to be gained by gradle-ifying a working IDEA project? Would I get faster build and startup times if I go through all this trouble? Can I avoid Gradle altogether, and forever (in the hopes that it doesn't somehow becomes mandatory).
I've been coding since the 80s. I've used all sorts of systems. I've build my own build systems. I'm not green here. I also hate having RTFM, if those FMs don't get to the point, and don't explain things well - as if the ones that wrote it, don't really understand it themselves - they know what buttons to press to get it going and that's it.
I find this thing one of the most frustrating things I have seen in a long time. The previous horrible build system I've tried in the past, and have avoided like the plague since, is Maven. The lack of clear thinking with these systems is astounding.
So, aside from these frustrations, what I'm after is: is it worth figuring out Gradle and applying all the workarounds and tweaks, so that I may, once I go through this grief, enjoy faster build and startup times of my application?
Or, would I be better off, avoid Gradle altogether, and instead, do something like place the startup of my application in a loop, so that every time I exit that the application that it re-runs main(). I would then use DCEVM to code on the fly while possible.
Lastly, I forgot to mention. Enabling automatic build in IDEA works really badly. A build seems to always involve re-building every single file, over and over. And setting build to automatic, all it ends up doing is it redoing this in the background over and over. There is nothing that happens in an asynchronous way, where when I finish editing a source file, and start the debug of the application, where IDEA has done any of the building in the background ahead of time, because it redoes the whole thing again after every single tiny little code change. Plus it makes the CPUs spike to near 100% constantly.
IDEA is a great IDE, but the whole compile thing is a total nightmare front to back.

Make sure to disable anti-virus protection in your Gradle caches directory and your IntelliJ project directory. For me, this reduces Java compilation time dramatically, because of all the intermediate files and JARs associated with Java builds. Excluding those same directories from Windows Search indexing can also help.
As of September, 2020, this is the list of default directory locations that should be exluded from virus scanning and Windows search indexing, published here.
Gradle cache: %USERPROFILE%\.gradle
Android Studio projects: %USERPROFILE%\AndroidStudioProjects
Android SDK: %USERPROFILE%\AppData\Local\Android\SDK
Android Studio system files: %USERPROFILE%\.AndroidStudio<version>\system
I published some instructions on my blog for doing this for Windows Defender and Windows Search Indexing, which I'll repost here for completeness:
Windows Defender - How to Exclude a Directory
Open Windows Defender Settings
Click "Virus & threat protection"
Click "Virus & threat protection settings"
Scroll down to "Exclusions" and click "Add or remove exclusions"
Click "Add an exclusion" and select "Folder" from the drop-down menu
Navigate to the directory to be excluded, or paste its path into the "Folder" text-box, and click "Select Folder"
Windows Search Indexing - How to Exclude A Directory
Open Windows Control Panel
Search for "index" and choose "Indexing Options"
Click the "Modify" button
Under the appropriate drive letter, navigate to the folder you want to exclude, and make sure it is unchecked
Double-check your work by verifying the directory shows up in the "Exclude" column of the "Summary of selected locations" table

Even I was facing the same issue and found one option in Intellij IDE, which closely solves my issue, but not fully.
Build Project Automatically and Compile in Parallel
Though it does not solve fully, but I see some performance improvement in build time, whenever I add/upgrade dependency and run the tests.

IntelliJ IDEA + Play Framework - how to speed up compilation

We just started using Play Framework 2 (Java) with IntelliJ IDEA (14.1.1) on a new project.
As it is now, simple tasks like running tests always seem to start with a 20 second long "make" step, and then the tests itself takes a few milliseconds.
According to the "messages" view, the Play 2.0 compiler is handling compilation, even though this is not checked in the Play2 options in IDEA.
Is this normal?
Are there any known quickfixes for how to make this work in a practical way?

A workaround was described in the issue mentioned in my last comment.
Disable play2 compiler in settings
Delete the play*xml files in the .idea folder for your project
Restart IDEA
If you reimport the project (e.g. when build.sbt changes), you'll have to delete the xml files and restart IDEA again.
This will presumably be fixed - look for updates in the YouTrack Issue

Should I commit files that are changed by Eclipse?

I inherited a Java project in the form of an Eclipse project. After changing the Tomcat configuration (from v6 to v7), Subclipse prompted me to commit the following files:
.classpath
org.eclipse.core.prefs
org.eclipse.common.project.facet.core.refs
org.eclipse.common.project.facet.core.xml
Will commiting them help my team members or will it mess with their workspace?
What is the best practice approach to this?

Generally speaking, you should check-in (and commit after changes) everything that does contribute to the build AND is not re-generateable by re-building completely AND is workstation-specific. (The implications of this statement depend on your build process/procedure, which is intended.)
This implies you should exclude everything that is re-generated upon full build etc. so it is not checked in (and not offered for check-in).

As a general rule, you should avoid committing files that contain user preferences, and project details that that Eclipse and/or your plugins can regenerate.
But in some cases things are a bit murky. For instance, the .classpath file can be the primary source of the Eclipse build path; e.g. if you have JAR files in your project tree rather than using Maven. (With Maven, the m2eclipse plugin generates the .classpath file from the dependency information in the POM file, and hence the file should not be checked in.)
Also, some of the facet stuff is borderline. For instance, in projects with JSPs and Javascripts, I have found it essential to change the facet properties to disable broken validators. And there's a good case for treating those changes as part of the project rather than as personal preferences.
Separation of group / project preferences from personal preferences is one area where (IMO) Eclipse is seriously deficient.

It is better not to commit those files as paths/settings may differ on different workstations.
You may wanna use some build tool to overcome this. (eg. Maven)
As if any of the team members are not using eclipse (using some other ide) , those files have no meaning for them.
If everyone commits different IDE settings, imagine what kind of mess it can cause.
EDIT:
More explanation;
I have worked in teams that people used NetBeans, Eclipse, IDEA...for a really long time and it is not really an option for them to change the IDE. It will only affect the productivity of that person.
When people get used to their IDEs they learn shorcuts, they know where to look for some functions (refactor/generate getter setter/implement override required methods....) so if you force them to use some other IDE it will just make things harder for them and slower for the overall process. IMHO and from my experience having a flexible codebase is always good. I am an eclipse guy and probably would not want to work with any other IDE as I know lots of shorcuts which makes thing real quicker/easier for me and those shorcuts are different on different IDEs.
All IDE files can be regenerated automatical by the IDE itself probably in just a couple of clicks.
And my current project has 3 developers, each using different IDEs eclipse(me), NetBeans, IDEA without any problems. I dont want to see IDEA or NetBeans config files which makes no sense for eclipse when I check out the source from repo. Likewise for them as well.

Yes, though do make sure that paths are relative in the workspace rather than absolute paths. Having these files in the workspace allows members of your team to work in the same environment as you are. It also makes setting up a new development environment much easier: you just check it out of source control and in Eclipse use 'Import... > Existing Projects into Workspace'
As #adamdunne mentioned, these files can contain environment specific paths. However it if you are careful to make sure paths are relative within your workspace, by using variables and by not importing external jars, i.e., by only including jars from projects in the workspace, then you should be okay. In my workspace we check in those files and have had a lot less issues setting up dev. environments since.

I work in a project where we commit the .classpath file since it is very useful that all developers use the same :) If you only use dependencies inside your workspace, this file uses relative paths and thus should be same on all machines. Even if this file might not be necessary to build (with ant e.g.) it´s very convenient to synchronize it.
In contrast the org.eclipse.core.prefs stores (afaik) project-specific, but personal preferences of developers which I would not check in.
With the facets I didn´t work yet in a real project, so I can´t tell. But in general, I think it depends on the information in the file and on the way you work.
If you are unsure, just try it. If you get conflicts in these files all day this is a hint you may not be on the perfect way.

These files can be very useful to share configurations between developers. The alternative is to either use Maven (which is a huge task for an established project) or to have constantly-outdated step-by-step instructions and new developers taking half a day until they can even build the project.
However, you should take care to ensure that these configurations are portable, i.e. contain no local paths. This can be done via the use of relative paths within the workspace, eclipse path variables and user libraries.

What we've done is ignore these files, as they may mess up the workspace of others on the project.
Ignoring them also makes your project cleaner, which I always like.

These files can contain environment specific paths so I would suggest not checking them in. On my current project we use ant scripts to create the project and do the initial checkout of all our code.

Eclipse not synchronized with file system changes made by ant's build.xml

We have a pretty big project. We use ant for setting the environment. One of the things that happens during the ant script is that a jar is copied to the lib folder of a project. Then, we use ant tasks: eclipse.refreshLocal and eclipse.incrementalBuild so that the jar now copied will resolve compilation problems in the project that refer to it.
Unfortunately, the projects seem unsynchronized with the filesystem. Project > properties > build path shows that the jar is present (probably information that is refresh while looking at the propertiesw) but the incrementalBuild didn't do what we expected it to do.
Also, sometimes a project that depends on this jar and we refreshed and built it (through ant) has a single compilation error: "The project cannot be resolved until build path errors are resolved". In other times, the are many compilation errors indicating the project hasn't noticed the new jar.
When I build it through eclipse, it is ok.
I was worried that maybe there are circular dependencies between the projects, but it appears the are no such dependencies (Build properties: the circular dependencies option is set to "Error", and the are no errors of this form in the projects).
Anybody is fimiliar with the problem?
What is the proper way to deal with this problem?
Thank you for your help.

If you right click on the build.xml and select Run as → Ant Build... you'll see a tab called Refresh. Just check the option Refresh resources upon completion.
(I use this myself when generating code in an ANT xslt target and it works like a charm.)
You may also want to check Refresh automatically under Window → Preferences → Workspace

Try Cleaning the entire workspace and refresh :)

One solution which might work for you is to modify the Eclipse launch config for the Ant build (Run -> External Tools -> External Tools Config), and explicitly configure it to refresh the workspace (or parts of it) when the Ant build finished. This removes the need to perform the refresh from within the Ant script itself, and it always triggers the correct build reaction when the script completes.

Eclipse does not generally like it very much if you change files in its workspace from outside. There is an option in the Eclipse Preferences to automatically sync with the filesystem, but that can be very intense with regards to IO.
And it will probably still not solve all your problems, esp. when you change library jars.
Even considering skaffman's suggestion with the Refresh on build completion, I am not sure this would really improve your situation, but give it a try.
From my experience, realistically I would have yet to see a solution in a big setup that really works and makes life easier instead of just creating a different set of problems.

To check in, or not check in, the entire Eclipse project?

I'm soon going to check in the very first commit of a new Java project. I work with Eclipse Ganymede and a bunch of plug ins are making things a little bit easier.
Previously I've been part of projects where the entire Eclipse project was checked in. It's quite convenient to get the project settings after a check out. However this approach still was not problem free:
I strongly suspect that some Eclipse configuration files would change without user interaction (from when I used Eclipse Europa), making them appear as changed (as they were changed, but not interactively) when it's time to do a commit.
There are settings unique to each development machine as well as settings global for all developers on a project. Keeping these apart was hard.
Sometime if the Eclipse version was different from others Eclipse would get angry and mess up the project configuration. Another case is that it change the format so it gets updated, and if commited messes up the configuration for others.
For this specific project I have another reason not to commit the project files:
There might be developers who prefer NetBeans which will join the project later. However they won't join within the coming months.
How do you organize this? What do you check into versioning control and what do you keep outside? What do you consider best practice in this kind of situation?

At a minimum you should be check-in the .project and .classpath files. If anybody on your team is hard-coding an external JAR location in the .classpath you should put them up against the wall and shoot them. I use Maven to manage my dependencies but if you are not using maven you should create user libraries for your external JARs with with a consistent naming convention.
After that you need to consider things on a plug-in by plug-in basis. For example I work with Spring so I always check-in the .springBeans and likewise for CheckStyle I always check-in the .checkstyle project.
It gets a bit trickier when it comes to the configuration in the .settings folder but I generally check-in the following if I change the default settings for my project and want them shared with the rest of the team:
.settings/org.eclipse.jdt.ui.prefs - it contains the settings for the import ordering
.settings/org.eclipse.jdt.core.prefs - it contains the settings for the compiler version
In general I haven't noticed Ganymede modifying files without me modifying the project preferences.

I recommend to use maven so that the entire life cycle is outside of any IDE. You can easily create an eclipse project with it on the command line and you can use whatever you want, if it's not eclipse. It has it's quirks but takes out a lot of bitterness when it comes to dependencies and build management.

In our world, we check in the entire Eclipse project and the entire parallel but separate Netbeans project. Our motivations for this were entirely focused on "when I do a checkout, I want a functional configuration immediately afterward." This means that we had to do some work:
Create runnable configurations for each primary IDE (people like what they like). This includes main class, working directory, VM parameters, etc.
Create useful start up scripts for all of our relevant scenarios.
Create edited datasets that don't cause the checkout to take too much longer (it's a big project).
This philosophy was worth cash money (or at least labor hours which are almost more valuable) when our new hire was able to check out the project from Subversion into Eclipse and immediately run a functional system with a (small) real data set without any fuss or bother on his part.
Follow up: this philosophy of "make the new guy's life easier" paid off again when he changed IDEs (he decided to try Netbeans after using Eclipse for quite a long time and decided to stick with it for a while). No configuration was required at all, he just opened the Netbeans project in the same directory that Eclipse had been pointing to. Elapsed switchover time: approximately 60 seconds.

I only ever check in things are done by humans, anything else that is generated (whether automaticly or not) should be easy to regenerate again and is liable to change (as you've stated). The only exeption to this is when the generated files are hard (requires alot of human intervention ;) ) to get it right. How ever things like this should really be automated some how.

Try to port your project to a build system like maven. It has everything you need to get the same experience of the project on every machine you use.
There are plugins for just everything. Like the eclipse plugin. You just type "mvn eclipse:eclipse" and the plugin generates your entire ready to work eclipse project.
To give the answer to your question. Never check in files that are not being used by your project at any time in the development cycle. That means that metadata files like eclipse properties etc. should never be checked in in a SCM.

I like checking in the .project, .classpath, and similar files only if they will be identical on any Eclipse user's machine anyway. (People using other IDEs should be able to check out and build your project regardless, but that issue is orthogonal to whether or not to check in Eclipse-only files.)
If different users working on the project will want to make changes or tweaks to their .project or .classpath or other files, I recommend that you do not check them into source control. It will only cause headaches in the long run.

I use IntelliJ, which has XML project files. I don't check those in, because they change frequently and are easy to recreate if I need to.
I don't check in JAR files. I keep those in a separate repository, a la Maven 2.
I don't check in WARs or JARs or javadocs or anything else that can be generated.
I do check in SQL and scripts and Java source and XML config.

I'd suggest having the actual project files ignored by the version control system due to the downsides you mentioned.
If there is enough consistent information in the project settings that there would be benefit from having it accessible, copy it to a location that Eclipse doesn't treat as special, and you'll have it available to work with on checkout (and copy back to where Eclipse will pay attention to it). There is a decent chance that keeping the actual project files separate from the controlled ones will result in loss of synch, so I'd only suggest this if there is clear benefit from having the settings available (or you're confident that you'll be able to keep them synchronised)

In our case, we used to check in the project files (.project and .classpath) to make it easy for all developers to create their project workspace. A common preferences file and team project set were located in source control as well, so creating your workspace was as simple as import preferences and import team project set. This worked very well, but does rely on everyone having a consistent environment, any customizations would have to be applied after the basic workspace is created.
We still do this for the most part, but Maven is now used so of course dependency management is handled via Maven instead. To avoid conflicting information, the .project and .classpath were removed from source control and are now generated via maven goals before we import the team project set. This would easily allow for different environments, as you would simply need scripts to generate the IDE specific portions based on the Maven configuration.
PS-For ease of maintenance though, I prefer having everyone use the same environment. Anything else inevitably becomes a full time maintenance job for someone.

Netbeans 6.5 has an improved Eclipse project import which is supposed to sync changes from Netbeans back to Eclipse: http://wiki.netbeans.org/NewAndNoteWorthyNB65#section-NewAndNoteWorthyNB65-EclipseProjectImportAndSynchronization

Don't. Only check in the source code of your projects.

As a response to:
"There are settings unique to each development machine as well as settings global for all developers on a project. Keeping these apart was hard."
Eclipse offers a number of ways to keep local settings manageable: Java Classpath Variables (Java > Build Path > Classpath Variables) are one, 'Linked Resources' (General > Workspace > Linked Resources) are another http://help.eclipse.org/stable/index.jsp?topic=/org.eclipse.platform.doc.user/concepts/concepts-13.htm Creating a README that states which settings to set before building/running the project works pretty well in my opinion.
Now how to make sure your continuous build system understands the changes that were made to the eclipse settings, thats another issue... (I have a separate build.xml for ant that I keep up to date by hand)