We have been working on our project with scons as build system for a few years. Scons is awesome and has improved our development process a lot.
There are ~15000 C++ source files initially and as the project evolves source files in other languages(Java/Scala) are added into the code base as well.
Then we developed our own builders which manage dependencies and invoke the external tool(javac, scalac) to build those sources, which works really well.
Recently I was working on the optimization of current build system and found performance difference between C++ and Java build process:
Environment setup:
Build server with 24 cores, Python 2.7.3, Scons 2.2.0
Command: scons --duplicate=soft-copy -j8
When building java code, CPU usage is easily high observed from top and spanning multiple cores:
Java Build
However, when building C++ code, CPU usage is always ~4% and running only on 1 core no matter how many jobs in scons:
C++ Build
I've been googling a lot on the internet but could not find something useful. Am I hitting the GIL issue in Python? I believe that each gcc/g++ command should be ran in a separate sub-process just like javac in our own builders, so there should not be GIL here. Is there any workaround to fully utilize multiple cores speeding up C++ build further?
Thanks in advance.
As WindLeeWT explained in one of his comments, the cause for the observed behaviour was that ccache was installed and configured on the server in question. It seems that most of the CPU usage during a C++ build was took by the compilation stage, which was fully avoided due to ccache. That's why no CPU usage for several cores could be seen within top.
As soon as they launched a "build from scratch" on another server without ccache, several cores were running at 90% with 'cc1plus -o ...' as expected.
No performance penalties (GIL etc.) were involved, and neither Python nor SCons degraded performance in any significant way.
Related
Who has successfully compiled a Java business project to native (e.g. using GCJ or Excelsior JET) and can share the pros and cons?
I image following advantages:
more speed (the user's machine does not need to compile byte code to native code)
less possibility to hack
no separate Java runtime required
and following disadvantages:
needs a special build for each platform to support (but that's required for SWT already)
some features like reflection might not work?
harder to locate bugs (what about stacktraces)?
I've used Excelsior JET to compile an SWT app to native for Windows.
This was some time ago, and they've improved the tool immensely since then, but JVM speed has since also improved commensurably. Memory usage not so much, but then even low end laptops have GB's of RAM these days and, although it might disgust our sense of frugality, it really doesn't matter if your small GUI app uses 20MB of memory on a box with at least 50x that amount, especially when you factor in the advantages of developing in this sort of environment.
The main reasons to static compile are startup time and memory usage. JET gave me these, but at the expense of a long build cycle, bugs in missing classes due to dynamic loading conflicting with static compilation (this is something I believe they've improved a lot) and platform-specific builds (you must build the Windows distro on Windows). Eventually Moore's Law and JVM speeds made these trade-off's not worth it, and I dropped this build option.
FWIW, stacktraces and reflection are fine, as is dynamic class loading if you make sure the compiler knows about the classes that are not statically referenced. If you're worried about decompilation, well, an obfuscator will likely give you as much mileage.
I have nothing but good things to say about JET and the people that make it though. If you need to do it, JET's a solid product.
I can think of 2 cons
Cons:
Compiling application needs time which would increases the software
development costs. Java originally use interpretor to run the code
which is far more easy to develop.
It doesn't have Java original debugger to debug your software. You
need to use other like GDB which is far more complicated. Again,
increase your development costs
I am writing a microbanchmark in Java, following the ‘rules’ on this page. The last rule states:
“Reduce noise in your measurements. Run your benchmark on a quiet machine…”
So now I’m wondering how to create a system that is as quiet as possible. Is a clean installation of windows 7 with unnecessary services disabled a good example of a quiet system, or are there better options?
A quiet machine is good, but perhaps unrealistic.
You should test a system which is as close to how you intend to use it as possible.
e.g. you should run the same hardware, OS and services as your target system.
If possibly run it on your target system, possible after hours if needed.
Linux gives you a lot more control over what runs on the system (e.g. it's trivial to run without a GUI, which I don't think is even possible on Windows), so if you want an ideal synthetic benchmark environment, it would be a better option.
But a clean Windows install (important: no virus scanner or other preinstalled crapware) with most services disabled should be good enough as well.
I use a desktop with eight cores to build a Java application using Ant (through a javac target). Is there a way to speed up the compilation by using more than one thread or process?
I know I can run several Ant tasks in parallel, but I don't think this can be applied to a single compilation target, or does it?
I don't know of any way to do tell ant itself to make effective use of multiple cores. But you can tell ant to use the Eclipse Compiler, which has support for multithreaded compilation built-in.
As long as the javac you are calling doesn't use all the cores it doesn't really matter what you say in Ant. You can use the compiler attribute to define which java compiler should be used for the task.
If you have several build targets you can use fork=yes to execute the target(s) externally.
http://ant.apache.org/manual/Tasks/javac.html#compilervalues
The documentation seems to indicate that it's unlikely to work correctly with javac.
Anyone trying to run large Ant task sequences in parallel, such as javadoc and javac at the same time, is implicitly taking on the task of identifying and fixing all concurrency bugs the tasks that they run.
Accordingly, while this task has uses, it should be considered an advanced task which should be used in certain batch-processing or testing situations, rather than an easy trick to speed up build times on a multiway CPU.
You can use Buck Build to increase your build speed and utilize multiple cores.
In a nutshell:
Buck is a build system developed and used by Facebook. It encourages
the creation of small, reusable modules consisting of code and
resources, and supports a variety of languages on many platforms.
Buck builds independent artifacts in parallel to take advantage of multiple
cores on your machine. Further, it reduces incremental build times by
keeping track of unchanged modules so that the minimal set of modules
is rebuilt.
Not as far as I know. The Eclipse compiler has some work done to speed up using multiple cores but it does not buy as much as you probably would like it to.
Question is, can you live with incremental compilation for development, and only recompile those that changed? The full rebuild can then be left to the build server.
I assume that it might not help much because javac can pull all files in memory and if it has to do this with multiple processes it's just doubling the effort. However if you want to compile two fairly separate pieces of Java code, then you can just do:
#!/usr/bin/env bash
javac file1.java &
javac file2.java &
javac file3.java &
wait;
if the 3 files have mostly different dependencies, then it might save time, if the dependencies overlap, then it probably doesn't save much time.
Is there a distributed compiler for Java, analogous to distcc for C/C++?
The direct answer to your question is "no". However, it probably would not help you anyway… compiling Java is very fast.
On a small project, the compilation is fast enough that you shouldn't really care. On a large project you would need to deal with throwing the file to compile over a network, and having to deal with potentially also throw across many megabytes of dependencies as well.
One thing that you can do to improve your compilation speed is to use the eclipse compiler instead of the Sun javac. The Eclipse compiler is multi-threaded, and so will, with luck, use all the cores of your machine.
It is probably also worth mentioning that Apple also recently reduced distcc support, since in general, on newer hardware, it cost more time to get the code somewhere else to compile and back, than it did to just do the compilation locally. To quote Apple:
The single-computer build performance of Xcode has been improved to the point that distributed building with Distributed Network Builds is slower than local builds in most circumstances.
Maybe Jikes would work for you. You can achieve very similar effects with a clever ant script and nfs like filesystem...
If you're annoyed with waiting a long time for your java compiles, then you might consider one of the following:
break your project up into several different jar files (in a hierarchic dependency). With any luck, your changes will only affect source in one of those jars, while the others can continue to serve as dependencies.
break your project into groups of sources, perhaps by package, and use Apache ant to coordinate your compiling. I was always too lazy to use it, but you can set up explicit dependency management to avoid re-compiling stuff for which .class files already exist and are newer than the source. The effort that goes into setting this up once can reap dividends within a few days if the project is large and compiles are chewing up a lot of your time.
As opposed to multi-coring, reducing the amount of code that you need to recompile will also reduce your PC's energy consumption and carbon footprint ;)
I did write the start of one for java6
http://www.pointdefence.net/jarc/index.html
It's distributed at the java compiler task. So it would work well with parallel compilation of independent Maven modules.
I think the parallel compilation of independent Maven modules should be quite easy using some simple scripts - just pull from version control, change dir and run mvn clean compile. Add mvn deploy to get the artifact to your artifact repository.
This should work even with dependent modules, will need some work on synchronization though.
My coworker noticed that when javac is compiling it only utilizes a single core. Is there anything like the -j command with the gcc for Java that will allow us to distribute the compiler workload across cores or processors? If not, do you think that this will ever be possible or is there some sort of fundamental restriction as a result of Java's design?
The environment is Linux with the Sun J2SE 1.6.0.11 jdk.
Although not exactly an answer to your question, some build environments like ant also support parallelization of builds, so if your dependency graph permits you can have multiple javac's running at the same time.