I was trying to reduce the time it takes for an ant build to complete. Most of the build time is taken by GWT compiler.
Following ant script is written on the lines of scripts found in official GWT examples.
Notice how two GWT modules are being passed to the Complier. When you run this script, the GWT compiler compiles the two modules sequentially.
<target name="gwtc" description="GWT compile to JavaScript">
<java failonerror="true" fork="true" classname="com.google.gwt.dev.Compiler">
........
........
<arg value="com.af.gwtmodules.dashboard.Dashboard" />
<arg value="com.af.gwtmodules.administration.Administration" />
<arg line=" -localWorkers 16" />
</java>
</target>
I changed the task to run 2 compile tasks in parallel and in each task I pass only one GWT module to the compiler.
<target name="gwtc" description="GWT compile to JavaScript">
<parallel threadsperprocessor="16">
<java failonerror="true" fork="true" classname="com.google.gwt.dev.Compiler">
........
........
<arg value="com.af.gwtmodules.dashboard.Dashboard" />
<arg line=" -localWorkers 16" />
</java>
<java failonerror="true" fork="true" classname="com.google.gwt.dev.Compiler">
........
........
<arg value="com.af.gwtmodules.administration.Administration" />
<arg line=" -localWorkers 16" />
</java>
</parallel>
</target>
This indeed runs faster as expected. However, I wonder whether the GWT compiler can do a better job at code optimization if it is given all modules at once instead of each module separately. For example, the two modules use a lot of common code. So if the compiler can see the entire code base at once, it can find more redundant code. In theory, it can create a single JS artefact for the common code and separate JS artifacts for code that is not common. This would have the effect of reducing download time for the user who accesses both modules as common JS artifact would be downloaded only once.
As far as I understand GWT modules are independent and so there would be no cross module optimizations. But the fact that GWT compiler internally does not parallelize this makes me think that there could be some cross module optimizations or other ramifications because of which Google engineers decided against parallelizing it beyond a limit.
I would like to know if parallelizing compile the way I have done, has any effect on quality of generated code.
If your CPU runs at 100% or you use all of available memory, it does not matter how many tasks you run in parallel. In fact, you may slow down the performance, not improve it, by pushing tasks in parallel.
You already set localWorkers to 16. That's a lot of parallel threads. By passing two tasks you simply double the number of threads. If you get any performance improvement from increasing this number - go for it, although your results look surprising (either your app is very small or your computer is a monster).
There are no optimization benefits from compiling modules sequentially vs in parallel, as far as I know. You can always verify it by looking the at the size of the compiled code.
You may find this post interesting:
GWT Compilation Performance
Related
I receive the following warning:
[javac] build.xml:9: warning: 'includeantruntime' was not set,
defaulting to build.sysclasspath=last; set to false for repeatable builds
What does this mean?
Ant Runtime
Simply set includeantruntime="false":
<javac includeantruntime="false" ...>...</javac>
If you have to use the javac-task multiple times you might want to consider using PreSetDef to define your own javac-task that always sets includeantruntime="false".
Additional Details
From http://www.coderanch.com/t/503097/tools/warning-includeantruntime-was-not-set:
That's caused by a misfeature
introduced in Ant 1.8. Just add an
attribute of that name to the javac
task, set it to false, and forget it
ever happened.
From http://ant.apache.org/manual/Tasks/javac.html:
Whether to include the Ant run-time
libraries in the classpath; defaults
to yes, unless build.sysclasspath is
set. It is usually best to set this to
false so the script's behavior is not
sensitive to the environment in which
it is run.
As #Daniel Kutik mentioned, presetdef is a good option. Especially if one is working on a project with many build.xml files which one cannot, or prefers not to, edit (e.g., those from third-parties.)
To use presetdef, add these lines in your top-level build.xml file:
<presetdef name="javac">
<javac includeantruntime="false" />
</presetdef>
Now all subsequent javac tasks will essentially inherit includeantruntime="false". If your projects do actually need ant runtime libraries, you can either add them explicitly to your build files OR set includeantruntime="true". The latter will also get rid of warnings.
Subsequent javac tasks can still explicitly change this if desired, for example:
<javac destdir="out" includeantruntime="true">
<src path="foo.java" />
<src path="bar.java" />
</javac>
I'd recommend against using ANT_OPTS. It works, but it defeats the purpose of the warning. The warning tells one that one's build might behave differently on another system. Using ANT_OPTS makes this even more likely because now every system needs to use ANT_OPTS in the same way. Also, ANT_OPTS will apply globally, suppressing warnings willy-nilly in all your projects
Chet Hosey wrote a nice explanation here:
Historically, Ant always included its own runtime in the classpath made available to the javac task. So any libraries included with Ant, and any libraries available to ant, are automatically in your build's classpath whether you like it or not.
It was decided that this probably wasn't what most people wanted. So now there's an option for it.
If you choose "true" (for includeantruntime), then at least you know that your build classpath will include the Ant runtime. If you choose "false" then you are accepting the fact that the build behavior will change between older versions and 1.8+.
As annoyed as you are to see this warning, you'd be even less happy if your builds broke entirely. Keeping this default behavior allows unmodified build files to work consistently between versions of Ant.
The answer from Daniel works just perfect. Here is a sample snippet that I added to my build.xml:
<target name="compile">
<mkdir dir="${classes.dir}"/>
<javac srcdir="${src.dir}" destdir="${classes.dir}" includeantruntime="false">
<!-- ^^^^^^^^^^^^^^^^^^^^^^^^^ -->
<classpath>
<path id="application" location="${jar.dir}/${ant.project.name}.jar"/>
<path id="junit" location="${lib.dir}/junit-4.9b2.jar"/>
</classpath>
</javac>
</target>
If you like me work from commandline the quick answer is executing
export ANT_OPTS=-Dbuild.sysclasspath=ignore
And then run your ant script again.
Use <property name="build.sysclasspath" value="last"/> in your build.xml file
For more details search includeAntRuntime in Ant javac
Other possible values could be found here
i faced this same, i check in in program and feature. there was an update has install for jdk1.8 which is not compatible with my old setting(jdk1.6.0) for ant in eclipse.
I install that update.
right now, my ant project is build success.
Try it, hope this will be helpful.
By specifying "-lib foo.jar" to ant I get the behaviour that the classes in foo.jar is added to the ant classloader and are available for various tasks taking a class name argument.
I'd like to be able to specify the same behaviour but only from inside build.xml (so we can do this on a vanilla ant).
For taskdefs we have functioning code looking like:
<taskdef resource="net/sf/antcontrib/antlib.xml" description="for/foreach tasks">
<classpath>
<pathelement location="${active.workspace}/ant-contrib-1.X/lib/ant-contrib.jar" />
</classpath>
</taskdef>
where the definition is completely provided from the ant-contrib.jar listed.
What is the equivalent mechanism for the "global" ant classpath?
(I have thought out that this is the way to get <javac> use ecj-3.5.jar to compile with on a JRE - Specifying the Eclipse compiler completely from _within_ build.xml - in a way compatible with ant 1.7. Better suggestions are welcome :)
EDIT: It appears that the about-to-be-released version 1.0 of ant4eclipse includes ecj. This does not answer the question, but may solve my basic problem.
if you ever decide to move to Ant 1.8 you can use following construction:
<import>
<javaresource name="resource_name.xml">
<classpath location="path_to_jar.jar" />
</javaresource>
</import>
I've posted an answer to the question that you've linked. I'll repeat a general solution here.
Reading Running Ant via Java, I think you can write a simple wrapper that will properly set a classpath and add any 3rd party library to the resulting class path.
Here I'm just cutting and pasting the sample from the above link with addition of the library to the classpath:
<java
classname="org.apache.tools.ant.launch.Launcher"
fork="true"
failonerror="true"
dir="${sub.builddir}"
timeout="4000000"
taskname="startAnt"
>
<classpath>
<pathelement location="${ant.home}/lib/ant-launcher.jar"/>
<pathelement location="/path/to/3rd-party-lib.jar"/>
</classpath>
<arg value="-buildfile"/>
<arg file="${sub.buildfile}"/>
<arg value="-Dthis=this"/>
<arg value="-Dthat=that"/>
<arg value="-Dbasedir=${sub.builddir}"/>
<arg value="-Dthe.other=the.other"/>
<arg value="${sub.target}"/>
</java>
I think you can even reuse the same build file by referencing ${ant.file}, just give a different target as an entry point.
EDIT
I think, this solution should only be used as a last resort measure. If task in general supports a <classpath> tag, then add required libraries locally to that specific task. Your gave a perfect example of this in your question with an ant-contrib taskdef.
As an experiment we want to build our products using the Eclipse java compiler (ecj-3.5.jar downloaded from eclipse.org) on the runtime version of Java 6 instead of the JDK, and as I understand it, it is a matter of adding this jar to the ant classpath, and setting the build.compiler property to point to the adapter.
By including
<property name="build.compiler" value="org.eclipse.jdt.core.JDTCompilerAdapter" />
in my build.xml and invoking ant with a JRE, I get the expected error that the adapter cannot be found, and by adding ecj-3.5.jar to the classpath in the Eclipse panel I can compile my code as expected. I believe the same functionality to be available with "-lib foo.jar" from the command line with modern ants.
Now, I want to specify from within build.xml that I want ecj-3.5.jar on my classpath satisfying the same as above. We can already do this with ant tasks, so I believe it is possible.
So the question is: How can I add to the classpath used by javac to locate the compiler only from within build.xml?
It appears that the upcoming ant4eclipse 1.0 includes the Eclipse compiler (which is what I wanted to use this for), so by upgrading to that (from 0.5) should solve the problem we have.
2010-09-24: Ant4Eclipse is still at M4 without indication of when the release will happen.
2011-12-01: We have now migrated from ant to maven. The build.xml scripts hit the complexity wall and a fresh approach was needed. Anyone needing to choose what to do - do not go the ant4eclipse path except for trivial projects.
2012-11-30: A year later, the maven experience is still mostly good. There is a lot of quirks and changes in mindset but most make sense in the context. Maven can specify the compiler level on individual projects easily. We are looking into using ecj instead of javac (for several reasons) but for most purposes javac works nicely.
One way is to specify a reference to a componentdef when using javac.
<componentdef name="ecj"
classname="org.eclipse.jdt.core.JDTCompilerAdapter"
classpath="ecj-3.7.1.jar" />
<javac ....>
<ecj/>
</javac>
Another option is to set build.compiler as you have or the compiler attribute for javac and then specify a compilerclasspath for javac. This is a normal path like structure to hold the classpath for loading your compiler adapter.
<javac compiler="org.eclipse.jdt.core.JDTCompilerAdapter" ....>
<compilerclasspath>
...
</compilerclasspath>
</javac>
See the javac task documentation in the Ant manual for more details. Note that both these both solutions only work from Ant 1.8 onwards.
Reading Running Ant via Java. I think you can write a simple wrapper that will properly set a classpath and add your jar file to the resulting class path.
Here I'm just cutting and pasting the sample from the above link with addition of the library that you are interested in to the classpath:
<java
classname="org.apache.tools.ant.launch.Launcher"
fork="true"
failonerror="true"
dir="${sub.builddir}"
timeout="4000000"
taskname="startAnt"
>
<classpath>
<pathelement location="${ant.home}/lib/ant-launcher.jar"/>
<pathelement location="/path/to/ecj-3.5.jar"/>
</classpath>
<arg value="-buildfile"/>
<arg file="${sub.buildfile}"/>
<arg value="-Dthis=this"/>
<arg value="-Dthat=that"/>
<arg value="-Dbasedir=${sub.builddir}"/>
<arg value="-Dthe.other=the.other"/>
<arg value="${sub.target}"/>
</java>
I think you can even reuse the same build file, just give a different target as an entry point.
We're using the task within our master build to invoke targets in separate ant builds for each of our sub-projects. So far so good, we get things built, we can even run JUnit tasks within each one and everybody is happy. However...
We want to take it to the next level, we would like to have a single JUnit report generated from the JUnit test result XML for all of our sub-projects but if we terminate the build whenever any of the sub-projects has a unit test failure, we never get to the point where we can generate a unified report. So that suggests that we would somehow note that unit tests failed within one or more of the sub-projects and not fail immediately but wait until the end of the master build to fail.
What mechanism exists for that communication from the child builds up to the master build though? Properties are immutable and everything else we think of (properties files we update, files we touch, etc.) seem like horrible kludges. Is there a way to do this that fits in nicely with Ant and doesn't seem like something horrible we grafted on?
The junit task supports the haltonerror and haltonfailure attributes, which, if set to false, will cause the ant script to continue to run even if a test fails. There is also an errorproperty and failureproperty you can set instead. You can then copy your junit reports to a master directory (with all success and failures), and use the fail task to fail if either of those properties have been set.
Something along the lines of:
<target name="run-tests" >
<junit printsummary="on" fork="yes" haltonfailure="false" haltonerror="false" errorproperty="test.failed" failureproperty="tests.failed" showoutput="true" maxmemory="512m">
<classpath refid="classpath" />
<formatter type="xml" />
<batchtest todir="test/reports">
<fileset dir="${build.classes.dir}">
<patternset refid="testfiles" />
</fileset>
</batchtest>
</junit>
<copy todir="test/master/reports" dir="test/reports" />
<fail if="tests.failed"/>
</target>
OK, I never got an answer on this one that I liked that much but we did end up finding a good solution we like a lot. We switched from using Anthill OS as our build server to Hudson and after we did that we were able to take advantage of a Hudson feature where it will aggregate JUnit results from any number of locations to produce a single report that goes with each build (whether successful or not). So, in short, use Hudson. It rocks!
I am running my junit tests via ant and they are running substantially slower than via the IDE. My ant call is:
<junit fork="yes" forkmode="once" printsummary="off">
<classpath refid="test.classpath"/>
<formatter type="brief" usefile="false"/>
<batchtest todir="${test.results.dir}/xml">
<formatter type="xml"/>
<fileset dir="src" includes="**/*Test.java" />
</batchtest>
</junit>
The same test that runs in near instantaneously in my IDE (0.067s) takes 4.632s when run through Ant. In the past, I've been able to speed up test problems like this by using the junit fork parameter but this doesn't seem to be helping in this case. What properties or parameters can I look at to speed up these tests?
More info:
I am using the reported time from the IDE vs. the time that the junit task outputs. This is not the sum total time reported at the end of the ant run.
So, bizarrely, this problem has resolved itself. What could have caused this problem? The system runs on a local disk so that is not the problem.
Here's a blind guess: try increasing the maximum heap size available to the forked VM by using a nested <jvmarg> tag to set the -Xmx option.
I'm guessing it's because your antscript is outputing results to XML files, whereas the IDE is keeping those in memory. It takes longer to write a file than to not write a file.
todir="${test.results.dir}/xml"
That's the part of the <batchtest> call that tells it to stick the results into that directory. It looks like leaving it off just tells it to stick the results in the "current directory", whatever that is. At first glance I didn't see anything to turn it all the way off.
Difficult to tell with that information. First thing I would do is look at the test results and determine if all the individual tests are running uniformly slower or if it can be narrowed down to a certain subset of test cases.
(The zero'th thing I would do is make sure that my ant task is using the same JVM as Eclipse and that the classpath dependencies and imported JARs are really and truly identical)
Maybe you are seeing that because Eclipse do incremental compiling and Ant don't. Can you confirm that this time is wasted only in the test target?
For the record, I found my problem. We have been using a code obfuscator for this project, and the string encryption portion of that obfuscator was set to "maximum". This slowed down any operation where strings were present.
Turning down the string encryption to a faster mode fixed the problem.
Try setting fork, forkmode and threads to these values:
<junit fork="yes" forkmode="perTest" printsummary="off" threads="4">
<classpath refid="test.classpath"/>
<formatter type="brief" usefile="false"/>
<batchtest todir="${test.results.dir}/xml">
<formatter type="xml"/>
<fileset dir="src" includes="**/*Test.java" />
</batchtest>
</junit>
Also see https://ant.apache.org/manual/Tasks/junit.html
For me, adding forkmode="once" for the <junit> element and adding usefile="false" for the <formatter> element makes the tests run much faster. Also remove the formatters that you don't need.