I am adding more than 20 files to a fileset with various includes and excludes file. This takes a bit so the dependent task on Ant fileset is skipped by Ant. It wont run the dependent operation at all. This is inefficient. Does anyone know a better way to make the fileset to add files and then perform dependent instead of skipping it?
A way to do that is to use sleep https://ant.apache.org/manual/Tasks/sleep.html. I cannot use waitfor.
Example:
<fileset id="scr" dir="dir1/*" casesensitive="yes"
includes="bin/*, lib/* excludes="dir1/*.py />
<foreach param="file" in="scr">
<echo message="${file}"/>
</foreach>
</fileset>
The above example will skip printing the file name completely since Ant is still adding files to fileset.
The foreach task is not part of ANT, it is a 3rd party extension and I'm not entirely certain it's designed to work the way you think.
Instead could I suggest you print your fileset contents as follows:
<apply executable="echo">
<fileset dir="dir1/*" casesensitive="yes" includes="bin/*, lib/* excludes="dir1/*.py />
</apply>
Or like this if you want to use a fileset reference:
<fileset id="scr" dir="dir1/*" casesensitive="yes" includes="bin/*, lib/* excludes="dir1/*.py />
<apply executable="echo">
<fileset refid="scr"/>
</apply>
Note:
Ant is not a programming language so looping is not fun. If you need complex logic I highly recommend embedding a scripting language. I don't recommend ant-contrib
I was trying to reduce the time it takes for an ant build to complete. Most of the build time is taken by GWT compiler.
Following ant script is written on the lines of scripts found in official GWT examples.
Notice how two GWT modules are being passed to the Complier. When you run this script, the GWT compiler compiles the two modules sequentially.
<target name="gwtc" description="GWT compile to JavaScript">
<java failonerror="true" fork="true" classname="com.google.gwt.dev.Compiler">
........
........
<arg value="com.af.gwtmodules.dashboard.Dashboard" />
<arg value="com.af.gwtmodules.administration.Administration" />
<arg line=" -localWorkers 16" />
</java>
</target>
I changed the task to run 2 compile tasks in parallel and in each task I pass only one GWT module to the compiler.
<target name="gwtc" description="GWT compile to JavaScript">
<parallel threadsperprocessor="16">
<java failonerror="true" fork="true" classname="com.google.gwt.dev.Compiler">
........
........
<arg value="com.af.gwtmodules.dashboard.Dashboard" />
<arg line=" -localWorkers 16" />
</java>
<java failonerror="true" fork="true" classname="com.google.gwt.dev.Compiler">
........
........
<arg value="com.af.gwtmodules.administration.Administration" />
<arg line=" -localWorkers 16" />
</java>
</parallel>
</target>
This indeed runs faster as expected. However, I wonder whether the GWT compiler can do a better job at code optimization if it is given all modules at once instead of each module separately. For example, the two modules use a lot of common code. So if the compiler can see the entire code base at once, it can find more redundant code. In theory, it can create a single JS artefact for the common code and separate JS artifacts for code that is not common. This would have the effect of reducing download time for the user who accesses both modules as common JS artifact would be downloaded only once.
As far as I understand GWT modules are independent and so there would be no cross module optimizations. But the fact that GWT compiler internally does not parallelize this makes me think that there could be some cross module optimizations or other ramifications because of which Google engineers decided against parallelizing it beyond a limit.
I would like to know if parallelizing compile the way I have done, has any effect on quality of generated code.
If your CPU runs at 100% or you use all of available memory, it does not matter how many tasks you run in parallel. In fact, you may slow down the performance, not improve it, by pushing tasks in parallel.
You already set localWorkers to 16. That's a lot of parallel threads. By passing two tasks you simply double the number of threads. If you get any performance improvement from increasing this number - go for it, although your results look surprising (either your app is very small or your computer is a monster).
There are no optimization benefits from compiling modules sequentially vs in parallel, as far as I know. You can always verify it by looking the at the size of the compiled code.
You may find this post interesting:
GWT Compilation Performance
I am using Mercurial HG for versioning my projects. To keep my version-number in my build/compiled programs uptodate I am using this approach. I've modified the build.xml to override a predefined target.
<target name="-pre-compile">
<!-- getting the version in a fancy format -->
<exec executable="hg" outputproperty="version.mercurial">
<arg value="parent"/>
<arg value="--template"/>
<arg value="{latesttag}.{rev}-{branch}-{node|short}"/>
</exec>
<echo level="warning">Version ${version.mercurial}</echo>
<manifest file="${manifest.file}" mode="update">
<attribute name="Implementation-Version" value="${version.mercurial}"/>
</manifest>
</target>
** note hg.exe is in my classpath.
To retrieve the version property I use the following code snippet in my mainclass
String version = Main.class.getPackage().getImplementationVersion();
When I build my project and run it with this java -jar myproject.jar the version number is correct and from that point everything is fine.
But when I run it in my IDE the version string ist null, so the manifest file isn't read.
So this makes me thinking if this approach is valid or is there another, even a simpler way to keep the version number uptodate?
EDIT: Is there a more sophisticated way to include the version number into your program?
Consider using the output attribute to Ant's <exec ..> task to write a properties file instead. Then you can use the <properties file="..."> method to read it in Ant, and can load it like a normal properties file when running in your IDE.
Good for you for not using the KeywordsExtension for this. It's a horrible solution in a DVCS world.
I want the javac task to use jars from the system classpath, by which I mean the classpath that is set in the shell's environment before ant is started. That classpath is
CLASSPATH=D:\local\lib\java\*;.;C:\lib\java\*;C:\lib\java\db\*
on my system. I have popular jars there that are used by many projects. The basic snippet I use in the build file is
<target name="build">
<mkdir dir="${obj}" />
<javac srcdir="${src}" destdir="${obj}"
includes="**/*.java"
excludes="**/package-info.java **/deprecated/*.java"
includeAntRuntime="no" debug="true" debuglevel="source,lines"
>
<compilerarg value="-Xlint"/>
</javac>
</target>
That way ant only passes the output directory as classpath.
[javac] '-classpath'
[javac] 'D:\dev\tbull-projects\jsonc\obj'
(jsonc is the project I'm working on, and D:\dev\tbull-projects\jsonc is the working directory.)
I browsed the documentation for a while and came up with two attempts. First one was adding the attribute classpath="${java.class.path}" to the javac tag. That would pass a tremendously long classpath to the compiler, listing every single jar from ant's own lib directory and finally tools.jar from the JDK. Not the classpath that I wanted.
The second shot was setting
<property name="build.sysclasspath" value="first" />
before javac was invoked, and that got me in the right direction. Now these lines were among the output:
dropping D:\dev\tbull-projects\jsonc\D:\local\lib\java\* from path as it doesn't exist
dropping D:\dev\tbull-projects\jsonc\C:\lib\java\* from path as it doesn't exist
dropping D:\dev\tbull-projects\jsonc\C:\lib\java\db\* from path as it doesn't exist
dropping D:\dev\tbull-projects\jsonc\C:\Program Files\Java\jdk1.6.0_18\jre\lib\sunrsasign.jar from path as it doesn't exist
dropping D:\dev\tbull-projects\jsonc\C:\Program Files\Java\jdk1.6.0_18\jre\classes from path as it doesn't exist
Well, you can imagine that these paths really don't exist. I just don't get why ant constructed them this way. It would know how to do path arithmetic on Windows, would it?
Maybe my approach is flawed more fundamentally, so I'll let you know what I'm actually after. So I'm developing this project (a library), which uses another library. The project is gonna be open source, so I want other developers to be able to build it after they have downloaded the dependency library and placed it somewhere in their classpath.
From what I saw in other questions about ant+classpath, it appears that it's a custom fashion to distribute the dependency libs with the source code (so the classpath can be just like ./libs). But I surely don't want to have jars in my git repo. So how could that be done?
Set includeJavaRuntime=true in javac task.
<target name="build">
<mkdir dir="${obj}" />
<javac srcdir="${src}" destdir="${obj}"
includes="**/*.java"
excludes="**/package-info.java **/deprecated/*.java"
includeAntRuntime="no" includeJavaRuntime="true"
debug="true" debuglevel="source,lines">
<compilerarg value="-Xlint"/>
</javac>
</target>
Why wouldn't you set CLASSPATH in Ant? It's perfectly suited to do just that. You're making a mistake if you do anything else. Not only will it work, bu your build.xml will document the requirements as well.
When javac compiles the code , it tries to find the files in rt.jar in a symbol file called ct.sym (which is also present in lib directory). some files are missing in this symbol file.
i have to add a compile option to ignore symbol file and look directly in rt.jar.
so i have used this option -XDignore.symbol.file for ant i put this value in javac tag. it works perfectly if you use eclipse or any other ide .
<compilerarg value="-XDignore.symbol.file"/>
So , whenever you get ClassNotFoundException in using classes from rt.jar , and if the class is still present there , just try to add this argument in java compiler
To reference rt.jar from ant you may use:
<fileset dir="${java.home}/lib" includes="rt.jar"/>
Original details were found here: http://www.javaroots.com/2013/01/javac-error-using-classes-from-rtjar.html
If someone is new to java/ANT world, people who suggest maven are idiots
whatever happened to KISS principle?
OP, instead of using javascript abomination try this
<project default="build">
<property name="src" value="src" />
<property name="obj" value="obj" />
<property name="parent.dir" value="/jakarta-tomcat/common/lib" />
<path id="project.class.path">
<pathelement location="lib/" />
<fileset dir="${parent.dir}" includes="**/*.jar" />
</path>
<target name="build">
<delete dir="${obj}" />
<mkdir dir="${obj}" />
<javac srcdir="${src}" destdir="${obj}" includes="**/*.java" excludes="**/package-info.java **/deprecated/*.java" debug="true" debuglevel="source,lines" classpathref="project.class.path" />
</target>
It's pretty clear the folks behind java, and (or at least), ant, really really really don't want to see $CLASSPATH end up as storage for user installed libraries of the sort that 95% of other mainstream languages (C/C++, perl, python, ruby, etc. etc.) use. So this is a tough paradigm to swallow if you are used to general programming in most other mainstream languages.
The disinclination goes so far that it is obvious ant intentionally strips $CLASSPATH out of the environment, but an easy way around this is to just use a different variable.
<property name="classpath" location="${env.JAVALIBPATH}"/>
This will then work, no fuss, with both <javac> and <java> commands (classpath="${classpath}) which is good, because if you try this instead:
<property name="classpath" location="${env.CLASSPATH}"/>
There is no includeAntRuntime="false" option to <java> which would allow this to work. You simply cannot get $CLASSPATH in and someone has gone to lengths to make sure of it (without, apparently, and yikes, adding in a ponderous javascript hack).
Of course that means you need to use a separate env variable and for your distributed/production version stick to the Java "Sorry no user libs!" paradigm. That's not a big problem if you use a variable name that, if it becomes involved, will almost certainly be undefined on the target system.
Alternatively, there are the Maven Ant Tasks. These will allow you to use Maven's dependency mechanism in a way that, IMO, is cleaner than Ivy. But it's still not a great solution.
Soo... seems I have to answer the question myself. Passing the original classpath to the javac task can be achieved with this:
<!-- load environment into the env property -->
<property environment="env" />
<javac srcdir="${src}" destdir="${obj}"
includes="**/*.java"
excludes="**/package-info.java **/deprecated/*.java"
includeAntRuntime="no" includeJavaRuntime="no"
debug="true" debuglevel="source,lines"
>
<!-- add -classpath option manually -->
<compilerarg value="-classpath" />
<compilerarg value="${env.CLASSPATH}" />
<compilerarg value="-Xlint"/>
</javac>
That does the trick at least so far that the javac task now gets passed the correct classpath. Yet it will still not work, javac now spits these complaints:
[javac] warning: [path] bad path element "D:\local\lib\java\*": no such file or directory
[javac] warning: [path] bad path element "C:\lib\java\*": no such file or directory
[javac] warning: [path] bad path element "C:\lib\java\db\*": no such file or directory
This is a straight lie, these paths do very much exist. I use them all the time, and if I manually craft an equivalent javac invocation at the shell, it works like a charm. I suspect ant's javac doesn't resolve the jar files in those directories. I have to examine that.
UPDATE
It is indeed as I suspected, the wildcard is not resolved to the individual present jar files by the javac task. I managed to do the resolving manually, and now it works as it should. And that resolving was in fact a struggle on its own. So I'll leave the solution here for those poor souls fighting the same stupidity, hopefully before they ask people that have nothing else to do than bullshitting around (yes Anon, talking about you).
Turns out, ant lacks the most basic functionality that you would expect from a build tool. Also turns out that I'm not the first one to notice that. While solutions are rare, there is a very good post about Using JavaScript to make Apache Ant less painful, which really saved my day. Yes, ant can indeed be scripted, which seems not to be widely known, although it is not kept secret. You can safely assume, that Javascript is already available without installing additional libraries if you run ant on Java 6.
Soo... down to business. Here is the thing:
<target name="expand_classpath">
<script language="javascript"><![CDATA[
// the original classpath
var ocp = java.lang.System.getenv("CLASSPATH");
// ... split in parts
var ocp_parts = ocp.split(project.getProperty("path.separator"));
// where our individual jar filenames go,
// together with pure directories from ocp_parts
var expanded_parts = [ ];
for each (var part in ocp_parts) {
if (part.endsWith('*')) {
var dir = part.substring(0, part.length() - 1);
var f = new java.io.File(dir);
// don't know how to construct a java.io.FilenameFilter,
// therefore filter the filenames manually
for each (var file in f.listFiles())
if (file.getPath().endsWith('.jar'))
expanded_parts.push(file.getPath());
} else
expanded_parts.push(part);
}
var expanded = expanded_parts.join(project.getProperty("path.separator"));
project.setProperty("classpath.expanded", expanded);
]]></script>
<!-- <echo message="classpath.expanded = ${classpath.expanded}" /> -->
</target>
<target name="build" depends="expand_classpath">
<mkdir dir="${obj}" />
<javac srcdir="${src}" destdir="${obj}"
classpath="${classpath.expanded}"
includes="**/*.java"
excludes="**/package-info.java **/deprecated/*.java"
includeAntRuntime="no" includeJavaRuntime="no"
debug="true" debuglevel="source,lines"
>
<compilerarg value="-Xlint"/>
<compilerarg value="-Xlint:-fallthrough"/>
</javac>
</target>
I will assume that your "popular" JARs are well-known open-source projects. This means that they're available in the Maven central repository.
While I believe that using Maven is the best answer to this question, you can also hack something using Ant's <get> task. For example, to download the JUnit JAR (may have typos):
<property name="dependency.dir" value="${basedir}/dependencies"/>
<property name="junit.jar" value="junit-4.8.2.jar"/>
<property name="junit.url" value="http://search.maven.org/remotecontent?filepath=junit/junit/4.8.2/${junit.jar}"/>
<target name="download.dependencies">
<mkdir dir="${dependency.dir}/>
<get url="${junit.url}" dest="${dependency.dir}/${junit.jar}"/>
</target>
Of course, if you do this then you'll have to carefully configure your build scripts so that you don't do the download with every run. And you'll increase load on the Maven Central repository.
As an experiment we want to build our products using the Eclipse java compiler (ecj-3.5.jar downloaded from eclipse.org) on the runtime version of Java 6 instead of the JDK, and as I understand it, it is a matter of adding this jar to the ant classpath, and setting the build.compiler property to point to the adapter.
By including
<property name="build.compiler" value="org.eclipse.jdt.core.JDTCompilerAdapter" />
in my build.xml and invoking ant with a JRE, I get the expected error that the adapter cannot be found, and by adding ecj-3.5.jar to the classpath in the Eclipse panel I can compile my code as expected. I believe the same functionality to be available with "-lib foo.jar" from the command line with modern ants.
Now, I want to specify from within build.xml that I want ecj-3.5.jar on my classpath satisfying the same as above. We can already do this with ant tasks, so I believe it is possible.
So the question is: How can I add to the classpath used by javac to locate the compiler only from within build.xml?
It appears that the upcoming ant4eclipse 1.0 includes the Eclipse compiler (which is what I wanted to use this for), so by upgrading to that (from 0.5) should solve the problem we have.
2010-09-24: Ant4Eclipse is still at M4 without indication of when the release will happen.
2011-12-01: We have now migrated from ant to maven. The build.xml scripts hit the complexity wall and a fresh approach was needed. Anyone needing to choose what to do - do not go the ant4eclipse path except for trivial projects.
2012-11-30: A year later, the maven experience is still mostly good. There is a lot of quirks and changes in mindset but most make sense in the context. Maven can specify the compiler level on individual projects easily. We are looking into using ecj instead of javac (for several reasons) but for most purposes javac works nicely.
One way is to specify a reference to a componentdef when using javac.
<componentdef name="ecj"
classname="org.eclipse.jdt.core.JDTCompilerAdapter"
classpath="ecj-3.7.1.jar" />
<javac ....>
<ecj/>
</javac>
Another option is to set build.compiler as you have or the compiler attribute for javac and then specify a compilerclasspath for javac. This is a normal path like structure to hold the classpath for loading your compiler adapter.
<javac compiler="org.eclipse.jdt.core.JDTCompilerAdapter" ....>
<compilerclasspath>
...
</compilerclasspath>
</javac>
See the javac task documentation in the Ant manual for more details. Note that both these both solutions only work from Ant 1.8 onwards.
Reading Running Ant via Java. I think you can write a simple wrapper that will properly set a classpath and add your jar file to the resulting class path.
Here I'm just cutting and pasting the sample from the above link with addition of the library that you are interested in to the classpath:
<java
classname="org.apache.tools.ant.launch.Launcher"
fork="true"
failonerror="true"
dir="${sub.builddir}"
timeout="4000000"
taskname="startAnt"
>
<classpath>
<pathelement location="${ant.home}/lib/ant-launcher.jar"/>
<pathelement location="/path/to/ecj-3.5.jar"/>
</classpath>
<arg value="-buildfile"/>
<arg file="${sub.buildfile}"/>
<arg value="-Dthis=this"/>
<arg value="-Dthat=that"/>
<arg value="-Dbasedir=${sub.builddir}"/>
<arg value="-Dthe.other=the.other"/>
<arg value="${sub.target}"/>
</java>
I think you can even reuse the same build file, just give a different target as an entry point.