We Keep Coding

build.xml for ivy in Netbeans

I am trying to integrate Apache Ivy in a Netbeans web app project.
My build.xml file follows the netbeans tutorial:
http://wiki.netbeans.org/FaqIvy
Specifically:
<target name="-ivy-retrieve">
<ivy:retrieve/> <!-- Load dependencies to the project -->
<pathconvert property="ivy.classpath.computed" dirsep="/" pathsep=":">
<path>
<fileset dir="lib" includes="*.jar"/>
</path>
<map from="${basedir}${file.separator}" to=""/>
</pathconvert>
<propertyfile file="nbproject/project.properties">
<entry operation="=" key="ivy.classpath" value="${ivy.classpath.computed}"/>
</propertyfile>
</target>
<target name="-pre-compile" depends="-ivy-retrieve"/>
<target name="-pre-compile-single" depends="-ivy-retrieve"/>
<target name="-post-clean">
<delete dir="lib"/>
</target>
Strangely, this updates ivy.classpath in the nbprojects.project.properties file but I can't seem to get the build's WEB-INF/lib folder to get populated with the .jar's that are listed as dependencies.
Strangely, I had this working the other day and was able to populate the build's WEB-INF/lib folder and now when I remove the dependency, the same jar files are included in the build (rather than removing them).
Any suggestions on how to change the build.xml file? Or anything else that will get Ivy successfully integrated into my build process (Ivy is detected: I see "resolution report" and "retrieving" in the output while building).
Some things I've tried:
I played around with different options in the above linked tutorial (tried including:
<property name="ivy.lib.dir" value="web/WEB-INF/lib/"/> in the -ivy-retrieve task or adding the <ivy:resolve/> task before <ivy:retrieve/>)

The ivy:retrieve task has a sync property which removes not longer relevant dependencies.
Also, doing a clean and build helps.

Ivy dependency as provided

Problem: I need to have a lib on the eclipse classpath that should not be deployed to Tomcat. (In a maven project it would be scope provided)
Explanation:
I've setup a project with some Ivy dependencies and had to externalize a configuration as JNI (mail/session) in order to do it I had to put the mail-1.4.7.jar inside the Tomcat lib folder.
The problem is that I have a dependency that add to my classpath the javax.mail-1.5.2.jar so I change it to:
<dependency org="org.apache.logging.log4j" name="log4j-core" rev="2.2">
<exclude org="com.sun.mail" name="javax.mail"/>
</dependency>
The problem now is that my project break (compilation errors) because of missing mail classes such as javax.mail.MessagingException
So I have to add the mail dependency but only to eclipse. I've tried some configurations as explained here from what I know from Maven behavior with no avail.
Keeping the mail dependency only in the project, breaks Tomcat, keeping it on both tomcat and project breaks project. When I manually remove it from my project lib folder (WEB-INF\lib), after deploy the project, it works properly.
Bottom line (after deploy):
tomcatFolder
|_lib
| |_...
| |_mail-1.4.7.jar
| |_...
|_webapps
|_myproject
|_WEB-INF
|_lib
|_...
|_javax.mail-1.5.2.jar //need to remove it at deploy time only
|_...
Can't change it to maven right now. But it is in process :)

This is really a duplicate of this question:
How to copy runtime libraries without the provided ones in IVY
But.. from your question I suspect you're not using ivy configuration mappings. This is unfortunate because this is the mechanism used by ivy to logically group dependencies into functional groupings, similar to how Maven maintains scopes. The following posting attempts to bridge this understanding
How are maven scopes mapped to ivy configurations by ivy
Furthermore you are also using Eclipse, which means that unless you're using the ivy plugin you effectively have two build mechanisms. (ivy and eclipse). I would recommend fixing your ANT build first and then look at how to maintain the Eclipse classpath second.
Example
The first section describes how configurations are declared and used in the ivy file and the second section explains how the ivy ANT tasks are used in the build logic.
ivy.xml
You should always declare ivy configurations and use these to control your classpaths. In my builds I always have at least three: compile, runtime and test. Notice how the extends attribute is used to create relationships between the configs, because runtime should also include the compile dependencies.
Adding an additional one for the provided scope jars is easy. Simple stand-alone configuration:
<ivy-module version="2.0">
<info organisation="com.myspotontheweb" module="demo"/>
<configurations>
<conf name="compile" description="Required to compile application"/>
<conf name="runtime" description="Additional run-time dependencies" extends="compile"/>
<conf name="test" description="Required for test only" extends="runtime"/>
<conf name="provided" description="Needed for compile, but will be present on the target platform."/>
</configurations>
<dependencies>
<!-- compile dependencies -->
<dependency org="org.slf4j" name="slf4j-api" rev="1.7.5" conf="compile->default"/>
<!-- runtime dependencies -->
<dependency org="org.slf4j" name="slf4j-log4j12" rev="1.7.5" conf="runtime->default"/>
<!-- test dependencies -->
<dependency org="junit" name="junit" rev="4.11" conf="test->default"/>
<!-- provided dependencies -->
<dependency org="org.apache.tomcat" name="servlet-api" rev="6.0.16" conf="provided->master"/>
</dependencies>
</ivy-module>
It's the configuration mappings that make things special. The simple explanation is that they fall into two basic types when pulling from a Maven repository:
conf="local_configuration->default"
conf="local_configuration->master"
The first means include the remote module and all its dependencies. The second means include the remote module and exclude it's dependencies. This means you don't need the following exclude trickery:
<dependency org="org.apache.logging.log4j" name="log4j-core" rev="2.2">
<exclude org="com.sun.mail" name="javax.mail"/>
</dependency>
You simply use the following, if all you want is the log4j-core jar:
<dependency org="org.apache.logging.log4j" name="log4j-core" rev="2.2" conf="provided->master"/>
Additional notes:
In ivy mapping to the remote "default" configuration will pull down only the jars you need. It will exclude optional dependencies and other stuff like javadocs.
Sometimes "excludes" are necessary when module authors get their dependencies wrong.
build.xml
The resolve target will pull down dependencies, generate a report and create the compile and test classpaths. Note the use of configurations to determine which jar groupings should be used:
<target name="resolve" description="Use ivy to resolve classpaths">
<ivy:resolve/>
<ivy:report todir='${build.dir}/ivy-reports' graph='false' xml='false'/>
<ivy:cachepath pathid="compile.path" conf="compile,provided"/>
<ivy:cachepath pathid="test.path" conf="test,provided"/>
</target>
These classpath references are then used by the compile target as normal:
<target name="compile" depends="resolve,resources" description="Compile code">
<mkdir dir="${build.dir}/classes"/>
<javac srcdir="${src.dir}" destdir="${build.dir}/classes" includeantruntime="false" debug="true" classpathref="compile.path"/>
</target>
<target name="compile-tests" depends="compile" description="Compile tests">
<mkdir dir="${build.dir}/test-classes"/>
<javac srcdir="${test.src.dir}" destdir="${build.dir}/test-classes" includeantruntime="false" debug="true">
<classpath>
<path refid="test.path"/>
<pathelement path="${build.dir}/classes"/>
</classpath>
</javac>
</target>
And the test target:
<target name="test" depends="compile-tests" description="Run unit tests">
<mkdir dir="${build.dir}/test-reports"/>
<junit printsummary="yes" haltonfailure="yes">
<classpath>
<path refid="test.path"/>
<pathelement path="${build.dir}/classes"/>
<pathelement path="${build.dir}/test-classes"/>
</classpath>
<formatter type="xml"/>
<batchtest fork="yes" todir="${build.dir}/test-reports">
<fileset dir="${test.src.dir}">
<include name="**/*Test*.java"/>
<exclude name="**/AllTests.java"/>
</fileset>
</batchtest>
</junit>
</target>
Lastly the ivy retrieve task is used to build the war file. Only the "runtime" configuration jars are used:
<target name="package" depends="test" description="Create the WAR file">
<ivy:retrieve pattern="${build.dir}/lib/[artifact].[ext]" conf="runtime"/>
<war destfile="${war.file}" webxml="${resources.dir}/web.xml">
<fileset dir="${resources.dir}" excludes="web.xml"/>
<lib dir="${build.dir}/lib"/>
</war>
</target>
In conclusion the cachepath ivy task is used to create classpath references based on the ivy configurations and the retrieve task is used when assembling the war file.

approch for inter-project dependencies

I have a console-based build system that's using Ant and Ivy and would like to avoid using Maven. I fetch my project's external dependencies using Ivy. My question is how best to handle my internal project dependencies. I.e. I build a number of library files (JARs) on which my modules are based and which unlikely to be used in another context. So these internal project dependencies are simply captured by build.xml files directly identifying the location of the needed JARs via relative paths (since everything's under the same source tree in the repository). E.g. I have Ant "code" like this:
<path id = "compile.classpath">
<fileset dir="${internal-project-dependency-a.dir}">
<include name ="*.jar"/>
</fileset>
<fileset dir="${internal-project-dependency-b.dir}">
<include name ="*.jar"/>
</fileset>
<fileset dir="${internal-project-dependency-c.dir}">
<include name ="*.jar"/>
</fileset>
</path>
My question is: is the above approach acceptable (from a best-practice point of view) or should I instead package and publish even my internal project dependencies in some lightweight "local-only" Ivy repository (if there's such a thing)? The way I've setup my system anyone with Ivy and Ant can checkout my sources from github and build everything from the console without needing to configure any other kind of Ivy repository information and I'd like to keep that property.

Ivy supports different kinds or repositories, described in the documentation.
local
shared
public
Without customization these are located under your "~/.ivy2" directory. So, I would recommend configuring each builds to publish to your local repository and simply reference dependencies as normal. All you need to do is ensure that on a fresh machine the build order is correct, so that the local repo is populated in the correct order (See the buildlist task).
Using the local repo is not very different from referencing the jars in known relative locations. It does have the benefit of making each project decoupled from the other.
Example
├── build.xml
└── ivy.xml
Project publishes 3 files to the local repository:
$ find ~/.ivy2/local -type f
/home/mark/.ivy2/local/myorg/hello/1.0/ivys/ivy.xml.sha1
/home/mark/.ivy2/local/myorg/hello/1.0/ivys/ivy.xml
/home/mark/.ivy2/local/myorg/hello/1.0/ivys/ivy.xml.md5
/home/mark/.ivy2/local/myorg/hello/1.0/docs/English.txt.md5
/home/mark/.ivy2/local/myorg/hello/1.0/docs/Spanish.txt.sha1
/home/mark/.ivy2/local/myorg/hello/1.0/docs/English.txt.sha1
/home/mark/.ivy2/local/myorg/hello/1.0/docs/Irish.txt.sha1
/home/mark/.ivy2/local/myorg/hello/1.0/docs/Spanish.txt.md5
/home/mark/.ivy2/local/myorg/hello/1.0/docs/English.txt
/home/mark/.ivy2/local/myorg/hello/1.0/docs/Spanish.txt
/home/mark/.ivy2/local/myorg/hello/1.0/docs/Irish.txt.md5
/home/mark/.ivy2/local/myorg/hello/1.0/docs/Irish.txt
build.xml
<project name="demo" default="publish" xmlns:ivy="antlib:org.apache.ivy.ant">
<target name="init">
<ivy:resolve/>
</target>
<target name="build" depends="init">
<mkdir dir="build"/>
<echo file="build/English.txt">Hello world</echo>
<echo file="build/Irish.txt">Dia dhuit</echo>
<echo file="build/Spanish.txt">Hola mundo</echo>
</target>
<target name="publish" depends="clean,build">
<ivy:publish pubrevision="1.0" status="release" resolver="local" >
<artifacts pattern="build/[artifact].[ext]"/>
</ivy:publish>
</target>
<target name="clean" description="Cleanup build files">
<delete dir="build"/>
</target>
</project>
ivy.xml
<ivy-module version="2.0">
<info organisation="myorg" module="hello"/>
<publications>
<artifact name="English" ext="txt" type="doc"/>
<artifact name="Irish" ext="txt" type="doc"/>
<artifact name="Spanish" ext="txt" type="doc"/>
</publications>
</ivy-module>

ivy build - multiple artifacts of different versions

We recently started using ant-ivy for a java webapp and the journey has been good so far. I have specified transitive='false' for all dependencies and listed log4j version 1.2.16 as a dependency. I have to use a recent version of log4j as I rely on an enhanced pattern layout.
Main problem - when the project is built, there are multiple versions of log4j in the ivy-lib folder. When looking at the package explorer in Eclipse, I see the right version and seems like it downloads only 1.2.16 from ivycachepath.
Any pointers? Do I need to remove each dependency individually and see which one brings older versions?
Thanks for the help.

I've included an example ant build which downloads log4j as a dependency, into a local directory called "lib".
$ ant
$ find lib
lib
lib/log4j-1.2.16.jar
As you can see only one version of the jar is present. Hope this helps in diagnosing your issue.
build.xml
<project xmlns:ivy="antlib:org.apache.ivy.ant" name="hello-ivy" default="retrieve">
<property name="lib.dir" location="lib"/>
<property name="report.dir" location="report"/>
<target name="retrieve" description="Retrieve dependencies locally">
<ivy:retrieve pattern="${lib.dir}/[artifact]-[revision](-[classifier]).[ext]"/>
<ivy:report todir="${report.dir}" graph="false"/>
</target>
<target name="clean" description="Remove build directories">
<delete dir="${lib.dir}"/>
<delete dir="${report.dir}"/>
</target>
<target name="clean-all" depends="clean" description="clean ivy cache">
<ivy:cleancache />
</target>
</project>
ivy.xml
<ivy-module version="2.0">
<info organisation="org.demo" module="demo"/>
<configurations defaultconfmapping="compile->default">
<conf name="compile" description="jars used for compilation"/>
</configurations>
<dependencies>
<dependency org="log4j" name="log4j" rev="1.2.16"/>
</dependencies>
</ivy-module>

Get ivy:retrieve to not copy a jar into two configurations?

I'm trying to set up my ant build so that the targets which run Ivy functions are completely separated from the ones that the continuous build and most developers run. I want one target to download updated dependencies, which I'll check in. Other targets will set up their classpath by including *.jar from the relevant directory.
I have two configurations:
<configurations>
<conf name="compile" />
<conf name="test" />
</configurations>
I have some dependencies:
<dependency
org="my.org"
name="some-lib"
rev="latest.release"
conf="compile->default" />
<dependency
org="my.org"
name="some-test-lib"
rev="latest.release"
conf="test->default" />
And I download those dependencies and their transitive dependencies using ivy:retrieve:
<ivy:retrieve
pattern="lib/[conf]/[type]/[artifact]-[revision].[ext]"
sync="true"
file="ivy.xml" />
The problem is that I'm seeing some duplicates jars between the compile and the test directories, which come from transitive dependencies. Since I want to check in all these jars and use them for creating classpaths, I'd like to avoid duplicates. Is this possible?
lib/compile/jar/some-lib-1.0.jar
lib/compile/jar/slf4j-api-1.5.11.jar
lib/test/jar/some-test-lib-1.0.jar
lib/test/jar/junit-4.7.jar
lib/test/jar/slf4j-api-1.5.11.jar

This is not duplication, each configuration is a separate set of jars and the ivy restrieve task is faithly creating each set....
Perhaps it would make more sense to create the classpaths directly, rather than populating a local lib directory.
Here's a snippet of my ANT build files:
<target name="get-dependencies">
<ivy:resolve/>
<ivy:cachepath pathid="compile.path" conf="compile" />
<ivy:cachepath pathid="test.path" conf="test" />
</target>
<target name="compile" depends="get-dependencies">
<javac srcdir="src" destdir="build/classes" classpathref="compile.path"/>
</target>
I normally only use the ivy retrieve task when I need to create a local copy of a set of jars, for example assembling a web app's directory:
<ivy:retrieve pattern="build/WEB_INF/lib/[artifact].[ext]" conf="runtime"/>
Update
Another alternative is to instruct ivy to exclude the slf4j module when downloading transient dependencies, as follows:
<dependency org="my.org" name="some-lib" rev="latest.release" conf="compile->default">
<exclude module="slf4j-api"/>
</dependency>
<dependency org="my.org" name="some-test-lib" rev="latest.release" conf="test->default">
<exclude module="slf4j-api"/>
</dependency>

If I can do this through Ivy directly, that would be best. For now I've solved the problem by just deleting duplicates using ant.

Try the following. Your test should extend compile
<dependency
org="my.org"
name="some-test-lib"
rev="latest.release"
conf="**test->compile**" />
If i am right IVY should find that test extends compile and would download slf4j only once.