We Keep Coding

Java - Apache Ivy not resolving dependencies correctly

I am unable to compile source with Ant via terminal as it seems like dependencies are not resolved correctly despite me issuing 'ant resolve or ant retrieve'?
My build.xml and ivy.xml below
build.xml
<!-- ANT HOME ENVIRONMENT VARIABLE -->
<property name="ant.home" value="${env.ANT_HOME}" />
<!-- IVY HOME DIRECTORY -->
<property name="ivy.home" value="${ant.home}" />
<!-- IVY2 JAR DIRECTORY (REPOSITORY) -->
<property name="ivy.default.ivy.user.dir" value="${user.home}/.ivy2"/>
<!-- DOWNLOAD IVY -->
<target name="setup" description="Install ivy">
<mkdir dir="${user.home}/.ivy2" />
<get dest="${ivy.home}/lib/ivy.jar" src="http://search.maven.org/remotecontent?filepath=org/apache/ivy/ivy/2.3.0/ivy-2.3.0.jar"/>
</target>
<!-- RESOLVE CLASSPATHS -->
<target name="resolve" description="Use ivy to resolve classpaths">
<ivy:resolve file="ivy.xml" />
<ivy:report todir='target/ivy-reports' graph='false' xml='false'/>
<ivy:cachepath pathid="ivy.path" conf="compile" />
</target>
<!-- RETRIEVE DEPENDANCIES AFTER RESOLVING-->
<target name="retrieve" depends="resolve" description="Use ivy to retrieve dependencies">
<ivy:retrieve sync="true" type="jar" />
</target>
<!-- COMPILE PROJECT -->
<target name="compile" depends="clean, retrieve">
<!-- Create build directory -->
<mkdir dir="target/${ant.project.name}" />
<!-- Compile source code -->
<javac includeantruntime="false" srcdir="src" debug="true" destdir="target/${ant.project.name}" >
<classpath>
<path refid="ivy.path" />
</classpath>
</javac>
</target>
<!-- CLEAN TARGET DIRECTORY -->
<target name="clean">
<delete dir="target/orderlycalls" />
<delete dir="target/classes" />
<delete dir="target/ivy-reports" />
</target>
<!-- CLEAN TARGET AND IVY CATCHE -->
<target name="clean-all" depends="clean" description="Additionally purge ivy cache">
<ivy:cleancache/>
</target>
ivy.xml
<ivy-module version="2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
<configurations>
<conf name="default" visibility="public" description="The single built artifact. Nothing else"/>
<conf name="compile" visibility="public" description="The master module and transitive dependencies"/>
<conf name="provided" visibility="public" description="Needed for compile. Will be provided outside or war"/>
<conf name="runtime" visibility="public" description="Not required for compile, but for runtime" extends="compile"/>
<conf name="default" visibility="public" description="The default configuration" extends="runtime"/>
<conf name="test" visibility="private" description="Required for testing" extends="runtime"/>
</configurations>
<dependencies>
<dependency org="net.sf.trove4j" name="trove4j" rev="3.0.3" conf="provided"/>
<dependency org="org.apache.tomcat.embed" name="tomcat-embed-core" rev="7.0.53" conf="provided"/>
</dependencies>
</ivy-module>
when i run 'ant compile' the on compilation ant is complaining that it can not find 'Servlet Context' which is part of tomcat.jar or 'TObject', 'THashMap' which are part of trove.jar and many more despite the fact that i am retrieving/resolving the jars in build.xml.
Another thing i have noticed is that inside my .ivy2/cache// there are no actual jar files. Only xml files
Any idea what i am doing wrong or not doing at all ?
Thanks

You have used the cachepath task to create an Ant Path, using dependencies associated with the "compile" configuration.
<ivy:cachepath pathid="ivy.path" conf="compile" />
Your issue lies with your ivy file, where you haven't specified any dependencies with a mapping to the "compile" configuration. This would explain why your javac task cannot see any jars.
<configurations>
...
<conf name="compile" .../>
<conf name="provided" .../>
...
</configurations>
<dependencies>
<dependency ... conf="provided"/>
<dependency ... conf="provided"/>
</dependencies>
I suggest creating an explicit mapping for each of your configurations, for example:
<!-- Compile dependencies -->
<dependency org="net.sf.trove4j" name="trove4j" rev="3.0.3" conf="compile->default"/>
<!-- provided dependencies -->
<dependency org="org.apache.tomcat.embed" name="tomcat-embed-core" rev="7.0.53" conf="provided->master"/>
The only mappings you'll ever need are to the following remote configurations:
default Remote jar plus its transitive dependencies
master Remote jar only
For more information on how ivy interprets remote maven modules please read the following:
How are maven scopes mapped to ivy configurations by ivy

Ivy conf struggle

I am currently struggling with apache ivy's conf settings. Maybe i'll explain
what i want to achieve first:
I want two configurations: compile, runtime
I want for each configuration that ivy resolves, all jars including the transitive dependencies
Here's something i tried:
<configurations>
<conf name="compile" visibility="public"/>
<conf name="runtime" visibility="public" extends="compile"/>
</configurations>
<dependencies>
<dependency org="org.apache.ivy" name="ivy" rev="2.4.0" conf="runtime->runtime, compile->compile"/>
</dependencies>
But it cannot even be parsed:
The ivy file '...\ivy.xml' could not be parsed:
Could please someone tell me how to declare the conf settings properly, so that i can resolve using either compile or runtime and get all jars including the transitive dependencies(in the example case only the jars of ivy).
Update
Ok, thanks for the links. This makes it a bit clearer, but i'm still stuck.
I changed the dependency declaration so that it resolves the ivy jars in scope 'default':
(I left out the "compile->default" declaration, as it seems there's a bug in the
parsing of ivy eclipse plugin)
<dependency org="org.apache.ivy" name="ivy" rev="2.4.0" conf="runtime->default"/>
and retrieved the jars (using the ivy eclipse plugin) with the settings
confs='*' and types='*'.
All i got was a single ivy-2.4.0.jar. I expected all runtime dependencies. I can't exactly tell which they are, as i don't understand the declarations in ivy's xml file like(the whole xml file is appended later):
<dependency org="org.apache.ant" name="ant" rev="1.7.1" force="true" conf="optional->compile(*),master(*)"/>
Here's how i understand the conf="runtime->default" declaration:
The ivy dependency is assigned to (local) conf 'runtime'
The ivy dependency is assigned to remote conf 'default'
Whenever i resolve (or retrieve) the dependency for conf 'runtime' i
get all of the remote dependencies including the transitive dependencies
for conf 'default', as the 'default' conf is declared as
"The remote module's artifact and all it's runtime transitive dependencies" link
Here's my ivy.xml:
<?xml version="1.0" encoding="ISO-8859-1"?>
<ivy-module version="2.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
<info organisation="My Name" module="jaculon.ivy" status="integration">
</info>
<configurations>
<conf name="default" visibility="public" description="runtime dependencies and master artifact can be used with this conf" extends="runtime,master"/>
<conf name="master" visibility="public" description="contains only the artifact published by this module itself, with no transitive dependencies"/>
<conf name="compile" visibility="public" description="this is the default scope, used if none is specified. Compile dependencies are available in all classpaths."/>
<conf name="provided" visibility="public" description="this is much like compile, but indicates you expect the JDK or a container to provide it. It is only available on the compilation classpath, and is not transitive."/>
<conf name="runtime" visibility="public" description="this scope indicates that the dependency is not required for compilation, but is for execution. It is in the runtime and test classpaths, but not the compile classpath." extends="compile"/>
<conf name="test" visibility="public" description="this scope indicates that the dependency is not required for normal use of the application, and is only available for the test compilation and execution phases." extends="runtime"/>
<conf name="system" visibility="public" description="this scope is similar to provided except that you have to provide the JAR which contains it explicitly. The artifact is always available and is not looked up in a repository."/>
<conf name="sources" visibility="public" description="this configuration contains the source artifact of this module, if any."/>
<conf name="javadoc" visibility="public" description="this configuration contains the javadoc artifact of this module, if any."/>
<conf name="optional" visibility="public" description="contains all optional dependencies"/>
</configurations>
<dependencies>
<!-- Need ivy to resolve the jars. -->
<dependency org="org.apache.ivy" name="ivy" rev="2.4.0" conf="runtime->default"/>
</dependencies>
</ivy-module>
Here's the content the ivy-2.4.0.xml:
<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven">
<info organisation="org.apache.ivy"
module="ivy"
revision="2.4.0"
status="release"
publication="20141222174010"
>
<description homepage="http://ant.apache.org/ivy/" />
<m:properties__project.build.sourceEncoding>UTF-8</m:properties__project.build.sourceEncoding>
<m:properties__organization.logo>http://www.apache.org/images/asf_logo_wide.gif</m:properties__organization.logo>
<m:properties__distMgmtSnapshotsName>Apache Development Snapshot Repository</m:properties__distMgmtSnapshotsName>
<m:properties__distMgmtSnapshotsUrl>https://repository.apache.org/content/repositories/snapshots</m:properties__distMgmtSnapshotsUrl>
<m:maven.plugins>org.apache.maven.plugins__maven-remote-resources-plugin__null|org.apache.maven.plugins__maven-remote-resources-plugin__null</m:maven.plugins>
<m:properties__sourceReleaseAssemblyDescriptor>source-release</m:properties__sourceReleaseAssemblyDescriptor>
</info>
<configurations>
<conf name="default" visibility="public" description="runtime dependencies and master artifact can be used with this conf" extends="runtime,master"/>
<conf name="master" visibility="public" description="contains only the artifact published by this module itself, with no transitive dependencies"/>
<conf name="compile" visibility="public" description="this is the default scope, used if none is specified. Compile dependencies are available in all classpaths."/>
<conf name="provided" visibility="public" description="this is much like compile, but indicates you expect the JDK or a container to provide it. It is only available on the compilation classpath, and is not transitive."/>
<conf name="runtime" visibility="public" description="this scope indicates that the dependency is not required for compilation, but is for execution. It is in the runtime and test classpaths, but not the compile classpath." extends="compile"/>
<conf name="test" visibility="private" description="this scope indicates that the dependency is not required for normal use of the application, and is only available for the test compilation and execution phases." extends="runtime"/>
<conf name="system" visibility="public" description="this scope is similar to provided except that you have to provide the JAR which contains it explicitly. The artifact is always available and is not looked up in a repository."/>
<conf name="sources" visibility="public" description="this configuration contains the source artifact of this module, if any."/>
<conf name="javadoc" visibility="public" description="this configuration contains the javadoc artifact of this module, if any."/>
<conf name="optional" visibility="public" description="contains all optional dependencies"/>
</configurations>
<publications>
<artifact name="ivy" type="jar" ext="jar" conf="master"/>
<artifact name="ivy" type="source" ext="jar" conf="sources" m:classifier="sources"/>
<artifact name="ivy" type="javadoc" ext="jar" conf="javadoc" m:classifier="javadoc"/>
</publications>
<dependencies>
<dependency org="org.apache.ant" name="ant" rev="1.7.1" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="org.apache.ant" name="ant-nodeps" rev="1.7.1" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="org.apache.ant" name="ant-trax" rev="1.7.1" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="commons-httpclient" name="commons-httpclient" rev="3.0" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="oro" name="oro" rev="2.0.8" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="commons-vfs" name="commons-vfs" rev="1.0" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="com.jcraft" name="jsch" rev="0.1.50" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="com.jcraft" name="jsch.agentproxy" rev="0.0.6" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="com.jcraft" name="jsch.agentproxy.connector-factory" rev="0.0.6" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="com.jcraft" name="jsch.agentproxy.jsch" rev="0.0.6" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="org.bouncycastle" name="bcpg-jdk14" rev="1.45" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="org.bouncycastle" name="bcprov-jdk14" rev="1.45" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="junit" name="junit" rev="3.8.2" force="true" conf="test->runtime(*),master(*)"/>
<dependency org="commons-lang" name="commons-lang" rev="2.6" force="true" conf="test->runtime(*),master(*)"/>
<dependency org="org.apache.ant" name="ant-testutil" rev="1.7.0" force="true" conf="test->runtime(*),master(*)">
<exclude org="*" module="*" name="*" type="*" ext="*" conf="" matcher="exact"/>
</dependency>
<dependency org="ant" name="ant-launcher" rev="1.6.2" force="true" conf="test->runtime(*),master(*)">
<exclude org="*" module="*" name="*" type="*" ext="*" conf="" matcher="exact"/>
</dependency>
<dependency org="ant-contrib" name="ant-contrib" rev="1.0b3" force="true" conf="test->runtime(*),master(*)">
<exclude org="*" module="*" name="*" type="*" ext="*" conf="" matcher="exact"/>
</dependency>
<dependency org="xerces" name="xercesImpl" rev="2.6.2" force="true" conf="test->runtime(*),master(*)"/>
<dependency org="xerces" name="xmlParserAPIs" rev="2.6.2" force="true" conf="test->runtime(*),master(*)"/>
</dependencies>
</ivy-module>

Sounds like an XML formatting problem.
The following examples demonstrate how ivy configurations can be used to manage separate "compile", "runtime" and "test" classpaths:
Ivy, what is the master configuration and why is it not pulling jvyaml?
Class not found with Ant, Ivy and JUnit - error in build.xml?

system scope equivalent for ivy

I am looking for the equivalent of a system scope dependency in Maven for Ivy. In Maven, declaring a dependency with system scope means that Maven will not include the dependency in the output, which is what I want. How can I achieve the same thing with Ivy?

I suspect you're talking about the Maven provided scope, not system.
provided This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scope provided because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.
Ivy's configurations are very flexible which means that are several ways to do this.
ivy.xml
I would model my configuration on the different types of jar that my build will use:
<configurations>
<conf name="compile" description="Required to compile application"/>
<conf name="runtime" description="Additional run-time dependencies" extends="compile"/>
<conf name="test" description="Required for test only" extends="runtime"/>
<conf name="provided" description="Additional compile time dependencies, implementation provided by target platform"/>
</configurations>
The dependencies are then assigned to each logical grouping using a "conf" mapping:
<!-- compile dependencies -->
<dependency org="org.slf4j" name="slf4j-api" rev="1.7.5" conf="compile->default"/>
Note this is where the "extends" attribute it useful, without it the following mapping would be required for a logging dependency:
<!-- compile dependencies -->
<dependency org="org.slf4j" name="slf4j-api" rev="1.7.5" conf="compile,runtime,test->default"/>
Dependencies provided by the target platform are special. This is why I create a standalone configuration for them:
<!-- compile dependencies -->
<dependency org="my.target.platform" name="makeitgo-api" rev="1.0" conf="provided->default"/>
For even more details on ivy configuration mappings see:
Ivy, what is the master configuration and why is it not pulling jvyaml?
build.xml
It's here where the classpaths are actually managed. (We could try and model the set relationships in various ivy configurations, but I'd argue this approach is simpler and gets the job done)
<target name="resolve" description="Use ivy to resolve classpaths">
<ivy:resolve/>
<ivy:report todir='${ivy.reports.dir}' graph='false' xml='false'/>
<ivy:cachepath pathid="compile.path" conf="compile,provided"/>
<ivy:cachepath pathid="test.path" conf="test,provided"/>
</target>
So just as the Maven documentation describes, add the provided dependencies to the compile and test path.
This means the "runtime" configuration only contains the dependencies that should be bundled:
<ivy:retrieve pattern="${build.dir}/WEB-INF/lib/[artifact]-[revision].[ext]" conf="runtime"/>

I guess that the only problem to deal with is to have a dependency which should be used at compile time but not at runtime, right ?
Here is an exemple of ivy.xml on how to deal with it:
<ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven">
<info organisation="com.acme" module="mymodule" />
<configurations>
<conf name="default" description="runtime" />
<conf name="runtime" description="Runtime configuration"/>
<conf name="compile" description="Used only for compilation" />
<conf name="test" extends="compile,runtime" description="Unit testing configuration" visibility="private" />
</configurations>
<dependencies defaultconf="runtime,compile->default">
<!-- Compile and runtime -->
<dependency org="commons-lang" name="commons-lang" rev="2.6" />
<!-- Only compile -->
<dependency org="javax.servlet" name="servlet-api" rev="2.5" conf="compile->default" />
</dependencies>
</ivy-module>

how to define ivy config for jar to be included only for compile not runtime

i'm facing an issue while deploying my app on tomcat container which already contains servlet-api and my app also depends on servlet-api due to different they clash.
Is it possible define ivy for my app to include servlet-api only during compilation not for runtime.
How to do that.
dependency org="java" name="servlet-api" rev="default" conf="compile"
passing conf as compile fails ivy validation.

Sounds like a dependency functionally similar to the Maven "provided" scope.
My advice is to create an extra confguration and mapping for example:
<configurations>
<conf name="compile" description="Compile dependencies"/>
<conf name="runtime" description="Runtime dependencies" extends="compile"/>
<conf name="provided" description="Provided dependencies"/>
</configurations>
<dependencies>
<dependency .. conf="compile->default"/>
<dependency .. conf="runtime->default"/>
<dependency .. conf="provided->default"/>
</dependencies>
Within your build file the classpath used for compilation can include the extra depenencies as follows:
<ivy:cachepath pathid="compile.path" conf="compile,provided"/>

Does Ivy's url resolver support transitive retrieval?

For some reason I can't seem to resolve the dependencies of my dependencies when using a url resolver to specify a repository's location. However, when using the ibiblio resolver, I am able to retrieve them.
For example:
<!-- Ivy File -->
<ivy-module version="1.0">
<info organisation="org.apache" module="chained-resolvers"/>
<dependencies>
<dependency org="commons-lang" name="commons-lang" rev="2.0" conf="default"/>
<dependency org="checkstyle" name="checkstyle" rev="5.0"/>
</dependencies>
</ivy-module>
<!-- ivysettings file -->
<ivysettings>
<settings defaultResolver="chained"/>
<resolvers>
<chain name="chained">
<url name="custom-repo">
<ivy pattern="http://my.internal.domain.name/ivy/[organisation]/[module]/[revision]/ivy-[revision].xml"/>
<artifact pattern="http://my.internal.domain.name/ivy/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"/>
</url>
<url name="ibiblio-mirror" m2compatible="true">
<artifact pattern="http://mirrors.ibiblio.org/pub/mirrors/maven2/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
</url>
<ibiblio name="ibiblio" m2compatible="true"/>
</chain>
</resolvers>
</ivysettings>
<!-- checkstyle ivy.xml file generated from pom via ivy:install task -->
<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="1.0" xmlns:m="http://ant.apache.org/ivy/maven">
<info organisation="checkstyle"
module="checkstyle"
revision="5.0"
status="release"
publication="20090509202448"
namespace="maven2"
>
<license name="GNU Lesser General Public License" url="http://www.gnu.org/licenses/lgpl.txt" />
<description homepage="http://checkstyle.sourceforge.net/">
Checkstyle is a development tool to help programmers write Java code that adheres to a coding standard
</description>
</info>
<configurations>
<conf name="default" visibility="public" description="runtime dependencies and master artifact can be used with this conf" extends="runtime,master"/>
<conf name="master" visibility="public" description="contains only the artifact published by this module itself, with no transitive dependencies"/>
<conf name="compile" visibility="public" description="this is the default scope, used if none is specified. Compile dependencies are available in all classpaths."/>
<conf name="provided" visibility="public" description="this is much like compile, but indicates you expect the JDK or a container to provide it. It is only available on the compilation classpath, and is not transitive."/>
<conf name="runtime" visibility="public" description="this scope indicates that the dependency is not required for compilation, but is for execution. It is in the runtime and test classpaths, but not the compile classpath." extends="compile"/>
<conf name="test" visibility="private" description="this scope indicates that the dependency is not required for normal use of the application, and is only available for the test compilation and execution phases." extends="runtime"/>
<conf name="system" visibility="public" description="this scope is similar to provided except that you have to provide the JAR which contains it explicitly. The artifact is always available and is not looked up in a repository."/>
<conf name="sources" visibility="public" description="this configuration contains the source artifact of this module, if any."/>
<conf name="javadoc" visibility="public" description="this configuration contains the javadoc artifact of this module, if any."/>
<conf name="optional" visibility="public" description="contains all optional dependencies"/>
</configurations>
<publications>
<artifact name="checkstyle" type="jar" ext="jar" conf="master"/>
</publications>
<dependencies>
<dependency org="antlr" name="antlr" rev="2.7.6" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
<dependency org="apache" name="commons-beanutils-core" rev="1.7.0" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
<dependency org="apache" name="commons-cli" rev="1.0" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
<dependency org="apache" name="commons-logging" rev="1.0.3" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
<dependency org="com.google.collections" name="google-collections" rev="0.9" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
</dependencies>
</ivy-module>
Using the "ibiblio" resolver I have no problem resolving my project's two dependencies (commons-lang 2.0 and checkstyle 5.0) and checkstyle's dependencies. However, when attempting to exclusively use the "custom-repo" or "ibiblio-mirror" resolvers, I am able to resolve my project's two explicitly defined dependencies, but not checkstyle's dependencies.
Is this possible? Any help would be greatly appreciated.

The reason is you didn't specify an ivy pattern for your 'ibibio-mirror' resolver. Your mirror should look somthing like (don't forget the [classifier] token):
<url name="ibiblio-mirror" m2compatible="true">
<ivy pattern="http://mirrors.ibiblio.org/pub/mirrors/maven2/[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]" />
<artifact pattern="http://mirrors.ibiblio.org/pub/mirrors/maven2/[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]" />
</url>
But you could also use the ibiblio resolver for your mirror:
<ibiblio name="ibiblio-mirror" root="http://mirrors.ibiblio.org/pub/mirrors/maven2/" m2compatible="true"/>
Maarten