We Keep Coding

Ivy conf struggle

I am currently struggling with apache ivy's conf settings. Maybe i'll explain
what i want to achieve first:
I want two configurations: compile, runtime
I want for each configuration that ivy resolves, all jars including the transitive dependencies
Here's something i tried:
<configurations>
<conf name="compile" visibility="public"/>
<conf name="runtime" visibility="public" extends="compile"/>
</configurations>
<dependencies>
<dependency org="org.apache.ivy" name="ivy" rev="2.4.0" conf="runtime->runtime, compile->compile"/>
</dependencies>
But it cannot even be parsed:
The ivy file '...\ivy.xml' could not be parsed:
Could please someone tell me how to declare the conf settings properly, so that i can resolve using either compile or runtime and get all jars including the transitive dependencies(in the example case only the jars of ivy).
Update
Ok, thanks for the links. This makes it a bit clearer, but i'm still stuck.
I changed the dependency declaration so that it resolves the ivy jars in scope 'default':
(I left out the "compile->default" declaration, as it seems there's a bug in the
parsing of ivy eclipse plugin)
<dependency org="org.apache.ivy" name="ivy" rev="2.4.0" conf="runtime->default"/>
and retrieved the jars (using the ivy eclipse plugin) with the settings
confs='*' and types='*'.
All i got was a single ivy-2.4.0.jar. I expected all runtime dependencies. I can't exactly tell which they are, as i don't understand the declarations in ivy's xml file like(the whole xml file is appended later):
<dependency org="org.apache.ant" name="ant" rev="1.7.1" force="true" conf="optional->compile(*),master(*)"/>
Here's how i understand the conf="runtime->default" declaration:
The ivy dependency is assigned to (local) conf 'runtime'
The ivy dependency is assigned to remote conf 'default'
Whenever i resolve (or retrieve) the dependency for conf 'runtime' i
get all of the remote dependencies including the transitive dependencies
for conf 'default', as the 'default' conf is declared as
"The remote module's artifact and all it's runtime transitive dependencies" link
Here's my ivy.xml:
<?xml version="1.0" encoding="ISO-8859-1"?>
<ivy-module version="2.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
<info organisation="My Name" module="jaculon.ivy" status="integration">
</info>
<configurations>
<conf name="default" visibility="public" description="runtime dependencies and master artifact can be used with this conf" extends="runtime,master"/>
<conf name="master" visibility="public" description="contains only the artifact published by this module itself, with no transitive dependencies"/>
<conf name="compile" visibility="public" description="this is the default scope, used if none is specified. Compile dependencies are available in all classpaths."/>
<conf name="provided" visibility="public" description="this is much like compile, but indicates you expect the JDK or a container to provide it. It is only available on the compilation classpath, and is not transitive."/>
<conf name="runtime" visibility="public" description="this scope indicates that the dependency is not required for compilation, but is for execution. It is in the runtime and test classpaths, but not the compile classpath." extends="compile"/>
<conf name="test" visibility="public" description="this scope indicates that the dependency is not required for normal use of the application, and is only available for the test compilation and execution phases." extends="runtime"/>
<conf name="system" visibility="public" description="this scope is similar to provided except that you have to provide the JAR which contains it explicitly. The artifact is always available and is not looked up in a repository."/>
<conf name="sources" visibility="public" description="this configuration contains the source artifact of this module, if any."/>
<conf name="javadoc" visibility="public" description="this configuration contains the javadoc artifact of this module, if any."/>
<conf name="optional" visibility="public" description="contains all optional dependencies"/>
</configurations>
<dependencies>
<!-- Need ivy to resolve the jars. -->
<dependency org="org.apache.ivy" name="ivy" rev="2.4.0" conf="runtime->default"/>
</dependencies>
</ivy-module>
Here's the content the ivy-2.4.0.xml:
<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven">
<info organisation="org.apache.ivy"
module="ivy"
revision="2.4.0"
status="release"
publication="20141222174010"
>
<description homepage="http://ant.apache.org/ivy/" />
<m:properties__project.build.sourceEncoding>UTF-8</m:properties__project.build.sourceEncoding>
<m:properties__organization.logo>http://www.apache.org/images/asf_logo_wide.gif</m:properties__organization.logo>
<m:properties__distMgmtSnapshotsName>Apache Development Snapshot Repository</m:properties__distMgmtSnapshotsName>
<m:properties__distMgmtSnapshotsUrl>https://repository.apache.org/content/repositories/snapshots</m:properties__distMgmtSnapshotsUrl>
<m:maven.plugins>org.apache.maven.plugins__maven-remote-resources-plugin__null|org.apache.maven.plugins__maven-remote-resources-plugin__null</m:maven.plugins>
<m:properties__sourceReleaseAssemblyDescriptor>source-release</m:properties__sourceReleaseAssemblyDescriptor>
</info>
<configurations>
<conf name="default" visibility="public" description="runtime dependencies and master artifact can be used with this conf" extends="runtime,master"/>
<conf name="master" visibility="public" description="contains only the artifact published by this module itself, with no transitive dependencies"/>
<conf name="compile" visibility="public" description="this is the default scope, used if none is specified. Compile dependencies are available in all classpaths."/>
<conf name="provided" visibility="public" description="this is much like compile, but indicates you expect the JDK or a container to provide it. It is only available on the compilation classpath, and is not transitive."/>
<conf name="runtime" visibility="public" description="this scope indicates that the dependency is not required for compilation, but is for execution. It is in the runtime and test classpaths, but not the compile classpath." extends="compile"/>
<conf name="test" visibility="private" description="this scope indicates that the dependency is not required for normal use of the application, and is only available for the test compilation and execution phases." extends="runtime"/>
<conf name="system" visibility="public" description="this scope is similar to provided except that you have to provide the JAR which contains it explicitly. The artifact is always available and is not looked up in a repository."/>
<conf name="sources" visibility="public" description="this configuration contains the source artifact of this module, if any."/>
<conf name="javadoc" visibility="public" description="this configuration contains the javadoc artifact of this module, if any."/>
<conf name="optional" visibility="public" description="contains all optional dependencies"/>
</configurations>
<publications>
<artifact name="ivy" type="jar" ext="jar" conf="master"/>
<artifact name="ivy" type="source" ext="jar" conf="sources" m:classifier="sources"/>
<artifact name="ivy" type="javadoc" ext="jar" conf="javadoc" m:classifier="javadoc"/>
</publications>
<dependencies>
<dependency org="org.apache.ant" name="ant" rev="1.7.1" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="org.apache.ant" name="ant-nodeps" rev="1.7.1" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="org.apache.ant" name="ant-trax" rev="1.7.1" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="commons-httpclient" name="commons-httpclient" rev="3.0" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="oro" name="oro" rev="2.0.8" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="commons-vfs" name="commons-vfs" rev="1.0" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="com.jcraft" name="jsch" rev="0.1.50" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="com.jcraft" name="jsch.agentproxy" rev="0.0.6" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="com.jcraft" name="jsch.agentproxy.connector-factory" rev="0.0.6" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="com.jcraft" name="jsch.agentproxy.jsch" rev="0.0.6" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="org.bouncycastle" name="bcpg-jdk14" rev="1.45" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="org.bouncycastle" name="bcprov-jdk14" rev="1.45" force="true" conf="optional->compile(*),master(*)"/>
<dependency org="junit" name="junit" rev="3.8.2" force="true" conf="test->runtime(*),master(*)"/>
<dependency org="commons-lang" name="commons-lang" rev="2.6" force="true" conf="test->runtime(*),master(*)"/>
<dependency org="org.apache.ant" name="ant-testutil" rev="1.7.0" force="true" conf="test->runtime(*),master(*)">
<exclude org="*" module="*" name="*" type="*" ext="*" conf="" matcher="exact"/>
</dependency>
<dependency org="ant" name="ant-launcher" rev="1.6.2" force="true" conf="test->runtime(*),master(*)">
<exclude org="*" module="*" name="*" type="*" ext="*" conf="" matcher="exact"/>
</dependency>
<dependency org="ant-contrib" name="ant-contrib" rev="1.0b3" force="true" conf="test->runtime(*),master(*)">
<exclude org="*" module="*" name="*" type="*" ext="*" conf="" matcher="exact"/>
</dependency>
<dependency org="xerces" name="xercesImpl" rev="2.6.2" force="true" conf="test->runtime(*),master(*)"/>
<dependency org="xerces" name="xmlParserAPIs" rev="2.6.2" force="true" conf="test->runtime(*),master(*)"/>
</dependencies>
</ivy-module>

Sounds like an XML formatting problem.
The following examples demonstrate how ivy configurations can be used to manage separate "compile", "runtime" and "test" classpaths:
Ivy, what is the master configuration and why is it not pulling jvyaml?
Class not found with Ant, Ivy and JUnit - error in build.xml?

system scope equivalent for ivy

I am looking for the equivalent of a system scope dependency in Maven for Ivy. In Maven, declaring a dependency with system scope means that Maven will not include the dependency in the output, which is what I want. How can I achieve the same thing with Ivy?

I suspect you're talking about the Maven provided scope, not system.
provided This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scope provided because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.
Ivy's configurations are very flexible which means that are several ways to do this.
ivy.xml
I would model my configuration on the different types of jar that my build will use:
<configurations>
<conf name="compile" description="Required to compile application"/>
<conf name="runtime" description="Additional run-time dependencies" extends="compile"/>
<conf name="test" description="Required for test only" extends="runtime"/>
<conf name="provided" description="Additional compile time dependencies, implementation provided by target platform"/>
</configurations>
The dependencies are then assigned to each logical grouping using a "conf" mapping:
<!-- compile dependencies -->
<dependency org="org.slf4j" name="slf4j-api" rev="1.7.5" conf="compile->default"/>
Note this is where the "extends" attribute it useful, without it the following mapping would be required for a logging dependency:
<!-- compile dependencies -->
<dependency org="org.slf4j" name="slf4j-api" rev="1.7.5" conf="compile,runtime,test->default"/>
Dependencies provided by the target platform are special. This is why I create a standalone configuration for them:
<!-- compile dependencies -->
<dependency org="my.target.platform" name="makeitgo-api" rev="1.0" conf="provided->default"/>
For even more details on ivy configuration mappings see:
Ivy, what is the master configuration and why is it not pulling jvyaml?
build.xml
It's here where the classpaths are actually managed. (We could try and model the set relationships in various ivy configurations, but I'd argue this approach is simpler and gets the job done)
<target name="resolve" description="Use ivy to resolve classpaths">
<ivy:resolve/>
<ivy:report todir='${ivy.reports.dir}' graph='false' xml='false'/>
<ivy:cachepath pathid="compile.path" conf="compile,provided"/>
<ivy:cachepath pathid="test.path" conf="test,provided"/>
</target>
So just as the Maven documentation describes, add the provided dependencies to the compile and test path.
This means the "runtime" configuration only contains the dependencies that should be bundled:
<ivy:retrieve pattern="${build.dir}/WEB-INF/lib/[artifact]-[revision].[ext]" conf="runtime"/>

I guess that the only problem to deal with is to have a dependency which should be used at compile time but not at runtime, right ?
Here is an exemple of ivy.xml on how to deal with it:
<ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven">
<info organisation="com.acme" module="mymodule" />
<configurations>
<conf name="default" description="runtime" />
<conf name="runtime" description="Runtime configuration"/>
<conf name="compile" description="Used only for compilation" />
<conf name="test" extends="compile,runtime" description="Unit testing configuration" visibility="private" />
</configurations>
<dependencies defaultconf="runtime,compile->default">
<!-- Compile and runtime -->
<dependency org="commons-lang" name="commons-lang" rev="2.6" />
<!-- Only compile -->
<dependency org="javax.servlet" name="servlet-api" rev="2.5" conf="compile->default" />
</dependencies>
</ivy-module>

how to define ivy config for jar to be included only for compile not runtime

i'm facing an issue while deploying my app on tomcat container which already contains servlet-api and my app also depends on servlet-api due to different they clash.
Is it possible define ivy for my app to include servlet-api only during compilation not for runtime.
How to do that.
dependency org="java" name="servlet-api" rev="default" conf="compile"
passing conf as compile fails ivy validation.

Sounds like a dependency functionally similar to the Maven "provided" scope.
My advice is to create an extra confguration and mapping for example:
<configurations>
<conf name="compile" description="Compile dependencies"/>
<conf name="runtime" description="Runtime dependencies" extends="compile"/>
<conf name="provided" description="Provided dependencies"/>
</configurations>
<dependencies>
<dependency .. conf="compile->default"/>
<dependency .. conf="runtime->default"/>
<dependency .. conf="provided->default"/>
</dependencies>
Within your build file the classpath used for compilation can include the extra depenencies as follows:
<ivy:cachepath pathid="compile.path" conf="compile,provided"/>

Find hidden dependencies in Ivy

I'm using Apache Ivy + IvyDE for getting my project's dependencies, which are:
<dependency org="com.google.guava" name="guava" rev="r08" />
<!-- logging -->
<dependency org="org.slf4j" name="jcl-over-slf4j" rev="1.6.1" />
<dependency org="ch.qos.logback" name="logback-classic" rev="0.9.27" />
<!-- database -->
<dependency org="org.hibernate" name="hibernate-entitymanager" rev="3.6.2.Final" />
<dependency org="org.hibernate" name="hibernate-validator" rev="4.1.0.Final" />
<dependency org="org.hibernate" name="hibernate-c3p0" rev="3.6.2.Final" />
<dependency org="mysql" name="mysql-connector-java" rev="5.1.14" />
Sources are the Maven and JBoss (Hibernate) repositories.
As you can see I'm using logback+SLF4J for logging, but for some reason Ivy will download log4j and slf4j-log4j as well, which causes a few small problem in my application.
Is there a way to see why this happens, to see which of the dependencies above depend on log4j? Can I get a dependency graph/tree generated from Ivy/IvyDE?
And is there then a way to prevent this from happening?

We have an ant target that looks like this:
<target name="report" depends="init">
<mkdir dir="report" />
<!--
The type attribute is optional, we're using it to exlude other dependcy types we're not interested in.
Note that each resolve uses that list (via a property) in our build.
-->
<ivy:resolve type="jar,ejb,tld,bundle"/>
<ivy:report todir="report" />
</target>
Then it's just a call ant report and Ivy will generate the report as HTML in the given directory.
Take a look at the Ivy documentation for ivy:report.
Edit:
To prevent the inclusion of those artifacts/dependencies, you could try transitive="false" on the <dependency ..> element, or use <exclude>. For example, we use Hibernate 3 but don't want to have JTA 1.1, so our ivy.xml containts this: <exclude module="jta"/> to exclude all transitive JTA dependencies.

I'd like to build on Thomas' answer and recommend adding a "conf" declaration to the dependencies:
<dependency org="org.slf4j" name="jcl-over-slf4j" rev="1.6.1" conf="default"/>
<dependency org="ch.qos.logback" name="logback-classic" rev="0.9.27" conf="default"/>
This will reduce the transitive dependencies to the default sub-set, which in Maven terminology is the jars on the "compile" scope.
Without this setting you get all the dependencies which includes lot of optional stuff you won't need.

Does Ivy's url resolver support transitive retrieval?

For some reason I can't seem to resolve the dependencies of my dependencies when using a url resolver to specify a repository's location. However, when using the ibiblio resolver, I am able to retrieve them.
For example:
<!-- Ivy File -->
<ivy-module version="1.0">
<info organisation="org.apache" module="chained-resolvers"/>
<dependencies>
<dependency org="commons-lang" name="commons-lang" rev="2.0" conf="default"/>
<dependency org="checkstyle" name="checkstyle" rev="5.0"/>
</dependencies>
</ivy-module>
<!-- ivysettings file -->
<ivysettings>
<settings defaultResolver="chained"/>
<resolvers>
<chain name="chained">
<url name="custom-repo">
<ivy pattern="http://my.internal.domain.name/ivy/[organisation]/[module]/[revision]/ivy-[revision].xml"/>
<artifact pattern="http://my.internal.domain.name/ivy/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"/>
</url>
<url name="ibiblio-mirror" m2compatible="true">
<artifact pattern="http://mirrors.ibiblio.org/pub/mirrors/maven2/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
</url>
<ibiblio name="ibiblio" m2compatible="true"/>
</chain>
</resolvers>
</ivysettings>
<!-- checkstyle ivy.xml file generated from pom via ivy:install task -->
<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="1.0" xmlns:m="http://ant.apache.org/ivy/maven">
<info organisation="checkstyle"
module="checkstyle"
revision="5.0"
status="release"
publication="20090509202448"
namespace="maven2"
>
<license name="GNU Lesser General Public License" url="http://www.gnu.org/licenses/lgpl.txt" />
<description homepage="http://checkstyle.sourceforge.net/">
Checkstyle is a development tool to help programmers write Java code that adheres to a coding standard
</description>
</info>
<configurations>
<conf name="default" visibility="public" description="runtime dependencies and master artifact can be used with this conf" extends="runtime,master"/>
<conf name="master" visibility="public" description="contains only the artifact published by this module itself, with no transitive dependencies"/>
<conf name="compile" visibility="public" description="this is the default scope, used if none is specified. Compile dependencies are available in all classpaths."/>
<conf name="provided" visibility="public" description="this is much like compile, but indicates you expect the JDK or a container to provide it. It is only available on the compilation classpath, and is not transitive."/>
<conf name="runtime" visibility="public" description="this scope indicates that the dependency is not required for compilation, but is for execution. It is in the runtime and test classpaths, but not the compile classpath." extends="compile"/>
<conf name="test" visibility="private" description="this scope indicates that the dependency is not required for normal use of the application, and is only available for the test compilation and execution phases." extends="runtime"/>
<conf name="system" visibility="public" description="this scope is similar to provided except that you have to provide the JAR which contains it explicitly. The artifact is always available and is not looked up in a repository."/>
<conf name="sources" visibility="public" description="this configuration contains the source artifact of this module, if any."/>
<conf name="javadoc" visibility="public" description="this configuration contains the javadoc artifact of this module, if any."/>
<conf name="optional" visibility="public" description="contains all optional dependencies"/>
</configurations>
<publications>
<artifact name="checkstyle" type="jar" ext="jar" conf="master"/>
</publications>
<dependencies>
<dependency org="antlr" name="antlr" rev="2.7.6" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
<dependency org="apache" name="commons-beanutils-core" rev="1.7.0" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
<dependency org="apache" name="commons-cli" rev="1.0" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
<dependency org="apache" name="commons-logging" rev="1.0.3" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
<dependency org="com.google.collections" name="google-collections" rev="0.9" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
</dependencies>
</ivy-module>
Using the "ibiblio" resolver I have no problem resolving my project's two dependencies (commons-lang 2.0 and checkstyle 5.0) and checkstyle's dependencies. However, when attempting to exclusively use the "custom-repo" or "ibiblio-mirror" resolvers, I am able to resolve my project's two explicitly defined dependencies, but not checkstyle's dependencies.
Is this possible? Any help would be greatly appreciated.

The reason is you didn't specify an ivy pattern for your 'ibibio-mirror' resolver. Your mirror should look somthing like (don't forget the [classifier] token):
<url name="ibiblio-mirror" m2compatible="true">
<ivy pattern="http://mirrors.ibiblio.org/pub/mirrors/maven2/[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]" />
<artifact pattern="http://mirrors.ibiblio.org/pub/mirrors/maven2/[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]" />
</url>
But you could also use the ibiblio resolver for your mirror:
<ibiblio name="ibiblio-mirror" root="http://mirrors.ibiblio.org/pub/mirrors/maven2/" m2compatible="true"/>
Maarten