We Keep Coding

Include a project JAR from filesystem in project using IVY fails to pull in sources and javadoc

So I've been trying this for the better part of a day, and I'm so close. I have two projects, and I want to publish one as a JAR to a shared file location and then pull it in to the other project including the source and javadoc .jar files that I am generating out. The actual ivy:publish is successful and the files get correctly created in the remote file system.
However when trying to include the JAR's in my 2nd project, I can only pull in the base JAR, not the JAR's that contain the sources and javadoc.
My files are named as follows: [projectname]-[version](-[classifier]) so at the remote location I get foo-1.0.0.jar, foo-1.0.0-sources.jar, foo-1.0.0-javadoc.jar & ivy-1.0.0.xml as well as .sha and .md5 files for each JAR, but my IVY is only pulling in the foo-1.0.0.jar.
I'd like to stress that I am successfully pulling in sources and javadocs off the mavenrepo remote repository automatically and that is works really well. I just don't get why it is refusing to pull down the source & javadoc JAR's into the IVY cache when it finds and pulls the class JAR down fine from the file system.
This is the resolver pattern that I am using to pull down the files off the remote file system is:
/path/to/remote/location/[organisation]/[module]/[revision]/[module]-[revision](-[classifier]).[ext]
and the pattern that I'm using to push the files to the shared location is:
/path/to/remote/location/[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]
I've been trawling the documentation and examples that people have, but the documentation may as well be written in Klingon for all the sense it makes, and the examples are either from 2010 or don't fetch sources from the file system.
Thoughts:
Its not a permissions issue (if it was, IVY wouldn't be able to pull anything down)
Its not a connection issue (see above)
It might be a problem with how I am publishing my project (If you think this is the case, I can provide much more detail in what I am doing with that, upon request)
I'm suspicious that IVY isn't pulling down (or appearing to read) the ivy-1.0.0.xml file on the remote filesystem at all. After all, that is where I'm declaring that the sources and javadoc JAR's exist.
Its not just this project, I've tried to follow the same process for a third project, only to have IVY only grab the class .jar file from the remote file system.
I could be not creating/naming files in a way that IVY is expecting to read. I had a look at the file structure that is used in mavencentral and attempted to copy how they named and laid out their files, but it doesn't work.
What I've tried:
Just about every combination of IVY patterns that I can think of.
Repeatedly deleting the cached files to see if IVY correctly resolves all files from the remote filesystem (it doesn't, it just regrabs the class JAR and generates an ivy-1.0.0.xml file in the cache directory)
Happy to post config and clarify exactly what I'm doing if you want me to etc, just drop a comment and ask for what you want.
IVY Config in second project:
<ivy-conf>
<property name="ivy.shared.dir"
value="/path/to/remote/repo/maven-repo/shared" />
<property name="ibiblio-maven2-root" value="http://repo1.maven.org/maven2/"
override="false" />
<property name="ibiblio-spring-core-root" value="http://maven.springframework.org/"
override="false" />
<property name="local.pattern"
value="[organisation]/[module]/[revision]/[module]-[revision](-[classifier]).[ext]"/>
<property name="maven2.pattern"
value="[organisation]/[module]/[revision]/[module]-[revision]" />
<property name="spring.pattern" value="org/[organisation]/[module]/[revision]/" />
<property name="maven2.pattern.ext" value="${maven2.pattern}(-[classifier]).[ext]" />
<property name="spring.pattern.ext" value="${spring.pattern}(-[classifier]).[ext]" />
<settings defaultResolver="default" />
<resolvers>
<filesystem name="shared" m2compatible="true">
<artifact pattern="${ivy.shared.dir}/${local.pattern}" />
</filesystem>
<ibiblio name="maven2" root="${ibiblio-maven2-root}" pattern="${maven2.pattern.ext}"
m2compatible="true" />
<ibiblio name="spring" m2compatible="true" pattern="${spring.pattern.ext}"
root="${ibiblio-spring-core-root}" />
<chain name="internal">
<resolver ref="shared" />
</chain>
<chain name="external">
<resolver ref="maven2" />
<resolver ref="spring" />
</chain>
<chain name="default" returnFirst="true">
<chain ref="external" />
<chain ref="internal" />
</chain>
</resolvers>
<modules>
<module organisation="company.com.au" name="name"
resolver="default" />
</modules>
</ivy-conf>
The only difference between the two IVY files (between both projects) is this line which is the pattern to push to the remote server:
<property name="shared-repo"
value="[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]" />
as well as changing the tag in the resolver to shared-repo.
IVY file in the remote location:
<ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
<info organisation="company" module="simple-email" revision="1.0.0" status="integration" publication="20140924154556">
</info>
<configurations defaultconfmapping="default">
<conf name="compile" visibility="private"/>
<conf name="test" extends="compile" visibility="private"/>
<conf name="master"/>
<conf name="runtime"/>
<conf name="default" extends="master,runtime"/>
<conf name="sources" visibility="public" description="this configuration contains the source artifact of this module, if any."/>
<conf name="javadoc" visibility="public" description="this configuration contains the javadoc artifact of this module, if any."/>
</configurations>
<publications>
<artifact name="simple-email" type="jar" ext="jar" conf="master"/>
<artifact name="simple-email" type="source" ext="jar" conf="sources" m:classifier="sources"/>
<artifact name="simple-email" type="javadoc" ext="jar" conf="javadoc" m:classifier="javadoc"/>
</publications>
<dependencies>
<!-- Unit Testing -->
<dependency org="junit" name="junit" rev="4.8.2" conf="test->default"/>
<!-- Log4j Logging -->
<dependency org="log4j" name="log4j" rev="1.2.13" conf="runtime->default;test->default;compile->default"/>
<!-- SLF Logging -->
<dependency org="org.slf4j" name="slf4j-simple" rev="1.6.1" conf="runtime->default;compile->default;test->default"/>
<dependency org="org.slf4j" name="slf4j-api" rev="1.6.1" conf="runtime->default;compile->default;test->default"/>
<dependency org="javamail" name="javamail" rev="1.4" conf="runtime->default;compile->default;test->default"/>
</dependencies>
</ivy-module>
IVY file that gets generated in the cache:
<ivy-module version="2.0">
<info organisation="company"
module="simple-email"
revision="1.0.0"
status="release"
publication="20140924162544"
default="true"
/>
<configurations>
<conf name="default" visibility="public"/>
</configurations>
<publications>
<artifact name="simple-email" type="jar" ext="jar" conf="default"/>
</publications>
</ivy-module>
Directory Structure on remote server:
Org
|--Module
|--Version
|--ivy-1.0.0.xml
|--ivy-1.0.0.md5
|--ivy-1.0.0.sha
|--simple-email-1.0.0-javadoc.jar
|--simple-email-1.0.0-javadoc.jar.md5
|--simple-email-1.0.0-javadoc.jar.sha
|--simple-email-1.0.0-sources.jar
|--simple-email-1.0.0-sources.jar.md5
|--simple-email-1.0.0-sources.jar.sha
|--simple-email-1.0.0.jar
|--simple-email-1.0.0.jar.md5
|--simple-email-1.0.0.jar.sha
Ivy Cache Directory structure:
Org
|--Module
|--ivy-1.0.0.xml
|--ivydata-1.0.0.properties
|--jars
|--simple-email-1.0.0.jar
Publish Task on the ANT script:
<target name="ivy-publish" depends="archive">
<ivy:publish resolver="shared" pubrevision="${project.jar.version}" overwrite="true">
<artifacts pattern="${dist.dir}/[artifact].[ext]"/>
</ivy:publish>
</target>

Explanation
Theory
First of all this setup is a bit complicated, but also familiar because it's how ivy translates Maven modules into ivy ones. The problem is understanding how Maven "scopes" are translated into ivy "configurations".
How are maven scopes mapped to ivy configurations by ivy
Your remote module
Addressing your specific question I think the issue is how you are downloading artefacts. Your remote module declares the following files:
<publications>
<artifact name="simple-email" type="jar" ext="jar" conf="master"/>
<artifact name="simple-email" type="source" ext="jar" conf="sources" m:classifier="sources"/>
<artifact name="simple-email" type="javadoc" ext="jar" conf="javadoc" m:classifier="javadoc"/>
</publications>
The magic part is the configurations. In this case you have a file associated with the following configurations:
master
sources
javadoc
Secondly the remote module has a rather complex set of configurations declared. Here are ones relevent to the "default" setup:
<configurations defaultconfmapping="default">
..
..
<conf name="master"/>
<conf name="runtime"/>
<conf name="default" extends="master,runtime"/>
..
..
</configurations>
So... Only the "master" artefact is included. This would explain why source code is excluded by default, which makes sense because normally users would want the compiled binaries.
How ivy downloads artefacts
Simple (I don't want to use configurations)
This is where we dig into the magic of dependency mappings. Most of the time users don't care about time. So to ignore them I normally recommend adding a conf="default" at the end of each dependency:
<dependency org="company" name="simple-email" rev="1.0.0" conf="default"/>
The creates the following relationship between me and the remote module:
<local "default" configuration> -> <remote "default" configuration>
In others words only give me defaults which are the compiled binaries excluding other more optional stuff like source and javadoc.
Using configurations
Once you understand configurations you're going to want to declare them locally. For example:
<configurations>
<conf name="compile" description="Required to compile application"/>
<conf name="sources" description="Source code"/>
</configurations>
We're stating that we have two buckets or logical groupings of dependencies.
Now we make our dependency declares more aware of our local configurations:
<dependency org="company" name="simple-email" rev="1.0.0" conf="compile->default;sources"/>
We now have 2 mappings:
<local "compile" configuration> -> <remote "default" configuration>
<local "sources" configuration> -> <remote "sources" configuration>
We can now reference or use these configurations separately in our ANT build file. For example create a classpath:
<ivy:cachepath pathid="compile.path" conf="compile"/>
Our put the sources jar inside the build directory using the retrieve task:
<ivy:retrieve pattern="build/src/[artifact]-[revision](-[classifier]).[ext]" conf="sources"/>
Example
In this contrived example I want to download the source jars into a build/src directory and the compile dependencies into a lib dir:
├── build
│   └── src
│   └── log4j-1.2.17-sources.jar
├── build.xml
├── ivy.xml
└── lib
└── log4j-1.2.17.jar
build.xml
<project name="demo" default="retrieve" xmlns:ivy="antlib:org.apache.ivy.ant">
<target name="retrieve">
<ivy:retrieve pattern="lib/[artifact]-[revision](-[classifier]).[ext]" conf="compile"/>
<ivy:retrieve pattern="build/src/[artifact]-[revision](-[classifier]).[ext]" conf="sources"/>
</target>
</project>
Note:
The files downloaded by ivy are grouped into "configurations". Each retrieve task has a different "conf" attribute.
Configurations are declared in the ivy file and mappings are defined on each dependency.
ivy.xml
<ivy-module version="2.0">
<info organisation="com.myspotontheweb" module="demo"/>
<configurations>
<conf name="compile" description="Required to compile application"/>
<conf name="sources" description="Source code"/>
</configurations>
<dependencies>
<dependency org="log4j" name="log4j" rev="1.2.17" conf="compile->default;sources" />
</dependencies>
</ivy-module>
Notes:
Ivy file creates 2 configurations
The magic is the "conf" attribute of the dependency. This creates the following mappings "compile->default" and "sources->sources". This means compile dependencies come from the remote default (usual setting) and the local sources come from the remote sources.

IVY Build, how do I use an extlib directory in the project?

There are several jar files, which are from a COTS product that we keep in our project's "extlib" folder at the root of the project. I want to have these included in ivy, but not put in the repository, just read from the project/extlib folder. Is this possible or do I need to add them to the artifactory?

Create an ivy settings file with the following content:
<ivysettings>
<settings defaultResolver='artifactory' />
<resolvers>
<ibiblio name='artifactory' m2compatible='true' root="http://my.artifactory.server"/>
<filesystem name='local'>
<artifact pattern='${ivy.settings.dir}/extlib/[artifact]' />
</filesystem>
</resolvers>
<modules>
<module organisation='NA' resolver='local' />
</modules>
</ivysettings>
And declare your dependencies as follows:
<dependencies>
<dependency org='org.apache.tomcat' name='tomcat-api' rev='7.0.21' />
..
..
<dependency org='NA' name='myspecial.jar' rev='NA' />
</dependencies>
The special "NA" organisation is configured to pull files from the "extlib" directory. The revision attribute is manadatory but generally meaningless when referring to files stored within the project directory.

How do I include an Ivy Dependency in subversion repo

So, since I've been unable to find a way to resolve our dependency issues by including everything from external sources I've turned to Eclipse / IvyDE for ant / Ivy integration.
With that said, I normally include a lib like this:
<dependency org="org.jsoup" name="jsoup" rev="1.6.3"/>
However what if I want to look at something in our own intranet?
Example, if the folder holding the jar is somewhere like this:
https://prdsvn01.company.intra.net/repo/libName/
and I want to include lib.jar into my folder.
I've been relatively unable to find ivysettings.xml in this implementation of eclipse, nor am I confident that I'd be able to get it right if I could.
Could someone help me with this?

The following ivy settings file:
<ivysettings>
<settings defaultResolver="central"/>
<resolvers>
<ibiblio name="central" m2compatible="true"/>
<url name="my-repo">
<artifact pattern="http://myserver/myrepo/[organisation]/[artifact]/[revision]/[artifact].[ext]"/>
</url>
</resolvers>
<modules>
<module organisation="org.mycompany" resolver="my-repo"/>
</modules>
</ivysettings>
Is configured to retrieve artifacts from Maven Central by default, and local artifacts from a HTTP server.
Update
ivy.xml
Nothing special in the ivy file. Just declare the dependencies and which configuration to associate them with:
<configurations>
<conf name="compile" description="Required to compile application"/>
</configurations>
<dependencies>
<!-- compile dependencies -->
<dependency org="org.slf4j" name="slf4j-api" rev="1.6.4" conf="compile->default"/>
<dependency org="org.mycompany" name="my-module" rev="1.0" conf="compile->default"/>
</dependencies>
Note:
It's ivy best practice to use configurations.
build.xml
<target name="resolve" dependencies="Resolve build dependencies">
<ivy:resolve/>
<ivy:report todir='build/reports' graph='false' xml='false'/>
<ivy:cachepath pathid="compile.path" conf="compile"/>
..
</target>
Notes:
The ivycachepath task transforms an ivy configuration into a populated ANT classpath. Very useful.
The ivy report task tells you the jars on the classpath(s)
Ivy resolve build output
All the magic is in the settings file. Running the build produces the following:
[ivy:resolve] found org.slf4j#slf4j-api;1.6.4 in central
[ivy:resolve] found org.mycompany#my-module;1.0 in my-repo
..
[ivy:resolve] downloading http://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.6.4/slf4j-api-1.6.4.jar ...
..
[ivy:resolve] downloading http://myserver/myrepo/org.mycompany/my-module/1.0/my-module.jar ...
Notes:
The my-repo resolver is used for modules with a "org.mycompany" groupId.
Everything else comes from the default resolver, Maven Central.

Where is Ivy repository for xuggle-utils.jar?

I am trying to compile Xuggler under windows. It's ivy.xml file contains the following dependency:
<dependency org="xuggle" name="xuggle-utils" rev="latest.integration" conf="test" changing="true"/>
Unfortunately, this dependency cannot be found int any repository, configured in this project.
Where is up to date version of repo for xuggler?

This will be fixed in Xuggler 5.3 (launches next week). It is already in the github project as [cross_compile 1489166].
https://github.com/xuggle/xuggle-xuggler
Enjoy.

There seems to be a file http://xuggle.googlecode.com/svn/trunk/build/java/ivysettings.xmlwhich suggests http://xuggle.googlecode.com/svn/trunk/repo/share/java/ as the repo, so you might try that one.
A problem might be that the repository might not be queryable and thus rev="latest.integration" might cause problems, since Ivy can't query the repo for the available versions.

The settings file in the project's source does not appear to be working.
I think discovered the locations of both the xuggle jars and the secondary repository used to host the module's transitive dependencies (contained in the xuggle module ivy.xml).
Example ivy.xml and ivysettings.xml files included below.
ivy.xml
<ivy-module version="2.0">
<info organisation="org.demo" module="demo"/>
<configurations>
<conf name="compile"/>
<conf name="runtime" extends="compile"/>
<conf name="test" extends="runtime"/>
</configurations>
<dependencies>
<dependency org="xuggle" name="xuggle-utils" rev="latest.integration" conf="test->default"/>
</dependencies>
</ivy-module>
ivysettings.xml
<ivysettings>
<settings defaultResolver="central"/>
<resolvers>
<ibiblio name="central" m2compatible="true"/>
<url name="xuggle">
<ivy pattern="http://xuggle.googlecode.com/svn/trunk/repo/share/java/[organisation]/[module]/[revision]/ivy-[revision].xml" />
<artifact pattern="http://xuggle.googlecode.com/svn/trunk/repo/share/java/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
</url>
<url name="red5">
<artifact pattern="http://red5.googlecode.com/svn/repository/[artifact]-[revision].[ext]" />
</url>
</resolvers>
<modules>
<module organisation="xuggle" resolver="xuggle" />
<module organisation="red5" resolver="red5" />
</modules>
</ivysettings>

Convert ivy.xml to pom.xml

I have a ivy.xml - https://gist.github.com/1898060
I also have the jar file related to this ivy.xml.
What i need is a mechanism to import this project to my maven repo and use it in my maven project.
SO basically if i am able to convert the ivy.xml to pom.xml , i might be able to get it work.
Is there some mechanism through which i can achieve this.
I am looking for something like a maven plugin to accomplish this task.
I know that there are ways we can edit the ivy.xml and build.xml to achieve this but then i dont want to do it , as the project is in a private repo.

What you really need to do is publish the jars built by ANT project into your Maven repository.
ant -Dproject.version=0.9.0-local-20120211095554 clean publish
I know you don't want to change the ANT build, but creating an extra "publish" target will properly integrate your ANT and Maven projects.
The two jar artifacts, published by your modified ANT build, could be consumed normally as follows:
<dependency>
<groupId>com.opengamma</groupId>
<artifactId>og-analytics</artifactId>
<version>0.9.0-local-20120211095554</version>
</dependency>
<dependency>
<groupId>com.opengamma</groupId>
<artifactId>og-analytics</artifactId>
<version>0.9.0-local-20120211095554</version>
<classifier>sources</classifier>
</dependency>
Modifications to your ANT build
ivy.xml
Main changes are to your publications section:
<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra">
<info organisation="com.opengamma" module="og-analytics"/>
<publications>
<artifact name="og-analytics" type="jar"/>
<artifact name="og-analytics" type="pom"/>
<artifact name="og-analytics" type="jar" e:classifier="sources"/>
</publications>
<dependencies>
<dependency name="og-util" rev="0.9.0-local-20120211095525" revConstraint="latest.integration"/>
<dependency org="org.jfree" name="jfreechart" rev="1.0.13"/>
<dependency org="cern" name="colt" rev="1.2.0"/>
<dependency org="cern" name="parallelcolt" rev="0.9.1"/>
<dependency org="latexlet" name="latexlet" rev="1.11"/>
<dependency org="org.apache.commons" name="commons-math" rev="2.1"/>
<dependency org="it.dexy" name="json-doclet" rev="0.3.1"/>
<dependency org="org.json" name="simple" rev="1.1"/>
<exclude org="org.junit"/>
</dependencies>
</ivy-module>
Notes:
The ANT project will now publish 3 files, jar, sources jar and the Maven POM
In Maven source jars have a "classifier" attributes that is set to "sources" (Not source). To facilitate this we're adding an ivy extra attribute.
No need for version and status information in the info tag header. This will be added by the publication step.
build.xml
<target name="prepare" description="Generate POM">
<fail message="Unset property: project.version" unless="project.version"/>
<ivy:deliver deliverpattern="${build.dir}/ivy.xml" pubrevision="${project.version}" status="release"/>
<ivy:makepom ivyfile="${build.dir}/ivy.xml" pomfile="${build.dir}/${ivy.module}.pom"/>
</target>
<target name="publish" depends="build,prepare" description="Upload to Nexus">
<ivy:publish resolver="nexus-deploy" pubrevision="${project.version}" overwrite="true" publishivy="false" >
<artifacts pattern="${build.dir}/[artifact](-[classifier]).[ext]"/>
</ivy:publish>
</target>
Notes:
The deliver task is optional, but recommended in case your ivy file contains dynamic revisions, such as "latest.release" or "latest.integration".
The makepoms task has powerful support for convert ivy configurations into Maven scopes. Does not apply in your case, but an incentive to learn more about ivy :-)
The publish task uses a specified pattern to find files specified in ivy's publications section.
ivysettings.xml
This is where you configure the location of the repositories and credentials to be used by publish build target.
<ivysettings>
<settings defaultResolver="nexus-central"/>
<credentials host="somehost" realm="Sonatype Nexus Repository Manager" username="????" passwd="????"/>
<resolvers>
<ibiblio name="nexus-central" root="http://somehost/nexus/content/repositories/central/" m2compatible="true"/>
<ibiblio name="nexus-deploy" root="http://somehost/nexus/content/repositories/repo" m2compatible="true"/>
</resolvers>
</ivysettings>
Notes:
Ivy downloads use the configured default resolver nexus-central.
The ivy publish task pushes to the Nexus repository called nexus-deploy
The security realm in this example matches Nexus Maven. Would be different for other repo managers.

Apache Ant itself provides a task to do this - makepom. Always helps to consult the documentation!