We Keep Coding

Ivy OutOfMemoryError during ivy:publish

I am having an issue with Ivy uploading large(ish) artifacts to our in-house artifact server (Artifactory 3.9.2). When we upload a 400MB file, we run out of Java heap space and it fails (thus failing our CI build). We keep nudging the upper limit up, but we're anticipating some significantly larger artifacts (~1GB in size) that is going to break our steady bumping in size.
We are using Ant 1.10, Ivy 2.4, OpenJSK 1.8.0.141, on a CentOS7 environment.
The issue has been documented with Ivy as IVY-1197 but it has not been fixed in a trunk build, so I want to put the described workaround:
The workaround is to always use ivy with commons-httpclient, commons-codec and commons-logging when the use case will involve uploading large files to sites requiring authentication
I want to add the commons-httpclient, commons-codec, and commons-logging to the Ivy classpath so that Ivy uses these during upload of the artifacts. Unfortunately, I am struggling to make this work. To be completely honest, I am a newcomer to the Java world, with classpaths being a foreign concept; I'm a C++ developer who has volunteered to fix this legacy part of our system.
We invoke our Ivy tasks through Ant. We pull in an ivysettings.xml file, whose relevant structure looks something like:
<ivysettings>
<classpath file="${build-utils.basedir}/ivy.lib/commons-httpclient-3.1.jar"/>
<classpath file="${build-utils.basedir}/ivy.lib/commons-codec-1.10.jar"/>
<classpath file="${build-utils.basedir}/ivy.lib/commons-logging-1.2.jar"/>
</ivysettings>
The ivysettings.xml is pulled in through a build-utils.xml file that is shared among our components:
<project name="build-utils" xmlns:ivy="antlib:org.apache.ivy.ant">
<taskdef resource="org/apache/ivy/ant/antlib.xml" uri="antlib:org.apache.ivy.ant">
<classpath>
<fileset dir="${build-utils.basedir}/ant.lib">
<include name="ivy*.jar"/>
</fileset>
</classpath>
</taskdef>
<ivy:settings file="${build-utils.basedir}/ivysettings.xml"/>
<target name="convert-POM-to-Ivy" > <!-- description="Convert a POM file to an Ivy file"; additionally verifies that we've pulled in Ivy correctly -->
<ivy:convertpom pomFile="pom.xml" ivyFile="ivyFileOut.xml" />
</target>
</project>
We also have an ivy.xml for each of our components that is pretty straightforward:
<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra">
<info organisation = "${groupId}"
module = "${artifactId}"
revision = "${version}}"/>
<configurations>
<include file="ivy-configurations.xml"/>
</configurations>
<!-- Some giant artifact -->
<artifact conf="${conf.el7}" type="tar.bz2" e:classifier="${conf.el7}"/>
<artifact type="pom"/>
<dependencies />
</ivy-module>
If I run the ant echoproperties target and grep for 'class.path' I get the following:
[echoproperties]
java.class.path=/usr/share/java/ant.jar\:/usr/share/java/ant-launcher.jar\:/usr/share/java/jaxp_parser_impl.jar\:/usr/share/java/xml-commons-apis.jar\:/usr/lib/jvm/java/lib/tools.jar\:/usr/share/ant/lib/ant-bootstrap.jar\:/usr/share/ant/lib/ant-launcher.jar\:/usr/share/ant/lib/ant.jar
[echoproperties]
java.class.path.ivy.instance=/usr/share/java/ant.jar\:/usr/share/java/ant-launcher.jar\:/usr/share/java/jaxp_parser_impl.jar\:/usr/share/java/xml-commons-apis.jar\:/usr/lib/jvm/java/lib/tools.jar\:/usr/share/ant/lib/ant-bootstrap.jar\:/usr/share/ant/lib/ant-launcher.jar\:/usr/share/ant/lib/ant.jar
With this setup, I still have issues with the upload, and I'm not sure if I can verify that I'm using the commons-httpclient.
Could someone provide some tips on how to get a jar into the Ivy classpath? Am I doing this wrong and I need to have it in the Ant classpath? If so, can someone point me in the right direction?

build.xml for ivy in Netbeans

I am trying to integrate Apache Ivy in a Netbeans web app project.
My build.xml file follows the netbeans tutorial:
http://wiki.netbeans.org/FaqIvy
Specifically:
<target name="-ivy-retrieve">
<ivy:retrieve/> <!-- Load dependencies to the project -->
<pathconvert property="ivy.classpath.computed" dirsep="/" pathsep=":">
<path>
<fileset dir="lib" includes="*.jar"/>
</path>
<map from="${basedir}${file.separator}" to=""/>
</pathconvert>
<propertyfile file="nbproject/project.properties">
<entry operation="=" key="ivy.classpath" value="${ivy.classpath.computed}"/>
</propertyfile>
</target>
<target name="-pre-compile" depends="-ivy-retrieve"/>
<target name="-pre-compile-single" depends="-ivy-retrieve"/>
<target name="-post-clean">
<delete dir="lib"/>
</target>
Strangely, this updates ivy.classpath in the nbprojects.project.properties file but I can't seem to get the build's WEB-INF/lib folder to get populated with the .jar's that are listed as dependencies.
Strangely, I had this working the other day and was able to populate the build's WEB-INF/lib folder and now when I remove the dependency, the same jar files are included in the build (rather than removing them).
Any suggestions on how to change the build.xml file? Or anything else that will get Ivy successfully integrated into my build process (Ivy is detected: I see "resolution report" and "retrieving" in the output while building).
Some things I've tried:
I played around with different options in the above linked tutorial (tried including:
<property name="ivy.lib.dir" value="web/WEB-INF/lib/"/> in the -ivy-retrieve task or adding the <ivy:resolve/> task before <ivy:retrieve/>)

The ivy:retrieve task has a sync property which removes not longer relevant dependencies.
Also, doing a clean and build helps.

Ivy dependency as provided

Problem: I need to have a lib on the eclipse classpath that should not be deployed to Tomcat. (In a maven project it would be scope provided)
Explanation:
I've setup a project with some Ivy dependencies and had to externalize a configuration as JNI (mail/session) in order to do it I had to put the mail-1.4.7.jar inside the Tomcat lib folder.
The problem is that I have a dependency that add to my classpath the javax.mail-1.5.2.jar so I change it to:
<dependency org="org.apache.logging.log4j" name="log4j-core" rev="2.2">
<exclude org="com.sun.mail" name="javax.mail"/>
</dependency>
The problem now is that my project break (compilation errors) because of missing mail classes such as javax.mail.MessagingException
So I have to add the mail dependency but only to eclipse. I've tried some configurations as explained here from what I know from Maven behavior with no avail.
Keeping the mail dependency only in the project, breaks Tomcat, keeping it on both tomcat and project breaks project. When I manually remove it from my project lib folder (WEB-INF\lib), after deploy the project, it works properly.
Bottom line (after deploy):
tomcatFolder
|_lib
| |_...
| |_mail-1.4.7.jar
| |_...
|_webapps
|_myproject
|_WEB-INF
|_lib
|_...
|_javax.mail-1.5.2.jar //need to remove it at deploy time only
|_...
Can't change it to maven right now. But it is in process :)

This is really a duplicate of this question:
How to copy runtime libraries without the provided ones in IVY
But.. from your question I suspect you're not using ivy configuration mappings. This is unfortunate because this is the mechanism used by ivy to logically group dependencies into functional groupings, similar to how Maven maintains scopes. The following posting attempts to bridge this understanding
How are maven scopes mapped to ivy configurations by ivy
Furthermore you are also using Eclipse, which means that unless you're using the ivy plugin you effectively have two build mechanisms. (ivy and eclipse). I would recommend fixing your ANT build first and then look at how to maintain the Eclipse classpath second.
Example
The first section describes how configurations are declared and used in the ivy file and the second section explains how the ivy ANT tasks are used in the build logic.
ivy.xml
You should always declare ivy configurations and use these to control your classpaths. In my builds I always have at least three: compile, runtime and test. Notice how the extends attribute is used to create relationships between the configs, because runtime should also include the compile dependencies.
Adding an additional one for the provided scope jars is easy. Simple stand-alone configuration:
<ivy-module version="2.0">
<info organisation="com.myspotontheweb" module="demo"/>
<configurations>
<conf name="compile" description="Required to compile application"/>
<conf name="runtime" description="Additional run-time dependencies" extends="compile"/>
<conf name="test" description="Required for test only" extends="runtime"/>
<conf name="provided" description="Needed for compile, but will be present on the target platform."/>
</configurations>
<dependencies>
<!-- compile dependencies -->
<dependency org="org.slf4j" name="slf4j-api" rev="1.7.5" conf="compile->default"/>
<!-- runtime dependencies -->
<dependency org="org.slf4j" name="slf4j-log4j12" rev="1.7.5" conf="runtime->default"/>
<!-- test dependencies -->
<dependency org="junit" name="junit" rev="4.11" conf="test->default"/>
<!-- provided dependencies -->
<dependency org="org.apache.tomcat" name="servlet-api" rev="6.0.16" conf="provided->master"/>
</dependencies>
</ivy-module>
It's the configuration mappings that make things special. The simple explanation is that they fall into two basic types when pulling from a Maven repository:
conf="local_configuration->default"
conf="local_configuration->master"
The first means include the remote module and all its dependencies. The second means include the remote module and exclude it's dependencies. This means you don't need the following exclude trickery:
<dependency org="org.apache.logging.log4j" name="log4j-core" rev="2.2">
<exclude org="com.sun.mail" name="javax.mail"/>
</dependency>
You simply use the following, if all you want is the log4j-core jar:
<dependency org="org.apache.logging.log4j" name="log4j-core" rev="2.2" conf="provided->master"/>
Additional notes:
In ivy mapping to the remote "default" configuration will pull down only the jars you need. It will exclude optional dependencies and other stuff like javadocs.
Sometimes "excludes" are necessary when module authors get their dependencies wrong.
build.xml
The resolve target will pull down dependencies, generate a report and create the compile and test classpaths. Note the use of configurations to determine which jar groupings should be used:
<target name="resolve" description="Use ivy to resolve classpaths">
<ivy:resolve/>
<ivy:report todir='${build.dir}/ivy-reports' graph='false' xml='false'/>
<ivy:cachepath pathid="compile.path" conf="compile,provided"/>
<ivy:cachepath pathid="test.path" conf="test,provided"/>
</target>
These classpath references are then used by the compile target as normal:
<target name="compile" depends="resolve,resources" description="Compile code">
<mkdir dir="${build.dir}/classes"/>
<javac srcdir="${src.dir}" destdir="${build.dir}/classes" includeantruntime="false" debug="true" classpathref="compile.path"/>
</target>
<target name="compile-tests" depends="compile" description="Compile tests">
<mkdir dir="${build.dir}/test-classes"/>
<javac srcdir="${test.src.dir}" destdir="${build.dir}/test-classes" includeantruntime="false" debug="true">
<classpath>
<path refid="test.path"/>
<pathelement path="${build.dir}/classes"/>
</classpath>
</javac>
</target>
And the test target:
<target name="test" depends="compile-tests" description="Run unit tests">
<mkdir dir="${build.dir}/test-reports"/>
<junit printsummary="yes" haltonfailure="yes">
<classpath>
<path refid="test.path"/>
<pathelement path="${build.dir}/classes"/>
<pathelement path="${build.dir}/test-classes"/>
</classpath>
<formatter type="xml"/>
<batchtest fork="yes" todir="${build.dir}/test-reports">
<fileset dir="${test.src.dir}">
<include name="**/*Test*.java"/>
<exclude name="**/AllTests.java"/>
</fileset>
</batchtest>
</junit>
</target>
Lastly the ivy retrieve task is used to build the war file. Only the "runtime" configuration jars are used:
<target name="package" depends="test" description="Create the WAR file">
<ivy:retrieve pattern="${build.dir}/lib/[artifact].[ext]" conf="runtime"/>
<war destfile="${war.file}" webxml="${resources.dir}/web.xml">
<fileset dir="${resources.dir}" excludes="web.xml"/>
<lib dir="${build.dir}/lib"/>
</war>
</target>
In conclusion the cachepath ivy task is used to create classpath references based on the ivy configurations and the retrieve task is used when assembling the war file.

Generating a classpath that dynamically chooses .jar files

I have a project where we store a local copy of some client API jars. These jars may or may not be the same as the final run-time versions used by the client. When I build the project locally for testing, I want to use the local copies of the jars. However, when I'm on-site with the client, I want to look at an external location from my project and check if the client updated jars are available. If they are we use them, if not we fall back on our local copies.
I have found a way to conditionally build a file-set containing the given jars, but I can't figure out how to get the file-set into my classpath.
Below is the script I have generated so far. I do not have access to the ant-contrib IF statements, so this is the only way I've found to conditionally build the file-set.
<condition property="externalFileExists">
<available file="[path to external file]"/>
</condition>
<target name="buildExternalFileset" if="externalFileExists">
<fileset id="fileset" includes="[path to external library]"/>
</target>
<target name="buildInternalFileset" unless="externalFileExists">
<fileset id="fileset" includes="[path to internal library]"/>
</target>
<target name="buildFileset depends="buildExternalFileset, buildInternalFileSet>
</target>
Once I have this file-set built, how do I include it in the classpath?
<path id="classpath">
<!-- other filesets -->
<!-- line that includes the conditional fileset by ID -->
</path>
According to the documentation for path-like structures you can include a path within another path, but I don't see an example for including an external file-set. Perhaps I've answered my question - - should I be building paths instead of file-sets? Additionally, file-set documentation doesn't actually include the option to add IDs, do they work?
If anyone has a better way to do this I'd be more than happy to hear about it!

I would suggest using a dependency manager rather than try and building your own.
Apache ivy is able to download dependencies from configured locations. It also caches downloaded files removing the need to store them locally if they are universally available from Maven Central.
Example
This project has an ivy settings file that pulls dependencies from either the local "lib" directory or from Maven central.
├── build.xml
├── ivysettings.xml
├── ivy.xml
└── lib
├── hamcrest-core-1.3.jar
├── junit-4.11.jar
├── log4j-1.2.17.jar
├── slf4j-api-1.7.5.jar
└── slf4j-log4j12-1.7.5.jar
build.xml
<project name="demo" default="resolve" xmlns:ivy="antlib:org.apache.ivy.ant">
<property name="build.dir" location="target"/>
<available classname="org.apache.ivy.Main" property="ivy.installed"/>
<target name="install-ivy" unless="ivy.installed">
<mkdir dir="${user.home}/.ant/lib"/>
<get dest="${user.home}/.ant/lib/ivy.jar" src="http://search.maven.org/remotecontent?filepath=org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar"/>
<fail message="Ivy installed run build again"/>
</target>
<target name="resolve" depends="install-ivy" description="Use ivy to resolve classpaths">
<ivy:resolve/>
<ivy:report todir='${build.dir}/ivy-reports' graph='false' xml='false'/>
<ivy:cachepath pathid="compile.path" conf="compile"/>
<ivy:cachepath pathid="test.path" conf="test"/>
</target>
<target name="clean">
<delete dir="${build.dir}"/>
</target>
<target name="clean-all" depends="clean">
<ivy:cleancache/>
</target>
</project>
Notes:
Ivy does not come bundled with ANT. The "install-ivy" task ensures it gets installed on the first ant run
The "cachepath" tasks will create classpaths for use in your javac, junit and java tasks.
The "report" and "cleancache" tasks are useful additional tasks.
ivy.xml
<ivy-module version="2.0">
<info organisation="com.myspotontheweb" module="demo"/>
<configurations>
<conf name="compile" description="Required to compile application"/>
<conf name="runtime" description="Additional run-time dependencies" extends="compile"/>
<conf name="test" description="Required for test only" extends="runtime"/>
</configurations>
<dependencies>
<!-- compile dependencies -->
<dependency org="org.slf4j" name="slf4j-api" rev="1.7.5" conf="compile->default"/>
<!-- runtime dependencies -->
<dependency org="org.slf4j" name="slf4j-log4j12" rev="1.7.5" conf="runtime->default"/>
<!-- test dependencies -->
<dependency org="junit" name="junit" rev="4.11" conf="test->default"/>
</dependencies>
</ivy-module>
Notes:
Uses configurations to keep dependencies separated. Note how this is used by the "cachepath" tasks to control dependency classpaths
ivysettings.xml
<ivysettings>
<settings defaultResolver="my-resolvers"/>
<resolvers>
<chain name="my-resolvers" returnFirst="true">
<filesystem name="local-lib">
<artifact pattern="${ivy.settings.dir}/lib/[artifact]-[revision].[ext]"/>
</filesystem>
<ibiblio name="central" m2compatible="true"/>
</chain>
</resolvers>
</ivysettings>
Notes:
This file is optional. By default Ivy will retrieve from Maven Central
The special chain resolver controls which source of dependencies is used. In this case the local "lib" directory is favoured over Maven Cental.

I'm not very familiar with ant, but how about a build configuration on ant ?
Try this site for a tutorial on how to do it: http://ant.apache.org/easyant/history/trunk/howto/BuildConfigurations.html

Try something like this
<classpath>
<fileset dir="[external-path1]">
<include name="**/*.jar"/>
</fileset>
<fileset dir="[external-path2]">
<include name="**/*.jar"/>
</fileset>
<fileset dir="[external-path3]">
<include name="**/*.jar"/>
</fileset>
</classpath>
found here
https://ant.apache.org/manual/using.html#path

Creating a WAR file using Java

How do I create a deployable WAR file for Apache Tomcat servers programmatically with Java?
Is there a library for such a task?
I am working on a small own IDE for special purposes. The IDE is written in Java and JavaScript, so I need to create the WAR file using those.

If you want to build it from code try to do it from the command line with
Runtime rt = Runtime.getRuntime();
Process pr = rt.exec("jar cvf /path/to/your/project/your-file.war");
Of course, the same thing would work using ANT or Maven (as long as those tools are installed on the final platform).
Edit: added improvement suggestion

I don't know of libraries, but a WAR file is just a ZIP file with a different ending.
Just create the inner folder structure and files (google the java code for that) and package as zip (java has methods for that too I think, again google) and rename the file from "myfile.zip" to "myfile.war"

I don't know how you would do it using the IDE you have. But a WAR file has the following structure:
web resources go to the root
project classes (including their package folders) go to a folder WEB-INF/classes
project dependency jars go to WEB-INF/lib
So if you want to build a WAR by hand, you need to create that file structure inside a zip file with a .war extension and you need to copy that to the proper location of the server to deploy it. Most servers also allow 'exploded deployment', meaning that you don't need an actual war file, you can just deploy the stuff to a directory with the same name as your war (IE. 'myapp.war').

You can do this a number of ways, for a quick example if you are using maven you just need to use <packaging>war</packaging>
You could just export the war as has been mentioned, but it's not exactly "programmatic".
If you're using Ant - you can find a tutorial for this here

<?xml version="1.0" ?>
<path id="compile.classpath">
<fileset dir="WebContent/WEB-INF/lib">
<include name="*.jar"/>
</fileset>
</path>
<target name="init">
<mkdir dir="build/classes"/>
<mkdir dir="dist" />
</target>
<target name="compile" depends="init" >
<javac destdir="build/classes" debug="true" srcdir="src">
<classpath refid="compile.classpath"/>
</javac>
</target>
<target name="war" depends="compile">
<war destfile="/APP/jboss-5.1.0.GA/server/all/deploy/DispatchActionEx.war" webxml="WebContent/WEB-INF/web.xml">
<fileset dir="WebContent"/>
<lib dir="WebContent/WEB-INF/lib"/>
<classes dir="build/classes"/>
</war>
</target>
<target name="clean">
</target>
Just write an build.xml file(i have give an example),
change "project name" and "war destfile" which will be ".../apache-tomcat/webapps/projectname.war"
put it in your project folder
open it in eclipse.
right click on it>>run as>>ant build
check whether the war file is created in the webapps folder in apache-tomcat