We Keep Coding

How to use Jenkins job(s) with dependent git repositories and multiple branches

We have 2 git repositories, Platform and US (we have other geo-specific ones as well which is why they are split, but they are not necessarily relevant here). US depends on Platform.
We are using git-flow (meaning new features are in their own branches like feature/some-product, develop branch is somewhat more stable and represents QA-ready builds, master branch is stable and for releases). (If a feature has both Platform and US parts, there will be a branch in each with the same name.) We decided that the Jenkins jobs for the features should not run mvn deploy because we don't want to publish them to the snapshot repository and probably shouldn't run mvn install because we don't want a different feature branch to grab it from Jenkins's local repo (this we are less sure about though). We believe they should only make sure everything compiles and that the unit tests pass (mvn verify).
This is where the problem comes in, because these are separate git repositories and we are not doing anything with the compiled jar (install or deploy),
how can we safely expose the compiled jars from the Platform job to the US without exposing them to other developers or jobs (or is this even a concern is only doing mvn install) or
how can one Jenkins job build Platform and US for a specific branch together?
If we only have a single actively developed branch (or we were using subversion) this would not be an issue.
Some ideas we have (and concerns with each)
For feature branches use a different version (e.g., 8.1.0-SNAPSHOT-some-product).
This seems like a lot of work for every feature branch.
It seems like it'd clog up the local repo with "stale" jars, and we would need to worry about purging them.
Somehow use git submodule to checkout Platform's and US's feature/some-product and either use mvn verify --reactor or a simple pom file with the top level projects as modules.
How to make Jenkins add the submodules?
If the submodules were already there, there would need to be a whole git repo for this, which seems redundant.
--reactor doesn't work always.
How to supply the pom file?
Just do mvn install.
feature/other-thing may only be on US, so after Platform feature/some-product publishes to Jenkins local repository (which may be very different from Platform develop, which US feature/other-thing would be built against normally), it would (We think) cause US feature/other-thing to fail (or pass!) in a false sense (supposing that if it were compiled against Platform develop it could possibly get a different result).

I have not had to address this issue personally.... here is my thoughts on how I would look at the issue:
If you MUST only have one job for both branches (a bad idea), you can use parameterized build plugin to pass in the text string "US" or "Platform" and have logic in a shell script that will check out the relevant repo's branch.
HOWEVER, this eliminates the ability to have repo polling kickoff the build. You would have to set up a build schedule on a cron and you would get a new build no matter what, even if the repo hasn't changed (unless your batch / shell script is smart enough to check for changes).
I don't see any reason NOT to have two separate Jenkins Jobs, one for each branch.
If one job needs access to the .jars (aka the build artifacts) then you can always reference the artifacts of any other jar from the job's "LATEST" URL on the jenkins server. Be sure the jobs specify what artifacts need to get archived.

The way I ended up solving this is using the maven versions plugin. I had to make sure all the modules were managed dependencies in the top-level project, but that may have been a different issue. Also, I am sure of this, the US project will need to explicitly declare its version even if it is the same as the parent.
They both poll git but the Platform job also triggers US if it built successfully.
The way the versions plugin works will require you to do this in two steps. Add 2 "Invoke top-level Maven targets" in the job, the second is the clean deploy. The first is a little different for the Platform and US.
Platform: mvn versions:set -DnewVersion=yourBranchName-${project.version}.
US: mvn versions:update-parent -DparentVersion=yourBranchName-${project.version} versions:set -DnewVersion=yourBranchName-${project.version}
If the branch only exists on the US repository, then obviously don't make the Platform one, and the US one is the same command as what the Platform one's would have been.
One final issue I ran into was originally I had the new version as ${project.version}-yourBranchName but the issue here was that the repository the job was deploying to only accepted snapshots and because the version didn't end in -SNAPSHOT it gave error code 400.

Avoid update POM versions for next development iteration with maven-release-plugin

I'm doing my first release with maven-release-plugin and git. Instead of tagging I'm using the release:branch goal. I'm agree with the complete relase-flow process except in one point: all local dependencies are automatically updated to the next SNAPSHOT version in the development branch.
This is my concrete situation: I'm working in a project with more than 20 maven projects. Let's say all of them are in SNAPSHOT version during development. I want to make a release with the maven plugin. After the release is done, all POM versions have been incremented and set again to SNAPSHOT. If I need to make a change in just one line of code of one project (fix a bug) and I make a new release, all projects (by default) must increment their version again when I've just changed one of them.
In my opinion, it makes more sense to continue the development with the state of the last release (all version without SNAPSHOT). In the next development iteration, I can set manually the next SNAPSHOT version only in these projects that I have changed. In the next release, only the changed projects will be "promoted" to a new version but not all of them.
I think this situation is not exceptional but I didn't found information how to do it (and I don't know if I'm breaking the "release" phylosophy...)
Somedy knows how to avoid update all POMs to the next SNAPSHOT version and leave the POM files as the relase version?
Thank you in advance!

The idea of a multimodule project is that all modules are part of the same release cycle. Even is some modules contain no changes, they will be part of the release. As some may say: versions are cheap.
The benefits are bigger compared to the ideal picture you described: it is much easier to manage the versions and to control inter-module dependencies.
Keep in mind that both trunk and branches should always have a SNAPSHOT version, since these are the working copies. If you use final versions, releases will fail, since you can't tag the same version twice, neither can you deploy(=upload) the same version twice. This is very important, because Maven relies on the definition that final versions are immutable, i.e. Maven will never download a final version again once it is available in the local repository.
So if some modules don't have the same release-cycle as the total multimodule project, you should move them out of this multimodule project (or release them all at once).

The differences between releasing individual modules, applications or application systems are summarized in my slideshare presentation http://www.slideshare.net/geertpante/version-mgmt-in-maven (slide 20 and following).
What we sometimes do is set the snapshot version to e.g. 2.0-SNAPSHOT, and do releases as 1.x.y. So in development, we always keep using 2.0-SNAPSHOT, but when we release, we choose older versions, e.g. 1.3, 1.4, and choose 2.0-SNAPSHOT as the next release version.
Also, if you use git, take a look at https://bitbucket.org/atlassian/jgit-flow/wiki/Home. One of the advantages is it keeps a single development branch, and only the releases are merged to the master branch:
mvn jgitflow:release-start -DreleaseVersion=1.2 -DdevelopmentVersion=2.0-SNAPSHOT
mvn jgitflow:release-finish -DreleaseVersion=1.2 -DdevelopmentVersion=2.0-SNAPSHOT

What is the Maven way for automatic project versions when doing continuous delivery?

I have a web application where we deploy to production whenever a feature is ready, sometimes that can be a couple of times a day, sometimes it can be a couple of weeks between releases.
Currently, we don't increment our version numbers for our project, and everything has been sitting at version 0.0.1-SNAPSHOT for well over a year.
I am wondering what is the Maven way for doing continuous delivery for a web apps. It seems overkill to bump up the version number on every commit, and never bumping the version number like we are doing now, also seems wrong.
What is the recommend best practice for this type of Maven usage?
The problem is actually a two-fold one:
Advancing project version number in individual pom.xml file (and there can be many).
Updating version number in all dependent components to use latest ones of each other.

I recommend the following presentation that discusses the practical realities of doing continuous delivery with Maven:
You tube presentation on CD with Maven
Slides
The key takeaway is each build is a potential release, so don't use snapshots.

This is my summary based on the video linked by Mark O'Connor's answer.
The solution requires a DVCS like git and a CI server like Jenkins.
Don't use snapshot builds in the Continuous Delivery pipeline and don't use the maven release plugin.
Snapshot versions such as 1.0-SNAPSHOT are turned into real versions such as 1.0.buildNumber where the buildNumber is the Jenkins job number.
Algorithm steps:
Jenkins clones the git repo with the source code, and say the source code has version 1.0-SNAPSHOT
Jenkins creates a git branch called 1.0.JENKINS-JOB-NUMBER so the snapshot version is turned into a real version 1.0.124
Jenkins invokes the maven versions plugin to change the version number in the pom.xml files from 1.0-SNAPSHOT to 1.0.JENKINS-JOB-NUMBER
Jenkins invokes mvn install
If the mvn install is a success then Jenkins will commit the branch 1.0.JENKINS-JOB-NUMBER and a real non-snapshot version is created with a proper tag in git to reproduce later. If the mvn install fails then Jenkins will just delete the newly created branch and fail the build.
I highly recommend the video linked from Mark's answer.

Starting from Maven 3.2.1 continuous delivery friendly versions are supported out of the box : https://issues.apache.org/jira/browse/MNG-5576
You can use 3 predefined variables in version:
${changelist}
${revision}
${sha1}
So what you basically do is :
Set your version to e.g. 1.0.0-${revision}. (You can use mvn versions:set to do it quickly and correctly in multi-module project.)
Put a property <revision>SNAPSHOT</revision> for local development.
In your CI environment run mvn clean install -Drevision=${BUILD_NUMBER} or something like this or even mvn clean verify -Drevision=${BUILD_NUMBER}.
You can use for example https://wiki.jenkins-ci.org/display/JENKINS/Version+Number+Plugin to generate interesting build numbers.
Once you find out that the build is stable (e.g. pass acceptance tests) you can push the version to Nexus or other repository. Any unstable builds just go to trash.

There are some great discussions and proposals how to deal with the maven version number and continuous delivery (CD) (I will add them after my part of the answer).
So first my opinion on SNAPSHOT versions. In maven a SNAPSHOT shows that this is currently under development to the specific version before the SNAPSHOT suffix. Because of this, tools like Nexus or the maven-release-plugin has a special treatment for SNAPSHOTS. For Nexus they are stored in a separate repository and its allowed to update multiple artefacts with the same SNAPSHOT release version. So a SNAPSHOT can change without you knowing about it (because you never increment any number in your pom). Because of this I do not recommend to use SNAPSHOT dependencies in a project especially in a CD world since the build is not reliable any more.
SNAPSHOT as project version would be a problem when your project is used by other ones, because of the above reasons.
An other problem of SNAPSHOT for me is that is not really traceable or reproducibly any more. When I see a version 0.0.1-SNAPSHOT in production I need to do some searching to find out when it was build from which revision it was build. When I find a releases of this software on a filesystem I need to have a look at the pom.properties or MANIFEST file to see if this is old garbage or maybe the latest and greatest version.
To avoid the manual change of the version number (especially when you build multiple builds a day) let the Build Server change the number for you. So for development I would go with a
<major>.<minor>-SNAPSHOT
version but when building a new release the Build Server could replace the SNAPSHOT with something more unique and traceable.
For example one of this:
<major>.<minor>-b<buildNumber>
<major>.<minor>-r<scmNumber>
So the major and minor number can be used for marketing issues or to just show that a new great milestone is reached and can be changed manually when ever you want it. And the buildNumber (number from your Continuous Integration server) or the scmNumber (Revision of SUbversion or GIT) make each release unique and traceable. When using the buildNumber or Subversion revision the project versions are even sortable (not with GIT numbers). With the buildNumber or the scmNumber is also kinda easy to see what changes are in this release.
An other example is the versioning of stackoverflow which use
<year>.<month>.<day>.<buildNumber>
And here the missing links:
Versioning in a Pipeline
Continuous Delivery and Maven

DON'T DO THIS!
<Major>.<minor>-<build>
will bite you in the backside because Maven treats anything after a hyphen as LEXICAL. This means version 1 will be lexically higher than 10.
This is bad as if you're asking for the latest version of something in maven, then the above point wins.
The solution is to use a decimal point instead of a hyphen preceding the build number.
DO THIS!
<Major>.<minor>.<build>
It's okay to have SNAPSHOT versions locally, but as part of a build, it's better to use
mvn versions:set -DnewVersion=${major}.${minor}.${build.number}
There are ways to derive the major/minor version from the pom, eg using help:evaluate and pipe to a environment variable before invoking versions:set. This is dirty, but I really scratched my head (and others in my team) to make it simpler, and (at the time) Maven wasn't mature enough to handle this. I believe Maven 2.3.1 might have something that go some way in helping this, so this info may no longer be relevant.
It's okay for a bunch of developers to release on the same major.minor version - but it's always good to be mindful that minor changes are non-breaking and major version changes have some breaking API change, or deprecation of functionality/behaviour.
From a Continuous Delivery perspective every build is potentially releasable, therefore every check-in should create a build.

At my work for web apps we currently use this versioning pattern:
<jenkins build num>-<git-short-hash>
Example: 247-262e37b9.
This is nice because it it gives you a version that is always unique and traceable back to the jenkins build and git revision that produced it.
In Maven 3.2.1+ they finally killed the warnings for using a ${property} as a version so that makes it really easy to build these. Simply change all your poms to use <version>${revision}</version> and build with -Drevision=whatever. The only issue with that is that in your released poms the version will stay at ${revision} in the actual pom file which can cause all sorts of weird issues. To solve this I wrote a simple maven plugin (https://github.com/jeffskj/cd-versions-maven-plugin) which does the variable replacement in the file.

As a starting point you may have a look at Maven: The Complete Reference. Project Versions.
Then there is a good post on versioning strategy.

Building Artifacts on Jenkins for Multiple Environments and Uploading them to S3

Environment:
Around a dozen services: both WARs and JARs (run with the Tanuki wrapper)
Each service is being deployed to development, staging, and production: the environments use different web.xml files, Spring profiles (set on the Tanuki wrapper),...
We want to store the artifacts on AWS S3 so they can be easily fetched by our EC2 instances: some services run on multiple machines, AutoScaling instances automatically fetch the latest version once they boot, development artifacts should automatically be deployed once they are updated,...
Our current deployment process looks like this:
Jenkins builds and tests our code
Once we are satisfied, we do a release with the Artifactory plugin (only for production deployments; development and staging artifacts are based on plain Jenkins builds)
We use the Promoted Builds plugin with 3 different promotion jobs for each project (development, staging, production) to build the artifact for the desired environment and then upload it to S3
This works mostly OK, however we've run into multiple annoyances:
The Artifactory plugin cannot tag and updated the source code due to not being able to commit to the repository (this might be specific to CloudBees, our Jenkins provider). Never mind - we can manually do that.
The Jenkins S3 plugin doesn't properly work with multiple upload targets (we are using one for each environment) - it will try to upload to all of them and duplicate the settings when you save the configuration: JENKINS-18563. Never mind - we are using a shell script for the upload, which is working fine.
The promotion jobs will sometimes fail, because they can be run on a different instance than the original build - either resulting in a failed (build because the files are missing) or in an outdated build (since an old version is being used). Again, this might happen due to the specific setup CloudBees is providing. This can be fixed by running the build again and hopefully having the promotion job run on the same instance as the its build this time (which is pure luck in my understanding).
We've been advised by CloudBees to do a code checkout before running the promotion job, since the workspace is ephemeral and it's not guaranteed that it exists or that it's up to date. However, shell scripts in promotion jobs don't seem to honor environment variables as stated in another StackOverflow thread, even though they are linked below the textarea field. So svn checkout ${SVN_URL} -r ${SVN_REVISION} doesn't work.
I'm pretty sure this can be fixed by some workaround, but surely I must be doing something terribly wrong. I assume many others are deploying their applications in a similar fashion and there must be a better way - without various workarounds and annoyances.
Thanks for any pointers and reading everything to the end!

The biggest issue you face is that you actually want to do a build as part of the promotion. The promoted builds plugin is designed to do a few small actions, primarily on the archived artifacts of your build, or else to perform tagging or other such actions.
The design ensures that it gets a workspace on a slave. But (and this is a general Jenkins thing not a CloudBees specific) firstly the workspace you get need not be the workspace that was used for the build you are trying to promote. It could be:
An empty workspace
The workspace of the most recent build of the project (which may be a newer build than you are trying to promote)
The workspace of an older build of the project (when you cannot get onto the slave that has the most recent build of the project)
Now which of these you get entirely depends on the load that your Jenkins cluster is under. When running on CloudBees, you are usually most likely to encounter either of the first two situations above.
The actions you place in your promotion should therefore be “self-sufficient”. So for example they will copy the archived artifacts into a specific directory and then use those copied artifacts to do their 'thing'. Or they will trigger a downstream build passing through parameters from the build that is promoted. Or they will perform some action on a remote server without using any local state (i.e. flipping the switch in a blue-green deployment)
In your case, you are fighting on multiple fronts. The second front you are fighting on is that you are fighting Maven.
Maven does not like it when a build profile being active produces a different, non-equivalent, artifact.
Maven can probably live with the release profile producing a version of the artifact with JavaScript minified and perhaps debug info stripped (though it would be better if it could be avoided)... but that is because the two artifacts are equivalent. A webapp with JavaScript minified should work just as well as one that is not minified (unless minification introduces bugs)
You use profiles to build completely different artifacts. This is bad as Maven does not store the active profile as part of the information that gets deployed to the Maven repository. Thus when you declare a dependency on one of the artifacts, you do not know which profile was active when that artifact was built. A common side-effect of this is that you end up with split-brain artifacts, where one part of a composite artifact is talking to the development DB and the other part is talking to the production DB. You can tell there is this bad code smell when developers routinely run mvn clean install -Pprod && mvn clean install -Pprod when they want to switch to building a production build because they have been burned by split-brain artifacts in the past. This is a symptom of bad design, not an issue with Maven (other than it would be nice to be able to create architecture specific builds of Maven artifacts... but they can pose their own issues)
The “Maven way” to handle the creation of different artifacts for different deployment environments is to use a separate module to repack the “generic” artifact. As is typical of Maven's subtle approach to handling bad architecture, you will need to add repack modules for each environment you want to target... which discourages having lots of different target environments... (i.e. if you have 10 webapps/services to deploy and three target deployment environments you will have 10 generic modules and 3x10 target specific modules... giving a 40 module build) and hopefully encourages you to find a way to have the artifact self-discover its correct configuration when deployed.
I suspect that the safest way to hack a solution is to rely on copying archived artifacts.
During the main build, create a file that will checkout the same revision as is being built, e.g.
echo '#!/usr/bin/env sh' > checkout.sh
echo "svn revert . -R" >> checkout.sh
echo "svn update --force -r ${SVN_REVISION}" >> checkout.sh
Though you may want to write something a little more robust, e.g. ensuring that you are in a SVN working copy, ensuring that the working copy is using the correct remote URI, removing any files that are not needed, etc
Ensure that the script you create is archived.
Add at the beginning of your promotion process, you copy the archived checkout.sh to the workspace and then run that script. The you should be able to do all your subsequent promotion steps as before.
A better solution would be to create build jobs for each of the promotion processes and pass the SVN revision through as a build parameter (but will require more work to set up for you (as you already have the promotion processes set up)
The best solution is to fix your architecture so that you produce artifacts that discover their correct configuration from their target environment, and thus you can just deploy the same artifact to each required environment without having to rebuild artifacts at all.

Completely override Maven Release lifecycle

In effort to customize the release of a project I came across the following article:
http://www.sonatype.com/people/2011/01/using-the-maven-release-plugin-things-to-know/
In essence the following assumptions are made for using the default release plugin:
Your codebase is going to be versioned and released as a “unit”.
You are using an SCM tool and a repository manager.
You are performing your release from a single, “versionable” unit in SCM.
You are using standard version numbers
You are publishing artifacts to a repository
In mostly all these cases, our project does not meet these assumptions. We want to use a custom version schema (independent of SCM, maven, etc). Deploy the artifacts to a filesystem (not a repository). Not have maven mess around with SCM at all. Etc.
As recommended, we should probably define our own release lifecycle. Therefore I am assuming we would need to override the maven default lifecycle release phase to run our plugin. I guess I am missing the location of the required documentation. Is this even possible?

What you're doing may be too radical to be worth doing within Maven, you may find yourself effectively writing a parallel build system simply triggered from Maven.
Assuming the effort is worth it - I would start by configuring the various plugins (most of Maven's functionality is delivered through plugins, triggered by phases in the lifecycle) and have them execute at a different goals within different phases. You may need to write Maven plugins to do whatever you want that doesn't already exist; which I expect you'll need to do in such a radical case.
http://maven.apache.org/guides/plugin/guide-java-plugin-development.html