I've a web-maven project 'A' that depends of a maven project 'B'. I'm using Jenkins to deploy. Before project 'A' deploy process, there is a pre-build parameter that builds the project 'B' and then proceeds with compilation. I'd need to indicate that the project 'A' will use the newly-compiled B.jar instead wich is defined on its pom.xml.
I saw this link How do I trigger another job from hudson as a pre-build step?, but I don't know how indicates the newly jar.
How can I do this?
The easy, but not guaranteed way is to force dependency updates (-U).
Project A pre builds causes B to build
B deploys B.jar to repository
A maven's build starts, check dependencies (-U), finds the most recent B which is the one you just built
A variation is to use a specific local repository shared between A and B jobs (in the advanced maven config in Jenkins), but that can be annoying if you want the jobs to execute on different slaves.
The more complex, but safer approach I'd recommend is to create a reactor project (i.e. a maven project that has A and B as modules), and build that. It can be tricky to set up if you're dealing with different repositories, but once that's done since A and B are executed in the same reactor you're guaranteed A will use the B that was just built.
Related
The ultimate goal is to find out a way to functional test a library.
I have a parent project A with sub modules B,C and D.
Let us assume that the module C has a dependency on module B (B being the library) and is included as a maven dependency using the <dependency> tag.
I am just curious to know when I start up the application server say Tomcat would a jar for B (along with others) be created and then C uses this jar?
I tried to monitor my directories to see whether jars were being created every time the server was started up but that doesn't seem to be the case.
If B isn't used as a jar then it would not exactly be a FT since the ultimate goal is to be able to use B as a library in other projects as well. I feel like this shouldn't be a problem since the code itself does not change but I'd like to get some insight into the same and comments if any. Thank you !
I hope I get it right, because you seem to mix building, deploying and testing
Maven will only build the current pom. It will not build dependency libraries but expect to find them in the repository.
Only if module A is a multi-module build, then B, C and D will be built when you build A.
Starting a tomcat will not trigger any build.
For a functional test you would build, package and deploy and execute your tests against the full application.
We are working on a legacy project and the first task is to setup a DevOps for the same.
The important thing is we are very new to this area.
We are planning to use jenkins and sonarqube for the purpose initially. Let me start with the requirements.
Currently the the project is sub divided into multiple projects (not modules)
We had to follow this build structure as there are no plans for re-organising it as a single multimodule maven project
Currently the builds and dependencies are managed manually
Eg: The project is subdivided in to 5 multi-module maven projects,
say A,B,C,D and E
1. A and C are completely independent and can be easly built
2. B depends on the artifact generated by A (jar) and has multiple maven profiles (say tomcat and websphere, it is a webservice module)
3. D depends on the artifact generated by C
4. E depends on A, B and D and has multiple maven profiles (say tomcat and websphere, it is a web project)
Based on jenkins documentation to handle this scenario, we are thinking about parameterized builds using “parameterized build plugin" and "extended choice parameter plugin" with the help of these plugins we are able to parameterize the profile name. But before each build, the builder waits for the profile parameter.
So we are still searching to find an good solution to
1. keep the dependency between projects an built the whole projects if there is any change in SCM (SVN). For that we are used "Build whenever a SNAPSHOT dependency is built" and "SCM polling option". Unfortunately this option seems not working in our case (we have given an interval of 5 min for scm polling but no build is happening based on test commits)
2. Even though we are able to parameterize the profile, this seems as a manual step (is there an option to automate this part too, ie. build with tomcat profile and websphere profile should happen sequentially).
We are struggling to find a solution to cater all these core requirements. Any pointer would be greatly appreciated.
Thanks,
San
My maven knowledge is limited, however since you didnt get any response yet, ill try to give some general advice.
There are usually multiple ways to reach some aim in Jenkins, each has its pros and cons. Choosing the most fitting solution depends on the specific requirements and your environment/setup.
However you first need something that just works, then you can refine it.
A quick result you get with the following
Everything in one job
Configure your subversion repo (Multiple are possible) to be checked out into your workspace
Enable Poll SCM trigger
Build your modules/projects via Execute shell build steps. (Failed builds can be handed to the job result by using Exit 1 on a Execute shell Build step.)
However keep in mind that this will prevent advanced functionality on a per project/module basis, such as mail notifications to the dev to blame. Or trend of metrics, like warnings or static code analysis.
The following solution is easier to extend in that direction.
Wrapper job around your various build jobs
Use Build step Trigger/call builds on other projects to build A, archive needed artifacts
Use Build step Trigger/call builds on other projects with some parameter tomcatto build B tomcat version, use Copy Artifact Plugin to copy over jar from A
...
Use Build step Trigger/call builds on other projects with some parameter tomcatto build E tomcat version. Use Copy Artifact Plugin to copy all needed artifacts, you can specify parameter there if you need artifact of i.e. B tomcat version
In this setup, monitoring the svn is an issue since if you trigger it from polling SCM, it will checkout it in your wrapper workspace while you dont actually need it checked out there, but in your build jobs.
Possible solution: Share the workspace between wrapper job and your build jobs, so the duplicate checkouts in the build jobs will find the files already in the right revision. However then you *need+ to make sure the downstream jobs are executed on the same machine (there are plugins to do so)
Or even more elegant: Use a post-commit hook (See here, section Post-comit hook) on your svn to notify jenkins of changes.
Edit: For the future, its worth looking into the Pipeline Plugin and its documentation for more complex builds, this is the engine for the upcoming jenkins version 2.0, see here.
I would create 5 different jobs for ABCDE.
As you mentioned A and C would be standalone jobs so I would just do mvn clean install/pkg/verify based on your need.
For B I would first build A and then invoke another maven target in build to build B
For D, I would first build C and then build D
Finally for E , i would use invoke top level mvn targets 5 times A , B,C,D and finally E
Edit:
Jenkins 2 is out and has a built-in support for pipelines.
A few pointers for your requirements:
"built the whole projects if there is any change in SCM"
Although Poll SCM usually requires less work, the proper way to do it is to use SVN hooks.
The solution works as follows:
First you enable the Trigger builds remotely feature and enter a random token in Authentication Token.
This allows you to trigger builds remotely using Jenkins REST API (http[s]://JENKINS_URL/job/BUILD_NAME/build?token=TOKEN)
Then you create a SVN hook (a script that runs whenever you commit) which triggers the build by sending a request to that URL (using curl,wget, python,...).
There are a lot of manuals on how to create SVN hooks, here's the first result on "SVN Hooks" from Google.
"keep the dependency between projects"
I would create a different Jenkins Job for each project separately, then make sure builds are executed in the required order.
I think the best way to order your builds (dependencies) is to create a Build Pipeline using the Pipeline Plugin (previously known as Workflow Plugin).
There is a lot to explain here, so it's better you read on your own. You can start here.
There are also other (simpler) solutions, like Build Flow Plugin or Parameterized Trigger Plugin which can help create dependencies between builds, but I think Pipeline is the newest and considered a best practice (it's definitely the most advanced solution).
Still, having said that, if you feel Pipeline is an overkill for you, go for the alternatives.
I would recommend making sure each build does a mvn install to the same local repo, and also deploys the artifact to Artifactory (hopefully you have one).
Automate parameterized builds: "build with tomcat profile and websphere profile"
To do that you'll need to create parameterized builds.
That's pretty easy to do, you just check This build is parameterized in your build config and add a MVN_PROFILE string/choice parameter.
After that you can trigger each build several times, with different parameters, using any one of the plugins mentioned in the previous bullet.
Extra Tip:
While hacking your way through this, consider using Job Configuration History Plugin, it can help review and revert changes made to the configuration.
Good luck, hope this helps :)
I would consider a bit different approach to fully de-couple the projects.
If you are able to create your internal artifactory, than I would consider in the maven build each on of the dependencies as a third party library exactly like it is done with any other external libraries you are using.
This way, each such project can be seperatly built and stored in the artifactory and when a dependent project will be built it will just take the right version as mentioned in the pom file.
This way you'll have different build process for each one of the projects and only relevant projects (relevant = changed) will be built.
I have three independent projects A, B and C. Projects B and C depend on project A jar. Every time there is a change in project A, i have to manually build it before building B and C.
I am trying to figure out, if there is a way to Build project A; every time i build project B or C independently. As these are independent projects, i don't think i have to use modules here.
If you need to rebuild B and C every time you introduce changes to A, I'd say, this is not a simple dependency between otherwise independent projects, and the coupling here is tighter than it seems.
You seem to know what maven modules are, and I would suggest to go that way.
Wrap all three of your projects with a ROOT maven project, and add them as modules to the root pom.xml.
Otherwise, think about Continuous Integration Server, like Jenkins or Hudson. In there, you can easily relate separate builds to trigger other builds, and also set them up, so they start automatically on every change comitted to version control system like SVN or GIT.
A scenario here would be quite simple:
SVN change => triggers build of A => triggers build of B, C
I have a Maven 3 multi-module project (~50 modules) which is stored in Git. Multiple developers are working on this code and building it, and we also have automated build machines that run cold builds on every push.
Most individual changelogs alter code in a fairly small number of modules, so it's a waste of time to rebuild the entire source tree with every change. However, I still want the final result of running the parent project build to be the same as if it had built the entire codebase. And I don't want to start manually versioning modules, as this would become a nightmare of criss-crossing version updates.
What I would like to do is add a plugin which intercepts some step in build or install, and takes a hash of the module contents (ideally pulled from Git), then looks in a shared binary repository for an artifact stored under that hash. If one is found, it uses that artifact and doesn't even execute the full build. If it finds nothing in the cache it performs the build as normal, then stores its artifact in the cache. It would also be good to rebuild any modules which have dependencies (direct or transient) which themselves had a cache miss.
Is there anything out there which does anything like this already? If not, what would be the cleanest way to go about adding it to Maven? It seems like plugins might be able to accomplish it, but for a couple pieces I'm having trouble finding the right way to attach to Maven. Specifically:
How can you intercept the "install" goal to check the cache, and only invoke the module's 'native' install goal on a cache miss?
How should a plugin pass state from one module to another regarding which cache misses have occurred in order to force rebuilds of dependencies with changes?
I'm also open to completely different ways to achieve the same end result (fewer redundant builds) although the more drastic the solution the less value it has for me in the near term.
I have previously implemented a more complicated solution with artifact version manipulation and deployment to private Maven repository. However, I think this will fit your needs better and is somewhat more simple:
Split your build into multiple builds (e.g., with a single build per module using maven -pl argument).
Setup parent-child relationships between these builds. (Bamboo even has additional support for figuring out Maven dependencies, but I'm not sure how it works.)
Configure Maven settings.xml to use a different local repository location - specify a new directory inside your build working directory. See docs: https://maven.apache.org/guides/mini/guide-configuring-maven.html
Use mvn install goal to ensure newly built artifacts are added to local repository
Use Bamboo artifact sharing to expose built artifacts from local repository - you should probably filter this to include only the package(s) you're interested in
Set dependent builds to download all artifacts from parent builds and put them into proper subdirectory of local repository (which is customized to be in working directory)
This should even work for feature branch builds thanks to the way Bamboo handles parent-child relations for branch builds.
Note that this implies that Maven will redownload all other dependencies, so you should use a proxy private Maven repository on local network, such as Artifactory or Nexus.
If you want, I can also describe the more complicated scenario I've already implemented that involves modifying artifact versions and deploying to private Maven repository.
The Jenkins plugin allows you to manage/minimize dependent builds
whenever a SNAPSHOT dependency is built (determined by Maven)
after other projects are built (manually via Jenkins jobs)
And if you do a 'mvn deploy' to save the build into your corporate Maven repo then you don't have to worry about dependencies when builds run on slave Jenkins machines. The result is that no module is ever built unless it or one of its dependencies has changed.
Hopefully you can apply these principles to a solution with Bamboo.
The problem is, in our company we have a project with multiple sub-modules, however one of the sub-modules is just a collection of API declarations and is meant for other (3rd praty) projects to use. I want to keep it as a sub-module because is easier to maintain and build (dependency and property inheritance). Other sub-modules in this project are also dependant on it.
The question I have is, if there exist a good practice or a nice way to execute a deploy phase that will upload just this sub-module to a different repository (can be duplicated too) without it having a dependency to parent pom.
What I have already tried:
I have already checked the deploy:deploy-file, but the problem is when it comes to SNAPSHOT builds. We wish to be able to publish SNAPSHOTS and release builds, and snapshots have different repository than release ones, but deploy-file goal can only have one url parameter. I do not wish to use different profile for snapshot deploy. Than I tried to use maven build-helper and its regex-property to be able to change the repository url if the version is a SNAPSHOT, but was unable to do so because of the plugin and regex limitations.
The last option is I can make a plugin for this, but I wish to know if there is a more elegant way to solve this the "maven way".
You can deploy this module separately but only for SNAPSHOT's for a release it does not make sense. The deployment of a module can be done via:
mvn -pl TheModuleYouWouldLikeToDeploy deploy
may be you need to add the option -am (also make dependencies) like:
mvn -am -pl TheModuleYouWouldLikeToDeploy deploy
Apart from that your approach sounds wrong cause if you are using a multi-module build why not deploying the whole build via mvn deploy ? May be it would be better to let do the job via a CI tool like Jenkins.