We currently have a big Maven 2 project that is rather a collection of many single standalone projects with complicated dependencies, with the exception of some common parent POMs for building. In the end we always have to ship the application as one piece, so I would rather like to convert it to one or a few big projects.
Does anyone have experience in how to optimize continuous integration builds for big projects. Is the incremental build functionality of Maven or Hudson any good? I would prefer not to wait always 2 hours when having made only a small change in one module.
On the other hand, to be sure, you always would have to rebuild and re-test at least all direct and indirect dependencies of a changed module. That is also what we are currently doing with Hudson, triggering automatically all dependent jobs.
Does a split up into multiple build jobs for the same project pay off? I generally do not like to have artifacts on the server where all the other generated stuff like reports, docs, etc. could possibly be out of date.
Thanks for any thoughts.
I just did some more testing and as I found out Maven does not really support incremental builds. Without any plugins Maven actually has a dangerous behavior. If you change the code in some module and compile without prior clean, dependent modules will not get rebuilt, meaning they then reference an old outdated version of the dependency and will not react to the updated code.
With the incremental build plugin it is possible to build without clean. Every module that changed gets rebuilt plus all dependents will be cleaned and rebuilt. However, in my case compiling uses only like 10% of the build time, 90% are for testing. And when I install/deploy all tests get executed again, so the time benefit from the incremental build plugin is very small.
So I still only see the option of splitting up builds in Hudson which is hardly ideal in my opinion.
I would highly suggest not to split up into different build jobs. This, in my experience, can get out of hand quickly with upstream and downstream dependencies. Incremental building works great for what you need it for. If the dependencies are set directly only artifacts that change and their dependencies are rebuilt.
I would split up the build jobs though if they are completely separate applications with no or very few dependnecies (if thats true then they shouldn't be under the same reactor and thus the incremental builds would not be possible)
Related
I've hand-rolled my project's build-system (mostly in Python + Hudson). One of the things I need to do quite often is fetch artifacts from upstream Hudson / Jenkins.
These artifacts could be almost anything - for example a zip-file full of business data to process or even an egg containing a load of python code which must be tested. Almost every important job in our system has upstream dependancies on artifacts produced by other Hudson jobs.
My manager has suggested that the next iteration of the build-system should replace some of my hand-rolled components with Ant. The purpose of this next iteration will be to reduce the complexity of our systems and bring them into line with the work of other teams who mainly use Java and Ant (and very little Python).
Also I'm personally keen to have an excuse to learn Ant. It seems like a really useful tool.
So in order not to re-invent the wheel one component I'm definitely going to need is an Ant task which can fetch an artifact from a particular Hudson build. Does such a thing exist. If it does not exist, is there something close to my requirement that I could customize? I'd rather not re-invent the wheel.
UPDATE1: We have a strong preference for 100% free, open-source tools. Everybody in the team is very happy with Ant, however Maven is something the team is trying to get away from.
The proper solution is to publish the artifacts from Hudson/Jenkins to an artifact manager, such as Nexus or Artifactory, and then pull the artifact versions with something like Ivy or Gradle.
If you must get the dependencies yourself, you could use the get task. Example:
<get src="http://jenkins/job/project-name/lastSuccessfulBuild/artifact/foo.jar"
dest="/path/to/local/file"/>
I do, however, agree with Stefan - dependency management is better accomplished by tools mentioned in his answer instead of manually pulling them down yourself using Ant.
If you are just starting to learn ANT then I would suggest you learn gradle instead, it has the dependency part already integrated and is far easier to work with. Furthermore if you need a specific ANT task you can simply call it from gradle or even import whole ant scripts.
Otherwise I agree with the answer from Stefan Kendall.
Background. My org uses Maven, Bamboo and Artifactory to support a continuous integration process. We rely on Maven's SNAPSHOT qualifier to help manage storage in Artifactory (rotate out old SNAPSHOT builds) and also to help keep cross-team integrations current (Maven checks for updates to SNAPSHOT dependencies automatically on each build).
Problem. One of the challenges we're having is around correctly promoting builds from environment to environment while continuing to use SNAPSHOT. Say that a tester deploys version 1.8.2-SNAPSHOT to a functional test environment, and it's at rev 1400 in Subversion. Let's say also that it passes functional test. By the time a tester decides to pull 1.8.2-SNAPSHOT from Artifactory into the performance testing environment, a developer could have committed a change to Subversion, so the actual binary in Artifactory is at a different rev. How do we ensure that the rev doesn't change out from under us when using SNAPSHOT builds?
Constraints. We obviously don't want to deploy different builds unknowingly. We also don't want to rebuild from source as we want to test the exact binary in performance test that we tested in functional test.
Approaches we've considered. The thought is that we want to stamp the versions with a fourth component, like 1.8.2.1400, where the fourth component is a Subversion rev. (As a side question, is there a Maven plugin or something else that does that automatically?) But if we do that, then essentially we lose the SNAPSHOT feature since Maven and Artifactory think that these are different versions.
We are using Scrum, so we deploy to the test environments very early (like day two or so). I don't think it makes sense to remove the SNAPSHOT qualifier that early in the dev cycle because we lose the SNAPSHOT benefits again.
Would appreciate knowing how other orgs solve this issue.
Just to circle back on this one, I wanted to share what we are doing.
Basically we deploy snapshot builds like 1.8.2-SNAPSHOT into the development environment. No other teams need to use these builds, so it is fine to leave -SNAPSHOT on them.
But any build that we deploy to a test environment (e.g. functional test, system test) or else production must include the revision; e.g., 1.8.2.1400. We call these "quads". The reason for insisting upon quads in test is that we can attach issues (features, bugfixes, etc.) to specific revisions so the testers know what to test. For production it's really just because we want to deploy exactly the same artifact that we tested, so that means we're deploying a quad.
Anyway hope that information is useful to somebody.
if you enable "uniqueVersion" for you snapshot builds, every snapshot deployed will have a unique id. you can use that to ensure you are deploying the correctly promote builds across environments.
and, as a side note, you can use the buildnumber-maven-plugin to add subversion buildnumbers to artifacts.
Rather than embed the build number of VCS revision in the artifact's version, we embed the CI build number in the META-INF/MANIFEST-MF file .
See for instance Using Hudson environment variables to identify your builds . Although the article is applicable to Jenkins/Hudson I believe it is trivial to port to Bamboo.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
What are the main benefits of using maven compared to let's say ant ?
It seems to be more of a annoyance than a helpful tool.
I use maven 2, with plain Eclipse Java EE (no m2eclipse), and tomcat.
Supporters of maven believe that
Maven lets you get your package dependencies easily
Maven forces you to have a standard directory structure
In my experience
Figuring out package dependencies is really not that hard. You rarely do it anyway. Probably once during project setup and few more during upgrades. With maven you'll end up fixing mismatched dependencies, badly written poms, and doing package exclusions anyway.
Slow FIX-COMPILE-DEPLOY-DEBUG cycle, which kills productivity. This is my main gripe. You make a change, the you have to wait for maven build to kick in and wait for it to deploy. No hot deployment whatsoever.
Or am I just doing it wrong ? Please point me to the right direction, I'm all ears.
Figuring out package dependencies is really not that hard. You rarely do it anyway. Probably once during project setup and few more during upgrades. With maven you'll end up fixing mismatched dependencies, badly written poms, and doing package exclusions anyway.
Not that hard... for toy projects. But the projects I work on have many, really many, of them, and I'm very glad to get them transitively, to have a standardized naming scheme for them. Managing all this manually by hand would be a nightmare.
And yes, sometimes you have to work on the convergence of dependencies. But think about it twice, this is not inherent to Maven, this is inherent to any system using dependencies (and I am talking about Java dependencies in general here).
So with Ant, you have to do the same work except that you have to do everything manually: grabbing some version of project A and its dependencies, grabbing some version of project B and its dependencies, figuring out yourself what exact versions they use, checking that they don't overlap, checking that they are not incompatible, etc. Welcome to hell.
On the other hand, Maven supports dependency management and will retrieve them transitively for me and gives me the tooling I need to manage the complexity inherent to dependency management: I can analyze a dependency tree, control the versions used in transitive dependencies, exclude some of them if required, control the converge across modules, etc. There is no magic. But at least you have support.
And don't forget that dependency management is only a small part of what Maven offers, there is much more (not even mentioning the other tools that integrates nicely with Maven, e.g. Sonar).
Slow FIX-COMPILE-DEPLOY-DEBUG cycle, which kills productivity. This is my main gripe. You make a change, the you have to wait for maven build to kick in and wait for it to deploy. No hot deployment whatsoever.
First, why do you use Maven like this? I don't. I use my IDE to write tests, code until they pass, refactor, deploy, hot deploy and run a local Maven build when I'm done, before to commit, to make sure I will not break the continuous build.
Second, I'm not sure using Ant would make things much better. And to my experience, modular Maven builds using binary dependencies gives me faster build time than typical monolithic Ant builds. Anyway, have a look at Maven Shell for a ready to (re)use Maven environment (which is awesome by the way).
So at end, and I'm sorry to say so, it's not really Maven that is killing your productivity, it's you misusing your tools. And if you're not happy with it, well, what can I say, don't use it. Personally, I'm using Maven since 2003 and I never looked back.
Maven can be considered as complete project development tool not just build tool like Ant.
You should use Eclipse IDE with maven plugin to fix all your problems.
Here are few advantages of Maven, quoted from the Benefits of using Maven page:
Henning
quick project setup, no complicated build.xml files, just a POM and go
all developers in a project use the same jar dependencies due to
centralized POM.
getting a number of reports and metrics for a project "for free"
reduce the size of source distributions, because jars can be
pulled from a central location
Emmanuel Venisse
a lot of goals are available so it isn't necessary to develop some
specific build process part contrary
to ANT we can reuse existing ANT tasks
in build process with antrun plugin
Jesse Mcconnell
Promotes modular design of code. by making it simple to manage mulitple
projects it allows the design to be
laid out into muliple logical parts,
weaving these parts together through
the use of dependency tracking in pom
files.
Enforces modular design of code. it is easy to pay lipservice to modular
code, but when the code is in seperate
compiling projects it is impossible to
cross pollinate references between
modules of code unless you
specifically allow for it in your
dependency management... there is no
'I'll just do this now and fix it
later' implementations.
Dependency Management is clearly declared. with the dependency
management mechanism you have to try
to screw up your jar
versioning...there is none of the
classic problem of 'which version of
this vendor jar is this?' And setting
it up on an existing project rips the
top off of the existing mess if it
exists when you are forced to make
'unknown' versions in your repository
to get things up and running...that or
lie to yourself that you know the
actual version of ABC.jar.
strong typed life cycle there is a strong defined lifecycle that a
software system goes thru from the
initiation of a build to the end...
and the users are allowed to mix and
match their system to the lifecycle
instead of cobble together their own
lifecycle.. this has the additional
benefit of allowing people to move
from one project to another and speak
using the same vocabulary in terms of
software building
Vincent Massol
Greater momentum: Ant is now legacy and not moving fast ahead. Maven is
forging ahead fast and there's a
potential of having lots of high-value
tools around Maven (CI, Dashboard
project, IDE integration, etc).
Figuring out dependencies for small projects is not hard. But once you start dealing with a dependency tree with hundreds of dependencies, things can easily get out of hand. (I'm speaking from experience here ...)
The other point is that if you use an IDE with incremental compilation and Maven support (like Eclipse + m2eclipse), then you should be able to set up edit/compile/hot deploy and test.
I personally don't do this because I've come to distrust this mode of development due to bad experiences in the past (pre Maven). Perhaps someone can comment on whether this actually works with Eclipse + m2eclipse.
Maven is one of the tools where you need to actually decide up front that you like it and want to use it, since you will spend quite some time learning it, and having made said decision once and for all will allow you to skip all kinds of doubt while learning (because you like it and want to use it)!
The strong conventions help in many places - like Hudson that can do wonders with Maven projects - but it may be hard to see initially.
edit: As of 2016 Maven is the only Java build tool where all three major IDEs can use the sources out of the box. In other words, using maven makes your build IDE-agnostic. This allows for e.g. using Netbeans profiling even if you normally work In eclipse
Maven advantages over ant are quite a few. I try to summarize them here.
Convention over Configuration
Maven uses a distinctive approach for the project layout and startup, that makes easy to just jump in a project. Usually it only takes the checkount and the maven command to get the artifacts of the project.
Project Modularization
Project conventions suggest (or better, force) the developer to modularize the project. Instead of a monolithic project you are often forced to divide your project in smaller sub components, which make it easier debug and manage the overall project structure
Dependency Management and Project Lifecycle
Overall, with a good SCM configuration and an internal repository, the dependency management is quite easy, and you are again forced to think in terms of Project Lifecycle - component versions, release management and so on. A little more complex than the ant something, but again, an improvement in quality of the project.
What is wrong with maven?
Maven is not easy. The build cycle (what gets done and when) is not so clear within the POM. Also, some issue arise with the quality of components and missing dependencies in public repositories.
The best approach (to me) is to have an internal repository for caching (and keeping) dependencies around, and to apply to release management of components. For projects bigger than the sample projects in a book, you will thank maven before or after
Maven can provide benefits for your build process by employing standard conventions and practices to accelerate your development cycle while at the same time helping you achieve a higher rate of success. For a more detailed look at how Maven can help you with your development process please refer to The Benefits of Using Maven.
Maven is a powerful project management tool that is based on POM (project object model). It is used for projects build, dependency and documentation.
It simplifies the build process like ANT. But it is too much advanced than ANT.
Maven helps to manage-
Builds,Documentation,Reporing,SCMs,Releases,Distribution.
- maven repository is a directory of packaged JAR file with pom.xml file. Maven searches for dependencies in the repositories.
I've never come across point 2? Can you explain why you think this affects deployment in any way. If anything maven allows you to structure your projects in a modularised way that actually allows hot fixes for bugs in a particular tier, and allows independent development of an API from the remainder of the project for example.
It is possible that you are trying to cram everything into a single module, in which case the problem isn't really maven at all, but the way you are using it.
This should have been a comment, but it wasn't fitting in a comment length, so I posted it as an answer.
All the benefits mentioned in other answers are achievable by simpler means than using maven. If, for-example, you are new to a project, you'll anyway spend more time creating project architecture, joining components, coding than downloading jars and copying them to lib folder. If you are experienced in your domain, then you already know how to start off the project with what libraries. I don't see any benefit of using maven, especially when it poses a lot of problems while automatically doing the "dependency management".
I only have intermediate level knowledge of maven, but I tell you, I have done large projects(like ERPs) without using maven.
I have recently joined a project that is using multiple different projects. A lot of these projects are depending on each other, using JAR files of the other project included in a library, so anytime you change one project, you have to then know which other projest use it and update them too. I would like to make this much easier, and was thinking about merging all this java code into one project in seperate packages. Is it possible to do this and then deploy only some of the packages in a jar. I would like to not deploy only part of it but have been sassked if this is possible.
Is there a better way to handle this?
Approach 1: Using Hudson
If you use a continuous integration server like Hudson, then you can configure upstream/downstream projects (see Terminology).
A project can have one or several downstream projcets. The downstream projects are added to the build queue if the current project is built successfully. It is possible to setup that it should add the downstream project to the call queue even if the current project is unstable (default is off).
What this means is, if someone checks in some code into one project, at least you would get early warning if it broke other builds.
Approach 2: Using Maven
If the projects are not too complex, then perhaps you could create a main project, and make these sub-projects child modules of this project. However, mangling a project into a form that Maven likes can be quite tricky.
If you use Eclipse (or any decent IDE) you can just make one project depend on another, and supply that configuration aspect in your SVN, and assume checkouts in your build scripts.
Note that if one project depends on a certain version of another project, the Jar file is a far simpler way to manage this. A major refactoring could immediately means lots of work in all the other projects to fix things, whereas you could just drop the new jar in to each project as required and do the migration work then.
I guess it probably all depends on the specific project, but I think I would keep all the projects separate. This help keep the whole system loosely coupled. You can use a tool such as maven to help manage all the dependencies between the projects. Managing dependencies like this is one of maven's main strengths.
Using Ant as your build tool, you can package your project any way that you want. However, leaving parts of your code out of the distribution seems like it would be error prone; you might accidentally leave out necessary classes (presumably, all of your classes are necessary).
In relation to keeping your code in different projects, I have a loose guideline. Keep the code that changes together in the same project and package it in its own jar file. This works best when some of your code can be broken out into utility libraries that change less frequently than your main application.
For example, you might have an application where you've generated web service client classes from a web service WSDL (using something like the Axis library). The web service interface will likely change infrequently, so you don't want to have the regeneration step reoccurring all the time in your main application build. Create a separate project for this piece so that you only have to recreate the web service client classes when the WSDL changes. Create a separate jar and use it in your main application. This style also allows other projects to reuse these utility modules.
When following this style, you should place a version number in the jar manifest so that you can keep track of which applications are using which versions of your module. Depending on how far you want to take this, you could also keep a text file in the jar that details the changes that have occurred for each revision (much like an open source library).
It's all possible (we had the same situation some years ago). How hard or easy it'll be depends on your IDE (refactoring, merging, organizing new project) and you build tool (deploying). We used IDEA as IDE and Ant as build tool and it wasn't too hard. One sunday (nobody working+committing), 2 people on one computer.
I'm not sure what you mean by
"deploy only some of the packages in a jar"
I think you will need all of them at runtime, won't you? As I understood they depend on each other.
I'm currently working on a project that contains many different Eclipse projects referencing each other to make up one large project. Is there a point where a developer should ask themselves if they should rethink the way their development project is structured?
NOTE: My project currently contains 25+ different Eclipse projects.
My general rule of thumb is I would create a new project for every reusable component. So for example if I have some isolated functionality that can be packaged say as a jar, I would create a new project so I can build,package and distribute the component independently.
Also, if there are certain projects that you do not need to make frequent changes to, you can build them only when required and keep them "closed" in eclipse to save time on indexing, etc. Even if you think that a certain component is not reusable, as long as it is separated from the rest of the code base in terms of logic/concerns you may be well served by just separating it out. Sometimes seemingly specific code might be reusable in another project or in a future version of the same project.
When compiled, a project would typically result in a jar. So if your application consists of potentially reusable components, it is ok to use a project for each.
I'm a big fan of using a lot of projects, I feel that this "breaks down" large things beyond what I can do with packages, and helps me orient and navigate.
Of course, if you're developing Eclipse plug-ins, everything would be a project anyway.
The only thing I would watch out for has to do with your source-control and it's ability to handle moves of files between projects. Subclipse had been giving me trouble with it, or maybe it's my SVN server that did.
If your project has that many sub-projects, or modules, needed to actually compose your final artifact then it is time to look at having something like Maven and setting up a multi-module project. It will a) allow you to work on each module independently without ide worries and allow easy setup in your ide (and others' IDEs) through the mvn eclipse:eclipse goal. In addition, when building your entire top level project, maven will be able to derive from list of dependencies you have described what modules need to be built in what order.
Here's a quick link via google and a link to the book Maven: The Definitive Guide, which will explain things in much better detail in chapter 6 (once you have the basics).
This will also force your project to not be explicitly tied to Eclipse. Being able to build independent from an ide means that any Joe Schmoe can come along and easily work with your code base using whatever tools he/she needs.
Create jars for the projects you don't work in often. That should greatly reduce the clutter. If you work on all the projects often, then you can add targets to your build that will jar up the respective projects for you, which condenses everything down to one file that you can then include on the class path.
An additional method is to create many different workspaces. The benefit of separate workspaces is that you can remove some of the visual clutter/ performance overhead of having lots of projects. You can use targets to jar up all of you projects and put them in a repository so you can reference them in each workspace.
At a former job the entire application was more then +170 projects. While it was rarely necessary to have all projects checked out locally, even the 30-40 projects constantly in our scope made reindexing, etc. very slow.
Yeesh. One Project for each Project. If you are using reusable projects, make them into a library for heavens sake. Break the none re-usable projects into packages, that's what they are there for.
That's a hard question and answers span from having one eclipse project at all to having one eclipse project for every single class.
My bottomline:
You can have too few projects,
and never too many (of course use
automation e.g. mvn eclipse:eclipse)
Use
-Declipse.useProjectReferences=true/false
when using maven to switch workspace
mode btw jar and project
dependencies
Use mvn release plugin to generate
consecutive releases (automatic
version increase)
Multiple projects gives you
independent versioning which is
extremely important. E.g. one dev may work on a new version of a
module while you still depends on
the previous one and you at some
point decide to upgrade to the newer
version(possibly by increasing its version in pom.xml dependency section). Or in other scenario if one
project contains a bug you downgrade
to its previous version.
Multiple projects makes you think
about the architecture more than if
you have just packages.
Multiple projects generally make
architectural problems evident more
than if you have just one project.
Anyone would like to comment on
this?
You never know if you project
evolves into OSGI/SOA/EDA where you
need separation.
Even if you're 100% sure that you
projects will be deployed as one jar
in an old way in a single jvm, it
still does not hurt(mvn assembly
plugin) to have multiple eclipse
projects for logically independent
pieces of code
BTW, the project I work on is divided into 24 eclipse projects.
Hell, we have more than 100. Projects don't cost anything.