When I distribute a source project with a POM I can define dependencies with version strings. The build will download those dependencies if they are not in the local repository, it will even verify the checksums of those downloaded files with the metadata from the same repository (-C).
However the build will download those dependencies from a number of public repositories (and proxies) and my users are at the mercy of those public services if they will return unmodified files.
I would like to have a way to record the checksums of all my build dependencies and ship them together with the POM (so I am sure the files are unaltered but I dont need to ship a copy of my local repository to builders).
Is there a Maven way to do this? Similiar to that, is there a easy way to archive my dependencies (copy of the local repository with all used artifacts and metadata files) so I can repeat my build even when the central repositories fail or ship them to offline customers?
(Both without the need for an repository proxy if possible. I know I can build something to do that, I just wonder if there is a infrastructure in maven for that already. Maybe shipping a local repository which does only contain the metadata files or similiar?)
NB: I am not looking for createChecksum on the generated artifacts, but on locking the checksums of the used dependencies. I found a maven-create-checksums plugin, but no corresponding verifyer.
There is BitcoinjEnforcerRules, which appears to be an plugin for Maven's enforcer plugin (that's right it's a plugin for a plugin). It works similar to gradle-witness, which means it compares the checksum of (hopefully) all used artifact dependencies to a list you have previously created.
While it's better then nothing, an ideal solution would be if one could simply set the checksum (or the openPG key used to sign the POM in case it's an snapshot artifact) for the POM of direct dependencies, and the POM of those dependencies would contain the checksum of it's artifacts, and the checksum (/openPG key, if snapshot) of the POM of its direct dependencies, and so on.
Note that if checksums are used, then they should be tied to a particular maven repository, since the same (non-changing, i.e. non-snapshot) artifact may have different checksums in different repositories.
Related
it seems to me there are 2 places where I might want to set the internal maven repo:
In maven's settings.xml inside the mirror tag;
In the project's pom file, inside repository and pluginRepository tags.
The question is, which one is right? or shall I put the internal repo in both places?
Thanks,
John
I don't recommend putting repository definitions in your pom. I have been bitten many times by a pom in my dependency list including repository definitions to repositories that I can't reach.
By putting these in settings.xml, you allow each developer the freedom to control which repositories are used when running a build. Since developers sometimes work disconnected or across a VPN, it can be desirable for this list of repositories to be different from machine to machine.
Remember also that the POM becomes immutable once a release is performed, effectively making the repository URL you defined permanent. Placing it in settings.xml allows your future team members the freedom to move the repository (or remove it).
In both :-(
You add the repo definition (server ID, URL and login credentials) to Maven's settings.xml and add a reference to that repo (by ID) inside your project's pom.xml. It's cumbersome, but it lets your credentials stay away from shared files.
Maven docs state:
The repositories for download and deployment are defined by the
repositories and distributionManagement elements of the POM. However,
certain settings such as username and password should not be
distributed along with the pom.xml. This type of information should
exist on the build server in the settings.xml.
I have a Maven 3 multi-module project (~50 modules) which is stored in Git. Multiple developers are working on this code and building it, and we also have automated build machines that run cold builds on every push.
Most individual changelogs alter code in a fairly small number of modules, so it's a waste of time to rebuild the entire source tree with every change. However, I still want the final result of running the parent project build to be the same as if it had built the entire codebase. And I don't want to start manually versioning modules, as this would become a nightmare of criss-crossing version updates.
What I would like to do is add a plugin which intercepts some step in build or install, and takes a hash of the module contents (ideally pulled from Git), then looks in a shared binary repository for an artifact stored under that hash. If one is found, it uses that artifact and doesn't even execute the full build. If it finds nothing in the cache it performs the build as normal, then stores its artifact in the cache. It would also be good to rebuild any modules which have dependencies (direct or transient) which themselves had a cache miss.
Is there anything out there which does anything like this already? If not, what would be the cleanest way to go about adding it to Maven? It seems like plugins might be able to accomplish it, but for a couple pieces I'm having trouble finding the right way to attach to Maven. Specifically:
How can you intercept the "install" goal to check the cache, and only invoke the module's 'native' install goal on a cache miss?
How should a plugin pass state from one module to another regarding which cache misses have occurred in order to force rebuilds of dependencies with changes?
I'm also open to completely different ways to achieve the same end result (fewer redundant builds) although the more drastic the solution the less value it has for me in the near term.
I have previously implemented a more complicated solution with artifact version manipulation and deployment to private Maven repository. However, I think this will fit your needs better and is somewhat more simple:
Split your build into multiple builds (e.g., with a single build per module using maven -pl argument).
Setup parent-child relationships between these builds. (Bamboo even has additional support for figuring out Maven dependencies, but I'm not sure how it works.)
Configure Maven settings.xml to use a different local repository location - specify a new directory inside your build working directory. See docs: https://maven.apache.org/guides/mini/guide-configuring-maven.html
Use mvn install goal to ensure newly built artifacts are added to local repository
Use Bamboo artifact sharing to expose built artifacts from local repository - you should probably filter this to include only the package(s) you're interested in
Set dependent builds to download all artifacts from parent builds and put them into proper subdirectory of local repository (which is customized to be in working directory)
This should even work for feature branch builds thanks to the way Bamboo handles parent-child relations for branch builds.
Note that this implies that Maven will redownload all other dependencies, so you should use a proxy private Maven repository on local network, such as Artifactory or Nexus.
If you want, I can also describe the more complicated scenario I've already implemented that involves modifying artifact versions and deploying to private Maven repository.
The Jenkins plugin allows you to manage/minimize dependent builds
whenever a SNAPSHOT dependency is built (determined by Maven)
after other projects are built (manually via Jenkins jobs)
And if you do a 'mvn deploy' to save the build into your corporate Maven repo then you don't have to worry about dependencies when builds run on slave Jenkins machines. The result is that no module is ever built unless it or one of its dependencies has changed.
Hopefully you can apply these principles to a solution with Bamboo.
I am working in a private network which doesn't have internet proxy.
I can not create a local repository as well which involves the bureaucracy, management won't allow it. I may be a long term fix but not the solution for the question I asked.
I can not keep maven as a build tool as it requires the direct or indirect internet connection.
I HAVE to use ANT for building the project hence using maven in offline mode also not an option for me.
But I still want to use the maven dependency management for collecting all the jars in a one archive smartly.
My plan is to generate a ZIP file containing dependencies resolved using maven. And then we will share this ZIP file to all developers working inside a private network which doesn't have internet connection.
To do so I will get a temporary access to a computer which is having internet connection and from there I will define a dummy POM with all the dependencies required.
Now the question is how do I generate a ZIP file ( not a single jar ) using maven which contains all the dependencies defined in POM.
while what youre suggesting is technically possible, it is (in my opinion) not the best solution to your problem.
your statement that
it requires the direct or indirect internet connection
is not accurate. what maven requires is a maven repository (or a set of them) to fetch stuff from. the best solution to your problem would be to install a local maven repository inside your organization's network. the 2 most popular choices for a loaclly-run maven repository seem to be nexus and artifactory - both offer free open source versions and paid supported pro versions.
once you set up a maven repository inside your organization's network and populate it with the artifacts you require you can simply configure all of your project's pom files to go to those repositories. for example, to configure maven to use your repo instead of maven central, you can do this:
<repositories>
<!-- override central -->
<repository>
<id>central</id>
<url>http://your.repo.location</url>
</repository>
</repositories>
you will need to map a plugin repository in a similar fashion.
its also possible to achieve this by configuring the maven settings.xml file in each user's home directory if you dont want this in the pom files but from my experience its less error-prone this way
use following command to build Maven project offline.
mvn -o package
Refer this and this for more information.
I ended up with a smart hack which lets me do dependency resolution and archiving!
I am creating a dummy maven web project with all the dependencies defined in pom xml.
Now the war packaging mode is used by default for web applications.
I simply install the maven project from internet facing machine.
I get all the dependencies and transitive dependencies in war file's "lib" directory with dependency naming version remaining unchanged !!!!
Copying and adding those files into an ANT project is a trivial task then..!
I am upgrading a project from Ant to Gradle. The project uses a aqapi13.jar(this is oracle aq jar. This is needed as the project reads from an oracle-queue and writes to an activemq queue.)
The ant project contains the jar aqapi13.jar in the libs folder. But iam trying to get this dependency from a repository instead of having it in the libs folder.
However, iam not able to find a repository which contains this jar. All the repositories that I have seen contain aqapi13-9i.jar, but not aqapi13.jar.
Anyone knows the difference between aqapi13.jar and aqapi13-9i.jar and how to get the needed aqapi13.jar from a repository.
Advance Thanks
Some dependencies are never found in public repositories because of the license they have.
One way to use this dependencies is to create your own repository (e.g. artifactory, nexus, archiva). Then you are free to put in every artifact you want (as long as you do not publish the repository). This repository can also serve as a mirror for maven.
Another way could be to mark this dependency as system scope.
How do I prevent Maven 2 from searching remote repositories for specific dependencies that are in the local repository only?
How do I prevent Maven 2 from searching remote repositories for specific depedencies that are in the local repository only
Well, actually, Maven won't unless:
they are SNAPSHOT dependencies in which case this is the expected behavior.
they are missing a .pom file in which case you can provide it or generate it (see questions below).
Related questions
How do I stop Maven 2.x from trying to retrieve non-existent pom.xml files for dependencies every build?
Maven install-file won’t generate pom.xml
set up nexus as a repository manager.
add addtional remote proxied repositories if necessary
add your local hosted repository (hosted on the nexus server)
define a group of repositories in the correct search sequence with your local repo's first.
change your builds to point at the nexus group url (use mirrorOf=* in your settings.xml)
run your build and let nexus manage the local vs remote dependency resolution
Use fixed version numbers in your POM for your remote dependencies or the local versions you want to fetch from the local repository.
Maven tries to be friendly and fetch the latest and greatest of whatever which has no version number specified.
For a quick fix to not be waiting for the internet to be downloaded each time you build you can use mvn -o to force an offline build, and then it will not lose time trying to fetch new versions.
The answer of #crowne is also very good advice, especially setting up your own nexus and making sure all remote repos are configured there so you will never have unpleasant surprises when a repo dissappears some day.
To prevent Maven from checking remote repositories at all, you can use the -o flag. Otherwise, Maven will check that any snapshot dependencies are up-to-date. You can use a dependency manager such as Nexus to get fine-grained control over dependency resolution. The repository section in your pom.xml or settings.xml file also has an updatePolicy element that allows you to configure how often Maven will check for updated dependencies.