Given a list of repository URLs, a group, a name, and a version number, is there a simple way to resolve the download URLs for the artifact and all of its dependencies including transitive dependencies from an external application? I'm looking to use these URLs in an launcher applicatiom to download artifacts from the official sources, decreasing bandwidth on our less-than-ideal servers in the process.
I've tried using Shrinkwrap, however that just returns a list of artifacts, not what repository it fetched from. I've also tried using aether however it appears you cannot disable local resolution, which becomes an issue when building a remote URL.
Update: I've looked a bit more into aether today, and while I do think it's possible to get to work, it feels like a lot of effort for something that I feel should be relatively simple, considering gradle and maven do it.
Related
I manage a large propriety system that's compromised of about a dozen services in java. We have a core set of java libs that these all share ), and all the components/apps are built using maven. Outside of the core SDK jars though each app has its own unique set of dependencies. I can't figure out what the best approach is to both building and deploying inside docker. Ideally I want the entire lifecycle in docker, using a multi-stage build approach. But, I can't see how to optimize this with the huge number of dependencies.
It looks like I can do 2 approaches.
Build as we have before, using maven and a common cache on the CI server (jenkins) so that dependencies are fetched once and cached, and accessible to all the apps. Then have a dockerfile for each app that just copies the product jar and it's dependencies (or a fat jar) into the container, and set it up to execute. Downside of this approach is that the build itself is something that could differ between developers and the CI server. Potentially use a local maven cache like nexus just to avoid pulling deps from the internet everytime? But that still doesn't solve the problem that a dev build won't necessarily match the CI build environment.
Use multi-stage dockerfile for each project. I've tried this, and it does work and I managed to get the maven dependencies layer to cache so that it doesn't fetch too often. Unfortunately that intermediate build layer was hitting 1-2gb per application, and I can't remove the 'dangling' intermediates from the daemon or all the caching is blowing away. It also means there's a tremendous amount of duplication in the jars that have to be downloaded for each application if something changes in the poms. (ie they all use junit and log4j and many other similarities)
Is there a way to solve this optimally that I'm not seeing? All the blogs I've found basically focus on the 2 approaches above (with some that focus on running maven itself in a container, which really doesn't solve anything for me). I'll probably need to end up going with Option 1 if there aren't any other good solutions.
I've checked around on stackoverflow and blogs, and everything I can find seems to assume that you're really just building a single app and not a suite of them, where it becomes important to not repeat the dependency downloads.
I think it is OK to use the .m2/repository filesystem cache as long as you set the --update-snapshots option in your maven build. It scales better, because you cache each .jar only once per build environment and not once per application. Additionally a change in a single dependency does not invalidate the entire cache, which would be the case if you use docker-layer-caching.
Unfortunately that cannot be combined well with multi-stage builds at the moment, but you are not the only one asking for it.
This issue requests adding a --volume option to the docker build command. This one asks for allowing instructions like this in the Dockerfile: RUN --mount=m2repo=/var/mvn/repo mvn install.
Both features would allow you to use the local maven filesystem cache during your multistage build.
For the moment I would advise to keep your option 1 as solution, unless you are facing many issues which are due to differing build environments.
I have many dependencies in my project, almost all of which can be fetched from a remote repository. However, one of the repositories used contains a dependency that I build a custom version of locally using provided tools for that (hence why I'm not just building it to a custom location, figuring that out is a last resort).
I would like to fetch all my dependencies from remote repositories, except for the one I am building locally. I cannot remove the repository that contains it, as it contains more of my dependencies that I DO want to update.
In addition, this is a project shared with others, so a system scope and a systemPath attribute aren't going to work.
I cannot run maven in offline mode, as I do want to update every other dependency regularly.
Thanks.
Let me start with one suggestion, do not try to override existing coordinates with your custom logic and hence the build. It will always add to the confusion and worst runtime exception will make it hard to debug the problem. You should
Change the coordinates of this custom build project - Best possible solution. This will be treated as any other dependency.
If you can't change the coordinates, change the version. Do add a qualifier to make it future proof for further release of this library from original source.
Always prefer to add this dependency in dependency management section. This will always give this library a preference over dependency mediation.
You could use relocation mechanism of distribution management. This will give a useful message to people. There always is a shared settings.xml across a company.
I am trying to set up a local maven repository using nexus to store ALL of the maven dependency libraries to be use in an offline environment. We need to be able to point our maven to this repository instead of trying to query the internet for a needed project library. There is no option of using the internet in our environment so I need an easy(ish) way to download literally every library maven could need in order to populate our repository. I am not asking for an opinion on what is the easiest, I just need a few methodologies that would work because I have no idea where to start with this other than spending the next year clicking links and downloading. I have found methods online that explain how to store project specific dependencies but I can't find a way to get ALL of the dependencies. I also know these libraries are continuous changing and be upgraded, so any ideas on how to keep them up to date without individually clicking and downloading each one would be great as well. Thanks!
MavenCentral is huge. Downloading it not only takes ages, they will block you if you try.
I understand very well that you cannot allow your developers to access MavenCentral during their development process. Our solution for this problem is as follows:
We have a proxy of MavenCentral that is only accessible to a few admins. If a developer needs a dependency that is not on our development Nexus, he writes a ticket to the admins. They check the dependency and if they approve, they start a job that copies the required dependency (together with all transitive dependencies) from the MavenCentral proxy to the development Nexus. This way, the admins have full control over what the developers use, but you need not copy MavenCentral as a whole.
I haven't downloaded spring in a while. I'm investigating gradle, but don't seem to have the hang of it yet, so i want to create a new spring framework project using java, Spring 4 and ant. I can't seem to find the place to download the binaries for Spring 4.
Are they just kicking ant developers under the bus?
Are they just kicking ant developers under the bus?
No. As Brian mentions, distribution zips are still available at http://repo.spring.io for those who have no other option. Ant users are encouraged to integrate Ivy into their build scripts to manage dependencies so that they will not need these dist zips. Ivy is capable of working against Maven-compatible artifact repositories to provide the same transitive dependency management benefits as Maven and Gradle do. Ant is a perfectly fine build solution for many folks, and we expect it will continue to be for some time. However, manually managing dependencies, i.e. downloading dist zips, storing jars on a network drive or checking them into source control is widely understood in the industry as a problematic approach.
We believe that the majority of Spring users are already using transitive dependency management solutions in one form or another. We continue to provide dist zips for those who have not yet been able to adopt this practice, but to be clear, it is intentional that we have not given those dist zips the first-class treatment on spring.io that they once had on springsource.org, because working with dist zips is simply an inferior way to manage an application's dependencies.
Spring is about helping application development teams eliminate unnecessary complexity. There are few things that can make developing an application more complex and frustrating than the "jar hell" that ensues from manual dependency management. Here are but a few examples of why this can be so painful:
Needing to be connected to the network drive where jars are stored (and thus, very often needing to be connected to the corporate VPN);
Or, in the case of jars being checked into the repository, having massive, unwieldy repositories to manage, typically resulting in many duplications of the same jars across different repositories;
Having no simple, universal and reliable way to (a) know what a dependency's version is and more importantly, (b) know whether that version of that dependency is compatible with all of the other jars in the application's dependency graph.
Maven, Gradle and Ivy are not a silver bullet for all dependency management issues, and naturally they come with their own complexity and learning curve. However, when given a choice, the vast majority of modern Java application developers agree that the benefits of using transitive dependency management outweigh their costs.
We hope that we've struck the right balance in our approach to guiding users how to consume Spring artifacts. We've shone the spotlight on what we (and most folks) consider to be best practices with dependency management by advertising Maven and Gradle syntax, but we've left the door to all comers by by continuing to publish distribution zips. We are however paying attention to feedback to make sure that this approach is in fact suitable for the majority of our users.
For additional information on this topic, see https://github.com/spring-projects/spring-framework/wiki/Downloading-Spring-artifacts.
And as a final note, we sometimes hear from folks that they need distribution zips because their company disallows access to public Maven repositories such as Maven Central (http://search.maven.org) or the Spring Repository (http://repo.spring.io). This is completely understandable, but the appropriate response to these constraints is not to keep development teams in the unproductive dark ages of manual dependency management. The correct solution is to stand up a private artifact repository within the corporate firewall. The leading contenders in this product space are JFrog's Artifactory and Sonatype's Nexus. We strongly recommend that any development team still forced into manual dependency management lobby their architecture teams to look into these products and adopt one of them. The benefits to productivity, build reproducibility, and indeed the ability for companies to effectively govern dependencies are dramatic.
Maven and Gradle are the preferred ways to use Spring Framework in your project, but dist ZIPs are still available on the official repository.
You can use the Maven config, go to any Maven repo, and download the JARs from there. For instance:
<dependencies>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>4.0.0.RELEASE</version>
</dependency>
</dependencies>
Tells me to look in /org/springframework/spring-context/4.0.0.RELEASE folder under any Maven 2 Repository. Using the example, I found this. You should be able to get all the JARs from there.
You can find the dependency information for Spring 4 here
From there you can download the binary jar and the jar with sources for manual dependency management.
I found two sources: the official download page here and OLEX.
This was after wasting lots of time trying to find the zip files via searches on the Spring site. There are no longer any direct download links there.
I have a system consisting of multiple web applications (war) and libraries (jar). All of them are using maven and are under my control (source code, built artifacts in Nexus,...). Let say that application A is using library L1 directly and L2 indirectly (it is used from L1). I can easily check the dependency tree top-down from the application, using maven's dependency:tree or graph:project plugins. But how can I check, who's using my library? From my example, I want to know, whether A is the only application (or library) using L1 and that L2 is used from L1 and from some other application, let say B. Is there any plugin for maven or nexus or should I try to write some script for that? What are your suggestions?
If you wish to achieve this on a repository level, Apache Archiva has a "used by" feature listed under project information
.
This is similar to what mvnrepository.com lists under its "used by" section of an artifact description.
Unfortunately, Nexus does not seem to provide an equivalent feature.
Now I suppose it would be a hassle to maintain yet another repository just for that, but then it would probably easier than what some other answers suggestions, such as writing a plugin to Nexus. I believe Archiva can be configured to proxy other repositories.
Update
In fact, there's also a plugin for Nexus to achieve the "used by" feature.
As far as I know nothing along these lines exists as an open source tool. You could write a Nexus plugin that traverses a repo and checks for usages of your component in all other components by iterating through all the pom's and analyzing them. This would be a rather heavy task to run though since it would have to look at all components and parse all the poms.
In a similar fashion you could do it on a local repository with some other tool. However it probably makes more sense to parse the contents of a repo manager rather than a local repository.
I don't think there's a Maven way to do this. That being said, there are ways of doing this or similar things. Here's a handful examples:
Open up your projects in your favorite IDE. For instance Eclipse will help you with impact analysis on a class level, which most of the time might be good enough
Use a simple "grep" on your source directory. This sounds a bit brusk (as well as stating the obvious), perhaps, but we've used this a lot
Use dependency analysis tools such as Sonargraph or Lattix
I am not aware of any public libraries for this job, so I wrote a customized app which does it for me.
I work with a distribution which involves more than 70 artifacts bundled together. Many times after modifying an artifact, I want to ensure changes are backward compatible (i.e. no compilation errors are introduced in dependent artifacts). To achieve this, it was crucial to know all dependents of modified artifact.
Hence, I wrote an app which scans through all artifacts under a directory(/subdirectories), extracts their pom.xml and searches (in dependency section of pom) for occurrence of modified artifact.
(I did this in java although shell/windows script can do this even more compactly.)
I'll be happy to share code on github, if that could be of any help.
One way that might suit your needs are to create a master-pom with all your maven projects. Then you run the following command on the master-pom:
mvn dependency:tree -DoutputType=graphml -DoutputFile=dependency.graphml
Open the generated file in yEd.
Used the instructions found here:
http://www.summa-tech.com/blog/2011/04/12/a-visual-maven-dependency-tree-view/
More interesting is probably: what would you do with this information? Inform the developers of A not to use library L1 or L2 anymore, because it has a critical bug?
In my opinion you should be able to create a blacklist of dependencies/parents/plugins on your repository manager. Once a project tries to deploy/upload itself with a blacklisted artifact, it should fail. I'm saying uploading and not downloading, because that might break a lot of projects. As far as I know, this is not yet available for any repository-manager.
One of the ways to approach this problem is outside Java itself : write an OS-level monitoring script that tracks each case of fopen() on the jar file under question! Assuming this is in a corporate environemnt, you might have to wait for a few weeks (!) to allow all using processes to access the library at least once!
On Windows, you might use Sysinternals Process Monitor to do this:
http://technet.microsoft.com/en-us/sysinternals/bb896645
On Unix variants, you would use DTrace or strace.
IMHO and also from my experience, looking for a technical solution for such a problem is often an overkill. If the reason why you want to know who is using your artifact(library) is because you want to ensure backward compatibility when you change an artifact or something similar, I think it is best done by communicating your changes using traditional channels and also encourage other teams who might be using your library to talk about it (project blogs, wiki, email, a well known location where documentations are put, Jour fixe etc.).
In theory, you could write a script that crawls though each project in your repository and then parses the maven build.xml (assuming they all use maven) and see whether they have defined a dependency to your artifact. If all the projects in your organization follows the standard maven structure, it should be easy to write one such script (though if any of those projects have a dependency to your artifact via a transitive dependency, things can get a bit more tricky).