I am trying to set up a local maven repository using nexus to store ALL of the maven dependency libraries to be use in an offline environment. We need to be able to point our maven to this repository instead of trying to query the internet for a needed project library. There is no option of using the internet in our environment so I need an easy(ish) way to download literally every library maven could need in order to populate our repository. I am not asking for an opinion on what is the easiest, I just need a few methodologies that would work because I have no idea where to start with this other than spending the next year clicking links and downloading. I have found methods online that explain how to store project specific dependencies but I can't find a way to get ALL of the dependencies. I also know these libraries are continuous changing and be upgraded, so any ideas on how to keep them up to date without individually clicking and downloading each one would be great as well. Thanks!
MavenCentral is huge. Downloading it not only takes ages, they will block you if you try.
I understand very well that you cannot allow your developers to access MavenCentral during their development process. Our solution for this problem is as follows:
We have a proxy of MavenCentral that is only accessible to a few admins. If a developer needs a dependency that is not on our development Nexus, he writes a ticket to the admins. They check the dependency and if they approve, they start a job that copies the required dependency (together with all transitive dependencies) from the MavenCentral proxy to the development Nexus. This way, the admins have full control over what the developers use, but you need not copy MavenCentral as a whole.
Related
Given a list of repository URLs, a group, a name, and a version number, is there a simple way to resolve the download URLs for the artifact and all of its dependencies including transitive dependencies from an external application? I'm looking to use these URLs in an launcher applicatiom to download artifacts from the official sources, decreasing bandwidth on our less-than-ideal servers in the process.
I've tried using Shrinkwrap, however that just returns a list of artifacts, not what repository it fetched from. I've also tried using aether however it appears you cannot disable local resolution, which becomes an issue when building a remote URL.
Update: I've looked a bit more into aether today, and while I do think it's possible to get to work, it feels like a lot of effort for something that I feel should be relatively simple, considering gradle and maven do it.
I have many dependencies in my project, almost all of which can be fetched from a remote repository. However, one of the repositories used contains a dependency that I build a custom version of locally using provided tools for that (hence why I'm not just building it to a custom location, figuring that out is a last resort).
I would like to fetch all my dependencies from remote repositories, except for the one I am building locally. I cannot remove the repository that contains it, as it contains more of my dependencies that I DO want to update.
In addition, this is a project shared with others, so a system scope and a systemPath attribute aren't going to work.
I cannot run maven in offline mode, as I do want to update every other dependency regularly.
Thanks.
Let me start with one suggestion, do not try to override existing coordinates with your custom logic and hence the build. It will always add to the confusion and worst runtime exception will make it hard to debug the problem. You should
Change the coordinates of this custom build project - Best possible solution. This will be treated as any other dependency.
If you can't change the coordinates, change the version. Do add a qualifier to make it future proof for further release of this library from original source.
Always prefer to add this dependency in dependency management section. This will always give this library a preference over dependency mediation.
You could use relocation mechanism of distribution management. This will give a useful message to people. There always is a shared settings.xml across a company.
I haven't downloaded spring in a while. I'm investigating gradle, but don't seem to have the hang of it yet, so i want to create a new spring framework project using java, Spring 4 and ant. I can't seem to find the place to download the binaries for Spring 4.
Are they just kicking ant developers under the bus?
Are they just kicking ant developers under the bus?
No. As Brian mentions, distribution zips are still available at http://repo.spring.io for those who have no other option. Ant users are encouraged to integrate Ivy into their build scripts to manage dependencies so that they will not need these dist zips. Ivy is capable of working against Maven-compatible artifact repositories to provide the same transitive dependency management benefits as Maven and Gradle do. Ant is a perfectly fine build solution for many folks, and we expect it will continue to be for some time. However, manually managing dependencies, i.e. downloading dist zips, storing jars on a network drive or checking them into source control is widely understood in the industry as a problematic approach.
We believe that the majority of Spring users are already using transitive dependency management solutions in one form or another. We continue to provide dist zips for those who have not yet been able to adopt this practice, but to be clear, it is intentional that we have not given those dist zips the first-class treatment on spring.io that they once had on springsource.org, because working with dist zips is simply an inferior way to manage an application's dependencies.
Spring is about helping application development teams eliminate unnecessary complexity. There are few things that can make developing an application more complex and frustrating than the "jar hell" that ensues from manual dependency management. Here are but a few examples of why this can be so painful:
Needing to be connected to the network drive where jars are stored (and thus, very often needing to be connected to the corporate VPN);
Or, in the case of jars being checked into the repository, having massive, unwieldy repositories to manage, typically resulting in many duplications of the same jars across different repositories;
Having no simple, universal and reliable way to (a) know what a dependency's version is and more importantly, (b) know whether that version of that dependency is compatible with all of the other jars in the application's dependency graph.
Maven, Gradle and Ivy are not a silver bullet for all dependency management issues, and naturally they come with their own complexity and learning curve. However, when given a choice, the vast majority of modern Java application developers agree that the benefits of using transitive dependency management outweigh their costs.
We hope that we've struck the right balance in our approach to guiding users how to consume Spring artifacts. We've shone the spotlight on what we (and most folks) consider to be best practices with dependency management by advertising Maven and Gradle syntax, but we've left the door to all comers by by continuing to publish distribution zips. We are however paying attention to feedback to make sure that this approach is in fact suitable for the majority of our users.
For additional information on this topic, see https://github.com/spring-projects/spring-framework/wiki/Downloading-Spring-artifacts.
And as a final note, we sometimes hear from folks that they need distribution zips because their company disallows access to public Maven repositories such as Maven Central (http://search.maven.org) or the Spring Repository (http://repo.spring.io). This is completely understandable, but the appropriate response to these constraints is not to keep development teams in the unproductive dark ages of manual dependency management. The correct solution is to stand up a private artifact repository within the corporate firewall. The leading contenders in this product space are JFrog's Artifactory and Sonatype's Nexus. We strongly recommend that any development team still forced into manual dependency management lobby their architecture teams to look into these products and adopt one of them. The benefits to productivity, build reproducibility, and indeed the ability for companies to effectively govern dependencies are dramatic.
Maven and Gradle are the preferred ways to use Spring Framework in your project, but dist ZIPs are still available on the official repository.
You can use the Maven config, go to any Maven repo, and download the JARs from there. For instance:
<dependencies>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>4.0.0.RELEASE</version>
</dependency>
</dependencies>
Tells me to look in /org/springframework/spring-context/4.0.0.RELEASE folder under any Maven 2 Repository. Using the example, I found this. You should be able to get all the JARs from there.
You can find the dependency information for Spring 4 here
From there you can download the binary jar and the jar with sources for manual dependency management.
I found two sources: the official download page here and OLEX.
This was after wasting lots of time trying to find the zip files via searches on the Spring site. There are no longer any direct download links there.
I have a system consisting of multiple web applications (war) and libraries (jar). All of them are using maven and are under my control (source code, built artifacts in Nexus,...). Let say that application A is using library L1 directly and L2 indirectly (it is used from L1). I can easily check the dependency tree top-down from the application, using maven's dependency:tree or graph:project plugins. But how can I check, who's using my library? From my example, I want to know, whether A is the only application (or library) using L1 and that L2 is used from L1 and from some other application, let say B. Is there any plugin for maven or nexus or should I try to write some script for that? What are your suggestions?
If you wish to achieve this on a repository level, Apache Archiva has a "used by" feature listed under project information
.
This is similar to what mvnrepository.com lists under its "used by" section of an artifact description.
Unfortunately, Nexus does not seem to provide an equivalent feature.
Now I suppose it would be a hassle to maintain yet another repository just for that, but then it would probably easier than what some other answers suggestions, such as writing a plugin to Nexus. I believe Archiva can be configured to proxy other repositories.
Update
In fact, there's also a plugin for Nexus to achieve the "used by" feature.
As far as I know nothing along these lines exists as an open source tool. You could write a Nexus plugin that traverses a repo and checks for usages of your component in all other components by iterating through all the pom's and analyzing them. This would be a rather heavy task to run though since it would have to look at all components and parse all the poms.
In a similar fashion you could do it on a local repository with some other tool. However it probably makes more sense to parse the contents of a repo manager rather than a local repository.
I don't think there's a Maven way to do this. That being said, there are ways of doing this or similar things. Here's a handful examples:
Open up your projects in your favorite IDE. For instance Eclipse will help you with impact analysis on a class level, which most of the time might be good enough
Use a simple "grep" on your source directory. This sounds a bit brusk (as well as stating the obvious), perhaps, but we've used this a lot
Use dependency analysis tools such as Sonargraph or Lattix
I am not aware of any public libraries for this job, so I wrote a customized app which does it for me.
I work with a distribution which involves more than 70 artifacts bundled together. Many times after modifying an artifact, I want to ensure changes are backward compatible (i.e. no compilation errors are introduced in dependent artifacts). To achieve this, it was crucial to know all dependents of modified artifact.
Hence, I wrote an app which scans through all artifacts under a directory(/subdirectories), extracts their pom.xml and searches (in dependency section of pom) for occurrence of modified artifact.
(I did this in java although shell/windows script can do this even more compactly.)
I'll be happy to share code on github, if that could be of any help.
One way that might suit your needs are to create a master-pom with all your maven projects. Then you run the following command on the master-pom:
mvn dependency:tree -DoutputType=graphml -DoutputFile=dependency.graphml
Open the generated file in yEd.
Used the instructions found here:
http://www.summa-tech.com/blog/2011/04/12/a-visual-maven-dependency-tree-view/
More interesting is probably: what would you do with this information? Inform the developers of A not to use library L1 or L2 anymore, because it has a critical bug?
In my opinion you should be able to create a blacklist of dependencies/parents/plugins on your repository manager. Once a project tries to deploy/upload itself with a blacklisted artifact, it should fail. I'm saying uploading and not downloading, because that might break a lot of projects. As far as I know, this is not yet available for any repository-manager.
One of the ways to approach this problem is outside Java itself : write an OS-level monitoring script that tracks each case of fopen() on the jar file under question! Assuming this is in a corporate environemnt, you might have to wait for a few weeks (!) to allow all using processes to access the library at least once!
On Windows, you might use Sysinternals Process Monitor to do this:
http://technet.microsoft.com/en-us/sysinternals/bb896645
On Unix variants, you would use DTrace or strace.
IMHO and also from my experience, looking for a technical solution for such a problem is often an overkill. If the reason why you want to know who is using your artifact(library) is because you want to ensure backward compatibility when you change an artifact or something similar, I think it is best done by communicating your changes using traditional channels and also encourage other teams who might be using your library to talk about it (project blogs, wiki, email, a well known location where documentations are put, Jour fixe etc.).
In theory, you could write a script that crawls though each project in your repository and then parses the maven build.xml (assuming they all use maven) and see whether they have defined a dependency to your artifact. If all the projects in your organization follows the standard maven structure, it should be easy to write one such script (though if any of those projects have a dependency to your artifact via a transitive dependency, things can get a bit more tricky).
Is there any to get the entire remote repository, say from http://repo1.maven.org/maven2/ , to my local repository (to %USERPROFILE%)?
From http://maven.apache.org/community.html:
Being a Good Maven Citizen
The concept of a public repository
built into the core architecture of
Maven makes it necessarily
community-centric. There are a few
simple things that Maven users may do
to help keep that community thriving.
Be a Kind Public Repository User
The best thing that a user can do is
to set up their own remote repository
mirror containing the projects needed.
There are several tools to make this
simpler, such as Nexus or Archiva.
This reduces strain on the Maven
central repository, and allows new
users to get acquainted with Maven
easier and quicker. This is especially
important for power-users and
corporations. The incentive behind
this is, controlling your own servers
can give you desired level of security
and more control over uptime,
resulting in a better experience for
your users. With that said, keep the
following sentiment in mind:
DO NOT wget THE ENTIRE REPOSITORY!
Please take only the jars you need. We
understand this is may entail more
work, but grabbing all 9+ Gigs of
binaries really kills our servers.
Host a Mirror
As an extention to the previous
statement, if you have access to a
large data repository with lots of
bandwidth, please consider becomming a
mirror for the Maven central
repository.
As you can imagine, thousands of users
downloading can put quite a strain on
one server. If you wish to be a
mirror, please file a request in the
Maven Project Administration JIRA
project.
Host a Public Repository
If you have any projects that you wish
others to use, host them on your own
public repository. That way, your
users can simply add your repository
to their own project repo list, and
viola! Maven can keep you and your
users in synch, growing your user-base
due simply to its new-found ease of
use.
Maybe if you explain why you'd like to get the whole central repo in %USERPROFILE% I could provide a better answer (if you you need to go offline, there are smarter solutions than getting 9+ gigs of artifacts). But for now, I don't get the point.
I just had to mirror a private product repo to make available on another machine. lftp worked just fine:
$ lftp http://my.repo.example.com/product/maven
mirror --no-empty-dirs --parallel=3 . my-local-repo
I'm not sure how you would do that using Maven. It might be useful to explain what you are trying to achieve - at a guess you want a local cache?
What you might be looking for really is a local repository manager like Sonatype's Nexus. These repository managers provide a local cache for remote repositories, as well as providing somewhere for you to put your code.
"wget -m http://site.to.mirror.com" should do it
-m stands for "mirror".