Is there any to get the entire remote repository, say from http://repo1.maven.org/maven2/ , to my local repository (to %USERPROFILE%)?
From http://maven.apache.org/community.html:
Being a Good Maven Citizen
The concept of a public repository
built into the core architecture of
Maven makes it necessarily
community-centric. There are a few
simple things that Maven users may do
to help keep that community thriving.
Be a Kind Public Repository User
The best thing that a user can do is
to set up their own remote repository
mirror containing the projects needed.
There are several tools to make this
simpler, such as Nexus or Archiva.
This reduces strain on the Maven
central repository, and allows new
users to get acquainted with Maven
easier and quicker. This is especially
important for power-users and
corporations. The incentive behind
this is, controlling your own servers
can give you desired level of security
and more control over uptime,
resulting in a better experience for
your users. With that said, keep the
following sentiment in mind:
DO NOT wget THE ENTIRE REPOSITORY!
Please take only the jars you need. We
understand this is may entail more
work, but grabbing all 9+ Gigs of
binaries really kills our servers.
Host a Mirror
As an extention to the previous
statement, if you have access to a
large data repository with lots of
bandwidth, please consider becomming a
mirror for the Maven central
repository.
As you can imagine, thousands of users
downloading can put quite a strain on
one server. If you wish to be a
mirror, please file a request in the
Maven Project Administration JIRA
project.
Host a Public Repository
If you have any projects that you wish
others to use, host them on your own
public repository. That way, your
users can simply add your repository
to their own project repo list, and
viola! Maven can keep you and your
users in synch, growing your user-base
due simply to its new-found ease of
use.
Maybe if you explain why you'd like to get the whole central repo in %USERPROFILE% I could provide a better answer (if you you need to go offline, there are smarter solutions than getting 9+ gigs of artifacts). But for now, I don't get the point.
I just had to mirror a private product repo to make available on another machine. lftp worked just fine:
$ lftp http://my.repo.example.com/product/maven
mirror --no-empty-dirs --parallel=3 . my-local-repo
I'm not sure how you would do that using Maven. It might be useful to explain what you are trying to achieve - at a guess you want a local cache?
What you might be looking for really is a local repository manager like Sonatype's Nexus. These repository managers provide a local cache for remote repositories, as well as providing somewhere for you to put your code.
"wget -m http://site.to.mirror.com" should do it
-m stands for "mirror".
Related
We recently migrated from SVN, with most code in a single repo, to git, with most projects in their own repos (about 70 of them). We build about a dozen different apps from this java source. The apps all run on *nix servers. We use maven and nexus to build. Many of us are struggling with developing features when that feature touches more than one repo. Here are a few of the challenges:
The developer has to branch each repo separately - we use the same name for all branches for one feature to make tracking less difficult.
One must update poms of all repos to point to the updated versions of each repo's artifact. If multiple people are working on the same branch, there can be a lot of merging others pom changes. When I commit a change to a repo, then the artifact is renamed to "-SNAPSHOT" which means more pom updates.
Changes need to be pushed in the right order or our automated builds will fail, e.g: repo A depends on a change to repo B; if repo A is pushed before repo B is built and deployed, then repo A won't build.
The person reviewing the feature has to look at changes in multiple repos.
When the feature is merged from its branch to, say, master, One has to remember all the repos that were touched.
It looks like switching to a mostly monorepo approach might be best, tho there are some drawbacks there:
Building the entire codebase with maven takes a looong time. (Why can't maven be more like make, only building things that have changed or whose dependencies have changed?)
Each push kicks off a big set of builds and many unit tests rather than just one repo's artifact build and test.
The developers who generally work in one or two repos prefer this new multi-repo world and will resist a change back.
I've looked into git submodules and sub trees, which don't seem to solve many of our issues (Not sure about Google Repo). Some of us use tools like "mu" to help. It would be sweet if there was a toolkit that would help developers maintain versions in poms, and track changes across repos.
Let me know if you have a set of procedures or tools you use to ease development in this kind of environment.
with most projects in their own repos (about 70 of them).`
For me this is where the problems start. My vote goes for minimising this number significantly.
If you really don't want a single repo (1 repo gets my vote) then you could separate the code base into n*change_often repos with 1*change_rarely repo. Keeping the n small is important. This way you would avoid rebuilding the bits that change rarely.
Also, even with the a single repo you don't need to reference everything by source and use binaries for base libraries. When a base library changes the person making the change could also update all the references in one go so that that all projects are up to date.
I am trying to set up a local maven repository using nexus to store ALL of the maven dependency libraries to be use in an offline environment. We need to be able to point our maven to this repository instead of trying to query the internet for a needed project library. There is no option of using the internet in our environment so I need an easy(ish) way to download literally every library maven could need in order to populate our repository. I am not asking for an opinion on what is the easiest, I just need a few methodologies that would work because I have no idea where to start with this other than spending the next year clicking links and downloading. I have found methods online that explain how to store project specific dependencies but I can't find a way to get ALL of the dependencies. I also know these libraries are continuous changing and be upgraded, so any ideas on how to keep them up to date without individually clicking and downloading each one would be great as well. Thanks!
MavenCentral is huge. Downloading it not only takes ages, they will block you if you try.
I understand very well that you cannot allow your developers to access MavenCentral during their development process. Our solution for this problem is as follows:
We have a proxy of MavenCentral that is only accessible to a few admins. If a developer needs a dependency that is not on our development Nexus, he writes a ticket to the admins. They check the dependency and if they approve, they start a job that copies the required dependency (together with all transitive dependencies) from the MavenCentral proxy to the development Nexus. This way, the admins have full control over what the developers use, but you need not copy MavenCentral as a whole.
I have a very well structured big project which is maintained in SVN in my team. And we not want to move to Git - which will further integrated with continious integration servers like Teamcity or Jenkins (yet to decide).
When I saw the svn I found that since svn allows you to create tags from anywhere - It has a lot of tags which only contains single-single projects.
The codebase is so huge that I import all the code in one git repo. One approach would be for each project - I can create seperate repos, or combine some of the project to one repo (not sure)
The thing I want to achieve is I want to retain all the history and break the repo so that it becomes easy to manage. (Also it ll be helpful in future while integration with jenkins for automated builds on every commit)
How can I ensure to retain history, have all the existing tags, and move to git
git-svn is the obvious solution to help with this migration but, in my opinion, this is better for a one-off migration from SVN to git.
If you want to keep both the SVN and git repositories alive and allow commits to both (keeping them both in sync), I would recommend SubGit.
When we moved from svn to git I found this stackoverflow post very helpful, specifically the second answer. I assume from your description that everything is under a single trunk. That does make it a little more complicated. I have not tried moving individual projects before, but you may be able to do it in one of two ways:
Migrate the entire svn repo, tags, branches, etc. to your local git repo (before you push to the remote). Then break up the projects by following the suggestions in (this) stackoverflow post. This should give you individual repos that you can then push to the remote repos.
You may be able to alter the steps when running the svn to git conversion and specify an individual project, but that seems dangerous and confusing since the tags/branches won't necessarily line up.
I have a system consisting of multiple web applications (war) and libraries (jar). All of them are using maven and are under my control (source code, built artifacts in Nexus,...). Let say that application A is using library L1 directly and L2 indirectly (it is used from L1). I can easily check the dependency tree top-down from the application, using maven's dependency:tree or graph:project plugins. But how can I check, who's using my library? From my example, I want to know, whether A is the only application (or library) using L1 and that L2 is used from L1 and from some other application, let say B. Is there any plugin for maven or nexus or should I try to write some script for that? What are your suggestions?
If you wish to achieve this on a repository level, Apache Archiva has a "used by" feature listed under project information
.
This is similar to what mvnrepository.com lists under its "used by" section of an artifact description.
Unfortunately, Nexus does not seem to provide an equivalent feature.
Now I suppose it would be a hassle to maintain yet another repository just for that, but then it would probably easier than what some other answers suggestions, such as writing a plugin to Nexus. I believe Archiva can be configured to proxy other repositories.
Update
In fact, there's also a plugin for Nexus to achieve the "used by" feature.
As far as I know nothing along these lines exists as an open source tool. You could write a Nexus plugin that traverses a repo and checks for usages of your component in all other components by iterating through all the pom's and analyzing them. This would be a rather heavy task to run though since it would have to look at all components and parse all the poms.
In a similar fashion you could do it on a local repository with some other tool. However it probably makes more sense to parse the contents of a repo manager rather than a local repository.
I don't think there's a Maven way to do this. That being said, there are ways of doing this or similar things. Here's a handful examples:
Open up your projects in your favorite IDE. For instance Eclipse will help you with impact analysis on a class level, which most of the time might be good enough
Use a simple "grep" on your source directory. This sounds a bit brusk (as well as stating the obvious), perhaps, but we've used this a lot
Use dependency analysis tools such as Sonargraph or Lattix
I am not aware of any public libraries for this job, so I wrote a customized app which does it for me.
I work with a distribution which involves more than 70 artifacts bundled together. Many times after modifying an artifact, I want to ensure changes are backward compatible (i.e. no compilation errors are introduced in dependent artifacts). To achieve this, it was crucial to know all dependents of modified artifact.
Hence, I wrote an app which scans through all artifacts under a directory(/subdirectories), extracts their pom.xml and searches (in dependency section of pom) for occurrence of modified artifact.
(I did this in java although shell/windows script can do this even more compactly.)
I'll be happy to share code on github, if that could be of any help.
One way that might suit your needs are to create a master-pom with all your maven projects. Then you run the following command on the master-pom:
mvn dependency:tree -DoutputType=graphml -DoutputFile=dependency.graphml
Open the generated file in yEd.
Used the instructions found here:
http://www.summa-tech.com/blog/2011/04/12/a-visual-maven-dependency-tree-view/
More interesting is probably: what would you do with this information? Inform the developers of A not to use library L1 or L2 anymore, because it has a critical bug?
In my opinion you should be able to create a blacklist of dependencies/parents/plugins on your repository manager. Once a project tries to deploy/upload itself with a blacklisted artifact, it should fail. I'm saying uploading and not downloading, because that might break a lot of projects. As far as I know, this is not yet available for any repository-manager.
One of the ways to approach this problem is outside Java itself : write an OS-level monitoring script that tracks each case of fopen() on the jar file under question! Assuming this is in a corporate environemnt, you might have to wait for a few weeks (!) to allow all using processes to access the library at least once!
On Windows, you might use Sysinternals Process Monitor to do this:
http://technet.microsoft.com/en-us/sysinternals/bb896645
On Unix variants, you would use DTrace or strace.
IMHO and also from my experience, looking for a technical solution for such a problem is often an overkill. If the reason why you want to know who is using your artifact(library) is because you want to ensure backward compatibility when you change an artifact or something similar, I think it is best done by communicating your changes using traditional channels and also encourage other teams who might be using your library to talk about it (project blogs, wiki, email, a well known location where documentations are put, Jour fixe etc.).
In theory, you could write a script that crawls though each project in your repository and then parses the maven build.xml (assuming they all use maven) and see whether they have defined a dependency to your artifact. If all the projects in your organization follows the standard maven structure, it should be easy to write one such script (though if any of those projects have a dependency to your artifact via a transitive dependency, things can get a bit more tricky).
At my school, we're starting a coding competition for our CS club to help out our freshmen learn the practices of fast turn around times, due dates, code comparison, and API documentation using Robocode. I was wondering if there was a way to allow each member to work on their own packages/robots and pull them into a central location for comparison and competition? I'd like to be able to pull all of the projects onto my laptop but allow club members to keep their packages separate and only work on their parts of it. A cross platform solution is ideal.
You could post your code on Github or Google Code for free and use that if you don't mind everyone being able to look at it. You could all have your own separate repositories out there and just view them there.
Otherwise, I would go with Subversion on a central server. There are plenty of free options out there like Unfuddle and Springloops that offer free plans. You could create a folder for each of you or create separate repositories. The downside on those free packages is the number of users.
Of course, the most customizable way would be to use your own server. The only problem with that is the complexity of configuring all of it. Subversion and Git are both pretty easy to set up though.
Git has a really nice feature called submodule.
It lets you group any number of unique git repositories into another repository. This way, all the programmers create their own repos, and then you can create a super project to link them all together.
There is a good description on how to do this at the Pro Git online book.