We Keep Coding

How to add 70 local jars on maven project?

why use Maven when you have such quantity of local jars?
So we have a client that have a lot of private jars and custom jars.
For example commons-langMyCompanyCustom.jar which is commons-lang.jar with 10 more classes in it.
So on their environment we use 100% Maven without local dependencies.
But on our site we have the jars for development in Eclipse and have Maven build with the public ones, but we do not have permission to add their jars in our organizational repository.
So we want to use the Maven good things like: compile,test, build uber-jar, add static code analysis, generate java-docs, sources-jars etc. not to do this thinks one by one with the help of Eclipse.
So we have 70 jar some of them are public if I get the effective pom on their environment I found 50 of them in Maven Central, but the other 20 are as I called "custom" jars. I searched for decision of course but found this:
<dependency>
<groupId>sample</groupId>
<artifactId>com.sample</artifactId>
<version>1.0</version>
<scope>system</scope>
<systemPath>${project.basedir}/src/main/resources/yourJar.jar</systemPath>
</dependency>
So for all 20 of them I have to add this in the development maven profile??
Is there a easy way like in Gradle where you can add all folder with its dependencies to the existing ones?
Also installing one by one in every developer's repo is not acceptable.

Please forget the system scope as mentioned before! Too problematic...
Ideally:
Ideally, all your developers have access to Repository Manager in your or their organization (if possible).
A central environment for your System Integration Testing, maybe?
Alternatively, you may have a central environment for testing where all the dependencies are provided. This approach can be used to simulate how a compilation would work as if it's in your client's environment. Plus you only setup jars once.
So on their environment we use 100% Maven without local dependencies.
But on our site we have the jars for development in Eclipse and have
Maven build with the public ones, but we do not have permission to add
their jars in our organizational repository.
According to what you're saying in the above-quoted excerpt I believe you want to have set in your build's pom.xml assuming that in the client setup the dependencies will be present.
Especially, as you indicate that the organization doesn't give you permission to add their jars in your repository, I would use the provided scope.
As stated in the Maven docs, the definition of a provided dependency is as followed:
This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scope provided because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.
So basically you assume that these dependencies will be present at your client's setup. However, this has some limitations. Meaning you can build solutions independently but cannot test it locally because you won't have the dependencies on your workstation.
If you won't even have access to the jars to configure your central environment ask if your client can provide a DEV/SIT environment.
None of the above? Inherit a parent pom.
To avoid the whole constant copy-paste process for every single (related) project, maven has the tools to centralize dependency and plugin configurations, one of such is by inheriting the configuration of a parent pom. As is explaining in the following documentation it is quite simple:
First you create a project with just a pom.xml where you define everything you wish to centralize (watch out, certain items have slight differences in their constructs);
Use as property of packaging tag the option pom: <packaging>pom</packaging>;
In the pom's that have to inherit these configurations set the parent configuration tags in <parent> ... </parent> (documentation is very clear with this);
Now everytime you update any "global" pom configuration only the parent version has to be updated on every project. As a result of this, you only need to configure everything once.
You can also apply this together with the abovementioned solutions, this way combining to find a solution that fits best to your needs.
But there is a big Maven world out there, so I advise a good read in its doc's to further acknowledge your possibilities. I remembered these situations because I've been in a similar situation you seem to be now.
Good luck!

Another alternative is the project RepoTree.
This one creates a Maven repository directory (not a server) from another directory which contains just the .jars. In other words, it creates the necessary .pom files and directory structure. It takes into account only the precise information from metadata contained in the archives (MANIFEST.MF, pom.xml).
Utility to recursively install artifacts from a directory into a local
Maven repository Based on Aether 1.7
This is 5 years old, but still should work fine.

TL;DR: MavenHoe creates a Maven repository server (not a directory) which serves the artefacts from a directory, guessing what you ask for if needed. The purpose is to avoid complicated version synchronizing - it simply takes whatever is closest to the requested G:A:V.
I have moved the MavenHoe project, which almost got lost with the decline of Google Code, to Github. Therefore I put it here for availability in the form of a full answer:
One of the options you have when dealing with conditions like that is to take whatever comes in form of a directory with .jar's and treat it as a repository.
Some time ago I have written a tool for that purpose. My situation was that we were building JBoss EAP and recompiled every single dependency.
That resulted in thousands of .jars which were most often the same as their Central counterpart (plus security and bug fixes).
I needed the tests to run against these artifacts rather than the Central ones. However, the Maven coordinates were the same.
Therefore, I wrote this "Maven repository/proxy" which provided the artifact if it found something that could be it, and if not, it proxied the request to Central.
It can derive the G:A:V from three sources:
MANIFEST.MF
META-INF/.../pom.xml
Location of the file in the directory, in combination with a configuration file like this:
jboss-managed.jar org/jboss/man/ jboss-managed 2.1.0.SP1 jboss-managed-2.1.0.SP1.jar
getopt.jar gnu-getopt/ getopt 1.0.12-brew getopt-1.0.12-brew.jar
jboss-kernel.jar org/jboss/microcontainer/ jboss-kernel 2.0.6.GA jboss-kernel-2.0.6.GA.jar
jboss-logging-spi.jar org/jboss/logging/ jboss-logging-spi 2.1.0.GA jboss-logging-spi-2.1.0.GA.jar
...
The first column is the filename in the .zip; Then groupId (with either slashes or dots), artifactId, version, artifact file name, respectively.
Your 70 files would be listed in this file.
See more information at this page:
https://rawgit.com/OndraZizka/MavenHoe/master/docs/README.html
The project is available here.
Feel free to fork and push further, if you don't find anything better.

Maven Plugin for project semantic versioning

I'm looking for a maven plugin that will help me manage version names and codes of every build that is made on our CI environment. Something that will be able to attach a prefix to the main version code or even update it (not changing the pom.xml). For example:
project version: 2.0.1
git/svn revision: 2342334
jar output: name-2.0.1-2342334.jar
maven repo: ../path/to/local/maven/repo/<package path>/2.0.1-2342334/
The main requirements to this plugin are:
Must be in Maven Repository (which means that NO additional setting required to add this plugin in my pom.xml and run maven)
Must not edit the pom, each time it's applied
A configuration file, would be great, so I could manage the versioning process
Must be able to edit the output file metadata (so the version will be applied as if it was written in the pom.xml file in the first place)
So far I found only maven-buildmetadata-pluging but unfortunately it's not in Maven Repo, so I'm stuck. Any help would be great.

Hosting your own maven repository is very easy, using either Nexus or Artifactory. You can also use the Artifactory cloud version (I'm not affiliated with them...) so it may solve your problem. BTW - a simple server with Apache does the trick as well, but with more work..,
Regarding the plugins: If you deploy snapshot applications then each gets its own version based on timestamp.
For releases another option is to run an svn info and put the result (or part of it) into the generated artifact. The information can then be accessed by the code.

If you change the version of your artifact the pom has to reflect the change, cause otherwise it's not reproducible.
If you change something in your build process (like added versions, whatever) it has to be reflected in the pom file. Otherwise you can not reproduce the build process with the same result.
You have written not to change the pom file but maintaining a separate file. So the questions is: Why not using the pom file itself, cause it's intended exactly for that purpose.
Furthermore all informations which you mentioned by the maven-buildmetadata-plugin can be achived by using existing maven plugins (like build-helper-maven-plugin, buildnumber-maven-plugin).
The SCM information can be used by using the buildnumber-maven-plugin which provides information like SCM revision number (SVN or GIT hash).
An on the other hand if you don't like to change your pom file manually you can use either the versions-maven-plugin or the maven-release-plugin which automatically can change informations in your pom file and handle all these things automatically.
To maintain metadata in your producted artifacts you can configure all plugins (like ear, war, jar) etc. more or less like this where the buildNumber is comming from buildnumber-maven-plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>${maven-jar-plugin.version}</version>
<configuration>
<archive>
<addMavenDescriptor>true</addMavenDescriptor>
<index>true</index>
<manifest>
<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
<addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
</manifest>
<manifestEntries>
<artifactId>${project.artifactId}</artifactId>
<groupId>${project.groupId}</groupId>
<version>${project.version}</version>
<buildNumber>${buildNumber}</buildNumber>
</manifestEntries>
</archive>
</configuration>
</plugin>
And of course if you really like to use Maven you should have to use an repository manager as already mentioned like Artifactory or Nexus which make life easier.

I just would like to add (although the question is 5 years old and already has an accepted answer) that the Buildmetadata Maven Plugin was not available on the Maven Repo at first, but it is now (since late 2013). People who would like to give it a try find the artifact at the following locations :
com.redhat.rcm.maven.plugin:buildmetadata-maven-plugin
de.smartics.maven.plugin:buildmetadata-maven-plugin
Please note that the name has changed from maven-buildmetadata-plugin to buildmetadata-maven-plugin due to naming conventions.
I'm one of the "original" authors of this plugin at smartics. If you would like to use it, you probably would like to use the fork provided by Red Hat. To my knowledge the two versions do not differ very much and they have not been synced since there is just so much other stuff to do and the plugin seems to be feature stable. ;-)
The source code for both versions is also available on GitHub:
release-engineering/buildmetadata-maven-plugin
smartics/buildmetadata-maven-plugin

As already stated, you have to change the version in the pom. One way of doing that, in combination with the release plugin is:
mvn \
se.bjurr.gitchangelog:git-changelog-maven-plugin:VERSION_HERE:semantic-version \
release:prepare release:perform
Using Git Changelog Maven Plugin

Smarter Native Dependency Handling with Maven

I'm currently in the midst of converting a large multi-module project (~100 sub-modules) to use Maven. Currently we use Ant + Ivy.
So far no major issues have cropped up and I'm comfortable that Maven is still a good fit. However, I wonder if there is a better way to handle native dependencies.
So far I have come to the following conclusions.
It's best to install each native dependency into the maven repo either as a standalone library or an archived package containing multiple dependencies.
Rather than get lost in declaring each and every dependency with the Maven dependency plugin, I opted to give each a classifier (e.g. natives-win32) and use the following in the parent POM:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.4</version>
<executions>
<execution>
<id>copy</id>
<phase>compile</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<includeScope>runtime</includeScope>
<includeClassifiers>natives-win32</includeClassifiers>
<outputDirectory>${project.build.directory}/natives</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
So far this seems to be a simple all-round solution that doesn't require too much messing about to add new native dependencies. It also offers me a simple all-round solution for managing natives. The only thing I must do is ensure that my /natives/ directory is defined on java.library.path.
One thing that bothers me (a little) about this approach is that all my native dependencies get copied around to each sub-module that expresses a transitive dependency on them, whilst my happy jar libraries are added to the classpath referenced to where they sit in my local repository (no copy required).
Is there no way to be smarter about this and have my natives referenced from withing their repository location (assuming I don't have them archived, i.e. dll). That would save a bunch of unnecessary copying about.
Are there any other potential gotchas' that I should be concerned about with the above approach?

Your snippet shows a goal attached to a build phase, not a dependency. Is the 'copy dependencies' goal in a super pom and inherited by all modules? There's no way to move it only to the modules which are going to be run/packaged as an app?

It could be, that I didn't got it. But why don't you deploy all your native libs into the repository at first. If the native libs are stable and change seldom, That could be done in a seperate reactor.
And afterwards you reference those native dependencies simply via GAV as any other dependency. Also the problem af unnecessary copying is solved by that.

I ended up using the maven natives plugin and dealing with the fact that I have redundant copies of the native libraries around the place. The reason for this was primarily due to the simplicity that the plugin offers and the fact that it also has a related eclipse plugin that sets up natives in developers eclipse environment without intervention.

switching to gradle from maven to manage a osgi big project (>200 bundles)

We have a big (~215 bundles and counting) osgi (felix+springdm) project, build with maven and maven-osgi plugin.
We've several problems with maven way:
1.　submodules pom have to inherit from parent pom to take advantage of common variables and dependencies (that's ok) but then parent pom has to include all bundles pom to be able to build everything in together. This kind of circular reference makes much hard to keep all in sync.
2.　the individual versioning of subbundles was so complex that it was decided (before I joined the project) to use the same version for all bundles. This means we now update version of all bundles for every release also if just a bunch of them are actually changed. This makes the whole concept of osgi a bit meanless IMHO. Please note that I'm not saying we continue to touch just a minority of bundles, we work on all of them but every release usually contains 1 or 2 features, that affects just some bundles.
3.　to do the package and the deploy of the final artifact we need still another submodule that imports all the bundles needed for the deploy (all but a few for tests and mocks).
[edited]
Note that this aggregation is different from the one in the main pom as it doesn't compile bundles but just pick them from the maven repository.
4.　the maven dependency system and the osgi plugin imports are sometimes hard to keep aligned. It's just too easy to forget an import or putting a wrong dependency.
[edited]
In every bundle pom there is a section like this:
`
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<extensions>true</extensions>
<configuration>
<instructions>
<Export-Package>
</Export-Package>
<Import-Package>
com.google.gson,
org.apache.log4j,
org.apache.log4j.spi,
org.dom4j,
com.myinterfaces
</Import-Package>
</instructions>
</configuration>
</plugin>`
For all those reasons, we are ok but not perfectly happy with maven. Recently someone proposed Gradle not as a panacea but as a definite improvements over the current situation.
Would you recommend moving to gradle? and in case which would be the best way?
Has someone else experienced the same situation? I think it should be common for all big projects with Osgi.
Disclaimer: I looked for similar questions like:
Buildr, Gradle or wait for Maven 3?
Looking for a good dev environment for OSGi bundles
Maven : OSGI, bundles and multi-modules projects
but either where where not about osgi submodules or not about gradle.

You can separate the parent and the aggregate maven modules, because currently your parent pom have two roles as you correctly observed. More information can be found in the Maven Introduction to POM.
I'm afraid that bundles version management cannot be become easier unless you use API Tools. Perhaps it would be great if API tools can be integrated as Maven plugin, but I'm not aware of any work in this area. So, you either touch all versions at once or update them each time it is needed. API Tools will greatly help here, but it works only for bundles, which can be imported as Plug-in Projects inside Eclipse.
So, will another aggregator module help here? You can setup several aggregators, which aggregate another aggregators, so you don't end up with one huge aggregator module which lists everything? Because, you may not want to deploy everything, you can setup what to exclude from deploying. Quick google search showed how to do it.
#Neil Bartlett already noted that bnd will take care of your manifest if you've setup your dependencies right. If you need extra tweaking of the defaults, you can always set BND instructions file.
You can put Tycho in the list of possible tools. It will help you with the dependency management, because you need to specify your dependencies in the Manifest only and it will permit you to use API Tools (but no integration yet). It will however require that you use p2 repositories if you want to skip some headaches (until Tycho has improved their support for depending on Maven artifacts).

What is a reasonable OSGi development workflow?

I'm using OSGi for my latest project at work, and it's pretty beautiful as far as modularity and functionality.
But I'm not happy with the development workflow. Eventually, I plan to have 30-50 separate bundles, arranged in a dependency graph - supposedly, this is what OSGi is designed for. But I can't figure out a clean way to manage dependencies at compile time.
Example: You have bundles A and B. B depends on packages defined in A. Each bundle is developed as a separate Java project.
In order to compile B, A has to be on the javac classpath.
Do you:
Reference the file system location of project A in B's build script?
Build A and throw the jar into B's lib directory?
Rely on Eclipse's "referenced projects" feature and always use Eclipse's classpath to build (ugh)
Use a common "lib" directory for all projects and dump the bundle jars there after compilation?
Set up a bundle repository, parse the manifest from the build script and pull down the required bundles from the repository?
No. 5 sounds the cleanest, but also like a lot of overhead.

My company has 100+ bundle projects and we use Eclipse to manage the dependencies. However, I don't recommend the "Required Plugins" approach to managing the dependencies. Your best bet is to create Plugin Projects. Export just the packages from each project that you want to be visible. Then on the import side do the following:
Open the Manifest editor
Goto the dependencies tab In the bottom left is a section called "Automated Management of Dependencies"
Add any plugins that the current plugin depends on there
Once you have code written, you can click the "add dependencies" link on that tab to auto-compute the imported packages.
If you run from Eclipse, this gets done automatically for you when you execute.
The benefits of this approach is that your built bundles are only using OSGi defined package import/export mechanism, as opposed to something from Eclipse.
If you want to learn more, I'd recommend going to this site and ordering the book. It's excellent.
http://equinoxosgi.org/

Well, do what you should have a long time before, separate implementation and API ... ok, this is not always that easy on existing systems but that model has a huge bang for your buck. Once your API is in a separate (much more stable) bundle/jar you can compile the clients and implementations against that bundle/jar.
One of the key qualities of a successful bundle is that it makes as little assumptions about the outside world as possible. This implies you do not have to compile against the bundles you run against in runtime, I have a preference to try hard to not do that. You should only compile against the bundles minimum set of dependencies. If assumptions are made they are explicit as imported packages and the use of services. Well designed OSGi systems attempt to use services for all inter-bundle communications. Not only does this model get rid of class loading issues it also makes your build setup more decoupled.
Unfortunately most code is written as libraries that have a rather wide interface because they hand code lots of the functionality that services provide out of the box like Factories and Listeners. This code has a tight relationship between implementation and API so you have to have the same on the class path during compile and in OSGi. One solution to this problem is to include this kind of code inside the bundle using it (but make sure no objects of this library leak to other bundles). A bit of extra memory consumption but it saves you from some headaches.
So with OSGi, try to create systems relying on services and compile against their service API, not an implementation bundle.

Basically, you can use:
source dependency (with Eclipse's "referenced projects")
binary dependency (using the jar of bundle A)
But since binary dependency is much cleaner, it is also the kind of dependency best managed by a release management framework like maven.
And you can integrate maven in your Eclipse project through m2eclipse.
The Maven plugin to use would then be: maven-bundle-plugin, that you can see in action in:
Using maven to create an osgi bundle (osgi felix sample)
Bundle Plugin for Maven
Getting the benefits of maven-bundle-plugin in other project types
How to build OSGi bundles using Maven Bundle Plugin
Consider this more real-world example using Felix' Log Service implementation.
The Log Service project is comprised of a single package: org.apache.felix.log.impl.
It has a dependency on the core OSGi interfaces as well as a dependency on the compendium OSGi interfaces for the specific log service interfaces. The following is its POM file:
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>org.apache.felix</groupId>
<artifactId>org.apache.felix.log</artifactId>
<packaging>bundle</packaging>
<name>Apache Felix Log Service</name>
<version>0.8.0-SNAPSHOT</version>
<description>
This bundle provides an implementation of the OSGi R4 Log service.
</description>
<dependencies>
<dependency>
<groupId>${pom.groupId}</groupId>
<artifactId>org.osgi.core</artifactId>
<version>0.8.0-incubator</version>
</dependency>
<dependency>
<groupId>${pom.groupId}</groupId>
<artifactId>org.osgi.compendium</artifactId>
<version>0.9.0-incubator-SNAPSHOT</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<extensions>true</extensions>
<configuration>
<instructions>
<Export-Package>org.osgi.service.log</Export-Package>
<Private-Package>org.apache.felix.log.impl</Private-Package>
<Bundle-SymbolicName>${pom.artifactId}</Bundle-SymbolicName>
<Bundle-Activator>${pom.artifactId}.impl.Activator</Bundle-Activator>
<Export-Service>org.osgi.service.log.LogService,org.osgi.service.log.LogReaderService</Export-Service>
</instructions>
</configuration>
</plugin>
</plugins>
</build>
</project>

There is a 6th option, which I've used for several projects, which is to use a single Eclipse project (not a plugin project, but just an ordinary Java project) and put all source code in there. A build file associated with the project will simply compile all code in a single pass and subsequently create bundles out of the compiled classes (using Bnd from Ant or from the soon to be released BndTools).
This has the downside that it does not honor visibility at development and compile time, but the upside that it's a really simple development model that gives you very fast build and deploy times.