I'm trying to run some unit tests with Apache Maven. I hoped this would be as simple as running the test "goal". But when I did that, maven complained that it could not download some dependencies and thus can't run my tests. This sounds fine, except that I have no idea why it decided I need those dependencies; they are not in my pom.xml, and I doubt they're in my transitive dependencies either. (I'm not sure about that last part; they very well might be in my transitive dependencies.)
Luckily, maven has the perfect tool for this: dependency:tree will tell us exactly which dependency is getting pulled in by what. Except for the small problem that maven thinks to itself "in order to build the tree, I have to resolve the dependencies first" so it tries (and fails) to download those very same dependencies so that it can build the part of the tree that's under them.
So now I don't have a tree, and I have no idea how to proceed from here.
How exactly would you think that maven could resolve transitive dependencies (= dependencies of dependencies) without resolving the dependencies first? Escpecially for the goal "test" also the dependency scope "test" has to be used, which is more then the default scope "compile".
You can use the goal dependency:go-offline to prepare for the offline mode. Maven downloads then all required dependencies. Find the detailed docs for that on https://maven.apache.org/plugins/maven-dependency-plugin/go-offline-mojo.html
You could also have a look at this answer to get another opinion on going online.
The main problem is maven downloads dependencies by demand, you may just check that by triggering different lifecycle phases like mvn initialize, mvn validate, mvn compile, mvn package and checking what maven is trying to download. Sometimes it is possible to figure out project dependencies via analysing project object model (pom), sometimes it is not, especially when plugins define their own dependencies either implicitly or explicitly, some examples below:
we may ask maven-dependency-plugin to download something via dependency:copy-dependencies
exec-maven-plugin has similar functionality: Running Java programs with the exec goal
maven-invoker-plugin may run poms which are part of project but not a part of reactor.
In short: neither maven plugin will able to download all required dependencies. The only "reliable" way to go offline is to run target goal and only then go offline, unfortunately even in this cases some weird things may happen, especially when you or dependency authors are using snapshot versions, version ranges, third-party repositories, etc (my own preference is to run maven with -llr flag to make it more reliable).
Related
I use both Intellij IDEA (2018.3.5) & Eclipse IDEs, but I prefer Intellij. I have a maven based Java project with multiple poms. I added some dependencies to one of the pom files. I need to find out if there are any dependency conflicts which could prevent the build from running when its deployed, and then exclude them. I tried the steps given below to find conflicts which could cause problems. Are they enough or do I need to do more ?
Check if there are any compile time dependency conflicts with mvn clean install -DskipTests. Build was successful with no errors.
Check if Intellij shows no problems under File > Project Structure > Problems. There are no problems.
I also saw the dependency tree with mvn dependency:tree -Dverbose. It has a lot of "omitted for duplicate" and "omitted for conflict with" items, but the build was successful. I don't see any errors though. Does this mean that everything is okay or do I have to do something more about these conflicts ?
The best way to tell if everything is fine with your application is to have good tests.
However normally one doesn't exclude transitive dependencies from project's <dependency> libraries. Doing it can potentially break the dependency in a subtle and hard to notice way. It's usually safer to remove the whole <dependency>.
There are few scenario when one should use <exclude>:
Dealing with incompatible transitive dependencies between different libraries e.g. A requires library C-1.0 but library B requires library C-2.0 while C-1.0 and C-2.0 can't coexist on the classpath.
Having transitive dependencies already provided by system e.g. deploying to Tomcat with additional JARs in the TOMCAT_HOME/lib directory.
If you decide to exclude a dependency it's important that you check the final artifact because sometimes plugins do weird things e.g. there were versions of maven-assembly-plugin affected by a bug that resulted in different dependencies being resolved during shaded JAR creation than maven-dependency-plugin used for compilation.
I found the Resolving step usually takes long time when I run sbt compile, such as this
[info] Resolving org.sonatype.oss#oss-parent;7 ...
Does anyone have ideas about what the resolving actually mean? And why is it necessary?
It is when ivy (which sbt uses for dependency management) looks through the dependency graph of the project to figure out all dependencies and transitive dependencies, and resolve any conflicts between those, for example, two dependencies might depend on different versions of the same library.
The result is cached, but is removed if you do a clean, so it will be recalculated on the next operation that needs to know the dependencies.
It is also possible to configure your project in ways which it so that sbt cannot cache the result but needs to redo those calculations every time you build, for example if you depend on snapshots and have configured sbt to update snapshots for every build.
I am new to using github and have been trying to figure out this question by looking at other people's repositories, but I cannot figure it out. When people fork/clone repositories in github to their local computers to develop on the project, is it expected that the cloned project is complete (ie. it has all of the files that it needs to run properly). For example, if I were to use a third-party library in the form of a .jar file, should I include that .jar file in the repository so that my code is ready to run when someone clones it, or is it better to just make a note that you are using such-and-such third-party libraries and the user will need to download those libraries elsewhere before they begin work. I am just trying to figure at the best practices for my code commits.
Thanks!
Basically it is as Chris said.
You should use a build system that has a package manager. This way you specify which dependencies you need and it downloads them automatically. Personally I have worked with maven and ant. So, here is my experience:
Apache Maven:
First word about maven, it is not a package manager. It is a build system. It just includes a package manager, because for java folks downloading the dependencies is part of the build process.
Maven comes with a nice set of defaults. This means you just use the archtype plugin to create a project ("mvn archetype:create" on the cli). Think of an archetype as a template for your project. You can choose what ever archetype suits your needs best. In case you use some framework, there is probably an archetype for it. Otherwise the simple-project archetype will be your choice. Afterwards your code goes to src/main/java, your test cases go to src/test/java and "mvn install" will build everything. Dependencies can be added to the pom in maven's dependency format. http://search.maven.org/ is the place to look for dependencies. If you find it there, you can simply copy the xml snippet to your pom.xml (which has been created by maven's archetype system for you).
In my experience, maven is the fastest way to get a project with dependencies and test execution set up. Also I never experienced that a maven build which worked on my machine failed somewhere else (except for computers which had year-old java versions). The charm is that maven's default lifecycle (or build cycle) covers all your needs. Also there are a lot of plugins for almost everything. However, you have a big problem if you want to do something that is not covered by maven's lifecycle. However, I only ever encountered that in mixed-language projects. As soon as you need anything but java, you're screwed.
Apache Ivy:
I've only ever used it together with Apache Ant. However, Ivy is a package manager, ant provides a build system. Ivy is integrated into ant as a plugin. While maven usually works out of the box, Ant requires you to write your build file manually. This allows for greater flexibility than maven, but comes with the prize of yet another file to write and maintain. Basically Ant files are as complicated as any source code, which means you should comment and document them. Otherwise you will not be able to maintain your build process later on.
Ivy itself is as easy as maven's dependency system. You have an xml file which defines your dependencies. As for maven, you can find the appropriate xml snippets on maven central http://search.maven.org/.
As a summary, I recommend Maven in case you have a simple Java Project. Ant is for cases where you need to do something special in your build.
I have a Maven 3 multi-module project (~50 modules) which is stored in Git. Multiple developers are working on this code and building it, and we also have automated build machines that run cold builds on every push.
Most individual changelogs alter code in a fairly small number of modules, so it's a waste of time to rebuild the entire source tree with every change. However, I still want the final result of running the parent project build to be the same as if it had built the entire codebase. And I don't want to start manually versioning modules, as this would become a nightmare of criss-crossing version updates.
What I would like to do is add a plugin which intercepts some step in build or install, and takes a hash of the module contents (ideally pulled from Git), then looks in a shared binary repository for an artifact stored under that hash. If one is found, it uses that artifact and doesn't even execute the full build. If it finds nothing in the cache it performs the build as normal, then stores its artifact in the cache. It would also be good to rebuild any modules which have dependencies (direct or transient) which themselves had a cache miss.
Is there anything out there which does anything like this already? If not, what would be the cleanest way to go about adding it to Maven? It seems like plugins might be able to accomplish it, but for a couple pieces I'm having trouble finding the right way to attach to Maven. Specifically:
How can you intercept the "install" goal to check the cache, and only invoke the module's 'native' install goal on a cache miss?
How should a plugin pass state from one module to another regarding which cache misses have occurred in order to force rebuilds of dependencies with changes?
I'm also open to completely different ways to achieve the same end result (fewer redundant builds) although the more drastic the solution the less value it has for me in the near term.
I have previously implemented a more complicated solution with artifact version manipulation and deployment to private Maven repository. However, I think this will fit your needs better and is somewhat more simple:
Split your build into multiple builds (e.g., with a single build per module using maven -pl argument).
Setup parent-child relationships between these builds. (Bamboo even has additional support for figuring out Maven dependencies, but I'm not sure how it works.)
Configure Maven settings.xml to use a different local repository location - specify a new directory inside your build working directory. See docs: https://maven.apache.org/guides/mini/guide-configuring-maven.html
Use mvn install goal to ensure newly built artifacts are added to local repository
Use Bamboo artifact sharing to expose built artifacts from local repository - you should probably filter this to include only the package(s) you're interested in
Set dependent builds to download all artifacts from parent builds and put them into proper subdirectory of local repository (which is customized to be in working directory)
This should even work for feature branch builds thanks to the way Bamboo handles parent-child relations for branch builds.
Note that this implies that Maven will redownload all other dependencies, so you should use a proxy private Maven repository on local network, such as Artifactory or Nexus.
If you want, I can also describe the more complicated scenario I've already implemented that involves modifying artifact versions and deploying to private Maven repository.
The Jenkins plugin allows you to manage/minimize dependent builds
whenever a SNAPSHOT dependency is built (determined by Maven)
after other projects are built (manually via Jenkins jobs)
And if you do a 'mvn deploy' to save the build into your corporate Maven repo then you don't have to worry about dependencies when builds run on slave Jenkins machines. The result is that no module is ever built unless it or one of its dependencies has changed.
Hopefully you can apply these principles to a solution with Bamboo.
The problem is, in our company we have a project with multiple sub-modules, however one of the sub-modules is just a collection of API declarations and is meant for other (3rd praty) projects to use. I want to keep it as a sub-module because is easier to maintain and build (dependency and property inheritance). Other sub-modules in this project are also dependant on it.
The question I have is, if there exist a good practice or a nice way to execute a deploy phase that will upload just this sub-module to a different repository (can be duplicated too) without it having a dependency to parent pom.
What I have already tried:
I have already checked the deploy:deploy-file, but the problem is when it comes to SNAPSHOT builds. We wish to be able to publish SNAPSHOTS and release builds, and snapshots have different repository than release ones, but deploy-file goal can only have one url parameter. I do not wish to use different profile for snapshot deploy. Than I tried to use maven build-helper and its regex-property to be able to change the repository url if the version is a SNAPSHOT, but was unable to do so because of the plugin and regex limitations.
The last option is I can make a plugin for this, but I wish to know if there is a more elegant way to solve this the "maven way".
You can deploy this module separately but only for SNAPSHOT's for a release it does not make sense. The deployment of a module can be done via:
mvn -pl TheModuleYouWouldLikeToDeploy deploy
may be you need to add the option -am (also make dependencies) like:
mvn -am -pl TheModuleYouWouldLikeToDeploy deploy
Apart from that your approach sounds wrong cause if you are using a multi-module build why not deploying the whole build via mvn deploy ? May be it would be better to let do the job via a CI tool like Jenkins.