I am sort of new to Docker and I am trying to understand some best practice recommendations regarding the build process in the Dockerfile.
I know that for a traditional JAVA application, we will typically use maven to pull dependent libraries from Nexus repository, source code from GIT or other CM and then build the application. Finally, the output .war file will again be stored in a tagged release folder in Nexus repository to be copied over to different environments (dev/qa/production etc.). But one challenge that remained was inconsistencies between the different environments and we are trying to use containers to solve that problem - meaning once it is tested in a container, we can just push that container up the pipeline to different environment and they should work without any issue since the container includes the environmental dependencies plus everything the web application needs as a package.
But my question is regarding the Dockerfile. So is the Dockerfile supposed to replace the function of Maven ? Should it include the complete build instructions for the application ? I know we still pull the source code from GIT and dependent libraries from Nexus by referencing them in the Dockerfile but where does the output .war file go ? Is there still a need to store that in Nexus ? Since we can version the Docker containers through some of the tools such as OpenShift and roll back as necessary, what will be the purpose of storing the output .war file in Nexus.
One scenario I can think of is related to patching. When an OS/security patch is available, we update the base Docker image and rebuild the Application specific Docker image to apply the patch. But what happens to the application in this case ? Do we need to rebuild the application every time we need to apply a patch ? If application build instructions are embedded into the Dockerfile, and we have to rebuild the application container image, doesn't the application also get rebuilt automatically ? Or can Dockerfile only execute certain instructions - skip build application part if no code change or when you tell it to skip etc.) . Should Dockerfile reference to a Nexus repository for the application .war file so in the case the .war file doesn't change, it doesn't rebuild ?
I will appreciate any help.
Thanks.
I would say it'd be a good idea to version control the application build artifacts (war, jar etc.) separately in Nexus and not doing the application build process as part of the docker build. We have a similar pipeline and the way we do it is this:
Checkout source code from SCM
Build with Maven and upload the build artifact to Nexus for version control
Inject the built artifact in the base Docker image(i.e. openjdk) as part of the docker build step
Push the image to the registry
CD kicks in and deploys the latest image
Should Dockerfile reference to a Nexus repository for the application .war file so in the case the .war file doesn't change, it doesn't
rebuild?
Yes, that's correct, you can use curl command in your Dockerfile to download the artifacts from Nexus, no need to rebuild application everytime you make change to your Dockerfile.
Related
I made a Java project which uses an own library, which is installed in my local m2 repository. Now, I'm trying to create a Docker image that generates the jar in a build stage. It fails because it can't resolve dependencies for project of my own library. Is there any way to tell maven (from container) to use my local m2 repository (outside container) so it could resolve my library dependency when I run mvn package?
Simple answer - you can't, because there is no way to mount a volume for docker build command.
I'm assuming your end goal is a Docker image that runs contains your project's built jar or war.
To accomplish that you need two Dockerfiles - one that can be used to create a container that will build your project and exit and a second one that will describe the image that you actually want - one that can be used to create a container that runs your project.
There is a nice article describing how to create and use the first one here. As
for the second one, that depends on whether your project builds as jar or a war and how it needs to be started/deployed.
There is two ways of managing libraries (including private):
(old way) copy your compiled library to your project lib directory and make sure your project pick it up.
use some repository management software. Good options is - Artifactory https://jfrog.com/artifactory or Nexus repository management - https://www.sonatype.com/product-nexus-repository
We have a core set of java libraries, published as jar files, that are used in multiple client projects. Both the library projects and the client projects are on our git server (GitLab EE). Sometimes clients and third-parties (such as sub-contractors) have access to the project source code, but we don't want them to have access to the source for the core java libraries.
So what I need to know is how to set up some kind of dependency or similar in git so that we can automatically pull in the latest version of the library file? I've been looking at git submodule, but that looks like it will pull in, or at least give access to, the full source of the library projects. I've also started looking at Maven, but I'm unclear whether that also needs access to the whole repository. Is there another/better way to do it? Or will Maven actually do what I want? Or do I need to publish my jar files somewhere else, and if so how can I automate that process, as I don't want to have to manually copy the jar file somewhere every time it's updated (which is often)?
Maven and a binary repository like Nexus is the common way to do what you're asking. Nexus stores binary files like .jar's .dll's etc and Maven has the capability to publish and retrieve specific versions of these binaries and include them when building the project. As Ryan commented, Maven Central is a public binary repository that you could publish to if you wanted the whole world access to your .jars, but with Nexus you have fine grained access control over your binaries.
If you build your own projects with Maven, you can configure it to build, test, and publish right to your Nexus repository with a single command mvn deploy. Then clients just have to update their project to point to the new version and Maven will download it and use it on the classpath. It's all very seamless, and I would definitely recommend refactoring your build process to use Maven or a similar product.
I have a Java web project that uses Maven standard directory layout: java files gets into java (actually: /src/main/java), resources into resources, web content into webapp.
Then we wanted to improve our web layer, by adding bower, sass, gulp etc. Our gulp build compiles scss, minimize javascripts, optimize images etc, everything what you would expect. But this introduced 1) another build tool, gulp and 2) generated files by gulp.
Question is how to organize such project? One way could be:
(A) gulp builds into webapp folder
In this solution, all javascript,images,scss files are stored in /src/main/assets and gets build into the /src/main/webapp. Both sources and gulp-generated files gets committed to the git. The gradle build is independent from gulp, and it is ok for the users that does not have gulp installed - like those who needs to work only on backend. Also, CI servers does not depend on gulp stuff.
(B) use gulp from gradle during build
In this solution, gulp is called from gradle. Gradle therefore builds everything. And you must use gradle every time when you want to try something. Also, every developer needs to have gulp installed, what may be a problem for developers using windows (as i've been told). Also CI server should know how to run gulp.
My team is torn between these two options. Does anyone have any working experience with either of these solutions?
I'm currently using Java + Grunt + Maven. I've found there are two ways of packaging your frontend with your backend and the same will certainly apply to Gulp.
In the end it's up to what's best for your project/team. From my experience I usually use option B when working with others since the decoupling is easily worth the other issues. When I'm doing my own side projects I always go for option A because it's just easier to launch one webserver and to run a local environment closer to what DEV/PROD is like.
A) Put your frontend into the webapp folder (ex. https://github.com/kdubb1337/maven-grunt-webapp)
Benefits - You can launch your backend and do development all in one place and working with Spring security is a snap, even without OAUTH. Fewer issues working with two webservers up on your local environment when they would normally be bundled into one port on other environments.
B) Keep your frontend in a different folder, or even in a different repo that you clone into the root folder of your backend repo. (ex. https://github.com/kdubb1337/maven-grunt) see the 'yo' folder
Benefits - Fantastic decoupling so front end developers can live in joy without having to even install java locally or worry about recompiling your backend. Works great if you want Travis (or your favourite CI app) to do unit tests on the backend and the frontend.
EDIT I've found this awesome plugin you can use with maven/gradle to build the frontend https://github.com/eirslett/frontend-maven-plugin. Seems like the way to go, will be refactoring my starter projects with this guy for grunt and gulp
The current best practice here is to treat your frontend build as a separate project and put it in its own Maven or Gradle module. Have your Java build system invoke the JavaScript tooling (e.g., with the maven-exec-plugin) and save the output into the appropriate directory in target or build. Bundle up the results in a jar and serve off the classpath.
If you're using Bower, you only need the base Node install on your CI server, and your Java build can invoke the necessary build process from there, fetching JS packages as needed. Don't forget to (1) use --save and (2) exclude the JS modules directory from source control.
In addition, I suggest looking at RaveJS, which manages your JavaScript build and keeps you from having to configure watchers and such during development.
My recommended best practice is using the com.github.eirslett.frontend-maven-plugin (or the maven grunt plugin) to call the grunt/gulp build from mvn (in process resources goal). When CI builds, it's all integrated, even npm etc can be installed in mvn's target so you don't have bother to configure your CI server for npm.
When developers build, then mostly still just use one maven command. For JS/CSS/HTML developers, after mvn clean install, they can run grunt/gulp "watch" in the background to get their JS changes reflected immediately in the browser without incurring any maven overhead (just the wicked fast gulp/grunt tasks).
Deploy the UI components on default Tomcat webapp directory.
Deploy the class files on "wtpwebapps" (default directory for uploading war through eclipse) directory.
Setting Eclipse
Go to server tab, open properties for tomcat
(source: scotch.io)
Make sure, the location should be [workspace metadata]
Then double click on tomcat to open Tomcat overview.
(source: scotch.io)
Set server location to "Use tomcat location".
Grunt/Gulp Setting
Use the copy task to copy the build UI file to<tomcat installation directory>/webapps/<contextRoot>/
https://pub.scotch.io/#ankur841/eclipse-tomcat-deployment-with-gruntgulp-build
I have a project with maven, this a multimodule project
Mainproject
project1 - jar
project2 - jar
project3 - web
Now that I finished the project I want to test it on the server but don't know how to upload them, on my computer I have added a plug for tomcat which deploys the war file to tomcat automatically, but the server doesn't have maven.
What is the way for moving to production with this kind of projects?
Should I just upload the target directories for each module?
Thanks
You're asking a few questions here. There is the "how do I test on a server" one, and there is "what is the way for moving to production" one. And they can be quite different.
I have assumed that the JAR files in the project are used by the WAR file and packaged within its WEB-INF/lib directory. If I'm wrong, that's cool. Just this sort of information is handy to know.
Maven is a build tool. It is not a deployment tool. What you have at the end of this, is a WAR file. If you run mvn install (or mvn deploy) you have a SNAPSHOT version of the WAR file. This would be suitable for quick, ad-hoc testing to other machines. But you would most likely use methods approriate for the hosting container for making them available. Note: a Maven DEPLOY is a different thing from a DEPLOY a war file to tomcat.
To my mind, if you're putting anything into production, or in front of a customer, or in front of a testing group, you should use Maven to make a Release of the product. That is, using the release plugin (via the release:prepare and release:plugin goals) to create a non-SNAPSHOT release of your artifact (in this case, a official version of the WAR file). Once you have that WAR file, you can migrate it to your production server in whatever way is easiest (copy, deploy into tomcat in the best way). You haven't mentioned if there are database requirements for this web application, and that would need to be considered before you change any production application.
But, once you have official versions, you have tags of source code, and you can accurately know what code is being run.
What I don't think you're going to get is being able to automatically copy the WAR file into a production server from your development environment. Here be dragons.
I'd running maven locally, and it's fantastic! The only issue that I have now is that I have a heavy stack and I have these Massive WAR files. Is there a good approach or best practice with regards to using Maven as a tool to sync your local dependencies with application running on the server?
For example, can I somehow create a goal that uploads the POM, and tells Maven to rebuild on the server side? We're running linux on both sides. Jetty locally and Tomcat remotely.
I'm sure the Apache guys thought of this... Your thoughts are much appreciated...
One tool you could look into is cargo, which can take the generated artifact that you've built and install it to a remote server instance. Cargo can be integrated into your maven build.
However, if you have multiple developers and just want the latest version of whatever is in the repository deployed to the server, I'd encourage you to at least investigate a continuous integration server and setting up a manual or nightly job to do deployments for you. A continuous integration server, such as Jenkins, can:
Check SVN for any code changes (if you want to enable SVN polling)
Perform a checkout (or revert + update) to receive the latest code from SVN.
Build the WAR file using your maven build.
Use either the Jenkins Deploy Plugin or a post build step to deploy the final WAR. Letting Cargo do the deploy via maven during the build is also an option.
Just some thoughts. I also highly recommend Nexus as an artifact repository. Since Jenkins is able to run your build as a Maven build, you can configure Jenkins to run the deploy goal and configure maven with the location of your deployed artifacts server (Nexus location) for the builds to be pushed to.
I think the right way would be to install the resulting war (if neccessary one per server config) into a repository (not SVN, a Maven repository like Nexus). With that you can setup simple scripts or maven configs to deploy the artifacts on your test or production servers without the need to rebuild them on the server.
Checkout the install plugin.