I have a Maven project which I want to build in a Docker container, so that I don't have to install Java, Maven, ... on my Jenkins system.
I build the Maven project as following:
docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock:ro
-v $(pwd):/opt/maven
-w /opt/maven
maven
mvn clean install
This works great, the build is correctly made. However, when doing this as my regular user, all files in the target/ directory are owned by root and no by jester, which is my current user.
Is there an easy way to fix this?
You chan check out the option -u, as in this gist, which wraps docker run in a script:
#!/bin/sh
set -e
test ":$DEBUG" != :true || set -x
# set image
set -- debian:jessie "$#"
# use current user and its groups at host
for v in /etc/group /etc/passwd; do
[ ! -r "$v" ] || set -- -v $v:$v:ro "$#"
done
set -- --user "`id -u`:`id -g`" "$#"
for g in `id -G`; do
set -- --group-add "$g" "$#"
done
set -- -v "$HOME":"$HOME" "$#"
exec docker run --rm -it "$#"
The goal is to make sure the container mounts the local user definition, and uses the same uid/gid.
Related
I'm trying to create a CI/CD Pipeline for a simple java/maven project.
The runner that I'm using is a docker runner.
I'm using a dockerfile to create a container which installs maven/java/etc.. and in this container the program should be tested.
Sorry for the question but I am new to CI/CD Pipelines in GitLab.
GitHub works just fine have a look: https://github.com/ni920/CICD-Test
Thank you
Here are the CI logs
...
Executing "step_script" stage of the job script
$ docker build --build-arg JAVA_VERSION=openjdk7
/bin/sh: eval: line 95: docker: not found
Cleaning up file based variables
ERROR: Job failed: exit code 127
Thats the .gitlab-ci.yml
stages:
- java7
# - java11
# - deploy
java7:
stage: java7
script:
- docker build --build-arg JAVA_VERSION=openjdk7
# tags:
# - docker
#java11:
# stage: java11
# script:
# - docker build --build-arg JAVA_VERSION=openjdk11
# tags:
# - docker
Thats the dockerfile
# Pull base image.
FROM alpine as build
ARG MAVEN_VERSION=3.6.1
ARG USER_HOME_DIR="/root"
ARG JAVA_VERSION=openjdk7
ARG BASE_URL=https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries
ENV HTTP_PROXY=#comment
ENV HTTPS_PROXY=#comment
# Install Java.
RUN apk --update --no-cache add JAVA_VERSION curl
RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
&& curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
&& tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \
&& rm -f /tmp/apache-maven.tar.gz \
&& ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
ENV MAVEN_HOME /usr/share/maven
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"
# Define working directory.
WORKDIR /data
# Define commonly used JAVA_HOME variable
ENV JAVA_HOME /usr/lib/jvm/default-jvm/
# Define default command.
CMD ["mvn", "--version"]
Running your pipelines using the Docker executor means that your jobs will run in a Docker container, but not that you will be able to execute docker commands.
If you need to run docker commands inside a GitLab CI job (read "inside a container") you will need Docker-in-Docker (often abbreviated DinD). It is a vast topic on itself but you can get started with GitLab CI's documentation: Use Docker to build Docker images
I always use DinD and have a minimal setup in my gitlab-ci.yml.
Using a docker image as a default:
image: docker:19.03.13
Define a default variable for TLS certificates:
variables:
DOCKER_TLS_CERTDIR: "/certs"
Then use a docker image as a service to enable DinD:
services:
- name: docker:19.03.13-dind
alias: docker
I wrote a few posts about using Docker-in-Docker on GitLab CI that you may find useful, but I still recommend to extensively read GitLab's documentation before reading them.
I have a Testng Selenium Project that is build using Maven. I am running this maven project using Maven Surefire plugin like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version>
<configuration>
<forkMode>never</forkMode>
<useFile>true</useFile>
<testFailureIgnore>true</testFailureIgnore>
<!-- Suite testng xml file to consider for test execution-->
<suiteXmlFiles>
<suiteXmlFile>${suiteXmlFile}</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>
What I need to do?
I need to run this Selenium project inside Docker Container. I need to move the complete source code to the container and run it from there. While running, I will be passing the path of testng xml file, that particular test alone should run. Post run, I need to take the result from docker image to local system (which we can do using docker cp ...).
What I have done so far?
I have created a docker image with maven, chrome, chromedriver. At run time, I am passing the testng XML file path and as expected that particular test case alone is running. But....
Once the program gets completed, the docker container is getting closed. docker ps shows no running containers. So, am not able to see the report.
What I want?
So, I want a way to avoid container from getting closed after the execution so that I can go into the docker container and see the report.
My Dockerfile:
FROM kshivaprasad/java
RUN apt-get update
RUN apt-get upgrade --fix-missing -y
RUN apt-get install -y curl
RUN apt-get install -y p7zip \
p7zip-full \
unace \
zip \
unzip
# Install Chrome for Selenium
RUN curl http://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_83.0.4103.116-1_amd64.deb -o /chrome.deb
RUN dpkg -i /chrome.deb || apt-get install -yf
RUN rm /chrome.deb
# Install chromedriver for Selenium
RUN mkdir -p /app/bin
RUN curl https://chromedriver.storage.googleapis.com/83.0.4103.39/chromedriver_linux64.zip -o /tmp/chromedriver.zip \
&& unzip /tmp/chromedriver.zip -d /app/bin/ \
&& rm /tmp/chromedriver.zip
ARG MAVEN_VERSION=3.6.3
# 2- Define a constant with the working directory
ARG USER_HOME_DIR="/root"
# 3- Define the SHA key to validate the maven download
ARG SHA=c35a1803a6e70a126e80b2b3ae33eed961f83ed74d18fcd16909b2d44d7dada3203f1ffe726c17ef8dcca2dcaa9fca676987befeadc9b9f759967a8cb77181c0
# 4- Define the URL where maven can be downloaded from
ARG BASE_URL=http://apachemirror.wuchna.com/maven/maven-3/${MAVEN_VERSION}/binaries
# 5- Create the directories, download maven, validate the download, install it, remove downloaded file and set links
RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
&& echo "Downlaoding maven" \
&& curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
\
&& echo "Checking download hash" \
&& echo "${SHA} /tmp/apache-maven.tar.gz" | sha512sum -c - \
\
&& echo "Unziping maven" \
&& tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \
\
&& echo "Cleaning and setting links" \
&& rm -f /tmp/apache-maven.tar.gz \
&& ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
# 6- Define environmental variables required by Maven, like Maven_Home directory and where the maven repo is located
ENV MAVEN_HOME /usr/share/maven
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"
COPY src /app/src
COPY pom.xml /app
COPY testng /app/testng
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /app/bin/chromedriver
#RUN mvn -f /app/pom.xml clean package
ENTRYPOINT ["/entrypoint.sh"]
That entrypoint.sh file is used to send the argument at run time. It consists of:
#!/bin/sh
mvn -f /app/pom.xml clean install -DsuiteXmlFile=$1
How I run this?
docker build -t my_image .
docker run -it my_image module/testng.xml
This process would be very time consuming when it comes to the quick time to market. I would suggest use zalenium https://opensource.zalando.com/zalenium/ in order to run your selenium script in the docker container and it has various features too like viewing the ongoing execution and retrieving the reports.
I just need to need bash command in the shell file (entrypoint.sh). It worked fine for me.
#!/bin/sh
mvn -f /app/pom.xml clean install -DsuiteXmlFile=$1
/bin/bash
I can properly launch this Java 8 application using this command from a bash shell:
java -cp "simple-queue-0.1-SNAPSHOT.jar:jms-1.1.jar:commons-logging-1.2.jar:activemq-all-5.13.3.jar"
-Dserver1="my1.domain.com"
-Dserver2="my2.domain.com"
-Dusername="user"
-Dpassword="passwd"
com.fusesource.activemq.exercises.simple.queue.SimpleProducer
I want to containerize this application, so here is my Dockerfile:
FROM store/oracle/serverjre:8
MAINTAINER <me#myco.com>
EXPOSE 4567
VOLUME /data
COPY build/libs/*.jar /usr/local/bin/
COPY /app/simple-queue-0.1-SNAPSHOT.jar /usr/local/bin/
CMD ["java", "-cp", "/usr/local/bin/simple-queue-0.1-SNAPSHOT.jar:/usr/local/bin/jms-1.1.jar:/usr/local/bin/commons-logging-1.2.jar:/usr/local/bin/activemq-all-5.13.3.jar", "-Dserver1=$SERVER1", -Dserver2="$SERVER2", -Dusername="$USER", -Dpassword="$PASSWORD"]
I start my container like this:
docker run -it --rm -e SERVER1=my1.domain.com -e SERVER2=my2.domain.com -e USER=user -e PASSWORD=passwd ecosystem/simple-queue-client:1.1 com.fusesource.activemq.exercises.simple.queue.SimpleProducer
And I get this error message:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"com.fusesource.activemq.exercises.simple.queue.SimpleProducer\": executable file not found in $PATH": unknown.
When I get into that container, I can see that my jar files are in fact inside /usr/local/bin directory which is in the PATH. The CLASSPATH in the container is empty...
What do I need to do to fix this?
I used info from the link provided by midelb above and ended up with two containers: one for SimpleProducer and another for SimpleReceiver.
Here is the Docker file for one:
FROM store/oracle/serverjre:8
MAINTAINER <james.depaul#maxar.com>
VOLUME /data
COPY build/libs/*.jar /usr/local/bin/
COPY /app/simple-queue-0.1-SNAPSHOT.jar /usr/local/bin/
ENTRYPOINT java -classpath /usr/local/bin/simple-queue-0.1-SNAPSHOT.jar:/usr/local/bin/jms-1.1.jar:/usr/local/bin/commons-logging-1.2.jar:/usr/local/bin/activemq-all-5.13.3.jar -Dserver1=$SERVER1 -Dserver2=$SERVER2 -Dusername=$USER -Dpassword=$PASSWORD com.fusesource.activemq.exercises.simple.queue.SimpleConsumer
Build
docker build -t mysystem/simple-client-consumer:1.0
And I call it like this now:
docker run -d --rm -e SERVER1=server-b0.domain.com -e SERVER2=server-b1.domain.com -e USER=user -e PASSWORD=passwd mysystem/simple-client-consumer:1.0
I have a docker run command, I am just wondering what the part -v "..":".." -w ".." maven mvn clean... is doing?
docker run --rm --name ${DOCKER_IMAGE_NAME}_build -v "$(pwd)":/usr/src/mymaven -w /usr/src/mymaven maven:3.6.0-jdk-8-alpine mvn clean install package
I think it is mapping the local volume on the OS "$(pwd)" to the docker folder inside the docker volume which is /usr/src/mymaven, then it executes the command with "w" inside the /usr/src/mymaven" folder but because of the mapping it executes inside the local $pwd directory right? so it executes the "maven:3.6.0-jdk-8-alpine mvn clean install package" part, which starts the mvn build process, but what is maven:3.6.0-jdk... I know that it is an image, but how does it know where to pull that image from and how is it possible to directly execute that command for maven thereafter for that image?
The -v part of that command binds a volume to the container you're starting. In this case -v "$(pwd)":/usr/src/mymaven
adds your current directory as /usr/src/mymaven to the container.
the -w part of the command sets your working dir within the container to /usr/src/mymaven
and the part after that actually
maven:3.6.0-jdk-8-alpine -> this is the container you're starting
mvn clean install package -> this is the command you're starting inside the container (so in the /usr/src/mymaven working directory)
basically this run command starts up a container, runs maven and does this with your current directory where you started as source to run in.
for more info about the arguments: docker doc on run arguments
I am getting this strange error at the end of the process of creating a docker image from a Dockerfile:
/bin/sh: 1: gradle: not found
INFO[0003] The command [/bin/sh -c gradle test jar] returned a non-zero code: 127
The relevant part of the Dockerfile:
FROM debian:jessie
[...]
RUN curl -L https://services.gradle.org/distributions/gradle-2.4-bin.zip -o gradle-2.4-bin.zip
RUN apt-get install -y unzip
RUN unzip gradle-2.4-bin.zip
RUN echo 'export GRADLE_HOME=/app/gradle-2.4' >> $HOME/.bashrc
RUN echo 'export PATH=$PATH:$GRADLE_HOME/bin' >> $HOME/.bashrc
RUN /bin/bash -c "source $HOME/.bashrc"
RUN gradle test jar
[...]
The command I am using is: docker build -t java_i .
The strange thing is that if:
I run a container from the previous image commenting out RUN gradle test jar (command: docker run -d -p 9093:8080 -p 9094:8081 --name java_c -i -t java_i),
then I log into that container (command: docker exec -it java_c bash),
then I manually check the gradle environment variables finding them,
then I manually run that commented out command from within the running container (gradle test jar):
I eventually get the expected output (the compiled java code in the build folder).
I am using Docker version 1.6.2
I solved the problem using the ENV docker instructions (link to the documentation).
ENV GRADLE_HOME=/app/gradle-2.4
ENV PATH=$PATH:$GRADLE_HOME/bin
This command /bin/bash -c "source $HOME/.bashrc" means that you create a new non-interactive process and run a command in it to set environment variables there. Which does not affect the parent process. As soon as variables are set, process exits. You can check this by running something like this:
RUN /bin/bash -c "source $HOME/.bashrc; env"
RUN env
What should be working is this option:
RUN source ~/.bashrc
And the reason why it works when you log in, is because the new process reads already updated ~/.bashrc.
I was trying to install same version with JDK 11.0.7 but gradle-2.4 does not work. and got below error
FAILURE: Build failed with an exception.
* What went wrong:
Could not determine java version from '11.0.7'.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
I install later version to fix the above issue after installation.
Posting as an answer might help someone else.
FROM openjdk:11.0.7-jdk
RUN apt-get update && apt-get install -y unzip
WORKDIR /gradle
RUN curl -L https://services.gradle.org/distributions/gradle-6.5.1-bin.zip -o gradle-6.5.1-bin.zip
RUN unzip gradle-6.5.1-bin.zip
ENV GRADLE_HOME=/gradle/gradle-6.5.1
ENV PATH=$PATH:$GRADLE_HOME/bin
RUN gradle --version
You can use multi-stage builds and the Gradle Docker image (no need to install Gradle...) to build the application then use the result in the runtime container:
# Build
FROM gradle AS build
WORKDIR /appbuild
COPY . /appbuild
RUN gradle --version
# here goes your build code
Once the Gradle build is done, switch to the runtime container:
# Runtime
FROM openjdk:8-jre-alpine
# more stuff here...
COPY --from=0 appbuild/<somepath>/some.jar application.jar
# more stuff here...
The COPY command copies the build artifacts from the build phase to the runtime container (in this case a jar file).