I've built a gitlab-ci yaml file, which works well. However the gradle version used in the image is different to that on my local machine. This is causing some unusual side effects, such as only some Java tests being run.
Here's my yaml file:
image: java:8-jdk
before_script:
- echo `pwd`
- export GRADLE_USER_HOME=`pwd`/.gradle
- rm -f .gradle/caches/modules-2/modules-2.lock
- rm -fr .gradle/caches/*/plugin-resolution/
cache:
paths:
- .gradle/wrapper
- .gradle/caches
build:
script:
- ./gradlew build
test:
stage: test
script:
- ./gradlew test
- cat build/jacocoHtml/index.html | grep -o 'Total[^%]*%'
artifacts:
paths:
- build/jacocoHtml
#deploy test coverage
pages:
stage: deploy
dependencies:
- test
script:
- mkdir public
- mkdir public/jacoco
- mv build/jacocoHtml/* public
artifacts:
paths:
- public
only:
- master
Currently, the build is being run on 4.10.3 but I need to update this to 5.1.1 to match my local setup.
Thanks in advance,
Sam
Use gradle docker image instead of java:8-jdk
For the version 5.1.1 with jdk8, use :
image: gradle:5.1.1-jdk8
Related
So i am quite new to the whole CI topic and im trying to run tests after my merge request in my pipeline. Except that when my command try to run the container i have an error.
So this is my gitlab-ci file
image: docker:18
variables:
REPO_URL: "registry.gitlab.com/xxxxx/xxxxxxx"
PROD_SRV: "xxx"
DEV_SRV: "xxx"
services:
- docker:dind
stages:
- build jar
- build docker image
- tests
maven-build:
only:
- work
- test
image: maven:3-jdk-8
stage: build jar
script: "mvn -Dmaven.test.skip=true package -B"
artifacts:
paths:
- target/*.jar
expire_in: 7 days
docker build:
stage: build docker image
only:
- work
- test
before_script:
- echo "$CI_REGISTRY_PASSWORD" | docker login --username "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin
- chmod a+x ./mvnw
script:
- docker info
- docker build -t $REPO_URL:$CI_COMMIT_SHA .
- docker images
- docker push $REPO_URL:$CI_COMMIT_SHA
after_script:
- docker logout registry.gitlab.com 2>/dev/null
- rm /root/.docker/config.json 2>/dev/null
tests job branch test:
stage: tests
only:
- test
variables:
GIT_STRATEGY: none
before_script:
- echo "$CI_REGISTRY_PASSWORD" | docker login --username "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin
script:
- docker pull $REPO_URL:$CI_COMMIT_SHA
- docker run -t $REPO_URL:$CI_COMMIT_SHA .
- docker push $REPO_URL:$CI_COMMIT_SHA
after_script:
- docker logout registry.gitlab.com 2>/dev/null
- rm /root/.docker/config.json 2>/dev/null
and this is my dockerfile
# syntax=docker/dockerfile:1
FROM openjdk:16-alpine3.13 as base
WORKDIR /amsn-counter
COPY .mvn/ .mvn
COPY mvnw pom.xml ./
RUN ./mvnw dependency:go-offline
COPY src ./src
FROM base as test
RUN ["./mvnw", "test"]
FROM base as development
CMD ["./mvnw", "spring-boot:run", "-Dspring-boot.run.profiles=mysql", "-Dspring-boot.run.jvmArguments='-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8000'"]
FROM base as build
RUN ./mvnw package
FROM openjdk:11-jre-slim as production
EXPOSE 8080
ADD /target/linkedin-0.0.1-SNAPSHOT.jar app.jar
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
and the error in the pipeline as well:
If you need any precision i will gladly answer, and like i said im new so don't hesitate to point if something is weird in the files or in my description in general
Edit: My branch are named {work}(local branch) and {test}
It looks as if there is no target specified when you are running docker build. Try this:
script:
- docker info
- image_name="$REPO_URL:$CI_COMMIT_SHA"
- docker build
--target test
--tag $image_name
.
- docker push $image_name
To use a integration test library https://www.testcontainers.org/" I need a image with java a docker installed at same time.
I'm trying o use this stage:
test:
stage: test
image: gradle:jdk16
services:
- docker:latest
script:
- docker --version
- chmod +x ./gradlew
- export GRADLE_USER_HOME=`pwd`/.gradle
- ./gradlew test --stacktrace
rules:
- !reference [.rules_merge_request, rules]
But It does not work:
$ docker --version
/scripts-33119345-2089057982/step_script: line 154: docker: command not found
Any help?
The image gradle:jdk16 does not include the docker client. You'll have to install it in your job. Additionally, you'll need to use the service docker:dind in your services: configuration (not docker:latest)
test:
stage: test
image: gradle:jdk16
services:
- docker:dind # use the docker-in-docker image
before_script: # install docker
- apt update && apt install --no-install-recommends -y docker.io
script:
- docker --version
Running this on gitlab.com runners, you should see an output like this:
When trying to run Firestore emulator in a Gitlab CI/CD pipeline I get the following error:
Firestore Emulator has exited because java is not installed, you can
install it from https://openjdk.java.net/install/
The question is, how do I install java in this env? I found a similar post, but there's no mention of the need of installing java, so I'm wondering if I'm missing something obvious.
This is how my .gitlab-ci.yml file looks like:
image: node:14.15.4
cache:
paths:
- functions/node_modules/
- project_name/node_modules/
before_script:
- npm i -g firebase-tools
.test_and_deploy:
script:
- cd functions
- npm i
- cd ..
- cd project_name
- npm i
- cd ..
- cd functions
- npm run build
- firebase emulators:exec -P project_name --only firestore "npm run test-once"
- cd ..
- cd project_name
- cp .env.project_name.local .env
- npm run build
- firebase --project project_name deploy
I was finally able to figure out how to install java in the docker image, now this part of the script in the yml file looks like this:
.test_and_deploy_functions:
script:
# this step installs java, necessary for the firebase emulator
- apt-get update && apt-get install -y openjdk-8-jdk
- cd functions
- npm i
- npm run build
- firebase use project_name
- firebase emulators:exec --only firestore "npm run test-once"
- npx firebase deploy --only functions
The question is very straightforward
What is the best way to execute liquibase migration in gitlab pipelines
what i have so far
but seems gitlab services immediately executes docker run, and docker run already requires db migration parameters
image: docker:19.03.1
stages:
- build
- db-migration
- deploy
services:
- docker:19.03.1-dind
- liquibase/liquibase:latest
variables:
DOCKER_TLS_CERTDIR: "/certs"
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_IMAGE_PATH: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
before_script:
- apk add --update python python-dev py-pip
- pip install awscli --upgrade
- $(aws ecr get-login --no-include-email --region $AWS_REGION | tr -d '\r')
build:
stage: build
script:
- docker build --tag $AWS_IMAGE_PATH/$CI_PROJECT_NAME:$CI_COMMIT_SHA --tag $AWS_IMAGE_PATH/$CI_PROJECT_NAME:latest .
- docker push $AWS_IMAGE_PATH/$CI_PROJECT_NAME:$CI_COMMIT_SHA
- docker push $AWS_IMAGE_PATH/$CI_PROJECT_NAME:latest
db-migration:
stage: db-migration
script:
- liquibase --changeLogFile=/src/main/resources/db/changelog/psql/changelog.yaml
--url="jdbc:postgresql://host:5432/db"
--username username --password $DB_PASSWORD update
deploy:
stage: deploy
script:
- echo "Deployed"
db-migration:
image: openjdk:8-jre-alpine
stage: db-migration
script:
- INIT_PATH=`pwd`
- apk add bash
- mkdir /liquibase
- mkdir /Downloads
- cd /Downloads
- wget "https://github.com/liquibase/liquibase/releases/download/liquibase-parent-3.7.0/liquibase-3.7.0-bin.zip"
- wget "https://repo1.maven.org/maven2/org/postgresql/postgresql/42.2.8/postgresql-42.2.8.jar"
- unzip liquibase-3.7.0-bin.zip -d /liquibase -q
- mv postgresql-42.2.8.jar /liquibase/lib/
- export PATH=$PATH:/liquibase
- liquibase --changeLogFile=$INIT_PATH/src/main/resources/db/changelog/psql/changelog.yaml
--url="jdbc:postgresql://host:port/db"
--username username --password $DB_PASSWORD update
only:
- master
I have the following .travis.yml file in my project, configured to build always and to deploy only when the build is triggered by a GitHub tag matching a certain version string:
language: java
sudo: false
cache:
directories:
- "$HOME/.cache"
jobs:
include:
- stage: build
os: linux
jdk: oraclejdk8
- stage: build
os: linux
jdk: openjdk8
- stage: build
os: linux
jdk: oraclejdk11
- stage: build
os: linux
jdk: openjdk11
- stage: deploy
os: linux
jdk: openjdk8
stages:
- build
- name: deploy
if: tag =~ ^[0-9]+\.[0-9]+\.[0-9]+
install:
- gpg --version
- mvn process-resources -B -V -e
script:
- mvn test -B -V -e
deploy:
- provider: script
script:
- openssl aes-256-cbc ... -in .travis.gpg.enc -out .travis.gpg -d
- openssl aes-256-cbc ... -in .travis.settings.xml.enc -out .travis.settings.xml -d
- gpg --import .travis.gpg
- cp .travis.settings.xml $HOME/.m2/settings.xml
- mvn clean deploy -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -P sign -B -V -e
skip_cleanup: true
- provider: releases
api_key:
secure: CKKEi9hoN...xbzZByUU80Q=
file_glob: true
file:
- $HOME/.m2/repository/path/to/project-*.pom
- $HOME/.m2/repository/path/to/porject-*.pom.asc
This worked fine when I pushed my branch: The Travis CI build ran and said it was skipping the deploy stage because the condition wasn't met. However, when I merged this branch into master, the Travis CI build ran against master (correct) and invoked the deploy stage (incorrect), attempting to deploy to Maven/GitHub (and failing due to what I believe is an unrelated error).
Oddly, it's not that it started the deploy job (it didn't; it only started the build jobs). It added the deploy stage to the first four (build) jobs (the ones that say stage: build). This is shown in the screenshot below, where you can see that it ran the four build jobs, and they all failed (specifically, they all failed on the deploy stage, but they shouldn't have tried to deploy; only the deploy job should deploy).
So, two questions:
What did I do wrong here that made it try to deploy from master even though it wasn't a tag?
Any idea what the following error is that (thankfully) made the deploy fail, and what I need to do to fix it?
Deploy error:
$ rvm $(travis_internal_ruby) --fuzzy do ruby -S gem install dpl
Successfully installed dpl-1.10.6
Parsing documentation for dpl-1.10.6
Installing ri documentation for dpl-1.10.6
Done installing documentation for dpl after 0 seconds
1 gem installed
dpl.1
Installing deploy dependencies
Successfully installed dpl-script-1.10.6
Parsing documentation for dpl-script-1.10.6
Installing ri documentation for dpl-script-1.10.6
Done installing documentation for dpl-script after 0 seconds
1 gem installed
/home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-1.10.6/lib/dpl/cli.rb:54:in `system': wrong first argument (ArgumentError)
from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-1.10.6/lib/dpl/cli.rb:54:in `shell'
from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-script-1.10.6/lib/dpl/provider/script.rb:19:in `push_app'
from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-1.10.6/lib/dpl/provider.rb:199:in `block in deploy'
from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-1.10.6/lib/dpl/cli.rb:41:in `fold'
from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-1.10.6/lib/dpl/provider.rb:199:in `deploy'
from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-1.10.6/lib/dpl/cli.rb:32:in `run'
from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-1.10.6/lib/dpl/cli.rb:7:in `run'
from /home/travis/.rvm/gems/ruby-2.4.1/gems/dpl-1.10.6/bin/dpl:5:in `<top (required)>'
from /home/travis/.rvm/gems/ruby-2.4.1/bin/dpl:23:in `load'
from /home/travis/.rvm/gems/ruby-2.4.1/bin/dpl:23:in `<main>'
So, I've discovered the answer to both questions.
First, the weird error was due to this:
deploy:
- provider: script
script:
- do something
- do something else
- do another thing
The normal script directive in Travis CI builds accepts a sequence, but the script directive within the script deploy provider is different and only accepts a single string, which it invokes as a single command. You can't even pass in a multi-line string. Still doesn't work. I filed this bug with Travis about that and, until that is fixed, found a temporary workaround for that issue using before_deploy (below).
Second, if you use the top-level deploy directive, then deployment will run as part of the build stage instead of as its own stage. The documentation is not clear on this, but it was an easy fix.
Finally, you can only encrypt a single file, not multiple files, so you if you have multiple secret files, you have to use a Tar archive and encrypt that.
Here's the working Travis build after I worked out all my issues:
language: java
sudo: false
cache:
directories:
- "$HOME/.cache"
jobs:
include:
- stage: build
os: linux
dist: trusty
jdk: oraclejdk8
- stage: build
os: linux
dist: trusty
jdk: openjdk8
- stage: build
os: linux
dist: xenial
jdk: oraclejdk11
- stage: build
os: linux
dist: xenial
jdk: openjdk11
- stage: deploy
os: linux
dist: xenial
jdk: openjdk8
before_deploy:
- openssl aes-256-cbc -in .travis.secrets.tar.enc -out .travis.secrets.tar -d
- tar -xvf .travis.secrets.tar
- rm .travis.secrets.tar
- gpg --import travis.gpg
- rm travis.gpg
- mv settings.xml $HOME/.m2/settings.xml
deploy:
- provider: script
script: "mvn clean deploy -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -P sign -B -V -e"
on:
tags: true
skip_cleanup: true
- provider: releases
api_key:
secure: CKKEi9hoN...xbzZByUU80Q=
file_glob: true
file:
- $HOME/.m2/repository/io/path/to/project-*.pom
- $HOME/.m2/repository/io/path/to/project-*.pom.asc
on:
tags: true
stages:
- build
- name: deploy
if: tag =~ ^[0-9]+\.[0-9]+(\.[0-9]+)?(-[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?$
install:
- gpg --version
- mvn process-resources -B -V -e
script:
- mvn test -B -V -e