I'm trying to work with gitlab CI/CD. I'm using Ubuntu server and Spring Boot with Maven. All is fine, runner starts pipeline jobs but it gets lots of errors with pattern "warning: failed to remove target/..." even if i call simple echo 'something' in .yaml pipeline script gitlab-ci.yaml. I found that if i remove /home/gitlab-runner/builds then all starts to work fine until /builds generated again. What am i doing wrong? I already tried to reinstall runner, making gitlab-user, different variations of script^ nothing works until i manually remove builds folder. However, there is also js frontend which is also on gitlab ci/cd and everything works fine there. Help me please!
Here is the error i get trying to get my java spring boot maven pipeline work:
enter image description here
gitlab-ci.yaml code here:
stages:
- test
- package
- deploy
# - sonar
test:
stage: test
only:
- master
- merge_requests
except:
- tags
script:
- echo 'test are running i swear!!!!!!'
- sudo mvn clean
- sudo systemctl stop socnet.service
package:
stage: package
only:
- master
except:
- tags
script:
- sudo mvn package -Dmaven.test.skip=true
deploy_to_server:
stage: deploy
only:
- master
except:
- tags
script:
- sudo systemctl restart socnet.service
Remove sudo from your .gitlab-ci.yml.
Using sudo there will execute mvn package as root user, hence all generated files have root as the owner.
When gitlab-runner picks up a job and proceeds to clean up previously generated files, it is still unprivileged and hence will fail to remove files owned by root.
You might want to add the following variables into your .gitlab-ci.yml file in order to change the location for Maven dependencies cache to inside the project directory:
variables:
MAVEN_OPTS: "-Dhttps.protocols=TLSv1.2 -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
MAVEN_CLI_OPTS: "--batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true"
https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Maven.gitlab-ci.yml
Related
name: demo
on:
push:
branches:
- 'pipeline-dev'
jobs:
conditional_step:
runs-on: 'ubuntu-20.04'
steps:
- uses: actions/checkout#v3
with:
repository: hello-world-with-gradle
- run: cd /home/runner/work/test-actions/test-actions
- uses: actions/setup-java#v3
with:
distribution: temurin
java-version: 8
- name: Setup Gradle
uses: gradle/gradle-build-action#v2
- name: Execute Gradle build
run: ./gradlew build
The above workflow gives a JAR as as a build for Application. Now do we have the capabilities to run this JAR in the workflow itself?
like
- run: Java -jar hello-world-0.1.0.jar // This gives an error
In our real use case we want to run the utility in pipeline and send its output result to other google bucket.
Tried out various java actions plugins but not able to run JAR yet. If anyone know how to run a JAR or any other alternative It would be helpful.
check if the case matters, use java, not Java:
- run: java -jar hello-world-0.1.0.jar
^^^
That way, the GitHub runner should find and execute java after applying actions/setup-java.
I am trying to setup my development environment using docker for a sprint boot application.
I am using intellij idea.
Here is the dockerfile.
FROM gradle:7.4.2-jdk18-alpine AS build
COPY --chown=gradle:gradle ./ /home/gradle/src
WORKDIR /home/gradle/src
RUN gradle build --no-daemon --debug
FROM openjdk:19-slim
EXPOSE 5097
EXPOSE 5005
RUN mkdir /app
COPY --from=build /home/gradle/src/build/libs/*.jar /app/spring-boot-application.jar
ENTRYPOINT ["java", "-Xdebug", "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005", "-jar", "/app/spring-boot-application.jar"]
and docker-compose.yml is:
version: '3.7'
services:
hmis-config-service:
build:
context: .
dockerfile: Dockerfile
image: 'hmis-config-service:0.0.0.1'
ports:
- "9060:5080"
- "8091:5005"
volumes:
- myapp:/home/gradle/src
environment:
db.url: 'jdbc:postgresql://host.docker.internal:5432/hms'
db.username: 'postgres'
db.password: 'pgsroot'
GRADLE_HOME: /usr/local/gradle
JAVA_HOME: /usr/lib/jvm/java
M2: /usr/local/apache-maven/bin
M2_HOME: /usr/local/apache-maven
volumes:
myapp:
Every time I try to build it, it takes a long time because it downloads all gradle dependencies. Hence even a small change takes a long time.
If project is started using
docker-compose up # without the --build flag
it starts instantaneously but the changes made aren't in the container.
I have tried to mount the volume /home/gradle/src to keep the dependencies synced but that didn't do anything.
What can be done to improve the build time and cache the dependencies?
Is there a reason you need to build the app in the docker context? If you build it outside (locally), and only copy the artefacts, you can leverage build caching from the (local) gradle daemon, which should increase the speeds significantly.
Essentially, you are building the project from scratch on each build/invocation.
i.e, don't build inside the Dockerfile, only copy the already built artefacts.
PS: If you have to build inside the docker context (for instance, if you don't have a JDK locally), you can try to figure out where gradle keeps its local cache and also sync/copy that over.
As you mentioned the issue is the build takes long time because it downloads dependencies each time.
If that is the case, you can try using multi-stage builds.
Basically you download the dependencies once then you just copy them each time. This also reduces the size of the image.
References:
https://docs.docker.com/develop/develop-images/multistage-build/
https://spring.io/guides/topicals/spring-boot-docker/
https://gist.github.com/msauza/6a906e879549e218c54868d81161afcb
https://hashedin.com/blog/a-guide-to-optimize-build-time-in-docker/
P.S: I have not tried this for Spring Boot applications, but they work well for applications based on Go.
I try to deploy the spring-boot project with GitLab CI/CD. But can't find any information that can help me.
So, I try to figure out how I can do it and created an AWS instance and run the gitlub-runner on it.
I created .gitlab-ci.yml and can copy files to the S3-bucket:
variables:
S3_BUCKET_NAME: "awesome-proj"
deploy:
script: aws s3 cp ./ s3://awesome-proj/ --acl bucket-owner-full-control --no-sign-request --recursive --exclude "*" --include "*.html"
tags:
- Tag1
I figure out I can use the console in a similar way and clone my project to the AWS server. I decided to clone the project from the git repo and build him on the server with 'maven'.
So I try to clone the project:
S3_BUCKET_NAME: "awesome-proj"
deploy:
script:
- sudo cd /home/ec2-user/aweproject
- sudo git init
- sudo git clone https://gitlab.com/lTer/cicdtestdev.git
tags:
- Tag1
These scripts even copied some data once.
For the second time, I got the error, but it is doesn't matter now. The problem is that I haven't any files after git clone.... So it not works how I hoped.
How to correctly use the console with GitLab-runner or deploy the project (spring-boot) with Gitlab ci/cd to the AWS?
In principle, you don't need to clone your project. When your CI pipeline runs, it already has the appropriate clone (of a branch, or the master, of whatever the CI has been instructed to run on).
The only thing you need is to compile and then deploy. For instance:
stages:
- compile
- deploy
compile:
stage: compile
script: mvn ...
artifacts:
paths:
- build/
expire_in: 1 week
deploy:
stage: deploy
script: aws s3 sync build/ s3://somebucket/some_place/build/
Of course, in real life, you may want to have a specific stage for testing etc.
I have a Spring Boot application with Gradle as a build tool and org.sglahn:gradle-dockerfile-plugin which I use to build and pull docker images
Everything works fine until this point
I have a property in build.gradle file which I use to set up build version number:
version = '0.0.1-SNAPSHOT'
I'm trying to access version property from Dockerfile but it is not available there:
FROM openjdk:9
RUN mkdir -p /usr/local/configserver
ADD /build/libs/config-server-${project.version}.jar /usr/local/configserver/
ADD run.sh run.sh
RUN chmod +x run.sh
CMD ./run.sh
I expect to see replaced value on the third line of Dockerfile but it doesn't happen. I get the following error:
Step 3/6 : ADD /build/libs/config-server-${project.version}.jar
/usr/local/configserver/ failed to process
"/build/libs/config-server-${project.version}.jar": missing ':' in
substitution
I also tried to put version property in gradle.properties file but this doesn't help either
When I replace the line with real build filename everything works as expected
ADD /build/libs/config-server-0.0.1-SNAPSHOT.jar
Can anybody advise please how to access gradle property in Dockerfile
I've a virtual server running a gitlab runner.
I've now added some GUI unit tests which run nicely on my pc but not on the virtual server.
It always exits with:
java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
Any idea how to get this running with gitlab-ci?
Edit:
My virtual server is running centos 7
This im my current .gitlab-ci.yml
image: kaiwinter/docker-java8-maven
#maven:3-jdk-7
before_script:
- "Xvfb :99 &"
- "export DISPLAY=:99"
maven_build:
stage: build
script:
- "mvn clean package"
- "cat target/site/jacoco/index.html"
Now, the headlessexception is gone but basically all unit tests are failing due to awt exceptions like:
Could not initialize class java.awt.image.IndexColorModel
Edit2:
I've added the headless option as suggested:
image: kaiwinter/docker-java8-maven
#maven:3-jdk-7
before_script:
- "Xvfb :99 &"
- "export DISPLAY=:99"
maven_build:
stage: build
script:
- "mvn clean package -Djava.awt.headless=true"
- "cat target/site/jacoco/index.html"
Now I get the headless exceptions again...
You try using xvfb program like in this post http://elementalselenium.com/tips/38-headless.
I have used xvfb to run browser from text terminal. Your case is basically the same.
The problem is that your program expects to be run in window environment, but you are running it from text terminal.
Finally I've found a solution!
I've created a DOCKER image which is prepared for GUI testing (using xvfb, thanks vbuhlev):
https://github.com/thorstenwagner/docker-java8-maven
In the .gitlab-ci.yml I've added the following lines:
before_script:
- "Xvfb :99 &"
- "export DISPLAY=:99"
You need to enable Headless Mode:
maven_build:
stage: build
script:
- "mvn clean package -Djava.awt.headless=true"