I have a use case where I want to run a jar file via Airflow, all of which has to live in a Docker Container on Mac.
I have tried installing java separately and also, tried mounting my JAVA_HOME(host) onto the container.
This is my docker-compose.yaml:
airflow:
image: 'puckel/docker-airflow:1.10.9'
hostname: airflow
container_name: airflow
volumes:
- ${PWD}/airflow/dags:/usr/local/airflow/dags
- ${JAVA_HOME}:/usr/local/bin/java //FWD MOUNTING JAVA_HOME
This way, I get a java directory inside /usr/local/bin/ with the data but java -version returns Permission denied.
Changing it to ${JAVA_HOME}/bin/java:/usr/local/bin/java returns exec format error.
What is the correct way to handle this use case?
I think that you're getting Permission denied because you are running docker with user airflow.
Can you try to run it as root? (this is risky! don't use in production - it is just an effort to temporary workaround). Avoid using root user!
airflow:
image: 'puckel/docker-airflow:1.10.9'
hostname: airflow
container_name: airflow
user: root
volumes:
- ${PWD}/airflow/dags:/usr/local/airflow/dags
- ${JAVA_HOME}:/usr/local/bin/java
EDIT:
Instead of mounting your local java, consider installation of a seperate one:
airflow:
build:
context: .
dockerfile: Dockerfile
hostname: airflow
container_name: airflow
volumes:
- ${PWD}/airflow/dags:/usr/local/airflow/dags
and add the Dockerfile in the same directory:
FROM puckel/docker-airflow:1.10.9
USER root
RUN mkdir -p /usr/share/man/man1
RUN apt-get update && apt-get install -y default-jdk && apt-get clean
USER airflow
Related
To use a integration test library https://www.testcontainers.org/" I need a image with java a docker installed at same time.
I'm trying o use this stage:
test:
stage: test
image: gradle:jdk16
services:
- docker:latest
script:
- docker --version
- chmod +x ./gradlew
- export GRADLE_USER_HOME=`pwd`/.gradle
- ./gradlew test --stacktrace
rules:
- !reference [.rules_merge_request, rules]
But It does not work:
$ docker --version
/scripts-33119345-2089057982/step_script: line 154: docker: command not found
Any help?
The image gradle:jdk16 does not include the docker client. You'll have to install it in your job. Additionally, you'll need to use the service docker:dind in your services: configuration (not docker:latest)
test:
stage: test
image: gradle:jdk16
services:
- docker:dind # use the docker-in-docker image
before_script: # install docker
- apt update && apt install --no-install-recommends -y docker.io
script:
- docker --version
Running this on gitlab.com runners, you should see an output like this:
I'm not able to detach from a Docker container after I attached to it.
Dockerfile to define the container:
FROM adoptopenjdk/openjdk16:debian
WORKDIR /app
STOPSIGNAL SIGTERM
RUN apt-get update && apt-get install -y curl && apt clean
CMD curl -so server.jar https://ci.tivy.ca/job/Airplane-1.17/lastSuccessfulBuild/artifact/launcher-airplane.jar && java -jar server.jar
Build the container with: docker build -t minecraft .
docker-compose file:
version: "3.7"
services:
mc:
container_name: mc
ports:
- 25565:25565
image: minecraft
volumes:
- type: bind
source: /root/mc
target: /app
Start the container with: docker-compose up mc
I tryed to attach to console with docker attach mc, it works. But I'm not unable to detach from screen. Ctrl+C not work. Ctrl+P and Ctrl+Q not work. Writing stop (command that will stop the java process) not working
I tryed to attach with docker attach --detach-keys="ctrl-d" mc but is not working
Java process never ends
Found fix myself. Need to insert stdin_open and ttyin docker-compose file. New docker-compose.yml:
version: "3.7"
services:
mc:
container_name: mc
ports:
- 25565:25565
image: minecraft
volumes:
- type: bind
source: /root/mc
target: /app
stdin_open: true
tty: true
Will set this answer as solution in two days because of stackoverflow rules
Hi I need to dockerize a system. the way I have to do this like below
steps:
up dynamodb local instance ( just for up ).
run a custom script to create tables ( have to go through this to create the tables ).
then run the system.
I wrote a compose file also. the way I did that was, like below
version: "3"
services:
dynamodb:
image: amazon/dynamodb-local
ports:
- "8000:8000"
networks:
- custom-network
volumes:
- "db-data:/home/dynamodblocal/data"
app:
container_name: my-app
build:
context: .
dockerfile: Dockerfile
args:
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
URL: ${URL}
env_file:
- docker.env
depends_on:
- dynamodb
networks:
- custom-network
volumes:
db-data:
networks:
custom-network:
docker file as below. ( sorry had to hide sensitive details )
FROM debian:buster
ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
ARG URL
RUN echo "deb http://ftp.us.debian.org/debian sid main" >> /etc/apt/sources.list
RUN apt-get update
RUN apt-get install openjdk-8-jdk maven awscli -y
RUN aws s3 cp ${URL} db-updater.jar
RUN echo local > input
# there are few lines of configs that wrote to input file
RUN cat input | java -jar db-updater.jar http://dynamodb:8000
WORKDIR /opt/app
COPY . .
RUN mvn package
EXPOSE 8080
CMD ["java","-cp","./app/target/app-1.0.0.jar:./app/target/lib/*"]
my problem is looks like dynamodb do not start before the script run. so script throws a error as can't connect to server.
if I could write a custom a dynamodb with executed script that is also great. please help
Commands in a Dockerfile can never interact with other Docker containers. The general pattern is that an image is built once and reused, so you could delete and recreate your DynamoDB container, or run the same image on a different system, and the database setup wouldn't have happened. Mechanically, the Dockerfile runs in an environment where it's not connected to the Compose networking system, so attempts at connecting between containers will generally fail with a "no such host" error.
A typical pattern is to use an entrypoint script to do first-time setup when the container launches. For example, you could write a simple shell script:
#!/bin/sh
# Seed the database
java -jar db-updater.jar http://dynamodb:8000 < input
# Run whatever the main container command is
exec "$#"
You can then include this in your Dockerfile:
COPY entrypoint.sh . # probably included in the `COPY . .` line
ENTRYPOINT ["./entrypoint.sh"] # must be JSON-array syntax
# replaces `RUN java -jar db-updater.jar`
CMD ["java", "-cp", ...] # as in the current Dockerfile
If you only need this to run once when you first set up the container stack, you could also seed the data on your host.
# Outside Docker
aws s3 cp ... db-updater.jar
./make-seed-data.sh > input
# Start the DynamoDB container (only)
docker-compose up -d dynamodb
# Load the seed data
java -jar db-updater.jar -url http://localhost:8000 < input
# Now start the rest of the application
docker-compose up -d
This would let you remove the code to build the input file and download the updater tool from your Dockerfile. It would also let you remove the AWS credentials from the build sequence (very important: it may be possible to find them in plain text looking at the image's docker history).
I am building an node mongo project.
I am using docker compose for the project.
here is my dockerFile
FROM node:carbon
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 8080
CMD [ "npm", "start" ]
Here is docker-compose.yml
version: "2"
services:
app:
container_name: app
restart: always
build: .
ports:
- "3000:3000"
links:
- mongo
mongo:
container_name: mongo
image: mongo
volumes:
- ./data:/data/db
ports:
- "27017:27017"
Here I also want to install java using docker-compose. As I will need java for elastic search and for other purposes. So can any one help about how to install java using docker-compose in this project.
Docker-compose is a tool used to launch multiple containers from a single .yaml file.
Add this line to your Dockerfile to install Java:
RUN apt-get update && \
apt-get install -y openjdk-8-jdk && \
apt-get install -y ant && \
apt-get clean;
I need to control the order of Docker containers instantiation, the problem is that I want to build a Jar file with the Docker maven container then pass that jar to an OpenJDK Docker container in order to build an image and then instantiate a MongoDB container and a Java-App container with the OpenJDK image generated before that communicates between them via docker-compose.
The problem is the Build always fails because some of the Unit tests talk to the database before it's initialized and since the tests fail the build also fails.
This is my dockerfile:
FROM maven:3.5-alpine
COPY ./ /app
RUN cd /app && mvn package
FROM openjdk:8
COPY spring-rest-iw-exam.jar /tmp/spring-rest-iw-exam.jar
EXPOSE 8087
ENTRYPOINT ["java", "-jar", "/tmp/spring-rest-iw-exam.jar"]
This is my Docker-Compose:
version: '2'
services:
mongodb:
image: mongo
container_name: iw_exam_mongo
restart: always
ports:
- "27017:27017"
environment:
- MONGO_INITDB_DATABASE=fizz_buzz_collection
volumes:
- /opt/iw-exam/data:/data/db
spring-app:
container_name: iw_exam_java_rest_api
build: ./
restart: always
ports:
- "8087:8087"
depends_on:
- mongodb
I tried with depends_on and did some other tests with a tool call dockerize but none of it works, the maven build always starts before docker-compose even start to instantiate mongodb.
This is the github repository of the proyect:
https://github.com/dsalasboscan/exam
I need to instantiate Mongodb first and THEN start with the maven build and java image generation.
I came across similar problem before, and would like to share my experience.
Basically, we need to wait for a while to make sure mongodb is completely boot up, here is the tool that you can leverage. It's fairly easy to use.