We Keep Coding

Executing command-line calls inside a Docker container from Java

I'm currently working on a microservice using the Quarkus framework for Java. Quarkus allows directly building a container which runs the application in JVM mode using Maven. This works very well, I can start the container and then query the exposed port from my host machine. The image is based on the ubi-minimal image provided by RedHat.
For data processing I need a utility called tabix which is included in a package called htslib. My first hunch was to install this inside of the container so whatever machine starts the container does not need to have tabix/htslib installed:
FROM registry.access.redhat.com/ubi8/ubi-minimal:8.3
ARG JAVA_PACKAGE=java-11-openjdk-headless
ARG RUN_JAVA_VERSION=1.3.8
ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en'
# Install java and the run-java script
# Also set up permissions for user `1001`
RUN microdnf install curl ca-certificates ${JAVA_PACKAGE} make tar gcc bzip2 \
&& microdnf update \
&& microdnf clean all \
&& mkdir /deployments \
&& chown 1001 /deployments \
&& chmod "g+rwX" /deployments \
&& chown 1001:root /deployments \
&& curl https://repo1.maven.org/maven2/io/fabric8/run-java-sh/${RUN_JAVA_VERSION}/run-java-sh-${RUN_JAVA_VERSION}-sh.sh -o /deployments/run-java.sh \
&& chown 1001 /deployments/run-java.sh \
&& chmod 540 /deployments/run-java.sh \
&& echo "securerandom.source=file:/dev/urandom" >> /etc/alternatives/jre/conf/security/java.security
# Configure the JAVA_OPTIONS, you can add -XshowSettings:vm to also display the heap size.
ENV JAVA_OPTIONS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
# We make four distinct layers so if there are application changes the library layers can be re-used
COPY --chown=1001 target/quarkus-app/lib/ /deployments/lib/
COPY --chown=1001 target/quarkus-app/*.jar /deployments/
COPY --chown=1001 target/quarkus-app/app/ /deployments/app/
COPY --chown=1001 target/quarkus-app/quarkus/ /deployments/quarkus/
# Set up HTSLib
RUN mkdir /package \
&& chown 1001 /package \
&& chmod g+rwX /package \
&& chown 1001:root /package \
&& curl -LJ https://github.com/samtools/htslib/releases/download/1.12/htslib-1.12.tar.bz2 -o /package/htslib.zip \
&& tar xjvf /package/htslib.zip \
&& ls -l
# Install HTSLib
WORKDIR /htslib-1.12/
RUN microdnf install zlib-devel bzip2-devel xz-devel \
&& microdnf update \
&& ./configure
RUN make \
&& make install
EXPOSE 8085
USER 1001
ENTRYPOINT [ "/deployments/run-java.sh" ]
This seems to work, at least I can call tabix when I shell into the container. However, the problem is that now I want to call it directly from the Java code. I tried using a ProcessBuilder:
String[] command = new String[] {
"tabix",
FILEPATH+filename
};
ProcessBuilder pb = new ProcessBuilder(command);
pb.redirectErrorStream(true);
pb.redirectInput(ProcessBuilder.Redirect.INHERIT);
try {
final Process p = pb.start();
p.waitFor();
} catch (Exception e) {
System.out.println("Exception occurred while creating tabix index file");
}
What I want the method here to do is to create an indexing file (.tbi) from a variant call format file that I mount into the container. This works on my machine, which has tabix installed. However, when I try doing this on another machine, it does not work and delivers no output even though I'm using the exact same container image and files. This is why I suspect that the ProcessBuilder is actually calling tabix on the host machine instead of inside the docker container, which seems pretty weird/unsafe to me, but I'm struggling to come up with another explanation.
In any case, I would appreciate any help on how to execute the process inside the container, if that is at all possible, or any other advice.
Thank you!

Adding GDAL to openjdk-alpine based container

I have Spring boot application running on container. The Dockerfile for it is like this:
FROM openjdk:8-jdk-alpine
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
CMD ["java", "-jar", "/app.jar"]
In the application there is ogr2ogr command so I want to install GDAL package to container.
I have no idea how I can do this.I feel that I approach the solution the most here.
In this solution, proj was in testing branch of Alpine but now it is in community branch as I understand. So, I added the dependencies of proj and modified Dockerfile like this:
FROM openjdk:8-jdk-alpine
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENV ROOTDIR /usr/local
ENV LD_LIBRARY_PATH /usr/local/lib
ENV GDAL_VERSION 3.1.2
ENV OPENJPEG_VERSION 2.3.0
# Load assets
WORKDIR $ROOTDIR/
RUN mkdir -p $ROOTDIR/src
RUN wget -qO- \
http://download.osgeo.org/gdal/${GDAL_VERSION}/gdal-${GDAL_VERSION}.tar.gz | \
tar -xzC $ROOTDIR/src/
RUN wget -qO- \
https://github.com/uclouvain/openjpeg/archive/v${OPENJPEG_VERSION}.tar.gz | \
tar -xzC $ROOTDIR/src/
RUN set -ex \
# Install runtime dependencies
&& apk add --update --no-cache \
git \
bash-completion \
python-dev \
python3-dev \
libffi-dev \
jpeg-dev \
openjpeg-dev \
libpng-dev \
linux-headers \
curl-dev \
musl \
libjpeg-turbo \
libcurl \
zlib \
libgcc \
libstdc++ \
musl \
sqlite-libs \
tiff \
&& apk add --no-cache \
--repository http://dl-cdn.alpinelinux.org/alpine/edge/community \
proj \
proj-dev \
# Install common build dependencies
&& apk add --no-cache --virtual .build-deps \
gcc \
cmake \
build-base \
make \
swig \
apache-ant \
# Compile and install OpenJPEG
&& cd src/openjpeg-${OPENJPEG_VERSION} \
&& mkdir build && cd build \
&& cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$ROOTDIR \
&& make -j3 && make -j3 install && make -j3 clean \
&& cd $ROOTDIR && rm -Rf src/openjpeg* \
# Compile and install GDAL
&& cd src/gdal-${GDAL_VERSION} \
&& ./configure --with-python --with-curl --with-java=${JAVA_HOME} --with-openjpeg=$ROOTDIR \
&& make -j3 && make -j3 install \
# Compile Python and Java bindings for GDAL
&& cd $ROOTDIR/src/gdal-${GDAL_VERSION}/swig/java && make -j3 && make -j3 install \
&& cd $ROOTDIR/src/gdal-${GDAL_VERSION}/swig/python \
&& python3 setup.py build \
&& python3 setup.py install \
&& cd $ROOTDIR && rm -Rf src/gdal* \
# Remove build dependencies
&& apk del .build-deps \
CMD ["java", "-jar", "/app.jar"]
Then the output of docker build is:
15 176.1 checking for ld used by GCC... /usr/x86_64-alpine-linux-musl/bin/ld -m elf_x86_64
#15 176.1 checking if the linker (/usr/x86_64-alpine-linux-musl/bin/ld -m elf_x86_64) is GNU ld... yes
#15 176.1 checking for shared library run path origin... /bin/sh: can't open './config.rpath': No such file or directory
#15 176.1 done
#15 176.1 checking for iconv... yes
#15 176.2 checking for working iconv... yes
#15 176.2 checking for iconv declaration...
#15 176.2 extern size_t iconv (iconv_t cd, char * *inbuf, size_t *inbytesleft, char * *outbuf, size_t *outbytesleft);
#15 176.3 using ICONV_CPP_CONST=""
#15 176.3 configure: Bash completions not requested
#15 176.3 checking for PROJ >= 6 library... checking for proj_create_from_wkt in -lproj... no
#15 176.3 checking for internal_proj_create_from_wkt in -lproj... no
#15 176.4 checking for internal_proj_create_from_wkt in -linternalproj... no
#15 176.4 configure: error: PROJ 6 symbols not found
------
executor failed running [/bin/sh -c set -ex && apk add --update --no-cache git bash-completion python-dev python3-dev libffi-dev jpeg-dev openjpeg-dev
libpng-dev linux-headers curl-dev musl libjpeg-turbo libcurl zlib libgcc libstdc++ musl sqlite-libs tiff && apk add --no-cache
--repository http://dl-cdn.alpinelinux.org/alpine/edge/community proj proj-dev && apk add --no-cache --virtual .build-deps gcc cmake build-base make
swig apache-ant && cd src/openjpeg-${OPENJPEG_VERSION} && mkdir build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$ROOTDIR && make -j3 && make -j3 install &
& make -j3 clean && cd $ROOTDIR && rm -Rf src/openjpeg* && cd src/gdal-${GDAL_VERSION} && ./configure --with-python --with-curl --with-java=${JAVA_HOME} --with-openjpeg=$ROOTDIR && make -j3 && m
ake -j3 install && cd $ROOTDIR/src/gdal-${GDAL_VERSION}/swig/java && make -j3 && make -j3 install && cd $ROOTDIR/src/gdal-${GDAL_VERSION}/swig/python && python3 setup.py build && python3 setup.p
y install && cd $ROOTDIR && rm -Rf src/gdal* && apk del .build-deps CMD ["java", "-jar", "-Dlogging.config=/logback.xml", "/app.jar"]]: exit code: 1
Let me emphasize that I do not know about Alpine Linux and GDAL. I tried to do something based on a solution I found.

OpenJDK 15 and Docker

I want to get Docker running with JDK 15 I am following https://github.com/markhobson/docker-maven-chrome/blob/master/jdk-15/Dockerfile but I get ADD failed: stat /var/lib/docker/tmp/docker-builder165793576/google-chrome.repo: no such file or directory
My Docker file is:
FROM maven:3.6.3-openjdk-15
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
## Google Chrome
ARG CHROME_VERSION=87.0.4280.88-1
ADD google-chrome.repo /etc/yum.repos.d/google-chrome.repo
RUN microdnf install -y google-chrome-stable-$CHROME_VERSION \
&& sed -i 's/"$HERE\/chrome"/"$HERE\/chrome" --no-sandbox/g' /opt/google/chrome/google-chrome
# ChromeDriver
ARG CHROME_DRIVER_VERSION=87.0.4280.88
RUN microdnf install -y unzip \
&& curl -s -o /tmp/chromedriver.zip https://chromedriver.storage.googleapis.com/$CHROME_DRIVER_VERSION/chromedriver_linux64.zip \
&& unzip /tmp/chromedriver.zip -d /opt \
&& rm /tmp/chromedriver.zip \
&& mv /opt/chromedriver /opt/chromedriver-$CHROME_DRIVER_VERSION \
&& chmod 755 /opt/chromedriver-$CHROME_DRIVER_VERSION \
&& ln -s /opt/chromedriver-$CHROME_DRIVER_VERSION /usr/bin/chromedriver
###
ENTRYPOINT ["java","-jar","/app.jar"]
Does anyone know how to fix? I do not have a file google-chrome.repo - does anyone know the best way to get that?

I got it to work with the following Docker File:
FROM maven:3.6.3-openjdk-15
# Google Chrome
ARG CHROME_VERSION=87.0.4280.88-1
ADD google-chrome.repo /etc/yum.repos.d/google-chrome.repo
RUN microdnf install -y google-chrome-stable-$CHROME_VERSION \
&& sed -i 's/"$HERE\/chrome"/"$HERE\/chrome" --no-sandbox/g' /opt/google/chrome/google-chrome
## ChromeDriver
ARG CHROME_DRIVER_VERSION=87.0.4280.88
RUN microdnf install -y unzip \
&& curl -s -o /tmp/chromedriver.zip https://chromedriver.storage.googleapis.com/$CHROME_DRIVER_VERSION/chromedriver_linux64.zip \
&& unzip /tmp/chromedriver.zip -d /opt \
&& rm /tmp/chromedriver.zip \
&& mv /opt/chromedriver /opt/chromedriver-$CHROME_DRIVER_VERSION \
&& chmod 755 /opt/chromedriver-$CHROME_DRIVER_VERSION \
&& ln -s /opt/chromedriver-$CHROME_DRIVER_VERSION /usr/bin/chromedriver
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
where google-chrome.repo is:
[google-chrome]
name=google-chrome
baseurl=http://dl.google.com/linux/chrome/rpm/stable/x86_64
enabled=1
gpgcheck=1
gpgkey=https://dl.google.com/linux/linux_signing_key.pub

How to download specific version in dockerfile?

I'm new at writing dockers. I need to download specific java version for it:
FROM alpine:3.9
RUN apk add bash && apk add openjdk8 && apk add R && apk add perl
this is working, however it downloads the latest version of java - 1.8.0_212.
I need the specific version 1.8.0_171, so I tried the below and it didn't work:
RUN apk add bash && apk add openjdk8=1.8.0_171 && apk add R && apk add perl
Anyone know how to get the specific version?

Try this out
ARG JAVA_VERSION_MAJOR=8
ARG JAVA_VERSION_MINOR=131
ARG JAVA_VERSION_BUILD=11
ARG JAVA_PACKAGE=server-jre
ARG JAVA_SHA256_SUM=a80634d17896fe26e432f6c2b589ef6485685b2e717c82cd36f8f747d40ec84b
ARG JAVA_URL_ELEMENT=d54c1d3a095b4ff2b6607d096fa8016
# Download and unarchive Java
RUN apk add --update curl && \
mkdir -p /opt
RUN wget -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/${JAVA_VERSION_MAJOR}u${JAVA_VERSION_MINOR}-b${JAVA_VERSION_BUILD}/${JAVA_URL_ELEMENT}/${JAVA_PACKAGE}-${JAVA_VERSION_MAJOR}u${JAVA_VERSION_MINOR}-linux-x64.tar.gz
RUN mv ${JAVA_PACKAGE}-${JAVA_VERSION_MAJOR}u${JAVA_VERSION_MINOR}-linux-x64.tar.gz java.tar.gz
RUN gunzip -c java.tar.gz | tar -xf - -C /opt && rm -f java.tar.gz && \
ln -s /opt/jdk1.${JAVA_VERSION_MAJOR}.0_${JAVA_VERSION_MINOR} /opt/jdk
RUN apk add unzip
RUN curl -L -C - -b "oraclelicense=accept-securebackup-cookie" -O http://download.oracle.com/otn-pub/java/jce/8/jce_policy-8.zip && \
unzip jce_policy-8.zip -d /tmp && \
cp /tmp/UnlimitedJCEPolicyJDK8/*.jar /opt/jdk/jre/lib/security/ && \
rm -rf jce_policy-8.zip /tmp/UnlimitedJCEPolicyJDK8 && \
apk del curl && \
rm -rf /var/cache/apk/*
ENV JAVA_HOME /opt/jdk
ENV PATH ${PATH}:${JAVA_HOME}/bin

this worked for me :
ENV JAVA_VERSION 10.0.2 # your java version
ENV JAVA_HOME /java
RUN apt-get update
RUN apt-get install -y wget cron vim unzip bzip2 && apt-get clean
RUN wget -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/${JAVA_VERSION_MAJOR}u${JAVA_VERSION_MINOR}-b${JAVA_VERSION_BUILD}/${JAVA_URL_ELEMENT}/${JAVA_PACKAGE}-${JAVA_VERSION_MAJOR}u${JAVA_VERSION_MINOR}-linux-x64.tar.gz \
&& mkdir -p $JAVA_HOME \
&& tar -xvzf jdk-10.0.2_linux-x64_bin.tar.gz -C $JAVA_HOME

Setting Up DB2 in Airflow

I am trying to connect DB2 as a data connection for airflow which is residing in a Docker Container (realizing that this is not nativly supported). I am developing on a Mac
I have added the connectionas seen in the below screenshot where the URL is host:port/databse.
I am then going to the Data Profiling > Ad Hoc Query to try and test the connection and I get the below.
In order to make sure the drivers were available, I mounted the folder where the jdbc driver is located to /usr/local/airflow/drivers in the docker-compose file.
I also made sure to include the below packages in my requirements.txt as these were required when I query from a jupyter notebook.
sasl
thrift_sasl
jaydebeapi
jpype1
ibm_db
ibm_db_sa
I can't figure out what I'm missing.
I've been through:
Airflow documentation
Unable to setup a DB2 / DashDB JDBC Connection in Apache Airflow
Apache Airflow db2 Connection
Apache Airflow db2 Connection
https://github.com/puckel/docker-airflow
Lots of GitHub issues
and many more resources but can't find anything that solves this
Here is my current Dockerfile. As indicated in the comments, JVM isn't installed in the Dockerfile so that may be the issue.
# VERSION 1.10.1
# AUTHOR: Matthieu "Puckel_" Roisil
# DESCRIPTION: Basic Airflow container
# BUILD: docker build --rm -t puckel/docker-airflow .
# SOURCE: https://github.com/puckel/docker-airflow
FROM python:3.6-slim
LABEL maintainer="Puckel_"
# Never prompts the user for choices on installation/configuration of packages
ENV DEBIAN_FRONTEND noninteractive
ENV TERM linux
# Airflow
ARG AIRFLOW_VERSION=1.10.1
ARG AIRFLOW_HOME=/usr/local/airflow
ARG AIRFLOW_DEPS=""
ARG PYTHON_DEPS=""
ENV AIRFLOW_GPL_UNIDECODE yes
# Define en_US.
ENV LANGUAGE en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8
ENV LC_CTYPE en_US.UTF-8
ENV LC_MESSAGES en_US.UTF-8
# Java
RUN apt-get update && apt-get install -y openjdk-7-jre-headless wget \
&& apt-get clean
ENV JAVA_HOME /usr/lib/jvm/java-7-openjdk-amd64
RUN set -ex \
&& buildDeps=' \
freetds-dev \
libkrb5-dev \
libsasl2-dev \
libssl-dev \
libffi-dev \
libpq-dev \
git \
' \
&& apt-get update -yqq \
&& apt-get upgrade -yqq \
&& apt-get install -yqq --no-install-recommends \
$buildDeps \
freetds-bin \
build-essential \
default-libmysqlclient-dev \
apt-utils \
curl \
rsync \
netcat \
locales \
&& sed -i 's/^# en_US.UTF-8 UTF-8$/en_US.UTF-8 UTF-8/g' /etc/locale.gen \
&& locale-gen \
&& update-locale LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 \
&& useradd -ms /bin/bash -d ${AIRFLOW_HOME} airflow \
&& pip install -U pip setuptools wheel \
&& pip install pytz \
&& pip install pyOpenSSL \
&& pip install ndg-httpsclient \
&& pip install pyasn1 \
&& pip install apache-airflow[crypto,celery,postgres,hive,jdbc,mysql,ssh${AIRFLOW_DEPS:+,}${AIRFLOW_DEPS}]==${AIRFLOW_VERSION} \
&& pip install 'redis>=2.10.5,<3' \
&& if [ -n "${PYTHON_DEPS}" ]; then pip install ${PYTHON_DEPS}; fi \
&& apt-get purge --auto-remove -yqq $buildDeps \
&& apt-get autoremove -yqq --purge \
&& apt-get clean \
&& rm -rf \
/var/lib/apt/lists/* \
/tmp/* \
/var/tmp/* \
/usr/share/man \
/usr/share/doc \
/usr/share/doc-base
COPY script/entrypoint.sh /entrypoint.sh
COPY config/airflow.cfg ${AIRFLOW_HOME}/airflow.cfg
COPY requirements.txt ${AIRFLOW_HOME}/requirements.txt
RUN pip install --upgrade pip && pip install -r requirements.txt
RUN chown -R airflow: ${AIRFLOW_HOME}
EXPOSE 8080 5555 8793
USER airflow
WORKDIR ${AIRFLOW_HOME}
ENTRYPOINT ["/entrypoint.sh"]
CMD ["webserver"] # set default arg for entrypoint

The jpype1 module requires a JVM, accessible on your $PATH -- try installing one and try again.

Finally figured it out. Below is what I ended up using to get Java installed. then I just mounted the folder with my driver in it.
# Java
RUN mkdir -p /usr/share/man/man1 && \
(echo "deb http://http.debian.net/debian stretch main" > /etc/apt/sources.list.d/backports.list) && \
apt-get update -y \
&& apt-get install --no-install-recommends -y build-essential libkrb5-dev libsasl2-dev libffi-dev default-libmysqlclient-dev vim-tiny gosu krb5-user openjdk-8-jre openjdk-8-jdk-headless openjdk-8-jdk openjdk-8-jre-headless \
&& apt-get clean
RUN apt-get install unzip -y && \
apt-get autoremove -y