Ok, I encountered linking problems while creating jib docker image.
I copy the files i want into container
jib {
allowInsecureRegistries = true
extraDirectories{
paths{
path{
from = file('jnetpcap/jib')
into = '/native'
}
}
}
.
.
.
and in other task, i point to those libraries
task cmdScript(type: CreateStartScripts) {
mainClassName = "cic.cs.unb.ca.ifm.Cmd"
applicationName = "cfm"
outputDir = new File(project.buildDir, 'scripts')
classpath = jar.outputs.files + project.configurations.runtime
defaultJvmOpts = ["-Djava.library.path=/native"]
}
I checked, and those libraries are added correctly to container. It is not a problem with copying libs, but setting up linker.
cmdScript sets correct linker if i build project with distTar, but i don't know how to set up linker when building it with jibDockerBuild.
I couldn't find anwer to my issue here so decided to seek help on SO.
UPDATE
I have found some clues here.
I have updated my jib task by adding
jib {
allowInsecureRegistries = true
extraDirectories{
paths{
path{
from = file('jnetpcap/jib')
into = '/native'
}
}
}
container.jvmFlags = ["-Djava.library.path=/native/*"]
But I keep getting the same error.
error message is
exception in thread main java.lang.unsatisfiedlinkerror 'long com.slytechs.library.NativeLibrary.dlopen(java.lang.String)'
The issue is largely unrelated to Jib. The root cause is missing required libraries inside the container environment.
First things first, it should be container.jvmFlags = ["-Djava.library.path=/native"] (not /native/* with the asterisk).
Now, jNetPcap is a Java wrapper around Libpcap and WinPcap libraries found on various Unix and Windows platforms. That is, on Linux (which is the OS of the container you are building), it depends on Libpcap and requires it to be installed on the system. Most OpenJDK container images (including the one Jib uses as a base image) does not come with Libpcap pre-installed, and I suspect the first problem being that you are not installing Libpcap into the container.
jNetPcap also requires loading other native libraries. In the case of my example below, they were the two .so shared object files that come with the jNetPcap package: libjnetpcap-pcap100.so and libjnetpcap.so.
For explanation, below is the complete example that creates a working container image.
Dockerfile
# This Dockerfile is only for demonstration.
FROM adoptopenjdk/openjdk11
# "libpcap-dev" includes the following files:
# - /usr/lib/x86_64-linux-gnu/libpcap.a
# - /usr/lib/x86_64-linux-gnu/libpcap.so -> libpcap.so.0.8
# - /usr/lib/x86_64-linux-gnu/libpcap.so.0.8 -> libpcap.so.1.8.1
# - /usr/lib/x86_64-linux-gnu/libpcap.so.1.8.1
RUN apt-get update && apt-get install -y libpcap-dev
# My machine is x86_64 running Linux.
RUN curl -o jnetpcap.tgz https://master.dl.sourceforge.net/project/jnetpcap/jnetpcap/1.4/jnetpcap-1.4.r1300-1.linux.x86_64.tgz
# The tar includes the following files:
# - jnetpcap-1.4.r1300/jnetpcap.jar
# - jnetpcap-1.4.r1300/libjnetpcap-pcap100.so
# - jnetpcap-1.4.r1300/libjnetpcap.so
RUN tar -zxvf jnetpcap.tgz
# .class file compiled with "javac -cp jnetpcap.jar MyMain.java"
COPY MyMain.class /my-app/
ENTRYPOINT ["java", "-cp", "/my-app:/jnetpcap-1.4.r1300/jnetpcap.jar", "-Djava.library.path=/jnetpcap-1.4.r1300", "MyMain"]
MyMain.java
import java.util.*;
import org.jnetpcap.*;
public class MyMain {
public static void main(String[] args) {
Pcap.findAllDevs(new ArrayList<>(), new StringBuilder());
System.out.println("SUCCESS!");
}
}
$ docker build -t test .
$ docker run --rm test
SUCCESS!
Therefore, as long as you copy necessary dependent libraries and have correct configuration, you should be able to make it work with Jib.
For installing Libpcap, I can think of a couple options:
Prepare a custom base image (for example, apt-get install libpcap-dev as above) and configure jib.from.image to use it.
Manually download and copy the libpcap.so file into, say, /usr/lib, using the extraDirectories feature. (You can even make your Gradle project dynamically download the file when building your project.)
For copying jNetPcap native libraries (libjnetpcap-pcap100.so and libjnetpcap.so), it's the same story. However, looks like you already manually downloaded and attempted copying them using the extraDirectories feature, so I guess you can continue doing so. But still, preparing a custom base image is another viable option. Note that in the example above, I configured -Djava.library.path=... for jNetPcap (BTW, there are many other ways to have Linux and JVM load shared libraries in an arbitrary directory), but if you copy the .so files into some standard locations (for example, /usr/lib), you wouldn't even need to set -Djava.library.path.
For all of the native libraries (.so files) above, make sure to download the right binaries compatible with the container architecture and OS (probably amd64 and Linux in your case).
Related
I am struggling to understand the startup time difference between JRE based application and JDK based application.
Think that I have an application with sample DockerFile
# syntax=docker/dockerfile:experimental
ARG BASE_IMAGE_FROM=<image>
FROM $BASE_IMAGE_FROM/maven-jdk11:0.0.2 as builder
WORKDIR /builder
COPY settings.xml /settings.xml
COPY . .
RUN --mount=type=cache,id=m2-cache,target=/root/.m2 /root/build.sh <app-name>
## app
FROM $BASE_IMAGE_FROM/<jre-name>
ARG APP_PATH=/builder
ARG DEPENDENCY=${APP_PATH}/target/dependecy
WORKDIR /server
COPY <somethings>
USER <some-user>
# update class to run
ENTRYPOINT exec java ${JAVA_OPTS} -cp app/config:app/lib/setup.jar:app:app/lib/* <app-name>
In the FROM $BASE_IMAGE_FROM/ part of the file, I specify JDK based or JRE based image. The current application is based on java 11 alpine JDK image. I've changed it with JRE based image. There is a big difference in size of the images.
I also thought there should be improvements in the startup time because of the overhead of the JDK. Yet, the startup time didn't change. I couldn't figure it why.
Any opinions?
Both the JDK and the JRE contain the same java runtime. The JDK contains additional tools like the javac compiler, javadoc documentation tool, and so on.
So in your setup, the JRE-based image will be much smaller, but when the container starts up, the ENTRYPOINT java ... starts the same JVM, loads the same class files, and runs the same application.
There's nothing different here other than the disk space required by the JDK functionality you're not using (if you need to pull the image, also the network I/O and time to download the unused files).
Context
I've done a java library that uses a C library with JNI.
The C library is compiled in linux into a .so file. This library needs cap_net_raw capabilities.
Goal
Execute a java process without additional privileges, that uses said java library. The actual processes that are going to use the library are existing processes already in prod and we don't want to give them more rights.
To test that, I've created a jar and run it both with and without sudo. As expected, it succeeds with but fail without it.
Steps to reproduce the test
Create a java class with a native method, let's call it SocketTester.java
static {
System.loadLibrary("SocketTester");
}
private native int socketTest();
Generate socketTester.h file with the command
javac -h . SocketTester.java
Create socketTester.c file that implements socketTester.h and which needs the cap_net_raw capabitily
Compile with
gcc -o libSocketTester.so socketTester.c -shared -I/usr/lib/jvm/java-14-openjdk-amd64/include -I/usr/lib/jvm/java-14-openjdk-amd64/include/linux
Move libSocketTester.so to /usr/lib
Run
sudo ldconfig
Set the cap
cd /usr/lib
sudo setcap cap_net_raw=epi libSocketTester.so
Create a Test.java class
public static void main(final String[] args) {
SocketTester tester = new SocketTester();
tester.socketTest();
}
Create a jar with SocketTester.java and Test.java
Run the test
java -cp socketTester.jar Test
What I've already tried
Adding cap to the .so lib
sudo setcap cap_net_raw=epi libSocketTester.so
Result: Failure
Adding cap to java
sudo setcap cap_net_raw=epi /usr/lib/jvm/java-14-openjdk-amd64/bin/java
Result: It works, but it's not what I want because now all java process have the capability (see bold in goal section).
The question
Why is adding the cap to the .so doesn't work? How else can I accomplish the goal?
A zillion years ago I figured out how to have PAM modules fork a helper program to do privileged things from unprivileged contexts. This is how pam_unix.so is able to invoke unix_chkpwd to help an unprivileged application (a screensaver, or screen) accept a user password to unlock under Linux.
More recently, I learned the trick to making shared library objects (libcap.so, pam_cap.so etc) work as standalone binaries. I've since been thinking about combining both of these techniques... Researching that, I came across this question. Since I was able to do it for the example task of an unprivileged program binding to port 80, I thought it might be of interest as an answer here.
I've done a full write up of how it works on the Fully Capable libcap distribution site, but it essentially boils down to three things:
make the .so file executable as a stand alone program, with its own file capability
include some code in the .so file that figures out its own filename when linked into another program (this uses a _GNU_SOURCE extension: dladdr())
create some mechanism for the library itself to fork/exec itself (I use libcap:cap_launch()) with a private communication mechanism (I use a Unix domain socket generated with socketpair()) back to the app-linked shared library code.
The flow is basically, app calls the .so function, that function invokes the .so file as a forked child and performs the privileged operation. It then returns the result to the app over the Unix domain socket and exits.
I'm struggling with the deployment of a spring app that needs to compile java code during runtime. My app calls the javac command when a user submits a solution to a problem, so it can later run java
I'm deploying to cloud foundry and using the java-buildpack, but unfortunately, it doesn't come with JDK, only JRE is available and that thing has no javac or java commands available.
Do you guys know a way on how to add JDK to cloud foundry, without having to write my own custom buildpack.
Thanks
I would suggest you use multi-buildpack support and use the apt-buildpack to install a JDK. It should work fine alongside the JBP. It just needs to be first in the list.
https://github.com/cloudfoundry/apt-buildpack
Example:
Create an apt.yml.
---
packages:
- openjdk-11-jdk-headless
Bundle that into your JAR, jar uf path/to/your/file.jar apt.yml. It should be added to the root of the JAR, so if you jar tf path/to/your/file.jar you should see just apt.yml and nothing prefixed to it.
Update your manifest.yml. Add the apt-buildpack first in the list.
---
applications:
- name: spring-music
memory: 1G
path: build/libs/spring-music-1.0.jar
buildpacks:
- https://github.com/cloudfoundry/apt-buildpack#v0.2.2
- java_buildpack
Then cf push. You should see the apt-buildpack run and install the JDK. It'll then be installed under ~/deps/0/lib/jvm/java-11-openjdk-amd64. It does not appear to end up on the PATH either, so use a full path to javac or update the path.
I'm setting up a .net Docker image, with some .net code within it. The code, however, needs access to a very specific version of Java Runtime (jre-7u9-windowsx64.exe).
I don't know exactly where to start in adding this executable into my dotnet Dockerfile.
The current Dockerfile for dotnet
FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build-env
WORKDIR /name
# Copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore
# Copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o out
# Build runtime image
FROM mcr.microsoft.com/dotnet/core/aspnet:2.2
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "name.dll"]
I would just like to get the JRE added to the Dockerfile so it is installed or available when Docker spins up.
Because the JRE forms part of your runtime environment, you're going to want to install it into your # Build runtime image.
Your ideal goal is to find the lowest common multiple, sufficient to run both ASP.NET and your .NET code and the JRE and your Java code.
Option #1: Find a (trusted) container image that runs both ASP.NET and JRE
Presumably, you've pursued this approach and been unsuccessful.
Option #1: Start from your currently working solution and add
I'm unfamiliar with mcr.microsoft.com/dotnet/core/aspnet but this may well continue to suffice as your baseline and it has the advantage that you know it will run your .NET app.
So, your process will be to determine what additionally -- if anything -- you'll need to install to be able to run jre-7u9-windowsx64.exe.
A hacky way to determine this would be to insert whatever the installer is for this binary (between lines 13-14). This will take the form RUN the jre-7u9-windowsx64.exe installer. Then, try to build your container and see what errors arise.
A more considered approach would be to identify whether you need to install additional packages to support jre-7u9-windowx64.exe and, if you do, you'll need to install those (using further RUN ...) commands beforehand.
Option #2: Start from a minimal baseline and add
Alternatively, you could start from a more foundational baseline. I assume the OS here is Windows rather than Linux. Presumably there's a minimal Windows container image?
Then you'd need to add whatever is needed to:
Get ASP.NET and your .NET code working
Get JRE and your Java code working
This would provide a more considered foundation for your image but at the cost of requiring you to solve two puzzles.
Option #3: Start from a working JRE image and add
Included for completeness but not-recommended in this case.
On OS X 10.8.2, I'm using JNA and gstreamer-java (through the Eclipse IDE) to load gstreamer libraries. At first I installed gstreamer using the SDK provided by gstreamer.com. However, I uninstalled this and use Macports to install it.
The SDK's libs were installed to:
/System/Library/Frameworks/GStreamer.framework/Versions/0.10-x64/lib
But that directory no longer exists.
Where Macports installed the libs to:
/opt/local/lib
Now, say I want to set java to know of this location via some environment variable. Is this possible? It seems so, because running this line:
System.out.println( System.getProperty("jna.library.path"));
Shows /System/Library/Frameworks/GStreamer.framework/Versions/0.10-x64/lib. But I cannot for the life of me figure out where that got set. It's not set as a runtime VM argument. My $PATH and $DYLD_LIBRARY_PATH, $LD_LIBRARY_PATH do not have this directory set. They actually have the path I want, /opt/local/lib/ set which has no effect it seems. Running env shows no variables with the Framework path either.
So, jna.library.path. Is it set externally? How can I change it--without setting it at runtime or via java command-line arguments? Zero points for telling me to symlink.
EDIT:
Searching through gstreamer-java's files led me to gstreamer-java.spec, which has this line:
sed -i [...] -e "s,\(run.jvmargs=-Djna.library.path=\).*,\1%{_libdir}:$(pkg-config --variable=pluginsdir gstreamer-0.10),"
Running the contained command pkg-config --variable=pluginsdir gstreamer-0.10 gets me this:
/opt/local/lib/gstreamer-0.10
Which is the correct path for plugins. Further running pkg-config --print-variables gstreamer-0.10 gives me all these:
typelibdir
datarootdir
exec_prefix
pluginsdir
datadir
prefix
libdir
includedir
girdir
toolsdir
Which are all at or under the correct /opt/local/ directory.
Have you tried running outside of Eclipse?