We Keep Coding

Modifying container cacerts at runtime

I have an containerized JVM application (multiple actually) running on kubernetes, which needs to trust additional custom/private CAs (which are not known beforehand, the application will be deployed in multiple unrelated data-centers that have their own PKI).
The cacerts in the base image are not writable at runtime.
Currently I see these options:
do not provide an option to modify cacerts, force the DCs to manage & inject their own cacert files via container volumes.
make cacerts file writeable at runtime and modify cacerts in entrypoint
do not use JDK TLS - set truststore at "client" level (e.g. CXF)
...?
Under the assumption the DCs will not run JVM apps only, they will not like to manage cacerts themselves, because cacerts is specific to JVM and they then potentially need to do that for every technology. Thus I do not really like that option.
The second option seems to be a quite pragmatic one - but I suspect that making the cacerts writable at runtime is a bad practice because an attacker could modify configuration s/he should not be able to.
The third option has it's limitations and intricacies because you need to make each each and every client configurable. (In case of CXF for example fetching the initial WSDL file does not seem to covered by CXF but by the JVM...) But this means if your client is not (properly) configurable this does not work.
Thus I am back at option 2.
My questions would be:
Is it a bad practice to have cacerts writeable at runtime?
Is there an option I missed that allows injecting (arbitrary) additional CAs into cacerts without making it writeable at runtime?
Note: I have asked about this on Security Stack Exchange, too, but got no response there.

Have you considered a combination of an init container and an emptyDir volume which is readOnly: true in the application container?
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
spec:
selector:
matchLabels:
app: app
template:
metadata:
labels:
app: app
spec:
initContainers:
- name: cacerts
image: cacerts-importer
volumeMounts:
- name: cacerts
mountPath: /media/cacerts
- name: cacert-imports
mountPath: /media/cacert-imports
containers:
- name: app
image: app:v1
volumeMounts:
- name: cacerts
mountPath: /media/cacerts
readOnly: true
volumes:
- name: cacerts
emptyDir: {}
- name: cacert-imports
configMap:
name: cacert-imports

Dockerized Spring Boot app not using mounted Kubernetes ConfigMap (application.properties)

I have a problem where in my dockerized Spring Boot application is not using the application.properties I stored in a configMap.
However, I can see and confirm that my configMap has been mounted properly in the right directory of my Spring Boot app when I enter the pod's shell.
Note that I have an application.properties by default wherein Kubernetes mounts / overwrites it later on.
It seems that the Spring Boot uses the first application.properties and when k8s overwrites it, apparently, it doesn't use it.
It seems that, apparently, what happens is:
run the .jar file inside the Dockerized Spring Boot app
use the first/default application.properties file on runtime
Kubernetes proceeds to mount the configMap
mount / overwrite success, but how will Spring Boot use this one since it's already running?
Here is the Dockerfile of my Spring Boot / Docker image for reference:
FROM maven:3.5.4-jdk-8-alpine
# Copy whole source code to the docker image
# Note of .dockerignore, this ensures that folders such as `target` is not copied
WORKDIR /usr/src/myproject
COPY . /usr/src/myproject/
RUN mvn clean package -DskipTests
WORKDIR /usr/src/my-project-app
RUN cp /usr/src/myproject/target/*.jar ./my-project-app.jar
EXPOSE 8080
CMD ["java", "-jar", "my-project-app.jar"]
Here's my Kubernetes deployment .yaml file for reference:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-project-api
namespace: my-cluster
labels:
app: my-project-api
spec:
replicas: 1
selector:
matchLabels:
app: my-project-api
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
app: my-project-api
spec:
containers:
- name: my-project-api
image: "my-project:latest"
imagePullPolicy: Always
env:
.
.
.
volumeMounts:
- name: my-project-config
mountPath: /usr/src/my-project/my-project-service/src/main/resources/config/application.properties
ports:
- containerPort: 8080
name: my-project-api
protocol: TCP
volumes:
# Name of the volume
- name: my-project-config
# Get a ConfigMap with this name and attach to this volume
configMap:
name: my-project-config
And my configMap for reference:
kind: ConfigMap
apiVersion: v1
data:
application.properties: |-
# This comment means that this is coming from k8s ConfigMap. Nice!
server.port=8999
.
.
.
.
metadata:
name: my-project-config
namespace: my-cluster
Any help is greatly appreciated... Thank you so much.. :)

The thing is that /src/main/resources/application.properties that your application uses is the one that is inside the jar file by default. If you open your jar, you should see it there.
That being said, your expectations to mount a /src/main/resources directory where your jar is are not going to be fulfilled, unfortunately.
These are the docs you should be looking at.
I won't go into much detail as it's explained pretty good in the docs but I will say that you are better off explicitly declaring your config location so that new people on the project know from where the config is coming from right off the bat.
You can do something like this:
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-project-api
labels:
app: my-project-api
spec:
selector:
matchLabels:
app: my-project-api
template:
metadata:
labels:
app: my-project-api
spec:
containers:
- name: my-project-api
image: "my-project:latest"
imagePullPolicy: Always
env:
- name: JAVA_OPTS
value: "-Dspring.config.location=/opt/config"
.
.
.
volumeMounts:
- name: my-project-config
mountPath: /opt/config
ports:
- containerPort: 8080
volumes:
- name: my-project-config
configMap:
name: my-project-config
Hope that helps,
Cheers!

I did slightly differently. I made sure I have mounted application.properties at config/. i.e; below is my example mounted application.properties (below commands show the values in pod - i.e; after kubectl exec -it into the pod)
/ # pwd
/
/ # cat config/application.properties
logback.access.enabled=false
management.endpoints.web.exposure.include=health, loggers, beans, configprops, env
Basically, the trick is based on the link in the above answer. Below is an excerpt from the link in which it does say application.properties will be picked from config/. So, I made sure my environment (dev, test, prod) specific config map was mounted at config/. Do note there is precedence for the below list (per the link: locations higher in the list override lower items)
A /config subdir of the current directory.
The current directory
A classpath /config package
The classpath root
Below is the config map definition (just pasted data section)
data:
application.properties: |+
logback.access.enabled={{.Values.logacbkAccessEnabled}}
management.endpoints.web.exposure.include=health, loggers, beans, configprops, env
And you can also see from actuator/env endpoint SpringBootApp did pick those values.
{
"name": "Config resource 'file [config/application.properties]' via location 'optional:file:./config/'",
"properties": {
"logback.access.enabled": {
"value": "false",
"origin": "URL [file:config/application.properties] - 1:24"
},
"management.endpoints.web.exposure.include": {
"value": "health, loggers, beans, configprops, env",
"origin": "URL [file:config/application.properties] - 2:43"
}
}
},

Provide javax.net.ssl.trustStorePassword from file (docker swarm setup) instead of environment property

I would like to inject a Truststore password to javax.net from a docker secret in a docker swarm setup.
Currently I have:
service-1:
image: service-1:${BUILD_VERSION:-latest}
deploy:
mode: global
environment:
- JAVA_OPTS=-Djavax.net.ssl.trustStore=/run/secrets/truststore -Djavax.net.ssl.trustStorePassword=changeit # <=== this is what I would like to change!
# later:
secrets:
truststore:
file: /opt/docker-secrets/trust-store.ts
truststore_pwd:
file: /opt/docker-secrets/trust-store-pwd.ts
Any ideas, on how to get rid of the line starting with JAVA_OPT (especially the parameter javax.net.ssl.trustStorePassword and to integrate this as a secret as well?
Thanks!

Single docker image thats being deployed as 2 different services thru kubernetes and helm. Change context path of app

We have a single docker image thats being deployed as 2 different services thru kubernetes and helm with names like "ServiceA" and "ServiceB". At the point deploy happens need to set the context path of Tomcat to be something different like /ServiceA and /ServiceB. How can this be done ? is there anything that can be set directly on the yaml ?
ex: Looks like below
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "fullname" . }}-bg
{{- include "labels" . }}
spec:
replicas: {{ .replicaCount }}
selector:
matchLabels:
app.kubernetes.io/name: {{ include "name" . }}-bg
app.kubernetes.io/instance: {{ .Release.Name }}
strategy:
type: Recreate
rollingUpdate: null
template:
metadata:
labels:
app.kubernetes.io/name: {{ include "name" . }}-bg
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
containers:
- name: {{ .Chart.Name }}
image: "{{ .image.repository }}:{{ .image.tag }}"
imagePullPolicy: {{ .image.pullPolicy }}
env:
- name: SERVICE_NAME
value: "ServiceB"
- name: background.jobs.enabled
value: "true"
envFrom:
- configMapRef:
name: {{ include "commmonBaseName" . }}-configmap
-

There are few approaches into setting up the context path of an app.
From the app itself: This depends on the language/framework/runtime your application uses. For example, if it's a traditional Java web application that runs on Tomcat, it would be served by default from the context path of the name of the .war file you put in the webapp directory. Or, if it is a Spring Boot 2.X app, you could set it up with the Spring Boot property server.servlet.context-path, which can be also passed via an environment variable, specifically SERVER_SERVLET_CONTEXT_PATH. So, to give an example, in the container in your deployment pod spec:
env:
- name: SERVER_SERVLET_CONTEXT_PATH
value: "/ServiceB"
However, this kind of app-specific settings are most of the times not needed in Kubernetes, since you can handle those concerns on the outer layers.
Using Ingress objects: If you have an Ingress controller running and properly configured, you can create an Ingress that will manage path prefix stripping, and other HTTP Layer7 concerns. This means, you can leave your application itself as-is (like serving from root context /) but configure the context path from the Ingress. An example is the following, assuming you use Nginx Ingress Controller
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: service-a-ingress
annotations:
ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: service-a.yourdomain.com
http:
paths:
- path: /ServiceA/(.*)
backend:
serviceName: service-a
servicePort: service-a-port
Note the capture group (.*) in the path, and $1 in the rewrite target - it will rewrite the request paths like /ServiceA/something to /something before forwarding the packet to your backend.
See this page to lear more about ingresses.
You can use an HTTP router software such as skipper to handle all this HTTP traffic configuration in-cluster.
If you use a service mesh solution such as Istio, they give you many ways to manage the traffic inside the mesh.

reference variables from pom.xml to deployment.yaml

i want to use maven pom.xml variables in a kubernetes deployment.yaml file. the variables i want to reference are ${project.artifactId} and ${project.version} which is pulled from
pom.xml
<groupId>my-project</groupId>
<artifactId>>my-project</artifactId>
<version>1.0.0-SNAPSHOT</version>
and this is what i want to achieve
deploment.yaml
apiVersion: v1
kind: Pod
spec:
containers:
- name: my-project
image: ${project.artifactId}:${project.version}
with this attempt i get an InvalidImageName error.
please advice on which better way of doing this.

I would say its issue with your deploment.yaml content.
I have use it (with nginx image) on K8s and get error below:
error: error when retrieving current configuration of:
Resource: "/v1, Resource=pods", GroupVersionKind: "/v1, Kind=Pod"
Name: "", Namespace: "default"
Object: &{map["apiVersion":"v1" "kind":"Pod" "spec":map["containers":[map["name":"test" "image":"nginx"]]] "metadata":map["namespace":"default" "annotations":map["kubectl.kubernetes.io/last-applied-configuration":""]]]}
from server for: "pod.yaml": resource name may not be empty
In your current file you have named only container. You have to specify your POD name using metadata.name. In metadata section you can also specify namespace.
apiVersion: v1
kind: Pod
metadata:
name: nginx-pod
spec:
containers:
- name: nginx
image: nginx
In addition keep in mind that kind: Pod and kind: Deployment are two different things (bit confused regarding your YAML file name).