We Keep Coding

Fabric8 KubernetesClient sends volumeName as empty string for PersistentVolumeClaim causing Forbidden: spec is immutable after creation

I want to apply the following yaml multiple times with the fabric8 kubernetes-client
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
storageClassName: my-storage-class
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
I apply the yaml using createOrReplace()
Config config = new ConfigBuilder()
.withMasterUrl("https://my-kubernetes-root:6443")
.withNamespace("my-namespace")
.withOauthToken(token)
.withTrustCerts(true)
.build();
KubernetesClient client = new DefaultKubernetesClient(config);
ClasspathResource resource = new ClasspathResource("my-pvc.yaml");
client.load(resource.getInputStream()).createOrReplace(); // this works
TimeUnit.MINUTES.sleep(1); // volumeName is dynamically assigned during this period
client.load(resource.getInputStream()).createOrReplace(); // this fails
This works the first time (when the PVC does not exist in the namespace) but fails the second time that createOrReplace() is called for the same yaml with the following error
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://my-kubernetes-root:6443/api/v1/namespaces/my-namespace/persistentvolumeclaims/my-pvc. Message: PersistentVolumeClaim "my-pvc" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
core.PersistentVolumeClaimSpec{
AccessModes: []core.PersistentVolumeAccessMode{"ReadWriteMany"},
Selector: nil,
Resources: core.ResourceRequirements{Requests: core.ResourceList{s"storage": {i: resource.int64Amount{value: 1073741824}, s: "1Gi", Format: "BinarySI"}}},
- VolumeName: "",
+ VolumeName: "pvc-b79ebfcb-d5cb-4450-9f17-d69ec10b8712",
StorageClassName: &"my-storage-class",
VolumeMode: &"Filesystem",
DataSource: nil,
}
Notice how "volumeName" is not present in the yaml (nil) but in the error message "volumeName" is changing from empty string to the dynamically assigned volumeName.
I can reproduce this exact same behavior using kubectl and empty string for volumeName
I can kubectl apply the following yaml as many times as I like
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
storageClassName: my-storage-class
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
But if I kubectl apply a yaml with volumeName of empty string it works the first time and fails the second time (The error message is the same as above)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
storageClassName: my-storage-class
volumeName: ""
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
How can I get KubernetesClient to behave the same as kubectl apply? Is there any way that I can apply the same PersistentVolumeClaim yaml multiple times with KubernetesClient?

As a workaround, I have switched to using a StatefulSet to manage my Pod which allows me to specify volumeClaimTemplates
eg the following yaml can be applied multiple times:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx
serviceName: "nginx"
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: k8s.gcr.io/nginx-slim:0.8
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "my-storage-class"
resources:
requests:
storage: 1Gi

Kubernetes API doesn't allow you to change some fields for a certain resource type at a certain state. For a PVC, after its state is Bound, the volumeName field in its spec is immutable; you can't change the volume your PVC is referencing.
When you apply a bare PersistentVolumeClaim manifest like the one in your question (a manifest that has no volumeName key or the value of its volumeName is nil), If the cluster has a default StorageClass, it automatically creates a corresponding PersistentVolume object that's the abstraction of your volume in Kubernetes from the underlying storage provider, then assigns its name to your PersistentVolumeClaim with the volumeName key.
After a PersistentVolumeClaim gets bound to a PersistentVolume and its volumeName field gets populated, you can not edit or patch its volumeName field anymore; that's the reason your kubectl apply command fails for the second time.
It's now clear that a Replace verb is not gonna work for a PVC in Bound state. You can Create one and edit or Patch 'some' of its fields (like increasing the size). I'm not familiar with Java SDK, but it should include some APIs for creating, getting, patching, and deleting a PVC.
You can first check for the existence of your PVC using the GetNamespacedPersistentVolumeClaim API; then, if it's not in the cluster, create it using CreateNamespacedPersistentVolumeClaim API. Referencing the axiom: "Explicit is better than Implicit."

I raised this issue on the facric8 kubernetes-client codebase and got an interesting response which may do what I want. Please note that I haven't tried yet since I have a workaround using StatefulSet and volumeClaimTemplates (see my other answer)
The problem is the replace operation. It is not a substitute for apply / patch. Instead of createOrReplace you can use the new support for server side apply
And the relevant section from the link
Server Side Apply Basic usage of server side apply is available via
Patchable. At it's simplest you just need to call:
client.services().withName("name").patch(PatchContext.of(PatchType.SERVER_SIDE_APPLY), service);
For any create or update. This can be a good alternative to
using createOrReplace as it is always a single api call and does not
issue a replace/PUT which can be problematic.
If the resources may be created or modified by something other than a
fabric8 patch, you will need to force your modifications:
client.services().withName("name").patch(new PatchContext.Builder().withPatchType(PatchType.SERVER_SIDE_APPLY).withForce(true).build(), service);

RabbitMQ throws an exception when sending to dead letter queue (DLQ) (maximum frame_size)

When I got a message from the queue and if an exception was thrown, I wanted to get the message again.
So, I create my consumer with a DLQ queue:
spring:
cloud:
stream:
bindings:
to-send-output:
destination: to-send-event
producer:
required-groups:
- to-send-event-group
to-send-input:
destination: to-send-event
group: to-send-event-group
consumer:
max-attempts: 1
requeueRejected: true
rabbit:
bindings:
# Forever retry
to-send-input:
consumer:
autoBindDlq: true
dlqTtl: 5000
dlqDeadLetterExchange:
maxConcurrency: 300
frameMaxHeadroom: 25000 # I added this as in the documentation
I added the property frameMaxHeadroom: 25000 as it says in the documentation, but it still does not work.
My springCloudVersion="Hoxton.RELEASE".
My dependency:
dependencies {
...
implementation "org.springframework.cloud:spring-cloud-starter-stream-rabbit"
...
}
In the repository on GitHub, I see the frameMaxHeadroom property in the property file.
I see that the code reduces the stack trace by the value I set (from a variable frameMaxHeadroom). I expected that I wasn't decreasing the stack trace, but increasing the value for the headers for the consumer, as written in the documentation. Why isn't it working as I wait?

frameMax is negotiated between the AMQP client and server; all headers must fit in one frame. You can increase it with the broker configuration.
Stack traces can be large and can easily exceed the frameMax alone; in order to leave room for other headers, the framework leaves at least 20,000 bytes (by default) free for other headers, by truncating the stack trace header if necessary.
If you are exceeding your frameMax, you must have other large headers. You need to increase the headroom to allow for those headers, so the stack trace is truncated further.

SCDF: Error handling when pod failed to start

I'm working on a service where it will call Spring Cloud Dataflow (SCDF) to spin off a new k8s Pod for Spring Batch job.
Map<String, String> properties = Map.of("testApp.cpu", cpu, "testApp.memory", memory);
LOGGER.info("Create task '{}' with definition '{}'", taskName, taskDefinition);
taskOperations.create(taskName, taskDefinition);
LOGGER.info("Launching task '{}' with properties {} and arguments '{}'", taskName, properties, args);
return taskOperations.launch(taskName, properties, args);
Everything works fine. The problem is, whenever we pull a non-existing image (eg: due to some connection issue), the pod failed to start AND we end up with pending tasks (with NO batch jobs created whatever)
For example, we will have tasks in the table task_execution (SCDF table) with empty end time
But no related jobs in batch_job_execution table.
It seems fine at first since no pod is created, we don't consume any resource. But as the number of "pending jobs" reached 20, we have the famous error:
Cannot launch task testApp. The maximum concurrent task executions is at its limit [20]
I'm trying to find a way to detect that the pod spin-off has failed (and hence we should mark the task as error), but to no avail.
Is there a way to detect if the task launch has failed when that task launch a new k8s pod?
UPDATE
Not sure if it is relevant, we are using SCDF 1.7.3.RELEASE
Describe the failed pod:
Name: podname-lp2nyowgmm
Namespace: my-namespace
Priority: 1000
Priority Class Name: test-cluster-default
Node: some-ip.compute.internal/XX.XXX.XXX.XX
Start Time: Thu, 14 Jan 2021 18:47:52 +0700
Labels: role=spring-app
spring-app-id=podname-lp2nyowgmm
spring-deployment-id=podname-lp2nyowgmm
task-name=podname
Annotations: iam.amazonaws.com/role: arn:aws:iam::XXXXXXXXXXXX:role/svc-XXXX-XXX-XX-XXXX-X-XXX-XXX-XXXXXXXXXXXXXXXXXXXX
kubernetes.io/psp: eks.privileged
Status: Pending
IP: XX.XXX.XXX.XXX
IPs:
IP: XX.XXX.XXX.XXX
Containers:
podname-lp2nyowgmm:
Container ID:
Image: image_host:XXX/mysystem/myapp:notExist
Image ID:
Port: <none>
Host Port: <none>
Args:
--spring.datasource.username=postgres
--spring.cloud.task.name=podname
--spring.datasource.url=jdbc:postgresql://...
--spring.datasource.driverClassName=org.postgresql.Driver
--spring.datasource.password=XXXX
--fileId=XXXXXXXXXXX
--spring.application.name=app-name
--fileName=file_name.csv
...
--spring.cloud.task.executionid=3
State: Waiting
Reason: ErrImagePull
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 8Gi
Requests:
cpu: 2
memory: 8Gi
Environment:
ELASTIC_SEARCH_PORT: 80
ELASTIC_SEARCH_PROTOCOL: http
SPRING_RABBITMQ_PORT: ${RABBITMQ_SERVICE_PORT}
ELASTIC_SEARCH_URL: elasticsearch
SPRING_PROFILES_ACTIVE: kubernetes
CLIENT_SECRET: ${CLIENT_SECRET}
SPRING_RABBITMQ_HOST: ${RABBITMQ_SERVICE_HOST}
RELEASE_ENV_NAME: QA_TEST
SPRING_CLOUD_APPLICATION_GUID: ${HOSTNAME}
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xxxxx(ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-xxxxx:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-xxxxx
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m22s default-scheduler Successfully assigned my-namespace/podname-lp2nyowgmm to some-ip.compute.internal
Normal Pulling 103s (x4 over 3m21s) kubelet Pulling image "image_host:XXX/mysystem/myapp:notExist"
Warning Failed 102s (x4 over 3m19s) kubelet Failed to pull image "image_host:XXX/mysystem/myapp:notExist": rpc error: code = Unknown desc = Error response from daemon: manifest for image_host:XXX/mysystem/myapp:notExist not found: manifest unknown: manifest unknown
Warning Failed 102s (x4 over 3m19s) kubelet Error: ErrImagePull
Normal BackOff 88s (x6 over 3m19s) kubelet Back-off pulling image "image_host:XXX/mysystem/myapp:notExist"
Warning Failed 73s (x7 over 3m19s) kubelet Error: ImagePullBackOff

1.7.3 is a very old release. We just released 2.7. The original logic used the task execution tables instead of the pod status. If the version you are using is subject to that, then it would explain what you are seeing. I strongly recommend an upgrade.

Thanks for the question. Looking at the source code, we don't include Pendingpods when calculating the current number of executing tasks. It may be something else is going on. 1) Could you run kubectl describe pod on a pod when it's in this state and post the result? (status details). 2) Is the deployer configured to create a job for each task? (false by default).

Elastic BeanStalk loading configuration from another region failed

I have uploaded a saved configuration file in a BeanStalk application in a region to another BeanStalk application in another region.
While loading that config I got an error
Stack named 'awseb-e-sme7w3eym3-stack' aborted operation. Current
state: 'CREATE_FAILED' Reason: The following resource(s) failed to
create: [AWSEBLoadBalancer]
Creating load balancer failed Reason: Property Listeners cannot be
empty Any idea about this issue ?
See the config file
AWSConfigurationTemplateVersion: 1.1.0.0
EnvironmentConfigurationMetadata:
DateCreated: '1580272974000'
DateModified: '1580273310143'
Description: xxxxxxxxxxxxxxxxxxxxx
EnvironmentTier:
Name: WebServer
Type: Standard
OptionSettings:
AWSEBAutoScalingGroup.aws:autoscaling:updatepolicy:rollingupdate:
MaxBatchSize: '1'
MinInstancesInService: '1'
RollingUpdateEnabled: true
RollingUpdateType: Health
AWSEBAutoScalingLaunchConfiguration.aws:autoscaling:launchconfiguration:
EC2KeyName: xxxxxxxxxxxxxxxxxxx
AWSEBCloudwatchAlarmHigh.aws:autoscaling:trigger:
UpperThreshold: '60'
AWSEBCloudwatchAlarmLow.aws:autoscaling:trigger:
BreachDuration: '2'
LowerThreshold: '25'
MeasureName: CPUUtilization
Period: '1'
Statistic: Maximum
Unit: Percent
AWSEBLoadBalancerSecurityGroup.aws:ec2:vpc:
VPCId: vpc-xxxxxxxxxxxxxxxx
AWSEBV2LoadBalancerListener.aws:elbv2:listener:default:
ListenerEnabled: false
AWSEBV2LoadBalancerListener443.aws:elbv2:listener:443:
SSLCertificateArns: arn:aws:acm:us-east-2:xxxxxxxxxxx:certificate/xxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxxxxx
AWSEBV2LoadBalancerTargetGroup.aws:elasticbeanstalk:environment:process:default:
HealthCheckPath: /rest/account/ping
MatcherHTTPCode: '200'
Port: '80'
Protocol: HTTP
aws:autoscaling:launchconfiguration:
IamInstanceProfile: aws-elasticbeanstalk-ec2-role
SecurityGroups:
- sg-xxxxxxxxxxxxx
aws:ec2:instances:
InstanceTypes: t2.small
aws:ec2:vpc:
ELBSubnets: subnet-xxxxxxxxxxxxxxxxxx,subnet-xxxxxxxxxxxxxxxxxx,subnet-xxxxxxxxxxxxxxx
Subnets: subnet-xxxxxxxxxxxxxxxxx,subnet-xxxxxxxxxxxxxxxxx,subnet-xxxxxxxxxxxxxxxxx
aws:elasticbeanstalk:application:environment:
JDBC_CONNECTION_STRING: jdbc:mysql://xxxxxxxxxxxxxxxxxxxxxxxxxxxx?user=xxxxxxxx&password=xxxxxxxxxxx&rewriteBatchedStatements=true&characterEncoding=UTF-8
aws.accessKeyId: xxxxxxxxxxxxxxxxxx
aws.secretKey: xxxxxxxxxxxxxxxxxxxx
com.aws.secretManger.secret.name: xxxxxxxxxxxxxxx
com.aws.secretManger.secret.region: us-east-2
com.decsond.loggly.token: xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx#xxxxx
com.decsond.metakey: xxxxxxxxxxxxxxxxx/XXX==
com.decsond.mode: debug
com.decsond.server.db.environment: aws
com.decsond.server.dpBinaryColumn: xxxxxxxxxxxx
com.decsond.server.environment: xxxxxxxxxx
com.decsond.server.type: pms
aws:elasticbeanstalk:container:tomcat:jvmoptions:
JVM Options: -XX:+CMSClassUnloadingEnabled -Dmvel.disable.jit=true -Ddrools.permgenThreshold=0
Xms: 512m
Xmx: 1024m
aws:elasticbeanstalk:environment:
LoadBalancerType: application
ServiceRole: arn:aws:iam::xxxxxxxxxxxxxx:role/aws-elasticbeanstalk-service-role
aws:elasticbeanstalk:healthreporting:system:
SystemType: enhanced
aws:elasticbeanstalk:managedactions:
ManagedActionsEnabled: true
PreferredStartTime: SAT:03:01
aws:elasticbeanstalk:managedactions:platformupdate:
InstanceRefreshEnabled: true
UpdateLevel: minor
aws:elasticbeanstalk:xray:
XRayEnabled: true
aws:elbv2:listener:443:
DefaultProcess: default
ListenerEnabled: true
Protocol: HTTPS
Rules: ''
SSLPolicy: ELBSecurityPolicy-2016-08
Platform:
PlatformArn: arn:aws:elasticbeanstalk:us-east-2::platform/Tomcat 8.5 with Java 8
running on 64bit Amazon Linux/3.3.1
Any idea about the issue ?

The most likely reason is that you are referencing objects in the region from where the config was saved from.
Is this the first EB application / environment in the new region?
If it is, it's worth first creating a test application and environment, using the features you want ... that will give EB a chance to create all the region specific behind-the-scenes magic it relies on.

Spring cloud stream RabbitMq - Set properties from source code

I am using Spring cloud stream with RabbitMQ.
I want to be able to configure message and query properties from source code and not from a property file (as they mention in their docs).
For example with the classic Java Client for RabbitMq i can do something like that to create a queue with the properties i want:
//qName, passive, durable, exclusive auto-delete
channel.queueDeclare("myQueue", true, false, false, , false , null);
Any ideas on how can i achieve the same thing using Spring cloud stream?

Inside of "application.yml" you can add all this values , following is example
spring:
cloud:
stream:
instance-count: 1
bindings:
input:
consumer:
concurrency: 2
maxAttempts: 1
group: geode-sink
destination: jdbc-event-result
binder: rabbit
rabbit:
bindings:
input:
consumer:
autoBindDlq: true
republishToDlq: true
requeueRejected: false
rabbitmq:
username: ur-user-name
password: ur-password
host: rabbitmq-url-replace-here
port: 5672
datasource:
platform: mysql
url: jdbc:mysql-url-replace-here
username: ur-user-name
password: ur-password
driverClassName: com.mysql.jdbc.Driver
datasource:
tomcat:
max-wait: 300
min-idle: 10
max-idle: 100
aggregator:
groupCount: 2
batchSize: 1000
batchTimeout: 1000
Updated :
https://cloud.spring.io/spring-cloud-static/spring-cloud-stream-binder-rabbit/2.2.0.M1/spring-cloud-stream-binder-rabbit.html
https://github.com/spring-projects/spring-xd/blob/master/spring-xd-dirt/src/main/resources/application.yml

After digging in their documentation and with the help of #vaquar khan I found out that the only way to do it is from your property file.
application.yml
spring:
cloud:
stream:
bindings:
queue_name :
destination: queue_name
group: your_group_name
durable-subscription : true
This will declare a durable, non-deleting and non-exclusive queue.