There might be times when the Kubernetes infrastructure may drift away from the Delphix Platform’s understood infrastructure state. This can be a scenario where the VDB can be seen as active; however, the underlying Kubernetes Pod is down. These kinds of scenarios are tricky to manage and handle accordingly. The Delphix Platform and Kubernetes clusters will try to stay in sync, however, errors may arise in complex scenarios. The following are situations that might arise.
The VDB is enabled but the Pod failed
Background: This is a common occurrence where, through the K8s Driver, you can see that the VDB enable operation succeeded, but the call to create the Pod failed.
Reason: This can be due to issues in the database’s Docker container or image.
Mitigation: The best way to resolve such a situation is to gracefully delete the Pod and let the K8s Driver handle the rollback of the Delphix Platform’s underlying objects.
kubectl delete -f pod.yaml
The Node is down
Background: When the node is down, there can be a case where the Pod gets rescheduled to a new node. In such a situation, we have seen the “multiattach error” among others.
Reason: Our solution mandates a single PVC be claimed by a single Pod. This is specified by setting the ‘Access-Mode’ of the PVC to ReadWriteOnce(RWO) or ReadWriteOncePod(RWOP).
Once a PVC is claimed by a Pod, no other Pods can claim that PVC again. If they try doing that, they will encounter a “multi-attach” error. We get this error because when a Pod claims a PVC, an entity in Kubernetes called ‘VolumeAttachment’ gets created between the PV and the Node where that claiming Pod resides.
Now in case of a Node failure, when the Node goes down, the pod fails. When Kubernetes detects a node failure and the Pod is in an unhealthy state, its auto-healing capability kicks in. This makes the Pod rescheduled to another healthy Node. While doing so, the previous VolumeAttachment doesn't get removed and stays as a lingering entity. This blocks the new Pod from creating a new VolumeAttachment for the same PVC. As a result, the old Pod stays in the Terminating state, and the new Pod gets stuck in the ContainerCreation state.
Mitigation: To resolve this, delete the Pod reference and the dangling volume attachment to the node that is down.
kubectl delete volumeattachments/<name> -n namespace kubectl delete pods/<name> -n namespace
Delete VolumeSnapshot - When READYTOUSE value is set to false
Background: If you are attempting to create a VolumeSnapshot, but the READYTOUSE property value is false, then deletion of the VolumeSnapshot will fail.
Reason: The deletion does not happen because of the request spamming from k8s.
Mitigation: Follow the below steps to clear the finalizers.
If VolumeSnapshots were created, run the following command:
kubectl patch volumesnapshot/<snapshot-name> --type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'
If VolumeSnapshotContents were created, run the following command:
kubectl patch volumesnapshotcontent/<volumesnapshotcontent-id…>--type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'
If VolumeSnapshots or VolumeSnapshotContents are still present after the above patch commands, then delete them individually using kubectl delete....
CrashloopBackOff error
Background: The driver goes into a CrashLoopBackOff state if there is any error from DCT or any other infrastructure during Installation.
Reason: It will retry in a loop till the DCT or infrastructure failures are resolved.
Mitigation: To find the root cause,the error’s stack needs to be investigated and fixed and the trace can be found in the driver’s or node-level logs.