If you have ever removed a node from a Kubernetes cluster and then added it back, you may have encountered some issues with persistent data. Persistent data is any data that survives beyond the lifecycle of a pod, such as databases, logs, or configuration files. Kubernetes uses persistent volumes (PVs) and persistent volume claims (PVCs) to manage persistent data across the cluster.
However, sometimes these resources may not be cleaned up properly when a node is deleted or drained. This can cause problems when you try to reuse the node for another cluster or add it back to the same cluster. For example, you may see errors like:
- Failed to attach volume \”pvc-1234\” on node \”node1\”: volume is already attached to node \”node2\”
- Failed to mount volume \”pvc-5678\” on pod \”pod1\”: mount failed: exit status 32
- Failed to create subPath directory for volumeMount \”data\” of container \”db\”: mkdir /var/lib/kubelet/pods/abcd-efgh/volumes/kubernetes.io~nfs/data: file exists
To avoid these issues, you need to clean up your old Kubernetes persistent data before adding a node back to a cluster. Here are some steps you can follow:
Step 1: Delete or unbind any PVCs associated with the node
The first step is to delete or unbind any PVCs that are associated with the node you want to remove. A PVC is a request for storage by a user or a pod. It binds to a PV that provides the actual storage backend. When you delete a PVC, it also releases the PV that it was bound to, unless the PV has a reclaim policy of Retain.
To list all the PVCs in your cluster, you can use the command:
kubectl get pvc --all-namespaces
To delete a PVC, you can use the command:
kubectl delete pvc <pvc-name> -n <namespace>
Alternatively, you can unbind a PVC from a PV without deleting it by editing the PVC spec and removing the
volumeName field. This will make the PVC available for binding to another PV.
To edit a PVC, you can use the command:
kubectl edit pvc <pvc-name> -n <namespace>
Step 2: Delete any PVs that are not bound to any PVCs
The next step is to delete any PVs that are not bound to any PVCs. A PV is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using storage classes. It is a resource in the cluster just like a node. PVs have a lifecycle independent of any pod that uses them.
To list all the PVs in your cluster, you can use the command:
kubectl get pv
To delete a PV, you can use the command:
kubectl delete pv <pv-name>
Note that deleting a PV does not necessarily delete the underlying storage device or volume. Depending on the type of storage and the reclaim policy of the PV, you may need to manually delete the storage device or volume from your cloud provider or storage server.
Step 3: Delete any leftover data on the node
The final step is to delete any leftover data on the node that you want to remove. This may include directories or files that were created by Kubernetes or by your applications. For example, you may need to delete:
/etc/cni/net.ddirectory that contains CNI (Container Network Interface) configuration files
/var/lib/kubeletdirectory that contains kubelet data such as pods, volumes, plugins, etc.
/var/lib/etcddirectory that contains etcd data if the node was running an etcd member
/var/lib/dockerdirectory that contains docker data such as images, containers, volumes, etc.
- Any other application-specific data directories or files that were mounted or created on the node
To delete these directories or files, you can use commands like:
sudo rm -rf /etc/cni/net.d sudo rm -rf /var/lib/kubelet sudo rm -rf /var/lib/etcd sudo rm -rf /var/lib/docker sudo rm -rf /path/to/your/application/data
Be careful when using these commands and make sure you are deleting only what you intend
Source: Conversation with Bing, 3/27/2023(1) A complete storage guide for your Kubernetes storage problems. https://bing.com/search?q=Kubernetes+persistent+data Accessed 3/27/2023.
(2) Persistent Volumes | Kubernetes. https://kubernetes.io/docs/concepts/storage/persistent-volumes/?ref=hack-slash Accessed 3/27/2023.
(3) Kubernetes Persistent Volumes – Ultimate Guide – Knowledge Base by …. https://phoenixnap.com/kb/kubernetes-persistent-volumes Accessed 3/27/2023.
(4) Data persistence on Kubernetes – SQL Server Big Data Clusters. https://learn.microsoft.com/en-us/sql/big-data-cluster/concept-data-persistence?view=sql-server-ver15 Accessed 3/27/2023.
(5) A complete storage guide for your Kubernetes storage problems. https://www.cncf.io/blog/2020/04/28/a-complete-storage-guide-for-your-kubernetes-storage-problems/ Accessed 3/27/2023.
(6) Data Persistence in Kubernetes | Kubernetes Volumes simply explained …. https://dev.to/techworld_with_nana/data-persistence-in-kubernetes-kubernetes-volumes-simply-explained-852 Accessed 3/27/2023.