Portworx to the rescue! Protect against accidental deletion in Kubernetes

Running Kubernetes in production and adopting a rich set of DevOps practices to accomplish organizational and functional flexibility means low touch automation, scripts and pipelines doing a lot of work. Infrastructure-as-code, continuous integration, and continuous deployment are mainstays in cloud-native data centers today, and while they do offer many benefits for teams, hiccups can occur within the automation scripts or with those who operate them. The good news is Portworx has you covered when those problems crop up.
Just like the feeling you get when you think you left your wallet on the grocery store floor or your phone in the backseat of the uber, it’s never a good feeling knowing something or someone may have accidentally deleted data. In fact, Kubernetes doesn’t make it easy on you when it sets its default reclaim policy to “Delete.” That means if a PVC object is accidentally deleted, so is the data within the PV that supports it.

Whether or not it’s buggy automation scripts, incorrect Kubernetes configurations, or just commands made by mistake, deletion of data can cause major headaches for operations teams – and, in the worst case, hefty fines.
With Portworx Enterprise 2.10, Portworx introduces the concept of a volume trash can with a configurable volume expiration for the volumes placed there. Yes, much like how your “Trash” app works on Mac OS X, it can save you from harmful accidental deletions. Here’s how it works.

Using Portworx Volume Trash Can with Kubernetes

For this example, we will use Kubernetes default reclaim policies as a potential reason why you want to enable the trash can feature. As mentioned before, even if the StorageClass object does not set a reclaim policy, it will use “Delete” as its default.

If the PersistentVolumeClaim objects that were dynamically provisioned from this storage class were to be accidentally deleted, the correlating PersistentVolume and its data would also be permanently erased. So, how can you configure Portworx to protect against this? The first thing to do is configure the volume expiration in the cluster settings within Portworx. This will set how long a volume remains in the trash can before it is permanently deleted. Keep in mind that volumes that remain in the trash can take up storage space to keep the data within the volumes safe. Your setting will depend on the trade-offs around how long to protect from deletions vs how much disk space to keep available. For our example, let’s configure volumes to remain in the trash can for 12 hours.

$ pxctl cluster options update --volume-expiration-minutes 720

Now, when a PVC is deleted, you should see the volume remain in the trash can, and those volumes can be seen using the following command:

$ pxctl v l --trashcan

DELETE TIME ID NAME SIZE HA SHARED ENCRYPTED IO_PRIORITY STATUS DELETE_TIMER
Tue Mar 29 15:50:48 UTC 2022 533540619086447686 pvc-6dc656c8-34f4-4b36-bc27-0d78f931c6c3-tc-1648569048 10 GiB 2 no no LOW up - detached 11h59m21s

To recover this volume, we can use the volume restore –trashcan command to indicate we want to restore from the trash can. Be sure to use the same PVC name to restore to if you plan to use it in the same application that was deleted.

$ pxctl v restore --trashcan pvc-6dc656c8-34f4-4b36-bc27-0d78f931c6c3-tc-1648569048 pvc-6dc656c8-34f4-4b36-bc27-0d78f931c6c3

Then, to restore back to Kubernetes, we can create a new PV object to be used by our application with the ID and Name of the volume. In this case, we can replace the PVC_NAME and VOL_ID with the volume name and ID. Make sure other options in this PV object match your accidentally deleted volume, such as the namespace, size, and storage class.

Once the PV object is created, it will point at our “restored-from-trash-can” volume within Portworx, and it can then be bound and used by your application in Kubernetes.

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS
pvc-6dc656c8-34f4-4b36-bc27-0d78f931c6c3   10Gi       RWO            Delete           Bound    default/postgres-data   px-postgres-sc           

Whether it’s faulty automation scripts in CI/CD pipelines or just an errant command run by a Kubernetes administrator, accidentally deleted data is a concern worth protecting against. Upgrade your Portworx cluster or adopt Portworx Enterprise 2.10 to help your team protect against these issues so you can fully focus on your cloud native adoption. Take a look at the trash can feature in action in the below demonstration.