Data Protection and Backup for Kubernetes: Using Local, Cloud and Application Consistent Snapshots

The concept of creating backups using snapshots has been around forever with traditional storage solutions. For cloud native applications, Portworx platform not only allows local snapshots but also supports backup/restore to or from any cloud object store. Even for multi-container apps, these snapshots are application-aware, providing an application-consistent, not simply crash-consistent copy. Both application-consistent local and cloud snapshots are recommended for production use-cases and it is important to understand the difference between the two as shown in the table below (assuming Kubernetes for now):

Going further, other platform features like an automated cluster to cluster data migration and disaster recovery are built on top of cloud snapshots. These cloud snapshots work with Amazon S3, Azure Blob, Google Cloud Storage or any S3 compatible object store.

Application consistent snapshots can also be tailored to meet application-specific needs. Here users can specify pre and post rules that are run on the application pods using the volumes. This allows users to quiesce the applications before the snapshot is taken and resume I/O after the snapshot is taken. The commands will be run in pods which are using the PVC being snapshotted.

Best practice recommendations

For production deployment, it is very important to schedule local and cloud snapshots on a periodic basis. Most customers schedule multiple local snapshots in a day and at least one snapshot to cloud daily. Since cloud snapshots can also be scheduled to happen at a particular time, it is recommended to schedule them at a time when the application data traffic is light to ensure the minimal load on the system and get backups done faster. Note that cloud snapshots store the incremental snapshots in the cloud and on import, can roll up all the snaps and import a point-in-time copy of the volume into the cluster.

Cluster Management UI

PX-Central, the Portworx Enterprise cluster management interface shows all the snapshots in the Snapshots tab as shown below. It offers a time-machine like the view of both local snapshots and cloud snapshots. Users can select any volume and click the snapshot icon on the volume details pane to create either local or cloud snapshots. Users have to configure the S3-compliant endpoint for the cloud provider of their choice by providing the required credentials. This requires them to create a secret in the secret store (KMS) of their choice e.g. Vault for establishing the credential. Once all this is set up, users can create cloud snapshots from the UI. For restore, users can restore cloud snapshot by choosing the snapshot and clicking the ‘Restore’ button.

Additionally, users can also create schedules for snapshots and manage them from the UI itself. This functionality supports local snapshots today and will be available for cloud snapshots in a future release.

Metrics in Grafana Dashboards

Starting version 2.0.3, following new metrics for cloud snapshots have been added. For any volume that is configured for cloud snapshots, users can select the volume in the Grafana Volume Dashboard and see the following metrics in the pre-created panels:

px_backup_stats_status
px_backup_stats_duration_seconds
px_backup_stats_size

Below is an example screenshot showing snapshot duration and size. This will show that incremental backups are smaller and take less time which brings down the average numbers over a period of time.

As you can see, the Portworx Enterprise platform offers data protection at multiple levels for cloud-native applications with automated controls and application-specific customizability. Not only that, these features have paved the way for the container industry’s only cluster migration and disaster recovery solution for applications and data in Kubernetes.

CLI Reference Documentation

Applications Consistent Local Snapshots – https://docs.portworx.com/portworx-enterprise/reference/CLI
Application Consistent Cloud Snapshots – https://docs.portworx.com/portworx-enterprise/reference/CLI