This is part one in a series about Openshift Virtualization with Portworx. The series will…
September 11, 2020
Live Migration, Backup and Restore for Openshift Virtualization
This is part two in a series about Openshift Virtualization with Portworx. The series will cover installation and usage, backup and restore, and data protection for virtual machines on Openshift. You can read part one here.
Enterprises are increasingly moving applications to containers, but many teams also have a huge investment in applications that run virtual machines (VMs). Many of these VMs often provide services to new and existing containerized applications as organizations break the monolith into microservices. Enterprises are diving head first into their container platforms, which often provide a seamless integration of computing, storage, networking, and security. Portworx often provides the data management and protection layer for these containerized applications. OpenShift Virtualization uses KubeVirt to offer a way for teams to run their VM-based applications side by side with their containerized apps as teams and applications grow and mature. This provides a single pane of glass for developers and operations alike. In this blog series, we’ll show you how Portworx can provide the data management layer for containers and for VMs. You can think of Portworx much like the v-suite of products in vSphere. Portworx can provide disks to VMs like vSAN, and data protection like vMotion but for both VMs and containers on OpenShift Container Platform.
- Please see Part 1 of the series for more detail on the install and configuration as well as how to create VMs using Portworx.
- Install Helm and PX-Backup
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 chmod 700 get_helm.sh ./get_helm.sh helm repo add portworx http://charts.portworx.io/ && helm repo update helm install px-backup portworx/px-backup --namespace px-backup --create-namespace
- Configure PX-Backup with credentials and backup location
Backup and Restore for VMs using PX-Backup
First, add the OCP cluster to PX-Backup by selecting “Add Cluster` at the top right of the PX-Backup interface. In the below image, the cluster we are working with is already added and named “openshift-cnv”.
Then, navigate to the namespace which is running the VMs you wish to backup.
Openshift virtual machines use CRDs to deploy into Kubernetes. We need to make sure PX-Backup is able to back these specific objects up, so let’s make sure and register these CRDs with our Portworx environment. This will allow PX-Backup to view, backup and restore these CRDs needed by virtual machines.
Apply both YAML files below, they will register VirtualMachineInstance and VirtualMachine CRDs with Portworx.
apiVersion: stork.libopenstorage.org/v1alpha1 kind: ApplicationRegistration metadata: name: cnv-virtualmachine-instance resources: - PodsPath: "" group: kubevirt.io version: v1alpha3 kind: VirtualMachineInstance keepStatus: false
apiVersion: stork.libopenstorage.org/v1alpha1 kind: ApplicationRegistration metadata: name: cnv-virtualmachine resources: - PodsPath: "" group: kubevirt.io version: v1alpha3 kind: VirtualMachine keepStatus: false
After you apply the specs above, you should be able to see the same VMs that you see in the Openshift UI in PX-Backup.
Here are the VMs within the Openshift console.
Here are the same VMs withn the PX-Backup UI.
Note, if you have many VMs within a single namespace and you want to backup only one of them, make sure and label the VM resources you need to backup. In this case, I’ve made sure the VM resources and PVCs have a label of vm: demo. Select this label within the PX-Backup UI to backup the needed VM resources.
Create a backup and wait for the backup to complete. You should see a green cloud icon on the left indicating successful backup of the VM.
You can also select Show Details from the specific backup to see what resources are within the backup.
Once your backup is complete, the VM can be deleted to simulate the need to restore a VM. Select “Delete Virtual Machine” from the Openshift VM menu.
Next, select the backup from PX-Backup and click “Restore”. Fill out the needed restore information such as the restore name and destination cluster. Leave the “replace existing resource” checkbox unchecked, we want the restore job to only replace what is missing (our deleted VM). Click restore to start the recovery.
The restore job should complete after some time, this will be indicated by the green cloud icon to the left of the restore job. Within the Openshift dashboard you will see the VM starting again if the restore completed successfully.
Watch this short demo of a backup and restore workflow for VMs on Openshift 4.5.
Enabling Live Migration with Portworx Shared Volumes
OpenShift virtualization enables live migration of VMs with the support from underlying storage classes. Portworx is one of these storage classes that supports the live migration features by using the sharedv4 flag for PVCs. Use the below storage class example to enable live migration workflows for Openshift virtual machines.
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: shared-px-replicated parameters: sharedv4: "true" repl: "2" provisioner: kubernetes.io/portworx-volume reclaimPolicy: Delete volumeBindingMode: Immediate
Make sure to update the AccessMode of the Openshift virtualization default data volume configuration to “<storage-class>.ReadWriteMany”. This is a requirement live migrations to work properly.
Once you have done that, you can create a PVC that uses this storage class. Then, a VM can use this PVC as its root disk. One example of this is to create a CDI based image that loads the Fedora cloud image into a PVC. Take note of the annotation “cdi.kubevirt.io/storage.import.endpoint“ which will load the Fedora image into the Portworx volume.
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: "fedora-disk1" labels: app: containerized-data-importer annotations: cdi.kubevirt.io/storage.import.endpoint: "https://dl.fedoraproject.org/pub/fedora/linux/releases/31/Cloud/x86_64/images/Fedora-Cloud-Base-31-1.9.x86_64.qcow2" spec: storageClassName: shared-px-replicated accessModes: - ReadWriteMany resources: requests: storage: 20Gi
Once it’s completed, you can select the above disk as your boot disk in the virtual machine wizard. Notice that it should show that it is using “Shared Access (RWX)” for its Access Mode.
Now, once the VM is running, simply click on the right menu bar and select “Migrate Virtual Machine”
Note that the above images show that the VM moved from node02 to node01. We can also get more information about this movement using kubectl and pxctl.
Using kubectl we can view the pods, this shows the Completed vm on node node02 and it is now running on node01
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES virt-launcher-vm1-5j5q8 1/1 Running 0 10m 10.128.4.45 node01.demoocp.portworx.dev <none> <none> virt-launcher-vm1-x5r9x 0/1 Completed 0 12m 10.131.4.37 node02.demoocp.portworx.dev <none> <none>
We can also inspect the volume using pxctl. This shows that there have been Volume Consumers on both node01 and node02 as expected and Potworx was able to provide the shared volume to both nodes.
# /opt/pwx/bin/pxctl v i 326738006419398040 Volume : 326738006419398040 Name : pvc-81825a14-abc8-4123-8ba7-4cb255fbde7a Size : 20 GiB Format : ext4 HA : 2 IO Priority : HIGH Creation time : Sep 10 18:06:25 UTC 2020 Shared : v4 Status : up State : Attached: 8a045618-657f-4e0f-89a4-3cdc75144b0a (10.180.5.20) Device Path : /dev/pxd/pxd326738006419398040 Reads : 3616 Reads MS : 1192 Bytes Read : 186867712 Writes : 6135 Writes MS : 36221 Bytes Written : 637800448 IOs in progress : 0 Bytes used : 1.3 GiB Replica sets on nodes: Set 0 Node : 10.180.5.20 (Pool eb8214d6-67f0-491c-bb37-9f7670fb88bc ) Node : 10.180.5.16 (Pool 80a656b7-b5b1-41e7-8d51-a3541c81c34a ) Replication Status : Up Volume consumers : - Name : virt-launcher-vm1-5j5q8 (de6e3e03-8023-4010-a01c-1fa7ac6ec09d) (Pod) Namespace : vms Running on : node01.demoocp.portworx.dev Controlled by : vm1 (VirtualMachineInstance) - Name : virt-launcher-vm1-cg6vf (c9d0a57c-55a2-49cf-891d-4378cab5fa2a) (Pod) Namespace : vms Running on : node02.demoocp.portworx.dev Controlled by : vm1 (VirtualMachineInstance)
Portworx often provides the data management and protection layer for containerized applications in Openshift. With OpenShift Virtualization, Portworx can offer a way for teams to run their VM-based applications side-by-side with their containerized apps with a single data management layer. This allows end users to get the same data protection and high availability for VMs and containers alike.