Portworx Guided Hands On-Labs. Register Now
Portworx is a cloud native storage platform to run persistent workloads deployed on a variety of orchestration engines, including Kubernetes. With Portworx, customers can manage the database of their choice on any infrastructure using any container scheduler. It provides a single data management layer for all stateful services, no matter where they run.
Kubemotion is one of the core building blocks of Portworx storage infrastructure. Introduced in Portworx Enterprise 2.0, it allows Kubernetes users to migrate application data and Kubernetes application configurations between clusters, enabling migration, backup & recovery, blue-green deployments, and more.
This step-by-step guide demonstrates how to move persistent volumes and Kubernetes resources associated with a stateful application from one Azure region to another.
Background
For enterprises, a common scenario is to run development and test environments in one cloud region and the production environment in another. Development teams may choose a region that’s geographically closer to them while deploying production applications in another region that has low latency for the users and customers.
Even though Kubernetes makes it easy to move stateless workloads across environments, achieving parity of stateful workloads remains a challenge.
For this walkthrough, we will move Kubernetes resources between Azure Kubernetes Service (AKS) clusters running in the East US and West US. The East US region is used by the development teams for dev/test, and the West US region for the production environment.
After thoroughly testing an application in dev/test, the team will use Portworx and Kubemotion to reliably move the storage volumes and application resources from the development to the production environment.
Exploring the Environments
We have two AKS clusters – dev and production – running in the East US region and West US region of Azure. Both of them have the latest version of Portworx cluster up and running.
The above AKS cluster represents the development environment running in the East US region of Azure.
The above AKS cluster represents the development environment running in the West US region of Azure.
The dev/test environment currently runs a LAMP-based content management system that needs to be migrated to production.
Tip: To navigate between two kubeconfig contexts representing different clusters, use the kubectx tool and rename the contexts with dev and prod labels.
kubectx kubectx dev=. kubectx kubectx prod=.
It runs two deployments—MySQL and WordPress—in the cms namespace.
For a detailed tutorial on configuring a highly available WordPress stack on Azure AKS, please refer to this guide.
The persistent volumes attached to these pods are backed by a Portworx storage cluster.
The below volume is attached to the MySQL pod.
For the WordPress CMS, there is a shared Portworx volume attached to the pods.
The fully configured application is accessed through the IP address of the load balancer.
Preparing the Source and Target Environments
Before we can migrate the resources, we need to configure the source and destination clusters.
Follow the below steps to prepare the environments.
Creating the Object Store Credentials
We need to create the object store credentials on both source and destination clusters. In order to create this, we have to retrieve the UUID of the destination cluster, which will be appended to the name of the credential.
To complete this step, create a storage account on Azure, either through the CLI or the Portal.
Make a note of the storage account name and the key associated with it.
Switch to prod cluster and run the below command to copy the UUID.
PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}') kubectl -n=kube-system exec $PX_POD -- /opt/pwx/bin/pxctl status
Let’s create the credentials on the production cluster while we are still using its context.
kubectx dev PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}') kubectl -n=kube-system exec $PX_POD -- /opt/pwx/bin/pxctl credentials create \ --provider azure \ --azure-account-name \ --azure-account-key \ clusterPair_56a45912-cf2b-484d-80c1-70d211dcb1dd
Make sure that the credential name follows the convention of ‘clusterPair_UUID’.
Switch to the development cluster and repeat the same steps to create the credentials for the source.
Retrieving the Target Cluster Token
The next step is to retrieve the cluster token from the production cluster, which is used to generate the YAML file with the cluster pair definition.
Let’s switch to the production cluster and run the command below to access the token.
kubectx prod PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}') kubectl -n=kube-system exec $PX_POD -- /opt/pwx/bin/pxctl cluster token show
Make a note of the cluster token and keep it in a safe place.
Getting the Load Balancer Endpoint for Portworx Service
We also need the IP address of the load balancer associated with the Portworx Service on the production cluster. We can retrieve that with the below command.
kubectl describe svc portworx-service -n kube-system
At this point, you should have the data below ready:
- Token of the destination cluster
- IP address of the load balancer pointing to portworx-service
Generating the ClusterPair Specification
Let’s generate the scaffolded YAML file from the production (destination) cluster. This will be applied to the development (source) cluster later.
Make sure you are still using the prod context, and run the command below.
kubectx prod storkctl generate clusterpair -n cms prodcluster > clusterpair.yaml
Open clusterpair.yaml and add the below details under options. They reflect the public IP address or DNS name of the load balancer in the destination cluster, port, and the token associated with the destination cluster.
Pairing the Clusters for Kubemotion
We are now ready to pair the clusters by applying the clusterpair specification to the source cluster.
Let’s switch to the source cluster (dev environment) to pair the clusters.
kubectx dev kubectl apply -f clusterpair.yaml kubectl get clusterpair -n cms
Verifying the Pairing Status
We can verify the pairing status with the storkctl CLI. Ensure that both storage-status and scheduler-status are ready with no errors.
storkctl -n=cms get clusterpair
Migrating the CMS Application from Source to Destination
Make sure that you are using the dev context, and follow the below steps to start the migration of the CMS application.
Starting the Migration Job
Create a YAML file called migration.yaml with the content below:
apiVersion: stork.libopenstorage.org/v1alpha1 kind: Migration metadata: name: cmsmigration namespace: cms spec: # This should be the name of the cluster pair created above clusterPair: prodcluster # If set to false this will migrate only the Portworx volumes. No PVCs, apps, etc will be migrated includeResources: true # If set to false, the deployments and stateful set replicas will be set to 0 on the destination. # There will be an annotation with "stork.openstorage.org/migrationReplicas" on the destinationto store the replica count from the source. startApplications: true # List of namespaces to migrate namespaces: - cms
This definition contains critical information like the name of the clusterpair, namespaces to be included in the migration, and the type of resources to be migrated.
Submit the YAML file to the dev cluster to initiate the migration job.
kubectl apply -f migration.yaml migration.stork.libopenstorage.org/cmsmigration created
Tracking and Monitoring the Migration Job
We can monitor the migration through storkctl.
storkctl get migration -n cms
Once the migration is done, storkctl reports the final number of volumes and resources migrated to the destination cluster.
To get detailed information on the migration, run the command below:
kubectl describe migration cmsmigration -n=cms Name: cmsmigration Namespace: cms Labels: Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"stork.libopenstorage.org/v1alpha1","kind":"Migration","metadata":{"annotations":{},"name":"cmsmigration","namespace":"cms"}... API Version: stork.libopenstorage.org/v1alpha1 Kind: Migration Metadata: Creation Timestamp: 2019-10-24T13:26:29Z Generation: 8 Resource Version: 596332 Self Link: /apis/stork.libopenstorage.org/v1alpha1/namespaces/cms/migrations/cmsmigration UID: e349cb67-f661-11e9-810c-aad62219ad75 Spec: Admin Cluster Pair: Cluster Pair: prodcluster Include Resources: true Include Volumes: true Namespaces: cms Post Exec Rule: Pre Exec Rule: Selectors: Start Applications: true Status: Finish Timestamp: 2019-10-24T13:27:25Z Resources: Group: core Kind: PersistentVolume Name: pvc-89ef8f76-f65d-11e9-810c-aad62219ad75 Namespace: Reason: Resource migrated successfully Status: Successful Version: v1 Group: core Kind: PersistentVolume Name: pvc-93d0b035-f65d-11e9-810c-aad62219ad75 Namespace: Reason: Resource migrated successfully Status: Successful Version: v1 Group: core Kind: PersistentVolumeClaim Name: px-mysql-pvc Namespace: cms Reason: Resource migrated successfully Status: Successful Version: v1 Group: core Kind: PersistentVolumeClaim Name: px-wp-pvc Namespace: cms Reason: Resource migrated successfully Status: Successful Version: v1 Group: core Kind: Service Name: mysql Namespace: cms Reason: Resource migrated successfully Status: Successful Version: v1 Group: core Kind: Service Name: wordpress Namespace: cms Reason: Resource migrated successfully Status: Successful Version: v1 Group: apps Kind: Deployment Name: mysql Namespace: cms Reason: Resource migrated successfully Status: Successful Version: v1 Group: apps Kind: Deployment Name: wordpress Namespace: cms Reason: Resource migrated successfully Status: Successful Version: v1 Stage: Final Status: Successful Volumes: Namespace: cms Persistent Volume Claim: px-mysql-pvc Reason: Migration successful for volume Status: Successful Volume: pvc-89ef8f76-f65d-11e9-810c-aad62219ad75 Namespace: cms Persistent Volume Claim: px-wp-pvc Reason: Migration successful for volume Status: Successful Volume: pvc-93d0b035-f65d-11e9-810c-aad62219ad75 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Successful 96s stork Volume pvc-89ef8f76-f65d-11e9-810c-aad62219ad75 migrated successfully Normal Successful 96s stork Volume pvc-93d0b035-f65d-11e9-810c-aad62219ad75 migrated successfully Normal Successful 94s stork /v1, Kind=PersistentVolume /pvc-89ef8f76-f65d-11e9-810c-aad62219ad75: Resource migrated successfully Normal Successful 94s stork /v1, Kind=PersistentVolume /pvc-93d0b035-f65d-11e9-810c-aad62219ad75: Resource migrated successfully Normal Successful 94s stork /v1, Kind=PersistentVolumeClaim cms/px-mysql-pvc: Resource migrated successfully Normal Successful 94s stork /v1, Kind=PersistentVolumeClaim cms/px-wp-pvc: Resource migrated successfully Normal Successful 94s stork /v1, Kind=Service cms/mysql: Resource migrated successfully Normal Successful 94s stork /v1, Kind=Service cms/wordpress: Resource migrated successfully Normal Successful 93s stork apps/v1, Kind=Deployment cms/mysql: Resource migrated successfully Normal Successful 93s stork (combined from similar events): apps/v1, Kind=Deployment cms/wordpress: Resource migrated successfully
Verifying the Migration in the Production Environment
Now we can switch the context to prod and check all the resources created within the cms namespace.
kubectx prod kubectl get all -n cms
You can also access the application by visiting the CNAME of the load balancer of the WordPress service.
Summary
Kubemotion extends the power of portability to stateful workloads. It can be used to seamlessly migrate volumes from on-premises to public cloud (hybrid) environments and cross-cloud platforms.
Share
Subscribe for Updates
About Us
Portworx is the leader in cloud native storage for containers.
Thanks for subscribing!
Janakiram MSV
Contributor | Certified Kubernetes Administrator (CKA) and Developer (CKAD)Explore Related Content:
- aks
- azure
- azure kubernetes service