Portworx & Red Hat Hands-on Labs Register Now
Portworx is a cloud native storage platform to run persistent workloads deployed on a variety of orchestration engines, including Kubernetes. With Portworx, customers can manage the database of their choice on any infrastructure using any container scheduler. It provides a single data management layer for all stateful services, no matter where they run.
Kubemotion is one of the core building blocks of Portworx storage infrastructure. Introduced in Portworx Enterprise 2.0, it allows Kubernetes users to migrate application data and Kubernetes application configurations between clusters, enabling migration, backup & recovery, blue-green deployments, and more.
This step-by-step guide demonstrates how to move persistent volumes and Kubernetes resources associated with a stateful application from one GCP region to another.
Background
For enterprises, it’s a common scenario to run development and test environments in one cloud region and the production environment in another. Development teams may choose a region that’s geographically closer to them while deploying production applications in another region that has low latency for the users and customers.
Even though Kubernetes makes it easy to move stateless workloads across environments, achieving parity of stateful workloads remains a challenge.
For this walkthrough, we will move Kubernetes resources between Google Kubernetes Engine (GKE) clusters running in Asia South (Mumbai) and Asia Southeast (Singapore). The Mumbai region is used by the development teams for dev/test and the Singapore region for the production environment.
After thoroughly testing an application in dev/test, the team will use Portworx and Kubemotion to reliably move the storage volumes and application resources from the development to the production environment.
Exploring the Environments
We have two GKE clusters—dev and production—running in the Mumbai and Singapore regions of Google Cloud Platform. Both of them have the latest version of Portworx cluster up and running.
The above GKE cluster represents the development environment running in the Mumbai (asia-south1) region of GCP.
The above GKE cluster represents the development environment running in the Singapore (asia-southeast1) region of GCP.
The dev/test environment currently runs a LAMP-based content management system that needs to be migrated to production.
Tip: To navigate between two kubeconfig contexts representing different clusters, use the kubectx tool and rename the contexts with dev and prod labels.
kubectx kubectx dev=. kubectx kubectx prod=.
It runs two deployments—MySQL and WordPress—in the cms namespace.
For a detailed tutorial on configuring a highly available WordPress stack on GKE, please refer to this guide.
The persistent volumes attached to these pods are backed by a Portworx storage cluster.
The volume below is attached to the MySQL pod.
For the WordPress CMS, there is a shared Portworx volume attached to the pods.
The fully configured application is accessed through the IP address of the load balancer.
We will now migrate this application along with its state and configuration from dev/test to production.
Preparing the Source and Target Environments
Before we can move the volumes, we need to configure the source and destination clusters.
Follow the below steps to prepare the environments.
Note: Kubemotion may also be configured with Kubernetes secrets. Since the scenario is based on Google Cloud Platform, we are demonstrating the integration with Google Cloud KMS and Service Accounts.
Enabling the GCP APIs
In your GCP account, make sure that the Google Cloud Key Management Service (KMS) APIs are enabled. Portworx integrates with Google Cloud KMS to store Portworx secrets for Volume Encryption and Cloud Credentials.
Creating the Service Account
After enabling the APIs, create a service account that has roles “compute admin” and “KMS admin” added.
The last step involves the creation of a file that contains the private key. Copy the downloaded account file in a directory gcloud-secrets/ and rename it gcloud.json to create a Kubernetes secret from it.
Configuring the KMS Keyring
Run the below command to create a keyring that will be used by Portworx. Make sure that while creating the asymmetric key you specify the purpose of the key as Asymmetric decrypt.
$ gcloud kms keyrings create portworx --location global
Creating the Kubernetes Secret
Run the command below on both the clusters to create a Kubernetes secret from the service account file.
$ kubectl -n kube-system create secret generic px-gcloud --from-file=gcloud-secrets/ --from-literal=gcloud-kms-resource-id=projects//locations//keyRings//cryptoKeys//cryptoKeyVersions/1
Make sure to replace the Project ID, Key Ring Name, and Asymmetric Key Name in the above command with the correct values.
Both the clusters now have a secret that provides access to Google Cloud Platform KMS and GCE APIs.
Patching Portworx Deployment
Now, we need to update the Portworx daemonset to provide access to KMS.
$ kubectl edit daemonset portworx -n kube-system
Add the “-secret_type”, “gcloud-kms” arguments to the portworx container in the daemonset.
containers: - args: - -c - testclusterid - -s - /dev/sdb - -x - kubernetes - -secret_type - gcloud-kms name: portworx
We are now ready to patch the Portworx daemonset to add the secret created in the previous step.
cat < patch.yaml spec: template: spec: containers: - name: portworx env: - name: GOOGLE_KMS_RESOURCE_ID valueFrom: secretKeyRef: name: px-gcloud key: gcloud-kms-resource-id - name: GOOGLE_APPLICATION_CREDENTIALS value: /etc/pwx/gce/gcloud.json volumeMounts: - mountPath: /etc/pwx/gce name: gcloud-certs volumes: - name: gcloud-certs secret: secretName: px-gcloud items: - key: gcloud.json path: gcloud.json EOF
Apply the patch and wait till all the Portworx pods are back in running state.
$ kubectl -n kube-system patch ds portworx --patch "$(cat patch.yaml)" --type=strategic
Make sure that you applied the above changes to both the source and destination clusters.
Download storkctl CLI before proceeding further. This utility will help us in dealing with the migration.
$ curl http://openstorage-stork.s3-website-us-east-1.amazonaws.com/storkctl/2.3.0/darwin/storkctl -o storkctl $ sudo mv storkctl /usr/local/bin $ sudo chmod +x /usr/local/bin/storkctl
Pairing GKE Clusters for Kubemotion
Before starting the migration, we need to pair the destination cluster with the source.
Getting the Cluster Token
Switch to the destination (prod) cluster and run the below commands to get the token.
$ kubectx prod $ PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}') $ kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl cluster token show -j
Save the token shown in the above output.
Getting the IP Address of GKE Node in Target Cluster
Make sure kubectl is still pointed to the prod environment, and run the below command to get the IP address of the first node in the GKE cluster. Make a note of the IP address.
kubectl get nodes -o jsonpath='{ $.items[0].status.addresses[?(@.type=="ExternalIP")].address }'
Creating and Verifying the Cluster Pair
Keep the public IP address of the load balancer and the cluster token retrieved from the above steps handy.
Let’s use storkctl on the destination cluster to generate a YAML template to create the cluster pair.
$ storkctl generate clusterpair -n cms remotecluster > clusterpair.yaml
Open clusterpair.yaml and add the below details under options. They reflect the public IP address of a node in the destination cluster, port, and the token associated with the destination cluster.
Let’s switch to the source cluster (dev environment) to pair the clusters.
$ storkctl generate clusterpair -n cms remotecluster > clusterpair.yaml
The output of <pre>kubectl get clusterpairs</pre> confirms that the pairing has been done.
We can also verify this with the storkctl CLI.
$ storkctl -n=cms get clusterpair
Congratulations! You have successfully paired the source and destination clusters. We are now ready to start the migration.
Migrating the CMS application from Source to Destination
Make sure that you are using the dev context and follow the below steps to start the migration of the CMS application.
Starting the Migration Job
Create a YAML file called migration.yaml with the below content:
apiVersion: stork.libopenstorage.org/v1alpha1 kind: Migration metadata: name: cmsmigration namespace: cms spec: # This should be the name of the cluster pair created above clusterPair: remotecluster # If set to false this will migrate only the Portworx volumes. No PVCs, apps, etc will be migrated includeResources: true # If set to false, the deployments and stateful set replicas will be set to 0 on the destination. # There will be an annotation with "stork.openstorage.org/migrationReplicas" on the destination to store the replica count from the source. startApplications: true # List of namespaces to migrate namespaces: - cms
This definition contains critical information, such as the name of the cluster pair, namespaces to be included in the migration, and the type of resources to be migrated.
Submit the YAML file to the dev cluster to initiate the migration job.
$ kubectl apply -f migration.yaml migration.stork.libopenstorage.org/cmsmigration created
Tracking and Monitoring the Migration Job
We can monitor the migration through storkctl.
$ storkctl get migration -n cms
Once the migration is done, storkctl reports the final number of volumes and resources migrated to the destination cluster.
To get detailed information on the migration, run the below command:
$ kubectl describe migration cmsmigration -n=cms Name: cmsmigration Namespace: cms Labels: Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"stork.libopenstorage.org/v1alpha1","kind":"Migration","metadata":{"annotations":{},"name":"cmsmigration","namespace":"cms"}... API Version: stork.libopenstorage.org/v1alpha1 Kind: Migration Metadata: Creation Timestamp: 2019-09-25T01:16:49Z Generation: 9 Resource Version: 283347 Self Link: /apis/stork.libopenstorage.org/v1alpha1/namespaces/cms/migrations/cmsmigration UID: 264a73f3-df32-11e9-ad33-42010aa0011e Spec: Cluster Pair: remotecluster Include Resources: true Include Volumes: true Namespaces: cms Post Exec Rule: Pre Exec Rule: Selectors: Start Applications: true Status: Finish Timestamp: 2019-09-25T01:17:40Z Resources: Group: core Kind: PersistentVolume Name: pvc-05b72faa-df32-11e9-ad33-42010aa0011e Namespace: Reason: Resource migrated successfully Status: Successful Version: v1 Group: core Kind: PersistentVolume Name: pvc-f9ae830d-df31-11e9-ad33-42010aa0011e Namespace: Reason: Resource migrated successfully Status: Successful Version: v1 Group: core Kind: PersistentVolumeClaim Name: px-mysql-pvc Namespace: cms Reason: Resource migrated successfully Status: Successful Version: v1 Group: core Kind: PersistentVolumeClaim Name: px-wp-pvc Namespace: cms Reason: Resource migrated successfully Status: Successful Version: v1 Group: core Kind: Secret Name: default-token-9699f Namespace: cms Reason: Resource migrated successfully Status: Successful Version: v1 Group: core Kind: Secret Name: default-token-kmkt5 Namespace: cms Reason: Resource migrated successfully Status: Successful Version: v1 Group: core Kind: Secret Name: default-token-v6gj2 Namespace: cms Reason: Resource migrated successfully Status: Successful Version: v1 Group: core Kind: Service Name: mysql Namespace: cms Reason: Resource migrated successfully Status: Successful Version: v1 Group: core Kind: Service Name: wordpress Namespace: cms Reason: Resource migrated successfully Status: Successful Version: v1 Group: apps Kind: Deployment Name: mysql Namespace: cms Reason: Resource migrated successfully Status: Successful Version: v1 Group: apps Kind: Deployment Name: wordpress Namespace: cms Reason: Resource migrated successfully Status: Successful Version: v1 Stage: Final Status: Successful Volumes: Namespace: cms Persistent Volume Claim: px-mysql-pvc Reason: Migration successful for volume Status: Successful Volume: pvc-f9ae830d-df31-11e9-ad33-42010aa0011e Namespace: cms Persistent Volume Claim: px-wp-pvc Reason: Migration successful for volume Status: Successful Volume: pvc-05b72faa-df32-11e9-ad33-42010aa0011e Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Successful 2m6s stork Volume pvc-f9ae830d-df31-11e9-ad33-42010aa0011e migrated successfully Normal Successful 2m6s stork Volume pvc-05b72faa-df32-11e9-ad33-42010aa0011e migrated successfully Normal Successful 2m3s stork /v1, Kind=PersistentVolume /pvc-05b72faa-df32-11e9-ad33-42010aa0011e: Resource migrated successfully Normal Successful 2m3s stork /v1, Kind=PersistentVolume /pvc-f9ae830d-df31-11e9-ad33-42010aa0011e: Resource migrated successfully Normal Successful 2m3s stork /v1, Kind=PersistentVolumeClaim cms/px-mysql-pvc: Resource migrated successfully Normal Successful 2m3s stork /v1, Kind=PersistentVolumeClaim cms/px-wp-pvc: Resource migrated successfully Normal Successful 2m3s stork /v1, Kind=Secret cms/default-token-9699f: Resource migrated successfully Normal Successful 2m3s stork /v1, Kind=Secret cms/default-token-kmkt5: Resource migrated successfully Normal Successful 2m2s stork /v1, Kind=Secret cms/default-token-v6gj2: Resource migrated successfully Normal Successful 2m (x4 over 2m2s) stork (combined from similar events): apps/v1, Kind=Deployment cms/wordpress: Resource migrated successfully
Kubemotion has successfully migrated all the Portworx Volumes, PVCs, Deployments, Secrets, and Services from source to the destination cluster.
Verifying the Migration in the Production Environment
Now we can switch the context to prod and check all the resources created within the cms namespace.
You can also access the application by visiting the Load Balancer IP of WordPress service.
Summary
Kubemotion extends the power of portability to stateful workloads. It can be used to seamlessly migrate volumes from on-premises to public cloud (hybrid) environments and cross-cloud platforms.
Share
Subscribe for Updates
About Us
Portworx is the leader in cloud native storage for containers.
Thanks for subscribing!
Janakiram MSV
Contributor | Certified Kubernetes Administrator (CKA) and Developer (CKAD)Explore Related Content:
- gke
- google kubernetes engine
- kubemotion
- kubernetes