How to Migrate Stateful Applications from One GCP Region to Another with Portworx Kubemotion

Portworx is a cloud native storage platform to run persistent workloads deployed on a variety of orchestration engines, including Kubernetes. With Portworx, customers can manage the database of their choice on any infrastructure using any container scheduler. It provides a single data management layer for all stateful services, no matter where they run.

Kubemotion is one of the core building blocks of Portworx storage infrastructure. Introduced in Portworx Enterprise 2.0, it allows Kubernetes users to migrate application data and Kubernetes application configurations between clusters, enabling migration, backup & recovery, blue-green deployments, and more.

This step-by-step guide demonstrates how to move persistent volumes and Kubernetes resources associated with a stateful application from one GCP region to another.

Background

For enterprises, it’s a common scenario to run development and test environments in one cloud region and the production environment in another. Development teams may choose a region that’s geographically closer to them while deploying production applications in another region that has low latency for the users and customers.

Even though Kubernetes makes it easy to move stateless workloads across environments, achieving parity of stateful workloads remains a challenge.

For this walkthrough, we will move Kubernetes resources between Google Kubernetes Engine (GKE) clusters running in Asia South (Mumbai) and Asia Southeast (Singapore). The Mumbai region is used by the development teams for dev/test and the Singapore region for the production environment.

After thoroughly testing an application in dev/test, the team will use Portworx and Kubemotion to reliably move the storage volumes and application resources from the development to the production environment.

Exploring the Environments

We have two GKE clusters—dev and production—running in the Mumbai and Singapore regions of Google Cloud Platform. Both of them have the latest version of Portworx cluster up and running.

The above GKE cluster represents the development environment running in the Mumbai (asia-south1) region of GCP.

The above GKE cluster represents the development environment running in the Singapore (asia-southeast1) region of GCP.

The dev/test environment currently runs a LAMP-based content management system that needs to be migrated to production.

Tip: To navigate between two kubeconfig contexts representing different clusters, use the kubectx tool and rename the contexts with dev and prod labels.

kubectx 
kubectx dev=.
kubectx 
kubectx prod=.

It runs two deployments—MySQL and WordPress—in the cms namespace.

For a detailed tutorial on configuring a highly available WordPress stack on GKE, please refer to this guide.

The persistent volumes attached to these pods are backed by a Portworx storage cluster.

The volume below is attached to the MySQL pod.

For the WordPress CMS, there is a shared Portworx volume attached to the pods.

The fully configured application is accessed through the IP address of the load balancer.

We will now migrate this application along with its state and configuration from dev/test to production.

Preparing the Source and Target Environments

Before we can move the volumes, we need to configure the source and destination clusters.

Follow the below steps to prepare the environments.

Note: Kubemotion may also be configured with Kubernetes secrets. Since the scenario is based on Google Cloud Platform, we are demonstrating the integration with Google Cloud KMS and Service Accounts.

Enabling the GCP APIs

In your GCP account, make sure that the Google Cloud Key Management Service (KMS) APIs are enabled. Portworx integrates with Google Cloud KMS to store Portworx secrets for Volume Encryption and Cloud Credentials.

Creating the Service Account

After enabling the APIs, create a service account that has roles “compute admin” and “KMS admin” added.

The last step involves the creation of a file that contains the private key. Copy the downloaded account file in a directory gcloud-secrets/ and rename it gcloud.json to create a Kubernetes secret from it.

Configuring the KMS Keyring

Run the below command to create a keyring that will be used by Portworx. Make sure that while creating the asymmetric key you specify the purpose of the key as Asymmetric decrypt.

$ gcloud kms keyrings create portworx --location global

Creating the Kubernetes Secret

Run the command below on both the clusters to create a Kubernetes secret from the service account file.

$ kubectl -n kube-system create secret generic px-gcloud --from-file=gcloud-secrets/ --from-literal=gcloud-kms-resource-id=projects//locations//keyRings//cryptoKeys//cryptoKeyVersions/1

Make sure to replace the Project ID, Key Ring Name, and Asymmetric Key Name in the above command with the correct values.

Both the clusters now have a secret that provides access to Google Cloud Platform KMS and GCE APIs.

Patching Portworx Deployment

Now, we need to update the Portworx daemonset to provide access to KMS.

$ kubectl edit daemonset portworx -n kube-system

Add the “-secret_type”, “gcloud-kms” arguments to the portworx container in the daemonset.

containers:
  - args:
    - -c
    - testclusterid
    - -s
    - /dev/sdb
    - -x
    - kubernetes
    - -secret_type
    - gcloud-kms
    name: portworx

 

We are now ready to patch the Portworx daemonset to add the secret created in the previous step.

cat < patch.yaml
spec:
  template:
    spec:
      containers:
      - name: portworx
        env:
          - name: GOOGLE_KMS_RESOURCE_ID
            valueFrom:
              secretKeyRef:
                name: px-gcloud
                key: gcloud-kms-resource-id
          - name: GOOGLE_APPLICATION_CREDENTIALS
            value: /etc/pwx/gce/gcloud.json
        volumeMounts:
          - mountPath: /etc/pwx/gce
            name: gcloud-certs
      volumes:
        - name: gcloud-certs
          secret:
            secretName: px-gcloud
            items:
              - key: gcloud.json
                path: gcloud.json
EOF

 

Apply the patch and wait till all the Portworx pods are back in running state.

$ kubectl -n kube-system patch ds portworx --patch "$(cat patch.yaml)" --type=strategic

Make sure that you applied the above changes to both the source and destination clusters.

Download storkctl CLI before proceeding further. This utility will help us in dealing with the migration.

$ curl http://openstorage-stork.s3-website-us-east-1.amazonaws.com/storkctl/2.3.0/darwin/storkctl -o storkctl
$ sudo mv storkctl /usr/local/bin
$ sudo chmod +x /usr/local/bin/storkctl

Pairing GKE Clusters for Kubemotion

Before starting the migration, we need to pair the destination cluster with the source.

Getting the Cluster Token

Switch to the destination (prod) cluster and run the below commands to get the token.

$ kubectx prod

$ PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')

$ kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl cluster token show -j

 

Save the token shown in the above output.

Getting the IP Address of GKE Node in Target Cluster

Make sure kubectl is still pointed to the prod environment, and run the below command to get the IP address of the first node in the GKE cluster. Make a note of the IP address.

kubectl get nodes -o jsonpath='{ $.items[0].status.addresses[?(@.type=="ExternalIP")].address }'

Creating and Verifying the Cluster Pair

Keep the public IP address of the load balancer and the cluster token retrieved from the above steps handy.

Let’s use storkctl on the destination cluster to generate a YAML template to create the cluster pair.

$ storkctl generate clusterpair -n cms remotecluster > clusterpair.yaml

Open clusterpair.yaml and add the below details under options. They reflect the public IP address of a node in the destination cluster, port, and the token associated with the destination cluster.

Let’s switch to the source cluster (dev environment) to pair the clusters.

$ storkctl generate clusterpair -n cms remotecluster > clusterpair.yaml

The output of <pre>kubectl get clusterpairs</pre> confirms that the pairing has been done.

We can also verify this with the storkctl CLI.

$ storkctl -n=cms get clusterpair

Congratulations! You have successfully paired the source and destination clusters. We are now ready to start the migration.

Migrating the CMS application from Source to Destination

Make sure that you are using the dev context and follow the below steps to start the migration of the CMS application.

Starting the Migration Job

Create a YAML file called migration.yaml with the below content:

apiVersion: stork.libopenstorage.org/v1alpha1
kind: Migration
metadata:
  name: cmsmigration
  namespace: cms
spec:
  # This should be the name of the cluster pair created above
  clusterPair: remotecluster
  # If set to false this will migrate only the Portworx volumes. No PVCs, apps, etc will be migrated
  includeResources: true
  # If set to false, the deployments and stateful set replicas will be set to 0 on the destination.
  # There will be an annotation with "stork.openstorage.org/migrationReplicas" on the destination to store the replica count from the source.
  startApplications: true
  # List of namespaces to migrate
  namespaces:
  - cms

 

This definition contains critical information, such as the name of the cluster pair, namespaces to be included in the migration, and the type of resources to be migrated.

Submit the YAML file to the dev cluster to initiate the migration job.

$ kubectl apply -f migration.yaml
migration.stork.libopenstorage.org/cmsmigration created

Tracking and Monitoring the Migration Job

We can monitor the migration through storkctl.

$ storkctl get migration -n cms

 

Once the migration is done, storkctl reports the final number of volumes and resources migrated to the destination cluster.

 

To get detailed information on the migration, run the below command:

$ kubectl describe migration cmsmigration -n=cms
Name:         cmsmigration
Namespace:    cms
Labels:       
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"stork.libopenstorage.org/v1alpha1","kind":"Migration","metadata":{"annotations":{},"name":"cmsmigration","namespace":"cms"}...
API Version:  stork.libopenstorage.org/v1alpha1
Kind:         Migration
Metadata:
  Creation Timestamp:  2019-09-25T01:16:49Z
  Generation:          9
  Resource Version:    283347
  Self Link:           /apis/stork.libopenstorage.org/v1alpha1/namespaces/cms/migrations/cmsmigration
  UID:                 264a73f3-df32-11e9-ad33-42010aa0011e
Spec:
  Cluster Pair:       remotecluster
  Include Resources:  true
  Include Volumes:    true
  Namespaces:
    cms
  Post Exec Rule:
  Pre Exec Rule:
  Selectors:           
  Start Applications:  true
Status:
  Finish Timestamp:  2019-09-25T01:17:40Z
  Resources:
    Group:      core
    Kind:       PersistentVolume
    Name:       pvc-05b72faa-df32-11e9-ad33-42010aa0011e
    Namespace:
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       PersistentVolume
    Name:       pvc-f9ae830d-df31-11e9-ad33-42010aa0011e
    Namespace:
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       PersistentVolumeClaim
    Name:       px-mysql-pvc
    Namespace:  cms
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       PersistentVolumeClaim
    Name:       px-wp-pvc
    Namespace:  cms
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       Secret
    Name:       default-token-9699f
    Namespace:  cms
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       Secret
    Name:       default-token-kmkt5
    Namespace:  cms
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       Secret
    Name:       default-token-v6gj2
    Namespace:  cms
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       Service
    Name:       mysql
    Namespace:  cms
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       Service
    Name:       wordpress
    Namespace:  cms
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      apps
    Kind:       Deployment
    Name:       mysql
    Namespace:  cms
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      apps
    Kind:       Deployment
    Name:       wordpress
    Namespace:  cms
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
  Stage:        Final
  Status:       Successful
  Volumes:
    Namespace:                cms
    Persistent Volume Claim:  px-mysql-pvc
    Reason:                   Migration successful for volume
    Status:                   Successful
    Volume:                   pvc-f9ae830d-df31-11e9-ad33-42010aa0011e
    Namespace:                cms
    Persistent Volume Claim:  px-wp-pvc
    Reason:                   Migration successful for volume
    Status:                   Successful
    Volume:                   pvc-05b72faa-df32-11e9-ad33-42010aa0011e
Events:
  Type    Reason      Age                From   Message
  ----    ------      ----               ----   -------
  Normal  Successful  2m6s               stork  Volume pvc-f9ae830d-df31-11e9-ad33-42010aa0011e migrated successfully
  Normal  Successful  2m6s               stork  Volume pvc-05b72faa-df32-11e9-ad33-42010aa0011e migrated successfully
  Normal  Successful  2m3s               stork  /v1, Kind=PersistentVolume /pvc-05b72faa-df32-11e9-ad33-42010aa0011e: Resource migrated successfully
  Normal  Successful  2m3s               stork  /v1, Kind=PersistentVolume /pvc-f9ae830d-df31-11e9-ad33-42010aa0011e: Resource migrated successfully
  Normal  Successful  2m3s               stork  /v1, Kind=PersistentVolumeClaim cms/px-mysql-pvc: Resource migrated successfully
  Normal  Successful  2m3s               stork  /v1, Kind=PersistentVolumeClaim cms/px-wp-pvc: Resource migrated successfully
  Normal  Successful  2m3s               stork  /v1, Kind=Secret cms/default-token-9699f: Resource migrated successfully
  Normal  Successful  2m3s               stork  /v1, Kind=Secret cms/default-token-kmkt5: Resource migrated successfully
  Normal  Successful  2m2s               stork  /v1, Kind=Secret cms/default-token-v6gj2: Resource migrated successfully
  Normal  Successful  2m (x4 over 2m2s)  stork  (combined from similar events): apps/v1, Kind=Deployment cms/wordpress: Resource migrated successfully

 

Kubemotion has successfully migrated all the Portworx Volumes, PVCs, Deployments, Secrets, and Services from source to the destination cluster.

Verifying the Migration in the Production Environment

Now we can switch the context to prod and check all the resources created within the cms namespace.

 

You can also access the application by visiting the Load Balancer IP of WordPress service.

Summary

Kubemotion extends the power of portability to stateful workloads. It can be used to seamlessly migrate volumes from on-premises to public cloud (hybrid) environments and cross-cloud platforms.