Portworx & Red Hat Hands-on Labs Register Now
This post is part of our ongoing series on running MariaDB on Kubernetes. We’ve published a number of articles about running MariaDB on Kubernetes for specific platforms and for specific use cases. If you are looking for a specific Kubernetes platform, check out these related articles.
Running HA MariaDB on Google Kubernetes Engine (GKE)
Running HA MariaDB on Amazon Elastic Container Service for Kubernetes (EKS)
Running HA MariaDB on Azure Kubernetes Service (AKS)
Running HA MariaDB with Rancher Kubernetes Engine (RKE)
And now, onto the post…
Red Hat OpenShift is a comprehensive enterprise-grade application platform built for containers powered by Kubernetes. OpenShift lets developers quickly build, develop, and deploy applications on nearly any infrastructure, public or private. It offers enterprises full control over their Kubernetes environments, whether they’re on-premise or in the public cloud, giving teams freedom to build and run applications anywhere.
Portworx recently achieved Red Hat certification for Red Hat OpenShift Container Platform and PX-Enterprise is available in the Red Hat Container Catalog. This certification enables enterprises to confidently run high-performance stateful applications like databases, big and fast data workloads, and machine learning applications on the Red Hat OpenShift Container Platform. Learn more about Portworx & OpenShift in our Product Brief.
This tutorial is a walk-through of the steps involved in deploying, managing, and backing up a highly available MariaDB database on OpenShift.
In summary, to backup and recover an HA MariaDB database on OpenShift you need to:
- Create an OpenShift cluster running at least three nodes
- Install a cloud native storage solution like Portworx as a DaemonSet on OpenShift
- Create a storage class defining your storage requirements like replication factor, snapshot policy, and performance profile
- Deploy MariaDB using Kubernetes
- Take a snapshot of Kubernetes PV
- Restore MariaDB using the Kubernetes persistent volume snapshot
How to install and configure an OpenShift Origin cluster
OpenShift Origin can be deployed in a variety of environments ranging from VirtualBox to a public cloud IaaS such as Amazon, Google, Azure. Refer to the official installation guide for the steps involved in setting up your own cluster. For this guide, we run an OpenShift Origin cluster in Microsoft Azure. Follow the instructions mentioned in Azure documentation.
Your OpenShift cluster setup should look similar to the below configuration. It is recommended that you run at least 3 nodes for the HA configuration.
$ oc get nodes NAME STATUS ROLES AGE VERSION mycluster-infra-0 Ready none 8d v1.9.1+a0ce1bc657 mycluster-master-0 Ready master 8d v1.9.1+a0ce1bc657 mycluster-node-0 Ready compute 8d v1.9.1+a0ce1bc657 mycluster-node-1 Ready compute 8d v1.9.1+a0ce1bc657 mycluster-node-2 Ready compute 8d v1.9.1+a0ce1bc657
Though almost all the steps can be performed through the OpenShift Console, we are using the oc CLI
. Please note that most of the kubectl
commands are available through oc
tool. You may find the tools used interchangeably.
Installing Portworx on OpenShift
Since OpenShift is based on Kubernetes, the steps involved in installing Portworx are not very different from the standard Kubernetes installation. Portworx documentation has a detailed guide with the prerequisites and all the steps to install on OpenShift.
Before proceeding further, ensure that Portworx is up and running on OpenShift.
$ oc get pods -n=kube-system -l name=portworx portworx-27wtw 1/1 Running 1 1d portworx-lfh6b 1/1 Running 0 1d portworx-q8j94 1/1 Running 0 1d
We can check the status of Portworx by running the following commands:
$ PX_POD=$(oc get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}') $ oc exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl status Status: PX is operational License: Trial (expires in 30 days) Node ID: mycluster-node-2 IP: 10.2.0.4 Local Storage Pool: 1 pool POOL IO_PRIORITY RAID_LEVEL USABLE USED STATUS ZONE REGION 0 LOW raid0 20 GiB 3.1 GiB Online default default Local Storage Devices: 1 device Device Path Media Type Size Last-Scan 0:1 /dev/sdd STORAGE_MEDIUM_MAGNETIC 20 GiB 06 Aug 18 16:58 UTC total - 20 GiB Cluster Summary Cluster ID: px-cluster-8764 Cluster UUID: c29f07f2-dada-4f36-8f29-1f1ba2d5e198 Scheduler: kubernetes Nodes: 3 node(s) with storage (3 online) IP ID StorageNode Used Capacity Status StorageStatus Version Kernel OS 10.2.0.4 mycluster-node-2 Yes 3.1 GiB 20 GiB Online Up (This node) 1.4.0.0-0753ff93.10.0-862.9.1.el7.x86_64 CentOS Linux 7 (Core) 10.2.0.6 mycluster-node-1 Yes 3.1 GiB 20 GiB Online Up 1.4.0.0-0753ff93.10.0-862.9.1.el7.x86_64 CentOS Linux 7 (Core) 10.2.0.5 mycluster-node-0 Yes 3.1 GiB 20 GiB Online Up 1.4.0.0-0753ff93.10.0-862.9.1.el7.x86_64 CentOS Linux 7 (Core) Global Storage Pool Total Used : 9.2 GiB Total Capacity : 60 GiB
Once OpenShift Origin cluster is up and running and Portworx is installed and configured, we will deploy a highly available MariaDB database.
Creating a Kubernetes storage class for MariaDB
Through storage class objects, an admin can define different classes of Portworx volumes that are offered in a cluster. These classes will be used during the dynamic provisioning of volumes. The storage class defines the replication factor, IO profile (e.g. for a database or a CMS), and priority (e.g. SSD or HDD). These parameters impact the availability and throughput of workload and can be specified for each volume. This is important because a production database will have different requirements than a development Jenkins cluster.
In this example, the storage class that we deploy has a replication factor of 3 with I/O profile set to “db,” and priority set to “high.” This means that the storage will be optimized for low latency database workloads like MariaDB and automatically placed on the highest performance storage available in the cluster. Notice that we also mention the filesystem, xfs in the storage class.
$ cat > px-mariadb-sc.yaml << EOF kind: StorageClass apiVersion: storage.k8s.io/v1beta1 metadata: name: px-ha-sc provisioner: kubernetes.io/portworx-volume parameters: repl: "3" io_profile: "db_remote" priority_io: "high" fs: "xfs" EOF
$ oc create -f px-mariadb-sc.yaml storageclass.storage.k8s.io "px-ha-sc" created $ oc get sc NAME PROVISIONER AGE px-ha-sc kubernetes.io/portworx-volume 10s stork-snapshot-sc stork-snapshot 3d
Create the storage class and verify its available in the default
namespace.
$ oc create -f px-mariadb-sc.yaml storageclass.storage.k8s.io "px-ha-sc" created $ oc get sc NAME PROVISIONER AGE generic (default) kubernetes.io/azure-disk 52m px-ha-sc kubernetes.io/portworx-volume 13s stork-snapshot-sc stork-snapshot 17m
Creating a MariaDB PVC on OpenShift
We can now create a Persistent Volume Claim (PVC) based on the Storage Class. Thanks to dynamic provisioning, the claims will be created without explicitly provisioning a persistent volume (PV).
$ cat > px-mariadb-pvc.yaml << EOF kind: PersistentVolumeClaim apiVersion: v1 metadata: name: px-mariadb-pvc annotations: volume.beta.kubernetes.io/storage-class: px-ha-sc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi EOF $ persistentvolumeclaim "px-mariadb-pvc" created $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE px-mariadb-pvc Bound pvc-739a7151-9a8d-11e8-9135-000d3a1a1cdf 1Gi RWO px-ha-sc 13s
Deploying MariaDB on OpenShift
Finally, let’s create a MariaDB instance as a Kubernetes deployment object. For simplicity’s sake, we will just be deploying a single mariadb pod. Because Portworx provides synchronous replication for High Availability, a single MariaDB instance might be the best deployment option for your MariaDB database. Portworx can also provide backing volumes for multi-node MariaDB cluster. The choice is yours.
$ cat > px-mariadb-app.yaml << EOF apiVersion: apps/v1 kind: Deployment metadata: name: mariadb spec: selector: matchLabels: app: mariadb strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate replicas: 1 template: metadata: labels: app: mariadb spec: schedulerName: stork containers: - name: mariadb image: mariadb:latest imagePullPolicy: "Always" env: - name: MYSQL_ROOT_PASSWORD value: password ports: - containerPort: 3306 volumeMounts: - mountPath: /var/lib/mysql name: mariadb-data volumes: - name: mariadb-data persistentVolumeClaim: claimName: px-mariadb-pvc EOF
$ oc create -f px-mariadb-app.yaml deployment "mariadb" created
The MariaDB deployment defined above is explicitly associated with the PVC, px-mariadb-pvc
created in the previous step.
This deployment creates a single pod running MariaDB backed by Portworx.
$ oc get pods NAME READY STATUS RESTARTS AGE docker-registry-2-7dwtc 1/1 Running 1 1d mariadb-654cc68f68-gxxbd 1/1 Running 0 22s registry-console-1-j9cg2 1/1 Running 1 1d router-1-lqsxs 1/1 Running 1 1d
We can inspect the Portworx volume by accessing the pxctl
tool running with the MariaDB pod.
$ VOL=`oc get pvc | grep px-mariadb-pvc | awk '{print $3}'` $ PX_POD=$(oc get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}') $ oc exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl volume inspect ${VOL} Volume : 984374852040473937 Name : pvc-739a7151-9a8d-11e8-9135-000d3a1a1cdf Size : 1.0 GiB Format : xfs HA : 3 IO Priority : LOW Creation time : Aug 7 22:01:35 UTC 2018 Shared : no Status : up State : Attached: mycluster-node-1 (10.2.0.6) Device Path : /dev/pxd/pxd984374852040473937 Labels : pvc=px-mariadb-pvc Reads : 138 Reads MS : 62 Bytes Read : 2371584 Writes : 659 Writes MS : 96013 Bytes Written : 172965888 IOs in progress : 0 Bytes used : 126 MiB Replica sets on nodes: Set 0 Node : 10.2.0.6 (Pool 0) Node : 10.2.0.5 (Pool 0) Node : 10.2.0.4 (Pool 0) Replication Status : Up Volume consumers : - Name : mariadb-654cc68f68-gxxbd (e9daceb6-9a8d-11e8-9135-000d3a1a1cdf) (Pod) Namespace : default Running on : mycluster-node-1 Controlled by : mariadb-654cc68f68 (ReplicaSet)
The screenshot looks similar to the one shown below:
Failing over MariaDB pod on OpenShift
Populating sample data
Let’s populate the database with some sample data.
We will first find the pod that’s running MariaDB to access the shell.
$ POD=`oc get pods -l app=mariadb | grep Running | grep 1/1 | awk '{print $1}'` $ oc exec -it $POD -- mariadb -uroot -ppassword Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 11 Server version: 10.4.6-MariaDB-1:10.4.6+maria~bionic mariadb.org binary distribution Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]>
Now that we are inside the shell, we can populate create a sample database and table.
MariaDB> CREATE DATABASE `classicmodels`; MariaDB> USE `classicmodels`; MariaDB> CREATE TABLE `offices` ( `officeCode` varchar(10) NOT NULL, `city` varchar(50) NOT NULL, `phone` varchar(50) NOT NULL, `addressLine1` varchar(50) NOT NULL, `addressLine2` varchar(50) DEFAULT NULL, `state` varchar(50) DEFAULT NULL, `country` varchar(50) NOT NULL, `postalCode` varchar(15) NOT NULL, `territory` varchar(10) NOT NULL, PRIMARY KEY (`officeCode`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1; Query OK, 0 rows affected (0.227 sec) MariaDB> insert into `offices`(`officeCode`,`city`,`phone`,`addressLine1`,`addressLine2`,`state`,`country`,`postalCode`,`territory`) values ('1','San Francisco','+1 650 219 4782','100 Market Street','Suite 300','CA','USA','94080','NA'), ('2','Boston','+1 215 837 0825','1550 Court Place','Suite 102','MA','USA','02107','NA'), ('3','NYC','+1 212 555 3000','523 East 53rd Street','apt. 5A','NY','USA','10022','NA'), ('4','Paris','+33 14 723 4404','43 Rue Jouffroy D\'abbans',NULL,NULL,'France','75017','EMEA'), ('5','Tokyo','+81 33 224 5000','4-1 Kioicho',NULL,'Chiyoda-Ku','Japan','102-8578','Japan'), ('6','Sydney','+61 2 9264 2451','5-11 Wentworth Avenue','Floor #2',NULL,'Australia','NSW 2010','APAC'), ('7','London','+44 20 7877 2041','25 Old Broad Street','Level 7',NULL,'UK','EC2N 1HN','EMEA'); Query OK, 7 rows affected (0.039 sec) Records: 7 Duplicates: 0 Warnings: 0
Let’s run a few queries on the table.
MariaDB> select `officeCode`,`city`,`phone`,`addressLine1`,`city` from `offices`; +------------+---------------+------------------+--------------------------+---------------+ | officeCode | city | phone | addressLine1 | city | +------------+---------------+------------------+--------------------------+---------------+ | 1 | San Francisco | +1 650 219 4782 | 100 Market Street | San Francisco | | 2 | Boston | +1 215 837 0825 | 1550 Court Place | Boston | | 3 | NYC | +1 212 555 3000 | 523 East 53rd Street | NYC | | 4 | Paris | +33 14 723 4404 | 43 Rue Jouffroy D'abbans | Paris | | 5 | Tokyo | +81 33 224 5000 | 4-1 Kioicho | Tokyo | | 6 | Sydney | +61 2 9264 2451 | 5-11 Wentworth Avenue | Sydney | | 7 | London | +44 20 7877 2041 | 25 Old Broad Street | London | +------------+---------------+------------------+--------------------------+---------------+ 7 rows in set (0.01 sec)
Find all the offices in USA.
MariaDB [classicmodels]> select `officeCode`, `city`, `phone` from `offices` where `country` = "USA"; +------------+---------------+-----------------+ | officeCode | city | phone | +------------+---------------+-----------------+ | 1 | San Francisco | +1 650 219 4782 | | 2 | Boston | +1 215 837 0825 | | 3 | NYC | +1 212 555 3000 | +------------+---------------+-----------------+ 3 rows in set (0.00 sec)
Exit from the MariaDB shell to return to the host.
Simulating node failure
Now, let’s simulate node failure by cordoning off the OpenShift node on which MariaDB is running.
$ NODE=`oc get pods -l app=mariadb -o wide | grep -v NAME | awk '{print $7}'` $ oc adm cordon ${NODE} node "mycluster-node-1" cordoned
The above command disabled scheduling on one of the nodes.
$ oc get nodes NAME STATUS ROLES AGE VERSION NAME STATUS ROLES AGE VERSION mycluster-infra-0 Ready 1h v1.9.1+a0ce1bc657 mycluster-master-0 Ready master 1h v1.9.1+a0ce1bc657 mycluster-node-0 Ready compute 1h v1.9.1+a0ce1bc657 mycluster-node-1 Ready,SchedulingDisabled compute 1h v1.9.1+a0ce1bc657 mycluster-node-2 Ready compute 1h v1.9.1+a0ce1bc657
Now, let’s go ahead and delete the mariadbDB pod.
$ POD=`oc get pods -l app=mariadb -o wide | grep -v NAME | awk '{print $1}'` $ oc delete pod ${POD} pod "mariadb-654cc68f68-gxxbd" deleted
As soon as the pod is deleted, it is relocated to the node with the replicated data. Storage Orchestrator for Kubernetes (STORK), a Portworx-contributed open source storage scheduler, ensures that the pod is co-located on the exact node where the data is stored. It ensures that an appropriate node is selected for scheduling the pod.
Let’s verify this by running the below command. We will notice that a new pod has been created and scheduled in a different node.
$ oc get pods -l app=mariadb -o wide NAME READY STATUS RESTARTS AGE IP NODE mariadb-97b758c4c-sssfg 1/1 Running 0 18s 10.129.0.7 mycluster-node-2
Let’s uncordon the node to bring it back to action.
$ oc adm uncordon ${NODE} node "mycluster-node-1" uncordoned
Finally, let’s verify that the data is still available.
Verifying that the data is intact
Let’s find the pod name and run the ‘exec’ command, and then access the MariaDB shell.
$ oc exec -it $POD -- mysql -uroot -ppassword Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 8 Server version: 10.4.6-MariaDB-1:10.4.6+maria~bionic mariadb.org binary distribution Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]>
We will query the database to verify that the data is intact.
MariaDB [none]> USE `classicmodels`; MariaDB [classicmodels]> select `officeCode`, `city`, `phone` from `offices` where `country` = "USA"; +------------+---------------+-----------------+ | officeCode | city | phone | +------------+---------------+-----------------+ | 1 | San Francisco | +1 650 219 4782 | | 2 | Boston | +1 215 837 0825 | | 3 | NYC | +1 212 555 3000 | +------------+---------------+-----------------+ 3 rows in set (0.00 sec)
Observe that the database table is still there and all the content is intact! Exit from the client shell to return to the host.
Taking Snapshots of a Kubernetes volume and restoring the database
Portworx supports creating snapshots for OpenShift PVCs.
Let’s create a snapshot for the PVC we created for MariaDB.
cat > px-mariadb-snap.yaml << EOF apiVersion: volumesnapshot.external-storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: px-mariadb-snapshot namespace: default spec: persistentVolumeClaimName: px-mariadb-pvc EOF
$ oc create -f px-mariadb-snap.yaml volumesnapshot.volumesnapshot.external-storage.k8s.io "px-mariadb-snapshot" created
Verify the creation of volume snapshot.
$ oc get volumesnapshot NAME AGE px-mariadb-snapshot 30s
$ oc get volumesnapshotdatas NAME AGE k8s-volume-snapshot-6ab731c7-9278-11e8-b018-e2f4b6cbb690 34s
With the snapshot in place, let’s go ahead and delete the database.
$ POD=`oc get pods -l app=mariadb | grep Running | grep 1/1 | awk '{print $1}'` $ oc exec -it $POD -- mysql -uroot -ppassword
Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 9 Server version: 10.4.6-MariaDB-1:10.4.6+maria~bionic mariadb.org binary distribution Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]>
drop database classicmodels;
Since snapshots are just like volumes, we can use it to start a new instance of MariaDB. Let’s create a new instance of MariaDB by restoring the snapshot data.
$ cat > px-mariadb-snap-pvc << EOF apiVersion: v1 kind: PersistentVolumeClaim metadata: name: px-mariadb-snap-clone annotations: snapshot.alpha.kubernetes.io/snapshot: px-mariadb-snapshot spec: accessModes: - ReadWriteOnce storageClassName: stork-snapshot-sc resources: requests: storage: 2Gi EOF $ oc create -f px-mariadb-snap-pvc.yaml persistentvolumeclaim "px-mariadb-snap-clone" created
From the new PVC, we will create a MariaDB pod.
$ cat < px-mariadb-snap-restore.yaml >> EOF apiVersion: apps/v1 kind: Deployment metadata: name: mariadb-snap spec: selector: matchLabels: app: mariadb-snap spec: strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate replicas: 1 template: metadata: labels: app: mariadb-snap spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: px/running operator: NotIn values: - "false" - key: px/enabled operator: NotIn values: - "false" spec: containers: - name: mariadb image: mariadb:latest imagePullPolicy: "Always" env: - name: MYSQL_ROOT_PASSWORD value: password ports: - containerPort: 3306 volumeMounts: - mountPath: /var/lib/mysql name: mariadb-data volumes: - name: mariadb-data persistentVolumeClaim: claimName: px-mariadb-snap-clone EOF
$ oc create -f px-mariadb-snap-restore.yaml deployment "mariadb-snap" created
Verify that the new pod is in running state.
$ oc get pods -l app=mariadb-snap NAME READY STATUS RESTARTS AGE mariadb-snap-5ddd6b6848-bb6wx 1/1 Running 0 30s
Finally, let’s access the sample data created earlier in the walkthrough.
$ POD=`oc get pods -l app=mariadb-snap | grep Running | grep 1/1 | awk '{print $1}'` $ oc exec -it $POD -- mysql -uroot -ppassword Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 8 Server version: 10.4.6-MariaDB-1:10.4.6+maria~bionic mariadb.org binary distribution Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> MariaDB [(none)]> USE `classicmodels`; MariaDB [classicmodels]> select `officeCode`, `city`, `phone` from `offices` where `country` = "USA"; +------------+---------------+-----------------+ | officeCode | city | phone | +------------+---------------+-----------------+ | 1 | San Francisco | +1 650 219 4782 | | 2 | Boston | +1 215 837 0825 | | 3 | NYC | +1 212 555 3000 | +------------+---------------+-----------------+ 3 rows in set (0.00 sec)
Notice that the collection is still there with the data intact.
We can also push the snapshot to Amazon S3 if we want to create a Disaster Recovery backup in another region. Portworx snapshots also work with any S3 compatible object storage, so the backup can go to a different cloud or even an on-premises data center. Alternatively, we can stretch a single Portworx cluster across two independent Kubernetes clusters for Zero RPO DR for Kubernetes.
Summary
Portworx can be easily deployed on Red Hat OpenShift to run stateful workloads in production, including mission-critical data management functions like backup and restore. Through the integration of Portworx and OpenShift, DevOps and DataOps teams can seamlessly run highly available database clusters in OpenShift. They can perform traditional operations such as volume expansion, snapshots, backup and recovery for the cloud-native applications.
Share
Subscribe for Updates
About Us
Portworx is the leader in cloud native storage for containers.
Thanks for subscribing!
Janakiram MSV
Contributor | Certified Kubernetes Administrator (CKA) and Developer (CKAD)Explore Related Content:
- kubernetes
- mariadb
- red hat
- red hat OpenShift