How to Run HA MariaDB on Amazon EKS

Contact Sales

How To

This post is part of our ongoing series on running MariaDB on Kubernetes. We’ve published a number of articles about running MariaDB on Kubernetes for specific platforms and for specific use cases. If you are looking for a specific Kubernetes platform, check out these related articles.

Running HA MariaDB on Google Kubernetes Engine (GKE)

Running HA MariaDB on Azure Kubernetes Service (AKS)

Running HA MariaDB on Red Hat OpenShift

Running HA MariaDB with Rancher Kubernetes Engine (RKE)

And now, onto the post…

Amazon EKS is a fully managed Kubernetes environment running in AWS. With Amazon EKS customers get a highly-available, secure Kubernetes control plane without needing to worry about provisioning, upgrades, or patching. Amazon EKS is certified by the Cloud Native Computing Foundation (CNCF) as Kubernetes conformant, which means it supports all existing plugins and tooling from the Kubernetes community, including Portworx.

Portworx, an EKS launch partner, is a cloud native storage platform to run persistent workloads deployed on a variety of orchestration engines including Kubernetes. With Portworx, customers can manage the database of their choice on any infrastructure using any container scheduler. It provides a single data management layer for all stateful services, no matter where they run.

This tutorial is a walk-through of the steps involved in deploying and managing a highly available MariaDB database on Amazon EKS.

In summary, to run HA MariaDB on Amazon you need to:

Install an EKS cluster by following instructions in the Amazon docs
Install a cloud native storage solution like Portworx as a DaemonSet on EKS
Create a storage class defining your storage requirements like replication factor, snapshot policy, and performance profile
Deploy MariaDB using Kubernetes
Test failover by killing or cordoning nodes in your cluster

How to set up an EKS cluster

Portworx is fully supported on Amazon EKS. Please follow the instructions to configure an Amazon EKS cluster.

You should have a three node Kubernetes cluster deployed based on the default EKS configuration.

$ kubectl get nodes
NAME                                            STATUS    ROLES     AGE       VERSION
ip-192-168-128-254.us-west-2.compute.internal   Ready     none    8d        v1.10.3
ip-192-168-217-164.us-west-2.compute.internal   Ready     none    8d        v1.10.3
ip-192-168-94-92.us-west-2.compute.internal     Ready     none    8d        v1.10.3

px-mariadb-eks-0

Installing Portworx in EKS

Installing Portworx on Amazon EKS is not very different from installing it on a Kubernetes cluster setup through Kops. Portworx EKS documentation has the steps involved in running the Portworx cluster in a Kubernetes environment deployed in AWS.

Portworx cluster need to be up and running on EKS before proceeding to the next step. The kube-system namespace should have the Portworx pods in running state.

$ kubectl get pods -n=kube-system -l name=portworx
NAME             READY     STATUS    RESTARTS   AGE
portworx-blqjh   1/1       Running   0          8d
portworx-c8bf2   1/1       Running   0          8d
portworx-z2j6z   1/1       Running   0          8d

px-mariadb-eks-1

Creating a storage class for MariaDB

Once the EKS cluster is up and running, and Portworx is installed and configured, we will deploy a highly available MariaDB database.

Through storage class objects, an admin can define different classes of Portworx volumes that are offered in a cluster. These classes will be used during the dynamic provisioning of volumes. The Storage Class defines the replication factor, I/O profile (e.g., for a database or a CMS), and priority (e.g., SSD or HDD). These parameters impact the availability and throughput of workloads and can be specified for each volume. This is important because a production database will have different requirements than a development Jenkins cluster.

In this example, the storage class that we deploy has a replication factor of 3 with I/O profile set to “db,” and priority set to “high.” This means that the storage will be optimized for low latency database workloads like MariaDB and automatically placed on the highest performance storage available in the cluster.

$ cat > px-mariadb-sc.yaml << EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
    name: px-ha-sc
provisioner: kubernetes.io/portworx-volume
parameters:
   repl: "3"
EOF

$ kubectl create -f px-mariadb-sc.yaml
storageclass.storage.k8s.io "px-ha-sc" created

$ kubectl get sc
NAME                PROVISIONER                     AGE
px-ha-sc            kubernetes.io/portworx-volume   10s
stork-snapshot-sc   stork-snapshot                  3d

Creating a MariaDB PVC on Kubernetes

We can now create a Persistent Volume Claim (PVC) based on the Storage Class. Thanks to dynamic provisioning, the claims will be created without explicitly provisioning Persistent Volume (PV).

$ cat > px-mariadb-pvc.yaml << EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
   name: px-mariadb-pvc
   annotations:
     volume.beta.kubernetes.io/storage-class: px-ha-sc
spec:
   accessModes:
     - ReadWriteOnce
   resources:
     requests:
       storage: 1Gi
EOF

$ kubectl create -f px-mariadb-pvc.yaml
persistentvolumeclaim "mariadb" created

$ kubectl get pvc
NAME           STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
px-mariadb-pvc   Bound     pvc-3a6788df-9274-11e8-8c5e-0253036635a0   1Gi        RWO            px-ha-sc       17s

Deploying MariaDB on EKS

Finally, let’s create a MariaDB instance as a Kubernetes deployment object. For simplicity’s sake, we will just be deploying a single MariaDB pod. Because Portworx provides synchronous replication for High Availability, a single MariaDB instance might be the best deployment option for your MariaDB database. Portworx can also provide backing volumes for multi-node MariaDB cluster. The choice is yours.

$ cat > px-mariadb-app.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mariadb
spec:
  selector:
    matchLabels:
      app: mariadb
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  replicas: 1
  template:
    metadata:
      labels:
        app: mariadb
    spec:
      schedulerName: stork
      containers:
      - name: mariadb
        image: mariadb:latest
        imagePullPolicy: "Always"
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: mariadb-data
      volumes:
      - name: mariadb-data
        persistentVolumeClaim:
          claimName: px-mariadb-pvc
EOF

$ kubectl create -f px-mariadb-app.yaml
deployment.extensions "mariadb" created

The MariaDB deployment defined above is explicitly associated with the PVC, px-mariadb-pvc created in the previous step.

This deployment creates a single pod running MariaDB backed by Portworx.

$ kubectl get pods
NAME                     READY     STATUS    RESTARTS   AGE
mariadb-dff54d66d-m9r6q   1/1       Running   0          6s

We can inspect the Portworx volume by accessing the pxctl tool running with the MariaDB pod.

$ VOL=`kubectl get pvc | grep px-mariadb-pvc | awk '{print $3}'`
$ PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
$ kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl volume inspect ${VOL}
Volume	:  760417490447442140
	Name            	 :  pvc-2298b6ec-9936-11e8-8c5e-0253036635a0
	Size            	 :  1.0 GiB
	Format          	 :  xfs
	HA              	 :  3
	IO Priority     	 :  LOW
	Creation time   	 :  Aug 6 05:04:08 UTC 2018
	Shared          	 :  no
	Status          	 :  up
	State           	 :  Attached: ip-192-168-95-234.us-west-2.compute.internal
	Device Path     	 :  /dev/pxd/pxd760417490447442140
	Labels          	 :  namespace=default,pvc=px-mariadb-pvc
	Reads           	 :  60
	Reads MS        	 :  20
	Bytes Read      	 :  294912
	Writes          	 :  512
	Writes MS       	 :  100644
	Bytes Written   	 :  172474368
	IOs in progress 	 :  0
	Bytes used      	 :  126 MiB
	Replica sets on nodes:
		Set  0
		  Node 		 :  192.168.95.234 (Pool 0)
		  Node 		 :  192.168.203.81 (Pool 0)
		  Node 		 :  192.168.185.157 (Pool 0)
	Replication Status	 :  Up

The screenshot looks similar to the one shown below:

px-mariadb-eks-2

The output from the above command confirms the creation of volumes that are backing MariaDB database instance.

Failing over MariaDB pod on Kubernetes

Populating sample data

Let’s populate the database with some sample data.

We will first find the pod that’s running MariaDB to access the shell.

$ POD=`kubectl get pods -l app=mariadb | grep Running | grep 1/1 | awk '{print $1}'`

$ kubectl exec -it $POD -- mysql -uroot -ppassword
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 11
Server version: 10.4.6-MariaDB-1:10.4.6+maria~bionic mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

Now that we are inside the shell, we can populate create a sample database and table.

MariaDB> CREATE DATABASE `classicmodels`;

MariaDB> USE `classicmodels`;

MariaDB> CREATE TABLE `offices` (
  `officeCode` varchar(10) NOT NULL,
  `city` varchar(50) NOT NULL,
  `phone` varchar(50) NOT NULL,
  `addressLine1` varchar(50) NOT NULL,
  `addressLine2` varchar(50) DEFAULT NULL,
  `state` varchar(50) DEFAULT NULL,
  `country` varchar(50) NOT NULL,
  `postalCode` varchar(15) NOT NULL,
  `territory` varchar(10) NOT NULL,
  PRIMARY KEY (`officeCode`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Query OK, 0 rows affected (0.227 sec)

MariaDB> insert  into `offices`(`officeCode`,`city`,`phone`,`addressLine1`,`addressLine2`,`state`,`country`,`postalCode`,`territory`) values 
('1','San Francisco','+1 650 219 4782','100 Market Street','Suite 300','CA','USA','94080','NA'),
('2','Boston','+1 215 837 0825','1550 Court Place','Suite 102','MA','USA','02107','NA'),
('3','NYC','+1 212 555 3000','523 East 53rd Street','apt. 5A','NY','USA','10022','NA'),
('4','Paris','+33 14 723 4404','43 Rue Jouffroy D\'abbans',NULL,NULL,'France','75017','EMEA'),
('5','Tokyo','+81 33 224 5000','4-1 Kioicho',NULL,'Chiyoda-Ku','Japan','102-8578','Japan'),
('6','Sydney','+61 2 9264 2451','5-11 Wentworth Avenue','Floor #2',NULL,'Australia','NSW 2010','APAC'),
('7','London','+44 20 7877 2041','25 Old Broad Street','Level 7',NULL,'UK','EC2N 1HN','EMEA');
Query OK, 7 rows affected (0.039 sec)
Records: 7  Duplicates: 0  Warnings: 0

Let’s run a few queries on the table.

MariaDB> select `officeCode`,`city`,`phone`,`addressLine1`,`city` from `offices`;
+------------+---------------+------------------+--------------------------+---------------+
| officeCode | city          | phone            | addressLine1             | city          |
+------------+---------------+------------------+--------------------------+---------------+
| 1          | San Francisco | +1 650 219 4782  | 100 Market Street        | San Francisco |
| 2          | Boston        | +1 215 837 0825  | 1550 Court Place         | Boston        |
| 3          | NYC           | +1 212 555 3000  | 523 East 53rd Street     | NYC           |
| 4          | Paris         | +33 14 723 4404  | 43 Rue Jouffroy D'abbans | Paris         |
| 5          | Tokyo         | +81 33 224 5000  | 4-1 Kioicho              | Tokyo         |
| 6          | Sydney        | +61 2 9264 2451  | 5-11 Wentworth Avenue    | Sydney        |
| 7          | London        | +44 20 7877 2041 | 25 Old Broad Street      | London        |
+------------+---------------+------------------+--------------------------+---------------+
7 rows in set (0.01 sec)

px-mariadb-eks-3

Find all the offices in USA.

MariaDB [classicmodels]> select `officeCode`, `city`, `phone`  from `offices` where `country` = "USA";
+------------+---------------+-----------------+
| officeCode | city          | phone           |
+------------+---------------+-----------------+
| 1          | San Francisco | +1 650 219 4782 |
| 2          | Boston        | +1 215 837 0825 |
| 3          | NYC           | +1 212 555 3000 |
+------------+---------------+-----------------+
3 rows in set (0.00 sec)

Exit from the MariaDB shell to return to the host.

Simulating node failure

Now, let’s simulate the node failure by cordoning off the node on which MariaDB is running.

$ NODE=`kubectl get pods -l app=mariadb -o wide | grep -v NAME | awk '{print $7}'`

$ kubectl cordon ${NODE}
node "ip-192-168-203-81.us-west-2.compute.internal" cordoned

The above command disabled scheduling on one of the nodes.

$ kubectl get nodes
NAME                                            STATUS                     ROLES     AGE       VERSION
ip-192-168-185-157.us-west-2.compute.internal   Ready                          4h        v1.10.3
ip-192-168-203-81.us-west-2.compute.internal    Ready,SchedulingDisabled       4h        v1.10.3
ip-192-168-95-234.us-west-2.compute.internal    Ready                          4h        v1.10.3

Now, let’s go ahead and delete the MariaDB pod.

$ POD=`kubectl get pods -l app=mariadb -o wide | grep -v NAME | awk '{print $1}'`
$ kubectl delete pod ${POD}
pod "mariadb-dff54d66d-m9r6q" deleted

As soon as the pod is deleted, it is relocated to the node with the replicated data. STorage ORchestrator for Kubernetes (STORK), a Portworx-contributed open source storage scheduler, co-locates the pod on the exact node where the data is stored. It ensures that an appropriate node is selected for scheduling the pod.

Let’s verify this by running the below command. We will notice that a new pod has been created and scheduled in a different node.

$ kubectl get pods -l app=mariadb -o wide
NAME                     READY     STATUS    RESTARTS   AGE       IP               NODE
mariadb-dff54d66d-tzvjw   1/1       Running   0          15s       192.168.86.169   ip-192-168-95-234.us-west-2.compute.internal

$ kubectl uncordon ${NODE}
node "ip-192-168-203-81.us-west-2.compute.internal" uncordoned

Finally, let’s verify that the data is still available.

Verifying that the data is intact

Let’s find the pod name and run the ‘exec’ command, and then access the MariaDB shell.

kubectl exec -it $POD -- mysql -uroot -ppassword
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 8
Server version: 10.4.6-MariaDB-1:10.4.6+maria~bionic mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

We will query the database to verify that the data is intact.

MariaDB [none]> USE `classicmodels`;
MariaDB [classicmodels]> select `officeCode`, `city`, `phone`  from `offices` where `country` = "USA";
+------------+---------------+-----------------+
| officeCode | city          | phone           |
+------------+---------------+-----------------+
| 1          | San Francisco | +1 650 219 4782 |
| 2          | Boston        | +1 215 837 0825 |
| 3          | NYC           | +1 212 555 3000 |
+------------+---------------+-----------------+
3 rows in set (0.00 sec)

Observe that the database table is still there and all the content is intact! Exit from the client shell to return to the host.

Performing Storage Operations on MariaDB

After testing end-to-end failover of the database, let’s perform StorageOps on our EKS cluster.

Expanding the Kubernetes Volume with no downtime

Currently, the Portworx volume that we created at the beginning is of 1Gib size. We will now expand it to double the storage capacity.

First, let’s get the volume name and inspect it through the pxctl tool.

If you have access, SSH into one of the nodes and run the following command.

$ POD=`/opt/pwx/bin/pxctl volume list --label pvc=px-mariadb-pvc | grep -v ID | awk '{print $1}'`
$ /opt/pwx/bin/pxctl v i $POD
Volume	:  760417490447442140
	Name            	 :  pvc-3a6788df-9274-11e8-8c5e-0253036635a0
	Size            	 :  1.0 GiB
	Format          	 :  xfs
	HA              	 :  3
	IO Priority     	 :  LOW
	Creation time   	 :  Jul 28 14:40:52 UTC 2018
	Shared          	 :  no
	Status          	 :  up
	State           	 :  Attached: ip-192-168-95-234.us-west-2.compute.internal
	Device Path     	 :  /dev/pxd/pxd150455926773027922
	Labels          	 :  namespace=default,pvc=px-mariadb-pvc
	Reads           	 :  188
	Reads MS        	 :  104
	Bytes Read      	 :  8458240
	Writes          	 :  23
	Writes MS       	 :  128
	Bytes Written   	 :  2347008
	IOs in progress 	 :  0
	Bytes used      	 :  126 MiB
	Replica sets on nodes:
		Set  0
		  Node 		 :  192.168.95.234 (Pool 0)
		  Node 		 :  192.168.203.81 (Pool 0)
		  Node 		 :  192.168.185.157 (Pool 0)
	Replication Status	 :  Up

Notice the current Portworx volume. It is 1GiB. Let’s expand it to 2GiB.

$ /opt/pwx/bin/pxctl volume update $POD --size=2
Update Volume: Volume update successful for volume 150455926773027922

Check the new volume size.

$ /opt/pwx/bin/pxctl v i $POD
Volume	:  760417490447442140
	Name            	 :  pvc-3a6788df-9274-11e8-8c5e-0253036635a0
	Size            	 :  2.0 GiB
	Format          	 :  xfs
	HA              	 :  3
	IO Priority     	 :  LOW
	Creation time   	 :  Jul 28 14:40:52 UTC 2018
	Shared          	 :  no
	Status          	 :  up
	State           	 :  Attached: ip-192-168-95-234.us-west-2.compute.internal
	Device Path     	 :  /dev/pxd/pxd150455926773027922
	Labels          	 :  namespace=default,pvc=px-mariadb-pvc
	Reads           	 :  200
	Reads MS        	 :  104
	Bytes Read      	 :  8507392
	Writes          	 :  60
	Writes MS       	 :  164
	Bytes Written   	 :  2498560
	IOs in progress 	 :  0
	Bytes used      	 :  126 MiB
	Replica sets on nodes:
		Set  0
		  Node 		 :  192.168.95.234 (Pool 0)
		  Node 		 :  192.168.203.81 (Pool 0)
		  Node 		 :  192.168.185.157 (Pool 0)
	Replication Status	 :  Up

px-mariadb-eks-4

Taking Snapshots of a Kubernetes volume and restoring the database

Portworx supports creating snapshots for Kubernetes PVCs.

Let’s create a snapshot for the Kubernetes PVC we created for MariaDB.

cat >  px-mariadb-snap.yaml << EOF
apiVersion: volumesnapshot.external-storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: px-mariadb-snapshot
  namespace: default
spec:
  persistentVolumeClaimName: px-mariadb-pvc
EOF

$ kubectl create -f px-mariadb-snap.yaml
volumesnapshot.volumesnapshot.external-storage.k8s.io "px-mariadb-snapshot" created

Verify the creation of volume snapshot.

$ kubectl get volumesnapshot
NAME                AGE
px-mariadb-snapshot   13s

$ kubectl get volumesnapshotdatas
NAME                                                       AGE
k8s-volume-snapshot-504e9e5f-a6ec-11e9-ab32-7a5327be2608   19s

With the snapshot in place, let’s go ahead and delete the database.

$ POD=`kubectl get pods -l app=mariadb | grep Running | grep 1/1 | awk '{print $1}'`
$ kubectl exec -it $POD -- mysql -uroot -ppassword
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 9
Server version: 10.4.6-MariaDB-1:10.4.6+maria~bionic mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

drop database classicmodels;

Since snapshots are just like volumes, we can use it to start a new instance of MariaDB. Let’s create a new instance of MariaDB by restoring the snapshot data.

$ cat > px-mariadb-snap-pvc << EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: px-mariadb-snap-clone
  annotations:
    snapshot.alpha.kubernetes.io/snapshot: px-mariadb-snapshot
spec:
  accessModes:
     - ReadWriteOnce
  storageClassName: stork-snapshot-sc
  resources:
    requests:
      storage: 2Gi
EOF

$ kubectl create -f px-mariadb-snap-pvc.yaml
persistentvolumeclaim "px-mariadb-snap-clone" created

From the new PVC, we will create a MariaDB pod.

$ cat < px-mariadb-snap-restore.yaml >> EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mariadb
spec:
  selector:
    matchLabels:
      app: mariadb
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  replicas: 1
  template:
    metadata:
      labels:
        app: mariadb-snap
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: px/running
                operator: NotIn
                values:
                - "false"
              - key: px/enabled
                operator: NotIn
                values:
                - "false"
    spec:
      containers:
      - name: mariadb
        image: mariadb:latest
        imagePullPolicy: "Always"
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: password       
        ports:
        - containerPort: 3306
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: mariadb-data
      volumes:
      - name: mariadb-data
        persistentVolumeClaim:
          claimName: px-mariadb-snap-clone
EOF

$ kubectl create -f px-mariadb-snap-restore.yaml
deployment.extensions "mariadb-snap" created

Verify that the new pod is in the running state.

$ kubectl get pods -l app=mariadb-snap
NAME                         READY     STATUS    RESTARTS   AGE
mariadb-snap-655ffd9d67-ff288   1/1       Running   0          15s

Finally, let’s access the sample data created earlier in the walkthrough.

$ POD=`kubectl get pods -l app=mariadb-snap | grep Running | grep 1/1 | awk '{print $1}'`
$ kubectl exec -it $POD -- mysql -uroot -ppassword
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 8
Server version: 10.4.6-MariaDB-1:10.4.6+maria~bionic mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

MariaDB [(none)]> USE `classicmodels`;
MariaDB [classicmodels]> select `officeCode`, `city`, `phone`  from `offices` where `country` = "USA";
+------------+---------------+-----------------+
| officeCode | city          | phone           |
+------------+---------------+-----------------+
| 1          | San Francisco | +1 650 219 4782 |
| 2          | Boston        | +1 215 837 0825 |
| 3          | NYC           | +1 212 555 3000 |
+------------+---------------+-----------------+
3 rows in set (0.00 sec)

Notice that the collection is still there with the data intact. We can also push the snapshot to Amazon S3 if we want to create a Disaster Recovery backup in another region. Portworx snapshots also work with any S3 compatible object storage, so the backup can go to a different cloud or even an on-premises data center. Alternatively, we can stretch a single Portworx cluster across two independent Kubernetes clusters for Zero RPO DR for Kubernetes.

Summary

Portworx can easily be deployed on Amazon Elastic Kubernetes Service to run stateful workloads in production. Through the integration of STORK, DevOps and StorageOps teams can seamlessly run highly-available database clusters in Amazon EKS. They can perform traditional operations such as volume expansion, snapshots, backup and recovery for the cloud-native applications.