How to Perform Failover of JupyterHub Deployment on Amazon Elastic Kubernetes Service

Contact Sales

How To

The Jupyter Notebook is an open-source web application that allows data scientists to create and share documents that contain live code, equations, visualizations, comments, and narrative text. It’s a powerful integrated development environment for data exploration, data processing, data analysis, machine learning, and analytics.

The JupyterHub is a multi-user platform that brings the power of Jupyter Notebook to enterprises. It’s a scalable, portable, customizable, and flexible environment to run Jupyter Notebooks in a multi-tenant model. JupyterHub can be deployed on bare metal servers, virtual machines, public cloud infrastructure, containers, and container orchestration engines. It can be deployed and managed in Kubernetes through Helm charts.

JupyterHub is a stateful workload that depends on a reliable persistence layer. When it is deployed in Kubernetes, JupyterHub needs a cloud native, scale-out data management layer.

Portworx is a cloud native storage platform to run persistent workloads deployed on a variety of orchestration engines, including Kubernetes. With Portworx, customers can manage the database of their choice on any infrastructure using any container scheduler. It provides a single data management layer for all stateful services, no matter where they run.

This guide is a continuation of the tutorial on deploying highly-available JupyterHub in EKS.

Uploading a Sample Dataset for the Notebook

Before attempting to perform a fail-over, let’s populate the environment with a sample dataset and a notebook.

Through the pod associated with user admin, we will upload a dataset to the shared storage location and verify its availability in user1’s environment.

Download the Pima Indian Diabetes dataset from Kaggle and save it as diabetes.csv.

We will copy it to the admin pod with the below command:

$ kubectl cp diabetes.csv jupyter-admin:/home/shared

jupyter

As shown below, the dataset—diabetes.csv—is now available to user1 in the /home/shared directory.

jupyter-node

We will now import a Jupyter Notebook into user1 environment. This contains Python code to explore and visualize the dataset copied in the previous step. Since the code in the Notebook has some dependencies, we need to install appropriate Python modules such as NumPy, Pandas, Matplotlib, and Seaborn.

Within the terminal window of user1, run the below commands to install the Python libraries.

$ pip install numpy pandas matplotlib seaborn

Upload the Jupyter Notebook and launch it. Since the Python modules are already installed, you can execute the code in the cells.

jupyter-database

We can download the SQLite database used by JupyterHub to explore its contents.

$ HUB_POD=$(kubectl get pods -l component=hub -o jsonpath='{.items[0].metadata.name}')
$ kubectl cp $HUB_POD:/srv/jupyterhub/jupyterhub.sqlite .

Download the DB Browser for SQLite to open the JupyterHub database.

SQLLite

Browsing the user table shows that admin and user1 as registered users of JupyterHub.

Exploring Portworx Volumes

Since all the PVCs are based on the Portworx storage class, they are provisioned as Portworx volumes.

Let’s look at the volumes mapped to the PVCs.

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl volume list

eks-node

The first two volumes with size 10 GiB represent the user directories that were dynamically created by the hub. The third volume is used by the hub to persist the SQLite database file. We can identify it by the size and the replication factor, as mentioned in the storage class. The last volume is the shared volume available to all users.

Performing Node Failover

We will deliberately simulate a failure by cordoning off the node running the hub pod and delete the current hub pod.

Since it is a part of the deployment, Kubernetes will automatically create a new pod and schedule it on a different node.

As soon as the pod is deleted, it is relocated to the node with the replicated data. STorage ORchestrator for Kubernetes (STORK), Portworx’s custom storage scheduler, allows co-locating the pod on the exact node where the data is stored. It ensures that an appropriate node is selected for scheduling the pod.

Let’s verify this by executing the steps of node cordoning and pod deletion. We will notice that a new pod has been created and scheduled in a different node.

We will first get the node that runs the hub pod.

$ NODE=`kubectl get pods -l component=hub -o wide | grep -v NAME | awk '{print $7}'`
echo $NODE
ip-172-31-24-96.ap-south-1.compute.internal

The below command cordons it off by disabling pod scheduling.

$ kubectl cordon $NODE
node/ip-192-168-80-126.us-west-2.compute.internal cordoned

Let’s go ahead and delete the hub pod that’s still running on the cordoned node.

$ kubectl delete pod $HUB_POD
pod "hub-7874f475b8-5gpx4" deleted

Kubernetes scheduled the pod in a different node, which can be verified with the below command:

$ kubectl get pod -l component=hub -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP           NODE                                           NOMINATED NODE   READINESS GATES
hub-599d577497-mpqm4   1/1     Running   0          23s   10.42.2.19   ip-172-31-24-143.ap-south-1.compute.internal

Download the SQLite database file and open it with the DB Browser. The data is intact!

HUB_POD=$(kubectl get pods -l component=hub -o jsonpath='{.items[0].metadata.name}')
kubectl cp $HUB_POD:/srv/jupyterhub/jupyterhub.sqlite .

You can repeat this task with one of the user pods to validate the failover process. It works exactly the same as the hub deletion.

Performing Storage Operations on JupyterHub

After testing end-to-end failover of the hub, let’s perform StorageOps on our GKE cluster.

Currently, the Portworx shared volume that we created at the beginning is of 1Gib size. We will now expand it to double the storage capacity.

First, let’s get the name associated with the PVC of shared volume and inspect it through the pxctl tool.

VOL=`kubectl get pvc | grep px-jhub-shared-vol | awk '{print $3}'`
kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl volume inspect $VOL
Volume	:  298147668021934078
	Name            	 :  pvc-7825d239-d2ec-11e9-be9b-0ac6c62f52b2
	Size            	 :  1.0 GiB
	Format          	 :  ext4
	HA              	 :  1
	IO Priority     	 :  LOW
	Creation time   	 :  Oct 14 10:31:27 UTC 2019
	Shared          	 :  yes
	Status          	 :  up
	State           	 :  Attached: 230ea89a-56b0-41a3-9d29-490def2676ef (192.168.80.126)
	Device Path     	 :  /dev/pxd/pxd298147668021934078
	Labels          	 :  io_profile=cms,namespace=default,pvc=px-jhub-shared-vol,repl=1,shared=true
	Reads           	 :  51
	Reads MS        	 :  4
	Bytes Read      	 :  1105920
	Writes          	 :  57
	Writes MS       	 :  56
	Bytes Written   	 :  16973824
	IOs in progress 	 :  0
	Bytes used      	 :  448 KiB
	Replica sets on nodes:
		Set 0
		  Node 		 : 192.168.80.126 (Pool 0)
	Replication Status	 :  Up
	Volume consumers	 :
		- Name           : jupyter-admin (7ba96390-d2ee-11e9-be9b-0ac6c62f52b2) (Pod)
		  Namespace      : default
		  Running on     : ip-172-31-22-152.ap-south-1.compute.internal

		- Name           : jupyter-user1 (d8f89f00-d2ee-11e9-be9b-0ac6c62f52b2) (Pod)
		  Namespace      : default
		  Running on     : ip-172-31-24-143.ap-south-1.compute.internal

eks-node

The output confirms that the shared flag is on for the volume and is being shared by two consumers—admin and user1.

Now, let’s expand this to 5GiB to add more capacity to the shared volume. We can use the same pxctl tool to perform this operation.

$ kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl volume update $VOL --size=5
Update Volume: Volume update successful for volume pvc-7825d239-d2ec-11e9-be9b-0ac6c62f52b2

As a part of storage operations, it is possible to take point-in-time snapshots of volumes that can be restored as new PVCs. Through Kubemotion, the entire JupyterHub environment can be migrated to a new Kubernetes cluster along with the PVCs and configuration data. The PX-DR capability enables synchronous or asynchronous replication of volumes for disaster recovery and business continuity.

Summary

Portworx can be easily deployed on Amazon Elastic Kubernetes Service to run stateful workloads in production. Through the integration of Portworx and EKS, DevOps and DataOps teams can seamlessly run highly available database clusters in Amazon Web Services. They can perform traditional operations such as volume expansion, snapshots, backup, and recovery for the cloud-native applications.