How to avoid noisy neighbor problems using Application IO Control

Over the past year, Kubernetes has crossed the chasm and is now in the early majority phase of market adoption. With this increased adoption for Kubernetes, we see an ever-increasing need for enterprise-grade Kubernetes storage solutions that enable organizations to deploy and run stateful applications on Kubernetes. Portworx has been a gold standard when it comes to Kubernetes storage, and we have continued to push the boundaries of innovation for our customers. We announced the general availability of our Portworx Enterprise 2.10 release today, and with the new release comes a whole new set of features that gives finer control to users for enabling a true multi-tenant PaaS environment, simplifying operational tasks, and enriching user experience.

As part of this new release, we introduced a new feature called Application IO Control, which provides users the ability to control the amount of IOPS or bandwidth available for each persistent volume running on a Portworx storage cluster. This is especially important for customers who are running a multi-tenant Kubernetes cluster for their Platform-as-a-Service (PaaS) deployments and might be vulnerable to noisy neighbor problems, where a single application or persistent volume consumes most of the storage bandwidth and IOPS available at the storage layer. This can lead other co-located applications to starve, which leads to performance degradation. With Portworx Application IO Control, users can now set IOPS or bandwidth limits for their persistent volume, both in a greenfield (new volumes) or a brownfield (existing volumes) deployment. For greenfield deployment, platform architects can set those IOPS or bandwidth maximums at the storage class level, and any persistent volumes that are dynamically provisioned using that storage class will inherit those parameters. And for brownfield deployments, platform architects or DevOps admins can use the pxctl utility and edit volume options and set those IOPS or bandwidth maximum values in real time.

In the remainder of this blog, we’ll look at how you can use Application IO Control on your Portworx clusters running v2.10 or later. To set the bandwidth limits in MB/s as part of your storage class definition, use the io_throttle_rd_bw and io_throttle_wr_bwparameters. Based on your application requirements, you can choose to set both parameters and assign them maximum values or simply choose to set either one of them.

##### Portworx storage class

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: bw-sc
provisioner: kubernetes.io/portworx-volume
parameters:
repl: "2"
io_throttle_rd_bw: "2" # MB/s maximum for read 
io_throttle_wr_bw: "2" # MB/s maximum for write
allowVolumeExpansion: true

In addition to bandwidth requirements, you can also set max read and write IOPS parameters in your storage class using the io_throttle_rd_iops and io_throttle_wr_iops variables. You can either set the bandwidth or the IOPS parameters in a storage class, but you cannot set both together.

<##### Portworx storage class

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: iops-sc
provisioner: kubernetes.io/portworx-volume
parameters:
repl: "2"
io_throttle_rd_iops: "500" # maximum read IOPS 
io_throttle_wr_iops: "500" # maximum write IOPS
allowVolumeExpansion: true

Once you apply these configurations and deploy these storage classes on your Portworx storage cluster, you can proceed to dynamically provision persistent volumes for your applications, and they will inherit these maximum values automatically, without any additional steps needed by the application developer or owner.

Now that we know how to create storage classes that leverage Application IO Control, let’s look at how you can use the pxctl utility as well as how to modify your existing “noisy” volumes and set these IOPS or bandwidth limits.

• Exec into one of the Portworx pods on your Kubernetes cluster.

PX_POD=$(kubectl get pods -n kube-system -l name=portworx -o \ 
jsonpath='{.items[0].metadata.name}')

kubectl exec -it $PX_POD -n kube-system -it -- bash

• List all the available volumes and identify the volume ID for the “noisy” volume.

pxctl volume list

• Next, use the volume update command to set the max IOPS or bandwidth values.

Note: You can set either the IOPS max limit or the bandwidth max limit, but not both on the same volume.

#To set the Max read IOPS to 1000 and write IOPS to 500, use: 
pxctl volume update --max_iops 1000,500 <<vol_id>> 

#To set the Max read bandwidth to 50 and write bandwidth to 20 MB/s, use:
pxctl volume update --max_bandwidth 50,20 <<vol_id>>

Once you set Application IO Control limits, you will see changes to the volume throughput in real-time, similar to the screenshot below:

• The syntax for both commands allows you to set different read and write limits for IOPS or bandwidth, but you can also use to following command to remove any limits you might have on Portworx volumes:

#To remove Max read and write IOPS, use: 
pxctl volume update --max_iops off,off <<vol_id>>

#To remove Max read and write bandwidth, use:
pxctl volume update --max_bandwidth off,off <<vol_id>>

To use this new capability, you can deploy a new Portworx cluster or upgrade your existing clusters to v2.10. If you want to see all of this in action, check out the demonstration below: