shutterstock

Kubernetes does a great job of orchestrating your containerized applications and deploying them across all the worker nodes in a cluster. If a node has enough compute capacity to run a specific Kubernetes pod, Kubernetes will schedule that application pod on that worker node. But what if none of the worker nodes in the cluster have enough available capacity to accept new application pods? At this point, Kubernetes will not be able to deploy your application pods, and they will be stuck in a pending state. In addition to this scenario, Kubernetes also does not have the capability to monitor and manage the storage utilization in your cluster. These are two huge problems when it comes to running applications on Kubernetes. This blog covers solutions from Portworx and AWS that will help users architect a solution that helps them remediate these concerns.

In this blog, we will look at how Portworx Autopilot and AWS Karpenter work together to help users build a solution on top of AWS EKS clusters that allows them to automatically expand persistent volumes or add more storage capacity to the Kubernetes clusters using Portworx Autopilot. The solution also automatically adds more CPU and memory resources by dynamically adding more worker nodes in the AWS EKS cluster using AWS Karpenter.

We can begin by developing a better understanding of automated storage capacity management with Portworx Autopilot. Autopilot is a rule-based engine that responds to changes from a monitoring source. Autopilot allows you to specify monitoring conditions as well as the actions it should take when those conditions occur, which means you can set simple IFTT rules against your EKS cluster and have Autopilot automatically perform a certain action for you if a certain condition has been met. Portworx Autopilot supports the following three use cases:

  • Automatically resizing PVCs when they are running out of capacity
  • Scaling Portworx storage pools to accommodate increasing usage
  • Rebalancing volumes across Portworx storage pools when they come unbalanced

To get started with Portworx Autopilot, first you will have to deploy Portworx on your Amazon EKS cluster and configure Prometheus and Grafana for monitoring. Once you have that up and running, use the following steps to configure Autopilot and create an Autopilot rule that will monitor the capacity utilization of a persistent volume and scale it up accordingly:

  • Use the following yaml file to deploy Portworx Autopilot on your Amazon EKS cluster. Verify the Prometheus endpoint set in the autopilot-config config map and ensure that it matches the service endpoint for Prometheus in your cluster.
# SOURCE: https://install.portworx.com/?comp=autopilot
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: autopilot-config
  namespace: kube-system
data:
  config.yaml: |-
    providers:
       - name: default
         type: prometheus
         params: url=http://px-prometheus:9090
    min_poll_interval: 2
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: autopilot-account
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ""
  labels:
    tier: control-plane
  name: autopilot
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: autopilot
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  replicas: 1
  template:
    metadata:
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ""
      labels:
        name: autopilot
        tier: control-plane
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "name"
                    operator: In
                    values:
                    - autopilot
              topologyKey: "kubernetes.io/hostname"
      hostPID: false
      containers:
      - command:
        - /autopilot
        - -f
        - ./etc/config/config.yaml
        - -log-level
        - debug
        imagePullPolicy: Always
        image: portworx/autopilot:1.3.1
        resources:
          requests:
            cpu: '0.1'
        securityContext:
          privileged: false
        name: autopilot
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
      serviceAccountName: autopilot-account
      volumes:
        - name: config-volume
          configMap:
            name: autopilot-config
            items:
            - key: config.yaml
              path: config.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: autopilot
  namespace: kube-system
  labels:
    name: autopilot-service
spec:
  ports:
    - name: autopilot
      protocol: TCP
      port: 9628
  selector:
    name: autopilot
    tier: control-plane
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: autopilot-role
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: autopilot-role-binding
subjects:
- kind: ServiceAccount
  name: autopilot-account
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: autopilot-role
  apiGroup: rbac.authorization.k8s.io
  • Once you apply this configuration, you can start creating AutopilotRules for individual applications or namespaces. An AutopilotRule has four main sections:
    • Selector: Matches labels on the objects that the rule should monitor.
    • Namespace Selector: Matches labels on the Kubernetes namespaces the rule should monitor. This is optional, and the default is all namespaces.
    • Conditions: These are the metrics for the objects to monitor.
    • Actions: These are what Autopilot will perform once the metric conditions are met.

Here is an example of an AutopilotRule that checks for a persistent volume with a label of app: postgres deployed in a namespace with a label of type: db. If the used capacity exceeds 50%, it doubles the size of the persistent volume untill it hits a maximum size of 400Gi.

apiVersion: autopilot.libopenstorage.org/v1alpha1
kind: AutopilotRule
metadata:
  name: volume-resize
spec:
  selector:
    matchLabels:
      app: postgres
  namespaceSelector:
    matchLabels:
      type: db
  conditions:
    expressions:
    - key: "100 * (px_volume_usage_bytes / px_volume_capacity_bytes)"
      operator: Gt
      values:
        - "50"
  actions:
  - name: openstorage.io.action.volume/resize
    params:
      scalepercentage: "100"
      maxsize: "400Gi"

Once you apply this AutopilotRule specification, Portworx will monitor the capacity utilization using Prometheus metrics for that persistent volume and automatically perform actions as needed.

  • In addition to expanding individual persistent volumes, Portworx Autopilot also allows you to put AutopilotRules in place that will direct Portworx to automatically expand the underlying storage pool if your applications are more storage intensive and you don’t want to add more EKS worker nodes in your cluster. Below is a sample AutopilotRule that monitors your storage pool utilization. If the available capacity falls below 50% and if the total capacity is still less than 2TB, it automatically creates and attaches EBS volumes to your EKS worker node to add 50% more capacity of your storage pool:
apiVersion: autopilot.libopenstorage.org/v1alpha1
kind: AutopilotRule
metadata:
  name: pool-expand
spec:
  enforcement: required
  ##### conditions are the symptoms to evaluate. All conditions are AND'ed
  conditions:
    expressions:
    # pool available capacity less than 50%
    - key: "100 * ( px_pool_stats_available_bytes/ px_pool_stats_total_bytes)"
      operator: Lt
      values:
        - "50"
    # pool total capacity should not exceed 2TB
    - key: "px_pool_stats_total_bytes/(1024*1024*1024)"
      operator: Lt
      values:
       - "2000"
  ##### action to perform when condition is true
  actions:
    - name: "openstorage.io.action.storagepool/expand"
      params:
        # resize pool by scalepercentage of current size
        scalepercentage: "50"
        # when scaling, add disks to the pool
        scaletype: "add-disk"

Now that you know how to automate storage capacity management, we can look at how we can leverage AWS Karpenter to add more nodes to our EKS cluster when we need more compute capacity for our application pods.

  • Use the eksctl instructions on Karpenter’s documentation site to deploy an EKS cluster. For our testing, we used the following eksctl configuration to deploy an EKS cluster with two different node groups. The first node group, “storage-nodes,” will include three nodes that will be used by Portworx to provide storage for your stateful applications. The second node group, “kar-bshah-ng,” will be used by AWS Karpenter to dynamically add more nodes to the EKS cluster to increase the amount of compute capacity available for your applications.
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: kar-bshah
  region: us-west-2
  version: "1.21"
  tags:
    karpenter.sh/discovery: kar-bshah
managedNodeGroups:
  - name: storage-nodes
    instanceType: m5.xlarge
    minSize: 3
    maxSize: 3
    desiredCapacity: 3
    volumeSize: 100
    amiFamily: AmazonLinux2
    labels: {role: worker, "portworx.io/node-type": "storage"}
    tags:
      nodegroup-role: worker
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess
        - arn:aws:iam::<<aws-account-id>>:policy/<<px-role>>
      withAddonPolicies:
        imageBuilder: true
        autoScaler: true
        ebs: true
        fsx: true
        efs: true
        albIngress: true
        cloudWatch: true
  - instanceType: m5.large
    amiFamily: AmazonLinux2
    name: kar-bshah-ng
    desiredCapacity: 1
    minSize: 1
    maxSize: 10
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess
        - arn:aws:iam::<<aws-account-id>>:policy/<<px-role>>
      withAddonPolicies:
        imageBuilder: true
        autoScaler: true
        ebs: true
        fsx: true
        efs: true
        albIngress: true
        cloudWatch: true
  • Once you have your EKS cluster deployed, use the steps on the documentation site to configure AWS Karpenter and have it add more nodes to your EKS cluster when you need more CPU and memory resources for your application.

If you want to see Portworx Autopilot and AWS Karpenter in action, watch the following video, where we demonstrate how you can scale your compute and storage capacity as and when you need it.

Share
Subscribe for Updates

About Us
Portworx is the leader in cloud native storage for containers.

Bhavin

Bhavin Shah

Sr. Technical Marketing Manager | Cloud Native BU, Pure Storage
link
container
March 29, 2022 How To
Application migration from Red Hat OpenShift or OSS Kubernetes to Amazon EKS and Amazon EKS Anywhere
Bhavin Shah
Bhavin Shah
link
EKS-A-EKS-blogimage
March 23, 2022 How To
How to build a hybrid cloud app migration solution using Amazon EKS, Amazon EKS Anywhere and Portworx
Bhavin Shah
Bhavin Shah
link
CapacityManagement-AutoPilot
January 7, 2020 Lightboard
Lightboard Session: Capacity Management (AutoPilot)
Ryan Wallner
Ryan Wallner