Implementing Data Security on Red Hat OpenShift

Implementing Data Security on Red Hat OpenShift

Portworx Enterprise is the data services platform businesses trust to run mission-critical applications in containers in production on Red Hat OpenShift. Only Portworx provides a fully integrated solution for persistent storage, data protection, disaster recovery, data security, cross-cloud and data migrations, and automated capacity management for applications running on Kubernetes. As a result, Portworx is the #1 most used Kubernetes data services platform by Global 2000 companies, including Carrefour, Comcast, GE Digital, Kroger, Lufthansa, and T-Mobile.

In this blog, you’ll learn how to effectively use Portworx to secure application volumes, data volume requests, and access as well as monitor and audit data management on Red Hat OpenShift with Portworx.

Security Is Key

According to the State of Kubernetes Security Report from Red Hat, nearly “94% of survey respondents have experienced a security incident in their Kubernetes and container environments during the last 12 months.”

In addition to this, not only are organizations who are running Kubernetes seeing security incidents, but “59% of respondents are most worried about unaddressed security and compliance needs or threats to containers.” What this should tell us is that security is a top concern when it comes to running Kubernetes in production.

With DevOps teams, where infrastructure and application architects all play key roles in the security of Kubernetes, it’s imperative to think about the valuable information flowing through these Kubernetes environments. While cluster security is a major portion of today’s security implementations, data security is also a core facet of the broader Kubernetes security concerns. Portworx understands this. Cloud native data management on Red Hat OpenShift is not immune to these security concerns and needs to be implemented with its own security strategies to protect important assets, information, and data within each and every stateful application in an organization.

PX-Security on Red Hat OpenShift

Portworx provides imperative data security pillars when it comes to securing data access and control within Kubernetes platforms such as OpenShift. These pillars include the following components:

Encryption

The importance of encryption is paramount to keeping information confidential—whether it’s at rest or in transit. Data within the PVCs of a Kubernetes application should be encrypted for those applications using sensitive information—such as those in the healthcare space using PII and PHI.

Encryption can happen in a few different places; the first is encrypting data at rest with encryption keys for cluster-wide encryption or per-pvc encryption with Portworx. To enable encryption for volumes with Portworx, a key must hold a passphrase used for encryption. This key can live in Kubernetes as a Kubernetes Secret or in external KMS systems such as Vault. Portworx supports a variety of secrets providers, which can be used for Portworx encryption secrets.

Encrypted volumes come in two flavors on Portworx enabled OpenShift clusters. One is a cluster-wide encrypted volume where all volumes share the same encryption key. The other is a per-volume encrypted volume where each volume has a unique secret. Per-volume encryption is great for multi-tenant environments where many teams may share a single OpenShift cluster.

Once encryption is enabled, a secrets provider is configured, and a cluster-wide secret is enabled. Enabling StorageClass based encryption is as easy as providing a storage class with secure: true.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: px-secure-sc
provisioner: kubernetes.io/portworx-volume
parameters:
  secure: "true"
  repl: "3"

You may also enable per-pvc volume encryption by using a secret per volume as well as using a CSI storage class to inject the provisioner and publish secrets and namespaces.

Then, your Portworx volumes will be encrypted and shown as Attributes: encrypted within Portworx.

 kubectl pxc pxctl v i pvc-98343cb7-c79b-42d2-8b01-2454cf607e82
>> Running pxctl on ip-10-0-155-63
Volume            :  707340059578410116
Name              :  pvc-98343cb7-c79b-42d2-8b01-2454cf607e82
Size              :  100 GiB
Format            :  ext4
HA                :  3
IO Priority       :  HIGH
Creation time     :  Aug 31 14:32:10 UTC 2021
Shared            :  no
Status            :  up
State             :  Attached: 60e56205-0e0c-4d43-a198-712c6bced423 (10.0.171.42)
Last Attached     :  Aug 31 14:35:01 UTC 2021
Attributes        :  encrypted
Device Path       :  /dev/mapper/pxd-enc707340059578410116
...<snip>

Portworx Enterprise volumes are one way Portworx enables encryption; another example is via PX-Backup. When backing up your OpenShift applications, PX-backup admins can provide a “Encryption Key” to a PX-Backup backup location so that data is sent encrypted in transit.

The example using Amazon S3 as a backup target shows the in-transit data when encrypted and unencrypted.

Encrypted

20:57:12.705491 IP ip-10-0-123-45.us-east-2.compute.internal.42328 > s3.us-east-2.amazonaws.com.https: Flags [P.], seq 1930:2511, ack 2829, win 1464, length 581
        0x0000:  4500 026d 423e 4000 3f06 8440 0a00 df57  E..mB>@.?..@...W
        0x0010:  34db 54da a558 01bb b978 b1b0 6872 bd02  4.T..X...x..hr..
        0x0020:  5018 05b8 756c 0000 1703 0302 4000 0000  P...ul......@...
        0x0030:  0000 0000 2aad 9c1a e84a ee27 d1ec c624  ....*....J.'...$
        0x0040:  023a da24 b206 36d8 3954 adec b894 a729  .:.$..6.9T.....)
        0x0050:  b7fe 731c 3f7f 1981 bb58 4fbe 1f11 2226  ..s.?....XO..."&

Unencrypted

20:53:17.639164 IP ip-10-0-123-45.us-east-2.compute.internal.57222 > s3.us-east-2.amazonaws.com.http: Flags [P.], seq 689:1428, ack 678, win 226, length 739: HTTP: PUT /backup-user-3/db-ops/test-unencrypted-01-3f8cc4b85224/namespaces.json HTTP/1.1
        0x0020:  5018 00e2 759a 0000 5055 5420 2f70 782d  P...u...PUT./px-
        0x0030:  6261 636b 7570 2d72 7761 6c6c 6e65 722d  backup-user-
        0x0040:  332f 6462 2d6f 7073 2f74 6573 742d 756e  3/db-ops/un
        0x0050:  656e 6372 7970 7465 642d 3031 2d32 3832  encrypted-01-282
        0x0080:  6363 3462 3835 3232 342f 6e61 6d65 7370  cc4b85224/namesp
        0x0090:  6163 6573 2e6a 736f 6e20 4854 5450 2f31  aces.json.HTTP/1
        0x0250:  3d0d 0a43 6f6e 7465 6e74 2d54 7970 653a  =..Content-Type:
        0x0260:  2074 6578 742f 706c 6169 6e3b 2063 6861  .text/plain;.cha
        0x0270:  7273 6574 3d75 7466 2d38 0d0a 582d 416d  rset=utf-8..X-Am

Authentication & Authorization

Validating that the user is who they say they are and then subsequently verifying that the user has the right to access, manipulate, or create a resource is the art of authenticating a user and then authorizing one. Authentication and authorization are key to any Kubernetes platform, and the same is true with OpenShift. When interacting with OpenShift itself, you must be an authorized user or admin, so why should it be any different when interacting with Persistent Storage?

Once the security enablement steps are complete, this will enable the Portworx-specific RBAC model. This RBAC model includes authentication and authorization for Portworx resources such as volumes, snapshots, cloud snapshots, and more.

To authenticate users in Portworx, PX-Security supports two types of token generation models: OIDC and self-generated tokens. OpenID Connect (or OIDC) is a standard model for user authentication and management and is a great solution for enterprise customers due to its integration with SAML 2.0, Active Directory, and/or LDAP. The second model is a self-generated token, where the administrator would generate a token using their own token authentication application. For convenience, Portworx provides a method of generating tokens using pxctl.

For Portworx to verify the tokens are valid, they must be signed with:

• A shared secret or
• An RSA private key or
• An ECDSA private key

The token will be created by the token administrator and will contain information about the user in the claims section. When Portworx receives a request from the user, it will check the token validity by verifying its signature, using either a shared secret or public key provided during configuration.

An example profile for a user may be the following. The user below is an “analytics-team” user, which is a “system.user” within Portworx. They belong to the “analytics-users” and “dbs-analytics” roles, which have specific access controls.

name: analytics-team
sub: analytics@purestorage.com/analytics
email: analytics@purestorage.com
roles: ["system.user"]
groups: ["analytics-users", “dbs-analytics”]

When a token is produced for this user, it can be made available within Kubernetes by creating a secret.

kubectl -n db-ops create secret generic px-user-token --from-literal=auth-token=<auth-token>

Authorization

The next step in this process is to verify that the token provided during a request (such as one to create a PVC) is valid.

Once the token has been determined to be valid, Portworx then checks if the user is authorized to make the request. The roles claim in the token must contain the name of an existing default or customer registered role in the Portworx system. A role is the name given to a set of RBAC rules that enable access to certain SDK calls. Custom roles can be created using pxctl or through the OpenStorage SDK.

As an example, if a token is not valid, users will see an “Access Denied” response with OpenShift, such as the one below when a PVC request is sent.

If a token was passed but is invalid or spoofed, a user will also be denied.

If a token is valid but has expired, a user will also be denied.

If a token is valid but has the wrong role for creating PVCs, a user will also be denied.

Ownership

Ownership is the model used for resource control. The model is composed of the owner and a list of groups and collaborators with access to the resource. Groups and collaborators can also have their access to a resource constrained by their access type. The following table defines the three access types supported:

For example, user1 could create a volume and give Read access to group1. This means that only user1 can mount the volume. However, group1 can clone the volume. When a volume is cloned, it is owned by the user who made the request.

Volume access can be viewed by looking at the access metadata of a volume. This can be done using the pxctl volume access show command.

Audit

Lastly, even with purpose-built security for Kubernetes, administrators should audit security by providing security audit logs. Portworx provides security audit and access logs so that organizations can  help protect critical data, identify security loopholes, create new security policies, and track the effectiveness of security strategies.

The logs are available on each Portoworx node at the following locations:

•  /var/lib/osd/log/security/openstorage-audit.log

•  /var/lib/osd/log/security/openstorage-access.log

Using Elasticsearch, Kibana, and Filebeat, these audit and access logs can be captured and loaded into Kibana dashboards for data security audit monitoring on OpenShift. First, apply a filebeat ConfigMap that targets both the access and audit logs on the Portworx nodes.

Then, deploy a Filebeat daemonset that uses the ConfigMap and send Portworx audit logs to Elasticsearch. Then use Kibana to dashboard the audit and access logs within your OpenShift cluster.

From here, you will be able to monitor your PX-Security audit logs.Remember the few PVC create requests that we made earlier that used invalid, expired, read-only tokens? Those are some examples of audit messages that will show up within your dashboards.

Authentication, authorization, encryption, ownership, and security auditing are critical components to any data security implementation in today’s cloud native data centers. By combining OpenShift Container Platform security architectures with Portworx Security, organizations can protect important information and assets now and in the future along with adhering to important governance and compliance regulations for applications and Kubernetes environments.