Portworx Container-Native CSI Driver for Nomad

An overview of Nomad CSI experiences to date for CSI plugin maintainers, community participants, and container orchestration platform users

Nomad CSI & Portworx

Nomad is a container orchestration platform similar to Kubernetes that is developed by HashiCorp. With the release of Nomad 1.1.0, support for the Container Storage Interface (CSI) is well underway. This version delivers support for two exciting new features: dynamic CSI volumes and snapshots. These features allow Nomad users to dynamically provision and snapshot volumes through any installed CSI spec-compliant plugins.

Portworx 2.8.0 and higher has first-class support for Nomad CSI with the Portworx CSI Driver. We have validated the following Portworx and Nomad features with Portworx 2.8.0 and Nomad 1.1.0:

• Creating and using dynamically provisioned volumes
• Registering and using pre-provisioned volumes
• Volume cloning & snapshots
• PX-Security
• PX sharedv4 volumes

Nomad CSI support is still in active development, and there are many new features on the roadmap.

Portworx CSI: Designed for any container orchestrator

The CSI spec was designed by the community to work for any container orchestrator (Kubernetes, Nomad, DCOS, etc.), but having a compliant spec for all orchestrators is only half the battle. It is up to the storage system to handle the intricacies of each orchestrator. The benefit of using a cloud-native CSI Driver like Portworx is that the underlying storage system behaves the same way for any container orchestration system. This is due to our common SDK server, which is used across all platforms. This SDK server supports authorization and authentication, and it can be interacted with from outside of the container orchestrator, allowing DevOps teams to even further customize their containerized stateful workloads with scripts.

We put this architecture to the test when adding support for Nomad CSI in Portworx 2.8.0. With only a few configurations and initialization changes, we were able to quickly extend support within our CSI driver from Kubernetes to Nomad as well. This is a testament not only to our storage system but also to the commitment of the CSI community to provide a common specification that truly works for any container orchestrator and storage plugin.

Nomad CSI driver installation

With Nomad, CSI plugins are easy to install and have first-class support as part of the Nomad job specification. The following describes an example of how simple it is to add a job with CSI in Nomad:

csi_plugin {
  id        = "portworx"
  type      = "monolith"
  mount_dir = "/var/lib/osd/csi"
}

This may come as a surprise to Kubernetes CSI storage administrators, who are used to installing several CSI sidecars and a handful of other configuration objects to get CSI running. Nomad took a monolithic approach to CSI, which makes installation and understanding the architecture far easier. This approach, of course, has its downsides: monolithic release cycles, a single large CSI codebase, and a single point of failure per node. We will see which approach ends up stronger in the end once Nomad CSI matures to the same feature parity as Kubernetes. Full installation documentation for the Portworx CSI driver can be found at our docs site.

Stateful storage with Nomad CSI

CSI driver volumes can be dynamically provisioned through Nomad with a simple volume create command. Various storage parameters—such as the volume access mode, size, name, and much more—can all be set through the volume specification in a .hcl file. Below is an example volume specification for creating a 1GiB volume that allows single-node read and writes access:

id           = "volume-1"
name         = "database"
type         = "csi"
plugin_id    = "portworx"
capacity_min = "1G"
capacity_max = "1G"
capability {
  access_mode     = "single-node-reader-only"
  attachment_mode = "file-system"
}
capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

The above Nomad CSI volume specification can be used to create a new CSI volume. The following command will take the specification, forward it to the desired CSI driver, and create a volume on the storage system backed by the CSI driver.

nomad volume create volume.hcl

Nomad community

While adding Portworx CSI support for Nomad, we found a few gaps in the ecosystem that limited some features of Portworx Security. Specifically, PX-Security depends highly on the container orchestrator to manage secrets for the CSI driver. These secrets contain user authentication information for the Portworx cluster and must be provided by the container orchestrator to authorize volume management. Portworx assisted with the implementation of this for a few CSI requests in Kubernetes and is now doing the same with Nomad.

So far, we have worked with HashiCorp to enable snapshot creation to pull secrets from the secrets provided during volume creation, and we have implemented the option to provide secrets to the Nomad list snapshots request. In addition, we’ve also fixed a few other minor bugs and docs issues. We look forward to continuing participation in the Nomad community!

How can you contribute to Nomad or CSI?

The CSI community meets monthly to discuss a variety of topics. Typically, the team will go over proposals for additions to the CSI spec. To learn more about participating in the CSI community, check out the CSI community repository. To contribute to Nomad, follow the development instructions in the Nomad repository.

Our experience working in these communities has been very positive and welcoming, and we highly encourage participation in them if you would like a new feature added to Nomad or CSI!