LCFS: The New Container Filesystem for Modern Datacenters

Docker downloads have exceeded 5 billion pulls — that’s amazing! Whether it’s five, six, or whatever-comes-next billion, containers are fundamental to this next wave of application modernization. Each download in those billions of downloads is handled by a storage driver. Every time a container starts, the storage driver handles what is equivalent to the container’s operating system (OS). Both stateless and stateful containers rely on storage drivers to then boot up the container.

Current storage drivers implement container behavior by mapping onto OS-designed file and storage systems. Since containers have different startup behaviors than OSs, container users must handle the leftovers. Admins have to deal with lots of maintenance because existing storage drivers bloat the OS memory, grinding to a halt as more containers are launched. Users track lists of workarounds and wade through a complex decision matrix to determine when and where to use a particular storage driver.

Today, we are open-sourcing a filesystem that is purpose-built for the container lifecycle, to encourage more innovation in this fundamental technology that boots all containers. We are calling it the Layer Cloning Filesystem (LCFS). Its aim is to improve the speed of downloading, booting, tearing-down, and building containers. Imagine if your laptop started 4x faster. The plan here is to improve the experience with stateless and stateful containers.

Here’s a picture of what we are measuring internally when we launch 100 fedora/apache containers. LCFS (green) is booting and tearing down 4x times faster than Overlay (red) and 11x faster than DeviceMapper (blue).

This is just a start. The future direction is to enhance LCFS with cluster-level operations, richer container statistics, and pave the way towards content integrity of container images:

• Cluster operations: where image pulls can be cooperatively satisfied by images across a group of servers instead of being isolated to a single server, as it is today.
• Management: easy garbage collection of images.
• Statistics: answering what are the most popular layers, and so on.
• Content integrity: ensuring container content has not been altered.

It is important to evolve this storage driver with the community. We welcome feedback and encourage others to use, modify, and contribute. You can find us on Github. We realize that advancing the use case of stateless containers might seem strange coming from Portworx. But building the modern data services layer for containers includes all forms of data. Over time, we see LCFS as an important technology for the community. With your feedback and support, we hope to advance the experience for all container users.