How GE Digital Tackled the Stateful / Stateless Problem for Industrial IoT

Alex Williams of The New Stack interviewed GE Predix and Portworx at MesosCon on September 15. You can read the article here and the interview transcript below:

Alex Williams: Hey, it’s Alex Williams at The New Stack, here at MesosCon in Los Angeles, California. It is coming to the close of the conference on a Friday afternoon and I’m lucky to have people here from Portworx and GE Digital. And so what I’d like to do is for you guys to introduce yourselves.

Goutham Rao: Sure. Hi, my name is Goutham Rao, I’m the CTO at Portworx.

Balajee Nagarajan: Hi, my name is Balajee Nagarajan, Director of Software Engineering at GE Predix.

Venkatesh Sivasubramanian: Hey, my name is Venkatesh Sivasubramanian, I lead the data platform for Predix.

AW: Great. The topic for today that I want to discuss is about storage in the maturing container technology ecosystem. We now have seen containers become widely used since their introduction, really as Docker three years ago. And now we’re seeing the need for companies to really think more about networking and storage, and Portworx has a particular focus on storage. And so perhaps, Gou, you can tell us a little bit about Portworx and what you guys do.

GR: Portworx builds cloud-native storage software. What that means is it’s a new kind of storage solution designed for applications that are deployed via a container orchestrator such as Kubernetes or DC/OS. Modern applications are service oriented. They’re packaged as containers and they’re inherently meant to be run on cloud native infrastructure in a distributed manner. Essentially what is fueling all of this is the need for enterprises to embrace DevOps and agile software development and deployment methodologies. Where Portworx comes in is we provide a storage layer that is purpose built for these kind of applications. The whole premise behind it is programmatic deployment or consumption of storage services for these applications like Cassandra, Postgres, Mongo, these type of applications that are packaged as Linux containers and deployed by a container orchestrator.

GR: These applications can programmatically specify the type of storage resources they want. They don’t need to know what kind of physical underlying infrastructure they’re running on. They don’t need to know what kind of public cloud provider they’re running on. Portworx provides that abstraction layer, something that the DevOps teams can directly deal with it. It gives them a uniform experience much like DC/OS gives a uniform compute experience.

AW: So how are you all working with Portworx? What’s the connection between GE Digital and Portworx?

BN: So before we go there, let me set the stage on what GE Digital is doing first on Predix. So Predix, we started on this journey of Predix a couple of years ago. Predix is a world-class highly secure industrial IOT platform that GE is building back in San Ramon, California. What does this give us as GE? GE has been the forefront in the industrial space for a long, long time. What we bring to the table is our industrial wealth knowledge…our domain expertise at GE is aviation, power, transportation, oil and gas, lighting, all this stuff. What we’re doing next is take all of that expertise to digitize the industrial internet itself. So we’re building a next-gen industrial internet. As part of that journey, we have a world-class Predix platform as a service, that we back in San Ramon are building. In that, we today use Cloud Foundry and we’re adding our stateful container architecture using Mesos. Portworx is our partner in crime, helping us to solve that stateful need in Predix.

AW: Why did you choose Portworx?

VS: So when we started looking at Mesos, it’s probably six to eight months ago. So we started looking Mesos particularly for the persistent store, the capability that it actually offers from an orchestration perspective. Because when Predix platform was originally started, we’ve always been with Cloud Foundry. There is a big community within Cloud Foundry and we have a large presence at Cloud Foundry and Cloud Foundry has been good for stateless applications. So when we started looking at Mesos, for us, one of the primary things is we looked at CSI, which is a container storage interface. That was actually available at that particular point, but it was still not ready. It was still not matured enough, so we stared, Okay, let’s… ” But we do need to do a single infrastructure. We need to have a single infrastructure that actually allows us to be able to run both stateless and stateful. And Portworx had a great solution with DVDI interfaces and that was ideal for us. And it also offers a few things that are actually super important from an industrial company perspective. One is security and compliance are super key for us. So we need a solution that actually supports encryption out of the box from a storage perspective, snapshotting out of the box available. And Portworx had those capabilities which actually made us gravitate towards them.

AW: What are some of the challenges that you’re seeing customers face with storage environments? We’re seeing more capabilities to use adapters for example into different storage environments. These plug-in type services. What are those issues that you’re seeing customers face when they start exploring how they actually can run these applications that are in these stateful environments, effectively using container technology?

GR: One of the fundamental tenants of what Portworx is building, is to actually provide that abstraction layer that sits on top of physical storage. The problem that people typically run into, is when they try and take a traditional volume and try and somehow orchestrate that into a container and let’s talk about that a little bit. Traditional storage, like an iSCSI LUN, these are meant to drive storage to a host computer to a operating system to a machine.

GR: They’re not designed to any SAN or NAS solution out there, its end user or its client is typically a host computer, not a container that’s been orchestrated by a container orchestrator that is meant to run in a multi-node environment. And let’s double click on why I say that. An application is never one single container, right? So take an example of Cassandra, it’s usually deployed as a set of containers. A Cassandra cluster could have three, 10, 15 Cassandra containers that are all orchestrated and deployed across maybe 15 different machines, potentially even in different availability zones.

GR: So to try and take a volume and somehow magically orchestrate this into a distributed application environment, you end up with a lot of problems, that ends up having the operators or the architects for the platforms as… Architects that are building the PAZ, have to go in and add significant glue around and making sure how to orchestrate these into these into the applications. And then you run into problems, exactly like Venkat was talking about, which are, now I want to take a snapshot, or I want to have a common encryption scheme across this particular application. Maybe the key used for that application is different for a completely different business unit that’s deploying a different application.

BN: That’s right.

GR: How do I provide these different keys in cloud neutral way and don’t burden my end users with making this a science project? That’s really what the value of the Portworx layer is. At the core, Portworx is a software-defined storage solution. Venkat can talked a little bit about CSI. CSI is going to be coming out in a couple of weeks. As soon as it’s out, Portworx will have one of the first implementations of CSI out there. In fact, in a couple of weeks we’ll be announcing a release for CSI.

GR: But all of these are meant to make the integration with the container orchestrator much more easier and to make the lives of Venkat and Balajee in managing their platform as a service really hands off. And everything is just happening on its own. They have end users their own internal customers that they have to catered to, and in their ideal world they’re going to put together a platform where their end users can go in and programmatically say, “I’m going to run my five node Cassandra cluster, give me a few terabytes or storage” and boom it happens. They don’t have to go in and provision it beforehand. The question you asked is what are problems people running into, that’s what they’re running into. When somebody has to run, for example, a five node Cassandra cluster, somebody has to go in and figure out, “Which machines are this going to run on? How do I get storage to it?” And then invariably those applications are then permanently pinned to those machines because they have consumed resources from it. And when a machine dies, somebody has to go in and figure out how to take it off of that and move it somewhere else, and you’re really not leveraging any of the cloud native benefits that these orchestrators want to get.

AW: Right, right. So how do you manage that underlying architecture with these storage considerations in mind? So if a node goes down, you can just take care of it?

BN: Sure. So one of the things that we started off was to build a cloud-agnostic platform, so that we don’t want to overburden, just like what Gou just said, we don’t want to pass on the onerous task of managing infrastructure on to our end customers as well. So we’ve abstracted out and there’s abstraction that we go on with every layer of Predix itself. So when we go apply Predix on a brand new point of presence, be it Amazon or Azure, we have automation that abstracts that out the infrastructure first. And then we go down and lay a common interface of our platform as a service layer, which is Cloud Foundry and Mesos. At the end of the day our end users interact with one or both and they consume Predix, no matter where Predix is deployed. Within that we provide constructs, for example the talk that we gave today was our open sellers broker implementation, which is essentially a broker marriage between Cloud Foundry and Mesos, where people living in the Cloud Foundry land can consume a back end service running on Portworx using our broker interface. That’s how we are solving that problem.

VS: That’s right. Just to add that we are planning to open source the open source program implementation at some point pretty soon. Can you sort of go back and touch the question again?

AW: Yeah I just was looking at understanding your own technical architecture and how you think about storage there? You maybe can tell us a little bit of more about your own technical architecture. What programming language are you using for the underlying architecture? What is that run time? And how you have to think about storage differently.

VS: Okay perfect. So Predix is an IOT platform that actually offers folks to be able to run any language, applications on any language pretty much, so it’s a microservice based architecture. But, to focus on the storage portion of this conversation… So prior to Portworx is our involvement or this thing with Portworx, so we’ll always run all our persistent store using an infrastructure that’s built on top of BOSH. Because Cloud Foundry doesn’t natively support running these sort of pin persistent clusters, so we got to basically set this up outside of Cloud Foundry, attach it to Cloud Foundry via our service broker. With this newer model, we’re able to essentially, one, use the the two-state scheduler from the Mesos itself to decentralize the scheduler portion and also use a solution like Portworx, so we can essentially go and do dynamic persistent volume portioning.

AW: Excellent.

BN: Maybe also you can mention your user frameworks which are…

VS: Oh yeah, absolutely. When we started looking at Portworx and Mesos, I think one of the very important things for us is the availability of at least some persistent stores or frameworks on Mesos, to actually use that as a starting point. So we looked at Elastic Search, we looked at the Cassandra framework and we looked at Redis and other persist frameworks. And we started working with both Mesos Mesosphere and Portworx, to make sure that these frameworks actually have a native support for a dynamic persistent volume. It started as a beta and then we’ve essentially made some good progress right now. So now we are at a position where we can actually go and spin these clusters. And some of the testing that we have actually done are pretty interesting, in terms of mission failovers and whatnot. So if a particular container actually goes down and comes back up somewhere else, we are now able to have the ability to be able to take that particular volume that’s there on that old failed container and attach it back to a new container. So your customers for most part, of course there are edge cases, these things, for most part are still able to go back and look at the system as if there was no change on the infrastracture we have.

AW: Gou, perhaps can you take us through how you guys built your own technical architecture to be able to provide the service that you do. Can you tell us about the underlying architecture itself?

GR: Sure, absolutely, there’s, at the core, is a distributed block software defined storage solution.

AW: Is it built in, Gou?

GR: No. Its built on C. C++ actually.

AW: Okay.

GR: Performance is very key for this environment. One of the architectural tenets is, look at how modern applications are built today. Just to kind of walk you through it. If you were to roll back 10 or 15 years ago, it wasn’t easy to get… If you had to get 10 or 15, 20 terabytes of storage, it would almost always come off of a network, it would come off either off of a SAN or a NAS. But over the past 10, 15 years, Intel’s ecosystem has made so many advances that now it’s easy to go buy a server and get 10, 15 terabytes of local capacity storage, and you want to be able to take advantage of this. In parallel, modern applications are built like Cassandra or Mongo or HDFS, and these inherently like to scale out and handle the scale out and the aggregation themselves. What I mean is, if you take a look at Cassandra, you would want to give it the illusion of direct attached storage or this notion of hyper-convergence. Keep these two things in mind, because these are important for me to describe how Portworx built out its software architecture. Our software architecture fundamentally enables people to run their applications hyper-converged. So, it’s more conducive for applications like Cassandra because you get that low latency access, your applications are running close to where their storage lives.

GR: Secondly the other thing that Portworx does is, it natively integrates with the scheduler. So it knows how to talk to either Mesosphere or Kubernetes, and work with the scheduler such that these applications are orchestrated in a sensible way. To give you an example of that, if you’re going to deploy a Cassandra cluster, you don’t want all the nodes in the same ring, to end up in the same availability zone or failure domain, or for example if it’s on-prem on the same rack. It’s Portworx’s architecture that lets you figure out how to distribute these applications sensibly. Also, the other thing is, typically people are aiming for close to 80 plus percent server utilization. What that means is, we have to allow for different types of of applications to co-reside on the same hardware, without having IOs conflict between them.

GR: So what Portworx does to mitigate this is, we don’t actually directly take the drive or disk or a physical LUN and give it to an application like other products would do, we provide a virtual software storage volume which is what the application actually talks to. This virtual volume then can implement all of the IO leveling and making sure that people aren’t stepping on each other’s toes, container granular encryption or container granular snapshots are all implemented at that level.

AW: Interesting. How do you see your platform maturing with storage technologies and what do you want to try to accomplish over the next six months or 12 months?

BN: Sure. To answer that better, the biggest challenge that we as platform operators and platform architects what we feel is, when we provide a platform for our end users, we want to make sure that platform is highly available. That’s our biggest challenge, that’s where Portworx comes in. Portworx provides a unified fabric of storage, so that we can then have end users not worry about how to consume an HA application. That’s what Portworx’s uses. And going down six months or a year down the road, we want to be able to have this running across the world in all other points presence as well, other Cloud Providers and then on-prem, definitely.

AW: So Gou, perhaps you can tell us about the next six months and talk about the issue of high availability, ’cause I know you brought this up in a story that you wrote for The New Stack about some of the drawbacks with Amazon EBS.

GR: Yeah I think the key thing to keep in mind here is stateful services with containers over the past 12 months has become such a hot topic, because people are now pushing these things into production at a faster than anticipated rate. The point that I wanted to make in that article is, day one deployment is always easy, right? It’s easy to put something together and say, “Here it’s working.” As time goes on, your systems age, you have various things that come into practice that we’ve learned over the past 12 months in working with GE and others. With what happens in production, drives fail and users are sloppy. They’ll start an application and leave things up and running, and over time space gets consumed, so it’s the day two operations that are difficult. And it’s very important for us to emphasize as an ecosystem over the next six months, where people can go wrong in the day two operations.

GR: And so the type of problems that we’ve seen people encounter are things like network partitions, or drive failures and then having applications all of a sudden have a storm for wanting to be scheduled on a set of other servers, and now all of a sudden you’re draining the storage resources from that server. So, you asked a question about where do I see things going in the next six months, it’s these types of operational practices that we want to jointly let the ecosystem and the community know of what to keep in mind, regardless of the solution that they use. Obviously in Portworx’s case, we’re very cognizant of these problems. We’ve been learning from our own end customers in how to make the product more resilient to these type of things. And I’ve got to tell you, this ecosystem is so fantastic because when there’s a real problem customers are willing to work with you. And they’ll show you the type of problems they have, so it’s important to learn from that, put it into product and my key answer to your question is, anticipating what problems can happen, and then making the product, be aware of it and react to it, and allow for admins to go in and fix any kind of physical infrastructure, but not have to encounter the burden of application level up-time and high availability, it’s those things that are important.

AW: Excellent, any last thoughts?

BN: I think no. You sort of nailed it on the head. I think as we move forward, we’re going to essentially start looking at running multiple types of workloads and people… Probably an application that’s very very high IO intensive, verses something that’s more Trooper intensive. And ability to be able to run all of this in this one cluster harmoniously is going to be the key thing for us. And then making sure this is highly available, it’s durable enough that we can actually relay for very business critical applications.

GR: And I think GE showcasing what they’ve build over here, that’s key, right? Because we work with a lot of customers and it’s the same thing that they’re trying to do which is… The key goal here that cloud architects like Venkat and Balajee are trying to do is, how do I get my end users that same agility that the Twitters, the Facebooks and the Googles of the world get? How do I build a programmable self-service layer that’s cloud agnostic, they can go from AWS to Microsoft to on-prem seamlessly and not impact their end users? And the more they talk about what they’ve done there, that also is important.

AW: Great, well thank you all for taking some time to talk with us today here at MesosCon. And we look forward to keeping in touch about all your progress.

BN: Cool. Thank you.

AW: Thanks.