Full Video Transcript

good morning good afternoon good evening uh thanks for joining us today we are going to talk today about jenkins on aws with portworx i’m excited to be joined today by chris kennedy and welles ciao from aws and i’m very excited to dive right in this conversation so with that let’s get started so i really like to start talking about containers and kubernetes and and everything that’s going on with data because data is really at the heart of what’s driving this revolution from the analog world to the digital world and it’s what’s bringing containers and kubernetes along with it as we see customers seek to convert their business or accelerate their business as we’ve shifted uh into this coven 19 time into this new global economy finding a way to harness that data and power and derive insight from that data is really the key to unlocking businesses of tomorrow and what we’re seeing is the successful companies are using a new cloud native stack to process this data into value and insights they’re using a stack that is built on hybrid cloud infrastructure that’s driven by automation with kubernetes and containers that’s using a wide variety of data apps and data services often open source technologies like cassandra and kafka and elasticsearch and they’re doing this with a set of new people and processes things like devops and git ops data pipelines and all of this comes together with containers in kubernetes so we’re seeing 95 of new apps are being developed in kubernetes 78 of cncf members are using kubernetes in production 81 are using multiple clouds and 85 of global businesses will be running containers in production by 2025 85 so this is a huge uptick from just 35 in 2019 but as cloud native scales and there’s always a but storage starts to have some issues and it’s really because the majority of businesses are scaling a traditional storage stack that wasn’t built for containers and kubernetes they have the same business requirements they still have to protect user data they still have to have mobility between clouds between data centers between on-prem and the public cloud they have to be able to deliver performance with strict slas have a z backup and recovery they have to do all the things they did in the traditional world but they’ve got to do it in this new cloud native world and those traditional storage systems just don’t fit the bill but unfortunately 75 of data management professionals believe that management for containers is the same as it is for virtual machines and we’re here here to tell you that it’s not it’s not at all the same what you find in the virtual world is maybe needing to scale to 100 volumes maybe needing to do 10 to 12 tasks a day maybe needing to put the data in one data center and never move it again in the container world in this app defined world the app is always moving the app may be on-prem that may go to the public cloud uh the app may be constantly being spun up and scaled being spun down i may be having hundreds of thousands of api api calls a day a week with hundreds of thousands of volumes so the scale the magnitude of this app defined world is dramatically different from this virtual world and we need a storage approach that is built for that app defined world and that’s what we’ve done here at portworx we’ve built a complete data services platform that can support you anywhere that amazon eks is deployed with any app running across the top across any aws region both on-prem with aws outpost or eks anywhere or on the public cloud and we do it to support every stage of the journey so whether you’re in the pilot stage the dev test stage going live in production scaling globally the portworx kubernetes data services platform was built from the ground up to provide those core business requirements you need to run the application in production things like the core storage backup dr application mobility infrastructure automation all the things you need to be confident in putting a containerized application in production at scale now if we double click a little bit on what’s underneath the hood in the platform it’s really six key modules that we bring to the table that we’re focused on delivering value to our customers the first is pxstore pxstore is focused on delivering that software-defined storage layer that automates container storage for developers and administrators so making it easy for you to have container optimized volumes for your container environment for your containerized applications the nexus px migrated px migrate is focused on making it easy for you to move clusters and applications between different locations so it might be between different availability zones it might be between an on-prem deployment and a public cloud deployment it may be between different data centers running different uh eks anywhere clusters whatever it is with px migrate we make it simple and easy uh for you to move your applications across those different locations with px secure we’re focused on making sure that we have the security apparatus you need to run containers in a safe secure fashion so implementing consistent encryption implementing role-based access control making sure that we’re lowering the risk by automating protection of your container applications and making sure your environment is safe and secure pfcr is focused on ensuring we have high availability between availability zones and regions and clouds so that we can get you to a zero rpo stance if your business requires it and eliminate downtime entirely the nexus px autopilot with px autopilot we’re very focused on making sure that you scale in an efficient and intelligent manner so your app today can grow with the storage as opposed to needing to get all the storage up front you can grow and auto scale that storage as you need it saving on storage costs as you as you scale up and scale down that application and the last is px backup px backup was built from the ground up to provide half consistent backup and recovery for cloud native applications both the data itself but also those key kubernetes artifacts that are necessary to have a complete restore and with px backup we make it simple and easy for you to back up your application and store it in any s3 enabled location so it’s simple and easy for you to back it up and also restore it back to wherever you want it to go now with all of that said there’s really four areas that we have focused heavily on uh partnering with aws to really make sure that we’re providing the best opportunity and the best environment for our customers the first is cross availability cross az high availability so by running portworx on top of amazon evs we make sure that your volumes are automatically replicated across availability zones providing you with high availability and faster failover for your kubernetes applications with backup and recovery uh which i just talked about we provide point and click uh backup and restore of those kubernetes applications across regions and we also provide that zero rpo disaster recovery that i mentioned a second ago uh next cross region migration again with that px migrate tool we’re able to let you migrate entire kubernetes applications including the data and app config between any aws region or aws outpost making it simple and easy for you to scale up and to move uh your data where it needs to fit and last is optimizing infrastructure so again with that px autopilot capability you can optimize your ebs usage and manage or spend intelligently provisioning storage when you need it where you need it so with that i’m going to turn it over to welly in a second to talk a little bit more about eks uh hopefully we’ll continue to keep your interest in in what portworx is doing throughout this presentation but uh just to put this here if you want to try portworx we have three trials and we also have an aws marketplace link available for you there so please do check us out please at the end of the presentation and without wally i’ll turn it over to you to talk about amazon eks</p><br /><br /><br /> <p>thanks andy hi again everyone my name is wily salve i work for aws as the senior solution partner solution architect uh now before we talk about uh jenkins or jenskin on containers later on i’d like to discuss how today our customers is using a kubernetes to to run the workload in aws</p><br /><br /><br /> <p>so over the years we see these explosions of the growth in kubernetes adoptions so kubernetes really simplify that deployment and management of your containers applications it allows you to do configure and run your applications at the production scale and really integrates with a lot of the networking storage for your applications they choose it because kubernetes offer this vibrant ecosystems and community it has a consistent open source api and also very broad flexibility with that said managing kubernetes at scale for the enterprise is hard you have to run the cluster and production scale it’s very complicated because you have to have a trained resource who can spend significant time looking at managing upgrades monitoring and scaling the control planes and also we talk about deploying and managing the lifecycle of dos patches security and etc in addition to that the operation team also needs to ensure that the workload is deployed in the secure manner without any exposure to any non vulnerability for example and this also further increases the operational overhead of running communities of your own and really all of this does not really move the business forward we want to be customer obsessed and our customer wants that to focus on the application itself and the idea that will allow them to deliver new products and not about this heavy lifting undifferentiated work</p><br /><br /><br /> <p>gonna go next night thanks so customers trust us to run their container workload and that’s because amazon eks provide this flexibility of kubernetes with the security and resiliency of being aws managed services there’s a couple thing that customers really like about eks first we don’t modify the open source communities this is a fully upstream certified conformant kubernetes we even backboard the security fixes to ensure we are consistent with the open source eks supports up to four different versions of the kubernetes which give you enough time and sufficient time to do a rollout upgrades and as part of this as well we backboard the patches fixes and upgrades to the community’s versions even for the one that the community might not currently support for example for any urgent security patches and then this is uh eks is a managed kubernetes experience where eks do a lot of the work to select the right kubernetes versions finally that the dependency and that’s really provide that security and stability that a lot of the customers are looking for and lastly the whole idea in here is to reduce the complexity of running kubernetes operations what eks provides is the menace experience with kubernetes that enables you to run the application with portability across a kubernetes environment</p><br /><br /><br /> <p>it’s really what’s the idea in here is to let you focus on what is that strategic operations rather than the undifferentiated heavy lifting eks proficient and scale your kubernetes control plan that’s including like the api servers the backend persistent layers and we do that across multiple aws availability zone which again talking about here high availability and also the file tolerance it automatically detects and replace unhealthy control panels and they provide the patch for that control plan itself eks also integrates with a lot of the services that will provide the scalability and security for the applications and eks also performed this manage in-place cluster upgrades for you and what this means is that it’s simplified at cluster operations so as well as of the updates of the eks configuration and security patches so i can see in here right you can kind of shift from doing a lot of the tactical work into doing a strategic operations because now you can focus on things such as product releases application security supply chains or maybe some sort of like customer connectivity that you’re looking for</p><br /><br /><br /> <p>aws provides a very broad services and tool for you to build the applications and containers and as well as a very deep integration with open source and also with aws partners such as sportwork we have the most choice in terms of compute maybe from far gate on ecs on eks i mean and also ec2 and you can choose from arm amd intel and you can also have like a special purpose hardware such as gpu-based instance uh eks provides integration with ebs and efs for storage you can also integrate natively with the aws security and authentication services such as qms for encryptions iem for the authentications and also lights such as a web or web application firewall for the network security eks provides uh automatic provisions of your network infrastructure so you can provision load balancer for example configure your private network and also set up like a route 53 for external access and chris later on we’ll talk a little bit more about how portwork used these services along with the eks to provide additional value for the customers</p><br /><br /><br /> <p>so where do we see a customers using eks today uh ecas really enables customers across a lot of the industry and a lot of the application types so we’re looking here from modern web applications or mobile app and it gets really helps customers to deliver this scalable experience for the end users and also if you look at the data processing in here the integrations and automatic tooling that eks provides helps because there’s a lot of this open source framework such as spark and kafka that can be deployed in kubernetes and make it very easy to process large volume of data for analytics and machine learning and if you look at machine learning itself kubernetes ecosystems support variety of this machine learning framework uh we’re talking here from model lifecycle mode building and training uh the deployments and also the inference not only that i also see a lot of customers now moving their gaming platform to eks and also such as iot and really what we see in here is that on all of these type of use cases what’s very important is to have application delivery and deployments cd is a very popular use cases where a customer can use eks to build the pipelines and also integrating with the framework such as jenkins now talking about running jenkins and eks is also a very popular uh use cases in here and that’s for many reasons such as scalability that i mentioned inside eks over and also the cost and security so with that said there’s also some challenges of running a jenkins on eks and then i’ll hand it over now to chris so he will talk about running ecas sorry running jenkins and eks and some of the use cases and challenges so with that chris i head it over to you thanks wally so um as kubernetes has come to prominence um of course your developers have come along with that and as developers have adopted uh kubernetes as their kind of primary platform to deploy applications and develop applications of course the tooling comes along and so jenkins is kind of one of the major industry standard tools to automate these continuous integration and continuous delivery pipelines</p><br /><br /><br /> <p>traditionally it was run in virtual machines on ec2 but as microservices and kubernetes have come into play jenkins has shifted their architecture and has gone to more of a master agent configuration and this has allowed us to actually bring jenkins into kubernetes as kind of a control plane for that cidc ci cd process so it really gives enterprises an advantage to achieve much better time to value much better deployments and much more consistent deployments and at speed as well but when you start bringing in these i o intensive workloads jenkins is is really one of them the build process can be quite memory intense uh there’s a real need for high availability because you know you’re talking about a group of developers who spend their day focused on this application and every moment that it’s down you could have a team of one to 250 people sitting idle and then ultimately to really get the benefits of of containerization and the flexibility of the cloud you need high performance and high performance access to your data and so that’s really where portworx comes in um next slide please so amazon eks is absolutely fantastic for scalability for that flexibility of being able to deploy across availability zones but when you start talking about handling high performance data high availability and a lot of the enterprise services that we’ve become accustomed to in traditional it shops um it becomes a lot more difficult and it becomes hard to meet slas you know you um as andy mentioned before handling container data is not the same as handling uh virtual machine data so really with with amazon they give us three options for storage we can use ebs volumes and if you don’t need that high availability that cross of uh availability zone resiliency ebs can definitely satisfy your needs but it’s when you scale to that need that point of needing that availability that we start to see some breakdowns now we can solve that with efs um efs is available and access accessible across availability zones but the problem is is that it wasn’t really designed and built for really heavy write i o patterns it’s more for reads and so that leaves a lot of customers asking you know how do i how do i ensure this application stays up it’s always available how do i keep my developers productive and you know how do i make sure that i’m really leveraging the resources uh that aws provides my team in the best possible way um and then you guys have all listened to the news and are well aware of what’s happening in the world ransomware malware it’s it’s rampant and we’re even even starting to see it get into container space and so ultimately you know these same enterprises are asking how do i back that up how do i protect my data and ensure that if something bad happens i can recover from that next</p><br /><br /><br /> <p>and so with portworx at the core we provide a consistent data platform across all of your eks instances and again as andy mentioned whether that’s in aws cloud as proper eks whether you’re deploying eksd into ecs instances or even if you’re doing uh eks on outposts or eks anywhere if you layer in portworx you get through the use of our various modules pxstore backup dr migrate and so forth all of these rich data services that are needed to run containers at scale at the level that enterprises are accustomed to next slide please</p><br /><br /><br /> <p>so if we kind of look at what portworx is doing when you deploy portworx on top of eks we deploy as an application in the actual cluster and we bootstrap as a daemon set and we will actually as you see across the bottom here we will actually consume their ebs volumes and create a abstraction layer or a storage virtualization layer within the the cluster and this allows us that ability to provide that cross availability replica or availability zone replication and it enables us to be able to dynamically provision our volumes all the way down to the container granular level so in an in a application like jenkins there’s a need for different types of volumes in a lot of cases you will see shared volumes like an nfs so that you can read and write from that as well as your deployment volumes which might be read write once and we give you that ability to do that at a container granular level to go a little further we’re also able to apply the security policies the rollback role-based access controls all at um any any level within the cluster that that you choose as you design it so you can kind of see here in the diagram and i just tossed this up but what we effectively do this peach part is port works and as you spin up instances in ebs we are creating a volume and you’ll see that we’re replicating that that’s these orange lines across the bottom so as soon as you request a pvc we start replicating that to one or two additional availability zones and then in the event that you have a pod failure we have an uh application built in the portal works called stork it’s our storage orchestrator for kubernetes and it will actually influence the kubernetes scheduler and tell it hey this node died and my application’s down we need to get that back up and running as quickly as possible and i have that on a node over here in as this example us east 1c so start it up right here and the data is available and this allows enterprises to ensure that their applications are always available but also always performant they’re not having to try to reach across availability zones for that data as it comes back online and more importantly they’re not having to rehydrate that that data to bring that application back online so not having to immediately go to a backup and do a restore next slide</p><br /><br /><br /> <p>and then the big thing that uh really and truly offers value for enterprise is our ability to provide for disaster recovery and business continuity again kind of as i mentioned before every moment that you’re unable to process software builds you have a team of developers that are costing you money and are idle and so what portworx has done is developed a means to be able to create cluster pairs across regions within aws and take advantage of those high bandwidth links that aws is able to provide in between those regions and then we will start replicating not only the data volumes but all of through stork all of the native kubernetes objects that make up that application and that will sit in your secondary cluster at another region uh in my example i’m using east one and west two um but it’ll sit there the deployments are there the data is there everything’s that that is needed to run the application is there and in the event that you were to lose east one you can easily go to east 2 and just bring your replica set setup so scale your jenkins deployment up and because this is all integrated with amazon everything will simply come back online and this kind of speaks towards um wellies comments about the ancillary services that aws brings to the table so in our demo that we’re going to take a look at here in a few minutes i’ve used one of my personal domains in route 53 and so i’m actually able to do this failover and because i’m utilizing aws is dns service and their hosting solutions i’m able to bring my application online behind my load balancers and and uh traffic controllers in aws from this other region and get my developers back to work very quickly but ultimately through this we can reach anywhere from a 5 to 15 minute rpo and the rto is however long it takes for your applications to spin up when you scale them so just a fantastic way to keep your developers busy and on a bad day but also to protect your business from from anything that may occur</p><br /><br /><br /> <p>and then ultimately you know backups are critical and uh to be candid in the process of building this demo i actually had to rely on my backups i was uh working on a pipeline and completely messed up my jenkins deployment and lo and behold i had a backup and i was able to restore and get back to work but portworx backup is really something special in this market because through our application awareness we know where all of the different components of your application live within your cluster not only the data volumes but each of the pods so if you look at something like a distributed database like cassandra where you may have five nodes running pods for your cassandra database portworx backup is able to know where those pods live and quiesce snapshot and back them up simultaneously so that you get an application consistent backup we then store that in aws 3 or any cloud adjacent aws or s3 compatible object store so we do have some customers who want to get their data um to a partner of aws that may offer adjacent co-location so that they’re not putting their eggs in all in one basket and aws enables that for us and you could easily write to a pure storage flashblade once you have your backups in place it’s as simple as if this eks cluster were to crash all you need to do is install portworx backup in another cluster point it to your s3 repository and it will catalog that repository and present them for restores so we have the ability to not only backup your applications but we can restore them to any other uh eks cluster that you might have in your enterprise</p><br /><br /><br /> <p>next slide and so with that we can actually take a look at this solution so why don’t we go ahead and hop into amazon</p><br /><br /><br /> <p>so now that we’ve logged into our management console we can actually take a look at how we’ve built this cluster to support jenkins so we’ll hop over to eks and take a look at our cluster first and if we take a look at the configuration you’ll notice that we have two node groups and so what we’ve done is con configured a set of nodes uh named controllers that will handle jenkins controllers and then agents for builds unit tests and various other tasks that you may need to schedule through jenkins and in our jenkins configuration we’ve actually set it up so that build jobs cannot run on this node group but instead will run from the agents if we click in and take a little look at the actual underlying ec2 instances</p><br /><br /><br /> <p>we’ll see our instant instances here and if we scroll over we’ll notice that for our controllers we have one in each of three regions and these are also our storage nodes so when we go over to our terminal and we take a look at our portworx cluster</p><br /><br /><br /> <p>then we’ll see the six nodes notice that we have three showing up as having storage here at 147 gigs each and then the three that have no storage now a really great feature about portworx is we can present persistent volumes to pods running on these storageless nodes from the storage pool that is presented by the other three nodes and so that gives us a lot of flexibility in the types of instances that we might introduce into eks for various tasks but let’s take a look at jenkins and</p><br /><br /><br /> <p>what we’re able to do to improve that</p><br /><br /><br /> <p>so i’ve already logged into jenkins and you you can see we’ve set up a few build jobs um this one has run 19 times this one 53 and this last one here 11. so i’m going to go ahead and kick these off real fast</p><br /><br /><br /> <p>and we’ll see those start running</p><br /><br /><br /> <p>now while these are wrapping up we are using harbor for our local registry and so i’ve published that here and we can log into harbor and go to our library and be able to see each of those build jobs and so and the tags so if we take a look you know tag 54</p><br /><br /><br /> <p>go back to the library same thing with pybot which happens to be uh that twitter bot build that was at the bottom and it has finished as well and so these are all successful and if i</p><br /><br /><br /> <p>refresh my dashboard will notice that the numbers have all increased so let’s take a look at just how resilient portworx allows our cluster to be when it’s spread across three different availability zones so first let’s see where jenkins is run running</p><br /><br /><br /> <p>so jenkins is on this node here so i’m going to go ahead and copy that to my clipboard</p><br /><br /><br /> <p>and then we’ll cordon that</p><br /><br /><br /> <p>so that node’s now been cordoned let’s uh delete our pod</p><br /><br /><br /> <p>and if we do a watch</p><br /><br /><br /> <p>we’ll see that’s already started coming back online</p><br /><br /><br /> <p>now if i refresh jenkins over here right now you’ll notice it’s down</p><br /><br /><br /> <p>but now that we see this is reached 2 of 2 the the instance is completely up and we can refresh and be able to sign back in now because we’re replicating that data between the various availability zones notice that my build numbers are there so my history of these jobs are still intact and i can kick another one off if i wanted to</p><br /><br /><br /> <p>and if we take a look at where</p><br /><br /><br /> <p>jenkins is now</p><br /><br /><br /> <p>it’s now instead of being on uh 244 239 it’s now on 30 to 47. so let’s take a look</p><br /><br /><br /> <p>and so it went from running in east 1a and instead failed over to our storage node in east 1b and is back online and or i’m sorry that’s backwards we went from east 1b to east 1a but regardless within just a few seconds we were able to recover all our history is intact and our developers are still working and being productive so um you know here we’re able to solve for that kind of regional cross availability zone high availability and that is just um something that is table stakes for an enterprise now the other com thing that we really bring uh to the table is backup so since we</p><br /><br /><br /> <p>have a ui available for both the cluster maintenance itself at day two type operations as well as our backup that’s all located at pxcentral now i’ve published this using pxcentral.6words.dev but you can publish this however you want we do support role-based access controls so you can create mappings through to your oidc or saml provider and be able to use your existing authentication mechanisms to set up various roles which will in turn allow you to give people access to different parts of the application or be able to configure backups for various namespaces and such but let’s take a look at lighthouse first lighthouse is our dashboard so here is our production cluster you see our six nodes and here are our uh</p><br /><br /><br /> <p>our storage nodes now we can do a lot of stuff in this we can see the drives that are mounted and look at it by node we could even</p><br /><br /><br /> <p>do things like go to a particular volume and take a snapshot i do like that portworx central is designed with multi-cluster deployments in mind you’ll notice i’ve got both my production and my dr cluster and for backup we’ve already set up a set of uh scheduled backups to protect our environment and there’s a few things to look at here so first off we’re able to set up schedule policies so right now i have two different nightly backups so that i can back up harbor and jenkins at two different times and then a weekly backup and then a monthly backup and these each have different retention policies but at the end of the day the you can build these out to meet the slas of and retention policies that are dictated by your business needs</p><br /><br /><br /> <p>we can also set up our storage location so in this case we’re going to have set up some credentials now i did create an i am service account for this so that i don’t have to worry about using someone’s credentials and then uh we set up a s3 bucket as well and so we can hop in and look we’re using this account which was the service account that i created this is the bucket i did put in an encryption key however it doesn’t show it to me when i go to edit and we’re using the class standard</p><br /><br /><br /> <p>to back up an application in portworx backup it is super simple you just jump into the cluster that you wish to take a backup from select the namespace or you can allow it a few moments to catalog at all of your namespaces and select individual items or deselect individual items so we’ll give this a moment</p><br /><br /><br /> <p>and so now that it’s listed those out if i did not want to include as an example this target group binding then i can remove that and what the target group binding is is actually a dynamic object that’s created by aws ingress controllers so i actually don’t want to back that up or it’s not needed so i can hit apply and then i can hit backup and from here i’ll give it a name</p><br /><br /><br /> <p>select</p><br /><br /><br /> <p>oh it doesn’t like my dashes</p><br /><br /><br /> <p>or the length</p><br /><br /><br /> <p>we’ll do it now we do have the ability to inject pre and post rules so these would be used with databases to do things like um qis that database app equals harbor so that we have a tag on it and let’s start that over</p><br /><br /><br /> <p>we’re going to do now again we don’t have any rules we’re going to use</p><br /><br /><br /> <p>put a tag in here</p><br /><br /><br /> <p>and create that backup and so that backup will start now you’ll notice we’ve got this timeline and it’s showing the last 24 hours but if i were to select the last 30 days it’ll actually show me all of my backups over that time period so i deployed this cluster and started around august 24th 23rd and you can see the backups have all been successful since then</p><br /><br /><br /> <p>so while this is backing up let’s hop over and take a look at what the restores look like so let’s say that</p><br /><br /><br /> <p>we want to</p><br /><br /><br /> <p>create a second instance of jenkins</p><br /><br /><br /> <p>or like what happened to me i messed up my jenkins and i need to recover from last night’s backup so we have our nightly backup here and i can come in and select restore give it a name</p><br /><br /><br /> <p>select my production cluster and tell it i want to replace existing resources and if i were to click restore then it would overwrite everything in the jenkins namespace that we captured so that would be all the role bindings config maps everything needed but i don’t want to do that because my jenkins instance is working so instead what i’m going to do is i’ll actually</p><br /><br /><br /> <p>restore this over to our dr cluster and i’m going to put it into a separate namespace</p><br /><br /><br /> <p>now one thing i do not want is</p><br /><br /><br /> <p>oh they are excluded</p><br /><br /><br /> <p>based on some tags that i put in place so we will go ahead and kick that off and so again this kind of demonstrates that multi-cluster power of px backup you know not only could have uh recovered from a mistake i might have made but in this case i can take my backups and restore them to another cluster in my enterprise and have those available for use so let’s take a look at backups</p><br /><br /><br /> <p>on our original cluster</p><br /><br /><br /> <p>and we’ll see that that backup was successful captured six volumes 56 resources and uh 1.6 gigs of data so you can see that is super simple to set up super simple to eat to run when we take a look at the schedules obviously obviously you can see these are schedules that i’ve created and the resulting backups have been successful so all of these applications are protected and can be recovered and when we take a look at this restore we’ll notice that it completed successfully in our dr cluster so if i go to back to backups and click into the dr cluster i have a new namespace that wasn’t here before and that came from that restore so that’s how we can use pxbackup to automate backups and ensure that we’re meeting our data protection and backup retention slas for our enterprises and finally let’s talk about uh business continuity and disaster recovery because downtime costs company money companies money lots of money and uh you know there’s a few simple steps that can be taken with portworx to ensure that your companies and more importantly with jenkins your developers don’t sit idle and that they’re able to continue to work so we do that by creating what we call a cluster pair and that’s done with our um stork application which we as i explained before is our storage uh orchestrator for kubernetes so if we take a look i’ve already set up replication to a second site so you’ll see that we have a cluster pair already created and this is uh</p><br /><br /><br /> <p>basically consists of the um cube cuddle or i’m sorry the cube config being provided to stork so that it’s able to talk to that remote cluster</p><br /><br /><br /> <p>and then once it’s able to talk to that cluster we actually pair both the kubernetes scheduler as well as the stork scheduler or the storage layer through stork and we create migrations so we can actually get those migrations</p><br /><br /><br /> <p>and we’ll note that you know within the last little bit we’ve had three times that jenkins has replicated based on our daily our interval and our weekly and the same for harbor and those are based on schedule policies so let’s take a look</p><br /><br /><br /> <p>so our schedule policies are here harbor uh and jenkins so if we do</p><br /><br /><br /> <p>we can take a look at that schedule and basically what we are saying is that we want</p><br /><br /><br /> <p>right here daily we’re going to replicate at 10 15 pm we’ll also replicate every 15 minutes monthly we’ll replicate on the 14th day of the month at 805 and then we’ll have a weekly full replication every thursday at 10 15 p.m now this is a little bit overkill as far as the number of different types of replications but i’ll put these in here to show you the possibilities most of our customers will use a daily full with um interval of maybe 15 minutes half an hour hour depending on what their uh recovery point objective is when we’re doing region to region replication so in this case from u.s east 1 to u.s west one our most frequent replication schedule is every five minutes and depending on the amount of data that you’re moving we generally like to say that that’s a a pretty trustworthy recovery point objective of every 15 minutes and so not only do we give you the ability to have this replica ready and let’s take a look at that real quick so we will expand this out a little and jump over to the other cluster</p><br /><br /><br /> <p>so i’ve now connected to my cluster in aws west one</p><br /><br /><br /> <p>and if you notice</p><br /><br /><br /> <p>we only have three nodes here so if i go back over to ec2 or aws console and to eks and switch to california</p><br /><br /><br /> <p>in order to help save i’ve configured my agent node group with a desired size of zero but they can expand to five nodes so if i needed to begin running workloads in that remote cluster based on how i’ve configured jenkins to run my my various kubernetes pods on the agent nodes auto scaling group would kick in and automatically deploy agent nodes for me to run jobs on</p><br /><br /><br /> <p>but one nice thing that we are able to do is we’re able to test this dr while keeping our um our developers busy in that production uh environment so let me switch back well first actually uh we were going to take a look so if we do cube cuddle get all in jenkins which is where our jenkins application is you’ll notice that we’ve got the service we’ve got the stateful set but there’s no pods and so if we get rep the replica sets for that</p><br /><br /><br /> <p>we’ll see that there’s none none in existence and that’s because as part of our awareness of kubernetes we know to create the stateful set but do not scale it up do not create replicas of that stateful set to spend which keeps the pods from actually starting up but tubes start them up and this is actually how we can test our disaster recovery we can in turn go back to our other cluster</p><br /><br /><br /> <p>and we can pause that migration schedule</p><br /><br /><br /> <p>using stork</p><br /><br /><br /> <p>and so now that we’ve paused that um basically the migration the replication from our production cluster to our dr cluster has been suspended our developers are still working they’re still producing code running jobs and uh in the meantime as part of our you know day two type operations we can switch back to that other uh cluster</p><br /><br /><br /> <p>and we can scale up</p><br /><br /><br /> <p>our replica set and check our dr site make sure that it’s uh that it’s working</p><br /><br /><br /> <p>so i’m actually going to scale my stateful set we’ll uh</p><br /><br /><br /> <p>wait on that because it does need to pull its images and everything</p><br /><br /><br /> <p>now again all the while</p><br /><br /><br /> <p>people are working in this instance of jenkins</p><br /><br /><br /> <p>and we’ve reached two of two now</p><br /><br /><br /> <p>if we did we’re actually failing over to this instance of jenkins i would now grab my in ingress uh manifests and apply them to the cluster to be able to redirect route 53 to these hosts and get my developers back into jenkins and get them working again but instead we just want to verify that the replication is working and that we’re ready for that dr so we’ll just port forward uh to the service and check it out that way</p><br /><br /><br /> <p>so now we have that port forward happening and we can go to</p><br /><br /><br /> <p>here</p><br /><br /><br /> <p>oh let me sign in as me</p><br /><br /><br /> <p>and we’ll see that we have our builds we have our build histories</p><br /><br /><br /> <p>we have all of our configuration and everything is in place</p><br /><br /><br /> <p>ready to go and if i were to click build now and start this then this auto scaling group would kick in create some nodes schedule my jobs and i would have my developers back to work so let’s go ahead and stop that port forward and scale that back down and so now that we have um proven to the business as part of our our checks and balances that we can indeed fail over to our dr site we want to re-enable that replication and get everything caught back up so if we go ahead and switch back to</p><br /><br /><br /> <p>our production cluster then we can go ahead and use the stork cuddle command line utility and resume</p><br /><br /><br /> <p>oops</p><br /><br /><br /> <p>and resume that migration and if we were to do a</p><br /><br /><br /> <p>then we’ll see that</p><br /><br /><br /> <p>29 seconds ago there was another replication and that was basically port works bringing our remote site back up to um</p><br /><br /><br /> <p>back up to nsync with our our primary site so as you can see when we combine eks and portworx we get a very robust container platform to be able to allow our developers to be their most productive selves thank you thanks chris uh thanks for walking us through that and i know we only have a short time left i wanted to leave a little time to ask a couple questions uh you know that we tend to hear from the field the first one i want to touch on is we talked a lot about about backup and restore i know with jenkins this idea of a golden image can be very powerful in helping you quickly and easily scale out your jenkins deployments or your jenkins work work jobs around the globe can you talk a little bit about how portworx helps enable that golden image and why it’s so important for developers in this in this world yeah so i mean with a golden image ah you know obviously you’ve got a predefined set of tools and configuration for your jenkins instance and by leveraging something like port works you can very easily</p><br /><br /><br /> <p>deploy those in any cluster in the same consistent way without having to refactor that template whether you’re in aws proper aws outposts or sorry welly um vanilla kubernetes in you know rackspace but that’s really where portworx comes in is we’re able to take any kubernetes distribution and provide a common data plane to where when you’re developing your applications you’re deploying your applications everything is consistent and the same</p><br /><br /><br /> <p>and the other question i wanted to ask and and we know that’s valuable the golden image but security you mentioned that you know throughout the presentation you mentioned it while i mentioned it it’s really vital to enterprises today you can talk a little bit about how portworx and aws are working together to leverage aws tooling along with what portworx brings to the table to make sure that the jenkins builds and everything folks are doing with aws and portworx is safe and secure yeah absolutely and that’s a great question andy um aws has spent a lot of effort developing their security framework and you know tools like their web application firewall uh identity access management you know these tools are all compa and their key management service are all tools that portworx is able to leverage so we do integrate with any oidc provider which is what i am is and we’re able to use that to define what groups within an organization can access or have the ability to create volumes um we can use it to you know kind of segregate teams between namespaces or between eks clusters and then we can take that a step further with the encryption component and be able to use aws key management service to provide encryption at rest for all of your workloads and tie that key at as granular level as is appropriate for your business needs so that could go all the way down to an individual container volume or it could be for the entire namespace or the entire cluster and we have the ability to do it with multiple keys so if you have multiple teams they may each have their own encryption key for their work area so it really gives a powerful tool to ensure that not only are the right people accessing the data but also that um it’s secure no matter where it lives and i like to add to that chris you mentioned a very good points about the use of like a key management services and everything there is api driven as you said like you can define the key policy to the certain user groups or the team and not only that the encryptions and the key management are handled by the qms itself but then all of the trail of the activities of encryption and decryptions and all of the api api calls that have been made to the kms is also being locked into a cloud trail so really you’ll have the enterprise grade not only you’re securing your key but you also have the log and audit trails of who is doing what inside of the cloud trail yeah and that’s true actually of portworx as well we we’re able to report most our you know operations in there and so cloudtrail can become a nice audit point um to to be able to go back and look in in the event of a security breach or an audit from from your uh friendly uh compliance organization based on your business</p><br /><br /><br /> <p>so just to recap i know we’re we’re tight on time here just to recap what everyone heard today we started talking about portworx we said we talked about how portworx help solves the solve those storage challenges as folks are moving into this cloud native architecture things around dr aj backup and restore and portworx is really positioned as a platform that can cross across across over any kubernetes distribution anywhere eks anywhere runs anywhere that aws outpost is any aws availability zoom and then we talk about how eks is the perfect solution for managed kubernetes so you are wanting to take advantage of everything kubernetes but rather than focus on the actual nuts and bolts of running kubernetes letting amazon do that with eks and letting you focus on running those workloads that are important to your business that are important to your industry helping you develop new applications and new innovations in your business and leaving the management of the of the back end to aws and then we talked about how jenkins really is at the forefront of moving from that traditional world to that kubernetes world jenkins is moving into that kubernetes space and it’s bringing along all of the benefit of jenkins but it also comes along with some of the challenges how do i make sure i have high performance how do i make sure i have high availability how do i make sure i can support memory intensive workloads and so by taking the power of portworx and the power of eks we talked about how we can provide that ha that dr that enterprise grade support so that you can successfully deploy jenkins at scale with your kubernetes deployment on eks with portworx and chris walked through a few of those examples for you so i hope this presentation you’ve really felt that and gotten to really see what we have to offer here together with with aws and portworx we have a few more resources for you if you’re interested as i mentioned earlier we do have a free trial that you can you can check out you can also get started on the aws marketplace we do have a portworx and aws resource page and on that page you’ll find not only topics about aws in general but you’ll find a great technical whitepaper deep dive that chris put together on jenkins and eks and really walking you through every step of how you get it set up how do you make sure that you’re deploying it correctly how do you make sure you’ve got all the pieces together to make portworx eks and jenkins work while in production uh and of course we do have some content on our youtube channel and we do have monthly demos where we touch on topics like this so please feel free to check those out so with that thanks for joining us today we really appreciate it and we hope to see you on portworx and eks in the future thanks a lot thanks ali thanks chris thanks<br /><br /><br /><br />