Engineering For Product Velocity: The Case For CI/CD
In their 2023 annual letter, Stripe mentioned that they deploy code to production up to 400 times per day, impacting services in the scale of handling trillions of dollars.
Let’s pause. What does 400 deployments a day actually mean, even?
Product Velocity Boils Down to Rapid Deployments
Each deployment contains new features, urgent bug fixes, code refactoring and much more. In other words, Stripe is releasing hundreds of product improvements per day. All that improvements will sooner or later reach the customer, and will (hopefully) provide uplift on Stripe’s bottom line.
According to DORA metrics, "high” performing software delivery refers to multiple deployments in a week, and “elite” delivery teams deploy “multiple” times a day. If that’s the standard for elite, then I argue that Stripe’s delivery team is “among the world’s best” at delivery.
Stripe achieves this through a culture of reliability, having an exhaustive test suite, and employing controlled, gradual rollout. In this article, we’ll dive deeper into the concept of DevOps, specifically CI/CD, and how organizations can adopt CI/CD to improve their product velocity.
Isolation: Faults in The Standard Way of Deploying Software
Not too long ago, deploying software was seen as a major headache for teams. In the past, software teams were split into development-specific and operations-specific teams. Development teams will write code, and operations will take it to production. Sounds simple, right?
Well, not really. These two teams were what you call “siloed” teams. They obviously will have different goals and incentives. Development (and product) wants to release as much features as fast as possible. Operations wants to ensure services are healthy and downtimes avoided.
To solve this issue, a new term called DevOps is coined. While “DevOps” often refer to a role, it’s more of a philosophy that organizations adopt. DevOps aims to encourage greater collaboration between everyone involved in software delivery, with a goal of delivering software faster and in a reliable manner.
Engineering for Velocity With CI/CD
One of the practices of DevOps is CI/CD, an acronym for Continuous Integration/Continuous Delivery. “Continuous Integration” is a practice to regularly integrate code into a target repository. Each time a new feature or change is made, development teams should not wait to merge the new code into the main codebase. Merging frequently into the main branch reduces long-lived branches, and reduces the risk of merge conflicts.
How would you ensure that the new change don’t break existing features, and ensure they are functionally correct? By having a comprehensive suite of automated tests! Coming back to the Stripe example, they evaluate every code change with 1.4 million tests. Each one of these tests check for the program’s correctness, and is executed automatically.
Another fun fact, Stripe uses half a million CPU cores to execute their 6 billion test daily runs! For illustration, my MacBook Pro “only” have 10 cores, and yet it still handles every task I throw at it without fail.
The cousin of CI, “Continuous Delivery” (or CD), involves automating the process of software release after the code has passed the CI stage. The most common implementation of CD is usually a pipeline, which contains a set of tasks such as building, testing, configuring, and deploying the code to production.
The deployment of software, which historically refers to the operations guys manually building artifacts, packaging software, and deploying the release to production environments, can now be automated with lower risk, lower non-determinism, and with greater velocity.
With proper CI/CD and a great DevOps team, it’s no wonder that Stripe is able to launch hundreds of product improvements a day.
Conclusion
Delivering software used to be hard. Development and operations teams were always at odds with each other, and every release cycle is haunted by concerns of service downtime and painful rollbacks. DevOps emerged as a philosophy, in part, to solve the problem of software delivery.
A key practice of DevOps is CI/CD, which involves developers “continuously” integrating their code to the main repository, and “continuously” deploy their services safely through a pipeline of automated tasks such as build, test, and packaging. Operations teams ensure the implementation of CI/CD by maintaining and improving the pipeline By implementing CI/CD properly, software teams are able to deliver new features, fixes, and other improvements at a rapid rate, sometimes up to hundreds of times per day.
Let’s give the mic to you: do you know how often your organization deploys to production? Are you satisfied with the speed at which your team delivers software? If not, maybe proper CI/CD is something you’d like to implement. We can’t all be Stripe, but we can all be “elite” at software delivery through proper DevOps practices such as CI/CD.