DevOps is dead! Long live platform engineering!
Here we go again: another technology hype cycle hailing The New Big Thing, and how The Old Big Thing is dead. But as someone who still believes in DevOps (despite observing many sub-optimal initiatives) and as someone who really does believe in modern platform engineering, I’d like to pick apart the topic in a bit more detail and with a bit of history from my time in this space. All histories are imperfect, but some are useful.
A Brief History of Platform Engineering
Building digital platforms as a way to deliver software at scale is not a recent invention, and it predates the emergence of the DevOps movement in the late 2000s. Many large tech companies whose primary business was building software realized decades ago that they could enable developer teams to build, ship, and operate applications more quickly and with higher quality by standardizing infrastructure, building self-service interfaces, providing increasingly higher-level abstractions focused on solving developer problems, and dedicating a team to maintaining all of this as a platform.
However, these teams had to build and operate those platforms entirely from scratch, which required a pool of technically sophisticated (and well-compensated) engineers with operations skills, executive support, and organizational focus, and who were relatively unencumbered with legacy IT, at least compared to your average enterprise company. The tools we take for granted these days around infrastructure as code didn’t exist and, as an industry, we hadn’t yet experienced the transformative impact of public cloud platforms.
In the early 2000s, Amazon created a shared internal IT platform to handle what was described as “undifferentiated heavy-lifting” so developers could better focus on shipping value to customers. A couple of years later, this became available to users outside of Amazon, and rapidly grew from a web application platform to providing infrastructure as a service, transforming our entire industry. Around 2003 or 2004, Google built a dedicated SRE organization and began work on Borg, but the company didn’t make these initiatives known publicly until it published a whitepaper on Borg in 2015 and a book on SRE a year or two later.
Cross-pollination of employees across modern big tech companies meant that these ideas and approaches started to spread – because they worked.
The Start of DevOps
The inefficiencies in many large enterprise development organizations and the problems posed to development teams by increasingly complex infrastructure had been apparent for a while. Attempts to solve this included self-service access to “golden image” virtual machines, self-service software catalogs, and the first few forays into PaaS, often with decent levels of success for new greenfield applications, but much less so for legacy and commercial off-the-shelf applications. It was a significant challenge to enforce mandates across varied internal landscapes (or at least, more varied than what Big Tech companies had), particularly given the regulatory burden under which many of them operated, with decades of relatively heavyweight and manual processes in place.
In parallel with much of this, but in somewhat different environments, the whole DevOps movement began to coalesce in the late 2000s, with one of the significant early moments being John Allspaw and Paul Hammond’s 2009 VelocityConf talk “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr” in which they emphasized the importance of communication, collaboration, and alignment of incentives between operations and development. An entire community emerged to investigate and advance these ideas, and an explosion of interesting open source projects occurred.
Most of this work happened out in the open, and these ideas were adopted by a wide variety of organizations. DevOps was influenced by and borrowed from many prior movements and frameworks, including Agile, Lean Manufacturing, The Toyota Way, and concepts from psychology, cognitive science, organizational dynamics, and industrial safety. This is why I’ve always thought that DevOps is best defined as a loose collection of evolving practices and processes that take context into account.
Scaling DevOps is Difficult
As we’ve tracked in the Puppet State of DevOps Reports for a number of years, while many companies have been successful in implementing DevOps principles and practices, a significant proportion of enterprises have become “stuck in the middle,” with decent success at the individual team level, but not consistently across the whole organization.
In 2018, we first identified that DevOps success within the enterprise required significant standardization on the way to providing self-service as part of our five stages evolutionary model.
The Rise of Platform Engineering
Platform engineering is not new, but it wasn’t a particularly accessible concept if you hadn’t experienced it for yourself. In 2019, a few of the major analyst firms began to identify it as a trend, and Manuel Pais and Matthew Skelton published Team Topologies – an in-depth examination of the topic based upon their extensive experience doing IT consulting and observing patterns that worked in practice. They not only mapped out the organizational structure, but provided prescriptive advice on organizational dynamics – which, in my experience, is the major stumbling block for more traditional organizations.
Platform Engineering and DevOps are Aligned
Given how many large companies have struggled to experience the benefits of DevOps across their organizations, and that this more prescriptive movement of modern platform engineering is proving to deliver value quickly, some have argued that “DevOps is dead” and that modern platform engineering has supplanted it.
This just isn’t true, and we do ourselves a disservice as an industry if we perpetuate it. DevOps has always borrowed ideas from other movements, and platform engineering is just another one to add to the list for organizations of a certain scale and complexity. If you’re working in a small company with a handful of developers (some who are more inclined to the operational side than others), there may be no need for you to take the platform approach, and the principles around DevOps are still a great guiding function.
DevOps is about using automation as a tool to align incentives and increase collaboration across all of the teams involved in the software delivery lifecycle in order to deliver software better, more quickly, and with less stress. Modern platform engineering has taken the already existing platform approach and added an explicit focus on treating the platform as a product rather than as a project, as well as clear guidance for where teams should interact via collaboration, and where they should interact via self-service interfaces.
Like DevOps, platform engineering makes heavy use of automation, focuses on collaboration, requires empathy across organizational functions, and keeps people rather than technology front and center. It’s perfectly aligned with DevOps, and is proving to be a viable way for many enterprises to do DevOps at scale, in highly complex and varied environments.
DevOps isn’t dead. It’s just evolving.
Get the full scoop on how platform engineering enables DevOps at scale in Puppet’s forthcoming State of DevOps Report: Platform Engineering Edition. Sign up now to get the report.