An Overview of DevOps, CI/CD, and DevSecOps
Unless you’ve been living in a cave for the last few years, you have likely heard of DevOps, maybe even the Three Ways of DevOps (systems thinking, amplifying feedback loops, and maintaining a culture of continual experimentation and learning – as shown below), and possibly even DevSecOps (perhaps only in passing). It’s a lot of trendy, buzzword-y terminology that has many organizations both excited and maybe a little confused about where all of this is leading.
Don’t worry. If this concern describes your organization, you are not alone, and you are right to question what is going on. This article is meant to provide some background and context to the cloud (no pun intended) of language and ideas that surround these, and other, concepts as you navigate a brave, new world within IT. We will also provide some simple analogies, stories, and practical recommendations throughout to help make these concepts real.
An Overview of DevOps and CI/CD
DevOps is no longer a new term. Lots of organizations are (or at least believe they are) practicing DevOps in some form. At its core, DevOps is a set of practices designed to bring development and operations teams closer together. The plan has always been to shift more of the development pipeline into the realm of operations. This effort includes a great deal of automation. You may be familiar with the term “continuous integration and continuous deployment” or CI/CD. This is just a fancy way of saying that new code is produced in real-time, and its deployment and release to production environments is fully automated (including testing).
The Promise and Pitfalls of DevSecOps
The concept of DevSecOps extends that lofty goal to include automated security gates for all code deployment, meant to ensure that newly deployed code complies with operational security requirements and processes as it is guided through the development pipeline. In theory, this sounds really great. Not only would such a system provide streamlined (much faster) deployment of code, it would ensure that all code deployed to production is secure, based on organizational policy and governance requirements.
In practice, success has been elusive, at best, for most organizations. As reality sets in across the industry, it is a useful exercise to explain the theory and the reality of DevSecOps. First and foremost, the promise of DevSecOps is to eliminate the dreaded “final gate” of information security, which has traditionally posed an enormous roadblock to increased development velocity. If the goal of operations has been to maintain the stability of systems (which is often in direct conflict with the goal of development teams), then the goal of security is really boiled down to the need to ensure that every system maintains a level of confidentiality, integrity, and availability that, quite frankly, makes operations seem like a cuddly puppy by comparison. Information security has always existed to say, “no”. There is no sugar-coating this fact. Policies are written in a negative voice. Risk assessments and vulnerability scans are conducted to find violations. And at the technical level, rules are written to deny access. Add to this the very real experience of most organizations where information security is a black hole from which nothing escapes—especially not helpful information. How can this possibly shift into a collaborative part of the development pipeline?
The Three Ways as They Apply to DevSecOps
If you haven’t read it, I recommend picking up a copy of Gene Kim’s treatise on DevOps, “The Phoenix Project”. It’s a fictional account of an organization discovering and embracing a DevOps mindset through a lot of trial and error (and failure). As a lifelong operations snob, this concept has been difficult for me to embrace; however, when a true DevOps mindset is embraced by an organization, the results are striking. I want to focus on a foundational concept of DevOps, known as the “Three Ways”, mostly as it relates to DevSecOps. This concept was originally defined and embraced as part of the DevOps movement, but its core principles extend very well into the expanded universe of information security.
The Three Ways of DevOps
DevSecOps and the First Way: Systems Thinking or Flow
The first of the Three Ways is termed “systems thinking” or “flow”. Simply stated, work always flows in one direction: downstream. It also means that no known defects should ever be passed downstream. Finally, we should always seek to increase the flow. If we think of this concept in non-technical terms, we can quickly see its value. Take dishes piling up in our sinks (not out of place as so many of us are working, and thus eating, at home these days). You can allow the sink to fill with dishes until you can’t fit anything more, and then choose to wash everything at once. This follows what we might consider a “waterfall” approach to dish washing. Get it all done at the end in one “big bang”. But what if your favorite knife is buried under 27 plates, bowls and glasses? It’s a serious pain to get to. Consider the annoyance of running out of dishes. Not to mention the overall frustration of seeing dishes pile up day after day.
The approach of “flow” says that we should wash the dishes as they enter the sink. The value stream is designed, then, in a way that we are continuously making our most frequently used (and likely most valuable) dishes available for use. If there are multiple people in the house, capable of washing dishes, then it follows that everyone should take responsibility to keep the sink in good order. This concept speaks to two very important pieces of the “flow” puzzle: 1) relationships and 2) mutual accountability. Relationships are such a key component of this concept. If we go back to the dishwashing example in a household of two or more people, we have to trust that each person will clean up after themselves, but also that the level of communication and collaboration is such that it ensures all parties are comfortable handling the dishes in circumstances where others are not available to do so. This practice leads directly into the second idea of mutual accountability.
We cannot force others to take care of their own dishes (let alone our dishes in cases where we cannot take care of it ourselves) as it creates a culture of suspicion and animosity. Instead, we must foster a collaborative environment where people embrace the value in keeping the sink clear of dishes. The same goes for modern development pipelines. Developers must take accountability for that which they create. If it is insecure, the result will be a vulnerable application. And the goal is to eliminate vulnerabilities in the pipeline, before anything gets to production. Of course, we cannot expect developers to be experts at security any more than we can expect security analysts and engineers to be expert coders. So, we have to put security gates in the pipeline itself, thus helping developers avoid costly vulnerabilities while allowing them the latitude to develop useful applications. This is the core of DevSecOps: incorporate security into the flow.
DevSecOps and the Second Way: Amplify Feedback Loops
If flow describes the development work getting from an idea to a customer (or from Dev to Ops), then feedback loops are the polar opposite. Feedback has always been a key ingredient in long-term customer relevance and improvement of products and services, but as DevOps matures, this idea elevates through speed and amplification. The Second Way is defined thus: Create, shorten, and amplify feedback loops. The key feature of adding security gates to the development pipeline is that when flaws or vulnerabilities are discovered as developers are writing code, fixes can be established in the moment, long before the deployment stage. This practice not only works to ensure more stable code, it also allows developers to keep moving. And let’s face it, the business likes development to continue to advance products and services.
Rather than run a slew of security tests at the conclusion of development (and probably QA tests), which serve primarily to stop the train before it has a chance to get up to full speed, we can ensure that the train is capable of running at full speed without self-destructing, which brings us back to the example of doing the dishes. If everyone waits around until the sink is completely full and no clean dishes are available for use—to then determine whether all the necessary equipment to actually wash the dishes is available—only to find that the house is completely out of dish soap—we have created a disastrous situation. If, however, someone checked on the dish soap (security in this case) when only a few dishes had been added to the sink, we could simply go buy some dish soap. The time spent solving the issue would likely not detract from our use of other clean dishes in the house, so no productivity is lost, and the result is that we have identified a security issue early in the lifecycle and corrected it. Thus, we are able to move forward with a more complete solution without large swaths of unproductive time spent waiting for the issue to be resolved.
DevSecOps and the Third Way: Culture of Continual Experimentation and Learning
The Third Way is more about culture than doing something specific. In order for all of the practices to work, it is imperative that an organization embrace the idea of learning from failure, rather than punishing everyone involved. Failure is part of the learning process, particularly when we’re attempting something new. No one will perform perfectly, and if we do not encourage our developers to take risks and experiment, we will never improve. This is one of the most difficult concepts of DevOps for those in a governance role (like security) to embrace. But without taking creative risks, we will quickly find ourselves stuck in a rut.
Bear with me as we return to the dish washing analogy. When preparing to actually wash dishes, some people like to fill up their sink with water and soap. Others prefer using a dish brush that stores soap in the handle. Still others may find that applying soap to the brush head itself or some other type of sponge is the way to go. In this case, we could try all three, measure the efficiency of (and problems with) each approach, and the most expedient solution would be adopted by everyone. This is the value of continual experimentation. Consider the person who came up with the idea of an automated machine that could wash numerous dishes without any human intervention. And then think of how far the dishwasher has come in the past hundred years. There’s a pretty good chance that there were many naysayers in that development lifecycle. Thankfully, appliance manufacturers were able to see the value in the advancement, and now we don’t really have to wash dishes in the sink anymore.
The same must be practiced by security teams. None of the gates and guardrails placed in the development pipeline should squash innovation. If one development team is most efficient using Docker, then build security gates that work with that platform. And allow for the reality that many new ideas will fail, sometimes spectacularly. It is critical that we not take developers to task simply because they tried something new that set off a thousand alarms in the security flow. It is far more productive and valuable to learn from the failure and adjust going forward. That’s the value in allowing people the space to take risks, fail, and learn in order to improve. And that culture is the best path forward.
The Journey to DevSecOps
There are a great many challenges to embracing a DevSecOps mindset. It’s not an easy path to take without buy-in at all levels and agreement with those working in development, operations, and security. Keys to success include relationship building—mostly between teams that have historically not gotten along well —and it demands a culture shift from blame and compliance to one of acceptance and encouragement. No small feat, and it takes time to get it right. If DevOps is already a part of your culture, keep in mind that expanding to include security is not just a matter of cramming three letters into the middle of the DevOps title.
To be successful with DevSecOps, information security team(s) must be involved in the conversation and agree that such a culture shift will be valuable to the organization. A DevSecOps approach will demand more of their team’s time, ongoing focus, and involvement throughout the life of DevOps projects. It will require them to work differently and be held accountable to regularly “do the dishes”. It will be an adjustment for everyone, particularly at the start, filled with failures, misunderstandings, arguments, and (hopefully) productive conversations, cooperation, and learning along the way. In the end, it’s about striving for the “greater good” – creating products and services that not only delight our customers, but also free them from the frustration and worry that comes from security concerns.