With many, many thanks to my former colleague and AppSec guru Jacob Salassi for some excellent education and ideas (through this podcast on YouTube, e.g.) and to Clint Gibler’s amazing curation of the AppSec world in tl;dr sec, here’s how I’m modeling roles and responsibilities between “Dev” teams and “AppSec” teams for product organizations.
To me, the conclusion is trivial: the buck stops with the Dev team. If you don’t start here, all is lost.
Taxonomy
A light-hearted but reasonably MECE classification of a typical tech product organization’s structure would be “what to build”, “how to build”, and “how to sell” teams. “How to build” teams are tautologically the foundation for product organizations.
A central perspective here is that every team other than “how to build” should try to make building more effective. Not because of something intrinsic to the role of builders or the people who are the builders but simply because the purpose of the organization is to build products.
Zeroing on the “how to build” box, this model fractally repeats — i.e., the “how to build” box itself has, as its core, developers. All else is about enabling those developers to build stuff that makes the business money. You could model it as something like this:
That sets the stage. And if you don’t already see in your imagination the glimmer of a box that says “AppSec” on it, sort of equivalent to Design and Test and so on, well, here you go:
Everything feeds into security and security feeds into everything. AppSec can generate requirements, influence product priorities, provide feedback on the finished product (and manage external communications, but that’s slightly outside the scope of this discussion) but Dev is responsible and AppSec is helping Dev meet that responsibility as effectively and efficiently as possible.
Structure Basics
First we’ll define a couple of terms, and then use language from the excellent and insightful Team Topologies (Skelton and Pais, September 2019, IT Revolution) to set up team structures along a maturity axis.
We’ll define the AppSec team as the team that concerns itself exclusively with information security-related concerns and the Dev team as essentially everyone else in the organization. (If you prefer, you can think of the dev team as just the developers. The conclusions will remain the same.)
Team Topologies defines four fundamental team types in any software engineering organization:
Stream-aligned — teams focused on a particular value stream, which may be as broad as a whole product or as narrow as a particular user persona
Platform — teams focused on building things for other teams to use
Complicated subsystem — teams with specialized knowledge in areas of the product with sufficiently complicated logic, tools, or construction
Enabling — teams that help other teams with their expertise (rather than code)
An organization can function with purely stream-aligned teams and most organizations do start this way. As they grow, platform and complicated subsystem teams evolve to manage the cognitive load of knowing “the whole product”. Finally enabling teams emerge to solve the problem of knowledge diffusion around horizontal needs, i.e., needs shared across multiple other types of teams.
A Simplified AppSec Maturity Model
With the organizational model and terminology established, we can now define the maturity model itself. In increasing order of maturity from bottom to top:
LEAST MATURE: security is owned by stream-aligned teams even if they lack knowledge — Initially, there’s not much of a choice. No one is going to pay for a highly secure product that… does nothing useful. So you do your best, take your chances, and build.
PRACTICAL: enabling teams of generalists emerge out of stream-aligned teams — over time, a product that survives beyond product-market fit needs to show that it is secure and the organization needs to show that it can manage and mitigate the risks of being attacked. The organization recognizes that it needs specialized knowledge, and devs with such knowledge or interest emerge to build and share that knowledge.
SUFFICIENT: enabling teams of specialists are created — with business success raising the cost of a security incident, security expertise is brought on to free developers from the cognitive load of dealing with security, itself a highly specialized domain. Now devs can focus on bearing the cognitive load of understanding customer needs and building the business logic powering the product. Many organizations simply stop here. And here lies the missed opportunity to vault to #5 (with an intentionally short step on #4).
A SLIPPERY STEP: platform teams create security platforms and the organization mandates their use — nothing, not even security, is immune to the one foundational truth in computer science: there is no problem that cannot be solved with another layer of abstraction (except the problem of too many layers of abstraction, of course). So a security platform team emerges, or is created, to build tools, libraries, platforms, services, and such to manage security through enforcement. ⚠️ This is a dangerous place. Use it a way station if you must, but just stay on #3 if you can’t get to #5! ⚠️
MOST MATURE: platform solutions are presented as paved roads — as the organization grows more enlightened and meets two criteria, namely (a) it trusts that devs understand and value security and (b) it adopts a product-centric view of the internal platform, platform solutions are repackaged as paved roads to solve the behavioral and social problems associated with mandates. More on this at the end.
Sidebar: compliance
What about compliance? That’s a topic for another day! In a nutshell, compliance requires a program rather than product view and is best structured almost from day one as an effort parallel to (but of course dependent on) product development. Compliance is a goal almost certainly unachievable with anything other than a dedicated specialist or team of specialists.
Structure
Adding it all up, here’s what an org at the highest maturity level might look like. Note also that, at scale, organizations may need multiple teams of each type.
Roles and Responsibilities
Maturity models and block diagrams of organizations are all well and good, but who ends up doing what? Here are some specific use cases that help explain the roles and responsibilities in an organization has climbed its way to the higher maturity levels. Remember that Dev is a proxy for everything that is not AppSec — product, e.g.
What are our security risks? Posture assessment: AppSec Investment decision: Dev Does vulnerability X need to be addressed right away? Technical assessment and recommendation: AppSec Compliance impact assessment and recommendation: Compliance Decision: Dev Implementation: Dev What's the best way to protect against risk X? Assessment of options: AppSec Definition of "best", i.e, establishing tradeoffs between the options: Dev Implementation support, e.g., build secure libraries: AppSec Platform Implementation: Dev Should we pursue compliance regime X? Compliance requirements definition: Compliance Investment decision: Dev
The Paved Road
Paved road approaches have recently been adopted with success. The philosophy is not dissimilar to that of DevOps and SRE, where the engineering or service teams are ultimately accountable even for security outcomes, but security teams define or build processes, practices, libraries, or even entire systems which make it easier for the engineering and service teams to achieve their targeted security outcomes. As an example, suppose a team needed to implement user account management. They could be mandated to use certain software that has been audited for security, a mandate that the product team then unthinkingly follows (and probably also whines about). The paved road approach, on the other hand, could be an audited and fully working user account management library or service that also just so happens to meet all the internal requirements not just for security but also, say, for observability. If this offering is just plain easier to adopt, why wouldn’t teams adopt it? Yes, you could off-road for fun but if your objective is to get to your destination quickly, wouldn’t you always choose the paved road?
As the authors of Nudge (Thaler and Sunnstein, September 2021, Yale University Press) say repeatedly: if you want something to be adopted, you have to foremost make it easy.
Paved roads are a great end goal, but the path to adopting this is strewn with every known version of You Are Not Netflix problems. Pursue with care!
The biggest implementation pitfall by far (based on my totally and absolutely very scientific research technique called “check out some good blogs”) may be described as the oxymoronic “mandate the paved road”. Its a special case of the “mandated platform” problem where internal platforms are built without their customers in mind (sidebar: do your internal platforms have product managers?) and are adopted only because of someone’s say-so. Beyond the psychological reasons that make the implementation of any mandate challenging, it creates the impression of transferring responsibility for security from the Dev team to the AppSec team, thus fighting against the fundamental value of Dev ownership.
So let your software go forth in the world! May your vulnerabilities be patched, your keys be private, and your encryption be quantum resistant.