Stop focusing on Agile vs Waterfall.

Like most working, functional humans, I sometimes read articles from other professionals in my field (writ large). Occasionally, I find experienced people, billed as experts, giving unsound or confusing advice that supports fundamental misunderstandings and misrepresents key concepts. Today, I’ve been inspired into action by a section in an article on TopTal titled 5 Common mistakes in Requirements Gathering. The author focuses for a time on biases in decision making, including the sunk cost fallacy. In the section titled “5. Believing Agile Is the Only Way” the author describes a situation where he focuses on Waterfall delivery vs. Agile behaviors in reaction to a crude request to “be agile” from a client. In this description, the author appears to commit the sunk-cost fallacy to prevent scope-creep.

I’ve shared the author’s resulting lesson learned below; honestly, I hesitated to quote it because it’s fundamentally flawed and appears to be a gross misunderstanding of Agility/Agile.

Lesson: Agile is one way to manage a product or service, but not the only way. At a certain point you need to finalize the requirements and move on to the next stage. How do you know when you’re done gathering requirements? It’s simple: when the requirements have been agreed upon with the customer—and no later. You can use Agile to develop your project, but you should employ a Waterfall-style delivery. Sometimes the best answer to the customer is, “Let’s talk about that on our next engagement,” or “We want you to realize value as soon as possible, so let’s not get distracted by new requirements right now.”

Michael Reeb - 5 Common mistakes in Requirements Gathering

I’d like to set the record straight for you on Agile vs. Waterfall and give you insight into a way to think about Agile that lets you focus on the goals and objectives rather than the terms. I’m going to assume you, the reader, are familiar with both Agile and Waterfall. And, frankly, that you probably have a strong opinion about them. So let’s clarify one thing up front.

Apples and oranges: Agility is an objective, Waterfall is a process.

I’m going to burst some bubbles very quickly here. Agile is a quality control philosophy. Like Lean, Six Sigma, Total Quality Management, and ISO 9000. At its core, it’s intended to help you deliver better quality software, more quickly, resulting in happier customers. Compare the 7 principles of ISO 9000, to the principles of Agile software described in the Agile Manifesto. You’ll find many similarities. There are no equivalent principles of Waterfall. Agility in software development is a natural consequence of those more formally described QC frameworks - whether businesses realize this or not.

“But… what about Scrum and Extreme Programming, and what not? What about sprints, etc?” you ask… These are processes that have been created with the intent of being agile. They are no more Agile than Waterfall without the underlying principles of quality that underpin agility. A schedule of regular activities, even a cyclical one does not make you agile, nor does a toolkit for decision making and design. Long cycle times don’t make you Waterfall either.

An agile process does not make you agile.

While working with BCG Digital Ventures (as an independent consultant), I’d heard a compelling introduction to Agility for members of our client team during kick off. One of the claimed differentiators shared in this introduction to agile was that the process of agile delivery isn’t inherently agile - and that while competitors might claim agility, they were not. If you deliver a big scope, predetermined solution, but simply chunk it into 2 week sprints - you are not agile, even though you are following an agile process! They termed this “Agilefall” - and while I’m sure it’s not unique or original, it’s where I’d heard the term the first time. In hindsight, I’ve participated in plenty of projects that were fundamentally Agilefall. Organizing around units of time, rather than value delivery inevitably incentivizes this behavior - and is a major risk for many sprint oriented teams.

Agilefall adds no value to your delivery (the scope was fixed!) but adds the overhead of agile project management on top of your existing work! Eek! That means you are less effective at delivering the same scope, than if you’d simply run it without pretending at agility.

Our TopTal Author has suggested exactly an Agilefall approach to projects when stating that you can “… use Agile to develop your project, but you should employ a Waterfall-style delivery.” This is a Bad Idea(tm).

Your Agile process is almost certainly actually Waterfall.

Perhaps you’re thinking: “What!?!? I follow all the principles of Agile! I follow a scrum process! I delay decisions until they need to be made! We are agile!”

The short answer is that any deliberate, goal oriented, data driven approach to development is almost always executed via Waterfall processes even when they are also executed via an Agile Process. “But how can that be?” you might ask. Well, I’m glad you did! The traveler (your idea) moves through a pipeline of discovery, decision, design, development, and delivery. Every. Single. Time. And as someone who executed Waterfall projects/products for a decade, I can assure you this is a waterfall process.

Now perhaps you are thinking: “But we have feedback loops and learn at each stage!” There’s nothing prohibitive of feedback loops in Waterfall projects. Nothing whatsoever. But I’m betting that once you’ve passed that decision stage - you usually end up at delivery. Feedback loops usually happen during discovery, are refined in design, and then happen after delivery. And guess what - this is exactly what a team in a waterfall project will do.

I hope you’re not too depressed. This is actually a good thing.

Most “Agile teams” are really just running many small Waterfall projects either in sequence or in parallel. And the more quickly they do so, the more quickly they learn from them, and discover new, tiny little baby waterfall projects to do. So cute!

Quality (broadly) is the goal of Agility.

A focus on agility is like wrapping a rubber band around the project / product. It’s saying that you’re going to apply a constant pressure to be more efficient, focus more on delivering value than on delivering a specification. And that pressure should encourage you to strip away the unnecessary, deliver sooner, and deliver more often. Sometimes the nature of your product, or the environment may resist that pressure. Plenty of regulatory regimes or high risk products (people’s lives!) can encourage longer initial cycle times. This doesn’t make those products or projects “Agile” or “Waterfall” just as you don’t compare “Six Sigma” to “Waterfall.”

At it’s most reductive, you might state that a Waterfall project is a single cycle with no feedback loop (though in practice, for me at least, we always got feedback and changes no matter how effectively we did discovery because the client LEARNED something from the project). So when you decide (in this sense) to execute in a waterfall mode vs. utilizing an “agile process” - assuming of course that you aren’t simply doing agilefall - you’re really simply saying that you don’t intend to learn anything for the duration of the project. And, that you care more about being on-time, on-budget, and on-scope, than about quality of outcomes. The author in the TopTal article has made this mistake.

Change is not your enemy.

An exclusive focus on quality against a predetermined scope suggests a fear of change. Perhaps it’s viewed as a hassle, pain, too much work, whatever. Our TopTal author used an example where the customer discovered new needs, and the experienced Project Manager recommended not to make changes, deliver the product as originally agreed, and then go re-scope for future revisions later. This example was intended to illustrate that “agile” is not always the way - or that Waterfall is good here. This obfuscates the collaborative goals between client and customer. The author’s sentiment to deliver value then iterate is spot on in the lesson learned, but the idea that scope must be adhered to first and foremost is plain wrong. It is true, in long cycle projects (whether waterfall or otherwise), changing your decisions often results in delays or reworks that can add some complexity to account for. If you’re billing on a project scope basis, this has to be closely managed, absolutely. So, ask yourself what’s better? The wrong product on-time, on-budget? Or the right product slightly late and more expensive? Lose your shirt on the first one, and swim in $$$ on the second.

Let’s step back to a quality lens here that is inclusive of outcomes. The client told the author that their product wasn’t meeting its needs. Effectively, they told the author that the product wasn’t high quality. Now, sure, no one realized what quality meant at the start of the project. They learned later on that a quality solution covered a different scope. The reaction of the author expresses a sentiment akin to: “We don’t care about overall quality… we care about delivering what we agreed to when we were all younger, possessed less wisdom, and were less aware of our needs.” To me, that’s not quality, that’s inflexibility and is an expression of the sunk cost fallacy / escalating commitment bias. And of course, there’s the not so subtle irony of this requirement gathering miss (normal actual thing to happen) being shared as a positive example in an article about not making mistakes in requirements gathering.

Mind you, I’m not telling you that you should cater to every request and change at any given time - I’m simply pointing out that total quality is measured in outcomes… not the letter of an original contract/scope/agreement that can be easily amended.

So what now?

I encourage you to get this Agile vs. Waterfall false dichotomy out of your head. Focus on quality over process, or inverted, find a process that best supports total quality for the work you’re doing. That won’t always be the same process - nor will it be the same for every team. When the idea of being agile comes up as a thing to do (especially in comparison to waterfall) - remember than what you’re really discussing is how to deliver quality. Agility always helps deliver quality, regardless of your execution process. Once you figure out what quality means for you (and probably your customer/user/client), it’ll be much easier to sort out a delivery process that improves your chances of quality outcomes. Perhaps that’s a ton of upfront research, and a pretty static execution phase. Perhaps that’s a continuous deployment environment, with daily cycle times. Focus on quality and the rest sorts itself out.