Today, you’ll be hard pressed to find product managers extolling the virtues of waterfall — a top-down software development framework organized around requirements documents and carried out by largely siloed teams of designers, software engineers, marketing teams and salespeople in gated, sequential phases.
In the late 1990s, however, attitudes were quite different.
“I remember it really clearly,” said Laura Ford, vice president of product and UX at the Seattle software company Navigating Cancer, who was at the University of Colorado at the time. “We learned waterfall as the software development lifecycle methodology. And then you get into the real world. And, you’re like, ‘That was a terrible idea.’”
Agile Vs. Waterfall
- Agile is a flexible software development approach focused on short cycles of product iteration informed by customer feedback.
- Waterfall is a top-down, structured software development methodology that proceeds through sequential phases: planning, design, development, testing, review, launch and maintenance.
But if it were a terrible idea, it was also a fashionable one with a long history in manufacturing. And it was used by companies like Amazon where Ford worked as a technical product and program manager for Amazon Web Services after graduating. It wasn’t until 2004, shortly after Allan Atlas, the first development manager for Amazon’s internet-connected storage service S3, came aboard, that practices began to shift.
Atlas trained employees in scrum: a popular agile framework in which a product owner defines outcomes the team is responsible for achieving — say, growing a customer business by 25 percent — and a scrum facilitator keeps the team focused and on track. While the process starts with a goal in mind, it is designed to accommodate change and encourage innovation. Teams are given wide latitude in how they organize themselves and achieve consensus-based outcomes, releasing software in one- or two-week cycles that allow them to make incremental changes to releases based on customer feedback.
The framework took hold quickly at Amazon, which has a focus on efficiency that aligns well with agile’s core principles.
“Amazon’s not going to say, ‘Your specs must be written this way; you must use this particular process,’” Ford said. “They’re going to say, ‘Build the best thing for the customer and do it as efficiently as possible.’”
Agile and Waterfall, Defined
What Is Agile Software Development?
Agile is a flexible software development approach focused on short cycles of product iteration informed by customer feedback. At each stage of development, stakeholder involvement helps guide decisions key to prioritizing, building and refining working software. Agile teams tend to function through rituals — daily stand-ups, retrospectives, demos — within short development cycles, or “sprints,” that typically last one or two weeks. While the process starts with a goal in mind, it is designed to accommodate change and encourage innovation. Teams across business departments work in close collaboration and are given wide latitude in how they organize themselves and achieve agreed-upon outcomes. Often, the goal is to produce shippable code by the end of a sprint.
What Is Waterfall Software Development?
Waterfall is a software development methodology that proceeds through sequential phases: planning, design, development, testing, review, launch and maintenance. It is typically “gated,” meaning each phase needs to be fully complete and approved before the next one can begin. Product requirements documents created early in the development cycle stipulate guidelines for deliverables and help define a project timeline and estimated cost. Since each phase is completed in sequence, teams tend to be compartmentalized and accountable for meeting their discrete outcomes. Customer feedback occurs only intermittently and is less central to development.
What’s Better: Agile or Waterfall? It’s Complicated.
The agile movement, the founding manifesto of which was written by a team of 17 software leaders in Snowbird, Utah in 2001, emphasizes “individuals and interactions above processes and tools; working software over comprehensive documentation; customer collaboration over contract negotiation; and responding to change over following a plan.”
Many companies have embraced agile because it is highly adaptive, said Megan Cook, who is head of product for Jira software teams at Atlassian and an agile coach. It focuses not just on solving problems, but on finding the right problems to solve — for instance, the first step in an agile process to create a website would ask whether a website is the best way to engage your customers, or whether they might be better served by a mobile app or something else entirely.
For proponents like Roni Ben Aharon, who is co-founder and chief product officer at Craft.io — a Tel Aviv-based product management software provider — the framework’s speed and responsiveness to shifting customer preferences and market conditions makes it unequivocally preferable to waterfall.
“When you do it right, you quickly see the benefits. Your deliverables can be every one or two weeks — at bigger companies, they can even be daily, or every few minutes or even seconds,” Ben Aharon said.
Still, agile can be difficult to apply in highly regulated industries such as healthcare, financial services and aerospace, where fixed budgets, strict safety protocols and rigorous rules for documentation and governance may demand a more rigid framework with stage-by-stage gates to ensure regulatory compliance and quality control.
“When you do it right, you quickly see the benefits.”
As Adam Dahlgren, senior vice president of product at Allstacks, put it: You have little room for error when building a pacemaker or a plane engine. In those cases, and even for education and gaming companies bound by strict release deadlines, a front-loaded waterfall approach, moving sequentially through standardized steps — product requirements, design, coding, verification and maintenance — is often the best way forward.
While debate flares over which approach is better, the reality may be that the answer depends on the level of uncertainty associated with an initiative or product. Morten Korsaa, a partner at Denmark-based management consultancy Whitebox, who has conducted performance interviews with leaders of more than 700 development projects, wrote in an email that the “religious war” between agile and waterfall is largely misplaced.
According to Korsaa, both frameworks are utopian: Agile is premised on the quixotic idea that teams are completely competent, have continuous access to stakeholders and don’t operate on fixed budgets or deadlines. Waterfall is rooted in the equally fanciful notion that customers know exactly what they want and can specify their preferences clearly from the start.
In practice, neither of these scenarios really hold water. Most software companies, he contends, operate somewhere in the middle, leaning toward agile if a project has more unknown variables and waterfall if there is more visibility surrounding the end result.
Here’s what you need to consider to find the right path for your team.
Benefits of Agile
Agile Is Effective at Removing Inefficiencies
Do you have a high degree of confidence your design can be developed as planned? If so, waterfall can work well.
“A lot of people say, ‘Waterfall is slow, agile is fast,’” Dahlgren said. “Actually, that’s really not true. If you’re really positive that you have [the right plan] and that it’s very unlikely to change, then you could use waterfall and create it pretty fast, actually.”
However, if you have less assurance in the outcome, waterfall can be risky. Since the framework requires sign-off at each stage of a project, Cook told me, it can lead to bottlenecks when things go wrong — for example, if a developer discovers during the coding phase that a proposed design can’t be feasibly engineered. When that happens, requirements need to be retooled. Then you’re essentially back to square one, having to repeat the development cycle, stage by stage.
“Agile makes it a lot easier to reevaluate that [feedback] and say, ‘We need to reconsider what we had planned and shift course more quickly.’”
Further, the time lag between when the code was originally written and verification testing, often weeks or months later, means developers have to go back and refresh themselves on dated code — a major time suck when it comes to flushing out bugs.
Agile’s short sprint cycles theoretically minimize those risks. Consider a health tracker app developed by the software company Navigating Cancer to help clinicians monitor patients’ medication adherence. In an early iteration of the app, patients were sent surveys inquiring as to their health and any side effects of prescribed medication. Results were sent to providers to guide diagnostic interventions. But the system provided more details than anyone would ever want. Care teams were getting bombarded with feedback, including complaints of fatigue that are common among cancer patients and generally not cause for intervention. Clinicians wanted more relevant data, not just more data in general.
“Agile makes it a lot easier to reevaluate that [feedback] and say, ‘We need to reconsider what we had planned and shift course more quickly,’” Ford said.
That’s what the design team at Navigating Cancer did, working in close collaboration with healthcare providers to improve the app’s functionality. In a later iteration, a check box let patients opt out from receiving clinical calls in response to survey data. According to Ford, the update improved the relevancy of patient data and reduced clinical call volume by 30 percent.
Agile Adapts to Market Shifts and Customer Demands
Paying attention to the wrong market signals can be the death knell of a software company. By delivering a minimum viable product — or “the smallest chunk of value that can be improved over time,” Dahlgren said — you can quickly understand whether your product appeals to users and is performing well against business metrics like activation and acquisition rates.
“Failed software delivery doesn’t just mean the thing didn’t work,” he said. “It might mean the market shifted, or there was something lost in translation about the customer’s need, and [the product] wasn’t validated the way it needed to be. We just delivered X and they wanted Y.”
Rapid iteration has obvious benefits for a consumer app or B2B SaaS product. When a product is falling flat, agile practices like sprint planning and user story mapping can help a team quickly reset expectations and align around new solutions. Change the form factor or streamline the onboarding flow and see how customers respond.
But even in legacy industries, companies are increasingly called on to respond to rapid shifts in consumer technology preferences and shorter production cycles, both of which benefit from an agile way of working, Cook said.
Take, for example, the recent rise of mobile banking apps. A financial firm invested in a waterfall approach might spend years refining the information architecture of their website, she said, only to find, upon the site’s release, that customers in 2022 prefer to do their online banking inside an app. An agile framework, in which the eyes of business analysts are embedded at each stage of the development process and outcomes are not pre-ordained but malleable, would likely prevent such misalignments.
“The big advantage behind agile is that you can release and test ideas very, very quickly with real-world customers,” Cook said. “And that allows you to be super nimble and pivot if you need to.”
Agile Is Effective When Customers Are invited Into the Process
Inside an agile framework, the software that first appears before customers doesn’t have to be flawless. In fact, modest imperfections, so long as they’re not stymying or offensive, can invite customers into the design process and help them feel invested in a product.
A recently released Jira feature, which helps programmers protect the integrity of a company’s codebase, is telling. Leveraging an integration with Git version control software, which tracks historical changes to a codebase, the Jira feature lets individuals work independently in task-based “branches.” Isolating these branches from the source code ensures bugs created by individuals don’t corrupt the main branch, disrupt frameworks or libraries, or otherwise slow down the team’s progress.
“If we used a waterfall approach and built the feature solely on the requirements, I think people would have just turned it off.”
The feature’s first iteration, however, suffered from excessive automation. Whenever a flagged issue was deemed “in progress,” the Jira feature automatically generated a new branch. This improved transparency, directly tying issues to their branch names in the source code. However, in some cases, it led to branches sprouting like weeds: a developer who might ordinarily knock out multiple tickets within a single branch suddenly had separate branches for the tiniest bugs and typos.
“By talking to customers and demoing this, we realized that was really annoying,” Cook said. “It wasn’t appreciated by the developers.”
In a later release, Jira’s product team created a link developers could click once they were ready to create an independent branch. At the time, branching was still a relatively young idea, and the appearance of the link provoked interest in the branch-per-task workflow, as an alternative to feature or release branching. It also nudged teams to conduct code reviews before code created in a branch could be merged into the mainline. As a result, code quality “skyrocketed,” she said.
“If we used a waterfall approach and built the feature solely on the requirements, I think people would have just turned it off, or they would have complained until we turned it off,” Cook said. “We would have missed that chance to help people write better quality code, which would have been a huge shame.”
The Benefits of Waterfall
Waterfall Is Effective When Customers and Stakeholders Have Limited Availability
Agile presumes that user and stakeholder feedback is readily available. In software, this is often a fair assumption, but not always — and in those cases, waterfall might be a better option. The sequential phases of a waterfall project, including staff assignments and schedules, are tightly scripted in product requirements documents. As a result, waterfall allows for intermittent participation in a project by stakeholders, as opposed to an all-hands-on-deck approach.
“With waterfall, you can have a more predictable engagement model and say, ‘OK, this is going to be our phase for definition. And we’re going to have a sign off and review, and this is what your involvement will look like,’” Ford said.
Overhauling a website to indicate a company’s legal name change or to reflect the merger of two companies consolidating their online portfolios are cases in which a waterfall approach is well suited, Cook said.
“There’s a really well-defined scope that’s not changing, the design isn’t changing,” Cook said. “And so you can easily define all those requirements up front, and then hand it over to the [engineering team] to go through these changes.”
Waterfall Is Effective in Low-Trust Environments
Even within an agile framework, customer involvement does not need to be as extensive as a meeting or interview. Product roadmapping and prioritization tools like Aha!, Trello, Jira, ProductPlan, Airfocus and Craft.io help product managers and owners make reliable predictions about user preferences and behavior. These tools often have data portals that aggregate survey and engagement data coming in externally from customers, and internal reports from sales, customer success and support teams.
In some cases, though, the key stakeholders for a product are the members of a company’s executive team or board, who need to be convinced of agile’s validity. While the model’s flexibility helps teams adapt to unforeseen shifts in customer or market preferences, it can also result in squishy deadlines and a cultural and operational resistance to being “done, done,” Cook said.
“A lot of folks out there, business leaders, CEOs who signed up for an agile transformation, will admit to you, ‘Those first few years, I just couldn’t get my organization to commit to a date because they were told to never commit to a date,’” Dahlgren added. “For the sake of the business, you have to be able to accommodate some commitments. There’s sort of a delivery repeatability factor that’s a calling card of highly effective organizations and allows them to call their shots a little bit, and not be really off.”
“Something about agile is people have to accept that we’re going to start building and we don’t know exactly where we’re going to end up ... And if you have a low-trust environment, it’s really, really hard.”
In most cases, it’s OK to miss deadlines on occasion. But teams should get better at making estimates over time.
“What is the spread from when you commit to something to when it actually ships?” he continued. “And then, on average, what’s the delta, in days, between when something landed and when you’d promised it?”
For teams transitioning to an agile framework, analysis of shipping trends using intelligent data software can help earn the trust of wary executives. Allstacks, for instance, predicts the production cycles of key features — a business objective executives typically find valuable.
Ultimately, though, if there is a lack of institutional trust in an agile approach, it’s not likely to take hold.
“Something about agile is people have to accept that we’re going to start building and we don’t know exactly where we’re going to end up,” Ford said. “And if you have a low-trust environment, it’s really, really hard.”
Waterfall Is Effective in Highly Regulated Industries, and for Projects With Inflexible Budgets and Deadlines
Trust is often a function of the goods or services a company is trading. Consider, for instance, a publicly traded financial services firm that maintains online portals for business and consumer transactions. A clearly defined release timetable, forecast perhaps six months to several years into the future, might be necessary to avoid upsetting customers whose money is at stake.
“If you’re just streaming out updates that are changing the way people are moving money around or reconciling transactions, that’s not great,” Dahlgren said. “You can’t deploy to production and actually roll out X feature functionality on an intraday basis. Think about the overhead for the customer and partnership teams just to keep those teams apprised of what’s changing. Those end users need real stability as to how things work.”
Medical devices, education software: same story. Even video games are often created in a waterfall framework.
“You can be attempting to build in an agile way, but you have a date out there,” Dahlgren said. “And you’re probably going to mostly waterfall toward that thing — in reality, in the way people really work.”
‘Agilefall’ Is Almost Never Effective
Mixing methodologies can get messy. Shane Drumm, a digital consultant based in Perth, Australia, said leaders at many companies want to be agile but aren’t willing to take the risks to get there. The result is a “fragile waterfall” approach in which business analysts stand in as product owners.
“The product owners are expected to create detailed user stories and sub-task to a degree where everything can be estimated by the engineering team,” he wrote. “This is just a rehashed version of a requirement specification in disguise as a user story.”
Agile operates in many different sub-frameworks, such as the Scaled Agile Framework (SAFe), Extreme Programming, Kanban and Shape Up. In practice, some of these are really hybrid agile-waterfall models that account for the continuity of legacy systems and the demands placed on large engineering teams, who must coordinate staggered rounds of integration testing, security testing and performance testing. But when these models are tethered to release dates, or scrum masters become overly pedantic in policing requirements created by product owners, Drumm said, focusing on outputs or “sprint points” rather than solving users’ problems, these hybrid approaches can backfire.
“This simply creates ‘feature factories’ focused on building features rather than solving problems for customers,” he said.
How to Make Agile or Waterfall Successful
So you’ve made a decision to go with either agile or waterfall. What do you do next?
Keys to Success in Agile
Start small. One of agile’s potential pitfalls is that companies can rush into it, trying to transform an entire organization overnight. Staff responsibilities change too quickly. The management structure becomes flatter and less hierarchical, but it comes as a culture shock, with people unsure of their roles.
“I always recommend starting with a small team first,” Cook said. “Take a pilot team on a non-critical project and run that in an agile way.”
This helps answer key questions before moving forward at an organizational scale: Does my funding model need to change? Am I happy with the team? What sort of reporting do I need from them to have visibility into their work and how it fits in with the whole company?
Hold off on prescribing a ‘best’ solution. Assessing market needs and a problem to be solved is a good place to start in agile development. At that point, “it would be easy to gather a number of requirements and release an MVP,” Drumm said, but “instead, the team should accept that they might not know all the requirements” and start experimenting. Experience maps can help further define the customer context, and tree diagrams can identify multiple paths to a desired outcome. These potential solutions can then be tested.
Make sure everyone owns the outcome. Ben Aharon said a common misstep among executives is presuming an agile framework applies solely to research and development teams, not product managers, UX designers and content writers. The 2021 book, Ask Your Developer: How to Harness the Power of Software Developers and Win the 21st Century by Twilio co-founder and CEO Jeff Lawson is adamant on this point, Ford said. When the product team alone is responsible for defining the product requirements and the engineering team is responsible for executing it, you are unlikely to achieve the best outcome. “You need to use the full brain power and capabilities of your team and, as a group, take shared accountability for what you deliver,” she said.
Strive for increased velocity. The health of an agile team is often measured by its velocity. During sprint planning, at the start of a new product iteration, team members sign up for a certain number of group-defined “points,” then, as they complete various tasks, burn them down. Accountability is often measured by the burn rate. Managing this rate through tools like Jira, so there is a steady decline from the beginning to the end of a sprint cycle, helps ensure the team is not taking on too much work, or the wrong type of work, and that key tasks like QA testing are not getting shortchanged.
Keys to Success in Waterfall
Use prototyping to define clear outcomes. Design problems encountered during coding are often difficult to respond to quickly, or at all, in waterfall, because roles and expectations have been so strictly defined up front. “So a way to set waterfall up for success is making sure the outcome you’re working toward is as real as possible during the definition stage,” Ford said. “Use something like prototyping, or proofs of concept, to do as much validation as you can during definition.” That way, “you can have more confidence when you get to implementation and delivery that you’re going to end up with the desired outcome.”
Define key milestones and provide development “off ramps.” “The challenge with waterfall is the inflexibility,” Ford said. “So if you have some checkpoints built in during implementation, where you say, ‘We’re going to look at this thing at a particular time and decide if we’re going to continue on one course or take an alternative,’ and know what each of those options look like, you can assess where revisions to the plan might be needed.”
Involve the full team at early stages. In a waterfall approach, participants are often assigned to select stages. Planning might involve members of the customer success, product and design teams, whereas software engineers might be brought in later. “And that can create a lot of challenges,” Ford said. “You’re not getting the full benefit of everyone’s brain being on the problem upfront.” To ensure the best outcome, it’s important to include team representatives in each of the phases to varying degrees.
“Engineering might have a smaller role during definition, but it’s really important that they be part of it,” Ford continued. “Otherwise, you’re just not going to end up with the best solution.”
