Should we ever use Waterfall?

Further ramblings on methodologies

As always — I’m afraid — it depends on definition, but also on what you want to use it for.

Tl;Dr

Should we use waterfall?

Not, for new, unique stuff (product design, feature development, unique goal driven initiatives, individual problem solving)

Yes, for consistently repeatable activities, such as application configuration or rollout or mass production

But in all cases should we apply the mindset and soft aspects of ‘Agility’ and certainly implement Lean.

Also, agile or iterative approaches can have shorter or longer iterations and can have different layers with, say, highly agile, short iterations for day to day delivery and more linear structure at top level. This can help if we face organisations that are not fully agile yet, in the process of transformation or if we are constrained by dependencies.

So it’s not an either — or, but more a ‘mix & match, what works’

What is ‘Waterfall’

I have posted before about whether Agile is better than Waterfall and outlined some differences in that post, but let’s recap.

When we say ‘waterfall’, we think of a linear methodology where project selection is followed by analysis by solution design, implementation, deployment and operation and eventually decommissioning.

This linear approach, which used to be the standard project management approach for literally all types of endeavours until the 2000s — and still is for most construction and many engineering initiatives — comes, generally speaking, with a number of paradigms:

• the project stages are sequential, and clearly delineated (gated)
• any change that requires going back into a previous stage is seen as aberration or mistake and discouraged
• all analysis and all design is done up front, i.e. before we start implementation (frequently called BigDesignUpFont)
• risks are managed by up-front analysis, the idea being that the more we plan and think up front, the better we can manage risks
• plans are being created upfront and then followed and tracked, again, the idea being that solid upfront planning will allow us to perfectly predict the future ant outcomes and that plans should be followed except in severe instances
• scope and time and cost and quality are considered fixed and non variable (in reality time and cost are the one’s that get moved if need be)
• oldschool waterfall methodologies are frequently high on process, hierarchy, governance and bureaucracy as this is seen to add control and reduce risk
• benefits are realised at the end once everything has been delivered

As you can see there are two things mixed in here:

• values and culture (how we think about empowerment, management, planning, risk and change)
• process (one step after the other)

For many projects this didn’t work

Over decades of software (and other) engineering it was found that linear (waterfall methodologies) do not work. The reason is complexity (unforeseeable behaviours that emerge as we go) and change intrinsic and unavoidable in literally all domains.

This frequently lead to undesirable outcomes, a lot of waste, late delivery or cost overrun in many cases, specifically in case of complex rather than complicated or obvious problems (see Cynefin).

A new world

In the 2000s we saw an emergence of what’s now called Agile methodologies. Best summarised as covering two aspects

• a different way to think about values and culture, shifting to accommodate, even appreciate change, enable early value delivery, short feedback cycles, more efficient ways of working
• a different way to think about delivery of work, mostly based on ideas of delivering in iterations

Tangentially related to this are the concepts of lean and product thinking that are worth exploring.

So is waterfall never right?

Generally speaking I would not recommend anyone to implement old-school, full fledged waterfall. Orthodox waterfall incurs too much risk and too much waste based on the assumption that we can get it right from the beginning, first time, and that things do not change.

There are cases where a more linear approach is more beneficial

1 Anything to do with repeatable activities such as rolling out or configuring an application or OTS product, anything that is highly repeatable (think mass production / assembly line) and standardised is best executed in linear fashion. Plan once and well, then execute consistently.
But, remember, you still want the good Agile stuff, like learning and continuous improvement, best, maybe articulated in Lean and Kanban.

Where, on the contrary you’d want Agile is for new feature or product delivery with a high degree of ‘newness’ or ‘unknowns’.

2 I think the software (agile) vs. engineering or construction (waterfall) way of thinking is a red herring. I have friends in all industries applying agile principles and processes. However, in cases where you have heavy dependencies or external constraints that force you into ‘stages’ (e.g. regulatory compliance) a certain level of linearity and work stages can be helpful. Also, of course, where the cost of change after the fact is prohibitive you might want to do more upfront thinking (which still doesn’t mitigate against the risk of change due to shift in context). To be honest, even here, we see an interesting shift where organisations like DoD or those in heavily regulated industries shift more and more to agile, getting away from waterfall…

However, there are cases where a highly agile approach isn’t suitable (just yet);

3 Not all teams or organisations are ready for the agile mindset of exploration and limited long-term visibility of where things will end up (not that waterfall sorts that, but it gives the appearance of it)
In this case a bit more linear structure as an overall wrapper around, a more agile approach ‘under the hood’ works well

4 Teams new to agile or organisations in the process of transformation may want to consider starting with ‘agilising’ day to day work and then make more over-arching changes.