One of the unfortunate side effects of the tension between agile and waterfall development methodologies has been a polarization of views on the subject. The reality is that many companies were starting to become more iterative in their development methodologies before the signing of the agile manifesto in 2001. This has pushed many traditional shops to focus on Waterfall as the “correct” way to develop software.
Matt Hotle, Dave Norton and I have just published a new research article titled “The End of Waterfall as We Know It” . This research explains the growing consensus that long duration Waterfall projects are the highest risk ways to deliver software. Despite Waterfall being an established methodology with over 50 years of usage, an alarming number of projects are delivered late or worse deliver software that falls far short of meeting the business need.
The classic “long tail” software project presents the business with a Sofie’s choice. After investing many months and millions of dollars in a project, the business is told that the project is behind schedule and will not be completed on time. Given the phased development approach, there is no software that is complete enough to be deployed. The business can either abandon the project and lose the sunk cost, or pour more money into the project.
Once the project is delivered, it is commonly found to not meet the business need due to a lack of understanding of the business problem, or changing business conditions. Many years of improving the process of requirements gathering have failed to eliminate this scenario.
This does not mean that all IT shops need to become agile. Iterative development will provide for more frequent feedback and avoid working for too long on the wrong solution. A focus on automated testing and continuous integration will ensure that there is something worth deploying on the scheduled release date. Agile will tighten the feedback loop even more.