I've been bemused watching the rise of agile over the last (10?) years since my background is in building software for large/multinational non-tech companies. They have VPs and accounting departments with budgets and they want to know exactly how much they're spending before they spend the first dollar. Whether or not they should, they don't see the uncertainties in estimating software development as their problem.
My solution to this is waterfall, and charging for a rigorous requirements analysis process up front, which I, in turn, estimate the cost of by having the client submit their requirements (as they know them so far) in detail in writing up front.
I've been surprised in (the practical version of) agile how often time is wasted revising the same feature 5x that could have been done once if a modicum of time and thought was put into business and technical requirements.
I am however a believer that people do things for a reason, it does probably have advantages for R&D or inside technical product companies.
I have worked on agile projects inside large non-technical corporates and there are several big advantages to agile in such an environment. For example:
1. Much less time spent trying to chase requirements that are technically infeasible within the budget and rough timescales. This on its own normally more than makes up for iterating and throwing away proof of concept implementations of features.
2. Much better visibility of progress. Because you are delivering something tangible that stakeholders can see and feel all the way through the development process, you get better feedback from your stakeholders, better buy-in from your sponsors, and fewer canceled projects.
3. Where projects are canceled, they tend to be canceled early where they are clearly not working, before millions has been spent.
The need for requirements doesn't go away with agile. If you start a project with no requirements, it will almost certainly fail. However the iterative process allows you to refine the requirements as you go, and the early and frequent feedback from your stakeholders makes the requirements better. It's easier to add or remove a requirement if you can see the software doing something wrong or not doing something it should do, rather than trying to figure it out on paper ahead of time.
Those benefits all make sense, but the main issue is just getting buy-in on that strategy from executives with budgets. I haven't had luck selling a product with a blank check or a range, all my customers have needed a fixed price and delivery date upfront.
The same dynamic happens with large residential/commercial construction projects FYI. They're expensive, complex, and hard to estimate. Overruns are common, but commitments on time and price are still made up front. In that industry they often remediate, if not solve, the problem by paying third-parties who are experts at estimation and of course everything that can possibly be spec'ed up front, is.
My solution to this is waterfall, and charging for a rigorous requirements analysis process up front, which I, in turn, estimate the cost of by having the client submit their requirements (as they know them so far) in detail in writing up front.
I've been surprised in (the practical version of) agile how often time is wasted revising the same feature 5x that could have been done once if a modicum of time and thought was put into business and technical requirements.
I am however a believer that people do things for a reason, it does probably have advantages for R&D or inside technical product companies.