Software and failure
As frequently reported by specialised literature, more often than not software development is a failure. Of course, this depends on the criteria you use to evaluate. But most specialist will admit that projects that suffer none of the following are rare: overtime, over budget, incomplete features, ... Why this is the case and how to improve the situation has been an area of research for the past decades. However, it seems that none of the findings have really succeeded, for various reasons. But first let's try to overview the software process scene.
To address the software industry problems, software engineers strove to set up software development processes which would provide greater control and predictability over the activity, thus preventing the unwanted above situations. These processes went through a number of stages during their development over time.
The waterfall era
The first step was reproducing what was done in other disciplines, such as civil engineering or manufacturing. This lead to the 'waterfall process' which outlines as follows: specification, design, implementation, test, deliver. Now, this was a good idea but fell short of at least one reallity. This is the fact that in software the cost of implementation does not outweigh the cost of the other preceding phases, specially design and analysis. This contrasts with civil engineering of course and also with manufacturing although in a slightly different way. Indeed, software development is basically always a prototyping activity because you never develop the same software twice and you rarely even use the same technologies twice (because information technologies evolve on a weekly/monthly scale). The problem is that when 'cost reduction' came into the game, the specification, analysis and design activities suffered a lot and therefore the result was poor foundations for the following implementation phase which in turn gave poor software.
The spiral era
This laid the path to a new generation of processes called 'spiral processes' because they acknowledged the fact that design wouldn't be completely validated (because it isn't economically and/or technically feasible in most cases) before implementation would start. So the idea is to create an initial design quite quickly which would be reworked as needed all along the implementation work. This avoided costly validation work of the design as its flaws would appear naturally during the development. This was a nice improvement which helped achieving software that replied to specifications with reasonable control. However, this approach didn't really satisfy everyone, and in particular not the end users who often realised at the end of the day that what they got was mostly what they'd asked for (with a few misunderstandings probably slipping in) but was in fact not really what they wanted and probably even less what they needed. Why is this? In fact we could rather say why shouldn't it be? As said, software is like prototyping, and in most cases you need a few prototypes before getting the wanted result. Also, software is often very complex (even relatively simple software), and it is difficult to imagine all the combinations of scenarios which can arise and so not all is really planned for.
The incremental and iterative era
This took industry a step further with the idea that the end user should be integrated in the full development cycle (and not only during specification, when it is too soon, and final delivery, when it is too late). Here came the 'incremental and iterative' processes. The iterative part is similar in concept to the spiral approach. The incremental aspect is the new cornerstone. Thanks to this concept the user will be delivered many versions of the software with each time a little more featured, hence the name incremental (this is incremental for the user). The advantage is that at each increment the user can test the software and provide feedback (change requests) which will be integrated for the next increment. This result is that the user should finish with what he needs (because he tests in situ).
This latest model (iterative and incremental process) is the one usually used today with a few variations like Agile and Predictive processes, but mainly they can be blended and merged into and from one another.
So, is the incremental and iterative approach the solution? Unfortunately not fully. Something has been forgotten along the path. The user now gets what he wants, but he doesn't really know a priori at what price (or in what time)! Basically, in many cases, the customer is told the outlines of what will be done and that this will be adapted along the way depending on the users' feedback. This is the accepted situation in most cases today.