Software engineering should not be compared with any engineering process, because it leads to harmful shortcuts. There are many analogies that are drawn with the automobile industry or the building sector.
But software engineering is a very particular industry: it is the kingdom of the invisible. Users do not see the code behind the software and the developers cannot always visualize clearly the interactions between various elements:
- the different components of the application
- all the configuration settings on the server
- the many ways the people will try to use the software
- the malicious tactics of hackers
- the caches
There is never a clear picture. It seems invisible, very obscure, and the potentiality of problems seem infinite… it looks like Space!
If the developers cannot have this clear picture, it is even more obscure for all the other stakeholders: project manager, business analyst, customer…
And there is this dichotomy between the product that you can see – web pages or screens – and the reality of the construction behind, which is the code, the databases, the server…The user sees only the visible side of the software. It is as when looking at someone very tanned and smiling: you can never know but she perhaps has a serious illness, that even she could not be aware of.
We must also add the initial complexity of not knowing exactly how one will build the product, because each software project is unique, and is in nature a prototype. The product that the programmer is building is a research, a test, and will be made of plenty of things that have to be invented by this programmer. By invented, I mean something that the programmer himself has never done before.
And the context of the product will never be similar for two products, so we really are talking about inventions: softwares are like human beings, you can never find two humans that are identical.
You cannot observe the complexity of the software you are building: you can only explore it, like an astronomer explores the Universe. Of course they draw maps or the different constellations, calculate distances, but every day they discover new planets, new phenomenons in the Universe.
I dare say that software complexity is quite identical to that. If not, how can we explain the gigantic disasters of some software projects?
Being aware of this, our experience should have taught us to be humble when involved in a new project, and to accept that we will build a prototype, perhaps a beta, that will eventually become a stable product.
In fact our reaction is often the contrary of humility: in reaction to our fear of uncertainty and darkness, we become arrogant. So we try to build very precise and rigid roadmaps, schedules, features lists. By doing this, we think we can constraint the developer, force him to make what we want him to do. It is like writing operating instructions of a new model of washing machine before the machine is built.
We should think differently: if we agree that the developer will first build a prototype, we should let him this liberty: the freedom to try the best solution to build this prototype.
Software programming is close to an art, but it is different: you have to create a melody, but you also have to guess the notes.