Software complexity cannot be avoided

Introduction : various thoughts on development, programming, creating.

Building a software is creating automated reactions to predefined user behaviours. Users have unpredictable and unexpected behaviours. So we must no think projects, but products: a product is never be finished, we have to make it live by adapting or creating new automated responses to new predefined user actions.

A software is like an helium balloon: it can be (re)shaped, it can explode with a single needle, and if you let it go, you loose control!

Design thinking  must anticipate testing but not get ahead with it. The risk is that we get lost  before event testing.

Pressured by time we rush to writing code syntax without preliminary design thinking. Confused thoughts will not result in effective coding.

How do we automate tasks that will have to respond to various behaviours, users, inputs…? That is the inherent uncertainty of programming.

Making errors is an absolute right. Testing is a prerequisite. Correcting is an absolute duty.

Refactoring is a good practice. It should not require the agreement of the managers.

Extensive, thorough  and exhaustive testing and adjustments can save a project that had a bad start.

If you’re tired of making wasteful estimates for your boss, wait untill he gets bored and become the boss; it worked for him.


Softwares are part of a vast eco-system from which the different parts are all created and maintained by humans: computers, programming languages, networks, protocols… Each of these is a sort of small universe that is constantly modified, improved, and extended by humans.

Their rules, limits and constraints are mostly a result of human rules, limits and constraints. This is one of the reasons of the complexity of software programming.

The programmer’s job is to gather a considerable knowledge and then constantly increase and update it, so that it can support his thought on how to find solutions to some problems that he discovers when the different tasks of a project are assigned to him.

This means that when he receives those tasks, the real complexity of his job begins: he must understand the task to do, find a way to do this task, and implement the solution.

I am of course talking of the complex tasks, the ones that he has never done before and that represent a sort of “mathematical” problem to solve. And these tasks are frequent in every software project.

We understand here that even when the team has sliced the project in many small tasks, the complexity still exists and has not been truly explored: a task is mainly seen by the project team as an output. It consists in the mental visualization of the result of an action, and not in the description of how the action should be done. If it was the case, the programmer would have nothing to do.

So a task is not the how-to. And the programmer’s job is not only to determine the how-to, but first to understand the what: what does this task mean in terms of programmatic problem. How to find a programmatic solution to that problem, and how to implement it (which is the easiest part).

The whole programmer’s life consists in finding solutions, but the tasks that can have a true impact on the schedule are the ones for which the programmer really has to work hard to find a solution. The degree of complexity does not depend on the size of the task (the “size” in terms of, for example, the number of data to be processed). A very “short” task on a sprint board could require a hard-to-discover algorithm to be properly addressed.

To find the solution the programmer uses different tools in the same time:

  •          his technical knowledge, which can never be exhaustive
  •          the specific research that he can make to improve his knowledge on this specific subject
  •          the discussions that he can have with pairs, that could have hints or even have had to solve a similar problem before
  •          his personal reflexion, intuition, intelligence, judgement…

From all this, it can result that once he has found and applied a suitable solution, the programmer has in the same time improved his knowledge, and another solution, more optimized,  pops in to his mind.

I have already seen this situation were the developer indicates the task as Done, and shortly afterwards asks for a new time-box to implement a better solution. In most cases, it is refused by the Project manager, because of schedule constraints. And of course it has resulted in generating frustration for the developer.

Refusing to implement this new solution is, in my opinion, opposed to the continuous improvement Agile principle: continuous improvement is not continuous learning. It means improving the product, continuously. So finding a better solution and not applying it is not continuous improvement.

We now see that the estimation of the length of the programming part of a software project cannot be accurate, even when the Team has divided the project into several small tasks. Because this linear slicing does not prefigure the non-linear programming activity that will be necessary for the realization of the tasks.

The tasks are mainly a list of problems for the programmer, and when the tasks are defined, the programming science begins.

So, even the best agile practitioners will never be able to reduce the programming complexity of a project. That is why even in the best Agile world, software estimation will always stay a guess.

Dealing with software uncertainty

Software engineering should not be compared with any engineering process, because it leads to harmful shortcuts. There are many analogies that are drawn with the automobile industry or the building sector.

But software engineering is a very particular industry: it is the kingdom of the invisible. Users do not see the code behind the software and the developers cannot always visualize clearly the interactions between various elements:

  • the different components of the application
  • all the configuration settings on the server
  • the many ways the people will try to use the software
  • the malicious tactics of hackers
  • the caches

There is never a clear picture. It seems invisible, very obscure, and the potentiality of problems seem infinite… it looks like Space!

If the developers cannot have this clear picture, it is even more obscure for all the other stakeholders: project manager, business analyst, customer…

And there is this dichotomy between the product that you can see – web pages or screens – and the reality of the construction behind, which is the code, the databases, the server…The user sees only the visible side of the software. It is as when looking at someone very tanned and smiling: you can never know but she perhaps has a serious illness, that even she could not be aware of.

We must also add the initial complexity of not knowing exactly how one will build the product, because each software project is unique, and is in nature a prototype. The product that the programmer is building is a research, a test, and will be made of plenty of things that have to be invented by this programmer. By invented, I mean something that the programmer himself has never done before.

And the context of the product will never be similar for two products, so we really are talking about inventions: softwares are like human beings, you can never find two humans that are identical.

You cannot observe the complexity of the software you are building: you can only explore it, like an astronomer explores the Universe. Of course they draw maps or the different constellations, calculate distances, but every day they discover new planets, new phenomenons in the Universe.

I dare say that software complexity is quite identical to that. If not, how can we explain the gigantic disasters of some software projects?

Being aware of this, our experience should have taught us to be humble when involved in a new project, and to accept that we will build a prototype, perhaps a beta, that will eventually become a stable product.

In fact our reaction is often the contrary of humility: in reaction to our fear of uncertainty and darkness, we become arrogant. So we try to build very precise and rigid roadmaps, schedules, features lists. By doing this, we think we can constraint the developer, force him to make what we want him to do. It is like writing operating instructions of a new model of washing machine before the machine is built.

We should think differently: if we agree that the developer will first build a prototype, we should let him this liberty: the freedom to try the best solution to build this prototype.

Software programming is close to an art, but it is different: you have to create a melody, but you also have to guess the notes.