[Off-Topic] The Humble Programmer, by Edsger W. Dijkstra

After writing my article on Info, Software Factory Is Nonsense, I received a retweet with a very cool link to a text I didn’t know. The Humble Programmer.

Of course, the author is well known, the great Edsger W. Dijkstra. He’s best known for the seminal paper A Case against the GO TO Statement. Anyway, The Humble Programmer was a speech he gave upon receiving the 1972 Alan Turing Award.

The text is fantastic and should be read in full, but I decided to pull a few excerpts to comment on. The most interesting thing is to read it with the context of the late 60s in mind, and how much of what he hopes for the future is something we, 50 years later, are still hoping for. I didn’t publish this text on Info for two reasons: first, because it’s more oriented toward programmers, and second, because this is one of my “Akita-sized” texts :-)

Two opinions about programming date from those days. I mention them now, I shall return to them later. The one opinion was that a really competent programmer should be puzzle-minded and very fond of clever tricks; the other opinion was that programming was nothing more than optimizing the efficiency of the computational process, in one direction or the other.

Unfortunately, the image of the programmer has changed, but to two extremes: we have the super-programmers, renowned authors, but the profession itself has become a commodity, an abundant and cheap consumer good, precisely because of what I commented on in the article about software factories. The artificial cheapening of the profession is leading to decay and a longer delay for research and evolution in the area, especially if we count that the competition to lower the price doesn’t come from raising the technical quality of processes and technologies, but mainly from using globalization to take the task to areas where human resources are cheaper, like India and China. It doesn’t mean everyone in the profession is a worker — those who grew on their own and evolved have respectable positions and tasks, but that’s due more to individual effort.

This latter opinion was the result of the frequent circumstance that, indeed, the equipment available was painfully slow, and in those days one often encountered the naïve expectation that, as soon as more powerful machines became available, programming would no longer be a problem, for then the effort to stretch the machine to its limits would no longer be necessary and that was all there was to programming, wasn’t it? But in the next decades something completely different happened: more powerful machines became available. But instead of finding ourselves in a state of perfect bliss with all programming problems solved, we found ourselves up to our necks in the software crisis! How come?

At least today no one assumes that just because machines will be better, programming will be simpler. But note something I always repeat: people think software problems are recent things, but the term “software crisis” was coined back in the 70s … and to this day it hasn’t been solved.

The vision is that, well before the seventies have run to completion, we shall be able to design and implement the kind of systems that are now straining our programming ability, at the expense of only a few percent in man-years of what they cost us now, and that besides that, these systems will be virtually free of bugs. These two improvements go hand in hand. In this latter respect software seems to be different from many other products, where as a rule a higher quality implies a higher price. Those who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result the programming process will become cheaper. If you want more effective programmers, you will discover that they should not waste their time debugging, they should not introduce the bugs to start with. In other words: both goals point to the same change.

“More effective programmers … should not waste their time debugging” — this is a conclusion that has existed for at least 5 decades, and yet to this day current programmers — trained in colleges to serve “factories,” and therefore to be tool workers — have hysterical fits when their tool doesn’t have specific debugging capabilities!

“Those who want really reliable software will discover that they must find means of avoiding the majority of bugs.” That is, efficiency doesn’t come from wanting to find bugs as quickly as possible, but from preventing them from being introduced in the first place. I know it may sound pedantic, but we have techniques for this, and the vast majority of programmers don’t use them — they’re not even taught in college. Search for Extreme Programming. To prevent bugs from being inserted by carelessness (which doesn’t mean eliminating 100% of bugs but avoiding most of them) there are techniques like Test First, Pair Programming, Continuous Integration, Tests to Avoid Regression Bugs, Acceptance Tests. Anyway, there are several simple and efficient techniques that the vast majority of the market isn’t even aware exist.

Now for the economic necessities. Nowadays one often encounters the opinion that in the sixties programming was an overpaid profession, and that in the coming years programmer salaries will come down. Usually this opinion is expressed in connection with the recession, but it could be a symptom of something different and perhaps even wholesome, viz., that perhaps the programmers of the past decade have not done so good a job as they should have done. Society is getting dissatisfied with the performance of programmers and their products. But there is another factor of much greater weight. In the present situation it is quite usual that for a specific system, the price to be paid for the development of software is of the same order of magnitude as the price of the hardware needed, and society more or less accepts that. But hardware manufacturers tell us that in the next decade hardware prices can be expected to drop by a factor of 10. If software development were to continue to be the same clumsy and expensive process as it is now, things would get completely out of balance. You cannot expect society to accept this, and therefore we must learn to program an order of magnitude more effectively. To put it in another way: as long as machines were the largest item on the budget, the programming profession could get away with its clumsy techniques, but that umbrella will very soon be folded.

And this has been our challenge in recent decades: how to “cheapen” the task of programming. But this can be taken the wrong way. One of the wrong ways is cheapening the labor force — just find poorer countries that charge less. Another way is cheapening the technology. That works up to an inflection point where we would need a new level of sophistication but we won’t have it because cheapening cut research and innovation in the area. The whole argument is based on “how not to make more expensive what today is already expensive” and for that one of the ways is to advance technology to, for example, demand fewer manual tasks that can be automated. To this day there are still “programmers” who waste their time opening the same windows and clicking the same “next, next, next” buttons every time they need to package a new version of software. That’s automatable, but since the programmer was trained to just follow procedures, most don’t find themselves capable of being creative and transforming a manual procedure into an automated script, for example. It’s a case where the “cheapening” of labor and its training is preventing the cheapening of processes and the advancement of technology. This was a problem in the 70s and still is in the 21st century.

Argument three is based on the constructive approach to the problem of program correctness. Today a usual technique is to make a program and then to test it. But: program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence. The only effective way to raise the confidence level of a program significantly is to give a convincing proof of its correctness. But one should not first make the program and then prove its correctness, because then the requirement of providing the proof would only increase the poor programmer’s burden. On the contrary: the programmer should let the correctness proof and the program grow hand in hand. Argument three is essentially based on the following observation. If one first asks oneself what the structure of a convincing proof would be and, having found this, then constructs a program satisfying this proof’s requirements, then these correctness concerns turn out to be a very effective heuristic guidance.

I have to admit Dijkstra is referring to formal mathematical proofs in this case. But I’d like to expand the concept. What he says is that the correct way would be first to create a “proof” and then implement the code that meets the requirements of that “proof.” I’d like to say that Dijkstra was practically a pioneer in the “Test First” concept of Extreme Programming, also known as TDD or “Test Driven Development” where the concept is: 1) first write a test that describes the requirement; 2) now we implement the code that makes the test fail; 3) then we implement the rest that makes the test pass; 4) move on to the next requirement. Dijkstra knew 50 years ago that this order of development is what minimizes the volume of bugs a posteriori — besides dozens of other benefits we’ve discovered today.

Argument four has to do with the way in which the amount of intellectual effort needed to design a program depends on the program length. It has been suggested that there is some kind of law of nature telling us that the amount of intellectual effort needed grows with the square of program length. But, thank goodness, no one has ever been able to prove this law. And this is because it need not be true. We all know that the only mental tool by means of which a very finite piece of reasoning can cover a myriad cases is called “abstraction”; as a result the effective exploitation of his powers of abstraction must be regarded as one of the most vital activities of a competent programmer. In this connection it might be worthwhile to point out that the purpose of abstracting is not to be vague, but to create a new semantic level in which one can be absolutely precise. (…) A second result was the identification of a number of patterns of abstraction that play a vital role in the whole process of program composition. Enough is now known about these patterns of abstraction that you could devote a lecture to each of them.

This is more complex but has to do with the programmer’s abstraction capacity. The concept itself is “abstract” and hard to define. The first part has to do with talent: a person who has no talent for programming will not be a programmer, period. Starting from the premise that the talent spark exists, now thousands of hours of practice are needed — and I mean practice of actual code and not just repeating procedures, but experimenting in the most different situations possible.

Only then will intuition emerge from experience, making it possible to identify patterns in code, opportunities for refactoring and optimization, creation of higher-level constructs to simplify the program, and so on. That path starts with the programmer following procedures, but must evolve rapidly into experimentation. This is fundamental.

As a curiosity, part of what he describes is what we today also know as “Design Patterns.”

Now for my fifth argument. It has to do with the influence of the tool we are trying to use upon our own thinking habits. I observe a cultural tradition, which in all probability has its roots in the Renaissance, to ignore this influence, to regard the human mind as the supreme and autonomous master of its artifacts. But if I start to analyse the thinking habits of myself and of my fellow human beings, I come, whether I like it or not, to a completely different conclusion, viz. that the tools we are trying to use and the language or notation we are using to express or record our thoughts, are the major factors determining what we can think or express at all! The analysis of the influence that programming languages have on the thinking habits of its users, and the recognition that, by now, brainpower is by far our scarcest resource, together give us a new collection of yardsticks for comparing the relative merits of various programming languages. The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague. (…) Another lesson we should have learned from the recent past is that the development of “richer” or “more powerful” programming languages was a mistake in the sense that these baroque monstrosities, these conglomerations of idiosyncrasies, are really unmanageable, both mechanically and mentally. I see a great future for very systematic and very modest programming languages.

A good programmer recognizes his limitations and looks for tools that better fit the problems. Most language wars start with arguments like this, but the problem is that the insistence on having “just one language” and piling on top of it everything everyone wants makes it a “baroque” monster, as Dijkstra puts it. Programmers — people — have limited brains, and we need to be able to express ourselves in code form. The more complicated the tool, the more of the mind is spent keeping the pieces in the head and less on creating elegant code.

Look at Dijkstra himself — up to the 70s, he literally saw the creation of computers, programmed in practically everything, from machine language to Fortran, Lisp, Algol. If back then he could know multiple languages in depth, I see no excuse for today, with all the extra resources we have, not to know an order of magnitude more languages and technologies and techniques.

He mentions more “modest” languages, and I’d say those are our current dynamic high-level languages like Ruby or Python. Abstractions that allow even larger systems with less complexity.

In order to complement this, I want to issue a warning to those who identify the difficulty of the programming task with the struggle against the inadequacies of our current tools, because they might conclude that, once our tools will be much more adequate, programming will no longer be a problem. Programming will remain very difficult, because once we have freed ourselves from the circumstantial cumbersomeness, we will find ourselves free to tackle the problems that are now well beyond our programming capacity.

And since the 70s we know there’s no silver bullet — it isn’t a new tool that will simply magically make the programming task orders of magnitude more efficient. There’s no free lunch.

This has already taught us a few lessons, and the one I have chosen to stress in this talk is the following. We shall do a much better programming job, provided that we approach the task with a full appreciation of its tremendous difficulty, provided that we stick to modest and elegant programming languages, provided that we respect the intrinsic limitations of the human mind and approach the task as Very Humble Programmers.

Programmers need to be Humble in the correct sense of the word: in acknowledging their own limitations and creating new techniques, technologies, and ways to perform the same work with more quality, more efficiency, breaking rules and traditions and creating new standards. Programmers who just follow what was taught, just the procedures, are arrogant because they think everything that could have been discovered has already been discovered.

I always repeat that a good programmer is dumb and lazy. Dumb because if they think they’re smart they’ll also think they already know everything, and if they already know everything why research more? And lazy because an overly hardworking programmer will follow the same procedure every day with great diligence, while a lazy one will get tired of doing that and will create an automated way to do the same work, leaving more time to rest.

I think Dijkstra would agree with that statement ;-)