Welcome, Reddit readers!
Please note that this paper was written specifically to spark discussion in a workshop focused on "reinventing computing." Its content should be taken with a small grain of salt.
On this dark horizon there are some significant beacons of hope. The Internet somehow managed to grow into a mass-market phenomenon without any corporations gaining a significant measure of control over it (yet). Linux and Perl have had perceptible impacts on the computing landscape. And while I would not hold up Perl as a shining example of what computing ought to be, it does demonstrate that it is possible for some individual's vision (or some small group of individuals' vision) of what computing ought to be (or what some aspect of it ought to be) to succeed.
Nonetheless, none of these efforts address what I see as the fundamental problem, which the wide gulf that separates programmers and users. As long as this gulf exists life will be miserable. The reason is simple: the money comes from users, so the users' desires trump the programmers'. Unfortunately, the producers of shoddy software have managed to hoodwink users into believing that software in *inherently* unreliable and brittle. Having to reboot your machine three times a day is like getting the oil changed on your car: annoying, but necessary maintenance. In such a world one has only three choices: either join the producers of shoddy software, become a consumer of shoddy software, or find a way to make a living that doesn't involve computers.
It may be that producing good software is just so much harder than producing shoddy software that the market has made a rational decision in opting for shoddy software because good software would cost so much more. In fact, the opposite is true: good software is almost always cheaper than shoddy software. For example, Linux, which is vastly more reliable than Windows, is free. Why doesn't the market flock to it? Because Linux (and unix in general) appeals to programmers, not users. Only people on the programmer side of the programmer-user divide understand enough about software to recognize the advantages of Linux. But these people don't drive the market, the users do.
There have been efforts in the past to "bring programming to the masses" (e.g. Logo) that have failed, prompting many to conclude that programming is just inherently difficult, and it is this inherent difficulty that leads to the programmer-user divide. Certainly some aspects of programming are difficult. It is probably unreasonable to expect the average user to grok the Y combinator. On the other hand, there have been some accidental successes at bringing programming to the masses. HTML, for example, is a programming language (not a general-purpose language to be sure, but a programming language nonetheless) that has proven accessible to many non-programmers, despite (or perhaps because of) not having been designed for that purpose.
Solving a math problem is an activity with a distinctive structure. For example, parts of the process are mechanizable and can be performed by a machine. Moreover, it is not necessary to build a new machine for each math problem one wishes to solve. It is possible to build a universal machine that can in principle solve any solvable math problem (for some suitable definition of solvable).
Another feature of solving math problems is that at some point the process generates a *result*, at which point the process *ends*. The economic value of solving math problems generally lies in the result, and not in the process. (There are exceptions, such as when one is solving math problems for entertainment.)
It turns out that the sort of machines one ends up building to solve math problems is also good for other things, like editing documents or playing video games. The structure of these activities is fundamentally different than the structure of math problem solving. The economic value of video game playing or document editing lies in the process, not the result. In the case of video game playing there is no result. In the case of document processing there is arguably a result (the document) but the value of computer assisted document processing lies in the fact that the process of creating the document is more efficient using a computer than using other tools like typewriters and carbon paper. The computer contributes little to the result, the content of the document (the exception being things like spelling correction, and even there it is not clear that the computer is making a net contribution).
The problem is that the methods we have developed for creating software, while they work reasonably well for results-oriented computations, don't work nearly as well for process-oriented ones. The fundamental problem is that our methods make an unwarranted assumption, namely, that the process of creating software, like the process of computing, *ends* and generates a *result* called a *program*. The fact that the parts of the software creation process itself can be couched as a math problem tends to reinforce this view.
There are existing systems that take tentative steps towards a program-free world. Users of Lisp and Smalltalk create software by "interacting with an environment" rather than "writing a program." Labview users create software by dragging and dropping graphical icons. The concept of "a Lisp program" or "a Labview program" is not so crisply defined as "a C program." But I think we need to go much further than just putting a graphical or interactive window dressing on business-as-usual.
"Program" manifests itself not only in the creation of software but also, and much more significantly, in its delivery. Delivering software is nowadays synonymous with delivering executable code, and more often than not, delivering a complete self-contained application. Programmers are trained to believe that getting the algorithm to work is not enough. You have to put the user interface on it before the program is "done". As a result, an enormous amount of programmer time (I conjecture that it is actually a majority of programmer's time, though I don't have any data) is wasted continually and laboriously reinventing the same things over and over again: window-based UI's and (bad) command-line parsers.
It doesn't have to be this way. Lisp programmers, for example, rarely write command line parsers because they can simply use S-expressions and use the Lisp reader to parse them. Moreover, Lisp programmers rarely have to learn new command line syntaxes for the same reason. S-expressions are a universal command-line syntax. Everything can be expressed as an S-expression.
Contrast this state of affairs with the unix command line syntax (or lack thereof). Because there is no universal syntax, the unix world is a Babel of dozens of different syntactic conventions. For example, sometimes the '-' sign is an operator (subtraction), sometimes it is a keyword indicator (the conventional command syntax), sometimes it indicates a function call (in conditionals in some shells), sometimes it denotes a range (e.g. in some regular expression syntaxes), etc. etc. etc. In lisp '-' has no special syntactic meaning. It is merely a constituent character like the letters of the alphabet.
(NOTE: S-expressions have been around long enough that it is clear that they will not bring about the synthesis I seek. I cite them merely as an existence proof that vast simplification of the current state of affairs is possible.)
The situation on the user side of things is not much different. Every application has a different set of user interface conventions. Things have gotten so complicated that people have started to take it for granted that they have to be "trained" to use a computer. Again, it doesn't have to be this way. In the early days of the Macintosh there were programs that required no training (and no documentation!) to use.
Today's applications require training because they have more features than MacDraw. Users have been conditioned to think that features are a good thing. A spreadsheet that has a spelling checker seems better than a spreadsheet that doesn't. I disagree. A spreadsheet shouldn't *have* a spelling checker. It should be able to somehow *interface* to a spelling checker, but I should be able to use that *same* spelling checker to check the spelling in the documents that I edit using my word processor, which shouldn't *have* a spelling checker either. For that matter, I should be able to use my word processor to edit the text inside a cell of my spreadsheet. I should be able to use my spreadsheet to update the numbers in my word processor. (Microsoft OLE actually lets you do some of these sorts of things, but the granularity is much coarser than what I envision.)
So, to sum up: today we live in a world where software is organized as different kinds of programs. There are operating systems, applications, middleware, compilers, but they are all programs. They tend to be big and encompass a lot of functionality, and making two of them work together to perform a task neither one can do alone is often a major chore. Programs are hard to write *and* hard to use, and the skills required to create a program are different from the skills required to use one.
Creating software (which is the same thing as "using a computer") consists of assembling modules to do the things you want to do. For example, one can take a module for a simple but inefficient compiler and connect it to a module that performs a certain class of optimization to obtain a more efficient compiler. Of course, what is called a compiler in my world is a different beast than what is called a compiler in the real world. They both produce machine code as output, but in my world the input to a compiler is not a programming language. (There are no programming languages because there are no programs.) There are various ways of specifying the behavior of modules, some of which are textual, but the "parsers" for these are of course modules as well and are therefore interchangeable. (This is very similar to Charles Simonyi's ideas for intentional programming, though Simonyi still wants to use these techniques for generating programs.)
One of the big conceptual differences between manipulating modules and writing programs is in the relationship between size and generality. Small programs (in terms of source code size) tend to run faster than larger ones, and have less capability. By contrast, a "small" module (that is, a module with a small amount of the human input, the modular equivalent of "source code") tends to run slow but have a lot of capability. In order to produce a module that runs fast you have to give the module compiler additional information, like type declarations. Engineering effort when writing programs tends to center on adding features and generality. By contrast, engineering effort in module design focuses on *removing* features and generality in order to gain efficiency. Thus, in the programming world, by default you get fast programs that don't work well. In the module world by default you get slower programs that do work well. By and large the fact that modules run less efficiently than programs isn't noticeable, since machines are so fast. But what is noticeable is that people have become accustomed to software that is robust and easy to understand. If someone overoptimizes a module by removing a necessary feature, computer users reject that module as unacceptable. Likewise, if someone produces a module that has too many features and is therefore too complex that module also fails.
A few other common concepts have gone by the wayside. For example, there are no more "files". Every interaction one has with a computer changes the state of some module, and by default modules know how to render themselves non-volatile by storing themselves in what we would call a database. By default, modules come with sophisticated access control, authentication, and revision control functionality (implemented by other modules). Thus, a given object knows who created it, who owns it, who is allowed to access it, who is allowed to modify it, and how to back out changes. Of course, it is possible to create modules without these capabilities, which would give you something close to what we would call an "in-memory object". (One could also make a module that behaved like a "file" by removing the authentication and revision control features, but you'd be considered a little backward if you did that.)