Do features you do not use confer benefits?

A baroque language

Common Lisp has some abstruse features that are rarely used. DEFINE-METHOD-COMBINATION allows the programmer to create his own method combinations. The condition system has RESTART-BIND in addition to RESTART-CASE.

This leads to the criticism that Common Lisp is a baroque language with too many features. Part of the logic of the critique is that programmers gain no benefit from the features that they do not use. This is too glib. Large programs need planning, which introduces subtle complications.

Why plan?

Why not plunge right in and get on with coding? At the heart of planning computer programs lies the idea of a contrast between some bits of code that are routine and other bits that are tricky. Coding the routine bits always goes smoothly. Coding the tricky bits sometimes ends in tears; the code cannot be written as originally designed and the work done on the tricky bit and on many routine parts of the program feeding in to it must be scrapped.

When we plan, we lightly sketch in the routine parts of the program, concentrating our efforts on anticipating the problems in the tricky bits. The benefit we obtain from planning is a reduction in the costs imposed upon us by the the discovery of show-stopping problems in the tricky bits. We lose the work we did implementing the tricky bit. That is unavoidable. We lose the work we did on the routine parts of the program feeding into the tricky bit, but we only sketched those parts, so the loss is greatly reduced compared to plunging into coding without looking ahead.

The effectiveness of planning

The benefit of planning flows from the routine parts of the code being mere sketches. The more that we can classify as routine, the more effective the the planning process will be at eliminating wasted effort.

When we plan we make two kinds of mistake.

1. Sometimes we think that a part of the program is tricky when it is actually routine. This causes us to spend effort filling in the details earlier than we should. If these details survive into the final program, the mistake costs us nothing. If this detailed worked is discarded later in the planning process we pay a price for the mistake. Had we seen that the work was routine we would only have sketched and only discarded a sketch.
2. Sometimes we think that a part of the program is routine when it is actually tricky. When we come to code it we may succeed, in which case our mistake has cost us nothing, or we may encounter a show-stopping problem and fail. Now we must discard much detailed work before trying again. This kind of mistake can be very expensive, emerging late in the project and impacting much detailed work

Language features and planning

Now that we are clear about how planning delivers its benefits we can look at how this interacts with the features present in or absent from our programming language.

For example, we may be called upon to decide whether a mildly ambitious plan, based around an unusual method combination is routine or tricky. We need to think ahead. A bug may show us that the method combination we had planned on is not in fact quite suitable. If the project is coded in a language with custom method combinations we may be able to anticipate that the code is routine. If problems arise, we can code our way out of trouble. If the project is coded in a language with a fixed set of method combinations we may call the code tricky and feel obliged to nail down the details early on.

The interest arises from the fact that planning needs to be fairly conservative. As we explained earlier the costs of the two kinds of error are different. We only call code routine if we have some notion of Plan B, which we have in reserve if things don't go smoothly. We are only justified in calling it Plan B if Plan A usually works. Sometimes Plan B is to use the advanced version of a language feature. In this case we gain an advantage, in the planning stage, from a feature that we don't actually get round to using.

That seems paradoxical. What are the ingredients that make this happen? There seem to be two.

1. The feature has to be never used in the ordinary sense that actually Fred used it a couple of months ago, but that doesn't count because it was a special case. If it is genuinely never used it can hardly be counted on as Plan B and doesn't help at the planning stage.
2. It seems most likely to happen with features that offer a more general version of a commonly used built-in feature or some way of brute forcing past rare problems.

DEFINE-MODIFY-MACRO has limitations. As Graham explains on page 169 of ANSI Common Lisp neither PUSH nor POP can be defined as modify-macros. The reason that you can plan with confidence is that you have DEFINE-SETF-EXPANDER available as Plan B. You might never use DEFINE-SETF-EXPANDER, but you benefit from it anyway because you can plan on the basis that code that is going to use DEFINE-MODIFY-MACRO is routine and leave the details until other parts of your plan are firmed up.

Ghost benefits are real!

Software projects require planning. Planning gains in effectiveness the more often you can declare "This bit of code is routine, we can fill in the details later." The right kind of advanced programming language feature helps with this, allowing you to plan with confidence. In this way programmers benefit from features that they don't use.