Wednesday, August 28, 2013

On Exploratory Design and Piecemeal Development

I have a soft spot for Lisp-like languages, I suspect I always will. I've now been programming in Scala for several months. And I sometimes find the language tedious. I've lately been thinking about the cathedral and the bazaar, worse is better, and other such essays in relation to how I feel about Scala vs Lisp.

Let's take a look at what Scala offers:

  1. An interesting combination of object oriented and functional programming styles.
  2. A powerful type calculus.
  3. Some interactive development via a REPL, though the REPL is fairly underpowered and almost an afterthought.

Common Lisp has, by contrast:

  1. A dynamic language with optional typing supported unevenly by the various implementations of the language.
  2. An object system that supports multiple inheritance, metaclasses, and multi-methods, with a rich vocabulary to describe how methods should interact.
  3. A language built with the REPL as a central feature. A Common Lisp development environment almost always has a REPL running in the background that can always be accessed.

The most significant downsides I've experienced for Scala are:

  1. Excruciatingly long compile times.
  2. The type system, while expressive, is also rigid.

I've run head-on into these problems with my current task, designing a DSL on top of an existing codebase. The codebase isn't very large, perhaps a couple of hundred substantial classes. Implementing a DSL, for anyone that's attempted this exercise, isn't a straightforward proposition. A DSL is about allowing a user to express their programming problem in a form that's most suitable for their problem domain, while hiding away much of the complexity of the software's underlying implementation. Like any API, implementing a DSL often requires adjusting the rest of the sofware a little bit to support the DSL. And it requires trying out a few different ideas to see which one sticks. It is a process of discovering how one might best help users express the problems in their domain.

Scala requires that the whole program compile before any part of the program is executable. It requires strong consistency in the elements of the program. Which means you can't just fire up a REPL, load up a few classes that are most relevant to the problem at hand, and play around with your implementation. If you're modifying the class definitions in any fashion, you have to make sure that the rest of the program is made consistent with your changes before you can run your experiment. The long compilation times, and inability to focus on just a few classes while experimenting, mean painfully slow experimentation in building up the DSL.

I feel strongly that Common Lisp (and many other Lisp oriented languages, including Clojure) do this better. Let's start with the proposition that your entire program isn't going to undergo development all the time. Development is a highly non-uniform activity. Once a part of the code reaches a certain level of maturity and functionality it is often set aside, while focus shifts to other problems. While under active development, code needn't be wholly consistent. This gives some space to experiment and rewrite the program many times until the right expression is found. On the other hand, touching relatively stable code should require much more care and attention on the part of the developer. The challenge is in structuring the implementation so that we have local stability in parts of the codebase.

Let's delve a little bit deeper into Common Lisp's technical implementation that makes this possible. Say we have a program with a couple of hundred classes, and we wish to evaluate a DSL implementation that changes the type of a commonly used class by removing one of its supertypes. We would start with loading the entire program into the development environment. Then we would interactively change the type of the class. To be clear, this means changing the type definition of the class, and impacting all existing implementations of that class. Common Lisp provides a type change protocol to support such an operation on objects. Parts of the program would no longer function correctly, but that's fine so long as the whole program isn't running. Since Common Lisp is dynamic it typically tolerates such inconsistency. Now we can implement a partial program to test if the implementation we have in mind can work. Rather than going through full compile cycles for the entire program, we can evaluate or compile just the bits we're working with, quickly verify if our approach might work, and either move forward or revert. We're changing the behavior of our program at runtime in order to evaluate alternate implementations.

Programming tool support made programming in Java palatable. Maybe one day we'll see better tool support for Scala, with automated support for refactoring type hierarchies and type parameters. For now we must do this manually.