flooey.org

A Closed and Common Lisp

Dec 31, 2021

I just recently finished up Advent of Code for 2021, and, like last year, I did it in a new-to-me language. This year’s choice was Common Lisp. My total previous exposure to Lisp was a couple weeks writing Scheme in my Programming Languages class at university, so it was almost entirely a novel experience.

Writing Lisp is like entering an alternate timeline where computing in the 1960s took a significantly different path. Rather than ALGOL becoming the favored language and spreading the syntax of blocks, assignment statements, function calls, and the rest to languages from C to JavaScript to Rust to Python, instead the world is based on lists, S-expressions, and parentheses (oh, the parentheses!).

It reminds me of nothing so much as my brief foray into learning Japanese. The vast majority of non-English natural language that I was exposed to growing up was from the Indo-European family (particularly the western European ones), so when I was first started learning Japanese, it was eye-opening. Almost every aspect of the language was unfamiliar, and it gave me a new appreciation for the great diversity of humanity.

Overall, I was greatly pleased with my choice of Common Lisp. It’s a very pleasant language to use, and it has some real high points. Below, I’ll discuss a few specific areas that caught my eye.

For those who are curious, I used sbcl as my implementation and Emacs with SLIME as my editor.

Programming Paradigm

Starting out on this project, my impression of Lisps were that they were heavily slanted toward functional programming, though with more escape hatches than something like Haskell. This is a bit more true for Scheme than for Common Lisp, but mainly it’s just wrong. While Common Lisp supports a functional programming style very well, it also happily supports a procedural style. Indeed, it marries the two wonderfully, letting you use the right tool for the job.

As a simple example, despite the origins of the name Lisp, it has arrays as a first class data structure, and they work just like in any other procedural language, so you can write efficient iterative code that would be difficult in a functional style just fine. Looping is also both well supported and entirely idiomatic from what I understand.

The great support for functional programming, though, does tend to mean that I wrote code in that way more often, which I see as a plus. I like functional programming, despite disliking many functional programming languages, and one of my main complaints with a lot of day-to-day workhorse languages is that they make it harder to write in a functional style than I’d like. Not so with Common Lisp.

Syntax

First things first: yes, there are tons of parentheses; my record in this year’s AoC was 14 closing parens in a row. And arithmetic without infix or suffix operators is a bit annoying. (+ (second start) (abs (- (* 2 (first start)) dest))) is just not as readable as start[1] + abs(2 * start[0] - dest).

But those are relatively minor — they don’t really cause problems in practice. Much more significant is also Lisp’s great innovation: everything is an S-expression.

Because everything is an S-expression, everything is structurally similar to everything else. Function definitions look like function calls; lambdas look like building lists; returning multiple values looks like a hash table lookup. Syntax highlighting helps to a certain extent, but for the most part the eye glazes over Lisp code and it all blurs together, which makes it harder to read and debug.

This is compounded by the fact that the same identifier can mean different things in different places. Common Lisp is a Lisp-2, meaning there are two different symbol namespaces — functions and values — which are used at different times. As well, you also have symbols, which are a type of value that’s normally written 'symbolname but are sometimes written bare, like with the case macro.

I once lost half an hour debugging a nonsensical return value because I had an errant (list in my code, which added an additional level of list nesting that wasn’t supposed to be there and discarded a secondary return value. That was very hard to find, because (list was also in my code legitimately in several places, both because I was constructing lists and because I was switching on the type of objects, and the typecase macro uses bare symbols. In a more ALGOL-like language that had separate syntaxes for constructing lists, switch statements, and function calls, the errant list would have been much more obvious.

Now, I don’t know that this is necessarily the wrong choice. The fact that everything is S-expressions is also what makes Lisp special: you can write macros that manipulate code in the same way you manipulate data, and even though I never wrote one it’s so obviously powerful just from using macros from the standard library that I’m willing to give it the benefit of the doubt. But it is a frustrating facet of the language.

Tooling

Tooling, on the other hand, is the place I most thought to myself, “How have I never heard about this before?!?” It’s incredible, and the principles aren’t Lisp-specific in the slightest, so I have no idea why every language hasn’t shamelessly stolen them.

Everything starts with the REPL, which (at least for sbcl) is useful but not extraordinary by any means. Most languages have a REPL, but using them to assist in writing software is usually annoying. Node’s REPL makes it awkward to import modules or run async code. Python’s REPL can’t edit function definitions sensibly. And every REPL I’ve ever used makes it hard to get the code out and into a source file once it’s working.

SLIME is the opposite. You write your code in your source file like normal, then there’s a keybinding (Ctrl-C Ctrl-C by default) to send the current top-level form — which is usually a function definition — to the REPL environment for evaluation. You then swap to the REPL and call it to make sure it’s doing what you need. And as a bonus, if something you do causes an error, sbcl’s REPL drops you into the debugger rather than just dumping the error to the console.

This enables you to write code in the way that I think is ideal: edit source in your source file using the tools in your text editor, test what you’ve got at any time using the tools in the REPL and debugger. When you’ve got everything just how you want it, there’s nothing more to do; the code is sitting in the source file it’s supposed to be in.

I’m not sure I can do it justice here, but it’s just so smooth. And the most frustrating thing to me is lots of modern languages could do this. Nothing prevents someone implementing this for Python or Node — they both can evaluate code coming from elsewhere, allow redefining functions, have structures that can easily let you determine what definition you’re in, and everything else. And yet, nobody has as far as I know. It’s deeply confusing.

Returning Multiple Values

A small feature I was surprised by is Common Lisp’s method of returning multiple values. Famously, Lisp has lists, so you can of course return multiple values yourself by just returning a list. But Common Lisp also supports an explicit multiple value return form. The key difference is that if you directly use the value from a function that returns multiple values, you only get the first return value.

This makes the second and later return values optional in a way that is very convenient. For instance, the truncate function, which implements integer division, returns the remainder from the division as a second value. So (print (truncate 5 2)) will print 2, but hiding in the shadows is a 1 as a second return value in case it’s needed.

This is cute, and at least as useful as the usual method. There are a lot of functions where this kind of thing shows up. For instance, the gethash function to get values from hash tables returns a true/false indicator as a second value for whether the value existed to allow you to distinguish between missing values and a value of nil stored in the hash table.

loop

I don’t know if the loop macro in Common Lisp is genius or madness. It’s the idiomatic way to do iteration, but the syntax doesn’t resemble Lisp in any way, the number of variations is as long as your arm, and you can mix and match every which way. All of these are valid loops:

And that’s not even the half of it. One (intentionally overblown) example I found looks like this:

(loop for i from 1 to 100
      if (evenp i)
        minimize i into min-even and 
        maximize i into max-even and
        unless (zerop (mod i 4))
          sum i into even-not-fours-total
        end
        and sum i into even-total
      else
        minimize i into min-odd and
        maximize i into max-odd and
        when (zerop (mod i 5)) 
          sum i into fives-total
        end
        and sum i into odd-total
      do (update-analysis min-even
                          max-even
                          min-odd
                          max-odd
                          even-total
                          odd-total
                          fives-total
                          even-not-fours-total))

Everything before the do there is part of the loop! And that do can be replaced with various alternative actions for the loop to take, like collect to collect values into a list or sum to sum then. In writing this section, I realized that many of the loops I wrote for AoC this year are suboptimal; I did something outside the loop that there’s a purpose-built loop keyword to do.

Libraries

The standard library in Lisp is quirky. Someone on Stack Overflow referred to it as a “no batteries included” language, and in a lot of respects, that’s correct. There’s no built-in function to split a string or flatten a nested list, which you can find in almost every language nowadays. But on the other hand, there are functions like 1+ (adds one to its argument), y-or-n-p (prompts the user to answer “y” or “n”), and short-site-name (returns the name of the machine’s location).

However, there is Quicklisp, which is a CPAN-style system for loading libraries that works really well, and everything I needed had a well-regarded library available for it. So it’s not like you need to implement everything yourself.

Overall, I’m a little disappointed. I think I align much more with Go’s philosophy here: the language is better when it provides a really comprehensive standard library you know you can trust. Having to rely on third parties to provide core functionality (let alone highly sensitive libraries like crypto or networking) is a bit of a downer.

Conclusion

I can see why Lisp has its diehard partisans, it’s a very nice language family with a lot of things to like in it, and Common Lisp seems like a rock solid Lisp. I’m actually surprised that it’s not used more than it is. Even though the syntax is unusual, I didn’t really have any problem picking it up, and what it gives you is pretty compelling. I’d recommend you give it a shot if you haven’t, especially if you’re already an Emacs user.

Honestly, the biggest thing holding me back from using it more is just that I don’t know what I’d do with it. I don’t write that much software from scratch these days. But I like Common Lisp enough that I expect my next personal project will use it.