The Beauty of Syntactical Macros

A programming language allows a really constrained form of expression and so (fortunately) it needs lesser training to understand a chunk of rhetorical code, if there is one. There is a limit to how much a programmer can let his/her words fly without breaking the intended low level syntax tree. But that limit definitely is above what I have hit, until now.

I have been playing with Hy, which is a Lispy dialect of Python, recently and started using macros. Put simply, they transform code to code. The input form usually is something we intend to write, the output being something the computer could understand. An example follows. Forget about the visual clutter if you are not familiar with s-expressions in Lisp and just go through the words, nesting and order.

(defmacro query-list [return-cond from source-list where check-cond is item]
  "SQL-ish query on list"
  `(try
    (let [item-index (.index (list (map (fn [it] ~check-cond) ~source-list)) ~item)]
      ((fn [it] ~return-cond) (nth ~source-list item-index)))
    (except [ValueError] False)))

Also forget about the dummy variables (from, where …) and hygiene (I am capturing it) if you are familiar with Lisp. For reference, here is a roughly equivalent code in Python for the code generated by that macro (forget representing this as list comprehension for a while, because its not about a specific language feature but how easily we can add another, possibly specific, feature):

try:
    item_index = list(map(check_cond, source_list)).index(item)
    return return_cond(source_list[item_index])
except ValueError:
    return False

What do I get from this macro? Here is a simple piece which uses this:

(query-list (= "dodo" it.name) from animals where it.extinction is "1660s")

Just by looking at it, you can sort of see what it is intended for. Indeed this is a variant of list comprehension. When I was working on the code that needed this construct, I was thinking of something on the lines of the above example in my mind. Usually, that something would have gotten converted to a function representation which would have a certain set of sensibly arranged and named arguments. But now that I have seen the above valid form, I don't think a function would have done justice to the exact expression in my mind and wouldn't have been that flexible in its usage.

Another example is with the case of docopt based argument parsing. Docopt allows you to write human readable usage instructions for a command line tool and parses it to a structure we can use in our program. Although the arguments passed into the command line can be structured in a nested way, the parsed dictionary returned by docopt is flat. Consider a program with usage instruction like the following (items separated by | are two options for representing the same thing):

program task (sub-task-one | sto)
program task (sub-task-two | stt)
program task-two

After parsing the arguments, docopt gives a dictionary like so:

args = {
    "task": False,
    "sub-task-one": False,
    "sub-task-two": False,
    "sto": False,
    "stt": False,
    "task-two": True
}

Now checking for the values mean using ~if~s & ~else~s. Many times its trivial. Sometimes the nesting can go deep and it becomes messier. When we think about what to do with these arguments, we think in terms of what each possible combination is going to do. We think about doing something when we get task AND (sub-task-one OR sto). Can we directly expose this to our program so that its readable? Surely, we could do

if args["task"] and (args["sub-task-one"] or args["sto"]):
    ...

This approach does more than nesting if s because the first test (args["task"]) is happening many times, but let's not worry about that since its not really a heavy computation. There is a repeating pattern here of args["<>"]. Repeated patterns are good for machines, but not for us. Its not that this tiny piece is hurting the readability, its just that I didn't think about this thing in my mind while planning to code. This thing actually came in when I did the transformations from my plan to a computer acceptable construct. Can we get rid of this then? A quick and simple fix is the following:

if check_args(args, "task", ["sub-task-one", "sto"]):
    ...

What the function check_args does is to collect all the parameters after args and put them in a list (call it *argv). Each of the items in that list is considered to be joined by AND s and the next level of nesting is joined by OR~s. These computations are done by ~getting the corresponding value of the string (the key) from args dictionary. This is fine. Probably will need some level of familiarity with the usage but its okay for this trivial case. What about a deeper level of argument nesting? In that case, for each list at any level, we could add a string representing the operation to apply on the list like and or or. For more complex transformations and when the arguments are not just boolean, instead of adding a string, we can just pass a function like the following:

if check_args(args, [func1, "task", [func2, "subtask", "st"]]):
    ...

See whats happening here? Our arguments are slowly beginning to take form of code themselves. Nothing wrong in that. But this is not really natural for Python and it looks out of place from the rest of the code as it needs a different mental model of whats happening here. Consider the same as a macro in Hy:

(if (check-args args (func1 "task" (func2 "subtask" "st")))
  (...))

The macro is also doing the same transformation of replacing the strings with a getter corresponding to the dictionary args, which is something like (get args "string"). But its doing nothing other than that. Just like I thought about the transformation. My first thought was to just run (func1 "task" (func "subtask" "st")) by using args as the context for interpreting the strings. This is not so in the case of a function. The mental model here is simpler because there is essentially just one, viz. of s-expressions.

The point is this, programming involves transforming our thoughts to code constructs and then writing them. Occasionally the output of our transformations get slightly messy and its feels bad to keep repeating the transformations. Making functions first class citizen puts us one level up while doing these transformations. Making code itself first class citizen puts us even higher. Considering you don't actually think in Python and are transforming your plain thoughts to code, the transformations like in the examples shown above are not at all natural to the approach of just writing functions. Using s-expressions we get a sweet spot of representation between what is sufficiently high level and what is understandable by a computer and allows us this syntactical freedom which is immensely beautiful to peruse, like a rhetorical device.

You shouldn't write a newspaper with poetic constructs because not everyone is looking to untangle a string of pearls every morning, however beautiful they might be. Reading a newspaper is not exactly reading as in "joyfully devouring words", its more of an information gathering mechanism and will go out of fashion if we invent something like an information drink.

There is a certain reason I wasn't seeing the importance of syntactical macros and it is the same reason non-standard constructs are avoided in popular programs. Probably its the same reason it will never be in popular usage for specific domains. A domain has certain needs which, when satisfied, remove the need of syntactical extensibility. SQL queries won't be replaced by Lisp because if you are going to recreate SQL syntax with Lisp, why even bother with the switch in the first place? Its only when you crave for extensibility, which is not very often in practical cases, does it actually reveals its true beauty.

The most popular languages are the ones with most set standards, certain important constraints and a lot of directly visible applications (not implying causality of the opposite side) and they keep gaining traction since its easier to get started and get going. To focus on the real problem we are solving. But sometimes, its good to just lay back and play around with the words till they entertain us on a very personal level. None of what I said is something new which others haven't already said about Lisp. The new thing, is just my personal realization of these facts. I don't actually believe now that I am seeking anything like performance (Hy has certain overheads, although common-lisp is crazy fast if used properly) or better productivity (though this is looking like a very visible long term side effect) with any of the Lisp variants I am using/going to use but just pure beauty. Probably it will stay that way for a long time.