Monday, March 10, 2025

Advanced CLOS — update-instance-for-changed-class

Like most object systems, instances in CLOS have a reference to their class. Unlike most most object systems, CLOS provides a protocol for changing that reference. Normally, this is a pretty insane thing to want to do. It effectively changes the class of the instance and it is pretty unlikely that the instance structure will be compatible with the new class. But there are two situations where you might want to do it anyway:

  • When you edit the class definition, you can arrange for the system to dynamically upgrade existing instances to the new class definition. This means you won't have to restart your lisp and rebuild all the instances from scratch. You can just reload the class definition and the instances will be seamlessly upgraded on the fly. This is much more pleasant experience for the developer.
  • While you normally don't want to change the class of an instance at runtime, there are some rare situations where it can make sense. A good example is the unified table interface. Instances are thin wrappers around a concrete table implementation. It makes sense to change a table instance from one concrete implementation to another. For instance, you might want to change a hash table to a assocation list. You can simply call change-class on the instance.

When the class changes, the representation will be wrong. This is where we add an :after method to update-instance-for-different-class:

(defmethod update-instance-for-different-class :after ((previous alist-table) (current plist-table) &rest initargs)
  (declare (ignore initargs))
  (setf (representation current) (alist-plist (representation previous))))
  
  ...etc...
> (defvar *foo* (make-instance 'alist-table :initial-contents '((:a . 420) (:b . 69))))
#<ALIST-TABLE 2 EQL>

> (representation *foo*)
((:A . 420) (:B . 69))

;; But I'd rather have a plist-table
  
> (change-class *foo* 'plist-table)
#<PLIST-TABLE 2 EQL>

> (representation *foo*)
(:a 420 :b 69)

;; And now I'd like a wttree-table

> (change-class *foo* 'wttree-table)
#<WTTREE-TABLE 2 EQUAL>

> (representation *foo*)
#(2 NIL :A #(1 NIL :B NIL 69) 420)

Naturally, you have to be judicious in your use of this feature of CLOS. You can easily construct nonsense objects. But some times it makes perfect sense,

Sunday, March 9, 2025

Unified table interface

On day 16 of the Advent of Code, I make use of a priority queue for Dijkstra's algorithm. I ported Stephen Adams's weight-balanced binary tree implementation from MIT Scheme to Common Lisp. Stephen Adams's implementation (and therefore my port of it) has the MIT license. Weight-balanced binary trees are a way to implement key-value maps with these properties:

  • The trees are immutable. This means that when you add or remove a key, you get a new tree with the change. The old tree is unchanged. This makes the trees easier to reason about and suitable for functional programming. For example, you can iterate over the tree without having to worry about mutating the tree during the iteration.
  • Most operations on the tree, and insertion, lookup, and deletion in particular, are O(log n). While theoretically not as fast as a hash table, n has to be quite large before log n becomes a big factor. In practice, a weight balanced binary tree is competitive with a hash table for any reasonably sized table.
  • Weight-balanced binary trees support set operations such as union, intersection, and difference. These operations run in O(log n) time as well.
  • Keys are stored in sorted order. This makes it easy to iterate from smallest to largest key (or in the direction).

But it occurred to me that I wanted a unified abstract interface to all the various table-like data structures that Common Lisp provides. You should be able to call a generic table/lookup on a property list, association list, hash table, or weight-balanced binary tree and have it do the right thing. I wrote a simple table package that provides this.

https://github.com/jrm-code-project/table

The package is documented in the `README.md` fie.

Saturday, March 8, 2025

Advent of Code 2024: Day 25

On day 25, we are given a set of locks and keys as ascii art. A typical lock looks like this:

.....
.#...
.##.#
.##.#
###.#
#####
#####

and a typical key looks like this:

#####
#####
##.#.
##.#.
##.#.
#..#.
.....

We read the input file with a little state machine that accumulates lines until a blank line or end of file is reached. It decides whether what it read was a lock or a key by looking to see if the first row is all #'s or not. If it is, it's a key, otherwise it's lock.

(defun read-input (pathname)
  (let ((package (find-package "ADVENT2024/DAY25")))
    (with-open-file (stream pathname)
      (let iter ((line (read-line stream nil))
                 (accum '())
                 (locks '())
                 (keys '()))
        (if line
            (let ((char-list (map 'list (lambda (c) (intern (string c) package)) line)))
              (if (null char-list)
                  (let ((item (make-grid (length accum) (length (first accum))
                                         :initial-contents (reverse accum))))
                    (if (every (lambda (s) (eq s '\#)) (first accum))
                        (iter (read-line stream nil)
                              '()
                              locks
                              (cons item keys))
                        (iter (read-line stream nil)
                              '()
                              (cons item locks)
                              keys)))
                  (iter (read-line stream nil)
                        (cons char-list accum)
                        locks
                        keys)))
            (let ((item (make-grid (length accum) (length (first accum))
                                   :initial-contents (reverse accum))))
              (if (every (lambda (s) (eq s '\#)) (first accum))
                  (values (reverse locks) (reverse (cons item keys)))
                  (values (reverse (cons item locks)) (reverse keys)))))))))

A key fits into a lock (but doesn't necessarily open it) if none of the '#'s in the key overlap with the '#'s in the lock. This is easily checked by iterating over the key and lock in parallel and ensuring that at least one of the characters is '.'.

(defun fits? (key lock)
  (collect-and (#M(lambda (k l)
                    (or (eql k '|.|) (eql l '|.|)))
                  (scan 'array key)
                  (scan 'array lock))))

For part 1, we are asked to find the number of key/lock combinations that result in a fit. We use map-product from the alexandria library to map the fits? predicate over the cartesian product of keys and locks. We then count the number of fits.

(defun part-1 ()
  (multiple-value-bind (locks keys) (read-input (input-pathname))
    (count t (map-product #'fits? keys locks))))

There is no part 2 for this problem.


We've arrived at the end of the 2024 Advent of Code. I started this series with two intents: to demonstrate an approach to solving the problems that is more idiomatic to Common Lisp, and to learn more about the series library. I don't claim my solutions are the best. They could all use some improvement, and I'm sure you code golfers can find numerous ways to shave strokes. But I think each solution is fairly reasonable and tries to show off how to effectively use Common Lisp in a number of simple prolems.

For these problems I purposefully avoided the loop macro and tried to use the series library as much as possible. I used named-let for the more complex iterations.

I was ultimately disappointed in series. I like the idea of automatically generating pipelines from a more functional style, but it simply hits the complexity wall far too quickly. For simple iterations, it's great, but for anything even slightly more complex, it becomes difficult to use.

The full source code I wrote is available on GitHub at https://github.com/jrm-code-project/Advent2024 Be aware that I have not included the puzzle input files. The code will not run without them. You can download the puzzle inputs from the Advent of Code website and put them in the appropriate directories, each in a file called input.txt

I'm curious to hear what you think of my solutions. If you have any comments or suggestions, please feel free to contact me via email or by leaving a comment.

Friday, March 7, 2025

Advent of Code 2024: Day 24

In day 24, we are given a set of equations that decribe some combinatorical logic. The first task is to read the input and parse out the combinatoric circuit and simulate it. To do this, I hijack the lisp reader. I create a readtable this is just like the standard Lisp readtable, but with these differences:

  • Case is not folded.
  • The colon character is no longer a package prefix marker, but rather a terminating macro character that inserts the token :colon into the stream.
  • The newline character is no longer a whitespace character, but rather a terminating macro character that inserts the token :newline into the stream.

These changes to the reader make it esay to parse the input file. We build a labels expression where each named quantity in the circuit (the wires) is a function of zero arguments. Simulating the solution is then just a matter of calling eval on the resulting expression.

(defun get-input (swaps input-pathname)
  (flet ((maybe-swap (symbol)
           (cond ((assoc symbol swaps) (cdr (assoc symbol swaps)))
                 ((rassoc symbol swaps) (car (rassoc symbol swaps)))
                 (t symbol))))

    (let ((*readtable* (copy-readtable nil)))
      (setf (readtable-case *readtable*) :preserve)
      (set-syntax-from-char #\: #\;)
      (set-macro-character #\: (lambda (stream char) (declare (ignore stream char)) :colon))
      (set-macro-character #\newline (lambda (stream char) (declare (ignore stream char)) :newline))

      (with-open-file (stream input-pathname :direction :input)
        (let iter ((token (read stream nil :eof))
                   (line '())
                   (gates '())
                   (wires '())
                   (outputs '()))
        
          (multiple-value-bind (line* gates* wires* outputs*)
              (if (or (eq token :eof) (eq token :newline))
                  (if line
                      (if (member :colon line)
                          (values '()
                                  gates
                                  (cons `(,(third line) () ,(first line)) wires)
                                  outputs)
                          (values '()
                                  (cons `(,(maybe-swap (first line)) ()
                                          (,(ecase (fourth line)
                                              (XOR 'logxor)
                                              (OR 'logior)
                                              (AND 'logand))
                                           ,@(list (list (third line)) (list (fifth line)))))
                                        gates)
                                  wires
                                  (if (and (symbolp token)
                                           (char= (char (symbol-name token) 0) #\z))
                                      (cons `(list ,(list token)) outputs)
                                      outputs)
                                  ))
                      (values '() gates wires outputs))
                  (values (cons token line) gates wires (if (and (symbolp token)
                                                                 (char= (char (symbol-name token) 0) #\z))
                                                            (cons (list token) outputs)
                                                            outputs)))
            (if (eq token :eof)
                `(labels (,@wires*
                          ,@gates*)
                   (fold-left (lambda (acc bit)
                                (+ (* 2 acc) bit))
                              0  (list ,@(sort outputs* #'string-greaterp :key (lambda (term) (symbol-name (car term)))))))
                (iter (read stream nil :eof) line* gates* wires* outputs*))))))))

For part 2, we are told that the circuit is supposed to add two binary numbers. We are also told that the circuit the circuit has four of its wires swapped. We are asked to find the swapped wires.

It is hard to understand what is going on because almost all the wires have random three-letter names. We start by renaming the wires so that they have a bit number prefixed to with them. If a gate has two numbered inputs where the numbers are equal, we propagate the number to the output of the gate.

Once the wires are numbered, we sort the wires by their numbers and print the wire list. The regular pattern of gates is instantly obvious, and the swapped wires are easy to spot. It isn't obvious how to find the swapped wires in the general case, but it is unnecessary to solve the puzzle, so there is no code for this.

Thursday, March 6, 2025

Advent of Code 2024: Day 23

For day 23 we’re going to look for cliques in a graph. A clique is a subset of vertices in a graph such that every pair of vertices in the clique is connected by an edge. In other words, a clique is a complete subgraph of the graph.

The graph is given as a list of edges. The graph is undirected, so the edge (a, b) is the same as the edge (b, a). We represent the graph as a hash table mapping vertices to a list of adjacent vertices.

;;; -*- Lisp -*-

(in-package "ADVENT2024/DAY23")

(defun get-input (input-pathname)
  (let ((neighbor-table (make-hash-table :test #’eql))
        (package (find-package "ADVENT2024/DAY23")))
    (iterate (((left right) (#2M(lambda (line) (values-list (str:split #\- line)))
                                (scan-file input-pathname #’read-line))))
      (let ((left*  (intern (string-upcase left)  package))
            (right* (intern (string-upcase right) package)))
        (push right* (gethash left* neighbor-table ’()))
        (push left* (gethash right* neighbor-table ’()))))
  neighbor-table))

Given a neighbor table, we can get a list of the two vertex cliques by looking at the keys and values of the hash table.

(defun two-vertex-cliques (neighbor-table)
  (collect-append
   (mapping (((vertex neighbors) (scan-hash neighbor-table)))
     (mappend (lambda (neighbor)
                (when (string-lessp (symbol-name vertex) (symbol-name neighbor))
                  (list (list vertex neighbor))))
              neighbors))))

Given a two vertex clique, we can find a three vertex clique by looking for a vertex that is connected to both vertices in the two vertex clique. We find the neighbors of each vertex in the clique and then take the intersection of the two lists of neighbors. We distribute this intersection over the two vertex clique to get the list of three vertex cliques. Note that each three vertex clique will appear three times in the list in different orders.

In Part 1, we count the number of three vertex cliques in the graph where one of the vertices begins with the letter ‘T’. We divide by three because we generate three vertex cliques in triplicate.

(defun part-1 ()
  (/ (count-if (lambda (clique)
                 (find-if (lambda (sym)
                            (char= #\T (char (symbol-name sym) 0)))
                          clique))
               (let ((neighbor-table (get-input (input-pathname))))
                 (mappend (lambda (clique)
                            (let ((left-neighbors (gethash (first clique) neighbor-table))
                                  (right-neighbors (gethash (second clique) neighbor-table)))
                              (map ’list (lambda (common-neighbor) (list* common-neighbor clique))
                                   (intersection left-neighbors right-neighbors))))
                          (two-vertex-cliques neighbor-table))))
     3))

For Part 2, we are to find the largest maximal clique. We use the Bron-Kerbosch algorithm to find the maximal cliques.

(defun bron-kerbosch (graph-vertices clique more-vertices excluded-vertices)
  (if (and (null more-vertices) (null excluded-vertices))
      (list clique)
      (let iter ((answer '())
                 (excluded-vertices excluded-vertices)
                 (more-vertices more-vertices))
        (if (null more-vertices)
            answer
            (let* ((this-vertex (car more-vertices))
                   (more-vertices* (cdr more-vertices))
                   (neighbors (gethash this-vertex graph-vertices)))
              (iter (append (bron-kerbosch graph-vertices
                                           (adjoin this-vertex clique)
                                           (intersection more-vertices* neighbors)
                                           (intersection excluded-vertices neighbors))
                            answer)
                (adjoin this-vertex excluded-vertices)
                more-vertices*))))))

(defun maximal-cliques (graph-vertices)
  (bron-kerbosch graph-vertices ’() (hash-table-keys graph-vertices) ’()))

Once we have found the maximal cliques, we can find the largest clique by sorting the cliques by length and taking the first one. We sort the vertices in the clique and print as a comma separated list.

(defun part-2 ()
  (format
   nil "~{~a~^,~}"
   (sort
    (first
     (sort
      (maximal-cliques (get-input (input-pathname)))
      #’> :key #’length))
    #’string-lessp :key #’symbol-name)))

Wednesday, March 5, 2025

Advent of Code 2024: Day 22

On Day 22 we are introduced to a simple pseudo-random number generator (PRNG) that uses this recurrance to generate pseudo-random numbers:

S1 = ((Xn << 6) ⊕ Xn) mod 224
S2 = ((S1 >> 5) ⊕ S1) mod 224
Xn+1 = ((S2 << 11) ⊕ S2) mod 224

We just define this as a simple function, but we are carful to put a check-type on the input to make sure it is a number in the correct range. This gives the compiler enough information to optimize the body of the generator to a sequence of inline fixed-point operations, avoid the overhead of a function call out to the generic arithmetic.

(defun next-pseudorandom (pseudorandom)
  (check-type pseudorandom (integer 0 (16777216)))
  (macrolet ((mix (a b) ‘(logxor ,a ,b))
             (prune (x) ‘(mod ,x 16777216)))
    (let* ((s1 (prune (mix (* pseudorandom 64) pseudorandom)))
           (s2 (prune (mix (floor s1 32) s1)))
           (s3 (prune (mix (* s2 2048) s2))))
      s3)))

We can generate a series of random numbers from a given seed:

(defun scan-pseudorandom (seed)
  (declare (optimizable-series-function))
  (scan-fn '(integer 0 (16777216))
           (lambda () seed)
           #'next-pseudorandom))

The nth pseudorandom number is the nth element in the series, i.e. the result of applying the next-pseudorandom function n times to the seed:

(defun nth-pseudorandom (seed n)
  (collect-nth n (scan-pseudorandom seed)))

Part 1 of the problem is to sum the 2000th pseudorandom numbers generated from seeds given in a file.

(defun part-1 ()
  (collect-sum (#Mnth-pseudorandom (scan-file (input-pathname)) (series 2000))))

For part 2, we're going to be simulating a market. The prices are single digit pseudorandom numbers:

(defun scan-prices (seed)
  (declare (optimizable-series-function))
  (#Mmod (scan-pseudorandom seed) (series 10)))

The bidders in our market are monkeys, and we read them from our input file:

(defun scan-monkeys (input-pathname)
  (declare (optimizable-series-function 2))
  (cotruncate (scan-range :from 0)
              (scan-file input-pathname)))

The seed that we read from the input pathname will be used to create a price series for each monkey.

Each monkey looks for trends in the market by looking at the last four price changes. If the last four prices changes match the trend the monkey looks for, the monkey will make a trade and get a profit of the current price.

For part 2, we assume all the monkeys look for the same trend. Some trend will maximize the total profit of all the monkeys. We want to know what that maximum profit is.

We'll proceed in two steps. First, we make a table that maps trends to profits for each monkey. We'll start with an empty table, then we'll iterate over the monkeys, adding the trend info for that monkey. Once we have the table, we'll iterate over all the possible trends and find the one that maximizes the total profit.

price-deltas is a series of the differences between the prices in the price series. We'll use this to determine the trend.

(defun price-deltas (price-series)
  (declare (optimizable-series-function)
           (off-line-port price-series))
  (mapping (((before after) (chunk 2 1 price-series)))
     (- after before)))

price-trends is a series of trends. The trend is simply a list of the last four price deltas.

(defun price-trends (price-series)
  (declare (optimizable-series-function)
           (off-line-port price-series))
  (mapping (((d1 d2 d3 d4) (chunk 4 1 (price-deltas price-series))))
           (list d1 d2 d3 d4)))

add-trend-info! adds the trend info for a monkey to the table. We'll look at a count of 2000 prices (minus the first four because there aren't enough to establish a trend). The key to an entry in the table will be taken from the price-trends. The value for an entry is the price after that trend. The table maps a trend to an alist that maps monkeys to profits, so once we know the trend, we look to see if an entry for the monkey already exists in the value. If it does, we're done. But if it doesn't, we add an entry for the monkey with the profit.

(defun add-trend-info! (table monkeyid seed)
  (iterate ((count (scan-range :from 4 :below 2001))
            (trend (price-trends (scan-prices seed)))
            (value (subseries (scan-prices seed) 4)))
    (declare (ignore count))
    (unless (assoc monkeyid (gethash trend table '()))
      (push (cons monkeyid value) (gethash trend table '())))))

Once we have added the trend info for all the monkeys, we find the entry in the table that maximizes the total profit.

(defun trend-table-maximum (table)
  (let ((best-score 0)
        (best-key nil))
    (maphash (lambda (key value)
               (let ((score (reduce #'+ (map 'list #'cdr value))))
                 (when (> score best-score)
                   (setq best-key key)
                   (setq best-score score))))
             table)
    (values best-key best-score)))

Finally, we can put it all together in the part-2 function:

(defun part-2 ()
  (multiple-value-bind (best-key best-value)
      (let ((table (make-hash-table :test #'equal)))
        (iterate (((monkeyid seed) (scan-monkeys (input-pathname))))
          (add-trend-info! table monkeyid seed))
        (trend-table-maximum table))
    (declare (ignore best-key))
    best-value))

Tuesday, March 4, 2025

Collate / index-list

I was talking to Arthur Gleckler last night and he mentioned that he had been making good use of a function he called index-list. This function takes two selector functions and a list of objects. The first selector extracts a key from each object, and the second selector extracts a value. A table is returned that maps the keys to a list of all the values that were associated with that key.

I had to laugh. I had written the same function a few month back. I called it collate.

Here is Arthur’s version in Scheme:

(define (index-list elements table choose-data choose-key)
  "Given a hash table ‘table’, walk a list of ‘elements’ E, using
‘choose-key’ to extract the key K from each E and ‘choose-data’ to
extract a list of data D from each E.  Store each K in ‘table’ along
with a list of all the elements of all the D for that K."
  (do-list (e elements)
    (hash-table-update!/default
     table
     (choose-key e)
     (lambda (previous) (append (choose-data e) previous))
     ’()))
  table)

And here is my version in Common Lisp:

(defun collate (list &key (key #’car) (test #’eql)
                               (merger (merge-adjoin :test #’eql)) (default nil))
  (let ((table (make-hash-table :test test)))
    (dolist (element list table)
      (let ((key (funcall key element)))
        (setf (gethash key table)
              (funcall merger (gethash key table default) element))))))

So how do they differ?

  • Arthur’s version takes the hash table as a parameter. This allows the caller to control the hash table’s properties. My version creates a hash table using the test parameter, which defaults to eql.
  • Arthur’s version uses choose-key to extract the key from each element. My version uses key, which is a keyword parameter defaulting to car. My choice was driven by the convention of Common Lisp sequence functions to take a :key parameter.
  • Arthur’s version uses a default value of ’() for the entries in the hash table. My version uses the :default keyword argument that defaults to ’().
  • Arthur’s version uses choose-data to extract the datum in each element. My version uses the :merger keyword argument to specify how to merge the entire element into the table. If you only want a subfield of the element, you can compose a selector function with a merger function.
  • Arthur’s version uses append to collect the data associated with each element. My version uses a merger function to merge the element into the entry in the hash table. The default merger is merge-adjoin, which uses adjoin to add the element to the list of elements associated with the key. merge-adjoin is paramterized by a test function that defaults to eql. If the test is true, the new data is not merged, so the result of (merge-adjoin #’eql) is a list with no duplicates.
  • If you instead specify a default of 0 and a merger of (lambda (existing new) (+ existing 1)) you get a histogram.
  • Another merger I make use of is merge-unique, which ensures that all copies of the data being merged are the same, raising a warning if they are not.
  • Finally, I occasionally make use of a higher-order merger called merge-list that takes a list of mergers and applies them elementwise to two lists to be merged. This allows you to create a singleton aggregate merged element where the subfields are merged using different strategies.

Like Arthur, I found this to be a very useful function. I was auditing a data set obtained from GitHub. It came in as a flat list of records of users. Each record was a list of GitHub org, GitHub ID, and SAML/SSO login. Many of our users inadvertently have multiple GitHub IDs associated with their accounts. I used my collate function to create a table that mapped SAML/SSO login to a list of all the GitHub IDs associated with that login, and the list of orgs where that mapping applies.