Arc Forumnew | comments | leaders | submitlogin
Thoughts on my first successful macro-generating macro
2 points by akkartik 5354 days ago | 4 comments
(Followup to http://arclanguage.org/item?id=13364, http://arclanguage.org/item?id=13377)

Here's mac in wart from a few commits ago, parsing keyword args, rest args, destructuring, optional args:

  (defmacro$ mac0(name params &body body)
    `(defmacro$ ,name(&rest ,$args)
       ,(compile-params params $args body)))
(defmacro$ simply replaces $args with an appropriate gensym: http://arclanguage.org/item?id=13342)

Inspired by Doug Hoyte, I wanted to implement once-only args so I could say, for example,

  (mac iflet(var o$expr then . rest)
    `(if ,$expr
       (let ,var ,$expr
         ,then)
       ,@rest))
instead of

  (mac iflet(var expr then . rest)
    `(let ,$expr ,expr
       (if ,$expr
         (let ,var ,$expr
           ,then)
         ,@rest)))
o$var automatically gets bound to gensym $var.

Here's how to do that, closely paralleling defmacro! at http://letoverlambda.com/lol.lisp:

  (defmacro$ mac(name params &body body)
    (let* (($os   (remove-if-not #'odollar-symbol-p (strip-rest params)))
           ($s  (mapcar #'odollar-symbol-to-dollar-symbol $os)))
      `(mac0 ,name ,params
         `(let ,(mapcar #'list (list ,@$s) (list ,@$os))
            ,(progn ,@body)))))
I have it, and yet I don't fully understand it. rocketnia helped immeasurably with http://arclanguage.org/item?id=13379[1] so now I get the ,(progn ,@body). It's to allow body to be a (third!) backquoted expression. But I still don't get the (list ,@$s). Why doesn't just $s work?

I think one reason nested backquotes are hard is that they expand into gibberish.

  wart> (macex1 '(mac foo(n) `(+ 1 ,n)))
  (mac0 foo (n)
    `(let* (sb-impl::backq-comma (mapcar #'list (list) (list)))
       ,(progn `(+ ,1 ,n))))
Looking at that you can't tell it's right. You can run the program and see that it works, of course, but when it doesn't work you can't tell if your attempted changes are on the right track. As a result I spent most of my time flailing around without the (list ,@var) pattern in my attempts. As before, any and all tips most appreciated.

[1] The clearest thing you've ever ever written, rocketnia :)



2 points by rocketnia 5354 days ago | link

But I still don't get the (list ,@$s). Why doesn't just $s work?

Suppose $os is the list '(o$a o$b o$c). Then $s is the list '($a $b $c), and the expansion will work like this:

  At macro definition time, while expanding the 'mac macro:
  
  `(mac0 ... `(let ,(mapcar #'list (list ,@$s) (list ,@$os)) ...))
  ==>
  (mac0 ...
    `(let ,(mapcar #'list (list $a $b $c) (list o$a o$b o$c)) ...))
  
  At macro usage time, assuming the expressions to evaluate once are 1,
  2, and 3 and the gensyms are 'a99, 'b99, and 'c99:
  
  `(let ,(mapcar #'list (list $a $b $c) (list o$a o$b o$c)) ...)
  ==>
  (let ((a99 1) (b99 2) (c99 3)) ...)
There are certainly lots of ways to get this wrong:

      At macro definition time:
  `(mac0 ... `(let ,(mapcar #'list $s $os) ...))
  (mac0 ... `(let ,(mapcar #'list $s $os) ...))
      At macro usage time:
  `(let ,(mapcar #'list $s $os) ...)
  Error: The variables '$s and '$os are unbound as non-functions.
  
      At macro definition time:
  `(mac0 ... `(let ,(mapcar #'list ,$s ,$os) ...))
  (mac0 ... `(let ,(mapcar #'list ($a $b $c) (o$a o$b o$c)) ...))
      At macro usage time:
  `(let ,(mapcar #'list ($a $b $c) (o$a o$b o$c)) ...)
  Error: The variables '$a and 'o$a are unbound as functions.
  
      At macro definition time:
  `(mac0 ... `(let ,(mapcar #'list ,@$s ,@$os) ...))
  (mac0 ... `(let ,(mapcar #'list $a $b $c o$a o$b o$c) ...))
      At macro usage time:
  `(let ,(mapcar #'list $a $b $c o$a o$b o$c) ...)
  Error: The values 'a99, 'b99, 'c99, 1, 2, and 3 aren't lists.
  
      At macro definition time:
  `(mac0 ... `(let ,(mapcar #'list ',$s ',$os) ...))
  (mac0 ... `(let ,(mapcar #'list '($a $b $c) '(o$a o$b o$c)) ...))
      At macro usage time:
  `(let ,(mapcar #'list '($a $b $c) '(o$a o$b o$c)) ...)
  (let (($a o$a) ($a o$b) ($c o$c)) ...)
      At runtime, in a place where the macro was used:
  Error: The variables 'o$a, 'o$b, and 'o$c are unbound as
    non-functions.
The tactic I suggest is just always thinking about what "gibberish" you want it to expand to. Find an expanded form that stretches far enough to cover all the use cases of the macro, and work backwards.

  (let ((a99 1) (b99 2) (c99 3)) ...)
We get here by taking the gensyms and the once-only expressions and zipping them together. To do that automatically, we can use (mapcar #'list <something> <something>).

  `(let ,(mapcar #'list <gensyms> <expressions>) ...)
We don't actually have lists of gensyms and expressions at that stage, though. We have variables that hold those things individually. We need to build the lists out of the individual pieces.

  `(let ,(mapcar #'list (list <gs> <gs> <gs>) (list <xp> <xp> <xp>)) ...)
Each gensym is held in a variable of the form '$foo, and each once-only expression is held in the corresponding variable of the form 'o$foo. Although we have direct access to those variables at this stage, we don't have the ability to dynamically choose which variables we use. The variables need to be calculated in the next level up:

  `(mac0 ... `(let ,(mapcar #'list (list ,@<gensym variables>)
                                   (list ,@<expression variables>))
                ...))
Thankfully, we can actually calculate <gensym variables> and <expression variables> directly at this stage, so we do that. Knowing we could get to this stage requires a bit of foresight, but hopefully practice makes it easier.

For the sake of example, there's an alternative approach we could take. Suppose I look at this...

  (let ((a99 1) (b99 2) (c99 3)) ...)
...and I initially think about it as a list of bindings rather than a zip of two corresponding lists. I might construct the list this way...

  `(let ,(list (list <gs> <xp>) (list <gs> <xp>) (list <gs> <xp>)) ...)
...with the intent of generating all those (list ...) calls from the next level up.

  `(mac0 ...
     `(let ,(list ,@(mapcar <function>
                            <list of individual binding creation info>))
        ...))
What function can I use to make a binding? The result needs to be an expression of the form (list <gs> <xp>), where <gs> and <xp> are symbols which resolve to the right variables in the next level down. At this point I might realize I need to zip a <gs> list with an <xp> list, and I'll end up going back to the other solution that way. But the symbol for the <gs> variable is determined from the symbol for the <xp> variable, using #'odollar-symbol-to-dollar-symbol, so I actually just need the <xp> symbol as input to the function, and I can get the list of <xp> symbols the same way you do:

  `(mac0 ...
     `(let ,(list ,@(mapcar (lambda (x)
                              `(list ,(odollar-symbol-to-dollar-symbol
                                        x)
                                     ,x))
                            (remove-if-not #'odollar-symbol-p
                              (strip-rest params))))
        ...))
Patterns like (map [do `(list ,foo._ ,_)] stuff) are pretty common in my Arc code, so this scenario is realistic. In fact, I often try to use quasiquotes iff I'm generating code, so there's a good chance I'd say (map [do ``(,,foo._ ,,_)] stuff) to make the intent clearer. That's right, I might make the intent clearer by using four instances of ` in one definition. Brag brag brag. :-p But really, if it's unclear, I don't know how to avoid it....

I think any helper macro that reduces the quasiquote depth needs at least two quasiquote levels itself (to hide the second level from the outside), but a simpler one can at least take away some of the distraction of the transformation itself:

  (mac0 letlists (vars vals . body)
    `((fn ,vars ,@body) ,@vals))
  
  (mac0 mac ...
    (let os ...
      `(mac0 ...
         `(letlists ,',(map o->dollar os) ,',os
            ...))))
Maybe that's significant enough?

-----

1 point by akkartik 5353 days ago | link

(haven't yet digested your comment)

think about what "gibberish" you want it to expand to

Just to clarify, the gibberish I was referring to was the (sbcl-specific) backq-comma call. It's not gibberish if I know what I want :)

Update: Oh, you're talking about stepping through the code. Yeah, I should just not be lazy and do that. I'm again reminded of that dijkstra comment from yesterday. My mental code simulation skill have atrophied with programming experience. Then again, Dijkstra hated how schools taught their students to step through code pretending to be the computer. Hence his more axiomatic approach to program semantics.

Hmm, I wonder if fallintothis's macro stepping utility would help with this..

Update 2: Ok, digested. You make a lot of sense. It's too bad, already the solution's starting to crystallize in my head as self-evident, and I'm forgetting what it was like to not know how to do this. Your clear comment is not helping :p

The idea that any helper macro would also need nested backquotes is interesting and explains vague memories of past experiences. I'm going to think about that.

Part of the problem is that I've needed nested backticks so infrequently. Perhaps I should just create some contrived examples to exercise this muscle.

-----

1 point by rocketnia 5353 days ago | link

Just to clarify, the gibberish I was referring to was the (sbcl-specific) backq-comma call. It's not gibberish if I know what I want :)

Ah, gotcha. ^_^ I thought that was part of it, but I thought maybe you also meant things like the (list) with no arguments and the (progn ...) with one subexpression. I dunno, I sorta consider that to be gibberish even if we want it. XD

Oh, you're talking about stepping the code.

I guess I am.... Maybe some debug printing would help out. ^^

Actually, maybe something like debug printing is a good way to do unit tests on macros. O_o Put in a meaningless form inside the macro, and have it short-circuit with an escape continuation (or call any other kind of function) if the right test is in progress. You get to verify, collect, or trace intermediate values without managing a spaghetti of one-use functions.

In its simplest form, this could be implemented as a global variable that's usually set to 'idfn.

It's too bad, already the solution's starting to crystallize in my head as self-evident, and I'm forgetting what it was like to not know how to do this.

I was actually really afraid of that. XP Maybe a good reason will dawn on one of us after a while.

Part of the problem is that I've needed nested backticks so infrequently. Perhaps I should just create some contrived examples to exercise this muscle.

The clearest uses of quasiquotation in Arc are 'eval, 'defset, and 'mac. As long as you need to combine two or three of those things, look out. :-p

My trickiest quasiquotation experience was when I wanted (= global!x 2) to effectively do (eval `(= ,'x ,2)) somehow. I forgot that we could actually embed arbitrary values directly into Arc expressions, so I decided to use a temporary global variable and go for (eval `(= ,'x temp)). For encapsulation's sake, I made the global variable a gensym, which meant I had to (eval ...) the whole (defset ...) form. Three layers, and I'm not sure a macro would help.

Once rntz pointed out that I didn't need the temporary variable (http://arclanguage.org/item?id=11612), I changed it to use two layers. But maybe it's still okay as a contrived example. ^_^

My three-layer code, just before I simplified it, is here: https://github.com/rocketnia/lathe/blob/82df9b715f8928cc87cd...

I wonder if quasiquotation layers are something that intrinsically come up infrequently, or if it just has to do with the way 'eval, 'defset, and 'mac come up infrequently. In fact, the reason I defined (= global!x 2) was so that I didn't have to mess with a bunch of quasiquotes each time I wanted to jump up to global scope using 'eval, so my most extreme usage of quasiquotes was a tool to make quasiquotes less frequent.

That example's probably an exception though. I just realized that making a helper macro to remove a layer of quasiquotes is a bit futile because it has to wrap both the root of the quoted code and the missing leaves, essentially becoming another quasiquote. (It could be I'm just not being creative enough.) If the macro abstracts away two layers of quasiquotes, it might be worthwhile, but again, it'll either have all the same "tricks" as two-layered quasiquotes or be restrictive and less general-purpose.

-----

2 points by akkartik 5353 days ago | link

"I thought maybe you also meant things like the (list) with no arguments and the (progn ...) with one subexpression."

Ah, macro generation time vs generated-macro-expansion time :)

I do use debug prints, but they actually seem ineffective for debugging macros. I'm not sure why. Hence the commented-out variants.

"I just realized that making a helper macro to remove a layer of quasiquotes is a bit futile.."

Yeah, I'm not sure either. Perhaps it's not the number of quasiquotes but something else about them. Perhaps there's easy and hard doubled-backticks.

-----