I was thinking about that and wanted to find a link earlier, but I can't find it. I hope someone will; I want to add a credit to that idea in Parendown's readme.

I seem to remember it being one of the early ideas for Arc that didn't make it to release, but maybe that's not right.

---

"I do like far better the idea of replacing withs with a '/' ssyntax for let."

I like it pretty well!

It reminds me of a `lets` macro I made in Lathe for Arc:

  (lets
    a (bind 1)
    (expr 1)
    b (bind 2)
    (expr 2)
    c (bind 3)
    (expr 3))

This right here

    (expr 1)
    b (bind 2)

       (expr 1)
    b  (bind 2)

With the syntax you're showing there, you at least have a punctuation character at the start of a line, which I think successfully breaks the illusion of nested parens for me.

---

It looks like I implemented `lets` because of this thread: http://arclanguage.org/item?id=11934

And would you look at that, ylando's macro (scope foo @ a b c @ d e f) is a shallow version of Parendown's macro (pd @ foo @ a b c @ d e f). XD (The `pd` macro works with any symbol, and I would usually use `/`.) I'm gonna have to add a credit to that in Parendown's readme too.

Less than a year later than that ylando thread, I started a Racket library with a macro (: foo : a b c : d e f): https://github.com/rocketnia/lathe/commit/afc713bef0163beec4...

mmm, not sure. It'd probably be easier to start with a working version (even if slow) and then remove source information from lists and see if anything breaks.

> In `load`, use `read-syntax`, extract the line number from that syntax value

That's a good point that I hadn't considered, thanks. Yeah, I guess I'm not as opposed to obscuring containment as I'd hypothesized ^_^

Maybe my reaction to (foo /a b c /d e f) is akin to that of a more hardcore lisper when faced with indentation-based s-expressions :)

Isn't there a dialect out there that uses a different bracket to mean 'close all open parens'? So that the example above would become (foo (a b c (d e f]. I can't quite place the memory.

I do like far better the idea of replacing withs with a '/' ssyntax for let. So that this:

  (def foo ()
    /a (bind 1)
    (expr 1)
    /b (bind 2)
    (expr 2)
    /c (bind 3)
    (expr 3))

  (def foo ()
    (let a (bind 1)
      (expr 1)
      (let b (bind 2)
        (expr 2)
        (let c (bind 3)
          (expr 3)))))

Where do you actually need source location information in order to get Arc function names to show up in the profiler?

Would it be okay to track it just on symbols, bypassing all this list conversion and almost all of the Arc built-ins' unwrapping steps (since not many operation have to look "inside" a symbol)?

If you do need it on cons cells, do you really need it directly on the tail cons cells of a macro body? I'd expect it to be most useful on the cons cells in functional position. If you don't need it on the tails, then it's no problem when the `apply` strips it.

Oh, you know what? How about this: In `load`, use `read-syntax`, extract the line number from that syntax value, and then use `syntax->datum` and expand like usual. While compiling that expression, turn `fn` into (let ([fn-150 (lambda ...)]) fn-150) or (procedure-rename (lambda ...) 'fn-150), replacing "150" here with whatever the source line number is. Then the `object-name` for the function will be "fn-150" and I bet it'll appear in the profiling data that way, which would at least give you the line number to work with.

If you want, and if that works, you can probably have `load` do a little bit of inspection to see if the expression is of the form (mac foo ...) or (def foo ...), which could let you create a more informative function name like `foo-150`.

I was curious, has anyone implemented "internal definitions", that is, a way to introduce a variable like `let`, but without having to indent the code that uses it?

For example, I may have a function like:

    (def foo ()
      (let a (...)
        (do-something a)
        (let b (... a ...)
          (do-something-else b)
          (let c (... b ...)
            etc...))))

In Racket, this could be written

    (define (foo)
      (define a (...))
      (do-something a)
      (define b (... a ...))
      (do-something-else b)
      (define c (... b ...))
      etc...)

In Arc, I'd like to use `var`, like this:

    (def foo ()
      (var a (...))
      (do-something a)
      (var b (... a ...))
      (do-something-else b)
      (var c (... b ...))
      etc...)

Thus, for example, `(fn () a b c)` would expand into `($fn () (do a b c))`, where `$fn` is the primitive, builtin function form.

Many macros such as `def` and `let` that have bodies expand into a `fn`s, and this change would mean that the body of these forms would also be a `do`.

For example, `(def foo () a b c)` expands (roughly) into `(assign foo (fn () a b c)`; which in turn would expand into `(assign foo ($fn () (do a b c)))`

Thus, in most places where we have a body where we might like to use `var`, there'd be a `do` in place which could implement it.

Why go to this trouble? `var` can't be a macro since a macro can only expand itself -- it doesn't have the ability to manipulate code that appears after it. However `do`, as a macro, can do whatever it wants with the code it's expanding, including looking to see if one of the expressions passed to it starts with `var`.

As macro, this could be an optional feature. Rather than being built in to the language where it was there whether you wanted it or not, if you didn't like internal definitions you wouldn't have the load the version of `do` which implemented `var`.

A further complication is what to do with lexical scope. If I write something like,

    (let var (fn (x) (prn "varnish " x)
      (var 42))

Just to clarify, in my original design a `var` would expand into a `let` (with the body of the let extending down to the bottom of the enclosing form), and thus the definitions wouldn't have the same scope.

Which isn't to say we couldn't do something different of course :)

- I'd like to be able to use the Racket profiler.

- For the profiler to be useful, functions need to be labeled in some way so that the profiler will show which functions are which.

- To identify functions, I need to propagate source location information from an Arc source code file through to Racket, which includes propagating the source location information through Arc macros.

One option would be to create a macro system designed to propagate source location information through macros. This of course is what Racket does.

In Arc, macros are defined with list primitives (car, cdr, cons), function calls (for example, `(mac let (var val . body) ...)`), and operations that can be built out of those (such as quasiquotation).

My hypothesis is that by allowing list primitives to operate on forms labeled with source location information, we can continue to use Arc macros.

This leads to an interesting issue however...

Consider

    (apply (fn args args) xs)

In Arc 3.2, and in my implementation, an Arc function compiles into a Racket function. E.g., `(fn args args)` becomes a Racket `(lambda args args)`.

Of course we don't have to do that. If we were writing an interpreter, for example, an Arc function would compile into some function object that'd be interpreted by the host language... an Arc function wouldn't turn into something that could be called as a Racket function directly.

But, if `(fn args args)` is implemented as a Racket function `(lambda args args)`, then to call the function with some list `xs` we need to use Racket's apply. But, of course, Racket's apply takes a Racket list. So in Arc 3.2, Arc's apply calls Racket's apply after translating the Arc list into a Racket list.

Leaving out a couple of steps, what in essence we end up with in Racket is the equivalent of:

    (apply (lambda args args) (ar-nil-terminate xs))

Now, for Amacx, I've invented my own representation for Arc lists. In my version, lists (that is, cons cells) can be labeled with where in a source code file they originated from. For example, if I read "(a b c)" from a file, I can inspect that list for source location information:

    > (prn x)
    (a b c)
    > (dump-srcloc x)
    foo.arc:1.0 (span 7) (a b c)
      foo.arc:1.1 (span 1) a
      foo.arc:1.3 (span 1) b
      foo.arc:1.5 (span 1) c

This is entirely internally consistent. E.g. (cdr x) returns a value which contains both the tail of the list `(b c)` and the source location information for the sublist.

But.

In a macro,

    (mac let (var val . body)
      `(with (,var ,val) ,@body))

    (mac let (var val . body)
      (dump-srcloc body)
      `(with (,var ,val) ,@body))

Why?

Because I stripped it.

    (apply (lambda args args) (ar-nil-terminate xs))

    > (apply (lambda args args) xs)
    ; apply: contract violation
    ;   expected: list?

So, a macro like `let` turns into the equivalent of

    (annotate 'mac
      (lambda (var val . body)
        ...))

Of course, like I said, I don't have to implement an Arc rest argument with a Racket rest argument. An Arc function that took a rest argument could turn into some other kind of object where I'd pass in the rest argument myself.

But that would be slower. Probably.

I can get the profiler to work (I think), but then I'd be profiling the slower version of the code.

Though the runtime that implements the extended form of lists with source location information is slower anyway because all of Arc's builtins need to unwrap their arguments.

I realize you also just said the example's bindings aren't mutually recursive, but I think if `a` and `c` "have the same scope" then `b` could depend on `c` as easily as it depends on `a`, so it seems to me we're talking about a design for mutual recursion. So what you're saying is the example could have been mutually recursive but it just happened not to be in this case, right? :)

Yeah, I think the most useful notions of "local definitions" would allow mutual recursion, just like top-level definitions do. It was only the specific, non-mutually-recursive example that led me to bring up Parendown at all.

---

The rest of this is a response to your points about Parendown. Some of your points in favor of not using Parendown are syntactic regularity, the clarity of containment relationships, and the avoidance of syntaxes that have little benefit other than saving typing. I'm a little surprised by those because they're some of the same reasons I use Parendown to begin with.

Let's compare the / reader macro with the `withs` s-expression macro.

I figure Lisp's syntactic regularity has to do with how long it takes to describe how the syntax works. Anyone can write any macro they want, but the macros that are simplest to describe will be the simplest to learn and implement, helping them propagate across Lisp dialects and thereby become part of what makes Lisp so syntactically regular in the first place.

- The / macro consumes s-expressions until it peeks ) and then it stops with a list of those s-expressions.

- The `withs` macro expects an even-length binding list of alternating variables and expressions and another expression. It returns the last expression modified by successively wrapping it lexical bindings of each variable-expression pair from that binding list, in reverse order.

Between these two, it seems to me `withs` takes more time and care to document, and hence puts more strain on a Lisp dialect's claim to syntactic regularity. (Not much more strain, but more strain than / does.)

A certain quirk of `withs` is that it creates several lexical scopes that are not surrouded by any parentheses. In this way, it obscures the containment relationship of those lexical scopes.

If we start with / and `withs` in the same language, then I think the only reason to `withs` is to save some typing.

So `withs` not only doesn't pull its weight in the language, but actively works against the language's syntactic regularity and containment relationship clarity.

For those reasons I prefer / over `withs`.

And if I don't bother to put `withs` in a language design, then whenever I need to write sequences of lexical bindings, I find / to be the clear choice for that.

Adding `withs` back into a language design isn't a big deal on its own; it's just slightly more work than not doing it, and I don't think its detriment to syntactic regularity and containment clarity are that bad. I could switch back from / to `withs` if it was just about this issue.

https://news.ycombinator.com/item?id=18842902

particularly:

a) Use indentation.

b) If indentation isn't absolutely clear, just use parens.

Here's another comment that may help clarify what I mean: https://news.ycombinator.com/item?id=8503353#8507385

(I have no objection to replacing parens with say square brackets or curlies or something. I just think introducing new delimiters isn't worthwhile unless we're getting something more for it than just minimizing typing or making indentation uniform. Syntactic uniformity is Lisp's strength.)

I can certainly see how I would feel differently when programming in certain domains (like CPS). But I'm not convinced ideas that work in a specific domain translate well to this thread.

"What's that about "the impression of a single unified block"? I think I could see what you mean if the bindings were all mutually recursive like `letrec`, because then nesting them in a specific order would be meaningless; they should be in a "single unified block" without a particular order. I'm not recommending Parendown for that, only for aw's original example with nested `let`."

Here's aw's original example, with indentation:

    (def foo ()
      (var a (...))
      (do-something a)
      (var b (... a ...))
      (do-something-else b)
      (var c (... b ...))
      etc...)

(I also like that aw is happy with parens ^_^)

Edit: Reflecting some more, another reason for my reaction is that you're obscuring containment relationships. That is inevitably a leaky abstraction; for example error messages may require knowing that `#/let` is starting a new scope. In which case I'd much rather make new scopes obvious with indentation.

I'd love to hear lots of feedback about how well or poorly Parendown works for people. :)

I'm trying to think of what kind of feedback would actually get me to change it. :-p Basically, I added it because I found it personally helped me a lot with maintaining continuation-passing style code, and the reason I found continuation-passing style necessary was so I could implement programming languages with different kinds of side effects. There are certainly other techniques for managing continuation-passing style, side effects, and language implementation, and I have some in mind to implement in Racket that I just haven't gotten to yet. Maybe some combination of techniques will take Parendown's place.

---

If it's just the #/ you object to, yeah, my preferred syntax would be / without the # in front. I use / because it looks like a tilted paren, making the kind of zig-zag that resembles how the code would look if it used traditional parens:

  (
    (
      (
  /
  /
  /

Do you think I should switch Parendown to use / like I really want to do? :) Is there perhaps another character you think would be better?

---

What's that about "the impression of a single unified block"? I think I could see what you mean if the bindings were all mutually recursive like `letrec`, because then nesting them in a specific order would be meaningless; they should be in a "single unified block" without a particular order. I'm not recommending Parendown for that, only for aw's original example with nested `let`. Even in Parendown-style Racket, I use local definitions for the sake of mutual recursion, like this:

  /...
  /let ()
    (define ...)
    (define ...)
    (define ...)
  /...

At the moment, Amacx loads arc.arc unpleasantly slowly. And I don’t even have all of arc.arc included yet.

I imagine it’s probably because the compiler is slow. (What else could it be?)

And I imagine there’s a good chance the compiler is slow because as the compiler recursively works its way down into forms being compiled, I functionally extend the compilation context in a simplistic way:

    (def functional-extend (g nk nv)
      (fn (k)
        (if (is k nk) nv (g k))))

I hope I can use the Racket profiler, if I can propagate function names and/or function source locations from Arc to Racket.

At the moment a typical profile trace looks like:

                                    for-loop [72]                                          5.1%
                                    ??? [1]                                               38.9%
                                    ??? [12]                                              56.0%
     [12] 1456(30.9%)  104(2.2%)  ??? /Users/andrew/w/amacx/arc/runtime.rkt:409:9
                                    ??? [12]                                              56.0%
                                    ??? [1]                                               42.2%

Today I ran into a weird issue. I was running the Racket profiler in DrRacket, and it worked at first, but then started to hang. So I closed DrRacket and restarted it, deleted all my `compiled` directories, reverted my source code back to an earlier version… and it still hung.

So I don’t know why it worked at first or why it stopped working.

It can't be a regular Arc macro, but Arc comes with at least two things I'd call macro systems already: `mac` of course, but also `defset`. Besides those, I keep thinking of Arc's destructuring syntax and reader syntax as macro systems, but that's just because they could easily be extended to be.

I would say it makes sense to add another macro system for local definitions. As a start, it could be a system where the code (do (var a 1) (var b 2) (+ a b)) would expand by calling the `var` macro with both the list (a 2) and the list ((var b 2) (+ a b)).

But I think that's an incomplete design. All it does is help with indentation, and I feel like a simpler solution for indentation is to have a "weak opening paren" syntactic sugar where (a b #/c d) means (a b (c d)):

  (def foo ()
    (let a (...)
    #/do (do-something a)
    #/let b (... a ...)
    #/do (do-something-else b)
    #/let c (... b ...)
      etc...))

(For posterity, here's a link specifically to the version of that readme at the time of this post: https://github.com/lathe/parendown-for-racket/blob/23526f8f5...)

While I think indentation reduction is one of the most important features internal definition contexts have, that benefit is less important once Parendown is around. I think their remaining benefit is that they make it easy to move code between the top level and local scopes without modifying it. I made use of this property pretty often in JavaScript; I would maintain a programming language runtime as a bunch of top-level JavaScript code and then have the compiler wrap it in a function body as it bundled that runtime together with the compiled code.

In Racket, getting the local definitions and module-level definitions to look alike means giving them the same support for making macro definitions and value definitions at every phase level, with the same kind of mutual recursion support.

They go to a lot of trouble to make this work in Racket, which I think ends up making mutual recursion into one of the primary reasons to use local definitions. They use partial expansion to figure out which variables are shadowed in this local scope before they proceed with a full expansion, and they track the set of scope boundaries that surround each macro-generated identifier so that those identifiers can be matched up to just the right local variable bindings.

The Arc top level works much differently than the Racket module level. Arc has no hygiene, and far from having a phase distinction, Arc's top level alternates between compiling and running each expression. To maintain the illusion that the top level is just another local scope, almost all local definitions would have to replicate that same alternation between run time and compile time, meaning that basically the whole set of local definitions would run at compile time. So local definitions usually would not be able to depend on local variables at all. We would have to coin new syntaxes like `loc=` and `locdef` for things that should interact with local variables, rather than using `=` and `def`.

Racket errors tend to report whatever location information they can, and Racket ports track four things: The name of the file, the line number, the column number, and the overall position in the stream. But line-counting isn't enabled by default on Racket streams, so the error you see doesn't contain the line number information.

I just pushed a commit[2] to Anarki that enables line-counting on the stream when an Arc files is loaded, which should mean you can now see the line and column of a reader error. A reader error on line 17, column 6 should now be printed like so:

  sample-file.arc:17:6: read: expected a closing "

For example, in Arc 3.2, `cons` creates an immutable Racket pair, `car` works with a Racket pair or a Racket symbol nil, and `scar` modifies an Arc cons with Racket's `unsafe-set-mcar!`.

These are all runtime decisions. We could create a different runtime. For example, Arc's `cons` could create a Racket mpair, and `scar` could use Racket's `set-mcar!`.

For Amacx I've written two runtimes so far. One I called "mpair" because it implements Arc lists using Racket's mpairs. The other I called "srcloc" because it allows source location information to be attached to Arc forms (lists and atoms).

Currently, in srcloc, source location information is attached to forms using Racket's syntax objects. Thus, in srcloc, Arc's `car` can be applied to a Racket syntax object which wraps a list or nil, and it will unwrap the syntax object and return the underlying value.

A container can be a simple Arc table, or some object like a Racket namespace which can be made to act like an Arc table. So suppose I have a container `c`, and I eval some code in c:

    (eval '(def foo () 123) c)

    > (c!foo)
    123

When I implemented Arc in Lumen (more info on Lumen: http://arclanguage.org/item?id=20935), this technique was ported over naturally. You can play with it here:

  git clone https://github.com/lumen-language/lumen
  git checkout 2018-10-29/arc
  npm i
  rlwrap bin/lumen-node # rlwrap is optional
  > (load "arc.l")
  function
  > (mac awhen (cond . body)
      `(let it ,cond
         (when it ,@body)))
  > (awhen 'hi (print it))
  hi
  > (awhen nil (print 'hi))
  > (awhen 1 (print it))
  1
  > (awhen 1 (print it)
      (let-macro ((awhen (cond . body)
                    `(let it ,cond
                        (unless it ,@body))))
        (awhen false (print it)))
      (awhen 42 (print it)))
  1
  false
  42
  >

https://imgur.com/a/CUccgz1

  (define-macro when (condition rest: body)
    `(if ,condition (do ,@body)))
  (when true
    (list 42
    (let-macro ((when (condition rest: body)
                  `(if (not ,condition) (do ,@body))))
      (when false
        21))
     (when true
       21)))

If there is interest, I can port the technique to arc3.2 and let the community take it from there. We're using arc3.2 over at https://www.laarc.io

I think what I describe there pretty much meshes with what you and waterhouse are describing here. :)

---

"And then "anarki-test" would be evaluated at compile time? Hmm..."

Yeah, I think basically every instance of local macros I've seen involves evaluating the expression at compile time.

That's even what Racket's `let-syntax` does, although in Racket's case it involves a little more detail since Racket enforces a strict separation between compile-time and run-time side effects. When Racket evaluates the expression at compile time (usually, in phase level 1) it first expands that expression in the phase level corresponding to the compile time of the compile time (phase level 2), and if that expression contains another `let-syntax`, then it starts expanding an expression in phase level 3 and so on.

Arc evaluates and expands everything in one phase, as far as Racket is concerned. It would make sense for `lexical-macro` to do its evaluation in the same phase too.

I notice Common Lisp's `macrolet` allows[1] inner `macrolet` macros to depend on outer ones, like this:

  (lexical-macro (foo) ''world
    (lexical-macro (bar) `(sym:+ "hello-" ,(foo))
      (bar)))
  ; could return the symbol 'hello-world

[1] http://www.lispworks.com/documentation/HyperSpec/Body/s_flet...

---

Here's a different take on the local macro concept by almkglor 10 years ago: http://arclanguage.org/item?id=3085

That implementation, `macwith`, is still around on Anarki's arc2.master branch, in the file lib/macrolet.arc: https://github.com/arclanguage/anarki/blob/af5b756e983807ba6...

It works by traversing the body s-expressions and expanding all occurrences it finds, leaving anything else alone. I think this means it tends to break if the code ever contains a list that looks like a call to that macro but isn't supposed to be, such as a quoted occurrence like '(my-macro), a binding occurrence like (fn (my-macro) ...), or even another `macwith` binding occurrence of the same macro name.

I don't prefer this to the other approach we're talking about, since after all it's within easy reach of the macro system to support local macros itself rather than with an error-prone code-walker like this.

However, `macwith` is a macro that makes sense in its own right. It's easy to work around many of the places the code walker runs into false positives, and if `macwith` came with support for an escape sequence, there would be easy workarounds in many other cases too.

   (define (ac-call fn args env)
     (let ((macfn (ac-macro? fn)))
  -    (cond (macfn
  +    (cond ((and macfn (not (lex? fn env)))
              (ac-mac-call macfn args env))

(define (ac-call fn args env) (let ((macfn (ac-macro? fn))) - (cond (macfn + (cond ((and macfn (not (lex? fn env))) (ac-mac-call macfn args env))

Now, if you wanted to define the macro locally, like this...

  (let-macro my-def (name args . body) `(= ,name (fn ,args ,@body))
    ... (my-def ...))

In theory, one could put arbitrarily complicated information, such as about deduced types of variables, into this "env" mapping, and implement some amount of compiler optimization that way.