Why does the article say it's nice to have lists-as-syntax, but then say Racket's syntax transformers are even better?

Why does the article say it's nice that X-expressions reveal "the sequential nature of the data elements" of a document, but then say it's nice that Scribble lets you embed Racket expressions at arbitrary places in your document?

IMO, all these things are nice options to have at the same time, and maybe that's what the author thinks too. It would be nice to hear more concrete reasons for preferring these features though, especially since concrete reasons are what the article is trying to offer.

---

"The prob­lem with Lisp flat­tery is that it makes sense only to ex­pe­ri­enced Lisp pro­gram­mers. To oth­ers—es­pe­cially those who are try­ing to de­cide whether to learn and use a Lisp—it just comes across as un­sub­stan­ti­ated hoodoo."

This is true, but certain kinds of experience bring the hoodoo feeling back. When I see the elegance of computational trinitarianism, it makes me want to use algebraic data types instead of s-expressions to represent program structure. Some programming languages are absolutely in the axiomatic style, even in the sense that they're specified fully as a half-page diagram with lots of Greek letters all over. Coming back to a Lisp, everything's messy again, and it makes it really hard to swallow the hype.

I'd say there are some specific benefits that Lisps have to offer, but people easily confuse these benefits with other accidental properties of the languages.

Most of the historic advantages of Lisp have already been plundered by other languages. What does "Lisp" even mean now that garbage collection, first-class anonymous functions, interactive debuggers, run time code loading, and package managers are available for basically every popular language?

Does "Lisp" just mean macros? Python, Groovy, Idris, Agda, Haskell, OCaml, etc. each have some variation of macros or AST manipulation, but does that make them Lisp dialects? What about cases like JavaScript and Ruby, which have extensive libraries for parsing and manipulating their own code but have no support built into the language?

Angular.js watches for changes to your first-class environments and uses them to re-macroexpand the DOM tree. That's pretty Lispy!

You might say that the difference isn't technical but cultural. Well, Java "managed to drag a lot of [C++ programmers] about halfway to Lisp." JavaScript was explicitly "Scheme in the browser" until it was dragged about halfway to Java. I think there's a case to be made that, culturally, these languages were Lisps doing the best they could.

Do we call them Lisp dialects? I don't think we do. I think we only call something a Lisp dialect if its syntax is structured as lists of lists and if it's not a Forth dialect.

So what are the concrete benefits of lists as syntax? I use them for specific reasons:

- If I were stranded on a new platform or otherwise compelled to reinvent a good syntax from a limited set of tools, it would be pretty easy to start off with parens. This is due to the sequentiality of text as a code representation format. Parens let us conceptually break up a piece of text into a bunch of text-with-holes-in-it pieces joined together. In other words, trees of interpolated strings. Lists are an extreme case of interpolated strings where there is no string left over; it's just some abstract sequential format consisting entirely of holes. I tried using interpolated strings as syntax, and the string-parsing operations made macroexpansion too inefficient, so I settled on lists as a consolation prize.

- When every syntax is a list, it doesn't matter if the syntax is standard or custom. Every syntax looks just as good or bad as every other. This might help people adopt new, innovative operators they might have scoffed at in another language. Racket is an example of a project that cultivates a variety of innovative syntaxes to coexist all at the same time, such as Typed Racket, web server's stateless servlets, and the legacy r5rs, r6rs, and mzscheme languages.

[1] https://github.com/arclanguage/anarki/issues/40

Before:

  $ racket -f boot.scm -e "(tl)"

  $ racket -t boot.scm -e "(tl)"

The new language construct...

  -(def editor ((o u the.me))
  +(def editor ((t u me))

Like, every time you type "fn" you could have typed "lambda", and that is indeed a certain kind of inconvenience.

In this case, every time you type "(t" you could have typed "(o" and "the." instead. So it has the slight benefits of a shorter name, and it has the added benefit that it makes you move your cursor less. (On the other hand, if you're refactoring an existing use of "(o", it makes you move your cursor more.)

This is the kind of thing I consider macro-expressible, rather than a language innovation. Libraries should be able to provide this. And in fact they can; my Lathe libraries have customizable pattern matching support already.

Yeah, Arc macros can't update the built-in pattern-matching syntax this way. My library does it by implementing its own special version of (fn ...) and such. For Hacker News in particular, it might have been easier to add (t ...) to the language rather than to overhaul the design of pattern-matching throughout the codebase to be extensible.

---

"You still have to declare which variables you plan to use in each function signature, so it's possible to look at the function and know exactly what state it depends on."

That's not even true. A call to (user-name ...) still depends on the state of the.me, and that's no longer visible in the signature.

It worries me that static knowledge would be used as a justification for this technique when the knowledge isn't even sound. That's where we get arbitrary language restrictions that have no real benefit.

I hope this static knowledge is meaningful in some weaker way that just wasn't quite articulated here.

---

  ; SXML
  (tagname (@ (attr "value") (attr2 "value2"))
    (tagname2)
    (tagname3 "data"))
  
  ; sml.arc's "old format"
  (tagname (@ attr "value" attr2 "value2")
    (tagname2)
    (tagname3 "data"))
  
  ; sml.arc's current example
  (tagname attr "value" attr2 "value2"
    (tagname2)
    (tagname3 "data"))

  ; X-expressions
  (tagname ((attr "value") (attr2 "value2"))
    (tagname2)
    (tagname3 "data"))

As for SXML, I think it's pretty nice that the SXML standard actually has well-defined ways to use doctypes, namespaces, processing instructions, and comments. That makes it more capable of representing an actual XML document. The spec is only a few pages long, and half of that is preamble, so it would be pretty simple to implement it faithfully.

Paul Graham has this to say about hygiene in the tutorial:

  Some people worry unduly about this kind of bug.  It caused the
  Scheme committee to adopt a plan for "hygienic" macros that was
  probably a mistake.  It seems to me that the solution is not to
  encourage the noob illusion that macro calls are function calls.
  People writing macros need to remember that macros live in the land
  of names.  Naturally in the land of names you have to worry about
  using the wrong names, just as in the land of values you have to
  remember not to use the wrong values-- for example, not to use zero
  as a divisor.

  (mac n-of (n expr)
    (w/uniq ga
      `(let ,ga nil
         (repeat ,n (push ,expr ,ga))
         (rev ,ga))))

It seems we don't particularly follow that rule here on the Arc Forum. For instance, a few of us have agreed that a certain behavior in Arc 3.1 is a bug: When we make a function call to a local variable, we don't want a global macro of the same name to take effect, which is what happens in Arc 3.1. If we were keeping locals and globals separate, we wouldn't even encounter this problem.

Which means that if we want to write macros that are hygienic, we can't write them in quite the way we see in arc.arc or the tutorial. If we're dedicated to hygiene, we might even want to rewrite arc.arc to fix its hygiene issues... but that's practically the whole language, so it effectively starts to be a new language project. The Anarki arc2.hygiene branch, Penknife, ar, Semi-Arc, and Nulan are several examples of Arc-based or Arc-inspired projects that pursued some kind of hygiene.

If we don't mind the lack of hygiene in arc.arc but only care about proper hygiene for our own new macros, it is possible to be diligent about hygiene in plain Arc 3.1 or Anarki:

  (mac n-of (n expr)
    (w/uniq ga
      (rep.let ga nil
         (rep.repeat n (rep.push expr ga))
         `(',rev ,ga))))

  (mac n-of (n expr)
    `( ',(fn (n expr)
             (let a nil
               (repeat n (push (expr) a))
               rev.a))
       ,n
       (fn () ,expr)))

  (mac qq-with args
    (let (body . rev-bindings) rev.args
      (let (vars vals) (apply map list (pair rev.rev-bindings))
        `(',list `',(fn ,vars ,body) ,@vals))))
  
  (mac n-of (n expr)
    (qq-with n n expr `(fn () ,expr)
      (let a nil
        (repeat n (push (expr) a))
        rev.a)))

It looks like the author forked their existing project Arcadia[1] to make Arc++, and difference has something to do with the memory management strategy (adopting C++'s shared_ptr).

  1 * larger - 5/2 * smaller = 0
  2 * larger - 2 * smaller = 12

  arc> (load "lib/math.arc")
  nil
  arc> (gauss-elim '((1 -5/2) (2 -2)) '(0 12))
  (10 4)

---

I came to Arc because I wanted to use only the best language, and Arc was the only sign I saw that anyone was ambitious enough to try to build the best language today. I stayed with Arc for a while because I could make macro frameworks that explored new ways to program in Arc, without feeling like I was messing up the way anyone else was programming in Arc. Only when I came up with language ideas that couldn't work in Arc did I start to write my own languages.

Then I moved away from Arc mainly because I wanted to practice coding in JavaScript. That was because I wanted to be able to spend my time programming even if I was stranded at a computer that didn't have Arc installed. It was sort of a theoretical concern at the time, but then it came in handy for employment, where Arc was unlikely to be an employer's platform of choice. :)

That nice phenomenon I experienced in Arc, where I could experiment with new ways to program without leaving the language... I think that's the essence of Arc for me. Arc is conducive to Greenspunning new languages. Not only is Arc homoiconic, but it also has few pretensions of being perfect yet, and it's explicitly in the stage of soliciting new suggestions for the core operators (or at least it was). Using Arc, I felt welcome to Greenspun new sets of operators as I liked.

Usually when I have strong opinions about what Arc should become, I just pursue those opinions myself in my own languages. I even intend for my language Staccato to be conducive to Greenspunning; part of its design even involves an explicit "please put all your nonstandard Staccato implementation experiments here" zone. But Arc's lack of pretension is a quality of its community. It's something delicate that I won't necessarily succeed at in Staccato even if I have no regrets about the technical design.

---

If there's any language that has that quality of Arc, I would guess it's Agda. I've seen quite a few type system experiments that are developed as DSLs in Agda, so Agda seems to have the kind of culture that likes remixing the core operators. While Agda's syntax is not homoiconic, I think it's flexible enough to make these DSLs rather seamless.

If Arc has an edge over Agda for Greenspunning, it's that it's easier to Greenspun a new implementation of Arc than a new implementation of Agda. From what I hear, Agda's implementation involves a termination checker that's sophisticated enough that many users take advantage of it without knowing where its limits are. Those users probably aren't prepared to reimplement it themselves.

I may be wrong on this, but it seems to me that static typing does not prevent macros at all (true or otherwise).

I switched Nulan to use static typing, yet it's using the same kind of macro system that it used when it was dynamically typed.

As far as I can tell, a macro is simply a compile-time function that accepts code and returns code. If so, then its static type is "Code -> Code".

In Nulan, the Code type might be defined like this:

  (TYPE Code
  | (*integer Integer)
  | (*number Number)
  | (*string String)
  | (*symbol String)
  | (*gensym Integer)
  | (*list (List Code)))

Within the macro's body, you can pattern match on the Code, you can map/filter on it just like in Arc, you can dynamically return Code, etc.

In Nulan, this is made easy with the & syntax, which is just a shorthand for the Code type:

  # These two are equivalent
  &1

  (*integer 1)


  # These two are equivalent
  &(foo 1 2)

  (*list [ (*symbol "foo") (*integer 1) (*integer 2) ])


  # These two are equivalent
  &(foo ~a ~b)

  (*list [ (*symbol "foo") a b ])

  # These two are equivalent
  (MATCH foo
  | &(foo 1 2)
      ...)

  (MATCH foo
  | (*list [ (*symbol "foo") (*integer 1) (*integer 2) ])
      ...)


  # These two are equivalent
  (MATCH foo
  | &(foo ~a ~@b)
      ...)

  (MATCH foo
  | (*list [ (*symbol "foo") a @b ])
      ...)

There are a couple of ways I think that can get more complicated:

In languages that resolve overloaded names in a type-directed way, I think the reconciling of static types and macros gets quite a bit more challenging due to the underlying challenge of reconciling name resolution with macros.

For instance, I think Idris has different meanings for the name (,) depending on if it occurs as a value of type (Type -> Type -> Type) or a value of type (a -> b -> (a, b)). In one case it's a constructor of tuple types (a, b), and in another case it's a constructor of tuple values (a, b).

Idris's macro system actually seems to have some special support for invoking the typechecker explicitly from macro code, which I think helps resolve names like those. I haven't actually succeeded in writing macros in Idris yet, and it seems to be a complicated system, so I'm not sure about what I'm saying.

Secondly, one potential goal of a macro system is that all the language's existing primitive syntaxes can turn out to be macros. That way, future versions of the language don't have to inherit all the crufty syntaxes of the versions before; they can stuff those into an optional library. If the language has sophisticated compile-time systems for type inference, term elaboration, name resolution, syntax highlighting, autocompletion, etc., then hopefully the macro system is expressive enough that the built-in syntaxes are indistinguishable from well-written macros.

Arc already gives us things like (macex ...) that can distinguish macros from built-in special forms, so maybe seamlessness isn't a priority in Arc. But if it were, and Arc had static types, we would probably want Arc to have type inference as well, which could complicate the macro system.

Macros deal with things at the syntax level, far earlier than any kind of type overloading.

As far as the macro is concerned, it simply sees the syntax

  (*list [ (*symbol ",") (*integer 1) (*integer 2) ])

Basically, I view type inference/checking/overloading occurring in a phase after macro expansion.

Is there any benefit to mixing the type overloading phase and the macro expansion phase?

----

It occurs to me that you might not be referring to a macro which expands to syntax, but instead using the syntax within the macro body itself.

In Nulan, that isn't a problem either. A macro is essentially a function from Code to Code, so the macro body is compiled/macro expanded/type checked just as if it were a function.

Thus any type overloading will happen inside the macro body, before the macro is ever run.

That does mean that the type checker must be able to overload based solely on the types found within the macro body. In other words, it cannot dynamically dispatch.

----

I don't think it's possible for the primitives to be macros, for the simple reason that the language needs axioms.

If the macro itself is being referred to by an overloaded name, then the name needs to be resolved before we know which macro to call.

(I'm not sure if macros' names can be overloaded in Idris; I'm just guessing. Actually, I don't know if Idris lets users define overloaded names. I just know a few built-in names like (,) are overloaded.)

---

"If you mean that the primitives should appear to be macros (even though they're implemented with compiler magic), then I agree with you."

But I think if your language allows for that, it's inconsistent with the way that macros work.

If you want powerful Arc-style macros, then the macro must be run before any types are known.

Any system which allows for that kind of type-based dispatch is different from a Lisp-style macro system, and is probably less powerful.

For example, you won't be able to create macros which create new variable bindings (like "let", "with", etc.) because the type of the variable is not known until after the macro is run.

I'm not sure what a system like that would look like, or if I would even call it a macro system.

I think it makes more sense to have two separate systems: a macro system that deals with syntax only, and an overloading system that deals with types. That's what Nulan does.

I have no experience with Idris or dependent types, so I may be completely wrong.

But from my understanding, a dependent type system allows for types to include values (which are evaluated at compile-time).

If so, then you might not even need macros at all to create the "," behavior.

Instead, you create a typeclass which will dispatch to the appropriate behavior:

  data Pair a b = pair a b

  interface Comma a b c where
    comma : a -> b -> c

  Comma Type Type Type where
    comma = Pair

  Comma a b (Pair a b) where
    comma = pair

  -- Type
  comma Integer Integer

  -- (Pair Integer Integer)
  comma 1 2

  foo : a -> b -> (comma a b)

I never said the overloading of "," was accomplished with macros.

The example I gave was of a macro whose name was overloaded in a type-directed way, similarly to the way "," is overloaded in Idris. (Maybe the macro itself is named ",".) My point is that if the macro system is designed to support that kind of overloading, then sometimes a macro call will depend on intermediate results obtained by the typechecker, so we can't run the macroexpander in a phase of its own.

---

For the sake of remembering Idris more accurately, I checked out the Idris docs to look for the particular "macro system" I was dealing with.

It turns out what I mean by the "macro system" in Idris is its system of elaborator extensions (http://docs.idris-lang.org/en/latest/reference/elaborator-re...). The elaborator's purpose is to fill in holes in the program, like the holes that remain when someone calls a function without supplying its implicit arguments.[1]

It's pretty natural to handle name overloading in the same phase as implicit arguments, because it's effectively a special case. Thanks to dependent types and implicit arguments, instead of having two differently typed functions named (,), you could have one function that implicitly takes a boolean:

  (,) : {isType : Bool} ->
    if isType then (... -> Type) else (... -> (a, b))
  (,) = ...

Another "macro system" I encountered in the docs was Idris's system of syntax extensions (http://docs.idris-lang.org/en/latest/tutorial/syntax.html). As far as I can tell, these apply before the elaborator and typechecker, but they might not be able to run arbitrary code or break hygiene. Maybe someday they'll gain those abilities and become a Nulan-like macro system.

I think Clojure fits most of the criteria that would lead someone to choose Arc. I think Clojure's main flaw compared to Arc is that it's a bit cumbersome to do iteration, because there's no general support for tail call optimization.

Arc has a few things positively going for it:

* Arc's implementation doesn't implement the whole language from scratch. Instead, syntaxes, data representations, and primitive operations are inherited from Racket, and most of the high-level tools are implemented in Arc itself as a library. What remains in the Arc implementation is a small, unintimidating core focused on some compilation and pattern-matching features. Since the core is small, it's easy to make certain modifications if needed. (Modifications to things inherited from Racket, like changing the reader syntax, are more challenging.)

* It extends s-expression syntax in a few minor ways. I think one of these extensions, the (a:b:c d) shortcut for (a (b (c d))), is particularly compelling. It tends to reduce lines of code, indentation, and parentheses all at once:

  (blah-blah-foo
    (blah-blah-bar
      blah-blah-baz))
  -->
  (blah-blah-foo:blah-blah-bar
    blah-blah-baz)

Huh, I didn't know that about Clojure!

We already have plenty of examples of first-class namespaces, like aw's implementation posted recently. So all this would take is an implementation of Arc in Arc. Do we have one of those? I thought I heard of one at some point.

My excitement is not because I think a pileup of various compilers in a single codebase is a great idea, but because I think it's easier to share code this way. Compiler hacks are prone to merge conflicts that inhibit code sharing, but sharing libraries is... well, not perfect in Arc either, but it's at least ameliorable by doing some simple search-and-replace renaming or by agreeing on a namespacing framework (like my framework in Lathe).

Here's the source for `and` (https://github.com/clojure/clojure/blob/clojure-1.7.0/src/cl...):

    (defmacro and
      "Evaluates exprs one at a time, from left to right. If a form
      returns logical false (nil or false), and returns that value and
      doesn't evaluate any of the other expressions, otherwise it returns
      the value of the last expr. (and) returns true."
      {:added "1.0"}
      ([] true)
      ([x] x)
      ([x & next]
       `(let [and# ~x]
          (if and# (and ~@next) and#))))

Based on that, I found this way to write aand, which seems to work at http://www.tutorialspoint.com/execute_clojure_online.php:

  (defmacro aand
    ([] true)
    ([x] x)
    ([x & next]
     `(let [~'it ~x]
        (if ~'it (aand ~@next) ~'it))))
  
  (prn (aand "woo" (list it it) (list it it it)))

  (defmacro capture-it
    ([] 'it))
  
  (prn (let [it "woo"] (capture-it)))