Can you give examples? If you give concrete examples we can try to improve matters.

Many libraries have a long header comment up top, like rocketnia's ns.arc.

Anarki comes with online help. Try saying `(help deftem)` at the commandline. Admittedly online help is patchy inside lib/ but make it a habit to check it if you aren't already.

Lastly, it's worth looking inside the CHANGES/ directory for major features.

> ..or to know how stable the code is, because no one seems to use most of it.

If by 'stable' you mean "we shouldn't change it", then nothing is stable, inside or out of the lib/ directory.

If by 'stable' you mean "working as intended", then tests are our proxy for intent. There's tons of tests under lib/ and I run the tests fairly often. If a library doesn't have tests then I have a lot less to say about it.

So "no one seems to use most of it" feels like an exaggeration. Most code under lib/ is constantly being monitored for bitrot.

> The dependencies for news should be libraries, but they still seem to bake in too many assumptions and interdependencies to be useful as standalone libraries.

I think here you're cargo-culting[1] notions of "good practice" without considering their costs. Not everything that other people use is a good idea for Arc. We have a tiny number of users, which makes it better to keep the community in a well-integrated codebase rather than fragment it across several "lines". Interfaces have a tendency to balkanize users on one side or other of them. We don't want that; our tiny community doesn't really benefit from the scaling benefits that interfaces provide.

If and when Arc grows 10x or something, and we have three different sets of html generation libraries, we can think about drawing better lines. Until then I suggest a different "best practice": YAGNI[2]

[1] https://en.wikipedia.org/wiki/Cargo_cult

Are you sure? After all, these issues are being reported as problematic already [1,2].

[1] http://arclanguage.org/item?id=20490 [2] http://arclanguage.org/item?id=20524 "(cough (body) cough)"

> If and when Arc grows 10x or something...

Did you perhaps have a similar opinion as well? I certainly recall other people asking for a package manager.

Currently, we break up the API boundaries in Anarki by function and by file. We document the function APIs with docstrings, and we write verified documentation for them with (examples ...). We verify the file APIs with "foo.arc.t" unit test files.

If we put more effort into documenting or verifying those boundaries, it would hardly be a big change in how we use the language. We could perhaps add a bunch of "foo.arc.doc" files or something which could compile into introductory guides to what foo.arc has to offer (whereas the docstrings still act as API documentation). And we could perhaps add an alternative to Anarki's (require ...) that would load a library in a mode that faster faster, abiding by an interface declared in a "foo.arc.i" file.

It seems like the most drastic change in philosophy would happen if we broke Anarki apart into sections where if someone's working on one section and discovers they're breaking something in another section, they don't feel like they can fix it themselves.

I suppose that happens whenever people put Arc libraries in their own separate GitHub repos. There are actually a number of those repos out there, including my own (Lathe). Even aw, who was a strong believer in avoiding unnecessary black-box interface boundaries, would still put things in various repositories, and the hackinator tool would collect them into one file tree.

It also happens naturally with poor knowledge sharing. Seems like despite my attempts at explaining ns.arc, I'm still the only one with a very clear idea of how it works and what it's good for, so other people can't necessarily fix it when it breaks.

(Come to think of it, that makes ns.arc a great test case for pulling out an Anarki library into its own repo. That'd save other Anarki maintainers the trouble of having to fix it to keep Anarki's tests running.)

This would be fine. Whether we consider something an API has nothing to do with how we divide code into repositories or sections. Perhaps the key is to know who is calling us. I'd be happy to provide a guarantee to anyone who "registers" their repository with Anarki: if it's easy for me to run your codebase, I'll check it everytime I make a change to Anarki, and ensure that it continues to pass all its tests. If I break an interface, I'll fix all its callers and either push my changes or send the author a PR.

Exactly! It's a huge problem in the programmer world today that we are indisciplined about this distinction. There's a major difference between making a boundary just to make edits easy and "APIs". APIs are basically signalling to most people that they shouldn't be crossing them, and guaranteeing that anybody calling them will be able to upgrade without modifying their code. That is worlds away from just not being able to see everything on a single screen.

Since we use words like "abstraction" and "interface" for both situations, it is easy to cross this line without any awareness that something big and irreversible has changed.

Maybe there's a difference somewhere in what we mean by "API." There's a distinction to make, but I wouldn't make that one.

I'm saying APIs arise naturally at boundaries people already feel aren't worth the effort to cross for various plausible reasons. Recognizing an API boundary is a positive thing: Then we can talk about it and help people understand the ways in which it's worth it for them to cross the boundary. But talking about it also makes it something people form as a concept in their mind and form expectations around, so it can become a more imposing boundary that way if we're not careful. We should seek a sweet spot.

The important distinction is that sometimes the reasons people don't cross boundaries are good ones, and sometimes they're baloney.

I think what you and I want to avoid for Anarki is documentation that gives people baloney reasons to think it's not worth it for them to edit the Anarki code.

While we can try to minimize the good reasons that exist, and we can encourage people to try it before they knock it, there's no point in denying that the good reasons sometimes exist.

If you ask me, the reason we have to settle for a "sweet spot" is totally a consequence of the fact that Anarki is made of text. Text loves bureaucracies. Someone sees text, they ask for the Cliff's Notes, and then someone has to maintain the Cliff's Notes.... So you've gotta get rid of the text, or you'll have an API boundary.

Programming without bureaucracy would be like having a conversation without writing it down. I believe in this. I think conversational programming is exactly what we can do to simplify the situation of programming in the world, and Era's my name for that project, but Anarki is a more traditional language, and it takes a more traditional trajectory into APIs.

Our documentation can try to convey "This is all subject to change! Edit it yourself, open an issue, or drop by on Arc Forum for some help! We count on contributions like yours to make Anarki a good language. Since others are welcome to do the same thing, watch out for changes in this space." We could put an "unstable, just like everything else here" tag on the corner of every single documentation entry.

Anarki is far from the only unstable codebase people have ever seen. Being a Node.js user, my code has regularly been broken by upgrades, and Node.js isn't even a version 0.0.x project. I think only a certain number of people would mind if Anarki promised stability and then couldn't deliver on it, let alone mind if Anarki made it clear in the API documentation that there was nothing stable to find there in the first place.

---

"guaranteeing that anybody calling them will be able to upgrade without modifying their code"

You know me well. I probably said that pretty insistently in the past. It's something obsess over and a major requirement I have for code I write (and hence all its dependencies), even though things like Node.js show me it's not a current norm in software.

It's the 100-year language ideal. Spend a lot of time to get the axioms just right, and then the language is stable and nobody ever feels the need to change it since there's nothing to improve on! In 100 years, Arc is supposed to be the language that's too perfect to be anything but the stablest language around.

I still believe in that ideal, but not in quite the same way. People will come up with a reason to promote their alternative language, even if it's just to say they're different than their parents. But I do believe languages can be designed to be more compatible with their forks, even to the point where it's difficult to say where one language ends and another begins. The resulting unified "ecosystem of all languages" could have a much simpler design than the "ecosystem of all languages" we currently have.

That's also part of my approach in Era. I make decisions in those codebases with the goal of simplifying the ecosystem of all languages.

I recommend (obj), (obj foo (list 'null)), (obj foo (list 'false)), and (obj foo (list (list))). Arc is good at representing optional values as subsingleton lists:

  ; Access a key that's known to be there.
  t!foo.0
  
  ; Access a key that isn't known to be there.
  (iflet (x) t!foo
    (do-something-then x)
    (do-something-else))

Meanwhile, I don't think there's any big deal when we don't have Arc-style booleans....

  ; Branch on the symbols 'false and 'true.
  (if (isnt 'false x)
    (do-something-then)
    (do-something-else))
  
  ; Alternative
  (case x true
    (do-something-then)
    (do-something-else))

We do already have booleans that work in Arc without any conversion. Please see http://arclanguage.org/item?id=20492

But now that I look closer at the ac.scm history (now ac.rkt in Anarki), I realize I was mistaken to believe Arc treated #f as a different value than nil. Turns out Arc has always equated #f and 'nil with `is`, counted them both as falsy, etc. So this library was already returning nil, from Arc's perspective.

There are some things that slip through the cracks. It looks like (type #f) has always given an "unknown type" error, as opposed to returning 'sym as it does for 'nil and '().

Really, #t and #f are not proper Arc booleans[0], so it makes sense that Arc can't tell what type they are.

You can configure the value Arc chooses for a JSON null with the $.json-null function, which I think is fine as JSON APIs might have differing semantics.

There are a lot of ways to conceive of what Arc "is" outside of the Racket implementations, but I think Arc implementations like Rainbow, Jarc, Arcueid, and so on tend to be inspired first by the rather small set of operations showcased in the tutorial and in arc.arc. (As the tutorial says, "The definitions in arc.arc are also an experiment in another way. They are the language spec.") Since #f isn't part of those, it's not something that an Arc implementation would necessarily focus on supporting, so there's a practical sense in which it's not a part of Arc our Arc code can rely on.

Really, #t and #f are not proper Arc anything, but the language apparently handles them so IMHO Arc should also be able to know what type they are. Otherwise, I fear, this will become a Hodge Podge language that will lose appeal.

I took a look and found fixes for the unit tests. Before I got into that debugging though, I noticed some problems with the JSON library that I'm not sure what to do with. It turns out those are unrelated to the test failures.

You might be familiar with Lathe as the name of my Arc utility libraries and their namespace system[2]. The concept behind the name Lathe was always related to trying to "smooth out" the language I was working in. (And I think originally it was directly related to the language Blade I was trying to design and build; I was smoothing out Arc to get it closer to Blade, or something.)

And at one point I started putting JavaScript utilities in Lathe as well. At the time, I thought stuffing Arc utilities and JavaScript utilities into the same repo was for the best, because I figured they might interact with each other somehow (e.g. one of them loading the other through an FFI of some kind). They never quite did. Even when I started putting Racket utilities in Lathe, I didn't ever invoke them from Arc or vice versa.

I'm finally breaking Lathe apart into multiple libraries, all under the "Lathe" GitHub organization[1]. I've got these so far:

- Lathe Comforts for Racket (little day-to-day utilities)

- Lathe Morphisms for Racket (algebraic or category-theoretic constructions)

- Lathe Ordinals for Racket (ordinal arithmetic)

Lathe Morphisms and Lathe Ordinals weren't ever part of the original Lathe repo[2]; they're all-new. And there isn't really that much to Lathe Morphisms yet anyhow; its design is still unstable at the most basic levels as I learn more about category theory.

Anyhow, you may notice "for Racket" is part of the library name, and the full GitHub repository name is something like lathe/lathe-comforts-for-racket. I'm organizing Lathe so that it's reasonable to add in libraries like "Lathe Comforts for JavaScript," "Lathe Comforts for Arc," and so on, without having to come up with a creative name for each and every library. :-p

Since Racket has a package repository, I drop the "for Racket" from the name of the library when I publish it there, so people can simply run `raco install lathe-comforts`. I would do the same thing if I were publishing a "for JavaScript" library on npm.

Anyhow, this blog post is a journal of the way I broke out Lathe Ordinals into its own library this week.

[1] https://github.com/lathe/

The gist of it is that the extensible quasiquotation syntax design I've been working on for a while now, which I've thought had something to do with higher category theory, does indeed seem very related.

All the times I've thought to myself "Why is this so hard to implement? Surely someone out there has answers..." it turns out that the people working on opetopic higher categories are exactly the people with those answers. So now some of the complexity that's made me doubt my approach, I can actually be confident about, and I've found some clear answers out there to things I never quite figured out on my own.

For instance, check out "Implementing the Opetopes," a PDF linked from http://ericfinster.github.io/. In there, Eric Finster describes a data structure called "SAddr," which is an address referencing a particular part of an opetopic structure, the same way you might use an integer to reference a particular element of a list.

Every so often I would think about what it would take to reference a particular element of what I've been calling a "hypertee," and I would come to the tentative conclusion that I'd need a list of lists of lists ... of lists of empty lists. That's exactly what Eric Finster's SAddr data structure is, so it looks like I don't need to worry that I've made a mistake somewhere; someone else has tested this idea already and had success. :)

Over the past week I've been going ahead with an implementation of the kind of quasiquotation system I've been attempting for all this time. It's going well. :) I look forward to having more to report at some point.

  ; my-file.arc
  (= n 2)
  (= my-definition (* n n))
  
  
  arc>
    (= my-definition
      (let my-ns (nsobj)
        
        ; Populate the namespace with the current namespace's bindings.
        (each k (ns-keys current-ns)
          ; Racket has a variable called _ that raises an error when
          ; used as an expression, and it looks like an Arc variable, so
          ; we skip it. This is a hack. Maybe it's time to change how
          ; the Arc namespace works. On the other hand, copying
          ; namespaces in this naive way is prone to this kind of
          ; problem, so perhaps it's this technique that should be
          ; changed.
          (unless (is k '||)
            (= my-ns.k current-ns.k)))
        
        ; Load the file.
        (w/current-ns my-ns (load "my-file.arc"))
        
        ; Get the specific things you want out of the namespace.
        my-ns!my-definition))
  4
  arc> n
  _n: undefined;
   cannot reference an identifier before its definition
    in module: "/home/nia/mine/drive/repo/mine/prog/repo/not-mine/anarki/ac.rkt"
    context...:
     /home/nia/mine/drive/repo/mine/prog/repo/not-mine/anarki/ac.rkt:1269:4

An "Arc namespace" is just a convenience wrapper over a Racket namespace that automatically converts between Arc variables `foo` and their corresponding Racket variables `_foo`.

For some overall background...

I wrote ns.arc when I didn't have much idea what Racket namespaces or modules could do, but I was at least sure that changing the compiled Arc code to more seamlessly interact with Racket's `current-namespace` would open up ways to load Arc libraries without them clobbering each other. It wouldn't be perfect because of things like unhygienic macros, but it seemed like a step in the right direction.

I went a little overboard with the idea that Racket namespaces and Racket modules could be manipulated like Arc tables. However, that was the only clear vision I had when I embarked on writing the ns.arc library, so I approximated it as well as I could anyway. In fact, I don't think the utilities for generating first-class modules (like `simple-mod` and `make-modecule`) are all that useful, because as I understand a little better now, Racket modules are as complicated as they are mainly to support separate compilation, so generating them at run time doesn't make much sense.

I approached this with the secondary goal of letting a Racket program (or a determined Arc program) instantiate multiple independent intances of Anarki. The ac.rkt module was the only place we were performing side effects when a Racket module was visited, and Racket's caching of modules makes it hard to repeat those side effects on demand, so I moved most of them into a procedure called `anarki-init`.

By adding one line to the example I gave...

  (= my-definition
    (let my-ns (nsobj)
      
      ; Load the Arc builtins into the namespace so we can evaluate
      ; code.
      (w/current-ns my-ns ($.anarki-init))

      ...))

I used issue #95 on GitHub to track this task, and I talk about it a little more there: https://github.com/arclanguage/anarki/issues/95

Before I started on that, I did a bunch of cleanup to get the Anarki unit tests and entrypoints running smoothly on all our CI platforms. To get started on this cleanup, I had a few questions hjek and akkartik were able to discuss with me on issue #94: https://github.com/arclanguage/anarki/issues/94

If the way Arc's lib/ directory has been used is any indication, the way to do it is:

- Start in the Arc directory when you run Arc, and never cd out of it.

- (load "lib/foo.arc"), or (require "lib/foo.arc") if you want to avoid running the same file multiple times

But I think for some Anarki users, the preferred technique has been somewhat different:

- Invoke Anarki from any directory.

- Ignore lib/ as much as possible. On occasion, load a library from there by using (load "path/to/arc/lib/foo.arc"), but a few libraries may make this difficult (e.g. if they need to load other libraries).

When I started writing Arc libraries, the first thing I wrote was a framework for keeping track of the location to load things relative to, so that my other libraries could load each other using relative paths regardless of which of the above techniques was in use. But the Lathe module system didn't catch on with anyone else. XD

Yeah, this makes sense if you're making something that only you use, but if I'm trying to make something a little more portable, like (as you mention) a library.

Arc's assignment operations are set up to acquire a global lock as they operate, achieved by use of an (atomic ...) block. This is so other threads can't observe the in-between states of operations like (swap ...) and (rotate ...). The documentation for 'atomic is here: https://arclanguage.github.io/ref/threading.html

Arc also has some support for continuations, which can serve as a kind of cooperative multithreading. Mainly, Arc just exposes 'ccc as a binding for Racket's 'call-with-current-continuation, and it uses Scheme's 'dynamic-wind to implement 'protect. These are documented here: https://arclanguage.github.io/ref/error.html