I was responding to this:

"Of the 14 files in the release of arc0.tar, the libs.arc file contained this:

  (map load '("strings.arc"
              "pprint.arc"
              "code.arc"
              "html.arc"
              "srv.arc"
              "app.arc"
              "prompt.arc"))

Can you give examples? If you give concrete examples we can try to improve matters.

Many libraries have a long header comment up top, like rocketnia's ns.arc.

Anarki comes with online help. Try saying `(help deftem)` at the commandline. Admittedly online help is patchy inside lib/ but make it a habit to check it if you aren't already.

Lastly, it's worth looking inside the CHANGES/ directory for major features.

> ..or to know how stable the code is, because no one seems to use most of it.

If by 'stable' you mean "we shouldn't change it", then nothing is stable, inside or out of the lib/ directory.

If by 'stable' you mean "working as intended", then tests are our proxy for intent. There's tons of tests under lib/ and I run the tests fairly often. If a library doesn't have tests then I have a lot less to say about it.

So "no one seems to use most of it" feels like an exaggeration. Most code under lib/ is constantly being monitored for bitrot.

> The dependencies for news should be libraries, but they still seem to bake in too many assumptions and interdependencies to be useful as standalone libraries.

I think here you're cargo-culting[1] notions of "good practice" without considering their costs. Not everything that other people use is a good idea for Arc. We have a tiny number of users, which makes it better to keep the community in a well-integrated codebase rather than fragment it across several "lines". Interfaces have a tendency to balkanize users on one side or other of them. We don't want that; our tiny community doesn't really benefit from the scaling benefits that interfaces provide.

If and when Arc grows 10x or something, and we have three different sets of html generation libraries, we can think about drawing better lines. Until then I suggest a different "best practice": YAGNI[2]

[1] https://en.wikipedia.org/wiki/Cargo_cult

Are you sure? After all, these issues are being reported as problematic already [1,2].

[1] http://arclanguage.org/item?id=20490 [2] http://arclanguage.org/item?id=20524 "(cough (body) cough)"

> If and when Arc grows 10x or something...

Did you perhaps have a similar opinion as well? I certainly recall other people asking for a package manager.

Currently, we break up the API boundaries in Anarki by function and by file. We document the function APIs with docstrings, and we write verified documentation for them with (examples ...). We verify the file APIs with "foo.arc.t" unit test files.

If we put more effort into documenting or verifying those boundaries, it would hardly be a big change in how we use the language. We could perhaps add a bunch of "foo.arc.doc" files or something which could compile into introductory guides to what foo.arc has to offer (whereas the docstrings still act as API documentation). And we could perhaps add an alternative to Anarki's (require ...) that would load a library in a mode that faster faster, abiding by an interface declared in a "foo.arc.i" file.

It seems like the most drastic change in philosophy would happen if we broke Anarki apart into sections where if someone's working on one section and discovers they're breaking something in another section, they don't feel like they can fix it themselves.

I suppose that happens whenever people put Arc libraries in their own separate GitHub repos. There are actually a number of those repos out there, including my own (Lathe). Even aw, who was a strong believer in avoiding unnecessary black-box interface boundaries, would still put things in various repositories, and the hackinator tool would collect them into one file tree.

It also happens naturally with poor knowledge sharing. Seems like despite my attempts at explaining ns.arc, I'm still the only one with a very clear idea of how it works and what it's good for, so other people can't necessarily fix it when it breaks.

(Come to think of it, that makes ns.arc a great test case for pulling out an Anarki library into its own repo. That'd save other Anarki maintainers the trouble of having to fix it to keep Anarki's tests running.)

This would be fine. Whether we consider something an API has nothing to do with how we divide code into repositories or sections. Perhaps the key is to know who is calling us. I'd be happy to provide a guarantee to anyone who "registers" their repository with Anarki: if it's easy for me to run your codebase, I'll check it everytime I make a change to Anarki, and ensure that it continues to pass all its tests. If I break an interface, I'll fix all its callers and either push my changes or send the author a PR.

Exactly! It's a huge problem in the programmer world today that we are indisciplined about this distinction. There's a major difference between making a boundary just to make edits easy and "APIs". APIs are basically signalling to most people that they shouldn't be crossing them, and guaranteeing that anybody calling them will be able to upgrade without modifying their code. That is worlds away from just not being able to see everything on a single screen.

Since we use words like "abstraction" and "interface" for both situations, it is easy to cross this line without any awareness that something big and irreversible has changed.

Maybe there's a difference somewhere in what we mean by "API." There's a distinction to make, but I wouldn't make that one.

I'm saying APIs arise naturally at boundaries people already feel aren't worth the effort to cross for various plausible reasons. Recognizing an API boundary is a positive thing: Then we can talk about it and help people understand the ways in which it's worth it for them to cross the boundary. But talking about it also makes it something people form as a concept in their mind and form expectations around, so it can become a more imposing boundary that way if we're not careful. We should seek a sweet spot.

The important distinction is that sometimes the reasons people don't cross boundaries are good ones, and sometimes they're baloney.

I think what you and I want to avoid for Anarki is documentation that gives people baloney reasons to think it's not worth it for them to edit the Anarki code.

While we can try to minimize the good reasons that exist, and we can encourage people to try it before they knock it, there's no point in denying that the good reasons sometimes exist.

If you ask me, the reason we have to settle for a "sweet spot" is totally a consequence of the fact that Anarki is made of text. Text loves bureaucracies. Someone sees text, they ask for the Cliff's Notes, and then someone has to maintain the Cliff's Notes.... So you've gotta get rid of the text, or you'll have an API boundary.

Programming without bureaucracy would be like having a conversation without writing it down. I believe in this. I think conversational programming is exactly what we can do to simplify the situation of programming in the world, and Era's my name for that project, but Anarki is a more traditional language, and it takes a more traditional trajectory into APIs.

Our documentation can try to convey "This is all subject to change! Edit it yourself, open an issue, or drop by on Arc Forum for some help! We count on contributions like yours to make Anarki a good language. Since others are welcome to do the same thing, watch out for changes in this space." We could put an "unstable, just like everything else here" tag on the corner of every single documentation entry.

Anarki is far from the only unstable codebase people have ever seen. Being a Node.js user, my code has regularly been broken by upgrades, and Node.js isn't even a version 0.0.x project. I think only a certain number of people would mind if Anarki promised stability and then couldn't deliver on it, let alone mind if Anarki made it clear in the API documentation that there was nothing stable to find there in the first place.

---

"guaranteeing that anybody calling them will be able to upgrade without modifying their code"

You know me well. I probably said that pretty insistently in the past. It's something obsess over and a major requirement I have for code I write (and hence all its dependencies), even though things like Node.js show me it's not a current norm in software.

It's the 100-year language ideal. Spend a lot of time to get the axioms just right, and then the language is stable and nobody ever feels the need to change it since there's nothing to improve on! In 100 years, Arc is supposed to be the language that's too perfect to be anything but the stablest language around.

I still believe in that ideal, but not in quite the same way. People will come up with a reason to promote their alternative language, even if it's just to say they're different than their parents. But I do believe languages can be designed to be more compatible with their forks, even to the point where it's difficult to say where one language ends and another begins. The resulting unified "ecosystem of all languages" could have a much simpler design than the "ecosystem of all languages" we currently have.

That's also part of my approach in Era. I make decisions in those codebases with the goal of simplifying the ecosystem of all languages.

Not allowing in obvious breakage is also a way to show respect for your collaborators, other people programming on the same codebase. When you come to Arc you would be demotivated if the news app didn't work, if you had to first get it working, figure out who broke it and how. It's the same with others. When we find a few hours to hack on something fun, we all prefer that it start out in a non-broken state. And we try to leave it in a non-broken state.

So I have a suggestion for you: everytime you create a PR, first try to break it yourself. Ask yourself what changed, and what it may affect. Try to catch the really obvious stuff yourself, like "does news load, does it look the same, can I submit a post?" Run the unit tests. Report in the commit message or PR what all you did by way of testing. These exercises will make you a better programmer. Tell yourself that the goal is to make changes in an efficient manner, solve the problem in a single PR, rather than require more changes to fix breakage in one change, that cause their own breakage, and on and on.

Certainly a lot of things could be tested more; this would help.

But also, we might want to set Github to require the tests to pass on a PR before letting it be merged. I could be wrong, but I think a PR can be merged even if the tests fail.

It isn't broken, though, at least not if "broken" implies an unusable state. I did test the forum before I pushed, I just didn't catch everything... I didn't even know prompt.arc existed, or what it was for, but apparently neither did you. ;)

Racket already has a built-in JSON parser[0] that works just fine with Arc.

    #hash((challenge_ts . 2018-08-03T12:59:44Z)
          (hostname . testkey.google.com) 
          (success . #t))

That said I haven't had a running install of arc in many years so I could be off base.

Anarki used to use Andrew's library from http://awwx.ws/fromjson (perhaps an older version of your link, i4cu) I switched it out back in 2009 for a faster replacement (https://github.com/arclanguage/anarki/commit/dad4dc6662), not realizing that I was breaking the read side in this rather obvious manner. Since then a bunch of work has happened on emitting JSON, but apparently nobody tried parsing JSON until now.

We could retrofit these bugfixes, but I think it would just give up the speed gains to parse things character by character. May be worth just returning to Andrew's version.

I mean, as long as Racket interop is going to be a thing, maybe using the boolean type isn't a bad idea.

    arc> (if $.#f 'foo 'bar)
    bar
    arc> (if $.#t 'foo 'bar)
    foo

Also, I don't get why the Anarki repository contains a JSON parser implemented in Racket, when the built-in Racket JSON parser works just fine with Arc:

    arc> ($ (require json))
    arc> ($.write-json (obj foo "bar"))
    {"foo":"bar"}#<void>
    arc> (w/instring in "{\"foo\":\"bar\"}" ($.read-json in))
    #hash((foo . "bar"))

It currently works but that's only because json.rkt has been modified.

    (def post-getjson (url params (o port stdin))
      (fromstring (post-url url params) 
        (read-json (port))))

    (def recaptcha-response (s r)
      (post-getjson "https://www.google.com/recaptcha/api/siteverify"
      (list 'secret s 'response r)))

    (def verify-captcha (req)
      (do
        (= resp (recaptcha-response recaptcha-secret* 
        (alref (req 'args) "g-recaptcha-response")))
      (if resp!success 
        (pr "success") 
        (pr "failure"))))

    arc> (w/instring in "{\"foo\": true}" ($.read-json in))
    #hash((foo . #t))

    #hash((foo . t))

I'd like to see perhaps a specific brief example, where using Racket booleans within Arc actually is a problem, because I haven't encountered any myself.

If I try to use the hash from your example in actual Arc code, it works fine:

    arc> (if ((w/instring in "{\"foo\": true}" ($.read-json in)) 'foo) 'yes 'no)
    yes
    arc> (if ((w/instring in "{\"foo\": false}" ($.read-json in)) 'foo) 'yes 'no)
    no

Ah, here's perhaps the only special-case we need to handle:

    arc> (= x (w/instring in "{\"foo\": null}" ($.read-json in)))
    #hash((foo . null))  # expected: #hash((foo . nil)) or #hash()
    arc> (if x!foo (prn "error"))
    error

Actually the Racket docs[0] clarifies the situation with null in JSON:

    (json-null) → any/c

    (json-null jsnull) → void?
      jsnull : any/c

If you set the JSON null to nil before running your example, it works as you'd expect:

    arc> ($.json-null nil)
    #<void>
    arc> (= x (w/instring in "{\"foo\": null}" ($.read-json in)))
    #hash((foo . nil))
    arc> (if x!foo (prn "error"))
    nil

I recommend (obj), (obj foo (list 'null)), (obj foo (list 'false)), and (obj foo (list (list))). Arc is good at representing optional values as subsingleton lists:

  ; Access a key that's known to be there.
  t!foo.0
  
  ; Access a key that isn't known to be there.
  (iflet (x) t!foo
    (do-something-then x)
    (do-something-else))

Meanwhile, I don't think there's any big deal when we don't have Arc-style booleans....

  ; Branch on the symbols 'false and 'true.
  (if (isnt 'false x)
    (do-something-then)
    (do-something-else))
  
  ; Alternative
  (case x true
    (do-something-then)
    (do-something-else))

We do already have booleans that work in Arc without any conversion. Please see http://arclanguage.org/item?id=20492

But now that I look closer at the ac.scm history (now ac.rkt in Anarki), I realize I was mistaken to believe Arc treated #f as a different value than nil. Turns out Arc has always equated #f and 'nil with `is`, counted them both as falsy, etc. So this library was already returning nil, from Arc's perspective.

There are some things that slip through the cracks. It looks like (type #f) has always given an "unknown type" error, as opposed to returning 'sym as it does for 'nil and '().

Really, #t and #f are not proper Arc booleans[0], so it makes sense that Arc can't tell what type they are.

You can configure the value Arc chooses for a JSON null with the $.json-null function, which I think is fine as JSON APIs might have differing semantics.

There are a lot of ways to conceive of what Arc "is" outside of the Racket implementations, but I think Arc implementations like Rainbow, Jarc, Arcueid, and so on tend to be inspired first by the rather small set of operations showcased in the tutorial and in arc.arc. (As the tutorial says, "The definitions in arc.arc are also an experiment in another way. They are the language spec.") Since #f isn't part of those, it's not something that an Arc implementation would necessarily focus on supporting, so there's a practical sense in which it's not a part of Arc our Arc code can rely on.

Really, #t and #f are not proper Arc anything, but the language apparently handles them so IMHO Arc should also be able to know what type they are. Otherwise, I fear, this will become a Hodge Podge language that will lose appeal.

You don't need to add any $ handlers to handle Racket booleans. They can be passed directly to Arc functions.

  (if #f 'foo 'bar)
 

    {"success": true,  
     "challenge_ts": "2018-08-03T22:31:05Z",  
     "hostname": "testkey.google.com"}

edit: I went ahead and pushed my current working branch[1] if anyone wants to see it.

    ($ (require json))
    
    (if
       ((w/instring json "{\"success\":false}"
                          ($.read-json json))
                          'success)
       'success ; proceed to login
       'fail    ; must be a robot then
     )

It might be useful if it only shows up on the login/signup page after failed attempts, but it might also be overkill. Personally, I prefer overt solutions to opaque ones like shadowbanning and throttling IPs. But mostly it seemed like a good way to figure out some basics (managing keys, how the app server manages the form loop, API responses and JSON.)

I get the impression many of the people who want to start a HN like clone are considering using it as a content delivery platform. For example http://arclanguage.org/item?id=20452 notes a revenue sharing model for specialized content. Now, if that's the case these site owners will have both spammers and content scrapers to contend with. So my initial comment was also referring content scrapers too.

Let me add some info to that.... I'm not sure if anyone noticed, but HN has implemented new session management strategies. You can see this as your login is now maintained across multiple devices, where the arc code (that we have access to) logs you out upon logging in elsewhere. I also believe that when pg handed over the HN code significant changes occurred including how session data is stored and how that data is utilized to integrate with cloudflare. Obviously I'm making big guesses, because I don't have access to the code, but I'm willing to bet the changes HN has put in place would surprise everyone here.

Maybe I am missing something.... Isn't captcha just a fairly simple robot test (and thus preventing spam)? Or are you suggesting something even simpler? Because I've run a few sites and had tried implementing very simple programmatic obstacles and it really didn't stop the spammers.

https://developers.google.com/recaptcha/docs/versions

I know you (akkartik) have a google account, because I remember when you moved your blog over to google's services (I think they call it 'circles' or some such). I also remember you created a news aggregator application that scraped content. Yes, I know, it was a long time ago in a galaxy far, far away..., but still...

I'm thinking that google identified your scraping work and deemed you a risky robot type, but they also probably correlated your IP from the scraping to your IP from your google services login and tagged you that way. So now, even if your IP changed, they'll continue to have you in their cross-hairs for, like, ever.

Other possible reasons:

a) My cookie acceptance policies are non-standard. (I no longer even remember what they are anymore.)

b) I'm often behind a VPN for work.

c) I'm often on my phone, or tethering from my phone.

Complaints about ReCaptcha are fairly common if you look on HN and so on. You don't have to have run a scraper to hit it, I don't think. I think you may be a robot from the future for never having problems with the pictures of signs and cars :p

Yeah I'm aware of the complaints, but in my mind HN wouldn't be the best resource of information for such an assessment. By default HN members are non-standard in most ways that would matter to ReCaptcha.

It's an interesting dilemma and one that I'm coming up on soon as I plan to release a new app in a few months time. In my case the intended audience for the app is very widespread and not specific to a tech audience. It could be that the vast majority of my users (if I get any - lol) would never have a problem, because the vast majority of people using the net don't know what a VPN is or how to change a cookie setting (just as examples).

I'll have to give it some more thought, but in the mean time, are you aware of any resources on the matter that would be more reflective than HN?

edit: I often find info like this [1]:

  "Different studies conducted by Stanford University, Webnographer and 
  Animoto, showed that there is an approximately 15% abandonment rate when the 
  users are faced with CAPTCHA challenge."

However, every step you put users through is a place where your funnel will leak. So in your place I wouldn't turn on captcha until I actually see spam being a problem.

agreed.

> I have zero urge to help maintain it in Anarki

It's really only a few line of code (probably smaller than a unit test) and it has already exposed json bugs, so I consider it a win all around.

I don't know why the tests keep failing, though, it works locally.

I took a look and found fixes for the unit tests. Before I got into that debugging though, I noticed some problems with the JSON library that I'm not sure what to do with. It turns out those are unrelated to the test failures.

PR = Pull Request.

> Can you link to the post you are referring to?

    arc> (load "lib/client.arc")
    arc> (cdr (mkreq "http://localhost:8080/"))

  (defun create-client (port buffer)
     (let ((socket (usocket:socket-connect "127.0.0.1" port
					 :protocol 
                                         :datagram
					 :element-type 
  '(unsigned-byte 8))))
    (unwind-protect
	 (progn
	   (format t "Sending data~%")
	   (replace buffer #(1 2 3 4 5 6 7 8))
	   (format t "Receiving data~%")
	   (usocket:socket-send socket buffer 8)
	   (usocket:socket-receive socket buffer 8)
	   (format t "~A~%" buffer))
      (usocket:socket-close socket))))