> Which is the same for voting and page generation. [...] Right now there's a big cost on the servers because all the work is done on the servers.
Yea, I don't think something like checking whether a comment is member of the list of hidden items by a user would really add workload of any significance.
> [...] you might as well say the same for voting + all the html creation and just write the whole thing in js where you only fetch data.
Does that not lead to an awful lot of traffic sometimes? (I'm imagining a version of News that would transmit all submitted content to let the client do the sorting and searching instead of doing it on the server.)
> I have no idea how to do it efficiently, though, since presumably each close/open operation would also mean an http request and possibly a file write.
Which is the same for voting and page generation.
ie. Right now there's a big cost on the servers because all the work is done on the servers. If you start looking at it from the perspective of not adopting that cost then you might as well say the same for voting + all the html creation and just write the whole thing in js where you only fetch data.
This is the slippery slope that lead me to writing apps in clojurescript. For me, the workload may get increased, but much of the operational costs get distributed across the users and on their hardware.
I'm working on improving the HTML, I don't know when if ever I'll get around to collapsible comments.
I was just pointing out that HN's own implementation is server-side, but doesn't have to be. As with voting, the js just makes changes to the DOM that shadow the actual work on the backend. I have no idea how to do it efficiently, though, since presumably each close/open operation would also mean an http request and possibly a file write.
> Storing state on the server would also allow for it to work w/o JavaScript.
I agree for this case (surprise, surprise...). This app was/is designed to work without js (mostly). If there's much more of a departure from this design and any real dependancy on js begins, well really the whole app should get re-written.
> But the interesting question would be where to store state? [...] It could probably also be done with local storage but that wouldn't persist across browsers.
Cool that you're working on this. Storing state on the server would also allow for it to work w/o JavaScript. (Yes, yes, I know, maybe everyone are not that much into progressive enhancement.)
>I would personally consider it less hacky to represent the comment tree as a tree of nested list elements in the HTML in the first place. I just don't get this fear of `<li>`.
Apparently pg did it in part to be "unPC" in regards to HTML. (see the linked quote in the 'inline javascript' thread[0].) It is weird that, in the name of exploratory programming, he chose the most complicated and finicky layout imaginable just to stick it to the W3c and their rules, man.
Fortunately, I think Anarki is already moving in the right direction. Table layout should be almost entirely removed (I think polls still use it) and I've been adding css classes and IDs to make the HTML more amenable to scripting. Unfortunately, this means we can't really use anything from HN without rewriting it. There's still a bit more work to do in removing old html like fontcolor, etc.
But the interesting question would be where to store state? HN appears to have endpoints that handle it (?collapse=id) so I assume there is just a list of closed ids for each user somewhere on the server. It could probably also be done with local storage but that wouldn't persist across browsers.
I was looking at implementing that in Anarki a while ago thinking it would be simple.
But take a look at how it's implemented on HN. It's quite ... hacky, to say it nicely.
First, the way comments are represented in the HTML is as table rows stacked on top of each other, and indentation is done by stretching a transparent 1x1 gif. Assuming collapsible comments are to be implemented in JavaScript, you're in a bit of a pickle, because the comment tree has been flattened into a long list thereby making it difficult process in a sane way.
So how does HN do it? Well, have a look at the `ind` function in hn.js[0]:
function ind (el) { return (byTag(el, 'img')[0] || {}).width }
Oh, yes, there is actually a function that returns the width of the invisible strecthed 1x1 gif used for indentation. And, `kidsOf` is implemented using `ind`, and so on.
Some time back, I changed Anarki to use fixed-width spaces for indentation in order to work on text-only browsers. I'm not sure that is any better, so if you want to take the same approach as HN, you'd have to revert Anarki back to invisible image indentation. And maybe you could just nab the collabsible comments code straight from hn.js?
I would personally consider it less hacky to represent the comment tree as a tree of nested list elements in the HTML in the first place. I just don't get this fear of `<li>`.
But whichever way you go about it, you should totally give it a shot!
But, well, even if you implement it in Anarki, it's not gonna go in this forum, I guess.
Hmm, all right. I didn't want to believe that was the point you were trying to make. In that case, I think I must not have conveyed anything very clearly in the `correcting-arc-read` comment.
I've been trying to respond to this, which was your response to that one:
"Racket also has mutable hashes created using `make-hash` rather than `hash`. It could just be that the tables are not serialised as something Racket reads as mutable hashes when deserialising it back again?"
In the `correcting-arc-read` comment, I used `make-hash` in the implementation of `correcting-arc-read`, so I assumed your first sentence was for the edification of others. I found something to respond to in the second, which kinda pattern-matched to a question I had on my mind already ("Can't the Racket reader just construct a mutable table since what was written was a mutable table?").
As for my response to your "No..?"...
One of the purposes of `correcting-arc-read` is that (when it's used as a drop-in replacement for Arc's `read`) it makes `readfile1` return mutable tables. So if anyone had to be convinced that a reader that returned mutable hashes could be implemented at all in Racket, I thought I had shown that already. When it looked like you might be trying to convince me of something I had already shown, I dismissed that idea and thought you were instead trying to clarify what your proposed alternative to `correcting-arc-read` was.
Seems like I've been making bad assumptions and that as a result I've been mostly talking to myself. Sorry about that. :)
Maybe I oughta clarify some more of the content of that `correcting-arc-read` comment, but I'm not sure what parts. And do you figure there are any points you were making that I could still respond to? I'd better not try to assume what those are again. :-p
> Looks like you're proposing to use a two-stage serialization format.
I wasn't really proposing anything, only pointing out that it's not the case that Racket never can read mutable hashes, and then illustrating that with a code example.
The Racket reader reads a seven-element list there, not a mutable hash.
Looks like you're proposing to use a two-stage serialization format. One stage is `read` and `write`, and the other is `serialize` and `deserialize`. At the level of language design, what's the point of designing it this way? (Why does Racket design it this way, anyway?)
I can see not wanting to have cyclic syntax or syntax-with-sharing in the core language just because it's a pain in the neck to parse and another pain in the neck to interpret. Maybe that's reason enough to have a separate `racket/serialize` library.
But isn't the main issue here that Arc's `read` creates immutable tables when the rest of the language only deals with mutable ones? The mutable tables go through `write` just fine, but `read` doesn't read the same kind of value that was written out. If and when this situation is improved, I don't see where `racket/serialize` would come into play.
"It could just be that the tables are not serialised as something Racket reads as mutable hashes when deserialising it back again?"
I'm pretty sure the Racket reader never reads a mutable hash, but that it's possible for a custom reader extension to do it.
Some of Racket's approach to module encapsulation depends on syntax objects being deeply immutable. In particular, a module can export a macro that expands to (set! my-private-module-binding 20) but which uses `syntax-protect` so that the client of that macro can't use the `my-private-module-binding` identifier for any other purpose. If the lists constituting a program's syntax were usually mutable, then it would be hard to stop the client from just mutating that expansion to make something like (list my-private-module-binding 20), giving it access to bindings that were meant to be private.
I think this is why Racket's `read-syntax` creates immutable data. As for why `read` does it too, I think it's just a case of `read` being a relatively unused feature in Racket. They don't have many use cases for `read` that they wouldn't rather use `read-syntax` for, so they don't usually have reasons for the behavior of `read` to diverge from the behavior of `read-syntax`.
All this being said, they could pretty easily add built-in syntaxes for mutable hashes, but I think it just hasn't come up. Who ever really wants to read a mutable value? In Racket, where immutable values are well-supported by the core libraries, you aren't gonna need it. In the rare case you want it, it's easy enough to do a deep traversal to build the mutable structure you're looking for (like my example `correcting-arc-read` does).
It only comes up as a particular problem in Arc. Arc's language design doesn't account for the existence of immutable values at all, so working around them when they appear can be a bit quirky.
I have an old fork (https://github.com/akkartik/arc) that has an extensible generic pair of functions called serialize and unserialize which emit not just the value but also tagged with its type. read and write are built atop them.
> I assume the problem with reading tables using `read` is that Racket's reader constructs immutable hashes.
Racket also has mutable hashes created using `make-hash`[0] rather than `hash`. It could just be that the tables are not serialised as something Racket reads as mutable hashes when deserialising it back again?
Even if you merge `writefile` and `save-table` like this, but not `readfile1` and `read-table`, then people still need to know, at development time, what type of data is in the file in order to read it, so they might as well use a type-specific way to write it as well. Unfortunately, merging `readfile1` and `read-table` isn't really possible, since their serialized representations overlap; they can't reconstitute information that was never written to the file to begin with.
From a bigger-picture point of view, this seems like it would become a non-issue once Arc had its own reader. I assume the problem with reading tables using `read` is that Racket's reader constructs immutable hashes. An Arc-specific reader would naturally construct Arc's mutable tables instead.
Doesn't Racket's reader give us similar problems in that it reads immutable strings and cons cells, too? So these problems could all be approached as a single project.
In the short term, it's not a project that would need a whole new reader. It could just be an adaptation of Racket's existing reader... something like this:
(define (correcting-arc-read in)
(let loop ([result (read in)])
(match result
; TODO: See if this should construct a Racket mutable cons cell
; instead (`mcons`). Right now this just creates an immutable
; one, which should be fine since Arc uses an unsafe technique
; to mutate those.
[(cons a b) (cons (loop a) (loop b))]
[ (? hash?)
(make-hash
(map
(match-lambda [(cons k v) (cons (loop k) (loop v))])
(hash->list result)))]
[ (? string?)
; We construct a new mutable string with the same content as
; `result`.
(substring result 0)]
; We handle tagged values, which are represented as mutable
; Racket vectors.
[(? vector?) (list->vector (map loop (vector->list result)))]
; We handle various atomic values. (TODO: Add more of these
; cases until we've accounted for every writable type Arc
; supports. Alternatively, just make this a catch-all
; `[_ result]`.)
[(? number?) result]
[(? symbol?) result])))
Writing Arc's `queue` type might be tricky, since that representation relies on sharing. It's possible queues (and other tagged values in general) should have a customized read and write behavior.
Yup, just rename the macro and search-and-replace all calls. Then try running all tests! (Updating incompatibilities at the top of arc.arc is also appreciated.)
As i4cu pointed out, it's because write can't handle tables. There _is_ a save-list, but it's called writefile, and it works not just on lists but most primitive datatypes. However, it doesn't work on tables.
The naming isn't very consistent here. Arguably writefile should be called save-file. We should preserve the write prefix for operations that write to opened Racket file ports (https://docs.racket-lang.org/reference/file-ports.html) and save for operations that write to provided filenames. That convention is followed by save-table and write-table.
And of course we could simplify names still further if we took i4cu's suggestion and had write work with tables.
The article uses patterns as an example of a non-expression syntax. Another would be what might be called access expressions that setf can use. And CL does have defsetf (as well as define-modify-macro and such) to extend it (as does Arc with = and defset).
Note that I say "access expression", though—it's designed such that (setf (car x) ...) will modify what (car x) subsequently would return (assuming that "..." doesn't rebind x). But there's also a symmetry in the pattern-matching stuff: if (cons x y) created an object, then the pattern (cons x y) will destructure the object and bind x and y to what they originally were. Perhaps that's simply good practice in designing new syntaxes.
> If that is something you have published, it'd be fun to see, whether it's finished or not.
I actually tried to look it out the other day during this conv, but it's buried somewhere unavailable right now. If I find/get to it I'll post.
> That multitude of libraries with similar purpose may be useful in some cases, sure, but also potentially a bit overwhelming for beginners, so I guess that's why I found it easier to get started with Racket.
Agreed. Navigating the volume libraries and the options available is a real pain in the beginning, but once you get past that, then it's not bad at all. At the same time, take a look at the quality of Clojure's Redis Carmine Library vs. Racket's Redis Libraries. Miles apart.
> Can these servlets live on a separate server so that the data can be shared between web servers?
Probably. I assume that serializable continuations[0] from stateless servlets can just be stored wherever, like in Redis or something, instead of in the memory of one server.
> I ported HN to Clojure
If that is something you have published, it'd be fun to see, whether it's finished or not.
> Uh oh, you're getting me interested in Racket now.
My impression is that Clojure is faster, less verbose partly due to clever syntax and provides more immutable data structures than Racket. But when it comes to documentation and error messages, I find Racket more coherent and comprehensible.
Say, if I wanted to connect to a SQL databse, with Racket I'd use the DB module[1], end of discussion. But with Clojure there's Korma, ClojureQL, Persist, HoneySQL, Yesql, a JDBC wrapper from Clojure contrib, SQLingvo, oj, Suricatta, aggregate, Hyperion, HugSQL, and probably a few more[2][3]. That multitude of libraries with similar purpose may be useful in some cases, sure, but also potentially a bit overwhelming for beginners, so I guess that's why I found it easier to get started with Racket.
When I was referring to load balancing and centralizing the data I was referring to many web servers sharing a centralized/external source for auth/session data.
I'm unfamiliar with racket's web server 'servlets'. The docs are little unclear (at least to me). Can these servlets live on a separate server so that the data can be shared between web servers? I'm guessing that was/is not a requirement for you, but I'm just interested in knowing if that's how it can work.
Uh oh, you're getting me interested in Racket now. I can't have that... I have too many projects :)
edit: I guess at the end of the day these servlets are web-servers right, so you can, even if you have to do it over http and build an api.
> I did the same thing, about 6 or 7 years ago, that you're doing now. I ported HN to Clojure (which is actually how I learned Clojure).
Cool!
> I needed to centralize the data for the authentication and fnid session info. I think Arc calls them fnids... You probably know better than I do now, but Arc has all this code to expire these session fnids and so, for me, Redis was just a good fit for that task.
The Racket web server is quite "batteries included" and comes with these different managers[0] for dealing with expiration of sessions/continuations, such as the LRU manager:
> The memory limit is set to `memory-threshold` bytes. Continuations start with 24 life points. Life points are deducted at the rate of one every 10 minutes, or one every 5 seconds when the memory limit is exceeded. Hence the maximum life time for a continuation is 4 hours, and the minimum is 2 minutes.
> If the load on the server spikes—as indicated by memory usage—the server will quickly expire continuations, until the memory is back under control. If the load stays low, it will still efficiently expire old continuations.
> PLOP (PLace Oriented Programming), which Datalog does address.
Yeah, I was thinking more along the lines that Datomic has built-in functionality to address the caching, cache eviction, and indexing that goes along with all that data accumulation. But you're correct, DataLog does accumulate facts.
> Interesting. Just checked news.arc, and yes `initload*` is set to 15000.
I did the same thing, about 6 or 7 years ago, that you're doing now. I ported HN to Clojure (which is actually how I learned Clojure). If memory serves me correctly when I was doing the work I realized I needed a real DB if I wanted to support load balancing. i.e. I needed to centralize the data for the authentication and fnid session info. I think Arc calls them fnids... You probably know better than I do now, but Arc has all this code to expire these session fnids and so, for me, Redis was just a good fit for that task.
Anyways, I'll be sure to take a look at the final result of your work.
> Datomic uses DataLog as part of its query language, but that's pretty much where the comparison should end. Things like "treating the database as a value", and features such as data accretion that Rich talks about have nothing to do with DataLog.
I'm not sure I totally agree with this. I think that apart from talking about the design of Datomic, he also has a more general point against what he calls PLOP (PLace Oriented Programming), which Datalog does address.
For example in plain Racket a value is lost if something else is put in its place:
Hickey is also mentioning how git doesn't do PLOP in that it doesn't throw out your commit history (without you asking it to do so).
> The reasons I mention Redis is that the HN app is very well suited to it. HN only keeps 'x' amount of data in memory. And in Redis the data lives in memory. Also Redis allows you to set expiry times on data for auto eviction [6].
Interesting. Just checked news.arc, and yes `initload*` is set to 15000. Interesting idea from Redis with expiry times. I'll check it out. I hadn't considered the scenario of storing text enough to max out on memory, because it would probably be premature optimisation, but good to keep in mind. I'd like to give Redis/Rackdis a try; thanks for the suggestion. I've been hosting an Etherpad Lite instance, and Redis was painless to setup.
> I'm pointing this out because it seems to me that you're doing (or are going to be doing) a lot of work that may not be worth it for what you're trying to accomplish.
Yes, my priorities here are definitely to make the code as brief and simple as possible, and to not have to do to much work. With plain Datalog it's very little work to timestamp a fact, and it's also kind of necessary, e.g. to figure out which fact is most recent, when previous facts are not removed. I'm just trying to get the gist of Hickey's ideas here.
