Gosh, lots has happened while I've been away from my computer all day.
Gensyms are easy; it's just
where GENSYM COUNTER is a global variable. So we can stop talking about that aspect of the algorithm.
Jens, just to be clear, at least as of now I don't expect to ask you to implement anything special to define macros programmatically. The metaprogramming tools we already have should be enough.
I do think that if the idea of a macro syntax catches on, you might end up wanting to implement a "Make a macro" GUI. That's something we can't do, I think, with metaprogramming. But I wouldn't even want you to do that until we had a solid joint understanding of what a macro should look like.
About the metaprogramming library getting in the way of the manual:
-
Let me be absolutely clear that what's gotten in the way of the manual, more and more seriously as time passes, is my own psychological inability to do any work at all over the past several years. I should have been keeping up with the software one minor release at a time, and then the major releases wouldn't have been such a killer problem. This has nothing to do with our differing ideas about what to do first; I'm not doing anything, not even debugging the streams version of my Mastermind project, which is just debugging, something I'm allegedly good at. Believe me, nobody is more upset about this state of affairs than I am.
-
The manual chapter on metaprogramming is currently terrible and needs a complete rewrite. Here I guess we have a relevant disagreement; I think that the metaprogramming tools are hard to use for anyone except you, and that meanwhile the rest of the world has had some pretty good ideas about how to design a metaprogramming capability that ordinary programmers can use and understand. They're not my ideas; I'm not doing Not-Invented-Here about this. So, yes, I think that the best and quickest route to good documentation about metaprogramming runs through the path of doing some user-level (library) software development first.
-
I started trying to work on the manual some time ago, and I did it by going through HISTORY.md in chronological order, and that just seemed like an insuperably hard slog, and it occurred to me that I might have better luck if instead I started with the big ideas that need work. It's not just metaprogramming, although that's the most extreme case; there is a chicken-and-egg problem about how to discuss hyperblocks, namely, that the underlying scalar blocks are easy Chapter-One stuff, whereas lists are a couple of chapters later. So either we start out sorta lying about the operators, documenting as if their domains were limited to scalars, and revisiting them after we have lists (and HOFs) under the reader's belt, or we move the discussion of lists way earlier and then talk about the complete domains of the operators. Right now the manual sort of does a little of each of those things. I need to tear out all that and redo it. My idea is that if I'm focusing on a few big ideas, maybe I can get myself to write about them, and get the manual not only in much better shape, but much better positioned to let other people file PRs documenting the myriad little details.
TeXInfo vs. markdown vs. whatever:
Again, what to do depends very much on how much one cares about producing a beautiful PDF. I agree with @sarpnt that by far the easiest thing to do would be to ditch the PDF altogether and just have an online, hyperlinked document with one (web) page per subsection. I'm reluctant to do that, because I want a beautiful PDF. TeX is clearly the best way to accomplish that goal, better than Word, because it gives you better control over positioning and formatting, and doesn't think it's smarter than you are. When I started in the documentation game, it was hard to get pictures in TeX documents because there was no standard notation for it; each implementation of TeX did it differently. But that's all under control now, and in principle it should be possible to get software to do the conversion for us. In practice, though, it'll take a lot of TLC, page by page, to get it perfect -- another Herculean task that I'll never manage. I did that for my books, and I'm glad I did it, but I was younger then and better able to do the impossible.
So, you know, maybe someone should just give me a gold watch and assign the entire documentation department to someone else. And maybe the someone else would just make a web document and be done with it. And I can start reading all the books I've bought but haven't read, and scanning in all my old film photographs, and so on.
Or maybe we can induce someone to do the work of taking the beautiful Word layout and reimplementing it in a medium that can be communally edited.
Or maybe Jens has to write the metaprogramming chapter.
Okay it's 12:30am and I'm going to bed now.