Metaprogramming library

[qw23]

The current Metaprogramming library (introduced in Snap! 10 IIRC) though useful is also kind of basic:

… and (from a user POV) undocumented: for starters, none of these blocks have any help text.

The same applies for blocks that may not be part of the Metaprogramming library, but are often (or even mostly) used for metaprogramming purposes, such as:

(the latter pair are from the Code to Blocks to Code library)

… most of those are also barely, or un-, documented.

Besides, the relevant Snap! data structures are mostly undocumented, too (which is an issue since they are a bit quirky IMO).

My impression is the Snap! development team could use some help here from the user community. A proposed program for this topic is to cooperate on:

1. documentation of all items mentioned above (help texts, Ref. Manual items);
2. collection of additional (useful) metaprogramming tools.

I would also like to experiment with the wiki editing option, esp. for help texts.

If you’re knowledgeable, or even just interested in what metaprogramming is and how you can utilize it, please feel free to participate. (Snap! staff are welcome too, of course)

[qw23]

Documentation texts (wiki)

Online help
Report a script containing all commands required to specify the ringed custom block. The result may be used and modified to create a new custom block that is a variation of an existing block.

Reference manual
(Like above + example:)
… [to be completed]

Online help
Report all custom blocks whose definition contains a call to the ringed block.
Caution: will only work if all input parameters of the ringed block are empty!

Reference Manual
…

Online help
Report a list of: 1. the currently executed script; 2. the block calling this script (1); 3. (if applicable:) the block calling the script that block (2) is part of; 4. (i/a:) the block calling the script that block (3) is part of, and so on. The final item is the current sprite.

Caution: the 1st item (i.e. the currently executed script) is the definition of the call stack block itself! the 2nd item is always . If you require the script in which you’re using call stack, that would be

.

Reference Manual
…

Online help
Report the value of a variable as visible to the caller of the current script.
Input: enter the name of the variable in the input slot.

Reference Manual
[These blocks (incl. `set _of caller to _”) are typically used in macro-like functions, like currently discussed in Ref. Man. 8.0, Ch. 11, section C]

Set the value of a variable as visible to the caller of the current script.
Input: enter the name of the variable in the 1st slot; the new value in the 2nd.

Reference Manual
[to be discussed together with get var () of caller]

Online help
Report a readable version of a nested list.
Use it for inspection of complicated data structures, such as a script that was split into blocks.

Online help
Insert a command, or a series of commands (input slot 1), into a script (slot 2), either before or after (3rd slot) each instance of an existing command within the script (slot 4).
Caution: will only work if the command in slot 4 is a single command, and its parameters are empty.

Reference Manual:
[To be discussed in what’s currently Ch. 11, section B of Ref. Manual 8.0. Together with obvious companion blocks (yet to be developed) for replacing and deleting a command, and another block for replacing a reporter)

Online help
Create a new custom block.
The 1st parameter is an upvar; use it to identify the block when subsequently adding or changing attributes (using SET _ OF BLOCK _ TO _ ). The 2nd (input) parameter is the label of the new block. The 3rd (input) parameter is the block definition, any parameter names may be specified by pressing the ring’s arrow.
The initial type of the new block will be 1 (= “command”), and category 10 (= “other”).
Any parameters within the label are to be specified with a single-character word: “_”; their initial type will be 0 (= “any”). If a block with the specified label already exists, the new block will be labeled “… (2)”.

Online help
Delete a custom block. Caution: all explicit calls to the block in code will be deleted.
If the specified block doesn’t exist, nothing will happen. If it does exist as a primitive block, an error will be raised.

Online help
Set (or change) an attribute of a custom block, to be used preferably immediately after creating it (using DEFINE).

Online help
Report an attribute of any block (either primitive or custom).

Reference manual
[The RM currently does not discuss: primitive, comment, custom?, global?, selector (all from the 1st group), replacables (2nd group), plus all of the attributes of the 3rd group (separators … translators). Besides there should be an appendix with conversion tables from numeric values to meaning, analogue to current Ch. VII, par. G]

Online help
Report a script including its (hidden) environment, that is to say its variables and their values. THIS SCRIPT reports the current script, i.e. the script containing the THIS SCRIPT block. THIS CALLER reports the expression that called the current script (or, if the current script was not called by any other, the current sprite). Finally, THIS CONTINUATION reports the remainder of the script the calling expression is part of, i.e. the part of that script after the calling expression.

Reference Manual
[Ch. X (Continuations) needs to be rewritten (except perhaps par. A) to explain the function and applications of THIS CONTINUATION. Ch. XI, par. C (macros) needs to be rewritten to explain THIS CALLER.]

General topics to be included in the Ref. Manual

• Data structure of code as split by blocks
• The variable scope mechanism

[jens]

Sorry to push back against this ... again. I don't think that metaprogramming is a powerful idea, let alone one that's even worthwhile making "accessible" to novices to the extent that it's being discussed in this forum. Metaprogramming is neither the pinnacle of advanced-ness nor a concept that's used anywhere in professional surroundings, also, as we keep finding out, it can have severe security implications. There are compelling reasons why metaprogramming has been declining since the 1980's. I have added metaprogramming capabilities (and keep adding them for the next release) mainly to enable me/us, i.e. the Snap! development team to quickly and easily make and maintain extensions and little DSLs (3D Beetle, Embroidery, OOP). Yes, you can do that, too, sure. But please, pretty please don't let kids in this forum believe that it's a powerful idea, and that it's even worthwhile studying. It is totally not!

What Snap! encourages learners to really dive deeply into are:

• Recursion & Higher Order Functions ("rings")
• Object Composition ("nested sprites") and Prototypal Inheritance
• Linear Algebra ("hyperblocks")

Those are powerful ideas, those are the Gospel of Programming. Let's not constantly divert attention from the earth-shaking wonders towards the petty quibbles of bored adults.

That said, of course, feel free to contribute whatever you can, but remember, there are parts of Snap that despite being "cool" are meant less for public consumption and more for use by the devs. Those include the JS source, the whole LISP syntax and most of the metaprogramming stuff.

The older I get the more I understand Mitch in his reluctance to add "advanced" features to Scratch, fearing that once these features are part of the language everybody will feel that they need to "master" them in order to be part of the community, and that not mastering them is seen as not being proficient.

[qw23]

So are you saying metaprogramming-enabling blocks are deliberately undocumented? (so as to prevent use by ordinary users)

[jens]

No, of course not. Documentation is always a good idea and it's direly needed and very welcome. This is just a reminder that @bh's documentation for the important, powerful, central concepts is still relevant and doesn't get old or outdated, even while Jens adds new toys.

[qw23]

@jens

[…] even while Jens adds new toys.

BTW, about one of the new toys … IMO call stack does something both complicating and useless: reporting from its own perspective instead of that of its caller.

This is the (still, IMO) improved version:

Or even (more flexible, and recursive):

[bh]

@jens

I don't think that metaprogramming is a powerful idea, let alone one that's even worthwhile making "accessible" to novices to the extent that it's being discussed in this forum. Metaprogramming is neither the pinnacle of advanced-ness nor a concept that's used anywhere in professional surroundings, also, as we keep finding out, it can have severe security implications. There are compelling reasons why metaprogramming has been declining since the 1980's.

I disagree strenuously with this. The reason you believe it is that you've been hanging out with Python programmers, or something. In the Lisp community, metaprogramming, in the form of macros, is still thriving, and is an important part of the "everything first class" philosophy.

In part what you wrote is straw-man argument. For example, metaprogramming doesn't have to be "the pinnacle of" advanced capabilities. But it's among the big ideas. Also, "as we keep finding out" strikes me as an exaggeration. I count one instance, so far.

I know you don't like "everything first class" as a slogan, partly because you don't like slogans at all. But I think maybe you've forgotten what a revelation it was when you finally really understood first class procedures!

I do agree with

• Recursion & Higher Order Functions ("rings")
• Object Composition ("nested sprites") and Prototypal Inheritance
• Linear Algebra ("hyperblocks")

Those are indeed gems of computer science and central to what Snap! offers. One you left out, because everyone outside of the Scratch community thinks it's too obvious to mention, is recursive data structures (lists of lists).

Actually I'm sad that the word "metaprogramming" has become the name under which we talk about macros. From my perspective, the big idea is first class code: the central Lisp idea that a program is just a list. That was kind of an accidental invention; McCarthy first conceptualized Lisp with the kludge of M-expressions and S-expressions, and thought it would be really hard to implement. But it turned out that if you just use S-expressions (code as data), implementing Lisp becomes trivial, much easier than implementing any other language. And once you have code-as-data as a central organizing idea, macros aren't a hairy obscure idea with a five-syllable name; if code is data, you can manipulate the code in your program.

The early version of macros, FEXPRs, is conceptually very simple: A macro is a function whose return value is a piece of code, which is then evaluated in place of the macro call, in the context of its caller. Also called "eval twice" macros; you eval the macro call itself, and then you eval the code you get back. This is how macros work in Berkeley Logo, and it was really simple and straightforward to implement them.

The later invention of hygienic macros was just an extension to the macro level of the switch from dynamic to lexical scope.

One of the costs of being a visual, block language is that code isn't data in this straightforward sense. That's why you had to invent SPLIT BY BLOCKS and JOIN of syntax lists. Also, using an explicit THIS CALLER instead of the implicit caller's-context evaluation of a FEXPR gives finer control, but at the cost of making the user have a more explicit model of what's entailed in switching contexts. All these things contribute to your sense that metaprogramming isn't for users. But macros are for users! Macros are an abstraction over code, just like all the other powerful abstractions we offer.

There's a reason why programming language experts all think in Scheme, treating every other language as just obfuscated Scheme. It's because that gives them abstractions over code!

So, yeah, that's why I think it's important to give users macros as an abstraction over metaprogramming, and why I think that'll make the whole idea much more documentable.

[sarpnt]

metaprogramming usually isn't common at runtime like what's done in snap, but it is very common at compile time. tons of rust libraries rely heavily on macros to make things extremely easy. check out rocket. you import the library, put the url above your function, and your webserver is done.

i would be extremely interested in custom hat blocks for exactly this reason. imagine having a 3d engine game project, where sprites can have hat blocks for things like "when touched by [sprite]". maybe even just write some custom data as a stack of blocks by not running them but instead using metaprogramming to parse the blocks directly. much nicer than trying to manage a super long list. it could work very similar to broadcasts except it looks much nicer and you can add custom inputs.

making a whole block based editor and not being able to use the blocks as data is a waste. block based editing is useful for a whole lot more than code, and complex projects usually need complex data.

[qw23]

@bh

you've been hanging out with Python programmers

10 Lashes!

O, by the way: I created this topic for user cooperation - not for staff rants, from post #3 (@jens: there’s a nice safe zone especially for those), or bromance breakdowns.

So, can we make a clean new start from here?

[jens]

I like Alan's remarks about metaprogramming (and I think you'll like them, too, because, as with everything Alan says even opposing viewpoints find themselves reinforced by his aphorisms^^)

[bh]

Oh now I'm confused, because he uses the words "metaprogramming" and "macros" as if they mean two different, and sorta competing, things. If that's the way you use them too, no wonder we're not communicating. I took the thrust of Alan's comment to be "of course you have macros but metaprogramming is a step beyond that." If we could agree on the "of course you have macros" part, that'd be great. :~)