Hygienic Macros Implementation in Snap

Continuing the discussion from Updating the Snap! Manual:

Long Quote

Alright, so based on this post I’ve been trying to make macros in Snap, and here is what I’ve got:

I started by playing around with the scoping of variables in custom blocks. I managed to create this:


which results in this:

The definition of “inherit caller context” first declares the existing script vars (aka “num”) (you’ll see why I had to use the create variables extension later) in the caller’s context and then runs its continuation in the caller’s context as well:

That does have side effects, however:

From there I realized my first problem - the means by which I had implemented the “inherit caller context” block didn’t allow for reporting:

macros script pic (5)

Thus, I devised a new, separate block for reporting in the caller:

The next step was getting the creation of script variables to work, because, as it was, this was what happened:



So, I began experimenting with ways to create a script variable in the caller’s context. However, neither this:

nor this:

actually worked. At last I realized why. Run (and tell) create their own kind of sub-context which notations like IF do not. This is best explained in pictures:

IF is doing something different from RUN - the code inside the c-shape doesn’t have a new sub-context - it has the same context as the code outside. This is good, expected behaviour. But it also means we can’t create script variables in a caller’s context. If we were to do this, for example:

We don’t gain access to the main script’s context, we create a new branch context from the main script’s caller. However, I have encountered this issue before and know the solution - the create variables library:

This also works from within custom blocks:

macros script pic (18)
Now all I had to do was create a block to replace every SCRIPT VARIABLES block with the block from the create variables library, as well as every report with my version of report:

Alright, let’s take stock. We are now ready to create a block that will take a desired definition and make it function in the caller - here’s an example of what that definition might look like as written by the user:

and then once modified via metaprogramming:

And it functions!

However, it is at this point that we start having problems. Because the Create Variables library’s script variable scoping works differently (which we need), there are some unintended side effects - script variables are created in the outermost context, not the desired caller context. Let’s take our “create foo variable” block from earlier, and see what happens when we wrap it in a ring:

Compare this to the built in script variables:

There are more problems too. I don’t know if the behaviour shown above is intended or not, so it could be patched at a later date. Additionally, let’s take a look at the IF/ELSE example @bh gave. The individual c-shape slots need to have their script variables and report blocks replaced with my versions, and this is something we have to do at runtime this time:

This means that our if notation will be slowed down and won’t be able to be visible stepped for debugging. This is far from ideal. We need a better way to run a script without creating a new context for it - like the primitive notations…

I tried modifying the definition of the primitive IF/ELSE to make the first input non-static, but this, unsurprisingly, didn’t work. The primitive definition, when toggled on, isn’t actually taking a ringified script as input, but something else:


This would be the case for all primitive notations. I needed a library notation instead. But most library notations are built in Snap and thus have the same problem:

I needed a notation that used an EXTENSION block for its definition. And then it hit me - SAFELY TRY:

I noted that I should run the script in the second input to avoid catching errors that should have been thrown, and then proceeded to make a “run w/o new context” block:


Taking this back to our IF/ELSE example, it works as well:


Reporting works too:

However, here is where things get a little tricky. You probably only want this kind of behaviour if your input is a c-shape. If it is a command or reporter ring, it might have input parameters and should probably behave like the built in RUN. Thus, I have written the REWRITE block to create a definition like this:

This also ensures that we don’t have to worry about how to add inputs to this version of run - if you’re adding inputs, that means it was in a grey ring and should get its own sub-context (and needs to have one for those pesky parameter not-variables). Here is how you use REWRITE at this time:

and its result:

I’ll be sharing the project and adding the hygienic bit soon.

This is an amazing explanation. As someone who has read the manual, I would honestly add this to it. Can’t wait for the project.
this doesn’t exactly seem like a collaboration though. You may want to switch the category to Share your Projects.

what happens if the stuff that is being ran errors?

The script you enter is in the catch, not try:

I honestly don’t know anything about macros aside from what I’ve read on the forums. I was thinking that @bh in particular could help with the user interface and making sure that what I’ve made is actually doing what macros are supposed to. We’ll see.

“True” and “false case” are already in the caller context until you split/join blocks.
Also, the term “hygienic” means “sandboxed” i.e., isolated from the parent context, as opposed to simple text-injected macros.

Can you express your problem upfront in terms of intended use cases or examples?
Looking at your final solution, nothing is left from the intended simplicity…
If I understand correctly, you are trying to address, so far, two main problems:

  1. Report from macro body as in the caller context
  2. Create variables in the context of the caller

I’m not sure if it’s your goal or an intermediate problem.

Yeah, that example @bh gave didn’t solve anything. “true case” and “false case” remained sub-contexts to the caller context (unable to report or create script variables in the caller).

Based on what I’d read from @bh, my understanding was that the “hygienic” par meant that

which I have plans to do. However, I am unclear as to whether this understanding was incorrect base on what you’ve written. Clarification would be appreciated.

Yeah, I plan on making the metaprogramming more complex to ensure the final definition is simpler.

Please bear in mind that I only know as much about macros as has been written on the forums by @bh. Thus far, all I’ve been trying to do is implement the concepts @bh showed in his original post. However, these are the two use cases I am aware of:

  1. Custom notations (I think that is the right word) - the “Iteration and Composition” library has plenty of examples. In CATCHTAG, for example, this happens:

    Using the blocks I’ve developed, this can be fixed:
  2. @bh has talked about how certain libraries have setup blocks that will only be used within custom block definitions. If I’m understanding correctly, this is the kind of thing that might be useful as a macro (bear in mind that this is a basic example). Let’s say you’re creating a colors library and always want to deal with your color inputs as RGBA. You make a macro:

    And then use it in your blocks:

Thanks. The extension block for that function is incredibly confusing.


Anyway, once it's finished, I would love to see the inherit caller context block to be added to the metaprogramming library.

In my first post, we were at the point where the IF/ELSE example was working properly, but this still doesn’t work:

That’s because my REWRITE block had abandoned the “script vars and report properly” block from earlier. This was because of the scoping concerns with the Create Variables library. After a little bit of fiddling, I got the “create foo variable” block to work:


macros script pic (44)

From there I designed a “macro variables” block for hybridized scoping:

macros script pic (46)
And then modified the REWRITE block accordingly so that this script produces the definition above:

We are now done the basic macro functionality. What we still have to do - hygienic macros and prettyifying the REWRITE function’s result.

Unfortunately, some stuff has come up IRL (in real life) and I won’t be active on Snap for a little while. When I return, I’ll return to this project (probably).

Wow! This is fantastic research. Thank you so much!

“Accessing caller-local variables and reporting a value from the caller” is the answer to “why do you need macros,” not the answer to “what is a macro?”

A macro is a rewriting of an input expression into an equivalent non-macro expression. It’s a little tricky giving examples in Snap! because historically we’ve invented a handful of features to solve specific problems that would be solved by macros in Lisp, especially upvars and unevaluated input types. So most simple examples can already be done in Snap! using those features. The classic case is the rewriting of
untitled script pic (7)
into


This has long been possible in Snap! without rewriting because of the way upvars work: the variable can have two different names, one inside and one outside the LET definition, and the extent of the upvar extends beyond the LET block itself, so it persists in the caller. But imagine that we didn’t have upvars. It would then be necessary to say

Note in this script that SCRIPT VARIABLES’s input isn’t an upvar, just a plain variable whose value becomes the name of the new script variable. Also note that SET’s first input isn’t a dropdown entry but rather a (plain) variable whose value specifies the name of the variable to change. So you’d say
untitled script pic (13)
and that would be equivalent to saying

(For a while in BYOB we did in fact allow dropping a variable oval onto the dropdown menu in the SET block, but users invariably thought that that meant to change the value of the variable whose name they could see in that first SET input, rather than to change the value of the variable whose name is the value of the variable they see there, so we disallowed it altogether.)

To answer @dardoro’s point about hygiene, note that the actual text “foo” doesn’t appear anywhere in the REWRITE block; instead, the name VARIABLE is used in the rewriter’s scope to represent the name FOO in the caller’s scope. If the user of this LET macro happened to use the name VARIABLE instead of the name FOO, there would be two different variables named VARIABLE: the one inside the rewrite rule, whose value is “VARIABLE”, and the one in the caller, whose value is 3.

This is all easier to put together in Lisp, where a program is just data. In Snap!, I’m making this up as I go along, but imagine that the input names in the header of the REWRITE AS are indented instead of raised, or something, to denote that they’re in the scope of the rewrite rule. And maybe there’d be a
untitled script pic (15)
block you could use to make more indented variables belonging to the rewrite rule itself.

Input groups are the latest Snap! feature to solve a macro-writing problem, namely inventing new syntax forms. Suppose I want a Lisp-style COND block for conditionals. In Lisp it looks like

(cond ((test1) (result1)) ((test2) (result2))... (else (last result)))

In Snap!, I’m imagining we could say

Things to note here: First of all, all of the inputs to the COND macro are unevaluated, automatically, by virtue of it being a macro. (In traditional Lisp, this is the only way to get unevaluated inputs; you can’t mark individual inputs as unevaluated.) So, in the IF expression that specifies the rewritten form of the macro, the
untitled script pic (17)
part is testing the unevaluated TEST1, i.e., testing whether the actual text “else” appears at this position in the macro call. If so, the expression in the VALUE1 slot of the macro call is reported as the rewritten text of the macro. If not, the stuff in the ring is reported as the rewritten text. No part of the original macro call is actually evaluated until, as the final step, whatever expression the rewrite rule provides is evaluated in the context of the caller.

The second thing to note is that, given input groups, COND could be written in Snap! as an ordinary procedure, because it doesn’t create variables in the caller and doesn’t report a value from the caller. But macros provide a different metaphor for thinking about new syntactic forms.

Here is the bible on hygienic macros:
hygienic-hopl.pdf (1.6 MB)

Note that “inherit caller context” actually pollutes the caller context with clones of block parameters. So it’s not hygienic by any means. Also not necessary for the intended purpose.
Based on the former example

Example color->RGBA can be implemented with the unevaluated inputs as a variable reference




Inlining a script into this script context as a side effect of the err_try() is very clever idea.
Also the extra context added to run , makes the edited primitives non working. So maybe “run inline” should be a standard part of the language.

So it’s not hygienic by any means.

Oh, sorry, I didn’t mean to disagree with you about @mark4sisb’s code. I was talking about my (hypothetical) version.