Streams library 2.0 development, part 1

[qw23]

@bh

But this:

generates all 10,000 numbers right now, because it makes a plain list, not a stream!

I’ve been aware of that. My demonstration was meant to show that a chain with two streams and a stream-to-stream (map) operation in between takes much more processing time than a chain with one stream and the mapping in front of or behind the stream. I had 10k items generated all at once in order to create some mass and make the runtime difference visible (1k would have been fine, too; but with just a single item the result would have been unreliable due to random influences).

Even so I made a mistake:

is not realistic, it should have been like:

Multiplicate stream items
I mentioned I was thinking about a spawn operation. The idea is a single input stream element may generate multiple output stream elements, individually mapped. So e.g. (. x .) -> (. x² (print x) .). Each of the outputs would also be conditional - it’s a kind of antagonist of keep. Such operation could facilitate splitting a stream: items may be marked such that consumers will recognize items that may be of relevance for them.

I dipped a toe in the water by constructing a simple item duplicator:

In the end stream-spawn should do something like the following, but with streams.

Actually, spawn can do anything list comprehension can do, and more.
Perhaps spawn is not the best of names, as it usually refers to new processes, not data items; multiply may be better, it’s just that it could be mixed up with multiplication as an arithmetic operator.

[bh]

OK, I see. I still would rather have a HOF compiler than a user-visible complicated toolkit. Nesting HOFs is such a great abstraction!

Yeah, not MULTIPLY. :~) Although it's ugly, I think the most expressive name would be ONE-TO-MANY. (Yeah, "many" could be zero. Still.) Or maybe CALL MULTIPLE?

That orange block needs work, too. I don't like the "else false" because (I'm making this complaint a lot lately...) it rules out a list of Booleans. I'd be inclined to have it not report anything, and have SPAWN (whatever it ends up being called) catch the resulting error.

And that always-empty input slot at the end just screams out for dynamic scope! Or maybe metaprogramming to add a not-user-visible input, or something. So just IF _ REPORT _. (No parentheses needed, especially since the ring groups whatever's inside it.)

[qw23]

@bh

That orange block needs work, too. I don't like the "else false" because (I'm making this complaint a lot lately...) it rules out a list of Booleans. I'd be inclined to have it not report anything, and have SPAWN (whatever it ends up being called) catch the resulting error.

“Else false” is not applicable any more now, and no potential results are ruled out. The output is always a list, carrying 0, 1 or more items. These items might be Booleans, or lists of Booleans, or whatever.

@bh

And that always-empty input slot at the end just screams out for dynamic scope! Or maybe metaprogramming to add a not-user-visible input, or something. So just IF _ REPORT _. (No parentheses needed, especially since the ring groups whatever's inside it.)

I don’t like the always-empty input slot either, but will also keep away from metaprogramming if possible - do you have an example of what is meant with dynamic scoping?
The parentheses were to signify that the result is always a list; they don’t do anything of course; but perhaps they only confuse.

[bh]

Oh that's much nicer!

Two name things: 1-2-MANY-PROJECTOR is misleading because, importantly, it might report an empty list. CALL MULTIPLE? Or of course it could be 0-1-2-MANY-PROJECTOR. :~)

And IF _ MAP _ is misleading because MAP implies calling a function for each item of a list, whereas this just calls the function once. (I read the code, I know that technically it doesn't call anything at all, but we're not telling the users that...) I'd make it IF _ CALL _ or even IF _ INCLUDE _ especially if you changed that second input to Any(Unevaluated).

---

Dynamic scope: In your previous version, the reason for the empty input slot at the end of ...W/INPUT () is that that orange block, in an inner ring, needs access to the input to the outer ring. The way we use empty slots as implicit placeholders for inputs makes this hard to talk about, so imagine your example looked like

So the inner procedure (the one that calls IF_REPORT) needs access to the variable VALUE from the outer procedure (the one that calls SPAWN). But if you just drag an orange VALUE oval into the definition of IF_REPORT, you'll get a no-such-variable error. That's the problem you're solving by putting VALUE into an input slot of IF_REPORT, so it can be used by the procedures it calls.

But if Snap! were dynamically scoped (instead of being lexically scoped), then IF_REPORT would automatically have access to the variables of the procedure that calls it, in this case SPAWN.

By the way, you don't have to go whole hog metaprogramming to get your caller's inputs. You can just say

Neat, huh? :~) This is a sort of backdoor dynamic scope. You can also ask your caller for the value of a specific variable with a ringed orange oval.

---

One thing I don't like about 1-2-MANY-PROJECTOR is that its input expressions have to be IF_REPORT_ calls. You can imagine wanting to include an expression that's not conditional on anything. I know, you can just put True in the first slot, or you can write another orange block with only one input that supplies the True itself. But as a naive user, I want to just drag in an expression without an orange block.

But maybe I'm just quibbling.

[qw23]

@bh

Two name things: 1-2-MANY-PROJECTOR is misleading because, importantly, it might report an empty list. CALL MULTIPLE? Or of course it could be 0-1-2-MANY-PROJECTOR. :~)

The block’s name is actually a pun on 1 (original value) to (= 2) many (image elements). And “many” in this context may be any non-negative integer, including 0. But if even you don’t understand it, the pun is probably to far-fetched.

@bh

I'd make it IF _ CALL _ or even IF _ INCLUDE _ especially if you changed that second input to Any(Unevaluated).

I did something like that.

Thx4 the explanation of dynamic scoping. I may not going to be using it here, but probably it will come in handy at some other occasion.

@bh

One thing I don't like about 1-2-MANY-PROJECTOR is that its input expressions have to be IF_REPORT_ calls. You can imagine wanting to include an expression that's not conditional on anything. I know, you can just put True in the first slot, or you can write another orange block with only one input that supplies the True itself. But as a naive user, I want to just drag in an expression without an orange block.

I see what you mean. The issue here is that if a user doesn’t apply an dedicated block, they must (in the current version) insert an expression producing a list of two lambda’s, which is even more complicated.
Alternatively (like in the previous version) the inserted expression would have to produce a list of 1 or 0 items, which isn’t obvious either.
Ideally the user would be enabled to insert any expression, such as or . The latter, however, always reports something (i.e. the value of the input). I tried a special version if (condition) then (expression), not reporting anything if the condition fails. However that causes an error (“hmm ... reporter didn’t report”) that can’t be caught by safely try … any suggestions?

Anomaly with Any (unevaluated)
For now I’m using an auxiliary block if (condition) apply (expression) (input types: Boolean (uneval.), and Any (uneval.), respectively). For the first input, I specified true as default; for the second I would like to have id of as default, but haven’t found out how yet - if no input is specified, the corresponding output will be: : when called, it produces NIL, and I don’t know how I can test for it in advance (I even tried comparing it with the output of an ad-hoc test reporter called Any (uneval.), but it was different (≠) ! the same holds for literals (constants), whereas comparing functions is OK. Try it here. My speculation: with Any (uneval.), if the input value is not a reporter, it is replaced internally by a unique ad-hoc reporter that will reproduce the value. But knowing so (if it is true) doesn’t solve the issue …

Why doesn’t this work? Solved! See post #118
I made a block to map the 1-2-many-projector (now renamed (item) -> 0 .. many images, using (rules...) over a stream. Though it’s very little code: it doesn’t work, even after hours of testing. More precisely: it gets stuck in an infinite loop. I am using a block heads (items) in front of stream (stream), which appears to work well (but I wonder if it actually does). Perhaps you (@bh), or anyone else reading this, could have a look - it’s probably something trivial I have consistently been missing. Same link as above. Thx in advance!

[bh]

@qw23

But if even you don’t understand it, the pun is probably too far-fetched.

Long ago I read about some alleged primitive tribe somewhere whose language has only the number names "one," "two," and "many," so that's what I thought the name meant, and yeah, the two/to pun went over my head. (I think I later learned that the story about that tribe wasn't true, but was a misunderstanding by some anthropologist.)

@qw23

However that causes an error (“hmm ... reporter didn’t report”) that can’t be caught by safely try

What? Really? I'll file a bug report.

All the rest of your post, which requires that I read your code, will have to wait until I'm actually awake. (This is a long process!)

[qw23]

@bh

Long ago I read about some alleged primitive tribe somewhere whose language has only the number names "one," "two," and "many,"

That was also part of the pun, of course. Plus “one too many”, and “want to many” (or much).

@bh

All the rest of your post, which requires that I read your code, will have to wait until I'm actually awake. (This is a long process!)

The secret is going to bed on time (or so they’ve told me )

[bh]

@qw23

The secret is going to bed on time (or so they’ve told me )

Yeah, I tried to do that last night, but ended up having a fight with my CPAP machine, which thinks there's a leak in my mask. That's another thing for me to deal with when I'm awake.

But we old people sleep more than you whippersnappers.

[bluebaritone21]

@bh

"one," "two," and "many,"

And toki pona has wan, tu, and mute as their only numbers.

[bh]

@bluebaritone21

And toki pona has wan, tu, and mute as their only numbers.

Huh, I wonder if it's deliberately copying this alleged tribe.

[qw23]

@qw23

I made a block to map the 1-2-many-projector (now renamed (item) -> 0 .. many images, using (rules...) over a stream. Though it’s very little code: it doesn’t work, even after hours of testing. More precisely: it gets stuck in an infinite loop. I am using a block heads (items) in front of stream (stream), which appears to work well (but I wonder if it actually does) …

Solved it!

The problem was with heads in front of stream, after all. The root of this complexity is that the Streams library in front of stream block has an unevaluated second input. Long story short: I had to code 2 helper blocks to get it to work properly. I especially like:

… which is a copy-by-definition of the library’s in front of stream block, but with a regular (evaluated) second input.

The other blocks are:

Application demo

[bh]

I made a slight change to your example (changing the True in the second clause to False):

@qw23

The root of this complexity is that the Streams library in front of stream block has an unevaluated second input.

Well of course! That's what makes them streams instead of lists. ;~)

[qw23]

@bh

@qw23

(1) head (2) tail, unevaluated OR link to tail, after evaluation.

This works unless the elements of the stream are scripts. …

I don’t see the issue about stream elements being scripts. E.g.:

[qw23]

@bh

I made a slight change to your example (changing the True in the second clause to False):

Thx4 bringing that to my attention - solved it now:

I complied my streams blocks (next post). Now back to Logic programming .

[qw23]

I compiled my Streams blocks here, with application examples.

SOMEHOW I LOST MOST OF MY BLOCKS JUST NOW.

Basic blocks

Extras

Alternative (more concise) implementation of streams
tail of and in front of are the only blocks with actually modified code, the others were adapted to refer to the modified blocks.

[bh]

@qw23

SOMEHOW I LOST MOST OF MY BLOCKS JUST NOW.

You didn't save the project in the dev version and then load it in the user version, did you?

Can we talk about block categories? I don't see why you put any of them in Control; you could make a case for a custom category, but if every library does that (as might be justifiable in each case) there will be forty gazillion categories and there won't be any colors left for the users! :~) I would just keep them all in Lists, not because streams are implemented using lists, but because a stream is, abstractly, a kind of list.

(I wanna get the details settled so you can post a library (export blocks) that I can replace the existing library with.)

[qw23]

@bh

You didn't save the project in the dev version and then load it in the user version, did you?

No I didn't.
In the meantime I restored most of it. It is now here.

@bh

Can we talk about block categories? I don't see why you put any of them in Control; you could make a case for a custom category, but if every library does that (as might be justifiable in each case) there will be forty gazillion categories and there won't be any colors left for the users! :~) I would just keep them all in Lists, not because streams are implemented using lists, but because a stream is, abstractly, a kind of list.

Assuming we're talking about the streams blocks ... I put them in a separate category "Streams" (for my own convenience; I agree the color was somewhat similar to "Control", changed it to reddish-pink Amaranth now - did you know that this is the name of a flower symbolizing immortality? not unfitting for a potentially infinite stream). But of course they can be recategorized to whatever you propose (even if the Lists category is obviously overloaded already, Snap! being more or less a LISP-derivate; not to mention Lists sharing the left IDE window with Variables and Other categories).

• the empty stream: reports (); still it is good practice to have this as a separate reporter;
• is stream (s) empty?: same;
• stream repeating (value = thing): a generator of constant values, like SICP p. 445 (ones);
• monitor (log↑) stream (s): this eavesdrops on an (intermediate) stream's output, as a debugging tool (we should have this for lists, too !);
• interleave streams <...> is from SICP, is generally useful & I will use it for logic programming.
• uniques of stream (s): I have 2 implementations, one elegant and the other much faster. Which would you prefer to include? or both?
• map (function) over keep (pred) from (s): the streams equivalent of Python's "list comprehension";
• keep (pred) from map (function) over (s): the opposite of the above;
• heads < … > in front of stream (s): attach 0 … many items as head in front of stream, one by one; I used this block for the next item on this list, and assume it may be of general use;
• process stream (s) using (operations), and project (rules) over stream (s): I'm not entirely satisfied with those two blocks, I may am going to replace them with a single Processor Of Everything.
• generate stream cascading using(function)over with (init) (inclusive = true) (#↑) until (finished = false): a generalized stream generator, inspired by the streams equivalent of cascade until ... from the List utilities library; added this just now, complementing stream with items from (start = 1), and stream repeating (value = thing); needs some more work;
• duplicate stream (s): I consider this a gimmick, wouldn't include it in an "official" library;
• stream sort: partial sorting with a stream; this is a borderline case IMO, may be presented as a demonstration of the magic of streams, just like sieve.
• zstream could be an alternative (more concise) implementation of the streams mechanism (really just a minor rewrite of tail of stream, and in front of stream); but not for now, perhaps.

[tethrarxitet]

@bluebaritone21

toki pona has wan, tu, and mute as their only numbers

Does that mean that toki pona counts off of base 3's?

[qw23]

And ala (nothing) = 0. Luka (hand) = 5. Toki pona, so I read, uses a decimal number system (from a minimalistic language I would have expected binary, but that would probably have made the language unnecessarily difficult to learn).

[tethrarxitet]

So would 4 be wan mute?