Streams library 2.0 development part 2 (Part 1)

I’m a skeptic. I don’t believe, I challenge. I’m gonna test my socks off. :smirk:

Update, first test results

  1. This appears to work much better now:

At least, it didn’t crash for a = 987654321, b = 0. (overnight test)
Stack size is stable at 4. Frames will still accumulate, at a rate of 33 per iteration (but then, I don’t know the definition of this indicator).

  1. This apparently works, too: (though I didn’t test it with really large numbers)

  1. By contrast, the block below crashes (stack overflow):

I would have expected the latter block to work, too, as both it and (1) are equivalent to Scheme:
(if (= a 0) b (recursive-sum (- a 1) (+ b 1))), or so I think.

So, apparently, Snap! ‘s tail call optimization only works under strict (even syntax) conditions. But, as far as I can see, it works now.

The next block crashes, like it was to be expected (not a proper tail call, I don’t see how it might have worked for you (bh)):

First tests with streams failed.
I’ll see if I can make that work.

Why with the old keep-stream (I suppose) ?
And did you try any stream operation yet?

Another update
I’ve tried a lot of things, even reconstructed streams-according-to-SICP, but haven’t been able to make large streams (hundreds of thousands of items) work, with either library version: Snap! keeps crashing. And yes, I used the dev version.

Yet another update
For comparison I tried my hand at programming streams in Scheme. I used LispPad Go 2.01 (a free iOS app claiming R7RS-compliance - BTW @callietastrophic: this is a nice app if you would like to program SICP examples, it took me perhaps an hour of experimenting to figure out how to write and store / retrieve Scheme code, which I never did before). It, too, crashes with some “larger” stream operations, e.g. :

(stream-cdr (stream-keep mod-fifty-k? (stream-numbers-from 1)))

with:

(define (stream-car s) (car s))
(define (stream-cdr s) (force (cdr s)))
(define the-empty-stream '())
(define (is-stream-empty? s) (eq? s the-empty-stream))
(define (stream-numbers-from b)
_(cons b (delay (stream-numbers-from (+ b 1)))))
(define (stream-keep pred s)
_(if (is-stream-empty? s) the-empty-stream
_ _if (pred (stream-car s))
_ _ _(cons (stream-car s)
_ _ _ _(delay (stream-keep pred (stream-cdr s))))
_ _ _ (stream-keep pred (stream-cdr s)))))
(define (mod-fifty-k? x) (= (modulo x 50000) 0))

LispPad Go will successfully process some other large stream operations, though. What I mean to say is, other Scheme implementations (at least this one) aren’t perfect either.

You may have noticed I didn’t use a cons-stream function. I had written one at first:

(define (cons-stream head tail) (cons head (delay tail)))

… but that didn’t work (probably it should have been a macro; haven’t figured out macros yet) - instead I’ve since directly called cons head (delay tail).

Of course I tried something equivalent in Snap!, but that didn’t change anything (which makes sense because Snap! has (limited) argument typing and can thus do things Scheme can’t; bh’s been hinting at introducing “hygienic macros” in Snap!, I wonder what that would add at all).

I'm waiting on other avenues at the moment. I'm considering resurrecting my other thread about creating a paint library, assuming it hasn't been auto-locked yet. (No inviting people to twitch, just discussions and I'll do that leg-work myself)

As interesting as this discussion is though, the road-blocks you keep hitting trying to massage sicp into snap just keep re-iterating my point, but until I can prove it or someone else (the other avenue I'm waiting on) can show me why I'm wrong? I'm going to keep my mouth shut.

SICP examples were written in Scheme, and no matter how often bh repeats the mantra “Snap! is really Scheme” … IMAO it isn’t, really really - even though Snap! is obviously heavily inspired by Scheme (and by some other languages). In many cases, Snap! is actually more powerful (i.e. facilitating expression of users’ ideas, not faster in terms of runtime). It’s also, I suppose, more complicated under-the-hood.

If you want to understand the ideas of SICP, it may help to implement the SICP code directly in Scheme. Writing Scheme code, tweaking it, and making mistakes along the way, will also help you to read and understand Scheme, and eventually translate SICP's Scheme code into a different language, such as Snap! (which is still really close, of course). At least, I suspect it works that way.

As for the “road blocks”, they mostly concern revising the revised Streams library, and they are very different by nature:

  • The original (and still official) version of the Streams library may have been written in an afternoon, mostly to demonstrate Eratosthenes' sieve. As it was never intended to use streams for real-time multi-process applications, tail of stream has a (unnecessarily) huge critical section, which was easily removed (by myself) for v2.
  • Neither was the original version tested (apparently) to produce large output data. When I tried this I ran into stack overflow. This I could also easily correct in the v2 code (by rewriting some recursive library procedures to iterative style).
  • Testing with large data I also ran into a problem with tail calls. Snap! 's tail call optimization (or elimination) mechanism was apparently buggy. Jens already fixed this in the dev version (10-240521), and it works well for recursive procedures ... just not for streams (yet). It's not a disaster, just a limitation.
  • A longer-standing issue is that upvars and streams don't go along too well - let's call it a "vulnerability". For v2 I'm using a partial solution by dardoro, which has its limitations, too; and I refrain from over-using upvars, which may be for the better anyway.

I guess I should add that I might never have encountered any of these issues if I hadn't written code for libraries; this has prompted me to test extensively, also with very large data sets and edge cases.

What I recognize from your struggle with abstraction is that the inner workings of Snap! are opaque to me - though the code is public, it’s written in JavaScript, which I don’t speak, and I don’t want to do jens’ job, and some issues may even be within JavaScript itself; even so it’s little frustrating sometimes.

You misunderstand, I DID that years ago. I did Brian's book on logo in logo too, but I did all of it in preperation for Snap. As you say, Snap isn't quite scheme

In many cases, Snap*!* is actually more powerful (i.e. facilitating expression of users’ ideas, not faster in terms of runtime). It’s also, I suppose, more complicated under-the-hood .

Yeah exactly. I'm waiting for the "parser" update in 10 or 11, but I'm still not sure it's going to be enough. For me anyway and as a niche case I absolutely do not want Jens to cater to me and me alone, that'd defeat the point of Snap! (edited to put parser in air quotes, because that is and isn't what it is)

My end goal has ALWAYS been my own tool and language seperate to snap, but taking the ideas that Snap has implemented and taking them in my own direction, and for ME and Me alone, I want to be able to drag and drop any source file into my ide and have text and block side to side, and I want this for a few reasons, mostly to study other people's code in a way that doesn't make my brain drip out of my ears.

I learn visually and I'm stubborn.

Hang on, I'm going to whip up an example, even though the new fortnite season just dropped lol. (Yay for modern machines, two games at once lol)

Snap certainly is not Scheme, but, as you correctly point out, heavily influenced by Scheme, and also by a bunch of other languages, especially Smalltalk and APL, and - not to forget - especially influenced by Scratch!. While Snap does share proper tail recursion with Scheme, it does not feature Scheme's nested variable scope, but instead limits local variables to individual scripts and functions, not to sub-parts of them. Also, Snap's continuations only feature a portion of Scheme's functionality. Therefore I always cringe when some folks claim that Snap is Scheme. It clearly isn't. Some of the divergences come out of Snap being a "live" graphical language, some out of parallelism, both of which Scheme doesn't have to deal with. Some features like upvars have no correlation in Scheme or Smalltalk, even though they might appear to be like other familiar features, therefore using upvars in a purely functional way as a notation for reified parameters is bound to fail. Personally I'd much rather prefer Snap to be seen as its own programming language, which it clearly has become, than a notation for some other one. Just like SICP was written for Scheme (or perhaps it was the other way), something should be written for Snap. Just as it would be a questionable idea to "translate" the C-language manual to, say, LOGO, it's not a meaningful idea to apply SICP to everything, especially to Snap. That said, I do believe that identifying and adapting great examples and powerful ideas in SICP and elsewhere (e.g. in CS LOGO Style!) is a wonderful idea.

Yes. I want to write that and have done for ages, maybe not in an official capacity, but the reason I keep trying SICP in snap! is because I want to translate across, but Snap! takes a lot of concepts that just were NOT present when that text book was written.

Iunno, I really love blocks and text makes my ears drip out my brain, but everyone insists that text is the standard. and I hate it.


Right. So this took way too long, but this is one of the things I've wanted to do for ages. Quake is the better example, but I don't post images of that game in this forum because I feel it's too gory, and Battlezone was, as far as I can find, rated G (for 11+)

Quake had a thing called QuakeC, which was a stripped down c parser that understood game logic and you could, provided you understood how to use it, write anything and everything in QuakeC and the game would interpret it and just do it. Levels, monsters, items, could be defined in QuakeC and the game would understand it.

What I've wanted to do is a lightweight Snap that did the same thing, and while I took this screenshot of a level in play, have an editor that could report in text or blocks.

Or even, if I ever build my own tool, build the prototype in that and have a lightweight version of that doing scripting.

Or in games like quake which game you dev-mode like snap! (The ingame console was also a parser and could take commands and change it on the fly, but instead of the pseudoDOS nature of that same console, mainly because Quake was a Dos game (at the time) You could just expose all the things like (chasecam 180) as a block.
I'm not entirely sure I'm getting the idea across to be brutally honest.

Snap/Visual Languages have SO MUCH potential, because you can embed the visuals into the visual and manipulate it that way.

And that's just one way I can envision it.

Look, I've been using computers for decades. I know how computers work, but coding is the one thing that I just CAN NOT use. Snap! tho? For me? Is the final piece of the puzzle to snap! into place... (Sorry but not even remotely sorry)

Aaaanyway, as usual I'm rambling.
(Also Also, this is why I keep saying what I want is ALSO well, well, WELL BEYOND THE Scope Of Snap! because it is. This is advanced advanced stuff. The reason I want something like snap or something visual that I created is because there SHOULD be an advanced snap! for kids and kids that never grew up (It's ME) to implement this)

Wow! At this point this would have to be a mod of Snap!!

Would that explain why I still can’t get very large stream operations (i.e. with over about 100k promises ever made on my iPad Pro-2) working without a crash? And why streams-with-upvars are extremely vulnerable with multiple streams using the same external names for upvars? (I’m just speculating wildly now)

Wait, what? Scheme isn't Javascript. The only sense in which it has nested variable scope is that it has nested procedures, each of which creates an environment frame when called, and we have that too.

I mean, of course we aren't exactly the same as Scheme in every detail. But different implementations of Scheme are allowed to be different. What it means to be a Scheme, in my view at least, is to have first class procedures, lists, and continuations. And macros. And tail call elimination. (EDIT: And lexical scope.) So, we're a Scheme! The pl gang didn't complain, when I said "Snap! is Scheme disguised as Scratch" at SPLASH, that our continuations aren't good enough. They just complained that we need hygienic macros.

That doesn't preclude us also being those other languages. :~) (In fact, in my occasional sleepless nights I worry that we are creeping toward being a PL/1, an attempt at being the unification of all programming languages. All we'd need is first class pointers, so we could be a C also.)

I think everyone understands the difference between a slogan and a design document. When I say we're a Scheme, I'm addressing people who think block languages have to be trivial; I'm saying that we have the expressive power not just of a text language, but of the ne plus ultra of text languages. And I guess I'm saying also that, like Lisps generally, we push in the direction of functional programming without strictly enforcing it.

If you use upvars you must be aware that they are script-scoped, they do not ever, under any circumstances, make a new scope. This is something I've been telling y'all for ages, and yet all you FP talibans never listen :slight_smile:

I don't know (and don't care) about "very large stream operations" with "promises" and whatnot, and I'm especially suspicious about all those hacks that desperately attempt to programmatically create or rebind variables, instead there must be a way more straightforward way for all of this. Don't think Scheme, think what it is you want to accomplish!

upvars are function arguments for c blocks and should be treated as such

what I mean is that in Snap the script vars block does not, under any circumstances, create a new scope that you can capture, neither does an upvar. Whereas in Scheme, as far as I understand it, this is exactly what's happening.

they totally are not that! Geez, are you even reading what I'm explaining. and upvar is a script variable. That's all. They are not a function parameter!!!!!!!!!

Jens, please don't get mad, especially not at the users.

Here's the part I think is true in what @tjc_games said: Remember that the canonical use case for upvars, the reason we invented them, was the FOR block. And it's definitely true that the purpose of the upvar in FOR is to drag it into the code in the loop.

DIfferent programming languages differ with respect to whether the loop variable of a FOR loop still exists after the loop finishes. This is a question to which there is no generally agreed right answer. You chose to implement upvars as a kind of script variable, following the semantics of script variables, and that's as right an answer as the other possibility. There are valid use cases for each approach. (It happens that some of my desired use cases would work better the other way, but that doesn't mean my way is right!)

an upvar is a script var of the caller, not a parameter of the function call. Always has been. If you "capture" an upvar and believe that you can somehow closurize it, you'll produce a memory leak. I bet that's exactly what's happening.

chill bro, chill

you shouldnt be angry when someone gets something wrong

(im gonna get banban'd)

In the new streams library, you mean? Could be, although I think streams 1.0, which is upvar-free, had some of the same problems.

This is the same argument we always have about everything. You're focused on how you implemented something, rather than on how a user sees it.

How do you make an upvar, anyway? There's no UPVAR block you can put in a script! The only way to make one is precisely in the "parameter of the function call" dialog box.

As an additional piece of evidence, I submit the history of the FOR EACH block. Originally it looked like this:
untitled script pic (2)
and, as with the other higher order procedures, you used empty slots in the code in the C-slot to say where to put the item. Then someone, Dan I think, pointed out that the word "item" in the block's title could be an upvar, as an alternative way to access the item. And then, years later, over my bitter objections, we changed it so that you couldn't use an empty slot to get the item any more. So now we are using an upvar precisely as the syntax for an argument to the implied procedure in the C-slot.

All of this is orthogonal to how upvars are implemented! How users use them is a completely different question from how they work.

nonono, implementation is not the argument at all, it's the fact that the upvar is shared by the block definition and the caller. It's not a one-way evaluation as other inputs, that's not an "implementation decision". If - some! - users make mistakes or wrong assumptions, then it's just that, a mistake or a wrong assumption.

upvars can be mutated both by the caller script and by the function:

unlike formal parameters upvars do not make a new scope:

“Taliban” is plural, and its meaning is: students.