@bh,
The following is probably a better implementation of in-front-of-stream as being a little faster, but it still crashes before it gets to a hundred thousand:
Oh. I didn't even think about that! Good point. @qw23, do you agree?
@bh,
The problem with the head of the stream not being garbage collected isn't unique to the memoization of streams, as I converted a couple of the basic stream implementations such as the stream-starting-at block to a CIS (Co-Inductive Stream as I think were first referenced in SICP - especially easy with the different, probably slightly slower, new stream primitives as with or without memoization works exactly the same, with the tail of the stream just a reporter/script) and still have the (apparently related to memory) crash problem...
I wrote the same stream implementations in pure JavaScript (actually in Fable generating JavaScript) and don't have a problem with garbage collection of the heads of the consumed stream, so the problem isn't in JavaScript...
So it seems something in the Snap! chain is doing something (perhaps to do with how captured free variables are stored for closure functions) that is causing this problem; for instance, if captured free variables are stored by modifying the object prototypes of the objects representing lists, then there could be a hard link from the tail value to the previous node, which would prevent garbage collection...
Let’s see if I get the portée of @gordonbgoodnew’s remark. Would that be obsolete parts of an “imperative list” (i.e. a Snap! list implemented in underlying Javascript as dynamic array?) are not garbage-collected, whereas similar parts of a Snap! list implemented-as-linked-list-in-JS are? I’m afraid I really couldn't say if that’s true - we’d have to ask @jens - and if it makes much of a difference in this case.
Earlier this year I experimented with a very different streams system for Snap! As you may remember it was several times more stable (and somewhat faster) than the current library’s. Here’s an example:
(this still works with 123,456 items or more on my antiquated iPad)
It had some drawbacks too (notably: more complicated definitions, therefore harder to debug, and - ironically! - sieve not being supported), so we decided we wouldn’t use it in the library for now.
No, I was testing with the current streams library, although I also compared with a slightly different implementation that tried to avoid the garbage collection not working for streams issue (apparently). The writing of the same algorithm in JavaScript (through Fable, but the JavaScript output is easy to check) was just to prove that this issue is something that Snap! does, and not a JavaScript garbage collection problem...
None of the implementations with which I was experimenting would interfere with implementing David Turner's sieve (as @bh likes) in any way, as I ran the David Turner sieve using my Fable implementation just to show how bad the computational complexity is...
The problem seems to be that the head of the list isn't consumed in a producer/consumer use of a stream, leading to the IDE crashing when iterating over something about a hundred thousand items...
Did you actually try my alternative streams script, as included in post 244?
I did, but it is slower than molasses such that one can't really run it over a big enough range to cause the kind of crash that I am seeing with the current streams library: it took over a two minutes to run to 1e4 let alone to about a hundred thousand where the problem was occurring...
I don't think that is the solution...
The "official" stream with the Shift 
"Llive coding" enabled, works for me up to
and break after 4GB of the memory allocated (1.2E6).
Your version
have a stable memory footprint of ~25MB.
I have no patience to wait 10 minutes for the result for 1E7 items.
The "official" stream with the Shift
and break after 4GB of the memory allocated (1.2E6).
Thank you for trying this. Yet another thing I didn't know about in this Shift "Live coding support" enabling option - It seems the current version does still use a lot of memory and breaks when memory use gets too high...
However, the experiment version still doesn't seem ideal if it's not as "functional" as streams should be...
This topic was automatically closed after reaching the maximum limit of 250 replies. Continue discussion at Streams library 2.0 development part 2 (Part 2).