Bug with "uniques of list" block: wrong results under some circumstances

[rmunn]

While working through the Advent of Code problems from 2015 in order to teach myself Snap!, I found a weird case where my answers to the day-3 problem were all off by exactly one too many. After double- and triple-checking my logic, I eventually discovered that under some circumstances (which I had managed to trigger with my project), the “uniques of (list)” block was returning an incorrect result, where there was one duplicate still left in the list. E.g. in a list of coordinates that I used for testing where (0,0) was repeated several times, the correct result would have been the list ( (0,0),(1,0),(0,1) ) but under some circumstances the uniques block was producing the list ( (0,0),(0,0),(1,0),(0,1) ) where two occurrences of (0,0) were appearing instead of just one.

This project was using only integer math (and only addition of positive or negative numbers, no multiplication or division) so there was no chance of this being a case of “one of those numbers looks like 0 but is really 0 + some tiny fraction, due to floating-point rounding errors”.

The project was on my local computer, not online, so I’ll need to upload the XML file in order to share it. If the forum software doesn’t allow that, well, I’ve uploaded the XML file (trimmed down to the bare minimum needed to reproduce the incorrect result) to the Github issue report here: Incorrect result from "uniques of list" in certain specific circumstances (repro attached) · Issue #3489 · jmoenig/Snap · GitHub

A sample of what’s going wrong:

Here we can see that the result variable contains a six-item list where the coordinates (0,0) appear four times. The uniques block should trim that down and leave only a single (0,0) in what it reports.

Here we can see that if we call uniques on a list that is equal to what is in the result variable, the result from uniques is correct: a list of the three unique coordinates, in which (0,0) only appears once.

Here we can see proof that what’s in the result block is equal to that list of six coordinates.

And here we can see the bug being demonstrated: even though uniques produces the right result when called with a list that’s equal to whatever is in result, when it’s called with the value that is actually in result then it somehow misses one duplicate, and leaves (0,0) in the list twice, for reasons I have not yet managed to figure out.

[dardoro]

Operator forces the list items to be a number.
Or

---

Minimal example of the problem with the type coercion for the recursive comparision

[jens]

Thanks a lot, @dardoro for this brilliant analysis!

And thanks, @rmunn , for finding and reporting this issue.

When Snap! computes the frequency distribution and the uniques of data sets it checks for a bunch of possible shortcuts so it doesn’t get bogged down for big data sets. Most of these optimizations involve scanning the data for whether it contains compound or atomic leafs (side remark: Being an ardent proponent of dynamically typed variables and lists for educational programming languages I have to grudgingly admit that statically typed data structures can be operated on much more efficiently). The optimization trick I’ve resorted to here is to check whether a (possibly deeply nested sub-) list can be encoded to a JSON string, and then to perform the frequency distribution directly on the set of strings. Alas, different atomic data types for numbers and stringified numbers have bitten me, as @dardoro has found out.

I’ve fixed this issue in my dev version by internally converting every string-number to an actual number when JSON-ifying a list. this will also reduce the overall size of the resulting JSON string for big data sets, and further reduce the disk size of Snap! projects. Since I’m currently working on a larger other project I’ll wait with deploying this change to production until the next minor release is ready, which might take a few more weeks. I hope you agree that this bug isn’t serious or urgent, as the use case in which it came to light seems rather edgy..

Thanks a lot, both of you, for working together and making Snap! better

edit: nvmd, I’ve just deployed the fix (v11.0.8) together with another one that addresses a recent bug that affected - among others - the TuneScope library. Please let me know if you still come across this problem. Thanks.

[rmunn]

Can confirm that the fix in commit f1459d7 fixed the bug; tested with both a local Git checkout of that commit, and with the v11.0.8 release on the Snap! website. I’ve closed the Git issue as completed; thanks for the quick fix!

[sathvikrias]

@jens

Since I’m currently working on a larger other project

ooh! I’m excited! also, what was the tunescope change? was that the letter ([all] v) of () thing?

(@ego-lay_atman-bay do you know why my snapblocks won’t work?)

[ego-lay_atman-bay]

@sathvikrias

(@ego-lay_atman-bay do you know why my snapblocks won’t work?)

It’s because you were typing it into the visual composer. I don’t know how to get snapblocks to work in it, so you have to use the source editor to use snapblocks (you can switch to it by pressing the A in the top right corner).

[sathvikrias]

oof.

[rmunn]

@sathvikrias

also, what was the tunescope change?

I’m not Jens but in this case I can answer with certainty: it was fixed an internal migration bug for old blocks that have since added … · jmoenig/Snap@01187f9 · GitHub, described as “fixed an internal migration bug for old blocks that have since added additional input slots with default values”. The “letter (all) of (text)” block was a feature that had just been added to the dev version, unrelated to either bugfix, and so was released at the same time as the bugfixes when Jens decided not to wait until the next release to get those bugfixes out. (Because while my bug was minor and only affected people in rare circumstances, the bug affecting Tunescope was going to affect a lot more people so it was more urgent to get a bugfix out for it. And since the fix for my bug was already in the dev version when the Tunescope bugfix got put in, the fix for my bug, and the new “letter (all) of (text)” block option, also got into the release when the dev version was made into an official release).