Speed execution problem with "remove duplicate from (list)" block


This block is not effective in its current form. If I run it on a large database (53000 records), it will take 8.5 minutes to run!


I noticed that inside this block there is an old version of the code present. I performed tests with the old code. Surprise: the same request takes 5.5 seconds!

current block (8.5 minutes)

old code (5.5 seconds)
I notice 1 thing with the old code: if i remove the "warp", 4.5 seconds to execute the code ! (??????? i don't understand...)

Was there a REMOVE DUPLICATES block in BYOB3 tools? If so, the old code is probably that from the BYOB version.

Have a go at writing a quicker one then :slight_smile:

Just thought - make sure you are testing the Snap7 dev version - no point in putting effort into present version

Sometimes, it just makes it more laggy than speedy.

It's worse in version 7: I have to press "wait" every 10 seconds!

I'm not a huge fan of those terribly inefficient library blocks!
Why don't you use the analyze reporter in the "frequency distribution analysis" library to make your own one:

unique items

I've just now tried it on a huge list of 10 Million (!) items and got the result instantly.

Thank's i will give it a try tomorrow...

I've just now updated the official "remove duplicates" reporter in the list library with my solution above. You might have to clear your cache to try it in the current v7 dev version.

Of course, JS code is blasting fast...

But removing duplicates from the sorted list is quite effective with

(it has a flaw - sometimes empty elements are removed)

A little error here... image

data, not my list...

8.5 minutes to 236 ms, good work !

thanks! Sorry for the glitch, I've just now fixed it, make sure to clear your cache so you get the correct version next time you load the list utilities library.