I am at a conference workshop using mBlock which has an extension that ties in with MSFT/Azure cog. services APIs. The extension has command / reporter blocks that do fun things like estimate an image's age, recognize speech / objects / specific spoken languages, etc. So lots of possibilities there... but "mBlock" is some sort of branch of Scratch run in China. For a variety of reasons, I'd much prefer to use these services with Snap! Any thoughts or knowledge of someone developing a snap-based extension leveraging these APIs?
Hi, welcome to the forum!
Ken Kahn (@toontalk) has done work interfacing Snap! with machine learning systems. If it's a web API it should be easy, using the URL block, which takes a URL as input and reports whatever the site sends back.
I did interface Snap! and image recognition services from Microsoft and Google (and originally from IBM but they changed their API and broke things). You can find the guide to using this at Adding image recognition to programs
But note that today one can do a good deal just locally in the browser including image recognition and object detection. See Adding machine learning models to programs for Snap! interfaces to this. The cloud services from Microsoft and Google do provide much more detailed information than the on-device models but one avoids the hassle of API keys and by running locally privacy is preserved and responsiveness increased.
Regarding speech recognition there is also a browser API that Chrome and Edge supports (and FireFox folks have been working on for years). See Adding listening to programs