I’m working on an update of the Streams library. If you are e.g. an electrical engineer, physicist, telecoms expert, weather buff, robotics specialist, intelligence agent … or just a smart student / programmer - may I borrow a few minutes of your brain power?
Streams, also called “delayed lists”, are potentially infinitely large datasets being created over a prolonged period of time. In other words: data originating, and being processed, in real time. At any given time, we can only know the part of the stream that was already produced.
Some examples of streams in the physical world:
surveillance camera images;
weather data;
process parameters measured in a chemical plant;
robot sensor data;
real-time sounds / speech.
You may have experience with any of these kinds of data, or other data that are generated and processed as a stream. If so, I have a few questions for you.
What I would ultimately want to know is how these data are typically processed, especially in real time …
Some (interrelated) things come to my mind:
pattern recognition;
statistics;
exception handling.
… and what facilities are sensible for a streams library to offer (such as: comparing a stream item with the preceding item).
So here are my questions to the community:
Are you familiar with any more types of data that are generated and processed as stream?
What are typical (real-time) operations on data streams?
Suppose you wanted to use a streams library for such operations, what features would help you most?
Presuming an API for reading the sensors, and one for instructing the actuators, I can imagine a streams-based mechanism taking care of the rest.
Let’s assume that’s possible … what would you want to use these previous readings for? And consequentially, which previous readings would you want to be able to read? Like data from a few seconds ago? last year? sometime in between? Raw data, aggregations, selections? Dong what with the data?
I may ask you about bird tracking laternote. But for now let’s get back to robotics. What data do your robot’s sensors produce, and what commands do you send to their actuators? Could you describe the processing that is done (or required) on the sensor data? Perhaps as part of a control loop: sensing → data processing → actuating → movement → sensing … (etc.) ?
Note: I may have misunderstood you, in that you actually use robotics technology as a means of tracking bird migration. If so, I apologize for my misinterpretation.
Streams can’t do miracles, of course. What can be done is that a streams-based monitoring system reads these sensors regularly and executes decision rules on what to save, and what not to, and for how long.
So as I understand it, your sensor measures past distance, is that the distance that your robot has walked? (or however it propels itself)
Supposing you would like to keep a record of “past distance”, for later reference, a streams-based monitoring system might have rules so as to:
skip a reading if it has not changed from the preceding reading;
keep every single reading during especially interesting episodes, like if the acceleration is beyond a certain threshold;
keep one reading for every second in all other cases.
Our discussion led me to experiment with real-time use of streams. I discovered, and repaired, a really annoying flaw in the definition of one of the basic Streams library blocks, that will only occur in a multiprocessing environment (such as in simulations, and monitoring-and-control applications). So that was really useful for my effort, thanks for prompting it!
I’ll get back to exploring what you could do with streams, and inform you as soon as I have something that might be of interest to your case.
for my misinterpretation.