3 Concepts

Now that you’ve installed the Astra SDK and ran through a brief tutorial on its basic use, let’s focus our lens on three core SDK concepts - streams, streamsets and readers. Understanding these concepts is vital to harnessing the power of the SDK with finesse and confidence. Furthermore, possessing a solid understanding of these concepts will help you to quickly pick up both more advanced and new SDK features as they are introduced in the future.

3.1 Streams

A stream is the sequential flow of frames from a particular data source. Imagine an old-time movie reel, with the video recorded as individual image snapshots in sequence on a long stripe of film. Each image snapshot is a frame, and you can imagine the full film as a stream, with the distinction that a stream doesn’t necessarily have a definite end.

By U. S. Navy. Photographed by Cmacd123 at en.wikipedia [Public domain], via Wikimedia Commons

Streams come in a variety of flavors. For instance, a color stream produces color frames, and a depth stream produces depth frames. Both of these stream types usually come from physical sensors (such as the Astra), but streams can also be generated by the SDK through “plugins” (more on plugins in a different section). For instance, the hand tracker in the SDK uses the depth stream from a sensor to generate a hand stream.

3.2 StreamSets

A streamset is a group of closely related streams. Continuing with the analogy above, modern movies aren’t only sequenced image frames - they have sound as well. The audio and video for a film are packaged together as a movie, since they are inextricably related to each other. In our terms, this combined package of audio and video can be thought of as a streamset.

While a streamset could be entirely composed of streams arriving from a physical sensor, it can also contain streams created through the SDK plugins. As an example, the Astra camera is represented by a streamset containing a depth stream, a color stream, as well as middleware-generated streams such as the hand stream.

Streamsets are addressed by a string-based unique identifier (URI) which allows developers to access multiple distinct streamsets at once in code. This becomes useful when working with multiple StreamSets, or perhaps addressing a server that produces higher-level streams created by a network of StreamSets.

3.3 StreamReaders

A stream reader is our window to the frames in the stream. Before we can actually look at each individual frame of any given type coming from our streamsets, we use the streamset to create a reader in order to access the frames of that stream. To help with understanding what a reader does, let’s continue with the film analogy one more time. While the audio/video film contains all of the information that we need to playback a movie, the film alone isn’t sufficient to presenting the movie. Another device is needed - a reel-to-reel projector. The projector not only advances the film and projects the images onto a screen, but also synchronizes the film’s audio to the video for a smooth, seamless performance.

By christian razukas from Honolulu, Hawaii (Hawaii Theatre Projection Booth) [CC BY-SA 2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons

A stream reader is functionally similar to a reel-to-reel projector. It consumes specified streams coming from a streamset (i.e. the film) and coordinates the actual delivery of each frame (the projector) to the application. Each reader can read multiple types of streams at once, and a streamset can create as many readers as are necessary for a given application. If more than one stream type is started by the reader, the reader will synchronize the streams before delivering them.

3.4 A Flexible, Scalable Pattern

By now, you should understand how the Astra SDK delivers data from the sensor to your application. As a quick review, individual packets of any type of data coming from a sensor are stored in frames, and these frames are delivered in streams that are grouped in streamsets and accessed by a stream reader.

Why this pattern, though? Two words - flexibility and scalability. Since streams can be of any type, even types that aren’t currently supported or aren’t even provided by sensors yet, the architecture can easily be extended through the addition of plugins to add new stream types. Even better, the new stream types will obey all of the same conventions as existing stream types, so no need to learn a new pattern every time new stream types are introduced.