The Media Kit: Real-Time Audio and Video in Haiku

Haiku’s Media Kit models audio and video work as connected media nodes. Producers create buffers, consumers receive them, filters do both, controllable nodes publish parameters, and time sources establish the clock used for synchronized performance. BMediaRoster is the application’s gateway for discovering, instantiating, connecting, and controlling those nodes.

The architecture is often summarized as “the media server routes everything,” but that is misleading. media_server coordinates the system roster and node relationships; actual media buffers are designed for efficient transfer between nodes, with buffer data commonly stored in shared memory to avoid repeated copies. Control messages and media payload flow are related but distinct paths.

Nodes declare roles, not one implementation shape

BMediaNode is the common base of the node protocol, but concrete nodes derive from specialized interfaces rather than directly from it alone. A BBufferProducer supplies buffers, a BBufferConsumer accepts them, and a filter implements both. BControllable exposes a parameter web, while BTimeSource provides timing.

A node can live inside an application or be created from a media add-on. Other applications refer to it through a media_node handle rather than sharing a C++ object pointer across protected address spaces. Registration with the roster makes its control endpoint known to the media system.

Node-kind flags describe capabilities such as producer, consumer, time source, physical input/output, or system mixer. They help discovery but do not replace querying endpoints and formats. Two producer nodes can generate completely different media and have incompatible connection requirements.

Node lifecycle is explicit: obtain or instantiate it, register where required, connect, preroll/start/seek/stop through roster/node protocol, disconnect, release, and unregister. Skipping teardown can leave resources, connections, or add-on instances alive beyond the application’s intended use.

Connections negotiate endpoints and formats

Producers expose outputs and consumers expose inputs. A connection is established only after both sides agree on a media_format compatible with their constraints. Raw audio includes sample format, rate, channel count, byte order, and buffer geometry; raw video includes dimensions, field rate, color space, and line layout. Encoded media has different details.

Format negotiation must tolerate wildcards and specialization. One side may initially accept a range, after which connection preparation selects concrete values. An application should not assume that a device supports its preferred rate or color space because another machine did.

If endpoints cannot agree, insert a converter node where one exists or tell the user which properties conflict. Silently reinterpreting bytes produces noise, wrong colors, or memory overrun. The negotiated format is the contract used to calculate frame sizes, buffer durations, and processing requirements.

Connection order has cleanup consequences. If producer preparation succeeds but consumer acceptance fails, roll back the producer state. The roster APIs return status for each stage; preserve the first failure and disconnect any partially established path.

Buffers use shared storage and recycling

A BBuffer contains a header plus access to media data. The header records fields including valid size and start time, with additional details depending on media type. Buffer groups manage reusable buffers so a real-time path does not allocate a new block for every audio period or video frame.

The Media Kit documentation describes buffer data in shared memory areas, allowing nodes in different teams to access the same underlying storage without copying the full payload at every hop. Passing a BBuffer transfers a reference/ownership role under the node protocol; it is not permission to keep the buffer forever.

Consumers recycle buffers when finished unless the protocol explicitly transfers them onward. A filter that modifies and forwards a buffer must obey connection rules and ensure downstream completion eventually returns it. Leaking one buffer at a time will exhaust the group and stall production.

Validate size_used against negotiated buffer capacity and frame geometry. Hardware, codecs, and third-party nodes are data boundaries. A timestamp or format header from another node cannot justify reading beyond mapped memory.

Time sources and performance time

The Media Kit distinguishes real time, media time, and performance time. Real time follows the system clock since boot for ordinary waits. Media time identifies a position in content. Performance time is the time at which an event should reach its final presentation according to a BTimeSource mapping.

Nodes in one chain normally share a time source so audio and video can agree on presentation. A producer stamps output buffers with the performance time at which their content should be performed. Consumers wait until that time or report/drop late data according to run mode and protocol.

The producer must account for downstream latency. If a speaker path needs several milliseconds from buffer receipt to audible output, the buffer has to leave upstream early enough to arrive before its presentation time. Filters report their processing plus downstream contribution so the chain can schedule coherently.

Do not compute long-running timestamps by repeatedly adding rounded durations when exact frame counts can be converted to time; accumulated rounding drift eventually creates discontinuities. The legacy Media Kit documentation explicitly recommends deriving timestamps from total frames and sample rate.

Latency has multiple sources

Algorithmic latency comes from a process that inherently needs future data or adds delay. Processing latency is time a node spends transforming a buffer. Scheduling latency is delay before the required thread runs. Downstream latency accumulates remaining processing and hardware presentation.

Reducing only buffer size can lower one component while increasing underrun risk and scheduling overhead. Increasing buffers can stabilize playback while making controls feel sluggish. Measure end-to-end latency, callback duration, late notices, dropped buffers, and queue depth together.

Real-time or display priorities should be used for deadline-facing bounded work, not decoding an entire file. Disk I/O, network access, allocation, format discovery, and GUI updates stay off the critical buffer thread. Prepare data ahead and communicate through bounded queues.

When a node receives a late notice, it must follow its run-mode policy—retimestamp, increase latency, decrease precision, drop data, or preserve recording—rather than ignoring the accumulating delay. The correct response depends on whether the task is live playback, capture, or offline processing.

media_server and media_addon_server

media_server maintains global coordination: node registration, roster operations, default/system nodes, connections, and media-system state. media_addon_server loads and hosts node flavors supplied by media add-ons. Separating add-on hosting from each application centralizes discovery and lifecycle.

Applications obtain their BMediaRoster and request operations through it. They should not send private messages to server ports or assume internal team IDs. Public roster and node APIs preserve compatibility and return recoverable errors if the media services are unavailable.

A server restart invalidates parts of the existing graph. Applications need a reinitialization path that releases stale handles, reacquires the roster/system nodes, renegotiates formats, and restores user intent. Continuing to send through old node IDs after a restart can only produce confusing failures.

The server is not a magical performance proxy. A poorly written node can block its control port, miss buffers, or corrupt shared data. Global coordination makes nodes discoverable; each implementation remains responsible for protocol correctness and bounded processing.

High-level and low-level application choices

Many applications do not need to implement a custom node. High-level classes and standard nodes can play sounds or connect existing devices and mixers. Building a node is appropriate for a new producer, consumer, effect, device integration, or processing stage that must participate in the shared graph.

A custom node commonly uses BMediaEventLooper to schedule timed events. This provides Media Kit event-queue behavior but still requires a complete node protocol: format negotiation, connection callbacks, buffer handling, latency reporting, start/stop/seek, time-source changes, and cleanup.

Controls exposed through BControllable should use stable parameter IDs, ranges, units, and automation behavior. A parameter web lets generic media UI inspect settings, but changing a parameter in real time must remain thread-safe and bounded.

Offline processing uses different timing rules. It can request buffers as fast as the graph permits rather than following wall-clock presentation, but it must still preserve ordering, format, and completion. Do not call real-time time-source logic blindly in offline mode.

Testing a media graph

Test every negotiated format and rejection path, not only the preferred stereo rate. Connect/disconnect repeatedly, start/stop/seek under load, change parameters, switch time sources, and restart media services. Verify buffer recycling and area cleanup with long runs.

Inject late buffers, short payloads, discontinuous timestamps, device removal, and allocation failures. Measure actual presentation timing at the output when latency claims matter. For A/V, track drift over long playback, not one minute of apparent sync.

Record the node flavors, negotiated formats, buffer sizes, run modes, time source, latencies, and Haiku revision in bug reports. “Media server failed” is too broad to distinguish negotiation, producer timing, consumer hardware, or add-on crash.

The Media Kit’s durable idea is a system-visible, negotiated graph with explicit timing and shared buffers. It enables composition across applications and add-ons without forcing every program to own hardware directly. Its real-time quality depends on faithfully implementing every boundary, not on the graph diagram alone.

Related:

Sources: