Skip to content
Haiku OSDeep Dive Published Updated 3 min readViews unavailable

BMessage Flattening and IPC: How Haiku Moves Typed Data Between Processes

An engineering guide to BMessage fields, flattening, messengers, reply semantics, validation, compatibility, and safe interprocess communication on Haiku.

BMessage is Haiku’s typed message container. It carries the what command code plus named fields whose values retain a type code, element count, and byte representation. The same abstraction supports in-process dispatch through a BLooper, communication through BMessenger, drag-and-drop, scripting, archival, and persistence. Its convenience does not remove protocol design: the sender and receiver still need a versioned, validated contract.

A message is a typed multimap

A field name may contain multiple values of the same type. AddString("path", ...) can therefore be called repeatedly, and the receiver can enumerate indexes. Names alone are not a schema. FindString("path", &value) fails if the field is absent or has the wrong type, so production code checks the returned status_t and constrains sizes and counts before allocating work.

enum : uint32 { kIndexRequest = 'idxr', kIndexReply = 'idxp' };

BMessage request(kIndexRequest);
request.AddInt32("protocol", 1);
request.AddString("path", "/boot/home/docs");
request.AddBool("recursive", true);

BMessage reply;
status_t sent = target.SendMessage(&request, &reply, 2000000, 5000000);

Four-character constants are readable command identifiers, not globally registered security boundaries. The receiver should also verify who is allowed to invoke sensitive operations through the surrounding application architecture. A well-formed message from an untrusted source is still untrusted input.

Flattening preserves representation, not behavior

A flattenable message can report FlattenedSize(), serialize into a sufficiently large buffer or BDataIO, and later be reconstructed with Unflatten(). Flattening converts the message graph into a transportable byte form; it does not serialize open file descriptors, C++ object pointers, live BMessenger targets, or application invariants into portable behavior.

ssize_t size = request.FlattenedSize();
std::vector<char> bytes(size);
if (request.Flatten(bytes.data(), size) == B_OK) {
    BMessage decoded;
    if (decoded.Unflatten(bytes.data()) == B_OK) {
        // Validate decoded.what and every required field here.
    }
}

Never unflatten arbitrary bytes and immediately perform privileged work. Put a hard bound on the input size, reject unknown top-level commands, validate required fields and ranges, and ignore or explicitly reject incompatible protocol versions. Persisted messages also need migration rules: a later binary may add optional fields, but it should not silently reinterpret an old field with a new meaning.

Delivery, replies, and timeouts

BMessenger names a handler/looper target and hides whether it is local or remote. Asynchronous SendMessage() avoids blocking the caller’s looper. A synchronous send with a reply is useful at process boundaries but must carry finite delivery and reply timeouts; otherwise an unavailable or wedged receiver can freeze the UI.

On the receiver side, MessageReceived() dispatches known commands and delegates unknown messages to the base class. A handler can use SendReply() while the incoming message is valid. Define whether an error is represented by the returned transport status, a reply command, a structured status field, or all three. Mixing these casually creates callers that mistake “message delivered” for “operation succeeded.”

Large payloads deserve a different transport or a file-backed handoff. Ports and message queues are finite, and repeatedly flattening megabytes increases memory copies and head-of-line blocking. Carry an identifier, pathname, or shared-area reference only when the receiving side can safely validate and authorize it.

Test the protocol at the byte and delivery layers. Round-trip every valid message through Flatten()/Unflatten(), compare repeated fields in order, truncate the buffer at every offset, mutate type codes, and inject unknown optional fields. Then test a dead target, a full queue, a late reply, receiver restart, and two callers using the same request identifier. The decoder should reject malformed input without leaking resources, while a compatible older receiver should ignore only fields explicitly designated optional.

A durable Haiku IPC protocol has stable command identifiers, an explicit protocol version, typed required and optional fields, size limits, timeouts, error semantics, and tests that feed malformed or old messages into the decoder. BMessage supplies the mechanism; compatibility and trust remain application responsibilities.

Related:

Sources:

Comments