Skip to content
RetrogamingDeep Dive Published Updated 6 min readViews unavailable

Nintendo 64 PI Cartridge DMA: Serialization, Completion, and Buffer Ownership

Trace ROM reads through the Nintendo 64 Peripheral Interface while keeping PI arbitration, asynchronous completion, alignment, and cache ownership distinct.

The Nintendo 64’s Peripheral Interface (PI) is the path between RDRAM and the cartridge ROM or other mass-storage address space. It is easy to discuss it as “cartridge DMA,” but that phrase hides several independent contracts: the device supports both DMA and CPU I/O operations, only one PI transaction can be active at a time, completion is asynchronous in the high-level SDK path, and the R4300 data cache does not automatically stay coherent with memory written by the DMA engine.

This article focuses on reading a known ROM range into main memory. It does not duplicate the broader cache-line ownership discussion in the existing N64 cache article, and it does not describe controller traffic through the separate SI/PIF path. The Nintendo 64 Programming Manual cited here is explicitly marked as a draft; its documented SDK behavior is valuable primary evidence, but should not be confused with a guarantee about every third-party SDK or every emulator implementation.

PI DMA and PI I/O are different transaction shapes

The manual describes two RCP I/O modes: DMA moves at least 64 bits between RDRAM and a device, while CPU I/O transfers 32 bits between the CPU and a device. PI supports both DMA and I/O reads/writes to ROM addresses. Use DMA for a bulk range; use a documented PI I/O routine when the need is a single word or a low-level device access. A word read is not merely a shorter DMA request, and the two paths must not race each other.

The manual is explicit about the race: while a ROM-to-RDRAM DMA is active, issuing an I/O read from a different ROM address can cause the current DMA transaction to fail. PI is therefore a shared device with mutual exclusion requirements, even if application code is split across threads. The high-level SDK functions exist to serialize this access. The Raw functions are exposed for a custom interface; their use transfers responsibility for that exclusion to the application.

Let the PI manager own an asynchronous read

Initialize the PI manager during OS startup with osCreatePiManager, supplying the manager thread’s priority, command queue, queue storage, and entry count. The queue used by the manager is distinct from the caller’s completion queue. osPiGetCmdQueue() returns the manager’s command queue when the manager has started, or NULL otherwise.

Each osPiStartDma call packages the PI device address, RDRAM address, direction, byte count, and caller-supplied completion queue into an I/O-message request. The PI manager serializes requests from multiple threads and posts the request’s completion message to that queue after the transfer. Normal-priority messages are appended to the command queue; OS_MESG_PRI_HIGH messages are inserted at its front. That is queue placement, not a promise that a high-priority request interrupts an operation already in progress. Review priority use and queue capacity when audio, graphics, and peripheral clients share the device.

The enqueue call can fail: the function reference specifies -1 if the PI manager has not been started, and otherwise returns the message-queue send result. Check that result before waiting, or a caller can block for a completion on a request that was never accepted. By contrast, the managed osPiReadIo and osPiWriteIo calls perform blocking 32-bit accesses with PI synchronization. Do not mix them with raw register access unless one owner explicitly coordinates both paths.

This libultra-style excerpt shows the request and completion boundaries. OS initialization has already started the PI manager, and the caller’s completion queue and message storage have been initialized. The request block and destination remain valid until completion; submission failure must be handled before waiting.

/* `dst` is dedicated, 16-byte/cache-line-aligned RDRAM for OS_READ. */
/* romAddress is 2-byte aligned; length is a multiple of 2. */
/* The PI manager and caller-owned completionQueue are initialized. */
osInvalDCache(dst, length);
if (osPiStartDma(&request, OS_MESG_PRI_NORMAL, OS_READ,
                 romAddress, dst, length, &completionQueue) != 0) {
    /* Report the enqueue error; there will be no completion to wait for. */
    return;
}
OSMesg completedRequest;
if (osRecvMesg(&completionQueue, &completedRequest, OS_MESG_BLOCK) != 0) {
    /* Handle an unexpected queue error before using the destination. */
    return;
}
/* Only now consume the bytes in `dst`. */

The manual’s request contract permits transfers up to 16 MiB, requires an even byte count and a two-byte-aligned PI device address, and requires an eight-byte-aligned RDRAM address. For OS_READ, it recommends stricter alignment: use a dedicated 16-byte/cache-line-aligned destination, with no unrelated state sharing any cache line touched by the transfer. The pre-DMA invalidation and ownership hazards are covered in Nintendo 64 DMA Cache Coherency: Keeping R4300, RSP, and RDRAM in Agreement; the PI-specific point here is to satisfy the request’s bus geometry as well as that separate cache contract. The completion receive is the boundary before consuming data. The excerpt assumes the address range is valid and the ROM interval exists in the cartridge image.

Do not submit a managed PI request and then issue a raw register transaction from another thread. The function reference describes osPiGetStatus() as low-level status observation, including DMA-busy, I/O-busy, and an error indication when I/O is issued while DMA is busy. That status is useful for diagnosing a raw path, but polling it is not mutual exclusion: a second thread can still race between a status read and its own register write. Keep managed clients behind the PI manager, or make one subsystem the explicit owner of every raw transaction.

The transfer length is the number of bytes in one request, not a promise that a high-level helper will split an oversized range. Chunk larger reads into valid requests and associate each completion with its own request state. If using a library helper that handles unaligned spans, treat that behavior as an SDK-specific implementation detail: at libdragon commit e356bf3f56f7afbf7e5246329562f145965cfdfc, the raw asynchronous path waits for PI busy to clear and programs the transfer registers, while the higher-level read helper has fallback handling for some misaligned transfers. Do not transfer that helper’s guarantees to libultra or to hardware generally.

Test the ownership sequence, not just the bytes

For a real target or emulator, fill two separate ROM ranges with deterministic, distinguishable patterns and issue normal-priority requests from two client threads through the initialized PI manager. Give each request its own message block, destination, and completion queue. Verify both submissions are accepted, each completion reaches the expected queue, and each destination contains exactly its assigned pattern before the buffer is reused. Add a later osPiReadIo operation after the DMA completion to check the managed 32-bit path without deliberately racing it against the transfer.

An emulator should model the manager command queue separately from active hardware state: queued requests, the selected request, PI DMA/I/O busy state, completion notification, and request owner. Preserve the documented normal-append/high-priority-front queue behavior, while avoiding an unsupported assumption that priority preempts an active transfer. A model that copies each request immediately without an active interval can hide an illegal raw I/O access during DMA. Cache invalidation and line ownership remain separate test concerns; use the existing coherency article for their test matrix rather than duplicating it here.

Related:

Sources:

Comments