Sega Master System VDP Sprites: Overflow Index, Collision Latch, and Status Reads

The Sega Master System VDP has a small sprite pipeline with two software-visible status effects: sprite overflow and sprite collision. They are often implemented as post-render checks over a finished bitmap, but the VDP discovers sprites while preparing a scanline and detects collisions as opaque object pixels overlap. The difference matters because software can read status during the frame, mask the display, change sprite attributes, or use the overflow index as a diagnostic signal.

The VDP has a hardware limit of eight sprites on a line in its standard display modes. A ninth intersecting sprite raises the overflow flag and records information about the sprite that exceeded the line budget in the status register’s low bits. Opaque sprite pixels that occupy the same horizontal position can set the collision flag. Reading the status port returns and clears status state, so the register is both a report and a synchronization action.

Sprite evaluation prepares the next display line

The VDP reads the sprite attribute table and determines which objects overlap a scanline. The table contains Y coordinates and, in the later mode, X positions and tile numbers. Object size, zoom, vertical coordinate wrap, and display mode affect the range test. The VDP retains a limited set of sprite numbers for drawing, while sprite pattern data is consumed by a later output stage. An implementation should keep selection order and pixel composition separate.

The maximum is eight sprites per line in standard SMS mode 4. If more candidates intersect, the VDP does not simply draw all of them and set a warning flag. Later sprites are omitted from the visible list, and overflow state reflects the first rejected candidate according to the chip’s status encoding. This is why a game may show flickering or missing objects as its sprite table order changes even when its total number of sprites is unchanged.

The line coordinate convention is offset relative to the visible line, and values near the top of the byte range can wrap to negative positions. Some modes use a sentinel Y value to stop parsing the table. These details should be checked against the exact VDP generation and display mode. A renderer that uses a normalized y <= line < y + height rectangle without applying the hardware’s coordinate interpretation can shift the first row, mishandle sprites above the top border, or scan too many entries.

The VDP has revisions with differences in sprite evaluation and buffering. Mode 4 behavior in the 315-5313 is not identical to the older TMS-like 315-5124 path. An emulator should select the correct model from the console configuration and preserve unresolved or revision-specific behavior as an explicit compatibility choice rather than merging all observations into a single “SMS VDP” rule.

Overflow is a timed status latch

The overflow status bit is set when sprite evaluation finds a candidate beyond the supported line capacity. It is not merely a count of the number of entries in the table. In the SMS status byte, the overflow indicator occupies a high bit, and low status bits can contain the index associated with the overflow event. That index is useful to software and must be generated from the VDP’s evaluation order, not from a host-side list after sorting sprites by X coordinate.

Overflow has timing. The candidate may be recognized during scanline preparation, while the status bit becomes externally visible at a defined horizontal point. A CPU read earlier in the scanline can therefore observe a different value than a read later, depending on the exact device. MAME’s VDP model keeps pending status and moves it into the visible status at the modeled horizontal event. That is a valuable architectural boundary even if a target core uses different measured offsets.

The status state is sticky until the documented clear operation, typically a status-register read or frame/reset behavior. Reading status also clears interrupt-related state, so a debugger or test ROM cannot read it repeatedly without changing the machine being measured. Provide a non-invasive internal trace for development and make guest-visible reads follow the hardware side effects exactly.

Do not derive the low status bits from the order in which a modern renderer visits pixels. The VDP processes sprite table entries in hardware order and its overflow behavior may report a table index, not the index of the first pixel actually drawn. A test should arrange nine eligible sprites with distinct indices, then reorder their table entries while keeping geometry constant. The visible subset and overflow report should change according to the scan algorithm.

Collision is about overlapping opaque pixels

The collision flag is set when two sprite objects produce nontransparent pixels at the same location. Bounding-box overlap is not sufficient: two sprites can have overlapping rectangles while their set pattern bits never coincide. Conversely, the collision may occur at a single pixel and disappear before a host frame renderer inspects the final image.

Collision detection depends on sprite pattern bytes, size, zoom, X coordinate, left-column shift, and the active display mode. Pattern value zero is transparent for collision purposes in the sprite path, while nonzero sprite pixels can collide even if their palette colors happen to match. Keep the collision test based on object opacity, not final RGB equality. Background pixels do not count as a sprite-to-sprite collision.

The collision flag can be raised before or during visible output depending on the VDP revision’s pipeline. A CPU read of the status register at a chosen horizontal position may see the flag as soon as the collision event has propagated. If the emulator computes collisions only after drawing the whole line, raster code that checks status in the middle of a line will be wrong. Schedule the event at the appropriate emulated beam position or update the visible status as the raster advances.

Horizontal shift and display blanking add useful edge tests. A sprite shifted left by the VDP register may collide at a different visible x; disabling the left column can suppress or alter output and collision behavior on some variants. Test pixels at x=0, at the column boundary, and at the last visible x. These cases also expose bugs where the renderer wraps an off-screen pixel into the opposite edge instead of clipping it.

A status-latch test model

This Python fixture separates selected sprites, rejected sprite index, collision events, and the guest-visible status read. It is intentionally a behavioral test shape, not a full SMS VDP. The machine-specific event scheduler should decide when pending overflow/collision becomes readable and should include the variant’s exact status encoding.

from dataclasses import dataclass, field


@dataclass
class SpriteLineStatus:
    selected: list[int] = field(default_factory=list)
    overflow_index: int | None = None
    collision: bool = False

    def consider_sprite(self, index, intersects):
        if not intersects:
            return
        if len(self.selected) < 8:
            self.selected.append(index)
        elif self.overflow_index is None:
            self.overflow_index = index

    def note_overlap(self, opaque_a, opaque_b, x):
        if 0 <= x < 256 and opaque_a and opaque_b:
            self.collision = True

    def status_value(self):
        value = 0
        if self.overflow_index is not None:
            value |= 0x40 | (self.overflow_index & 0x1F)
        if self.collision:
            value |= 0x20
        return value

    def read_and_clear(self):
        value = self.status_value()
        self.overflow_index = None
        self.collision = False
        return value


line = SpriteLineStatus()
for sprite in range(9):
    line.consider_sprite(sprite, intersects=True)
assert line.selected == list(range(8))
assert line.status_value() & 0x40
assert line.read_and_clear() & 0x40
assert line.status_value() == 0

The example intentionally does not claim the exact low-bit mapping of every SMS VDP revision. Keep the first rejected sprite identity in the trace, then encode the visible register value through a model-specific function. Collision should be driven by pixel-level coverage, not the simplified note_overlap arguments alone.

Validation strategy

Use a test pattern with eight sprites on one line and a ninth just outside the line. Move the ninth sprite one pixel into the range, then adjust its table index while keeping the geometry fixed. Verify the selected list, overflow bit, low status bits, and actual visible pixels independently. Repeat for 8x8 and 8x16 modes, zoom, sprite shift, line wrap, and the Y-table terminator where applicable.

For collision, start with two sprites whose bounding boxes overlap but whose pattern pixels are disjoint; the flag must remain clear. Then move one nontransparent pixel into the other’s pixel position and read the status port immediately before and after that beam location. Test same-color and different-color overlap, because color values should not determine collision. Add three sprites at one pixel to verify that the boolean latch remains set until a status read rather than counting collision pairs.

Finally test read side effects. Read status with pending vertical interrupt, overflow, and collision state in combinations; assert the return byte before clearing, the post-read latch values, and interrupt output. A diagnostic trace should include VDP model, frame and line, horizontal position, table index, evaluated pixel coverage, pending flags, visible status, and the exact guest read timestamp.

Acceptance criteria

A reliable Master System VDP models sprite-table scan order, per-line capacity, model-specific overflow index, pixel-accurate collision, pending-to-visible status timing, and status-read side effects. It can reproduce the same reported index and collision cycle from a compact fixture. With those conditions tested, a missing sprite or raster effect can be traced to selection, pixel opacity, VDP revision, or status timing rather than hidden in the renderer.

Related:

Sources: