Game Boy Advance Windows and Blending: Layer Masks, Priority, and Pixel Rules

The Game Boy Advance’s window registers are not ordinary rectangular clipping rectangles. They classify each screen pixel into a region, and that region supplies a six-bit permission mask for backgrounds, objects, and color effects. The compositor still applies its normal layer priority after that classification. Alpha blending is a separate stage: it uses a top-layer set, a bottom-layer set, a blend mode, and fixed-point weights. An emulator that treats a window as a scissor box or applies alpha to the final RGB image will get common effects wrong even when the same layers look correct with windows disabled.

The useful mental model is a per-pixel pipeline: determine the active window region, filter candidate layers and effects through its mask, resolve visible pixels using GBA priority and transparency rules, then apply the selected color effect where the hardware permits it. These stages should be explicit in both an emulator and a graphics program.

The register groups and their separate jobs

DISPCNT has independent master enables for BG0-BG3, OBJ, WIN0, WIN1, and the OBJ window. The layer enables are not replaced by the window registers: a layer must be enabled globally and allowed in the pixel’s window region. Conversely, enabling a layer in WININ or WINOUT cannot make a globally disabled layer appear. The window enable bits are controls for region classification, not layer bits.

WIN0H and WIN1H each pack an 8-bit left coordinate in bits 8-15 and an 8-bit right coordinate in bits 0-7. WIN0V and WIN1V use the same layout for top and bottom. For an ordinary non-wrapping rectangle, the left and top edges are included while the right and bottom edges are excluded. On the 240-by-160 display this means the right edge may be 240 and the bottom edge may be 160; a rectangle covering the full screen is conceptually (left=0, right=240, top=0, bottom=160).

The geometry registers define WIN0 and WIN1 only. They do not define the OBJ window, whose region comes from the nontransparent pixels of OBJ entries configured in object-window mode. Those objects mark pixels for classification; they are not drawn as ordinary colored sprites. WININ describes permissions inside WIN0 and WIN1. WINOUT describes permissions outside both rectangular windows and, in its upper byte, inside the OBJ-window region. This split is easy to miss when packing the 16-bit registers.

Classify pixels before applying masks

Each visible pixel belongs to one effective window region. Where rectangular windows overlap, WIN0 wins over WIN1. If neither rectangular window contains the pixel, a nontransparent OBJ-window marker may classify it as OBJ-window; otherwise the pixel is in WINOUT. The priority is about region selection, not about which background or sprite is visually in front. After region selection, the relevant six-bit control group says whether BG0, BG1, BG2, BG3, OBJ, and color special effects are allowed at that pixel.

This means overlapping regions are not combined by OR-ing their enable masks. If WIN0 covers WIN1, the WIN0 mask applies at that pixel, even if WIN1 would allow additional layers. Similarly, an OBJ-window sprite does not simply sit above WIN0 to reveal its own mask: rectangular window priority still determines the region. Keep a small, explicit function for the region choice and test it independently from the compositor.

The six permission bits are also not a replacement for ordinary layer priority. If two backgrounds remain enabled, their BG control priority still decides which opaque pixel is on top. Transparent palette entries do not block lower layers in the same way as visible pixels. A window controls eligibility; it does not reorder eligible layers or manufacture an opaque pixel. When the enabled layers in a region contribute no visible pixel, the backdrop color remains the result.

The window’s color-effect permission is independent of its layer permissions. It is possible to show a layer in a region while disabling blending there, or to permit a color effect while restricting which layers can participate. In an emulator, carry the effect-enable flag forward with the region rather than folding it into the layer mask.

Boundary behavior and scanline-shaped effects

For normal rectangles, use the documented half-open interval: left <= x < right and top <= y < bottom. This avoids the classic one-pixel seam caused by interpreting the right and bottom coordinates as inclusive. Reversed bounds are not necessarily an empty rectangle. The GBA references describe wrapping behavior: when the right value is smaller than the left, the horizontal window includes the portions on either side of the excluded interval; the analogous rule applies vertically. Reversing both axes can create a cross-shaped excluded area. If software relies on unusual values near the screen edge, validate against hardware rather than silently normalizing them as an empty rectangle.

The fixed registers describe rectangles, but a game can update them during horizontal blanking to make a region vary by scanline. This is one way to create a shape that is not rectangular, such as a widening cone of light or a horizontal reveal. Such effects depend on timing: the update must land in the intended blanking interval, and an emulator needs to model the register value used for each rendered line. A screenshot-only test can miss a one-line transition error; capture the boundary row and adjacent rows separately.

There is a documented caution here: some older programming notes report hardware edge cases when vertical bounds are near or beyond the visible range, and observed emulator behavior has not always matched those reports. Treat ordinary in-range rectangles as the portable baseline. For reversed or out-of-range bounds, consult a register reference and a hardware test suite, and describe any emulator-specific compatibility behavior rather than asserting that one intuitive clipping rule is universal.

Color effects are not generic transparency

BLDCNT selects a mode and identifies candidate top (first target) and bottom (second target) layers. The modes are: no effect, alpha blend, brighten toward white, and darken toward black. BLDALPHA contains EVA and EVB, each a 5-bit unsigned coefficient used as a 4.4 fixed-point value; values above 16 are not valid weights. BLDY supplies the coefficient for brightness effects, also limited to the hardware’s 0-to-16 range. These are global effect parameters, while the window effect bit decides whether the effect can operate at a particular pixel.

For alpha mode, a useful per-channel model is min(31, (EVA * A + EVB * B) >> 4) for 5-bit color components. It is not a conventional per-pixel alpha channel stored alongside every color. The first and second targets are layer classes selected by BLDCNT; the visible pixels must also satisfy normal composition constraints. In particular, the A target must be in front of an eligible B target for the expected blend to occur. If the blend looks like the foreground is unchanged, inspect layer priorities and target bits before changing the coefficients.

Semi-transparent OBJ pixels have special behavior: OBJ attributes can request blending, and the hardware’s sprite rules are not identical to simply applying the current BLDCNT operation to every sprite pixel. Do not implement OBJ blending as a blanket post-process over the final framebuffer. Follow the documented OBJ and first/second-target rules, and include tests for overlap and non-overlap. Windows can further disable the effect region by region, so a correct blended overlap outside a window is not evidence that the window mask is correct.

A minimal register setup

The following example assumes mode 0 backgrounds have already been configured, BG0 is in front of BG1, both are enabled in DISPCNT, and palette data is valid. It gives WIN0 the central rectangle, allows BG0/BG1 and effects there, and leaves the OBJ window disabled. Register writes should be made at a safe point in the frame when the program expects the new state to take effect.

#include <stdint.h>

#define REG16(addr) (*(volatile uint16_t *)(addr))
#define DISPCNT   REG16(0x04000000)
#define WIN0H     REG16(0x04000040)
#define WIN0V     REG16(0x04000044)
#define WININ     REG16(0x04000048)
#define WINOUT    REG16(0x0400004A)
#define BLDCNT    REG16(0x04000050)
#define BLDALPHA  REG16(0x04000052)

enum {
    DCNT_BG0 = 1u << 8,
    DCNT_BG1 = 1u << 9,
    DCNT_OBJ = 1u << 12,
    DCNT_WIN0 = 1u << 13,
    WIN_BG0 = 1u << 0,
    WIN_BG1 = 1u << 1,
    WIN_OBJ = 1u << 4,
    WIN_EFFECT = 1u << 5,
    BLD_ALPHA = 1u << 6,
    BLD_A_BG0 = 1u << 0,
    BLD_B_BG1 = 1u << 9
};

static void setup_window_and_blend(void)
{
    /* Right and bottom are exclusive: the rectangle is x=32..207, y=24..135. */
    WIN0H = (32u << 8) | 208u;
    WIN0V = (24u << 8) | 136u;

    /* WIN0: BG0, BG1, and color effects are permitted. */
    WININ = WIN_BG0 | WIN_BG1 | WIN_EFFECT;

    /* WINOUT byte: keep the same layers and effects outside WIN0. */
    WINOUT = WIN_BG0 | WIN_BG1 | WIN_EFFECT;

    /* BG0 is first target; BG1 is second target; mode 1 is alpha blend. */
    BLDCNT = BLD_A_BG0 | BLD_B_BG1 | BLD_ALPHA;
    BLDALPHA = 8u | (8u << 8);  /* EVA=8/16, EVB=8/16 */

    /* Preserve mode and any other required enables in real code. */
    DISPCNT |= DCNT_BG0 | DCNT_BG1 | DCNT_WIN0;
}

This is a register illustration, not a complete display initialization routine. A production renderer should compose DISPCNT from a known state instead of inheriting unknown bits, write all four WININ and WINOUT permission groups deliberately, and set BLDY when using brightness modes. If an OBJ window is enabled, configure at least one object in object-window mode and set its WINOUT mask; otherwise the OBJ-window region has no useful shape. Also remember that WININ and WINOUT are write-only in common hardware references, so keep software-side state if later updates need to preserve unrelated bits.

An emulator test plan that catches pipeline mistakes

Start with pure region tests. Check pixels just before, on, and just after each left, right, top, and bottom edge. Verify that right and bottom are excluded. Add overlapping rectangles and assert WIN0 precedence, then add a nontransparent OBJ-window marker and verify that WINOUT applies only where no higher-priority rectangular window owns the pixel. Use reversed bounds as a separate compatibility test rather than treating them as a malformed version of the normal case.

Next test the masks. For each of WIN0, WIN1, WINOUT, and OBJ-window, independently disable each background, OBJ, and color effect. Confirm the global DISPCNT layer bit still gates the result. Give BG0 and BG1 different solid colors and known priority values; test both possible priority orders and a transparent pixel so the compositor’s regular rules are exercised after the window mask.

For blending, use small 5-bit test colors whose expected channel results are easy to calculate. Exercise EVA/EVB at 0, intermediate values, and 16; test clamping and reject or mask invalid values according to the hardware model. Check a valid top/bottom order, then reverse the order and confirm the effect does not incorrectly blend a behind layer into a front one. Separately test brighten and darken, semi-transparent OBJ overlap, and a window that allows the layers but blocks color effects. Compare the emulator output with a hardware-derived test ROM or a well-documented reference capture before claiming exact pixel parity.

Finally test timed updates. Change a boundary once per scanline and render a diagnostic pattern with a contrasting color on each side. Record the line where the old and new window values take effect. Repeat for both rectangular windows and, if supported, an OBJ window whose marker moves. Keep raster update timing tests separate from static classification tests: if a line is wrong, that tells you whether the bug is in interval logic or register-update timing.

GBA windows are best modeled as a priority-ordered region classifier with independent layer/effect masks, followed by the console’s normal pixel-priority and color-effect logic. Preserving those boundaries makes simple rectangles, x-ray effects, OBJ-window masks, fades, and raster-shaped reveals testable without turning all of them into a generic host-GPU scissor or alpha operation.

Related:

Sources: