DOS EXEC Overlays: Caller-Owned Memory and Explicit Relocation

DOS INT 21h with AX=4B03h loads an overlay into memory selected by the caller. It does not perform the complete child-process lifecycle associated with AX=4B00h. A program using overlays must manage the destination, relocation assumptions, entry point, calling convention, and lifetime of the loaded code. The loader cannot infer an application-level plugin interface from an executable header.

This article examines the FreeDOS kernel implementation reviewed on October 11, 2026. Its source provides direct evidence for the behavior described below; implementation observations are not blanket guarantees about every historical DOS clone. Test the exact kernel and toolchain you intend to support. The assembly fragment is a calling-sequence illustration, not a complete tested overlay manager.

An overlay load is not a process launch

Ordinary EXEC can create a child Program Segment Prefix, clone an environment, arrange the child’s initial stack and entry address, and transfer control. FreeDOS’s MZ overlay path deliberately bypasses those operations. After reading and relocating the image, it closes the executable file and returns success before the child-PSP creation and control-transfer path.

That difference changes how the loaded code must behave. It does not receive a newly initialized process context merely because its input file has an MZ header. A module that expects a standalone executable’s startup routine, environment setup, stack initialization, and termination flow is not automatically callable as an overlay.

The caller still has its existing PSP, handles, environment, and execution context. An overlay is code placed into that process’s address space. The application must explicitly call an entry point compatible with its own protocol and arrange a return to its resident code.

The parameter block has two independent words

The kernel’s exec_blk union defines an overlay load structure containing load_seg and reloc, both 16-bit words. The first identifies the paragraph-aligned destination segment. The second is the relocation factor added to MZ relocation words.

These fields often have the same value in a simple module linked with a zero-based segment model, but they answer different questions. load_seg answers where the image bytes go. reloc answers what adjustment to add to segment values identified by the executable’s relocation table. An overlay format or linker arrangement can require a different relationship. Do not derive the factor from a filename or set it to a conventional constant without understanding the module’s link model.

The interrupt interface passes the null-terminated pathname in DS:DX and the parameter block in ES:BX. The following fragment assumes a caller whose DS addresses its own resident data and whose destination was already allocated and sized:

; 16-bit real-mode, Intel syntax: illustrative integration fragment.
; destination_segment has been obtained by the caller.
; This module's link contract requires relocation by its load segment.
mov ax, [destination_segment]
mov [overlay_parameters], ax
mov [overlay_parameters+2], ax

push ds
pop es
mov bx, overlay_parameters
mov dx, overlay_path
mov ax, 4B03h
int 21h
jc overlay_load_failed

; Loading succeeded. Calling the module is a separate operation.
; Do not jump to an assumed entry point here.

overlay_parameters: dw 0, 0
overlay_path: db 'MODULE.OVL', 0
destination_segment: dw 0

Real integration must preserve the registers required by the caller and its toolchain. Place error handling so execution cannot fall into the embedded data. Capture the carry flag and error result before another DOS call changes them. The zero destination in this fragment is a placeholder to initialize, not a valid safe destination for a runnable example.

Allocate and validate the destination before loading

In DosExeLoader, the overlay branch takes its memory segment directly from load_seg rather than allocating a new arena or child environment. This leaves sizing and ownership with the caller. A successful return should not be interpreted as proof that the caller supplied a safe buffer.

Determine the image size from the actual executable structure, excluding its header, and account for any runtime storage required by the module’s own contract. Validate relocation targets against the image you intend to load. An application that accepts arbitrary external MZ files needs substantially stronger validation than an application loading modules produced by its own controlled build.

Keep the resident loader, parameter block, error handler, and active stack outside the overlay region. Replacing the memory containing a return address or executing instruction is not a supported hot-swap mechanism. A module manager should record each buffer’s segment, capacity, currently loaded module, and active-call count before it permits replacement.

In conventional DOS, this separation is especially important because the overlay participates in the same unprotected address space. A malformed relocation entry can address unrelated memory. DOS overlays are a memory-management technique, not a sandbox for untrusted extensions.

Understand the MZ relocation operation

The FreeDOS loader reads the executable image after seeking past exHeaderSize paragraphs. For each relocation table entry, it reads an offset and segment pair. In overlay mode, the target word is located at load_seg + relocation_segment, at the specified offset. The loader adds the supplied reloc factor to that word.

For example, consider a controlled synthetic image loaded at segment 3000h, with a relocation entry pointing to offset 0010h in image-relative segment 0002h. The word to modify is at 3002h:0010h. If its original value is 0005h and the agreed relocation factor is 3000h, the resulting word is 3005h.

This calculation is illustrative, not measured runtime output. It shows why the address of the word being patched and the segment value added to that word are separate operations. A far pointer’s offset does not become a physical address merely because its segment word is relocated.

The kernel source also makes an important boundary visible: it returns from the overlay path without applying the ordinary executable’s initial stack and start-address setup to a new process. If your module uses header entry fields to publish an entry location, your own manager must interpret and validate them under a documented convention.

Define a callable module interface

An overlay manager needs an explicit entry offset or export table, near-versus-far call rules, register preservation, data-segment expectations, stack requirements, and error reporting. It must also define whether a module can retain pointers into its own region after returning.

For a replaceable overlay, resident code must not keep a function pointer into the old module and call it after the buffer has been reused. A practical design uses a generation identifier and invalidates exported pointers on replacement. This is an application design recommendation, not a feature supplied by DOS EXEC.

Termination requires equal care. A standalone program’s INT 21h termination call targets a process, not an overlay-local return convention. Modules should return through the agreed calling sequence. Test failure paths as carefully as the normal function result, including what happens if a module detects unavailable data or an unsupported host ABI version.

Do not confuse API modes with library constants

FreeDOS’s header also defines P_OVERLAY as a mode for spawnxx-style functions, describing replacement of a parent by a child. Its value is 2. That is not the interrupt subfunction AL=03h. The source’s DosExec rejects mode 2 as an invalid executable mode.

Names reused across library and kernel APIs are not interchangeable numeric contracts. When reviewing old C examples, identify which function consumes the constant before translating the code into direct interrupt calls.

Verify with a disposable DOS environment

Build a tiny known MZ module with one documented entry and a relocation-bearing far pointer. Record the linker options, header fields, relocation entries, and image size. Run it inside a disposable VM or emulator using the target FreeDOS kernel, never against the only copy of valuable data.

First load without calling it. Inspect destination bytes and relocated words, comparing them with your independent calculation. Then call the entry, verify the agreed result and preserved registers, and confirm that resident state remains usable. Add repeated replacement and error cases: missing file, malformed module rejected by the manager, insufficient destination capacity rejected before EXEC, and attempts to replace an active module.

If loading fails, retain the error code and inspect the path and image structure. If loading succeeds but the call fails, inspect relocation, entry selection, stack, and data-segment assumptions. Do not keep changing relocation factors experimentally until a crash disappears. Recover by restarting the disposable process, then repair the violated contract from evidence.

Related:

Sources: