Skip to content
WindowsDeep Dive Published Updated 3 min readViews unavailable

Windows I/O Completion Ports: Building Correct Overlapped-I/O Workers

Understand IOCP queue and worker semantics, preserve OVERLAPPED lifetimes, and distinguish failed I/O packets from completion-port timeouts.

An I/O completion port (IOCP) is a kernel-managed queue used to collect completion packets for asynchronous I/O. A process associates one or more handles with a port, starts operations with OVERLAPPED structures, and uses worker threads to dequeue the result. IOCP is not the operation itself: the operation is initiated by an API such as ReadFile, while the port provides a scalable completion and dispatch mechanism.

The core correctness rule is lifetime. The OVERLAPPED structure, its buffer, and any request state referenced by the completion key must remain valid until the completion is dequeued or a documented cancellation path has completed. Returning from the initiating call with ERROR_IO_PENDING means the operation is still in flight, not that the buffer can be reused.

Create the port and associate handles

Create a port with CreateIoCompletionPort using INVALID_HANDLE_VALUE, then associate each file, socket, or other supported handle with the port. The completion key is application-owned context associated with the handle. The concurrency value limits how many associated worker threads may run at once; it does not limit how many I/O requests can be outstanding.

Handles used for overlapped file I/O must be opened with FILE_FLAG_OVERLAPPED. Initialize one OVERLAPPED and request object per outstanding operation, and do not reuse either while the request is pending. A synchronous return must also be handled according to the chosen notification policy; do not assume every operation follows the pending path.

Dequeue and classify completions

GetQueuedCompletionStatus returns a packet’s byte count, completion key, and OVERLAPPED pointer. Its return value must be interpreted together with that pointer:

// Worker-loop fragment: port is valid; report/handle/finish helpers are application-defined.
DWORD bytesTransferred = 0;
ULONG_PTR completionKey = 0;
OVERLAPPED* overlapped = nullptr;

BOOL ok = GetQueuedCompletionStatus(
    port, &bytesTransferred, &completionKey, &overlapped, INFINITE);

if (overlapped == nullptr) {
    if (!ok) {
        DWORD waitError = GetLastError();
        // The wait failed; no I/O completion was dequeued.
        report_wait_failure(waitError);
    } else {
        // For example, an application control packet was posted.
        handle_control_packet(completionKey);
    }
} else if (!ok) {
    DWORD operationError = GetLastError();
    // A failed I/O completion packet was dequeued; retire this request.
    finish_failed_request(overlapped, operationError);
} else {
    finish_successful_request(overlapped, completionKey, bytesTransferred);
}

When GetQueuedCompletionStatus returns FALSE with a non-null OVERLAPPED pointer, it dequeued a completion packet for a failed I/O operation. Capture GetLastError immediately. When the pointer is null, there was no I/O completion packet to process; the call may have timed out or failed while waiting. This distinction prevents a common bug where an application leaks the request object for failed operations or dereferences null after a timeout.

Queue order is not a promise about worker order

The port’s packet queue is FIFO, while waiting threads are released in LIFO order. The concurrency setting controls runnable worker concurrency and can help keep a CPU-bound pool from overscheduling. It does not create ordering guarantees for application work after a worker dequeues a packet; multiple workers can process completions concurrently and finish in a different order.

Use the completion key to recover per-handle context and the OVERLAPPED pointer to recover per-operation context. Avoid using a raw pointer as a key without a clear ownership protocol. Treat posted control packets as a separate message kind and ensure shutdown packets cannot be confused with actual I/O requests.

Cancellation and teardown

CancelIoEx requests cancellation, but the request storage cannot be freed merely because cancellation was requested. The operation may already have completed, or a completion notification may still be delivered. Retire the request only after the completion/cancellation outcome is known. Close every associated handle and the completion-port handle during orderly shutdown, and make worker exit explicit.

For simpler workloads, the Windows thread-pool I/O API wraps IOCP and manages worker lifecycle. For raw IOCP, test immediate completion, ERROR_IO_PENDING, device removal, cancellation races, zero-byte reads, failed completion packets, and shutdown while workers are blocked. Record operation IDs and handle ownership so production hangs can be diagnosed without guessing.

Related:

Sources:

Comments