Skip to content
WindowsDeep Dive Published Updated 6 min readViews unavailable

Windows Thread-Pool I/O: Pair Overlapped Requests With Completion Callbacks

Pair every overlapped I/O request with Windows thread-pool notifications, including immediate success, cancellation, callback ownership, and shutdown.

Windows thread-pool I/O connects overlapped I/O completions to a callback without requiring an application-managed completion-port worker loop. The API is compact, but its accounting rules are strict: before each asynchronous operation, call StartThreadpoolIo; when an operation will not produce a callback, balance that notification with CancelThreadpoolIo; and keep the OVERLAPPED structure, buffer, and request state alive until the operation is genuinely complete.

This facility does not initiate I/O, own your request objects, or cancel a pending operation when a callback wait returns. It associates a file handle with a PTP_IO object and routes eligible completions to the callback you provide. Treat every request as an explicit state machine rather than assuming one API call equals one guaranteed callback.

Bind an overlapped handle and callback

Create the file handle for overlapped I/O, normally with FILE_FLAG_OVERLAPPED, then pass it and a callback to CreateThreadpoolIo. The handle must remain valid while I/O is submitted. Each concurrent operation needs its own OVERLAPPED and storage; do not reuse a single structure just because the initiating function has returned ERROR_IO_PENDING.

#include <windows.h>
#include <array>
#include <cstddef>
#include <new>

struct ReadRequest {
    OVERLAPPED overlapped{}; // First member for this example's cast.
    std::array<std::byte, 4096> buffer{};
};

VOID CALLBACK ReadCompleted(PTP_CALLBACK_INSTANCE, void*, void* overlapped,
                            ULONG result, ULONG_PTR bytesTransferred, PTP_IO) {
    auto* request = reinterpret_cast<ReadRequest*>(overlapped);
    if (result == NO_ERROR) {
        Consume(request->buffer.data(), static_cast<size_t>(bytesTransferred));
    } else {
        RecordReadFailure(result);
    }
    delete request; // Completion path owns reclamation.
}

PTP_IO CreateIo(HANDLE file, void* context) {
    PTP_IO io = CreateThreadpoolIo(file, ReadCompleted, context, nullptr);
    if (io == nullptr) {
        ReportSetupFailure(GetLastError());
    }
    return io;
}

This fragment assumes the request layout shown, ReportSetupFailure as an application-defined setup helper, and a callback context that outlives the I/O object. A production request should be non-copyable, use a clear ownership type, and make error reporting safe for a thread-pool callback. If OVERLAPPED is not the first member, recover the containing request with a checked container-of pattern such as CONTAINING_RECORD, not a cast to the request’s beginning.

Start accounting before every operation

Call StartThreadpoolIo immediately before issuing each asynchronous operation on the associated handle. Starting it afterwards races a fast completion and can make the pool miss the notification; omitting it can corrupt memory. If the API returns failure with an error other than ERROR_IO_PENDING, no completion will arrive for that operation, so call CancelThreadpoolIo and reclaim the request on the initiating path.

bool SubmitRead(HANDLE file, PTP_IO io) {
    auto* request = new (std::nothrow) ReadRequest{};
    if (request == nullptr) return false;

    StartThreadpoolIo(io);
    if (!ReadFile(file, request->buffer.data(),
                  static_cast<DWORD>(request->buffer.size()), nullptr,
                  &request->overlapped)) {
        const DWORD error = GetLastError(); // Capture before another Win32 call.
        if (error != ERROR_IO_PENDING) {
            CancelThreadpoolIo(io);
            delete request;
            return false;
        }
    }
    return true;
}

The code assumes default completion-port notification behavior. A successful return from an overlapped ReadFile can mean the operation completed inline. Normally, the completion notification still arrives. If the handle was configured with FILE_SKIP_COMPLETION_PORT_ON_SUCCESS, however, immediate success does not call the thread-pool callback. In that mode, the caller must detect immediate success, call CancelThreadpoolIo to balance the start notification, and process or retire that request synchronously. Do not both process it inline and allow the callback to free it.

Also ensure the handle was opened for overlapped I/O. ReadFile requires a valid, distinct OVERLAPPED for each concurrent operation. For a pending result, ownership transfers to the completion path only after submission is accepted. Preserve the request’s buffer and metadata until then; cancellation being requested is not the same as completion.

The zero-initialized OVERLAPPED in this sample requests a read at offset zero. For a seekable file, set Offset and OffsetHigh to the byte range owned by that request before submitting it; Windows does not advance those fields automatically for overlapped operations. Streams that do not support byte offsets ignore them. If reads are parallel, assign non-overlapping ranges or use an application-level sequencing rule rather than depending on a shared file position. A successful callback can report fewer bytes than requested, particularly near end-of-file. Consume the reported byte count, not the buffer’s capacity, and schedule another read only when the file format and application policy require one.

Interpret callback results as operation outcomes

The callback receives the OVERLAPPED pointer, a Win32 error code, and a byte count. A nonzero result is an I/O failure or cancellation outcome, not evidence that the callback itself failed. Record or route the result before reclaiming the request. Callback invocations may run concurrently, so shared state needs its own synchronization policy, and callback code must not assume request completions arrive in submission order.

If the application has several operations in flight, track them separately. An operation that fails synchronously is reclaimed by the submitter after cancelling the notification. A queued completion is reclaimed by the callback. An inline success under skip-on-success is reclaimed by the synchronous path after balancing the notification. This one-owner rule prevents both leaks and double-frees.

Cancellation and shutdown have two layers

Shutdown begins by preventing new submissions. CancelIoEx can request cancellation of outstanding operations on the handle, optionally targeting a particular OVERLAPPED. It returns before canceled operations necessarily finish; a completion or cancellation result still has to reach the request’s owner. ERROR_NOT_FOUND can mean that the operation has already completed or no matching request remains, so it is not a signal to free every request blindly.

WaitForThreadpoolIoCallbacks(io, TRUE) cancels callbacks that are queued but have not started. It does not cancel the underlying pending I/O. Microsoft explicitly warns not to free an OVERLAPPED while the request is pending. With the TRUE option, arrange another reliable way to determine that each operation completed before freeing its request; otherwise use the normal callback path and wait for those callbacks to retire the requests.

A safe high-level teardown is:

  1. Stop producers from submitting new I/O and mark the service as shutting down.
  2. Request cancellation with CancelIoEx if policy requires it; keep all request objects and buffers alive.
  3. Drain outstanding operations and their callbacks. If using WaitForThreadpoolIoCallbacks(..., FALSE), callbacks run to completion and can reclaim their requests. Do not confuse this with canceling I/O.
  4. Once no overlapped operation remains, close the associated handle and then call CloseThreadpoolIo. Do not issue any more overlapped operations after closing the thread-pool I/O object.

The exact producer barrier and outstanding-request counter are application responsibilities. Protect the transition so a submitter cannot pass the shutdown check and call StartThreadpoolIo after the I/O object begins closing. A join or condition variable can coordinate the non-real-time control plane, but callback code must still be able to make progress while shutdown waits.

Test the awkward paths deliberately

Exercise a request that completes immediately, one that returns ERROR_IO_PENDING, a synchronous error, a cancellation race, and shutdown while operations are in flight. If the application enables skip-on-success, test both the inline-success and pending-success cases separately. Assert that every successful StartThreadpoolIo is paired with either a completion callback or the appropriate CancelThreadpoolIo path, and that each request is reclaimed exactly once.

The thread-pool I/O APIs date to Windows Vista / Windows Server 2008; Microsoft documents _WIN32_WINNT >= 0x0600 for the interface. However, the ReadFile sample above passes NULL for lpNumberOfBytesRead, and Microsoft documents that this parameter cannot be NULL on Windows 7. As written, target Windows 8 or later; if Windows 7 compatibility is required, use a separately tested compatibility path rather than inferring support from the older thread-pool API minimum. Test the actual Windows SDK and runtime because a cross-compiler cannot reproduce kernel completion timing or cancellation races.

Related:

Sources:

Comments