IIS Failed Request Tracing: Capture and Read Request-Pipeline Failures

IIS Failed Request Tracing (FREB), also called Failed Request Tracing, records detailed events for requests that match a configured failure condition. It is useful when an IIS access log says a request returned a status code but does not reveal which module, handler, authentication stage, rewrite rule, or application pipeline event produced it. A FREB trace is a request-scoped diagnostic artifact, not a general access log, packet capture, application crash dump, or proof that the browser received the same response the origin server generated.

The trace is especially valuable because a request crosses multiple layers. HTTP.sys receives and queues HTTP traffic before it reaches IIS worker processing. IIS modules and handlers then process the request, and an application can call downstream services or return its own response. A status code in an IIS log and a response generated by HTTP.sys or an upstream proxy are not the same incident. Start from the request’s timestamp, site binding, URL, client, HTTP status and substatus, and trace only the layer that can produce the behavior.

FREB has operational cost. Verbose providers can generate large XML files and capture request metadata that should be handled as sensitive diagnostic data. A rule that traces every successful request can consume disk and expose more information than necessary. Install the IIS Tracing role service where required, enable tracing for only the affected site, use the narrowest useful status/path/time condition, set a file-count limit, reproduce a small number of requests, and disable or remove the temporary rule when the evidence is collected.

Separate IIS request failures from pre-IIS responses

First inspect the IIS W3C log for the matching row, including date/time, site, URI stem and query, method, client address, user agent, status, substatus, Win32 status, and time-taken fields if configured. The status and substatus often narrow the class of failure. If there is no corresponding IIS log row, check HTTPERR logs and edge or proxy telemetry before assuming that the application pool is responsible.

Microsoft documents that HTTP.sys can generate a 4xx response before the request reaches the IIS worker pipeline. Such responses might be absent from IIS W3C logs but present in HTTPERR logs. A client HAR can provide a clue: a Microsoft-HttpApi/2.0 response header is evidence to investigate HTTP.sys. FREB is not a substitute for HTTPERR or a network trace when the request never reached the configured IIS pipeline.

When an IIS log row exists, correlate it with the website ID and binding, application path, application pool, worker PID, application deployment version, and Application/System event records. An app-pool crash may cause several requests to fail but does not explain which module set a particular status. FREB can show a request-pipeline event sequence; pair it with worker-process and application logs when the failure is process initialization, an unhandled exception, or a downstream dependency.

Create a rule that captures the incident without becoming the incident

The IIS Tracing role service must be installed before Failed Request Tracing can collect the full pipeline events. In IIS Manager, select the target site, open Failed Request Tracing Rules, and add a rule for the narrowest content path and response condition that reproduces the failure. For an intermittent failure, use the observed status code or a bounded time condition rather than tracing all requests indefinitely. The site must also have Failed Request Tracing enabled with a known log directory and a maximum trace-file count.

An example rule for ASP.NET .aspx content traces only HTTP 500 responses and requests warning-level ASP.NET pipeline areas:

<system.webServer>
  <tracing>
    <traceFailedRequests>
      <add path="*.aspx">
        <traceAreas>
          <add provider="ASPNET"
               areas="Infrastructure,Module,Page,AppServices"
               verbosity="Warning" />
        </traceAreas>
        <failureDefinitions statusCodes="500" />
      </add>
    </traceFailedRequests>
  </tracing>
</system.webServer>

This configuration is scoped to ASP.NET .aspx requests and does not trace every static, native, or reverse-proxy request. The Tracing role service and site-level Failed Request Tracing logging settings must already be configured. For a native IIS module, classic ASP, or ARR rewrite failure, choose the providers and areas that correspond to that request path. Verify the configuration at the intended site or application level; a web.config fragment can be inherited or locked differently in a real IIS hierarchy.

Verbose tracing is appropriate for a short controlled reproduction, not a permanent default. IIS writes a trace when the configured failure condition is met. A large percentage range such as 400-600 may be useful in a brief incident window when the failure code is not yet known, but it can include many unrelated requests. Start with a narrow filter when evidence is available and widen only if the first pass does not capture the event.

Store traces on a volume with monitored free space and an access control list that limits readers to authorized operators. The default path is commonly under the system drive’s inetpub FailedReqLogFiles directory; configure and document the actual directory instead of assuming defaults. Each site has a W3SVC directory corresponding to its numeric site ID. Verify that the worker and logging components can write to the location, that the retention count is suitable, and that log rotation or collection will not delete files before analysis.

Reproduce one request and identify the decisive event

Reproduce with a known URL, method, authentication mode, client, and timestamp. Capture the client’s response and correlation identifier if the application emits one. Do not repeatedly load a destructive POST or checkout action simply to fill the trace directory. For production, coordinate with the application owner and use a read-only or test request where possible.

Open the FREB XML trace with the included XSL stylesheet in a supported browser or copy both the trace and stylesheet to an analysis workstation. On Server Core, transfer the artifact and stylesheet to a machine with a browser rather than changing the server installation to add a UI. Read the event timeline in order. Look for module notifications, URL rewrite decisions, authentication/authorization events, handler selection, managed pipeline events, response headers, and the event that records the final status.

The last visible event is not automatically the cause. IIS can log cleanup and response-finalization events after the module that failed. Find the first event whose result or status changes unexpectedly, then examine the immediately preceding modules and the request data they saw. The trace can identify a module and stage, but it may not include the application’s full exception, SQL statement, downstream response body, or root cause. Follow up in application logs, Windows events, database telemetry, or dependency traces.

Treat each trace as one request observation. If two traces differ, compare URI, method, authentication, host header, client, route, process, and configuration version. A success trace from a different server or app pool does not disprove a failure on the affected node. For load-balanced sites, pin or record the backend instance and correlate the trace filename with the site ID and host.

Read 4xx and 5xx traces with the right boundary

For 4xx results, distinguish authentication failures, authorization denials, invalid URL/request parsing, missing resources, request filtering, and application-defined client errors. A 401 challenge sequence can involve more than one request and multiple authentication modules. A 403 can be generated by authorization rules, IP restrictions, request filtering, filesystem ACLs, or application logic. Compare status and substatus, trace events, and the identity that actually accesses the content; changing file permissions globally is not a diagnosis.

For 404, determine whether the requested path maps to the expected site and application, whether rewrite rules changed it, whether the handler receives it, and whether the file is present with the appropriate ACL. A 404 can intentionally hide a protected resource. Do not expose detailed error pages to remote clients simply to see more diagnostics; Microsoft warns that remote detailed errors can disclose sensitive server information. Use FREB and local-only detailed errors where appropriate, then revert temporary diagnostic changes.

For 500-series responses, identify whether IIS, a runtime handler, custom module, or application code produced the status. A request trace can establish the pipeline sequence, while an application log or dump can reveal the exception and state that caused it. Distinguish an HTTP 500 response from a worker process crash or rapid-fail protection event: one failed request can leave the pool running; repeated process startup failures may prevent later requests from reaching the application at all.

For time-based failures, set the trace condition to capture requests exceeding a deliberate threshold. A long request can be blocked on the application, file I/O, a database, an external API, thread-pool starvation, or lock contention. FREB shows elapsed pipeline stages but does not by itself prove a CPU bottleneck or identify the owner of every wait. Correlate with application tracing, ETW/PerfView, request timing, and resource metrics if the trace only shows time accumulating in a broad module.

Protect trace data and bound retention

FREB output can include requested URLs, headers, authentication-related metadata, server paths, and module details. Review the trace as operationally sensitive. Do not attach raw traces to a public ticket or repository. Copy only the necessary files into an access-controlled incident workspace, redact tokens and personal data, and follow the organization’s retention policy.

Configure a maximum number of trace files and an appropriate maximum file size. A size limit can truncate a trace before the interesting event, so inspect whether the XML ends cleanly and whether the logging configuration reports truncation. If a trace is truncated, reduce verbosity, narrow the URL/provider set, increase a justified per-file bound temporarily, or capture fewer requests. Do not set unlimited trace sizes on a production server as a convenience.

Create a change record with the site, exact rule, status/path/time condition, provider areas and verbosity, directory, file-count limit, operator, start time, and planned removal time. After the reproduction, export required evidence, remove the temporary broad rule or restore the approved baseline, and verify that the trace files stop growing. A diagnostic rule left enabled can become a storage, privacy, and performance issue months after the incident.

FREB incident workflow

Match the client failure to IIS W3C logs, HTTPERR logs, proxy data, site ID, and server; determine whether the request reached IIS.
Install or verify the IIS Tracing role service and enable logging only for the affected site.
Configure a narrow URL/status/time rule, appropriate providers, limited verbosity, a known directory, and retention limits.
Reproduce a small number of controlled requests and record exact timestamps, host, backend, and correlation IDs.
Read the FREB timeline, identify the first unexpected pipeline result, and corroborate it with application, process, and dependency evidence.
Protect and retain only the needed trace files; remove temporary rules and verify disk and logging return to baseline.

FREB works best as an instrumented observation of one request path. It gives an operator the sequence that a normal access log omits, but disciplined scoping, layer separation, and correlation are what turn the XML into a defensible diagnosis.

Related:

Sources: