Windows Firewall IPsec Connection Security: Stage Authentication and Isolation

Windows Defender Firewall connection security rules use IPsec to authenticate peers and, depending on policy, protect traffic with integrity and encryption. They are distinct from ordinary firewall rules: a firewall rule determines whether matching traffic is allowed or blocked, while an IPsec rule negotiates security associations (SAs) between endpoints. A firewall allow rule with -Authentication Required depends on a matching IPsec policy. Creating only one half can leave traffic blocked or permit traffic without the protection an operator expected.

IPsec is useful for host-to-host protection, server isolation, and domain isolation, but broad policy can sever management, DNS, cluster, backup, or application paths if peer identity or cryptographic proposals do not match. Start with a precise communication matrix and a narrow pilot. Do not deploy an “all endpoints, require inbound” rule across a production network as an experiment. Preserve an out-of-band management path and a tested rollback before enforcing authentication or encryption.

Map the intended traffic and trust boundary

Define the source and destination device groups, direction, protocol, ports, profile, authentication identity, integrity/encryption requirements, exceptions, and business owner. Include infrastructure dependencies such as domain controllers, DNS, time, certificate revocation, monitoring, backup, management, cluster heartbeat, storage, and software distribution. Establish how devices authenticate when they are not domain joined, are across a trust boundary, or are temporarily offline. A domain-isolation design must not prevent an endpoint from reaching the services it needs to obtain domain credentials and policy.

Separate the main and quick mode decisions. Main mode establishes an authenticated channel between peers; quick mode negotiates protection for traffic. Windows PowerShell models authentication proposals, phase 1 and phase 2 authentication sets, cryptographic sets, IPsec rules, and firewall rules as separate objects. Default proposal sets may be sufficient for a compatible Windows-only deployment, but interoperability requirements, algorithm policy, and peer versions should be tested rather than assumed.

Prefer Kerberos or certificate authentication for managed environments where supported and correctly provisioned. A pre-shared key has difficult rotation and distribution properties; never embed one in scripts, Group Policy preference files, tickets, or source control. If a narrowly controlled use case requires a pre-shared key, protect it as a secret, define rotation and recovery, and avoid reusing it across unrelated peers. Confirm machine or user identities match the intended security boundary.

Inspect effective policy and live SAs

Before creating or changing policy, record which Group Policy Object or local policy store supplies the rule and which profile applies to each interface. A rule configured in a domain GPO can coexist with local rules and other policy stores. Use -TracePolicyStore where supported to identify the source. Inventory current rules, matching firewall filters, active main-mode SAs, and quick-mode SAs. Preserve this evidence so that a failed negotiation can be compared with the before state.

The following commands are read-only examples for a scoped endpoint pair. The filter values are documentation-only addresses and ports; replace them with the approved communication tuple when collecting production evidence:

$remoteAddress = '192.0.2.25'

Find-NetIPsecRule -RemoteAddress $remoteAddress `
    -Protocol TCP -RemotePort 443 |
    Format-List *

Get-NetIPsecMainModeSA |
    Format-List *

Get-NetIPsecQuickModeSA |
    Select-Object LocalEndpoint, RemoteEndpoint, LocalPort, RemotePort,
        FirstIntegrityAlgorithm, FirstCipherAlgorithm,
        SecondIntegrityAlgorithm, SecondCipherAlgorithm

Confirm the property names against the installed NetSecurity module; versions can expose different SA details. Find-NetIPsecRule helps identify policies that match an address and traffic tuple, but it is not proof that the expected rule negotiated successfully. Active SAs show established security associations at query time, not every packet or failed attempt. Correlate them with firewall policy, network traces, Windows Filtering Platform events, and application behavior.

If an expected SA is missing, first verify endpoint addresses, DNS, routing, profile classification, policy refresh, and whether both endpoints have compatible authentication and cryptographic proposals. Confirm that no device in the path performs NAT or protocol filtering in a way that conflicts with the selected negotiation. Do not clear all SAs to “refresh” the policy until you have captured the failure and understand the impact on existing connections.

Design firewall and IPsec rules as a pair

Start with a narrow firewall rule for the application traffic and a connection security rule scoped to the same peers. For a secure firewall rule, Microsoft’s New-NetFirewallRule supports -Authentication Required; the matching IPsec rule must authenticate the traffic. Encryption requirements need corresponding IPsec quick-mode protection, and the firewall rule’s security settings must match the intended outcome. If one endpoint requests protection and the other requires it, test both inbound and outbound flows and account for how Windows evaluates the rule direction.

For domain isolation, define computer groups and authentication policy in a managed GPO. Stage an allow-request policy before requiring it where appropriate, so operators can verify that expected devices negotiate IPsec. Do not use “request” as a permanent substitute for a documented enforcement policy. Keep a bootstrap path for systems that are not yet domain-authenticated and include emergency access that does not rely exclusively on the policy being changed.

When building custom proposals, choose algorithms supported by every endpoint and approved by current organizational cryptographic policy. Avoid legacy algorithms shown in old samples unless compatibility requirements have been assessed and the security owner has accepted them. Test negotiation on the exact Windows Server and client versions. A configuration that parses successfully may still fail to negotiate because the two peers have no common proposal or authentication method.

Pilot policy in an isolated scope

Create a dedicated test GPO or policy store and link it only to pilot systems. Export or document the pre-change firewall and IPsec configuration. Include endpoints that represent different sites, network profiles, server roles, and management paths. Verify successful traffic in both directions, the expected authentication identity, integrity and encryption mode, and application-level health. Test restart, sleep/resume where relevant, machine password rotation, certificate renewal, and policy refresh.

Monitor both peers. Confirm that the expected main-mode and quick-mode SAs appear, the correct firewall rule matches, and nonmember or unauthenticated peers are blocked only where intended. Test legitimate negative cases using controlled test hosts. Ensure a management workstation can still administer the pilot, and confirm that DNS, domain join, time, Group Policy, monitoring, and backup paths are not unintentionally isolated.

Roll out by a small cohort, then expand after reviewing logs and support reports. Use distinct GPOs or clear policy boundaries for server isolation, user endpoints, and special networks. Maintain a tested rollback that can be applied from an independent management path. A broad policy can cause a feedback loop if an endpoint needs domain services to obtain the policy that blocks its current traffic.

Diagnose negotiation and firewall failures

If the firewall drops traffic, identify whether the block came from the firewall filter, an IPsec requirement, an authentication failure, or an upstream device. Use Find-NetIPsecRule with the specific remote endpoint and port; inspect Get-NetFirewallRule and its associated filters; and compare the policy source on both peers. Review the Security and Microsoft-Windows-Windows Firewall With Advanced Security event channels according to the configured audit policy. A permitted TCP handshake without IPsec does not prove a protected connection, and an active SA does not prove the intended firewall authorization is in effect.

If main mode fails, check identity and phase 1 authentication. For Kerberos, inspect DNS, time, domain trust, computer account health, and SPNs. For certificates, inspect chain, EKU, subject identity, validity, private key access, and revocation reachability. For quick mode failures, compare phase 2 proposals, traffic selectors, endpoint scope, and encryption requirements. Verify that the traffic selectors on both hosts cover the same addresses and protocol/port tuple.

If only one direction fails, compare local and remote policy, profile, firewall direction, and whether the endpoint is behind NAT. If the application works but is unprotected, verify the connection-security rule is actually matching and inspect the negotiated SA. If the application fails after enforcement, use a controlled test with the same source, destination, user, and process rather than disabling the firewall globally. Preserve the failing connection evidence before changing policy.

Rollback and ongoing lifecycle

Rollback should be a known GPO unlink, policy disable, or documented alternate rule set, not a command improvised after remote access disappears. Keep a maintenance console or out-of-band path, test the emergency account, and ensure the security team can reach the pilot. For a rollback, verify that old sessions and SAs behave as expected and that weakening protection does not persist beyond the incident window.

Maintain an inventory of policy names, owners, peer groups, authentication methods, algorithms, allowed traffic, exceptions, expiration dates, and GPO links. Revalidate after device group changes, host renames, domain migrations, certificate rotation, firewall baseline updates, and Windows upgrades. Remove unused policies and secrets through change control. Review group membership and policy source regularly so that decommissioned or repurposed hosts do not retain trust they no longer need.

Acceptance requires matching firewall and IPsec scope; approved peer identity and algorithms; active, expected SAs; demonstrated application success; verified negative behavior; central event evidence; an independent recovery path; and operator signoff. Record the exact host and policy versions tested. Re-run the pilot after material endpoint, network, or cryptographic changes.

Related:

Sources: