Linux ARP and Neighbor Tables: Diagnose Resolution Without Blind Flushing

An application timeout does not prove that ARP is broken. Before Linux can send an IP packet to a directly attached destination, it needs a link-layer address for the next hop. For IPv4, that mapping is normally learned with Address Resolution Protocol (ARP); IPv6 uses Neighbor Discovery (ND). Linux exposes both through its neighbor subsystem and the ip neigh command. The table is scoped to a network namespace and link context, so an entry on the host or on the wrong interface may have no bearing on the packet that failed.

The production-grade workflow is to identify the route and next hop first, inspect the matching neighbor entry, observe resolution traffic on the correct Layer-2 segment, and only then change configuration. Flushing a table can briefly hide stale state, but it can also trigger a burst of broadcasts, erase useful evidence, and fail to address a VLAN, bridge, duplicate-address, or route-selection problem.

Resolve the actual next hop

Applications send to an IP destination. The routing decision determines whether that destination is on-link or which gateway receives the packet. Neighbor resolution is for that next hop, not necessarily the remote server. Ask the kernel for the route using the same source, mark, VRF, or namespace context as the affected flow when those policies are in use:

DEST=198.51.100.40
ip route get "$DEST"

The address above is from a documentation range. Replace it with the affected destination. A result such as via 192.0.2.1 dev ens192 src 192.0.2.25 means the local host resolves the gateway on ens192; it does not ARP for the remote server. If the route has no via, the destination is treated as on-link for that route and is itself the neighbor lookup target. Inspect all rules and tables if this result differs from the intended path:

ip rule show
ip route show table all
ip -details link show

Do not infer that a host is directly attached merely because two IP addresses look similar. Prefix length, policy rules, VRF membership, bridges, VLAN devices, and network namespaces define the actual path. In a container, inspect from the container’s network namespace and then walk the veth/bridge path toward the physical uplink. A host-side ip neigh listing cannot substitute for that view.

Read the neighbor entry in context

Use the device from ip route get to narrow the display and query one target without changing state:

NEXT_HOP=192.0.2.1
IFACE=ens192
ip -s neigh show to "$NEXT_HOP" dev "$IFACE"
ip neigh get "$NEXT_HOP" dev "$IFACE"

ip neigh reports protocol-to-link-layer bindings and Neighbor Unreachability Detection (NUD) state. IPv4 entries are commonly called the ARP table. A state is a point in a lifecycle, not a universal health verdict:

REACHABLE means the kernel currently considers the neighbor reachable according to its confirmation state and timer.
STALE is a usable mapping that has not recently been confirmed. It is not, by itself, an outage; use can trigger validation.
DELAY postpones probing while waiting for upper-layer confirmation; PROBE means validation probes are being sent.
INCOMPLETE means address resolution has not completed, while FAILED means the attempt exhausted its probes without succeeding.
PERMANENT and some externally managed entries have different lifecycle rules from dynamically learned entries; do not overwrite them as a generic repair.

The exact output depends on kernel and iproute2 versions, address family, and userspace controllers. Compare repeated snapshots with timestamps rather than treating a single STALE line as proof. ip monitor neigh can record state changes while you reproduce the failure:

ip -ts monitor neigh

If the entry changes between INCOMPLETE and FAILED, resolution is not receiving an accepted answer in the context being observed. That narrows the question; it does not yet tell whether the request left the right interface, whether the intended peer saw it, or whether a reply returned.

Observe requests and replies on the correct link

Capture the protocol exchange on the routed interface. For an IPv4 ARP test, record Ethernet headers as well as the ARP fields:

sudo tcpdump -nn -e -i "$IFACE" arp

In a controlled change window, generate a single targeted lookup from the same namespace and interface context, then correlate packet timestamps with ip monitor neigh. Avoid a broad ip neigh flush as a diagnostic stimulus. A capture showing repeated “who-has” requests without a reply indicates a failure somewhere between the selected interface and the expected responder. A reply visible on the wire but not reflected in the table points toward a different local context or policy, competing responses, or a capture taken on a path that is not the one the kernel uses.

Check the full Layer-2 path: switch port and VLAN, tagged/untagged expectations, bridge membership, bond/team state, Wi-Fi client isolation, hypervisor port groups, and any proxy-ARP design. If a gateway is expected, verify its address and MAC from an independent trusted source such as the network inventory or switch/router telemetry. A packet capture alone cannot establish that a responder is authorized.

For a suspected duplicate IPv4 address, correlate ARP probes and announcements with DHCP lease records, static-address inventory, and switch MAC learning. RFC 5227 defines IPv4 address conflict detection behavior, but the presence or absence of one observed probe does not prove that every endpoint follows it. Do not “fix” the symptom by pinning a permanent neighbor entry: that masks address ownership conflicts and can make failover or legitimate MAC changes fail.

Separate route, link, and resolver failures

Use a decision sequence that keeps different layers distinct:

ip route get DEST returns an unexpected interface or gateway: investigate policy routing, source selection, VRF, route metrics, or the manager that owns the route.
The route is correct but the next-hop entry is INCOMPLETE/FAILED: capture ARP or ND on that exact link and verify the expected L2 peer and VLAN.
The neighbor entry resolves, but traffic still fails: inspect firewall/ACL policy, MTU, transport behavior, service health, and the return route. A valid MAC mapping proves only next-hop resolution.
The failure exists only inside a container or service: compare namespace-specific routes and neighbor entries, then follow the veth/bridge path. Do not modify the host’s table until the failing namespace is known.

For IPv6, inspect the corresponding entry with ip -6 neigh show dev "$IFACE" and capture Neighbor Solicitation/Advertisement messages with sudo tcpdump -nn -e -i "$IFACE" icmp6. IPv6 neighbor resolution is part of ICMPv6 Neighbor Discovery, not ARP. Router discovery, Duplicate Address Detection, and next-hop reachability are related control-plane functions but have distinct messages and failure conditions. A link-local IPv6 address also requires the correct interface scope for tests and lookups.

Understand neighbor garbage collection before tuning

Large virtualized or directly connected networks can approach neighbor-table thresholds. Kernel documentation describes gc_thresh1 as a minimum retained-entry threshold, gc_thresh2 as the point where collection becomes more aggressive, and gc_thresh3 as the maximum number of non-permanent entries; the documented default values are 128, 512, and 1024 for the default table. gc_stale_time affects when unused entries become eligible for garbage collection. These are kernel defaults, not a sizing recommendation for every distribution or topology.

Read effective values in the relevant network namespace and account for per-interface overrides before proposing a change:

for key in gc_thresh1 gc_thresh2 gc_thresh3 gc_stale_time; do
    path="/proc/sys/net/ipv4/neigh/default/$key"
    if [ -r "$path" ]; then
        printf '%s=' "$key"
        cat "$path"
    fi
done

The example reads the IPv4 default neighbor parameters in the current namespace; it does not enumerate every interface-specific override or IPv6 setting. Gather actual neighbor population, churn, dropped/failed resolutions, and interface/namespace scale before changing thresholds. If a justified adjustment is needed, record the current value, scope, owner, persistence mechanism, rollback, and a load test. Raising a limit without understanding churn can conceal a topology leak or an unintended broadcast domain.

Recover without destroying evidence

Prefer correcting the identified fault: restore the intended VLAN, route, peer address, bridge port, or manager-owned configuration. If a confirmed stale dynamic entry must be refreshed, first capture the existing state and define a narrow target. ip neigh flush deletes matching entries and causes future traffic to resolve them again; it is state-changing and can affect every flow that shares the selected scope. Never use an unqualified flush on a production node as a first-line repair.

After the correction, verify the route, the neighbor mapping and its transition, packet exchange, and the actual application transaction. Repeat after a link flap or planned failover if that is part of the incident. Preserve the route output, namespace/interface identity, timestamps, capture, and before/after entry so the result is reproducible.

Production acceptance checklist

Confirm the affected process’s network namespace and the route lookup for its real destination and source context.
Record the interface, next hop, address family, VRF/bridge/VLAN path, and network manager that owns the link.
Collect a timestamped neighbor snapshot and event stream before changing state.
Capture only the relevant ARP or ICMPv6 ND exchange on the correct L2 boundary.
Verify peer identity and address ownership with independent operational records.
Change only the configuration implicated by evidence; retain rollback and out-of-band access for remote changes.
Prove DNS, transport, service response, return path, and failover behavior separately from neighbor resolution.
Document the exact kernel/iproute2 versions and compare behavior in the same namespace after reboot or interface recreation when persistence matters.

“The ping worked after flushing ARP” is not an acceptance test. A robust resolution explains which next hop the route selected, why the old mapping was wrong or absent, what fixed the underlying condition, and how the service behaves through the next normal lifecycle event.

Related:

Sources: