Linux SO_TXTIME and ETF: Schedule Packets at the Egress Queue

SO_TIMESTAMPING tells an application when selected packet events happened. SO_TXTIME answers a different question: when should a packet be eligible to leave the Linux traffic-control path? Paired with the Earliest TxTime First (etf) queueing discipline, it lets a sender attach an intended transmission time to each packet and lets the qdisc order, delay, or reject packets according to that schedule.

This is a scheduling interface, not a latency guarantee. CPU scheduling, qdisc placement, driver behavior, clock synchronization, queue contention, and NIC support all affect the result. A successful sendmsg() means the kernel accepted the datagram; it does not prove that a frame appeared on the wire at its requested time.

Keep the clock and packet contracts aligned

There are three pieces to the contract. The application supplies a per-packet transmit time in ETF’s clock domain. In the Linux v6.12 implementation cited here, ETF requires CLOCK_TAI; it rejects dynamic clock IDs and other clock IDs. The socket’s SO_TXTIME option declares that reference clock and policy flags, and the ETF qdisc must use the same clock and a matching deadline-mode policy. With socket checks enabled, a mismatch is not a small offset to compensate with delta: ETF drops packets if the socket clock or deadline mode differs from the qdisc configuration. The kernel also exposes skip_sock_check, which bypasses this validation rather than reconciling mismatched clocks or policies; the examples below do not request it.

The timestamp is an absolute nanosecond value in CLOCK_TAI for this setup, not a relative delay. Do not compute it from CLOCK_MONOTONIC and hand it to a socket configured for CLOCK_TAI, or compare it directly with a NIC’s PHC timestamp. Verify the relationship between the system’s TAI clock and any peer or hardware clock, and keep the chosen domain consistent through scheduling and measurement.

The following excerpt shows the API boundary. fd, iov, and next_tai_txtime_ns are provided by the surrounding sender; the timestamp must be calculated in CLOCK_TAI because that is the clock selected below. Error handling and socket creation are abbreviated, but the control-message fields follow the Linux kernel self-test.

#define _GNU_SOURCE
#include <linux/net_tstamp.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/uio.h>
#include <time.h>

int configure_txtime(int fd)
{
    struct sock_txtime policy = {
        .clockid = CLOCK_TAI,
        .flags = SOF_TXTIME_REPORT_ERRORS,
    };
    if (setsockopt(fd, SOL_SOCKET, SO_TXTIME, &policy, sizeof(policy)) == -1) {
        perror("setsockopt SO_TXTIME");
        return -1;
    }
    return 0;
}

int send_scheduled(int fd, struct iovec *iov, uint64_t next_tai_txtime_ns)
{
    uint64_t txtime_ns = next_tai_txtime_ns;
    char control[CMSG_SPACE(sizeof(txtime_ns))] = {0};
    struct msghdr msg = {
        .msg_iov = iov,
        .msg_iovlen = 1,
        .msg_control = control,
        .msg_controllen = sizeof(control),
    };
    struct cmsghdr *cmsg = CMSG_FIRSTHDR(&msg);
    cmsg->cmsg_level = SOL_SOCKET;
    cmsg->cmsg_type = SCM_TXTIME;
    cmsg->cmsg_len = CMSG_LEN(sizeof(txtime_ns));
    memcpy(CMSG_DATA(cmsg), &txtime_ns, sizeof(txtime_ns));

    if (sendmsg(fd, &msg, 0) == -1) {
        perror("sendmsg");
        return -1;
    }
    return 0;
}

The example omits socket creation and message payload setup; fd must be a socket configured for time-based transmission, and the caller must compute the absolute transmit time in CLOCK_TAI. In production, set the socket policy once during initialization rather than on every send, initialize each message deliberately, ensure the transmit time is still in the future, and drain the socket error queue when reporting is enabled. Linux’s self-test uses SO_TXTIME with SCM_TXTIME and reads transmit errors from that queue. The error queue is part of observability: without it, a late or invalid packet can look like a normal successful send at the application boundary.

Place ETF under the traffic class that will transmit the flow

ETF is normally attached below a classifying qdisc such as mqprio; it does not decide by itself which traffic class should carry every flow. First inspect the existing hierarchy and the NIC’s clock/timestamp context:

tc -d qdisc show dev eno1
tc -s qdisc show dev eno1
ethtool -T eno1
ethtool -i eno1

ethtool -T reports timestamping capabilities and any exposed PTP Hardware Clock; it does not by itself prove that the NIC supports ETF launch-time offload. ethtool -i identifies the driver and firmware to check against their documentation. The parent handle in the next command is only an example. Use it only if the intended mqprio configuration actually created that class and the interface supports the chosen setup:

tc qdisc replace dev eno1 parent 100:1 etf \
    clockid CLOCK_TAI delta 300000

The 300,000-nanosecond delta means ETF schedules its wake-up before the requested transmit time, giving the kernel and driver lead time to process the packet. It is a tuning value, not a universal constant. Measure under the intended CPU and network load, then adjust based on observed scheduling error and missed deadlines. Replacing a qdisc changes live packet handling; test in a network namespace or maintenance window, preserve the prior configuration, and prepare a rollback before applying it to a production interface.

If the driver and NIC implement time-based launch arbitration, the offload option asks ETF to configure it:

tc qdisc replace dev eno1 parent 100:1 etf \
    clockid CLOCK_TAI delta 300000 offload

This is not portable proof that hardware scheduling is active. Capability is device-, driver-, and firmware-dependent. Confirm that the command succeeds, inspect qdisc statistics and driver counters, and measure the actual transmit path with appropriately synchronized capture hardware. A software qdisc experiment in a pair of network namespaces validates application and kernel behavior, not a NIC’s physical launch-time accuracy.

Strict transmit times and deadlines are different policies

By default, ETF applies strict transmit-time semantics: a packet whose time is already in the past, or that expires while waiting in the qdisc, is dropped. deadline_mode changes the interpretation from a precise scheduled instant to a latest acceptable time; when the qdisc dequeues a packet, it changes the packet’s transmit time to “now.” That can be appropriate when lateness is acceptable but missing the deadline is not. It is not a way to make late packets meet an earlier schedule.

Choose the mode from the protocol’s requirement. A TSN stream that must leave at a planned instant and an interactive packet that is merely useful before a cutoff do not share the same policy. The socket’s SOF_TXTIME_DEADLINE_MODE flag and the qdisc option must describe a compatible contract. Keep error reporting enabled while validating the behavior, and read the kernel’s extended error details instead of inferring the drop reason from a timeout alone.

Validate scheduling without confusing it with timestamping

Start with a disposable topology and numbered UDP payloads. Run the sender and receiver in separate network namespaces connected by a virtual Ethernet pair, schedule a controlled series of future timestamps, and record requested times, receive order, losses, and error-queue reports. Repeat with past timestamps and deliberate congestion to exercise miss handling. The kernel self-test is a useful reference for the socket-option and control-message setup, but it is a test fixture rather than evidence about the application’s deadline budget.

For a physical link, repeat with the exact NIC, driver, firmware, queue mapping, qdisc hierarchy, and clock source intended for deployment. Record whether timestamps and scheduled times use a system clock, TAI, or a PHC. Compare requested time with a synchronized hardware observation when possible, and report distributions and tail errors instead of presenting the smallest sample as a guarantee. Also verify that normal traffic remains classified into the intended queues after the qdisc change.

ETF is useful when the application can produce meaningful deadlines and the platform can honor the selected timing contract. It is not a substitute for clock discipline, traffic-class design, queue budgeting, or end-to-end measurement. Treat every supported NIC configuration as a separate integration target, retain a conventional fallback path, and fail visibly when the kernel or driver cannot provide the requested behavior.

Related:

Sources:

Keep the clock and packet contracts aligned

Place ETF under the traffic class that will transmit the flow

Strict transmit times and deadlines are different policies

Validate scheduling without confusing it with timestamping

Comments