Skip to content
Haiku OSDeep Dive Published Updated 8 min readViews unavailable

Haiku R1/beta6 O_CLOFORK: Descriptor Inheritance at fork()

Use Haiku R1/beta6 O_CLOFORK to control descriptor inheritance at fork(), distinguish it from close-on-exec, and test its concurrency limits.

Haiku R1/beta6 lists O_CLOFORK among its POSIX compatibility additions. The flag marks a file descriptor to be omitted from the child descriptor table created by fork(). That is different from O_CLOEXEC, which requests closure during exec. The distinction matters in supervisors and multithreaded services: a child that retains an unrelated pipe, file, or socket can keep a resource alive, delay end-of-file, or gain access the child was not meant to receive.

This guide is specifically about the Haiku R1/beta6 POSIX descriptor interface and its current source implementation, not a portable recipe for every Unix. The release notes describe Haiku as beta-quality. In addition, the source-level audit below finds a possible gap between POSIX’s creation-time close-on-fork protection and the beta6 file-descriptor setup path. Treat that as a concrete verification item, not as a proven runtime exploit or a claim about later Haiku revisions.

Separate fork inheritance from exec inheritance

FD_CLOFORK is a per-descriptor flag. O_CLOFORK asks an operation that creates a descriptor, such as open(), to set that flag as part of descriptor creation. Haiku’s POSIX header also defines F_GETFD, F_SETFD, and F_DUPFD_CLOFORK. With F_GETFD, inspect both FD_CLOEXEC and FD_CLOFORK; with F_SETFD, pass the complete flag value you intend to retain rather than assuming it only adds a bit.

The two policies act at different transitions:

  • FD_CLOFORK prevents that descriptor from being copied into a child created by fork(). It does not close the parent’s descriptor.
  • FD_CLOEXEC closes the descriptor when the process image is replaced by an exec operation. By itself, it does not keep a descriptor out of a child between fork() and exec().
  • Neither flag is an access-control boundary. A child can still open a pathname for which it has permission, and threads in the same team continue to share the process descriptor table.

POSIX.1-2024 describes the forked process as receiving its own descriptor table except for descriptors marked FD_CLOFORK. The child descriptors that are copied refer to the same open file descriptions as their parent counterparts. This affects more than confidentiality: an inherited pipe writer can keep a reader from seeing end-of-file, and an inherited file reference can keep an object busy after the parent believes it has shut down.

Use the creation flag for a descriptor that should stay private

The following small program checks a serial, non-concurrent case on Haiku R1/beta6. Pass the path of a readable file. It requests both close-on-fork and close-on-exec so the example can verify that they are separate flags; the child tests the fork behavior before calling any exec function. Each F_GETFD call supplies (size_t)0: POSIX makes the third argument optional for this command, but the pinned beta6 libroot implementation unconditionally reads a size_t vararg. This typed dummy is an implementation-specific workaround for that source-level defect, not a portable fcntl() convention.

#include <errno.h>
#include <fcntl.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <unistd.h>

int
main(int argc, char *argv[])
{
    if (argc != 2) {
        fprintf(stderr, "usage: %s readable-file\n", argv[0]);
        return EXIT_FAILURE;
    }

    int fd = open(argv[1], O_RDONLY | O_CLOEXEC | O_CLOFORK);
    if (fd < 0) {
        perror("open");
        return EXIT_FAILURE;
    }

    int flags = fcntl(fd, F_GETFD, (size_t)0);
    if (flags < 0) {
        perror("fcntl(F_GETFD)");
        close(fd);
        return EXIT_FAILURE;
    }
    if ((flags & (FD_CLOEXEC | FD_CLOFORK))
            != (FD_CLOEXEC | FD_CLOFORK)) {
        fprintf(stderr, "open did not set both requested descriptor flags\n");
        close(fd);
        return EXIT_FAILURE;
    }

    pid_t child = fork();
    if (child < 0) {
        perror("fork");
        close(fd);
        return EXIT_FAILURE;
    }
    if (child == 0) {
        errno = 0;
        int childFlags = fcntl(fd, F_GETFD, (size_t)0);
        _exit(childFlags == -1 && errno == EBADF ? EXIT_SUCCESS : EXIT_FAILURE);
    }

    int status;
    pid_t waited;
    do {
        waited = waitpid(child, &status, 0);
    } while (waited < 0 && errno == EINTR);

    if (waited < 0) {
        perror("waitpid");
        close(fd);
        return EXIT_FAILURE;
    }

    int parentFlags = fcntl(fd, F_GETFD, (size_t)0);
    int ok = WIFEXITED(status) && WEXITSTATUS(status) == EXIT_SUCCESS
        && parentFlags >= 0 && (parentFlags & FD_CLOFORK) != 0;
    if (close(fd) != 0) {
        perror("close");
        return EXIT_FAILURE;
    }
    return ok ? EXIT_SUCCESS : EXIT_FAILURE;
}

In the expected serial result, the child receives EBADF for the descriptor while the parent still has it open and marked FD_CLOFORK. A failure can mean the flag was not applied, the running image does not implement the expected release behavior, or the test ran on a different platform. Record the exact Haiku release/build and rerun on the target image. This check does not exercise simultaneous descriptor creation and fork(); that limitation is important for beta6.

If the child must intentionally use one endpoint of a pipe or socket pair, do not mark every endpoint close-on-fork and then assume communication will still work. Choose the descriptors the child is meant to receive, preserve those deliberately, and test end-of-file and shutdown behavior. Haiku beta6 also advertises close-on-fork socket and received-descriptor flags, but their protocol details are outside this article’s open()-and-fork() scope.

A beta6 source-audit caveat for concurrent creation

POSIX specifies creation-time close-on-fork flags as a way to avoid a race in which one thread creates a descriptor and another thread forks before a later fcntl(F_SETFD) call marks it. Haiku R1/beta6 declares the flag and its kernel fork path omits descriptors whose FD_CLOFORK bit is already set. However, the pinned beta6 source path deserves scrutiny before using this as a hard no-leak guarantee in a multithreaded process.

In get_new_fd(), Haiku calls new_fd() to insert the descriptor into the team’s file-descriptor table. That helper acquires and releases the I/O-context lock. Only after it returns does get_new_fd() acquire the lock again and set the close-on-fork bit from O_CLOFORK. The fork path calls vfs_new_io_context(parent, false), which takes the parent’s I/O-context lock and tests that bit while copying descriptors. A fork on another thread could therefore take its snapshot after insertion but before the flag-setting critical section. The same publish-then-mark ordering is visible in the pinned F_DUPFD_CLOFORK path: it obtains the new descriptor before separately setting its close-on-fork bit. In the pinned pipe2(O_CLOFORK) path, both endpoints are opened through open_vnode() and get_new_fd(), so that path also reaches the same window. These are source-level findings for the cited beta6 revision, not proof that a particular binary reproduces the interleaving or that later revisions retain it. Other descriptor-receive paths need their own source trace; do not assume they share this exact implementation.

Until the exact target build is tested or the upstream implementation is confirmed fixed, do not rely on O_CLOFORK alone as a race-free security boundary against concurrent fork activity. If every descriptor-creation path and every fork in the application is under your control, a process-wide gate around both operations can serialize them as a temporary mitigation. A library that forks without using that gate defeats the mitigation, so it is not a substitute for an implementation-level fix. Prefer a launch design that does not expose sensitive descriptors to uncontrolled process creation, and report a reproducible issue with the release/build, test program, and source revision.

Validate the exact boundary in a disposable system

Start with the serial program above to confirm the basic expected behavior. A userspace test that learns the descriptor number only after open() returns cannot inspect the insert-before-mark interval: by then the flag-setting code has already run. To test that exact interleaving in a disposable VM, use an instrumented test kernel that pauses after publishing the descriptor but before setting the bit, records the assigned slot for the harness, and lets another thread fork and inspect that slot. Keep the probe limited to a harmless test file; children must not read or modify it. A child that sees the descriptor is evidence to investigate, while a stress test without a barrier-controlled observation that never reproduces the race is not proof that the interleaving is impossible.

Repeat the source trace for each descriptor-producing path your application uses. In the pinned beta6 tree, F_DUPFD_CLOFORK has its own insert-then-mark sequence, and pipe2(O_CLOFORK) reaches get_new_fd() for its endpoints. The pinned AF_UNIX SCM_RIGHTS receive path is also visible: unix_process_ancillary_data() calls new_fd() for each received descriptor, then later obtains a WriteLocker and sets FD_CLOFORK when MSG_CMSG_CLOFORK was requested. That is the same source-level publish-then-mark shape for this path, not proof of runtime reproduction; other descriptor-producing providers and protocols still need their own trace. Also test normal inheritance intentionally: a descriptor without FD_CLOFORK should be present in a forked child, while the parent must retain its own copy in both cases. Do not use production secrets or sensitive resources for the test.

Capture three outcomes separately: whether open() succeeded, what F_GETFD reports in the parent, and whether the child can inspect the descriptor before exec. That separation catches an application that merely assumed a flag was set, as well as an implementation or timing issue. A successful serial test proves the ordinary flag behavior for that case; it does not validate race-free creation under concurrent forks.

The practical rule is to treat close-on-fork as an explicit inheritance policy, not as a sandbox. Haiku R1/beta6 exposes useful POSIX.1-2024 interfaces and a testable child-table behavior, but production code must verify the installed implementation’s concurrency guarantee before depending on it to prevent descriptor leakage.

Related:

Sources:

Comments