AXUIElement on macOS: Build Reliable Accessibility Automation Clients

AXUIElement is the client side of macOS accessibility automation. It lets assistive applications query another process for semantic information about its interface and invoke supported accessibility actions. That is different from implementing accessibility in your own AppKit views: a custom accessibility element is a provider, while an AXUIElement client consumes the tree exported by an application. The distinction matters because remote UI trees are dynamic, permission-gated, and not a stable substitute for a documented application API.

Use the API for assistive technology, testing, or a clear user-authorized automation feature. Do not scrape another app’s pixels or accessibility tree to collect private content without an explicit purpose and consent. A robust client asks only for the attributes and actions it needs, handles target changes and messaging failures, and never assumes the other app will expose every role or notification.

Establish trust before querying

Accessibility access is controlled by macOS privacy settings. AXIsProcessTrusted() answers whether the current process is trusted; AXIsProcessTrustedWithOptions can request that the system prompt, but the user remains in control. Check trust before constructing a large automation workflow and give a concise explanation of why accessibility access is needed. A menu-bar utility that reads and modifies interface elements should disclose those capabilities clearly.

An untrusted process should not spin on the trust check, repeatedly show dialogs, or treat a missing result as an empty UI tree. Model permission as a distinct application state. Provide a user-triggered retry after the user changes System Settings, and be prepared for trust to be revoked while the process is running. Do not attempt to alter TCC databases or use root privileges to bypass the setting.

Start with a specific target process. AXUIElementCreateApplication(pid) creates an application-level element, while AXUIElementCreateSystemWide() refers to the system-wide accessibility object. Prefer an explicit process identifier obtained from a verified running application over a broad system-wide search. Verify that the process is still the expected product before reading or acting on its interface.

import ApplicationServices

func applicationElement(pid: pid_t) throws -> AXUIElement {
    guard AXIsProcessTrusted() else {
        throw AutomationError.accessibilityPermissionRequired
    }
    return AXUIElementCreateApplication(pid)
}

enum AutomationError: Error {
    case accessibilityPermissionRequired
    case attributeUnavailable
}

This code only checks the current trust state and creates a reference. It does not prove that a target responds or that the requested command is safe. Keep those checks at the operation boundary.

Query attributes defensively

Accessibility attributes are typed values, not a promise that every element has every property. The root may expose a focused window; a button may expose a title and enabled state; a text field may expose a value. Query only what your feature requires, check each AXError, and validate the returned Core Foundation type before casting it. kAXErrorAttributeUnsupported and kAXErrorNoValue should be handled as normal capability differences rather than treated as a corrupt system.

Remote messaging can also fail. kAXErrorCannotComplete can indicate that the target is busy, unresponsive, or blocked behind interaction; kAXErrorInvalidUIElement means a previously acquired reference is no longer usable. Reacquire the tree from the application root after a structural transition instead of retaining element handles indefinitely. Never retry a non-idempotent action such as pressing a “Delete” button without first checking whether the first attempt already completed.

A small helper can make error handling explicit:

import ApplicationServices
import CoreFoundation
import Foundation

enum AttributeError: Error {
    case unavailable
}

func copyStringAttribute(_ element: AXUIElement, _ name: CFString) throws -> String {
    var value: CFTypeRef?
    let error = AXUIElementCopyAttributeValue(element, name, &value)
    guard error == .success, let value else {
        throw AttributeError.unavailable
    }
    guard CFGetTypeID(value) == CFStringGetTypeID() else {
        throw AttributeError.unavailable
    }
    return value as! String
}

In production, preserve the underlying AXError in a structured diagnostic and map it to a user-meaningful state. Avoid logging the returned string: it may be a document name, email address, account balance, or another sensitive value. Log the target bundle ID, requested attribute name, outcome category, and latency instead.

Treat element identity as transient

The accessibility hierarchy is a semantic view of an application’s current state. A window can close, a table can reload, or a virtualized cell can be recycled after a query. A role and title are useful for locating a control but are not globally unique identifiers. When possible, combine a stable target process, container role, relationship, and application-specific identifier. Revalidate the element immediately before an action and verify the postcondition after it.

Avoid indexing into arrays and assuming that row number remains associated with a record. Sorting, filtering, localization, and asynchronous refreshes can reorder items. Prefer a stable description such as a unique accessibility identifier that the target app publishes. If the target does not expose enough semantic information, report the limitation rather than clicking at an inferred screen coordinate.

Parameterized attributes can expose richer navigation, including text ranges and children. They can also be expensive. Use bounded traversal: cap depth and node count, stop after the target is found, and impose a deadline for each remote request. Do not recursively enumerate an entire application’s interface every time a user presses a hotkey. Cache only data whose freshness you can validate, and clear it when the focused window or target process changes.

Observe changes without missing transitions

AXObserver can receive notifications from an application, but notification support varies by element and target. Register for the smallest set of changes needed, such as focused-window or value changes, and add the observer’s run-loop source to a run loop you own. Retain the observer and callback context while it is registered. On teardown, remove registrations and the run-loop source before releasing those objects.

Notifications are signals to re-query, not complete event-sourced truth. The target can change between a notification and your next request, and notifications may not cover every mutation. Build the callback to mark relevant state dirty and schedule a bounded refresh on your own serial queue. Avoid performing a long tree walk inside the callback. If the target restarts, becomes inaccessible, or emits an unsupported notification error, discard stale references and reconstruct state from the root.

Test races deliberately: close a window while a notification is pending, switch the focused app during traversal, and terminate the target after obtaining an element. The client should recover to a clear “target unavailable” state without dereferencing stale Core Foundation objects, repeatedly prompting, or sending actions to the wrong process.

Invoke actions as transactions

Query AXUIElementCopyActionNames before using an action, then call AXUIElementPerformAction only when the action is present and the element still satisfies the expected role and state. The advertised list is the target’s current capability declaration, not proof that the action will succeed. The user may have closed a dialog or changed selection after the query.

Before a destructive action, give the user an opportunity to review the intended target. A client that types text into a focused field must verify focus, field role, and permitted target application first; otherwise a password or private note may be entered into the wrong window. For reversible changes, preserve enough state to support undo. For irreversible actions, require a direct user gesture and confirm the result using a fresh query.

Do not synthesize keyboard events as a universal fallback when accessibility actions are unavailable. Keyboard injection has separate privacy controls and can activate unrelated controls if focus changes. Likewise, coordinate-based clicks are fragile under multiple displays, scaling, window movement, and overlays. Prefer semantic actions; if no supported action exists, explain the limitation and leave control with the user.

Test the actual client contract

Test with permission unset, granted, denied, and revoked. Exercise different target application versions, window roles, localization, display scaling, slow responses, modal sheets, and process relaunch. Include targets that expose the requested attribute as an unexpected type or do not implement the action. The correct behavior is an explicit unsupported or unavailable result, not a crash or a guessed action.

Separate provider and client QA. For your own AppKit controls, inspect the tree with Accessibility Inspector and verify role, label, value, frame, and actions. For the automation client, use a controlled fixture app whose accessibility tree and state changes are known. This allows tests to assert that a query returns the expected stable identifier and that a successful action is reflected by a new state.

Finally, minimize collection. Accessibility can reveal a broad amount of on-screen information. Keep data in memory only while needed, do not upload trees or snapshots, and make any telemetry aggregate and opt-in. A well-engineered AX client is an assistive interface with an explicit trust boundary, not a hidden general-purpose screen scraper.

Related:

Sources: