Skip to content
SRE & DevOpsDeep Dive Published Updated 6 min readViews unavailable

Cloudflare Durable Object Alarms: At-Least-Once Scheduling and Retry-Safe Work

Build durable per-object schedules around one alarm, persisted work records, replay-safe effects, and bounded recovery rather than timer assumptions.

A Durable Object alarm is a persistent wake-up signal for one object identity, not an in-memory timer and not a general-purpose cron expression. Cloudflare documents at-least-once execution: an uncaught exception can cause the alarm() handler to run again, with exponential backoff and a bounded automatic retry count. A handler must therefore tolerate replay even when the work it initiated may already have reached an external system.

An alarm is also singular: each Durable Object can have one scheduled alarm at a time. That does not limit the object to one business event. Store many due times as object state, use the alarm as the next wake-up point, process due records, and move the alarm to the earliest remaining deadline. This pattern fits per-customer reminders, retry queues, lease cleanup, and batching when the state truly belongs to one object key. It is not a substitute for Cron Triggers when the schedule is global and regular, nor a distributed queue across unrelated object identities.

Store work as durable state, not as a timer callback

An in-memory setTimeout() disappears when an isolate is evicted or replaced. An alarm belongs to the Durable Object storage lifecycle and wakes the object later. Keep the work description in storage as well: a stable event ID, due time, state, and any idempotency key needed by a downstream service. The alarm says “look for due work”; the stored record says what remains to be done.

When inserting or rescheduling an event, compare it with the object’s current alarm. If no alarm exists, or the new event is earlier, set the alarm to the new due time. A later event should not move the next wake-up later and strand work that is already due. The following API-shaped fragment assumes the object has a storage binding and a saveEvent(event) helper that persists a record with id and dueAt fields and handles errors in the application’s chosen storage model.

async function scheduleEvent(event: { id: string; dueAt: number }) {
  if (!Number.isFinite(event.dueAt) || event.dueAt < 0) {
    throw new TypeError("dueAt must be a non-negative epoch time in milliseconds");
  }

  await saveEvent(event);
  const alarmAt = await storage.getAlarm();
  if (alarmAt === null || event.dueAt < alarmAt) {
    await storage.setAlarm(event.dueAt);
  }
}

This fragment illustrates ordering, not a universal transaction guarantee. If the event write succeeds but setting the alarm fails, a later request or reconciliation pass must notice the unscheduled record. If the alarm is set first but the event write fails, the handler may wake to find no matching work. Make either sequence safe and test failures between the two operations. Durable Object storage supports transactions, but whether an application can place a particular alarm operation and its data mutation in one transaction depends on the documented storage API and backend in use; do not assume atomicity from two adjacent await statements.

Process due records and schedule the next wake-up

At the start of alarm(), read the current time once and select records whose persisted due time is at or before that instant. Process a bounded batch so one large backlog does not turn a single invocation into an unbounded operation. For each record, preserve its identity through the external request. If the destination supports idempotency keys, send the stable event ID; otherwise store enough state to reconcile an ambiguous result before retrying.

Do not delete a work record merely because a request was sent. A network timeout can happen after the remote system commits. Conversely, do not mark it complete before the effect is confirmed. Model states such as pending, attempting, complete, and needs-reconciliation only if their transitions have precise meanings and can recover after object restart. For a large batch, persist progress after each bounded unit and schedule another near-term alarm before returning. That turns backlog processing into resumable work rather than relying on one handler invocation to finish everything.

After processing, calculate the earliest remaining due time and set that as the next alarm. For recurring jobs, advance from the job’s intended schedule rather than repeatedly using “current time plus interval” if drift matters. Decide explicitly whether a missed interval should be skipped, coalesced, or replayed; that is product policy, not an automatic alarm guarantee. If work fails transiently, either let an uncaught exception use Cloudflare’s bounded retry behavior or catch the error and persist a retry deadline before scheduling another alarm. The latter is necessary for application-controlled backoff and recovery after the platform’s finite automatic retries are exhausted.

Understand retry state and cancellation boundaries

Cloudflare’s alarm callback provides retry information such as retryCount and isRetry. Use it for observability and carefully bounded policy, not as the only source of job state. The retry count describes the current alarm attempt; the persisted event record is what survives later requests, object restarts, and application-level retry schedules.

deleteAlarm() removes a scheduled alarm but does not cancel a handler already executing. A cancel operation should therefore update durable job state first, and the handler should re-check that state immediately before an expensive or externally visible action. Even that check has a time-of-check boundary: an external request already in flight may not be retractable. If the operation is financially or operationally consequential, use a destination-side idempotency or cancellation protocol rather than treating alarm deletion as rollback.

Avoid scheduling from a constructor without checking whether an alarm already exists. Cloudflare documents that a constructor can run when an object wakes before its existing alarm() callback; a constructor that unconditionally sets a new time can overwrite that pending schedule. Keep constructor work limited to safe initialization and perform explicit reconciliation in a request or alarm path.

Operational example: an idempotent reminder queue

For a per-account reminder object, use the account’s stable object name, persist reminder IDs and due times, and set the alarm to the earliest pending reminder. When the alarm fires:

  1. Read a snapshot of due reminder IDs, ordered by due time and ID for deterministic diagnostics.
  2. Re-read each record before sending, because a user may have canceled or edited it after the snapshot.
  3. Send a stable reminder ID to the delivery service as its idempotency key.
  4. Persist delivery success or a classified retry time; do not translate authentication failure, permanent invalid-recipient responses, and transient service errors into the same retry loop.
  5. Re-read the earliest pending due time and schedule it, or clear the alarm only after proving no pending work remains.

This is a design example, not a claim that an email provider supports a particular idempotency header. Verify that contract with the provider. If it has no deduplication mechanism, document the duplicate-delivery window and prefer a reconciliation query when available. Do not store a sensitive bearer token in an alarm timestamp, object name, log line, or unprotected event payload.

Instrument object identity in a privacy-safe form, event ID, scheduled time, retry count, handler duration, backlog age, and terminal outcome. Alert on old pending records and repeated retry exhaustion, not on every expected wake-up. Load-test a downstream outage, an exception after a successful remote commit, a cancel racing the handler, a newly inserted earlier event, and an object with many due records. Verify after each case that work is neither silently stranded nor duplicated beyond the documented delivery contract.

Durable Object alarms provide a useful per-object scheduling primitive. Production reliability comes from the surrounding state machine: persisted work, idempotent effects, bounded processing, explicit retry policy, and a recovery path when a handler or dependency fails.

Related:

Sources:

Comments