Skip to content
SRE & DevOpsDeep Dive Published Updated 8 min readViews unavailable

Amazon SQS FIFO Queues in Production: Ordering, Deduplication, and Failure Recovery

Design SQS FIFO ordering groups, bounded deduplication, visibility timeouts, and DLQ recovery while keeping business effects replay-safe.

An Amazon SQS FIFO queue is useful when a workflow needs ordered messages, but its guarantee is narrower than “every business operation happens exactly once.” Ordering is scoped to a MessageGroupId; producer deduplication suppresses duplicate sends only within a documented interval; and a consumer can receive a message again when it fails to delete the message before visibility expires. A robust design treats those as separate controls and makes the application effect replay-safe.

This guide covers queue-level ordering and failure recovery for SQS FIFO. It does not assume a particular SDK, Lambda event-source mapping, or database. Those integrations have their own batching and retry semantics; verify them independently before applying these examples.

Define the ordering domain before choosing a group ID

SQS preserves the sequence of messages within a FIFO message group. Different groups can make progress independently, so a group ID should represent the smallest business entity whose operations must be serialized. For example, use a stable account or order identifier when events for that account or order must be applied in sequence. If every message uses one constant group ID, one slow or repeatedly failing message can hold back the whole queue. If every message gets a random group ID, the queue no longer serializes operations that belong together.

Make the group key stable across producer restarts and deployments. Do not derive it from a transient worker ID or a timestamp. If the system has both a tenant and an aggregate identifier, include enough namespacing to avoid collisions, such as tenant-42:order-817. Keep sensitive personal data out of group IDs; use opaque identifiers instead.

FIFO ordering begins at the queue boundary. If two independent producers race to publish updates for the same group, the queue preserves the accepted order, not an external database’s intended order. Put a monotonically increasing aggregate version or event sequence in the message body, and let the consumer reject, defer, or reconcile stale and unexpected versions. That check detects producer-side reordering that SQS cannot infer from application intent.

Before enabling high throughput, verify that the workflow can safely process distinct groups concurrently. More message groups create more opportunities for parallel work, but concurrency can still be limited by consumer capacity, downstream quotas, or a hot group. Native SQS CloudWatch metrics are queue-dimensioned, not per-message-group backlog or age. If you need those per-group signals, instrument them in the application rather than assuming that a FIFO queue distributes work evenly.

Use deduplication as a bounded producer safeguard

FIFO producer deduplication helps when a sender times out after a SendMessage request and cannot tell whether SQS accepted it. A retry using the same explicit MessageDeduplicationId within the five-minute deduplication interval does not introduce a second queued copy. A stable application event ID is usually a better choice than a hash of incidental serialization details.

For high-throughput FIFO, set DeduplicationScope to messageGroup and FifoThroughputLimit to perMessageGroupId; AWS requires these settings for that mode. Because deduplication is then scoped to a group, a retry must reuse both its original MessageGroupId and its original MessageDeduplicationId. Keep the group ID tied to the stable ordering key (for example, the account/order aggregate) and the deduplication ID tied to the stable event identity. Reusing only the deduplication ID while changing the group can put the retry in a different deduplication and ordering domain, so it may not be suppressed as intended.

With content-based deduplication enabled, SQS computes a SHA-256 hash from the message body. Message attributes are not included in that hash. Consequently, two messages with identical bodies but different attributes can be treated as duplicates during the interval, while a retry that changes body serialization can evade deduplication. If attributes carry business meaning or identical bodies can represent separate events, use an explicit deduplication ID that reflects the event identity.

The five-minute interval is not a permanent idempotency ledger. It cannot deduplicate a replay after that interval, and it says nothing about an external database, payment provider, or email service. Store processed event IDs or aggregate versions at the system that owns the business effect. If a consumer writes to a database, commit the effect and its idempotency record together where the database supports that transaction. Otherwise, define a reconciliation procedure for the ambiguous case where the remote operation succeeded but the consumer did not record or acknowledge it.

Set visibility from observed processing time

When a consumer receives a message, the message remains in SQS but becomes temporarily invisible to other consumers. If the consumer does not delete it before the visibility timeout expires, it becomes eligible for another receive. For FIFO, later messages in that same group are not made available while the earlier message remains in flight; they can proceed after it is deleted or its visibility timeout expires. A stuck handler therefore creates a per-group stall, even while other groups continue.

Set the initial timeout based on measured processing duration plus realistic variance, not an optimistic average. For work with variable duration, a consumer can extend visibility with ChangeMessageVisibility while it is still making progress. SQS caps the total visibility period at 12 hours from the first receive; extending the timeout does not restart that clock. If a task may exceed that boundary, split it into resumable steps or use a workflow service designed for longer-running work.

Delete only after the consumer has durably completed the operation or persisted enough state to resume it safely. Consider the failure window: the consumer commits a database update, then crashes before DeleteMessage. SQS can redeliver the message, so the consumer must recognize the already-applied event and return a successful acknowledgement without repeating a non-idempotent effect. By contrast, deleting first and committing later can lose work if the consumer fails between those actions.

Use dead-letter queues without hiding ordering loss

A redrive policy moves a message to a dead-letter queue after its receive count exceeds maxReceiveCount. This is valuable for isolating poison messages, but removing an event from its FIFO group changes that group’s sequence. AWS explicitly warns against using a DLQ when exact message order or operation order must not be broken. Decide whether the business process can tolerate a gap, whether later events should stop, and who is authorized to repair or replay the failed event.

Choose maxReceiveCount based on classified failures and expected transient recovery, not an arbitrary value of one. A temporary dependency outage can consume receives without indicating a poison message. Conversely, an infinite retry loop can pin one group and consume capacity. Alert on DLQ arrival and oldest-message age, and include the group ID, event ID, receive count, and application error class in structured diagnostics without logging secrets or full sensitive payloads.

Retention settings also matter during investigation. For FIFO queues, the enqueue timestamp resets when SQS moves a message to a DLQ, and the DLQ age metric reports time since that move. Choose a retention period that gives responders enough time to diagnose the issue and verify the original queue’s retention policy. Redriving should be an explicit recovery action: first correct or quarantine the cause, then verify whether reinserting the event can safely precede or follow newer events in that group.

Rehearse the failure windows

Use a non-production queue and representative downstream state to test at least these cases:

  1. Publish an event, force the producer to time out, and retry with the same deduplication ID. Confirm the result within and beyond the five-minute interval.
  2. Deliver two events for one group and one event for another. Hold the first group message in flight, then verify that the second group can still progress while the first remains ordered.
  3. Commit a consumer side effect and terminate the worker before message deletion. Confirm that redelivery does not duplicate the business effect.
  4. Let a visibility timeout expire while processing is still active. Verify duplicate-worker behavior and test a heartbeat extension, including the 12-hour total boundary.
  5. Route a repeatedly failing message to a DLQ. Confirm the expected ordering gap, alarm, retention behavior, and reviewed repair procedure before replaying it.

Use native queue-level CloudWatch metrics for visible backlog, oldest-message age, in-flight messages, receives, and DLQ depth. SQS metrics use the QueueName dimension; the FIFO-only ApproximateNumberOfGroupsWithInflightMessages is an aggregate count of active groups, not a per-group backlog or age metric. To identify a particular blocked group or track its processing age, add application-side instrumentation (for example, emit an opaque group key plus enqueue and completion timestamps). Avoid putting personal data in metric dimensions, and control cardinality and cost. These signals together can distinguish a global consumer-capacity problem from a single hot or blocked group. A FIFO queue provides useful ordering and producer-deduplication primitives; production correctness still depends on stable event identity, replay-safe effects, measured timeout policy, and an explicit answer to what happens when one ordered operation cannot be completed.

Related:

Sources:

Comments