Skip to content
Tech HistoryHistory Published Updated 5 min readViews unavailable

Tandem NonStop Process Pairs: Checkpointed State, Not Lockstep Execution

Trace Tandem primary/backup checkpoints, duplicate-request handling, and the hardware and transaction assumptions that bounded recovery.

Tandem’s NonStop reputation came from a system design that combined redundant hardware with software recovery. One part of that design was the process pair: a primary process performed work while a backup on another processor received enough checkpoint information to take over if the primary or its processor failed. The mechanism was not a second CPU executing the same instruction stream in lockstep. Correct recovery depended on which operations had been checkpointed and how the application handled requests that could not safely be repeated.

Joel F. Bartlett’s 1981 paper, written while he was at Tandem Computers, describes the process-pair protocol and its assumptions. It is a useful primary source because it gives a concrete request sequence, not only a later product summary. Its details describe that system generation; they should not be projected onto every later HPE NonStop product without checking the applicable Guardian and transaction-manager documentation.

A software protocol over a redundant machine

Bartlett describes nodes built from multiple processors and I/O controller modules connected by redundant buses. Each processor had its own memory, power supply, and I/O channel. His fault model assumes that a single component failure is contained to the affected processor, bus, or controller. That hardware arrangement provided paths for recovery, but it did not itself preserve every application’s intermediate state.

A process pair attached a primary and backup process to a symbolic name associated with a resource or application server. Requests went to the primary, which controlled the resource. At appropriate points, it checkpointed request and state information to the backup through the message system. If the primary process or its processor failed, the backup could take over under the protocol. The message system’s name table then routed later requests to the new primary.

That architecture has three separate parts: hardware must provide an alternate processor and communication path; the operating system must detect failure and redirect the named request; and the application or server must checkpoint the state needed to resume correctly. Calling all three simply “redundancy” hides which component is responsible for what.

Sequence numbers made a non-idempotent request recoverable

The paper’s disk-server example shows why a backup needs more than a copy of the current data. A requestor and server each track sequence numbers; the server also saves completion status. That state lets a replacement process distinguish a request it has already performed from one it still needs to complete.

In simplified form, a write request follows this sequence:

  1. The requestor sends a uniquely sequenced request to the server’s primary.
  2. The server checks saved sequence and status. If the request is a duplicate, it returns the saved result instead of repeating the operation.
  3. For a new request, the server performs the work and checkpoints the request and updated record to its backup.
  4. The disk write and completion status are recorded; the result and sequence state are checkpointed before the reply completes the requestor’s own state.

The example matters because a key-sequenced file write is not safely repeatable by default: repeating it can produce a duplicate-key error. Sequence tracking turns retry into a protocol with remembered outcomes. If a requestor fails after the server performs an operation but before receiving the reply, its backup can resend the request and obtain the saved status instead of applying the write again.

This is not a universal “exactly once” property created merely by running two processes. The paper explicitly requires the server to track non-retryable requests and save enough status to recognize duplicates. The necessary history depends on the operation and resource. An application that sends a payment request, prints a check, or emits a message to an external system needs its own deduplication or transaction boundary; an internal process pair cannot infer whether an unrelated side effect already happened.

Checkpoint boundaries define the recovery story

A checkpoint is a recovery boundary, not a transparent copy of every instruction or local variable. If the primary fails between checkpoints, the backup’s view reflects the last state the application actually made recoverable. The code and storage model therefore have to agree on what a “completed” request means, how in-flight work is detected, and whether a retry is safe.

Later HPE NonStop Transaction Management Facility documentation describes explicit checkpoint placement for a single-threaded transactional requester. One checkpoint can establish a clean boundary before beginning a transaction; another, placed before commit, records the in-progress transaction state. The documented recovery behavior is to abort and restart the whole transaction when failure occurs during it, or continue at the next transaction when failure happens between transactions. That is a transaction-manager contract, not a claim that every original NonStop process pair automatically resumed arbitrary work at an instruction boundary.

The same documentation warns that transaction-manager database consistency is not fault tolerance for every outside effect. A message sent across a data communication link, for example, may require process-pair recovery in addition to database rollback. Separate durable state and idempotency keys are still necessary when an operation crosses system boundaries.

The fault model was deliberately bounded

The 1981 paper’s design goal covers contained single faults and assumes they can be detected, diagnosed, and repaired online. It also identifies contamination before fault detection and operator or service errors as serious risks. A second simultaneous failure, an uncontained fault, corrupt application logic, a bad recovery procedure, or a human mistake is not made harmless by the existence of a backup process.

Bartlett closes by declining to make broad reliability or availability claims on behalf of every deployment. That caution is still appropriate. Process pairs describe a recovery mechanism; they do not establish a particular uptime percentage, zero-loss guarantee, or recovery time for a system whose workload, checkpoint placement, disk configuration, and external dependencies have not been measured.

The historical lesson is therefore more specific than “two copies are safer.” Tandem paired a modular multiprocessor design with message routing and application-aware checkpoints. That combination let a successor process take over requests using recorded state, while leaving the difficult questions of idempotency, transaction boundaries, and failure assumptions visible to system designers.

Related:

Sources:

Comments