Terraform State in Production: Backends, Locking, Drift, Import, and Recovery
A complete operational model for tfstate files: what they contain, how remote backends coordinate teams, and how to migrate or recover state without corrupting it.
Terraform state is the mapping between resource addresses in configuration and objects managed through provider APIs. It also stores dependency metadata and cached attribute values needed to plan efficiently. The JSON snapshot is an internal persistence format, not a file operators should hand-edit.
Local and remote state
The default local backend writes terraform.tfstate and normally keeps the prior snapshot as terraform.tfstate.backup. That is convenient for experiments but unsafe for a team: laptops fail, copies diverge, and concurrent applies can race. A remote backend centralizes the current snapshot and can add access control, encryption, versioning, auditability, and locking. Capabilities differ by backend; locking is not universal merely because storage is remote.
State contains sensitive data
Marking an output sensitive redacts normal CLI display but does not remove the value from state. Provider-returned credentials and application secrets may be present in clear JSON inside the encrypted storage envelope. Restrict state access, encrypt transport and storage, keep it out of Git, rotate exposed credentials, and prefer designs that avoid placing secret material in managed attributes.
Refresh, drift, and imports
Plan and apply normally refresh tracked objects before proposing changes. For an explicit drift-only review, current Terraform uses terraform plan -refresh-only; the standalone terraform refresh command is deprecated because it directly updates state without the review step. Import establishes a binding for an existing remote object, using an import block or terraform import according to the workflow. It does not automatically author a complete, correct resource configuration.
The plan command itself does not commit refreshed state. Review the refresh-only plan and use a refresh-only apply to persist accepted state changes. If configuration should own the real-world value, change configuration instead of accepting drift into state.
Moving and migrating safely
Use terraform state mv, rm, show, list, pull, and related commands instead of editing JSON. Initialize a backend change and let Terraform migrate the snapshot after taking a verified backup. Preserve lineage and serial protections. terraform state push and force-unlock are recovery tools: confirm no other writer is active, save the remote snapshot first, and document why normal safeguards were bypassed.
Recovery practice
Enable backend versioning and test restoration in an isolated workspace. If a backend write fails and Terraform emits a local recovery state, stop further applies until that snapshot is reconciled. The safest state incident is one rehearsed before production: named owners, locked writes, retained versions, tested credentials, and a written rollback sequence.
Backend design and access separation
Evaluate storage, locking, encryption, versioning, audit logs, availability, workspace behavior, and authentication for the exact backend. Remote storage does not guarantee locking. For the S3 backend, current Terraform can use an S3 lockfile with use_lockfile; DynamoDB-based locking is deprecated. Enable bucket versioning and grant state and lock object permissions only to plan/apply identities that require them.
Keep backend credentials out of configuration and -backend-config values that can persist under .terraform or plan artifacts. Prefer environment or workload identity mechanisms supported by the backend. Separate state administration from ordinary infrastructure changes, and protect deletion or lifecycle changes on the storage itself through an independent control plane.
State boundaries and dependencies
One state snapshot creates one locking and failure domain. Split unrelated systems when ownership, change cadence, credentials, or blast radius differs, but do not fragment until every output becomes a fragile manual copy. Publish only required outputs through a stable interface and avoid giving consumers broad read access to a sensitive state file.
Terraform workspaces create multiple state instances for one configuration; they are not automatically a complete environment isolation strategy. Accounts, projects, credentials, policy, and backends may still need separation. Record the exact workspace and backend key in automation so a production apply cannot silently run against the default workspace.
Safe import, removal, and refactoring
An import binds an address to an existing remote object. Write the intended resource configuration, import under a reviewed plan, and reconcile every proposed difference before apply. Removing an address with terraform state rm makes Terraform forget the object; it does not delete the remote object. A later plan may propose creating a replacement with the same name.
Use moved blocks for durable configuration refactors and state commands only when the declarative mechanism cannot express recovery. Before state mv, state rm, state push, or force-unlock, stop automation, identify the lock owner, pull and hash a backup, record serial and lineage, and prepare a rollback. Never copy a snapshot between unrelated lineages merely because the JSON looks compatible.
A recovery drill
Create an isolated test backend, provision disposable infrastructure, and simulate accidental state deletion, stale locking, a failed backend write, and an imported object. Restore a version, reinitialize, run a refresh-only plan, and prove the next normal plan is understood. Do not test by pointing an alternate configuration at production objects.
During a real incident, preserve failing logs, plan, local recovery state, remote versions, lock metadata, Terraform and provider versions, and cloud audit records. Choose one authoritative snapshot, reconcile it under a single writer, and only then resume automation. Recovery is complete when configuration, state, and reality produce an explainable no-op or approved plan.
Automate a read-only state inventory and backend health check, but do not schedule unattended repair commands. Alert on failed locks, denied backend access, unexpected state deletion, disabled versioning, and apply runs outside the approved pipeline. Review who can read state as seriously as who can apply infrastructure, because read access may disclose credentials and private topology.
Treat the state format as Terraform-owned data, not as a stable public API for hand editing. When forensic inspection is necessary, preserve the original bytes and use supported terraform show, state pull, and state subcommands; validate any recovery against an isolated backend before replacing the authoritative snapshot.
Related: Recovering from a Stuck Terraform State Lock · Fixing Terraform Provider Version Conflicts
Sources: Terraform state, backends and locking, S3 backend, refresh-only mode, import existing resources, state commands