Infrastructure as Code: Terraform State, Drift, and Idempotency

Terraform’s core promise — describe your infrastructure declaratively, and it figures out how to make reality match — depends entirely on one artifact most users interact with far less than they should: the state file. Understanding what it actually is, and why it exists, explains almost every non-obvious Terraform behavior, from drift to locking to why terraform apply sometimes wants to destroy and recreate something you didn’t touch.

Why state exists at all

Terraform’s configuration files describe desired infrastructure, but cloud APIs don’t expose “give me everything you created because of this specific Terraform run” as a query. State is Terraform’s own record of exactly what it created and what real-world resource IDs correspond to each configuration block — without it, Terraform would have no way to know that the aws_instance.web block in your .tf file corresponds to i-0abc123def456 in AWS.

terraform show
terraform state list

aws_instance.web
aws_security_group.web_sg
aws_db_instance.primary

The plan: a three-way diff

terraform plan is fundamentally a three-way comparison: the configuration (what you wrote), the state file (what Terraform last recorded), and — critically — a fresh refresh of real infrastructure (what actually exists right now). Discrepancies between the state file and real infrastructure are drift; discrepancies between configuration and state are planned changes.

terraform plan -out=tfplan
terraform apply tfplan

Drift: when reality diverges from state

Drift happens whenever something changes infrastructure outside Terraform’s awareness — a manual console edit, another automation tool, an auto-scaling event — leaving the state file describing something that no longer matches reality. Terraform’s refresh step is what detects this, and a plan after drift will show Terraform “correcting” the resource back toward what configuration says it should be, which can be surprising if you don’t expect a manual change to be reverted.

terraform plan -refresh-only

-refresh-only updates state to match reality without proposing any configuration-driven changes — the right first step when drift is suspected, letting you see exactly what changed before deciding whether to accept the drift (by updating configuration to match) or let Terraform revert it.

Idempotency: why running apply twice should be a no-op

A well-behaved Terraform configuration is idempotent — running apply against already-correct infrastructure produces no changes at all, because Terraform’s plan is always a diff against current state, not a blind “create everything again” operation. This property is what makes Terraform safe to run repeatedly (in CI, on a schedule, or manually) without fear of duplicating resources — a directly analogous guarantee to a well-written configuration management playbook (Ansible, Puppet) being safe to re-run.

terraform apply
# No changes. Your infrastructure matches the configuration.

Remote state and locking: the multi-person problem

A local state file works for solo experimentation but fails immediately with more than one person or automation pipeline running Terraform against the same infrastructure — two concurrent apply runs reading and writing the same local file corrupt each other’s view of reality. Remote state backends (S3+DynamoDB, Terraform Cloud, GCS) solve this with shared storage plus state locking, preventing two runs from modifying state concurrently:

terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
  }
}

A lock held by a stuck or crashed run is the classic operational headache this introduces — terraform force-unlock exists specifically for that recovery scenario, deliberately requiring a manual, explicit step rather than ever silently breaking a lock automatically.

Why some changes force a destroy-and-recreate

Certain resource attribute changes can’t be applied in-place against the real cloud API at all — changing an AWS instance’s availability_zone, for instance, isn’t something the EC2 API supports as a modification, so Terraform’s provider schema marks that attribute as ForceNew, meaning any change to it requires destroying the old resource and creating a new one:

terraform plan
# -/+ aws_instance.web (new resource required)
#     ~ availability_zone: "us-east-1a" -> "us-east-1b"  # forces replacement

This is provider-schema-driven, not a Terraform-wide policy — the same kind of attribute change might update in-place on a different resource type or a different cloud provider entirely, depending on what that specific API actually supports.

Modules and state organization

Splitting infrastructure into modules (reusable, parameterized configuration bundles) and separate state files per environment/component (rather than one enormous shared state) limits the blast radius of both mistakes and lock contention — a broken apply against the networking module’s state doesn’t block or risk the completely separate database module’s state.

Why this design has become the industry default

Terraform’s state-centric model is what lets “infrastructure as code” mean something stronger than “a script that creates infrastructure” — it’s a system that continuously reconciles declared intent against both its own history and observed reality, catching drift, guaranteeing idempotent re-application, and coordinating safely across a team. Nearly every criticism of Terraform in practice — mysterious forced replacements, drift surprises, lock contention — traces back to a gap in understanding exactly what the state file is doing, which is precisely why it’s the first thing worth understanding deeply rather than treating as an opaque implementation detail.