Skip to content
SRE & DevOpsDeep Dive Published Updated 6 min readViews unavailable

AWS S3 Multipart Upload Cleanup: Inventory, Abort, and Lifecycle Boundaries

Find and safely retire abandoned S3 multipart uploads without confusing upload parts with completed objects or replacing unrelated lifecycle rules.

An Amazon S3 multipart upload is a stateful session, not a partially visible object. S3 retains uploaded parts after CreateMultipartUpload; it assembles the object only after a successful CompleteMultipartUpload. A producer that crashes, loses its upload ID, or never finishes can therefore leave billable part storage behind without creating an object that ordinary GetObject or object-inventory workflows can see.

Cleanup must distinguish three things: an in-progress upload identified by bucket, key, and upload ID; the parts belonging to that upload; and any completed object version already stored under the same key. AbortMultipartUpload acts on the first two. It is not a general object-delete operation and must not be used as a substitute for object retention or version cleanup.

Inventory the upload sessions, not the object namespace

Use ListMultipartUploads to find sessions that have been initiated but neither completed nor aborted. S3 returns at most 1,000 uploads in one API response. A truncated response includes continuation markers, so an API client that performs its own pagination must continue with both NextKeyMarker and NextUploadIdMarker for general-purpose buckets. The AWS CLI paginates its modeled operations by default; do not add --no-paginate to an inventory procedure that must cover the entire prefix.

Start with a narrow, read-only prefix that belongs to the service or migration being investigated:

aws s3api list-multipart-uploads \
  --bucket 'REPLACE_WITH_REVIEWED_BUCKET' \
  --prefix 'staging/import-2026-10/' \
  --query 'Uploads[].{Key:Key,UploadId:UploadId,Initiated:Initiated,InitiatorId:Initiator.ID}' \
  --output table

Treat the returned upload ID as an opaque identifier. Record the bucket owner/account, region, exact key, initiation time, initiating principal when available, and any matching application job or deployment record. A key prefix is a review boundary, not an authorization boundary: a prefix can contain work owned by multiple writers, and two concurrent multipart sessions can target the same key. Do not infer that an upload is abandoned only because it is old; reconcile with the job system and its retry/recovery state first.

For large buckets, retain the complete paginated result and count it before taking action. Check the command’s exit status and output truncation behavior, and compare the number of returned sessions with an independent operational signal such as the uploader’s job ledger. A single page or a console view filtered to one prefix is not proof that every incomplete upload has been found.

Abort one reviewed upload and verify its parts

Once an owner confirms that a specific session is abandoned and no completion attempt is in flight, abort that exact tuple. Keep the command separate from discovery so a reviewer can compare the selected key and upload ID with the inventory:

aws s3api abort-multipart-upload \
  --bucket 'REPLACE_WITH_REVIEWED_BUCKET' \
  --key 'staging/import-2026-10/object-00042.bin' \
  --upload-id 'REPLACE_WITH_REVIEWED_UPLOAD_ID'

aws s3api list-parts \
  --bucket 'REPLACE_WITH_REVIEWED_BUCKET' \
  --key 'staging/import-2026-10/object-00042.bin' \
  --upload-id 'REPLACE_WITH_REVIEWED_UPLOAD_ID'

This is a mutating operation: once aborted, that upload ID cannot accept additional parts. AWS documents a race when part uploads are already in progress; such an upload may still succeed while the abort is being processed. AWS therefore recommends checking ListParts and repeating the abort if parts remain. Treat a concurrent completion or a NoSuchUpload response as a reconciliation signal, not as permission to retry against a different upload ID. Confirm whether a completed object now exists through the ordinary object/version path used by the workload.

Do not automate a broad list | xargs abort pipeline as an incident shortcut. A shell pipeline can lose pagination, mishandle keys with spaces or unusual characters, and act on an upload that the owner has resumed between review and execution. For a controlled cleanup job, persist a reviewed candidate manifest, bind each action to bucket/key/upload ID, make retries idempotent at the workflow level, and preserve per-session responses for audit.

Use lifecycle as a guardrail, not as an immediate repair

An S3 lifecycle rule with AbortIncompleteMultipartUpload makes an upload eligible for abort after a configured number of days from initiation. AWS states that this rule also applies to uploads already in progress when the rule is added. The action has no effect on completed objects; it removes incomplete upload parts rather than expiring an object. Lifecycle processing is asynchronous, so it is a cost-control safety net, not a synchronous cleanup confirmation for an incident.

A prefix-scoped rule can limit the cleanup policy to an upload namespace owned by a particular service. Tags are not available as the selection mechanism for this lifecycle action, so avoid designing a cleanup policy around a tag filter. Choose the age threshold from the longest legitimate upload and retry window, then test with a disposable bucket or a controlled test prefix before applying it to a production bucket.

One operational trap is the configuration API itself: PutBucketLifecycleConfiguration creates a new configuration or replaces the existing bucket lifecycle configuration. It does not merge one rule into the existing set. Retrieve the current configuration, preserve every intended rule and filter, review the complete replacement document, and only then apply it through the normal change process. A technically correct abort rule can still remove an unrelated transition or expiration rule if the replacement body omits it.

Before applying the configuration, inspect the current state without writing:

aws s3api get-bucket-lifecycle-configuration \
  --bucket 'REPLACE_WITH_REVIEWED_BUCKET'

After an approved change, retrieve the configuration again and verify the rule ID, enabled status, filter, and DaysAfterInitiation. Also inspect existing multipart uploads to confirm the rule’s scope is appropriate. Do not treat a successful configuration API response as evidence that every old upload has already been removed.

Build an acceptance test around object integrity

In a test bucket, start a multipart upload, upload at least one small test part, and verify that the object is absent before completion. Abort the session, list its parts, and confirm the incomplete upload is no longer usable. Separately complete another upload under a test key and verify that the lifecycle abort action leaves the completed object untouched. The goal is to test both sides of the boundary: cleanup removes uncommitted part state but preserves completed data.

For production acceptance, record the bucket type, account and region, lifecycle document revision, rule age, owner for each upload namespace, and the expected maximum legitimate transfer duration. Alert on old sessions and on repeated cleanup failures, but avoid sending raw object keys or upload IDs into broad logs if they reveal customer or workload information. Reconcile retained part storage after cleanup rather than using a disappearing listing as the only evidence.

Multipart-upload hygiene is complete when the producer can resume or abandon a session deliberately, an operator can identify the exact session before aborting it, lifecycle cleanup preserves existing policy, and tests prove that incomplete parts rather than completed object data are the target.

Related:

Sources:

Comments