Thanos Sidecar: Separate Live Prometheus Queries from Object-Store History
Connect Prometheus to Thanos through a sidecar, distinguish live StoreAPI data from uploaded blocks, and verify labels, retention, and query paths.
Adding a Thanos sidecar beside Prometheus gives a Querier access to recent series through the sidecar’s StoreAPI. Giving that sidecar object-store credentials adds a second, optional behavior: upload completed Prometheus TSDB blocks to a bucket. These are related but different data paths. The sidecar does not turn a bucket into a Prometheus replica, and it does not serve historical bucket contents by itself. A Thanos Store Gateway reads those older blocks for queries; a Compactor manages their long-term block lifecycle.
The distinction matters during failures. A live panel can look healthy because the Querier still reaches Prometheus through the sidecar while uploads have stopped. Conversely, a historical graph can work through Store Gateway when the local Prometheus is down, but it will not contain the unshipped head data. A sound rollout tests both paths and records where each time range is expected to come from.
Draw the data path before deploying it
Prometheus scrapes targets, writes samples into its local WAL and head, and periodically cuts TSDB blocks. A sidecar colocated with it exposes the recent data through Thanos StoreAPI. With object storage configured, the sidecar ships completed blocks after Prometheus creates them; the Thanos documentation describes the usual block cadence as two hours. The Querier can talk to a sidecar for live data and a Store Gateway for bucket-backed history. The gateway synchronizes block metadata from the bucket and serves historical series through the same StoreAPI family.
Keep Prometheus local disk persistent. Thanos warns that object storage does not make a sidecar-backed Prometheus stateless: losing the local TSDB volume before the current head becomes a block and is uploaded can lose roughly the recent block interval. An ordinary process restart with the volume intact is different, because Prometheus can replay its write-ahead log (WAL). Object storage protects uploaded blocks; it cannot reconstruct samples that never reached it. Treat the local WAL, Prometheus retention, sidecar upload lag, bucket durability, and Store Gateway availability as separate controls.
Make source identity stable
Set Prometheus global.external_labels so each Prometheus source has a unique and persistent label set across the entire Thanos deployment. An HA pair needs a label distinguishing its two replicas, while labels describing the shared cluster or environment should agree where appropriate. Do not assign the same complete external-label set to two different Prometheus instances. Thanos uses those labels in uploaded block metadata and the Compactor’s grouping; collisions can make independently produced blocks appear to belong to one stream and can halt compaction on overlaps.
For example, these values illustrate identity, not a complete Prometheus configuration:
global:
external_labels:
cluster: west
replica: prometheus-aThe second member of the HA pair would use a different replica value. Keep an instance’s labels stable through restarts and replacements; a gratuitous rename creates a new block stream. Check the final running Prometheus configuration and the sidecar’s observed metadata, not only a Helm values file. A templating error can produce two identical replicas even when the intended values differ.
Thanos Querier can deduplicate HA series when configured with the replica label. This is a query-time presentation rule, not permission to give both replicas the same source identity in object storage. Test with deduplication on and off: without it, the two replica series should remain distinguishable; with it, the expected logical series should be returned.
Configure upload as a storage migration, not a checkbox
The Thanos sidecar documentation requires object-store configuration for upload and recommends equal Prometheus --storage.tsdb.min-block-duration and --storage.tsdb.max-block-duration values, normally 2h, to disable local compaction of the blocks the sidecar ships. Leaving Prometheus to compact those blocks while Thanos also manages them in object storage risks overlapping or inconsistent block histories. Do not copy those flags to a Prometheus instance that has no planned sidecar upload; the recommendation is conditional on this architecture.
The same guidance recommends local retention of at least three times the minimum block duration - six hours with two-hour blocks - so temporary bucket connectivity problems do not immediately discard blocks before upload resumes. That is a minimum resilience recommendation, not a universal sizing formula. Estimate the WAL and head, series cardinality, scrape rate, filesystem overhead, and the longest plausible outage. If the bucket is unavailable longer than the local retention window, uploaded history may have an unrecoverable gap. Alert on the actual upload backlog and failures, not just bucket reachability.
Store object-store configuration in a protected file or the platform’s secret mechanism and pass a path through --objstore.config-file; avoid embedding access keys in command-line arguments or article examples. Grant the sidecar only the permissions it needs to write and inspect its blocks. Thanos documents that the Compactor, not the sidecar or Store Gateway, should be the component authorized to delete bucket data. Review the actual backend’s credential and encryption model before rollout.
For existing long-retention Prometheus data, do not assume changing the two block-duration flags retroactively makes old compacted blocks safe to upload. Thanos documents a separate --shipper.upload-compacted migration path with special cautions. Inventory existing blocks and plan a tested migration rather than enabling a hidden or exceptional flag during an incident.
Verify the live and historical query routes separately
First confirm Prometheus itself serves a known series for the recent period. Then check that the sidecar is reachable through the Querier’s configured StoreAPI endpoint and that the Querier displays the expected external labels and time range. Finally check Store Gateway discovery, bucket synchronization, and a range whose completed block should already exist in object storage. The Store Gateway may not see a new block instantly; its synchronization loop and the object store’s consistency behavior introduce delay.
A safe smoke test does not need to mutate production metrics. Use a known stable expression such as up, record its labels at a recent timestamp, and compare the same series through Prometheus and Thanos with an appropriate query window. A historical query should target a time older than the local head and after at least one successful upload. If there is no old data yet, report the historical path as untested; a recent query cannot prove it. Compare a query with deduplication disabled and enabled when validating an HA pair.
Distinguish failures by layer. If Prometheus has data but Thanos lacks only recent data, check sidecar reachability and Querier endpoint discovery. If recent data works but history is absent, check sidecar shipping logs and metrics, bucket objects, Store Gateway readiness, and the selected time window. If bucket blocks exist but a query returns partial data, inspect Store Gateway synchronization and Querier warnings before changing retention. Do not delete objects, rewrite metadata, or disable compaction to make a graph look complete.
The Compactor must be treated as a separate owner. Thanos warns that concurrent compactors against the same block stream are not generally safe; a misconfigured HA rollout can create overlapping blocks and require manual repair. Its retention and downsampling policies are also independent of Prometheus’s local retention. A production readiness review should therefore name exactly one owner for each data lifecycle: Prometheus for local TSDB, sidecar for upload, Store Gateway for historical reads, and Compactor for bucket compaction and retention.
Related:
- Prometheus TSDB Capacity and Retention: WAL, Blocks, and Backups
- Prometheus Federation: Hierarchical Aggregation, Selectors, and Failure Boundaries
Sources: