How to Set Up Canary Analysis with Automated Rollback

The manual canary deployment approach covered elsewhere on this blog requires a human watching metrics and deciding when to proceed or roll back — Flagger automates that entire decision loop based on metric thresholds you define once.

Step 1: install Flagger

helm repo add flagger https://flagger.app
helm repo update
helm upgrade --install flagger flagger/flagger \
  --namespace istio-system \
  --set meshProvider=istio

Flagger works with several service mesh and ingress options (Istio, Linkerd, NGINX Ingress, App Mesh) — this example assumes Istio for traffic splitting.

Pin a reviewed chart version in production instead of silently accepting a newer chart, and verify that the version supports your Kubernetes and Istio releases. Flagger does not install or validate the entire service mesh for you: traffic routing, telemetry, CRDs, and permissions must already be healthy. Confirm the controller is Ready before continuing:

kubectl rollout status deployment/flagger -n istio-system
kubectl get crd canaries.flagger.app

Step 2: deploy your application normally first

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  selector:
    matchLabels: {app: myapp}
  template:
    metadata:
      labels: {app: myapp}
    spec:
      containers:
        - name: myapp
          image: registry.example.com/myapp@sha256:STABLE_IMAGE_DIGEST
          readinessProbe:
            httpGet: {path: /ready, port: 8080}

Flagger takes over managing this Deployment’s rollout process once a Canary resource references it — the Deployment itself doesn’t need special annotations beforehand.

Step 3: define a Canary resource with promotion criteria

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: myapp
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  service:
    port: 80
  analysis:
    interval: 1m
    threshold: 5
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
        interval: 1m
      - name: request-duration
        thresholdRange:
          max: 500
        interval: 1m

stepWeight: 10 shifts 10% more traffic to the canary every interval; the metrics block defines the actual pass/fail criteria — here, a 99% success rate and under 500ms latency, checked every minute.

Those thresholds are examples, not universal defaults. Choose them from the service’s SLO, normal traffic volume, and measurement noise. A success rate computed from two requests is not meaningful, and a 500 ms average can conceal a bad tail. Flagger’s built-in request-duration check uses the metric exposed by the selected provider; confirm the exact aggregation and units in that provider’s integration. Add application outcomes—such as checkout failures or queue age—when transport-level success is insufficient.

Step 4: trigger a canary rollout by updating the deployment

kubectl set image deployment/myapp \
  myapp=registry.example.com/myapp@sha256:CANARY_IMAGE_DIGEST

Flagger detects the change automatically and begins the analysis-driven rollout — no separate command needed to “start” a canary.

Step 5: watch Flagger’s automated progression

kubectl describe canary myapp

This shows the current traffic weight, the metrics being evaluated each interval, and whether the canary is progressing, holding, or has failed.

Step 6: understand what happens on a metrics failure

If metric checks fail often enough to reach threshold (5, in this example), Flagger automatically rolls back — routing traffic back to the primary and scaling down the failed canary. Do not describe this as necessarily five consecutive observations: the threshold is a failed-check limit during analysis, and its precise behavior also depends on skipped or unavailable metric queries. Test failure behavior deliberately before trusting it in production.

An automated traffic rollback is not a time machine. It cannot undo an incompatible database migration, a message published to another system, or data written using a new format. Use expand-and-contract migrations, keep APIs backward compatible throughout analysis, and provide a separate recovery plan for stateful side effects.

Step 7: understand what happens on success

Once the canary reaches 100% traffic weight without triggering a rollback, Flagger promotes it — the canary version becomes the new stable baseline, and the whole cycle is ready for the next deployment.

Step 8: add a webhook for custom pre/post-rollout checks

analysis:
  webhooks:
    - name: smoke-test
      url: http://flagger-loadtester.test/
      metadata:
        cmd: "curl -sf http://myapp-canary/health"

Webhooks let you run arbitrary checks (smoke tests, load tests) as part of the automated gate, beyond just the built-in metric thresholds.

Keep webhook endpoints cluster-internal, authenticated where appropriate, tightly authorized, and bounded by timeouts. A command supplied through metadata is powerful: treat the Canary manifest as privileged configuration and require review. Smoke tests should be idempotent and must not create real orders, charge cards, or corrupt production state.

Operate the controller, not only the rollout

Monitor the Flagger controller itself and alert on stalled or failed analyses. Preserve Canary events and controller logs long enough to explain why a release promoted or failed. Use a PodDisruptionBudget and appropriate replicas for the routing and metrics dependencies; a missing metrics backend should fail safely rather than approve an unobserved release.

Before enabling automated sync from CI, run a controlled exercise: deploy a version that returns a known error, verify the metric breach, confirm stable traffic restoration, and inspect the final primary image digest. Then test a good version and ensure promotion completes. This distinguishes a valid manifest from a genuinely working safety loop.

Why automating the decision, not just the traffic shift, is the actual improvement here

Manually watching dashboards during a canary rollout doesn’t scale past a small number of deployments, and human judgment during an incident is often slower and less consistent than a pre-defined threshold. Flagger’s real contribution isn’t the traffic-splitting mechanism itself (which plain Kubernetes primitives can also achieve) — it’s replacing “a human decides whether this canary is healthy” with a fast, consistent, metrics-driven decision made automatically, every single deployment.

Related:

Sources: