How to Set Up Prometheus and Grafana Monitoring for Kubernetes

Prometheus collects and stores metrics; Grafana visualizes them. This walks through deploying both into a Kubernetes cluster using the community-maintained Helm chart that bundles them together sensibly.

Step 1: add the Helm repository

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Step 2: install the kube-prometheus-stack chart

helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --version CHART_VERSION \
  --values monitoring-values.yaml

This single chart deploys Prometheus, Grafana, Alertmanager, and a set of pre-configured dashboards and scrape targets for core Kubernetes metrics — a considerably faster starting point than configuring each component independently from scratch.

Replace CHART_VERSION with a reviewed version and commit the values file. The chart changes independently from Prometheus, Grafana, and the operator, so read its release notes and CRD upgrade instructions. Production values should define resource requests and limits, persistent volumes, retention by time or size, replica strategy, and disruption behavior instead of accepting development-friendly defaults.

Step 3: verify the components are running

kubectl get pods -n monitoring

You should see pods for Prometheus, Grafana, Alertmanager, the operator, kube-state-metrics, and node-exporter. Running is only a process state: wait for readiness, inspect failed events, and confirm Prometheus targets are Up. Some control-plane targets are unavailable or configured differently on managed services; disable or adapt only the relevant scrape rules rather than ignoring permanent alerts.

Step 4: access the Grafana dashboard

kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

Retrieve the generated admin password from the chart’s Kubernetes Secret according to the chart notes rather than assuming a hard-coded default. Port forwarding is suitable for bootstrap. For shared access, use TLS, SSO, least-privilege Grafana roles, network restrictions, and an external secrets workflow; do not publish the service with its bootstrap credential.

Step 5: explore the pre-built dashboards

Grafana → Dashboards → browse the pre-loaded set, including
  Kubernetes cluster overview, node metrics, and pod-level
  resource usage

The kube-prometheus-stack chart ships with a substantial set of dashboards already configured against Prometheus as a data source — worth exploring before building custom ones, since many common needs are already covered.

Step 6: query metrics directly with PromQL

Grafana → Explore → select the Prometheus data source →
sum by (namespace, pod) (
  rate(container_cpu_usage_seconds_total{container!="", image!=""}[5m])
)

PromQL is Prometheus’s query language. This example aggregates CPU rate by pod and namespace over five minutes. Validate recording and dashboard queries against the labels actually present in the cluster; exporter and Kubernetes upgrades can change series availability. Avoid unbounded labels such as user IDs, raw URLs, request IDs, or arbitrary error text because cardinality drives memory, storage, and query cost.

Step 7: add a custom scrape target for your own application

# ServiceMonitor custom resource, if your app exposes /metrics
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp-monitor
  namespace: monitoring
  labels:
    release: monitoring
spec:
  namespaceSelector:
    matchNames: [myapp]
  selector:
    matchLabels: {app: myapp}
  endpoints:
    - port: metrics
      path: /metrics

A ServiceMonitor (provided by the Prometheus Operator this chart installs) tells Prometheus to automatically discover and scrape your own application’s metrics endpoint, the same declarative way Kubernetes handles most other configuration.

Whether Prometheus selects this object depends on the chart’s serviceMonitorSelector and namespace selector. The matching Service must expose a named port called metrics; the ServiceMonitor selects Services, not Deployments. Check the Prometheus Targets page and operator logs instead of assuming that an accepted custom resource is being scraped.

Step 8: set up an alert rule

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: high-error-rate
  namespace: monitoring
  labels:
    release: monitoring
spec:
  groups:
    - name: myapp.rules
      rules:
        - alert: HighErrorRate
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m])) > 0.05
          for: 10m

This alert fires if the 5xx error rate exceeds 5% sustained for 10 minutes — routed through Alertmanager to whatever notification channel (Slack, email, PagerDuty) is configured separately.

Add a minimum traffic condition to avoid unstable ratios at very low volume and scope labels to the intended service. Every page needs an owner, severity, runbook, and tested Alertmanager route. Exercise the notification path end to end, including inhibition and silence behavior; a valid PrometheusRule does not prove anyone will receive the alert.

Plan retention, backup, and failure behavior

Local Prometheus storage is not automatically a durable, multi-cluster archive. Size retention from measured ingestion, test volume loss, and keep headroom for compaction. If long retention or global querying is required, evaluate a supported remote-write or long-term system and understand its consistency and cost.

Back up Grafana dashboards and alert configuration that are not already declarative. Test restoring the monitoring stack and verify it can alert when part of itself is unavailable. Monitor scrape failures, rule evaluation errors, rejected samples, disk fill, WAL replay, notification failures, and certificate expiry. Observability that fails silently during an incident is worse than a visible monitoring outage.

Why the combined Helm chart is worth using over assembling components manually

Prometheus, Grafana, Alertmanager, and the various exporters that feed them metrics each have their own configuration surface — getting all of them correctly wired together (data sources, scrape configs, dashboard provisioning) by hand is a substantial amount of glue work that the kube-prometheus-stack chart has already solved and battle-tested across a huge number of real deployments, making it the sensible starting point for nearly any Kubernetes cluster’s monitoring setup rather than assembling the same stack manually from scratch.

Related:

Sources: