cAdvisor¶

🐳 What is cAdvisor?¶

cAdvisor (Container Advisor) is an open-source container monitoring tool developed by Google. It runs as a daemon on a host and provides insights into:

• Resource usage (CPU, memory, network, disk I/O)
• Performance characteristics of running containers
• Per-container statistics in real time

👉 cAdvisor is often used as a metrics source for Prometheus in Kubernetes and Docker environments.

It’s lightweight, designed to run inside a container, and exposes metrics at an HTTP endpoint (/metrics) in Prometheus format.

🧐 Why Do We Need cAdvisor?¶

In modern containerized environments (Docker, Kubernetes):

• Containers are ephemeral (come and go quickly).
• Multiple containers share the same host.
• We need visibility into resource consumption per container (not just at the host level).

Without cAdvisor:

• You can see host metrics (via node_exporter) but not per-container usage.
• Hard to debug issues like “which container is consuming all the CPU?”

👉 cAdvisor provides fine-grained container-level metrics, making it critical for container monitoring.

📊 What Does cAdvisor Monitor?¶

cAdvisor collects and exposes:

• CPU usage (total, per core, throttling).
• Memory usage (working set, cache, limits).
• Filesystem usage (per container).
• Network stats (bytes sent/received, packets).
• Container lifecycle stats (start, stop, restart).
• Custom labels (Kubernetes adds pod, namespace, container name).

🔧 How cAdvisor Works¶

1. Runs as a daemon (usually a Docker container).

2. Reads metrics from the Linux kernel cgroups and container runtime (Docker, containerd, CRI-O).

3. Exposes metrics on an HTTP endpoint:

4. JSON API → /api/v1.3/subcontainers (legacy).

5. Prometheus format → /metrics.

6. Tools like Prometheus scrape these metrics.

7. Grafana visualizes them in dashboards.

🔗 Architecture Overview¶

+-------------------+
|   Linux Kernel    |
| (cgroups, stats)  |
+---------+---------+
          |
          v
+---------+---------+
|   cAdvisor        |  (container metrics daemon)
+---------+---------+
          |
          v
+-------------------+
| /metrics endpoint |  --> Scraped by Prometheus
+---------+---------+
          |
     +----v-----+
     | Grafana  |   (visualization)
     +----------+

🔄 Metric Flow: From Container → cAdvisor → Prometheus → Grafana¶

sequenceDiagram
    participant Cont as Container
    participant Kernel as Linux cgroups
    participant Cad as cAdvisor (/metrics)
    participant Prom as Prometheus
    participant Graf as Grafana
    participant User as DevOps/SRE

    Cont->>Kernel: Resource usage (CPU, mem, I/O)
    Cad->>Kernel: Collect stats via cgroups
    Prom->>Cad: Scrape /metrics
    Prom->>Prom: Store in TSDB
    Graf->>Prom: Query metrics
    Prom-->>Graf: Return data
    Graf-->>User: Show dashboards

🏗️ Prometheus + cAdvisor Architecture¶

flowchart TD

    subgraph Containers["Running Containers (Docker/K8s)"]
        C1["Container A"]
        C2["Container B"]
        C3["Container C"]
    end

    subgraph Kernel["Linux Kernel (cgroups, namespaces)"]
        M1["CPU usage"]
        M2["Memory usage"]
        M3["Disk I/O"]
        M4["Network I/O"]
    end

    Containers --> Kernel

    subgraph cAdvisor["cAdvisor Daemon (per-node)"]
        Cad["/metrics endpoint"]
    end

    Kernel --> Cad

    subgraph Prometheus["Prometheus Server"]
        P["Scrape & Store Metrics (TSDB)"]
    end

    Cad --> P

    subgraph Grafana["Grafana Dashboards"]
        G["Visualize Metrics"]
    end

    Prometheus --> G

    subgraph Alertmanager["Alertmanager"]
        A["Send Alerts (Slack, Email, PagerDuty)"]
    end

    Prometheus --> A

🔎 Explanation of the Flow¶

1. Containers → generate resource usage.
2. Linux Kernel (cgroups) → tracks CPU, memory, disk, and network per container.
3. cAdvisor → collects these stats and exposes them at /metrics.
4. Prometheus → scrapes cAdvisor metrics regularly and stores them in its TSDB.
5. Grafana → queries Prometheus for dashboards (per-container CPU/memory, etc.).
6. Alertmanager → triggers alerts if rules match (e.g., container using >90% memory).

👉 This stack gives complete visibility into container performance.

🏗️ Kubernetes + cAdvisor + Prometheus Architecture¶

flowchart TD

    subgraph Node["Kubernetes Node"]
        subgraph Pods["Pods & Containers"]
            P1["Pod A: Container A1, A2"]
            P2["Pod B: Container B1"]
        end

        subgraph Kubelet["Kubelet"]
            Cad["cAdvisor (built-in)"]
        end

        Kernel["Linux Kernel (cgroups, namespaces)"]
    end

    Pods --> Kernel
    Kernel --> Cad

    subgraph Prometheus["Prometheus Server"]
        P["Scrape & Store Metrics"]
    end

    Cad --> P

    subgraph Grafana["Grafana Dashboards"]
        G["Visualize Metrics"]
    end

    Prometheus --> G

    subgraph Alertmanager["Alertmanager"]
        A["Send Alerts (Slack, Email, PagerDuty)"]
    end

    Prometheus --> A

🔎 Explanation of the Kubernetes Flow¶

1. Pods (containers) run on each node.
2. Linux kernel (cgroups) tracks per-container resource usage (CPU, memory, I/O).

3. Kubelet runs on every node and integrates cAdvisor internally.

4. Exposes metrics at:

   • /metrics/cadvisor → container-level
   • /metrics → node & kubelet metrics
   • Prometheus scrapes kubelet endpoints across all nodes.
   • Grafana builds dashboards (per-pod, per-container, per-namespace).
   • Alertmanager notifies when resource thresholds are breached.

👉 In Kubernetes, you don’t usually run standalone cAdvisor. Instead, kubelet already provides cAdvisor-powered metrics.

📈 Example Metrics from cAdvisor¶

CPU¶

• container_cpu_usage_seconds_total → Total CPU time consumed.
• container_cpu_cfs_throttled_seconds_total → Time container spent throttled.

Memory¶

• container_memory_usage_bytes → Total memory usage.
• container_memory_working_set_bytes → Memory actively used.

Network¶

• container_network_receive_bytes_total
• container_network_transmit_bytes_total

Filesystem¶

• container_fs_usage_bytes
• container_fs_reads_bytes_total

⚙️ Running cAdvisor¶

Docker Example¶

docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  gcr.io/cadvisor/cadvisor:latest

👉 Access metrics at:

• UI: http://localhost:8080
• Prometheus metrics: http://localhost:8080/metrics

🔎 cAdvisor in Kubernetes¶

In Kubernetes, cAdvisor is built into the kubelet:

• Every node’s kubelet runs cAdvisor.
• Metrics are exposed at :4194/metrics/cadvisor.
• Prometheus scrapes these endpoints.

👉 Many Kubernetes setups use kubelet’s cAdvisor integration rather than running cAdvisor as a separate container.

🔍 Key Strengths of cAdvisor¶

• Container-native: built specifically for container monitoring.
• Lightweight: low resource overhead.
• Rich metrics: CPU, memory, disk, network at per-container level.
• Easy integration: works out-of-the-box with Prometheus.
• UI dashboard: basic real-time web UI included.

⚠️ Limitations & Watch Outs¶

• Short-term storage only → cAdvisor itself doesn’t persist data (only in-memory). → Must use Prometheus or other TSDB for historical metrics.

• Scalability → Standalone cAdvisor isn’t meant for very large clusters.

• Limited security → Exposes metrics unauthenticated (protect behind reverse proxy).

• Deprecation concern → Standalone cAdvisor development has slowed since metrics are now integrated into Kubernetes kubelet.

🔔 Example Prometheus Scrape Config for cAdvisor¶

scrape_configs:
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['localhost:8080']

📊 Grafana Dashboards for cAdvisor¶

Grafana has many pre-built dashboards (via Grafana.com) for cAdvisor metrics:

• Container CPU & memory usage
• Disk & network performance
• Top N containers by resource consumption
• Per-pod or per-namespace breakdown

👉 Example dashboard ID: 893 (Google cAdvisor).

🛠️ Alternatives to cAdvisor¶

👉 In Kubernetes, cAdvisor + kubelet is usually enough.

🛡️ Security Best Practices¶

• ❌ Don’t expose cAdvisor directly to the internet.
• ✅ Run behind a reverse proxy (Nginx, Traefik).
• ✅ Scrape metrics only from trusted networks.
• ✅ Use Kubernetes RBAC with kubelet metrics proxying.

🧾 cAdvisor Cheat Sheet¶

✅ Core Concepts¶

📊 Example PromQL¶

rate(container_cpu_usage_seconds_total[1m])   # CPU usage per second
container_memory_usage_bytes                  # Current memory usage
container_network_receive_bytes_total         # Network RX bytes

🎯 Final Takeaway¶

cAdvisor is:

• Essential for per-container resource monitoring.
• Lightweight and easy to deploy.
• Integrated into Kubernetes via kubelet.
• Best used with Prometheus + Grafana for long-term monitoring.

👉 Think of cAdvisor as the container-level metrics engine that feeds Prometheus, while node_exporter gives you host-level metrics.