Monitoring with cAdvisor, Prometheus and Grafana on Docker

Introduction

Monitoring a system is crucial after deploying a product to a production environment. Keeping an eye on system metrics like logs, CPU, RAM, disks, etc, helps identify the system's status, performance issues, and provides timely solutions to ensure stable operations.

While cloud providers like Google, Amazon, or Azure offer built-in monitoring systems, if your company needs to manage multiple applications/systems/containers and desires a centralized monitoring system for easier management, using cAdvisor, Prometheus, and Grafana is a sensible choice. These three popular open-source tools are widely used by DevOps teams, especially for monitoring container applications.

cAdvisor

Developed by Google, cAdvisor is an open-source project used to analyze resource usage, performance, and other metrics from container applications, providing an overview of all running containers.

Find more details here

Prometheus

Prometheus is a toolkit for system monitoring and alerting based on system metrics. It supports organizing data over time, sending alerts via email, SMS, etc. Prometheus can integrate with various tools, including cAdvisor.

While cAdvisor fetches system information, Prometheus provides functionalities to manipulate that information such as visualizing data and triggering alerts.

Find more details here

Grafana

Grafana is a platform specialized in data visualization, supporting the creation of beautiful dashboards for web viewing, facilitating professional data analysis. Grafana can integrate with various data sources like MySQL, MongoDB, and Prometheus, etc.

So, the combination of cAdvisor, Prometheus, and Grafana works like this:

cAdvisor gathers system info.
Prometheus scrapes data from cAdvisor.
Grafana is used to visualize data from Prometheus (while Prometheus has monitoring dashboards, Grafana excels in data visualization, hence the need for Grafana).

Setup Docker

First, let's create a docker-compose.yml file as follows:

services:
  cAdvisor:
    image: gcr.io/cadvisor/cadvisor
    container_name: cAdvisor
    ports:
      - 8080:8080
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /etc/machine-id:/etc/machine-id:ro
      - /var/lib/dbus/machine-id:/var/lib/dbus/machine-id:ro
  prometheus:
    image: prom/prometheus
    ports:
      - 9090:9090
    command:
      - --config.file=/etc/prometheus/prometheus.yml
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    depends_on:
      - cAdvisor
  grafana:
    image: grafana/grafana
    ports:
      - 3000:3000
    restart: always
    depends_on:
      - prometheus

You can see that I’ve defined three services corresponding to cAdvisor, Prometheus, and Grafana. In this setup, Prometheus depends on cAdvisor, and Grafana depends on Prometheus.

Next, create a `prometheus.yml` file in the same location as your `docker-compose.yml`. This file will configure Prometheus to scrape data from cAdvisor.

scrape_configs:
- job_name: cAdvisor
  scrape_interval: 10s
  static_configs:
  - targets:
    - cAdvisor:8080

The content of this file defines a job where Prometheus will scrape data from cAdvisor every 10 seconds, using cAdvisor:8080 as defined in the docker-compose.yml file.

Next, start the services: