Monitoring with cAdvisor, Prometheus and Grafana on Docker

Introduction

Monitoring a system is crucial after deploying a product to a production environment. Keeping an eye on system metrics like logs, CPU, RAM, disks, etc, helps identify the system's status, performance issues, and provides timely solutions to ensure stable operations.

While cloud providers like Google, Amazon, or Azure offer built-in monitoring systems, if your company needs to manage multiple applications/systems/containers and desires a centralized monitoring system for easier management, using cAdvisor, Prometheus, and Grafana is a sensible choice. These three popular open-source tools are widely used by DevOps teams, especially for monitoring container applications.

cAdvisor

Developed by Google, cAdvisor is an open-source project used to analyze resource usage, performance, and other metrics from container applications, providing an overview of all running containers.

Find more details here


Prometheus

Prometheus is a toolkit for system monitoring and alerting based on system metrics. It supports organizing data over time, sending alerts via email, SMS, etc. Prometheus can integrate with various tools, including cAdvisor.

While cAdvisor fetches system information, Prometheus provides functionalities to manipulate that information such as visualizing data and triggering alerts.

Find more details here


Grafana

Grafana is a platform specialized in data visualization, supporting the creation of beautiful dashboards for web viewing, facilitating professional data analysis. Grafana can integrate with various data sources like MySQL, MongoDB, and Prometheus, etc.

So, the combination of cAdvisor, Prometheus, and Grafana works like this:

  1. cAdvisor gathers system info.
  2. Prometheus scrapes data from cAdvisor.
  3. Grafana is used to visualize data from Prometheus (while Prometheus has monitoring dashboards, Grafana excels in data visualization, hence the need for Grafana).


Setup Docker

First, let's create a docker-compose.yml file as follows:

services:
cAdvisor:
image: gcr.io/cadvisor/cadvisor
container_name: cAdvisor
ports:
- 8080:8080
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- /etc/machine-id:/etc/machine-id:ro
- /var/lib/dbus/machine-id:/var/lib/dbus/machine-id:ro
prometheus:
image: prom/prometheus
ports:
- 9090:9090
command:
- --config.file=/etc/prometheus/prometheus.yml
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
depends_on:
- cAdvisor
grafana:
image: grafana/grafana
ports:
- 3000:3000
restart: always
depends_on:
- prometheus

You can see that I’ve defined three services corresponding to cAdvisor, Prometheus, and Grafana. In this setup, Prometheus depends on cAdvisor, and Grafana depends on Prometheus.


Next, create a `prometheus.yml` file in the same location as your `docker-compose.yml`. This file will configure Prometheus to scrape data from cAdvisor.

scrape_configs:
- job_name: cAdvisor
scrape_interval: 10s
static_configs:
- targets:
- cAdvisor:8080

The content of this file defines a job where Prometheus will scrape data from cAdvisor every 10 seconds, using cAdvisor:8080 as defined in the docker-compose.yml file.


Next, start the services:

docker compose up -d


After successfully executing, you can access each service as follows:

cAdvisor

Prometheus


Grafana


However, the main platform you need to use is Grafana, so it's fine to just focus on learning how to use Grafana.

If you found this post helpful, please show your appreciation by sharing and commenting!

Comments

Post a Comment

Popular posts from this blog

Kubernetes Practice Series

Image
Introduction Kubernetes (often abbreviated as K8s ) is an open-source system designed to automate the deployment, scaling, and management of containerized applications. This page serves as a collection of articles related to Kubernetes (K8s) , including theoretical concepts and practical guides on using Kubernetes to set up essential tools for the software development process. I will continue to update this series with new articles as ideas come to mind, to ensure the series becomes more comprehensive. The articles are arranged in increasing order of difficulty to make it easier for you to follow. If you have time, it's recommended that you start from the beginning of the series to acquire the necessary knowledge and information that will prepare you for the subsequent articles. Key Topics Covered in This Series Basic Knowledge : Fundamental concepts, common commands, etc. Resources : Pod, Deployment, Service, StatefulSet, Ingress, etc. Pod Autoscaler : Horizontal vs. Vertical sc
Read more

NodeJS Practice Series

Image
Introduction NodeJS is an open-source and cross-platform JavaScript runtime environment . Here are some key points about NodeJS : V8 Engine : NodeJS runs on the V8 JavaScript engine , which is also the core of Google Chrome . This allows NodeJS to be highly performant. Asynchronous and Non-Blocking : NodeJS  uses an event-driven, non-blocking I/O model. It’s lightweight and efficient, making it ideal for data-intensive real-time applications. Single-Threaded : NodeJS  runs in a single process, handling multiple requests without creating new threads. It eliminates waiting and continues with the next request. Common Language : Frontend developers who write JavaScript for browsers can use the same language for server-side code in NodeJS . You can even use the latest ECMAScript standards without waiting for browser updates. This page is designed to compile articles related to NodeJS , including how to integrate it with various libraries and relevant tech stacks. I will continue to u
Read more

Docker Practice Series

Image
Introduction Docker is an open platform for developing, shipping, and running applications. This page is dedicated to compiling articles related to Docker , covering both the theoretical aspects and practical applications of Docker in setting up popular tools essential for software development. I will be updating this series with more articles in the future as new ideas for topics arise. The articles are arranged in increasing order of difficulty, making it easier for you to follow along. If you have the time, I recommend starting from the beginning of the series to ensure that you grasp the necessary knowledge and information needed for the more advanced articles later on. Here are some key topics in the series that you need to explore to effectively use Docker : Basic knowledge Docker commands Docker Compose Building Docker images Performance improvement Integration with CI/CD Once you have a solid understanding of the foundational knowledge, the extended topics (including advanced
Read more

React Practice Series

Image
Introduction React is a JavaScript library created by Facebook , often referred to as the most popular frontend framework today. This page aims to gather articles related to ReactJS , covering topics such as theory, features, and commonly used packages in the process of building ReactJS applications. I will update this series with more articles in the future as new ideas for content come up. The articles are arranged in increasing order of difficulty, so if you have time, it's recommended to start from the beginning of the series. This will ensure you grasp the essential knowledge and information needed for the subsequent articles. Here are some key topics in the series that you need to explore to effectively use ReactJS : Fundamental : React Hook, React Context, Lazy load, etc. State management : redux, mobx, recoil, etc. Middleware libraries : redux-thunk, redux-saga, redux-observable, etc. Popular packages : react-query, immer, styled-components, etc. Rendering techniques : C
Read more

Sitemap

Image
A page aggregator designed for easy reference and search, compiling all blog posts for convenient access.
Read more

Setting up Kubernetes Dashboard with Kind

Image
Introduction In a previous article, I guided you through using Helm to deploy on Google Kubernetes Engine . However, if you want to cut down costs by using Kubernetes in your local environment instead of relying on a cloud provider during development, then Kind is your go-to. There are several tools to help set up Kubernetes locally, such as MiniKube , Kind , K3S , KubeAdm , and more. Each tool has its own pros and cons. In this article, I'll walk you through using Kind to quickly set up a Kubernetes cluster on Docker . Kind stands out for its compactness, making Kubernetes start up quickly, being user-friendly, and supporting the latest Kubernetes versions. Working with Kind Firstly, follow the instructions here to install Kind according to your operating system. If you're using Ubuntu , execute the command: [ $( uname -m ) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.23.0/kind-linux-amd64 chmod +x ./kind sudo mv ./kind /usr/local/bin/ki
Read more

Explaining Async/Await in JavaScript in 10 Minutes

Image
Introduction For a long time, we've relied on callbacks to deal with asynchronous code in JavaScript. As a result, many of us have faced dreadful experiences dealing with deeply nested functions ( callback hell ). Callback hell Callbacks come with a lot of drawbacks. When we have multiple asynchronous operations, callbacks have to wait for each other to execute, prolonging the completion time. Additionally, writing nested callbacks makes our code messy and hard to maintain. In the ES6 (ECMAScript 2015) version, JavaScript introduced Promises. This was a fantastic replacement for callbacks and was quickly embraced by the community. Our new code looks similar to the old one, making it easier to follow and maintain. However, the issues with callbacks weren't completely resolved. Since the ES7 (ECMAScript 2016) version, Async/Await has been added to make asynchronous code in JavaScript look and feel easier to use. What is Async/Await? Async/Await is a feature of JavaScript that
Read more

Create API Gateway with fast-gateway

Image
Introduction In this article, I will guide you on how to use fast-gateway to deploy a simple API Gateway on NodeJS along with express . The advantage of an API Gateway is that it acts as an intermediary layer to hide the rest of the system, including services, commonly used in Microservices architecture. Example Usage This is the Microservices model after deployment: First, install the package yarn add fast-gateway Next, define the ports that will be used. import * as express from 'express' import * as gateway from 'fast-gateway' const portGateway = 5000 const portService1 = 5001 const portService2 = 5002 Define service 1 as follows: const startService1 = (): void => { const app = express () app . get ( '/list' , ( req , res ) => { const items = [ { id: 1 , name: 'service 1 value 1' , }, { id: 2 , name: 'service 1 value 2' , }, ] re
Read more

Deploying a NodeJS Server on Google Kubernetes Engine

Image
Introduction to GKE Google Kubernetes Engine (GKE) is a managed Kubernetes service provided by Google Cloud Platform , facilitating simple and efficient deployment of Docker images. We only need to provide some configuration for the number of nodes, machine types, and replicas to use. Some Concepts Cluster A Cluster is a collection of Nodes where Kubernetes can deploy applications. A cluster includes at least one Master Node and multiple Worker Nodes . The Master Node is used to manage the Worker Nodes . Node A Node is a server in the Kubernetes Cluster. Nodes can be physical servers or virtual machines. Each Node runs  Kubernetes , which is responsible for communication between the Master Node and Worker Node , as well as managing Pods and containers running on it. Pod A Pod is the smallest deployable unit in Kubernetes . Each Pod contains one or more containers, typically Docker containers. Containers in the same Pod share a network namespace, meaning they have the same
Read more

What is react-query? Why should we use react-query?

Image
Introduction You might have heard or seen others mention the react-query library quite often. Why should we use react-query when we can use useEffect to fetch data from an API ? This article will answer that question. What is react-query? react-query is a set of hooks for fetching, caching, and updating asynchronous data in React . Simply put, react-query is a library for fetching , caching , and updating asynchronous data when used in React . Why should we use react-query? Normally, there are many ways to fetch data from a server. Some popular methods include using useEffect and useState , or using a state management library like redux . export const Page = () => { const [ data , setData ] = useState (); useEffect (() => { const loadData = async () => { const response = await fetch ( hostAPI ). then ( response => response . json ()); setData ( response ) } loadData () }) return < DisplayData /> } This me
Read more