Server Monitoring using Prometheus and Grafana

Setting up Prometheus and Grafana in ubuntu using Docker

In today’s technology-driven world, servers play a pivotal role in ensuring the smooth functioning of businesses, organizations, and websites. However, the health and performance of servers can be fragile, leading to potential downtime, security breaches, and inefficiencies. That’s where server monitoring comes into play. In this blog, we will explore the importance of server monitoring and dive into the detailed setup process of two popular monitoring tools — Grafana and Prometheus.

Server monitoring is the process of tracking and analyzing server performance metrics to detect anomalies, identify bottlenecks, and proactively address issues.

Introduction to Grafana and Prometheus

In the rapidly evolving world of technology, system administrators, DevOps engineers, and IT teams face the constant challenge of ensuring the reliability and performance of their applications and infrastructure. This is where monitoring tools like Grafana and Prometheus step in to provide comprehensive solutions for tracking and visualizing critical metrics. In this introduction, we will explore the key features and functionalities of Grafana and Prometheus, two powerful tools that have become indispensable in the realm of monitoring and observability.

What is Grafana?

Grafana is an open-source analytics and visualization platform that empowers users to create customizable, real-time dashboards for monitoring and analyzing data from various sources. Originally designed for monitoring Graphite metrics, Grafana has evolved to support a wide range of data sources, including Prometheus, InfluxDB, Elasticsearch, and many others.

What is Prometheus?

Prometheus is an open-source monitoring and alerting toolkit initially developed at SoundCloud. It is designed to gather time-series metrics from various sources, store them efficiently, and offer a powerful querying language called PromQL for data retrieval and analysis.

Prometheus doesn’t use protocols such as SMNP or some sort of agent service. Instead, it pulls (scrapes) metrics from a client (target) over http and places the data into its local time series database that you can query using its own DSL.

Prometheus uses exporters that are installed and configured on the clients in order to convert and expose their metrics in a Prometheus format. The Prometheus server then scrapes the exporter for metrics.

By default Prometheus comes with a UI that can be accessed on port 9090 on the Prometheus server. Users also have the ability to build dashboards and integrate their favorite visualization software, such as Grafana.

Let’s recap some of the important terms :

• Prometheus Server : The main server that scrapes and stores the scraped metrics in a time series DB.
• Scrape : Prometheus server uses a pulling method to retrieve metrics.
• Target : The Prometheus servers clients that it retrieves info from.
• Exporter : Target libraries that convert and export existing metrics into Prometheus format.
• Alert Manager : Component responsible for handling alerts.

Setting up Prometheus, Node exporter, and grafana using docker-compose file

Step 1: Installing Docker

Before you install Docker Engine for the first time on a new host machine, you need to set up the Docker repository. Afterward, you can install and update Docker from the repository.

sudo apt update

Next, install a few prerequisite packages which let apt use packages over HTTPS:

sudo apt install apt-transport-https ca-certificates curl software-properties-common

Then add the GPG key for the official Docker repository to your system:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Add the Docker repository to APT sources:

sudo add-apt-repository "deb [arch=amd64]
https://download.docker.com/linux/ubuntu focal stable"

This will also update our package database with the Docker packages from the newly added repo.

Make sure you are about to install from the Docker repo instead of the default Ubuntu repo:

apt-cache policy docker-ce

Notice that docker-ce is not installed, but the candidate for installation is from the Docker.

Finally, install Docker:

sudo apt install docker-ce

Check the status of docker:

sudo systemctl status docker

Initially you won’t have any images and containers.

To view images:

sudo docker images

To view running containers:

sudo docker ps

Step 2: Installing Docker-Compose

We don’t need to do much while installing docker-compose. Just follow these steps:

sudo apt-get update -y
sudo apt-get install docker-compose -y

Check your docker-compose version after its installed:

sudo docker-compose --version

Step 3: Setting up prometheus and grafana:

After installing docker-compose we can easily setup prometheus and grafana using docker-compose.yml file. While we are doing this let’s try to monitor our host machine (server) as well. We need node_exporter for this.

Lets configure docker-compose.yml and start up our containers

sudo mkdir prometheus
cd prometheus
sudo nano docker-compose.yml

Now insert the following code into it:

version: '3.3'
volumes:
  prometheus-data:
    driver: local
  grafana-data:
    driver: local
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./config:/etc/prometheus/
      - prometheus-data:/prometheus
    networks:
      - prometheus-network
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    volumes:
      - grafana-data:/var/lib/grafana
    networks:
      - prometheus-network
    ports:
      - "3000:3000"
  node_exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: node_exporter
    command:
      - '--path.rootfs=/host'
    pid: host
    ports:
      - "9100:9100"
    restart: unless-stopped
    volumes:
      - '/:/host:ro,rslave'
    networks:
      - prometheus-network
networks:
  prometheus-network:
    driver: bridge

In our docker-compose file:

version: Version 3.3 is specified.

volumes: Two volumes prometheus-data and grafana-data is created locally. It is created automatically in directory /var/lib/docker/volumes/

services: Two services prometheus and grafana is created; image and container name is specified for them respectively.

restart: restart is set to unless stopped i.e. if the container was running before the reboot, the container would be restarted once the system restarted.

volumes:

For Prometheus: — ./config:/etc/prometheus/ : A folder named config (inside our current folder of host/server) is mapped with /etc/Prometheus (inside the container prometheus) — prometheus-data:/prometheus : Our previously created volume prometheus-data is mapped with /Prometheus (inside the container prometheus)

For grafana: — grafana-data:/var/lib/grafana : Our previously created volume grafana-data is mapped with /var/lib/grafana (inside the container grafana)

For node_exporter: — ‘/:/host:ro,rslave’ : Root directory of the host i.e. ‘/’ is mapped as ‘/host’ in readonly mode as a slave (inside the container node_exporter).

pid: pid of node_exporter is shared with host. This is done for process monitoring.

command: In case of node_exporter. Command ‘–path.rootfs=/host’ is specified and it is executed while our container starts . It is done for monitoring the root disk of our host.

networks: All containers reside in same network i.e. prometheus-network

ports: Port mapping is done from host port to container port.

networks: A network named prometheus-network is defined whose driver mode is bridge.

Also, we need a config file for Prometheus. So let’s create it.

mkdir config
sudo nano config/prometheus.yml

Copy the following code inside it:

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds.
  
  # scrape_timeout is set to the global default (10s).
  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['prometheus:9090']
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['node_exporter:9100']

Now lets start our containers:

docker-compose up --build –d

and check the status:

docker ps

Looks like everything is running fine. Now go to http://yourserverip:9090/ to access Prometheus:

Go to Status->targets.

Here we can see that both our targets prometheus and node_exporter is up. It means that prometheus can scrape data from these endpoints. However, we won’t be discussing about prometheus for now. We will be using grafana to display the metrics that is scraped by prometheus.

Let’s login to our grafana server.

Goto http://yourserverip:3000/

Default username and password is admin.

Grafana homepage

Now you have to create a Prometheus data source:

• Click on the Grafana logo to open the sidebar.
• Click on “Data Sources” in the connection sidebar.
• Choose “Add New”.
• Select “Prometheus” as the data source.
• Set the Prometheus server URL (in our case: http://prometheus:9090) This url works only because grafana and prometheus are in same network in docker.
• Click on Save and test.

Your settings should look like this:

You are now ready to create your first dashboard from the information collected by Prometheus. Now let’s setup our dashboard for host monitoring.

How to Import a Grafana Dashboard?

A dashboard displays statistics for the client node using a more effective and standardized layout. It is certainly possible to create a custom dashboard. However, Prometheus has already created a dashboard to support the Node Exporter statistics. The Node Exporter Full dashboard neatly graphs most of the values collected from the client nodes. It is much less work to import this premade dashboard than to create a custom one.

• Go to the Grafana dashboard. Select the Dashboard icon, consisting of four squares, and choose + Import.
• In the Import via grafana.com box, enter the ID 1860. Then select the Load button.
• This ID helps us to export full node_exporter dashboard from grafana dashboard library. You can browser more dashboards on https://grafana.com/grafana/dashboards/.
• Click on load.
• Set name, set the correct prometheus and click on import:

After importing you can see a dashboard. This dashboard is readymade dashboard and contains a lot of information.

Your dashboard has been loaded along with various metrics such as CPU utilization, memory utilization, network traffic utilization, etc. There are a lot of various metrics that you can explore and customize as you need.

That's it!! Its really easy to monitor your server using this method.