How to Install Prometheus on Ubuntu Server 22.04 LTS

[LHammonds]

Project Abandoned: Did not have a nice overview of everything like Nagios

WARNING - I am new to Prometheus and figuring it out and keeping notes while I do so.

To discuss this thread, please participate here: Ubuntu Forums

High-level overview

This thread will cover installation of a dedicated Ubuntu server, the Prometheus monitoring system and agents on various systems to monitor. This will replace my Nagios monitoring system. The server will be installed inside a virtual machine in vSphere running on ESXi servers. Notes will also be supplied for doing the same thing for VirtualBox on a Windows 10 PC.

* Install Prometheus (9090)
* Install Alert Manager (9093)
* Install Grafana (3000)
* Install Nginx (80)
* Configure Basic Authentication with Nginx
* Configure Nginx as reverse proxy for Prometheus services
* Install Node Exporter on Linux (9100)
* Install Node Exporter on Windows (9182)
* Monitor switches
* Dashboard Configuration

Tools utilized in this process

• Ubuntu Server 22.04 LTS, 64-bit
• Prometheus 2.31.2
• Portable PuTTY 0.77
• VMware vSphere 6.0.0
• VirtualBox 6.1.36

Helpful links

The list below are sources of information that was helpful in the creation of this document.

• Prometheus Documentation
• Grafana Documentation
• Install prometheus and grafana on ubuntu
• How To Install Grafana 8 on Ubuntu 22.04|20.04|18.04
• ?
• ?
• Ubuntu Documentation
• Ubuntu Firewall Basics

Assumptions

This documentation will need to make use of some very-specific information that will most-likely be different for each person / location. And as such, this information will be noted in this section. They will be highlighted in red throughout the document as a reminder that you should plug-in your own value rather than actually using these "place-holder" values.

Under no circumstance should you use the actual values listed below. They are place-holders for the real thing. This is just a checklist template you need to have answered before you start the install process.

Wherever you see RED in this document, you need to substitute it for you will use in your environment.

• Ubuntu Server name: srv-monitor
• Ubuntu Server IP address: 192.168.107.57
• Ubuntu Admin ID: administrator
• Ubuntu Admin Password: myadminpass

It is also assumed the reader knows how to use the VI editor. If not, you will need to beef up your skill set or use a different editor in place of it.

[LHammonds]

Install Prometheus from the repository:

Code: Select all

sudo apt -y install prometheus

Make sure it is running:

Code: Select all

systemctl status prometheus

The configuration file should be located here:

Code: Select all

sudo vi /etc/prometheus/prometheus.yaml

Code: Select all

# Sample config for Prometheus.

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'example'

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    scrape_timeout: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
      - targets: ['localhost:9090']

  - job_name: node
    # If prometheus-node-exporter is installed, grab stats about the local
    # machine by default.
    static_configs:
      - targets: ['localhost:9100']

If you modify the configuration file, be sure to validate the syntax with this command:

Code: Select all

promtool check config /etc/prometheus/prometheus.yml

Prometheus configuration file uses the YAML format. It strictly forbids tabs and requires 2 spaces for indentation otherwise Prometheus will fail to start with formatting errors so use a YAML validator to make sure the config has no formatting issues.

After validating the config file syntax, tell the service to reload its config file:

Code: Select all

sudo systemctl reload prometheus

Reset Prometheus Metrics

If you need to delete all metrics and start over (such as going from test to production), you and perform the following to empty the database of existing metrics collected:

Code: Select all

sudo systemctl stop prometheus
rm -rf /var/lib/prometheus/metrics2/*
sudo systemctl start prometheus

[LHammonds]

Nginx will serve as a reverse proxy and authenticator.

Enable port 80 in the firewall:

Code: Select all

sudo vi /var/scripts/prod/en-firewall.sh

Find this line and remove the comment at the beginning of the line and save the file:

Code: Select all

ufw allow proto tcp to any port 80 comment 'HTTP Service' 1>/dev/null 2>&1

Run the script to refresh the rules:

Code: Select all

sudo /var/scripts/prod/en-firewall.sh

Install Nginx:

Code: Select all

sudo apt -y install nginx

Make sure the service started and is active:

Code: Select all

systemctl status nginx

Open a web browser on another machine in the network, enter the IP of the server into the URL and make sure you see something like this:

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

Install the htpassword tool for basic authentication support:

Code: Select all

sudo apt -y install apache2-utils

Create credentials for a web user:

sudo htpasswd -c /etc/nginx/.htpasswd myadmin

Link Nginx to Prometheus:

Code: Select all

sudo cp /etc/nginx/sites-available/default /etc/nginx/sites-available/prometheus
sudo chown root:root /etc/nginx/sites-available/prometheus
sudo chmod 644 /etc/nginx/sites-available/prometheus
sudo vi /etc/nginx/sites-available/prometheus

Find this location block:

Code: Select all

  location / {
    # First attempt to serve request as file, then
    # as directory, then fall back to displaying a 404.
    try_files $uri $uri/ =404;
  }

Replace it with this block:

Code: Select all

  location / {
    auth_basic "Prometheus server authentication";
    auth_basic_user_file /etc/nginx/.htpasswd;
    proxy_pass http://localhost:9090;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
  }

Verify that the syntax is valid in the configuration files using this command:

Code: Select all

sudo nginx -t

Disable the default Nginx site by removing the link:

Code: Select all

sudo rm /etc/nginx/sites-enabled/default

Enable the Prometheus site by creating a link:

Code: Select all

sudo ln -s /etc/nginx/sites-available/prometheus /etc/nginx/sites-enabled/

Restart the Nginx service:

Code: Select all

sudo systemctl restart nginx

[LHammonds]

On the Prometheus server, install Alert Manager:

Code: Select all

sudo apt -y install prometheus-alertmanager

Edit the Alert Manager service configuration:

Code: Select all

sudo vi /etc/default/prometheus-alertmanager

Find this:

Code: Select all

ARGS=""

Change to:

Code: Select all

ARGS="--cluster.listen-address=0.0.0.0:9094 --web.listen-address=0.0.0.0:9093 --config.file=/etc/prometheus/alertmanager.yml --storage.path=/var/lib/prometheus/alertmanager/"

Edit the Alert Manager configuration:

Code: Select all

sudo vi /etc/prometheus/alertmanager.yml

Set the SMTP information to match your mail server:

Code: Select all

global:
  smtp_smarthost: '192.168.107.25:25'
  smtp_from: 'alertmanager@mydomain.com'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'password'

Restart the Alert Manager service:

Code: Select all

sudo systemctl restart prometheus-alertmanager

Create System Down Alert

On the Prometheus server, create a rules file that will be imported into the Prometheus configuration:

Code: Select all

sudo touch /etc/prometheus/rules-downtime.yml
sudo chown root:root /etc/prometheus/rules-downtime.yml
sudo chmod 644 /etc/prometheus/rules-downtime.yml
sudo cat<<EOF>>/etc/prometheus/rules-downtime.yml
groups:
 - name: Downtime
   rules:
   - alert: InstanceDown
     expr: up == 0
     for: 1m
EOF

Edit the Prometheus configuration file:

Code: Select all

sudo vi /etc/prometheus/prometheus.yml

Find this:

Code: Select all

rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

Change to this:

Code: Select all

rule_files:
 - "rules-downtime.yml"

Reload the Prometheus service:

Code: Select all

sudo systemctl reload prometheus

[LHammonds]

Obtain the missing required packages:

Code: Select all

sudo apt install -y apt-transport-https

Obtain the GPG Key:

Code: Select all

wget -O- https://packages.grafana.com/gpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

NOTE: This is the better way to manage 3rd-party keys to ensure trust is only given to their packages and not allow identically-named packages in the official repositories to be superseded.

Update repository and install Grafana:

Code: Select all

sudo apt update
sudo apt install -y grafana

Start the service and verify Grafana is active:

Code: Select all

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server

Configure the service to auto-start whenever the operating system boots up:

Code: Select all

sudo systemctl enable grafana-server.service

Enable port 3000 on the firewall:

Code: Select all

ufw allow proto tcp to any port 3000 comment 'Grafana' 1>/dev/null 2>&1

Use a web browser on your PC to access Grafana's URL:

Code: Select all

192.168.107.57:3000

When prompted for login credentials, enter the following:

• Login: admin
• Password: admin

You will then be prompted to change the admin password. Go ahead and set your secret admin password.

Importing Official Dashboards

You can create or modify your own dashboard but let's start by importing one of the many dashboards on the Grafana web site.

Login to your Grafana site: 192.168.107.57:3000
On the far left, select Dashboards -> Browse
Then click the "Import" button.
Type in the ID from the Grafana site (which is in the URL....13106 for example) and click the "Load" button. Click the "Prometheus" datasource and click the "Import" button.

Here are some of the commonly-used dashboards:

• MariaDB Cluster Overview (13106) (Requires: prometheus-mysqld-exporter)
• MySQL / MariaDB Workload (14621) (Requires: prometheus-mysqld-exporter)
• Apache Dashboard (3894) (Requires: prometheus-apache-exporter)
• Linux Dashboard (8919) (Requires: prometheus-node-exporter) (new / updated)
• Linux Dashboard (159) (Requires: prometheus-node-exporter) (simplistic / old)
• SNMP Stats Dashboard (11169) (Requires: prometheus-snmp-exporter)
• HAProxy2 Full (12693) (Requires: prometheus-haproxy-exporter)
• Website Monitoring (13041) (Requires: prometheus-blackbox-exporter)
• Windows Node (12422) (Requires: windows-exporter)

[LHammonds]

Install MySQL Exporter on Database server

Code: Select all

sudo apt install -y prometheus-mysqld-exporter

Create Prometheus database account

Code: Select all

sudo mysql

Code: Select all

CREATE USER 'prometheus-mysqld-exporter'@'localhost' IDENTIFIED BY 'YourStrongDBPasswordHere' WITH MAX_USER_CONNECTIONS 2;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'prometheus-mysqld-exporter'@'localhost';
FLUSH PRIVILEGES;
EXIT

Create database credential file:

Code: Select all

sudo touch /etc/prometheus-mysqld-exporter.cnf
sudo chown root:prometheus /etc/prometheus-mysqld-exporter.cnf
sudo chmod 640 /etc/prometheus-mysqld-exporter.cnf
sudo cat<<EOF>>/etc/prometheus-mysqld-exporter.cnf
[client]
user=prometheus-mysqld-exporter
password=YourStrongDBPasswordHere
EOF

Tell the exporter service to utilize the credential file:

Code: Select all

sudo vi /etc/default/prometheus-mysqld-exporter

Find this:

Code: Select all

ARGS=""

Change to:

Code: Select all

ARGS="--config.my-cnf /etc/prometheus-mysqld-exporter.cnf --web.listen-address=0.0.0.0:9104"

Restart the exporter service:

Code: Select all

sudo systemctl restart prometheus-mysqld-exporter

Modify the firewall to allow Prometheus to connect to the new port:

Code: Select all

ufw allow from 192.168.107.57 proto tcp to any port 9104 comment 'Prometheus' 1>/dev/null 2>&1

On the Prometheus server, edit the configuration file:

Code: Select all

sudo vi /etc/prometheus/prometheus.yml

Add the following and save:
NOTE: job_name needs to be unique among all jobs.

Code: Select all

  - job_name: srv-mariadb-db
    static_configs:
      - targets: ['192.168.107.20:9104']

Verify the configuration syntax is valid:

Code: Select all

promtool check config /etc/prometheus/prometheus.yml

Reload the Prometheus service configuration:

Code: Select all

sudo systemctl reload prometheus

Import Grafana dashboard for MySQL
You can create or modify your own dashboard but let's start by importing the MariaDB Cluster Overview and MySQL/MariaDB Workload dashboards.
Login to your Grafana site: 192.168.107.57:3000
On the far left, select Dashboards -> Browse
Then click the "Import" button.
Type in the ID from the Grafana site (which is in the URL....13106 in this case) and click the "Load" button. Click the "Prometheus" datasource and click the "Import" button.
On the far left, select Dashboards -> Browse
Then click the "Import" button.
Type in the ID from the Grafana site (which is in the URL....14621 in this case) and click the "Load" button. Click the "Prometheus" datasource and click the "Import" button.

Reference: Monitoring MySQL

[LHammonds]

Install Apache Exporter on the server running Apache2

Code: Select all

sudo apt install -y prometheus-apache-exporter

Edit the service configuration:

Code: Select all

sudo vi /etc/default/prometheus-apache-exporter

Find this:

Code: Select all

ARGS=''

Change to this:

Code: Select all

ARGS='-telemetry.address :9117'

Restart the service:

Code: Select all

sudo systemctl restart prometheus-apache-exporter

If you want to make sure the service is running under the port specified, you can verify with this command:

Code: Select all

ss -tunlp | grep prometheus

Configure the firewall to allow only Prometheus to connect over this port:

Code: Select all

ufw allow from 192.168.107.57 proto tcp to any port 9117 comment 'Prometheus' 1>/dev/null 2>&1

On the Prometheus server, edit the configuration file:

Code: Select all

sudo vi /etc/prometheus/prometheus.yml

Add the following and save:
NOTE: job_name needs to be unique among all jobs.

Code: Select all

  - job_name: srv-web-apache
    static_configs:
      - targets: ['192.168.107.20:9117']

Verify the configuration syntax is valid:

Code: Select all

promtool check config /etc/prometheus/prometheus.yml

Reload the Prometheus service configuration:

Code: Select all

sudo systemctl reload prometheus

Import Grafana dashboard for Apache

You can create or modify your own dashboard but let's start by importing an existing Apache dashboard.
Login to your Grafana site: 192.168.107.57:3000
On the far left, select Dashboards -> Browse
Then click the "Import" button.
Type 3894 in the ID from the Grafana site and click the "Load" button. Click the "Prometheus" datasource and click the "Import" button.

Reference: Monitoring Apache

[LHammonds]

Install Node Exporter on Linux server

Code: Select all

sudo apt install -y prometheus-node-exporter

Check to make sure the service is running / active:

Code: Select all

systemctl status prometheus-node-exporter

You can see what version you are running with this command:

Code: Select all

prometheus-node-exporter --version
node_exporter, version 0.18.1+ds (branch: debian/sid, revision: 0.18.1+ds-2)
  build user:       pkg-go-maintainers@lists.alioth.debian.org
  build date:       20200221-05:56:03
  go version:       go1.13.8

Firewall:

printf "Allowing Prometheus connections\n" ufw allow from 192.168.107.57 proto tcp to any port 9100 comment 'Prometheus' 1>/dev/null 2>&1

NOTE: This firewall rule only let's the Prometheus server to connect to the 9182 port which is much more secure than allowing any machine to connect to it.
Connect Prometheus server to Linux server.
On the Prometheus server, edit the configuration file:

Code: Select all

sudo vi /etc/prometheus/prometheus.yml

Add the following and save:
NOTE: job_name needs to be unique among all jobs.

Code: Select all

  - job_name: srv-linux1
    static_configs:
      - targets: ['192.168.107.50:9100']
        labels:
          alias: srv-linux1

Verify the configuration syntax is valid:

Code: Select all

promtool check config /etc/prometheus/prometheus.yml

Reload the Prometheus service configuration:

Code: Select all

sudo systemctl reload prometheus

Import Grafana dashboard for Linux

Reference: Monitor Linux Server

[LHammonds]

Install Node Exporter on Windows server

1. Download the latest release of Windows Exporter
   32-bit servers would be like windows_exporter-0.??.?-386.msi
   64-bit servers would be like windows_exporter-0.??.?-amd64.msi
2. Open a Command Prompt as Administrator.
3. At the Command Prompt, create a configuration folder:

   Code: Select all

   mkdir C:\prometheus

4. At the Command Prompt, open a new configuration file:

   Code: Select all

   notepad C:\prometheus\config.yml

5. Copy/Paste the following into the config file:

   Code: Select all

   ---
   # Note this is not an exhaustive list of all configuration values
   collectors:
     enabled: cpu,cs,logical_disk,net,os,service,system,textfile
   collector:
     service:
       services-where: Name='windows_exporter'
     scheduled_task:
       blacklist: /Microsoft/.+
   log:
     level: debug
   scrape:
     timeout-margin: 0.5
   telemetry:
     addr: ":9182"
     path: /metrics
     max-requests: 5
   

6. Double-click the downloaded installer and let it complete.
7. Modify the service so it uses the config file. At the Command Prompt, type the following:

   Code: Select all

   sc stop windows_exporter
   sc config windows_exporter binPath="C:\Program Files\windows_exporter\windows_exporter.exe --log.format logger:eventlog?name=windows_exporter --config.file=C:\prometheus\config.yml"
   sc start windows_exporter

8. At the Command Prompt, modify the firewall rules to only allow the Prometheus server to access port 9182 by typing the following:
   netsh advfirewall firewall add rule name="Prometheus 9182 TCP" dir=in action=allow localport=9182 remoteport=any protocol=tcp remoteip=192.168.107.57 profile=Domain,Public,Private

Connect Prometheus server to Windows server.
On the Prometheus server, edit the configuration file:

Code: Select all

sudo vi /etc/prometheus/prometheus.yml

Add the following and save:
NOTE: job_name needs to be unique among all jobs.

Code: Select all

  - job_name: srv-windows1
    static_configs:
      - targets: ['192.168.107.51:9182']
        labels:
          alias: srv-windows1

Verify the configuration syntax is valid:

Code: Select all

promtool check config /etc/prometheus/prometheus.yml

Reload the Prometheus service configuration:

Code: Select all

sudo systemctl reload prometheus

Import Grafana dashboard for Windows

[LHammonds]

SNMP is an abbreviation for Simple Network Management Protocol. Most switches, routers, modems, printers, servers and workstations support the SNMP protocol. How to enable and configure SNMP on various devices is not covered here. We will just assume each device being monitored has it enabled and configured already.

Install SNMP Exporter on the Prometheus server

Code: Select all

sudo apt install -y prometheus-snmp-exporter

Check to make sure the service is running / active:

Code: Select all

systemctl status prometheus-snmp-exporter

Since this is running on the Prometheus server, you will not need to add a firewall rule for port 9116 but if you installed it on a different machine, you would need to allow connections from the Prometheus server through the firewall.

Edit the SNMP Exporter configuration file:

Code: Select all

sudo vi /etc/default/prometheus-snmp-exporter

Find this:

Code: Select all

ARGS=""

Change it to this:

Code: Select all

ARGS="--web.listen-address=":9116""

Restart the service and make sure it is active:

Code: Select all

sudo systemctl restart prometheus-snmp-exporter
systemctl status prometheus-snmp-exporter

If you want to make sure the service is running under the port specified, you can verify with this command:

Code: Select all

ss -tunlp | grep prometheus

On the Prometheus server, edit the configuration file:

Code: Select all

sudo vi /etc/prometheus/prometheus.yml

Add the following and save...assuming the smnp device to monitor is 192.168.107.1:
NOTE: job_name needs to be unique among all jobs.

Code: Select all

  - job_name: sw-snmp
    static_configs:
      - targets:
        - 192.168.107.1
    metrics_path: /snmp
    params:
      module: [if_mib]
    relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target__]
      target_label: instance
    - target_label: __address__
      replacement: localhost:9116

Verify the configuration syntax is valid:

Code: Select all

promtool check config /etc/prometheus/prometheus.yml

Reload the Prometheus service configuration:

Code: Select all

sudo systemctl reload prometheus

Import Grafana dashboard for monitoring SNMP devices.
You can create or modify your own dashboard but let's start by importing an existing SNMP Stats dashboard.
Login to your Grafana site: 192.168.107.57:3000
On the far left, select Dashboards -> Browse
Then click the "Import" button.
Type 11169 in the ID from the Grafana site and click the "Load" button. Click the "Prometheus" datasource and click the "Import" button.