How to Install Prometheus on Ubuntu Server 22.04 LTS

Post Reply
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1098
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

How to Install Prometheus on Ubuntu Server 22.04 LTS

Post: # 1079Post LHammonds »

Project Abandoned: Did not have a nice overview of everything like Nagios

WARNING - I am new to Prometheus and figuring it out and keeping notes while I do so.

To discuss this thread, please participate here: Ubuntu Forums

High-level overview

This thread will cover installation of a dedicated Ubuntu server, the Prometheus monitoring system and agents on various systems to monitor. This will replace my Nagios monitoring system. The server will be installed inside a virtual machine in vSphere running on ESXi servers. Notes will also be supplied for doing the same thing for VirtualBox on a Windows 10 PC.

* Install Prometheus (9090)
* Install Alert Manager (9093)
* Install Grafana (3000)
* Install Nginx (80)
* Configure Basic Authentication with Nginx
* Configure Nginx as reverse proxy for Prometheus services
* Install Node Exporter on Linux (9100)
* Install Node Exporter on Windows (9182)
* Monitor switches
* Dashboard Configuration

Tools utilized in this process
Helpful links

The list below are sources of information that was helpful in the creation of this document.
Assumptions

This documentation will need to make use of some very-specific information that will most-likely be different for each person / location. And as such, this information will be noted in this section. They will be highlighted in red throughout the document as a reminder that you should plug-in your own value rather than actually using these "place-holder" values.

Under no circumstance should you use the actual values listed below. They are place-holders for the real thing. This is just a checklist template you need to have answered before you start the install process.

Wherever you see RED in this document, you need to substitute it for you will use in your environment.
  • Ubuntu Server name: srv-monitor
  • Ubuntu Server IP address: 192.168.107.57
  • Ubuntu Admin ID: administrator
  • Ubuntu Admin Password: myadminpass
It is also assumed the reader knows how to use the VI editor. If not, you will need to beef up your skill set or use a different editor in place of it.
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1098
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Install Prometheus

Post: # 1080Post LHammonds »

Install Prometheus from the repository:

Code: Select all

sudo apt -y install prometheus
Make sure it is running:

Code: Select all

systemctl status prometheus
The configuration file should be located here:

Code: Select all

sudo vi /etc/prometheus/prometheus.yaml

Code: Select all

# Sample config for Prometheus.

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'example'

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    scrape_timeout: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
      - targets: ['localhost:9090']

  - job_name: node
    # If prometheus-node-exporter is installed, grab stats about the local
    # machine by default.
    static_configs:
      - targets: ['localhost:9100']
If you modify the configuration file, be sure to validate the syntax with this command:

Code: Select all

promtool check config /etc/prometheus/prometheus.yml
Prometheus configuration file uses the YAML format. It strictly forbids tabs and requires 2 spaces for indentation otherwise Prometheus will fail to start with formatting errors so use a YAML validator to make sure the config has no formatting issues.

After validating the config file syntax, tell the service to reload its config file:

Code: Select all

sudo systemctl reload prometheus
Reset Prometheus Metrics

If you need to delete all metrics and start over (such as going from test to production), you and perform the following to empty the database of existing metrics collected:

Code: Select all

sudo systemctl stop prometheus
rm -rf /var/lib/prometheus/metrics2/*
sudo systemctl start prometheus
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1098
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Install Nginx

Post: # 1081Post LHammonds »

Nginx will serve as a reverse proxy and authenticator.

Enable port 80 in the firewall:

Code: Select all

sudo vi /var/scripts/prod/en-firewall.sh
Find this line and remove the comment at the beginning of the line and save the file:

Code: Select all

ufw allow proto tcp to any port 80 comment 'HTTP Service' 1>/dev/null 2>&1
Run the script to refresh the rules:

Code: Select all

sudo /var/scripts/prod/en-firewall.sh
Install Nginx:

Code: Select all

sudo apt -y install nginx
Make sure the service started and is active:

Code: Select all

systemctl status nginx
Open a web browser on another machine in the network, enter the IP of the server into the URL and make sure you see something like this:
Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.
Install the htpassword tool for basic authentication support:

Code: Select all

sudo apt -y install apache2-utils
Create credentials for a web user:
sudo htpasswd -c /etc/nginx/.htpasswd myadmin
Link Nginx to Prometheus:

Code: Select all

sudo cp /etc/nginx/sites-available/default /etc/nginx/sites-available/prometheus
sudo chown root:root /etc/nginx/sites-available/prometheus
sudo chmod 644 /etc/nginx/sites-available/prometheus
sudo vi /etc/nginx/sites-available/prometheus
Find this location block:

Code: Select all

  location / {
    # First attempt to serve request as file, then
    # as directory, then fall back to displaying a 404.
    try_files $uri $uri/ =404;
  }
Replace it with this block:

Code: Select all

  location / {
    auth_basic "Prometheus server authentication";
    auth_basic_user_file /etc/nginx/.htpasswd;
    proxy_pass http://localhost:9090;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
  }
Verify that the syntax is valid in the configuration files using this command:

Code: Select all

sudo nginx -t
Disable the default Nginx site by removing the link:

Code: Select all

sudo rm /etc/nginx/sites-enabled/default
Enable the Prometheus site by creating a link:

Code: Select all

sudo ln -s /etc/nginx/sites-available/prometheus /etc/nginx/sites-enabled/
Restart the Nginx service:

Code: Select all

sudo systemctl restart nginx
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1098
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Install Alert Manager

Post: # 1082Post LHammonds »

On the Prometheus server, install Alert Manager:

Code: Select all

sudo apt -y install prometheus-alertmanager
Edit the Alert Manager service configuration:

Code: Select all

sudo vi /etc/default/prometheus-alertmanager
Find this:

Code: Select all

ARGS=""
Change to:

Code: Select all

ARGS="--cluster.listen-address=0.0.0.0:9094 --web.listen-address=0.0.0.0:9093 --config.file=/etc/prometheus/alertmanager.yml --storage.path=/var/lib/prometheus/alertmanager/"
Edit the Alert Manager configuration:

Code: Select all

sudo vi /etc/prometheus/alertmanager.yml
Set the SMTP information to match your mail server:

Code: Select all

global:
  smtp_smarthost: '192.168.107.25:25'
  smtp_from: 'alertmanager@mydomain.com'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'password'
Restart the Alert Manager service:

Code: Select all

sudo systemctl restart prometheus-alertmanager
Create System Down Alert

On the Prometheus server, create a rules file that will be imported into the Prometheus configuration:

Code: Select all

sudo touch /etc/prometheus/rules-downtime.yml
sudo chown root:root /etc/prometheus/rules-downtime.yml
sudo chmod 644 /etc/prometheus/rules-downtime.yml
sudo cat<<EOF>>/etc/prometheus/rules-downtime.yml
groups:
 - name: Downtime
   rules:
   - alert: InstanceDown
     expr: up == 0
     for: 1m
EOF
Edit the Prometheus configuration file:

Code: Select all

sudo vi /etc/prometheus/prometheus.yml
Find this:

Code: Select all

rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
Change to this:

Code: Select all

rule_files:
 - "rules-downtime.yml"
Reload the Prometheus service:

Code: Select all

sudo systemctl reload prometheus
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1098
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Install Grafana

Post: # 1083Post LHammonds »

Obtain the missing required packages:

Code: Select all

sudo apt install -y apt-transport-https
Obtain the GPG Key:

Code: Select all

wget -O- https://packages.grafana.com/gpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
NOTE: This is the better way to manage 3rd-party keys to ensure trust is only given to their packages and not allow identically-named packages in the official repositories to be superseded.

Update repository and install Grafana:

Code: Select all

sudo apt update
sudo apt install -y grafana
Start the service and verify Grafana is active:

Code: Select all

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server
Configure the service to auto-start whenever the operating system boots up:

Code: Select all

sudo systemctl enable grafana-server.service
Enable port 3000 on the firewall:

Code: Select all

ufw allow proto tcp to any port 3000 comment 'Grafana' 1>/dev/null 2>&1
Use a web browser on your PC to access Grafana's URL:

Code: Select all

192.168.107.57:3000
When prompted for login credentials, enter the following:
  • Login: admin
  • Password: admin
You will then be prompted to change the admin password. Go ahead and set your secret admin password.

Importing Official Dashboards

You can create or modify your own dashboard but let's start by importing one of the many dashboards on the Grafana web site.

Login to your Grafana site: 192.168.107.57:3000
On the far left, select Dashboards -> Browse
Then click the "Import" button.
Type in the ID from the Grafana site (which is in the URL....13106 for example) and click the "Load" button. Click the "Prometheus" datasource and click the "Import" button.

Here are some of the commonly-used dashboards:
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1098
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Monitor MariaDB/MySQL

Post: # 1084Post LHammonds »

Install MySQL Exporter on Database server

Code: Select all

sudo apt install -y prometheus-mysqld-exporter
Create Prometheus database account

Code: Select all

sudo mysql

Code: Select all

CREATE USER 'prometheus-mysqld-exporter'@'localhost' IDENTIFIED BY 'YourStrongDBPasswordHere' WITH MAX_USER_CONNECTIONS 2;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'prometheus-mysqld-exporter'@'localhost';
FLUSH PRIVILEGES;
EXIT
Create database credential file:

Code: Select all

sudo touch /etc/prometheus-mysqld-exporter.cnf
sudo chown root:prometheus /etc/prometheus-mysqld-exporter.cnf
sudo chmod 640 /etc/prometheus-mysqld-exporter.cnf
sudo cat<<EOF>>/etc/prometheus-mysqld-exporter.cnf
[client]
user=prometheus-mysqld-exporter
password=YourStrongDBPasswordHere
EOF
Tell the exporter service to utilize the credential file:

Code: Select all

sudo vi /etc/default/prometheus-mysqld-exporter
Find this:

Code: Select all

ARGS=""
Change to:

Code: Select all

ARGS="--config.my-cnf /etc/prometheus-mysqld-exporter.cnf --web.listen-address=0.0.0.0:9104"
Restart the exporter service:

Code: Select all

sudo systemctl restart prometheus-mysqld-exporter
Modify the firewall to allow Prometheus to connect to the new port:

Code: Select all

ufw allow from 192.168.107.57 proto tcp to any port 9104 comment 'Prometheus' 1>/dev/null 2>&1
On the Prometheus server, edit the configuration file:

Code: Select all

sudo vi /etc/prometheus/prometheus.yml
Add the following and save:
NOTE: job_name needs to be unique among all jobs.

Code: Select all

  - job_name: srv-mariadb-db
    static_configs:
      - targets: ['192.168.107.20:9104']
Verify the configuration syntax is valid:

Code: Select all

promtool check config /etc/prometheus/prometheus.yml
Reload the Prometheus service configuration:

Code: Select all

sudo systemctl reload prometheus
Import Grafana dashboard for MySQL
You can create or modify your own dashboard but let's start by importing the MariaDB Cluster Overview and MySQL/MariaDB Workload dashboards.
Login to your Grafana site: 192.168.107.57:3000
On the far left, select Dashboards -> Browse
Then click the "Import" button.
Type in the ID from the Grafana site (which is in the URL....13106 in this case) and click the "Load" button. Click the "Prometheus" datasource and click the "Import" button.
On the far left, select Dashboards -> Browse
Then click the "Import" button.
Type in the ID from the Grafana site (which is in the URL....14621 in this case) and click the "Load" button. Click the "Prometheus" datasource and click the "Import" button.

Reference: Monitoring MySQL
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1098
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Monitor Apache Web Server

Post: # 1085Post LHammonds »

Install Apache Exporter on the server running Apache2

Code: Select all

sudo apt install -y prometheus-apache-exporter
Edit the service configuration:

Code: Select all

sudo vi /etc/default/prometheus-apache-exporter
Find this:

Code: Select all

ARGS=''
Change to this:

Code: Select all

ARGS='-telemetry.address :9117'
Restart the service:

Code: Select all

sudo systemctl restart prometheus-apache-exporter
If you want to make sure the service is running under the port specified, you can verify with this command:

Code: Select all

ss -tunlp | grep prometheus
Configure the firewall to allow only Prometheus to connect over this port:

Code: Select all

ufw allow from 192.168.107.57 proto tcp to any port 9117 comment 'Prometheus' 1>/dev/null 2>&1
On the Prometheus server, edit the configuration file:

Code: Select all

sudo vi /etc/prometheus/prometheus.yml
Add the following and save:
NOTE: job_name needs to be unique among all jobs.

Code: Select all

  - job_name: srv-web-apache
    static_configs:
      - targets: ['192.168.107.20:9117']
Verify the configuration syntax is valid:

Code: Select all

promtool check config /etc/prometheus/prometheus.yml
Reload the Prometheus service configuration:

Code: Select all

sudo systemctl reload prometheus
Import Grafana dashboard for Apache

You can create or modify your own dashboard but let's start by importing an existing Apache dashboard.
Login to your Grafana site: 192.168.107.57:3000
On the far left, select Dashboards -> Browse
Then click the "Import" button.
Type 3894 in the ID from the Grafana site and click the "Load" button. Click the "Prometheus" datasource and click the "Import" button.

Reference: Monitoring Apache
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1098
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Monitor Linux Server

Post: # 1086Post LHammonds »

Install Node Exporter on Linux server

Code: Select all

sudo apt install -y prometheus-node-exporter
Check to make sure the service is running / active:

Code: Select all

systemctl status prometheus-node-exporter
You can see what version you are running with this command:

Code: Select all

prometheus-node-exporter --version
node_exporter, version 0.18.1+ds (branch: debian/sid, revision: 0.18.1+ds-2)
  build user:       pkg-go-maintainers@lists.alioth.debian.org
  build date:       20200221-05:56:03
  go version:       go1.13.8
Firewall:
printf "Allowing Prometheus connections\n" ufw allow from 192.168.107.57 proto tcp to any port 9100 comment 'Prometheus' 1>/dev/null 2>&1
NOTE: This firewall rule only let's the Prometheus server to connect to the 9182 port which is much more secure than allowing any machine to connect to it.
Connect Prometheus server to Linux server.
On the Prometheus server, edit the configuration file:

Code: Select all

sudo vi /etc/prometheus/prometheus.yml
Add the following and save:
NOTE: job_name needs to be unique among all jobs.

Code: Select all

  - job_name: srv-linux1
    static_configs:
      - targets: ['192.168.107.50:9100']
        labels:
          alias: srv-linux1
Verify the configuration syntax is valid:

Code: Select all

promtool check config /etc/prometheus/prometheus.yml
Reload the Prometheus service configuration:

Code: Select all

sudo systemctl reload prometheus
Import Grafana dashboard for Linux

Reference: Monitor Linux Server
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1098
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Monitor Windows Server

Post: # 1087Post LHammonds »

Install Node Exporter on Windows server
  1. Download the latest release of Windows Exporter
    32-bit servers would be like windows_exporter-0.??.?-386.msi
    64-bit servers would be like windows_exporter-0.??.?-amd64.msi
  2. Open a Command Prompt as Administrator.
  3. At the Command Prompt, create a configuration folder:

    Code: Select all

    mkdir C:\prometheus
  4. At the Command Prompt, open a new configuration file:

    Code: Select all

    notepad C:\prometheus\config.yml
  5. Copy/Paste the following into the config file:

    Code: Select all

    ---
    # Note this is not an exhaustive list of all configuration values
    collectors:
      enabled: cpu,cs,logical_disk,net,os,service,system,textfile
    collector:
      service:
        services-where: Name='windows_exporter'
      scheduled_task:
        blacklist: /Microsoft/.+
    log:
      level: debug
    scrape:
      timeout-margin: 0.5
    telemetry:
      addr: ":9182"
      path: /metrics
      max-requests: 5
    
  6. Double-click the downloaded installer and let it complete.
  7. Modify the service so it uses the config file. At the Command Prompt, type the following:

    Code: Select all

    sc stop windows_exporter
    sc config windows_exporter binPath="C:\Program Files\windows_exporter\windows_exporter.exe --log.format logger:eventlog?name=windows_exporter --config.file=C:\prometheus\config.yml"
    sc start windows_exporter
  8. At the Command Prompt, modify the firewall rules to only allow the Prometheus server to access port 9182 by typing the following:
    netsh advfirewall firewall add rule name="Prometheus 9182 TCP" dir=in action=allow localport=9182 remoteport=any protocol=tcp remoteip=192.168.107.57 profile=Domain,Public,Private
Connect Prometheus server to Windows server.
On the Prometheus server, edit the configuration file:

Code: Select all

sudo vi /etc/prometheus/prometheus.yml
Add the following and save:
NOTE: job_name needs to be unique among all jobs.

Code: Select all

  - job_name: srv-windows1
    static_configs:
      - targets: ['192.168.107.51:9182']
        labels:
          alias: srv-windows1
Verify the configuration syntax is valid:

Code: Select all

promtool check config /etc/prometheus/prometheus.yml
Reload the Prometheus service configuration:

Code: Select all

sudo systemctl reload prometheus
Import Grafana dashboard for Windows
User avatar
LHammonds
Site Admin
Site Admin
Posts: 1098
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Monitor SNMP Devices

Post: # 1088Post LHammonds »

SNMP is an abbreviation for Simple Network Management Protocol. Most switches, routers, modems, printers, servers and workstations support the SNMP protocol. How to enable and configure SNMP on various devices is not covered here. We will just assume each device being monitored has it enabled and configured already.

Install SNMP Exporter on the Prometheus server

Code: Select all

sudo apt install -y prometheus-snmp-exporter
Check to make sure the service is running / active:

Code: Select all

systemctl status prometheus-snmp-exporter
Since this is running on the Prometheus server, you will not need to add a firewall rule for port 9116 but if you installed it on a different machine, you would need to allow connections from the Prometheus server through the firewall.

Edit the SNMP Exporter configuration file:

Code: Select all

sudo vi /etc/default/prometheus-snmp-exporter
Find this:

Code: Select all

ARGS=""
Change it to this:

Code: Select all

ARGS="--web.listen-address=":9116""
Restart the service and make sure it is active:

Code: Select all

sudo systemctl restart prometheus-snmp-exporter
systemctl status prometheus-snmp-exporter
If you want to make sure the service is running under the port specified, you can verify with this command:

Code: Select all

ss -tunlp | grep prometheus
On the Prometheus server, edit the configuration file:

Code: Select all

sudo vi /etc/prometheus/prometheus.yml
Add the following and save...assuming the smnp device to monitor is 192.168.107.1:
NOTE: job_name needs to be unique among all jobs.

Code: Select all

  - job_name: sw-snmp
    static_configs:
      - targets:
        - 192.168.107.1
    metrics_path: /snmp
    params:
      module: [if_mib]
    relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target__]
      target_label: instance
    - target_label: __address__
      replacement: localhost:9116
Verify the configuration syntax is valid:

Code: Select all

promtool check config /etc/prometheus/prometheus.yml
Reload the Prometheus service configuration:

Code: Select all

sudo systemctl reload prometheus
Import Grafana dashboard for monitoring SNMP devices.
You can create or modify your own dashboard but let's start by importing an existing SNMP Stats dashboard.
Login to your Grafana site: 192.168.107.57:3000
On the far left, select Dashboards -> Browse
Then click the "Import" button.
Type 11169 in the ID from the Grafana site and click the "Load" button. Click the "Prometheus" datasource and click the "Import" button.
Post Reply