Collecting local system stats with Prometheus on Arch Linux

Install Prometheus

sudo pacman -S prometheus

Increase metrics retention time and listen only on loopback interface (the first ExecStart entry clears the existing ExecStart value):

sudo systemctl edit prometheus

[Service]
ExecStart=
ExecStart=/usr/bin/prometheus \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/var/lib/prometheus/data \
    --storage.tsdb.retention.time=1y \
    --web.listen-address=127.0.0.1:9090

Start Prometheus and confirm it's listening on the loopback interface only:

sudo systemctl start prometheus
sudo ss -lptun
sudo systemctl enable prometheus

Confirm the prometheus target is healthy: http://localhost:9090/targets

Install Grafana

sudo pacman -S grafana

Configure Grafana:

sudo vim /etc/grafana.ini

[server]
http_addr = 127.0.0.1
[auth.anonymous]
enabled = true
org_role = Editor

Start Grafana and confirm it's listening only on the loopback interface:

sudo systemctl start grafana
sudo ss -lptun
sudo systemctl enable grafana

Add Prometheus data source

Go to http://localhost:3000/login and log in with username admin and password admin. Change the password to keep it from nagging you every time you log in.

Go to Configuration -> Data Sources and click "Add data source". Choose "Prometheus" and use the default settings.

Go to Create -> Import and import Grafana.com dashboard ID "2" (Prometheus Stats). Select the Prometheus data source.

Confirm that stats appear under the new dashboard.

Install node exporter

The node exporter collects system metrics.

sudo pacman -S prometheus-node-exporter

Configure it to only listen on the loopback interface:

sudo vim /etc/conf.d/prometheus-node-exporter

NODE_EXPORTER_ARGS="--web.listen-address=127.0.0.1:9100"

Start the service and confirm it's only listening on the loopback interface:

sudo systemctl start prometheus-node-exporter
sudo ss -lptun
sudo systemctl enable prometheus-node-exporter

Go to http://localhost:9100/metrics and confirm that metrics appear.

Configure Prometheus to pull from the node exporter:

sudo vim /etc/prometheus/prometheus.yml

scrape_configs:
  - job_name: 'node'
    static_configs:
    - targets: ['localhost:9100']

Reload Prometheus to make it aware of the new target:

sudo systemctl reload prometheus

Go to http://localhost:9090/targets and confirm that the localhost target appears.

Add to Grafana

Go to Grafana and import a new dashboard (http://localhost:3000/dashboard/import). Use Grafana.com dashboard ID "1860" (Node Exporter Full). Select the "Prometheus" data source. Go to the new dashboard and confirm that the last 5 minutes of metrics look sane.

Install blackbox exporter

The blackbox exporter probes arbitrary network endpoints.

sudo pacman -S prometheus-blackbox-exporter

Edit the systemd unit to set a config file and only listen on the loopback interface:

sudo systemctl edit prometheus-blackbox-exporter

[Service]
ExecStart=
ExecStart=/usr/bin/prometheus-blackbox-exporter \
    --config.file="/etc/prometheus/blackbox.yml" \
    --web.listen-address=127.0.0.1:9115

Configure a generic IPv4 ICMP (ping) probe:

sudo vim /etc/prometheus/blackbox.yml

modules:
  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: "ip4"

sudo chown root:prometheus /etc/prometheus/blackbox.yml

Start the service and confirm it's only listening on the loopback interface:

sudo systemctl start prometheus-blackbox-exporter
sudo ss -lptun
sudo systemctl enable prometheus-blackbox-exporter

Confirm that it's working: * Go to http://localhost:9115/metrics and confirm that metrics about the blackbox exporter service are present. * Go to http://localhost:9115/probe?target=prometheus.io&module=icmp and confirm that the ICMP probe works. Afterward, it should appear in the "Recent Probes" section on http://localhost:9115/.

Configure blackbox exporter

Configure Prometheus to pull from the blackbox exporter. Replace the values in the targets list with appropriate values for your network setup. The gateway and next hop can be determined with mtr -b 8.8.8.8 (8.8.8.8 is Google's public DNS server, which is generally reliable).

sudo vim /etc/prometheus/prometheus.yml

scrape_configs:
  - job_name: 'blackbox_meta'
    static_configs:
      - targets: ['localhost:9115']
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets: ['8.8.8.8', '<<NEXT_HOP_IP>>', '<<YOUR_GATEWAY_IP>>', '127.0.0.1']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 'localhost:9115'

The blackbox_meta job collects metrics about the exporter itself. The blackbox job collects results from the ICMP probes.

The /probe endpoint requires two URL parameters:

• The module parameter is set to a fixed value of icmp in the params section. This matches the module name configured in /etc/prometheus/blackbox.yml.
• The target parameter is set in the relabel_configs section. Read the relabel_configs docs for full details on how it works:
  • When Prometheus collects metrics, it iterates over the targets listed in the static_configs section. __address__ is the address of the target currently being polled. The first relabel sets the target URL parameter to be this value (the label __param_NAME is passed as the URL parameter NAME).
  • The second relabel copies this value to the instance label so that we don't end up with all the metrics being attributed to localhost:9115.
  • The third relabel sets the __address__ to localhost:9115, since that's where the /probe endpoint is located. Without this, Prometheus would try to request metrics from the target (http://8.8.8.8/probe?target=8.8.8.8&module=icmp) rather than the blackbox exporter (http://localhost:9115/probe?target=8.8.8.8&module=icmp).

Reload Prometheus to start collecting metrics:

sudo systemctl reload prometheus

Go to http://localhost:9090/targets and confirm that the blackbox and blackbox_meta targets appear with all of the endpoints you configured above (in my configuration, there are a total of 5 endpoints across both).

Add to Grafana

Go to Grafana and import a new dashboard (http://localhost:3000/dashboard/import). Use Grafana.com dashboard ID "7587" (Prometheus Blackbox Exporter). Select the "Prometheus" data source. Go to the new dashboard and confirm that the last 5 minutes of metrics look sane.