Skip to content

Generic messaging

This scenario demonstrates a generic scenario of a monitoring service to collect and visualize IoT device metrics using Grafana and Prometheus.

Prerequisites

Architecture

architecture

Usage

Use Docker Compose in your local environment to build a monitoring service that collects and visualizes IoT device metrics. The service is defined in compose.yaml, and the configuration for each service is in the configs directory.

Service Description
Grafana Dashboard for visualizing metrics
Prometheus Monitoring service for collecting metrics
Node Exporter Agent for collecting node metrics
Alertmanager Monitoring service for alert notifications
Prometheus Client Agent for simulating virtual devices

Run the following commands to build the monitoring service in your local environment:

# Clone the repository and navigate to it
git clone https://github.com/ks6088ts-labs/iot-monitoring.git
cd iot-monitoring

# Build the monitoring service with Docker Compose
docker compose up -d

Access Grafana at http://localhost:3000 and log in. The default username and password are both admin. You will be prompted to change the password upon first login.

After logging in, open Dashboards > Services > Node Exporter Full from the Grafana home to view the node-exporter metrics.

node-exporter-full

Open Dashboards > Services > Prometheus Client to view the prometheus client metrics.

Prometheus Client

Customization

Notification Settings

Slack

Use Slack's Incoming Webhooks to send alerts. Obtain a Slack Webhook URL and set it in the slack_api_url field of alertmanager.yml.

Edit Alert Notifications

Alerts are defined in Alertmanager's alert_rules.yml with the following rules:

  • InstanceDown: Triggered when metrics are not collected for a certain period
  • APIHighRequestLatency: Triggered when API latency exceeds a threshold for a certain period

Refer to the Alertmanager Configuration documentation to edit alert rules.

Simulate Node Metrics Changes

To test alerts, change the Node Exporter metrics. Run the following commands to increase CPU usage and observe metric changes:

# Log in to the Node Exporter container
docker compose exec node-exporter sh

# Increase CPU load to change metrics
yes > /dev/null &

# Check the PID and kill the process to clean up
ps
kill <PID0>

Test Alerts

To trigger an InstanceDown alert, stop the Node Exporter with the following commands:

# Stop the Node Exporter
docker compose stop node-exporter

An alert will be triggered after a certain period.

Implement an Exporter

To collect metrics other than Node Exporter, implement an Exporter. An Exporter is an HTTP server that provides metrics to Prometheus.

prometheus-client container is an HTTP server that provides time-varying metrics.

References

Prometheus

Grafana