Prometheus + Grafana monitoring stack for Substrate-based blockchain nodes. Simple, unified configuration that works out of the box.
- 📊 Prometheus - Metrics collection and storage (60 days retention)
- 📈 Grafana - Metrics visualization with pre-configured dashboards
- 🖥️ Node Exporter - System metrics (CPU, RAM, Disk, Network)
- 🔒 Nginx Reverse Proxy - Prometheus protected with Basic Auth + Rate Limiting
- 🎯 Network Dashboards - Pre-configured dashboards for multiple blockchain networks
- 🎨 Quantus Branding - Custom logo, colors, and styling matching Quantus design
- ⚡ Single Setup - One configuration, works everywhere
# 1. Clone repository
git clone <your-repo-url>
cd monitoring
# 2. (Optional) Customize credentials, SMTP, Telegram, Rocket & alert emails
cp .env.example .env
nano .env # Set passwords, SMTP, TELEGRAM_*, ROCKET_WEBHOOK_URL, ALERT_EMAIL_ADDRESSES
# 3. Start the stack
docker compose up -d
# 4. Access services
open http://localhost:3000 # Grafana (login: admin / admin)
open http://localhost:9091 # Prometheus (admin / prometheus)That's it! 🎉
Notes:
- Grafana: Login required (
admin/adminby default). Anonymous access is disabled. - Public dashboards: Use Grafana’s built-in Public Dashboard share for selected boards only (see Public Dashboards). Explore, alerting, and other dashboards stay private.
- Prometheus: Secured with Basic Auth (
admin/prometheus)
- Grafana: http://localhost:3000 (login required)
- Prometheus: http://localhost:9091 (Basic Auth:
admin/prometheus) - Node Exporter: http://localhost:9100/metrics (metrics endpoint)
Grafana stays login-only (GF_AUTH_ANONYMOUS_ENABLED=false). To share a view without giving out credentials, use Grafana’s Public dashboard feature on the same instance (same host / Cloudflare Tunnel URL).
- Log in to Grafana and open the dashboard (e.g. Overview → Service Status).
- Click Share → Public dashboard.
- Enable the public link and copy the URL (
/public-dashboards/<accessToken>). - Share that URL. Visitors can view that dashboard only — they cannot open Explore, alerting, or other dashboards without logging in.
Public share state is stored in Grafana’s database (not in the provisioned JSON). Enabling it once is enough; the token persists across restarts.
| Safe to public-share | Keep private |
|---|---|
| Service Status (uptime / operational status only) | Infrastructure host boards (CPU, mem, disk, network, hostnames) |
| Selected Chain boards that only show public chain health (e.g. Chain Health), after review | Applications boards with balances, process internals, or endpoint inventories (Faucet, Explorer, Quests full boards) |
| Monitoring Stack, Support Host, and any board that exposes capacity or topology |
Do not public-share Quersi Host, Senoti Host, Subsquid Host, or other Infrastructure/Applications dashboards as-is.
The stack monitors:
- Prometheus - Self-monitoring (metrics collection system)
- Node Exporter - Docker host system metrics
- CPU usage and load averages
- Memory usage and availability
- Disk usage and I/O
- Network traffic (receive/transmit)
- System uptime
- Remote Blockchain Nodes - Heisenberg and Dirac networks
- Node metrics (system resources, peers, network I/O)
- Substrate metrics (block production, finalization)
- Mining metrics (hashrate, difficulty)
- Support Services - Telemetry and monitoring infrastructure
- Telemetry Host (qm-telemetry.quantus.cat) - VPS system metrics
- Telemetry Backend (feed-telemetry.quantus.cat) - Application metrics
- Connected nodes/feeds/shards
- Message rates and dropped messages
- Service availability
Edit prometheus/prometheus.yml to add your own node targets:
scrape_configs:
# Add your nodes here
- job_name: 'my-validator'
scrape_interval: 10s
static_configs:
- targets: ['validator1.example.com:9615']
labels:
instance: 'validator-1'
chain: 'polkadot'
role: 'validator'Reload Prometheus:
# With authentication
curl -u admin:prometheus -X POST http://localhost:9091/-/reloadCF_ACCESS_CLIENT_ID and CF_ACCESS_CLIENT_SECRET are required — docker compose will fail if either is missing or empty. Set them in .env:
CF_ACCESS_CLIENT_ID=your_client_id
CF_ACCESS_CLIENT_SECRET=your_client_secretOn start, Prometheus writes these into /etc/prometheus/secrets/. For any scrape job protected by Access, add the same http_headers block used on telemetry-host in prometheus/prometheus.yml:
http_headers:
CF-Access-Client-Id:
files:
- /etc/prometheus/secrets/cf_access_client_id
CF-Access-Client-Secret:
files:
- /etc/prometheus/secrets/cf_access_client_secretThen recreate Prometheus (so secrets are rewritten) and reload if you only changed the YAML:
docker compose up -d prometheus
curl -u admin:prometheus -X POST http://localhost:9091/-/reloadOptional - create .env from .env.example:
cp .env.example .envKey variables (see .env.example for the full list, including SMTP, Telegram, and Rocket):
# Grafana Configuration
GRAFANA_ADMIN_PASSWORD=admin
# Prometheus Basic Auth (via Nginx)
# Credentials are generated at nginx container startup
PROMETHEUS_USER=admin
PROMETHEUS_PASSWORD=prometheus
# Cloudflare Access service token (protected /metrics scrapes)
CF_ACCESS_CLIENT_ID=
CF_ACCESS_CLIENT_SECRET=
# Production alert routing (see Alert Routing below)
TELEGRAM_BOT_TOKEN=
TELEGRAM_CHAT_ID=
ROCKET_WEBHOOK_URL=Security Tip: For production, use strong credentials:
PROMETHEUS_USER=monitoring_$(openssl rand -hex 8)
PROMETHEUS_PASSWORD=$(openssl rand -base64 32)To enable email notifications in Grafana, configure SMTP settings in your .env file:
# SMTP Configuration for Grafana Email Notifications
SMTP_ENABLED=true
SMTP_HOST=smtp.example.com:587
SMTP_USER=your-email@example.com
SMTP_PASSWORD=your_smtp_password_here
SMTP_FROM_ADDRESS=your-email@example.com
SMTP_FROM_NAME=Grafana Monitoring
SMTP_STARTTLS_POLICY=MandatoryStartTLS
# Alert Email Addresses (comma-separated)
ALERT_EMAIL_ADDRESSES=admin@example.com, alerts@example.comNote: Copy .env.example to .env and update with your SMTP credentials and alert email addresses:
cp .env.example .env
nano .env # Edit SMTP settings and ALERT_EMAIL_ADDRESSESAfter configuring SMTP, recreate Grafana so it picks up the new env:
docker compose up -d grafanaTo test email notifications:
- Go to Grafana → Alerting → Contact points
- Click "New contact point"
- Select "Email" as the type
- Enter test email address
- Click "Test" to send a test email
Grafana has built-in Telegram support for the highest-priority business alert: No New Blocks (critical). Other critical alerts go to Email only; warnings go to Rocket.Chat.
Setup Steps:
1. Create a Telegram Bot:
# Open Telegram and message @BotFather
/newbot
# Follow the instructions
# You'll receive a bot token like: 123456789:ABCdefGHIjklMNOpqrsTUVwxyz2. Get your Chat ID:
# Send any message to your bot in Telegram
# Then visit this URL in your browser (replace <YOUR_BOT_TOKEN>):
https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates
# Look for "chat":{"id":123456789} in the JSON response
# The number is your Chat ID3. Add to your .env file:
# Telegram Configuration
TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrsTUVwxyz
TELEGRAM_CHAT_ID=1234567894. Recreate Grafana (reloads .env; use --build if you also changed contact-point files):
docker compose up -d --build grafanaMessage Format:
🚨 No New Blocks
Status: firing
Severity: critical
Chain: dirac
Instance: a1-qm-dirac.quantus.cat
📋 No new blocks on dirac for 7+ minutes
Check block production immediately
🔗 View in Grafana
To test:
- Go to Grafana → Alerting → Contact points
- Find "Telegram Notifications"
- Click "Test" to send a test message
Non-critical alerts (warnings and chain-matched non-critical routes) go to Rocket.Chat via an Incoming Webhook. Grafana uses a generic webhook contact point (not Slack) so Rocket’s {"success":true} response is not treated as a failure.
Grafana 11.3 posts a fixed JSON envelope (version, title, message, state, …). A script-less Rocket Incoming Webhook only builds chat content from text / msg, so you must enable a Rocket-side script that maps Grafana’s title and message into a Rocket attachment. Without that script you can get an empty channel message while Grafana still records HTTP 2xx success (and the long repeat_interval suppresses another send).
Setup Steps:
- In Rocket.Chat: Administration → Workspace → Integrations → Incoming Webhook.
- Set Post to Channel / Post as as needed.
- Turn Script Enabled on.
- Paste the contents of
grafana/rocket-incoming-webhook.script.jsinto the Script field and save. - Copy the webhook URL into your
.envfile:
# Rocket.Chat Incoming Webhook
ROCKET_WEBHOOK_URL=https://rocket.example.com/hooks/xxxx/yyyy- Recreate Grafana so it picks up the new env and baked-in contact-point config:
docker compose up -d --build grafanaMessage Format (attachment: title = Grafana title, body = Grafana message):
🚨 High CPU Usage — FIRING
Severity: warning
Chain: planck
Instance: example-host
📋 CPU usage high
…
🔗 [View in Grafana](…)
Resolved alerts use a green attachment (state: ok); firing uses red (state: alerting).
To test:
- Go to Grafana → Alerting → Contact points
- Find "Rocket Notifications"
- Click "Test" and confirm a non-empty message appears in the Rocket channel
- Optionally resolve a real warning alert and confirm the green resolved attachment
When both Telegram (TELEGRAM_BOT_TOKEN + TELEGRAM_CHAT_ID) and ROCKET_WEBHOOK_URL are set, Grafana loads production policies (policies.production.yml):
- 🔴 No New Blocks (critical) → Email + Telegram
- 🔴 Other critical → Email only
- 🟡 Warnings / non-critical → Rocket.Chat
- Default receiver → Rocket.Chat
- Dirac / Planck → 2 min
group_wait - Heisenberg → 10 min
group_wait
If either Telegram or Rocket is missing, Grafana falls back to email-only local policies (policies.local.yml). Contact points for whichever channels are configured are still provisioned, but routing only uses Email until both are set.
Alerts are configured via provisioning files in grafana/provisioning/alerting/:
Pre-configured Alerts:
Node Health:
- 🔴 Node Down - Triggers when a node is unreachable for 5+ minutes
- 🔴 No New Blocks - Fires when no new blocks have been produced for 7+ minutes (rule); first Telegram notification arrives ~10 min after the last block (7 min threshold + 1 min
for:+ ~2 mingroup_wait) - 🟡 Low Peer Count - Triggers when peer count drops below 3
System Resources:
- 🔴 Low Disk Space - Triggers when disk usage exceeds 85%
- 🟡 High CPU Usage - Triggers when CPU usage exceeds 80% for 15+ minutes
- 🟡 High Memory Usage - Triggers when memory usage exceeds 90%
Support Services:
- 🔴 Telemetry Host Down - Triggers when telemetry host is unreachable for 5+ minutes
Customizing Alert Email:
Alert email addresses are configured in your .env file. Edit the ALERT_EMAIL_ADDRESSES variable:
# Single email
ALERT_EMAIL_ADDRESSES=your-email@example.com
# Multiple emails (comma-separated)
ALERT_EMAIL_ADDRESSES=email1@example.com, email2@example.com, team@example.comAfter editing .env, rebuild and restart Grafana:
docker compose up -d --build grafanaAdding Custom Alerts:
Edit grafana/provisioning/alerting/rules.yml. Use the reduce + threshold pattern:
- uid: custom-alert
title: My Custom Alert
condition: C # Final threshold step
data:
# Step A: Prometheus query
- refId: A
datasourceUid: prometheus
model:
datasource:
type: prometheus
uid: prometheus
expr: your_prometheus_query_here
refId: A
instant: false
range: true
# Step B: Reduce to single value
- refId: B
datasourceUid: __expr__
model:
datasource:
type: __expr__
uid: __expr__
expression: A
reducer: last # or min, max, mean
refId: B
type: reduce
# Step C: Threshold comparison
- refId: C
datasourceUid: __expr__
model:
datasource:
type: __expr__
uid: __expr__
conditions:
- evaluator:
params: [threshold_value]
type: gt # gt (>), lt (<), eq (=)
operator:
type: and
query:
params: [C]
reducer:
params: []
type: last
type: query
expression: B
refId: C
type: threshold
for: 5m
annotations:
description: 'Alert description with {{ $value }}'
summary: 'Alert summary'
labels:
severity: warning # or critical
# Omit notification_settings so production/local notification policies choose the receiverAlert Notification Policies:
Policies are assembled at container start from policies.production.yml or policies.local.yml (see Alert Routing above). Production priorities:
| Network | Priority | First Notification | Repeat Interval |
|---|---|---|---|
| Dirac / Planck 🔴 | Highest | 2 minutes | once until resolved (8736h) |
| Heisenberg 🟡 | Medium | 10 minutes | once until resolved (8736h) |
Fallback by severity (if no chain label):
- Critical alerts (severity=critical): 10s wait, once until resolved
- Warning alerts (severity=warning): 30s wait → Rocket.Chat, once until resolved
After changing alert configuration (rules, contact points, or policies under grafana/provisioning/alerting/), rebuild and recreate Grafana so the image picks up the files:
docker compose up -d --build grafanaTroubleshooting Alert Provisioning:
If you see errors like UNIQUE constraint failed: alert_rule.guid, it means alerts were already created in Grafana UI and conflict with provisioned alerts. To fix:
# Option 1: Reset Grafana data (loses all UI changes)
docker compose down
docker volume rm monitoring_grafana-data
docker compose up -d
# Option 2: Change UIDs in rules.yml if you want to keep existing alerts
# Edit each alert's 'uid' field to a unique valueNote: With provisioning, manage alerts through YAML files instead of the UI. UI changes may conflict with provisioned configuration.
Place JSON dashboard files in grafana/dashboards/ directory. They will be automatically loaded on startup.
You can export dashboards from:
- Grafana Dashboard Repository
- Your existing Grafana instance
# All services
docker compose logs -f
# Specific service
docker compose logs -f prometheus
docker compose logs -f grafana# All services
docker compose restart
# Specific service
docker compose restart prometheus# Stop services
docker compose down
# Stop and remove data volumes (caution!)
docker compose down -vdocker compose pull
docker compose up -d- Prometheus data: Stored in Docker volume
prometheus-data(60 days retention, 30GB max) - Grafana data: Stored in Docker volume
grafana-data(dashboards, datasources, settings)
To backup:
# Backup Prometheus
docker run --rm -v monitoring_prometheus-data:/data -v $(pwd):/backup alpine tar czf /backup/prometheus-backup.tar.gz /data
# Backup Grafana
docker run --rm -v monitoring_grafana-data:/data -v $(pwd):/backup alpine tar czf /backup/grafana-backup.tar.gz /dataThe monitoring stack is fully customized with Quantus branding:
- Custom Logo: Quantus logo replaces default Grafana branding
- Custom Favicon: Quantus icon appears in browser tabs
- App Title: "Quantus Monitoring" instead of "Grafana"
- Login Subtitle: "Blockchain Network Monitoring"
The dashboards use Quantus color scheme:
- Blue (
#0000ff,#1f1fa3) - Healthy/OK state - Pink (
#ed4cce) - Warning state - Yellow (
#ffe91f) - Critical state - Dark Background (
#0c1014) - Main background
Last Block Time (seconds):
- 🔵 Blue (< 3 min) - Normal block production
- 🩷 Pink (3-10 min) - Slow block production
- 💛 Yellow (> 10 min) - Critical delay
Uptime (percentage over 30 days):
- 🔵 Blue (> 90%) - Excellent availability
- 🩷 Pink (50-90%) - Degraded service
- 💛 Yellow (< 50%) - Critical downtime
All branding assets are located in grafana/branding/:
grafana/branding/
├── logo.svg # Sidebar logo (SVG → grafana_icon.svg)
├── logo.png # Apple touch icon (180×180)
├── favicon.ico # Browser tab icon
├── fav32.png # 32×32 PNG for Grafana’s fav32 slot
├── quantus_login_dark.svg # Login background (dark theme)
├── quantus_login_light.svg # Login background (light theme)
└── quantus-favicon.svg # Optional source for regenerating raster iconsTo customize:
- Replace files in
grafana/branding/with your own - Rebuild the Grafana image (assets are baked in at build time):
docker compose up -d --build grafana - Hard refresh the browser (Ctrl+Shift+R / Cmd+Shift+R)
Branding is applied in grafana/Dockerfile via COPY into /usr/share/grafana/public/img/ (same idea as commit 53a13823). For the login background, docker-compose.yml also bind-mounts quantus_login_*.svg onto g8_login_*.svg.
Grafana serves /public/img/*.svg with long browser cache headers (Cache-Control: public, max-age=14400 and similar). Cloudflare will cache those responses (cf-cache-status: HIT). After you change login artwork or icons on the origin, visitors can still see the old g8_login_dark.svg until the edge cache expires or you purge cache for those URLs (or add a Cache Rule to bypass or shorten TTL for /public/img/*).
monitoring/
├── docker-compose.yml # Main configuration
├── prometheus/
│ └── prometheus.yml # Prometheus scrape configs
├── nginx/
│ ├── nginx.conf # Nginx reverse proxy config
│ ├── Dockerfile # Custom nginx image with htpasswd
│ └── docker-entrypoint.sh # Auth generation script
├── grafana/
│ ├── dashboards/ # Pre-loaded dashboards (by concern)
│ │ ├── overview/ # Home / multi-chain summary
│ │ ├── chains/ # Chain dashboards (chain selector)
│ │ ├── infrastructure/ # Hosts & telemetry
│ │ └── applications/ # Faucet, explorer, quests
│ ├── branding/ # Quantus branding assets
│ │ ├── logo.svg # Sidebar logo (SVG)
│ │ ├── logo.png # Apple touch icon
│ │ ├── favicon.ico # Favicon
│ │ └── fav32.png # 32×32 favicon PNG
│ ├── rocket-incoming-webhook.script.js # Paste into Rocket Incoming Webhook (Script Enabled)
│ └── provisioning/ # Auto-configuration
│ ├── datasources/ # Prometheus datasource
│ ├── dashboards/ # Dashboard providers
│ └── alerting/ # Alert templates (assembled at container start)
│ ├── rules.yml # Alert rules
│ ├── contactpoints.base.yml
│ ├── contactpoints.telegram.fragment.yml
│ ├── contactpoints.rocket.fragment.yml
│ ├── policies.local.yml # Email-only (local/testing)
│ └── policies.production.yml # Email / Telegram / Rocket routing
├── .env.example # Environment variables template
├── .gitignore
└── README.md
Dashboards are grouped by concern, not by network. Chain-specific views use a Chain dropdown (planck / heisenberg / dirac).
Quantus Network Overview — first page after login:
- Chain height, last block age, and uptime for Planck, Heisenberg, and Dirac
- Telemetry host status and connected nodes
- Refreshes every 10 seconds
Service Status — public-safe status for chains and support services (intended for Grafana Public Dashboard sharing):
- Chains: Planck / Heisenberg / Dirac (Chain 1–2 + Node 1–2 each)
- Quersi; Senoti units (App / DB / MQ / Watcher / Core); Explorer units (Indexer / API 1–2 / DB / Chain + sync); Faucet; Quests; Telemetry
- Explorer DB uses
max(up)across blue/green (only one active outside cutover; matches alerts) - Per-unit UP/DOWN, 30d availability %, and coarse success/error rates only — no host capacity, balances, or internal topology
All chain dashboards share a chain selector and link to each other via the Chains dropdown:
| Dashboard | What it covers |
|---|---|
| Chain Health | Height, block age, peers, syncing, difficulty, uptime |
| Consensus & Mining | Hashrate, difficulty, block time, mining duration (QPoW) |
| Node Operations | CPU/memory, block pipeline, trie cache, runtime performance |
| Network & Peers | P2P connections, bandwidth, Kademlia, sync peers |
| Transactions | TXPool activity and RPC sessions |
| Dashboard | What it covers |
|---|---|
| Monitoring Stack | Docker host running Prometheus/Grafana |
| Telemetry | Telemetry VPS host + backend message feeds |
| Support Host | Support server system metrics |
| Senoti Host | Senoti fleet system metrics |
| Subsquid Host | Subsquid fleet system metrics |
| Quersi Host | Quersi wallet remote-config system metrics |
| Dashboard | What it covers |
|---|---|
| Faucet | Request rates, transfers, balance, rejections |
| Explorer | Subsquid sync, RPC, Node.js performance |
| Quests | HTTP request rates, errors, latency |
Edit docker-compose.yml:
services:
prometheus:
command:
- '--storage.tsdb.retention.time=90d' # Change retention period
- '--storage.tsdb.retention.size=50GB' # Change max sizeBy default, services are accessible from localhost. To expose on your network, edit docker-compose.yml:
ports:
- "0.0.0.0:3000:3000" # Instead of "3000:3000"- Check target status: http://localhost:9091/targets (use Basic Auth)
- Verify target is accessible from Prometheus container
- Check Prometheus logs:
docker compose logs prometheus
This means rate limiting is too strict. Current settings allow 30 requests/second (burst 50), which should be enough. If you still see errors:
- Check nginx logs:
docker compose logs nginx - Adjust rate limits in
nginx/nginx.confif needed - Restart nginx:
docker compose restart nginx
Prometheus is protected with Basic Auth. Use credentials from .env:
# Default credentials
Username: admin
Password: prometheus
# Or check your .env file
cat .env | grep PROMETHEUS- Verify Prometheus datasource: Grafana → Configuration → Data Sources
- Check if Prometheus is scraping: http://localhost:9091/targets (use Basic Auth)
- Adjust time range in dashboard
On Linux, add to each service in docker-compose.yml:
extra_hosts:
- "host.docker.internal:host-gateway"If Node Exporter can't read system metrics, ensure proper volume mounts:
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/host:roThis stack includes built-in security (Nginx + Basic Auth + Rate Limiting). For production:
- ✅ Grafana login required - Anonymous access is disabled; dashboards, Explore, and alerting need credentials
- ✅ Prometheus Basic Auth - Already configured (change credentials in
.env) - ✅ Rate Limiting - 30 req/sec, prevents bruteforce attacks
⚠️ Strong Credentials - The compose defaults (admin/adminfor Grafana,prometheusandgrafanafallbacks) are for local dev only. Override them in.envbefore any production/internet-exposed deploy:GRAFANA_ADMIN_PASSWORD=$(openssl rand -base64 32) POSTGRES_PASSWORD=$(openssl rand -base64 32) PROMETHEUS_USER=monitoring_$(openssl rand -hex 8) PROMETHEUS_PASSWORD=$(openssl rand -base64 32)
⚠️ SSL/TLS - Use Cloudflare Tunnel or reverse proxy (Caddy, Traefik)⚠️ Firewall - Restrict ports or use VPN
# Prometheus is already secured with Basic Auth
# Add Cloudflare Tunnel for SSL + DDoS protection
# See: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/
# Your monitoring stays private, Cloudflare handles SSL- Increase retention if needed: Edit
docker-compose.ymlstorage settings - Setup backups for Docker volumes
- Monitor the monitoring - Set up alerting for stack availability
- Regular updates:
docker compose pull && docker compose up -d
# 1. Edit .env
nano .env # Change PROMETHEUS_USER and PROMETHEUS_PASSWORD
# 2. Restart nginx (generates new htpasswd)
docker compose restart nginx
# 3. Verify
curl -u newuser:newpass http://localhost:9091/Internet → Cloudflare (SSL/DDoS) → Nginx (Auth/Rate Limit) → Prometheus
Defense in Depth: Basic Auth + Rate Limiting + Cloudflare = Enterprise-grade security
- Docker
- Docker Compose
- 2GB+ RAM recommended
- ~30GB disk space for default retention settings
- Substrate
- Polkadot
- Kusama
- Any Substrate-based parachain
- Generic Prometheus metrics
See LICENSE file for details.
Issues and pull requests welcome!
For Substrate/Polkadot metrics documentation: