diff --git a/servers/fr1.md b/servers/fr1.md index ab9a478..8a35706 100644 --- a/servers/fr1.md +++ b/servers/fr1.md @@ -108,7 +108,7 @@ backend=systemd banaction=ufw - Стек: **Prometheus + Grafana + node-exporter + cAdvisor + Alertmanager + Pushgateway** (docker compose). - **Алерты:** Alertmanager → Telegram (бот @ultimate_log_bot, chat 607015). Правила `prometheus/rules/alerts.yml`: TargetDown, HostHighCPU, HostLowMemory, HostDiskLow/Critical; группа `rpi` (с 2026-10-11) — SMART и бэкапы малины: PushMetricsStale, SmartHealthFailed, SmartBadSectors(+Growing), SmartUdmaCrcErrors, SmartSelftestFailed, SmartTemperatureHigh, BackupFailed, BackupStale. Токен — в `~/monitoring/alertmanager/telegram_token` (chown 65534, не в git). - Prometheus: retention 90d, слушает `127.0.0.1:9090`. Job'ы: `node` и `cadvisor` (file_sd, instance'ы **fr1 и ru1**), `openvpn` (file_sd, только ru1 — `85.198.109.86:9176`), `gitea` (static, `gitea:3000` через внешнюю docker-сеть `monitoring_monitoring`), `prometheus` (self). На 2026-09-08 все таргеты up, активных алертов нет. -- Grafana: `127.0.0.1:3000`, доступ снаружи через **https://mon.ruzzy.dev** (nginx + Let's Encrypt). Datasource Prometheus (uid `prometheus`) и дашборды (Node Exporter Full, cAdvisor, Network/Traffic, Gitea, OpenVPN — папка Infra) провижатся автоматически. Пароль admin — в `~/monitoring/docker-compose.yml` на сервере (в репо вынесен в `${GF_ADMIN_PASSWORD}`). +- Grafana: `127.0.0.1:3000`, доступ снаружи через **https://mon.ruzzy.dev** (nginx + Let's Encrypt). Datasource Prometheus (uid `prometheus`) и дашборды (Node Exporter Full, cAdvisor, Network/Traffic, Gitea, OpenVPN, **RPi — диск и бэкап OneDrive** (`rpi-disk.json`, с 2026-10-11) — папка Infra) провижатся автоматически. Пароль admin — в `~/monitoring/docker-compose.yml` на сервере (в репо вынесен в `${GF_ADMIN_PASSWORD}`). - **Тома:** `monitoring_prometheus_data` — **6.5 ГБ** (при retention 90d; самый крупный том на сервере, учитывать при планировании диска), `monitoring_grafana_data` — 50 МБ, `monitoring_alertmanager_data` — пустой. - **Multi-server:** таргеты через file_sd (`prometheus/targets/{node,cadvisor}/*.yml`) — новый сервер = добавить файл + `curl -X POST .../-/reload`. См. `stacks/monitoring/README.md`. - **node-exporter — `network_mode: host`** (видит реальные интерфейсы хоста: eth0, wg0-туннель и т.д.; иначе показывал бы только сеть docker-бриджа). Слушает host:9100, доступ ограничен ufw (`from 172.16.0.0/12 to port 9100`), Prometheus скрейпит через `host.docker.internal:9100`. nodename берётся с хоста (fr1). diff --git a/servers/rpi.md b/servers/rpi.md index 845f4ff..f1cbe60 100644 --- a/servers/rpi.md +++ b/servers/rpi.md @@ -40,6 +40,8 @@ OneDrive на внешний HDD. Мониторится с fr1 через pushg - `smart-metrics` (таймер 15 мин): smartctl `-n standby` → метрики `smart_*` → job `smart`. Спящий диск не будится: отдаются значения из кэша `/var/lib/smart-metrics/` + `smart_disk_standby=1`. - `backup-metrics ` — `ExecStopPost=+` у юнитов бэкапа: `backup_last_exit_code`, `backup_last_run_timestamp`, `backup_last_success_timestamp` (время успеха хранится локально в `/var/lib/backup-metrics/`, т.к. PUT заменяет группу целиком). - Доставка: `metrics-push ` = ssh с ключом `/etc/metrics-push/id_ed25519` (root, 600) на `rus@161.97.93.252`; host key fr1 закреплён в `/etc/metrics-push/known_hosts`. На fr1 ключ ограничен forced command (`servers/fr1.md` → Мониторинг). +- `smart-metrics` заодно отдаёт `backup_disk_size_bytes`/`backup_disk_avail_bytes` для `/mnt/backup` (statfs, диск не будит). +- Дашборд: https://mon.ruzzy.dev/d/rpi-disk (SMART, износ за сутки, свежесть бэкапов, заполнение). - Алерты — группа `rpi` в `stacks/monitoring/prometheus/rules/alerts.yml`. ## Агент Astra (с 2026-10-11) diff --git a/stacks/monitoring/grafana/provisioning/dashboards/rpi-disk.json b/stacks/monitoring/grafana/provisioning/dashboards/rpi-disk.json new file mode 100644 index 0000000..d058ca3 --- /dev/null +++ b/stacks/monitoring/grafana/provisioning/dashboards/rpi-disk.json @@ -0,0 +1,1039 @@ +{ + "uid": "rpi-disk", + "title": "RPi — диск и бэкап OneDrive", + "tags": [ + "rpi", + "smart", + "backup" + ], + "timezone": "browser", + "schemaVersion": 39, + "refresh": "5m", + "time": { + "from": "now-7d", + "to": "now" + }, + "editable": true, + "templating": { + "list": [ + { + "name": "instance", + "type": "query", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "query": { + "query": "label_values(smart_health_passed, instance)", + "refId": "i" + }, + "refresh": 2, + "current": {} + }, + { + "name": "serial", + "type": "query", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "query": { + "query": "label_values(smart_health_passed{instance=\"$instance\"}, serial)", + "refId": "s" + }, + "refresh": 2, + "includeAll": true, + "multi": true, + "current": { + "text": "All", + "value": "$__all" + } + } + ] + }, + "panels": [ + { + "type": "row", + "title": "Диск — состояние SMART", + "collapsed": false, + "gridPos": { + "x": 0, + "y": 0, + "w": 24, + "h": 1 + }, + "id": 100, + "panels": [] + }, + { + "id": 1, + "type": "stat", + "title": "SMART health", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 0, + "y": 1, + "w": 6, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "expr": "smart_health_passed{instance=\"$instance\",serial=~\"$serial\"}", + "legendFormat": "{{model}}", + "instant": true, + "range": false + } + ], + "fieldConfig": { + "defaults": { + "mappings": [ + { + "type": "value", + "options": { + "1": { + "text": "PASSED", + "color": "green" + }, + "0": { + "text": "FAILED", + "color": "red" + } + } + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": null, + "color": "red" + }, + { + "value": 1, + "color": "green" + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + }, + "description": "Общий вердикт прошивки диска. FAILED — менять немедленно." + }, + { + "id": 2, + "type": "stat", + "title": "Температура", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 6, + "y": 1, + "w": 4, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "expr": "smart_temperature_celsius{instance=\"$instance\",serial=~\"$serial\"}", + "legendFormat": "{{model}}", + "instant": true, + "range": false + } + ], + "fieldConfig": { + "defaults": { + "unit": "celsius", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": null, + "color": "green" + }, + { + "value": 50, + "color": "orange" + }, + { + "value": 55, + "color": "red" + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + } + }, + { + "id": 3, + "type": "stat", + "title": "Переназначенные / ожидающие / нечитаемые сектора", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 10, + "y": 1, + "w": 10, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "expr": "smart_attribute_raw{instance=\"$instance\",serial=~\"$serial\",id=~\"5|197|198\"}", + "legendFormat": "{{name}}", + "instant": true, + "range": false + } + ], + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": null, + "color": "green" + }, + { + "value": 1, + "color": "red" + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + }, + "description": "Любое значение > 0 — диск начал сыпаться. Рост — срочно делать вторую копию." + }, + { + "id": 4, + "type": "stat", + "title": "Последний самотест", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 20, + "y": 1, + "w": 4, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "expr": "smart_selftest_last_passed{instance=\"$instance\",serial=~\"$serial\"}", + "legendFormat": "", + "instant": true, + "range": false + } + ], + "fieldConfig": { + "defaults": { + "mappings": [ + { + "type": "value", + "options": { + "1": { + "text": "OK", + "color": "green" + }, + "0": { + "text": "FAILED", + "color": "red" + } + } + }, + { + "type": "special", + "options": { + "match": "null", + "result": { + "text": "ещё не было", + "color": "text" + } + } + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": null, + "color": "red" + }, + { + "value": 1, + "color": "green" + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + }, + "description": "Короткий — по субботам 03:00, длинный — 1-го числа 05:00 (smartd)." + }, + { + "id": 5, + "type": "stat", + "title": "Наработка", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 0, + "y": 5, + "w": 4, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "expr": "smart_power_on_hours{instance=\"$instance\",serial=~\"$serial\"}*3600", + "legendFormat": "", + "instant": true, + "range": false + } + ], + "fieldConfig": { + "defaults": { + "unit": "s", + "decimals": 1 + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "none", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + } + }, + { + "id": 6, + "type": "stat", + "title": "Старт/стоп (раскрутки)", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 4, + "y": 5, + "w": 5, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "expr": "smart_attribute_raw{instance=\"$instance\",serial=~\"$serial\",id=\"4\"}", + "legendFormat": "", + "instant": true, + "range": false + } + ], + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "none", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + } + }, + { + "id": 7, + "type": "stat", + "title": "Парковки головок", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 9, + "y": 5, + "w": 5, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "expr": "smart_attribute_raw{instance=\"$instance\",serial=~\"$serial\",id=\"193\"}", + "legendFormat": "", + "instant": true, + "range": false + } + ], + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "none", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + }, + "description": "Типичный ресурс 2.5\" — ~300 000." + }, + { + "id": 8, + "type": "stat", + "title": "CRC-ошибки USB", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 14, + "y": 5, + "w": 4, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "expr": "smart_attribute_raw{instance=\"$instance\",serial=~\"$serial\",id=\"199\"}", + "legendFormat": "", + "instant": true, + "range": false + } + ], + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": null, + "color": "green" + }, + { + "value": 1, + "color": "orange" + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + }, + "description": "Растут — кабель, USB-мост или питание." + }, + { + "id": 9, + "type": "stat", + "title": "Диск сейчас", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 18, + "y": 5, + "w": 6, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "expr": "smart_disk_standby{instance=\"$instance\",serial=~\"$serial\"}", + "legendFormat": "", + "instant": true, + "range": false + } + ], + "fieldConfig": { + "defaults": { + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "крутится", + "color": "blue" + }, + "1": { + "text": "спит (standby)", + "color": "purple" + } + } + } + ] + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + }, + "description": "Спящий диск не будим: SMART-значения выше — с последнего опроса, когда он крутился." + }, + { + "type": "row", + "title": "Бэкап OneDrive", + "collapsed": false, + "gridPos": { + "x": 0, + "y": 9, + "w": 24, + "h": 1 + }, + "id": 109, + "panels": [] + }, + { + "id": 10, + "type": "stat", + "title": "С последнего успешного sync", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 0, + "y": 10, + "w": 6, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "expr": "time() - backup_last_success_timestamp{instance=\"$instance\",job=\"backup_onedrive\"}", + "legendFormat": "", + "instant": true, + "range": false + } + ], + "fieldConfig": { + "defaults": { + "unit": "s", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": null, + "color": "green" + }, + { + "value": 28800, + "color": "orange" + }, + { + "value": 43200, + "color": "red" + } + ] + }, + "noValue": "ещё не было" + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + } + }, + { + "id": 11, + "type": "stat", + "title": "С последней проверки целостности", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 6, + "y": 10, + "w": 6, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "expr": "time() - backup_last_success_timestamp{instance=\"$instance\",job=\"backup_onedrive_verify\"}", + "legendFormat": "", + "instant": true, + "range": false + } + ], + "fieldConfig": { + "defaults": { + "unit": "s", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": null, + "color": "green" + }, + { + "value": 648000.0, + "color": "orange" + }, + { + "value": 691200, + "color": "red" + } + ] + }, + "noValue": "ещё не было" + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + } + }, + { + "id": 12, + "type": "stat", + "title": "Код последнего прогона", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 12, + "y": 10, + "w": 6, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "expr": "backup_last_exit_code{instance=\"$instance\"}", + "legendFormat": "{{job}}", + "instant": true, + "range": false + } + ], + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": null, + "color": "green" + }, + { + "value": 1, + "color": "red" + } + ] + }, + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "OK" + }, + "2": { + "text": "диск >90%" + } + } + } + ] + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + } + }, + { + "id": 13, + "type": "stat", + "title": "Свежесть метрик", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 18, + "y": 10, + "w": 6, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "expr": "time() - push_time_seconds{instance=\"$instance\"}", + "legendFormat": "{{job}}", + "instant": true, + "range": false + } + ], + "fieldConfig": { + "defaults": { + "unit": "s", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": null, + "color": "green" + }, + { + "value": 3600, + "color": "orange" + }, + { + "value": 10800, + "color": "red" + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + }, + "description": "Сколько назад малина последний раз пушила. Больше 3 ч — малина или канал лежат." + }, + { + "type": "row", + "title": "Динамика", + "collapsed": false, + "gridPos": { + "x": 0, + "y": 14, + "w": 24, + "h": 1 + }, + "id": 114, + "panels": [] + }, + { + "id": 14, + "type": "timeseries", + "title": "Температура", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 0, + "y": 15, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "smart_temperature_celsius{instance=\"$instance\",serial=~\"$serial\"}", + "legendFormat": "{{model}}", + "instant": false, + "range": true + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "lineWidth": 2, + "fillOpacity": 10, + "spanNulls": true, + "thresholdsStyle": { + "mode": "line" + } + }, + "unit": "celsius", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": null, + "color": "green" + }, + { + "value": 55, + "color": "red" + } + ] + } + }, + "overrides": [] + }, + "options": {} + }, + { + "id": 15, + "type": "timeseries", + "title": "Износ: прирост за сутки", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 12, + "y": 15, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "increase(smart_attribute_raw{instance=\"$instance\",serial=~\"$serial\",id=\"4\"}[1d])", + "legendFormat": "раскрутки", + "instant": false, + "range": true + }, + { + "refId": "B", + "expr": "increase(smart_attribute_raw{instance=\"$instance\",serial=~\"$serial\",id=\"193\"}[1d])", + "legendFormat": "парковки головок", + "instant": false, + "range": true + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "lineWidth": 2, + "fillOpacity": 10, + "spanNulls": true + } + }, + "overrides": [] + }, + "options": {}, + "description": "Для прогноза ресурса: раскруток в сутки × 365 против запаса." + }, + { + "id": 16, + "type": "timeseries", + "title": "Плохие сектора", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 0, + "y": 23, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "smart_attribute_raw{instance=\"$instance\",serial=~\"$serial\",id=~\"5|197|198\"}", + "legendFormat": "{{name}}", + "instant": false, + "range": true + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "lineWidth": 2, + "fillOpacity": 10, + "spanNulls": true + } + }, + "overrides": [] + }, + "options": {} + }, + { + "id": 17, + "type": "timeseries", + "title": "Заполнение диска бэкапа", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "gridPos": { + "x": 12, + "y": 23, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "1 - backup_disk_avail_bytes{instance=\"$instance\"} / backup_disk_size_bytes{instance=\"$instance\"}", + "legendFormat": "{{mountpoint}}", + "instant": false, + "range": true + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "lineWidth": 2, + "fillOpacity": 10, + "spanNulls": true, + "thresholdsStyle": { + "mode": "line" + } + }, + "unit": "percentunit", + "min": 0, + "max": 1, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": null, + "color": "green" + }, + { + "value": 0.9, + "color": "red" + } + ] + } + }, + "overrides": [] + }, + "options": {} + } + ] +} \ No newline at end of file diff --git a/stacks/rpi-backup/bin/smart-metrics b/stacks/rpi-backup/bin/smart-metrics index 738e460..0600480 100755 --- a/stacks/rpi-backup/bin/smart-metrics +++ b/stacks/rpi-backup/bin/smart-metrics @@ -33,4 +33,10 @@ for dev in /dev/sd?; do [ -f "$f" ] && cat "$f" >> "$out" echo "smart_disk_standby{serial=\"$serial\",device=\"${dev#/dev/}\"} $standby" >> "$out" done +# заполнение диска бэкапа (statfs — из памяти, спящий диск не будит) +if mountpoint -q /mnt/backup; then + read -r size avail < <(df -B1 --output=size,avail /mnt/backup | tail -1) + echo "backup_disk_size_bytes{mountpoint=\"/mnt/backup\"} $size" >> "$out" + echo "backup_disk_avail_bytes{mountpoint=\"/mnt/backup\"} $avail" >> "$out" +fi metrics-push smart < "$out"