监控告警踩坑记录

作者: 醉月思发布日期: 2022-09-15 分类: 技术实践

凌晨三点被邮件吵醒，原来是磁盘脚本误报——同一条告警连发七封，值班同事已经麻木了。

我的答案是在那次误报之后才慢慢想清楚的，后面才换成 Prometheus + Grafana。

最初的问题

脚本监控

简单的 Shell 脚本监控。

#!/bin/bash
# 检查服务是否运行
if ! systemctl is-active --quiet myapp; then
    echo "myapp is not running" | mail -s "Service Alert" [email protected]
    systemctl start myapp
fi

# 检查磁盘使用率
disk_usage=$(df -h / | awk 'NR==2 {print $5}' | cut -d'%' -f1)
if [ $disk_usage -gt 80 ]; then
    echo "Disk usage is ${disk_usage}%" | mail -s "Disk Alert" [email protected]
fi

问题：

数据不持久化
无法做趋势分析
维护困难

Prometheus

为什么用 Prometheus

Pull 模式，易于扩展，数据持久化，支持多维度数据。

部署 Prometheus

# 使用 Docker 部署
docker run -d --name prometheus \
  -p 9090:9090 \
  -v /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus:latest

Prometheus 配置

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'myapp'
    static_configs:
      - targets: ['localhost:8080']
        labels:
          app: myapp
          env: production

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['localhost:9100']

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093

rule_files:
  - "alerts/*.yml"

暴露指标

Go 应用

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "net/http"
)

var (
    httpRequestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests",
        },
        []string{"method", "path", "status"},
    )

    httpRequestDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "http_request_duration_seconds",
            Help: "HTTP request duration in seconds",
            Buckets: prometheus.DefBuckets,
        },
        []string{"method", "path"},
    )
)

func init() {
    prometheus.MustRegister(httpRequestsTotal)
    prometheus.MustRegister(httpRequestDuration)
}

func middleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        
        ww := &responseWriter{ResponseWriter: w}
        next.ServeHTTP(ww, r)
        
        duration := time.Since(start).Seconds()
        
        httpRequestsTotal.WithLabelValues(
            r.Method, r.URL.Path, http.StatusText(ww.status),
        ).Inc()
        
        httpRequestDuration.WithLabelValues(
            r.Method, r.URL.Path,
        ).Observe(duration)
    })
}

func main() {
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":8080", nil)
}

Node.js 应用

const promClient = require('prom-client');

// 创建指标
const httpRequestsTotal = new promClient.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'path', 'status'],
});

const httpRequestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration in seconds',
  labelNames: ['method', 'path'],
  buckets: [0.1, 0.5, 1, 2, 5],
});

// 中间件
function metricsMiddleware(req, res, next) {
  const start = Date.now();
  
  const originalSend = res.send;
  res.send = function (data) {
    const duration = (Date.now() - start) / 1000;
    
    httpRequestsTotal.inc({
      method: req.method,
      path: req.path,
      status: res.statusCode,
    });
    
    httpRequestDuration.observe({
      method: req.method,
      path: req.path,
    }, duration);
    
    originalSend.call(this, data);
  };
  
  next();
}

// 暴露指标
app.get('/metrics', (req, res) => {
  res.set('Content-Type', promClient.register.contentType);
  res.end(promClient.register.metrics());
});

Node Exporter

监控服务器资源。

# 安装 Node Exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.3.1.linux-amd64.tar.gz
cd node_exporter-1.3.1.linux-amd64
./node_exporter

# 验证
curl http://localhost:9100/metrics

Grafana

部署 Grafana

docker run -d --name grafana \
  -p 3000:3000 \
  --link prometheus:prometheus \
  grafana/grafana:latest

添加 Prometheus 数据源

登录 Grafana（admin/admin）
Configuration -> Data Sources -> Add data source
选择 Prometheus
输入 URL: http://prometheus:9090
Save & Test

创建仪表盘

面板配置：

{
  "dashboard": {
    "title": "MyApp Dashboard",
    "panels": [
      {
        "title": "QPS",
        "targets": [
          {
            "expr": "rate(http_requests_total[1m])",
            "legendFormat": "{{method}} {{path}}"
          }
        ],
        "type": "graph"
      },
      {
        "title": "Response Time",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))",
            "legendFormat": "P95"
          },
          {
            "expr": "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))",
            "legendFormat": "P99"
          }
        ],
        "type": "graph"
      },
      {
        "title": "Error Rate",
        "targets": [
          {
            "expr": "rate(http_requests_total{status=~\"5..\"}[5m]) / rate(http_requests_total[5m]) * 100",
            "legendFormat": "Error Rate %"
          }
        ],
        "type": "graph"
      }
    ]
  }
}

告警规则

Prometheus 告警

# alerts/myapp.yml
groups:
  - name: myapp
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }} for the last 5 minutes."

      - alert: HighResponseTime
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High response time detected"
          description: "P95 response time is {{ $value }}s for the last 10 minutes."

      - alert: ServiceDown
        expr: up{job="myapp"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service is down"
          description: "MyApp has been down for more than 1 minute."

Alertmanager

# 部署 Alertmanager
docker run -d --name alertmanager \
  -p 9093:9093 \
  -v /etc/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml \
  prom/alertmanager:latest

配置邮件通知：

# alertmanager.yml
global:
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: 'password'

route:
  receiver: 'default-receiver'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h

receivers:
  - name: 'default-receiver'
    email_configs:
      - to: '[email protected]'
        headers:
          Subject: '[Alert] {{ .GroupLabels.alertname }}'

配置 Webhook 通知：

receivers:
  - name: 'webhook-receiver'
    webhook_configs:
      - url: 'https://hooks.example.com/webhook'
        send_resolved: true

踩过的坑

坑一：指标太多

一开始什么都监控，指标太多，查询慢。

解决：只监控关键指标。

// 不要这样做
for _, user := range users {
    prometheus.NewCounter(prometheus.CounterOpts{
        Name: fmt.Sprintf("user_%d_requests", user.ID),
    })
}

// 应该这样做
userRequestsTotal := prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "user_requests_total",
    },
    []string{"user_id"},
)
userRequestsTotal.WithLabelValues(strconv.Itoa(user.ID)).Inc()

坑二：告警太多

阈值设得太低，告警太多，团队麻木。

解决：合理设置阈值，使用告警分组。

# 使用告警分组
route:
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

# 使用告警抑制
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'service']

坑三：数据丢失

Prometheus 数据丢了，无法做趋势分析。

解决：

配置数据保留时间
使用远程存储
定期备份

# prometheus.yml
global:
  external_labels:
    monitor: 'myapp-monitor'

# 远程存储
remote_write:
  - url: http://thanos-receiver:19291/api/v1/receive

# 数据保留
storage:
  tsdb:
    retention.time: 30d

监控指标分类

RED 方法

Rate：请求速率
Errors：错误率
Duration：响应时间

# 请求速率
rate(http_requests_total[5m])

# 错误率
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])

# 响应时间
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

USE 方法

Utilization：资源使用率
Saturation：饱和度
Errors：错误

# CPU 使用率
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# 内存使用率
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# 磁盘使用率
(1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) * 100

写在最后

监控告警这东西，不是技术问题，是运维文化问题。

解决了：

系统可见性
问题及时发现
趋势分析

带来了：

运维成本
警报疲劳
复杂度增加

实施之前先评估：

监控目标
团队能力
资源预算
告警策略

不是所有指标都需要告警，有时候趋势分析就够用。

这次监控告警改造花了三周，从脚本监控到 Prometheus + Grafana。改造完成后，故障发现时间从 30 分钟降到 5 分钟，MTTR 从 2 小时降到 30 分钟。

版权声明: 本文首发于指尖魔法屋-监控告警踩坑记录（https://blog.thinkmoon.cn/post/67-monitoring-alerting-prometheus-grafana-practice/）转载或引用必须申明原指尖魔法屋来源及源地址！

从ELK走到Loki：日志管理笔记

从ELK走到Loki：日志管理笔记一旦进项目，好看的架构图就没那么管用了。

📁 技术实践

📅 2022-09-12

AI模型监控：这次怎么落地的

他们去年上线的推荐模型，看起来一切正常：预测延迟在100ms以内，接口错误率接近0，监控面板绿油油的。