https://besterry.com/tags/prometheus/Recent content in Prometheus on Besterry — Linux & DevOps NotesHugoen-usMon, 10 Jun 2024 00:00:00 +0000https://besterry.com/posts/prometheus-alerts/Mon, 10 Jun 2024 00:00:00 +0000https://besterry.com/posts/prometheus-alerts/<p>Most alerts are noise. The hardest part of monitoring is deciding what NOT to alert on. Here is the framework I use.</p> <h2 id="rule-1-every-alert-must-be-actionable">Rule 1: Every alert must be actionable</h2> <p>If you get paged and there is nothing to do, the alert should not exist. Either fix the root cause, automate the response, or let it be a metric trend instead of a page.</p> <h2 id="rule-2-alert-on-user-visible-symptoms">Rule 2: Alert on user-visible symptoms</h2> <p>Instead of HighCPUUsage, prefer HighRequestLatency. CPU usage high with good latency means the system is working as designed. Latency high means users are hurting.</p>