Observability

The three pillars of observability are talked about a lot. Which one to reach for depends on the question you’re answering. Metrics: for “is it broken and how much” Aggregated numerical data over time. Good for: Dashboards and alerts Trends (is latency increasing week-over-week?) Capacity planning Not good for: Explaining why a specific request was slow Finding causality between events Stack: Prometheus + Grafana remains the default. OpenTelemetry Metrics if you want vendor-neutral instrumentation. ...

Most Grafana dashboards are bad. Too many panels, unclear queries, inconsistent color schemes, no clear purpose. Here are the principles I apply now. Rule 1: Every dashboard has one question Start by writing down: “What question does this dashboard answer?” Good: “Is the order service healthy right now?” “How is the nightly ETL job progressing?” “What is the cost trend for our compute in the last 30 days?” Bad: “Production metrics” “Database overview” If you can’t state the question in one sentence, you don’t know what the dashboard is for. ...