Practice: Design Metrics Monitoring System
Metrics monitoring system inspired by Prometheus and Facebook's Gorilla time-series database (VLDB 2015): buffered ingestion with cardinality limits, compressed time-series storage, cached dashboard queries, rule-based alerting, and downsampled cold storage. A teaching design.
Build this system yourself on the canvas below. Add components, connect them, and run traffic simulations to see where your design holds up and where it breaks.
Functional requirements
- Ingest metric data points from mobile/web client apps and backend servers
- Store time-series metrics with tags (service, host, region, env)
- Query metrics by time range, tag filters, and aggregation functions (avg, p99, sum)
- Render dashboards showing metric graphs over configurable time windows
- Evaluate alert rules on live metrics — trigger notification when threshold crossed
- Send alert notifications via email, Slack, PagerDuty
- Aggregate raw metrics into rollups (1-min, 5-min, 1-hour) for efficient long-range queries
- Archive older metrics to cold storage; query cold + hot data transparently
Non-functional requirements
- Ingest 1M data points per second sustained
- Dashboard query response < 200ms p99 for last 1-hour window
- Alert evaluation latency < 30s from metric ingestion to notification
- Metrics available for query within 5s of ingestion
- Hot storage retains 30 days of raw data; cold storage retains 2 years of rollups
- 99.9% uptime for alert evaluation — missed alerts are safety-critical