System Design Interview
Metrics monitoring system inspired by Prometheus and Facebook's Gorilla time-series database (VLDB 2015): buffered ingestion with cardinality limits, compressed time-series storage, cached dashboard queries, rule-based alerting, and downsampled cold storage. A teaching design.
See why a relational table of data points can't keep up with metrics.
What was missing: Nothing yet: one server ingests and answers queries.
Why that's risky: 180K inserts per second, plus indexes on every label, exceed any single relational database; a heavy dashboard query slows ingestion for everyone.
What gets added: A metrics server that inserts each data point as a row and runs dashboard queries against the same table.
Trade-offs: Familiar SQL, but rows of 50+ bytes per point waste storage and I/O.
Separate ingestion from storage and store points in a format built for time series.
What was missing: Ingestion and queries shared one server, and data points were stored as rows.
Why that's risky: Every dashboard refresh recomputes its queries from raw chunks, and there is no alerting.
What gets added: An ingestion service that validates batches and enforces per-team series limits, a durable Kafka-style buffer, storage writers that write in large batches, a time-series database storing compressed chunks per series (Gorilla-style, about 1.4 bytes per point), and a query service on a separate path.
Trade-offs: Data is visible a few seconds after it's pushed, in exchange for ingestion that never waits on storage.
Make dashboards fast without recomputing the same windows on every refresh.
What was missing: No stored dashboards, and every refresh recomputed everything.
Why that's risky: Nobody is told when something breaks: problems are only seen if someone is looking.
What gets added: A dashboards service and database for dashboard definitions, and a query cache that stores results per fixed time window so a refresh only computes the newest window.
Trade-offs: Cached windows can miss a late-arriving point; the newest window is never cached.
Detect problems automatically and notify the right people once.
What was missing: Nothing evaluated conditions or notified anyone.
Why that's risky: All raw data stays in the time-series database forever, so storage grows without bound and long-range queries get slow.
What gets added: Alert rules in their own database, a rule evaluator that checks rules over the newest data as it arrives, and an alert manager that deduplicates, groups and silences alerts before notifying on-call.
Trade-offs: Evaluating on 30-second windows trades a little detection delay for far less evaluation work.
Keep a year of history cheaply and make long-range queries fast.
What was missing: Raw data was kept forever at full resolution.
Why that's risky: One misbehaving service can still push a flood of points, and nothing watches the monitoring system itself.
What gets added: A downsampler that rolls raw points up into 1-minute and 1-hour aggregates and moves data older than 15 days into object storage; queries over long ranges read the downsampled data.
Trade-offs: Old data loses fine detail: a spike shorter than a minute disappears from data older than 15 days.
Protect ingestion from runaway clients and watch the watcher.
What was missing: No limit on push rate, and no monitoring of the monitoring system.
Why that's risky: Remaining risk is regional: a region's monitoring should keep working when that region has an outage, which is when it's needed most.
What gets added: A rate limiter on ingestion per team, and a separate, small meta-monitoring setup watching ingest lag, buffer depth and rule evaluation delay.
Trade-offs: Rate-limited data is dropped at the edge; teams see their limit and the drop count.
You're in the middle of an interview session. Leaving now will end your current attempt.
Explore concept overviews, real-system examples, key tradeoffs, and interview talking points for each roadmap section.
You've conquered this phase. These are the skills you now own:
