Practice: Design Real-time Analytics System
Near real-time analytics that ingests billions of events, aggregates them into time-windowed metrics, and powers live dashboards with sub-minute freshness.
Build this system yourself on the canvas below. Add components, connect them, and run traffic simulations to see where your design holds up and where it breaks.
Functional requirements
- Ingest high-throughput event streams via authenticated API.
- Durably buffer events and support at-least-once delivery to processors.
- Aggregate metrics in tumbling/sliding windows with late-event handling.
- Power live dashboards and APIs with sub-minute freshness.
- Support ad-hoc drill-down on raw events for debugging.
- Backfill and reprocess history when aggregation logic changes.
- Detect anomalies and trigger alerts on threshold/ML rules.
- Multitenant isolation with per-tenant quotas and cost controls.
Non-functional requirements
- Availability: 99.95%+; single AZ failure should not halt ingest or queries.
- Latency: ingest API p99 < 150 ms; dashboard API p95 < 300 ms; end-to-end freshness < 60 s.
- Durability: raw events persisted with replication and backups; immutability for audit.
- Scalability: independently scale ingest, queues, processors, and query tiers across multi-AZ deployments.
- Consistency: at-least-once across stream; idempotent processors; read-after-write for configs.
- Security: TLS 1.2+ in transit, encryption at rest, per-tenant API keys/OAuth, and RBAC for dashboards.
- Resilience: timeouts, circuit breakers, retries/backoff, DLQs, and backpressure.
- Observability: traces/logs/metrics for queue depth, lag, watermark, window lateness, and query p99.
