Intermediate System design concept · Handling Scale & Bottlenecks · 37 mins read

Asynchronous Processing

Not every piece of work belongs in the user-facing request path.

Open the interactive lesson See it in Design Library Management System

Event-Driven Processing

Learn how systems react to published events so producers and consumers evolve independently at scale.

Intuition

A checkout write often triggers several follow-up actions: email, analytics, inventory, shipping, and search updates. Doing all of that inline turns one request into a long chain of dependencies and makes every downstream slowdown user-visible. Event-driven processing lets the writer publish a fact and move on while other services react asynchronously. That keeps the producer fast and loosely coupled, but it also means downstream views catch up later, so teams must design for eventual consistency.

Mental Model

Treat the producer as saying “this happened” on a shared event stream. Each consumer decides whether that fact matters and handles its own local update without the producer coordinating the workflow. Think of it like: It is like posting a gate change on an airport board: passengers, cleaners, and staff all react, but the airline does not call each group one by one.

Building Blocks

A producer emits an event only after its own state change is durable. That keeps the event tied to a real business fact instead of a best-effort side effect.
A broker or log stores and distributes the event. The producer knows the schema and topic, not the full list of downstream consumers.
Consumers subscribe and translate the event into local actions such as notifications, projections, or integrations. Each consumer owns its own retries, failures, and pace.
A stable event contract plus replayable offsets make the system survivable over time. New consumers can catch up from history, and existing ones can recover after crashes.

Definitions

Event

A record of a fact that already happened in the domain.

Good names are past tense, such as OrderPlaced or InvoicePaid.
An event describes what happened, not which service must act next.
The schema becomes a shared contract across many consumers.

Producer

The service that changes the source of truth and publishes the event.

A producer should not depend on knowing every consumer.
If the write and publish steps are not coordinated, events can be lost.
Outbox-style publishing is a common fix for that gap.

Consumer

A service or worker that reacts to the event and performs follow-up work.

Consumers often maintain their own projections or side effects.
They must handle duplicates because at-least-once delivery is common.
Lag tells you how stale the consumer’s view is.

Eventual Consistency

A model where different parts of the system agree after propagation delay.

Search, analytics, or email may update after the main write succeeds.
The key design question is what can be stale, and for how long.
Monitoring lag is how teams make that delay visible.

Patterns

Publish domain events after commit so downstream systems react to durable facts, not tentative work.
Use fan-out when several teams need the same fact for different purposes such as search, billing, and analytics.
Build read models from the event stream when query shape differs from the write model.

Strategies

Name events as business facts, because consumers reason better about what happened than about implicit commands.
Version schemas conservatively so old consumers continue working while new ones adopt extra fields.
Make handlers idempotent and watch consumer lag, because replay and redelivery are normal operations.

Why event-driven systems trade coordination for independence

The main win is that the producer stops coordinating every downstream action in the request path. It can commit its own change, publish one event, and let subscribers process later. That cuts tail latency and isolates failures because one slow consumer no longer blocks the main user flow.

The cost is that correctness moves from “everyone finished before response” to “everyone will converge later.” Consumers may fail independently, replay old events, or observe the same event twice. That is why event-driven design always comes with idempotency, schema evolution, and lag monitoring.

A strong design answer therefore names both sides of the trade: faster writes and easier fan-out on one side, plus eventual consistency and async debugging on the other.

Tradeoffs

Loose coupling vs harder tracing: adding consumers is easy, but following one business action across many async hops is harder.
Fast producer latency vs stale derived views: the write path improves, but projections and notifications update later.
Replayability vs operational overhead: retained logs help recovery and new consumers, but they add storage and debugging complexity.

Real World

Apache Kafka is often used for durable event streams where multiple consumers keep their own offsets and process at different speeds.
Amazon SNS plus SQS fan-out is a common pattern for sending one business event to several isolated downstream queues.

Interview

Questions interviewers ask

When is event-driven processing a better fit than synchronous RPC?
How do you keep an event producer from losing important events after a successful write?
Why does event-driven architecture usually imply eventual consistency?
How is pub/sub fan-out different from handing one job to one worker?

What a strong answer covers

Candidates should explain producer, broker, and consumer roles; connect pub/sub to loose coupling; and mention eventual consistency, idempotent consumers, durable publishing, and lag monitoring.

Common traps

Describing events as if they were synchronous function calls with a different transport.
Assuming every consumer sees the event instantly and in one universal order.
Using event-driven design for work that actually requires an immediate synchronous answer.

Quiz

Why do teams often move from synchronous callbacks to event-driven processing?

Because events guarantee exactly-once delivery
Because producers can finish without waiting for every downstream action
Because events remove the need for data contracts
Because consumers never fail independently

The producer only needs to publish the fact, not wait for every subscriber. That lowers request latency and reduces runtime coupling.

What is the main consistency implication of event-driven systems?

Every read becomes strongly consistent
Consumers update before the producer commits
Different services may reflect the change at different times
The broker prevents stale reads automatically

Consumers process the event later, so derived views can lag behind the source of truth. Eventual consistency is a core tradeoff, not a bug to wish away.

Which statement best distinguishes an event from a command?

An event describes a fact that happened, while a command tells a specific target what to do
An event must always be synchronous, while a command is always async
An event cannot have a schema, while a command can
An event is only used inside databases

Good event names describe business facts in past tense. Commands are directional and imply control over a specific consumer.

Why must event consumers be idempotent?

Because pub/sub systems never preserve ordering
Because one event can be delivered or replayed more than once
Because events are too large to deserialize twice
Because consumers always run under a global lock

At-least-once delivery and replay are common. Idempotent handlers make duplicate processing safe.

What does consumer lag measure?

How many producers are publishing per second
How much memory the event bus uses
How far a consumer is behind the latest available events
How many schemas have been versioned

Lag quantifies delay between published reality and processed reality. It is a practical proxy for downstream staleness.