Availability Thinking

Building Blocks

• Primary / Leader: The authoritative copy that accepts writes and coordinates ordering.
• Replica / Follower: A secondary copy that replays the primary's writes and can serve reads or take over.
• Write-Ahead Log: An ordered stream of changes that replicas apply to stay in sync.
• Replication Lag: The delay between a write on the primary and its visibility on a replica.
• Quorum: A minimum count of replicas that must acknowledge a write before it is considered durable.

Definitions

Replication

The process of copying and maintaining data across multiple nodes or locations.

• Can be synchronous or asynchronous.
• Used for fault tolerance, read scaling, and geo-locality.
• Introduces consistency and conflict challenges.

Synchronous Replication

A write is acknowledged only after it has reached one or more replicas.

• Stronger durability guarantee.
• Adds write latency, especially across regions.
• Can reduce availability if replicas are unreachable.

Asynchronous Replication

A write is acknowledged after reaching the primary; replicas catch up in the background.

• Lower write latency.
• Replicas can lag, so failover may lose recent writes.
• Common for geo-distributed and read-scaling setups.

Split Brain

A failure state where disconnected partitions each believe they are the primary and accept independent writes.

• Produces divergent data that must be reconciled.
• Prevented by quorums, fencing, and leader election.

Bonus Points

• Read replicas offload read traffic from the primary but may serve slightly stale data.
• Chain replication forwards writes through a chain of nodes to balance throughput and consistency.
• Multi-leader replication allows writes in multiple regions but complicates conflict resolution.
• Witness nodes vote in quorums without storing full data, saving storage.
• RPO and RTO measure how much data you can lose and how fast you must recover.

Patterns

• Primary-Replica Replication — When you need one authoritative writer and many readers.
• Read Replicas for Scale — When read volume exceeds what one node can serve.
• Cross-Region Replication — When disaster recovery and local reads matter more than immediate consistency.

Strategies

• Match Replication Mode to Criticality When: When different data has different durability needs. How: Use synchronous replication for financial ledgers and asynchronous replication for analytics and caches. Example: A bank uses synchronous replication for transactions and async replication for audit logs.
• Monitor Replication Lag When: Whenever replicas serve reads or take over on failover. How: Alert when lag exceeds a threshold and route critical reads away from stale replicas. Example: A dashboard shows replica lag in seconds; an alert fires if any replica falls behind by more than five seconds.
• Plan Failover Before You Need It When: As soon as you add replicas. How: Define how a replica is promoted, how clients find the new primary, and how the old primary is fenced. Example: Redis Sentinel monitors Redis nodes and automatically promotes a replica when the master fails.

Tradeoffs

Real World

Quiz

What is the main purpose of replication?

1. To keep multiple copies of data for fault tolerance and scale
2. To compress data for storage savings
3. To replace backups entirely
4. To eliminate the need for a primary node

Replication creates redundant copies, improving fault tolerance and allowing reads to be spread across nodes.

Which replication style waits for replicas to acknowledge a write before confirming it?

1. Synchronous replication
2. Asynchronous replication
3. Lazy replication
4. Best-effort replication

Synchronous replication confirms a write only after replicas have received it, trading latency for durability.

What does replication lag measure?

1. The delay between a primary write and replica visibility
2. The number of replicas in the cluster
3. The time to promote a replica
4. The bandwidth between nodes

Replication lag is the time it takes for a replica to apply a write that has already succeeded on the primary.

A danger of asynchronous replication during failover is:

1. Recent writes that did not reach a replica may be lost
2. All replicas will have identical data
3. Writes become impossible
4. Read traffic stops entirely

Because async replication acknowledges before replicas catch up, unpropagated writes can be lost when the primary fails.

Split brain occurs when:

1. Two partitions each believe they are the primary and accept writes
2. A replica is slightly behind the primary
3. A backup is restored to the wrong node
4. The primary fails without a replica

Split brain happens when disconnected nodes both act as authoritative primaries, producing divergent data.

Building Blocks

• Active Node: The node currently handling read and write traffic.
• Passive / Standby Node: A node that keeps a copy of state and can be promoted when the active node fails.
• Heartbeat: A periodic signal that proves the active node is healthy.
• Virtual IP / DNS Failover: A mechanism that redirects traffic to the new active node after promotion.
• Fencing: Making sure the old active node cannot write after a failover.

Definitions

Active-Passive

An availability pattern where one node serves traffic and another node stands by ready to take over.

• Simpler than active-active because writes are serialized on one node.
• Standby resources are underutilized until failover.
• Common for databases and stateful services.

Failover

The process of switching traffic from a failed node to a healthy node.

• Can be automatic or manual.
• Detection time, promotion time, and client redirection all affect RTO.
• Must be tested regularly to remain reliable.

Heartbeat

A regular health signal sent from the active node to a monitor or the standby.

• Missed heartbeats trigger failover evaluation.
• Must distinguish node failure from network partition.
• Often paired with additional health checks.

Recovery Point Objective (RPO)

The maximum amount of data loss acceptable during a failure, usually measured in time.

• For synchronous replication, RPO can be near zero.
• For async replication, RPO equals the replication lag at failure time.

Bonus Points

• Hot standby is fully running and ready to take over in seconds.
• Warm standby is running but needs a short startup or promotion step.
• Cold standby is not running until needed, giving the slowest recovery.
• Floating IP lets the same IP move from the old active node to the new one.
• Health checks should test real traffic, not just process liveness.

Patterns

• Hot Standby — When recovery time must be very short.
• Warm Standby — When you can tolerate a few minutes of downtime.
• Cold Standby — When cost matters more than recovery speed.

Strategies

• Automate Failover Carefully When: When you want fast recovery without human delay. How: Use a monitor, quorums, and fencing to decide when to promote the standby. Example: Patroni watches PostgreSQL nodes and promotes a replica only when it holds a leader lock in etcd.
• Keep the Standby Warm When: When RTO is important. How: Run the passive node with the same software and data so promotion is fast. Example: A hot standby database applies the primary's log continuously.
• Test Failover Regularly When: Always. How: Schedule drills that actually promote the standby and redirect traffic. Example: A team runs a monthly failover drill during a low-traffic window.

Tradeoffs

Real World

Quiz

In active-passive architecture, the passive node:

1. Stays in sync but does not serve traffic until promoted
2. Serves half of the read traffic
3. Processes writes independently
4. Is only a configuration backup

The passive node is a standby that can take over if the active node fails.

What signal is commonly used to detect that the active node has failed?

1. Heartbeat
2. Replication lag
3. Checksum
4. Load average

A heartbeat is a periodic signal; missed heartbeats trigger failover checks.

Which standby type can take over fastest?

1. Hot standby
2. Warm standby
3. Cold standby
4. Offline backup

A hot standby is already running and up to date, so promotion is quickest.

The main capacity downside of active-passive is:

1. Half the resources are idle until a failure
2. It requires more code changes than active-active
3. It cannot handle read traffic
4. It needs no replication

Until failover, the passive node is not serving traffic, so part of the fleet is underutilized.

Fencing is used after failover to:

1. Prevent the old active node from writing
2. Speed up replication
3. Route traffic to both nodes
4. Increase the heartbeat frequency

Fencing isolates the failed node so it cannot rejoin and write stale data after a new primary is chosen.

Building Blocks

• Load Balancer: Distributes incoming requests across active nodes and removes unhealthy ones.
• Shared-Nothing State: Each node owns a partition of data so nodes rarely conflict with each other.
• Gossip or Consensus: Protocols that keep nodes aware of membership and state changes.
• Conflict Resolution: Rules that decide the final value when concurrent writes touch the same data.
• Partition Tolerance: The ability to keep operating when the network splits nodes into isolated groups.

Definitions

Active-Active

An availability pattern where multiple nodes serve traffic simultaneously.

• Improves resource utilization and horizontal scalability.
• Requires handling concurrent writes and state synchronization.
• Common for stateless services and sharded data stores.

Shared-Nothing Architecture

A design where each node is independent and self-sufficient, sharing no resources with others.

• Nodes do not contend for a central database or lock manager.
• State is partitioned so each node owns its slice.
• Failures are isolated to individual nodes.

Conflict Resolution

The process of merging or choosing a winner when concurrent writes produce divergent versions.

• Strategies include last-write-wins, application merge, and CRDTs.
• Often needed in multi-master or geo-distributed setups.
• Requires understanding causality and concurrency.

Quorum

A minimum number of nodes that must agree for an operation to proceed.

• Helps active-active systems avoid split brain.
• Common in distributed databases like Cassandra and DynamoDB.
• Tradeoff between consistency and availability.

Bonus Points

• Database active-active is much harder than stateless active-active because writes can conflict.
• CRDTs allow replicas to merge without coordination.
• Sharding reduces contention by partitioning data so each shard has a single writer.
• Geographic active-active places nodes close to users but complicates consistency.
• Idempotency is essential so retrying a request on another node does not create duplicate effects.

Patterns

• Shared-Nothing Services — When each request can be handled independently.
• Multi-Master Replication — When writes must be accepted in multiple regions.
• Sharded Active-Active — When data is too large or hot for a single primary.

Strategies

• Design for Idempotency When: When requests may be retried on different active nodes. How: Make operations safe to repeat using unique request IDs or natural keys. Example: A payment processor uses idempotency keys so retrying a charge does not double-bill.
• Partition Data to Reduce Conflicts When: When active-active writes could collide. How: Use sharding, user pinning, or entity affinity so related writes go to the same node. Example: A game server assigns each match to a single shard, avoiding cross-node conflicts.
• Use Sticky Sessions Where Needed When: When a user must see their own writes immediately. How: Route a user's requests to the same node for a short window after a write. Example: A chat app pins a user to the datacenter where their message was posted for a few seconds.

Tradeoffs

Real World

Quiz

In active-active architecture, nodes are:

1. All serving traffic simultaneously
2. One active and one passive
3. Only used during failover
4. Read-only replicas

Active-active means multiple nodes handle traffic at the same time.

Stateful active-active is harder than stateless active-active mainly because of:

1. Concurrent writes and conflict resolution
2. Higher CPU usage
3. Larger log files
4. Slower DNS resolution

When nodes share mutable state, concurrent writes can conflict and require resolution.

A shared-nothing architecture avoids conflicts by:

1. Partitioning data so each node owns its slice
2. Using a single global lock
3. Disabling writes on all but one node
4. Storing everything in one central database

Shared-nothing designs partition data and state so nodes do not contend for the same resources.

Which technique helps retries across active nodes stay safe?

1. Idempotency
2. Encryption
3. Compression
4. Caching

Idempotent operations can be retried on different nodes without causing duplicate effects.

Quorums in active-active systems mainly help with:

1. Avoiding split brain and bounding consistency
2. Compressing data
3. Speeding up DNS lookups
4. Caching static assets

Quorums ensure overlapping agreement between nodes, preventing divergent authoritative decisions.

Building Blocks

• Health Checks: Probes that determine whether a node is alive and healthy enough to serve traffic.
• Leader Election: A process that chooses one node to be authoritative when multiple candidates exist.
• Fencing: Isolating the failed node so it cannot rejoin and corrupt data after a replacement takes over.
• DNS / Load-Balancer Routing: Mechanisms that redirect clients to the new healthy node.
• Graceful Shutdown: Allowing a node to finish in-flight work before it stops receiving traffic.

Definitions

Failover

The process of switching from a failed component to a redundant or standby component.

• Can be automatic or manual.
• Detection, decision, and switching all contribute to downtime.
• Must be tested to stay reliable.

Recovery Time Objective (RTO)

The maximum acceptable time between a failure and service restoration.

• Includes detection, decision, promotion, and client redirection.
• Often defined per service in an SLA.
• Shorter RTO usually requires more automation and cost.

Graceful Degradation

Reducing functionality instead of failing completely when a component is unavailable.

• Improves perceived availability.
• Example: read-only mode when writes cannot be processed.
• Must be designed into the product.

Fencing Token

A monotonic token given to the current primary; stale tokens are rejected.

• Prevents a delayed or partitioned old primary from writing.
• Common in distributed locks and storage systems.

Bonus Points

• Canary failover routes a small percentage of traffic first to verify health before full cutover.
• Chaos engineering deliberately triggers failures to validate failover procedures.
• Runbooks document human steps for cases where automatic failover is too risky.
• Automatic failover is fast but can make bad decisions during network partitions.
• MTTR (mean time to recover) is usually more important than MTBF (mean time between failures).

Patterns

• Automatic Failover with Quorum — When recovery speed matters and a consensus cluster can judge health.
• Manual Failover — When data safety is more important than speed and a human must confirm.
• Graceful Degradation — When full functionality cannot be maintained but partial service is valuable.

Strategies

• Test Failover Paths Regularly When: Always. How: Run scheduled drills and chaos experiments that actually exercise detection, promotion, and routing. Example: Netflix's Chaos Monkey terminates production instances to verify automatic recovery.
• Use Quorums to Avoid Split Brain When: When automatic failover could promote two nodes simultaneously. How: Require a majority of witnesses or a distributed lock before promotion. Example: MongoDB replica sets use a majority vote to elect a new primary.
• Document Runbooks for Edge Cases When: When automatic failover is disabled or uncertain. How: Write step-by-step procedures for operators, including rollback steps. Example: A runbook explains how to promote a replica, update DNS, and reattach the old primary as a new replica.

Tradeoffs

Real World

Quiz

Which of these is NOT a typical failover stage?

1. Compiling source code
2. Detecting failure
3. Promoting a replacement
4. Redirecting traffic

Failover involves detection, decision, promotion, and redirection, not recompilation.

RTO stands for:

1. Recovery Time Objective
2. Replication Timeout Offset
3. Resource Turnover Objective
4. Redundant Target Operation

Recovery Time Objective is the maximum acceptable time to restore service after a failure.

A fencing token prevents:

1. An old primary from writing after failover
2. Clients from reading stale data
3. Replicas from joining the cluster
4. Health checks from running

Fencing tokens reject writes from a node that has lost authority after a new primary is chosen.

Why is manual failover sometimes preferred over automatic failover?

1. It avoids mistaken promotions during ambiguous failures
2. It is always faster
3. It requires no runbooks
4. It prevents replication

A human can investigate ambiguous failures before promoting a standby, reducing the risk of split brain or data loss.

Graceful degradation helps availability by:

1. Offering reduced functionality instead of complete failure
2. Eliminating the need for replicas
3. Speeding up every request
4. Preventing all errors

Graceful degradation keeps a subset of the service running when some components fail, improving perceived availability.