Availability in Numbers

Building Blocks

• Availability Percentage: The fraction of time a system is up and correctly handling requests, usually measured over a one-year window.
• Allowed Downtime Budget: The maximum outage time that still keeps you within your target percentage. 99.9% allows about 8.7 hours per year; 99.99% allows about 52 minutes.
• SLA and SLO: An SLA is a contractual promise to customers, often with penalties. An SLO is an internal reliability target used to guide engineering and prioritize incident response.
• Error Budget: The inverse of your SLO: the amount of unreliability you are allowed to consume. It helps balance reliability work against feature velocity.
• Cost of Downtime: Revenue lost, reputation damage, and engineering hours spent during outages. This number determines whether pursuing another nine is worth it.

Definitions

Availability

The proportion of time that a system is operational and performing its required function under agreed conditions.

• Usually expressed as a percentage such as 99.9% or 99.99%.
• Must be measured from the user's perspective, not just the server's heartbeat.

Nines

A shorthand for the number of consecutive 9 digits in an availability target. 99.99% is called 'four nines'.

• Each additional nine reduces the yearly downtime budget by a factor of ten.
• Going from two nines to five nines is a 1,000x improvement in allowed downtime.

Service Level Objective (SLO)

An internal reliability target that defines how available a service should be over a measurement period.

• SLOs drive alerting thresholds and incident severity.
• They should be ambitious but achievable; missing them constantly erodes trust.

Service Level Agreement (SLA)

A contractual commitment to customers that defines availability targets and any penalties for missing them.

• SLAs are usually looser than internal SLOs to leave a safety margin.
• Breaching an SLA can trigger refunds, credits, or legal exposure.

Error Budget

The amount of unreliability a team is allowed to introduce before it must slow down feature work and focus on reliability.

• If your SLO is 99.9%, your error budget is 0.1% downtime.
• Error budgets protect teams from perfectionism and from recklessness.

Bonus Points

• Downtime per year: 99% ≈ 3.65 days, 99.9% ≈ 8.76 hours, 99.99% ≈ 52.6 minutes, 99.999% ≈ 5.26 minutes.
• Durability is not availability: S3's eleven nines of durability describe data loss probability, not uptime. A system can have perfect durability and still be down.
• User-perceived availability: A backend may report 99.99% uptime while a buggy mobile client makes users think the service is broken.
• Planned maintenance: Some SLAs exclude scheduled downtime; clarify what counts before comparing numbers across vendors.
• Higher nines need better change management: Most large outages are caused by deployments and configuration changes, not random hardware failures.

Patterns

• Pick the Right Target for the Workload — When setting or revising an SLO.
• Spend Error Budgets Deliberately — When reliability work conflicts with shipping features.
• Define Availability from the User's View — When measuring whether the product is actually usable.

Strategies

• Start with a Realistic SLO When: At the beginning of a project or reliability review. How: Estimate the cost of downtime and the cost of improving reliability. Choose an SLO that is better than 'best effort' but not stricter than the business justifies. Example: A startup might begin at 99.5% and tighten to 99.9% once it lands enterprise customers.
• Build Observability Before Chasing Nines When: Before declaring that you have hit a new availability target. How: You cannot improve what you cannot measure. Instrument user-facing success rates, latency distributions, and error budgets before investing in redundancy. Example: A team notices that 90% of 'unavailability' is actually a slow dependency timing out, so they add circuit breakers before buying more replicas.
• Invest Reliability Improvements Where Failure Hurts Most When: When prioritizing reliability work across a large system. How: Focus on the critical paths that directly affect revenue or user trust. Non-critical features can live with lower targets and graceful degradation. Example: A streaming service protects the playback path with redundant origins while keeping the comments section on a simpler stack.

Tradeoffs

Real World

Quiz

Approximately how much downtime per year is allowed for 99.99% availability?

1. About 52 minutes
2. About 8.7 hours
3. About 5 minutes
4. About 3.6 days

99.99% (four nines) allows roughly 52.6 minutes of downtime per year.

Which statement best describes an SLO?

1. An internal reliability target used to guide engineering priorities
2. A contractual promise with financial penalties
3. The maximum number of nodes allowed in a cluster
4. The percentage of requests that must complete in under one second

An SLO is an internal objective. An SLA is the external contract that may include penalties.

What is an error budget?

1. The amount of unreliability a team is allowed before slowing feature work
2. The total number of errors a service can log in a day
3. The budget allocated to buy redundant servers
4. The difference between SLA and SLO

An error budget is the inverse of an SLO; it quantifies how much downtime is acceptable and balances reliability with velocity.

Why might user-perceived availability differ from backend uptime?

1. Client bugs, third-party failures, or slow dependencies can break the user experience even when servers are healthy
2. Backend uptime is always higher than user-perceived availability
3. Users measure availability with different calendars
4. Backend uptime excludes successful responses

User-perceived availability is end-to-end; it includes clients, networks, and dependencies.

Which is generally the biggest driver of needing more nines?

1. Business cost of downtime
2. Number of developers on the team
3. Choice of programming language
4. Length of the README

The right number of nines is driven by how much outages cost the business and what customers expect.

Building Blocks

• Series Availability: The total availability of a chain of components is the product of each component's availability: A_total = A1 × A2 × A3.
• Parallel Availability: The total availability of redundant components is one minus the probability that every component fails: A_total = 1 − (1−A1)(1−A2).
• Complement Rule: If a component is 99.9% available, the chance it is down is 0.1%. Parallel availability uses those failure probabilities, not success probabilities.
• Single Point of Failure: A component that sits alone in series. If it fails, the whole system fails regardless of how reliable the rest of the stack is.
• Critical Path: The sequence of components that must all work for the user request to succeed. Availability improvements here have the biggest impact.

Definitions

Series Availability

The availability of components arranged end-to-end where every component must be healthy for the system to function.

• Calculated by multiplying individual availabilities.
• Even reliable components chained together produce a lower total availability.

Parallel Availability

The availability of redundant components where the system continues to work if at least one component is healthy.

• Calculated using the complement of all components failing simultaneously.
• Adding parallel copies gives diminishing but significant gains.

Single Point of Failure (SPOF)

A component whose failure will take down the entire system.

• Removing SPOFs usually means adding parallel redundancy or automated failover.
• Common SPOFs include single databases, single load balancers, and single network links.

Redundancy

Duplicating critical components so that a failure does not stop the system.

• Can be active-active, active-passive, or standby.
• Only improves availability if the redundant units fail independently.

N+1 Redundancy

Having at least one spare component beyond the minimum needed to handle the load.

• N is the number of units required; +1 is the extra buffer.
• Common in power supplies, load balancers, and database replicas.

Bonus Points

• Hybrid architectures: A common pattern is a series of parallel pairs, e.g., dual load balancers → dual application servers → database cluster. Each parallel pair raises the total availability of its stage.
• Correlated failures break the math: Two replicas in the same rack share power and network. If one fails, the other is more likely to fail too, so the parallel gain is smaller than the formula suggests.
• Failover must actually work: A standby node that does not take over automatically adds no availability. Test failover paths regularly.
• Microservices multiply series risk: Each additional service in the call chain can become a serial dependency. A chain of five 99.99% services is roughly 99.95% available overall.
• Availability and capacity are linked: A single surviving node in a parallel pair must be able to handle the full load; otherwise you are available but degraded.

Patterns

• Parallel Pair on the Critical Path — When a single component failure would take down the service.
• Series Minimization — When designing a request flow.
• Failover with Health Checks — When you have active-passive redundancy.

Strategies

• Map the Critical Path First When: Before adding redundancy anywhere. How: Draw the components a request must pass through. Identify every SPOF. Add parallel redundancy to the ones on the critical path first. Example: An e-commerce checkout needs DNS, CDN, load balancer, app server, payment service, and database. DNS and the database are SPOFs unless redundant.
• Isolate Failure Domains When: When deploying parallel replicas. How: Place replicas in different racks, availability zones, or regions so they do not share fate. Example: A multi-AZ database cluster can survive the loss of an entire data center.
• Test Failover in Production-like Conditions When: Continuously, not just during launch. How: Run game days or chaos engineering exercises that terminate nodes and verify that traffic reroutes and services recover. Example: Netflix's Chaos Monkey randomly terminates instances to ensure automated failover works.

Tradeoffs

Real World

Quiz

Two services each with 99.9% availability are connected in series. What is the approximate total availability?

1. 99.8%
2. 99.99%
3. 99.9999%
4. 99.0%

Series availability is A1 × A2. 0.999 × 0.999 ≈ 0.998, so about 99.8%.

Two 99.9% available components are placed in parallel. What is the approximate total availability?

1. 99.9999%
2. 99.9%
3. 99.8%
4. 99.99%

Parallel availability is 1 − (0.001)² = 0.999999, roughly 99.9999%.

Which topology is most likely to create a single point of failure?

1. A single database that every request must read from
2. Two active-active load balancers
3. A multi-AZ replicated cluster
4. A CDN with hundreds of edge caches

A single required database is a single point of failure because its failure brings the whole system down.

What does N+1 redundancy mean?

1. One extra component beyond what is needed to handle the load
2. One node for every user
3. Adding one more nine of availability
4. One network link per availability zone

N+1 means you have the required N units plus one spare.

Why can correlated failures reduce the real availability of a redundant system?

1. Replicas may share power, network, or code and fail together
2. Redundant systems always have lower availability
3. Correlated failures increase component reliability
4. They have no effect because the math accounts for them

If replicas share failure domains, they can all fail at once, making redundancy less effective than the formula predicts.