Site Reliability Engineer Top 100 Interview Questions

Communicate early, regularly and in plain language, through fixed channels, so stakeholders get what they need without interrupting responders.

sequenceDiagram
  participant R as Responders
  participant I as Incident commander
  participant C as Communications lead
  participant S as Stakeholders and status page
  R->>I: Findings and progress
  I->>C: Current impact and plan
  C->>S: Update with next update time
  S-->>C: Questions
  C->>I: Only questions that need a decision

1. One incident channel for responders, and a separate channel or page for stakeholders.
2. Regular updates on a stated schedule, such as every 30 minutes, even when there is no news, because silence causes people to interrupt responders.
3. A standard format:
   • What is affected and how badly.
   • What is being done now.
   • What users or customers should do, if anything.
   • When the next update will come.
4. Customer communication through a status page or support teams, using wording approved in advance for common scenarios.
5. Avoid speculation: state what is known, and correct earlier statements openly if they prove wrong.
6. Close the loop: announce resolution, summarize impact and say when the review will be shared.

The communications lead owns this, so the people fixing the system can stay focused on the fix.

Complex systems rarely fail for one reason. An incident usually needs several conditions at once, such as a bug, a missing alert and a risky process, so a review that names only one "root cause" leaves the others in place to combine again.

flowchart LR
  A["Trigger: bad configuration value"] --> E["Incident"]
  B["Detection gap: no alert on the symptom"] --> E
  C["Slow mitigation: manual rollback"] --> E
  D["Process gap: change skipped staged rollout"] --> E

How to find contributing factors:

1. Build a detailed timeline from logs, alerts, chat messages and deployments, with what people knew at each moment.
2. Ask "why was this possible?" at each step, not just "what broke?"
3. Look across categories:
   • Technical: the trigger, such as a bad configuration value.
   • Detection: why monitoring did not catch it sooner.
   • Mitigation: what made recovery slow, such as unclear runbooks or slow rollbacks.
   • Process: reviews or tests that let the change through.
   • Organizational: unclear ownership or time pressure.
4. Separate what went well too, so useful practices are reinforced.

Output: a list of factors, each with an action that removes or weakens it, prioritized by how much it would have reduced the impact. Use blameless language: people acted reasonably given what they knew, and the system allowed the outcome.

Treat action items as real work with owners, deadlines and tracking, and review their completion as seriously as the incidents themselves. Reviews whose actions never happen teach teams that reviews do not matter.

Practices:

1. Fewer, better actions: a handful of concrete, prioritized items beats a long list nobody can finish.
2. Specific and verifiable: "add an alert when the queue's oldest message is older than five minutes" rather than "improve monitoring".
3. Owner and due date for each, agreed by the owning team's manager, not assigned by the reviewer alone.
4. Tracked in the team's normal backlog, labeled as incident follow-ups.
5. Regular review: a monthly report of open and overdue items across teams, shared with engineering leadership.
6. Close with evidence, such as a link to the change and a test showing the alert fires.

When items stall: escalate through the error budget policy or leadership review, since an unfixed contributing factor is a known risk that has already caused one incident.

Give leaders and teams a consistent view of how reliable services are against their objectives, where risk is growing and where investment is needed, instead of a collection of incompatible dashboards.

What to include:

1. Objective attainment: for each service and critical journey, whether the objective was met over the last period, and how much error budget was used.
2. Trends: budget use over several periods, to show improvement or decline.
3. Incidents: count and severity, time to detect and time to restore, grouped by contributing factor category.
4. Coverage: the share of services with objectives, alerts tied to them and runbooks.
5. On-call health: pages per shift and off-hours pages by team.
6. Open risks: overdue incident actions and known reliability gaps.

Making it useful:

• Standardize indicator definitions, so numbers are comparable.
• Highlight exceptions rather than listing every service.
• Review it monthly with engineering leadership, and link decisions, such as reliability investment, to the data.

Use the error budget as the shared rule: while the budget is healthy, teams prioritize features; when it runs low, reliability work takes priority. This turns an argument about opinions into a decision based on agreed data.

Making the balance work:

1. Agree objectives with product owners, so they own the reliability target as much as engineers do.
2. Reserve a fixed share of capacity for reliability and operational improvements, such as 20%, regardless of budget, so problems do not pile up.
3. Prioritize by impact: rank reliability work by how much budget it would have saved, using incident data.
4. Make reliability visible in planning, with the cost of incidents in user and business terms.
5. Celebrate prevention: reliability work that avoids incidents is invisible unless someone points it out.

When there is disagreement: escalate to the leaders who signed the error budget policy, with data on budget use and the risks of delaying fixes. Most disagreements disappear once both sides see the same numbers.

Forecast peak demand, measure how much load one unit of capacity handles within the objectives, then provision enough units for the forecast plus headroom for failures and surprises. Repeat on a schedule, because both demand and efficiency change.

1. Find the driver: the business metric that load follows, such as active users or orders.
2. Forecast peak load, not average load, including seasonal peaks and planned events.
3. Measure capacity per instance with a load test at the latency objective, not at the point where the service breaks.
4. Set a target utilization, such as 70%, to absorb spikes and slow scale-ups.
5. Add failure headroom: enough to lose an availability zone, or the largest single failure domain, at peak.
6. Check every bottleneck: databases, queues, connection limits and third-party quotas, not just application servers.
7. Track lead times: some resources, such as reserved hardware or quota increases, take weeks to obtain.

Compare the forecast with actual load each period, and revise the model when the error is large.

On average, the number of requests in progress equals the arrival rate times the time each request spends in the system. This simple relationship holds for almost any stable system, and it explains why slow dependencies exhaust worker pools and connection limits.

concurrency = throughput × latency

How to use it:

• Sizing pools: worker threads, connection pools and concurrency limits must cover the expected concurrency, with margin.
• Understanding slowdowns: if latency doubles at the same traffic, concurrency doubles too, and fixed-size pools fill up.
• Finding the maximum throughput: with a fixed number of workers, the highest sustainable rate is workers divided by latency.

What it reveals about incidents: a slow dependency does not just slow requests down; it multiplies the number of requests holding resources. Once pools are full, new requests queue or fail, even though the service itself is healthy.

Practical defenses: timeouts that cap latency, concurrency limits per dependency and load shedding when queues grow.

When requests arrive at random times, they sometimes bunch together. At low utilization the server clears bunches quickly; near full utilization there is no spare capacity to catch up, so queues build and waiting time grows much faster than the load.

For a simple single-server queue with random arrivals and service times, the average time in the system is:

time in system = service time / (1 − utilization)

What this means for operations:

• Going from 50% to 90% utilization makes average latency five times worse, not twice as bad.
• Tail latency rises even faster than the average.
• Running hot, at 85% to 95%, leaves no room for spikes, garbage collection pauses or the loss of an instance.

Practical implications:

1. Set utilization targets well below 100%, usually 60% to 75% for latency-sensitive services.
2. Alert on queue length and waiting time, not just on utilization.
3. Use load shedding so excess requests fail fast instead of waiting in long queues.
4. Reduce variability in service times, since variability makes queues worse at every utilization level.

Increase realistic load in steps until the service stops meeting its objectives, then keep going carefully to see how it fails. Knowing both the safe limit and the failure behavior is what makes a load test useful.

1. Realistic traffic: replay or model the real mix of requests, sizes and user behavior, including cache misses.
2. Production-like environment: same instance types, configuration, data volumes and dependencies, or a carefully isolated test in production.
3. Step the load: hold each level long enough for caches, autoscaling and garbage collection to settle.
4. Watch every layer: latency percentiles, errors, saturation of processors, memory, connection pools, queues and dependencies.
5. Find the knee: the load where latency starts rising sharply; set capacity plans below it.
6. Push past the limit: check that the service degrades gracefully, shedding load and recovering when load drops, rather than collapsing.

Common mistakes:

• Testing with one repeated request, which stays in cache and looks unrealistically fast.
• Load generators that become the bottleneck themselves.
• Forgetting downstream quotas, so a test overloads a shared dependency and affects other teams.

Record results with the build version, and rerun after significant changes.

Load shedding means rejecting some requests quickly when a service is overloaded, so it can keep serving the rest within its objectives. Without it, every request slows down, timeouts multiply and the service can collapse entirely.

How to detect overload:

• Queue length or time spent waiting in the queue.
• Concurrency above a limit.
• Latency above the objective for requests already in progress.

How to choose what to drop:

1. Priority classes: keep critical traffic, such as checkout and payment, and drop less important work first, such as recommendations, analytics or prefetching.
2. Request age: drop requests that have already waited longer than the client's deadline, since the answer would arrive too late anyway.
3. Retries before new requests: retries add load during overload, so shedding them first reduces amplification.
4. Fairness: limit each client to a share of capacity, so one heavy client cannot crowd out others.

How to reject well: respond quickly with a clear status, such as 503 with a suggested retry delay, and record shed requests as a metric. Rejecting early, before doing expensive work, is what makes shedding effective.

A cascading failure starts in one component and spreads, as overloaded or failing parts push extra load or waiting onto others. Prevention means limiting how much one failure can affect its neighbors.

flowchart TD
  A["One replica fails"] --> B["Remaining replicas take extra load"]
  B --> C["They slow down and time out"]
  C --> D["Clients retry, adding more load"]
  D --> E["More replicas fail"]
  E --> B

Defenses:

1. Capacity headroom, so the loss of one replica or zone does not overload the rest.
2. Timeouts on every call, so slow dependencies do not hold resources indefinitely.
3. Limited retries with backoff and jitter, plus retry budgets that cap retries as a share of traffic.
4. Load shedding that rejects excess work cheaply.
5. Circuit breakers that stop calling a failing dependency for a while.
6. Bulkheads: separate pools per dependency or tenant, so one cannot consume everything.
7. Graceful degradation: drop optional features rather than failing whole requests.
8. Slow, careful recovery: bring capacity back gradually, since a recovering service can be knocked over again by a surge.

Test these behaviors with load tests and fault injection, since they only matter when things are already going wrong.

Retries help with brief, random failures, but during an overload they add load to a dependency that is already struggling. When several layers each retry, the extra load multiplies, and the dependency cannot recover even after the original cause has passed.

How the amplification works: if each of three layers makes up to three attempts, one user request can become up to 27 requests at the bottom layer.

A retry budget limits retries to a share of normal traffic, such as 10%, per client. When failures exceed what the budget allows, the client stops retrying and fails fast, so retries never more than slightly increase the load.

Other practices:

1. Retry at one layer only, usually the one closest to the user, or the one that knows whether retrying is safe.
2. Exponential backoff with jitter, so retries spread out instead of arriving in waves.
3. Retry only safe failures: idempotent operations and transient errors, not validation errors.
4. Respect server hints, such as a suggested retry delay or a signal not to retry.
5. Count retries as a metric, since a rising retry rate is an early warning.

Give each request a deadline, propagate the remaining time to every downstream call and make each service stop working when the deadline passes. Fixed, independent timeouts at each layer either cut off useful work too early or let services keep working on requests the user has already abandoned.

sequenceDiagram
  participant U as User
  participant G as Gateway
  participant A as Service A
  participant B as Service B
  U->>G: Request with a 2,000 ms deadline
  G->>A: Remaining 1,900 ms
  A->>A: Database query takes 300 ms
  A->>B: Remaining 1,500 ms, minus a margin
  B-->>A: Response
  A-->>G: Response
  G-->>U: Response within the deadline

Rules of thumb:

1. Outer timeouts are longer than inner ones, so the inner call fails first and the caller can still respond, perhaps with a fallback.
2. Pass the deadline, not a fixed timeout, in request metadata.
3. Check before expensive work: if the remaining time is too short, fail immediately.
4. Base timeouts on measured latency, such as a little above the 99th percentile, rather than guesses.
5. Include retries in the budget: two attempts must both fit within the deadline.

Graceful degradation means a service keeps its most important functions working when parts of the system fail or are overloaded, by turning off or simplifying less important features. Users get a reduced experience instead of an error page.

Design steps:

1. Rank features by importance for each user journey: for a store, product details and checkout are essential, while recommendations and reviews are optional.
2. Classify dependencies as hard or soft, and make every soft dependency optional in code, with a timeout and a fallback.
3. Define fallbacks: cached data, default content, simpler algorithms or a message such as "reviews are temporarily unavailable".
4. Add switches: configuration flags that operators can use to disable expensive features during incidents or traffic peaks.
5. Shed optional work automatically when load or latency crosses thresholds.

Testing it:

• Disable each soft dependency in a test environment and confirm the journey still works.
• Run game days where optional services are switched off under load.
• Monitor how often fallbacks are used, so a silently degraded feature does not go unnoticed for weeks.

A bulkhead splits resources, such as threads, connections or whole instances, into separate pools, so one overloaded or failing part can use up only its own pool. The name comes from ship compartments that keep one leak from sinking the whole vessel.

flowchart LR
  A["Incoming requests"] --> B["Service"]
  B --> C["Pool for payments: 50 connections"]
  B --> D["Pool for recommendations: 20 connections"]
  B --> E["Pool for search: 30 connections"]
  C --> F["Payment provider"]
  D --> G["Slow recommendation service"]
  E --> H["Search cluster"]

When the recommendation service becomes slow, only its 20 connections fill up; payments and search keep working.

Common forms:

• Per-dependency pools for connections and workers.
• Per-tenant or per-client limits, so one customer's traffic cannot starve others.
• Separate deployments for critical and non-critical workloads, such as dedicated instances for checkout.
• Cell-based architecture: independent copies of the whole stack, each serving a subset of users, so a failure affects only one cell.

Trade-off: bulkheads waste some capacity, since idle pools cannot help busy ones; size them from measured usage and revisit regularly.

A thundering herd happens when many clients act at the same moment, such as all retrying when a service comes back, or all requesting the same missing cache entry. The synchronized burst can overload the system again just as it recovers.

Common triggers:

• A service restart, after which every client reconnects at once.
• A cache flush or expiry of a popular key, sending all requests to the database.
• Scheduled jobs that start on the hour across thousands of machines.

Defenses:

1. Jitter: randomize retry delays, reconnection times, cache expiry times and job start times.
2. Request coalescing: only one request loads a missing key, while others wait for its result.
3. Serve stale data while refreshing: keep returning the old value for a short time while one request fetches the new one.
4. Gradual recovery: admit traffic in steps after an outage, using load shedding or rate limits until the system is warm.
5. Warm caches before use: preload popular entries after a deployment or restart.
6. Connection limits on servers, so reconnecting clients are spread out rather than all accepted at once.

A metastable failure is a state where the system stays broken even after the original trigger is gone, because the failure itself creates the load that keeps it broken. Typical sustaining loops are retries, cache misses and work piling up in queues.

stateDiagram-v2
  state "Stable: normal load" as S
  state "Vulnerable: high load, little headroom" as V
  state "Metastable: failing under its own retries" as M
  [*] --> S
  S --> V: Load grows
  V --> S: Load falls
  V --> M: Trigger, such as a brief slowdown
  M --> M: Retries and timeouts keep load high
  M --> S: Load is cut deliberately

Why it is dangerous: fixing the trigger, such as restarting a slow database, does not help, because the retry load alone exceeds capacity.

How to recover:

1. Cut the sustaining load: stop retries, shed traffic, or block a share of clients temporarily.
2. Let queues drain and caches refill.
3. Restore traffic gradually, watching latency.

How to prevent it: retry budgets, load shedding, capacity headroom and limits on queue sizes, so the system cannot build up an unrecoverable backlog.

Limit each client's share of capacity, so a bug, a traffic spike or an abusive caller in one client cannot degrade the service for everyone else.

Techniques:

1. Per-client rate limits: requests per second per API key or tenant, enforced at the edge, with clear rejection responses.
2. Quotas: daily or monthly usage caps for expensive operations, such as exports.
3. Concurrency limits per client, which protect against slow, expensive requests better than rate limits.
4. Fair queuing: when overloaded, serve clients in turn rather than first come first served, so a flood from one client does not push others to the back.
5. Cost-aware limits: count expensive requests, such as large searches, as several units.
6. Separate capacity for the largest clients, so their load is isolated and planned.

Operating it:

• Monitor usage by client, and alert when one client's share jumps.
• Agree limits with internal teams and document them for external customers.
• Provide headers or responses that tell clients their remaining allowance, so well-behaved clients can slow down themselves.

Health checks decide whether an instance receives traffic or gets restarted, so a badly designed check can remove healthy instances or restart a whole fleet at once. Keep checks focused on what the instance itself controls.

Common mistakes:

• Checking a shared database in the liveness check: a database blip restarts every instance at once, turning a small problem into a total outage.
• Expensive checks that consume capacity at high frequency.
• Too aggressive thresholds: one slow response under load marks instances unhealthy, pushing more load onto the rest.

Safeguards: require several consecutive failures before acting, and have load balancers keep sending traffic to all instances if every instance looks unhealthy, since a shared cause is then more likely than all instances failing at once.

A dependency that is down fails fast: calls return errors immediately, and well-designed services switch to fallbacks. A slow dependency holds every call open, consuming threads, connections and memory while callers wait, so the slowness spreads upstream until callers are also unable to serve requests.

Why slowness is so damaging:

• Resources held per request grow with latency, so pools fill quickly.
• Health checks often still pass, so traffic keeps flowing to the struggling path.
• Retries after timeouts add more load to the slow dependency.

Defenses:

1. Tight timeouts based on the dependency's normal latency.
2. Concurrency limits per dependency, so only a bounded number of requests can wait on it.
3. Circuit breakers that open on high latency as well as errors.
4. Fallbacks that are used when the timeout or limit triggers.
5. Latency-based alerts for dependencies, so slowness is noticed before resources run out.

Testing: inject latency, not just errors, in fault-injection experiments; many systems handle a dead dependency well and a slow one badly.