Site Reliability Engineer Top 100 Interview Questions
100 site reliability engineer interview questions with structured answers, comparison tables, flow diagrams, worked calculations and runnable Python and shell. Covers service level objectives and error budgets, alerting, incident response, capacity and queueing, resilience patterns, observability, systems debugging and disaster recovery, from the basics to real outages.
Questions 81-100 of 100
Q81.On-call engineers are paged at night for problems that resolve on their own. What do you do?
Self-resolving pages are noise: they wake people without needing action, and they train responders to ignore alerts. Find out why each one fires, then fix the alert or the underlying behavior.
- List the alerts that paged and resolved without action over the last month.
- For each, ask:
- Is there real user impact during the condition? If not, it should not page.
- Is the window too short, so brief spikes trigger it?
- Is it a cause-based alert, such as high processor use, that rarely matters to users?
- Fix the alert:
- Switch to symptom-based, objective-based alerts.
- Use multi-window burn rates, or require the condition to persist.
- Downgrade to a ticket or dashboard if action can wait until morning.
- Fix the system: if the condition is a real but brief problem, such as nightly batch contention, remove it at the source.
- Track paging health: pages per shift, night pages and the share of actionable pages, reviewed weekly.
The goal is that every page means "a person must do something now".
Q82.A service needs restarting every week because its memory grows. Should you automate the restart or fix the leak?
Do both, in that order: automate safe restarts to remove the immediate toil and risk, then fix the leak, since automated restarts only hide a defect that may get worse.
Automating restarts safely:
- Restart instances one at a time, with readiness checks, so capacity is never reduced too much.
- Trigger on memory thresholds rather than a fixed schedule, so faster growth is handled too.
- Alert if restarts become more frequent, which signals the leak is worsening.
- Record each restart, so the toil is visible.
Why the leak still needs fixing:
- Growth rates often increase with traffic or new features, shrinking the time between restarts.
- Restarts lose in-memory state, such as caches, and can cause latency spikes.
- A leak may involve resources other than memory, such as connections, that cause harder failures.
Prioritizing the fix: estimate the engineering cost of finding the leak against the ongoing operational cost and risk. Put the fix on the backlog with an owner and a date, and treat the automation as temporary.
Q83.Customers report errors, but every dashboard is green. How do you investigate?
The monitoring is not measuring what customers experience. Either the failures happen before requests reach your instrumentation, or your indicators exclude the failing cases.
Where failures can hide:
- Before your servers: name resolution, certificates, content delivery, client networks or a load balancer rejecting requests.
- In the client: app crashes, script errors in browsers or a bad client release.
- In a segment: one region, one client version or one customer, too small to move global averages.
- Excluded by definition: errors classified as client errors, or a journey not covered by indicators.
- Partial failures: a successful response with wrong or missing content.
How to investigate:
- Get concrete examples: times, user identifiers, request IDs and screenshots.
- Look for those requests in edge and load balancer logs; if absent, the failure happened before them.
- Run synthetic probes from outside, in the affected regions.
- Break dashboards down by client version, region and customer.
Afterwards: add client-side and synthetic indicators, and review what the objectives exclude.
Interview tip: stating that customer reports are a monitoring signal, and that green dashboards plus unhappy users means the dashboards are wrong, is exactly the mindset interviewers look for.
Q84.A cache cluster restart caused a database overload. What happened, and how do you prevent it?
With the cache empty, every request became a database query at once. The database was sized for the cache's normal miss rate, not for full traffic, so it overloaded, and retries made it worse.
Why it escalated:
- Popular keys were requested by thousands of clients at the same moment, each going to the database.
- Slow database responses caused timeouts and retries, multiplying the load.
- Even after the cache came back, it filled slowly because the database could not answer quickly.
Prevention:
- Restart cache nodes one at a time, never the whole cluster, so most keys stay cached.
- Coalesce misses, so one request per key goes to the database while others wait.
- Warm the cache from a snapshot or with background loading before taking traffic.
- Protect the database with concurrency limits and load shedding.
- Serve stale values briefly when the database is slow.
- Add the scenario to game days, with the cache flushed under realistic load.
Example
The service handles 20,000 requests per second with a 97% hit rate; the database comfortably handles 2,000 queries per second.
- Normal database load:
20,000 × 0.03 = 600queries per second. - After the full restart: 20,000 queries per second, ten times capacity.
- Restarting one of eight nodes at a time empties only about an eighth of the keys, adding about
20,000 × 0.97 / 8 ≈ 2,400queries per second at worst, briefly. - With miss coalescing, many requests share each key's single load, keeping the extra load within the database's capacity.
Q85.How would you calculate burn rates over several windows and decide whether to page, in Python?
Compute each window's error rate, divide by the error rate the objective allows, then apply paired-window rules: page for a fast burn when both a long and a short window exceed a high threshold, and page or ticket for a slower burn when both exceed a lower one.
Typical rules for a 30-day objective:
- Fast burn: one-hour and five-minute windows both above 14.4.
- Slower burn: six-hour and thirty-minute windows both above 6.
Implementation notes:
- Read good and total counts per window from the metrics system, rather than averaging ratios.
- Handle windows with no traffic, which should not alert.
- Include the burn rates and the budget remaining in the alert text.
- Test the rules against past incidents, to check they would have fired in time and not during normal noise.
Example
The fast-burn rule does not fire because the one-hour window is just below its threshold, but the slower-burn rule does.
def burn_rate(bad, total, objective): return (bad / total) / (1 - objective) objective = 0.999 windows = {"5m": (90, 6_000), "1h": (1_000, 72_000), "30m": (300, 36_000), "6h": (2_600, 430_000)} rates = {name: burn_rate(bad, total, objective) for name, (bad, total) in windows.items()} for name, rate in rates.items(): print(name, round(rate, 1)) fast = rates["1h"] > 14.4 and rates["5m"] > 14.4 slow = rates["6h"] > 6 and rates["30m"] > 6 print("fast page:", fast, "| slow page:", slow) # Output: # 5m 15.0 # 1h 13.9 # 30m 8.3 # 6h 6.0 # fast page: False | slow page: TrueThe six-hour rate prints as 6.0 after rounding, but its exact value, about 6.05, is above the threshold.
Q86.How would you implement a retry budget in Python?
Give each client a token balance: every normal request adds a small fraction of a token, such as 0.1, and every retry spends one whole token. Retries are allowed only while tokens remain, so over time retries cannot exceed about 10% of requests, however many failures occur.
Design points:
- Ratio: the share of extra load you accept from retries.
- Cap: a maximum balance, so a long healthy period cannot save up a huge burst of retries.
- Scope: usually one budget per client process and per dependency.
- Combine with backoff and jitter, which spread out the retries the budget allows.
Why it works: during a brief, random failure, there are few retries and the budget is rarely exhausted. During an overload, failures are frequent, the budget runs dry and the client stops adding retry traffic, giving the dependency room to recover.
Expose the balance and the number of denied retries as metrics, since denied retries are a strong signal that a dependency is unhealthy.
Example
A dependency fails half of all first attempts; the budget limits retries to roughly one request in ten.
class RetryBudget: def __init__(self, ratio=0.1, max_tokens=10): self.ratio = ratio self.max_tokens = max_tokens self.tokens = max_tokens def on_request(self): self.tokens = min(self.max_tokens, self.tokens + self.ratio) def try_spend(self): if self.tokens >= 1: self.tokens -= 1 return True return False budget = RetryBudget() failures = retries = 0 for i in range(1_000): budget.on_request() if i % 2 == 0: failures += 1 if budget.try_spend(): retries += 1 print(failures, retries, f"{retries / 1_000:.1%}") # 500 109 10.9%Without the budget, all 500 failures would be retried, adding 50% load to a struggling dependency.
Q87.How would you calculate the availability of a dependency graph in Python?
Write two small functions: series availability, the product of the parts, and parallel availability, one minus the product of the parts' unavailability. Then compose them following the request path, to see which component limits the whole journey.
What the calculation is good for:
- Comparing design options, such as adding a replica or a second provider.
- Showing which component dominates the overall unavailability.
- Checking whether an objective is achievable before committing to it.
Limits to remember:
- Parallel formulas assume independent failures; shared causes, such as one configuration system, make real availability lower.
- Failover takes time, which the formulas ignore.
- Partial failures and slowness are not captured.
Use the results to guide design discussions, and compare them with measured availability from incident history.
Example
A checkout path runs through a load balancer, three application replicas, a database and two payment providers with failover.
from math import prod def series(*parts): return prod(parts) def parallel(*parts): return 1 - prod(1 - p for p in parts) load_balancer = 0.9999 app_tier = parallel(0.995, 0.995, 0.995) database = 0.9995 payments = parallel(0.999, 0.998) checkout = series(load_balancer, app_tier, database, payments) print(f"{app_tier:.7f}") # 0.9999999 print(f"{payments:.6f}") # 0.999998 print(f"{checkout:.4%}") # 99.9398%The database and the load balancer dominate the result, so they are the places to invest next.
Q88.How would you compare canary and baseline latency percentiles in Python to catch a regression?
Collect latency samples from canary and baseline instances over the same period, compute matching percentiles and flag the canary when a percentile rises by more than an agreed share, provided both groups have enough samples. Comparing tail percentiles matters, since many regressions hit only slow requests.
Design choices:
- Same window and similar traffic for both groups, so time-of-day effects cancel out.
- Minimum sample size, such as several thousand requests, before deciding.
- Relative and absolute thresholds: for example, a regression when the 99th percentile rises by more than 20% and by more than 20 ms.
- Several percentiles: the median for overall slowdowns, the 90th and 99th for tail effects.
Turning it into a rollback rule: run the comparison at each rollout stage, and stop or roll back when the regression is confirmed. Record the comparison with the release, so later investigations can see what was checked.
Example
The median barely moves, but the tail grows sharply in the canary.
import statistics baseline = [42, 45, 47, 48, 50, 51, 53, 55, 58, 60, 62, 65, 70, 75, 90, 120, 150, 210, 260, 300] canary = [44, 46, 48, 49, 51, 53, 55, 57, 60, 63, 66, 70, 78, 88, 110, 160, 220, 300, 380, 450] def percentile(samples, q): return statistics.quantiles(samples, n=100, method="inclusive")[q - 1] for q in (50, 90, 99): base, new = percentile(baseline, q), percentile(canary, q) change = (new - base) / base flag = "REGRESSION" if change > 0.20 and new - base > 20 else "ok" print(f"p{q}: {base:.0f} -> {new:.0f} ms ({change:+.0%}) {flag}") # Output: # p50: 61 -> 64 ms (+6%) ok # p90: 215 -> 308 ms (+43%) REGRESSION # p99: 292 -> 437 ms (+49%) REGRESSIONReal comparisons use thousands of samples; twenty per group only illustrates the logic.
Q89.How would you design a service level objective platform for many teams?
Let teams declare objectives in a standard format, generate the measurement, dashboards and alerts automatically and report budget status centrally. Teams then spend their time choosing good objectives, not building monitoring.
flowchart LR A["Objective definitions in version control"] --> B["Platform generates queries"] B --> C["Metrics store: good and total events"] C --> D["Burn-rate alert rules"] C --> E["Dashboards per service and journey"] C --> F["Budget reports for leadership"] D --> G["Paging and ticket routing"]Components:
- Definition format: service, journey, indicator as good and total event queries, target, window and owner.
- Validation: check definitions in the pipeline, such as queries that return data and targets within allowed ranges.
- Generated alerts: multi-window burn-rate rules with standard thresholds, routed to the owner.
- Dashboards: objective attainment, budget remaining and burn history, with deployment annotations.
- Reports: fleet-wide views of objectives met, budgets exhausted and coverage gaps.
- Policy hooks: release tooling that reads budget status, for example warning or blocking risky deployments when a budget is exhausted.
Start with a few critical journeys, prove the value, then expand.
Q90.How would you design a globally available service with multi-region failover?
Serve users from several regions, route each user to a healthy nearby region and design the data layer so a region can be lost without losing critical data or overloading the survivors.
flowchart TD A["Users worldwide"] --> B["Global traffic routing with health checks"] B --> C["Region A: full stack"] B --> D["Region B: full stack"] B --> E["Region C: full stack"] C <--> F["Replicated data layer"] D <--> F E <--> FKey decisions:
- Routing: latency-based routing with health checks that remove a failing region automatically; keep record lifetimes short for faster shifts.
- Capacity: each region sized so the remaining regions absorb a lost one at peak.
- Data: decide per data type whether it needs one primary region with replicas, or writes in every region with conflict resolution.
- Statelessness: keep sessions and caches region-independent, or rebuildable.
- Dependencies: every dependency available in each region, including identity, secrets and configuration.
- Failover practice: regular drills that drain a region under real traffic.
Trade-offs: multi-region writes add consistency complexity and cost; many services choose a single write region with fast, tested failover instead.
Q91.How would you design an incident management process for a growing company?
Create a lightweight, consistent process that anyone can start, with clear roles, severity levels, communication rules and reviews, then improve it through practice and incident data.
Core elements:
- Declaring incidents: anyone can declare one through a simple command or form, which creates a channel and notifies on-call responders.
- Severity levels based on user impact, with examples.
- Roles: incident commander, operations lead, communications lead and scribe, combined for small incidents.
- On-call rotations for each service, plus a pool of trained incident commanders for major incidents.
- Communication templates for internal updates and status pages.
- Reviews: a blameless review for every major incident, with tracked actions.
- Learning: a searchable archive of reviews and periodic analysis of patterns.
Growing it:
- Start simple; a heavy process is ignored by small teams.
- Train people with drills and shadowing.
- Measure time to detect, time to mitigate and action completion rates.
- Adjust severity definitions and roles as the organization and its systems grow.
Q92.How would you design synthetic monitoring for a critical user journey?
Run scripted probes that perform the journey from outside your network, on a schedule, from the regions where users are. They detect failures even when real traffic is low, and they see problems that internal metrics miss, such as name resolution, certificates and content delivery.
flowchart LR A["Probe schedulers in several regions"] --> B["Scripted journey: load, search, add to cart"] B --> C["Public endpoints"] B --> D["Results: success, step timings"] D --> E["Metrics and alerts"] D --> F["Saved failure details"]Design points:
- Cover the critical path, step by step, with assertions on content, not just status codes.
- Use test accounts and data that never affect real customers or metrics, and clean up any side effects such as test orders.
- Probe from several locations, and alert only when several fail, to avoid alerts from one probe's network.
- Record step timings and save details of failures, such as response bodies, for debugging.
- Run often enough to detect problems quickly, such as every minute for the most important journeys.
- Keep scripts maintained: update them with product changes, or they will fail for the wrong reasons.
Synthetic results complement, but do not replace, objectives based on real traffic.
Q93.How do you introduce site reliability practices into an organization that has none?
Start small with one important service, show measurable improvements and expand by helping teams adopt practices themselves, rather than imposing a large framework at once.
- Pick a service with visible pain: frequent incidents or noisy on-call.
- Define a few objectives with its product owner, and measure them.
- Fix alerting: move to symptom-based alerts tied to the objectives, and remove noise.
- Run blameless reviews for incidents, with tracked actions.
- Measure toil and automate the worst of it.
- Share results: fewer pages, faster recovery and budgets met, in terms leaders understand.
- Scale through enablement: templates, training, readiness reviews and a shared objective platform.
Embedded engineers Consulting team Platform team Reliability engineers sit inside product teams A central team advises teams on request A central team builds shared tools and standards Deep context, but hard to scale Scales well, but less hands-on Scales best, but needs teams to adopt tools Most organizations combine these models as they grow.
Q94.How do you work with product teams to agree on service level objectives?
Frame objectives as product decisions about what users need, supported by data on current performance and on what each level of reliability costs. Product owners should own the targets with engineering, not receive them as technical constraints.
How to run the conversation:
- Start with user journeys: which ones matter most, and what does a bad experience look like for each?
- Show current data: availability and latency over recent months, and incidents users noticed.
- Explain the trade-off: each extra nine costs more engineering time and slows feature delivery.
- Propose a target slightly below current performance, with an error budget and a policy for what happens when it runs out.
- Agree on the policy explicitly, including who can approve exceptions.
- Review regularly: compare budget use with customer feedback, and adjust.
Signs it is working: product owners ask about budget status in planning, and reliability work is prioritized without escalation when the budget is low.
Q95.How do you measure and improve the on-call experience?
Track on-call load and quality with data, ask responders directly how it feels and treat bad on-call as a reliability problem with engineering fixes.
Metrics to track:
- Pages per shift, split into working hours and nights.
- Actionable rate: the share of pages that required action.
- Time to acknowledge and time to mitigate.
- Repeat alerts: the same alert firing often without a permanent fix.
- Escalations: how often the primary needs help, which may mean missing documentation or access.
Qualitative input:
- A short survey at the end of each shift: stress level, interruptions and what would have helped.
- Handoff notes that mention recurring issues.
Improving it:
- Review alerts weekly, deleting or fixing noisy ones.
- Improve runbooks wherever responders got stuck.
- Fix the top recurring causes of pages as engineering work.
- Balance rotations, and compensate night work fairly.
- Set a team target, such as no more than two pages per shift on average, and act when it is exceeded.
Q96.What reliability metrics should you report to leadership?
Report a small set of metrics that connect reliability to customer experience and business risk, with trends and context, rather than raw technical data.
- Objective attainment for the most important user journeys, with budget used over the period.
- Customer-impacting incidents: count, severity and total impact in user minutes or failed transactions.
- Time to detect and time to restore for major incidents, as trends.
- Top reliability risks from the risk register, with planned mitigations.
- Incident action completion: open and overdue follow-ups.
- On-call health: pages per shift and night pages, as an indicator of sustainability.
- Cost of downtime estimates for major incidents, where available.
Presenting it well:
- Lead with what changed and what decisions are needed.
- Show trends over several periods, not single numbers.
- Explain incidents in business terms: what customers experienced and what it cost.
- Keep definitions stable, so trends remain meaningful.
Q97.How do you decide when a system needs a redesign rather than more fixes?
A redesign is justified when incidents keep coming from the same structural cause, fixes no longer reduce their rate and the cost of operating the system is rising faster than its value.
Signals that point to redesign:
- Recurring incident patterns: reviews keep naming the same architectural limitation, such as a single shared database.
- Capacity limits: growth requires heroic efforts each time, or the design cannot scale further.
- Rising toil: operational work grows with traffic despite automation.
- Fragility: small changes routinely cause unexpected failures, and teams fear deploying.
- Error budgets exhausted period after period despite reliability work.
How to make the case:
- Quantify the cost of the current design: incident impact, toil hours and slowed delivery.
- Estimate the redesign's cost and risk, including migration.
- Propose an incremental path, such as moving one journey or one data domain at a time, rather than a full rewrite.
- Define success metrics before starting, such as fewer incidents of the recurring type.
Rewrites have their own risks, so keep the old system stable during migration and avoid large, all-at-once cutovers.
Q98.How do reliability, availability and durability differ?
They describe different promises, and a system can be strong in one while weak in another. Clear terms help teams set the right objectives for each part of a system.
Reliability Availability Durability The system does what users expect, correctly and consistently, over time The system is up and responding when users need it Stored data is not lost once accepted Includes correctness, latency and availability together Usually measured as the share of successful requests or of time up Usually measured as the probability of losing data over a year Example: payments always charge the right amount, quickly Example: checkout responds successfully 99.95% of the time Example: confirmed orders survive any single disk, host or zone failure Why the distinction matters:
- A storage system can be briefly unavailable without losing data, which is often acceptable; losing data is not.
- A service can be available but unreliable, answering quickly with wrong results.
- Different targets need different designs: availability needs redundancy and fast failover; durability needs replication, backups and integrity checks.
Q99.How should you explain a site reliability project you personally worked on?
Tell one project as a story: the reliability problem, how you measured it, what you changed, the trade-offs you made and the measurable results.
- Context: the service, its users and scale, and why reliability mattered.
- Problem: for example, frequent incidents, noisy on-call or a capacity limit, with numbers.
- Your role: what you personally designed, built or led.
- Approach: how you measured the problem, which options you considered and why you chose yours.
- A hard moment: an unexpected failure mode, a disagreement or a migration risk, and how you handled it.
- Results: concrete outcomes, such as "pages per shift fell from nine to two" or "failover time dropped from 40 minutes to 4".
- Reflection: what you would do differently now.
What the interviewer is listening for:
- Data-driven reasoning, starting from measurements.
- Understanding of failure modes and trade-offs.
- Collaboration with product and development teams.
Interview tip: have the key numbers ready, since reliability work is judged by measured outcomes more than by technology choices.
Q100.How do you discuss a major incident you led?
Walk through the incident chronologically, showing how you coordinated the response, made decisions under uncertainty and turned the review into lasting improvements.
- Detection: how the incident was noticed, and how quickly.
- Impact: what users experienced, for how long and how severe it was.
- Coordination: how you assigned roles, kept communication flowing and avoided duplicated effort.
- Decisions: the mitigation options you considered, the one you chose and why, including any that did not work.
- Resolution: how service was restored and how you confirmed it.
- Review: the contributing factors you found, using blameless language.
- Follow-up: the actions that came out of it, and evidence that they reduced the risk.
What the interviewer is listening for:
- Calm leadership and clear communication under pressure.
- Prioritizing mitigation over diagnosis while users are affected.
- Honest reflection, including what could have gone better.
- Systemic fixes rather than blaming individuals.