Site Reliability Engineer Top 100 Interview Questions

A site reliability engineer keeps services reliable at scale by treating operations as an engineering problem: measuring reliability, automating repetitive work and changing systems so failures are rarer and smaller.

Typical responsibilities:

• Service level objectives: defining what "reliable enough" means with product teams, and tracking it.
• Incident response: being on call, leading incidents and running reviews that produce lasting fixes.
• Capacity and performance: forecasting demand and making sure systems scale before they hit limits.
• Resilience design: timeouts, retries, load shedding, redundancy and failover.
• Automation: removing toil, the manual and repetitive operational work that grows with the service.
• Release safety: progressive rollouts, automatic rollback and production readiness reviews.
• Observability: making sure teams can see and debug what their systems are doing.

The balance matters: a common guideline caps operational work at about half of an engineer's time, so the rest goes to engineering that reduces future operational load.

Start from what users experience during a critical journey, such as checkout, and measure the share of events that were good from their point of view. A good indicator is a ratio of good events to valid events, so it reads naturally as a percentage.

Principles:

1. Few indicators per journey, usually availability and latency, plus freshness or correctness where they matter.
2. Measure close to the user, such as at the load balancer or in the client, so failures before your servers are counted.
3. Exclude invalid events, such as requests with bad credentials, but never exclude errors your system caused.
4. Use thresholds, not averages, for latency, since averages hide the slow experiences users notice.

Validate indicators against real user pain: past incidents should show up clearly as drops.

Each measurement point sees a different slice of failures. The closer to the user, the more failures you capture, but the noisier and harder to control the data becomes.

Practical approach:

1. Use load balancer metrics as the main indicator for most services, since they capture server crashes and overload.
2. Add synthetic probes that run the journey from outside, which catch problems even with no real traffic.
3. Use client data to validate, especially for mobile apps and regions with poor networks.
4. Keep server metrics for debugging, since they explain why requests failed.

When the measurements disagree, investigate: a gap between client and load balancer data often reveals problems with networks, content delivery or certificates.

Start with a reasonable target based on what users need and what the system currently achieves, measure for a few weeks, then adjust. The first target is a hypothesis, not a promise.

1. Measure current performance for a few weeks, even before agreeing a target.
2. Ask what users need: an internal batch report tolerates much more failure than checkout.
3. Check dependencies: a service cannot be more reliable than the hard dependencies it needs for every request.
4. Pick a target slightly below current performance, so the error budget is not exhausted by normal operation, while still catching real regressions.
5. Review after one or two periods, comparing budget burn with how users felt: if users complained while the objective was met, tighten it or change the indicator; if the budget burned while nobody noticed, loosen it.

Avoid common mistakes:

• Choosing many nines because they sound good; each extra nine costs far more engineering effort.
• Setting the target equal to the service level agreement promised to customers; keep the internal objective stricter, so you have warning before contractual penalties.

An error budget policy is a short agreement, signed off by product and engineering leaders, that says how the team changes its priorities as the budget is used up. Agreeing it in advance turns a stressful negotiation during an incident into following a plan.

flowchart TD
  A["Budget healthy"] --> B["Normal feature work and releases"]
  C["Budget burning fast"] --> D["Investigate, slow risky releases"]
  E["Budget exhausted"] --> F["Freeze non-critical changes"]
  F --> G["Prioritize reliability fixes until budget recovers"]

What the policy should state:

1. The objectives it covers and the measurement window, such as 30 rolling days.
2. Thresholds and actions: for example, at 50% used, review upcoming risky changes; at 100%, freeze releases except fixes and security patches.
3. Exceptions: who can approve an urgent release during a freeze.
4. Ownership: who checks the budget and announces changes in status.
5. Escalation: what happens if a single incident uses more than a set share of the budget, such as a mandatory review.

Using it well: the budget is also permission to take risk. If it is mostly unused, the team can release faster or run experiments, instead of making the system more reliable than users need.

Burn rate is how fast the error budget is being consumed, relative to the rate that would use exactly the whole budget over the objective's window. A burn rate of 1 uses the budget exactly on time; a burn rate of 10 uses it ten times faster.

burn rate = observed error rate / (1 − objective)

Why alert on burn rate:

• It ties alerts to user impact as defined by the objective, not to arbitrary thresholds.
• It scales alert urgency: a fast burn deserves a page, a slow burn deserves a ticket.

Typical alert design for a 30-day window:

• Page when 2% of the budget is used within one hour, which is a burn rate of about 14.4.
• Page when 5% is used within six hours, a burn rate of 6.
• Ticket when 10% is used within three days, a burn rate of 1.

Choosing thresholds: compute how long the budget would last at each burn rate, and decide how much warning each situation needs. A burn rate of 14.4 empties a 30-day budget in about two days, which needs attention now.

A single window forces a bad trade-off: a short window reacts fast but pages for brief blips, while a long window is stable but keeps alerting long after the problem is fixed. Requiring both a long and a short window to exceed the threshold gives fast, accurate alerts that also clear quickly.

How it works:

• Long window, such as one hour: confirms that enough budget has been used to matter.
• Short window, such as five minutes, usually a twelfth of the long one: confirms the problem is still happening right now.
• Page only when both exceed the burn-rate threshold.

What it fixes:

1. Brief spikes: a few minutes of errors raise the short window but not the long one, so nobody is woken for a blip that has already passed.
2. Stale alerts: after recovery, the short window drops below the threshold within minutes, so the alert resolves even though the long window still includes the bad period.
3. Sustained problems: both windows rise together, so real issues still page quickly.

Pair each threshold with its own window pair, such as one hour and five minutes for fast burns, and six hours and thirty minutes for slower ones.

When a request needs every component, the components are in series, and their availabilities multiply. When any one of several redundant components can serve a request, they are in parallel, and the system fails only if all of them fail at once.

• Series: A = A1 × A2 × … × An
• Parallel: A = 1 − (1 − A1) × (1 − A2) × … × (1 − An)

Lessons for design:

• Every hard dependency lowers the ceiling: a chain of good components can still be mediocre overall.
• Redundancy raises availability dramatically, but only if failures are independent; two replicas sharing a power supply, a configuration or a bad release fail together.
• Removing a hard dependency, through caching or graceful degradation, often helps more than making it more reliable.

Caveats: these formulas assume independent failures and ignore failover time, so treat results as upper bounds, and validate them against measured incident history.

If your service calls a dependency on every request and cannot work without it, your availability can be no better than that dependency's, and in practice somewhat worse. Before committing to an objective, list hard dependencies and their objectives.

Ways to break the constraint:

1. Graceful degradation: serve a reduced response when the dependency fails, such as hiding recommendations instead of failing the page.
2. Caching: serve recent values when the source is unavailable, if slightly stale data is acceptable.
3. Asynchronous processing: accept the request, queue the work and complete it when the dependency recovers.
4. Redundant providers: two independent sources, with failover.
5. Negotiate: ask the dependency's owners for a stronger objective, with evidence of why you need it.

Practices:

• Record each dependency as hard or soft, and test soft ones by disabling them in a staging environment.
• Track dependency availability separately, so incidents can be attributed accurately.
• Set your own objective with margin below the product of your hard dependencies.

For batch and data systems, users care about whether results arrive on time and are correct, not about request success. Define indicators around completion time, freshness and data quality.

Useful indicators:

• Timeliness: the share of runs that complete before their deadline, such as daily reports ready by 7 in the morning.
• Freshness: the share of time, or of reads, where data is no older than a threshold, such as dashboard data less than one hour old.
• Completeness: the share of expected records or partitions present.
• Correctness: the share of runs passing data quality checks, such as totals reconciling with source systems.

Setting the objective:

1. Base deadlines on what consumers need, such as a finance team's morning review.
2. Measure over enough runs to be meaningful: for a daily job, a monthly window gives only about 30 data points, so consider a quarter.
3. Alert on predicted lateness, such as a job running slower than usual, not only on missed deadlines.

Example statement: "99% of daily sales partitions are complete and validated by 6:00, measured over 90 days."

Latency objectives should count the share of requests faster than a threshold, computed from all requests in the window. The common mistakes are averaging percentiles and computing percentiles per server, both of which give numbers that no single request distribution actually has.

Pitfalls:

• Averaging percentiles: the average of hourly 99th percentiles is not the 99th percentile of the day.
• Per-instance percentiles: combining percentiles from many servers is not mathematically valid.
• Averages: a mean of 80 ms can hide 5% of requests taking two seconds.

What to do instead:

1. Record histograms with buckets that include the threshold, such as 100, 200, 300 and 500 ms; bucket counts can be added across servers and time windows.
2. Define the objective as a ratio: requests under 300 ms divided by all valid requests, which is exact if 300 ms is a bucket boundary.
3. Estimate percentiles from merged histograms for dashboards, knowing they are approximate within a bucket.
4. Choose bucket boundaries deliberately, denser around the threshold.

When one user request waits for responses from many backends in parallel, the slowest of them decides the total time. Even if each backend is slow only 1% of the time, a request that touches many of them will very often hit at least one slow response.

P(at least one slow) = 1 − (1 − p)^n, where p is the chance that a single call is slow and n is the number of calls.

What this means:

• A backend's 99th percentile becomes the typical experience for requests with large fan-out.
• Improving the median of backends does little; reducing their tail matters most.

Techniques to reduce the impact:

1. Hedged requests: send a second request to another replica if the first is slow, and use whichever answers first.
2. Partial results: return with the responses received by a deadline, for search or feeds where completeness is not essential.
3. Reduce fan-out: group data so fewer backends are needed per request.
4. Remove tail causes: garbage collection pauses, noisy neighbors, background compaction and lock contention.

Toil is operational work that is manual, repetitive, automatable, reactive, without lasting value and growing with the service. Restarting a stuck job by hand every week is toil; designing a fix that stops it getting stuck is engineering.

How to measure it:

1. Ask engineers to log time on operational tasks for a few weeks, by category.
2. Count interrupts: tickets, manual requests and pages per week.
3. Track the share of team time spent on toil, with a target cap, such as 50%.

How to reduce it:

• Eliminate: fix the cause, such as the bug that makes the job hang.
• Automate: script or build self-service for repeated requests, such as access grants or environment setup.
• Change the process: batch requests, or remove approvals that add no value.
• Push back on design: services that need regular manual care should not pass production readiness review.

Prioritize by frequency times time per occurrence, weighted by risk, since manual steps during incidents are both slow and error-prone.

Interview tip: giving a concrete example, such as "we spent six hours a week on certificate renewals until we automated them", is far more convincing than defining toil.

Rank candidate tasks by the time they consume, the risk of doing them by hand and how often they block other people, then compare that with the cost of building and maintaining the automation.

Factors to weigh:

1. Frequency and duration: total hours per year spent on the task.
2. Error risk: manual steps that can cause outages, such as editing production configuration, deserve priority even if rare.
3. Waiting time for others: requests that block developers, such as creating environments, have hidden costs.
4. Incident relevance: steps performed during incidents should be automated or scripted, since people make mistakes under pressure.
5. Stability of the task: automating a process that changes every month costs more to maintain.

Approach:

• Start with a documented manual procedure, then a script, then full self-service, improving step by step.
• Measure the time saved after automation, to justify the next project.
• Remember that automation needs maintenance; budget for it.

A production readiness review checks that a service can be operated safely before it takes real traffic, or before a reliability team agrees to share its on-call. It uses a standard checklist, so expectations are clear and consistent across teams.

Areas the review covers:

1. Objectives: indicators and targets agreed with the product owner, with dashboards.
2. Monitoring and alerts: symptom-based alerts tied to the objectives, with runbooks.
3. Capacity: load tested to expected peak plus margin, with autoscaling or a capacity plan.
4. Resilience: timeouts, retries with backoff, graceful degradation and tested failover.
5. Deployments: automated, progressive and quickly reversible.
6. Dependencies: listed, with their objectives and the behavior when each fails.
7. Data: backups that have been restored in a test, and documented recovery objectives.
8. Security: access controls, secret handling and patching process.
9. Operations: on-call rotation, escalation paths and incident procedures.

Running it well:

• Make the checklist available early, so teams design for it rather than discovering gaps at the end.
• Classify gaps as launch blockers or follow-ups with owners and dates.
• Repeat a lighter review after major changes.

On-call is sustainable when the load is low enough that engineers can respond well, recover afterwards and still do project work. A burned-out rotation produces slow responses, mistakes and people leaving.

Signs of a healthy rotation:

• A small number of pages per shift, typically no more than a couple of real incidents.
• Most pages are actionable and lead to a fix.
• Nights are rarely interrupted.
• Engineers feel prepared, with runbooks and access they need.

Practices that help:

1. Enough people: at least six to eight in a single-site rotation, so each person is on call about one week in six or eight.
2. Follow-the-sun: split shifts across time zones, so nobody is regularly woken at night.
3. Primary and secondary: a backup who can take over if the primary is unavailable.
4. Compensation and recovery time after heavy shifts.
5. Handoffs: a short summary of open issues at each shift change.
6. Review alerts weekly: fix or delete noisy alerts, and track pages per shift as a metric.
7. Training: shadow shifts and practice incidents for new members.

When on-call load exceeds the target, the policy should be to spend engineering time reducing it, not to ask people to cope.

Page only when a person must act now to prevent or stop user impact. Everything else is a ticket for working hours or a dashboard signal for investigation.

Questions to ask for each alert:

1. Does it indicate real or imminent user impact?
2. Can the on-call engineer do something about it?
3. Would it be acceptable to handle it next morning?
4. Has it fired falsely in the past month?

Alert on symptoms users feel, such as errors and latency, rather than on causes, such as high processor use, which may or may not matter.

An actionable alert tells the responder what is wrong for users, how bad it is and where to start, so they can act within minutes without reconstructing context.

What the alert should contain:

1. A clear summary: "Checkout error budget burning at 15 times the allowed rate", not "metric above threshold".
2. Impact: affected service, journey, region and the current error or latency level.
3. Since when: the start time of the condition.
4. Links: the relevant dashboard filtered to the right time and service, recent deployments and the runbook.
5. Owner and severity, so routing is correct.

Design rules:

• Alert on symptoms, and use cause-based signals in the runbook to guide diagnosis.
• Require persistence over a short window, to avoid alerts on single spikes.
• Make it testable: fire it deliberately in a staging environment to confirm routing and content.
• Review after every page: was it timely, was it actionable, did the runbook help?

Signs an alert needs work: responders routinely acknowledge it and do nothing, or always ask the same first question that the alert could have answered.

Separating roles stops one person from trying to debug, coordinate and communicate all at once, which slows everything down during serious incidents. For small incidents one person may hold several roles; for major ones, assign them explicitly.

How it works in practice:

1. The first responder is commander until they hand the role over explicitly.
2. Everyone knows who the commander is, announced in the incident channel.
3. Handoffs of any role are explicit, with a summary.
4. The commander declares when the incident is resolved and starts the review process.

Practicing these roles in drills makes them natural when a real incident happens.

Severity should follow user and business impact, using a short, written scale everyone applies the same way. It decides who gets involved, how often to communicate and how deep the review must be.

Typical levels:

• Severity 1: a critical journey is down or badly degraded for many users, data loss is happening, or there is a security breach. All hands, executive notification, public communication.
• Severity 2: significant degradation for some users or a non-critical feature down. Dedicated responders and regular internal updates.
• Severity 3: minor impact with a workaround, or a risk that could become serious. Handled by the owning team during working hours.

Rules that keep it consistent:

1. Base the level on impact, not on how hard the problem is to fix.
2. Declare quickly, and upgrade or downgrade as facts emerge; starting too high is cheaper than starting too low.
3. Include concrete examples for each level, such as "checkout errors above 5% for more than five minutes".
4. Review severity choices in incident reviews, and refine the scale.

Interview tip: saying you would rather declare an incident early and downgrade it than wait for certainty shows good incident judgment.