Site Reliability Engineer Top 100 Interview Questions

Expired certificates break connections instantly and completely, often for services nobody has touched in months. Prevention is almost entirely about inventory and automation.

1. Inventory every certificate: public endpoints, internal services, load balancers, client certificates and certificates embedded in devices or partner integrations.
2. Automate renewal and deployment, with short lifetimes, so renewal happens often and is well practiced.
3. Monitor expiry from the outside: probes that connect to each endpoint and read the certificate actually served, since renewed files that were never reloaded still expire.
4. Alert early: for example, a ticket 30 days before expiry and a page at 7 days for anything not renewed automatically.
5. Watch the whole chain: intermediate certificates and trusted roots can expire too.
6. Test reloads: confirm services pick up new certificates without restarts, or schedule rolling restarts after renewal.

After an expiry incident, add the missed certificate to the inventory and probes, and ask why automation did not cover it.

Clock skew is the difference between the clocks of different machines. Even with time synchronization, clocks drift apart by milliseconds or more, and a host with a broken synchronization setup can be off by minutes.

Practices:

1. Run time synchronization on every host, with monitoring of each host's offset and alerts above a threshold.
2. Use monotonic clocks to measure durations within a process.
3. Never rely on wall-clock time from different machines to decide correctness.
4. Include clock offset in incident checklists when errors involve tokens, expiry or ordering.

Intermittent resets usually come from something closing connections that the other side still thinks are open, from devices along the path or from overloaded endpoints. Find out who sent the reset and when, then match it to a timeout or limit.

Common causes:

• Idle timeouts: a load balancer, firewall or address translation device drops idle connections, and the next request on them fails.
• Keep-alive mismatch: a server closes idle connections sooner than the client expects.
• Connection tracking limits: a host or network device runs out of entries and drops new or old connections.
• Overload: a server's listen queue is full, so it refuses or resets new connections.
• Path problems: packet size mismatches along a route, which break large transfers but not small ones.

How to investigate:

1. Correlate resets with timing: after idle periods, at high load or for large payloads.
2. Capture traffic on both ends for a sample of failures to see which side sent the reset.
3. Compare idle timeouts at every hop, and make clients close idle connections first.
4. Check device and kernel counters for dropped connections.

Retrying idempotent requests once on a reset hides the residual cases, but fix the timeout mismatch first.

Most database incidents involve replication, failover or connection exhaustion. Plan and test each one explicitly, since database failures affect every service above them.

sequenceDiagram
  participant A as Application
  participant P as Primary
  participant R as Replica
  participant M as Failover manager
  A->>P: Writes
  P->>R: Replicate changes
  M->>P: Health check fails
  M->>R: Confirm replica is caught up
  M->>R: Promote to primary
  M->>A: Update endpoint
  A->>R: Writes resume

Replication:

• Monitor replication lag, and alert when it exceeds what reads from replicas can tolerate.
• Send reads that must see the latest data to the primary.

Failover:

• Automate it, with checks that prevent two primaries from accepting writes at the same time.
• Test it regularly, and measure how long applications take to reconnect.

Connections:

• Limit pool sizes per instance, so the total stays under the database's maximum even at full scale-out.
• Use a connection proxy when there are many application instances.

Other essentials: query timeouts, slow-query monitoring, capacity planning for storage and tested backups.

Stabilize first, then find whether the cause is too many clients, connections held too long or leaked connections, and fix the sizing so it cannot happen again at full scale.

Immediate steps:

1. Identify which applications hold the most connections, and how many are idle versus active.
2. Kill clearly idle or stuck sessions if safe.
3. Stop the scale-out that is adding connections, or reduce pool sizes on the largest consumers.

Common causes:

• Pool sizes multiplied by instance count: autoscaling adds instances, each with a full pool.
• Long transactions or slow queries holding connections.
• Leaks: code paths that do not return connections to the pool.
• Retry storms opening new connections during a slowdown.

Lasting fixes:

1. Size pools from the database's limit: maximum connections, minus a reserve for administration, divided by the maximum instance count.
2. Add a connection proxy that multiplexes many client connections onto fewer database connections.
3. Set statement and idle-in-transaction timeouts.
4. Alert on connection usage as a share of the maximum.

Queues decouple producers from consumers and absorb bursts, but they can hide failures: messages pile up, poison messages block progress, and duplicates arrive after retries. Reliability comes from designing consumers for these cases.

flowchart LR
  A["Producer"] --> B["Main queue"]
  B --> C["Consumer"]
  C -->|Success| D["Acknowledge and delete"]
  C -->|Failure| E{"Attempts below limit?"}
  E -->|Yes| B
  E -->|No| F["Dead-letter queue"]
  F --> G["Alert and investigate"]

Key practices:

1. Idempotent consumers: most queues deliver at least once, so processing the same message twice must be safe, for example by recording processed message IDs.
2. Acknowledge after processing, not before, so crashes do not lose messages.
3. Visibility timeouts longer than processing time, so messages are not redelivered while still being handled.
4. Retry limits and a dead-letter queue, so one bad message cannot block the queue forever.
5. Backpressure: limit queue size or producer rate, so a consumer outage does not grow an unbounded backlog.
6. Monitoring: age of the oldest message, depth, processing rate and dead-letter arrivals.

The most useful signal is the age of the oldest unprocessed message, because it directly measures how late work is. Queue depth alone can mislead: a deep queue that is draining fast may be fine, while a shallow queue with a stuck message is not.

Signals to track:

• Age of oldest message, compared with the freshness the business needs.
• Arrival rate and processing rate: if arrivals exceed processing, the backlog grows.
• Depth, as context for the other signals.
• Dead-letter arrivals, which indicate messages that cannot be processed.
• Consumer errors and retries.

Alerting:

1. Page when the oldest message age threatens the objective, such as orders older than ten minutes.
2. Ticket when arrivals exceed processing for a sustained period, before age becomes a problem.
3. Alert on any growth in the dead-letter queue for critical flows.

Estimating recovery: the time to clear a backlog is the backlog divided by the difference between processing and arrival rates, which tells responders whether adding consumers will help in time.

Treat the migration as a long-running production workload with its own capacity limits, monitoring and rollback plan, so it never competes with user traffic in ways that break objectives.

1. Rehearse on a copy of production data, measuring speed and checking results.
2. Make it resumable: process in batches with checkpoints, so a failure restarts from the last batch, not the beginning.
3. Make it idempotent: rerunning a batch gives the same result, with no duplicates.
4. Throttle: limit rows per second, and pause automatically when database latency, replication lag or error rates rise.
5. Run during quieter periods when possible, but design for it to be safe at any time.
6. Validate continuously: compare counts and checksums between source and target, and sample records.
7. Keep the old path working until the migration is verified, with a clear switch-over and switch-back plan.
8. Communicate: tell on-call teams what is running, its expected impact and how to pause it.

The most common failure is overloading a shared database, so the throttle and its automatic pause are the most important parts.

A backup is only as good as your last successful restore. Trustworthy backups are automated, isolated from the systems they protect, regularly restored and monitored.

Practices:

1. Automate backups for every critical data store, with schedules that meet the recovery point objective.
2. Isolate copies: store backups in a separate account or location, with write-once settings where possible, so a mistake or an attacker who reaches production cannot delete them.
3. Encrypt backups and manage the keys separately, making sure the keys themselves are recoverable.
4. Restore regularly and automatically: restore into an isolated environment, run integrity checks and compare record counts with production.
5. Measure restore time against the recovery time objective, including the time to download large backups.
6. Monitor: alert on failed, late or unusually small backups.
7. Cover everything needed to recover: configuration, secrets, schemas and infrastructure definitions, not just the data.

Common surprises discovered in restore tests: missing tables, backups that stopped months ago, restores far slower than expected and lost encryption keys.

Third-party services, such as payment providers, messaging services and identity providers, are dependencies you cannot fix yourself. Reduce how much a failure hurts, and make sure you know about failures quickly.

1. Classify each dependency as hard or soft for every user journey.
2. Know their commitments: published objectives, support terms and status channels.
3. Monitor them yourself: track latency and error rates of your calls, rather than relying on their status page.
4. Protect yourself: timeouts, retries with backoff, circuit breakers and concurrency limits on every call.
5. Degrade gracefully: queue work for later, such as sending emails when the provider recovers, or hide optional features.
6. Have alternatives for critical paths: a second payment provider or a fallback login method, tested regularly.
7. Watch quotas and rate limits, which can be reached during your own traffic peaks.
8. Plan communications: templates for telling users when a provider's outage affects them.

Include third-party failures in game days, since teams often discover hidden hard dependencies only when a provider goes down.

Keep a risk register that estimates, for each known risk, how often it might happen and how much user impact it would cause, then rank by expected impact and the cost of reducing it.

How to build the register:

1. Collect risks from incident history, readiness reviews, game days and team knowledge.
2. Estimate frequency and impact roughly; precision matters less than consistency.
3. Compare expected impact with the error budget, to see which risks could consume most of it.
4. Estimate the cost of each mitigation, and prioritize the best impact reduction per unit of effort.

Keep it current: update estimates after incidents and mitigations, and review the register quarterly with engineering leadership.

Estimate the direct revenue lost during the outage, subtract what customers recover later, then add indirect costs such as support, compensation and long-term customer loss. The number turns reliability debates into business decisions.

Components:

1. Lost transactions: revenue per minute at the time of day the outage happens, since peaks cost far more than quiet hours.
2. Recovery: the share of customers who come back and complete their purchase later.
3. Contract penalties: service credits owed under agreements.
4. Operational cost: responder time, support contacts and refunds.
5. Reputation and churn: harder to measure, but estimated from customer behavior after past incidents.

How to use it:

• Compare the expected cost of downtime with the cost of reliability investments, such as a second region.
• Choose objectives that reflect the business value of each journey.
• Prioritize incident follow-ups by their potential cost.

Use ranges rather than single numbers, and update them with data from real incidents.

Fit a trend to recent usage, project when it will reach capacity and alert on the time remaining rather than on a fixed usage percentage. A disk at 90% that grows slowly is less urgent than one at 60% that fills in two days.

How to do it:

1. Collect usage over a window long enough to show the trend but recent enough to reflect current behavior, such as the last one or two weeks.
2. Fit a simple trend, such as a straight line, or use the recent maximum growth rate for a cautious estimate.
3. Project time to exhaustion: remaining capacity divided by the growth rate.
4. Alert by lead time: a ticket when less than the time needed to add capacity remains, such as 14 days, and a page when only hours remain.

Pitfalls:

• Sudden step changes, such as a new log level, break trends; use a short window too.
• Cyclical patterns, such as nightly batch writes, need daily peaks rather than raw samples.
• Some resources, such as connection limits, fail suddenly rather than gradually.

Read each request line, count it as good if it succeeded and was faster than the latency threshold, then divide good requests by all requests. A short script is useful for quick checks during incidents or for services without a metrics pipeline yet.

Steps:

1. Decide the fields: here the third field is the status code and the fourth is latency in milliseconds.
2. Define a good request: status below 500 and latency at or below the threshold.
3. Count good and total requests, and print the ratio.
4. Compare it with the objective.

Making it reliable:

• Exclude invalid traffic, such as health checks, if the objective does not include it.
• Process logs in time windows matching the objective, such as one day.
• For production reporting, move the same logic into the metrics pipeline, so results are continuous and consistent.

Treating client errors such as 404 as good is a deliberate choice: the service answered correctly, so they usually do not count against reliability.

Group requests by day, classify each as good or bad using the objective's definition, report each day's ratio and compute how much of the period's error budget has been used. The budget used is the number of bad requests divided by the number the objective allows.

Calculation:

• Allowed bad requests: (1 − objective) × total requests.
• Budget used: bad requests / allowed bad requests.

Making it production-ready:

1. Read counts from the metrics system rather than raw logs, for completeness and speed.
2. Use the objective's real window, such as 30 rolling days, not calendar months, unless that is the agreement.
3. Show the days that consumed most of the budget, linked to incidents.
4. Publish the report automatically, so teams see budget status without asking.

A budget above 100% means the objective was missed for the period, which should trigger the error budget policy.

Treat it as an incident if the burn is still happening, then find what consumed the budget and decide, using the error budget policy, how the team's priorities change for the rest of the period.

1. Check whether it is ongoing: if the burn rate is still high, mitigate first, with a rollback, failover or load shedding.
2. Find the cause: look at when the bad events happened, which endpoints, regions and versions they affected, and what changed.
3. Confirm the measurement: a broken probe, a misconfigured indicator or a bot flood can burn budget without real user impact; fix the measurement if so, but do not erase genuine failures.
4. Apply the policy: with 60% left for the rest of the month, the policy may require slowing risky releases or prioritizing the fix.
5. Review the incident and assign actions, particularly for detection, if the burn was noticed late.
6. Communicate budget status to product owners, so changes to plans are expected rather than surprising.

Interview tip: saying you would check that the indicator itself was accurate, before changing team plans, shows maturity without dodging accountability.

A daily pattern points to something scheduled or to a daily traffic peak. Line up the latency increase with everything that happens at that time, in your service and in its dependencies.

Candidates to check:

1. Scheduled jobs: backups, batch exports, report generation, cache rebuilds or log rotation on the same hosts or databases.
2. Traffic peaks: the daily busiest hour, perhaps combined with too little capacity or slow autoscaling.
3. Dependency schedules: another team's batch job hitting a shared database or network link.
4. Cache expiry: entries with fixed lifetimes created at the same time each day, expiring together.
5. Maintenance tasks: database statistics updates, compaction or index rebuilding.
6. Certificate or credential refreshes that briefly slow connections.

How to confirm:

• Compare resource use on databases and hosts at that hour with other hours.
• Check job schedulers and scheduled-task configurations across teams for that time.
• Temporarily move a suspected job and see whether the spike moves with it.

Typical fixes: reschedule or throttle the batch job, isolate it on replicas, add jitter to expiry times or scale ahead of the daily peak.

If the same version works elsewhere, the difference is in that region's environment, data or traffic. Mitigate by rolling back or shifting traffic away, then compare the region with healthy ones.

1. Mitigate: roll back the region, or move its traffic to healthy regions if they have capacity.
2. Configuration: region-specific settings, feature flags or secrets that differ, including ones that were never updated.
3. Dependencies: a regional database, cache or third-party endpoint that behaves differently, or is on a different version.
4. Infrastructure: different instance types, network rules, quotas or a capacity shortage in that region.
5. Data: region-specific data, such as a locale, currency or character set, that triggers a code path not exercised elsewhere.
6. Traffic: different client versions or request mixes in that region.
7. Rollout order: if that region received the change first, it may simply be the first to show a general bug.

Prevent it next time: keep regional configuration in version control with validation, and roll out region by region with health checks between steps, so a regional problem stays regional.

The backup region was not truly ready to carry the primary region's traffic: capacity, caches, quotas or dependencies were sized for its normal load, not for the combined load after failover.

flowchart TD
  A["Primary region fails"] --> B["All traffic shifts to backup region"]
  B --> C["Caches are cold, autoscaling is slow"]
  C --> D["Databases and dependencies overload"]
  D --> E["Errors and retries add more load"]
  E --> F["Backup region fails too"]

Common causes:

• Capacity sized for normal regional traffic, relying on autoscaling that is too slow or limited by quotas.
• Cold caches sending all reads to databases.
• Connection limits on databases and third-party services.
• Retries from clients during the outage adding a surge on top.

Prevention:

1. Provision for failover load, or keep enough warm capacity plus fast scaling with confirmed quotas.
2. Shift traffic gradually when possible, rather than all at once.
3. Warm caches in the backup region by serving some real traffic there all the time.
4. Shed load during failover, prioritizing critical journeys.
5. Test full failover under realistic load regularly, not just the switching mechanism.

When one dependency can take down everything, the architecture has a single point of failure in practice, even if each service looks independent. Reduce how much each service depends on it and how many services one failure can reach.

Ways to reduce blast radius:

1. Make it a soft dependency where possible: caches, defaults or queued work, so services keep running in a reduced mode.
2. Partition the dependency: separate instances per domain or per tier, so critical services do not share a cluster with batch workloads.
3. Cell-based architecture: split users across independent copies of the stack, so one failing cell affects only its share of users.
4. Bulkheads and timeouts in every client, so a slow dependency cannot exhaust callers' resources.
5. Staged changes to the dependency, rolled out to one partition at a time.
6. Map dependencies: keep an up-to-date graph showing which services and journeys rely on each shared component.

flowchart LR
  A["Users"] --> B["Cell 1: full stack, 25% of users"]
  A --> C["Cell 2: full stack, 25% of users"]
  A --> D["Cell 3: full stack, 25% of users"]
  A --> E["Cell 4: full stack, 25% of users"]

With cells, a failure in one shared component affects a quarter of users rather than everyone.