Site Reliability Engineer Top 100 Interview Questions

A hedged request sends a duplicate of a slow request to a different replica after a short delay, and uses whichever response arrives first. Since slowness is often caused by one replica's temporary state, such as a garbage collection pause, the duplicate usually returns quickly.

How to configure it:

1. Delay before hedging: set it near the 95th percentile of normal latency, so only the slowest few percent of requests are duplicated.
2. Cancel the loser: when one response arrives, cancel the other request to save work.
3. Limit hedging: cap the share of hedged requests, so hedging cannot double load during a general slowdown.

When to use it:

• Read requests that are safe to repeat.
• Services with variable latency across replicas, such as storage or search backends.

When not to use it:

• Non-idempotent writes, which could be applied twice.
• When the whole service is overloaded: hedges add load exactly when there is no spare capacity.

A cache that absorbs most of the traffic becomes a hidden dependency: if it empties or fails, the backend suddenly receives far more load than it was sized for. Design so the system survives the cache being cold or missing.

Risks:

• Cold cache after restarts or deployments sends all requests to the database.
• Synchronized expiry of popular keys creates bursts of misses.
• Stale or inconsistent data after updates.
• Cache cluster failure taking down services that cannot run without it.

Practices:

1. Size the backend for realistic miss rates, or protect it with load shedding and request coalescing when the hit rate drops.
2. Warm caches before shifting traffic to new instances.
3. Add jitter to expiry times, and serve stale data while refreshing.
4. Monitor hit rate and alert on drops, since it predicts backend overload.
5. Restart cache nodes one at a time, never all at once.
6. Test with the cache disabled at reduced traffic, so you know the real limits.

Treat a planned peak as a project: forecast the load, test at that load, remove known risks and prepare people and playbooks before the day.

1. Forecast: estimate peak requests per second for each critical journey, using past events and marketing plans, and add a safety margin.
2. Load test the full journey at the forecast plus margin, including dependencies and third-party services.
3. Pre-scale: raise capacity ahead of time, since autoscaling may respond too slowly to a sudden surge, and confirm cloud quotas.
4. Reduce change risk: freeze non-essential deployments and configuration changes around the event.
5. Prepare degradation switches: decide which optional features to turn off if needed, and practice turning them off.
6. Protect the backend: queues for order processing, rate limits and a waiting room for extreme surges.
7. Staff the event: an on-call team with clear roles, a war room channel and direct contacts at key providers.
8. Review afterwards: compare forecast with actual load, and record lessons for the next event.

A chaos experiment deliberately introduces a failure to test a hypothesis about how the system behaves, such as "if one cache node fails, checkout latency stays within the objective". Safety comes from starting small, defining stop conditions and observing closely.

flowchart TD
  A["Define steady state: key metrics"] --> B["State hypothesis"]
  B --> C["Choose smallest useful scope"]
  C --> D["Set abort conditions"]
  D --> E["Inject failure"]
  E --> F{"Metrics within limits?"}
  F -->|No| G["Abort and restore"]
  F -->|Yes| H["Record results"]
  G --> H
  H --> I["Fix weaknesses, expand scope"]

Safety practices:

1. Start in test environments, then move to production with a small blast radius, such as one instance or a small share of traffic.
2. Abort automatically when objectives are threatened.
3. Announce experiments to on-call teams and stakeholders.
4. Run during working hours, with the owning team present.
5. Avoid stacking risks: do not experiment during incidents, peaks or freezes.

Typical experiments: terminating instances, adding latency to a dependency, failing a zone, filling a disk and expiring credentials.

Build confidence before the first real page through shadowing, practice incidents and good documentation, then keep skills fresh with regular drills. Untrained responders are slower, more stressed and more likely to make an incident worse.

A typical path for new on-call engineers:

1. Learn the systems: architecture walkthroughs, dashboards, runbooks and the most common past incidents.
2. Shadow shifts: follow an experienced engineer's shift, joining every page and incident as an observer.
3. Reverse shadow: take the lead while the experienced engineer watches and helps when needed.
4. Practice incidents: role-play sessions where a facilitator describes a failure step by step, and the trainee decides what to check and do next.
5. Access check: confirm the trainee has every permission and tool before their first solo shift.

Training incident commanders:

• Teach coordination skills separately from technical skills: assigning roles, running regular updates and making decisions with incomplete information.
• Let trainees command low-severity incidents first, with a mentor available.
• Review their incidents together afterwards, focusing on coordination.

Keeping skills fresh: rotate practice scenarios monthly, update runbooks after every drill and track how confident engineers feel through short surveys.

Spread instances and data across zones, with enough capacity in the remaining zones to carry full peak load when one zone disappears, and make sure no hidden component lives in only one zone.

Design checklist:

1. Capacity: with N zones, each zone must hold at least 1 / (N − 1) of peak load, so the survivors can absorb a lost zone.
2. Stateless tiers: spread replicas evenly, with load balancers that stop routing to a failed zone automatically.
3. Data stores: synchronous replicas in other zones, with automatic failover tested regularly.
4. Hidden single points: caches, queues, configuration services, network gateways and batch schedulers often run in one zone by default.
5. Zone-local traffic: keep calls within a zone where possible for latency, but make sure they can cross zones when needed.

Testing: simulate a zone loss by draining traffic from one zone during working hours, and check that latency and errors stay within objectives.

Choose by how quickly you must recover, how much data you can afford to lose and how much complexity and cost you can accept.

Questions that decide it:

1. What recovery time and data loss limits does the business need?
2. Can the data model handle writes in several regions, or must there be one primary?
3. Is the team able to run regular failover tests?

Whichever you choose, practice failovers regularly; an untested standby region often fails when needed.

Run real recovery exercises on a schedule, measure how long they take and how much data would be lost, and compare the results with the recovery objectives. A plan that has never been executed is a hypothesis.

What to test:

1. Backup restores: restore a production backup into an isolated environment, check integrity and time the process.
2. Failover: switch a service to its standby region or replica, then switch back.
3. Dependency loss: run without a key third-party service.
4. Access and tooling: confirm that people can reach consoles, credentials and runbooks when the primary environment is down.
5. Data consistency: verify that recovered data matches expectations, such as order counts.

Measuring:

• Actual recovery time against the recovery time objective.
• Actual data loss window against the recovery point objective.
• Manual steps that slowed the process.

Making it routine:

• Automate restore tests, for example weekly for critical databases.
• Run full failover exercises at least a few times a year.
• Turn every gap into an action item, and repeat the test after fixes.

Many major outages come from configuration, not code. Treat configuration with the same rigor as code: review it, validate it automatically and roll it out progressively with fast rollback.

1. Store configuration as code in version control, with review and history.
2. Validate before applying: schema checks, value ranges and tests that load the configuration into the service in a test environment.
3. Roll out progressively: one instance, one zone or one region first, with health checks between stages.
4. Make rollback instant: previous versions available and applied with one action.
5. Separate dangerous settings, such as traffic weights or feature switches affecting all users, with extra approval.
6. Avoid global changes in one step: a change that applies everywhere at once removes every chance to catch problems early.
7. Record changes as events on dashboards, so responders see them immediately during incidents.

Common causes of configuration outages: typos in values, missing entries, removing settings that something still uses and applying a change meant for one environment to another.

Compare the new version's signals with the current version's during the same period, and roll back automatically when the difference is both statistically clear and large enough to matter. Comparing against fixed thresholds alone misses regressions during quiet periods and raises false alarms during busy ones.

Signals to compare:

• Error rate and latency percentiles for the new version against the baseline.
• Key business signals, such as checkout completions per request.
• Resource use, such as memory growth, which can reveal leaks.

Designing the criteria:

1. Same time, same traffic: compare canary and baseline instances serving similar traffic during the same window.
2. Enough data: wait for a minimum number of requests before deciding, so noise does not trigger rollbacks.
3. Statistical test plus minimum effect: require both a clear difference and a meaningful size, such as 50% more errors.
4. Fast path for severe failures: roll back immediately on crashes or very high error rates, without waiting for statistics.
5. Log every decision, with the data behind it.

Narrow the problem from the user-facing symptom down to one component, using data at each step rather than guessing. The aim is to find where the extra time is spent, then why.

flowchart TD
  A["Latency objective breached"] --> B["Scope: which journeys, regions, clients, versions?"]
  B --> C["Correlate with changes: deploys, config, traffic"]
  C --> D["Use traces to find the slow span"]
  D --> E["Check that component's saturation and errors"]
  E --> F{"Waiting on a dependency?"}
  F -->|Yes| G["Repeat the analysis on the dependency"]
  F -->|No| H["Profile the component itself"]

1. Scope the impact: is it every request or a segment, such as one region, endpoint or client version?
2. Check what changed: deployments, configuration, traffic shape and dependency changes around the start time.
3. Follow traces: compare slow and normal requests to find the span that grew.
4. Inspect the component: saturation of processors, memory, threads, connections and queues, plus garbage collection.
5. Recurse: if it is waiting on something else, repeat the process there.
6. Mitigate first: roll back, shift traffic or shed load while investigating further.

Keep notes with timestamps during the investigation; they become the incident timeline.

They are two checklists that make investigations systematic. USE looks at resources, such as processors and disks; RED looks at services and the requests they handle. Together they cover both "what is busy?" and "what are users experiencing?".

How to use them together:

1. Start with RED on the affected service to confirm the symptom: higher latency, more errors or a traffic change.
2. Apply USE to that service's resources and its dependencies' resources, looking for saturation in particular.
3. Saturation is often the key signal: a resource at 60% utilization with a growing queue is a bottleneck, while one at 90% with no queue may be fine.

Build dashboards in the same structure for every service, so responders know where to look.

Every unique combination of label values creates a separate time series, so the number of series is the product of the number of values of each label. Labels with many possible values, such as user IDs, make the monitoring system slow, expensive or unstable.

Rules for labels:

1. Use labels with a small, bounded set of values: service, endpoint template, method, status class, region.
2. Never use unbounded values: user IDs, request IDs, full URLs with parameters, error messages or timestamps.
3. Normalize paths: use the route template, such as /orders/{id}, not the actual path.
4. Group fine-grained values: status classes such as 2xx and 5xx instead of every status code, when that is enough for alerting.
5. Aggregate early: for objective metrics, per-zone totals are usually enough; keep per-instance detail in a shorter-retention store.

For high-cardinality questions, such as "which customers had errors?", use logs or traces, which are designed for per-event detail.

Monitor series counts per metric, and reject or alert on sudden growth, which usually means someone added an unbounded label.

Storing every trace at high traffic is expensive, but random sampling throws away most of the rare errors and slow requests you need. Combine sampling methods so normal traffic is represented cheaply and unusual traffic is kept.

A practical combination:

1. Keep a small random share of all traces, such as 1%, for baselines.
2. Keep all traces with errors and those above a latency threshold, using tail-based decisions.
3. Keep all traces for important low-volume journeys, such as payments.
4. Allow forced sampling for debugging, through a request header used by engineers.

Record the sampling rate with each trace, so counts derived from traces can be scaled correctly.

Each log line should be a machine-readable record with consistent fields, so logs can be searched, filtered and joined with metrics and traces during an incident.

Fields every line should have:

• Timestamp in a standard format with time zone, at millisecond precision.
• Severity level, used consistently.
• Service, version and instance, so a problem can be tied to a release or host.
• Request and trace IDs, to follow one request across services.
• Event name describing what happened, such as payment_declined.
• Relevant context: route template, status, duration, and identifiers such as an order ID, where allowed.

Practices:

1. Log events, not prose: consistent event names and fields, rather than free-form sentences that differ every time.
2. Never log secrets or sensitive personal data; mask fields such as card numbers automatically.
3. Control volume: sample repetitive success logs, keep all errors.
4. Include the error's cause and stack trace in one record, not spread across many lines.
5. Agree on a shared schema across services, so queries work everywhere.

First decide whether the growth is a real leak or normal behavior, such as caches filling up, then find which process and which part of it is growing, and mitigate before the host runs out of memory.

1. Confirm the trend: memory growing steadily across days and restarts, rather than rising and leveling off.
2. Identify the process: compare resident memory per process over time.
3. Check the type of memory: application heap, file system cache, which the operating system frees when needed, or memory outside the heap such as buffers and native libraries.
4. Correlate with events: a release, a traffic change or a new feature that started the growth.
5. Use the language's tools: heap snapshots compared over time, or allocation profiling, to find which objects accumulate.
6. Check limits: container memory limits and how the process sizes itself, since a runtime that ignores the limit can be killed abruptly.

Mitigation while you fix it: scheduled rolling restarts before memory runs out, with alerts on the growth rate. Treat this as temporary toil, and track the real fix as an action item.

Load average counts tasks that are running or waiting to run, and on many systems it also counts tasks waiting in uninterruptible states, usually for disk or network storage. High load with idle processors usually means many tasks are blocked on input and output, not on computation.

Likely causes:

• Slow or saturated disks: heavy writes, a failing disk or a noisy neighbor on shared storage.
• Network storage problems: a remote file system that is slow or unreachable, leaving processes stuck.
• Swapping: the host is short of memory and pages are moving to and from disk.
• Lock contention in the kernel or on files.

How to investigate:

1. Check the share of processor time spent waiting for input and output.
2. Look at disk utilization, queue length and latency per device.
3. List processes in uninterruptible waiting states, and see which files or mounts they are using.
4. Check memory pressure and swap activity.
5. Check recent changes: backups, log rotation, batch jobs or new storage mounts.

The fix depends on the cause: faster or separate storage, reducing write volume, more memory or fixing the unreachable mount.

Every open file, socket and pipe uses a file descriptor, and each process has a limit. Running out causes errors such as failures to accept connections or open files, often at the worst time: under high load.

1. Confirm it: errors mentioning too many open files, and the process's descriptor count at or near its limit.
2. Find what is open: group the open descriptors by type, such as network sockets, regular files or pipes, and by destination.
3. Look for leaks: a count that grows steadily, even at constant traffic, points to connections or files that are never closed.
4. Look for load-driven growth: a count that rises with traffic and slow dependencies points to too many connections in progress, often because calls wait without timeouts.
5. Check connection states: many sockets waiting to close suggest connections opened and closed per request instead of reused.

Fixes:

• Close resources reliably, with language features that guarantee cleanup.
• Reuse connections through pools with a maximum size.
• Add timeouts so slow calls release sockets.
• Raise the limit only after fixing leaks, and set it explicitly in the service's configuration.
• Alert when usage passes a share of the limit, such as 80%.

Each outgoing connection uses a local port from a limited range. After a connection closes, the side that closed it keeps the port in a waiting state for a while, so a service that opens and closes many short connections to the same destination can run out of ports, and new connections fail.

Symptoms:

• Errors such as "cannot assign requested address" on outgoing connections.
• Many sockets in the post-close waiting state toward one destination.
• Failures that appear only at high request rates.

Fixes, in order of preference:

1. Reuse connections: keep-alive connection pools remove the problem at its source.
2. Spread destinations: more destination addresses or ports multiply the available combinations.
3. Widen the local port range, a short-term relief.
4. Tune reuse settings for the waiting state carefully, understanding the protocol risks.

Monitor socket counts by state, and include outgoing connection counts in load tests, since this problem rarely appears at low traffic.

Almost every call starts with a name lookup, so when name resolution is slow or failing, everything fails at once, often in confusing ways. Reduce how often services depend on live lookups, and make sure lookup failures fail fast.

Practices:

1. Cache results locally on each host, respecting record lifetimes, so brief resolver outages do not affect running services.
2. Reuse connections, which avoids a lookup for every request.
3. Set short lookup timeouts and limited retries, so a dead resolver does not stall requests for many seconds.
4. Run redundant resolvers in each zone, and monitor their latency and error rates.
5. Be careful with negative caching: a failed lookup cached for a long time can keep a service broken after the record is fixed.
6. Stage record changes: lower record lifetimes before planned changes, and validate records before publishing.

During incidents: when many unrelated services fail together, check name resolution early; it is a common shared dependency that is easy to overlook.

Include resolver failures in fault-injection tests, so you know how each service behaves when lookups fail.