DevOps Engineer Top 100 Interview Questions

Monitoring watches known signals and alerts when they cross thresholds. Observability is the ability to understand any internal state of a system from its outputs, including problems nobody predicted.

How they work together:

• Monitoring tells you that the checkout error rate has jumped.
• Observability lets you slice that error rate by version, region and customer type, and follow a failing request through every service to find the cause.

A system is observable when engineers can answer new questions without shipping new code to collect more data.

Metrics, logs and traces are the three main types of telemetry. Each answers a different kind of question, and investigations usually move from one to the next.

How they connect in an investigation:

1. A metric alert shows the error rate rising.
2. Traces for failing requests show which service and call is slow or failing.
3. Logs from that service, linked by the trace identifier, show the exact error.

Adding a shared request or trace identifier to all three is what makes moving between them quick.

The four golden signals are a small set of user-facing measurements that, together, show whether a service is healthy. If you can only monitor four things, monitor these.

Points that matter in practice:

• Measure the latency of successful and failed requests separately, because fast errors can make average latency look better while users suffer.
• Use percentiles rather than averages for latency, since averages hide the slow tail that frustrates users.
• Saturation is the early warning: a queue that keeps growing predicts failure before errors appear.

These signals suit request-driven services; batch jobs are better measured by throughput, run duration and how long work waits before it starts.

They form a chain from measurement to promise. An SLI measures a service's behaviour, an SLO is the internal target for that measurement, and an SLA is a contract with customers that has consequences if it is broken.

flowchart LR
  A["SLI: measured reliability"] -->|compared with| B["SLO: internal target"]
  B -->|kept stricter than| C["SLA: customer promise"]
  B --> D["Error budget for releases"]

Key rules:

• Define SLIs from the user's point of view, such as successful and fast page loads, not server CPU.
• Set the SLO stricter than the SLA, so the team reacts before customers are owed credits.
• Do not aim for 100%: every extra nine costs much more, and users often cannot notice the difference.

An error budget is the amount of unreliability an SLO allows over a period. It turns reliability into a quantity the team can spend on releases and experiments, and it gives a clear rule for when to slow down.

error_budget = 1 - SLO

• SLO: the target, such as 99.9%, written as 0.999.
• Error budget: the allowed share of failures, here 0.1% of requests or of time.
• Budget remaining: the allowed failures minus the failures that have already happened in the period.

How teams use it:

• Budget left: release normally, and take reasonable risks.
• Budget nearly spent: slow down risky changes and prioritise reliability work.
• Budget exhausted: freeze feature releases, except fixes, until the window recovers.

The budget makes the trade-off between speed and stability objective: both product and engineering agree on the SLO in advance, so nobody has to argue about it during an incident.

Availability is the share of time a service works. Each extra nine cuts the allowed downtime by a factor of ten, while the cost of achieving it rises steeply.

allowed_downtime = (1 - availability) * period

What the numbers mean in practice:

• At 99.99%, a single slow, manual recovery uses the whole monthly budget, so detection and failover must be automatic.
• A service can never be more available than the dependencies it needs, multiplied together.
• Redundancy raises availability: two independent copies, where either is enough, fail together far less often than one.

Agree on how downtime is measured, such as from the user's side and whether partial outages count, before promising any number.

Page a person only for problems that are urgent, user-facing and actionable; everything else goes to a ticket or a dashboard. Alert fatigue, where people ignore alerts because most are noise, is a real reliability risk.

1. Alert on symptoms, not causes: page on high error rates or slow responses that users feel, not on high CPU that may be harmless.
2. Base alerts on SLOs: alert when the error budget is being spent fast enough to matter, using burn rate, the speed of spending compared with an even pace.
3. Use two windows: a fast-burn alert pages quickly for severe problems, while a slow-burn alert opens a ticket for gradual decline.
4. Make every page actionable: include what is wrong, the impact, a dashboard link and a runbook.
5. Review regularly: after each on-call shift, remove or tune alerts that fired without needing action.

Trade-offs:

• Fewer alerts reduce noise but risk missing early warnings, so low-urgency signals still create tickets.
• Burn-rate alerts are more precise but harder to explain than simple thresholds.

Interview tip: say that an alert nobody acts on should be deleted or downgraded, because it trains people to ignore pages.

The goal during an incident is to restore service first and find the root cause later, with clear roles so effort is not duplicated.

1. Acknowledge and assess: confirm the alert is real and judge severity from user impact.
2. Declare and assign roles: an incident lead coordinates, one person investigates, and one handles communication.
3. Communicate: post an initial status for users and stakeholders, then update at regular intervals.
4. Mitigate: roll back, fail over, scale up or disable a feature to stop the impact, even before the cause is known.
5. Verify recovery: confirm with metrics that users are no longer affected.
6. Close and review: record the timeline and schedule a blameless review.

flowchart TD
  A["Alert fires"] --> B["Assess user impact"]
  B --> C["Declare incident and assign roles"]
  C --> D["Communicate status"]
  C --> E["Mitigate impact"]
  E --> F{"Service restored?"}
  F -->|no| E
  F -->|yes| G["Monitor and close"]
  G --> H["Blameless review"]

Trade-offs:

• Rolling back fast may lose evidence, so capture logs and metrics before or while mitigating.
• Too many people in an incident slows it down; the lead should keep the response group small and focused.

A blameless postmortem is a written review after an incident that focuses on how the system and processes allowed the failure, not on who made a mistake. People share facts honestly only when they are not afraid of punishment, and honest facts are what lead to real fixes.

What it should contain:

1. Summary: what happened, in two or three sentences.
2. Impact: who was affected, for how long, and how badly, in numbers.
3. Timeline: detection, key decisions and recovery, with timestamps.
4. Root causes and contributing factors: usually several, such as a missing test, a confusing alert and a risky default.
5. What went well: practices worth keeping.
6. Action items: specific changes, each with an owner and a due date.
7. Lessons: what the team now understands that it did not before.

What the interviewer is listening for:

• Language like "the deploy process allowed an untested config" rather than "the engineer forgot".
• Action items that change systems, such as adding a check, rather than "be more careful".
• Follow-through: action items tracked to completion.

Mean time to detect (MTTD) is the average time from a failure starting to someone knowing about it. Mean time to restore (MTTR) is the average time from the failure starting to service being back.

MTTD = sum of detection times / number of incidents

MTTR = sum of restore times / number of incidents

How to reduce detection time:

• Alert on user-facing symptoms with SLO-based burn-rate rules.
• Add synthetic checks that test key journeys every minute.
• Make sure alerts reach a person reliably, with escalation if unacknowledged.

How to reduce restore time:

• Keep rollback one command away and practise it.
• Write runbooks for common failures and link them from alerts.
• Use feature flags to switch off risky functionality quickly.
• Build observability so the failing component is found in minutes.

Means are easily skewed by one very long incident, so track the median too, and look at the slowest incidents individually.

Distributed tracing records the path of a single request through every service it touches. Each service records a span for its own work, and all spans share one trace identifier, so the full journey can be shown as a timeline.

1. The first service creates a trace identifier and a root span when the request arrives.
2. It passes the context on: the trace identifier and its span identifier travel in request headers to every downstream call.
3. Each service records a span with start time, duration, status and useful attributes, linked to its parent span.
4. Spans are sent to a collector, which assembles them into one trace.
5. The trace is shown as a waterfall, revealing which call was slow or failed.

sequenceDiagram
  participant G as Gateway
  participant O as Orders service
  participant P as Payments service
  participant D as Database
  G->>O: Request with trace id
  O->>D: Query, child span
  D-->>O: Result
  O->>P: Charge, child span
  P-->>O: Approved
  O-->>G: Response

Practical points:

• Every service must forward the trace context, or the trace breaks into pieces.
• Most systems keep only a sample of traces to limit cost, but keep all traces for errors and slow requests.

Send logs from every source to one central system, in a structured format with consistent fields, so they can be searched and correlated in one place instead of logging in to each server.

1. Write to standard output: containers write logs to the console, and the platform captures them.
2. Use structured logs: each line is a record with fields such as timestamp, level, service, version and request identifier.
3. Collect with agents: a lightweight agent on each node ships logs to the central system, buffering if it is unavailable.
4. Index and store: recent logs stay searchable; older logs move to cheaper storage according to a retention policy.
5. Search and alert: query by field, build dashboards, and create alerts for important patterns.

Good practices:

• Include a request or trace identifier in every line, so one request can be followed across services.
• Never log secrets, passwords or full personal data; mask them at the source.
• Control volume: debug logs everywhere can cost more than the servers that produce them.

Shifting security left means moving security checks earlier in the development process, towards the left of the timeline, so problems are found when they are cheapest to fix. A vulnerability caught while writing code costs minutes; the same one found in production can cost an incident.

Checks at each stage:

• While coding: editor plugins and pre-commit hooks flag secrets and risky patterns.
• On every commit: static analysis of the code, dependency vulnerability scanning and secret scanning.
• On build: container image scanning and generation of a software bill of materials.
• Before production: dynamic testing of the running application and infrastructure policy checks.
• In production: runtime monitoring, because some problems only appear with real traffic.

flowchart LR
  A["Code"] --> B["Commit checks"]
  B --> C["Build and image scan"]
  C --> D["Test environment scans"]
  D --> E["Production monitoring"]
  B -->|feedback in minutes| A

Shifting left does not mean removing later checks; it means most issues are caught early, with later stages as a safety net. Checks must be fast and low-noise, or developers start to bypass them.

The software supply chain is everything that goes into your software: dependencies, build tools, the pipeline and the artifacts it produces. Securing it means making sure every piece comes from where you think it does and has not been tampered with.

Practical steps:

1. Review new dependencies before adding them, preferring well-maintained ones with few transitive dependencies.
2. Update dependencies regularly, with automated pull requests and tests.
3. Allow production to run only artifacts signed by the trusted pipeline.

The goal is a verifiable chain from a reviewed commit to the running artifact, so any step that was skipped or altered is detected.

Least privilege means every identity, human or machine, gets only the permissions it needs, for only as long as it needs them. Pipelines and service accounts are attractive targets because they often hold powerful credentials.

1. One identity per purpose: separate identities for the build pipeline, each environment's deployment and each running service.
2. Narrow permissions: grant specific actions on specific resources, not administrator access.
3. Short-lived credentials: issue tokens per job or per session instead of long-lived keys.
4. Environment separation: pipelines triggered by unreviewed changes never receive production credentials.
5. Review usage: compare granted permissions with the permissions actually used, and remove the rest.
6. Audit and alert: log every use of sensitive permissions and alert on unusual activity.

Trade-offs:

• Fine-grained permissions take effort to maintain, and overly tight ones break pipelines; start broad in development and tighten using real usage data.
• Short-lived credentials need an identity trust setup between systems, but they remove the worst risk: a leaked key that works for years.

Keep secrets in a dedicated store, give each service its own, and rotate them on a schedule using a sequence that keeps the old and new values valid together for a short overlap.

1. Central store: all credentials live in an encrypted secret store with access control and audit logs.
2. Per-service secrets: each service has its own credentials, so one leak does not expose others.
3. Dynamic secrets where possible: generate short-lived database credentials on demand, so there is nothing long-lived to steal.
4. Rotate with overlap: create the new secret, update consumers, then revoke the old one.
5. Automate rotation: scheduled rotation makes it routine instead of a risky manual event.
6. Reload without restart: services re-read secrets from mounted files or the store, or restart gradually.

Trade-offs:

• Frequent rotation limits exposure but adds moving parts; automate it before increasing the frequency.
• Dynamic secrets are the most secure but require the secret store to be highly available, since services cannot start without it.

Find which filesystem is full, then which directories and files are using the space, and remove or move data only after you know what it is. Deleting the wrong file can break an application or remove data you need.

1. Check usage per filesystem: confirm which mount point is full; it may be a separate log or data volume.
2. Find large directories: measure the size of top-level directories and drill down into the largest.
3. Look for usual suspects: growing log files, core dumps, old releases, caches, temporary files and container images.
4. Check for deleted files still held open: a process can keep writing to a log that was deleted, so the space is not freed until the process restarts or reopens it.
5. Free space safely: compress or move old logs, clean caches with the proper command, and truncate an open log instead of deleting it.
6. Prevent recurrence: add log rotation, retention rules and an alert at around 80% usage.

Trade-offs:

• Expanding the disk is a quick fix that buys time, but without rotation the same disk fills again.
• Deleting old logs frees space immediately but may remove evidence needed for an investigation; archive them first if in doubt.

Confirm whether the CPU is really busy and with what kind of work, find the responsible process, then find what that process is doing, and only then decide whether to fix, scale or restart.

1. Check overall usage and load: separate user time, system time and time waiting for disk, since high disk wait is not a CPU problem.
2. Find the top processes: list processes sorted by CPU use and note the process ID and owner.
3. Look inside the process: check its threads, recent logs and whether its traffic has increased.
4. Correlate with changes: a recent deployment, a scheduled job, a traffic spike or a retry storm.
5. Profile if needed: take a CPU profile of the application to find the hot code path.
6. Act: roll back a bad change, scale out, stop a runaway job, or fix inefficient code.

flowchart TD
  A["High CPU alert"] --> B{"Mostly disk wait?"}
  B -->|yes| C["Investigate storage instead"]
  B -->|no| D["Find top processes"]
  D --> E{"Expected process?"}
  E -->|no| F["Check for runaway job or compromise"]
  E -->|yes| G["Compare traffic and recent changes"]
  G --> H["Profile, then fix or scale"]

An unknown process using a lot of CPU can be a sign of a compromise, such as a cryptocurrency miner, so treat it as a security event before simply stopping it.

Access logs record one line per request, and the status code sits in a fixed field. A short awk program can count each code and compute the share of server errors, which are codes starting with 5.

Approach:

1. Read the log line by line and take the status code from its field; in the common log format it is the ninth space-separated field.
2. Count requests per status code and in total.
3. Count codes that start with 5 as server errors.
4. At the end, print each code with its count, then the error count and percentage.
5. Guard against an empty log, so the percentage never divides by zero.

Key points:

• Use field positions that match your log format; a custom format may place the status elsewhere.
• Sort the output, because awk does not keep its arrays in any order.
• For large or rotated logs, read compressed files through a decompression stream instead of extracting them first.

The same pattern answers many operational questions, such as requests per path, per client address or per minute.

Treat it as an incident caused by the deployment until proven otherwise: confirm the impact, roll back quickly, and investigate once users are no longer affected.

1. Minute 0 to 2, confirm: check that the error rise started with the deployment and is real, not a monitoring glitch.
2. Minute 2 to 4, declare and communicate: open an incident, take or assign the lead role, and tell the team that a rollback is being considered.
3. Minute 4 to 10, mitigate: roll back to the previous artifact or turn off the feature flag, and stop any further rollout.
4. Minute 10 to 15, verify: confirm that error rates are back to the baseline and capture logs, traces and the diff of the release for analysis.
5. Afterwards: find the root cause, add a test or a canary check that would have caught it, and roll forward with a fix.

flowchart TD
  A["Error rate tripled after deploy"] --> B["Confirm timing and impact"]
  B --> C["Declare incident and stop rollout"]
  C --> D{"Feature flag can isolate it?"}
  D -->|yes| E["Turn off flag"]
  D -->|no| F["Roll back to previous artifact"]
  E --> G["Verify error rate recovered"]
  F --> G
  G --> H["Investigate and fix forward"]

Interview tip: interviewers listen for "roll back first, debug later"; trying to fix forward under pressure usually makes outages longer.