DevOps Engineer Top 100 Interview Questions

A flaky test passes and fails on the same code without any change, usually because of timing, shared state, test order or external dependencies. Flaky tests are dangerous because people learn to ignore red builds, and then real failures slip through.

Common causes:

• Fixed sleeps or timeouts instead of waiting for a condition.
• Tests that depend on each other or on leftover data.
• Calls to real networks, clocks or random values without control.
• Resource limits on shared build machines.

How to deal with one:

1. Detect: rerun failed tests automatically and record which ones pass on retry; those are flaky.
2. Quarantine: move the test out of the blocking pipeline and create a ticket with an owner and a deadline.
3. Fix the cause: wait for conditions, isolate data, fake time and external services.
4. Restore it: return the test to the pipeline once it passes reliably many times in a row.

Interview tip: say that automatic retries hide flakiness; they are useful for measuring it but should not become the permanent fix.

The test pyramid is a guide to the proportion of test types: many fast unit tests at the base, fewer integration tests in the middle, and a small number of slow end-to-end tests at the top. It keeps a pipeline fast while still testing the system as a whole.

How it shapes the pipeline:

• Unit tests run on every commit and must finish in a few minutes.
• Integration tests run after packaging, often against temporary databases or service stand-ins.
• End-to-end tests run against a deployed test environment and cover only the critical paths, such as sign-up or checkout.

An inverted pyramid, with many end-to-end tests and few unit tests, leads to slow, flaky pipelines where a failure does not say which part of the code is wrong.

Store secrets in a dedicated secret store, give each pipeline only the secrets it needs, and inject them at runtime without ever writing them into code, logs or artifacts.

1. Central storage: keep secrets in the pipeline system's encrypted store or a separate secret manager, never in the repository.
2. Least privilege: scope each secret to the specific project, branch and environment that needs it.
3. Short-lived credentials: prefer tokens issued per job through workload identity federation over long-lived keys.
4. Masking: make sure the pipeline hides secret values in its logs, and avoid printing environment variables.
5. Protected branches only: production credentials are available only to jobs running from protected branches, not to pull requests from forks.
6. Rotation and audit: rotate secrets regularly and record who or what used them.

Trade-offs:

• A separate secret manager adds setup work, but it centralises rotation and auditing across many pipelines.
• Short-lived credentials remove the risk of leaked long-term keys but need identity trust between the pipeline and the target environment.

A build is reproducible when the same source code always produces the same output, no matter when or where it is built. This lets you trust that a rebuilt artifact matches the original and makes tampering easier to detect.

• Pinned dependencies: a lock file records exact versions and checksums, so builds never pick up newer releases unexpectedly.
• Pinned tools: the compiler, runtime and build tools have fixed versions, often by building inside a versioned container image.
• Clean environment: each build starts from a fresh, isolated workspace rather than a machine with leftover state.
• No hidden inputs: avoid embedding the current time, random values or machine names in the output, or fix them to known values.
• Recorded provenance: store the commit, dependency list and build settings with the artifact.

If two independent builds of the same commit produce byte-for-byte identical artifacts, the build is fully reproducible. Many teams settle for a weaker but useful goal: identical behaviour and recorded inputs.

Dependency caching saves downloaded libraries or build outputs from one pipeline run and restores them in later runs, so jobs skip work that has not changed. The cache is stored under a key, usually derived from the files that define dependencies.

1. Compute a key: for example, a hash of the lock file plus the operating system and tool version.
2. Restore: if a cache with that key exists, unpack it before installing dependencies.
3. Install: the package manager only downloads what is missing.
4. Save: after a successful run, store the cache under the key for future runs.

What can go wrong:

• Stale cache: a key that ignores the lock file restores outdated dependencies and hides real problems.
• Poisoned cache: an untrusted branch writes a cache that a protected branch later restores; restrict who can write caches used by release builds.
• Too large: restoring a huge cache can take longer than installing from scratch.
• Non-portable contents: caching compiled outputs across different machine types can produce broken builds.

A good rule is to cache what is expensive to fetch and cheap to verify, and to key it on everything that affects its contents.

Retrying with exponential backoff means waiting longer after each failed attempt, for example 1, 2, 4 and 8 seconds, and giving up after a fixed number of tries. It handles short outages such as a dependency restarting, without hammering a service that is already struggling.

Key points of a good retry function:

• A maximum number of attempts, so a permanent failure still fails the job.
• A growing delay, usually doubling each time, often with a cap on the longest wait.
• The command's exit status decides success; the function returns non-zero when all attempts fail.
• Messages on standard error, so the command's own output stays clean.
• Jitter in production code: a small random addition to each delay stops many clients from retrying at the same instant.

Retry only operations that are safe to repeat, such as reads or idempotent writes; retrying a payment without an idempotency key could charge a customer twice.

Both keep every change in a releasable state through an automated pipeline. The difference is the last step: in continuous delivery a person decides when to release to production, while in continuous deployment every change that passes the pipeline goes to production automatically.

When to use which:

• Choose continuous delivery when releases must align with business events, regulatory sign-off, or customer communication.
• Choose continuous deployment when automated tests and monitoring are strong and the team wants the shortest feedback loop.

In both, deployment and release can be separated: code can be deployed but hidden behind a feature flag until it is switched on for users.

A blue-green deployment runs two identical production environments. One (blue) serves all traffic while the new version is deployed to the other (green); once green is verified, traffic switches to it in one step.

1. Blue is live: all users reach the current version.
2. Deploy to green: install the new version on the idle environment.
3. Verify green: run smoke tests and health checks against it without real users.
4. Switch traffic: point the load balancer or router at green.
5. Watch: monitor errors and latency; blue stays running as a fallback.
6. Roll back or retire: switch back to blue if problems appear, or recycle blue for the next release.

flowchart LR
  A["Users"] --> B["Router"]
  B -->|live| C["Blue: current version"]
  B -.->|after switch| D["Green: new version"]
  D --> E{"Healthy?"}
  E -->|yes| F["Keep green live"]
  E -->|no| G["Switch back to blue"]

Trade-offs:

• Rollback is instant, but you need capacity for two full environments during the release.
• Shared databases are the hard part: schema changes must work with both versions at once.

A canary release sends a small share of real traffic to the new version while most users stay on the old one. If the canary's health matches the old version, its share grows step by step; if not, the canary is removed and few users were affected.

1. Deploy the canary alongside the current version.
2. Route a small share of traffic, such as 5%, to it.
3. Compare health signals: error rate, latency and key business metrics against the current version over the same period.
4. Decide: continue to 25%, 50% and 100%, or roll back.

flowchart TD
  A["Deploy canary"] --> B["Send small traffic share"]
  B --> C["Compare canary with current version"]
  C --> D{"Within thresholds?"}
  D -->|yes| E{"At 100%?"}
  E -->|no| F["Increase traffic share"]
  F --> C
  E -->|yes| G["Release complete"]
  D -->|no| H["Roll back canary"]

Base the decision on thresholds agreed in advance, and wait for enough requests to be statistically meaningful; 20 requests are not enough to judge an error rate.

All three replace a running version without downtime, but they differ in how traffic moves to the new version, how fast you can undo it, and how much extra capacity they need.

When to use which:

• Use rolling updates as a simple default for stateless services with good health checks.
• Use blue-green when instant rollback matters and you can afford double capacity.
• Use canary releases for high-traffic services where real-user metrics are the best test.

All three require the new version to work with the same database and with other services' current versions during the transition.

A feature flag is a runtime switch that turns a piece of functionality on or off without deploying new code. It separates deploying code from releasing a feature to users.

Common uses:

• Hiding unfinished work so it can be merged to the main branch early.
• Gradual rollout to a percentage of users, a region or internal staff first.
• Kill switch to disable a misbehaving feature instantly.
• Experiments comparing two variants of a feature.

Risks and how to manage them:

• Flag debt: old flags pile up and make the code hard to follow; give every flag an owner and a removal date.
• Untested combinations: many flags create many possible states; test the states actually used in production.
• Configuration mistakes: a wrong flag value can break production as surely as bad code, so flag changes need audit logs and permission controls.
• Slow lookups: flags evaluated on every request should be cached locally, with a safe default if the flag service is unavailable.

Interview tip: stress that a release flag should be deleted once the feature is fully on; long-lived flags are a separate, deliberate design choice.

The fastest rollback is redeploying the previous known-good artifact through the same automated pipeline, or switching traffic back to it if it is still running. Speed comes from preparing for rollback before the release, not from improvising during an incident.

1. Detect: alerts on error rate and latency fire within minutes of the release.
2. Decide quickly: if the release is the likely cause, roll back first and investigate afterwards.
3. Roll back: switch traffic to the old version, or redeploy the previous artifact by its version identifier.
4. Disable flags: if the new behaviour is behind a feature flag, turning it off may be faster than a full rollback.
5. Verify: confirm that metrics return to normal.
6. Follow up: find the cause, add a test that would have caught it, and roll forward with a fix.

What makes rollback safe:

• Every release keeps the previous artifact available and tagged.
• Database changes are backward compatible, so the old code still works against the new schema.
• Rollback is practised, not first tried during an outage.

Use the expand and contract pattern: change the schema in small, backward-compatible steps, so both the old and the new application versions work at every point during the rollout.

1. Expand: add the new column or table without removing anything. New columns are nullable or have defaults.
2. Deploy code that writes to both: the application writes old and new structures, and still reads the old one.
3. Backfill: copy existing data into the new structure in small batches to avoid long locks.
4. Switch reads: deploy code that reads from the new structure.
5. Stop writing the old structure: deploy code that only uses the new one.
6. Contract: drop the old column or table in a later release, once rollback to the old code is no longer needed.

flowchart LR
  A["Add new column"] --> B["Write to both"]
  B --> C["Backfill old rows"]
  C --> D["Read from new column"]
  D --> E["Stop writing old column"]
  E --> F["Drop old column"]

Trade-offs:

• It takes several releases instead of one, but every step can be rolled back on its own.
• Large tables need online schema-change techniques and batched backfills to avoid locking.

A deployment gate is a condition that must be met before a change moves to the next environment. Gates can be automated checks or human approvals, and good pipelines use automation wherever a rule can be written down.

What belongs before production:

• Automated tests, scans and a successful deployment to a production-like environment.
• Policy checks, such as the artifact being built by the trusted pipeline.
• A human approval only where it adds real judgement, such as legal or customer-facing timing.

Too many manual gates slow delivery and teach people to approve without looking. Each manual gate should have a clear reason, a named approver, and a plan to automate it if possible.

In pull-based deployment, the desired state of every environment is stored as declarative files in a repository, and an agent inside the environment continuously compares the running state with those files and fixes any difference. Deploying becomes a reviewed change to the repository rather than a command sent from a pipeline.

flowchart LR
  A["Developer"] -->|merges change| B["Configuration repository"]
  C["Sync agent in cluster"] -->|pulls desired state| B
  C -->|compares and applies| D["Running workloads"]
  D -->|reports actual state| C

Benefits:

• Audit trail: every deployment is a commit with a review and an author.
• Drift correction: manual changes in the cluster are reverted to the declared state.
• Fewer credentials outside: the pipeline does not need administrator access to production, because the agent pulls from inside.
• Easy rollback: revert the commit and the agent restores the previous state.

Trade-offs:

• Secrets cannot be stored as plain text in the repository, so they need encryption or an external secret store.
• Debugging needs both the repository history and the agent's sync status.

Build the artifact once, store it with an immutable version, and deploy that exact artifact to each environment in turn, changing only configuration. Promotion means moving a version forward after it passes the checks of the current environment.

1. Build and version: the pipeline creates the artifact and records its commit and checksum.
2. Deploy to development: run automated tests against it.
3. Promote to staging: deploy the same version and run integration, performance and smoke tests in a production-like setup.
4. Approve: automated gates and any required sign-off.
5. Promote to production: deploy the same version, often gradually.

flowchart LR
  A["Build once"] --> B["Artifact repository"]
  B --> C["Development"]
  C -->|tests pass| D["Staging"]
  D -->|gates pass| E["Production"]
  F["Per-environment configuration"] --> C
  F --> D
  F --> E

Key rules:

• Configuration such as database addresses and credentials comes from each environment, never baked into the artifact.
• The version identifier, not a moving label, decides what is deployed, so every environment's contents are known.
• Promotion records show which version is in each environment at any time.

A deployment freeze is a period when no changes, or only emergency fixes, may be released, such as during a major sale, a holiday season or year-end processing. It reduces the chance of a change-related incident when the business can least afford one.

When a freeze helps:

• Very high-traffic events where even a small incident costs a lot.
• Periods when key staff are away and recovery would be slow.
• Events with external commitments, such as a launch with partners.

When it is a bad idea:

• As a substitute for safe releases: if releases are risky every day, the fix is better testing and rollback, not more freezes.
• Long freezes: changes pile up, and the first release afterwards is large and dangerous.
• Blocking fixes: security patches and incident fixes must still have a fast, approved path.

Better habits around a freeze:

1. Release smaller changes before it begins rather than rushing large ones in.
2. Keep the pipeline running so it is ready for emergency fixes.
3. After it ends, release in small batches rather than all at once.

A virtual machine emulates a whole computer with its own operating system kernel, while a container is an isolated process that shares the host's kernel. Containers are lighter and start faster; virtual machines give stronger isolation.

When to use which:

• Use containers to package and run many services efficiently with consistent environments.
• Use virtual machines when workloads need a different kernel or operating system, or strong isolation between untrusted tenants.
• Combine both: containers commonly run inside virtual machines in cloud environments, getting fast deployments and a strong outer boundary.

Containers are built on kernel features: namespaces limit what a process can see, and control groups limit how much CPU and memory it can use.

A container image is a read-only package of a filesystem and settings used to start containers. It is built as a stack of layers, where each build step adds one layer containing only the files that step changed.

How layers behave:

• Layers are cached: if a step and everything before it are unchanged, the build reuses the stored layer instead of running the step again.
• Layers are shared: many images built on the same base layer store it only once on a host or registry.
• Layers are immutable: deleting a file in a later step hides it but does not shrink earlier layers.

Why order matters:

1. Put steps that change rarely, such as the base image and dependency installation, first.
2. Put steps that change often, such as copying application code, last.
3. Remove temporary files in the same step that creates them, or they remain in a lower layer.

A well-ordered build can drop from minutes to seconds, because most of it is served from cache.

A multi-stage build uses one temporary stage with the full toolchain to compile or package the application, then copies only the finished output into a small, clean final image. The compilers, source code and build caches never reach production.

Benefits:

• Smaller images: the final image contains only the runtime and the application, often a fraction of the size.
• Smaller attack surface: no compilers, package managers or shells that an attacker could use.
• One build definition: the build environment is versioned with the code instead of living on a build server.
• Cleaner layers: secrets or tokens used only to fetch build dependencies stay in the discarded stage.

flowchart LR
  A["Build stage with full toolchain"] -->|compile| B["Application binary"]
  B -->|copy only output| C["Minimal runtime stage"]
  C --> D["Final image"]
  A -.->|discarded| E["Tools and source not shipped"]

Trade-offs:

• A minimal final image may lack debugging tools, so troubleshooting needs a separate debug image or attaching a temporary tools container.
• Build-time secrets still need care, because the build stage's history can be inspected if it is pushed anywhere.