DevOps Engineer Top 100 Interview Questions

Start from a minimal base, include only what the application needs at runtime, and keep the image patched and scanned. Smaller images pull faster and give attackers fewer tools.

Additional practices:

• Exclude files from the build context with an ignore file, so secrets and test data are never copied.
• Make the root filesystem read-only at runtime and mount writable volumes only where needed.
• Remove shells and package managers from production images where the application does not need them.

Rebuild images regularly even when the code has not changed, because the base image keeps receiving security fixes.

A tag is a movable label that can point to a different image tomorrow; a digest is a hash of the image's contents, so it always identifies exactly one image. Deploying by digest guarantees that every environment and every restart runs the same bits.

When to use which:

• Use version tags for people to read and search, and never move a release tag once published.
• Use digests in deployment manifests, or record the digest that a tag resolved to at deploy time.

The "latest" tag is the worst choice for production: two replicas started an hour apart can run different code, and nobody can tell which version is live.

By default, many images run their process as root inside the container. If an attacker breaks out of the application or the container, root privileges make it far easier to damage the host or other containers.

Hardening measures:

• Run as a non-root user: create a dedicated user in the image and run the process as that user.
• Drop kernel capabilities: remove all capabilities and add back only the few the process needs.
• Read-only root filesystem: the process cannot modify its own binaries or plant files.
• No privilege escalation: block setuid programs from gaining extra rights.
• Resource limits: cap CPU, memory and process count so one container cannot starve the host.
• No privileged mode or host mounts: never mount the container runtime's socket or the host's root filesystem into a container.
• Minimal image: fewer binaries means fewer tools for an attacker.

These settings follow the principle of least privilege. Most can be enforced cluster-wide by an admission policy, so an insecure workload is rejected before it starts.

Interview tip: mention that root inside a container is not root on the host only when user namespaces are configured; without them, it is the same user ID.

When a platform stops a container, it sends SIGTERM to the main process, waits a grace period, often 30 seconds, then sends SIGKILL, which cannot be caught. A well-behaved service catches SIGTERM, stops taking new work, finishes current requests and exits.

Why process ID 1 matters:

• The container's main process runs as PID 1, which the kernel treats specially: it gets no default signal handlers, so an unhandled SIGTERM is ignored instead of stopping it.
• If the start command runs through a shell wrapper, the shell may become PID 1 and not forward signals to the application, so the application is killed abruptly after the grace period.
• PID 1 must also collect finished child processes; otherwise they remain as zombie processes.

What to do:

1. Start the application directly, or replace the shell with it using exec, so it receives signals.
2. Handle SIGTERM in the application: stop accepting work, drain in-flight requests, close connections.
3. Use a minimal init process as PID 1 when the application cannot reap child processes.
4. Keep shutdown shorter than the grace period.

A container's own writable layer is deleted with the container, so anything that must survive a restart or replacement goes in storage mounted from outside the container.

When to use which:

• Use named volumes rather than bind mounts in production, because they do not depend on host paths.
• Use network volumes in orchestrated clusters, where a container may be rescheduled to another machine.
• Keep stateless services stateless: store data in databases or object storage rather than local volumes.

Volumes protect against container restarts, not against host failure or mistakes, so they still need backups.

Each container gets its own network namespace with a virtual network interface. The container runtime connects these interfaces to virtual networks on the host and uses network address translation for traffic leaving or entering the host.

• Bridge network: containers on the same user-defined bridge network reach each other by container name through built-in DNS.
• Port publishing: a host port is mapped to a container port, so external clients reach the container through the host's address.
• Outbound traffic: containers reach the internet through the host, with their addresses translated to the host's.
• Host networking: the container shares the host's network stack directly, which is faster but removes network isolation.
• No network: the container has only a loopback interface, for jobs that must not make network calls.

flowchart LR
  A["External client"] -->|host port 8080| B["Host network"]
  B -->|translated to port 80| C["Web container"]
  C -->|by name over bridge| D["API container"]
  D -->|by name over bridge| E["Database container"]

Only publish the ports that outside clients need; internal services such as databases should be reachable only on the internal network.

Work from the evidence the platform already keeps: the exit code, the last logs and the recent events. Most restart loops come from a crash on startup, a failed health check, missing configuration or running out of memory.

1. Check status and exit code: the code points to the kind of failure.
2. Read the logs of the previous run: the crash message is usually in the last lines before exit.
3. Check events: look for failed health checks, image pull errors or memory kills.
4. Compare configuration: environment variables, secrets and mounted files against a working environment.
5. Run it interactively: start the same image with a shell or debug entry point and run the start command by hand.

flowchart TD
  A["Container restarting"] --> B{"Exit code 137?"}
  B -->|yes| C["Check memory limit and usage"]
  B -->|no| D{"Errors in last logs?"}
  D -->|yes| E["Fix config or code error"]
  D -->|no| F["Check health probe settings"]

A container registry stores and distributes container images, organised into repositories with tags and digests. Pipelines push images to it, and hosts or clusters pull images from it when they start containers.

Good management practices:

• Access control: only the trusted pipeline can push production images; runtime environments get read-only pull access.
• Immutable tags: once a release tag is pushed, it cannot be overwritten, so a version always means the same image.
• Vulnerability scanning: scan images on push and on a schedule, because new vulnerabilities are found in old images.
• Signing and verification: sign images in the pipeline and verify signatures before deployment.
• Retention rules: delete untagged and old development images automatically, but keep every version still deployed or needed for rollback.
• Regional mirrors or caches: keep a copy near each cluster for faster, more reliable pulls.

A private registry close to production also removes a dependency on public registries, which may rate-limit pulls or become unavailable during an incident.

A container orchestrator runs containers across a group of machines and keeps them in the state you declared, so you describe what should run instead of starting each container by hand.

• Scheduling: decides which machine each container runs on, based on available resources and placement rules.
• Self-healing: restarts failed containers and replaces those on machines that go down.
• Scaling: changes the number of replicas manually or automatically based on load.
• Service discovery and load balancing: gives a group of containers one stable name and spreads traffic across healthy replicas.
• Rolling updates and rollbacks: replaces a version gradually and can return to the previous one.
• Configuration and secrets: delivers settings and credentials to containers at runtime.
• Storage: attaches persistent volumes to the containers that need them.

All of this runs as a control loop: the orchestrator constantly compares the actual state with the desired state and acts to close the gap. That loop is why declarative configuration is the normal way to work with an orchestrator.

Placement happens in two phases: first the scheduler filters out nodes that cannot run the workload, then it scores the remaining nodes and picks the best one.

1. Filter: remove nodes without enough free CPU or memory for the workload's requests, nodes marked to repel workloads, and nodes that fail required placement rules such as a needed hardware type.
2. Score: rank the remaining nodes by preferences such as spreading replicas across zones, keeping related workloads together, and balancing resource use.
3. Bind: assign the workload to the highest-scoring node, where the node agent pulls the image and starts the containers.
4. No fit: if no node passes the filter, the workload stays pending until resources free up or new nodes are added.

flowchart TD
  A["New workload"] --> B["Filter nodes"]
  B --> C{"Any node fits?"}
  C -->|no| D["Stay pending"]
  C -->|yes| E["Score nodes"]
  E --> F["Bind to best node"]
  F --> G["Node starts containers"]

Scheduling uses the requested resources, not actual usage, so accurate requests are essential: requests set too high waste capacity, and requests set too low overload nodes.

All three are health checks the orchestrator runs against a container, but each triggers a different action. Mixing them up is a common cause of restart loops and dropped traffic.

When to use which:

• Use readiness to stop sending traffic during warm-up, overload or a lost database connection, without restarting.
• Use liveness only for states the process cannot recover from, such as a deadlock.
• Use startup for slow-starting applications, so liveness does not kill them before they are ready.

Never make liveness depend on a shared dependency such as the database: if the database goes down, every replica fails liveness and is restarted at once, turning a partial outage into a total one.

A request is the amount of CPU and memory the scheduler reserves for a container when placing it on a node. A limit is the maximum the container may use. Requests drive scheduling; limits are enforced at runtime.

CPU is measured in cores or millicores, where 1000 millicores is one core. Memory is measured in bytes, usually mebibytes or gibibytes.

How to set them:

1. Measure real usage under normal and peak load.
2. Set requests near typical usage, so the scheduler packs nodes accurately.
3. Set memory limits with headroom above peak usage, since hitting them kills the container.
4. Be careful with tight CPU limits on latency-sensitive services, because throttling adds delay even when the node is idle.

A container with no requests can be placed on an already busy node and is the first to be evicted when the node runs short of memory.

A horizontal autoscaler compares a measured metric, such as average CPU use, with a target value and scales the replica count in proportion to the difference.

desired_replicas = ceil(current_replicas * current_metric / target_metric)

• current_replicas: how many replicas run now.
• current_metric: the average of the metric across those replicas, such as 90% CPU.
• target_metric: the value you want each replica to average, such as 60% CPU.
• ceil: round up, so capacity is never too low.

Behaviour that prevents constant changes:

• Tolerance: small differences, often within 10% of the target, cause no change.
• Stabilisation window: scale-down waits several minutes to confirm lower load, while scale-up reacts quickly.
• Bounds: minimum and maximum replica counts cap the result.

CPU percentage is relative to each container's CPU request, so wrong requests make autoscaling wrong too. For queue workers, a metric such as queue length per replica is often a better signal than CPU.

Horizontal scaling adds more instances of a service; vertical scaling gives an existing instance more CPU or memory. Horizontal scaling is the default for modern services, but it only works when the service is designed for it.

When to use which:

• Scale horizontally for stateless web services and workers, where instances are interchangeable behind a load balancer.
• Scale vertically for components that are hard to split, such as a single primary database, or as a quick fix while a redesign happens.
• Combine them: right-size each instance vertically, then add instances horizontally as load grows.

Service discovery lets workloads find each other by a stable name even though individual containers come and go with changing addresses. The cluster tracks healthy endpoints and resolves a service name to them.

1. A service is defined with a stable name and a selector that matches a group of workloads.
2. The cluster tracks endpoints: as replicas start and pass readiness checks, their addresses are added; failed ones are removed.
3. Internal DNS returns a stable virtual address for the service name.
4. Traffic is load-balanced: connections to the virtual address are spread across the current healthy endpoints.

sequenceDiagram
  participant C as Client container
  participant D as Cluster DNS
  participant S as Service address
  participant P as Healthy replica
  C->>D: Look up orders service
  D-->>C: Stable virtual address
  C->>S: Send request
  S->>P: Forward to one replica
  P-->>C: Response

Clients should use the service name, never a replica's own address, because replica addresses change on every restart. Readiness checks are what keep traffic away from replicas that cannot serve.

Keep configuration and secrets outside the image and inject them at runtime, so the same image runs in every environment. Clusters provide separate objects for plain configuration and for sensitive values.

Good practices:

• Enable encryption at rest for cluster secrets and limit who can read them.
• Prefer mounting secrets as files over environment variables, so they do not leak through logs or child processes.
• Sync secrets from an external store for rotation and audit, rather than editing them by hand.
• Roll workloads automatically when configuration changes, so no replica runs with stale values.

A common mistake is treating the cluster's secret object as encrypted storage; by default it is only base64-encoded.

A rolling update replaces old replicas with new ones in small batches, starting a new replica and waiting for it to become ready before removing an old one. Two settings control how fast it goes and how much spare capacity it uses.

• Maximum surge: how many extra replicas may exist above the desired count during the update.
• Maximum unavailable: how many replicas may be unavailable below the desired count.

1. Start new replicas, up to the surge limit.
2. Wait until they pass readiness checks.
3. Remove old replicas, keeping availability within the unavailable limit.
4. Repeat until all replicas run the new version.
5. Pause or stop automatically if new replicas never become ready.

Tuning the speed:

• A higher surge finishes faster but needs more spare capacity.
• Zero unavailable keeps full capacity throughout but needs surge capacity.
• A minimum ready time adds a delay after each new replica becomes ready, catching crashes that appear a little after startup.

When values are given as percentages, surge is usually rounded up and unavailable rounded down, which keeps capacity safe.

Stateful services need things that stateless ones do not: a stable identity, storage that follows each replica, and a careful order for starting, stopping and updating. Orchestrators provide a dedicated controller for this, but much of the risk remains with the team.

What a stateful workload controller provides:

• Stable names: each replica keeps the same name and network identity across restarts, such as db-0, db-1 and db-2.
• Dedicated volumes: each replica gets its own persistent volume, which is reattached when it is rescheduled.
• Ordered operations: replicas start, stop and update one at a time, in a defined order.

What you still need to handle:

1. Replication and failover: the database itself must replicate data and elect a new primary.
2. Backups and restores: volumes are not backups; test restoring regularly.
3. Storage performance: choose volume types with suitable throughput and latency.
4. Disruption limits: stop maintenance from taking down too many replicas at once.
5. Operational automation: an operator, a controller that encodes expert procedures, can automate upgrades and failover.

Many teams use a managed database service instead, because it removes most of this work; running databases on an orchestrator makes sense when the team has the skills and a strong reason.

A namespace is a logical partition of a cluster that groups related resources, such as everything belonging to one team or one environment. Names must be unique only within a namespace, and many policies are applied per namespace.

What namespaces are used for:

• Separating teams or applications so their resources do not collide or get changed by mistake.
• Access control: roles grant permissions within a namespace, so a team can manage only its own workloads.
• Policies: network rules and security settings can apply to a whole namespace.

How quotas and limits protect the cluster:

• A resource quota caps the total CPU, memory, storage and object count a namespace may use, so one team cannot consume the whole cluster.
• Default limits give containers sensible requests and limits when a workload does not set them.
• Without quotas, one bad deployment, such as a runaway autoscaler, can starve every other team.

Namespaces are not a strong security boundary: workloads in different namespaces still share nodes and the kernel. Untrusted tenants need separate clusters or extra isolation.

External traffic enters through a load balancer at the cluster edge, is routed by rules on host name and path to the right service, and is then spread across that service's healthy replicas.

1. DNS points the public host name to the external load balancer's address.
2. The load balancer receives the connection and forwards it to the cluster's edge proxy.
3. The edge proxy applies routing rules, such as sending /api to the API service and everything else to the web service, and usually terminates TLS.
4. The service forwards the request to one of its ready replicas.

flowchart LR
  A["User"] --> B["Public DNS"]
  B --> C["External load balancer"]
  C --> D["Edge proxy with routing rules"]
  D -->|path api| E["API service"]
  D -->|other paths| F["Web service"]
  E --> G["API replicas"]
  F --> H["Web replicas"]

Design points:

• Terminate TLS at the edge and manage certificates automatically, with renewal before expiry.
• Expose only the edge proxy publicly; internal services stay on the cluster network.
• Add rate limits and request size limits at the edge to protect services behind it.