Cloud Engineer Top 100 Interview Questions

Combine three layers: prevent insecure settings before deployment, detect them continuously in running accounts, and correct them quickly, automatically where it is safe.

Making it work:

• Prioritize by exposure and data sensitivity, so a public bucket holding customer data beats an unencrypted test volume.
• Route findings to owners using tags; a central queue nobody owns just grows.
• Track time to fix by severity, and report it per team.
• Allow documented exceptions with expiry dates, so real needs do not turn into ignored alerts.

Assume it has already been used: automated scanners find exposed keys within minutes. Disable the key at once, then work out what was done with it, and remove anything the attacker created.

1. Deactivate the key immediately. Deactivating rather than deleting keeps the record for the investigation.
2. Search the audit logs for every use of the key: actions, regions, source addresses and timing.
3. Revoke what it created: new users, keys, roles, trust policies and any temporary sessions issued from it.
4. Find and remove resources the attacker launched. Unexpected servers in unusual regions often mean cryptocurrency mining.
5. Check data access: look for reads from storage and databases to decide whether this is a data breach.
6. Rotate the credentials the workload legitimately needs, ideally replacing the key with a role.
7. Clean the repository history, and contact the provider, who can help with abuse and unexpected charges.

flowchart TD
  A["Leaked key reported"] --> B["Deactivate key"]
  B --> C["Review audit logs for its use"]
  C --> D{"Used by anyone else?"}
  D -->|no| E["Replace with a role and close"]
  D -->|yes| F["Remove persistence and attacker resources"]
  F --> G["Assess data access"]
  G --> H["Notify and review"]

Prevention: secret scanning that blocks pushes containing keys, and roles with federation instead of long-lived keys.

Identify which data must stay in which jurisdiction, keep every copy of it there, including backups, replicas, logs and keys, and enforce that with guardrails rather than relying on everyone remembering.

• Classify the data to know which datasets are in scope and which rules apply.
• Choose compliant regions for primary storage, and check each service's data handling, since some features process data in other regions.
• Block other regions with organization guardrails for the accounts that hold regulated data.
• Follow every copy: cross-region replication, backups, disaster recovery sites, analytics exports and log destinations must all stay in approved regions.
• Check global services: DNS, content delivery caches, identity services and support tickets can move data across borders.
• Keep encryption keys in region, and consider customer-managed or externally held keys if the rules require control over provider access.
• Document and audit: maintain a record of where each dataset lives, and test it with configuration scans.

Residency and disaster recovery can conflict: if regulation allows only one country with one region, plan resilience across zones within that region. Otherwise you need a second in-country location.

Use TLS for every service-to-service connection, even inside private networks, and use mutual TLS where services must prove their identity to each other. Automate certificates, because manual certificate handling is what usually breaks.

• TLS everywhere: private networks still carry attacker traffic once one host is compromised, and many regulations require encryption in transit regardless of the network.
• Mutual TLS: both client and server present certificates, so each service knows exactly which workload is calling it. This becomes the basis of service-level authorization.
• A private certificate authority issues internal certificates, so no internal name needs a public certificate.
• Short-lived certificates with automatic rotation limit the damage of a stolen key and remove expiry outages.
• A service mesh can handle all of this with sidecar proxies, without code changes, at the cost of added latency and operational complexity.
• Managed services: require TLS on database and cache connections, and reject plaintext connections at the server.

Some providers encrypt traffic between their facilities or between certain instance types automatically. Check exactly what is covered before counting on it for compliance.

Components in a chain that all must work multiply their availabilities, so the chain is weaker than its weakest link. Redundant components in parallel fail only if all of them fail, so redundancy raises availability sharply.

• In series: A = A1 × A2 × ... × An
• In parallel: A = 1 − (1 − A1) × (1 − A2) × ... × (1 − An)
• Downtime per year: (1 − A) × 8,760 hours

What each term means:

• A1, A2, ... are the availabilities of individual components, as fractions such as 0.9995.
• Series means every component is needed for the request to succeed, such as a load balancer, then an app server, then a database.
• Parallel means any one of the redundant copies is enough, such as two app servers behind a load balancer.

When to use it: to find which component limits the whole design, and to check whether a target, such as 99.95%, is even possible. The formulas assume independent failures. Shared dependencies, such as one zone or one configuration change, make real availability lower than calculated.

Multi-zone protects against the failure of a data center within a region and is the standard for production. Multi-region also protects against a whole-region outage or disaster, but costs far more in money, complexity and data consistency, so it needs a clear business reason.

When to use which:

• Multi-zone for every production system that matters; it is cheap insurance.
• Multi-region when the cost of a regional outage exceeds the cost of building and running a second region, when users are spread worldwide and need low latency, or when regulation demands geographic separation.
• A middle path keeps a cold or pilot-light copy in a second region: backups and infrastructure code ready to deploy, without full running capacity.

The four strategies trade cost against how quickly you recover and how much data you can lose. Moving from backup-and-restore to active-active cuts recovery time from hours to near zero, while cost rises at each step.

What each needs to work:

• Backup and restore: infrastructure code to rebuild everything, and tested restore times.
• Pilot light: continuous data replication, plus automation to start and scale the remaining services.
• Warm standby: automated scale-up and a tested traffic switch.
• Active-active: an application designed for writes in several regions, with conflict handling or partitioned data ownership.

Most organizations mix strategies: active-active or warm standby for the few systems that earn money every minute, and backup-and-restore for internal tools.

Start from each system's recovery time objective, how long it may be down, and recovery point objective, how much data it may lose. Then pick the cheapest strategy that meets both, and prove it with a test.

flowchart TD
  A["System with RTO and RPO"] --> B{"RTO under a few minutes?"}
  B -->|yes| C{"Users must write in both regions?"}
  C -->|yes| D["Active-active"]
  C -->|no| E["Warm standby"]
  B -->|no| F{"RTO under a few hours?"}
  F -->|yes| G["Pilot light"]
  F -->|no| H["Backup and restore"]

1. Set objectives with the business, based on the cost of downtime and data loss for each system, not on technical preference.
2. Match data protection to the recovery point: continuous replication for minutes, frequent backups for hours.
3. Match standby capacity to the recovery time: the less time allowed, the more must already be running.
4. Include dependencies: identity, DNS, secrets, keys and third-party services must be available in the recovery region too.
5. Test regularly and measure the real recovery time against the objective.

The strategy is only as strong as its last successful test. An untested plan usually fails on something small, such as a missing permission or quota.

Health checks probe each instance or endpoint on a schedule. After a set number of consecutive failures, traffic stops going to it, and after a set number of successes it is restored. Load balancers apply this to instances, autoscaling replaces unhealthy ones, and DNS or global load balancers apply it to whole regions.

sequenceDiagram
  participant L as Load balancer
  participant I as Instance
  participant S as Autoscaling
  L->>I: Health check every 10 seconds
  I-->>L: Fails three times in a row
  L->>L: Stop routing to the instance
  S->>I: Mark unhealthy and terminate
  S->>S: Launch replacement
  L->>S: New instance passes checks
  L->>L: Resume routing

Designing good checks:

• Shallow checks confirm the process is up and can serve. They are fast and rarely wrong.
• Deep checks also test dependencies, such as the database. They catch more problems but are risky: if a shared dependency fails, every instance fails its check and the load balancer removes them all.
• Fail open: many load balancers keep sending traffic to all instances when every one is unhealthy, because some service is better than none.
• Tune thresholds and intervals to balance detection speed against flapping on brief blips.
• Give new instances a grace period so slow-starting applications are not killed before they are ready.

Retries help with brief failures but can turn a short outage into a long one: if every client retries immediately, the recovering service is flooded again. Exponential backoff spaces retries out further each time, and jitter randomizes the wait so clients do not retry in synchronized waves.

Key points:

• Backoff: the wait ceiling doubles per attempt, ceiling = min(cap, base × 2^attempt), with a cap so waits stay reasonable.
• Full jitter: wait a random time between zero and the ceiling, which spreads clients evenly.
• Limit attempts, and give up with a clear error, ideally within the caller's own timeout.
• Retry only safe cases: timeouts, throttling and server errors, never client errors such as invalid input.
• Make operations idempotent before retrying writes, or a retry may duplicate the action.
• Retry at one layer only; retries at every layer of a call chain multiply the load.

Software development kits for cloud services usually include backoff with jitter. Configure it rather than writing your own, but understand what it does.

An idempotency key is a unique value the client sends with a request that must not be performed twice, such as a payment. The server stores the key with the result of the first request, and any retry with the same key gets that stored result instead of a second payment.

sequenceDiagram
  participant C as Client
  participant A as Payments API
  participant D as Key store
  C->>A: Charge 50 with key K1
  A->>D: Save K1 as in progress
  A->>A: Process payment
  A->>D: Save K1 with result
  A--xC: Response lost on the network
  C->>A: Retry charge with key K1
  A->>D: Look up K1
  D-->>A: Stored result
  A->>C: Original result, no second charge

Implementation details:

1. The client generates the key once per logical operation, such as a random identifier, and reuses it for every retry.
2. Store the key atomically before doing the work, using a unique constraint, so two simultaneous retries cannot both proceed.
3. Return the stored response for repeats, including errors from the first attempt.
4. Scope keys per client or account, and check that a retry carries the same request body.
5. Expire keys after a reasonable period, such as 24 hours.

Idempotency is what makes retries, queues with at-least-once delivery and failover safe for operations that change state.

A circuit breaker wraps calls to a dependency and stops making them when that dependency is clearly failing, returning a fast error or fallback instead. This stops threads piling up on timeouts and gives the dependency room to recover.

stateDiagram-v2
  [*] --> Closed
  Closed --> Open: Failure rate crosses threshold
  Open --> HalfOpen: Wait period ends
  HalfOpen --> Closed: Trial calls succeed
  HalfOpen --> Open: Trial calls fail

1. Closed: calls pass through normally, and failures are counted over a rolling window.
2. Open: once failures cross a threshold, such as 50% of the last 20 calls, calls fail immediately without touching the dependency.
3. Half-open: after a wait, a few trial calls are allowed. Success closes the circuit; failure opens it again.

Using it well:

• Provide a fallback: cached data, a default value or a degraded feature, such as hiding recommendations while the recommendation service is down.
• Use one breaker per dependency, so one failing service does not block calls to healthy ones.
• Combine it with timeouts and bulkheads: short timeouts bound each call, and separate thread or connection pools stop one dependency exhausting them all.
• Expose its state in metrics, since an open breaker is an important operational signal.

A queue between producers and workers lets requests arrive faster than they can be processed. The backlog grows during a spike and drains afterwards, so workers run at a steady, safe rate instead of failing under peak load. This is often called queue-based load leveling.

• backlog growth per minute = arrival rate − processing rate
• drain time = backlog / (processing rate − arrival rate after the spike)

What makes it work:

• Scale workers on queue depth or the age of the oldest message, not CPU, so capacity follows the backlog.
• Set a dead-letter queue for messages that fail repeatedly, so one bad message cannot block the rest.
• Make consumers idempotent, since most queues deliver at least once and duplicates happen.
• Set visibility timeouts longer than the processing time, so messages are not handed to a second worker mid-processing.
• Alert on message age: the oldest message's age shows how far behind users' requests are.

When not to use it: anything where the user needs the result immediately in the same request, such as checking a password. Queues suit work that can finish seconds or minutes later, such as sending emails or generating reports.

Chaos engineering deliberately injects failures, such as killed instances, added latency or a lost zone, to check that a system behaves as its designers believe. A game day is a planned session where a team runs such experiments together and practises the response.

1. Define steady state: the user-facing metrics that show the system is healthy, such as checkout success rate and latency.
2. Form a hypothesis: "If one zone's app servers fail, the success rate stays above 99.5% and recovery needs no human action."
3. Limit the blast radius: start in staging, then production with a small share of traffic or a single service.
4. Set abort conditions and a one-step rollback before starting.
5. Inject the failure and watch the metrics, dashboards and alerts.
6. Compare with the hypothesis: did alerts fire, did automation work, did people know what to do?
7. Fix and repeat: every surprise becomes a work item, and the experiment runs again to confirm the fix.

Prerequisites: good monitoring and a basically resilient design. Injecting failures into a system with no redundancy only proves what you already know.

Interview tip: stress the hypothesis and the abort conditions; chaos engineering is careful experimentation, not randomly breaking production.

Restore them regularly, automatically and completely, into an isolated environment, then check the data and measure how long it took. A backup is proven only by a successful restore.

1. Automate restore tests on a schedule: restore the latest backup of each critical system into an isolated account.
2. Validate the data: row counts, checksums and application-level queries, not just "the restore job succeeded".
3. Measure restore time against the recovery time objective, since large restores often take far longer than expected.
4. Test the whole chain: encryption keys, permissions, network access and credentials needed in the recovery account. Missing access to a key is a classic failure.
5. Run full disaster recovery drills periodically, restoring a complete application, not just one database.
6. Keep runbooks current, and have someone other than the author run them.
7. Report results so failed tests get fixed, not quietly ignored.

Test restoring to an older point too. If corruption went unnoticed for weeks, you need restore points from before it began.

Decide how quickly the application must recover and how much data it may lose, then keep its data replicated, its infrastructure deployable and its traffic switchable in a second region, and practise the switch.

1. Set objectives per component with the business, since not everything needs regional resilience.
2. Replicate data: asynchronous database replicas and object storage replication to the second region.
3. Make infrastructure reproducible with code, so the secondary region matches the primary.
4. Replicate dependencies: identity, secrets, encryption keys, container images and configuration must all exist in the second region.
5. Prepare capacity: pre-provisioned, or at least confirmed quotas, since everyone scales up in a large outage.
6. Switch traffic with global load balancing or DNS failover.
7. Write and practise the runbook, including who decides to fail over and how to fail back.

flowchart LR
  A["Users"] --> B["Global traffic routing"]
  B -->|normal| C["Primary region"]
  B -.->|failover| D["Secondary region"]
  C -->|async replication| D

Trade-offs: asynchronous replication means recent writes can be lost on failover, so agree on how to reconcile them afterwards. Automatic regional failover risks flapping on partial outages, so many teams keep the final decision with a human.

Move the state out of the servers, into shared, managed stores, so any server can handle any request and servers can be added or removed freely.

• Sessions: store them in a shared cache or database, or use signed tokens held by the client, instead of server memory.
• Uploaded files: write them to object storage rather than the local disk.
• Caches: local in-memory caches are fine if they can be rebuilt, but shared data that must be consistent belongs in a shared cache.
• Scheduled jobs: run them in a separate worker or use leader election, so ten servers do not send the same report ten times.
• Locks and counters: use the database or a coordination service, not process memory.
• The database itself: scale reads with replicas and caching, and writes with partitioning or sharding when a single primary is no longer enough.

Sticky sessions, where the load balancer pins a user to one server, are a quick fix. They lose users' sessions when that server fails and spread load unevenly, so treat them as a temporary step.

Once servers hold no unique state, autoscaling, rolling deployments and zone failures all become routine.

A statically stable system keeps working through a failure without needing to change anything, such as launching servers, updating DNS or calling management APIs. It matters because those management functions are often degraded during the same large outage.

How to achieve it:

• Pre-provision capacity across zones, so losing one zone leaves enough running capacity, rather than relying on autoscaling to replace it quickly.
• Avoid control-plane dependencies on the recovery path: reading configuration or credentials from a management API during failover can fail just when you need it.
• Cache what the data plane needs, such as configuration, credentials and routing data, so services keep running if their source becomes unavailable.
• Keep failover routes in place, with health-checked routing configured in advance, so recovery needs no new records.

The trade-off is cost: running spare capacity all the time costs more than scaling on demand. The decision depends on how much an outage would cost compared with that standing capacity.

Static stability is a mindset. For each failure, ask whether the system survives as it is right now, with nothing new created.

FinOps is the practice of managing cloud spending as a shared responsibility between engineering, finance and the business, so decisions balance speed, quality and cost with timely data. In the cloud every engineer makes spending decisions, so cost cannot stay a finance-only topic.

How responsibility is shared:

• Engineering teams own their usage: they see their costs, fix waste and consider cost in design reviews.
• A central FinOps function provides tooling and data, negotiates commitments and discounts, and sets standards such as tagging.
• Finance forecasts and budgets with engineering input, rather than receiving surprises at month-end.
• Leadership sets targets in business terms, such as cost per customer, rather than flat cuts.

Principles that make it work: cost data that is timely and visible to every team, decisions driven by business value rather than the lowest bill, and a culture where cost is a design constraint like latency, discussed without blame.

Both allocate cloud costs to the teams that cause them. Showback reports each team's costs for awareness, while chargeback actually bills those costs to the team's budget.

When to use which:

• Start with showback while tagging and allocation rules are being established; billing teams for numbers they do not trust creates conflict.
• Move to chargeback once allocation is accurate and teams can influence their costs, so the incentive is fair.

Shared costs need a clear rule in both models. Networking hubs, security tooling and support plans are usually split by usage or by each team's share of direct spend, and the rule must be published so teams can predict their charges.