MLOps Engineer Top 100 Interview Questions

State cost per thousand predictions, so it can be compared across models and tracked as traffic grows. Then reduce it by using hardware more fully, matching capacity to traffic and making the model cheaper to run.

Estimating: add up hourly costs of all replicas, including idle ones, divide by the predictions served in the same period and multiply by 1,000.

Reducing cost:

1. Batching: raises throughput per replica, especially on accelerators.
2. Autoscaling: scale down at night and on quiet days, with a sensible minimum.
3. Right-sizing: use smaller or cheaper hardware when the model does not need accelerators.
4. Model efficiency: lower-precision weights, smaller distilled models or pruning, validated against quality gates.
5. Caching and batch precomputation for repeated inputs or known entities.
6. Interruptible capacity for extra replicas above the baseline.

Track quality alongside cost, so savings never come at the expense of an unnoticed accuracy drop.

Compare the measured value the model adds against its full cost of ownership, including people, not just servers. Many models keep running long after their benefit has faded, quietly consuming budget and on-call time.

Value side:

• The improvement over the simplest alternative, such as a rule or the previous model, not over doing nothing.
• Measured through experiments or holdouts where possible, rather than assumed from offline metrics.

Cost side:

1. Serving and training infrastructure.
2. Data pipelines and storage specific to the model.
3. Engineering time for maintenance, retraining, incidents and on-call.
4. Governance work, such as audits and documentation.
5. Opportunity cost: the team could build something else.

Decision options: keep and invest, simplify to cut cost, merge with another model, replace with a rule or retire it.

Review value and cost for each production model at least once a year.

Run a baseline of replicas on regular capacity that can serve normal traffic alone, and add interruptible replicas for extra throughput and peaks. When interruptible machines are reclaimed, the baseline keeps the service up while replacements start.

Practices:

1. Spread across machine types and zones, since interruptions often hit one pool at a time.
2. Handle the reclaim notice: stop accepting new requests, finish in-flight ones and deregister from the load balancer.
3. Fast startup: cached images and weights, so replacements join quickly.
4. Monitor the share of traffic on interruptible replicas, and alert when the baseline alone would be overloaded.
5. Keep batch scoring flexible: batch jobs can run almost entirely on interruptible capacity, since they can be retried.

Test the setup by deliberately removing interruptible replicas during peak traffic and confirming latency stays within limits.

Measure the model's real resource use under realistic load, then choose the cheapest hardware and settings that meet the latency target with a safety margin. Teams often default to the largest machine and never revisit the choice.

1. Profile one replica: memory for weights and activations, processor and accelerator use, and latency at increasing load.
2. Try smaller and different hardware: many models meet their targets on general-purpose processors, especially with batching and efficient runtimes.
3. Tune concurrency and batch size: the right settings often double throughput on the same hardware.
4. Pack models: if one model leaves an accelerator mostly idle, host several models on it.
5. Set requests and limits for memory and processors from measurements, with headroom for spikes, so the scheduler can pack replicas efficiently without out-of-memory kills.
6. Revisit after changes: new model versions, libraries or traffic patterns can change the best choice.

What to compare: cost per thousand predictions at the latency target, not raw speed. A cheaper machine with slightly higher latency can be the right choice if the latency budget allows it.

Treat prompts as code: store them in version control, test them against an evaluation set before release and roll them out gradually with monitoring. A one-word change to a prompt can change behavior as much as a new model.

1. Version everything that shapes output: the prompt template, system instructions, examples, model identifier and generation settings such as temperature.
2. Review changes through the normal code review process, with a short description of the intended behavior change.
3. Evaluate automatically against a fixed set of test cases with expected properties, comparing the new version with the current one.
4. Roll out gradually: a small share of traffic first, comparing quality signals, cost and latency.
5. Log the version with every response, so issues can be traced to the prompt that produced them.
6. Keep rollback instant: switching back to the previous version should be a configuration change, not a redeploy.

Watch for: changes in the underlying model when a provider updates it, which can alter behavior even if your prompt did not change; pin model versions where the provider allows it.

Maintain a versioned set of test cases, run every candidate configuration against them automatically and compare results with the current production version before release. It turns "it seems better" into evidence.

flowchart LR
  A["Prompt or model change"] --> B["Run evaluation set"]
  B --> C["Deterministic checks: format, length, forbidden content"]
  B --> D["Graded checks: rubric scoring by reviewers or a judge model"]
  C --> E["Compare with production version"]
  D --> E
  E --> F{"Better or equal on all gates?"}
  F -->|Yes| G["Approve for gradual rollout"]
  F -->|No| H["Block and report regressions"]

Building the evaluation set:

• Real user requests, anonymized, covering common and difficult cases.
• Known failure cases from past incidents.
• Adversarial cases, such as attempts to override instructions.

Running it well:

1. Fix generation settings and repeat sampled cases a few times, since outputs vary between runs.
2. Track cost and latency alongside quality.
3. Calibrate automated graders against human ratings regularly.
4. Grow the set whenever production reveals a new failure type.

Combine standard service monitoring with signals specific to generated output: cost, quality indicators, safety flags and user feedback. Generated text has no single correct answer, so monitoring relies on many partial signals.

Signals to track:

• Service: latency to first token and to completion, error and timeout rates, and provider rate-limit responses.
• Cost: input and output tokens per request and per feature, with budget alerts.
• Output checks: format validity, response length, refusal rates and automated safety classifier flags.
• User feedback: ratings, retries, edits and copy actions, which indicate whether output was useful.
• Sampled quality review: a small share of conversations graded by people or by an automated grader.
• Inputs: topic mix and attempts to override instructions, which change over time.

Practices:

1. Tag every metric with the prompt version and model identifier.
2. Alert on sudden changes in refusals, safety flags or cost per request, which often follow provider-side changes.
3. Protect privacy in logs: mask personal data and restrict access.
4. Feed reviewed failures back into the evaluation set.

A gateway gives applications one internal interface for calling language models, while centralizing routing, security, cost control and logging. Teams build features without each one integrating and governing every provider separately.

flowchart LR
  A["Application features"] --> B["Model gateway"]
  B --> C["Policy: authentication, quotas, data rules"]
  C --> D["Router"]
  D --> E["Primary model provider"]
  D --> F["Fallback provider or self-hosted model"]
  B --> G["Logs, costs and metrics"]

Main responsibilities:

• One API: a consistent request and response format across models.
• Routing: choose models by task, cost or latency, and fail over when a provider has an outage.
• Quotas and budgets: per team and per feature, with alerts and hard limits.
• Security and data rules: keep secrets in one place, mask personal data and block disallowed data from leaving the company.
• Caching: reuse responses for repeated identical requests where safe.
• Observability: log usage, latency, errors and costs, tagged by feature.

Risks: the gateway becomes critical infrastructure, so it needs high availability, careful capacity planning and low added latency.

Caching saves cost and latency for repeated requests, but generated answers can depend on context, user permissions and time. Cache only where a reused answer is genuinely correct for the new request.

Rules for safety:

1. Include in the key everything that changes the answer: prompt version, model identifier, settings and retrieved documents.
2. Never share cached answers across users when responses depend on personal data or permissions.
3. Set expiry times that match how quickly the underlying information changes.
4. Invalidate entries when source documents or prompts change.
5. Monitor hit rates and sample cached responses for quality.

Record where every dataset and pretrained model came from, under what terms it may be used, and which production models depend on it. License problems discovered after launch can force a model to be withdrawn.

What to record for each asset:

• Source, version and date obtained.
• License or contract terms: commercial use, redistribution, attribution and restrictions on certain uses.
• Whether it contains personal data, and the legal basis for using it.
• Known limitations, such as language coverage or collection biases.

Practices:

1. Approval before use: new external datasets and pretrained models go through a lightweight review with legal and security input.
2. Lineage: link each production model to the datasets and base models it was built from, so a license change shows its impact immediately.
3. Scan downloaded weights and files for safety, and load them only in isolated environments until approved.
4. Respect attribution and notice requirements in products and documentation.
5. Re-check terms when licenses or usage plans change.

A runbook lets whoever is on call diagnose and mitigate common model problems quickly, without needing the model's original authors. It should be short, specific and tested.

1. Overview: what the model does, its consumers, its owners and its risk tier.
2. Dashboards and logs: direct links, with what normal looks like.
3. Common symptoms and first checks:
   • Latency or errors: check recent deployments, dependencies and resource limits.
   • Prediction shift: check feature health and upstream pipeline status.
   • Outcome drop: check thresholds, business rule changes and label delays.
4. Mitigations, from safest to most drastic: roll back to the previous model version, switch to a fallback model or rule, adjust a threshold with approval, or disable the feature.
5. Commands and permissions: exactly how to perform each mitigation, and who is allowed to.
6. Escalation: who to contact for data, model and infrastructure problems.
7. Communication: which teams and stakeholders to notify.

Practice the runbook with game days, and update it after every incident.

Interview tip: naming a fallback model or rule as a mitigation, rather than only "roll back", shows you design for failure in advance.

Machine learning incidents cross team boundaries: the failing part may be data, the model or the serving infrastructure. Define who responds to each kind of signal, and give the first responder the tools to triage across all three.

Making it work:

1. One entry point: a primary on-call engineer who triages using shared dashboards and runbooks, then pulls in the right owner.
2. Clear severity levels, tied to user and business impact.
3. Safe mitigations available to the first responder, such as rollback or fallback, without waiting for specialists.
4. Sustainable load: track pages per shift, fix noisy alerts and keep rotations large enough.
5. Review incidents together, since root causes often sit between teams.

Measure how quickly and safely teams move models into production and keep them healthy, in the same way delivery metrics measure software teams. Platform health is about the users of the platform, not just its uptime.

Delivery metrics:

• Lead time: from a promising experiment to the model serving production traffic.
• Release frequency: how often models and pipelines are updated.
• Change failure rate: the share of model releases that cause incidents or rollbacks.
• Time to restore: how long it takes to recover from a model incident.

Operational metrics:

• Pipeline success rate and retry rate.
• Training queue time and accelerator utilization.
• Serving availability and latency across models.
• Share of production models with monitoring, lineage and an owner.

User experience:

• Time for a new team to deploy its first model.
• Survey results and support requests from data scientists.

Use these numbers to prioritize platform work, such as fixing the slowest step in the path to production.

Run both stacks in parallel, prove that the new one produces the same predictions and meets performance targets, then shift traffic gradually with a fast way back.

flowchart LR
  A["Deploy model on new stack"] --> B["Parity tests on recorded requests"]
  B --> C["Shadow live traffic and compare"]
  C --> D["Canary: small share of live traffic"]
  D --> E["Gradual shift to 100%"]
  E --> F["Decommission old stack after a waiting period"]

Parity checks:

1. Predictions: the same inputs give the same outputs within a tight tolerance; differences usually come from preprocessing, library versions or numerical precision.
2. Edge cases: missing values, unusual categories and very large inputs behave identically.
3. Performance: latency and throughput at production load.
4. Operations: metrics, logs, alerts and runbooks work on the new stack.

Migration order: start with low-risk, low-traffic models to learn, then move critical ones. Keep the old stack ready until every model has run on the new one through a full business cycle, such as a month-end.

Retire a model deliberately: confirm no one still depends on it, move consumers to a replacement, stop traffic, then archive its artifacts and records according to retention rules. Forgotten models keep consuming resources and can be called by surprise.

1. Decide: low measured value, a replacement model or a changed business process are common reasons.
2. Find consumers: use serving logs and the model inventory to list every caller.
3. Announce a timeline, with a migration path and a contact.
4. Reduce and stop traffic: monitor for remaining calls, and contact any caller still active.
5. Remove infrastructure: endpoints, scheduled training jobs, feature pipelines only it uses and alerts.
6. Archive: keep the artifact, lineage, documentation and decision logs as long as legal and audit rules require.
7. Update the inventory to show the model as retired, with the reason and date.

Watch for hidden dependencies: batch outputs read by other teams' jobs, features shared with other models and reports that depend on its scores.

Find which step got slower, then compare that step's inputs, resources and code between a fast run and a slow run. Pipelines usually slow down because data grew, a cache stopped working, resources shrank or a code change made a step less efficient.

1. Compare step durations between recent fast and slow runs; usually one or two steps account for the whole increase.
2. Check data volume: row counts and file sizes for that step's inputs; a duplicated join or a missing date filter can multiply data silently.
3. Check caching: a changed cache key, such as a new image digest, can force expensive steps to rerun every time.
4. Check resources: fewer workers, smaller machines, lower quotas or longer queue times on a busy cluster.
5. Check the code: commits to that step since the last fast run, such as a new feature computed row by row.
6. Check storage: many small files, a new remote region or throttled reads.

Fix the cause rather than adding hardware, then add a duration alert per step, so the next slowdown is noticed after one run instead of many.

The offline evaluation did not represent production. Check whether the data, the features or the conditions differed between evaluation and serving, before concluding the model itself is worse.

1. Feature parity: compare logged serving features with the features used in evaluation for the same entities; missing values, different defaults or different transformations are the most common cause.
2. Freshness: features that were complete offline may be delayed or partial at serving time.
3. Evaluation data: was the test set recent and drawn from the same population as live traffic, and was it free of leakage?
4. Thresholds: a different score scale with an unchanged threshold changes decisions.
5. Canary assignment: check that canary users were comparable to control users, and that the sample was large enough.
6. Feedback effects: the old model shaped the historical data, such as which cases were reviewed, so offline labels may favor models that agree with it.

Reproduce a sample of canary predictions offline with the exact logged inputs; if they match the live outputs, the difference is in the data, not in serving.

Compare the old and new versions on the same requests and the same hardware, and break latency into stages. The cause is usually a larger model, a change in preprocessing or a serving setting that changed with the release.

1. Model size and architecture: more layers, larger embeddings or longer input sequences increase compute per request.
2. Input-dependent cost: if latency grew mainly for large inputs, the new model may scale worse with input length.
3. Batching settings: a new configuration may have changed the maximum batch size or wait.
4. Memory pressure: a larger model can leave less room for batches or caches, or cause swapping and garbage collection pauses.
5. Warm-up and compilation: missing warm-up can make early requests slow after every scale-up.
6. Feature lookups: a new feature from a slower store adds a dependency on the critical path.
7. Library changes: a new runtime version may have lost an optimized code path.

Profile one request end to end in both versions; the stage that grew is where to look.

Identical inputs should give identical outputs from the same model version, so differences point to replicas that are not actually the same, or to inputs that are not actually identical by the time they reach the model.

Common causes:

• Mixed model versions: a rollout stalled halfway, or some replicas failed to load the new artifact and kept the old one.
• Different dependencies: images rebuilt at different times with unpinned libraries, or different hardware types with different numerical behavior.
• Local state: per-replica caches, feature values fetched at slightly different times or in-memory counters.
• Randomness: sampling or dropout accidentally left active at inference time.
• Configuration drift: a threshold or feature flag set differently on some replicas.

How to find it:

1. Send the same request to each replica directly, and compare model version, image digest and configuration reported in the responses.
2. Log the version and a configuration hash on every prediction, so mismatches are visible in aggregate.
3. Fix the rollout process: readiness checks that confirm the expected version, and deployment by immutable image digest.

Unless the system tracks feature freshness, the model kept predicting with six-hour-old features without any error. For slowly changing features that may be harmless; for features about recent behavior, such as transactions in the last hour, predictions could be badly wrong.

What the system should do:

1. Detect it: compare each feature's latest timestamp with its freshness target, and alert before the target is breached.
2. Degrade deliberately: for critical stale features, switch to a fallback, such as a model trained without those features, or conservative rules.
3. Flag affected predictions in logs, so they can be excluded from monitoring and analysis, or reviewed.
4. Communicate: tell downstream owners that decisions during the window used degraded inputs.
5. Recover cleanly: when the pipeline catches up, confirm values are correct before returning to the main model.

Afterwards:

• Find the delay's root cause and add an earlier warning signal.
• Estimate the impact by comparing decisions made during the window with what the model would have decided with fresh features.
• Train the model with realistic staleness, so moderate delays are less harmful.