MLOps Engineer Top 100 Interview Questions

Caching helps when the same inputs recur and predictions can be slightly stale without harm. It cuts latency and cost, but it can serve outdated or wrong answers if inputs and invalidation are not handled carefully.

Rules for safe caching:

1. Include every model input and the model version in the cache key, so a new model never serves old predictions.
2. Set expiry times that match how quickly the underlying features change.
3. Never cache predictions that depend on fast-changing signals, such as fraud checks on recent activity.
4. Track the hit rate and the age of served predictions.
5. Keep personalized predictions isolated per user, so one user's result never reaches another.

Treat it as a pipeline that reads a snapshot, scores in parallel partitions, validates the output and then publishes it atomically, so downstream systems either see a complete new set of scores or the previous complete set.

flowchart LR
  A["Snapshot customer features"] --> B["Split into partitions"]
  B --> C["Score partitions in parallel"]
  C --> D["Validate output: counts, ranges, distribution"]
  D --> E{"Checks pass?"}
  E -->|Yes| F["Publish new scores atomically"]
  E -->|No| G["Keep previous scores and alert"]

Design points:

1. Fixed inputs: score from a dated feature snapshot, and record it with the output.
2. Parallel partitions: split by customer ID ranges or hash, sized so each finishes in minutes and can be retried independently.
3. Model version pinned: load the approved version by identifier, never "latest".
4. Output validation: row count matches the input, no missing scores and the score distribution is close to the previous night's.
5. Atomic publish: write to a new dated table or location, then switch a pointer, so readers never see partial results.
6. Freshness monitoring: alert if scores are not published by the time downstream jobs need them.

Keep a few previous runs, so a bad night can be rolled back by switching the pointer.

Make each run's output a pure function of its inputs, and write it to a location determined by those inputs, replacing any earlier output for the same partition. Rerunning a day then produces exactly the same result instead of duplicates.

Key practices:

• Partition by business date and model version, such as one partition per scoring date.
• Overwrite, never append: a rerun replaces the whole partition, written to a temporary location first and swapped in when complete.
• Deterministic inputs: read the feature snapshot for that date, not current tables.
• Stable identifiers: each output row carries the entity ID, date and model version.

Backfilling:

1. Score historical dates with the model version you are evaluating, using that date's features.
2. Run partitions in parallel with a concurrency limit, so the backfill does not starve daily jobs.
3. Track progress per partition, so failures are retried individually.
4. Keep backfilled outputs separate from production outputs until they are validated.

Backfilled predictions let you evaluate a new model on past periods, or rebuild outputs after a bug, without special one-off scripts.

Edge deployment moves the model to devices you do not control, with limited memory, battery and connectivity. Models must be small, updates must be safe to roll out gradually, and you need telemetry to know how they behave.

Practices for on-device models:

1. Compress: reduce model size with lower-precision weights and smaller architectures, then confirm accuracy on device-like hardware.
2. Ship models separately from the app, so they can be updated without an app release, with checksums and signatures.
3. Roll out gradually by device group, with a kill switch and the previous model kept for fallback.
4. Collect privacy-respecting telemetry: latency, crashes, model version and aggregate prediction statistics.
5. Support old versions for a long time, and make sure server components accept outputs from all versions still in use.

Replay realistic requests at increasing rates until latency or errors exceed their limits, and record the highest rate a replica sustains within the latency target. Use real request mixes, since input size often changes cost dramatically.

How to run it:

1. Realistic payloads: sample real requests, with their natural spread of sizes, rather than one fixed example.
2. Warm-up: exclude the first minutes, so cold-start effects do not distort results.
3. Step the load: increase requests per second gradually, holding each step long enough to stabilize.
4. Test dependencies: include feature lookups, or capacity limits elsewhere will be missed.

What to measure:

• Latency percentiles at each load level, especially the 99th.
• Error and timeout rates.
• Throughput per replica at the latency target.
• Resource use: processor, memory, accelerator and batch sizes.

Turning results into capacity: the number of requests in flight equals the request rate times latency. Plan replicas so each runs at about 70% of its measured maximum, leaving room for spikes and the loss of a replica.

Separate the model version from the API contract. A new model can ship behind the same contract as long as inputs, outputs and their meaning stay compatible; anything that changes the contract needs a new API version that clients adopt deliberately.

Practices:

1. Explicit schemas for requests and responses, versioned and shared with clients.
2. Return the model version in every response, so clients and logs can attribute behavior.
3. Run old and new API versions side by side during migration, with a published end date for the old one.
4. Contract tests in the pipeline that send recorded client requests to the new model and check that responses stay valid.

Score meaning is the change most often missed: if downstream rules use thresholds, a recalibrated score can break them while every field still validates.

Downstream systems often act on fixed thresholds, such as blocking transactions scoring above 0.8. If the new model's scores are distributed differently, the same threshold produces a different number of actions, even when the model ranks cases better.

How to handle it:

1. Compare score distributions of old and new models on the same recent data before release.
2. Calibrate the new model, so its scores match observed outcome rates, such as "0.8 means about 80% of these are fraud".
3. Or map thresholds: choose the new threshold that produces the same action rate, then adjust deliberately if the business wants a different rate.
4. Tell downstream owners, such as review teams whose workload depends on the number of flagged cases.
5. Monitor action rates during the canary, with rollback limits.

A model that ranks better but floods a review team with three times the cases can be a net loss until thresholds are adjusted.

Record the exact feature values the model received for each prediction, together with the request ID, timestamp and model version. Later, join these logs with outcomes to build training data that reflects what the model really saw in production.

Why it helps:

• Training on logged features removes a whole class of mismatches between training and serving pipelines.
• It captures real-world staleness and missing values, which recomputed historical features hide.
• It provides an audit trail for individual predictions.

How to do it well:

1. Log asynchronously, so logging never adds latency to predictions.
2. Use a stable schema with feature names and versions.
3. Join with outcomes by request ID once labels arrive, such as a purchase or a chargeback.
4. Sample when volume is huge, keeping all rare but important cases, such as flagged predictions.
5. Apply privacy rules: exclude or protect personal data, and enforce retention limits.

Limitation: new features cannot be learned from old logs, since they were never logged; they still need a historical backfill.

Ship the feature first, prove it works in production, and only then switch on the model that uses it. Deploying a new feature pipeline and a new model together makes failures hard to diagnose and roll back.

flowchart TD
  A["Build feature pipeline and backfill history"] --> B["Deploy feature to production, unused"]
  B --> C["Monitor freshness, null rate and distribution"]
  C --> D["Compare live values with backfilled values"]
  D --> E["Train model with the feature"]
  E --> F["Shadow, then canary the new model"]
  F --> G["Full rollout"]

Why this order works:

• The feature's live behavior, including delays and missing values, is known before the model depends on it.
• Differences between backfilled and live values are found before they distort training.
• If the model needs rolling back, the feature can stay, since the old model ignores it.

Also plan for failure: the model needs defined behavior when the feature is missing, either a default value it saw during training or a fallback to the previous model.

Use a routing layer that selects the model version for each request based on explicit rules, keep each version independently deployable and record which version served every prediction.

Practices:

1. One routing decision point, so rules are not scattered across clients.
2. Configuration changes reviewed like code, with an audit trail, since routing changes alter behavior for real users.
3. A default route for requests that match no rule.
4. Per-version monitoring, since an aggregate can hide one bad version.
5. Limit the number of versions: each one needs retraining, monitoring and support, so merge them when their differences no longer pay off.

Log the route and model version with each prediction, so outcomes can be analyzed per version.

Monitor four layers, because each one catches failures the others miss. A model can be fast and error-free while making useless predictions, and accurate predictions can still fail to help the business.

How to combine them:

1. Alert on service health and severe input problems immediately.
2. Review prediction and data drift daily, with alerts for large shifts.
3. Track outcome metrics on dashboards and in regular model reviews.
4. Tag every metric with the model version, so a release can be compared with the previous one.

When true labels arrive late, input and prediction monitoring are the early warning system.

Broken features rarely cause errors. They show up as sudden changes in simple statistics: more missing values, a flood of one default value, or a range that suddenly shifts. Monitoring those statistics per feature, against a reference, catches most pipeline breaks quickly.

Checks per feature, per time window:

• Null rate compared with its normal level.
• Constant or default values: the share of the most common value, which jumps when a join fails and a default such as 0 is filled in.
• Range and distribution: minimum, maximum and percentiles compared with a reference window.
• Category frequencies: new categories appearing, or a known category disappearing.
• Freshness: the age of the newest value.

Making the alerts useful:

1. Compute references from a stable recent period, and refresh them deliberately.
2. Set thresholds per feature, since some are naturally sparse.
3. Report which features broke and since when, so the owning team can act.
4. Weight alerts by feature importance, since a broken minor feature matters less.

Compare the two distributions directly, for example by the largest vertical gap between their cumulative distribution curves. The gap is 0 when the distributions match and approaches 1 when they do not overlap at all, and it needs no assumptions about their shape.

How to use it:

1. Reference window: predictions from a stable period, such as the weeks after a validated release.
2. Current window: recent predictions, large enough to be stable but recent enough to respond quickly, such as the last day.
3. Threshold: with large windows, formal significance tests flag tiny, harmless differences, so set a practical threshold from historical variation instead.
4. Segments: compare per important segment too, since overall distributions can hide shifts.

Interpreting a shift:

• A shift in predictions with no shift in inputs suggests a model or serving change.
• A shift in both suggests the population or an upstream pipeline changed.
• Some shifts are expected, such as seasonal behavior, so annotate known events.

Pair distribution checks with outcome metrics when labels arrive, since a shifted distribution is a warning, not proof of worse accuracy.

Alert only on conditions that need someone to act, set thresholds from the metric's own history and route everything else to dashboards and scheduled reviews. Alert fatigue causes real incidents to be ignored.

Principles:

1. Page for user impact: service outages, severe input breaks and large shifts in decisions, such as the approval rate halving.
2. Ticket, do not page, for gradual drift and moderate data changes, reviewed during working hours.
3. Use history: set thresholds from the normal variation of each metric, such as the range seen over the last eight weeks, rather than a single universal number.
4. Require persistence: alert when a condition lasts for several windows, not on a single noisy point.
5. Group related alerts: one upstream failure can break twenty features; alert once, with the list.
6. Account for seasonality: compare with the same hour or weekday, not just the previous window.

Keep the system healthy:

• Review every alert after the fact: was it real, and was the threshold right?
• Delete or tune alerts that are never acted upon.

Interview tip: describing which alerts page a person and which only open a ticket shows you have operated real monitoring, not just designed it.

Show the chain from the model's behavior to the business result it is meant to change, with the same time ranges and segments on every chart, so a viewer can see where a problem starts.

A useful dashboard structure:

1. Headline outcome: the business metric the model serves, such as fraud losses as a share of sales.
2. Decisions: how many actions the model drives, such as blocked or flagged transactions, and their rate.
3. Decision quality: precision of actions, confirmed by later labels or human reviewers, and estimates of missed cases.
4. Model health: score distributions, drift and feature health.
5. Service health: latency and errors.

Making it trustworthy:

• Annotate releases, threshold changes and known events on every chart.
• Show label delay: recent quality metrics are incomplete, so mark them as provisional.
• Break results down by model version and key segments.
• Agree on the definitions with business owners, so the dashboard is the reference everyone uses.

When outcomes move, this structure tells you whether the cause is the model, the volume of cases or the business environment.

Log a structured record for each prediction with everything needed to explain it later: request ID, timestamp, model version, input features, output and the decision taken. Without this record, investigating a complaint or an incident becomes guesswork.

What each record should contain:

• Identity: request ID and, where allowed, a pseudonymous entity ID.
• Versions: model version, feature definitions version and routing rule.
• Inputs: feature values as received, including which ones used defaults.
• Outputs: raw scores, the final decision and any business rule that overrode the model.
• Context: latency and the serving replica.

Operational practices:

1. Write logs asynchronously to durable storage, so logging never slows predictions.
2. Sample high-volume routine predictions if needed, but keep all predictions that led to significant actions.
3. Protect personal data: restrict access, mask sensitive fields and enforce retention periods.
4. Make logs queryable by request ID and by time, so investigations take minutes.
5. Join logs with outcomes later for monitoring and training.

Start from the logged prediction record and walk backwards through the lineage: the inputs it received, where each input came from and which model version produced the result. Reproduce the prediction offline, then change one thing at a time.

flowchart TD
  A["Complaint or flagged prediction"] --> B["Find prediction log by request ID"]
  B --> C["Check model version and routing"]
  B --> D["Check logged feature values"]
  D --> E{"Features correct at source?"}
  E -->|No| F["Trace the feature pipeline and its inputs"]
  E -->|Yes| G["Reproduce prediction offline"]
  G --> H["Explain behavior: similar training cases, feature effects"]

Typical findings:

• A feature had a default value because of a failed lookup.
• An outdated model version was still served by one replica.
• A business rule overrode the model.
• The model genuinely got it wrong, because similar cases were rare or mislabeled in training.

Turning findings into improvements: add a monitoring check for data problems, a behavioral test for model mistakes and a regression example to the evaluation set, so the same failure is caught automatically next time.

Governance makes sure every model in production is known, owned, documented, approved and reviewed, with stricter controls for models whose mistakes can hurt people or the business. Applying the heaviest process to every model slows teams without improving safety.

Core elements for every model:

1. Inventory: a registry of all production models, their owners, purpose and risk tier.
2. Documentation: intended use, training data, evaluation results, limitations and known failure modes.
3. Approval records: who approved each release and on what evidence.
4. Monitoring and incident handling, with defined responsibilities.
5. Retirement: a process for decommissioning models that are no longer needed.

Automate as much as possible through the pipeline, so governance is a by-product of releasing, not separate paperwork.

Apply least privilege to every asset in the machine learning lifecycle, with identities for people and for pipelines, and grant access per project rather than across the whole platform.

Assets and typical access rules:

• Raw data with personal information: only data engineering pipelines and a few approved people.
• Training datasets: project members, read-only, preferably pseudonymized.
• Feature stores: read access per feature group; write access only for the owning pipeline.
• Model registry: pipelines create versions; promotion to production requires an approver role.
• Serving endpoints: callers authenticate, and each client is allowed only the models it needs.

Practices:

1. Use service identities for pipelines, with short-lived credentials instead of shared keys.
2. Separate development and production environments, with production data unavailable in development by default.
3. Review access regularly, and remove it automatically when people change projects.
4. Log access to sensitive datasets and model artifacts.
5. Treat model artifacts as sensitive: they can leak training data and represent valuable intellectual property.

For any individual decision, you should be able to show which model version made it, with which inputs, under which approved process, and what the model's documented performance was at the time. Build this into normal logging and release processes, so audits do not require reconstruction.

What the trail must connect:

1. The decision: prediction log with request ID, timestamp, inputs, output and final action.
2. The model version: registry entry with lineage to code, training data snapshot and environment.
3. Validation evidence: evaluation results, slice and fairness analyses and the approval record.
4. Changes: history of thresholds, business rules and routing configuration, with who changed them and when.
5. Monitoring history: what was known about performance and drift at the time.

Making it robust:

• Store records in tamper-evident, access-controlled storage with defined retention periods.
• Keep the model artifact itself, since exact retraining may not be possible years later.
• Be able to produce a human-readable explanation of the main factors behind a decision.
• Test the process with a mock audit: pick a past decision and reconstruct its full story.