MLOps Engineer Top 100 Interview Questions

An MLOps engineer owns the path that takes a model from an experiment to a reliable production service, and keeps it healthy afterwards. Data scientists focus on whether a model is good; DevOps engineers focus on whether software runs reliably. MLOps covers the gap: data, models and code that change on different schedules and fail in different ways.

Typical responsibilities:

• Pipelines: automated, repeatable training, evaluation and batch scoring jobs.
• Lineage: linking every deployed model to its code, data, configuration and environment.
• Release: quality gates, shadow and canary rollouts, and fast rollback for models.
• Serving: model servers that meet latency, throughput and cost targets.
• Monitoring: service health plus data quality, prediction behavior and business outcomes.
• Platform: shared tools and templates, so each team does not rebuild the same infrastructure.
• Governance: access control, audit trails, approvals and data deletion handling.

The defining skill is treating data and models as production dependencies that can break silently, without errors, while the service itself looks healthy.

Teams typically progress from manual work to automated training, then to automated delivery of the pipeline itself. Each level removes a class of human error and shortens the time from idea to production.

How to move up a level:

1. Make one model fully reproducible, with versioned code, data and environment.
2. Turn its notebook steps into pipeline steps with clear inputs and outputs.
3. Add automated validation and a registry, so promotion is a controlled step.
4. Put the pipeline code itself under tests and automated deployment.

Not every model needs the top level; a quarterly report model may be fine with a simpler process.

They automate different things. Continuous training produces new model versions from new data with the same code. Continuous delivery ships new code, such as a new feature pipeline or serving logic, through testing into production.

Why the distinction matters:

• A retrained model needs evaluation against the current model, not just passing tests.
• A code change may need retraining before release, because it changes features.
• Both need rollback paths, and the registry must record which code and data produced each model.

Most mature setups run both: continuous delivery deploys the training pipeline, and that pipeline then retrains models on its own schedule.

Model lineage is the recorded chain of everything that produced a deployed model: code version, training data snapshot, feature definitions, configuration, environment and evaluation results. It lets you answer "why does this model behave this way?" and rebuild it exactly.

flowchart LR
  A["Code commit"] --> E["Training run"]
  B["Data snapshot ID"] --> E
  C["Feature definitions version"] --> E
  D["Config and environment image"] --> E
  E --> F["Model version with metrics"]
  F --> G["Deployment record"]

How to capture it:

1. Automatically, in the pipeline: each run records its inputs' identifiers, never relying on people to type them.
2. Immutable references: commit hashes, dataset snapshot IDs and image digests, not names such as "latest".
3. Store with the model: the registry entry links to the run, and the deployment record links to the registry entry.
4. Include evaluation: metrics, slice results and the dataset they were measured on.

What it enables: reproducing a model for an audit, finding every model affected by a bad data snapshot, and comparing two versions by exactly what changed between them.

Give every training run a record with the same structure, in a shared tracking system, so anyone can compare runs and trace a promising result back to how it was produced.

What each run should log:

• Identity: a run ID, owner, project and a short description of the hypothesis.
• Inputs: code commit, data snapshot ID, feature set version and every hyperparameter.
• Environment: container image digest and hardware type.
• Outputs: metrics over training, final evaluation metrics, slice metrics and artifacts such as the model file and plots.

Team conventions that keep it usable:

1. Log automatically from training code through a shared helper, so nothing depends on memory.
2. Agree on metric names and evaluation datasets, so comparisons are like for like.
3. Group runs by experiment, with the question being tested.
4. Tag the runs that led to decisions, and link promoted models to their runs.
5. Clean up or archive abandoned runs, keeping the record but hiding noise.

Tracking pays off when someone asks months later why a model was chosen, or when a regression needs comparing against an older run.

Pin everything you control, record everything you cannot, and decide how much variation is acceptable. Bit-for-bit identical results are often impractical on accelerators, but results within a known tolerance are achievable.

Sources of variation and how to control them:

1. Randomness: fix seeds for every random number generator used, including data shuffling and weight initialization.
2. Data order and content: train from an immutable snapshot, with a deterministic split and a recorded shuffle seed.
3. Libraries: run training in a pinned container image, referenced by digest.
4. Hardware: some parallel operations on accelerators are nondeterministic because the order of floating-point additions varies; deterministic modes exist but are slower.
5. Distributed training: the number of workers changes batch composition and reduction order, so record it.

Practical standard:

• Re-run a training job twice and measure the spread of the final metric; use that spread as the tolerance for "reproduced".
• Treat a metric difference larger than that tolerance between versions as a real change.
• For audits, store the trained artifact itself, since exact retraining may not be possible later.

Hash a stable identifier, such as a customer ID, and assign the record to a set based on the hash value. The same ID always produces the same hash, so a record never moves between training and validation when data is refreshed or reprocessed.

Why it matters:

• Random splits redone on each run let validation records leak into training over time, inflating metrics.
• Splitting by row instead of by entity lets the same customer appear in both sets, which also inflates metrics.
• Hash-based splits are reproducible without storing a list of IDs per set.

Design choices:

• Split key: the entity whose records must stay together, such as customer, device or session.
• Salt: add a project-specific string before hashing, so different projects get independent splits.
• Time-based problems: for forecasting, split by time instead, since future data must never train a model evaluated on the past.

Check the resulting proportions and the distribution of key features in each set, since hashing guarantees stability, not balance.

Run automated checks between data arrival and training, and decide in advance which failures stop the pipeline and which only raise a warning. Bad data produces a bad model without any error, so this gate is one of the most valuable in the pipeline.

What to check:

1. Schema: column names, types and allowed values.
2. Volume: row counts per partition compared with recent history.
3. Completeness: missing-value rates per column.
4. Ranges and distributions: values within valid bounds, and distributions compared with a reference window.
5. Label sanity: class balance and label delay.

Store each check's results with the run, so a later model problem can be traced to the data it was trained on.

Treat feature code like any other production code: small pure functions, tested with hand-written inputs whose correct outputs you know. Feature bugs are dangerous because the model still trains and serves, just with wrong inputs.

What to test:

• Typical values: the main calculation, with a result computed by hand.
• Edge cases: missing values, zero, empty lists, very large values and boundary dates.
• Time logic: that a feature uses only data available at the prediction time, such as orders strictly before the cutoff.
• Categorical handling: unseen categories map to a defined fallback rather than raising an error.
• Consistency: the same function gives the same answer in batch and online code paths, ideally because they share the code.

Habits that help:

1. Keep transformations as functions separate from data loading, so they are easy to test.
2. Use small fixtures in the test file instead of production data.
3. Run the tests in continuous integration for every change to feature code.

Compute the feature for every historical point at which the model would have made a prediction, using only data that existed at that moment. A backfill that uses today's data for past dates leaks the future and makes the model look better than it will be.

1. Define the feature as of a timestamp, so the same code serves backfill, training and live computation.
2. Use event-time data: raw events with timestamps, or snapshots of tables as they were, rather than current state tables that have been overwritten.
3. Process in partitions, such as one day at a time, so failures can be retried and progress tracked.
4. Make each partition idempotent: rerunning a day overwrites that day's output instead of adding duplicates.
5. Validate against live values: for recent dates, compare backfilled values with values the live pipeline produced; they should match.
6. Record the backfill's code version, so training runs know exactly which definition they used.

Watch for: source data whose history was deleted or changed, late-arriving events, and feature definitions that depend on systems which did not exist in the past. Document the earliest reliable date, and train only on data after it.

Define how old each feature may be, monitor its actual age at serving time, and give the system a planned behavior for stale data. A model that silently uses yesterday's features during a fast-moving event can make confidently wrong decisions.

flowchart TD
  A["Prediction request"] --> B["Fetch features with their timestamps"]
  B --> C{"Within freshness limit?"}
  C -->|Yes| D["Predict normally"]
  C -->|No| E{"Critical feature?"}
  E -->|No| F["Use fallback value and flag the prediction"]
  E -->|Yes| G["Use a simpler fallback model or rule"]
  F --> H["Record staleness metric"]
  G --> H

Practices:

1. Freshness targets per feature, such as five minutes for transaction counts and one day for profile features.
2. Store timestamps with values, so serving code can compute each feature's age.
3. Alert on pipeline delays before they cross the limit, not after.
4. Train with realistic staleness: if serving features are typically ten minutes old, build training data with the same delay.
5. Track flagged predictions, so stale periods can be excluded from monitoring and analysis.

They differ in when they are computed, which sets how fresh they can be and how much they cost to run.

Design guidance:

• Choose the slowest computation that is fresh enough; streaming pipelines bring real operational cost.
• Combine them: many models use batch features for history, streaming features for recent behavior and request features for context.
• Keep one definition per feature, so the training data and the serving path compute it the same way.
• Test how the model behaves when each type is missing, since each fails differently.

Delete the person's data from every store you control, stop it from entering future training, and decide, with legal guidance, whether existing models must be retrained. Planning for this upfront is far cheaper than tracing data after a request arrives.

1. Map where personal data lives: raw tables, feature stores, training snapshots, logs, caches and evaluation sets.
2. Delete or anonymize at the source, and propagate deletions to derived datasets.
3. Exclude from future snapshots: keep a list of deleted identifiers that training pipelines filter out.
4. Assess existing models: many policies accept that the model is retrained on the next scheduled cycle; stricter requirements or memorizing models, such as those that can reproduce training text, may need prompt retraining.
5. Record what was done and when, for auditing.

Designing for it:

• Keep training snapshots for a limited time, so old copies do not accumulate.
• Use identifiers that can be mapped back to people only through a protected table, so deleting that mapping breaks the link.
• Prefer features that aggregate many people, which carry less individual information.

Use the least personal data the model needs, protect what remains with access controls and technical safeguards, and document why each sensitive field is used.

Minimize first:

• Drop direct identifiers, such as names, phone numbers and emails, before data reaches training environments.
• Replace identifiers with pseudonymous keys when records must be joined.
• Aggregate or bucket values, such as age bands instead of birth dates, when precision is not needed.

Protect what remains:

1. Access: separate roles for raw personal data and for pseudonymized training data, with access granted per project and reviewed regularly.
2. Environments: keep sensitive data in controlled storage and compute, and block copying to laptops or unmanaged notebooks.
3. Encryption: at rest and in transit, with keys managed centrally.
4. Logging: record who accessed which datasets.
5. Retention: delete snapshots after a defined period.

Check the model too: models can memorize rare records, and features such as location can reveal sensitive traits indirectly. Review high-risk models before release, and test whether outputs can reveal training data.

Represent the pipeline as a graph of steps with explicit inputs and outputs, run by an orchestrator that handles scheduling, retries and dependencies. Every step should produce the same output for the same input and write it to a location named by the run, so retries never corrupt earlier results.

flowchart LR
  A["Extract data snapshot"] --> B["Validate data"]
  B --> C["Build features"]
  C --> D["Train model"]
  D --> E["Evaluate against production model"]
  E --> F{"Passes quality gate?"}
  F -->|Yes| G["Register candidate"]
  F -->|No| H["Stop and notify owners"]

What makes steps safe to rerun:

1. Immutable inputs: steps read snapshots or versioned datasets, not tables that change underneath them.
2. Run-scoped outputs: each step writes to a path including the run ID, never overwriting another run's output.
3. Atomic completion: write to a temporary location, then mark the output complete in one step, so partial results are never read.
4. No hidden side effects: steps such as registration or notifications happen once, at the end, after checks pass.
5. Recorded parameters: every input is logged, so a rerun is truly the same run.

Interview tip: interviewers often probe what happens when a step fails halfway; describing run-scoped outputs and an atomic completion marker answers that directly.

A pipeline step can be skipped when it has already run with exactly the same inputs, reusing the stored output. The orchestrator computes a key from the step's code version, parameters and input artifact identifiers, and looks it up before running.

Benefits:

• Fast iteration: changing only the training step reuses expensive data extraction and feature building.
• Lower cost: shared steps run once across many experiments.

When caching gives wrong results:

• Inputs that are not in the key: a step that reads "the latest data" or calls an external service produces different output with the same key.
• Code changes not captured: if the key includes only the step's own file, a change in a shared helper module is missed.
• Environment changes: a new library version can change results with identical code.
• Nondeterministic steps: caching hides variation that a rerun would expose.

How to keep it safe:

1. Make every input explicit, including dataset snapshot IDs and the container image digest.
2. Disable caching for steps whose purpose is to fetch fresh data.
3. Allow a forced rerun, and record in each run whether outputs came from the cache.

Distributed jobs need all their workers at once, fail if any worker fails and compete with other teams for scarce accelerators. Reliability comes from scheduling them as a unit, recovering from checkpoints and enforcing fair use.

Scheduling:

• All-or-nothing placement: start a job only when every worker it needs is available, so partial jobs do not hold hardware while waiting.
• Quotas and priorities: per-team limits and a priority order, so urgent production retraining can preempt experiments.
• Topology awareness: place workers on machines with fast connections between them, since communication often limits speed.

Recovering from failure:

1. Checkpoint model and optimizer state regularly to shared storage.
2. Restart the whole job from the latest checkpoint when a worker fails, automatically and with a retry limit.
3. Detect hung workers with heartbeats and timeouts, since a stuck worker stalls everyone.

Operating it:

• Track queue time, utilization and failure reasons per team.
• Give each job resource requests that match its real needs, and kill jobs that idle while holding hardware.
• Keep small debug runs on cheaper resources, so expensive clusters are not used for trial and error.

Interruptible capacity is much cheaper but can be reclaimed with little notice. Jobs that checkpoint often, resume automatically and tolerate a changing number of workers can capture most of the savings with little lost work.

1. Checkpoint regularly: save model weights, optimizer state, the learning rate schedule position, random states and the data position.
2. Handle the warning: many platforms give a short notice before reclaiming a machine; use it to write a final checkpoint.
3. Resume automatically: the job looks for the latest complete checkpoint at startup and continues from there.
4. Write checkpoints atomically, so an interrupted save never becomes the latest checkpoint.
5. Mix capacity: keep coordination components on regular machines and put workers on interruptible ones.

Choosing the checkpoint interval: frequent checkpoints waste time writing; rare ones lose more work per interruption. Aim for the expected lost work plus checkpoint time to stay a small share of total runtime.

Not every job suits this: very large synchronized jobs can lose too much time to restarts when interruptions are frequent.

Low utilization usually means the accelerator is waiting for something: input data, the host processor, other workers or storage. Profile one training step to see where time goes, then fix the slowest part.

1. Check the input pipeline: if utilization rises when training on a small in-memory dataset, data loading is the bottleneck.
2. Look at host processor usage: decoding images, parsing records or applying augmentation on too few processes starves the accelerator.
3. Check storage reads: many small files over a network are slow; larger sharded files and local caching help.
4. Check batch size: small batches leave the accelerator underused; increase it if memory and model quality allow.
5. Check communication: in distributed training, workers may spend much of each step synchronizing gradients.
6. Check synchronization points: frequent logging that copies values back to the host stalls the pipeline.

Interview tip: saying "first I check whether the accelerator is waiting for data" before suggesting bigger hardware shows practical experience.

Typical fixes: more parallel data-loading workers, prefetching the next batches while the current one trains, preprocessing data once ahead of time, larger file shards, and overlapping communication with computation.

Measure throughput on a short trial run, then scale it to the full job, adding overhead for evaluation, checkpointing and failures. Estimates prevent surprises and help choose between hardware options.

1. Measure throughput: samples per second per accelerator, from a few hundred steps after warm-up.
2. Total work: dataset size times the number of passes over the data.
3. Time: total samples divided by the combined throughput of all accelerators. Scaling across machines is rarely perfect, so measure with the real worker count.
4. Overhead: add 10% to 20% for evaluation, checkpoints and restarts.
5. Cost: accelerator count times hours times price per accelerator-hour, plus storage and data transfer.

Using the estimate:

• Compare options, such as more accelerators for a shorter time, or interruptible capacity.
• Set a budget alert at a little above the estimate, to catch runaway jobs.
• Compare actual with estimated after each run, which improves later estimates.