AI Engineer Top 100 Interview Questions

Both are problems with derivative magnitudes during training. Very small gradients can prevent useful learning, while very large gradients can produce unstable parameter updates.

When to use which:

• Investigate vanishing gradients when parameters receive consistently tiny updates, examining saturating activations, initialization, and the depth of dependency paths.
• Investigate exploding gradients when norms spike or numerical values become invalid, checking inputs, loss calculations, learning rates, and gradient clipping.
• Use residual connections, suitable normalization, and well-chosen initialization where appropriate, then verify that the intervention actually improves training and validation behavior.

Clipping limits excessively large gradients; it does not restore a useful signal that has already vanished.

Both can increase the number of examples contributing to an update. Gradient accumulation does this over several smaller forward and backward passes instead of one larger pass.

When to use which:

• Use a larger direct batch when available memory and hardware utilization make it efficient, while checking how the changed update frequency affects optimization.
• Use accumulation when activation memory limits batch size, scaling the loss or gradients consistently and updating at the intended accumulation boundary.
• Verify behavior with batch-dependent layers, random operations, schedules, and numerical precision, because accumulated microbatches are not always identical to one large batch.

Track effective batch size separately from microbatch size so that experiments and learning-rate decisions remain interpretable.

I would begin by reusing the pretrained representation and changing only what the task requires. Limited data makes careful validation and overfitting control especially important.

1. Confirm that the available model representation and permitted data are relevant to the target task, including input format and label definitions.
2. Establish a frozen-representation baseline with a small prediction head. Compare its errors with the task requirements before updating more parameters.
3. Unfreeze selected components or use a compact adaptation method if the baseline misses important domain behavior, with conservative training settings.
4. Evaluate on independent examples and relevant groups, checking generalization and regressions. Stop expanding adaptation when complexity outpaces reliable improvement.

flowchart TD
  A["Inspect task and data"]
  B["Train small task head"]
  C{"Quality sufficient?"}
  D["Keep frozen representation"]
  E["Adapt selected parameters"]
  F["Check held-out behavior"]
  A --> B
  B --> C
  C -->|yes| D
  C -->|no| E
  E --> F

Trade-offs:

• Freezing parameters reduces compute and overfitting risk, but can limit adaptation when the new domain differs substantially from pretraining.
• Updating more parameters increases flexibility while demanding better data, stronger validation, and a larger training budget.

Attention scores the relationship between a query and candidate keys, then uses normalized scores to combine their values. The result is a context-dependent representation.

1. Project input representations into queries, keys, and values using learned transformations; these projections determine what compatibility and content mean for each attention head.
2. Calculate query–key dot products and scale by the square root of key dimension, helping control the magnitude of the resulting scores.
3. Apply any required mask before normalization, preventing access to forbidden positions such as future tokens in a causal prediction task.
4. Normalize permitted scores and use them as weights for the value vectors, then combine head outputs based on the architecture.

flowchart TD
  A["Project representations"]
  B["Calculate scaled scores"]
  C{"Restricted positions?"}
  D["Mask restricted scores"]
  E["Normalize scores"]
  F["Combine value vectors"]
  A --> B
  B --> C
  C -->|yes| D
  D --> E
  C -->|no| E
  E --> F

• The formula is attention = softmax(Q * transpose(K) / sqrt(key_dimension)) * V, with masking added before normalization when required.
• Attention weights describe this computation and should not automatically be interpreted as a faithful causal explanation of the whole model.

Attention must distinguish not only which elements are present but also their ordering. Positional information lets the architecture represent relationships that depend on where elements occur.

• Order sensitivity: rearranging the same words can change meaning, so a representation based only on token identity is insufficient for many language tasks.
• Integration: architectures can incorporate position through added representations or position-dependent transformations of the attention calculation.
• Generalization: the method and training distribution influence behavior at longer lengths; accepting a longer input does not prove it will be used reliably.
• Evaluation: test order-sensitive cases, long-distance relationships, and positions near context boundaries rather than assuming a larger context limit solves these challenges.

Position handling is part of the model architecture and must remain compatible with the representation learned during training.

The architectures differ in how they represent an input and produce an output sequence. Their training objectives and attention constraints shape their usual strengths.

When to use which:

• Use an encoder-oriented representation when the task primarily requires understanding an available input, such as extraction, matching, or classification.
• Use a decoder-oriented model when generating a continuation is the central operation, accounting for sequential generation and its serving costs.
• Use an encoder–decoder design when a distinct input sequence conditions an output sequence and the available training setup supports that structure.

These are architectural tendencies rather than absolute task restrictions, so compare available approaches on the actual application workload.

Tokenization maps input into the units the model processes. The correct tokenizer and formatting are part of the model interface, not interchangeable preprocessing details.

• Compatibility: token identifiers, special markers, and conversation formatting must match the representation expected by the deployed model.
• Budgeting: token counts affect input limits, output capacity, latency, and often cost, while word or character counts provide only rough estimates.
• Coverage: different languages, identifiers, and unusual symbols can produce very different token lengths, changing effective context capacity across users.
• Data handling: chunking, truncation, padding, and masking should respect both token limits and meaningful content boundaries so that important evidence is not silently removed.

Count tokens using the actual serving representation when accuracy matters, and test representative languages and document types.

These stages improve different aspects of model behavior. Pretraining builds broad representations, supervised fine-tuning teaches desired responses, and preference alignment favors responses judged more suitable.

When to use which:

• Use pretrained capabilities as the starting point when they already cover much of the task, reducing the need for expensive broad training.
• Use supervised examples when the desired output behavior can be demonstrated reliably, including difficult cases and appropriate handling of missing information.
• Use preference signals when ranking alternative responses captures qualities that a single target answer does not express adequately, and evaluate unintended regressions.

These stages change parameters; supplying new context during a request changes the input rather than updating the model’s weights.

These settings control how a next token is selected from model scores. They influence variability and diversity, but do not establish factual correctness.

When to use which:

• Use supported low-variability settings when consistency matters, while still validating structure and evidence because a stable response can be consistently wrong.
• Use broader sampling when diverse proposals are useful and the application has a way to filter, evaluate, or review the generated candidates.
• Change settings deliberately and compare held-out results, checking the serving interface because supported controls and deterministic behavior depend on implementation.

Neither low temperature nor a fixed seed guarantees identical behavior across changing models, hardware, or serving configurations.

I would choose a strategy based on whether the task needs selected facts or coverage of the entire document. Simply cutting the text can remove decisive evidence.

1. Reserve space for instructions, conversation state, tool results, and the response using the actual token accounting of the selected model.
2. For a focused question, retrieve relevant sections and retain titles, source identifiers, and exceptions needed to interpret their contents correctly.
3. For a whole-document task, process meaningful sections and combine structured intermediate results while tracking coverage and unresolved inconsistencies.
4. Evaluate information loss at boundaries and during summarization, and explicitly handle cases where required evidence cannot fit or cannot be recovered reliably.

flowchart TD
  A["Determine task scope"]
  B{"Whole document required?"}
  C["Process sections"]
  D["Retrieve relevant passages"]
  E["Assemble within budget"]
  F["Verify evidence coverage"]
  A --> B
  B -->|yes| C
  B -->|no| D
  C --> E
  D --> E
  E --> F

Trade-offs:

• Selective retrieval saves tokens but can miss relevant passages, while hierarchical processing improves coverage at the cost of extra calls.
• Summaries reduce length but can remove qualifiers, so retain authoritative references and verify important claims against their original context.

An effective prompt defines the task, relevant context, expected output, and handling of uncertainty. A testable prompt produces behavior that can be evaluated against explicit criteria.

• Clarity: state the required decision or transformation directly, avoiding conflicting requirements and unnecessary background that obscures the actual task.
• Evidence boundaries: separate instructions from untrusted input and specify what the model should do when required information is absent or contradictory.
• Demonstrations: add representative examples only when they clarify behavior, including difficult cases rather than showing only easy successful outputs.
• Evaluation: version changes and compare them on held-out cases using a stable rubric, because a convincing answer to one prompt is not evidence of broad reliability.

I would define a schema, use supported output constraints, and validate the result in application code. Correct syntax alone does not establish valid meaning or supporting evidence.

1. Define required fields, types, allowed values, and missing-information behavior. Keep the schema consistent with the downstream business rules.
2. Request constrained output when supported and explicitly handle truncated responses, refusals, unavailable results, and other incomplete generation states.
3. Validate structure and then check semantic constraints, such as allowable dates or whether an extracted value actually appears in the input.
4. Use bounded correction attempts or a review path when validation fails. Store accepted data only after authorization and all required checks succeed.

flowchart TD
  A["Generate structured result"]
  B{"Schema valid?"}
  C["Bounded correction"]
  D{"Evidence and rules valid?"}
  E["Accept result"]
  F["Escalate or reject"]
  A --> B
  B -->|no| C
  C --> A
  B -->|yes| D
  D -->|yes| E
  D -->|no| F

Trade-offs:

• Stricter constraints improve machine readability but cannot prevent invented values that still satisfy the schema.
• Repeated correction attempts can increase cost and latency, so define a retry limit and a clear fallback.

A hallucination is content that is false, unsupported, or inconsistent with the evidence required for the task. Fluent generation does not guarantee factual accuracy.

• Missing information: the model may produce a plausible continuation when the prompt lacks the facts needed for a supported answer.
• Ambiguity: unclear questions, conflicting sources, or poorly specified tasks can lead to an answer that sounds coherent but resolves the uncertainty incorrectly.
• Retrieval failures: incomplete, outdated, or irrelevant context can cause an apparently grounded system to generate inaccurate claims or misleading citations.
• Mitigation: use trustworthy evidence, appropriate tools, explicit abstention behavior, and claim-level evaluation. Reduced sampling variability alone does not remove unsupported content.

Evaluate the system’s willingness to decline unsupported answers as well as its performance on questions with sufficient evidence.

The choice depends on whether the main problem is task instruction, access to evidence, or persistent model behavior. These approaches can also be combined.

When to use which:

• Use prompting first when the base capability is sufficient and clearer instructions or a few examples can reliably express the required behavior.
• Use retrieval when answers need current, private, or traceable facts, with careful attention to ingestion, permission checks, and retrieval quality.
• Use fine-tuning when representative examples can improve consistent task behavior, then evaluate regressions and whether the adaptation justifies its maintenance cost.

Fine-tuning does not automatically provide fresh facts, and retrieval does not automatically teach a model every required output behavior.

I would define the target behavior before collecting examples and prioritize correctness and coverage over raw volume. The evaluation set must remain independent of dataset improvement.

1. Write labeling and response guidelines, then review representative examples for consistent interpretation of instructions, evidence, and acceptable abstention.
2. Remove duplicates, protect sensitive fields, and separate related sources or entities across partitions so that evaluation does not reward memorization.
3. Check input formatting, truncation, target masking, and class or task balance. Ensure the loss is applied to the intended portions of each example.
4. Compare the adapted model with its starting baseline on difficult cases and general capabilities, tracking quality changes and possible regressions.

flowchart TD
  A["Define desired behavior"]
  B["Review training examples"]
  C{"Labels consistent?"}
  D["Revise guidelines"]
  E["Partition and format"]
  F["Train and evaluate"]
  A --> B
  B --> C
  C -->|no| D
  D --> B
  C -->|yes| E
  E --> F

Trade-offs:

• More examples improve coverage only when their quality and relevance are adequate; repeated low-quality patterns can reinforce the wrong behavior.
• Narrow specialization can improve one task while degrading broader behavior, so evaluation should include both the target task and essential retained capabilities.

Low-rank adaptation learns a compact update while selected base weights remain frozen. Updating all weights offers greater flexibility but usually requires more trainable state and memory.

When to use which:

• Use low-rank adaptation when compute or memory is constrained and a compact learned update can capture the task-specific change you need.
• Use full updating when the task requires broader adaptation and sufficient data, training resources, and regression evaluation are available.
• Evaluate target components, update rank, and deployment compatibility rather than assuming parameter reduction translates directly into equal quality or lower serving latency.

Quantizing the frozen base can reduce memory further, but activations and adapter training still consume memory and computation.

Quantization represents values with fewer numerical levels or fewer bits. It can reduce model memory requirements, but its quality and speed effects depend on the workload and runtime.

• Scope: weights, activations, and cached intermediate values can be quantized separately, with different impacts on storage and computation.
• Accuracy: rounding, clipping, and limited ranges can affect sensitive calculations, so evaluate important tasks and edge cases after conversion.
• Performance: smaller representations may reduce bandwidth demands, but conversion overhead or unsupported arithmetic can prevent the expected speed improvement.
• Validation: benchmark the deployed hardware, request lengths, and concurrency rather than relying only on theoretical bit savings or model-file size.

Total serving memory also includes temporary workspaces, activations, and caches, so weight compression alone does not determine capacity.

These techniques change different parts of the system. Distillation transfers behavior, pruning removes components, and quantization reduces the precision used to represent values.

When to use which:

• Use distillation when a smaller model can learn the required task behavior from permitted teacher outputs or representations, with independent quality checks.
• Use pruning when the resulting reduced structure is supported efficiently by the serving implementation, since isolated removed weights may not improve latency.
• Use quantization when memory or bandwidth is a bottleneck and the target hardware can process the reduced representation without unacceptable quality loss.

The techniques can be combined, but each combination needs evaluation because theoretical compression ratios do not directly predict end-to-end speed.

A key–value cache stores attention representations for previously processed tokens so they can be reused during later decoding steps. It trades memory for reduced repeated computation.

• Reuse: generating a new token need not recompute the keys and values already produced for the earlier context.
• Growth: cache requirements depend on sequence length, concurrency, layer count, numerical precision, and the architecture’s arrangement of attention heads.
• Limits: caching does not eliminate attention over previous positions, nor does it store a complete answer that can always be reused for another request.
• Management: paging, offloading, shared-prefix handling, or reduced precision can change capacity and latency, so measure the actual serving workload and isolation boundaries.

Long simultaneous conversations can exhaust cache capacity even when the model’s weights comfortably fit in device memory.

Multiply the parameter count by the number of bytes used per parameter. This estimates raw weight storage, not the total memory needed to train or serve the model.

• Formula: weight_bytes = parameter_count * bits_per_parameter / 8, assuming a uniform representation across the counted parameters.
• Units: decimal gigabytes use one billion bytes, while binary gibibytes use 2 ** 30 bytes; specify which convention you report.
• Overhead: quantization scales, metadata, alignment, temporary buffers, activations, and attention caches add memory beyond the simple weight estimate.
• Training: gradients and optimizer state can increase memory substantially, so do not use raw weight size as a training-capacity estimate.