Data Engineer Top 100 Interview Questions

A narrow transformation can compute an output partition from a limited corresponding set of input partitions. A wide transformation usually requires data redistribution across partitions.

When to use which:

• Use local filters, projections, and maps early when they preserve correctness and reduce the records or fields that later operators must exchange.
• Use wide operations when the business calculation needs global grouping, redistribution, ordering, or a join that cannot be satisfied locally.
• Reduce unnecessary exchanges by aligning compatible data layouts and combining work where the engine can optimize it, while verifying the physical plan.

A wide operation is not inherently a mistake. Its cost depends on exchanged bytes, partition balance, serialization, and available memory, not just the number of transformation calls.

Join strategy depends on input sizes, key distribution, memory, ordering, and engine support. The smallest input after filtering is often more relevant than the original table size.

When to use which:

• Use a broadcast join when one filtered input is small enough for every relevant worker, accounting for its in-memory representation and concurrent workloads.
• Use a repartitioned hash join when neither side should be broadcast and balanced partitions allow efficient local hash matching.
• Use sort-merge when large inputs and available ordering make it suitable, particularly where hash-building memory would be problematic.

Verify estimates, skew, join type, and runtime adaptation. Broadcasting a table that unexpectedly grows can turn a previously fast query into widespread memory failures.

I would compare work across tasks to determine whether a small number of keys or partitions dominate execution. Adding workers does not necessarily split one oversized key group.

1. Inspect task durations, input sizes, shuffle bytes, spill volume, and key frequencies to distinguish skew from slow infrastructure or broad resource shortage.
2. Identify the operation and key responsible, including placeholder or missing-key values that may concentrate unrelated records into one partition.
3. Apply a suitable correction such as early aggregation, filtering irrelevant hot keys, supported skew handling, broadcasting a small side, or carefully designed key salting.
4. Verify join completeness and aggregate equivalence, then measure tail-task duration, memory, and total processing cost on representative data.

flowchart TD
A["Compare partition and task metrics"]
B{"A few tasks dominate?"}
C["Inspect hot keys and operation"]
D["Investigate general capacity"]
E["Apply targeted redistribution"]
F["Verify results and task balance"]
A --> B
B -->|yes| C
B -->|no| D
C --> E
E --> F

Trade-offs:

• Salting can distribute a hot key but may require replicating matching records and a later aggregation step.
• More partitions can reduce average size while leaving an unsplit hot key largely unchanged.

Caching primarily avoids repeated computation. Checkpointing records a recoverable processing state or materialized boundary, depending on the engine and operation.

When to use which:

• Cache an expensive intermediate result when several later operations reuse it and the saved work exceeds the memory, serialization, and materialization costs.
• Use a durable checkpoint when recovery needs a consistent saved position or state, or when supported batch execution benefits from truncating long dependency chains.
• Remove caches that are no longer useful and manage checkpoint retention according to recovery, compatibility, and restart requirements.

An in-memory cache is not a durable backup. A local checkpoint may also provide weaker recovery than a reliable shared checkpoint, so verify the exact guarantee rather than relying on the name.

These timestamps refer to different stages in a record’s journey. Choosing the wrong one can shift events into the wrong reporting window or hide delivery delays.

When to use which:

• Use event time when reports must reflect when activity actually happened, while validating producer clocks and handling records that arrive out of order.
• Use ingestion time to measure arrival patterns and source delays, retaining it separately from the event timestamp rather than overwriting the original.
• Use processing time to measure runtime behavior or implement explicitly processing-time-based operations whose meaning does not require business event chronology.

Track differences between clocks to locate delays. A pipeline can process new arrivals quickly while still serving stale business events that were delayed upstream.

Windows group an unbounded event stream into manageable analytical units. The choice determines whether events share fixed periods, overlapping periods, or activity-based sessions.

When to use which:

• Use tumbling windows for nonoverlapping summaries such as hourly totals when each event should contribute to one period in that series.
• Use sliding windows for frequently refreshed trends, recognizing that overlapping results reuse events and should not simply be summed together.
• Use session windows when an inactivity gap defines meaningful activity, and plan for late events that can extend or merge previously identified sessions.

Specify event or processing time, interval endpoints, allowed lateness, and result-update behavior. Window width alone does not define when a result becomes final.

A watermark represents an engine’s progress estimate for event time and helps decide when to finalize results or remove old state. It is not proof that no older event can ever arrive.

• Progress rule: a common strategy subtracts an allowed delay from the maximum observed event timestamp, although actual propagation and combination rules are engine-specific.
• State management: operators can retire state once the relevant window and lateness conditions are satisfied, preventing indefinite growth.
• Late records: events beyond the supported lateness boundary may be dropped, side-output, or handled through correction logic according to the configured operator.
• Multiple inputs: slow or idle inputs can hold progress back, so understand watermark combination, idle-source treatment, and the resulting completeness trade-off.

Configure the delay using observed arrival behavior and business tolerance, then monitor excluded or corrected late data.

These guarantees describe how failures and retries affect processing outcomes. State the source, processing, and destination boundaries before claiming an end-to-end guarantee.

When to use which:

• Use at-most-once behavior only when occasional loss is acceptable under the stated failure model and the lower recovery cost has real value.
• Use at-least-once delivery with idempotent sinks when replay is expected and stable identities or deterministic replacements can prevent duplicate effects.
• Use transactional or coordinated exactly-once mechanisms when correctness requires one committed effect, checking that every destination participates in the guarantee.

Exactly-once effects do not mean code executes only once. An external email or API call can still repeat if it is outside the coordinated commit boundary.

Partitions divide a stream into independently ordered logs that can be processed in parallel. In a partition-assigned consumer group, each partition is assigned to one active consumer at a time.

• Parallelism: more partitions can provide more independent work, but adding consumers beyond the available assigned partitions does not increase useful parallelism for that group.
• Ordering: ordering is generally defined within a partition, so events requiring per-entity order should use a stable compatible partitioning rule.
• Independent consumption: separate consumer groups maintain their own progress and can process the same stream for different applications.
• Reassignment: failures and membership changes can move partitions between consumers, requiring correct checkpointing and duplicate-safe recovery.

A partition position is meaningful within that partition, not as a universal event timestamp. Changing partition counts or routing rules can also affect assumptions about per-key ordering.

Recovery should restore a consistent combination of input position, operator state, and committed output. Restoring only the input offset can lose or duplicate effects when the other components disagree.

1. Select the latest complete checkpoint that is compatible with the deployed processing logic and available source history.
2. Restore keyed state, timers, and source positions according to the engine’s recovery protocol, rather than combining fragments from different checkpoints.
3. Resolve or replay uncommitted work through a transactional or idempotent destination so uncertain writes cannot create duplicate logical effects.
4. Resume consumption, measure catch-up progress, and reconcile outputs or control totals before declaring full recovery.

Checkpoints require durable storage, retention, and tested restore procedures. An incompatible state change or expired source log may require a controlled rebuild rather than an ordinary restart.

flowchart TD
A["Detect processor failure"]
B["Load complete checkpoint"]
C{"State and history compatible?"}
D["Restore state and positions"]
E["Replay through safe output commit"]
F["Reconcile and resume service"]
G["Run controlled rebuild"]
A --> B
B --> C
C -->|yes| D
C -->|no| G
D --> E
E --> F

I would identify where sustained processing capacity falls below arrival rate. Lag is a symptom; backpressure is a mechanism that slows upstream work when downstream capacity is insufficient.

1. Compare arrival and completion rates over time, along with backlog age and per-partition lag, to separate a temporary burst from continuing overload.
2. Inspect source reads, transformations, state access, network transfer, and sink commits for the slow stage, checking skew and external throttling.
3. Increase effective capacity or reduce unnecessary work at that stage, while preserving ordering, quality checks, and duplicate-safe output behavior.
4. Verify that spare throughput can drain the backlog before source retention expires, and monitor recovery rather than only current CPU utilization.

flowchart TD
A["Measure arrival rate and backlog age"]
B{"Sustained overload?"}
C["Locate constrained stage"]
D["Monitor burst recovery"]
E["Improve effective stage capacity"]
F["Verify backlog drains safely"]
A --> B
B -->|yes| C
B -->|no| D
C --> E
E --> F

Trade-offs:

• More consumers help only when partitions, sink capacity, and shared resources allow additional useful parallelism.
• Larger buffers absorb bursts but do not fix a sustained throughput deficit and can increase failure recovery time.

A streaming deduplicator tracks stable event identities for a defined retention horizon. Bounded state means the guarantee is also bounded unless another durable system handles older replays.

• Identity: use a producer-assigned event identifier or another proven business key; identical payloads can still represent two legitimate events.
• State: store previously accepted identifiers and the metadata needed to expire them, partitioned consistently so duplicates reach the same logical state.
• Retention: align expiration with expected lateness, outage recovery, and replay requirements; processing-time expiry and event-time expiry have different behavior.
• Older replays: route out-of-horizon records to a durable uniqueness check, controlled correction path, or explicit policy rather than claiming unlimited duplicate prevention.

Monitor state size and duplicate rates. A growing identifier set without cleanup can exhaust resources even when ordinary per-event processing is fast.

A stream-to-stream join may need to retain a record until its counterpart arrives. Without bounds on matching time and progress, the required state can grow indefinitely.

• Join condition: combine the business key with an appropriate event-time relationship, such as a response occurring within a bounded interval after a request.
• State cleanup: use the engine’s watermark and time-constraint rules to determine when no permitted future match remains for a stored record.
• Outer results: unmatched outputs may be delayed until the matching horizon is considered complete; output timing is part of the business behavior.
• Correctness: account for duplicate matches, skew, late counterparts, and clock errors, then test how the chosen engine handles records outside the supported window.

A stream joined to a finite reference table has a different state problem, but still needs an explicit policy for refreshing and historically interpreting that reference data.

I would apply the agreed correction policy using retained source evidence and a versioned or updateable result. Silently ignoring the event can leave business totals permanently wrong.

1. Determine whether the record is valid, new, and within the business correction horizon, distinguishing it from a duplicate or malformed timestamp.
2. Identify the affected window and recompute it from retained authoritative events, or apply a provably correct incremental adjustment.
3. Publish a replacement, retraction and addition, or a new result version according to downstream capabilities, preserving a clear audit trail.
4. Notify or refresh dependent reports as required, and measure late-event frequency to reassess the original lateness and finalization policy.

flowchart TD
A["Receive beyond horizon event"]
B{"Valid correction required?"}
C["Identify affected aggregate"]
D["Record rejected or duplicate input"]
E["Recompute or adjust safely"]
F["Publish correction and audit trail"]
A --> B
B -->|yes| C
B -->|no| D
C --> E
E --> F

Trade-offs:

• Longer open windows reduce later corrections but retain more state and delay final results.
• Append-only consumers may need explicit result versions because an additional row alone can double-count a correction.

A directed acyclic graph, or DAG, represents tasks and their dependency relationships without circular paths. It describes when work is eligible to run, not necessarily how its internal computation is distributed.

• Dependencies: connect tasks through real data or control requirements, such as validated ingestion preceding a transformation that consumes the published dataset.
• Execution identity: associate each run with a defined input interval or version so retries and historical runs do not accidentally read the latest changing data.
• Operational behavior: define timeouts, retries, concurrency limits, ownership, and acceptable outcomes for skipped or failed upstream work.
• Data readiness: distinguish a schedule firing from the required input actually being complete; use explicit availability or completion conditions where necessary.

A scheduler cannot make an unsafe task reliable. Tasks still need predictable side effects, durable outputs, and correct publication boundaries.

I would retry recoverable failures within a bounded budget and make uncertain outcomes safe to inspect or repeat. Retrying every failure indefinitely can worsen an incident.

1. Classify failures into transient conditions, permanent input or authorization errors, and ambiguous completion where the side effect may already have occurred.
2. Set task and dependency timeouts based on expected runtime and the end-to-end deadline, including cancellation or cleanup behavior for abandoned work.
3. Use bounded exponential backoff with jitter for transient failures, honoring service guidance and limiting concurrent retry pressure.
4. Before repeating a write, check its durable outcome or use an idempotent operation; escalate exhausted attempts with actionable context and ownership.

flowchart TD
A["Task fails or times out"]
B{"Outcome already committed?"}
C["Confirm existing success"]
D{"Failure recoverable?"}
E["Back off within retry budget"]
F["Stop and alert owner"]
G["Run safe retry"]
A --> B
B -->|yes| C
B -->|no| D
D -->|yes| E
D -->|no| F
E --> G

Trade-offs:

• Longer retry budgets can survive brief outages but may consume the freshness budget and hide persistent failures.
• Short timeouts detect problems sooner but can create unnecessary retries when legitimate workloads occasionally run longer.

I would run a bounded, reproducible historical computation with controlled publication and resource isolation. A backfill should not change meaning merely because it runs on a different day.

1. Define the affected interval, source versions, transformation version, dependencies, and expected reconciliation checks before scheduling historical work.
2. Process explicit partitions with separate concurrency or resource limits, avoiding uncontrolled competition with current ingestion and serving workloads.
3. Write to staged outputs and validate completeness, uniqueness, totals, and interactions with any current updates affecting the same interval.
4. Publish through coordinated replacement or merging, refresh dependent datasets, and retain the run manifest and rollback information.

flowchart TD
A["Define historical scope and versions"]
B["Allocate bounded resources"]
C["Build staged partitions"]
D{"Validation passes?"}
E["Publish coordinated replacement"]
F["Refresh dependent outputs"]
G["Investigate without publishing"]
A --> B
B --> C
C --> D
D -->|yes| E
D -->|no| G
E --> F

Trade-offs:

• Parallel backfills finish sooner but can overload the source or delay current data unless capacity is explicitly controlled.
• Reusing current code is convenient only when its definitions and source assumptions are correct for the historical period.

Quality should be defined against how the dataset will be used. A syntactically valid record can still be incomplete, stale, duplicated, or wrong about the real world.

• Completeness and uniqueness: measure required coverage and missing fields, and verify that identifiers are unique at the declared business grain.
• Validity and consistency: check types, ranges, allowed values, referential relationships, and agreement of units or definitions across related datasets.
• Timeliness: measure delivery against the required business interval, including source delays rather than only the latest successful job timestamp.
• Accuracy: compare important values with trustworthy reference evidence or reconciliation controls; schema checks alone cannot establish real-world correctness.

Set severity and response policies for each check, including whether failures block publication or create a visible warning. Monitor quality by meaningful segments so overall averages do not conceal a broken source or region.

A useful test strategy covers transformation logic, component interactions, data assumptions, and operational recovery. Tests should target plausible failures rather than simply repeat implementation details.

• Unit tests: exercise deterministic transformations with normal, missing, duplicate, and boundary inputs, including business rules for time, units, and rounding.
• Integration tests: verify source parsing, destination writes, permissions, schema handling, and transaction or commit behavior using representative dependencies.
• Data and contract checks: assert grain uniqueness, required fields, allowed values, relationships, and reconciled control totals at publication boundaries.
• Recovery tests: simulate retries, partial writes, late changes, and restart from saved progress to confirm safe reruns and consistent outputs.

Use small fixtures for precise cases and representative larger samples for performance risks. Keep expected outputs independent enough to detect a mistake in the transformation being tested.

A successful job can still publish stale, incomplete, or incorrect data. Monitoring should connect technical execution to the service consumers expect from the dataset.

• Freshness and completeness: track the latest expected business interval, source arrival lag, missing partitions, accepted and rejected counts, and reconciliation status.
• Quality and meaning: monitor key violations, unusual distributions, schema changes, and important business totals, with thresholds suited to known seasonal behavior.
• Performance and cost: measure duration, throughput, tail-task time, retries, resource use, and cost per useful unit of delivered data.
• Service ownership: define measurable objectives, alert routing, runbooks, and escalation based on consumer impact rather than generating an alert for every harmless fluctuation.

An availability target, a freshness target, and a correctness target are different promises. Report them separately so one healthy metric cannot mask another failure.