Data Scientist Top 100 Interview Questions

When a measurement is extreme partly because of luck, the next measurement tends to be closer to the average, even if nothing changed. Any intervention aimed at the most extreme cases will appear to work, because they would have moved toward the average anyway.

Where it shows up:

• Coaching the worst-performing sales teams, then celebrating their improvement.
• Targeting customers whose spending dropped sharply, then measuring a rebound.
• Investigating the slowest week of the year, which is often followed by a normal week.
• Rewarding the best performers, who then seem to "slip" the next period.

How to measure the real effect:

1. Keep a comparable control group of extreme cases that receive no intervention, ideally by random assignment.
2. Compare the change in the treated group with the change in the control group, not with its own past.
3. Use longer baselines, which are less driven by a single lucky or unlucky period.

Interview tip: when a case study says "we targeted the lowest performers and they improved", asking "compared with which control group?" is exactly the response interviewers look for.

Look at the data visually first, then decide whether normality actually matters for the method you plan to use.

How to check:

• Histogram: look for skew, long tails, multiple peaks or a hard boundary at zero.
• Quantile plot: plot sample quantiles against the quantiles of a normal distribution; points on a straight line indicate normality, and curving ends reveal heavy tails or skew.
• Summary statistics: a mean far from the median, or high skewness, signals asymmetry.
• Formal tests: with large samples they reject normality for trivial deviations, so treat them as a minor input.

What to do if it is not normal:

1. Check whether it matters: tests on means with large samples rely on the normality of the mean, not of individual values.
2. Transform skewed positive data, often with a logarithm, and analyze on that scale.
3. Use methods that do not assume normality, such as rank-based tests or bootstrap intervals.
4. Investigate multiple peaks, which often mean two populations that should be analyzed separately.

Bootstrapping estimates the uncertainty of a statistic by resampling the observed data with replacement many times and recalculating the statistic each time. The spread of those recalculated values approximates the statistic's sampling distribution, without needing a formula.

1. Draw a sample of the same size as the data, with replacement.
2. Compute the statistic, such as a median or a ratio.
3. Repeat thousands of times.
4. Take the 2.5th and 97.5th percentiles of the results as a 95% interval.

When it helps:

• Statistics without simple formulas, such as medians, percentiles and ratios of sums.
• Skewed data where normal approximations are doubtful.

Cautions:

• Resample the independent unit: if users have many sessions, resample users, not sessions.
• Very small samples give unreliable intervals, because the data may not represent the tails.
• Fix the random seed so the result can be reproduced.

A permutation test asks how unusual the observed difference between groups is compared with differences produced by randomly reshuffling the group labels. If the treatment had no effect, the labels would be arbitrary, so the reshuffled differences show what chance alone produces.

1. Compute the observed difference, such as the difference in means.
2. Pool all observations and reassign them to groups of the original sizes, many times or in every possible way.
3. Compute the difference for each reassignment.
4. The p-value is the share of reassignments with a difference at least as extreme as the observed one.

When it is useful:

• Small samples, where normal approximations are questionable.
• Unusual statistics, such as a difference in medians or in percentiles.
• Explaining significance intuitively to non-statisticians.

Limits: it assumes the observations are exchangeable under the null hypothesis, so it is not valid when groups differ in other ways, and exhaustive reassignment is only possible for small groups.

Frequentist inference treats the true effect as fixed and asks how often a procedure would give misleading results over repeated samples. Prior-based inference treats the effect as uncertain, starts from a prior distribution and updates it with the data into a posterior distribution, which directly states how probable each effect size is.

Practical view:

• With plenty of data and weak priors, both approaches usually lead to the same decision.
• Posterior statements, such as "93% chance the new page converts better", are easier for stakeholders to understand.
• Priors help with small samples, by pulling noisy estimates toward realistic values based on past experiments.
• Either approach fails if the experiment design or the data is flawed.

Write the whole plan before any user sees the change: the hypothesis, the metrics, the sample size, the analysis and the decision rule. Deciding these after seeing the data turns an experiment into a search for a favorable story.

flowchart TD
  A["Hypothesis: change X will move metric Y"] --> B["Choose primary and guardrail metrics"]
  B --> C["Pick randomization unit and audience"]
  C --> D["Calculate sample size and duration"]
  D --> E["Run checks: assignment balance and logging"]
  E --> F["Run for the planned duration"]
  F --> G["Analyze as planned"]
  G --> H["Decide: ship, iterate or stop"]

Key elements of the plan:

• Hypothesis: what changes, for whom, and why you expect the metric to move.
• Metrics: one primary metric for the decision, plus guardrails that must not get worse.
• Sample size and duration: based on the smallest effect worth detecting, rounded up to full weeks.
• Decision rule: for example, ship if the primary metric improves significantly and no guardrail drops by more than an agreed margin.

Run an A/A test or a short ramp first when the experiment setup is new, to confirm that assignment and logging work.

Each metric has a different job in the decision. Agree on them in advance, so nobody can pick the most flattering metric after the results arrive.

What makes a good primary metric:

• It moves within the test's duration and has low enough variance to detect realistic effects.
• It reflects real value, not just activity; more clicks can mean confusion.
• It is defined per randomization unit, such as per user, so the statistics are valid.

Guardrail practice:

• Set a tolerance for each guardrail, such as "no more than 1% worse".
• Include data-quality guardrails, such as the share of users assigned to each group, which catch broken experiments.

Calculate the sample size needed to detect the smallest effect worth acting on, divide by the eligible traffic per day, then round up to whole weeks. Never stop early just because the result looks good.

Sample size per group for a conversion rate:

n = (z_alpha + z_power)² × (p1(1 − p1) + p2(1 − p2)) / (p2 − p1)²

• p1: the baseline rate; p2: the rate you want to be able to detect.
• z_alpha: 1.96 for a two-sided test at the 5% level.
• z_power: 0.84 for 80% power.

Rules for the duration:

• Full weeks: behavior differs by weekday, so cover at least one complete weekly cycle, usually two.
• Novelty: if the change is visible and new, run long enough for the initial curiosity to fade.
• Eligible traffic only: count only users who actually reach the changed experience.
• Seasonal events: avoid running a test only during a sale or holiday unless that is the target situation.

The minimum detectable effect is the smallest true change a test can reliably detect with its sample size, significance level and power. It should be set by the business: the smallest change that would justify shipping, maintaining and supporting the feature.

How it relates to sample size:

• The required sample grows with the inverse square of the effect: halving the minimum detectable effect needs about four times as many users.
• Noisy metrics, such as revenue per user, have larger minimum detectable effects than conversion rates for the same traffic.

How to choose it:

1. Estimate the value of a change of each size, such as the revenue from a 1% lift.
2. Compare it with the cost of building and maintaining the feature.
3. Check what is realistic: past experiments on similar changes show typical effect sizes.
4. If the traffic needed is unrealistic, test a bolder change, pick a more sensitive metric or use variance reduction.

A common mistake is setting the minimum detectable effect to whatever the available traffic allows, which produces tests that can only detect implausibly large effects.

Each look at the results is another chance for random noise to cross the significance threshold. If you stop as soon as the p-value dips below 0.05, the real false positive rate is far above 5%; with about five looks it is roughly 14%, and with continuous monitoring it approaches certainty over a long enough test.

Why it happens: the p-value wanders up and down during a test, even when there is no effect. Stopping on the first low point captures noise.

What to do instead:

1. Fixed horizon: calculate the sample size in advance and analyze once at the end; dashboards may show progress, but decisions wait.
2. Group sequential designs: plan a few interim looks with stricter thresholds at each look, so the overall false positive rate stays at 5%.
3. Always-valid methods: sequential tests designed for continuous monitoring keep their error guarantees at every look, at some cost in power.
4. Safety stops only: allow early stopping for serious harm on guardrails, which is a different decision from declaring a win.

Interview tip: saying that peeking is fine if you use a method designed for it, and explaining why the plain fixed-sample test is not, shows a mature understanding.

A sample ratio mismatch happens when the share of users in each group differs from the designed split by more than chance allows. It means assignment or logging is broken, so the experiment's results cannot be trusted, however good they look.

Common causes:

• A bug or redirect that drops users from one variant before they are logged.
• The treatment crashing or loading slowly, so fewer of its users are recorded.
• Bots or caching affecting groups differently.
• Users being reassigned between groups when they change device or log in.

How to detect it:

1. Compare observed group counts with the expected split using a chi-square goodness-of-fit test.
2. Use a strict threshold, such as p below 0.001, since checks run on every experiment.
3. Check the ratio over time and by segment, such as browser or platform, to locate the cause.

Treat a mismatch as a stop sign: find and fix the cause, then rerun the test, rather than adjusting the results.

Every additional comparison adds another chance of a false positive. With 20 independent metrics each tested at the 5% level, the chance of at least one false "win" is about 64%, even if the change does nothing.

Practical habits:

• Pre-register one primary metric, which is tested at the full significance level.
• Treat secondary metrics and segment results as exploratory unless corrected.
• With several variants, compare each to control and correct for the number of comparisons.
• Report how many metrics and segments were examined, so readers can judge how surprising any single result is.

They are temporary reactions to change that make early experiment results unrepresentative. A novelty effect inflates early engagement, because users explore something new. A primacy effect depresses early results, because experienced users resist a change to a familiar workflow before they adapt.

How to detect them:

• Plot the treatment effect by day of the test; a steady decline suggests novelty, and a steady rise suggests primacy.
• Compare new users, who have no habits to break, with existing users.
• Look at the effect by user cohort, based on the day each user first saw the change.

How to handle them:

1. Run the test for longer, until the daily effect stabilizes.
2. Base the decision on new users, or on later weeks, when habits have settled.
3. Use a long-term holdout to confirm the effect persists after launch.
4. For major redesigns, plan a longer test from the start, since adaptation can take weeks.

Ignoring these effects is a common reason a feature's measured gain fails to appear after launch.

Standard A/B tests assume one user's treatment does not affect another user's outcome. In marketplaces, social products and shared-inventory systems, that assumption fails, and the measured effect is biased, often overstated.

flowchart LR
  A["Treatment riders get a discount"] --> B["More treatment ride requests"]
  B --> C["Shared pool of drivers"]
  C --> D["Fewer drivers free for control riders"]
  D --> E["Control looks worse than normal"]
  E --> F["Measured effect is exaggerated"]

Typical sources of interference:

• Shared supply: treatment users take inventory, drivers or delivery slots that control users would have used.
• Social spillover: treated users share content or invite friends who are in control.
• Pricing and auctions: treatment bids change prices for everyone.

How cluster randomization helps:

• Randomize whole groups, such as cities, regions or time slots, so interactions happen mostly within a group.
• Analyze at the cluster level, because the effective sample is the number of clusters, not users, which greatly reduces power.
• Alternatives include switching the treatment on and off over time for a whole market and comparing periods.

Randomize at the level where the experience must stay consistent and where outcomes are independent. The analysis unit should normally match the randomization unit.

Things to check:

• Use a stable identifier; if users switch devices, a device-based unit may assign one person to both groups.
• If you randomize by user but compute a per-session metric, correct the variance, because sessions from the same user are correlated.
• Assign users only when they reach the changed experience, so unaffected users do not dilute the effect.
• Keep assignment sticky for the whole test; reshuffling users midway mixes the two experiences and contaminates both groups.

When the randomization unit is the user but the metric divides two totals, such as revenue divided by sessions, the observations are not independent. Treating each session as a separate sample understates the variance and produces too many false positives.

Correct approaches:

1. Delta method: approximate the variance of a ratio of two means from the user-level means, variances and covariance of the numerator and denominator.
2. Bootstrap by user: resample users, not sessions, and recompute the ratio each time.
3. Change the metric: use a per-user metric such as revenue per user when it answers the business question.

Interpretation pitfalls:

• A ratio can rise because the denominator fell: revenue per session rises if the change reduces casual sessions, even when total revenue drops.
• Always report the numerator and denominator separately next to the ratio.
• Check whether the treatment changes who appears in the denominator, such as more users reaching checkout.

The same care applies to click-through rate, average order value and any metric where one user contributes many events.

Users differ a lot in their baseline behavior, and that variation is noise in the experiment. Adjusting each user's outcome by their behavior before the test removes the predictable part, so the remaining noise is smaller and the same effect becomes easier to detect.

How it works:

1. For each user, measure a pre-period value of the metric, such as spending in the previous four weeks.
2. Compute an adjusted outcome: adjusted = outcome − θ × (pre_value − mean_pre_value), where θ is the regression slope of outcome on pre-period value.
3. Compare adjusted outcomes between groups with the usual test.

Why it is valid: randomization makes the pre-period values balanced between groups, so the adjustment removes noise without introducing bias.

What to expect: the variance shrinks by a factor of 1 − ρ², where ρ is the correlation between pre-period and test-period values.

Limits: new users have no history, so the gain is smaller for tests on new users; and the covariate must be measured before assignment, never during the test.

Statistical significance only says the effect is probably not zero. Whether to ship depends on practical significance: is the effect large enough to justify its costs and risks, given the uncertainty around it?

1. Look at the confidence interval, not just the point estimate. If the whole interval sits below the effect that would be worth acting on, the result is real but not useful.
2. Translate it into business value, such as annual revenue or hours saved, using the interval's low and high ends.
3. Weigh the costs: engineering maintenance, added complexity, performance and any guardrail movements.
4. Consider reversibility: a cheap, easily reversed change can ship on a small gain; a complex one needs a stronger case.
5. Check for side effects, such as a lift in clicks that comes with more refunds.

Common recommendation patterns:

• Ship if the change simplifies something or the gain is cheap to keep.
• Do not ship if it adds complexity for a negligible gain.
• Iterate on a bolder version if the direction is promising.

Large-traffic products often detect tiny effects, so a pre-agreed minimum worthwhile effect prevents shipping trivia.

Break the result into its components and find which user behavior changed. Revenue per user equals conversion rate times average order value, so a gain in one can be outweighed by a loss in the other.

1. Decompose: compare conversion, average order value, items per order and discounts used, between groups.
2. Look at who converted: the change may have attracted smaller, price-sensitive buyers.
3. Check margins: revenue can fall while profit rises if lower-value orders carry better margins, or vice versa.
4. Think long term: new buyers may return later, so look at repeat purchases if the test ran long enough, or plan a longer holdout.
5. Return to the goal: if the strategy is acquiring customers, a modest revenue drop may be acceptable; if it is profitability, it is not.

The result is not contradictory; it is a trade-off, and the decision belongs to whoever owns the business goal, informed by clearly presented components.

A confounder is a variable that influences both the treatment and the outcome, creating an association that is not caused by the treatment. Without adjustment, the estimated effect mixes the treatment's impact with the confounder's.

flowchart TD
  A["Company size, the confounder"] --> B["Buys premium support"]
  A --> C["Renews the contract"]
  B --> C

Large companies are both more likely to buy premium support and more likely to renew anyway, so comparing renewal rates with and without premium support overstates its effect.

Ways to adjust:

1. Stratification: compare within groups of similar size, then combine the results.
2. Regression: include the confounder as a control variable.
3. Matching or weighting: compare treated customers with similar untreated ones.
4. Designs that use timing or rules, such as difference-in-differences or a threshold-based comparison.

Limits: adjustment only handles confounders you measure. Do not adjust for variables caused by the treatment, such as support tickets opened after buying support, since that can hide the effect or create bias.