Skip to content

Cookies on Jobverse

We use cookies that are needed to sign you in and keep the site working. With your permission we’d also use an analytics cookie to count job views, and Google advertising cookies to show ads. The site works the same whatever you choose. Read the Cookie Policy.

Jobverse
Sign in

Data Scientist Top 100 Interview Questions

100 data scientist interview questions with structured answers, comparison tables, flow diagrams, worked calculations and runnable Python and SQL. Covers statistics, probability, A/B testing, causal inference, product metrics, SQL analytics, exploratory analysis, modeling for decisions and communication, from the basics to real business scenarios.

100 questions

Questions 61-80 of 100

  1. Q61.How do you choose a chart type for comparisons, trends, distributions and relationships?

    Start from the question the chart must answer, then pick the simplest form that makes the answer visible at a glance.

    ComparisonTrendDistributionRelationship
    Bar chart, sorted by value, with a zero baselineLine chart with time on the horizontal axisHistogram, box plot or percentile chartScatter plot, with a trend line if useful
    Revenue by regionWeekly active users over a yearDelivery times across ordersDiscount size against order value
    Avoid pie charts for more than three or four partsAvoid uneven time intervals that distort slopesAvoid showing only the meanAvoid implying causation from a pattern

    Design habits:

    • Put the main message in the title, such as "Mobile conversion fell 12% after the release".
    • Highlight the important series and grey out the context.
    • Label lines and bars directly instead of relying on a legend.
    • Use consistent colors across charts for the same category.
    • Show uncertainty, such as confidence bands, when decisions depend on small differences.
  2. Q62.What makes a chart misleading, and how do you avoid it?

    A chart misleads when its design creates an impression the data does not support, even if every number on it is correct.

    Common problems:

    • Truncated axes on bar charts: starting at a value other than zero exaggerates small differences, because bar length is read as magnitude.
    • Dual axes: two scales on one chart can make any two series look related.
    • Cherry-picked time windows: starting at a low point makes ordinary growth look dramatic.
    • Missing context: a rising count without the total, such as complaints growing while customers grow faster.
    • Area and 3D effects: distort proportions and make values hard to compare.
    • Too many categories: colors become impossible to tell apart.

    How to avoid them:

    1. Start bar charts at zero; line charts may zoom in, but label the axis clearly.
    2. Show rates alongside counts when the base changes.
    3. Use consistent time ranges and explain any chosen window.
    4. Add annotations for events that explain shifts, such as releases or holidays.

    Interview tip: when asked to critique a dashboard, interviewers want you to name the decision it should support and check whether the chart leads a reader to the right one.

  3. Q63.How do you segment customers by recency, frequency and monetary value?

    Score each customer on how recently they bought, how often they buy and how much they spend, then combine the scores into segments that call for different actions. It is simple, explainable and often a strong baseline before any clustering or modeling.

    1. Compute the three values over a fixed window, such as the last year.
    2. Score each one, usually by quantiles within your customer base or by business-defined cutoffs.
    3. Reverse recency, since fewer days since the last purchase is better.
    4. Map score patterns to segments with clear actions.

    Typical segments and actions:

    • Champions (recent, frequent, high spend): early access and referral programs.
    • New customers (recent, one order): onboarding and a second-purchase nudge.
    • At risk (not recent, but historically valuable): targeted win-back offers.

    Validate the segments by checking whether they differ in future behavior, such as next-quarter spend, and refresh them regularly.

    Example

    Four customers are scored from 1 to 3 on each dimension using fixed cutoffs.

    customers = {
        "C1": {"days_since": 5, "orders": 12, "spend": 48_000},
        "C2": {"days_since": 40, "orders": 3, "spend": 6_000},
        "C3": {"days_since": 200, "orders": 8, "spend": 30_000},
        "C4": {"days_since": 12, "orders": 1, "spend": 900},
    }
    
    
    def score(value, cutoffs, higher_is_better=True):
        level = 1 + sum(value >= c for c in cutoffs)
        return level if higher_is_better else 4 - level
    
    
    def segment(r, f, m):
        if r == 3 and f == 3:
            return "champion"
        if r == 1 and (f >= 2 or m >= 2):
            return "at risk"
        if r == 3 and f == 1:
            return "new"
        return "regular"
    
    
    for cid, c in customers.items():
        r = score(c["days_since"], [30, 90], higher_is_better=False)
        f = score(c["orders"], [3, 10])
        m = score(c["spend"], [5_000, 25_000])
        print(cid, f"{r}{f}{m}", segment(r, f, m))
    # Output:
    # C1 333 champion
    # C2 222 regular
    # C3 123 at risk
    # C4 311 new
  4. Q64.How do you estimate customer lifetime value?

    Customer lifetime value estimates the total margin a customer will generate over their relationship with the business, discounted to today's value. It guides how much to spend acquiring and retaining customers.

    A simple model for repeat purchasing:

    • Annual margin per customer m = average order value × orders per year × gross margin.
    • Annual retention rate r and discount rate d.
    • CLV = m × (1 + d) / (1 + d − r), which adds the first year's margin to each later year's margin, weighted by the chance of still being a customer and discounted.

    Making it trustworthy:

    1. Use margin, not revenue, since revenue overstates value.
    2. Estimate retention by cohort, because early churn is usually much higher than later churn.
    3. Segment it: customers acquired through different channels often have very different values.
    4. Compare with acquisition cost: a ratio of lifetime value to acquisition cost above about 3 is a common rule of thumb for healthy growth.

    For subscription businesses, monthly retention and monthly margin work the same way; for irregular buyers, a probabilistic model of purchase timing is more accurate.

    Example

    Customers place 4 orders per year at 1,500 rupees with a 30% margin; annual retention is 70% and the discount rate is 10%.

    1. Annual margin: 1,500 × 4 × 0.30 = 1,800 rupees.
    2. Lifetime value: 1,800 × 1.1 / (1.1 − 0.7) = 1,800 × 2.75 = 4,950 rupees.
    3. With an acquisition cost of 2,000 rupees, the ratio is about 2.5, below the common target of 3.
    4. Raising retention to 75% gives 1,800 × 1.1 / 0.35 ≈ 5,657 rupees, showing how strongly retention drives value.
  5. Q65.How do you build a churn analysis that leads to action rather than just a prediction?

    Define churn clearly, find which behaviors precede it, and connect each finding to an intervention that can be tested. A churn score with no plan for what to do with it rarely changes anything.

    flowchart LR
      A["Define churn and its time window"] --> B["Build cohorts with early behaviors"]
      B --> C["Find leading signals of churn"]
      C --> D["Design interventions per signal"]
      D --> E["Test interventions with holdouts"]
      E --> F["Scale what changes retention"]
    1. Define churn: for subscriptions, cancellation or non-renewal; for non-contractual products, no purchase in a period chosen from the natural repurchase cycle.
    2. Find leading signals: declining usage, failed payments, support complaints or never adopting a key feature.
    3. Separate correlation from cause: users who never adopt a feature may churn for other reasons; an experiment that drives adoption tells you whether adoption matters.
    4. Prioritize by value: focus retention effort on customers with high value and a real chance of being saved.
    5. Measure with holdouts: compare retention between contacted customers and a random group that was not contacted.

    Report retention lift and its cost, not model accuracy, as the measure of success.

  6. Q66.How do interaction terms change the interpretation of a regression?

    An interaction term lets the effect of one variable depend on the value of another. Without it, a regression assumes each variable has the same effect for everyone, which is often unrealistic.

    spend = b0 + b1 × premium + b2 × tenure + b3 × premium × tenure

    How to read the coefficients:

    • b1 is the effect of premium only when tenure is zero, not the general premium effect.
    • b2 is the effect of one more month of tenure for non-premium customers.
    • b3 is how much the tenure effect changes for premium customers, and equivalently how much the premium effect grows per month of tenure.

    Good practice:

    1. Center continuous variables, such as tenure minus its average, so main effects describe a typical customer.
    2. Report effects at meaningful values, such as the premium effect at 1, 12 and 24 months.
    3. Plot predicted values by group, which is far easier to understand than coefficients.
    4. Add interactions for a reason, since searching through many of them invites false findings.

    Example

    A fitted model is spend = 500 + 200 × premium + 30 × tenure + 20 × premium × tenure, with spend in rupees per month and tenure in months.

    1. Premium effect at 0 months: 200 rupees.
    2. Premium effect at 12 months: 200 + 20 × 12 = 440 rupees.
    3. Tenure effect for non-premium customers: 30 rupees per month; for premium customers: 30 + 20 = 50 rupees per month.
    4. Reporting "premium adds 200 rupees" would understate its effect for most established customers.
  7. Q67.How do you interpret regression coefficients when the target is log-transformed?

    When the outcome is the natural logarithm of a value, such as spend, each coefficient describes a proportional change rather than an absolute one. A coefficient b on a yes-or-no variable means the outcome is multiplied by e^b, which is a change of (e^b − 1) × 100 percent.

    • Small coefficients: b × 100 is a close approximation, so 0.05 is roughly a 5% increase.
    • Large coefficients: the approximation breaks down, so always convert with the exponential.
    • Both sides logged: when both the outcome and a predictor are logged, the coefficient is an elasticity, the percentage change in the outcome for a 1% change in the predictor.
    • Predicting back: converting predicted logs back to the original scale gives something closer to the median than the mean, so averages need a correction.

    Why log the target at all:

    • Skewed positive outcomes become closer to symmetric.
    • Effects that are naturally proportional, such as a discount raising spending by a percentage, become linear.
    • Outliers have less influence on the fit.

    Example

    A model of log monthly spend gives a coefficient of 0.05 for using the mobile app and 0.40 for being on a premium plan.

    1. App users: e^0.05 − 1 ≈ 0.051, about 5.1% higher spend, close to the quick estimate of 5%.
    2. Premium plan: e^0.40 − 1 ≈ 0.492, about 49% higher spend, well above the naive reading of 40%.
    3. For a customer spending 2,000 rupees, the premium effect is roughly 984 rupees, not 800.
  8. Q68.What is survival analysis, and why do you need it for time-to-event data?

    Survival analysis models the time until an event, such as churn, a repeat purchase or a device failure, while correctly handling censoring: cases where the event has not happened yet when observation ends. Ignoring censored cases, or treating them as events, gives biased answers.

    Why ordinary methods fail:

    • Averaging the tenure of churned customers ignores current customers, who may stay much longer.
    • Treating active customers as non-churners for a fixed window wastes information about how long they have already stayed.

    The key output: a survival curve, showing the share of customers still active at each time. A standard estimate multiplies, at each event time, the share of at-risk customers who did not churn.

    Common uses:

    • Comparing retention curves between plans or signup cohorts.
    • Estimating median customer lifetime, which feeds lifetime value.
    • Modeling how factors such as plan type speed up or slow down churn, with regression models built for time-to-event data.

    Define the start time and event precisely, since a vague definition of "churn" makes any survival model unreliable.

    Example

    Eight customers are observed; three are still active, which makes them censored.

    from collections import Counter
    
    customers = [(2, True), (3, False), (4, True), (4, True), (5, False), (6, True), (8, False), (9, True)]
    
    churn_counts = Counter(months for months, churned in customers if churned)
    survival = 1.0
    for t in sorted(churn_counts):
        at_risk = sum(1 for months, _ in customers if months >= t)
        survival *= 1 - churn_counts[t] / at_risk
        print(t, at_risk, churn_counts[t], round(survival, 3))
    # Output:
    # 2 8 1 0.875
    # 4 6 2 0.583
    # 6 3 1 0.389
    # 9 1 1 0.0

    Each tuple is the months observed and whether the customer churned. Customers who are still active count as at risk until they leave observation, then drop out without being counted as churned.

  9. Q69.How does uplift modeling identify customers who respond because of a treatment?

    Uplift modeling predicts the change in a customer's behavior caused by a treatment, such as a discount, rather than the behavior itself. A standard churn or purchase model finds people likely to act, many of whom would act anyway.

    PersuadableSure thingLost causeDo not disturb
    Buys only if treatedBuys whether treated or notDoes not buy either wayBuys only if left alone
    The only group worth targetingTreating wastes the discountTreating wastes the contactTreating actively loses the sale

    How it is built:

    1. Run a randomized test where some customers receive the treatment and others do not.
    2. Model the outcome separately for treated and control customers, or with a single model that includes treatment as an input, and estimate the difference for each customer.
    3. Rank customers by predicted uplift and target the top of the list.

    How to evaluate it: plot the cumulative gain from targeting the top-ranked customers, measured on held-out experimental data, since individual uplift can never be observed directly.

    Uplift needs randomized data and large samples, because it estimates a difference between two noisy predictions.

  10. Q70.How do you measure which products are bought together, using support, confidence and lift?

    These three measures summarize how often items appear together in baskets, which guides cross-selling, bundling and store layout.

    • Support of a pair: the share of all baskets that contain both items.
    • Confidence of a rule "A then B": among baskets with A, the share that also contain B.
    • Lift: confidence divided by the overall share of baskets containing B. Lift above 1 means A makes B more likely; lift below 1 means less likely.

    Why lift matters: a rule can have high confidence just because the second item is popular. If almost everyone buys butter, "bread then butter" has high confidence even when bread buyers are no more likely to buy butter than anyone else.

    Practical points:

    • Filter by minimum support first, so rare combinations do not produce noisy, extreme lifts.
    • Validate promising rules with an experiment, such as a recommendation test, since co-purchase is not causation.
    • Compute measures per segment or season, as baskets differ by context.

    Example

    In five baskets, "bread then butter" has high confidence but a lift below 1.

    baskets = [
        {"bread", "butter"},
        {"bread", "butter", "jam"},
        {"bread"},
        {"milk", "butter"},
        {"bread", "butter", "milk"},
    ]
    
    
    def share(*items):
        return sum(set(items) <= b for b in baskets) / len(baskets)
    
    
    for a, b in [("bread", "butter"), ("jam", "bread")]:
        support = share(a, b)
        confidence = support / share(a)
        lift = confidence / share(b)
        print(f"{a} -> {b}: support={support:.2f} confidence={confidence:.2f} lift={lift:.2f}")
    # Output:
    # bread -> butter: support=0.60 confidence=0.75 lift=0.94
    # jam -> bread: support=0.20 confidence=1.00 lift=1.25
  11. Q71.Why is averaging averages a mistake, and how do you combine rates correctly?

    A simple average of group rates gives every group equal weight, regardless of size. When groups differ in size, the result does not represent the overall rate, and small groups can dominate it.

    Correct ways to combine rates:

    • Pooled rate: add up the numerators and denominators across groups, then divide, such as total conversions divided by total visitors.
    • Weighted average: weight each group's rate by its denominator, which gives the same answer as pooling.

    When an unweighted average is right: when each group deserves equal weight by design, such as the average experience across stores regardless of traffic. State that choice explicitly.

    Where the mistake hides:

    • Dashboards that average daily rates across a month, while days have different traffic.
    • Averages of percentiles, such as averaging daily 95th percentile latencies, which do not give the monthly 95th percentile.
    • Averaging ratios per user, which differs from the ratio of totals.

    Keep numerators and denominators in your data model, so any level of aggregation can be computed correctly.

    Example

    Store A converts 5 of 50 visitors, a rate of 10%; store B converts 19 of 950 visitors, a rate of 2%.

    1. Simple average of rates: (10% + 2%) / 2 = 6%.
    2. Pooled rate: (5 + 19) / (50 + 950) = 24 / 1,000 = 2.4%.
    3. The simple average is 2.5 times too high, because the small store counts as much as the large one.
  12. Q72.How do you define a metric precisely enough that two analysts get the same number?

    Write down every choice the calculation depends on, and keep the definition in one shared place, ideally as code that everyone reuses. Most "conflicting numbers" between teams come from unstated differences in definitions.

    What a complete definition specifies:

    1. Numerator and denominator: for example, completed orders divided by sessions with a product view.
    2. Unit: per user, per session or per order.
    3. Filters: whether internal staff, test accounts, bots, cancelled orders and refunds are excluded.
    4. Time: which timestamp is used, the time zone, and how late-arriving data is handled.
    5. Deduplication: how repeated events from retries or multiple devices are treated.
    6. Edge cases: zero denominators, currency conversion and partial periods.

    Keeping definitions consistent:

    • Store metric logic in a shared, version-controlled layer that dashboards and analyses query, instead of rewriting the logic in each report.
    • Give each metric an owner who approves changes.
    • Annotate dashboards when a definition changes, and backfill history when possible, so trends are not broken by a silent shift.
  13. Q73.Why do partial periods and late-arriving data distort recent trends?

    The most recent period on a chart is often incomplete, either because it is still in progress or because some events are recorded days later. Both effects make the latest point look like a decline when nothing has changed.

    Common sources:

    • Period in progress: the current week or month has only some of its days.
    • Late events: conversions attributed days after a click, delayed payment confirmations, or mobile events uploaded when a device reconnects.
    • Pipeline delays: data that lands hours after the dashboard refreshes.

    How to handle them:

    1. Exclude or visually mark incomplete periods on charts.
    2. Compare like with like, such as the first three days of this week with the first three days of last week.
    3. Measure the typical completeness curve, such as how much of a day's conversions arrive within 1, 3 and 7 days, and wait until data is mostly complete before deciding.
    4. Report metrics that mature over time with an "as of day N" definition, such as conversions within seven days of signup.

    Example

    A store typically gets 7,000 orders per week, spread evenly across the days, and about 30% of each day's orders are recorded during the following two days.

    1. On Wednesday evening, the dashboard shows 2,400 orders so far this week.
    2. Against a full week of 7,000, that looks like a 66% drop.
    3. A typical Monday to Wednesday has about 7,000 × 3 / 7 = 3,000 orders.
    4. Roughly 30% of Tuesday's and Wednesday's 2,000 orders are not recorded yet, so about 3,000 − 600 = 2,400 should be visible now.
    5. The week is on track, and no alert should fire.
  14. Q74.How do you make a quick estimate of a market size or volume with little data?

    Break the quantity into a chain of factors you can estimate or look up, multiply them, and give a range rather than a single number. The structure of the estimate matters more than any single input, because it shows which assumption drives the answer.

    1. Clarify the target: for example, daily food delivery orders in one large city, not revenue and not all meals.
    2. Choose a decomposition: population to households, households that order online, orders per household per week.
    3. Estimate each factor, using known figures where possible and stating your assumptions.
    4. Multiply, then sanity-check against anything known, such as the capacity of local restaurants.
    5. Give a range: low and high values for the most uncertain factors.
    6. Identify the swing factor: the input that changes the answer most, which is what to research first.

    Approaches:

    • Top-down: start from the total population and narrow down.
    • Bottom-up: start from supply, such as restaurants times orders per restaurant per day.

    Doing both and comparing the results is a strong way to catch a bad assumption.

    Example

    Estimate daily food delivery orders in a city of 10 million people.

    1. Households: 10,000,000 / 4 ≈ 2,500,000.
    2. Households that order delivery at all: about 40%, giving 1,000,000.
    3. Orders per ordering household: about 1.5 per week, giving 1,500,000 orders per week.
    4. Daily orders: 1,500,000 / 7 ≈ 214,000.
    5. With 25% to 55% of households ordering and 1 to 2 orders per week, the range is roughly 90,000 to 390,000 per day, so the ordering share is the factor to research first.
  15. Q75.How do you detect anomalies in a daily business metric that has a weekly pattern?

    Compare each day with what is normal for that day of the week, not with the previous day, since a quiet Monday after a busy Sunday is not an anomaly. Flag a day when it differs from that expectation by more than a threshold set from historical variation.

    1. Build an expectation: the median of the same weekday over the last few weeks, which resists one-off spikes.
    2. Measure the deviation: the percentage difference, or the difference divided by a robust measure of spread.
    3. Set thresholds from history: choose a level that would have flagged a manageable number of past days, and check that known incidents would have been caught.
    4. Handle known events: holidays and campaigns need separate expectations, or annotations that suppress alerts.

    Making alerts useful:

    • Alert on a few key metrics, and break them down automatically by segment when one fires.
    • Require a sustained deviation for noisy metrics, such as two consecutive hours.
    • Review false alarms regularly and adjust thresholds.

    Example

    Three normal weeks set the expectation for each weekday; the latest week has a drop on day 4 and a spike on day 7.

    import statistics
    
    weekly_pattern = [100, 104, 98, 102, 110, 150, 160]
    history = [v + offset for offset in (0, 2, -3) for v in weekly_pattern]
    latest_week = [101, 106, 97, 70, 112, 148, 205]
    
    for day, value in enumerate(latest_week):
        expected = statistics.median(history[week * 7 + day] for week in range(3))
        change = (value - expected) / expected
        if abs(change) > 0.2:
            print(f"day {day + 1}: {value} vs expected {expected} ({change:+.0%})")
    # Output:
    # day 4: 70 vs expected 102 (-31%)
    # day 7: 205 vs expected 160 (+28%)
  16. Q76.How would you test the difference between two conversion rates using only Python's standard library?

    Use a two-proportion z-test: compute each group's rate, the pooled rate under the null hypothesis of no difference, the standard error of the difference, and a z-score that converts to a p-value through the normal distribution. Report a confidence interval for the difference as well, since it shows the size and uncertainty of the effect.

    Steps:

    1. Rates: p_a = conversions_a / n_a and p_b = conversions_b / n_b.
    2. Pooled rate for the test: total conversions divided by total users.
    3. z = (p_b − p_a) / √(pooled × (1 − pooled) × (1/n_a + 1/n_b)).
    4. Two-sided p-value: 2 × (1 − Φ(|z|)), where Φ is the standard normal cumulative distribution.
    5. Confidence interval: use the unpooled standard error, since it describes the difference itself rather than the null hypothesis.

    When the test is valid: observations are independent, users are counted once, and each group has at least a few dozen conversions and non-conversions. For very small counts, use an exact test instead.

    Example

    Control converts 1,000 of 10,000 users; treatment converts 1,090 of 10,000.

    from statistics import NormalDist
    
    
    def two_proportion_test(conv_a, n_a, conv_b, n_b):
        p_a, p_b = conv_a / n_a, conv_b / n_b
        pooled = (conv_a + conv_b) / (n_a + n_b)
        se_pooled = (pooled * (1 - pooled) * (1 / n_a + 1 / n_b)) ** 0.5
        z = (p_b - p_a) / se_pooled
        p_value = 2 * (1 - NormalDist().cdf(abs(z)))
        se_diff = (p_a * (1 - p_a) / n_a + p_b * (1 - p_b) / n_b) ** 0.5
        margin = NormalDist().inv_cdf(0.975) * se_diff
        return p_b - p_a, z, p_value, (p_b - p_a - margin, p_b - p_a + margin)
    
    
    diff, z, p, (low, high) = two_proportion_test(1_000, 10_000, 1_090, 10_000)
    print(f"lift={diff:.4f} z={z:.2f} p={p:.4f}")
    print(f"95% CI: {low:.4f} to {high:.4f}")
    # Output:
    # lift=0.0090 z=2.08 p=0.0375
    # 95% CI: 0.0005 to 0.0175

    The lift of 0.9 points is significant at the 5% level, but the interval runs from almost zero to 1.75 points, so the true effect could be small.

  17. Q77.How would you fit and sanity-check a simple linear regression in Python?

    For one predictor, the standard library's statistics.linear_regression returns the least-squares slope and intercept, and statistics.correlation gives the correlation, whose square is R-squared. Fitting is the easy part; checking whether the line is trustworthy and how far it can be used takes more care.

    Sanity checks:

    1. Plot the data with the fitted line, and look for curves, clusters or outliers that a single line hides.
    2. Look at residuals, the gaps between actual and predicted values; patterns in them mean the model misses structure.
    3. Interpret the slope in business units, such as extra orders per thousand rupees of ad spend.
    4. Stay within the data's range: predicting far beyond the observed values assumes the relationship continues unchanged, which it rarely does, for example when advertising saturates.
    5. Remember causation: a slope from observational data measures association; spend may rise in weeks that were already busy.

    A high R-squared on five points says little; small samples can fit almost perfectly by chance, so judge the result against domain knowledge and more data.

    Example

    Five weeks of ad spend, in thousands of rupees, and orders.

    import statistics
    
    ad_spend = [10, 20, 30, 40, 50]
    orders = [120, 150, 190, 210, 250]
    
    slope, intercept = statistics.linear_regression(ad_spend, orders)
    r = statistics.correlation(ad_spend, orders)
    
    print(round(slope, 2), round(intercept, 1))  # 3.2 88.0
    print(round(r ** 2, 3))  # 0.992
    print(round(intercept + slope * 60))  # 280

    Each extra thousand rupees is associated with about 3.2 more orders. The prediction for 60 thousand lies outside the observed range, so it should be treated as a rough guess.

  18. Q78.How would you draw a stratified sample and weight it back to the population in Python?

    Group units by stratum, sample from each group separately, then give every sampled unit a weight equal to its stratum's population size divided by the stratum's sample size. Weighting corrects for deliberately sampling some groups more heavily than their share of the population.

    Why sample disproportionately: a small but important group, such as premium customers, needs enough sampled units for its own estimate. Sampling it at a higher rate achieves that, but an unweighted average would then over-represent it.

    Steps:

    1. Split the population into strata, such as plans or regions.
    2. Draw a random sample of a chosen size from each stratum, using a fixed seed so the sample can be reproduced.
    3. Compute each stratum's weight: population count divided by sample count.
    4. Estimate population totals or means with the weights.

    Checks: confirm every stratum has enough sampled units, compare weighted sample totals with known population totals, and report results per stratum as well as overall.

    Example

    Premium users are 20% of the population but half of the sample, so the unweighted mean overstates spending.

    import random
    import statistics
    from collections import defaultdict
    
    random.seed(42)
    users = []
    for i in range(1, 101):
        if i % 5 == 0:
            users.append(("premium", 400 + (i % 9) * 25))
        else:
            users.append(("free", 40 + (i % 6) * 5))
    
    by_plan = defaultdict(list)
    for plan, spend in users:
        by_plan[plan].append(spend)
    
    sample = {plan: random.sample(values, 10) for plan, values in sorted(by_plan.items())}
    weights = {plan: len(by_plan[plan]) / len(values) for plan, values in sample.items()}
    
    unweighted = statistics.mean(v for values in sample.values() for v in values)
    weighted = sum(weights[p] * sum(vals) for p, vals in sample.items()) / len(users)
    
    print(weights)  # {'free': 8.0, 'premium': 2.0}
    print(round(statistics.mean(s for _, s in users), 1))  # 141.3
    print(f"{unweighted:.1f} {weighted:.1f}")  # 275.0 141.5
  19. Q79.How would you show by simulation that peeking at an A/B test inflates false positives?

    Simulate many experiments in which there is no true effect, check the test statistic at several interim looks, and count how often any look crosses the significance threshold. With one look the rate matches the 5% level; with more looks it climbs steadily.

    How the simulation works:

    1. Under the null hypothesis, the standardized difference between groups after each batch of users behaves like a running sum of independent standard normal steps.
    2. At look k, the z-score is the running sum divided by √k.
    3. A trial is a false positive if any look exceeds the critical value of about 1.96.
    4. Repeat thousands of times and take the share of false positives.

    What it teaches:

    • The error rate depends on the number of looks, not on the analyst's intentions.
    • Methods for sequential testing raise the threshold at each look, so the overall rate stays at 5%.
    • Simulation is a quick way to check any analysis plan before using it on real decisions, including unusual metrics and stopping rules.

    Example

    Twenty thousand null experiments per setting, with one, five or twenty equally spaced looks.

    import random
    from statistics import NormalDist
    
    random.seed(1)
    critical = NormalDist().inv_cdf(0.975)
    
    
    def false_positive_rate(looks, trials=20_000):
        hits = 0
        for _ in range(trials):
            total = 0.0
            for k in range(1, looks + 1):
                total += random.gauss(0, 1)
                if abs(total / k ** 0.5) > critical:
                    hits += 1
                    break
        return hits / trials
    
    
    for looks in (1, 5, 20):
        print(looks, round(false_positive_rate(looks), 3))
    # Output:
    # 1 0.049
    # 5 0.141
    # 20 0.247
  20. Q80.An A/A test, where both groups get the same experience, shows a significant difference. What do you conclude?

    One significant A/A result is expected about 5% of the time, so a single occurrence proves little. A pattern of them, or a large difference, means the experiment system or the analysis is broken, and every A/B test run on it is suspect.

    First, check how often it happens: run many A/A comparisons, for example by re-randomizing past data; p-values should be roughly uniform, with about 5% below 0.05.

    If they are too often significant, look for:

    • Wrong variance: analyzing sessions or page views when users were randomized, which understates uncertainty.
    • Assignment problems: a sample ratio mismatch, users switching groups or bots concentrated in one group.
    • Logging differences: one group's events recorded differently, for example through a different code path.
    • Outliers: a few extreme users dominating a revenue metric.
    • Carryover: groups reused from a previous experiment, so their past treatment still affects behavior.

    What to do: pause decisions that depend on the system, fix the cause, and rerun the A/A test. A healthy experimentation program runs A/A checks routinely, because they are the cheapest way to catch problems that would otherwise produce confident, wrong results.