Product Manager Interviews Module 3 – Metrics, analytics and experiments

Diagnosing a sudden metric drop

Diagnose a metric drop in a product interview: check it is real, split it by segment and time with SQL, weigh internal and external causes, spot mix shifts.

Intermediate
20 minutes
Examples run with Python 3.14.8, Pyodide 314.0.7 and sql.js 1.14.2
By MySmartCoPilot

Free: Check that the drop is real before you explain it · Split by segment and time to find where the drop lives · Internal causes or external ones: test the likeliest first · 2 more sections. With a pass: 2 in-depth parts and 3 interview answers, about 13 minutes.

What you will learn

Check whether a metric drop is real before explaining it
Isolate a drop by time, segment and platform with SQL
Separate internal causes from external ones and test the likeliest first
Explain a drop caused by a change in the mix of users

Before you start

Product metrics and the north star metric

What you’ll learn · Before you start

What you will learn

Check whether a metric drop is real before explaining it
Isolate a drop by time, segment and platform with SQL
Separate internal causes from external ones and test the likeliest first
Explain a drop caused by a change in the mix of users

Before you start

Product metrics and the north star metric

On this page

An invented grocery app’s orders fell 13.9 % in one week, from 15,528 to 13,370. Asked why, a weak answer lists ten possible causes; a strong one runs checks in an order that rules causes out, and reaches the cause with data.

Below, you check that a drop is real, isolate it by segment and time with SQL, separate internal causes from external ones, and explain a drop that comes from a change in the mix of users rather than from anything breaking.

Check that the drop is real before you explain it

Many drops happen only in the data. A metric’s definition changed, an event stopped being logged after a release, yesterday’s data has not fully arrived, or automated traffic was filtered out: each moves the dashboard while users behave as before. The HEART paper lists such traps of log-based metrics, among them “ensuring that all of the important user actions are being logged”. GOV.UK’s Service Manual asks analysts to make sure data is collected accurately, to check for statistical significance and to test for errors or inconsistencies, and then to say what happened, why, and what to do about it.

Two checks catch most of these. Compare the metric with a second source that counts the same thing another way, such as payments received against “order placed” events, and compare the drop with the metric’s normal swing, such as the same weekday in earlier weeks. Then follow the rest of the order:

Interactive · stage by stage Diagnosing a drop in orders, stage by stage Step down the column with the grocery app's drop in orders; the side arrows show where a diagnosis goes back.

The drawing shows the whole scene; its 8 steps are listed below.

Every step as text

The drop. Orders of an invented grocery app fell from 15,528 in week 1 to 13,370 in week 2, 13.9 % fewer. The first question is whether the fall happened at all, or only in the data.
Is it real?. Check the metric's definition, any change to logging, data that has not fully arrived, and a second source such as payments received. A tracking change looks exactly like a change in users, and then the fix is in the data, not the product.
Where is it?. Split by platform: Android orders fell 27.0 %, iOS rose 0.1 % and the web stayed at 0.0 %. The drop lives on Android. Had every segment fallen alike, the next place to look would be causes outside the product.
When did it start?. Android's fall began on day 8 and deepened each day until day 12. A one-day dip suggests an outage or a holiday, a step suggests a single change; this gradual slope has the shape of a release that reaches more users each day.
An internal cause. The release log shows Android 7.4 going out from day 8. On the days both versions ran, 7.4 converted 4.5 % of sessions into orders and 7.3 converted 7.5 %, so the same calendar treated them differently.
External causes, checked. A festival, the weather or a competitor would have reached iOS and web shoppers too, and both held. An external cause does not fit this drop, which is why the order checks the internal changes first here.
A change in the mix?. A mix shift explains a total that falls while every segment holds, because a weaker segment grew. Here one segment's own rate fell, so mix is not the answer, but it is the stage that catches drops with no broken part.
Confirm it. The explanation must predict something else that a second source can show, such as fewer payments from Android 7.4. Only then is it the cause, and only then does the team size the loss and decide what to do.

Split by segment and time to find where the drop lives

A drop that sits in one segment points at its cause. Split by platform, app version, region, new or returning users and acquisition channel, then look at when it started: a one-day dip suggests an outage or a holiday, a step a single change, a slope a change that reached more users each day. The query splits the grocery app’s orders by platform, then compares Android’s two versions day by day:

Orders by week and platform, then Android by day and version SQL · drop/metric_drop.sql

-- Fourteen days of an invented grocery app: sessions and orders a day, by platform and app version.
-- Weekends (days 6, 7, 13 and 14) bring 20 % more sessions, and every day varies by up to 4 %. Android version
-- 7.4 reached 10 %, 25 %, 50 % and 75 % of Android sessions on days 8 to 11, and all of them from day 12.
CREATE TABLE daily (day INTEGER, platform TEXT, version TEXT, sessions INTEGER, orders INTEGER);

INSERT INTO daily
WITH RECURSIVE days(day) AS (SELECT 1 UNION ALL SELECT day + 1 FROM days WHERE day < 14),
segments(platform, version, base, rate) AS (
  VALUES ('android', '7.3', 14000, 0.075), ('android', '7.4', 14000, 0.045),
         ('ios', '7.3', 9000, 0.080), ('web', 'web', 5000, 0.060)
),
rollout(day, share) AS (VALUES (8, 0.10), (9, 0.25), (10, 0.50), (11, 0.75)),
sized AS (
  SELECT d.day, s.platform, s.version, s.rate,
         s.base * (CASE WHEN d.day IN (6, 7, 13, 14) THEN 1.2 ELSE 1.0 END)
                * (1 + ((d.day * d.day * 3 + s.base / 1000) % 9 - 4) / 100.0)
                * (CASE WHEN s.platform <> 'android' THEN 1.0
                        WHEN s.version = '7.4' THEN coalesce(r.share, CASE WHEN d.day >= 12 THEN 1.0 ELSE 0.0 END)
                        ELSE 1.0 - coalesce(r.share, CASE WHEN d.day >= 12 THEN 1.0 ELSE 0.0 END) END) AS sessions
  FROM days AS d CROSS JOIN segments AS s LEFT JOIN rollout AS r ON r.day = d.day
)
SELECT day, platform, version, CAST(round(sessions) AS INTEGER), CAST(round(sessions * rate) AS INTEGER)
FROM sized WHERE sessions > 0;

-- Where: orders in each week, by platform, and the change.
SELECT platform,
       sum(CASE WHEN day <= 7 THEN orders ELSE 0 END) AS week_1,
       sum(CASE WHEN day >= 8 THEN orders ELSE 0 END) AS week_2,
       round(100.0 * (sum(CASE WHEN day >= 8 THEN orders ELSE 0 END)
                      - sum(CASE WHEN day <= 7 THEN orders ELSE 0 END))
             / sum(CASE WHEN day <= 7 THEN orders ELSE 0 END), 1) AS change_pct
FROM daily
GROUP BY platform
UNION ALL
SELECT 'all',
       sum(CASE WHEN day <= 7 THEN orders ELSE 0 END),
       sum(CASE WHEN day >= 8 THEN orders ELSE 0 END),
       round(100.0 * (sum(CASE WHEN day >= 8 THEN orders ELSE 0 END)
                      - sum(CASE WHEN day <= 7 THEN orders ELSE 0 END))
             / sum(CASE WHEN day <= 7 THEN orders ELSE 0 END), 1)
FROM daily;

-- When, and which version: Android conversion by day and version, from the last day before the release.
SELECT day, version, sessions, orders, round(100.0 * orders / sessions, 1) AS rate_pct
FROM daily
WHERE platform = 'android' AND day >= 7
ORDER BY day, version;

Output

┌──────────┬────────┬────────┬────────────┐
│ platform │ week_1 │ week_2 │ change_pct │
├──────────┼────────┼────────┼────────────┤
│ android  │ 8012   │ 5849   │ -27.0      │
│ ios      │ 5227   │ 5231   │ 0.1        │
│ web      │ 2289   │ 2290   │ 0.0        │
│ all      │ 15528  │ 13370  │ -13.9      │
└──────────┴────────┴────────┴────────────┘
┌─────┬─────────┬──────────┬────────┬──────────┐
│ day │ version │ sessions │ orders │ rate_pct │
├─────┼─────────┼──────────┼────────┼──────────┤
│ 7   │ 7.3     │ 17472    │ 1310   │ 7.5      │
│ 8   │ 7.3     │ 13104    │ 983    │ 7.5      │
│ 8   │ 7.4     │ 1456     │ 66     │ 4.5      │
│ 9   │ 7.3     │ 10605    │ 795    │ 7.5      │
│ 9   │ 7.4     │ 3535     │ 159    │ 4.5      │
│ 10  │ 7.3     │ 7280     │ 546    │ 7.5      │
│ 10  │ 7.4     │ 7280     │ 328    │ 4.5      │
│ 11  │ 7.3     │ 3640     │ 273    │ 7.5      │
│ 11  │ 7.4     │ 10920    │ 491    │ 4.5      │
│ 12  │ 7.4     │ 14140    │ 636    │ 4.5      │
│ 13  │ 7.4     │ 17472    │ 786    │ 4.5      │
│ 14  │ 7.4     │ 17472    │ 786    │ 4.5      │
└─────┴─────────┴──────────┴────────┴──────────┘

Recorded with sql.js 1.14.2 (SQLite 3.49.1) on macOS 26 arm64. To run it yourself: sqlite3 -box :memory: < metric_drop.sql

The whole fall is on Android, 27.0 %, while iOS and the web held. On days 8 to 11, when both versions ran, 7.3 turned 7.5 % of sessions into orders and 7.4 only 4.5 %. Comparing versions on the same days takes the calendar out of the comparison: Google’s SRE workbook calls time “one of the biggest sources of change in observed metrics”, which is why it compares a canary with a control running at the same time.

Checkpoint

Check: Checking that a drop is real · Isolating a drop by segment and time

Question 1 of 2 A dashboard shows sign-ups down 30 % since yesterday's app release. What do you check first?
Choose one answer.

Survey users about the new version Ask marketing whether a campaign ended Whether sign-ups are still counted the same way, through the event's logging after the release and a second source such as new accounts in the user database Roll the release back at once

Show the answer: Question 1 of 2

Answer: Whether sign-ups are still counted the same way, through the event's logging after the release and a second source such as new accounts in the user database

Is it real? Check the definition, the logging, late data and a second source before explaining a drop.
Question 2 of 2 Why compare Android 7.3 with 7.4 on days 8 to 11, rather than week 1 with week 2?
Choose one answer.

Both versions ran on those days, so the calendar affected them alike and the difference belongs to the version Because 7.3 has more users Because SQL cannot compare weeks Because week 2 has more days

Show the answer: Question 2 of 2

Answer: Both versions ran on those days, so the calendar affected them alike and the difference belongs to the version

A before-and-after comparison mixes the release with everything else that changed that week. Two groups on the same days differ only in the version.

Internal causes or external ones: test the likeliest first

Internal causes are the team’s own changes: a release, an experiment, a price change, an outage, a paused campaign. Check them against the change log, the experiment list and the incident record. External causes come from outside: a season, a festival, a school holiday, the weather, a competitor’s launch. Check them against a calendar, the same weeks last year and the segments that none of the team’s changes reached.

Test first the cause that fits both where and when. The grocery drop sits in one segment and follows one release, so the release goes first; a festival would have reached iOS and web shoppers too.

Pattern	Points to	Check first
One segment, from a release	That release	Its versions on the same days
Every segment, from one day	A holiday, a change for everyone, an outage	The calendar and the change log
Each segment holds, the total falls	A change in the mix	Each segment’s share

A change in the mix can move the total while every segment holds

Sometimes no segment got worse. When a segment with a low rate grows, the overall rate falls although each segment’s rate holds. This is a mix shift: A change in an overall rate that comes from segments' shares changing while each segment's own rate holds, such as conversion falling because more visitors arrive from a channel that converts less often., an everyday case of Simpson’s paradox, in which an association “disappears or reverses when the population is divided into subpopulations”. The program splits a shop’s conversion rate: The share of users who complete a step, or a whole funnel, out of those who started it. 120 passes bought by 1,000 people who opened an app is a conversion rate of 12 %. More in “Improving an existing product” change into a mix effect, the shares moving at last week’s rates, and a rate effect, the rates moving at this week’s shares:

A fall in conversion split into a mix effect and a rate effect Python · mix/mix_shift.py

# Overall conversion fell. Did any channel get worse, or did the mix of
# visitors change? Invented numbers for two weeks of a shop's visitors and
# orders by channel. The change is split into a mix effect (the shares moved,
# at last week's rates) and a rate effect (the rates moved, at this week's shares).
WEEKS = {
    "week 1": {"search ads": (7000, 420), "social video": (3000, 60)},
    "week 2": {"search ads": (5000, 310), "social video": (7000, 140)},
}


def summary(week):
    visitors = sum(v for v, _ in week.values())
    shares = {c: v / visitors for c, (v, _) in week.items()}
    rates = {c: o / v for c, (v, o) in week.items()}
    overall = sum(o for _, o in week.values()) / visitors
    return shares, rates, overall


before, after = WEEKS["week 1"], WEEKS["week 2"]
s1, r1, all1 = summary(before)
s2, r2, all2 = summary(after)
for channel in before:
    print(f"{channel:12}  share {s1[channel]:5.1%} -> {s2[channel]:5.1%}   "
          f"conversion {r1[channel]:4.1%} -> {r2[channel]:4.1%}")
print(f"Overall conversion: {all1:.2%} -> {all2:.2%}, "
      f"{(all2 - all1) * 100:+.2f} points")

mix = sum((s2[c] - s1[c]) * r1[c] for c in before)
rate = sum(s2[c] * (r2[c] - r1[c]) for c in before)
print(f"Mix effect:  {mix * 100:+.2f} points")
print(f"Rate effect: {rate * 100:+.2f} points")
print(f"Together:    {(mix + rate) * 100:+.2f} points")

Output

search ads    share 70.0% -> 41.7%   conversion 6.0% -> 6.2%
social video  share 30.0% -> 58.3%   conversion 2.0% -> 2.0%
Overall conversion: 4.80% -> 3.75%, -1.05 points
Mix effect:  -1.13 points
Rate effect: +0.08 points
Together:    -1.05 points

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 mix_shift.py

Both channels converted as well as before or better, yet overall conversion fell from 4.80 % to 3.75 %. The mix effect is −1.13 points, because social video, which converts at 2.0 %, grew from 30.0 % to 58.3 % of visitors; the rate effect is +0.08 points. Nothing broke, so the question for the team is whether the extra social-video visitors were worth what they cost.

Checkpoint

Check: Internal and external causes · Mix shifts

Question 1 of 2 Orders fell 9 % in every segment from one Monday, and no release went out that week. Where do you look first?
Choose one answer.

Each channel's share of visitors The calendar and the change log, for anything that reached everyone, such as a holiday, a fee change or an outage The newest app version Next week's planned release

Show the answer: Question 1 of 2

Answer: The calendar and the change log, for anything that reached everyone, such as a holiday, a fee change or an outage

Match the cause to the pattern: one segment points at a change that reached only it; every segment points at a change for everyone or at the calendar.
Question 2 of 2 Half of a shop's visitors come from search, converting at 6 %, and half from social, at 2 %. Next week the shares are 25 % and 75 %, with the same rates. What is the overall conversion now, in percent?
Type a number.
Your answer %

A hint

Weight each channel's rate by its new share.

Show the answer: Question 2 of 2

Answer: 3 % (anything from 2.95 to 3.05 counts)

0.25 × 6 % + 0.75 × 2 % = 1.5 % + 1.5 % = 3.0 %, down from 4.0 %. No channel got worse; the whole fall is the mix.

Practise: find the segment that fell most

Write the query that ranks segments by how much their orders changed between two weeks, and run the sample tests:

Exercise · Easy · SQL

Find the segment whose orders fell most between two weeks

The table daily has one row per day and segment:

day: the day, from 1 to 14;
segment: a segment of users, such as android or web;
orders: the orders that segment placed that day.

Days 1 to 7 are week 1 and days 8 to 14 are week 2. Write a query that returns one row per segment, with four columns:

segment;
week_1: the segment's orders in week 1;
week_2: its orders in week 2;
change: week_2 - week_1.

A segment with no rows in one of the weeks shows 0 for that week, not NULL. Sort the rows by change, the largest fall first, and segments with the same change by name. The order of the rows is checked, so the first row is the segment whose orders fell most.

Starter code · fell_most.sql

-- One row per segment: segment, week_1, week_2, change (week_2 - week_1),
-- the largest fall first, then by segment name.
SELECT segment, sum(orders) AS week_1, 0 AS week_2, 0 AS change
FROM daily
GROUP BY segment;

The sample tests · tests.yaml

# Sample tests: each one builds a small daily table, runs your query on it and compares the rows it returns, in order.
tests:
  - name: three segments, one of them falling
    ordered: true
    setup: |
      CREATE TABLE daily (day INTEGER, segment TEXT, orders INTEGER);
      INSERT INTO daily VALUES
        (1, 'android', 50), (4, 'android', 60), (9, 'android', 30), (12, 'android', 35),
        (2, 'ios', 40), (6, 'ios', 40), (10, 'ios', 41), (13, 'ios', 42),
        (3, 'web', 20), (11, 'web', 22);
    expected:
      columns: [segment, week_1, week_2, change]
      rows:
        - ['android', 110, 65, -45]
        - ['web', 20, 22, 2]
        - ['ios', 80, 83, 3]
  - name: a segment with no orders in week 2 shows 0 and falls most
    ordered: true
    setup: |
      CREATE TABLE daily (day INTEGER, segment TEXT, orders INTEGER);
      INSERT INTO daily VALUES
        (1, 'tablet', 12), (5, 'tablet', 8),
        (2, 'web', 30), (9, 'web', 25),
        (8, 'smart-tv', 4);
    expected:
      columns: [segment, week_1, week_2, change]
      rows:
        - ['tablet', 20, 0, -20]
        - ['web', 30, 25, -5]
        - ['smart-tv', 0, 4, 4]
  - name: equal changes are sorted by name, and day 7 and day 8 fall in different weeks
    ordered: true
    setup: |
      CREATE TABLE daily (day INTEGER, segment TEXT, orders INTEGER);
      INSERT INTO daily VALUES
        (7, 'north', 10), (8, 'north', 4),
        (7, 'east', 9), (8, 'east', 3),
        (7, 'south', 5), (8, 'south', 5);
    expected:
      columns: [segment, week_1, week_2, change]
      rows:
        - ['east', 9, 3, -6]
        - ['north', 10, 4, -6]
        - ['south', 5, 5, 0]

A hint

Group by segment and add up the orders of each week with conditional aggregation: sum(CASE WHEN day <= 7 THEN orders ELSE 0 END). The ELSE 0 is what gives a week without rows a 0 instead of NULL. Then sort by the difference of the two sums.

The sample tests run on this device, in your browser (sql.js): nothing is sent to mysmartcopilot.com. The first run downloads SQLite (about 0.7 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.

Then work through the self-check below, which starts with whichever idea you have answered worst so far.

Three drops diagnosed on the lesson's data Pro: opens with a Pro, Premium or Ultimate pass

The method runs three times on the lesson’s own data. Each case comes with the query or program that settles it, its recorded output and the write-up an interviewer expects.

In this part:

The Android release that broke payment, found and sized
Ruling out the other causes, with the same data
A fall in conversion that was a cheaper channel growing
A drop in web orders that was a broken event
The write-up, for the first case

3 runnable examples (Python, SQL) · about 5 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a pass I have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

The interviewer's follow-ups, answered Pro: opens with a Pro, Premium or Ultimate pass

Interviewers push a diagnosis with three follow-ups: what if the drop is in every segment, what if two causes overlap, and what you would tell leadership first. Each has an answer that holds up.

In this part:

Is it a drop or the normal swing? Checked with numbers
When every segment fell by the same share
When two causes overlap, with a decision table
What to tell leadership in the first hour

1 runnable example (Python) · about 4 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a pass I have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

Interview questions

The warm-up is answered in full. For the other three, talk through your order of checks before you open the first step:

Fresher to mid · Warm-up

A key metric fell 20 percent overnight. What is the first thing you check, and why?

Check that the fall is real before explaining it, because a drop that large overnight is often a change in the data rather than in users. Look at four things: whether the metric's definition or query changed; whether a release changed how the event is logged; whether the last day's data has fully arrived, since late data makes the newest day look low; and whether a second source that counts the same thing another way, such as payments for orders, fell as well. Compare the day with the same weekday in earlier weeks too, so that a normal weekly dip is not taken for a drop. If the second source held while the metric fell, fix the tracking; if both fell, go on to where and when the drop started.
Fresher to senior

Daily orders fell 12 percent last week. Walk through how you would find the cause, and what you would do if it is limited to one app version.

Model answer, Pro. Model answers open with any pass; prices are in the notice on this page. First step and follow-ups
Mid to staff

Checkout conversion fell from 3.2 to 2.9 percent, yet conversion within every acquisition channel is flat. What could cause this, and how would you confirm it?

Model answer, Pro. First step and follow-ups
Mid to senior

Sign-ups dropped in the week of a major festival, and a release went out the same week. How do you tell which one caused the drop?

Model answer, Pro. First step and follow-ups

Flashcards · 7 cards

Recall it, then check

Say the answer to yourself before you look: recalling it is what makes it stick.

What is the first thing to check when a metric drops?

Whether the drop is real. A changed definition, an event that stopped being logged after a release, data that has not fully arrived or filtered bot traffic can all move the dashboard while users behave as before.
Which two checks catch most drops that exist only in the data?

Compare the metric with a second source that counts the same thing another way, such as payments received against order events, and compare the drop with the metric's usual swing, such as the same weekday in earlier weeks.
What does the shape of a drop over time suggest?

A dip lasting one day points at an outage or a holiday, a step at a single change, and a slope at a change that reached more users each day.
Into which segments should you split a metric drop?

Platform, app version, region, new or returning users and acquisition channel. In the grocery example the whole fall sat on Android, down 27.0 %, and on the same days the new version turned 4.5 % of sessions into orders against 7.5 % for the old one.
How do you separate internal from external causes of a drop?

Internal causes are the team's own releases, experiments, price changes, outages and campaigns, checked against the change log and the experiment list. External ones are seasons, festivals, weather or a competitor, checked against a calendar and last year's weeks. Test first the cause that fits both where and when.
Why compare two app versions on the same days?

Time is one of the biggest sources of change in observed metrics, so comparing versions that ran side by side takes the calendar out of the comparison.
What is a mix shift, and how can it lower conversion when no segment got worse?

A change in the share each segment holds: when a low-converting group grows, the overall rate falls although every segment holds steady. In the lesson's shop, social-video visitors grew from 30.0 % to 58.3 %, pulling conversion from 4.80 % to 3.75 %, a mix effect of −1.13 points.

Key takeaways

Check that a drop is real first: the definition, logging, late data and a second source.
Split by segment and time: where the drop sits and how it started point at the cause.
Test first the cause that fits both where and when, and compare groups on the same days.
A total can fall while every segment holds; split such a change into mix and rate.

Terms in this lesson

mix shift: A change in an overall rate that comes from segments' shares changing while each segment's own rate holds, such as conversion falling because more visitors arrive from a channel that converts less often.

References

How to set performance metrics for your service (Service Manual) (GOV.UK (Government Digital Service))
Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications (Google Research (Rodden, Hutchinson and Fu))
The Site Reliability Workbook, chapter 16: Canarying Releases (Google (O'Reilly Media))
Simpson's Paradox (Stanford Encyclopedia of Philosophy)
Built-in Aggregate Functions (SQLite)

Report a problem with this lesson

In this module: Metrics, analytics and experiments

Product metrics and the north star metric Free lesson, Pro answers: 20 minutes
Success metrics for a new feature Free core, Pro depth: 5 of 15 minutes free
Diagnosing a sudden metric drop Free core, Pro depth: 7 of 20 minutes free
Funnels, cohorts and retention Free core, Pro depth: 6 of 15 minutes free
SQL for product manager interviews Free core, Pro depth: 13 of 35 minutes free
Designing an A/B test Free core, Pro depth: 8 of 20 minutes free
Reading A/B test results Free core, Pro depth: 6 of 15 minutes free
Metric trade-offs and launch decisions Free core, Pro depth: 8 of 20 minutes free

Your country

Diagnosing a sudden metric drop

What you will learn

Before you start

What you will learn

Before you start

Check that the drop is real before you explain it

Split by segment and time to find where the drop lives

Internal causes or external ones: test the likeliest first

A change in the mix can move the total while every segment holds

Practise: find the segment that fell most

Find the segment whose orders fell most between two weeks

Results of the sample tests

Three drops diagnosed on the lesson's data Pro: opens with a Pro, Premium or Ultimate pass

The interviewer's follow-ups, answered Pro: opens with a Pro, Premium or Ultimate pass

Interview questions

A key metric fell 20 percent overnight. What is the first thing you check, and why?

Daily orders fell 12 percent last week. Walk through how you would find the cause, and what you would do if it is limited to one app version.

Checkout conversion fell from 3.2 to 2.9 percent, yet conversion within every acquisition channel is flat. What could cause this, and how would you confirm it?

Sign-ups dropped in the week of a major festival, and a release went out the same week. How do you tell which one caused the drop?

Key takeaways

References

Related tools