Product metrics and the north star metric

A food-delivery app can show five million app opens a month on its dashboard while it loses customers, because opening an app is not getting dinner. Product metrics tell a team every week whether people get what the product promises, and interviewers ask you to choose them because the choice shows whether you understand the product.

Below, you build the metric tree of an invented food-delivery app: one north star metric, the three input metrics that multiply into it and two guardrails that catch harm. Then you pick measures for an experience with the HEART categories, compute DAU, MAU and their ratio from a month of events, and read how a listed company defines its own active users.

A north star metric counts the value customers get

Start with what customers come for. People open a food-delivery app to get a meal that arrives on time with nothing missing, so a strong north star for it is good orders a week: orders delivered on time, complete and not refunded. App opens, menu views and downloads say that people came, not that they got anything.

The term comes from product analytics practice. The North Star Playbook, a product analytics vendor’s own guide, defines a north star metricnorth star metric: The one metric a product team chooses to count the value its customers get, such as good orders delivered in a week. The team's work should raise it, and it tends to move before revenue does. as “a single critical rate, count, or ratio” that stands for the product’s strategy, and gives it three qualities: it represents the value users get, product and marketing teams can influence it, and it is a leading indicator of revenue. Its checklist puts seven tests to a candidate. Here they are in short, tried on good orders a week:

The playbook also asks which “game” a product plays, because the game points at the value. Here is one north star for an invented product of each kind:

Input metrics are the levers that multiply into it

A north star moves slowly and no single team owns it, so it is split into input metricsinput metric: One of the few factors that produce a north star metric, such as orders per customer, and that a team can move through its daily work.: the few factors that produce it and that a team can change in its daily work. The playbook suggests “typically a set of three to five”, and adds that the north star “is not immediately actionable—the inputs are”. For the food app, three inputs multiply into the north star: ordering customers × orders per customer × the share of orders that are good. Step through the tree as it is built:

Because the inputs multiply, 5 % more of any one of them gives 5 % more good orders: 42,000 customers at the same 1.8 orders and 85 % give 64,260 good orders a week instead of 61,200. What differs between the inputs is how hard each rise is to win and what it can break, so each input gets its own team and its own guardrail.

Guardrails catch the harm the north star would miss

Each input can be pushed in a way that raises the north star and hurts the product. A guardrail metricguardrail metric: A metric that must not get worse while a team pushes another metric up, such as notification opt-outs while it sends more offers. If it does get worse, the change is stopped or rethought. is a metric that must not get worse while you push another one up. In the tree, a team that sends more offers as notifications raises orders per customer, and the opt-outs show what the offers cost; a team that promises later delivery times raises the share of orders that arrive “on time”, and the median promised time shows the trick. Good guardrails are the metrics a push would damage first:

Common guardrails across products are crashes, complaints, unsubscribes, load time and cost per order.

HEART turns an experience goal into measures

Interviewers often ask for metrics of an experience rather than of a whole product: “How would you measure the new checkout?” Rodden, Hutchinson and Fu’s paper on user-centred metrics gives five categories to choose from, the HEARTHEART framework: Five categories of user-centred metrics: Happiness, Engagement, Adoption, Retention and Task success. A team reaches a metric through goals first, then signals of success, then the metric that counts the signal. framework: Happiness, Engagement, Adoption, Retention and Task success. Teams reach a metric in three steps, goals first, then signals (what users would do or feel if the goal were met), then the metric that counts the signal. Here is one signal and one metric per category for the food app:

The paper advises reporting Engagement as an average per user rather than a total, because an increase in a total “could be a result of more users, not more usage”. It also says not every category fits every product: Engagement may mean little in a work tool that people must use. And it contrasts HEART with business and technical health measures such as page views, uptime and latency, which are too indirect to judge a change to the experience.

DAU, MAU and their ratio, computed from a month of events

Daily active usersdaily active users (DAU): The number of different users who were active on a day, by the product's own definition of active. Monthly active users (MAU) are the different users active in a month. (DAU) are the different users active on a day; monthly active users (MAU) are the different users active in the month. Their ratio, average DAU ÷ MAU, is often called stickinessstickiness (DAU/MAU): Average daily active users divided by monthly active users: the share of a month's users who are active on a typical day. At 15 %, a monthly user is active on about 4.5 days of 30.: the share of a month’s users who are active on a typical day. Both depend on what “active” means. The program counts both for a month of app opens by 12 invented users of the food app, then counts again with “active” meaning “placed an order”:

# DAU, MAU and DAU/MAU for one 30-day month of an invented food-delivery app.
# Each user's string lists the days on which they opened the app, one entry per
# open; a "*" marks an open that ended in an order. Change a string and run again.
OPENS = {
    "u01": "1 3* 3 6 8 10 12 15* 15* 18 21 24 27*",
    "u02": "2 5 9* 9* 13 16 20* 23 28",
    "u03": "1 4* 4 11 17 22* 29*",
    "u04": "2 7* 14* 19 25* 30",
    "u05": "5* 5 12 19 26*",
    "u06": "3 10 16* 22 29",
    "u07": "6 6 13* 27",
    "u08": "8 18* 28",
    "u09": "11 20 30*",
    "u10": "14 21*",
    "u11": "9 24",
    "u12": "17",
}
DAYS = 30


def active_days(only_orders):
    """Set of (user, day) pairs: a user counts once per day, however often they act."""
    pairs = set()
    for user, entries in OPENS.items():
        for entry in entries.split():
            if only_orders and not entry.endswith("*"):
                continue
            pairs.add((user, int(entry.rstrip("*"))))
    return pairs


def stickiness(pairs):
    """Average daily active users over every day of the month, MAU and their ratio."""
    dau = len(pairs) / DAYS  # the sum of each day's active users, divided by all 30 days
    mau = len({user for user, _ in pairs})
    return dau, mau, dau / mau


opens = sum(len(entries.split()) for entries in OPENS.values())
dau, mau, ratio = stickiness(active_days(only_orders=False))
print(f"A month of app opens: {opens} opens by {mau} users over {DAYS} days")
print(f"Daily active users, averaged: {dau:.2f} ({len(active_days(False))} user-days)")
print(f"MAU: {mau} users, DAU/MAU: {ratio:.1%}")
print(f"Counting opens, not users: {opens / DAYS:.2f} a day, {opens / DAYS / mau:.1%} (too high)")

dau, mau, ratio = stickiness(active_days(only_orders=True))
print(f"Active means ordering: DAU {dau:.2f}, MAU {mau}, DAU/MAU {ratio:.1%}")

# A listed company's own averages for one quarter (millions), from its shareholder letter.
published_dau, published_mau = 58.7, 140.6
print(f"Published averages: {published_dau} million DAUs, {published_mau} million MAUs")
print(f"DAU/MAU from them: {published_dau / published_mau:.1%}")

A month of app opens: 60 opens by 12 users over 30 days
Daily active users, averaged: 1.80 (54 user-days)
MAU: 12 users, DAU/MAU: 15.0%
Counting opens, not users: 2.00 a day, 16.7% (too high)
Active means ordering: DAU 0.60, MAU 10, DAU/MAU 6.0%
Published averages: 58.7 million DAUs, 140.6 million MAUs
DAU/MAU from them: 41.7%

Opens give 15.0 %: an average day had 1.80 active users, and 12 users were active in the month, so a monthly user was active on about 4.5 days of 30. Counting opens instead of users gives 16.7 %, too high: 6 of the 60 opens were a second open by the same user on the same day, such as u01’s two opens on day 3, and a user counts once a day. Counting orders gives 6.0 %: one log, two definitions, numbers more than twice apart. For a food app, 15 % is not bad news, because few people order dinner every day; the ratio says how often people come back, not what they get. Change an entry in OPENS and run it again.

Each company defines its own active user

Duolingo’s shareholder letter for one quarter, filed with the US Securities and Exchange Commission, counts as DAUs the “unique users who engage with” its app or the learning part of its website “each calendar day”, and reports them as the average over the period. It reports averages of 58.7 million DAUs and 140.6 million MAUs; the last lines of the program divide them, 41.7 %. The letter itself reports no such ratio. It evaluates engagement mainly with DAUs, treats MAUs as a supplementary metric, and calls its own Current User Retention Rate, the share of recently active daily users who come back the next day, its main measure of stickiness.

So two companies’ DAU can differ in what counts as activity, where a day ends and whether the number is an average or the last day’s count. Read the definition before you compare numbers or quote one in an answer. Set side by side, the program’s two definitions and the letter’s differ in every row:

Public services publish their measures too. GOV.UK’s Service Manual names four that UK government services must collect and publish: cost per transaction, user satisfaction, completion rate and digital take-up. It advises each team to pick three or four metrics that answer one question: “is this service working?”

Totals that only grow cannot show that a change worked

Downloads since launch, registered users and orders ever rise every week whatever the team does, so they cannot tell whether last week’s change helped. These are vanity metricsvanity metric: A metric that makes a product look good but cannot guide a decision, usually a total that only grows, such as downloads since launch. A rate or a count in a fixed period can fall, so it can show whether a change helped.: they make a product look good and decide nothing. The HEART paper’s advice is to normalise, because “raw counts will go up as your user base grows”: use ratios, percentages or averages per user instead. The playbook’s own list of vanity metrics includes downloads and registered users, and also daily active users and page views, which can fall but say nothing about the value customers get.

A decision metric can fall, which is what lets it show that something went wrong. Rates, averages per user and counts in a fixed period can; so can the cohorts of this module’s funnels and cohorts lesson, which follow each week’s new users on their own.

Practise: compute stickiness

Write the function that computes DAU/MAU from a month of events, and run the sample tests:

stickiness([("a", 1), ("a", 1), ("b", 1), ("a", 2)], 2)  # 0.75: 1.5 users a day on average, 2 in the month
stickiness([("a", 1)], 4)                                 # 0.25: days without events count as 0

def stickiness(events, days):
    """Return average daily active users divided by monthly active users.

    events: a list of (user, day) pairs, one per event; a user may repeat on a day
    days: the number of days in the month, days without events included
    Returns 0.0 when there are no events.
    """
    # Your code here.
    return 0.0

from stickiness import stickiness


def test_repeat_events_count_once():
    """two events by a on day 1 are one active user that day"""
    assert stickiness([("a", 1), ("a", 1), ("b", 1), ("a", 2)], 2) == 0.75


def test_quiet_days_count():
    """days without any event still count in the average"""
    assert stickiness([("a", 1)], 4) == 0.25


def test_no_events():
    """an empty month has nothing to divide"""
    assert stickiness([], 30) == 0.0


def test_daily_user():
    """one user active on every day of a 5-day month and one who came once"""
    events = [("a", d) for d in range(1, 6)] + [("b", 3)]
    assert stickiness(events, 5) == 0.6


def test_three_users_once_each():
    """three users, each active on one day of 30"""
    assert abs(stickiness([("a", 2), ("b", 9), ("c", 30)], 30) - 1 / 30) < 1e-12

Then answer the self-check, one question at a time:

Interview questions

Metric questions open with a choice: which number, and why that one. The warm-up below is answered in full. For the two after it, give your whole answer out loud, then compare it with the first step and the follow-ups:

Key takeaways

• A north star metric counts the value customers get, such as good orders a week, and leads revenue rather than repeating it.
• Input metrics are the few factors that produce the north star and that a team can move; for the food app they multiply, so each team owns one.
• Every input can be pushed in a harmful way, so each push gets a guardrail that the push would move the wrong way.
• HEART gives five categories of experience measures, chosen through goals, then signals, then metrics.
• DAU, MAU and their ratio depend on what “active” means: the same log gave 15.0 % for opens and 6.0 % for orders, so read each company’s definition first.
• Totals that only grow are vanity metrics; rates, averages per user and counts in a fixed period can show a change.