Writing the analysis: from notebook to decision memo

Trace the original four-store memo’s evidence and caveats, then meet the six original summary conditions.

Lead with the decision and keep its evidence traceable

A decision memodecision memo: A short argument that leads with a recommendation, connects it to evidence and limitations, and states the next action. turns an analysis into a choice that someone can make. It opens with the answer to the question the reader asked, then gives the evidence for it, the caveats that limit it and the next step, in that order, because many readers decide from the first lines and never reach the end. The notebook does not go to the reader. It becomes the package behind the memo: the code, the lock file and the results file from which every quoted number comes. This lesson rewrites a weak summary of a free delivery trial into a strong one, checks every number in both drafts against the results with a script, and shows how to present uncertainty, coverage and causation honestly. It ends with a package that reruns the whole analysis from nothing and an executive summary of your own.

The example is an invented trial: two of four stores offered free delivery for four weeks, and the regional manager wants to know whether all stores should. The memo answers that question in four parts.

• Answer. The recommendation, in the first sentence, phrased as something the reader can approve or reject.
• Evidence. The few numbers that support the answer, each with the comparison that gives it meaning: before and after, trial stores and the others. Give each chart one message and a headline title that states it: the UK Government Analysis Function’s chart guidance asks for a headline title that communicates the main message and warns against too many messages in one chart.
• Caveats. What the data covers and what it cannot tell, stated specifically. The Analysis Function’s guidance on quality asks for vital caveats to come up front and advises saying what you can and cannot conclude; “care must be taken” on its own, it says, is not enough.
• Next step. What to do now, and what result would change the answer.

Definitions, the data source with its licence and the method go to an appendix, where a careful reader can check them without slowing everyone else down.

Keep the original recommendation, four-store evidence and proposed next action in distinct memo parts.

A rewrite should preserve the results and make their limits visible

Both drafts below describe the same results. The first is the kind of paragraph a notebook often ends with:

We analysed the store data from the last couple of months and found some interesting trends. The free delivery trial seems to have driven a big jump in order values of around 28%, and customers clearly liked it. Orders also went up. There were a few data issues, such as missing customer IDs, but they probably do not change the picture. It might be worth thinking about rolling free delivery out more widely if the costs work out.

Recommendation: extend free delivery to stores C and D for an eight-week trial before deciding on a permanent change. In the two trial stores, average order value rose 9.1% in the four trial weeks, from 630.97 to 688.60, while it fell 1.7% in the two stores without the offer, a gap of 10.8 points. Orders per week rose 6.6% in the trial stores and fell 2.9% elsewhere, both within the 7.4% by which a store's weekly orders varied before the trial. Each free delivery costs 38, and the regional manager chose the trial stores, so part of the gap may come from the stores themselves. A second trial in C and D will show whether the gap holds and whether the extra order value pays for the deliveries.

The rewrite also follows the plain-language advice of Digital.gov, the U.S. government’s guide: active voice, so it is clear who does what, the present tense where possible, and verbs rather than nouns made from verbs: “we compared the stores”, not “a comparison of the stores was carried out”.

Trace the unsupported 28 and the rewrite’s nine matched tokens before reviewing the sentences’ meanings.

A token match does not establish the right metric or denominator

Writing well is not enough if a number is wrong. The analysis writes its figures to results.json, and a short script checks that every number a draft quotes has claim provenanceclaim provenance: The link from a sentence's metric, value, unit and population to the analysis result and assumptions supporting it. when it appears there, at the precision the draft shows it:

{
  "orders": 4821,
  "trial_aov_before": 630.97,
  "trial_aov_during": 688.6,
  "trial_aov_change_pct": 9.1,
  "other_aov_change_pct": -1.7,
  "difference_points": 10.8,
  "trial_orders_per_week_change_pct": 6.6,
  "other_orders_per_week_change_pct": -2.9,
  "weekly_orders_variation_pct": 7.4,
  "guest_share_pct": 19.9,
  "delivery_cost_per_order": 38,
  "trial_weeks": 4,
  "stores_in_trial": 2
}

"""Every number a draft quotes must be in the results, at the precision the draft shows it."""
import json
import re

with open("results.json") as f:
    results = {k: v for k, v in json.load(f).items() if isinstance(v, (int, float))}


def sources(token):
    """The result keys that match a quoted number; the sign is carried by words such as "rose" and "fell"."""
    value = float(token.replace(",", ""))
    decimals = len(token.split(".")[1]) if "." in token else 0
    return [key for key, v in results.items() if round(abs(v), decimals) == value]


for name in ["weak.txt", "strong.txt"]:
    with open(name) as f:
        text = f.read()
    sentences = len(re.findall(r"[.!?](?:\s|$)", text))
    numbers = re.findall(r"\d+(?:,\d{2,3})*(?:\.\d+)?", text)  # 1,234 and 12,34,567 too
    print(f"{name}: {len(text.split())} words, {sentences} sentences, {len(numbers)} number{'s' if len(numbers) != 1 else ''}")
    for token in numbers:
        found = sources(token)
        print(f"  {token:>7}  {', '.join(found) if found else 'NOT IN THE RESULTS'}")

weak.txt: 78 words, 5 sentences, 1 number
       28  NOT IN THE RESULTS
strong.txt: 128 words, 5 sentences, 9 numbers
      9.1  trial_aov_change_pct
   630.97  trial_aov_before
   688.60  trial_aov_during
      1.7  other_aov_change_pct
     10.8  difference_points
      6.6  trial_orders_per_week_change_pct
      2.9  other_orders_per_week_change_pct
      7.4  weekly_orders_variation_pct
       38  delivery_cost_per_order

The weak draft quotes one number, and it is not in the results. 28 % may come from an earlier version of the analysis, another period or a slip; the check cannot tell which, only that it must be traced or removed. All nine numbers of the rewrite are found, each with the result it comes from. The check rounds each result to the precision the draft shows, so 688.60 matches 688.6, and a quoted 9.1 would match 9.14 but not 9.16. It ignores signs and does not parse the meaning of words such as “fell”. Check the direction, metric, unit and denominator separately; a matching token alone does not establish those relationships.

Three more habits make the memo right as well as readable, and each has its own section after this one. Uncertainty is stated as a range the reader can use. Coverage is stated with its denominator, such as the share of orders that have a customer ID. Causal words require a justified identifying design and assumptions, whether based on random assignment or a suitable observational argument.

Match the original tokens to their named metrics before accepting the quoted 10.8-point comparison.

Uncertainty, coverage and causal words, written with numbers Pro: opens with a Pro, Premium or Ultimate pass

From notebook to package, with a rubric for the summary Pro: opens with a Pro, Premium or Ultimate pass

Practise the original reporting contracts

Interview questions

Key takeaways

• A decision memo has four parts in reading order: answer, evidence, caveats, next step; definitions and the method go to an appendix.
• Give every number its comparison, and use one chart per claim with a title that states the finding.
• Name caveats specifically and say what the data cannot tell.
• Check every quoted number against the results file with a script before the memo goes out.