How chatbots write: predicting the next token

Ask a chat app to finish “Chai tastes best with” five times and you can get five different endings, each one as smooth as the last. The app is not changing its mind: it builds every reply one small piece of text at a time, and each piece is drawn from a list of likely next pieces.

You will step through that loop on a small table of chances, watch a tiny model write fluent sentences nobody ever said, see what each stage of training adds, read the hidden instructions an app puts in front of your message, and use temperaturetemperature: A setting that sharpens or flattens a model's chances for the next token before one is drawn: low values favour the top tokens, high values spread the picks, and 0 always takes the top token. to predict when the same question gets the same answer.

A chatbot writes one token at a time

A chat app runs on a large language modellarge language model: A model trained on enormous amounts of text to predict the next token, over and over. Chat apps are built on models of this kind., and the model does one thing over and over: given the text so far, it gives every tokentoken: A piece of text that a model reads or writes as one unit: often a whole common word, sometimes part of a longer word, a punctuation mark or a single byte of a rarer character. More in “Tokens, context windows and AI memory” (a word, or a piece of one) a score for coming next, one token is chosen, and it is added to the text. The textbook Speech and Language Processing lists the steps for every reply: the prompt is put together and cut into tokens, the network turns it into chances for the next token, one token is chosen and appended, and the loop goes round again until the model produces an end token. Step through the loop on this lesson’s example:

The numbers in the figure come from a small table of made-up scores for eight endings. The program turns the scores into chances at three temperatures, then draws 20 endings at each of them and at temperature 0:

"""A toy next-token table: how temperature turns a model's scores into chances, and chances into picks.

A language model gives every token of its vocabulary a score (a logit) for coming next. Dividing the scores
by the temperature and taking the softmax gives the chances; a token is then drawn at random with those
chances. Temperature 0 means "always take the top token". The eight scores below are made up for this
example; a real model scores tens of thousands of tokens at every step.
"""
import math
import random
from collections import Counter

CONTEXT = "Chai tastes best with"
SCORES = {"ginger": 2.0, "milk": 1.6, "cardamom": 1.3, "friends": 1.0,
          "biscuits": 0.8, "rain": 0.4, "sugar": 0.2, "lemon": -0.5}
TEMPERATURES = [0.5, 1.0, 1.5]
TRIES = 20


def chances(temperature):
    exps = {t: math.exp(s / temperature) for t, s in SCORES.items()}
    total = sum(exps.values())
    return {t: e / total for t, e in exps.items()}


def pick(rng, temperature):
    if temperature == 0:
        return max(SCORES, key=SCORES.get)
    probs = chances(temperature)
    return rng.choices(list(probs), weights=list(probs.values()))[0]


print(f'Chances of each next token after "{CONTEXT}"')
print(f"{'token':<10}" + "".join(f"{f'temp {t}':>10}" for t in TEMPERATURES))
table = {t: chances(t) for t in TEMPERATURES}
for token in SCORES:
    print(f"{token:<10}" + "".join(f"{table[t][token]:>10.1%}" for t in TEMPERATURES))

print(f"\n{TRIES} tries at each temperature (random seed 42)")
for temperature in [0] + TEMPERATURES:
    rng = random.Random(42)
    counts = Counter(pick(rng, temperature) for _ in range(TRIES))
    seen = ", ".join(f"{t} {n}" for t, n in counts.most_common())
    kinds = f"{len(counts)} different ending{'s' if len(counts) > 1 else ''}"
    print(f"temp {temperature}: {kinds}: {seen}")

Chances of each next token after "Chai tastes best with"
token       temp 0.5  temp 1.0  temp 1.5
ginger         50.1%     30.4%     23.9%
milk           22.5%     20.4%     18.3%
cardamom       12.3%     15.1%     15.0%
friends         6.8%     11.2%     12.3%
biscuits        4.5%      9.2%     10.7%
rain            2.0%      6.1%      8.2%
sugar           1.4%      5.0%      7.2%
lemon           0.3%      2.5%      4.5%

20 tries at each temperature (random seed 42)
temp 0: 1 different ending: ginger 20
temp 0.5: 4 different endings: ginger 11, milk 6, cardamom 2, friends 1
temp 1.0: 6 different endings: ginger 10, cardamom 4, friends 2, milk 2, rain 1, biscuits 1
temp 1.5: 7 different endings: ginger 9, friends 4, cardamom 3, milk 1, biscuits 1, sugar 1, rain 1

Read the column for temperature 1.0 first. “ginger” is the likeliest ending at 30.4 %, yet the other seven share the remaining 69.6 %, so most single draws are not the top ending. A real model does the same with a vocabulary of tens of thousands of tokens or more, at every token of every reply.

Fluent text is not a checked fact

The loop picks likely tokens, not true ones. NIST’s profile of generative AI calls the result “confabulation”, “confidently stated but erroneous or false content”, and says such errors are “a natural result of the way generative models are designed”, because the models produce text that follows the statistical patterns of their training data. You can watch it happen in a model small enough to read. It learned which word follows which from six true sentences about buses and trains, then wrote eight of its own:

"""A tiny language model that writes fluent sentences which nobody ever said.

It learns from six true sentences by counting which word follows which (a bigram model, the simplest
next-word predictor). To write, it starts at "the" and keeps picking a next word with the chances it
learned, until it reaches the full stop. Then each sentence it wrote is compared with the six it learned.
"""
import random
from collections import Counter, defaultdict

TRUE_SENTENCES = [
    "the red bus leaves at seven .",
    "the blue train leaves at nine .",
    "the red train stops at platform two .",
    "the blue bus stops at platform one .",
    "the school bus leaves at eight .",
    "the night train leaves at ten .",
]

follows = defaultdict(Counter)
for sentence in TRUE_SENTENCES:
    words = sentence.split()
    for a, b in zip(words, words[1:]):
        follows[a][b] += 1


def write(rng):
    words = ["the"]
    while words[-1] != "." and len(words) < 12:
        options = follows[words[-1]]
        words.append(rng.choices(list(options), weights=list(options.values()))[0])
    return " ".join(words)


def show(sentence):
    return sentence[0].upper() + sentence[1:-2] + "."


print('Words the model saw after "at", with their counts:', dict(follows["at"]))
rng = random.Random(11)
written = [write(rng) for _ in range(8)]
print("\nEight sentences it wrote (random seed 11):")
for s in written:
    status = "one of the six true sentences" if s in TRUE_SENTENCES else "NEW: no sentence it learned says this"
    print(f"  {show(s):<40} {status}")
new = sum(s not in TRUE_SENTENCES for s in written)
print(f"\n{new} of 8 are new. Every word pair in them comes from a true sentence; nothing checks the whole.")

Words the model saw after "at", with their counts: {'seven': 1, 'nine': 1, 'platform': 2, 'eight': 1, 'ten': 1}

Eight sentences it wrote (random seed 11):
  The blue bus stops at platform one.      one of the six true sentences
  The blue bus stops at nine.              NEW: no sentence it learned says this
  The school bus leaves at ten.            NEW: no sentence it learned says this
  The blue train leaves at seven.          NEW: no sentence it learned says this
  The red bus leaves at ten.               NEW: no sentence it learned says this
  The blue train leaves at nine.           one of the six true sentences
  The night train stops at nine.           NEW: no sentence it learned says this
  The red train leaves at platform one.    NEW: no sentence it learned says this

6 of 8 are new. Every word pair in them comes from a true sentence; nothing checks the whole.

Six of the eight are new, and every one reads like a line of a timetable. “The blue bus stops at nine.” is built entirely from word pairs that appear in true sentences (“blue bus”, “bus stops”, “stops at”, “at nine”), but no sentence it learned says so, and nothing in the model checks a sentence as a whole.

A large language model keeps far more of the context in view, which is why most of what it writes makes sense, but its basic move is the same: continue the text plausibly. Even about something it learned nothing on, a plausible continuation is always available, and it can carry an invented name, date or number.

Training stages turn a text predictor into an assistant

A model that has only learned to predict text continues a request instead of answering it. The textbook shows failures from early work: asked to explain the moon landing to a six-year-old, a model of that kind could reply with another request, to explain the theory of gravity to a six-year-old. Further training stages change that. Walk through all four stages, from pretraining on:

Each stage leaves a mark you can notice as a user:

• Pretrainingpretraining: The first and largest stage of training a language model: learning to predict the next word in huge amounts of text, which gives it broad patterns up to the point the text was collected. decides what the model has picked up about the world, and only up to the point its text was collected, its training cutofftraining cutoff: The point in time up to which a model's training text was collected. The model knows nothing later unless an app adds it to the input, for example from a web search.. A model has no clock and no internet of its own: unless the app adds search results, your files or the date to its input, it cannot know today’s news, or even today’s date.
• Instruction tuninginstruction tuning: Training a pretrained model further on instructions paired with good responses, so that it answers requests instead of merely continuing the text. is why it answers in the shape you asked for: a list when you ask for a list, a translation when you ask for one.
• Preference trainingpreference training: Training a model towards the answers that people, or another model, chose as better when comparing two answers to the same request. It shapes an assistant's manner. shapes its manner. In a study of the method, people preferred the answers of a model trained on their demonstrations and rankings to those of a model with 100 times as many parameters that had learned from text alone. It has a side effect: Sharma and others found that people and the models trained to imitate their choices both sometimes prefer convincingly written answers that agree with the user over correct ones, so a model can learn to agree too easily. The sycophancy lesson of the next module tests that.

Hidden instructions make two apps behave differently

An app does not need to train a model of its own: it can talk to one that someone else trained, in text. Hugging Face’s guide to chat templates puts it plainly: chat models still continue a sequence of tokens, and the list of messages an app passes in is turned into one such sequence, in which a system message holds “directives on how the model should act”. That message, the system promptsystem prompt: Instructions that an app places before your messages to steer the model, such as a role, rules and the format of replies. It is usually hidden from the person typing., is written by the app’s makers and is usually hidden from you. Step through one message to a proof-reading app:

The program builds what two apps send to the same model for the same sentence:

"""What a model actually reads when two apps built on it get the same message.

A chat app sends the model one long text: its own instructions (the system prompt), anything else it adds,
the conversation so far and your new message, joined with markers for who said what. The markers below
follow the shape of real chat templates; each model family has its own. Both apps use the same model.
"""

APPS = {
    "Homework helper": [
        ("system", "You help school students with homework. Explain each step in simple words. "
                   "Never give the final answer to a maths problem; ask a question that leads to it."),
        ("system", "Today's date and the student's class are added here by the app."),
    ],
    "Magazine proof-reader": [
        ("system", "You are a strict proof-reader for a school magazine. Correct spelling, grammar and "
                   "punctuation only. Reply with the corrected text and nothing else."),
    ],
}
MESSAGE = "their going to the libary tomorow to study for the test"


def build(app_messages, user_text):
    turns = app_messages + [("user", user_text)]
    return "".join(f"<|{role}|>\n{text}\n" for role, text in turns) + "<|assistant|>\n"


for app, hidden in APPS.items():
    text = build(hidden, MESSAGE)
    share = len(MESSAGE) / len(text)
    print(f"--- {app}: the model reads {len(text)} characters; your message is {len(MESSAGE)} of them ({share:.0%})")
    print(text)

--- Homework helper: the model reads 321 characters; your message is 55 of them (17%)
<|system|>
You help school students with homework. Explain each step in simple words. Never give the final answer to a maths problem; ask a question that leads to it.
<|system|>
Today's date and the student's class are added here by the app.
<|user|>
their going to the libary tomorow to study for the test
<|assistant|>

--- Magazine proof-reader: the model reads 239 characters; your message is 55 of them (23%)
<|system|>
You are a strict proof-reader for a school magazine. Correct spelling, grammar and punctuation only. Reply with the corrected text and nothing else.
<|user|>
their going to the libary tomorow to study for the test
<|assistant|>

Your 55 characters are 17 % of what the homework helper’s model reads and 23 % of the proof-reader’s. Apps add more than instructions: the date, the results of a web search, files you attached, saved notes about you and the earlier messages of the conversation all arrive the same way, as text around your message. OWASP’s guidance on system prompt leakage describes system prompts as instructions that steer a model for an application, and warns that a system prompt “should not be considered a secret, nor should it be used as a security control”.

Temperature decides how often the same question gets the same answer

Look again at the program’s table. Temperature divides every score before the scores become chances: below 1 the top tokens gain, above 1 the chances flatten out, and as it approaches 0 the top token’s chance approaches 100 %, as chapter 7 of the textbook explains. At temperature 0.5 “ginger” holds 50.1 %; at 1.5 it holds 23.9 %. Temperature 0 means always taking the top token, so its 20 draws gave “ginger” 20 times, while 0.5 gave 4 different endings, 1.0 gave 6 and 1.5 gave 7.

Why not always take the top token? The textbook describes the trade-off: text built from the likeliest words is judged more accurate and coherent, and also duller and more repetitive, and Holtzman and others found that always choosing the likeliest words “leads to text that is bland and strangely repetitive”. So apps draw at random, with a temperature and other settings their makers choose, and many chat apps do not show these settings. The everyday way to see the spread is the regenerate button: each regeneration is a new draw.

How often would 20 regenerations show the top ending? Treat each one as a try with the program’s chance at temperature 1, rounded to 0.30:

On average 6 of 20 regenerations show “ginger”, and the chance of 10 or more, as in the program’s own run, is 0.048, about 1 in 21. Set the chance to 0.50, close to the top token’s 50.1 % at temperature 0.5, and 10 or more becomes the likelier outcome (0.588).

Practise: settings, stages and the loop

Two quick checks on the program’s recorded runs and the training stages, then the lesson’s self-check:

The self-check below opens with the idea you have answered least well so far:

Interview questions

Warm-up (fresher to mid-level): Why can the same question get two different answers from a chatbot?

A chatbot writes its reply one token at a time, and at each step it draws the next token at random from a list of likely ones, so two runs can take different paths from the very first word. How spread out the draws are depends on the temperature and similar settings that the app’s makers choose. The answers can also differ for reasons outside the model: two apps may wrap the same model in different hidden instructions, one may search the web while the other does not, the earlier messages of the conversation may differ, and the model behind an app may have been updated. A good answer adds the consequence: if a question matters, regenerate it or ask it in other words, and treat answers that disagree as a sign to check a source, not to pick the one you like.

Two workplace questions build on this lesson:

Key takeaways

• A chatbot writes one token at a time: the model scores every token for coming next, one is drawn and added, and the loop repeats until an end token.
• Fluent is not checked: a word-pair model built six new timetable sentences from true word pairs, and nothing in the loop tests a sentence against the facts.
• Pretraining gives broad patterns up to a training cutoff, instruction tuning makes the model answer requests, and preference training shapes its manner, including a pull towards agreeing with you.
• An app puts hidden instructions and other text around your message, so two apps on one model can behave very differently.
• Temperature spreads or sharpens the chances: at 0 the same continuation comes out every time, right or wrong, and higher values give more different answers.