AI Literacy & Prompting Module 3 – Prompting: from a first try to reliable results
Iterating, chaining and self-review
How to improve AI results step by step: specific feedback instead of "make it better", chains of small prompts with checks, and why self-review is no proof.
What you will learn
- Improve a result through short rounds of specific feedback instead of rewriting the prompt from scratch
- Split a big task into a chain of smaller prompts with a check between the steps
- Ask the AI to question you first, to review its draft against your rubric and to offer alternatives
- Choose a check from outside the model for any step whose numbers or facts matter
- Save prompts that work in a personal prompt library, with test cases to run again after updates
Before you start
What you’ll learn · Before you start
What you will learn
- Improve a result through short rounds of specific feedback instead of rewriting the prompt from scratch
- Split a big task into a chain of smaller prompts with a check between the steps
- Ask the AI to question you first, to review its draft against your rubric and to offer alternatives
- Choose a check from outside the model for any step whose numbers or facts matter
- Save prompts that work in a personal prompt library, with test cases to run again after updates
Before you start
On this page
When Madaan and others let AI models give feedback on their own answers to maths word problems, one model’s feedback said “everything looks good” for 94 % of the answers, wrong ones included. Improving a result step by step works, but only when each step brings something new: your specific feedback, a smaller task, a check from outside the model.
This lesson turns that into habits: feedback that names the change, chains of small prompts with a check between the steps, questions the AI asks you first, checks that do not rely on the model’s view of its own work, a fresh start when a chat drifts, and a library of prompts that you test again after every update.
Specific feedback beats “make it better”
A draft notice comes back at 160 words, with a point about parking nobody asked for and no deadline. “Make it better” leaves the AI to guess what “better” means. “Cut it to at most 100 words, drop the point about parking, and add the deadline, Friday at 5 pm” names three changes, and each can be checked in the next draft.
The research points the same way. In Self-Refine, Madaan and others had a model draft, give feedback on its draft and revise, round after round, which improved results by about 20 points on average across seven tasks. They then compared three kinds of feedback, specific, generic and none, on three of those tasks:
| Task | Specific | Generic | None |
|---|---|---|---|
| Rewriting a review with the opposite sentiment | 43.2 | 31.2 | 0 |
| Making up acronyms | 56.4 | 54.0 | 48.0 |
| Speeding up code | 27.5 | 26.0 | 24.8 |
Specific feedback did best on all three, and on the first task revising with no feedback at all failed completely. Their example of generic feedback was “Improve the efficiency of the code”; the specific one was “Avoid repeated calculations in the for loop”. Your own feedback works the same way, so write it like this:
- One line per problem, naming the exact change: the number, the sentence, the item.
- Keep what worked: “Keep the heading and the first bullet as they are.”
- Feed back the failed checks of your test, one line each, as the lesson on the anatomy of a clear prompt showed.
Checkpoint
Check: Specific feedback
Chain small prompts, with a check between steps
A two-day trip for 40 students and 4 teachers is too much for one prompt: an outline, a cost table, a check and a note to parents all at once leave nowhere to stop and look. prompt chaining: Splitting a task into a sequence of smaller prompts, in which the checked output of one step becomes the input of the next: for example an outline, then a draft, then a check, then the final version. splits it into steps, and the checked output of each step becomes the input of the next. In a study of 20 people, Wu and others found that chaining “improved the quality of task outcomes” and made the work more transparent and easier to control. Breaking a problem into smaller ones helps the model too: with least-to-most prompting, Zhou and others reached at least 99 % on a test where a single chain of reasoning reached 16 %. Step through the trip’s chain:
The drawing shows the whole scene; its 5 steps are listed below.
Every step as text
- An outline first. The first prompt asks only for an outline of the costs, with headings for transport, entry tickets, the stay and meals. The teacher reads it before anything else is written; a missing heading costs one line to fix now.
- The cost table. The approved outline and the teacher's prices go into the second prompt, which asks for a table of item, quantity, price and line total, with the total at the end.
- A check that rebuilds. The check does not reread the draft, whose lines do add up to its total. It rebuilds every line from the prices and the head counts, and the meals line fails. 176 meals at 120 rupees cost 21,120, not 19,120.
- Back to step 2. The failed line goes back with exact feedback, "Meals are 176 x 120 = 21,120; correct the line and the total." The corrected total is 58,020 rupees, 1,020 over the budget of 57,000.
- Only checked numbers go on. The note to parents is written from the checked table, at 1,451 rupees for each of the 40 students, with the 1,020 over the budget to be settled first.
Step 3 is the one that matters. The program below is that check: it first rereads the draft the way a quick review would, then rebuilds every line from the teacher’s prices and head counts:
"""Step 3 of a chain: check a draft cost table against the given prices.
A teacher is planning a two-day trip for 40 students and 4 teachers with
a chain of prompts: an outline, then a cost table from the prices the
teacher gave, then a check, then a note to parents. The draft table is
step 2's output, written for this lesson with one slip in it. The check
does not trust the table: it rebuilds every line from the prices and
the head counts, then compares.
"""
STUDENTS, TEACHERS = 40, 4
PEOPLE = STUDENTS + TEACHERS
BUDGET = 57_000
# The prices the teacher gave, in rupees, and how many of each are needed.
PRICES = {
"Bus hire, two days": (18_000, 1),
"Science centre entry": (60, STUDENTS), # teachers do not pay
"Fort entry": (25, PEOPLE),
"Dormitory, one night": (350, PEOPLE),
"Meals, four each": (120, PEOPLE * 4),
}
# Step 2's draft: item, quantity, price, line total; then its own total.
DRAFT = [
("Bus hire, two days", 1, 18_000, 18_000),
("Science centre entry", 40, 60, 2_400),
("Fort entry", 44, 25, 1_100),
("Dormitory, one night", 44, 350, 15_400),
("Meals, four each", 176, 120, 19_120),
]
DRAFT_TOTAL = 56_020
DRAFT_SAYS_WITHIN_BUDGET = True
print("A re-read of the draft: does its total add up its own lines?")
print(f" lines add to {sum(row[3] for row in DRAFT):,}; "
f"the draft says {DRAFT_TOTAL:,}: no slip found this way")
print("\nA rebuild from the given prices:")
total = 0
for item, quantity, price, line_total in DRAFT:
given_price, needed = PRICES[item]
right = given_price * needed
total += right
problems = []
if price != given_price:
problems.append(f"price should be {given_price:,}")
if quantity != needed:
problems.append(f"quantity should be {needed}")
if line_total != right:
problems.append(f"{needed} x {given_price:,} = {right:,}, "
f"not {line_total:,}")
print(f" {'FAIL' if problems else 'ok '} {item}: "
f"{'; '.join(problems) or f'{right:,}'}")
print(f"\nRight total: {total:,} rupees (draft: {DRAFT_TOTAL:,})")
within = total <= BUDGET
print(f"Budget {BUDGET:,}: {'within' if within else 'over'} by "
f"{abs(BUDGET - total):,} rupees "
f"(the draft said {'within' if DRAFT_SAYS_WITHIN_BUDGET else 'over'})")
share = -(-total // STUDENTS) # rounded up to a whole rupee
print(f"Each of the {STUDENTS} students pays {share:,} rupees, rounded up")Output
A re-read of the draft: does its total add up its own lines? lines add to 56,020; the draft says 56,020: no slip found this way A rebuild from the given prices: ok Bus hire, two days: 18,000 ok Science centre entry: 2,400 ok Fort entry: 1,100 ok Dormitory, one night: 15,400 FAIL Meals, four each: 176 x 120 = 21,120, not 19,120 Right total: 58,020 rupees (draft: 56,020) Budget 57,000: over by 1,020 rupees (the draft said within) Each of the 40 students pays 1,451 rupees, rounded up
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 trip_check.py
The re-read finds nothing, because the draft’s lines really do add up to its total of 56,020. The rebuild finds the meals line 2,000 rupees short, and with it the decision changes: the right total is 58,020 rupees, over the budget of 57,000 by 1,020, where the draft said it was within. Each student’s share is 1,451 rupees, rounded up.
Key idea
Check each step before it feeds the next, and check by rebuilding from the source: a draft can agree with itself and still be wrong.
Checkpoint
Check: Chains with a check between steps
Ask the AI to question you first
Most gaps in a request are ones you did not notice, and the model seldom points them out unprompted. In the tests of Kuhn and others, models given an unclear question tended to answer it wrongly rather than ask; prompted to spot the ambiguity and put a question first, they answered more accurately. You can ask for that in any prompt: “Before you start, ask me up to three questions about anything you need to know.” Two more requests turn a single answer into choices:
- “Give me three different openings, each in one sentence,” when the right one is a matter of judgement.
- “List what you assumed,” after a reply, to find the guesses it made.
Step through a planning request that uses both:
The drawing shows the whole scene; its 5 steps are listed below.
Every step as text
- Questions first. The request ends: "Before you start, ask me up to three questions about anything you need to know." Without that line, the AI would write the outline at once and fill every gap with a guess.
- The AI asks. It asks how many students and teachers are going, what the budget is, and whether there are food needs to plan for.
- You answer once. One message answers all three: 40 students and 4 teachers, a budget of 57,000 rupees, vegetarian meals only. Each answer replaces a guess.
- The outline. The outline now rests on your facts. You read it and approve it before the cost table is written.
- Options, not one answer. Once the total is 1,020 over budget, ask for choices instead of a single fix: "Give me three ways to save 1,020 rupees, with what each one changes for the students."
The three questions cost one short message and replaced three guesses, the head count, the budget and the meals. The options at the end leave the decision about the 1,020 rupees where it belongs, with the teacher.
Checkpoint
Check: Questions first, then options
A model’s review of its own work is not a check
Asking the AI to review its draft against a rubric of yours finds some problems: “Check the draft against these five points and list every one it fails; do not rewrite it.” It works best for parts that are visible on the page, such as a missing heading, the wrong tone or a point the brief asked for. It is weakest where the model’s own error is the problem. Huang and others found that without feedback from outside, models “struggle to self-correct their responses”, and that their performance sometimes got worse after trying. In Self-Refine, the maths answers barely improved, because the model’s feedback so seldom found an error; when an outside signal said which answers were wrong, the gains were much larger.
That is self-correction: Asking a model to review and revise its own answer. Without feedback from outside the model, such as a recalculation or a comparison with a source, models often miss their own mistakes.: useful for a first pass, never the last check for numbers or facts. Answer three questions about a step you want to check, and the table rules out the checks that cannot catch its slips:
Answer the questions above: the table then marks what fits your case and says why the rest does not.
| Check | Arithmetic slips | Claims the source lacks | Missing parts | Time it takes |
|---|---|---|---|---|
| Ask the AI to recheck its draft What it looks at: its own draft | Poor one model said "everything looks good" for 94 % of its maths answers | Poor rereads the same claims it wrote | Mixed may notice a missing part if it rereads the brief | Good seconds |
| AI review against your rubric What it looks at: its draft and your list of checks | Poor "totals are right" is still judged by the model that added them | Mixed can flag a claim with no quote, but not prove it | Good checks each part your rubric lists | Good seconds |
| Recompute with a calculator What it looks at: the numbers you gave | Good rebuilt the meals line: 21,120, not 19,120 | Poor checks numbers only | Poor says nothing about parts left out | Mixed a minute or two, or one run of a program |
| Compare with the source What it looks at: the document the facts came from | Mixed finds a wrong price, not a wrong product | Good finds a claim the source does not make | Mixed shows what the source has that the draft left out | Mixed minutes for a long source |
| Ask someone who knows What it looks at: their own knowledge of the task | Mixed only if they recompute | Good knows what a document leaves unsaid | Good knows what the readers will need | Poor their time, and a wait |
How the answers decide
- “Does the step produce numbers that someone will rely on?” Yes, totals, amounts or counts: rules out Ask the AI to recheck its draft and AI review against your rubric. Numbers need a recomputation. The trip table's lines added up to its own total and still hid a wrong meals line.
- “Do its claims come from a document, a rule or a price list?” Yes, from a source I have: rules out Ask the AI to recheck its draft and adds a caveat to AI review against your rubric. Check claims against the source itself; a review by the model that wrote them rereads the same claims.
- “Will the result go to someone outside your team?” Yes, parents, clients or the public: adds a caveat to Ask the AI to recheck its draft and AI review against your rubric. Anything that leaves your team gets a check from outside the model as well, a recomputation, the source or a person.
In short: numbers are recomputed, claims are compared with their source, and anything that leaves your team gets one check from outside the model, whatever the AI said about its own work.
Asking 'Are you sure?' as the check
Wrong: “Are you sure the total is right?” The same model judges the same sum, and a confident “yes” proves nothing. Right: recompute the total from the prices yourself, or ask for every line with its quantity and price and check them in a calculator.
Checkpoint
Check: What self-review can and cannot catch
Start a fresh chat with a summary when one drifts
Long chats go wrong in a particular way. Laban and others found that when models “take a wrong turn in a conversation, they get lost and do not recover”, and the two tips they give users are to try again in a new conversation and to consolidate everything said so far into one message before retrying. A drifting chat also carries its old mistakes with it, as the lesson on tokens, context windows and memory explained: the earlier messages are all read again with each new reply. Step through the move:
The drawing shows the whole scene; its 5 steps are listed below.
Every step as text
- A chat that drifted. Forty messages in, the budget chat has drifted. The outline changed twice, and an old bus price keeps coming back into the tables.
- Ask for one summary. "Please consolidate everything I have told you so far into one message: the decisions, the prices and the open questions."
- Read and correct it. The summary arrives as one message. You correct it yourself, the old bus price included, because the summary is the only thing that carries over.
- Start again with it. The corrected summary and the next request go into a new chat as its first message, so it starts with every decision and none of the wrong turns.
- Why it works. Laban and others found that models which take a wrong turn in a conversation tend not to recover, and that a new conversation repeating the same information can do much better.
The summary is the only thing that carries over, so read it and correct it before you paste it. Start fresh when the same mistake comes back after you corrected it, when replies contradict decisions already made, or when the chat has grown so long that its first messages no longer seem to count.
Checkpoint
Check: Starting fresh with a summary
Keep the prompts that work, and test them again
A prompt that worked once is worth keeping, with what you need to trust it again. A prompt library: A personal or team collection of prompts that worked, each kept with its purpose, the tool and model it was used with, when it was last tested and the test cases it passed. keeps each one with these fields:
| Field | For the trip budget |
|---|---|
| Name and purpose | Trip budget: costs and each student’s share, from given prices |
| The prompt | The four steps of the chain, with the prices left as a slot to fill |
| Tool and model | The app and the model it ran on, as the app names them |
| Last tested | The date of the last run of the test cases |
| Test cases | This trip, 58,020 rupees and 1,451 per student; a one-day trip with no stay |
| Checks | Each line is quantity times price; the total; each student’s share |
The test cases matter because prompts are fragile. Changing only a prompt’s formatting moved one model’s accuracy by up to 76 points in tests by Sclar and others, and across 20 models Mizrahi and others found results that change with the wording of the instruction. The model inside an app also changes, as the lesson on models, apps and AI features explained. So run the test cases again after any update, before you rely on the prompt. Here is how one teacher’s trip-budget prompt grew:
- v1
A one-line request
"Make a budget for a school trip." Without the prices, any reply has to invent them, so v1 is kept only as a note of what not to do.
- v2
A six-part brief brief
The job, the readers (the teacher and the principal), the background (40 students, 4 teachers, every price), the limits, a table of item, quantity, price and line total, and a test: every line total is quantity times price.
- v3
A chain of four prompts chain
Outline, cost table, check, note to parents. The check step rebuilds every line from the prices, which is how the meals slip of 2,000 rupees was caught.
- v4
Tagged, with two test cases tested
The prices sit inside <prices> tags, and two test cases are saved with the prompt: this trip (58,020 rupees, 1,451 per student) and a one-day trip with no stay, each with the totals it must produce.
- v5
Re-tested after an update re-tested
When the app moved to a newer model, the teacher ran both test cases again before using the prompt. A check that fails sends the prompt back for a fix, and the library notes what changed.
The timeline has 5 events, from v1 to v5.
Checkpoint
Check: A tested prompt library
Practise: check a picnic’s cost table
The exercise holds a picnic’s prices and a draft cost table from step 2 of a chain, with slips in it. Act as step 3: fix the numbers so that every line matches the prices and the head counts, then the total and the budget decision. The sample tests rebuild every line, as the lesson’s check did:
Exercise · Easy · JavaScript
Check a picnic's cost table against its prices
A Class 6 picnic is planned with a chain of prompts, and table.mjs holds step 2's draft cost table. You are step 3, the check. The facts the teacher gave:
- 30 students and 3 teachers, 33 people in all.
- Bus hire: 6,500 rupees for the day, one bus.
- Zoo entry: 40 rupees a student; teachers do not pay.
- Lunch: 110 rupees a person. Snacks: 30 rupees a person.
- The budget is 12,200 rupees.
Fix the numbers in table.mjs, and nothing else, so that:
- every
priceis the price given above; - every
quantitymatches the head count it needs; - every
totalisquantitytimesprice; grandTotalis the sum of the line totals;withinBudgetistrueonly if the grand total is at most the budget.
Notice that the draft already agrees with itself: its line totals add up to its grand total. The sample tests do what the lesson's check did, and rebuild every line from the facts above.
Starter code · table.mjs
// Step 2's draft cost table for the picnic, in rupees. Fix the numbers so the table matches the given facts.
export const lines = [
{ item: 'Bus hire', quantity: 1, price: 6500, total: 6500 },
{ item: 'Zoo entry', quantity: 33, price: 40, total: 1320 },
{ item: 'Lunch', quantity: 33, price: 110, total: 3330 },
{ item: 'Snacks', quantity: 33, price: 30, total: 990 },
];
export const grandTotal = 12140;
export const withinBudget = true;The sample tests · table.test.mjs
import { test, assert } from 'mysmartcopilot:test';
import { lines, grandTotal, withinBudget } from './table.mjs';
const STUDENTS = 30;
const PEOPLE = STUDENTS + 3;
const BUDGET = 12200;
// The facts the teacher gave: the price of each item and how many of it are needed.
const FACTS = {
'Bus hire': { price: 6500, quantity: 1 },
'Zoo entry': { price: 40, quantity: STUDENTS },
Lunch: { price: 110, quantity: PEOPLE },
Snacks: { price: 30, quantity: PEOPLE },
};
const line = (item) => lines.find((l) => l.item === item);
test('the four items are all there', () => {
for (const item of Object.keys(FACTS)) assert.ok(line(item), `the table has no line for ${item}`);
});
test('every price is the price given', () => {
for (const [item, fact] of Object.entries(FACTS)) {
assert.ok(line(item)?.price === fact.price, `${item}: the price given is ${fact.price}`);
}
});
test('every quantity matches the head count', () => {
for (const [item, fact] of Object.entries(FACTS)) {
assert.ok(line(item)?.quantity === fact.quantity, `${item}: how many people need it?`);
}
});
test('every line total is quantity times price', () => {
for (const l of lines) {
assert.ok(l.total === l.quantity * l.price, `${l.item}: ${l.quantity} x ${l.price} is not ${l.total}`);
}
});
test('the grand total adds up the lines', () => {
const sum = lines.reduce((n, l) => n + l.total, 0);
assert.ok(grandTotal === sum, `the lines add up to ${sum}, not ${grandTotal}`);
});
test('the budget decision follows from the total', () => {
assert.ok(withinBudget === grandTotal <= BUDGET, `a total of ${grandTotal} against a budget of ${BUDGET}: is it within?`);
});A hint
Rebuild each line by hand: how many people need it, times its price. Zoo entry is for the 30 students only, and lunch is 33 times 110. Then add the four line totals and compare the sum with 12,200.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (QuickJS): nothing is sent to mysmartcopilot.com. The first run downloads JavaScript (about 0.6 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
To finish, the self-check covers the six habits of this lesson and opens with the one your earlier answers point to:
Practice that adapts to you
Practise this lesson
16 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up (fresher to mid-level): Why is an AI’s review of its own draft not enough as a final check?
Because the same model that made a mistake is asked to find it. Asked to review its own work, a model often repeats its reasoning and approves it: in one study, a model’s feedback on its maths answers said “everything looks good” almost every time, and another study found that without outside feedback, models struggle to correct their reasoning and sometimes get worse. A self-review against your rubric is still useful for visible problems, such as a missing part, the wrong tone or the wrong layout. But numbers need recomputing, claims need comparing with their source, and anything that goes outside the team needs a check that does not depend on the model, such as a calculator, the document itself or a person who knows the subject.
Two workplace questions build on this lesson:
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Fresher to mid
An AI tool produced a 40-row event budget, then "double-checked" it and said it was correct, but the total was wrong. How would you set up the work so that this does not happen again?
-
Mid to senior
Your team keeps a shared document of prompts that work. After the AI tool updates, some of them start giving worse results. How would you organise and test the team's prompt library?
Key takeaways
- Feedback that names the change works better than “make it better”: specific feedback beat generic feedback on all three tasks of the Self-Refine comparison.
- A chain of small prompts with a check after each step lets you catch a slip before it spreads: the trip table agreed with itself, and only a rebuild from the prices found the meals line 2,000 rupees short.
- Asking the AI to question you first replaces guesses with your facts, and asking for options keeps decisions with you.
- A model’s review of its own work is a first pass, not a check: numbers are recomputed and claims compared with their source.
- When a chat drifts, consolidate the decisions into one corrected summary and start a new chat with it.
- Keep prompts that work in a library with test cases, and run them again after every update.
Terms in this lesson
- prompt chaining
- Splitting a task into a sequence of smaller prompts, in which the checked output of one step becomes the input of the next: for example an outline, then a draft, then a check, then the final version.
- prompt library
- A personal or team collection of prompts that worked, each kept with its purpose, the tool and model it was used with, when it was last tested and the test cases it passed.
- self-correction
- Asking a model to review and revise its own answer. Without feedback from outside the model, such as a recalculation or a comparison with a source, models often miss their own mistakes.
References
- Self-Refine: Iterative Refinement with Self-Feedback (NeurIPS (Madaan and others))
- Large Language Models Cannot Self-Correct Reasoning Yet (ICLR (Huang and others; arXiv))
- AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (arXiv (Wu, Terry and Cai))
- Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (ICLR (Zhou and others; arXiv))
- CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models (arXiv (Kuhn, Gal and Farquhar))
- LLMs Get Lost In Multi-Turn Conversation (arXiv (Laban, Hayashi, Zhou and Neville))
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design (ICLR (Sclar, Choi, Tsvetkov and Suhr; arXiv))
- State of What Art? A Call for Multi-Prompt LLM Evaluation (Transactions of the Association for Computational Linguistics (Mizrahi and others; arXiv))
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress