Studying with AI without losing skills

In a field experiment with nearly 1,000 high-school maths students, those who practised with an ordinary AI chatbot scored 48 % higher on their practice problems, and then 17 % lower than students who never had it, once the chatbot was gone for the exam. Students given an AI tutor that asked and hinted instead of answering did as well on the exam as those who never had AI.

This lesson shows you how to get the tutor and avoid the crutch. You will write prompts that make an AI tool ask questions, give hints and check your working instead of answering for you; see what the research found and why; plan a week in which your own attempts and recall from memory come first; and spot the moment when AI help starts to replace the thinking you need in order to learn.

Help that raises practice scores can lower exam scores

The experiment, by Bastani and others, ran in four class sessions of 90 minutes each at a large high school. Some students practised with a plain chatbot; others had a tutor version, whose prompt held each problem’s solution, the common mistakes and hints for them, and an instruction not to give the whole solution away. Then everyone took an exam on similar problems with no AI at all (Bastani and others). Step through both routes:

The difference was not the tool’s intelligence; both used the same model. It was what the students did with it. With the plain chatbot, students most often asked for the answer and copied it, so practice went well and the skill never formed; the authors call it using AI as a “crutch”. With the tutor, students attempted answers and asked for help, so the thinking stayed theirs. The plain chatbot was also often wrong: asked ten times for the answer to each of the 57 practice problems, it was right only 51 % of the time, so copying taught wrong methods as well.

The last step of the figure is the most useful warning. Students in both groups believed they had learned more than the exam showed. A session in which AI did the hard part feels productive, because everything got done; whether you learned anything shows only when the help is gone. Students in this study were at school, but the trap is the same for anyone learning a skill, at college or at work.

A tutor prompt asks for questions and hints, not answers

Any chat tool can behave like the study’s tutor if you ask it to. A tutor prompttutor prompt: A prompt that asks an AI tool to teach rather than answer: to ask one question at a time, give hints instead of the answer and point to the first mistake in your working without fixing it. sets the rules before the problem: ask me one question at a time, give a hint and not the answer, point to the first mistake in my working. The difference from an answer prompt is in what it asks the tool to do:

If you have the worked solution or the marking scheme from your course, give it to the tutor with the instruction not to reveal it, as the study’s tutor had: its hints then come from a correct solution. The exchange below, written for this lesson, works one practice problem of a first statistics course, the chance of at least one six in four rolls of a die:

The tutor asked, nudged and confirmed; the student did every step and could explain the mistake afterwards. The last step matters too: a tutor’s confirmation is not a check, so the answer is checked another way. Counting every case is the surest check when the cases can be listed:

"""Check an answer without asking the AI again: count every case.

The practice problem of the lesson: what is the chance of at least one six in
four rolls of a fair die? A common first attempt adds 1/6 four times. The
program lists all 6 x 6 x 6 x 6 = 1,296 equally likely results, counts those
with at least one six, and compares the count with both methods.
"""
from fractions import Fraction
from itertools import product

results = list(product(range(1, 7), repeat=4))
with_six = sum(1 for r in results if 6 in r)
count = Fraction(with_six, len(results))

added = 4 * Fraction(1, 6)
complement = 1 - Fraction(5, 6) ** 4

print(f"all results: {len(results)}, with at least one six: {with_six}")
print(f"counted:       {count} = {float(count):.4f}")
print(f"1 - (5/6)^4:   {complement} = {float(complement):.4f}")
print(f"4 x 1/6:       {added} = {float(added):.4f}")
twice = sum(1 for r in results if r.count(6) >= 2)
print(f"results with two or more sixes: {twice}")
print(f"adding 1/6 four times counts those results more than once, so it is too high by {float(added - count):.4f}")

all results: 1296, with at least one six: 671
counted:       671/1296 = 0.5177
1 - (5/6)^4:   671/1296 = 0.5177
4 x 1/6:       2/3 = 0.6667
results with two or more sixes: 171
adding 1/6 four times counts those results more than once, so it is too high by 0.1489

All 1,296 results of four rolls, counted one by one, give 671 with at least one six: 0.5177, the same as 1 − (5/6)⁴. Adding 1/6 four times gives 2/3 = 0.6667, too high by 0.1489, because it counts each of the 171 results with two or more sixes more than once. A count like this does not depend on the AI being right.

Practise first, recall from memory, then let AI check

What builds a skill is effort of the right kind: trying before you are shown, and recalling from memory instead of reading again. In a classic experiment, students who read a text and then recalled it from memory remembered it better after two days and after a week than students who read it again, although rereading worked better five minutes later and made students more confident (Roediger and Karpicke). When Dunlosky and colleagues weighed the evidence for ten common techniques, two got their highest rating, testing yourself and spreading practice over time, while rereading, highlighting and summarising were among those that got their lowest (Dunlosky and others). The two habits are known as retrieval practiceretrieval practice: Learning by recalling what you know without looking, such as answering questions with your notes closed. It strengthens memory more than reading the same material again. and spaced practicespaced practice: Practice spread over several sessions on different days, so that each topic comes back after a gap, instead of one long session..

So AI’s place in a study week is after your own effort: to check your working, to quiz you on what you learned days ago, and to explain what you got wrong. Walk through such a week for the two topics of the statistics course:

The program below checks two plans for the same week against five rules: your own try before any AI help, no answer prompts, recall on two days or more, each topic worked on two days or more, and a closed-book test at the end. Both plans were written for this lesson, and both take 300 minutes:

"""Check a week's study plan against five rules from the lesson.

Each row of a plan is one session: the day, the topic ("both" for the two
topics), the activity and how AI is used (none, tutor or answer). Activities:
learn (read the course material), practise (solve problems), check (an AI tutor
reads your working), quiz and test (recall without notes), reread. The two
plans were written for this lesson and take 300 minutes each.
"""
import csv
from pathlib import Path

TOPICS = ["probability", "statistics"]
ACTIVE = {"practise", "check", "quiz", "test"}


def topics_of(row):
    return TOPICS if row["topic"] == "both" else [row["topic"]]


def check(rows):
    results = []
    late = [t for t in TOPICS
            if next((r for r in rows if t in topics_of(r) and r["activity"] in {"practise", "check"}), {}).get("ai") != "none"]
    results.append(("your own try before any AI help", not late, f"{', '.join(late)} began with AI help"))
    answers = [r for r in rows if r["ai"] == "answer"]
    results.append(("no answer prompts", not answers, f"{len(answers)} sessions ask for answers"))
    recall_days = {r["day"] for r in rows if r["activity"] in {"quiz", "test"}}
    results.append(("recall on two days or more", len(recall_days) >= 2, f"recall on {len(recall_days)} days"))
    thin = [t for t in TOPICS if len({r["day"] for r in rows if t in topics_of(r) and r["activity"] in ACTIVE}) < 2]
    results.append(("each topic worked on two days", not thin, f"{', '.join(thin)} on fewer days"))
    last = rows[-1]
    results.append(("a closed-book test at the end", last["activity"] == "test" and last["ai"] == "none", "no final test"))
    return results


here = Path(__file__).parent
for n, name in enumerate(["plan_a.csv", "plan_b.csv"]):
    if n:
        print()
    with open(here / name, encoding="utf-8", newline="") as f:
        rows = list(csv.DictReader(f))
    minutes = sum(int(r["minutes"]) for r in rows)
    print(f"{name}: {len(rows)} sessions, {minutes} minutes")
    for rule, ok, why in check(rows):
        print(f"  {rule:<34} {'ok' if ok else 'FAIL: ' + why}")

plan_a.csv: 7 sessions, 300 minutes
  your own try before any AI help    FAIL: probability, statistics began with AI help
  no answer prompts                  FAIL: 2 sessions ask for answers
  recall on two days or more         FAIL: recall on 0 days
  each topic worked on two days      FAIL: probability, statistics on fewer days
  a closed-book test at the end      FAIL: no final test

plan_b.csv: 9 sessions, 300 minutes
  your own try before any AI help    ok
  no answer prompts                  ok
  recall on two days or more         ok
  each topic worked on two days      ok
  a closed-book test at the end      ok

day,topic,activity,ai,minutes
1,probability,learn,none,40
1,probability,practise,answer,40
2,probability,reread,none,30
3,statistics,learn,none,40
3,statistics,practise,answer,40
5,statistics,reread,none,30
6,both,reread,none,80

day,topic,activity,ai,minutes
1,probability,learn,none,40
1,probability,practise,none,40
2,probability,check,tutor,30
3,statistics,learn,none,40
3,statistics,practise,none,40
4,statistics,check,tutor,30
4,probability,quiz,tutor,20
6,statistics,quiz,tutor,20
7,both,test,none,40

Plan A fails all five rules: it asks AI for the answers, then rereads, and never recalls anything from memory. Plan B passes all five in the same 300 minutes. The difference is not time; it is the order of the work and who does it.

Signs that AI is doing the thinking for you

The crutch is easy to slip into, because each step of it feels reasonable. Watch for these signs:

• you reach for the AI before you have tried the problem yourself;
• you cannot solve a similar problem the next day without it;
• you accept an explanation that you could not repeat in your own words;
• your practice scores rise while your marks on tests without AI do not;
• you feel you have learned a topic, but cannot recall its main points with your notes closed.

Each sign has the same fix: put your own effort back first, and give the AI the job of a tutor. Which use of AI fits depends on where you are with a problem. Answer three questions about the one in front of you:

In short: before your own try, ask for a hint or the idea explained, never the answer; after it, ask for a check of your working or a quiz; and when you will be tested without AI, do the thinking now, because the study’s plain-chatbot group scored 17 % lower once the help was gone.

Practise: plan a study week

The exercise gives you a student’s plan for a week before a closed-book test: it asks AI for the answers and spends the rest of the week rereading. Rewrite it into a week that builds the skill. The sample tests hold it to the rules of this lesson: your own try before any AI help, no answer prompts, tutor prompts that ask for questions, hints or a check, recall on two days or more, each topic on two days or more, and a test without AI at the end. They cannot tell whether the plan fits a real week; that judgement is yours.

// The student's study week: one line per session. Day | topic | activity | AI use | prompt (or -).
// Activities: learn, practise, check, quiz, test, reread. AI use: none, tutor or answer.
export const plan = `Day 1 | probability | learn | none | -
Day 1 | probability | practise | answer | Solve these problems and give me the answers
Day 2 | probability | reread | none | -
Day 3 | statistics | learn | none | -
Day 3 | statistics | practise | answer | Do these questions for me
Day 6 | both | reread | none | -`;

import { test, assert } from 'mysmartcopilot:test';
import { plan } from './week.mjs';

// The plan's lines as sessions, held to the rules of the lesson.
const TOPICS = ['probability', 'statistics'];
const ACTIVITIES = ['learn', 'practise', 'check', 'quiz', 'test', 'reread'];
const rows = () =>
  String(plan)
    .split('\n')
    .map((l) => l.trim())
    .filter(Boolean)
    .map((l) => {
      const [day = '', topic = '', activity = '', ai = '', prompt = ''] = l.split('|').map((s) => s.trim());
      return { line: l, day: Number(day.replace(/^day\s*/i, '')), topic, activity, ai, prompt };
    });
const about = (r, t) => r.topic === t || r.topic === 'both';
const days = (list) => new Set(list.map((r) => r.day)).size;

test('every line has its five parts', () => {
  for (const r of rows()) {
    assert.ok(Number.isInteger(r.day) && r.day >= 1 && r.day <= 7, `"${r.line}": the day is a number from 1 to 7`);
    assert.ok([...TOPICS, 'both'].includes(r.topic), `"${r.line}": the topic is probability, statistics or both`);
    assert.ok(ACTIVITIES.includes(r.activity), `"${r.line}": the activity is one of ${ACTIVITIES.join(', ')}`);
    assert.ok(['none', 'tutor', 'answer'].includes(r.ai), `"${r.line}": the AI use is none, tutor or answer`);
  }
});

test('your own try before any AI help', () => {
  for (const t of TOPICS) {
    const first = rows().filter((r) => about(r, t) && ['practise', 'check'].includes(r.activity)).sort((a, b) => a.day - b.day)[0];
    assert.ok(first && first.activity === 'practise' && first.ai === 'none', `${t}: start with a practise session with no AI, before any AI help`);
  }
});

test('no session asks for answers', () => {
  const bad = rows().filter((r) => r.ai === 'answer');
  assert.ok(bad.length === 0, `${bad.length} sessions ask AI for answers: use none or tutor`);
});

test('tutor prompts ask for questions, hints or a check', () => {
  for (const r of rows().filter((x) => x.ai === 'tutor')) {
    assert.ok(/hint|question|mistake|quiz|check/i.test(r.prompt), `"${r.prompt}": ask for questions, a hint or a check of your working`);
    assert.ok(!/give me the answers?|tell me the answers?|solve (it|this|these)|do (it|this|these)( questions)? for me|write (it|my)/i.test(r.prompt), `"${r.prompt}": a tutor prompt does not ask for the answer`);
  }
});

test('recall without notes on two days or more', () => {
  const n = days(rows().filter((r) => ['quiz', 'test'].includes(r.activity)));
  assert.ok(n >= 2, `quizzes and tests fall on ${n} days: put recall on two days or more`);
});

test('each topic worked on two days or more', () => {
  for (const t of TOPICS) {
    const n = days(rows().filter((r) => about(r, t) && ['practise', 'check', 'quiz', 'test'].includes(r.activity)));
    assert.ok(n >= 2, `${t} is practised, checked or recalled on ${n} day(s): give it two days or more`);
  }
});

test('a closed-book test at the end', () => {
  const list = rows();
  const last = list[list.length - 1];
  assert.ok(last && last.activity === 'test' && last.ai === 'none', 'end the week with a test that uses no AI');
});

Then test yourself on the research, tutor prompts, study routines and the warning signs. The questions begin where your answers have been weakest:

Interview questions

Warm-up (fresher to mid-level): What is the difference between asking an AI tool for the answer and using it as a tutor, and why does it matter for learning?

Asking for the answer hands the AI the thinking: you get a result, and the skill is never practised. Using it as a tutor keeps the thinking yours: you set rules such as one question at a time, hints instead of answers and your first mistake pointed out but not fixed, and you do every step. It matters because learning comes from the effort of trying and recalling. In a field experiment with nearly 1,000 students, those who practised with a plain chatbot mostly copied its answers and later scored 17 % lower on an exam without AI than students who never had it, while a tutor that gave hints instead of answers avoided that loss. The plain chatbot was also wrong almost half the time, so any final answer still needs checking another way.

Two workplace questions build on this lesson:

Key takeaways

• AI help can lift practice scores and still leave you worse off without it: in the field experiment, the plain-chatbot group scored 17 % lower on the exam, while the hint-giving tutor’s group did not.
• A tutor prompt sets the rules first, questions one at a time, hints instead of answers, your first mistake pointed out, and any final answer is checked another way, such as by counting.
• Build the week on your own effort: try first, recall from memory on later days, and let AI check and quiz you afterwards; recall beat rereading after two days and a week.
• Watch for the signs of the crutch, such as reaching for AI before trying or failing a similar problem without it, and put your own attempt back first.