What AI does well and where it slips

The same chat app that drafts a polite leave request for you can miscount the letter a in “Mahabaleshwar”, which has four. Researchers who tested many models on a long list of words found that most of them could not count a letter that appears more than twice in a word, while the same models write fluent paragraphs on almost any subject.

You will sort everyday tasks into those AI tools usually handle well and those that need checking, see why their ability is uneven, predict where an answer is likely to slip (counts, sums, dates, and rare, recent or local facts), and pick the check that catches each kind of slip before you rely on the answer.

AI tools are strongest at language work you can judge

Two large studies measured what happens when people do real work with a chat model. In one, 453 college-educated professionals did writing tasks from their own jobs: those with the tool took 40 % less time and produced work rated 18 % better. In another, 5,172 customer-support agents with an AI assistant resolved 15 % more issues an hour on average; the least experienced gained most, while the most experienced gained a little speed and lost a little quality.

These tasks have three things in common: the input and the output are language, many different answers are good enough, and a person can judge the result by reading it. Work of that shape is where a fast, fluent writer helps most. Work that has exactly one right answer, which the reader cannot see by reading, is where it needs checking:

The legal figures come from Dahl and others, who asked models verifiable questions about real court cases. The textbook Speech and Language Processing adds that language models work worse in languages other than English, presumably because English dominates their training data, so the translation row holds best for language pairs with a lot of text behind them.

Ability is uneven, even between tasks that look alike

You might expect a tool that writes a good market report to manage anything easier than that. A preregistered experiment with 758 consultants of a global firm shows why that expectation fails: Dell’Acqua and others report two opposite results for tasks that looked similarly hard:

The authors call the edge of what AI does well a jagged technology frontierjagged frontier: The uneven edge of what AI does well. Tasks that look equally hard to a person can fall inside it, where AI help improves the work, or outside it, where the same help makes the work worse.: it does not follow how hard a task looks to a person, so two tasks that seem alike can fall on opposite sides of it. Outside the frontier, people who worked with AI did worse than people who worked alone.

The opening example is the same effect in miniature. Letter counting is trivial for a person and slips for a model, while a fluent paragraph on a technical subject, which takes a person real effort, is routine for it. Difficulty for people is a poor guide to difficulty for models, so the only way to know where your task falls is to try the tool on cases whose answers you already know.

Letters slip because models read tokens, not letters

A model never sees the letters you typed. A tokenizertokenizer: The program that cuts text into tokens before a model reads it, and turns tokens back into text afterwards. Its pieces are learned from its own training text, so they differ from model to model. More in “Tokens, context windows and AI memory” first cuts your text into tokenstoken: A piece of text that a model reads or writes as one unit: often a whole common word, sometimes part of a longer word, a punctuation mark or a single byte of a rarer character. More in “Tokens, context windows and AI memory”: Hugging Face’s guide to tokenization notes that common words stay whole while rare words are split into smaller pieces, and the lesson on tokens shows how those pieces are learned. Each piece then becomes a number. Follow one question from your keyboard to the reply:

To count the a’s, the model would need the spelling of every piece and a running tally of the letters. Fu and others found that models recognise letters but cannot count them reliably, with errors growing with the number of letters in a word and most models failing when a letter appears more than twice. The CUTE benchmark by Edman, Schmid and Fraser found the same split: models seem to know how their tokens are spelled, yet fail to use that knowledge when they must change a word letter by letter, by inserting, deleting or swapping letters. Writing a word backwards is the same kind of task.

The program below works out eight quick tasks of the kinds that slip, letter by letter and digit by digit. Ask any AI app the same eight questions in a new chat and mark its answers against this table:

"""Eight quick tasks of the kinds chat models slip on, each answered exactly.

A chat model reads tokens, pieces of words, and writes a likely answer. These
tasks need letters, digits or a calendar handled exactly, so this program
works each one out step by step. Ask an AI app the same eight questions and
mark its answers against the table.
"""
from datetime import date


def count_letter(word, letter):
    return word.lower().count(letter)


def cab_fare(km, first_km=2, first_fare=50, per_km=18):
    """50 rupees for the first 2 km, then 18 rupees for every km after."""
    return first_fare + max(0, km - first_km) * per_km


independence_day = date(1947, 8, 15)

TASKS = [
    ("counting", 'How many times does "a" appear in "Mahabaleshwar"?',
     count_letter("Mahabaleshwar", "a")),
    ("counting", 'How many times does "r" appear in "Tiruchirappalli"?',
     count_letter("Tiruchirappalli", "r")),
    ("counting", 'How many letters are in "Thiruvananthapuram"?',
     len("Thiruvananthapuram")),
    ("letters", 'Write "Kanyakumari" backwards.',
     "Kanyakumari"[::-1].lower()),
    ("sums", "What is 347 x 286?",
     f"{347 * 286:,}"),
    ("sums", "What is 4,857 x 3,926?",
     f"{4857 * 3926:,}"),
    ("calendar", "What weekday was India's first Independence Day?",
     independence_day.strftime("%A")),
    ("given numbers", "50 rupees for 2 km, then 18 a km: a 13.5 km fare?",
     f"{cab_fare(13.5):g} rupees"),
]

for number, (kind, question, answer) in enumerate(TASKS, start=1):
    print(f"{number}. [{kind}] {question}")
    print(f"   Exact answer: {answer}")

print()
print("Mahabaleshwar, letter by letter:", " ".join("Mahabaleshwar"))
print("Positions of a:", [i + 1 for i, ch in enumerate("mahabaleshwar")
                          if ch == "a"])
digits = len("347") * len("286")
print(f"347 x 286 by hand: {digits} single-digit products, then carries")

1. [counting] How many times does "a" appear in "Mahabaleshwar"?
   Exact answer: 4
2. [counting] How many times does "r" appear in "Tiruchirappalli"?
   Exact answer: 2
3. [counting] How many letters are in "Thiruvananthapuram"?
   Exact answer: 18
4. [letters] Write "Kanyakumari" backwards.
   Exact answer: iramukaynak
5. [sums] What is 347 x 286?
   Exact answer: 99,242
6. [sums] What is 4,857 x 3,926?
   Exact answer: 19,068,582
7. [calendar] What weekday was India's first Independence Day?
   Exact answer: Friday
8. [given numbers] 50 rupees for 2 km, then 18 a km: a 13.5 km fare?
   Exact answer: 257 rupees

Mahabaleshwar, letter by letter: M a h a b a l e s h w a r
Positions of a: [2, 4, 6, 12]
347 x 286 by hand: 9 single-digit products, then carries

Mahabaleshwar has its a’s at positions 2, 4, 6 and 12, which any program finds exactly. Two apps can give different answers to these questions: one that runs code behind the scenes lets a program count, while one that only writes text gives you the model’s guess.

Long sums and dates slip one step at a time

Multiplication by hand is many small steps. The program’s 347 x 286 needs 9 single-digit products and the carries between them, and 4,857 x 3,926 needs 16. A model writes the answer token by token, so a step it gets wrong anywhere in that chain spoils the product. Dziri and others found that the models they tested reached only 55 % and 59 % accuracy on three-digit by three-digit multiplication, and that performance fell from near perfect to zero as problems grew. They argue that errors accumulate with every step of a problem built from smaller ones.

Here is a simple model of that argument. Suppose each single-digit product is right 97 % of the time, an assumption chosen only to show the shape. The figure starts at the 9 products of 347 x 286 and gives the chance that all 9 are right; set the tries to 16 and then 25, keeping “at least” equal to the tries:

With 9 steps the chance that all are right is about 0.76; with 16 it falls to about 0.61, and with the 25 steps of a five-digit by five-digit product to about 0.47. Even a small error per step makes long sums unreliable. How numbers are cut into tokens matters too: Singh and Strouse found that writing numbers with commas, which changes how they are split, largely improved one family of models’ arithmetic.

Dates are sums in disguise. “Which weekday was India’s first Independence Day?” means counting days across years, leap years and months of different lengths; the program’s answer is Friday.

Rare, recent and local facts are the weakest

A model answers a factual question from patterns in its training text, so it does best on facts that text repeats. Kandpal and others found that a model’s accuracy on a factual question tracks how many documents about that fact it saw in training, and Mallen and others found that making models larger barely helps with such long-tail factslong-tail fact: A fact that appears in few documents, such as a small town's bus fare or a minor local rule. Models answer such facts far less reliably than facts that are written about often., while models that look up documents do far better on them. Facts can be weak spots for three more reasons: they happened after the model’s training cutofftraining cutoff: The point in time up to which a model's training text was collected. The model knows nothing later unless an app adds it to the input, for example from a web search. More in “How chatbots write: predicting the next token”, they are local and change often, or they depend on details of your case that you never gave. Here is each kind of slip with its example and its check:

Match the check to the way an answer can fail

Each kind of slip has a check that catches it, and reading the answer again catches almost none of them. Pick one AI answer you want to use, answer the five questions about it, and the table rules out the checks that cannot catch its likely slips:

In short: recompute numbers, run code and formulas on cases you know, compare a summary with the text you gave, look up recent, local or rare facts with the body that issues them, and bring in a qualified person when health, money or law is at stake. Reading it again is the check for wording, tone and ideas, and for nothing else.

Practise: sort six task cards

Six task cards from an ordinary week. For each, decide whether it is likely fine with a read-through, needs a check before use, or should not be handed to an AI tool at all; every answer says why:

Then try the lesson’s self-check, which asks first about whichever of its six ideas has given you the most trouble:

Interview questions

Warm-up (fresher to mid-level): Which tasks can you usually hand to an AI tool, and which need close checking?

Hand it work whose result you can judge by reading or check quickly: drafting and rewriting, summarising a text you supply, translating between widely used languages, explaining a concept, brainstorming and reformatting. Large studies found such writing and support work done faster and often better with AI help. Check closely anything with exactly one right answer that reading cannot reveal: counts, sums and dates, because the model reads tokens and writes likely text rather than computing; rare, recent or local facts such as fees, rules and timetables, which may be missing from its training text or out of date; and legal, medical and money specifics, where errors are costly. Ability is uneven, so test the tool on cases whose answers you know before trusting it with a new kind of task, and match the check to the slip: recompute, test, compare with the source, or ask a qualified person.

Two workplace questions build on this lesson:

Key takeaways

• AI tools help most with language work you can judge by reading: drafting, summarising text you supply, translating between widely used languages, explaining and reformatting.
• Ability is uneven: in a study of 758 consultants, AI help made 18 tasks faster and better but made people 19 % less likely to solve one task that looked just as hard.
• Letters slip because a model reads tokens, pieces of words turned into numbers, and writes a likely answer instead of counting.
• Long sums and dates slip because they are many small steps, and one wrong step anywhere spoils the result.
• Rare, recent and local facts, such as fees, rules and timetables, are the weakest, and so is anything that depends on details of your case.
• Match the check to the slip: recompute, test, compare with your text, use the official source, or ask a qualified person; reading again checks only the wording.