AI search and deep-research tools

When Liu, Zhang and Liang audited the answers of four AI search tools, only 51.5 % of the sentences were fully supported by their citations, and only 74.5 % of the citations supported the sentence they were attached to (Liu and others). A footnote beside a sentence looks like evidence. About one time in four it was not.

This lesson shows how an AI search answer and a long deep-research report are put together from the pages a tool retrieves, and where each goes wrong. You will audit an answer citation by citation, recognise old, copied and weak sources, size a sample check of a long report with a model you can change, and decide when to stop reading the AI and read the originals yourself.

An AI search answer is written from the pages it retrieved

An AI search tool does two jobs in a row. First it searches: it turns your question into queries and gets back pages, much as a search engine would. Then a language model writes an answer from passages of those pages and attaches links to its sentences. Researchers call this retrieval-augmented generationretrieval-augmented generation: A way of building AI answers in two steps: a search first retrieves documents or web pages, then a model writes the answer from passages of them, often with links to the pages it used.; Lewis and others found that models writing from retrieved text were more specific and more factual than models writing from memory alone. Follow one question through the two jobs:

Each job has its own way to fail. Retrieval can bring back the wrong pages: an old blog instead of the department’s page, a copy instead of the original. Writing can misreport a good page: a figure that the page does not give, or a sentence that no page supports at all. A citation tells you which page the tool used, not which job went right.

A deep-research tooldeep-research tool: An AI tool that plans many searches in a loop, reads what each one returns and writes a long report with footnotes. Its report rests on retrieved pages, like any AI search answer. runs the same two jobs many times in a loop. Huang and others describe such agents as combining planning, searches that build on each other, repeated tool use and “the generation of structured analytical reports” (Huang and others). The result is long, organised and full of footnotes, and it rests on the same retrieved pages, many more of them. Java and others, who built a benchmark of 100 such tasks, argue that the defining feature is broad searching, not report length; the best system they tested scored 0.55 on a scale on which 1 means every expected key claim found and nothing wrong (Java and others).

A citation can sit beside a sentence it does not support

The Tow Center for Digital Journalism ran 1,600 queries through eight AI search tools, asking each to identify the source of a news extract. Together they answered more than 60 % of the queries incorrectly, were more likely to give a wrong answer than to admit they did not know, and sometimes cited copied versions of articles instead of the original publisher (Tow Center). The paid versions they tested gave confidently wrong answers more often than the others.

So audit an answer one sentence at a time, with four questions for each citation: does the page say what the sentence says, how old is the page, is it a primary source, and is it independent or a copy of another page? The program below audits the scholarship answer from the figure above; the answer and the three pages it read were written for this lesson and saved, so it runs without a network:

"""Audit an AI search answer: does each cited page support its sentence?

A student asked an AI search tool whether she can still apply for a state
merit scholarship, and what the income limit is. The tool read three pages
and wrote a five-sentence answer with citations. The pages and the answer are
examples written for this lesson, saved here so the program runs without a
network.

For each sentence the program asks the questions of a citation audit:
  1. Does the cited page say what the sentence says?
  2. Is the page the newest source on the point, or has a newer one changed it?
  3. Is it a primary source, or does it only repeat another page?
"""

PAGES = {
    1: {"site": "the scholarship department's page", "year": 2026,
        "primary": True, "repeats": None,
        "text": "Applications for the merit scholarship are open until "
                "30 November. The family income limit is 8,00,000 rupees "
                "a year. Students must have scored at least 60 per cent in "
                "their last examination."},
    2: {"site": "a coaching centre's blog", "year": 2024,
        "primary": False, "repeats": None,
        "text": "The merit scholarship's family income limit is 6,00,000 "
                "rupees a year, and applications close on 31 October."},
    3: {"site": "a news site", "year": 2025,
        "primary": False, "repeats": 2,
        "text": "According to a popular coaching blog, the scholarship's "
                "family income limit is 6,00,000 rupees a year."},
}

# sentence, cited page, the point it makes, what it says on that point
ANSWER = [
    ("Applications are open until 30 November.", 1,
     "applications", "30 November"),
    ("The family income limit is 6,00,000 rupees a year.", 2,
     "income limit", "6,00,000 rupees"),
    ("News reports confirm the 6,00,000 rupee limit.", 3,
     "income limit", "6,00,000 rupees"),
    ("Students need at least 75 per cent in their last examination.", 1,
     "at least", "75 per cent"),
    ("Late applications are accepted with a fee.", 1,
     "late applications", "with a fee"),
]


def sentence_about(text, point):
    for sentence in text.split(". "):
        if point in sentence.lower():
            return sentence.rstrip(".")
    return None


def newest_primary(point):
    """The newest primary page that speaks to the point, if any."""
    found = [(p["year"], n) for n, p in PAGES.items()
             if p["primary"] and sentence_about(p["text"], point)]
    return max(found)[1] if found else None


sound = 0
for i, (sentence, cited, point, says) in enumerate(ANSWER, start=1):
    page = PAGES[cited]
    print(f"{i}. {sentence} [{cited}]")
    print(f"   page {cited}: {page['site']}, {page['year']}, "
          f"{'primary' if page['primary'] else 'secondary'}")
    passage = sentence_about(page["text"], point)
    if passage is None:
        print("   DOES NOT SUPPORT: the page says nothing on this point\n")
        continue
    if says not in passage:
        print(f'   DOES NOT SUPPORT: the page says "{passage}"\n')
        continue
    flags = []
    best = newest_primary(point)
    if best and best != cited:
        newer = sentence_about(PAGES[best]["text"], point)
        if says not in newer:
            flags.append(f"OUTDATED: page {best} "
                         f"({PAGES[best]['year']}, primary) says "
                         f'"{newer}"')
    if page["repeats"]:
        flags.append(f"NOT INDEPENDENT: it repeats page {page['repeats']}")
    if flags:
        print("   supports the words, but")
        for flag in flags:
            print(f"   {flag}")
        print()
    else:
        sound += 1
        print("   SUPPORTS it, from a current primary source\n")

print(f"Sound citations: {sound} of {len(ANSWER)} sentences")

1. Applications are open until 30 November. [1]
   page 1: the scholarship department's page, 2026, primary
   SUPPORTS it, from a current primary source

2. The family income limit is 6,00,000 rupees a year. [2]
   page 2: a coaching centre's blog, 2024, secondary
   supports the words, but
   OUTDATED: page 1 (2026, primary) says "The family income limit is 8,00,000 rupees a year"

3. News reports confirm the 6,00,000 rupee limit. [3]
   page 3: a news site, 2025, secondary
   supports the words, but
   OUTDATED: page 1 (2026, primary) says "The family income limit is 8,00,000 rupees a year"
   NOT INDEPENDENT: it repeats page 2

4. Students need at least 75 per cent in their last examination. [1]
   page 1: the scholarship department's page, 2026, primary
   DOES NOT SUPPORT: the page says "Students must have scored at least 60 per cent in their last examination"

5. Late applications are accepted with a fee. [1]
   page 1: the scholarship department's page, 2026, primary
   DOES NOT SUPPORT: the page says nothing on this point

Sound citations: 1 of 5 sentences

One citation of five is sound. Two pages support their sentences’ words but are out of date, and one of those also repeats the other. Two sentences cite the department’s page for things it does not say: it asks for 60 per cent, not 75, and it says nothing at all about late applications. The fifth sentence is the kind of slip a reader never checks, because it carries the most trustworthy link in the answer.

Old, copied and weak sources look as good as current ones

An answer’s citations all look the same: a number in brackets, a site name, a link. The pages behind them do not. In the scholarship answer the income limit seems to have two sources behind it, a coaching blog and a news item. Step through where each came from:

Three things went wrong, and the citations showed none of them:

• Out of date. The blog described an earlier year’s rules. Fees, limits, dates and eligibility often change from year to year, and a page keeps saying the old figure long after it changed.
• Copied. The news item repeated the blog. Copies add no confirmation, however many sites repeat a figure. Circular sourcingSources that seem to confirm one another but all trace back to the same original, so that a single claim appears to have several independent sources. is easy to miss because each copy looks independent.
• Weak. A page with no author, no date and no source for its own figures is a lead at best.

NIST’s profile of generative AI expects information of high integrity to trace back to its original source and to say how long it stays valid: it “creates reasonable expectations about when its validity may expire” (NIST AI 600-1). Ask the same of every citation: when was the page written, does it name where its figure comes from, and is there a newer primary page on the same point?

A long report needs a sample check, not trust

A deep-research report may have 40 footnotes. Checking every one takes as long as writing the report, and checking none makes it a guess with formatting. A sample check sits between the two, and its strength can be worked out. Say a report’s footnotes fail at the rate Liu and others measured, about one in four. The model below gives the chance that checking 8 footnotes picked at random finds at least one bad one; here a “success” is a check that finds a bad footnote:

At one bad footnote in four, 8 checks find at least one with a chance of 0.8999, about nine times in ten, so a sloppy report is caught quickly. Drag the chance of a bad footnote down to 0.1 and the same 8 checks find one only 0.5695 of the time; at 0.05, only 0.3366. A clean sample does not prove a report is clean. The model treats every check as an independent draw, as if the report had far more footnotes; picking 8 different footnotes out of 40 makes a catch a little more likely, so its chances lean to the cautious side.

Use the sample in two ways. Check every footnote behind a claim you will act on, whatever the sample says. Then check a random handful of the rest: if one fails, check them all or put the report aside. And ask what a good report on the question should have cited, such as the official page of the body that decides, and look for it in the footnotes.

Read the originals before you act

An AI search answer is a good way to find sources and a poor way to decide. Stop and read the original sources yourself when any of these signals appears; the scholarship answer showed four of the five:

When you read, start from the primary source: the department’s own page, the published rules, the judgment, the study. Use the AI answer as a list of leads, and judge each lead as the lesson on a risk-based routine for checking AI shows.

Practise: audit two more answers

Each question gives one cited sentence and what its page says. Decide whether the citation holds; every answer says why:

Then a self-check on how AI search is built, citations, sources, sampling and originals. It opens with whichever of them your answers show is weakest:

Interview questions

Warm-up (fresher to mid-level): How does an AI search answer differ from a list of search results, and what does that mean for checking it?

A list of search results gives you pages, and you read them and decide. An AI search answer adds a second step: a model reads passages of the pages the search found and writes sentences from them, with links attached. That makes it faster to read and easier to trust than it deserves, because each link shows which page was used, not whether the page supports its sentence. An answer can fail in retrieval, by choosing an old, copied or weak page, or in writing, by stating what no page says. So you check it citation by citation: does the page say it, how old is it, is it primary, and is it independent? For anything you will act on, read the original source yourself.

Two workplace questions build on this lesson:

Key takeaways

• An AI search answer is retrieval followed by writing, and it can fail in either: old or wrong pages, or sentences no page supports; a deep-research report repeats the same steps many times.
• A citation shows which page was used, not that it supports the sentence: in one audit only 74.5 % of citations supported their sentences, and in the scholarship answer only one of five citations was sound.
• Check each citation’s page for what it says, its age, whether it is primary and whether it only repeats another page.
• In a long report, check every footnote behind what you will act on, and judge the rest by a random sample, which catches a sloppy report quickly and cannot prove a good one clean.
• Read the original sources yourself before you act, when sources disagree, or when the answer cites only copies and summaries.