PDFs: extract text and tables, merge, generate

The practice site’s weekly market bulletin is a one-page PDF with a table of 40 prices. pypdf reads its text as 290 short lines, one value each, and nothing in them says which value belongs to which column; pdfplumber, which reads where every word sits on the page, gives back the 41 rows of the table, header included, cell by cell.

Here you extract the bulletin both ways and check every value against the data it was printed from, and see why a scanned page gives no text at all. After that come tables that run across pages, merging and stamping files, generating report PDFs, and opening PDFs from the web safely.

A PDF page holds text at positions, not a table

A PDF describes how a page looks: draw these characters, in this font, at this point. The pypdf documentation puts it plainly: the text within a PDF is absolutely positioned, the file has no semantic layer, and tables are typically just positioned text (pypdf). So a program gets the characters back, and the table has to be rebuilt from where they are. The program reads the bulletin three ways, then a page made from a picture:

"""What a PDF page gives a program: pypdf 6.19.0's text, the same page laid out, words with their positions from
pdfplumber 0.11.10, and a scanned page, which is a picture with no text in it."""
import pdfplumber
import pypdf
from PIL import Image, ImageDraw

page = pypdf.PdfReader("bulletin.pdf").pages[0]
lines = page.extract_text().splitlines()
print(f"pypdf, extract_text(): {len(lines)} lines; lines 11 to 17, the first row of the table:")
print(" ", lines[10:17])
laid_out = page.extract_text(extraction_mode="layout").splitlines()
row = next(line.strip() for line in laid_out if "Green chilli" in line)
print('pypdf, extraction_mode="layout", the Green chilli row:')
print(" ", row)
print(f"  split on spaces: {len(row.split())} pieces for 7 columns")

with pdfplumber.open("bulletin.pdf") as pdf:
    words = pdf.pages[0].extract_words()
first = next(w for w in words if w["text"] == "Green")
print("pdfplumber, the words of that row and where each starts (x0, in points from the left edge):")
print(" ", ", ".join(f"{w['text']} {w['x0']:.0f}" for w in words if abs(w["top"] - first["top"]) < 1))

# A scanned page is a picture of text: draw one, save it as a PDF and ask both libraries for its text.
picture = Image.new("L", (900, 120), 255)
ImageDraw.Draw(picture).text((30, 50), "Onion  Red  Nimbu Nagar  1,270  1,535  1,440  112.9", fill=0)
picture.save("scanned.pdf")
with pdfplumber.open("scanned.pdf") as pdf:
    scanned = pdf.pages[0]
    print(f"scanned.pdf: pypdf text {pypdf.PdfReader('scanned.pdf').pages[0].extract_text()!r}, "
          f"pdfplumber characters {len(scanned.chars)}, images {len(scanned.images)}")

pypdf, extract_text(): 290 lines; lines 11 to 17, the first row of the table:
  ['Onion', 'Red', 'Nimbu Nagar', '1,270', '1,535', '1,440', '112.9']
pypdf, extraction_mode="layout", the Green chilli row:
  Green chilli     Local            Tulsi Peth           1,150        1,350        1,260          163.3
  split on spaces: 9 pieces for 7 columns
pdfplumber, the words of that row and where each starts (x0, in points from the left edge):
  Green 118, chilli 144, Local 178, Tulsi 237, Peth 257, 1,150 312, 1,350 357, 1,260 402, 163.3 456
scanned.pdf: pypdf text '', pdfplumber characters 0, images 1

extract_text() gave 290 lines, one value each, in the order they were drawn: the row is there only as seven lines in a row. The layout mode lines the values up again, but splitting that line on spaces gives 9 pieces for 7 columns, because Green chilli and Tulsi Peth hold spaces themselves. pdfplumber’s words carry their positions, and positions decide the columns: Green and chilli start at 118 and 144 points, both left of Local at 178, where the Variety column begins.

The scanned page gave nothing at all: no characters, one image. Its text layertext layer: The characters a PDF page draws as text, each with a font and a position, which programs can read back. A scanned page is a picture and has none, so text extraction returns nothing until OCR adds one. is missing, because the page is a picture of text, and pypdf’s documentation says it will never extract text from images. Such pages need OCR, which is outside this track; for a single file, MySmartCoPilot’s OCR tool makes a scan searchable:

pdfplumber rebuilds the rows, and the source data checks them

pdfplumber finds a table from the ruling linesruling lines: The lines a page draws between the rows and columns of a table. pdfplumber's default table setting builds its cells from them, so a table printed without lines needs other settings. the page draws, in five steps its README describes: find the lines, merge the ones that nearly overlap, find where they cross, take the smallest rectangles between crossings as cells, and group touching cells into tables. Step through them with the bulletin’s own counts:

Every value comes back as text, so the program reads the numbers, 2,03,020 as 203020 whatever the grouping, and then compares each record with the data the bulletin was printed from:

"""The bulletin's table read with pdfplumber 0.11.10, then checked value by value against the data it was printed
from (bulletin_data.py)."""
import pdfplumber

from bulletin_data import HEADER, ROWS


def number(text):
    """2,150 and 2,03,020 are read the same way: the commas only group the digits."""
    return int(text.replace(",", ""))


with pdfplumber.open("bulletin.pdf") as pdf:
    page = pdf.pages[0]
    finder = page.debug_tablefinder()
    kinds = [edge["orientation"] for edge in finder.edges]
    print(f"lines on the page: {kinds.count('v')} vertical, {kinds.count('h')} horizontal; "
          f"crossings {len(finder.intersections)}; cells {len(finder.cells)}")
    table = page.extract_table()

print(f"extract_table(): {len(table)} rows of {len(table[0])} cells")
print("row 1:", table[0])
print("row 14:", table[13])
records = [(c, v, m, number(lo), number(hi), number(mo), float(t)) for c, v, m, lo, hi, mo, t in table[1:]]
same = sum(got == want for got, want in zip(records, ROWS))
print(f"header as printed: {table[0] == HEADER}; rows equal to the source: {same} of {len(ROWS)}")
print("row 14 as a record:", records[12])

lines on the page: 8 vertical, 42 horizontal; crossings 336; cells 287
extract_table(): 41 rows of 7 cells
row 1: ['Commodity', 'Variety', 'Market', 'Min', 'Max', 'Modal', 'Arrivals (t)']
row 14: ['Cardamom', 'Bold', 'Nimbu Nagar', '1,89,110', '2,15,360', '2,03,020', '23.5']
header as printed: True; rows equal to the source: 40 of 40
row 14 as a record: ('Cardamom', 'Bold', 'Nimbu Nagar', 189110, 215360, 203020, 23.5)

"""The practice site's market bulletins, as the data they were printed from. Every market and price is invented.

ROWS is week 18's one-page bulletin (bulletin.pdf): 40 rows of prices in rupees per quintal and arrivals in tonnes.
BOARD is the three-page board of the same week (board.pdf): 120 rows, printed without ruling lines.
"""
import random

ITEMS = [("Onion", "Red"), ("Potato", "Jyoti"), ("Tomato", "Hybrid"), ("Green chilli", "Local"), ("Brinjal", "Round"),
         ("Cabbage", "Golden Acre"), ("Cauliflower", "Local"), ("Wheat", "Lokwan"), ("Bengal gram", "Desi"),
         ("Soybean", "Yellow"), ("Cotton", "Long staple"), ("Turmeric", "Finger"), ("Cardamom", "Bold"),
         ("Garlic", "Ooty"), ("Ginger", "Fresh"), ("Maize", "Yellow"), ("Groundnut", "Bold"), ("Mustard", "Black"),
         ("Coriander", "Badami"), ("Cumin", "Bold")]
MARKETS = ["Nimbu Nagar", "Kesar Ganj", "Amla Pur", "Tulsi Peth", "Neem Chowk", "Kamal Bagh"]
BASE = {"Cardamom": 210000, "Cumin": 24000, "Turmeric": 13500, "Cotton": 7400, "Garlic": 9800, "Ginger": 6200,
        "Mustard": 5900, "Coriander": 7600, "Groundnut": 6400, "Bengal gram": 5600, "Soybean": 4700, "Wheat": 2600,
        "Maize": 2200}  # rupees a quintal; vegetables get 900 to 3,000
HEADER = ["Commodity", "Variety", "Market", "Min", "Max", "Modal", "Arrivals (t)"]


def rows(count, seed):
    rng = random.Random(seed)
    out = []
    for n in range(count):
        commodity, variety = ITEMS[n % len(ITEMS)]
        market = MARKETS[(n // len(ITEMS) + n) % len(MARKETS)]
        base = BASE.get(commodity, 900 + 100 * rng.randrange(0, 22))
        low = base - base // 10 + 10 * rng.randrange(0, 20)
        high = low + base // 8 + 10 * rng.randrange(0, 20)
        modal = low + (high - low) * rng.randrange(40, 80) // 100 // 10 * 10
        arrivals = rng.randrange(5, 4000) / 10  # tonnes, one decimal
        out.append((commodity, variety, market, low, high, modal, arrivals))
    return out


ROWS = rows(40, 20261004)
BOARD = rows(120, 20261018)


def grouped(n):
    """Rupees with Indian digit grouping, as the bulletins print them: 2,150 and 2,10,000."""
    s = str(n)
    if len(s) <= 3:
        return s
    head, tail = s[:-3], s[-3:]
    parts = []
    while len(head) > 2:
        parts.insert(0, head[-2:])
        head = head[:-2]
    return ",".join([head, *parts, tail]) if head else ",".join([*parts, tail])

The counts match the figure: 8 vertical and 42 horizontal lines, 336 crossings, 287 cells, 41 rows of 7, and all 40 rows equal the source. A job rarely has the source to compare with, which is why its checks come from the PDF itself.

When a page draws no lines, pdfplumber can guess columns from where the words line up, its text setting, or take column edges that your code gives it. pypdf’s PdfWriter merges, splits and stamps files page by page. Merging copies the pages of several files into one, splitting copies chosen pages into files of their own, and a stamp draws one page, such as a mark that says the data is for practice, over each page of another. A report PDF can be written with ReportLab in Python or pdf-lib in Node. Generating it from the stored records and reading it back proves that it says what the records say. And a PDF from the web is untrusted input, read with limits.

Tables across pages and without lines, checked value by value Pro: opens with a Pro, Premium or Ultimate pass

Merging, splitting and stamping, and what each step keeps Pro: opens with a Pro, Premium or Ultimate pass

Generating report PDFs, and reading downloaded ones safely Pro: opens with a Pro, Premium or Ultimate pass

Practise reading PDFs

Test yourself on text at positions and on rebuilding a table, starting where your answers have been weakest:

Interview questions

Warm-up (fresher): Why is getting a table out of a PDF harder than getting its text?

Because a PDF stores how a page looks, not what it means. Its text is a set of characters drawn at positions, and a table is only those characters lined up, sometimes with lines drawn between them. Text extraction returns the characters in drawing order, so the rows and columns are gone and a cell with a space in it cannot be told from two cells. A table extractor such as pdfplumber rebuilds the cells from the lines the page draws, or from where words line up, and every value still comes back as text to be read and checked. A scanned page is harder still: it has no text at all until OCR adds it.

Three harder questions start from this lesson: shifted prices after a supplier changed its template, invoices with no text, and a PDF that took down a worker. The module’s bank gives the first step of each:

Key takeaways

• A PDF page holds characters at positions; text extraction loses the table, and a scanned page has no text at all.
• pdfplumber rebuilds a ruled table from its lines, and every value comes back as text for your code to read.
• Check every extracted table, against its source while you build the job and against the PDF’s own numbers after.