Web Scraping & Browser Automation (Python and JavaScript) Module 2 – HTTP, sessions, pagination and APIs
Sessions, cookies and HTML forms
How cookies make a session and where a jar sends them, how to submit GET and POST forms with hidden fields and tokens, and when a login means stop.
What you will learn
- Explain how Set-Cookie and Cookie headers make a session, and predict which requests a cookie jar sends each cookie with
- Build the GET or POST request a form sends, with its hidden fields, its selected options and the button pressed
- Log in to your own lab account with its form token, download your own data and sign out again
- Protect saved sessions as you protect passwords, and recognise when a login means stop
Before you start
What you’ll learn · Before you start
What you will learn
- Explain how Set-Cookie and Cookie headers make a session, and predict which requests a cookie jar sends each cookie with
- Build the GET or POST request a form sends, with its hidden fields, its selected options and the button pressed
- Log in to your own lab account with its form token, download your own data and sign out again
- Protect saved sessions as you protect passwords, and recognise when a login means stop
Before you start
On this page
Send the practice shop’s sign-in form back without the cookie it set a moment earlier and the shop answers 403;
send it with the cookie but let fetch follow the redirect by itself and you end up, with status 200, on the
sign-in page again. Sessions, cookies and forms decide whether a collector is one visitor or a stranger on every
request.
Here you watch a cookie travel and see where a cookie jar sends it, build the requests that forms send, hidden fields and tokens included, and sign in to your own test account to download your own orders. You also see why a saved session is as secret as a password, and when a login is a reason to stop rather than to sign in.
A cookie is the site’s note, and a jar sends it back
HTTP keeps no memory between requests. A site that wants to recognise you sends a cookie: A name and value that a server sends in a Set-Cookie header and a client returns in a Cookie header on later requests, within the host, path, security and lifetime limits its attributes set.:
RFC 6265 describes how a server passes name and value pairs, called cookies, to a client in a Set-Cookie header, and
the client returns them in a Cookie header on later requests. A session: A record a server keeps for one visitor, found again on each request through an id the client sends back in a cookie. Signing in attaches an account to it; signing out deletes it. is nothing
more: a record on the server, found by an id the client carries back. Step through two visits to the shop’s sign-in
page:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- A first visit. The program asks for the sign-in page. HTTP itself remembers nothing between requests, so to the shop this is a stranger.
- The shop starts a session. The shop makes a record for this visitor, keyed by a random id, and sends the id back in a Set-Cookie header: nb_session=0df8…, with Path=/, Secure, HttpOnly and SameSite=Lax.
- The jar keeps it. urllib's HTTPCookieProcessor stores the cookie in its CookieJar, together with the host that set it and its attributes. Without a jar the header would simply be dropped.
- The next request. Before the next request to the same host, the jar adds a Cookie header with the stored value. The program wrote no code for this: the handler does it for every request.
- The shop recognises you. The shop looks the id up and finds the same record, so the two requests belong to one session. That is all a session is: a record on the server and an id the client carries.
- Without a jar. A client with no jar, such as urllib without the cookie handler or Node's fetch, sends no Cookie header, and every request looks like a new visitor. Requests sessions and HTTPX clients keep a jar for you.
Clients differ in who keeps the cookies. Python’s urllib keeps them only when you add an HTTPCookieProcessor,
whose cookie jar: The store in which an HTTP client keeps the cookies it receives and applies their attributes to choose which to send with each later request, such as Python's http.cookiejar.CookieJar. holds them; a Requests Session keeps the cookies for every request
sent through it, and an HTTPX Client lists cookie persistence across requests among its features. Node’s fetch keeps
none: on a server, the Set-Cookie values reach your code through response.headers.getSetCookie(), which MDN
describes for server environments such as Node.js, and sending them back is your job. You will see what that costs in
the sign-in section below.
A cookie without Expires or Max-Age is a session cookie: RFC 6265 says it expires at the end of the current
session, as the client defines it. For a script, that usually means when the program ends, unless it saves the jar.
HttpOnly does not hide a cookie from your program
HttpOnly keeps a cookie out of the page’s own scripts, such as document.cookie in a browser. A collector is not
a page script, so it sees and sends HttpOnly cookies like any other. The attribute protects the site’s visitors
from injected scripts, not the cookie from its owner’s program.
A cookie goes back only where its attributes allow
A jar does not send every cookie with every request. RFC 6265 lets the server scope each one: Path limits it to a
set of paths, Domain to a host and its subdomains (a cookie set without Domain goes back only to the host that set
it), Secure to encrypted connections, and Max-Age or Expires to a lifetime. The shop’s /cookies/demo page sets
four cookies with four different scopes. The program below collects them with urllib’s jar, then asks the jar which
Cookie header it would add to six requests, without sending any of them:
sessions/cookie_jar.py
"""Four cookies from the shop, and the Cookie header urllib's jar would send with each of six requests."""
import http.cookiejar
import urllib.request
import nimbu_lab as lab
jar = http.cookiejar.CookieJar()
shop = lab.opener(urllib.request.HTTPCookieProcessor(jar))
with shop.open(lab.SITE + "/cookies/demo", timeout=10) as response:
for line in response.headers.get_all("Set-Cookie"):
print("Set-Cookie:", line)
print()
print(f"{'cookie':10} {'domain':30} {'path':11} {'secure':7} kept")
for c in sorted(jar, key=lambda c: c.name):
domain = c.domain + ("" if c.domain_specified else " (host only)")
print(f"{c.name:10} {domain:30} {c.path:11} {str(c.secure):7} {'until the session ends' if c.discard else 'a day'}")
print()
for url in ["https://bazaar.nimbu.test/", "https://bazaar.nimbu.test/products/?page=2",
"https://bazaar.nimbu.test/products/NB-0001/", "http://bazaar.nimbu.test/products/",
"https://api.nimbu.test/v1/prices", "https://nimbu.test.example.org/"]:
request = urllib.request.Request(url)
jar.add_cookie_header(request) # what the jar would send: nothing is sent here
print(f"{url:46} {request.get_header('Cookie') or '(no cookies)'}")Output
Set-Cookie: nb_seen=1; Path=/ Set-Cookie: nb_region=south; Path=/products/ Set-Cookie: nb_theme=dark; Path=/; Secure Set-Cookie: nb_group=retail; Domain=nimbu.test; Path=/; Max-Age=86400 cookie domain path secure kept nb_group .nimbu.test / False a day nb_region bazaar.nimbu.test (host only) /products/ False until the session ends nb_seen bazaar.nimbu.test (host only) / False until the session ends nb_theme bazaar.nimbu.test (host only) / True until the session ends https://bazaar.nimbu.test/ nb_seen=1; nb_theme=dark; nb_group=retail https://bazaar.nimbu.test/products/?page=2 nb_region=south; nb_seen=1; nb_theme=dark; nb_group=retail https://bazaar.nimbu.test/products/NB-0001/ nb_region=south; nb_seen=1; nb_theme=dark; nb_group=retail http://bazaar.nimbu.test/products/ nb_region=south; nb_seen=1; nb_group=retail https://api.nimbu.test/v1/prices nb_group=retail https://nimbu.test.example.org/ (no cookies)
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 cookie_jar.py
nimbu_lab.py (the shop) · sessions/nimbu_lab.py
"""Nimbu Bazaar, the practice shop of this track. Every product, price and person in it is invented.
The shop is a WSGI application (PEP 3333): app() answers one request. opener() builds Python's own HTTP client,
urllib.request, with its usual handlers for redirects, errors and cookies, and one change: the last step hands each
request to app() instead of opening a socket. The client code is real, nothing leaves your program, and any host but
the shop's gets 421 Misdirected Request.
"""
import fnmatch
import hashlib
import html
import http.client
import http.cookies
import io
import itertools
import json
import socket
import urllib.request
import urllib.response
from urllib.parse import parse_qs, urlsplit
HOST = "bazaar.nimbu.test"
SITE = f"https://{HOST}"
KINDS = ["Basmati rice", "Toor dal", "Green tea", "Jaggery", "Turmeric", "Mustard oil", "Poha", "Rock salt",
"Cashews", "Cardamom", "Ragi flour", "Ghee", "Honey", "Peanuts", "Coffee", "Raisins", "Sesame seeds",
"Tamarind", "Cumin", "Black pepper", "Rolled oats", "Cloves", "Fennel", "Moong dal", "Chana dal",
"Wheat flour", "Semolina", "Coriander", "Red chilli", "Almonds", "Dates", "Coconut oil", "Brown rice",
"Millet", "Flax seeds", "Cinnamon", "Ginger powder", "Vermicelli", "Masoor dal", "Sago"]
SIZES = [250, 500, 1000, 2000, 5000, 10000] # grams
LOG = [] # what the shop saw: one entry per request
class Clock:
"""The shop's clock, in seconds. A slow page moves it on instead of waiting, so every example runs at once."""
def __init__(self):
self.seconds = 0.0
def now(self):
return self.seconds
def sleep(self, seconds):
self.seconds += seconds
clock = Clock()
def products():
"""The 240 products: 40 kinds in 6 pack sizes, each price in paise (100 paise = 1 rupee)."""
out = []
for k, kind in enumerate(KINDS):
per_kg = 60 + (k * 37) % 900 # rupees a kilogram
for grams in SIZES:
size = f"{grams // 1000} kg" if grams >= 1000 else f"{grams} g"
out.append({"sku": f"NB-{len(out) + 1:04d}", "name": f"{kind} {size}",
"paise": per_kg * grams // 10 + 1000})
return out
def rupees(paise):
return f"₹{paise // 100:,}.{paise % 100:02d}"
class Asked:
"""One request as the shop's pages see it."""
def __init__(self, environ):
self.method, self.path = environ["REQUEST_METHOD"], environ["PATH_INFO"]
self.query = {k: v[-1] for k, v in parse_qs(environ.get("QUERY_STRING", "")).items()}
self.headers = {k[5:].replace("_", "-").title(): v for k, v in environ.items() if k.startswith("HTTP_")}
size = int(environ.get("CONTENT_LENGTH") or 0)
self.form = {k: v[-1] for k, v in parse_qs(environ["wsgi.input"].read(size).decode()).items()}
self.cookies = {k: m.value for k, m in http.cookies.SimpleCookie(environ.get("HTTP_COOKIE", "")).items()}
self.wait = environ.get("lab.wait", clock.sleep) # the shop takes this long before its next bytes
PAGES = {} # path pattern ("/products/NB-*/" matches every product) -> function(asked) -> (status, headers, body)
def page(pattern):
def register(fn):
PAGES[pattern] = fn
return fn
return register
def find_page(path):
exact = PAGES.get(path)
return exact or next((fn for p, fn in PAGES.items() if "*" in p and fnmatch.fnmatchcase(path, p)), None)
def text(status, body, kind="text/plain; charset=utf-8", extra=()):
data = body.encode() if isinstance(body, str) else body
return status, [("Content-Type", kind), ("Content-Length", str(len(data))), *extra], data
def not_found(asked):
return text("404 Not Found", "no such page\n")
def app(environ, start_response):
"""The shop: answers one request, and writes it into LOG first."""
asked = Asked(environ)
if environ.get("HTTP_HOST") != HOST:
status, headers, body = text("421 Misdirected Request", "not a Nimbu Bazaar address\n")
else:
status, headers, body = (find_page(asked.path) or not_found)(asked)
target = asked.path + ("?" + environ["QUERY_STRING"] if environ.get("QUERY_STRING") else "")
LOG.append({"line": f"{asked.method} {target} HTTP/1.1", "headers": environ.get("lab.sent", []),
"status": int(status[:3])})
start_response(status, headers)
return [body] if isinstance(body, bytes) else body
class Stream(io.RawIOBase):
"""A response body read chunk by chunk, as from a socket: a slow chunk can time out in the middle of read()."""
def __init__(self, chunks):
self.chunks, self.rest = iter(chunks), b""
def readable(self):
return True
def readinto(self, buffer):
while not self.rest:
self.rest = next(self.chunks, None)
if self.rest is None:
self.rest = b""
return 0
n = min(len(buffer), len(self.rest))
buffer[:n], self.rest = self.rest[:n], self.rest[n:]
return n
class ShopTransport(urllib.request.AbstractHTTPHandler):
"""urllib's last step, with app() where the socket would be. Everything before it is urllib's own code."""
handler_order = 100 # runs before any handler that would open a real connection
http_request = https_request = urllib.request.AbstractHTTPHandler.do_request_
def https_open(self, req):
# The headers urllib's do_open() writes to the socket, and the one http.client adds by itself.
headers = dict(req.unredirected_hdrs)
headers.update({k: v for k, v in req.headers.items() if k not in headers})
headers["Connection"] = "close"
sent = [(k.title(), v) for k, v in headers.items()]
if "Accept-encoding" not in headers:
sent.insert(0, ("Accept-Encoding", "identity"))
timeout = socket.getdefaulttimeout() if req.timeout is socket._GLOBAL_DEFAULT_TIMEOUT else req.timeout
def wait(seconds):
"""A socket in timeout mode gives up when one wait for bytes lasts longer than the timeout."""
if timeout is not None and seconds > timeout:
clock.sleep(timeout)
raise TimeoutError("timed out")
clock.sleep(seconds)
parts, data = urlsplit(req.full_url), req.data or b""
environ = {"REQUEST_METHOD": req.get_method(), "PATH_INFO": parts.path or "/", "QUERY_STRING": parts.query,
"SERVER_NAME": parts.hostname or "", "SERVER_PORT": str(parts.port or 443),
"SERVER_PROTOCOL": "HTTP/1.1", "wsgi.url_scheme": parts.scheme, "wsgi.input": io.BytesIO(data),
"CONTENT_TYPE": headers.get("Content-type", ""), "CONTENT_LENGTH": str(len(data)),
"lab.wait": wait, "lab.sent": sent}
environ.update({"HTTP_" + k.upper().replace("-", "_"): v for k, v in sent
if k not in ("Content-Type", "Content-Length")})
answer = {}
body = app(environ, lambda status, hdrs, exc_info=None: answer.update(status=status, headers=hdrs))
raw = "".join(f"{k}: {v}\r\n" for k, v in answer["headers"]) + "\r\n"
message = http.client.parse_headers(io.BytesIO(raw.encode("iso-8859-1")))
response = urllib.response.addinfourl(io.BufferedReader(Stream(body)), message, req.full_url,
int(answer["status"][:3]))
response.reason = response.msg = answer["status"][4:] # as urllib's own responses carry them
return response
http_open = https_open
DEFAULTS = [urllib.request.HTTPDefaultErrorHandler, urllib.request.HTTPRedirectHandler,
urllib.request.HTTPErrorProcessor, urllib.request.UnknownHandler]
def opener(*handlers):
"""Like urllib.request.build_opener(*handlers), with the shop in place of the network: urllib's usual handlers,
except those a handler you pass replaces (a subclass of HTTPRedirectHandler, say)."""
director = urllib.request.OpenerDirector()
kept = [d for d in DEFAULTS if not any(isinstance(h, d) for h in handlers)]
for handler in [ShopTransport(), *(d() for d in kept), *handlers]:
director.add_handler(handler)
return director
# ---- The pages this lesson's examples ask for ----------------------------------------------------------------------
ROBOTS = """\
User-agent: *
Disallow: /account/
Disallow: /people/
Allow: /products/
Sitemap: https://bazaar.nimbu.test/sitemap.xml
"""
ACCOUNT = {"email": "learner@example.org", "password": "practise-on-the-lab-7311"} # the lab's test account
ORDERS = "order,placed_ist,items,total_inr\nNB-O-1041,2026-09-02 18:40,3,412.50\nNB-O-1077,2026-09-19 09:15,1,96.00\n" \
"NB-O-1102,2026-10-01 20:05,5,1240.00\n"
SESSIONS = {} # session id -> {"csrf": the session's form token, "user": the signed-in email or None}
SERIAL = itertools.count()
def new_session(user=None):
"""A session and its form token. A real shop draws both with the secrets module; these repeat on every run."""
sid = hashlib.sha256(f"lab-session-{next(SERIAL)}".encode()).hexdigest()[:16]
SESSIONS[sid] = {"csrf": hashlib.sha256(f"lab-csrf-{sid}".encode()).hexdigest()[:24], "user": user}
return sid
def session_cookie(sid):
return ("Set-Cookie", f"nb_session={sid}; Path=/; Secure; HttpOnly; SameSite=Lax")
def html_page(title, body, status="200 OK", extra=()):
return text(status, f"<!doctype html>\n<title>{title} · Nimbu Bazaar</title>\n{body}\n", "text/html; charset=utf-8",
extra)
def see_other(where, extra=()):
return text("303 See Other", "", extra=[("Location", where), *extra])
@page("/robots.txt")
def robots_txt(asked):
return text("200 OK", ROBOTS)
@page("/")
def home(asked):
return html_page("Home", '<p><a href="/products/">Products</a> · <a href="/account/">Your account</a></p>')
@page("/cookies/demo")
def cookie_demo(asked): # four cookies, each with other attributes
return text("200 OK", "four cookies set\n", extra=[
("Set-Cookie", "nb_seen=1; Path=/"),
("Set-Cookie", "nb_region=south; Path=/products/"),
("Set-Cookie", "nb_theme=dark; Path=/; Secure"),
("Set-Cookie", "nb_group=retail; Domain=nimbu.test; Path=/; Max-Age=86400")])
@page("/search")
def search(asked): # a GET form: the query travels in the address
form = ('<form action="/search" method="get"><input name="q" value="">'
'<select name="sort"><option value="name">Name</option><option value="price" selected>Price</option>'
'</select><button>Search</button></form>')
if "q" not in asked.query:
return html_page("Search", form)
found = [p for p in products() if asked.query["q"].lower() in p["name"].lower()]
found.sort(key=lambda p: p["paise"] if asked.query.get("sort") == "price" else p["name"])
rows = "".join(f"<li>{p['name']}: {rupees(p['paise'])}</li>" for p in found[:5])
return html_page("Search", f"{form}<p>{len(found)} products</p><ul>{rows}</ul>")
BOARD_STATE = "Ym9hcmQ9bWFuZGktc291dGg7dmlldz10YWJsZQ==" # the page's own state, echoed back as __VIEWSTATE does
MARKETS = {"Bengaluru": 4210, "Chennai": 4185, "Hyderabad": 4302, "Kochi": 4390} # onion, paise a kilogram
@page("/prices/board")
def board(asked): # a POST form with hidden fields, a select, a checkbox, a disabled field and two buttons
if asked.method == "GET":
options = "".join(f'<option{" selected" if m == "Chennai" else ""}>{m}</option>' for m in MARKETS)
return html_page("Price board", f'''<form action="/prices/board" method="post">
<input type="hidden" name="__VIEWSTATE" value="{BOARD_STATE}">
<input type="hidden" name="board" value="mandi-south">
<select name="market">{options}</select>
<input type="checkbox" name="organic" value="yes">
<input name="region" value="South" disabled>
<button type="submit" name="action" value="show">Show prices</button>
<button type="submit" name="action" value="csv">Download CSV</button>
</form>''')
if asked.form.get("__VIEWSTATE") != BOARD_STATE:
return text("400 Bad Request", "the form's state is missing: load the board again\n")
market = asked.form.get("market", "")
if market not in MARKETS:
return text("400 Bad Request", "unknown market\n")
line = f"{market},onion,{MARKETS[market] / 100:.2f}"
if asked.form.get("action") == "csv":
return text("200 OK", f"market,item,inr_per_kg\n{line}\n", "text/csv; charset=utf-8")
return html_page("Price board", f"<table><tr><td>{market}</td><td>onion</td><td>{rupees(MARKETS[market])}"
"/kg</td></tr></table>")
def signed_in(asked):
session = SESSIONS.get(asked.cookies.get("nb_session", ""))
return session if session and session["user"] else None
def login_form(sid, message=""):
note = f'<p class="error">{message}</p>' if message else ""
return f'''{note}<form action="/account/login" method="post">
<input type="hidden" name="csrf_token" value="{SESSIONS[sid]["csrf"]}">
<input type="hidden" name="next" value="/account/">
<input type="email" name="email"><input type="password" name="password">
<button type="submit" name="action" value="sign-in">Sign in</button>
</form>'''
@page("/account/login")
def login(asked):
sid = asked.cookies.get("nb_session", "")
if asked.method == "GET":
if sid not in SESSIONS:
sid = new_session()
return html_page("Sign in", login_form(sid), extra=[session_cookie(sid)])
session = SESSIONS.get(sid)
if not session or asked.form.get("csrf_token") != session["csrf"]:
return text("403 Forbidden", "this form has expired or did not come from this site\n")
if (asked.form.get("email"), asked.form.get("password")) != (ACCOUNT["email"], ACCOUNT["password"]):
return html_page("Sign in", login_form(sid, "Email or password is wrong."))
del SESSIONS[sid] # a new session id after signing in, so an id seen before it is worth nothing
return see_other(asked.form.get("next", "/account/"), [session_cookie(new_session(ACCOUNT["email"]))])
@page("/account/")
def account(asked):
session = signed_in(asked)
if not session:
return see_other("/account/login?next=/account/")
return html_page("Your account", f'''<p>Signed in as {html.escape(session["user"])}.</p>
<p><a href="/account/orders.csv">Download your orders as CSV</a>. Account holders may fetch this file with their own
scripts, at most once an hour.</p>
<form action="/account/logout" method="post"><input type="hidden" name="csrf_token" value="{session["csrf"]}">
<button>Sign out</button></form>''')
@page("/account/orders.csv")
def orders(asked):
if not signed_in(asked):
return see_other("/account/login?next=/account/orders.csv")
return text("200 OK", ORDERS, "text/csv; charset=utf-8")
@page("/account/logout")
def logout(asked):
sid = asked.cookies.get("nb_session", "")
session = SESSIONS.get(sid)
if asked.method != "POST" or not session or asked.form.get("csrf_token") != session["csrf"]:
return text("403 Forbidden", "sign out with the form on your account page\n")
del SESSIONS[sid] # the session ends on the server, so a copy of the cookie stops working too
return see_other("/", [("Set-Cookie", "nb_session=; Path=/; Max-Age=0")])Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Read the last block row by row. nb_region went back only under /products/; nb_theme, marked Secure, stayed
off the plain http:// request; and nb_group, the only cookie with a Domain, was the only one sent to
api.nimbu.test. Nothing went to nimbu.test.example.org, a host whose name merely contains the shop’s.
| Attribute | Example from the shop | Goes back to |
|---|---|---|
| none of the below | nb_seen=1; Path=/ |
the host that set it, every path, until the session ends |
Path |
nb_region=south; Path=/products/ |
only paths under /products/ |
Secure |
nb_theme=dark; Path=/; Secure |
only https:// requests |
Domain |
nb_group=retail; Domain=nimbu.test |
nimbu.test and every host under it |
Max-Age |
Max-Age=86400 |
for a day, even across runs if the jar is saved; Max-Age=0 deletes it |
Setting the Cookie header by hand on every request
A hand-written Cookie header goes with every request, to every path and every host your code calls, which is
exactly what Path, Domain and Secure exist to prevent. Use a jar, which applies the scope rules; in Node,
where fetch has none, keep one jar per site and send each cookie only to its own host.
Checkpoint
Check: Cookies and sessions · Where a jar sends each cookie
A form is a request written in HTML
A form is a description of a request: its action says where to send it, its method how, and its fields what. The
HTML standard fills in what is missing. The method defaults to GET; the encoding to
application/x-www-form-urlencoded, name=value pairs joined by &; and an empty action becomes the address of the
page the form is on. A GET form puts the pairs in the address’s query, a POST form in the request body.
Not every field travels. When the standard builds the list of pairs to send, it skips a field that is disabled, one without a name, a checkbox or radio button that is not checked, and any button except the one that submitted the form. On some platforms, pressing Enter in a text box submits the form with its default button, which the standard defines as its first submit button. Follow four fields of the shop’s price board through those rules:
The drawing shows the whole scene; its 5 steps are listed below.
Every step as text
- A hidden field. __VIEWSTATE is a hidden input with a name and a value. It passes every rule, so it is sent, although nobody sees it on the page.
- A disabled field. The region input says disabled. A disabled field is never sent, whatever its value, so South does not travel.
- An unchecked box. The organic checkbox is not ticked. An unchecked box sends nothing at all: not organic=no, not an empty value, nothing.
- The other button. Pressing Show prices sends action=show. The Download CSV button has a name too, but only the button that submits the form is sent.
- The body. What is left is encoded as name=value pairs joined by &: __VIEWSTATE, board, market and action, the body the shop expects from its own form.
The program below sends two of the shop’s forms the way a browser would: the search form as a GET, and the price
board as a POST with its two hidden fields, read from the page first. The last request leaves one hidden field out:
"""Two forms of the shop sent as a browser would send them: a GET search and a POST board with hidden fields."""
import urllib.error
import urllib.request
from html.parser import HTMLParser
from urllib.parse import urlencode
import nimbu_lab as lab
BOT = {"User-Agent": "NimbuLearner/1.0 (+https://example.org/my-bot; you@example.org)"}
shop = lab.opener()
class HiddenFields(HTMLParser):
"""The hidden inputs of a page, name -> value: the fields a person never sees but the form always sends."""
def __init__(self):
super().__init__()
self.fields = {}
def handle_starttag(self, tag, attrs):
a = dict(attrs)
if tag == "input" and a.get("type") == "hidden":
self.fields[a["name"]] = a.get("value", "")
def send(method, path, fields):
"""A form submission: GET puts the fields in the address, POST in the body, both form-encoded."""
query = urlencode(fields)
if method == "GET":
request = urllib.request.Request(f"{lab.SITE}{path}?{query}", headers=BOT)
else:
request = urllib.request.Request(lab.SITE + path, data=query.encode(), headers=BOT, method="POST")
try:
with shop.open(request, timeout=10) as response:
body = response.read().decode()
except urllib.error.HTTPError as error:
body = f"{error.code}: {error.read().decode().strip()}"
sent = lab.LOG[-1]
print(sent["line"])
if method == "POST":
print(" Content-Type:", dict(sent["headers"])["Content-Type"])
print(" body:", query)
return body
page = send("GET", "/search", {"q": "dal", "sort": "price"})
print(" ->", page[page.index("<p>") + 3:page.index("</p>")], "found")
print()
hidden = HiddenFields()
with shop.open(urllib.request.Request(lab.SITE + "/prices/board", headers=BOT), timeout=10) as response:
hidden.feed(response.read().decode())
print("hidden fields of the board:", sorted(hidden.fields))
page = send("POST", "/prices/board", {**hidden.fields, "market": "Kochi", "action": "show"})
print(" ->", page[page.index("<td>"):page.index("</table>")])
print()
without_state = {k: v for k, v in hidden.fields.items() if k != "__VIEWSTATE"}
print(" ->", send("POST", "/prices/board", {**without_state, "market": "Kochi", "action": "show"}))Output
GET /search?q=dal&sort=price HTTP/1.1 -> 24 products found hidden fields of the board: ['__VIEWSTATE', 'board'] POST /prices/board HTTP/1.1 Content-Type: application/x-www-form-urlencoded body: __VIEWSTATE=Ym9hcmQ9bWFuZGktc291dGg7dmlldz10YWJsZQ%3D%3D&board=mandi-south&market=Kochi&action=show -> <td>Kochi</td><td>onion</td><td>₹43.90/kg</td></tr> POST /prices/board HTTP/1.1 Content-Type: application/x-www-form-urlencoded body: board=mandi-south&market=Kochi&action=show -> 400: the form's state is missing: load the board again
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 forms.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The search arrived as /search?q=dal&sort=price, and the board as a form-encoded body. urlencode turned the ==
at the end of __VIEWSTATE into %3D%3D, because = separates names from values in the body. Without that one
hidden field the shop answered 400: it could not tell which board the request was about.
Read the form, then send it
Fetch the page that holds the form, read its fields, and send them all back with only your own changes. A request written from memory goes stale the day the site adds a field or renames one; one built from the current form does not.
Hidden fields and tokens are part of the form
The board’s __VIEWSTATE carries the page’s own state, a style some frameworks use, and the shop needs it back to
know which board you were looking at. The sign-in form carries another kind of hidden field, a token. OWASP’s
cross-site request forgery cheat sheet explains why: a forged request is one in which another site tricks a signed-in
person’s browser into an action they did not choose, and the cheat sheet calls the synchronizer token pattern “one of
the most popular and recommended methods to mitigate CSRF”. The server generates a CSRF token: A secret value a server puts in a form as a hidden field and checks when the form comes back, so that a request forged by another site, which cannot read the value, is refused. for the session, puts it in the
form as a hidden field and checks it when the form comes back. A request without it, or with a token from another
session, is refused.
For a collector this changes one thing: a form you are allowed to submit must be sent with the token from the same page, in the same session, with the session’s cookie. RFC 6265 calls cookies a form of ambient authority, sent with every request to their site whoever caused the request; the token is the part another site cannot read, which is why it works.
| Hidden field | Who made it | What happens without it |
|---|---|---|
__VIEWSTATE on the price board |
the page, to remember its own state | 400: “the form’s state is missing” |
csrf_token on the sign-in form |
the server, for this session only | 403: “this form has expired or did not come from this site” |
next on the sign-in form |
the page, to say where to go after | the shop sends you to /account/ anyway |
Key idea
A form is a request the site has written for you: send every field it contains, hidden ones and the token included, from the same page and session, and change only the values a person would type or choose.
Checkpoint
Check: The request a form describes · Hidden fields and form tokens
Sign in only to your own account, and sign out after
Signing in with a program is legitimate where the account is yours, or a test account, or where its owner has given
written permission, and where the site’s terms allow scripts. Prefer an export or an API whenever the site offers one.
The practice shop’s account page says account holders may fetch their orders file with their own scripts, at most
once an hour. Its robots.txt asks crawlers to stay out of /account/, and a crawler should: a script that signs in as
you to fetch your own export is not crawling the site, but on a real site its terms decide, so read them first. Step
through the sign-in:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- The sign-in form. GET /account/login returns the form and a session cookie. The form holds a hidden csrf_token tied to that session.
- The POST. The program sends the form back with every field: the token, next, your email and password from the environment, and the button. The jar adds the cookie.
- Token and session agree. The shop checks that the token belongs to the session in the cookie. A POST without the cookie, or with a token from another session, gets 403.
- 303 and a new id. The password is right, so the shop starts a new session for the signed-in account, sends its id in Set-Cookie and answers 303 See Other to /account/.
- Your own data. With the new cookie, GET /account/orders.csv returns your 3 orders. The account page says account holders may fetch this file with their own scripts, at most once an hour.
- Signing out. The sign-out form also carries the token. The shop deletes the session and sends Max-Age=0, which removes the cookie from the jar; the next request for the orders goes to the sign-in page.
The program reads the email and password from the environment, never from its own text. In the lab they come from
lab_account.env when the shell has not set them, the way a .env file that git ignores works for a real account:
sessions/own_orders.py
"""Sign in to your own test account, download your own orders, sign out: a session in urllib with a cookie jar."""
import http.cookiejar
import os
import urllib.request
from html.parser import HTMLParser
from pathlib import Path
from urllib.parse import urlencode
import nimbu_lab as lab
# Credentials come from the environment; in the lab, from lab_account.env when the shell has not set them.
for line in Path("lab_account.env").read_text(encoding="utf-8").splitlines():
if line and not line.startswith("#"):
name, _, value = line.partition("=")
os.environ.setdefault(name, value)
BOT = {"User-Agent": "NimbuLearner/1.0 (+https://example.org/my-bot; you@example.org)"}
jar = http.cookiejar.CookieJar()
shop = lab.opener(urllib.request.HTTPCookieProcessor(jar))
class FormToken(HTMLParser):
def __init__(self):
super().__init__()
self.token = None
def handle_starttag(self, tag, attrs):
a = dict(attrs)
if tag == "input" and a.get("name") == "csrf_token":
self.token = a["value"]
def get(path):
with shop.open(urllib.request.Request(lab.SITE + path, headers=BOT), timeout=10) as response:
return response.url.removeprefix(lab.SITE), response.read().decode()
def post(path, fields):
data = urlencode(fields).encode()
with shop.open(urllib.request.Request(lab.SITE + path, data=data, headers=BOT), timeout=10) as response:
return response.url.removeprefix(lab.SITE), response.read().decode()
def token_of(page):
reader = FormToken()
reader.feed(page)
return reader.token
def cookie():
"""The session cookie, shortened: a full session id in a log is a key to the account."""
value = next((c.value for c in jar if c.name == "nb_session"), None)
return value[:4] + "…" if value else "(none)"
where, page = get("/account/login")
print("1. sign-in page; session cookie", cookie(), "and a form token")
where, page = post("/account/login", {"csrf_token": token_of(page), "next": "/account/", "action": "sign-in",
"email": os.environ["NIMBU_EMAIL"], "password": os.environ["NIMBU_PASSWORD"]})
print("2. signed in, redirected to", where, "with a new session cookie", cookie())
where, orders = get("/account/orders.csv")
print("3. downloaded", where, "with", len(orders.splitlines()) - 1, "orders:")
print(" " + orders.splitlines()[1])
where, page = post("/account/logout", {"csrf_token": token_of(get("/account/")[1])})
print("4. signed out; session cookie now", cookie())
print("5. the orders again:", "redirected to " + get("/account/orders.csv")[0])Output
1. sign-in page; session cookie 0df8… and a form token 2. signed in, redirected to /account/ with a new session cookie 70d9… 3. downloaded /account/orders.csv with 3 orders: NB-O-1041,2026-09-02 18:40,3,412.50 4. signed out; session cookie now (none) 5. the orders again: redirected to /account/login?next=/account/orders.csv
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 own_orders.py
lab_account.env (the lab's test account) · sessions/lab_account.env
# The practice shop's test account. Your real .env file holds your own values and is listed in .gitignore.
NIMBU_EMAIL=learner@example.org
NIMBU_PASSWORD=practise-on-the-lab-7311Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Two cookies appeared because the shop gave the session a new id once you had signed in. OWASP’s session management
cheat sheet requires a new session id after any change of privilege, such as signing in, so that an id seen before
the sign-in is worth nothing afterwards. Signing out sent Max-Age=0, which removed the cookie from the jar, and the server deleted
the session itself, as the same cheat sheet requires on log-out.
In Node, fetch keeps no jar, and following redirects by itself hides the Set-Cookie of the 303 that carries the
new session id. Three attempts at the same sign-in, against the JavaScript version of the shop:
node/cookies_fetch.mjs
// Node's fetch keeps no cookies: every request carries only the headers you give it, three ways to sign in.
import { ACCOUNT, SITE, shop } from './nimbu_lab.mjs';
const lab = { dispatcher: shop, signal: AbortSignal.timeout(10_000) }; // a real request leaves dispatcher out
const jar = new Map(); // name -> value, for this one site
function remember(response) {
for (const line of response.headers.getSetCookie()) {
const [name, value] = line.split(';')[0].split('=');
if (value) jar.set(name, value);
else jar.delete(name);
}
}
async function signIn({ sendCookies, redirect }) {
jar.clear();
const form = await fetch(`${SITE}/account/login`, lab);
remember(form);
const csrf_token = (await form.text()).match(/name="csrf_token" value="([^"]+)"/)[1];
const headers = sendCookies ? { cookie: [...jar].map(([k, v]) => `${k}=${v}`).join('; ') } : {};
const post = await fetch(`${SITE}/account/login`, { ...lab, method: 'POST', redirect, headers, body: new URLSearchParams({ csrf_token, ...ACCOUNT }) });
remember(post);
await post.body?.cancel();
const ended = new URL(post.url);
if (post.status !== 303) return `${post.status}, ended at ${ended.pathname}${ended.search}`;
const next = await fetch(SITE + post.headers.get('location'), { ...lab, headers: { cookie: `nb_session=${jar.get('nb_session')}` } });
return `${next.status} at ${new URL(next.url).pathname}: ${(await next.text()).split('\n')[1]}`;
}
const tries = [
['no cookies sent back', { sendCookies: false, redirect: 'follow' }],
['cookie sent, redirect followed by fetch', { sendCookies: true, redirect: 'follow' }],
['cookie sent, redirect followed by hand', { sendCookies: true, redirect: 'manual' }],
];
for (const [label, how] of tries) console.log(`${label}:`.padEnd(42), await signIn(how));Output
no cookies sent back: 403, ended at /account/login cookie sent, redirect followed by fetch: 200, ended at /account/login?next=/account/ cookie sent, redirect followed by hand: 200 at /account/: <p>Signed in as learner@example.org.</p>
Recorded with Node.js 24.21.0 on macOS 26 arm64. To run it yourself: mise exec node@24.21.0 -- node cookies_fetch.mjs
nimbu_lab.mjs (the shop for fetch) · node/nimbu_lab.mjs
// Nimbu Bazaar's sign-in pages for Node's fetch. Every account and order in it is invented.
//
// `shop` is a dispatcher, the object Node's fetch hands each request to (its `dispatcher` option, which Node's
// documentation describes). It answers from the pages below instead of opening a socket; fetch's own code does the
// rest: its headers, its redirects. Nothing leaves the program.
import { createHash } from 'node:crypto';
export const SITE = 'https://bazaar.nimbu.test';
export const ACCOUNT = { email: 'learner@example.org', password: 'practise-on-the-lab-7311' }; // the lab's test account
export const LOG = []; // what the shop received: one entry per request
const sessions = new Map(); // session id -> { csrf, user }
let serial = 0;
function newSession(user = null) {
// A real shop draws both values at random; these repeat on every run.
const sid = createHash('sha256').update(`lab-session-${serial++}`).digest('hex').slice(0, 16);
sessions.set(sid, { csrf: createHash('sha256').update(`lab-csrf-${sid}`).digest('hex').slice(0, 24), user });
return sid;
}
const cookieOf = (headers) => Object.fromEntries((headers.cookie ?? '').split(/;\s*/).filter(Boolean).map((c) => c.split('=')));
const setCookie = (sid) => `nb_session=${sid}; Path=/; Secure; HttpOnly; SameSite=Lax`;
const html = (body) => ({ status: 200, headers: [['content-type', 'text/html; charset=utf-8']], body: `<!doctype html>\n${body}\n` });
const seeOther = (location, extra = []) => ({ status: 303, headers: [['location', location], ...extra], body: '' });
function answer(method, url, headers, body) {
const sid = cookieOf(headers).nb_session ?? '';
if (url.pathname === '/account/login' && method === 'GET') {
const id = sessions.has(sid) ? sid : newSession();
const page = html(`<form method="post"><input type="hidden" name="csrf_token" value="${sessions.get(id).csrf}">…</form>`);
page.headers.push(['set-cookie', setCookie(id)]);
return page;
}
if (url.pathname === '/account/login' && method === 'POST') {
const form = new URLSearchParams(body);
const session = sessions.get(sid);
if (!session || form.get('csrf_token') !== session.csrf) return { status: 403, headers: [['content-type', 'text/plain']], body: 'this form has expired or did not come from this site\n' };
if (form.get('email') !== ACCOUNT.email || form.get('password') !== ACCOUNT.password) return html('<p>Email or password is wrong.</p>');
sessions.delete(sid); // a new session id after signing in
return seeOther('/account/', [['set-cookie', setCookie(newSession(ACCOUNT.email))]]);
}
if (url.pathname === '/account/') {
const user = sessions.get(sid)?.user;
return user ? html(`<p>Signed in as ${user}.</p>`) : seeOther('/account/login?next=/account/');
}
return { status: 404, headers: [['content-type', 'text/plain']], body: 'no such page\n' };
}
// fetch passes a request body as chunks to read; collect them into one string.
async function bodyOf(body) {
if (body == null || typeof body === 'string') return body ?? '';
const parts = [];
for await (const chunk of body) parts.push(Buffer.from(chunk));
return Buffer.concat(parts).toString();
}
export const shop = {
dispatch(options, handler) {
const headers = Object.fromEntries(Object.entries(options.headers ?? {}).map(([k, v]) => [k.toLowerCase(), v]));
LOG.push({ line: `${options.method} ${options.path}`, cookie: headers.cookie ?? null });
const url = new URL(options.path, options.origin);
handler.onConnect(() => {});
bodyOf(options.body).then((body) => {
const a = url.host === new URL(SITE).host ? answer(options.method, url, headers, body) : { status: 421, headers: [], body: '' };
const raw = [...a.headers, ['content-length', String(Buffer.byteLength(a.body))]].flatMap(([k, v]) => [Buffer.from(k), Buffer.from(v)]);
handler.onHeaders(a.status, raw, () => {}, '');
if (a.body) handler.onData(Buffer.from(a.body));
handler.onComplete([]);
});
return true;
},
close() {},
destroy() {},
};The second attempt is the dangerous one: it ended with 200, on the sign-in page. Only redirect: "manual" let the
program see the 303, keep its new cookie and send it to /account/.
| You meet | It means | What you do |
|---|---|---|
| a sign-in page on someone else’s data | the data is not offered to you | stop; ask the owner or find another source |
| a CAPTCHA, a code sent to a phone or a “verify you are human” page | the site wants a person here | stop; do not try to get past it |
| terms that forbid automated access to accounts | the contract says no | use the site’s export or API, or ask in writing |
| your own account, with terms that allow scripts | you may automate it | sign in, take your data, sign out |
Checkpoint
Check: Signing in to your own account
A saved session is as secret as a password
A collector that runs every hour need not sign in every hour: it can save its cookie jar and load it next time. That file is then as good as the password. OWASP’s session management cheat sheet points out that, for as long as it lasts, a session id is worth as much as the strongest sign-in the application offers, and Playwright’s documentation warns that the cookies and headers in its saved browser state could let someone impersonate you. Step through what the next program does with such a file:
The drawing shows the whole scene; its 5 steps are listed below.
Every step as text
- Signed in. The collector signs in with the password, and the shop gives it session 70d9…. Only that id travels with later requests, never the password.
- The jar is saved. MozillaCookieJar.save(ignore_discard=True) writes the session cookie to session.txt, and os.chmod(path, 0o600) leaves it readable by you alone: -rw-------.
- Someone has the file. Another program loads the file. It has no password and needs none: the cookie inside is the proof the shop asks for.
- Signed in as you. Its request carries the cookie, and the shop answers as it answers you: Signed in as learner@example.org. A leaked session file is a leaked sign-in.
- Signing out ends it. The collector signs out, and the shop deletes session 70d9…. The file still holds the cookie, but it opens nothing now: the next request goes to the sign-in page.
The program saves the signed-in jar with MozillaCookieJar (passing ignore_discard=True, without which save()
skips session cookies), restricts the file to its owner, and hands it to a second program that knows no password:
"""A saved cookie jar is a key to the account: whoever holds the file is signed in, with no password."""
import http.cookiejar
import os
import stat
import urllib.request
from pathlib import Path
from urllib.parse import urlencode
import nimbu_lab as lab
ACCOUNT = dict(line.split("=", 1) for line in Path("lab_account.env").read_text(encoding="utf-8").splitlines()
if line and not line.startswith("#"))
def client(jar):
return lab.opener(urllib.request.HTTPCookieProcessor(jar))
def page(shop, path, fields=None):
data = urlencode(fields).encode() if fields else None
with shop.open(urllib.request.Request(lab.SITE + path, data=data), timeout=10) as response:
return response.url.removeprefix(lab.SITE), response.read().decode()
def token(html):
return html.split('name="csrf_token" value="')[1].split('"')[0]
# Sign in once and save the session, as a collector might between runs.
jar = http.cookiejar.MozillaCookieJar("session.txt")
shop = client(jar)
form = page(shop, "/account/login")[1]
page(shop, "/account/login", {"csrf_token": token(form), "email": ACCOUNT["NIMBU_EMAIL"],
"password": ACCOUNT["NIMBU_PASSWORD"], "next": "/account/"})
jar.save(ignore_discard=True) # session cookies are skipped unless you ask for them
os.chmod("session.txt", 0o600) # readable and writable by you alone
print("session.txt:", stat.filemode(os.stat("session.txt").st_mode))
for line in Path("session.txt").read_text(encoding="utf-8").splitlines():
if line and not line.startswith("#"):
fields = line.split("\t")
print(" holds", fields[5], "for", fields[0], "=", fields[6][:4] + "…")
# A second program loads the file, and the shop treats it as you.
other = http.cookiejar.MozillaCookieJar("session.txt")
other.load(ignore_discard=True)
where, html = page(client(other), "/account/")
print("loaded by another program:", where, "->", html.split("<p>")[1].split("</p>")[0])
# Signing out ends the session on the server: the copy in the file stops working.
page(shop, "/account/logout", {"csrf_token": token(page(shop, "/account/")[1])})
print("after signing out, the file opens:", page(client(other), "/account/")[0])
Path("session.txt").unlink()Output
session.txt: -rw------- holds nb_session for bazaar.nimbu.test = 70d9… loaded by another program: /account/ -> Signed in as learner@example.org. after signing out, the file opens: /account/login?next=/account/
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 session_file.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The second program was signed in as learner@example.org with nothing but the file, and stopped being signed in only
when the session ended on the server. Treat session files as you treat passwords:
- keep them out of git (Playwright’s documentation recommends a folder listed in
.gitignore) and out of logs, issue reports and screenshots; - give them the mode
0o600, readable by you alone, as the program did; - sign out when a job is done, so that a copied file opens nothing;
- delete a file you no longer need, and treat a leaked one as a leaked password: sign out everywhere and change the password.
Never print a session cookie in full
Both Python examples printed only the first four characters of the session id. A full cookie value in a log or a bug report lets anyone who reads it sign in as you until the session ends.
Checkpoint
Check: Saved sessions kept secret
Practise: build a form’s request
The exercise asks for the function that the forms section used by hand: given a page’s HTML, build the request its first form sends, with every rule of the HTML standard for hidden fields, disabled fields, checkboxes, selects and buttons. The sample tests use the shop’s search form, its price board and its sign-in form:
Exercise · Medium · Python
Build the request a browser would send for a form
formreq.py already reads the forms of a page with read_forms(html): each form is a dictionary with its action and method as written (empty strings when missing) and its fields in document order. A field has its tag (input, button, select or textarea) and its attrs, a dictionary in which a value-less attribute such as disabled or selected maps to ""; a select also has options (each with attrs and text), a textarea its text. option_value(option) gives an option's value: its value attribute, else its text.
Write build_form_request(html, page_url, values=None, submitter=None) for the first form of the page. Return {"method": ..., "url": ..., "body": ...} the way the HTML standard builds a submission:
- The method is
"POST"when the form sayspostin any case, else"GET". The action is resolved againstpage_urlwithurljoin; a missing or empty action meanspage_urlitself. - Fields are sent in document order, and a field that is
disabledor has nonameis never sent. - An input sends
values[name]when given, else itsvalueattribute (""when absent). A checkbox or radio button is sent only when checked:values[name]set toTrueor to its value checks it,Falseunchecks it, and otherwise the HTML'scheckedattribute decides. Its value defaults to"on". - A select sends
values[name]when given, else every selected option that is not disabled; a select withoutmultipleand with no selected option sends its first option. A textarea sends its text, orvalues[name]. - Of the submit buttons (
<input type="submit">, and<button>without a type or withtype="submit"), only the one pressed is sent: the one whose value issubmitter, else the first. - The fields are encoded with
urlencode. A GET puts them in the action's query, replacing any query it had, with"body": None; a POST sends them as the body.
The sample tests use the shop's search form, its price board and its sign-in form, then the HTML's own defaults, options grouped under <optgroup> and a multiple select.
Starter code · formreq.py
from html.parser import HTMLParser
from urllib.parse import urlencode, urljoin, urlsplit, urlunsplit
class FormReader(HTMLParser):
"""Every <form> of a page with its fields in document order, as they are written in the HTML."""
def __init__(self):
super().__init__()
self.forms, self.select, self.textarea = [], None, None
def handle_starttag(self, tag, attrs):
a = {k: ("" if v is None else v) for k, v in attrs}
if tag == "form":
self.forms.append({"action": a.get("action", ""), "method": a.get("method", ""), "fields": []})
elif not self.forms:
return
elif tag in ("input", "button"):
self.forms[-1]["fields"].append({"tag": tag, "attrs": a})
elif tag == "select":
self.select = {"tag": "select", "attrs": a, "options": []}
self.forms[-1]["fields"].append(self.select)
elif tag == "option" and self.select is not None:
self.select["options"].append({"attrs": a, "text": ""})
elif tag == "textarea":
self.textarea = {"tag": "textarea", "attrs": a, "text": ""}
self.forms[-1]["fields"].append(self.textarea)
def handle_data(self, data):
if self.textarea is not None:
self.textarea["text"] += data
elif self.select is not None and self.select["options"]:
self.select["options"][-1]["text"] += data
def handle_endtag(self, tag):
if tag == "select":
self.select = None
elif tag == "textarea":
self.textarea = None
def read_forms(html):
reader = FormReader()
reader.feed(html)
return reader.forms
def option_value(option):
return option["attrs"]["value"] if "value" in option["attrs"] else " ".join(option["text"].split())
def build_form_request(html, page_url, values=None, submitter=None):
"""{"method", "url", "body"} for the first form of the page, sent with `values` typed in and the button whose
value is `submitter` pressed (the first submit button when it is None)."""
form = read_forms(html)[0]
# Replace this: a request that sends no fields loses every hidden field and token the form needs.
return {"method": "GET", "url": page_url, "body": None}The sample tests · test_formreq.py
from formreq import build_form_request
SHOP = "https://bazaar.nimbu.test"
SEARCH = ('<form action="/search" method="get"><input name="q" value="">'
'<select name="sort"><option value="name">Name</option><option value="price" selected>Price</option>'
'</select><button>Search</button></form>')
BOARD = '''<form action="/prices/board" method="post">
<input type="hidden" name="__VIEWSTATE" value="Ym9hcmQ9bWFuZGktc291dGg7dmlldz10YWJsZQ==">
<input type="hidden" name="board" value="mandi-south">
<select name="market"><option>Bengaluru</option><option selected>Chennai</option><option>Hyderabad</option><option>Kochi</option></select>
<input type="checkbox" name="organic" value="yes">
<input name="region" value="South" disabled>
<button type="submit" name="action" value="show">Show prices</button>
<button type="submit" name="action" value="csv">Download CSV</button>
</form>'''
LOGIN = '''<form action="/account/login" method="post">
<input type="hidden" name="csrf_token" value="5b0e9c1d2a7f43e8a61c9d04">
<input type="hidden" name="next" value="/account/">
<input type="email" name="email"><input type="password" name="password">
<button type="submit" name="action" value="sign-in">Sign in</button>
</form>'''
def test_a_get_form_puts_the_fields_in_the_address():
"""the search form: GET, the typed query, the selected sort; a button without a name sends nothing"""
got = build_form_request(SEARCH, SHOP + "/search", {"q": "dal"})
assert got == {"method": "GET", "url": SHOP + "/search?q=dal&sort=price", "body": None}, f"got {got}"
def test_a_post_form_sends_hidden_fields_and_the_pressed_button():
"""the price board: both hidden fields, the selected market, no unchecked box, no disabled field, one button"""
got = build_form_request(BOARD, SHOP + "/prices/board")
assert got["method"] == "POST" and got["url"] == SHOP + "/prices/board", "method and action come from the form"
assert got["body"] == ("__VIEWSTATE=Ym9hcmQ9bWFuZGktc291dGg7dmlldz10YWJsZQ%3D%3D&board=mandi-south"
"&market=Chennai&action=show"), f"the first submit button is pressed by default; got {got['body']}"
def test_values_and_another_button():
"""values change a select and tick a box; submitter presses the other button"""
got = build_form_request(BOARD, SHOP + "/prices/board", {"market": "Kochi", "organic": True}, submitter="csv")
assert (got["body"] or "").endswith("&market=Kochi&organic=yes&action=csv"), f"got {got['body']}"
assert "region=" not in (got["body"] or ""), "a disabled field is never sent"
def test_the_login_form():
"""every field in document order: token, next, email, password, button"""
got = build_form_request(LOGIN, SHOP + "/account/login", {"email": "learner@example.org", "password": "pw"})
assert got["body"] == ("csrf_token=5b0e9c1d2a7f43e8a61c9d04&next=%2Faccount%2F&email=learner%40example.org"
"&password=pw&action=sign-in"), f"got {got['body']}"
def test_html_defaults():
"""no action means the page's own address; an option without value sends its text; a box without value sends on"""
html = ('<form><select name="size"><option> 1 kg </option><option>5 kg</option></select>'
'<input type="checkbox" name="gift" checked><textarea name="note">Ring twice</textarea>'
'<input name="nameless-is-fine" value="x"><input value="no name"></form>')
got = build_form_request(html, SHOP + "/order?step=2")
assert got == {"method": "GET", "url": SHOP + "/order?size=1+kg&gift=on¬e=Ring+twice&nameless-is-fine=x",
"body": None}, f"got {got}"
def test_options_in_groups():
"""options grouped under an optgroup still belong to the select; with none selected, the first is sent"""
html = ('<form action="/search"><select name="kind"><optgroup label="Dals"><option>Toor dal</option>'
'<option>Moong dal</option></optgroup><optgroup label="Rice"><option>Basmati</option></optgroup>'
'</select></form>')
got = build_form_request(html, SHOP + "/")
assert got == {"method": "GET", "url": SHOP + "/search?kind=Toor+dal", "body": None}, f"got {got}"
def test_multiple_select_and_relative_action():
"""a multiple select sends every selected option; the action resolves against the page"""
html = ('<form method="POST" action="filter"><select name="tag" multiple><option selected>dal</option>'
'<option>rice</option><option selected disabled>tea</option><option selected>oil</option></select></form>')
got = build_form_request(html, SHOP + "/products/")
assert got == {"method": "POST", "url": SHOP + "/products/filter", "body": "tag=dal&tag=oil"}, f"got {got}"A hint
Write a helper entries(form, values, submitter) that returns a list of (name, value) pairs, and loop over form["fields"] once, in order. Decide first whether a field is skipped ("disabled" in attrs, or no name), then handle each kind. Find the pressed button before the loop: the first submit button whose value equals submitter, or the first submit button when submitter is None; in the loop, send a button only if it is that one.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
The self-check comes next. It picks its first question from the idea your checkpoints found hardest, of the six here:
Practice that adapts to you
Practise this lesson
12 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up (fresher): What is a cookie, and how does a client decide where to send it back?
A cookie is a name and value pair that a server sends in a Set-Cookie header and the client stores and returns in a
Cookie header on later requests, which is how a site recognises one visitor across stateless HTTP requests. The
client sends a cookie back only where its attributes allow: to the host that set it, or with a Domain attribute to
that domain and its subdomains; only under its Path; only over HTTPS if it is Secure; and only until its Max-Age
or Expires, or the end of the session when it has neither. A cookie jar, such as Python’s http.cookiejar or a
Requests session, applies those rules for you; a session cookie is as sensitive as the password that created it.
Two harder questions grow out of this lesson: automating the download of your team’s own invoices from a supplier’s
portal, and a form that works in a browser but answers 403 to a script. Their first steps and follow-ups wait on
the question page of this module:
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Fresher to mid
Your manager asks you to automate the monthly download of your team's own invoices from a supplier's web portal, which needs a login. How do you approach it, from the first message to the running job?
-
Fresher to mid
A form on a site you are allowed to automate works in your browser, but your script's submission of the same form gets 403. How do you find the cause?
Key takeaways
- A session is a record on the server and an id in a cookie: the server sends it with
Set-Cookie, and a client with a jar sends it back inCookie; urllib needsHTTPCookieProcessor, Requests sessions and HTTPX clients keep a jar, and Node’sfetchkeeps none. - A jar sends each cookie only where its attributes allow: its host, or a
Domainand its subdomains, paths under itsPath, HTTPS only whenSecure, and only until it expires. - A form describes a request:
GETputs its fields in the query,POSTin a form-encoded body, and disabled, nameless, unchecked and unpressed fields are left out. - Hidden fields and tokens are part of the form; send them from the same page and session, or the site answers
400or403. - Sign in only to your own or a test account where the terms allow scripts, prefer an export, sign out after, and stop at another person’s login, a CAPTCHA or a code check.
- A saved session file is as good as a password: keep it out of git and logs, give it mode
0o600, and end the session on the server when the job is done.
Terms in this lesson
- cookie
- A name and value that a server sends in a Set-Cookie header and a client returns in a Cookie header on later requests, within the host, path, security and lifetime limits its attributes set.
- cookie jar
- The store in which an HTTP client keeps the cookies it receives and applies their attributes to choose which to send with each later request, such as Python's http.cookiejar.CookieJar.
- CSRF token
- A secret value a server puts in a form as a hidden field and checks when the form comes back, so that a request forged by another site, which cannot read the value, is refused.
- session
- A record a server keeps for one visitor, found again on each request through an id the client sends back in a cookie. Signing in attaches an account to it; signing out deletes it.
References
- RFC 6265: HTTP State Management Mechanism (cookies) (IETF)
- http.cookiejar: Cookie handling for HTTP clients (Python 3.14) (Python Software Foundation)
- urllib.request: HTTPCookieProcessor (Python 3.14) (Python Software Foundation)
- Requests: Advanced usage (session objects) (Requests project)
- HTTPX: Clients (HTTPX project (Encode))
- Headers: getSetCookie() method (Mozilla (MDN Web Docs))
- HTML Standard: form submission (constructing the entry list, implicit submission) (WHATWG)
- Cross-Site Request Forgery Prevention Cheat Sheet (OWASP Foundation)
- Session Management Cheat Sheet (OWASP Foundation)
- Playwright for Python: Authentication (saved browser state) (Microsoft (Playwright))
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress