PostgreSQL in Depth Module 2 – PostgreSQL data types, chosen well
Arrays, enums, domains and composite types
When a PostgreSQL array beats a join table and how GIN serves it, a domain that checks a PIN code once, safe enum changes, and composite row values.
What you will learn
- Decide between an array column and a join table, and index the array with GIN
- Query arrays with unnest WITH ORDINALITY and the array functions of PostgreSQL 18, and send lists as one array with = ANY
- Validate a PIN code once with a domain that every column of the type shares
- Add a value to an enum type safely, and name the changes ALTER TYPE cannot make
- Use composite types and ROW values in SQL functions and hand clients plain columns
Before you start
What you’ll learn · Before you start
What you will learn
- Decide between an array column and a join table, and index the array with GIN
- Query arrays with unnest WITH ORDINALITY and the array functions of PostgreSQL 18, and send lists as one array with = ANY
- Validate a PIN code once with a domain that every column of the type shares
- Add a value to an enum type safely, and name the changes ALTER TYPE cannot make
- Use composite types and ROW values in SQL functions and hand clients plain columns
Before you start
On this page
Finding the 1,428 of 50,000 products tagged both tea and organic takes one GIN index lookup when the tags are an
array, and a text search in which LIKE '%tea%' also matches teapot when they are the string 'tea,organic'. A
status column of plain text, likewise, stores shiped as readily as shipped. PostgreSQL has four kinds of type for
values like these: arrays keep a short list inside the row, enums a fixed and ordered set of labels, domains a base
type with a rule every value must pass, and composite types several named fields that travel together as one value.
On a PostgreSQL 18.6 server you will measure a GIN-indexed tag array against a join table of the same tags, put the rule of a six-digit PIN code into one domain for two tables, add a value to an enum around the commit it needs, and read a composite value’s fields in the forms that the parser and client drivers expect.
GIN answers containment on an array column
The first run gives each of 50,000 products two tags in a text[] column, ARRAY['tea', 'organic'] for product 35,
and indexes the column with GIN: Generalized Inverted Index: an index that stores, for every element of a composite value (an array item, a jsonb key or value, a word of a tsvector), the rows that contain it. More in “GIN, GiST, SP-GiST, BRIN, hash and bloom”. GIN keeps every distinct element once, followed by the
rows whose array contains it, so a question about elements inside the arrays becomes a lookup of a few lists. Which
questions the index can take is set by its operator class: The part of an index definition that says which operators the index can search with for a column's type, such as jsonb_path_ops for GIN. The catalogs pg_opclass and pg_amop list them. More in “GIN, GiST, SP-GiST, BRIN, hash and bloom”: the run lists the operators
of array_ops, the default class for arrays, and then asks for the products tagged both tea and organic:
tags_session.py
"""Tags as a text[] column with a GIN index, on PostgreSQL 18.6: runs tags.sql on a throwaway cluster."""
from pg_session import Server
# Plans that are the same on every run: no autovacuum, no parallel workers or JIT, statistics read from every row,
# and UTC as the time zone.
SETTINGS = {"autovacuum": "off", "max_parallel_workers_per_gather": 0, "jit": "off",
"default_statistics_target": 1000, "timezone": "'UTC'"}
with Server(settings=SETTINGS) as pg:
pg.up()
pg.psql("tags.sql", errors=0) Output
$ psql -X -a -f tags.sql
-- 50,000 products, each with a category tag and one more tag.
CREATE TABLE products (id int PRIMARY KEY, name text NOT NULL, tags text[] NOT NULL);
CREATE TABLE
INSERT INTO products
SELECT n, 'product ' || n,
ARRAY[(ARRAY['tea', 'coffee', 'snacks', 'spices', 'sweets'])[1 + n % 5],
(ARRAY['organic', 'gift', 'vegan', 'sale', 'new', 'local', 'imported'])[1 + n % 7]]
FROM generate_series(1, 50000) AS n;
INSERT 0 50000
CREATE INDEX products_tags ON products USING gin (tags);
CREATE INDEX
VACUUM ANALYZE products;
VACUUM
-- The operators of array_ops, the default GIN operator class for arrays.
SELECT amopopr::regoperator AS operator FROM pg_amop
WHERE amopfamily = (SELECT opcfamily FROM pg_opclass WHERE opcname = 'array_ops'
AND opcmethod = (SELECT oid FROM pg_am WHERE amname = 'gin'))
ORDER BY amopstrategy;
operator
-----------------------
&&(anyarray,anyarray)
@>(anyarray,anyarray)
<@(anyarray,anyarray)
=(anyarray,anyarray)
(4 rows)
-- Products tagged both tea and organic.
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF, BUFFERS OFF)
SELECT count(*) FROM products WHERE tags @> ARRAY['tea', 'organic'];
QUERY PLAN
------------------------------------------------------------------------------
Aggregate (actual rows=1.00 loops=1)
-> Bitmap Heap Scan on products (actual rows=1428.00 loops=1)
Recheck Cond: (tags @> '{tea,organic}'::text[])
Heap Blocks: exact=562
-> Bitmap Index Scan on products_tags (actual rows=1428.00 loops=1)
Index Cond: (tags @> '{tea,organic}'::text[])
Index Searches: 1
(7 rows)
-- The same question for one tag, written with = ANY.
EXPLAIN (COSTS OFF) SELECT count(*) FROM products WHERE 'tea' = ANY (tags);
QUERY PLAN
--------------------------------------------
Aggregate
-> Seq Scan on products
Filter: ('tea'::text = ANY (tags))
(3 rows)
EXPLAIN (COSTS OFF) SELECT count(*) FROM products WHERE tags @> ARRAY['tea'];
QUERY PLAN
-----------------------------------------------------
Aggregate
-> Bitmap Heap Scan on products
Recheck Cond: (tags @> '{tea}'::text[])
-> Bitmap Index Scan on products_tags
Index Cond: (tags @> '{tea}'::text[])
(5 rows)
-- Every table and index of this design, and their total.
SELECT coalesce(relname, 'total') AS relation, pg_size_pretty(sum(pg_relation_size(oid))) AS size
FROM pg_class WHERE relnamespace = 'public'::regnamespace GROUP BY ROLLUP (relname) ORDER BY relname;
relation | size
---------------+---------
products | 4496 kB
products_pkey | 1112 kB
products_tags | 192 kB
total | 5800 kB
(4 rows)
-- The same tags as one comma-separated string per product: smaller, but searched by pattern.
CREATE TABLE products_csv AS SELECT id, name, array_to_string(tags, ',') AS tags FROM products;
SELECT 50000
SELECT pg_size_pretty(pg_relation_size('products_csv')) AS csv_table,
'teapot,gift' LIKE '%tea%' AS teapot_matches_tea;
csv_table | teapot_matches_tea
-----------+--------------------
3072 kB | t
(1 row)
-- Renaming one tag rewrites every row that carries it.
UPDATE products SET tags = array_replace(tags, 'sale', 'offer') WHERE tags @> ARRAY['sale'];
UPDATE 7143
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 tags_session.py
tags.sql
-- 50,000 products, each with a category tag and one more tag.
CREATE TABLE products (id int PRIMARY KEY, name text NOT NULL, tags text[] NOT NULL);
INSERT INTO products
SELECT n, 'product ' || n,
ARRAY[(ARRAY['tea', 'coffee', 'snacks', 'spices', 'sweets'])[1 + n % 5],
(ARRAY['organic', 'gift', 'vegan', 'sale', 'new', 'local', 'imported'])[1 + n % 7]]
FROM generate_series(1, 50000) AS n;
CREATE INDEX products_tags ON products USING gin (tags);
VACUUM ANALYZE products;
-- The operators of array_ops, the default GIN operator class for arrays.
SELECT amopopr::regoperator AS operator FROM pg_amop
WHERE amopfamily = (SELECT opcfamily FROM pg_opclass WHERE opcname = 'array_ops'
AND opcmethod = (SELECT oid FROM pg_am WHERE amname = 'gin'))
ORDER BY amopstrategy;
-- Products tagged both tea and organic.
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF, BUFFERS OFF)
SELECT count(*) FROM products WHERE tags @> ARRAY['tea', 'organic'];
-- The same question for one tag, written with = ANY.
EXPLAIN (COSTS OFF) SELECT count(*) FROM products WHERE 'tea' = ANY (tags);
EXPLAIN (COSTS OFF) SELECT count(*) FROM products WHERE tags @> ARRAY['tea'];
-- Every table and index of this design, and their total.
SELECT coalesce(relname, 'total') AS relation, pg_size_pretty(sum(pg_relation_size(oid))) AS size
FROM pg_class WHERE relnamespace = 'public'::regnamespace GROUP BY ROLLUP (relname) ORDER BY relname;
-- The same tags as one comma-separated string per product: smaller, but searched by pattern.
CREATE TABLE products_csv AS SELECT id, name, array_to_string(tags, ',') AS tags FROM products;
SELECT pg_size_pretty(pg_relation_size('products_csv')) AS csv_table,
'teapot,gift' LIKE '%tea%' AS teapot_matches_tea;
-- Renaming one tag rewrites every row that carries it.
UPDATE products SET tags = array_replace(tags, 'sale', 'offer') WHERE tags @> ARRAY['sale']; array_ops holds four operators: && (the two arrays share an element), @> (the left array contains every element
of the right one), <@ (the reverse) and =. The containment query went through a Bitmap Index Scan on
products_tags and found 1,428 products in 562 heap blocks, the products whose number is a multiple of 35. The whole
design takes 5,800 kB: 4,496 kB of table, 1,112 kB of primary key and only 192 kB of GIN index, because the index
stores each of the 12 distinct tags once with the rows that carry it. A copy with the tags as comma-separated text took
3,072 kB without any index, less again, but it can be searched only by pattern, and one check of the run shows why
that misleads: 'teapot,gift' LIKE '%tea%' is true.
The same question for one tag, written as 'tea' = ANY (tags), scanned the whole table. That condition compares a
text value with each element through the = of text, which no array operator class holds; tags @> ARRAY['tea'] asks
for the same rows with an operator of array_ops, and the index served it.
= ANY on the array column
WHERE 'tea' = ANY (tags) reads every row even with a GIN index on tags. Write WHERE tags @> ARRAY['tea']: the
same rows, asked with an operator of array_ops, which is the form the index serves.
The last statement renamed the tag sale to offer. array_replace() rewrote 7,143 rows, every product that carried the
tag, and each new row version put its entries into the GIN index again. An array copies its values into every row that
holds them, which costs nothing while the values stay as they are and a rewrite of each copy when one changes.
Checkpoint
Check: GIN on array columns
A join table wins once each tag has a life of its own
The PostgreSQL manual warns that arrays are not sets, and that a need to search them for single elements may point to
a missing table with one row per element. The second run builds that design for the same 50,000 products, a
tags table of 12 rows and a join table: A table with one row for each pairing of rows from two other tables, such as one row per product and tag, holding a foreign key to each side. Facts about the pairing itself can be columns of it., product_tags, with one row per product and
tag, 100,000 rows with a foreign key to each side:
jointable_session.py
"""The tags of tags.sql as a join table, on PostgreSQL 18.6: runs jointable.sql on a throwaway cluster."""
from pg_session import Server
# The settings of tags_session.py, so the two plans are read on equal terms.
SETTINGS = {"autovacuum": "off", "max_parallel_workers_per_gather": 0, "jit": "off",
"default_statistics_target": 1000, "timezone": "'UTC'"}
with Server(settings=SETTINGS) as pg:
pg.up()
# One error is expected: the foreign key refuses a tag id that is not in tags.
pg.psql("jointable.sql", errors=1) Output
$ psql -X -a -f jointable.sql
-- The same 50,000 products, with their tags as rows of a join table.
CREATE TABLE products (id int PRIMARY KEY, name text NOT NULL);
CREATE TABLE
CREATE TABLE tags (id int PRIMARY KEY, name text NOT NULL UNIQUE);
CREATE TABLE
CREATE TABLE product_tags (
product_id int NOT NULL REFERENCES products,
tag_id int NOT NULL REFERENCES tags,
PRIMARY KEY (product_id, tag_id));
CREATE TABLE
INSERT INTO products SELECT n, 'product ' || n FROM generate_series(1, 50000) AS n;
INSERT 0 50000
INSERT INTO tags (id, name)
SELECT id, name FROM unnest(ARRAY['tea', 'coffee', 'snacks', 'spices', 'sweets', 'organic',
'gift', 'vegan', 'sale', 'new', 'local', 'imported'])
WITH ORDINALITY AS t(name, id);
INSERT 0 12
INSERT INTO product_tags SELECT n, 1 + n % 5 FROM generate_series(1, 50000) AS n;
INSERT 0 50000
INSERT INTO product_tags SELECT n, 6 + n % 7 FROM generate_series(1, 50000) AS n;
INSERT 0 50000
CREATE INDEX product_tags_tag ON product_tags (tag_id, product_id);
CREATE INDEX
VACUUM ANALYZE products, tags, product_tags;
VACUUM
-- Products tagged both tea and organic: one row per tag, grouped back per product.
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF, BUFFERS OFF)
SELECT count(*) FROM (
SELECT pt.product_id FROM product_tags pt JOIN tags t ON t.id = pt.tag_id
WHERE t.name IN ('tea', 'organic')
GROUP BY pt.product_id HAVING count(*) = 2) AS both_tags;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------
Aggregate (actual rows=1.00 loops=1)
-> HashAggregate (actual rows=1428.00 loops=1)
Group Key: pt.product_id
Filter: (count(*) = 2)
Batches: 1 Memory Usage: 1305kB
Rows Removed by Filter: 14286
-> Nested Loop (actual rows=17142.00 loops=1)
-> Seq Scan on tags t (actual rows=2.00 loops=1)
Filter: (name = ANY ('{tea,organic}'::text[]))
Rows Removed by Filter: 10
-> Index Only Scan using product_tags_tag on product_tags pt (actual rows=8571.00 loops=2)
Index Cond: (tag_id = t.id)
Heap Fetches: 0
Index Searches: 2
(14 rows)
-- Every table and index of this design, and their total.
SELECT coalesce(relname, 'total') AS relation, pg_size_pretty(sum(pg_relation_size(oid))) AS size
FROM pg_class WHERE relnamespace = 'public'::regnamespace GROUP BY ROLLUP (relname) ORDER BY relname;
relation | size
-------------------+------------
product_tags | 3544 kB
product_tags_pkey | 3312 kB
product_tags_tag | 2208 kB
products | 2544 kB
products_pkey | 1112 kB
tags | 8192 bytes
tags_name_key | 16 kB
tags_pkey | 16 kB
total | 12 MB
(9 rows)
-- A tag that does not exist is refused by the foreign key.
INSERT INTO product_tags VALUES (1, 99);
psql:jointable.sql:27: ERROR: insert or update on table "product_tags" violates foreign key constraint "product_tags_tag_id_fkey"
DETAIL: Key (tag_id)=(99) is not present in table "tags".
-- Renaming a tag changes one row.
UPDATE tags SET name = 'offer' WHERE name = 'sale';
UPDATE 1
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 jointable_session.py
jointable.sql
-- The same 50,000 products, with their tags as rows of a join table.
CREATE TABLE products (id int PRIMARY KEY, name text NOT NULL);
CREATE TABLE tags (id int PRIMARY KEY, name text NOT NULL UNIQUE);
CREATE TABLE product_tags (
product_id int NOT NULL REFERENCES products,
tag_id int NOT NULL REFERENCES tags,
PRIMARY KEY (product_id, tag_id));
INSERT INTO products SELECT n, 'product ' || n FROM generate_series(1, 50000) AS n;
INSERT INTO tags (id, name)
SELECT id, name FROM unnest(ARRAY['tea', 'coffee', 'snacks', 'spices', 'sweets', 'organic',
'gift', 'vegan', 'sale', 'new', 'local', 'imported'])
WITH ORDINALITY AS t(name, id);
INSERT INTO product_tags SELECT n, 1 + n % 5 FROM generate_series(1, 50000) AS n;
INSERT INTO product_tags SELECT n, 6 + n % 7 FROM generate_series(1, 50000) AS n;
CREATE INDEX product_tags_tag ON product_tags (tag_id, product_id);
VACUUM ANALYZE products, tags, product_tags;
-- Products tagged both tea and organic: one row per tag, grouped back per product.
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF, BUFFERS OFF)
SELECT count(*) FROM (
SELECT pt.product_id FROM product_tags pt JOIN tags t ON t.id = pt.tag_id
WHERE t.name IN ('tea', 'organic')
GROUP BY pt.product_id HAVING count(*) = 2) AS both_tags;
-- Every table and index of this design, and their total.
SELECT coalesce(relname, 'total') AS relation, pg_size_pretty(sum(pg_relation_size(oid))) AS size
FROM pg_class WHERE relnamespace = 'public'::regnamespace GROUP BY ROLLUP (relname) ORDER BY relname;
-- A tag that does not exist is refused by the foreign key.
INSERT INTO product_tags VALUES (1, 99);
-- Renaming a tag changes one row.
UPDATE tags SET name = 'offer' WHERE name = 'sale'; The same question now needs a grouping. The plan read 17,142 rows of product_tags through an index-only scan of
product_tags_tag, 8,571 for each tag on average, grouped them by product and kept the 1,428 products that had both
tags, after the filter removed 14,286 that had only one. The design takes 12 MB against the array’s 5,800 kB, because
every pair is a row of its own with two B-trees over it.
In return it does three things the array could not. The foreign key refused tag 99, which does not exist, since a
foreign key checks that the values of a column match a row of the other table, and an array’s value is the whole
array. The rename was UPDATE 1 in tags instead of 7,143 rewritten products. And a tag can have columns of its own,
a display name or the team that owns it, in one row that every product refers to.
Which design fits depends on what the application asks of its tags. Answer the five questions of the table for a list of your own:
Answer the questions above: the table then marks what fits your case and says why the rest does not.
| Design | Items with tags A and B | Space on disk | Only tags that exist | Facts about each tag | Renaming a tag |
|---|---|---|---|---|---|
| Comma-separated text Stored as: tags text, 'tea,organic' | Poor only patterns, and LIKE '%tea%' matched teapot too | Good 3,072 kB without an index, the smallest of the three | Poor every spelling is stored as it was typed | Poor nowhere to keep them | Poor string surgery on every row that holds the tag |
| Array column Stored as: tags text[] with a GIN index | Good tags @> ARRAY['tea', 'organic'] read 1,428 rows from the GIN bitmap | Good 5,800 kB in all, the GIN index 192 kB of it | Poor a foreign key compares whole values, never each element | Poor an element is only a string | Poor UPDATE 7143: every row that carries the tag |
| Join table Stored as: tags and product_tags | Mixed GROUP BY with HAVING count(*) = 2 grouped 17,142 rows | Mixed 12 MB in all: 100,000 pairs and two B-trees on them | Good the foreign key refused tag 99 | Good columns on tags, one row per tag | Good UPDATE 1, in tags |
How the answers decide
- “Do you look up the items that carry a given tag?” Yes, by tag: rules out Comma-separated text. A string can be searched only by pattern, and LIKE '%tea%' matched teapot; an array or a join table answers exactly.
- “Does each tag need facts of its own, such as a display name or the team that owns it?” Yes, facts per tag: rules out Comma-separated text and Array column. Facts about each tag need a row per tag, which a table of tags holds and an array element or a string cannot.
- “Do tags change apart from the rows that carry them, by being renamed or retired?” Yes, they have a life of their own: rules out Comma-separated text and adds a caveat to Array column. Renaming one tag rewrote 7,143 product rows in the array design and one row of tags in the join design; a string needs surgery in every row.
- “Must every tag come from a list of allowed tags that keeps growing?” Yes, a growing list: rules out Comma-separated text and Array column. Only a join table can point at a table of allowed tags with a foreign key; a CHECK with <@ holds an array to a list written into it.
- “How many tags can one row hold?” Hundreds or more: rules out Comma-separated text and adds a caveat to Array column. Every change rewrites the whole value, an array or a string; for many elements the manual suggests a table with one row per item.
- “Does each tag need facts of its own, such as a display name or the team that owns it?” No, the word is enough and “Do tags change apart from the rows that carry them, by being renamed or retired?” No, they change with the row: adds a caveat to Join table. Nothing here needs a row per tag, so a join table works but costs more: 12 MB against 5,800 kB, and a grouping for every two-tag search.
In short: a short list that is read and written with its row and searched by containment belongs in an array with a GIN index, and items with facts, references or changes of their own belong in rows. A comma-separated text column does neither job well.
Checkpoint
Check: Array column or join table
Arrays turn into rows, and PostgreSQL 18 sorts them
Arrays meet ordinary SQL in two directions. A list of values can go into a query as one array, WHERE id = ANY (ARRAY[17, 42, 99]), which a driver binds as a single parameter however long the list; psycopg 3 adapts a Python list
to an array and recommends = ANY (%s) over IN, which also works for an empty list. In the other direction
unnest() turns an array into rows, and WITH ORDINALITY adds each element’s position. The third run keeps the items
of 10,000 baskets in the order they were added:
arrayfns_session.py
"""Array functions on PostgreSQL 18.6, including two new in 18: runs arrayfns.sql on a throwaway cluster."""
from pg_session import Server
# No autovacuum and no parallel workers, so the plans and counts are the same on every run; UTC as the time zone.
SETTINGS = {"autovacuum": "off", "max_parallel_workers_per_gather": 0, "jit": "off", "timezone": "'UTC'"}
with Server(settings=SETTINGS) as pg:
pg.up()
pg.psql("arrayfns.sql", errors=0) Output
$ psql -X -a -f arrayfns.sql
-- 10,000 baskets, each with its items in the order they were added.
CREATE TABLE baskets (id int PRIMARY KEY, items text[] NOT NULL);
CREATE TABLE
INSERT INTO baskets
SELECT n, CASE n % 3 WHEN 0 THEN ARRAY['tea', 'spices', 'tea', 'sugar']
WHEN 1 THEN ARRAY['coffee', 'sweets'] ELSE '{}' END
FROM generate_series(1, 10000) AS n;
INSERT 0 10000
VACUUM ANALYZE baskets;
VACUUM
-- A list of ids: = ANY (array) and IN (list) use the same B-tree.
EXPLAIN (COSTS OFF) SELECT id FROM baskets WHERE id = ANY (ARRAY[17, 42, 99]);
QUERY PLAN
----------------------------------------------------
Index Only Scan using baskets_pkey on baskets
Index Cond: (id = ANY ('{17,42,99}'::integer[]))
(2 rows)
EXPLAIN (COSTS OFF) SELECT id FROM baskets WHERE id IN (17, 42, 99);
QUERY PLAN
----------------------------------------------------
Index Only Scan using baskets_pkey on baskets
Index Cond: (id = ANY ('{17,42,99}'::integer[]))
(2 rows)
-- An array as numbered rows.
SELECT b.id, i.item, i.pos
FROM baskets AS b, unnest(b.items) WITH ORDINALITY AS i(item, pos) WHERE b.id = 3;
id | item | pos
----+--------+-----
3 | tea | 1
3 | spices | 2
3 | tea | 3
3 | sugar | 4
(4 rows)
-- New in PostgreSQL 18: array_sort(), array_reverse(), min() and max() of arrays.
SELECT items, array_sort(items) AS sorted, array_sort(items, true) AS descending,
array_reverse(items) AS reversed FROM baskets WHERE id = 3;
items | sorted | descending | reversed
------------------------+------------------------+------------------------+------------------------
{tea,spices,tea,sugar} | {spices,sugar,tea,tea} | {tea,tea,sugar,spices} | {sugar,tea,spices,tea}
(1 row)
SELECT min(items), max(items) FROM baskets;
min | max
-----+------------------------
{} | {tea,spices,tea,sugar}
(1 row)
-- An empty array has no dimensions and so no length, but its cardinality is 0.
SELECT array_ndims('{}'::text[]) AS dimensions, array_length('{}'::text[], 1) AS length,
cardinality('{}'::text[]) AS cardinality;
dimensions | length | cardinality
------------+--------+-------------
| | 0
(1 row)
SELECT count(*) FILTER (WHERE array_length(items, 1) = 0) AS by_length,
count(*) FILTER (WHERE cardinality(items) = 0) AS by_cardinality FROM baskets;
by_length | by_cardinality
-----------+----------------
0 | 3333
(1 row)
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 arrayfns_session.py
arrayfns.sql
-- 10,000 baskets, each with its items in the order they were added.
CREATE TABLE baskets (id int PRIMARY KEY, items text[] NOT NULL);
INSERT INTO baskets
SELECT n, CASE n % 3 WHEN 0 THEN ARRAY['tea', 'spices', 'tea', 'sugar']
WHEN 1 THEN ARRAY['coffee', 'sweets'] ELSE '{}' END
FROM generate_series(1, 10000) AS n;
VACUUM ANALYZE baskets;
-- A list of ids: = ANY (array) and IN (list) use the same B-tree.
EXPLAIN (COSTS OFF) SELECT id FROM baskets WHERE id = ANY (ARRAY[17, 42, 99]);
EXPLAIN (COSTS OFF) SELECT id FROM baskets WHERE id IN (17, 42, 99);
-- An array as numbered rows.
SELECT b.id, i.item, i.pos
FROM baskets AS b, unnest(b.items) WITH ORDINALITY AS i(item, pos) WHERE b.id = 3;
-- New in PostgreSQL 18: array_sort(), array_reverse(), min() and max() of arrays.
SELECT items, array_sort(items) AS sorted, array_sort(items, true) AS descending,
array_reverse(items) AS reversed FROM baskets WHERE id = 3;
SELECT min(items), max(items) FROM baskets;
-- An empty array has no dimensions and so no length, but its cardinality is 0.
SELECT array_ndims('{}'::text[]) AS dimensions, array_length('{}'::text[], 1) AS length,
cardinality('{}'::text[]) AS cardinality;
SELECT count(*) FILTER (WHERE array_length(items, 1) = 0) AS by_length,
count(*) FILTER (WHERE cardinality(items) = 0) AS by_cardinality FROM baskets; Both lists of ids ran as the same plan, an index-only scan of the primary key with Index Cond: (id = ANY ('{17,42,99}'::integer[])): PostgreSQL turns an IN list of constants into an array, and the B-tree searches it
value by value. For basket 3, unnest() returned tea, spices, tea and sugar as four rows numbered 1 to 4, the repeated
tea kept in both of its places.
PostgreSQL 18 adds array_sort(), which sorted that basket to {spices,sugar,tea,tea}, and with true as its second
argument to {tea,tea,sugar,spices}, and array_reverse(), which only turned it round, to {sugar,tea,spices,tea}.
min() and max() now take arrays too and compare them element by element: the smallest basket was the empty one,
{}, and the largest the one that starts with tea.
New in PostgreSQL 18
array_sort(), array_reverse(), and min() and max() over arrays and composite types are new in PostgreSQL 18.
On older servers, sort an array with unnest() and array_agg(... ORDER BY ...).
The last two queries show the trap of empty arrays. array_ndims() returned NULL for '{}': an empty array has no
dimensions, so array_length(items, 1) returned NULL for it too, and the filter array_length(items, 1) = 0 counted
none of the 3,333 empty baskets, while cardinality(items), the number of elements, returned 0 and counted all of
them.
Testing for an empty array with array_length()
WHERE array_length(items, 1) = 0 never matches, because the length of a dimension that an empty array does not
have is NULL. Write WHERE cardinality(items) = 0, or compare with the empty array, items = '{}'.
Checkpoint
Check: ANY, unnest and array functions
A domain checks a value once for every column of the type
A domain: A data type made from another type with constraints, NOT NULL and a default added. A value is checked when it is converted to the domain, so every column, cast and argument of the type follows one rule. is a data type made from another one with constraints added. CREATE DOMAIN pin_code AS text CHECK (VALUE ~ '^[1-9][0-9]{5}$') is text that must be six digits, the first of them 1 to 9, and
every column declared as pin_code gets that rule, as does every cast to the type, so the rule is written once
however many tables hold PIN codes. Domains suit any value with one rule and many homes: postal codes, phone numbers,
amounts that must be positive.
A six-digit PIN code
India’s postal code, the PIN code, has six digits written after the name of the locality. In the Universal Postal
Union’s sheet for India, the first digit stands for the state or province, and the digits after it lead on to the
sorting and forwarding centre and then to the delivery office. pin_code
also refuses a first digit of 0, as the contacts table of
From app code: COPY, pipelining, RETURNING does
with an integer column and a range check instead of a pattern.
The fourth run uses the domain in two tables, customers and stores, and writes one good code and one bad one:
pincode_session.py
"""A domain for PIN codes on PostgreSQL 18.6: runs pincode.sql on a throwaway cluster."""
from pg_session import Server
SETTINGS = {"autovacuum": "off", "timezone": "'UTC'"}
with Server(settings=SETTINGS) as pg:
pg.up()
# One error is expected: the store's PIN code starts with 0, which the domain's CHECK refuses.
pg.psql("pincode.sql", errors=1) Output
$ psql -X -a -f pincode.sql
-- A PIN code: six digits, the first of them 1 to 9.
CREATE DOMAIN pin_code AS text CHECK (VALUE ~ '^[1-9][0-9]{5}$');
CREATE DOMAIN
CREATE TABLE customers (id int PRIMARY KEY, name text NOT NULL, pin pin_code NOT NULL);
CREATE TABLE
CREATE TABLE stores (id int PRIMARY KEY, city text NOT NULL, pin pin_code NOT NULL);
CREATE TABLE
INSERT INTO customers VALUES (1, 'Asha Rao', '411001');
INSERT 0 1
INSERT INTO stores VALUES (1, 'Pune', '011001');
psql:pincode.sql:6: ERROR: value for domain pin_code violates check constraint "pin_code_check"
-- Test candidate values without raising an error.
SELECT v AS value, pg_input_is_valid(v, 'pin_code') AS valid
FROM (VALUES ('411001'), ('011001'), ('41100'), ('4110012'), ('41l001'), (' 411001')) AS t(v);
value | valid
---------+-------
411001 | t
011001 | f
41100 | f
4110012 | f
41l001 | f
411001 | f
(6 rows)
SELECT message, sql_error_code FROM pg_input_error_info('011001', 'pin_code');
message | sql_error_code
----------------------------------------------------------------------+----------------
value for domain pin_code violates check constraint "pin_code_check" | 23514
(1 row)
-- One rule, two columns: the catalog knows every column of the domain.
SELECT table_name, column_name FROM information_schema.columns
WHERE domain_name = 'pin_code' ORDER BY table_name;
table_name | column_name
------------+-------------
customers | pin
stores | pin
(2 rows)
SELECT conname, pg_get_constraintdef(oid) FROM pg_constraint WHERE contypid = 'pin_code'::regtype;
conname | pg_get_constraintdef
----------------+-------------------------------------------
pin_code_check | CHECK ((VALUE ~ '^[1-9][0-9]{5}$'::text))
(1 row)
-- Text operators work on the domain, and their results are plain text.
SELECT pg_typeof(pin) AS stored, pg_typeof(pin || '-1') AS concatenated FROM customers;
stored | concatenated
----------+--------------
pin_code | text
(1 row)
-- The domain itself allows NULL; NOT NULL belongs on each column.
SELECT NULL::pin_code IS NULL AS null_is_allowed;
null_is_allowed
-----------------
t
(1 row)
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 pincode_session.py
pincode.sql
-- A PIN code: six digits, the first of them 1 to 9.
CREATE DOMAIN pin_code AS text CHECK (VALUE ~ '^[1-9][0-9]{5}$');
CREATE TABLE customers (id int PRIMARY KEY, name text NOT NULL, pin pin_code NOT NULL);
CREATE TABLE stores (id int PRIMARY KEY, city text NOT NULL, pin pin_code NOT NULL);
INSERT INTO customers VALUES (1, 'Asha Rao', '411001');
INSERT INTO stores VALUES (1, 'Pune', '011001');
-- Test candidate values without raising an error.
SELECT v AS value, pg_input_is_valid(v, 'pin_code') AS valid
FROM (VALUES ('411001'), ('011001'), ('41100'), ('4110012'), ('41l001'), (' 411001')) AS t(v);
SELECT message, sql_error_code FROM pg_input_error_info('011001', 'pin_code');
-- One rule, two columns: the catalog knows every column of the domain.
SELECT table_name, column_name FROM information_schema.columns
WHERE domain_name = 'pin_code' ORDER BY table_name;
SELECT conname, pg_get_constraintdef(oid) FROM pg_constraint WHERE contypid = 'pin_code'::regtype;
-- Text operators work on the domain, and their results are plain text.
SELECT pg_typeof(pin) AS stored, pg_typeof(pin || '-1') AS concatenated FROM customers;
-- The domain itself allows NULL; NOT NULL belongs on each column.
SELECT NULL::pin_code IS NULL AS null_is_allowed; 411001 went into customers. 011001, with its leading 0, stopped the store’s insert with value for domain pin_code violates check constraint "pin_code_check", and the catalog lists one constraint, pin_code_check, for the two
columns customers.pin and stores.pin. Follow both values through the conversion to the domain:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- Two values arrive. A form sends 411001 for a customer and 011001 for a store. Both are valid text, so nothing has checked them yet.
- Conversion runs the rule. Writing a value into a pin_code column converts it to the domain, and so does a cast with ::pin_code. Each conversion evaluates the domain's CHECK, with VALUE standing for the value being converted.
- 411001 matches. The pattern asks for six digits, the first of them 1 to 9. 411001 matches, the CHECK is true and the row is written, INSERT 0 1.
- 011001 is refused. 011001 starts with 0, so the CHECK is false. The INSERT stops with value for domain pin_code violates check constraint pin_code_check, SQLSTATE 23514, and the store's row is not written.
- NULL is no violation. A CHECK that comes out NULL counts as success, so NULL::pin_code is accepted. The manual advises allowing NULL in the domain and putting NOT NULL on each column, as both tables of the run do.
- Test a value first. pg_input_is_valid() runs the same conversion and returns false instead of raising the error, and pg_input_error_info() gives the message and the code 23514. Of six candidates, only 411001 came back true.
- One rule for two tables. customers.pin and stores.pin both use the one constraint pin_code_check, so a change to the rule is one ALTER DOMAIN, and every column of the type follows it.
Two details of the run decide how a domain is written. NULL::pin_code was accepted, because a CHECK that comes out
NULL counts as success; the manual advises leaving NULL allowed in the domain and putting NOT NULL on each column,
as both tables do, because an outer join or an empty subquery can produce a NULL of the domain type anyway. And
pin || '-1' was plain text: an operator of the base type sees the domain value as text, and only a conversion back
to the domain checks the rule again.
pg_input_is_valid() tests a candidate without raising an error, which suits a form that wants to say which field is
wrong: of the six values the run tried, only 411001 was valid, and pg_input_error_info() returned the message and
SQLSTATE 23514, check_violation. A later change of the rule is one ALTER DOMAIN ... ADD CONSTRAINT, which checks
every stored value of every column of the type, or, added NOT VALID, leaves the stored values until ALTER DOMAIN ... VALIDATE CONSTRAINT reads them. One cost comes with the checks: adding a column of a domain that has constraints
to an existing table rewrites the table and its indexes, where a plain column without a default does not.
Key idea
A domain moves a rule from each column to the type: every column, cast and function argument of the type checks it, and one ALTER DOMAIN changes it everywhere.
Checkpoint
Check: Domains and their checks
Enums keep their declared order, and a new value waits for a commit
An enum type: A data type with a fixed list of labels, made with CREATE TYPE ... AS ENUM. Values take 4 bytes and compare in the order the labels were declared; labels can be added and renamed, never removed. is a fixed list of labels, such as the stages of an order. Each value takes 4
bytes in a row, whatever the length of its label; labels are case-sensitive and, in a standard build, at most 63 bytes
long, and values compare in the order they were declared. Enums suit a short list that changes rarely and mostly
grows. The fifth run creates order_status with pending, paid, shipped and cancelled, then adds a value, renames one
and tries to drop one:
enum_session.py
"""Changing an enum type on PostgreSQL 18.6: runs enum.sql on a throwaway cluster."""
from pg_session import Server
SETTINGS = {"autovacuum": "off", "timezone": "'UTC'"}
with Server(settings=SETTINGS) as pg:
pg.up()
# Two errors are expected: the new value used before its transaction commits, and DROP VALUE, which does not exist.
pg.psql("enum.sql", errors=2) Output
$ psql -X -a -f enum.sql
-- Four statuses, in the order an order moves through them.
CREATE TYPE order_status AS ENUM ('pending', 'paid', 'shipped', 'cancelled');
CREATE TYPE
CREATE TABLE orders (id int PRIMARY KEY, status order_status NOT NULL);
CREATE TABLE
INSERT INTO orders VALUES (1, 'shipped'), (2, 'pending'), (3, 'cancelled'), (4, 'paid');
INSERT 0 4
-- Enum values sort in the order they were declared, not alphabetically.
SELECT id, status FROM orders ORDER BY status;
id | status
----+-----------
2 | pending
4 | paid
1 | shipped
3 | cancelled
(4 rows)
SELECT id, status FROM orders WHERE status > 'paid' ORDER BY id;
id | status
----+-----------
1 | shipped
3 | cancelled
(2 rows)
-- A new value inside a transaction cannot be used before that transaction commits.
BEGIN;
BEGIN
ALTER TYPE order_status ADD VALUE 'refunded' BEFORE 'cancelled';
ALTER TYPE
UPDATE orders SET status = 'refunded' WHERE id = 3;
psql:enum.sql:11: ERROR: unsafe use of new value "refunded" of enum type order_status
LINE 1: UPDATE orders SET status = 'refunded' WHERE id = 3;
^
HINT: New enum values must be committed before they can be used.
ROLLBACK;
ROLLBACK
-- Committed first, it can be used at once, and the table keeps its data file.
SELECT pg_relation_filenode('orders') AS file_before \gset
ALTER TYPE order_status ADD VALUE 'refunded' BEFORE 'cancelled';
ALTER TYPE
SELECT pg_relation_filenode('orders') = :file_before AS same_file;
same_file
-----------
t
(1 row)
UPDATE orders SET status = 'refunded' WHERE id = 3;
UPDATE 1
SELECT enum_range(NULL::order_status);
enum_range
-------------------------------------------
{pending,paid,shipped,refunded,cancelled}
(1 row)
SELECT enumlabel, enumsortorder FROM pg_enum
WHERE enumtypid = 'order_status'::regtype ORDER BY enumsortorder;
enumlabel | enumsortorder
-----------+---------------
pending | 1
paid | 2
shipped | 3
refunded | 3.5
cancelled | 4
(5 rows)
-- A value can be renamed, but there is no command to drop one.
ALTER TYPE order_status RENAME VALUE 'shipped' TO 'dispatched';
ALTER TYPE
ALTER TYPE order_status DROP VALUE 'refunded';
psql:enum.sql:23: ERROR: dropping an enum value is not implemented
LINE 1: ALTER TYPE order_status DROP VALUE 'refunded';
^
SELECT enum_range(NULL::order_status);
enum_range
----------------------------------------------
{pending,paid,dispatched,refunded,cancelled}
(1 row)
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 enum_session.py
enum.sql
-- Four statuses, in the order an order moves through them.
CREATE TYPE order_status AS ENUM ('pending', 'paid', 'shipped', 'cancelled');
CREATE TABLE orders (id int PRIMARY KEY, status order_status NOT NULL);
INSERT INTO orders VALUES (1, 'shipped'), (2, 'pending'), (3, 'cancelled'), (4, 'paid');
-- Enum values sort in the order they were declared, not alphabetically.
SELECT id, status FROM orders ORDER BY status;
SELECT id, status FROM orders WHERE status > 'paid' ORDER BY id;
-- A new value inside a transaction cannot be used before that transaction commits.
BEGIN;
ALTER TYPE order_status ADD VALUE 'refunded' BEFORE 'cancelled';
UPDATE orders SET status = 'refunded' WHERE id = 3;
ROLLBACK;
-- Committed first, it can be used at once, and the table keeps its data file.
SELECT pg_relation_filenode('orders') AS file_before \gset
ALTER TYPE order_status ADD VALUE 'refunded' BEFORE 'cancelled';
SELECT pg_relation_filenode('orders') = :file_before AS same_file;
UPDATE orders SET status = 'refunded' WHERE id = 3;
SELECT enum_range(NULL::order_status);
SELECT enumlabel, enumsortorder FROM pg_enum
WHERE enumtypid = 'order_status'::regtype ORDER BY enumsortorder;
-- A value can be renamed, but there is no command to drop one.
ALTER TYPE order_status RENAME VALUE 'shipped' TO 'dispatched';
ALTER TYPE order_status DROP VALUE 'refunded';
SELECT enum_range(NULL::order_status); ORDER BY status returned pending, paid, shipped, cancelled, the order of the declaration and not of the alphabet,
and status > 'paid' found the shipped and the cancelled order. Step through the changes that came next:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- Four labels, in order. order_status starts with pending, paid, shipped and cancelled, stored in the pg_enum catalog with sort orders 1 to 4. ORDER BY status follows those numbers, not the alphabet, and status > paid found the shipped and the cancelled order.
- ADD VALUE inside BEGIN. Inside BEGIN, ADD VALUE 'refunded' BEFORE 'cancelled' writes a fifth pg_enum row with sort order 3.5, halfway between shipped and cancelled. Until the transaction commits, no other session sees it.
- Used too early. The UPDATE that moves order 3 to refunded runs in the same transaction and fails with unsafe use of new value, and the hint says that new enum values must be committed before they can be used.
- ROLLBACK takes it away. The error aborted the transaction, so ROLLBACK was the only way on, and it removed the new label as well. The type was back to its four values.
- Commit first, then use. Run on its own, outside BEGIN, ALTER TYPE commits at once and the table keeps its data file, so no row is rewritten. The next statement may use the label, UPDATE 1, and enum_range() then listed pending, paid, shipped, refunded, cancelled.
- Rename it, never drop it. RENAME VALUE turned shipped into dispatched in place. DROP VALUE stopped with dropping an enum value is not implemented, so a label stays in the type once it is added.
ADD VALUE 'refunded' BEFORE 'cancelled' put the new label between shipped and cancelled. Added inside a transaction
block, a value cannot be used until that transaction has committed, so the UPDATE in the same transaction failed
with unsafe use of new value, and the ROLLBACK that the error forced took the new label away as well. Run on its
own, outside BEGIN, the ALTER TYPE committed at once, the table kept the same data file, so no row was rewritten,
and the next statement could use refunded. The
system catalog: One of the tables in the pg_catalog schema where PostgreSQL records its objects, such as pg_class for relations, pg_attribute for columns and pg_constraint for constraints. They can be queried like any table. More in “System catalogs, pages and TOAST” pg_enum shows how the place was found: the new label got sort order 3.5,
halfway between its neighbours, and the manual notes that comparisons involving a value placed this way can be a
little slower than comparisons of the original values.
Two changes cannot be made to an enum at all. A value can be renamed, as shipped became dispatched in its old place,
but not removed: DROP VALUE stopped with dropping an enum value is not implemented. Nor can the existing values be
put in another order, short of dropping and re-creating the type. A list that must also shrink therefore fits better
as rows of a lookup table, referred to by a foreign key, or as text held to a CHECK constraint.
Drivers see enums as strings: psycopg 3, for one, loads an enum value as a Python string without being told about the type, but an array of an enum type only after the program registers the array’s type.
Checkpoint
Check: Enum order and new values
A composite value travels as one value with named fields
A composite type: A data type made of named fields, each with its own type, like the columns of a row. Every table has one under its own name, and CREATE TYPE ... AS (...) makes a standalone one. is a list of named fields with their types, the shape of a row.
Every table has one under its own name, and CREATE TYPE order_line AS (item text, price numeric(10,2), qty int)
makes a standalone one. A row constructor: An expression that builds a composite value from its fields, written ROW(a, b, c) or, with two fields or more, (a, b, c), often cast to a named composite type. such as ROW('Masala tea', 120.00, 2)
builds a value of it, and the key word ROW may be left out when there are two fields or more. The last run stores one
line per order, with prices in rupees, and reads it in several ways:
composite_session.py
"""A composite type and ROW values on PostgreSQL 18.6: runs composite.sql on a throwaway cluster."""
from pg_session import Server
SETTINGS = {"autovacuum": "off", "timezone": "'UTC'"}
with Server(settings=SETTINGS) as pg:
pg.up()
# One error is expected: line.price without parentheses, which the parser reads as a table's column.
pg.psql("composite.sql", errors=1) Output
$ psql -X -a -f composite.sql
-- One line of an order as a composite type: an item, its price in rupees, a quantity.
CREATE TYPE order_line AS (item text, price numeric(10,2), qty int);
CREATE TYPE
CREATE TABLE orders (id int PRIMARY KEY, line order_line NOT NULL);
CREATE TABLE
INSERT INTO orders VALUES (1, ROW('Masala tea', 120.00, 2)), (2, ('Coffee, filter', 250, 1)),
(3, ROW('Cardamom', 90.50, 3));
INSERT 0 3
-- The whole value prints in its text form, with quotes where a field needs them.
SELECT id, line FROM orders ORDER BY id;
id | line
----+-----------------------------
1 | ("Masala tea",120.00,2)
2 | ("Coffee, filter",250.00,1)
3 | (Cardamom,90.50,3)
(3 rows)
-- Without parentheses, line is read as a table name.
SELECT line.price FROM orders;
psql:composite.sql:9: ERROR: missing FROM-clause entry for table "line"
LINE 1: SELECT line.price FROM orders;
^
SELECT id, (line).item, (line).price * (line).qty AS total FROM orders ORDER BY id;
id | item | total
----+----------------+--------
1 | Masala tea | 240.00
2 | Coffee, filter | 250.00
3 | Cardamom | 271.50
(3 rows)
-- A function takes the whole line as one argument.
CREATE FUNCTION line_total(l order_line) RETURNS numeric
LANGUAGE sql IMMUTABLE RETURN (l).price * (l).qty;
CREATE FUNCTION
SELECT line_total(ROW('Ginger', 40, 5)::order_line) AS ginger;
ginger
--------
200.00
(1 row)
-- New in PostgreSQL 18: max() of a composite compares field by field.
SELECT max(line) FROM orders;
max
-------------------------
("Masala tea",120.00,2)
(1 row)
-- (line).* hands a client plain columns instead of one text value.
SELECT o.id, (o.line).* FROM orders AS o WHERE o.id = 2;
id | item | price | qty
----+----------------+--------+-----
2 | Coffee, filter | 250.00 | 1
(1 row)
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 composite_session.py
composite.sql
-- One line of an order as a composite type: an item, its price in rupees, a quantity.
CREATE TYPE order_line AS (item text, price numeric(10,2), qty int);
CREATE TABLE orders (id int PRIMARY KEY, line order_line NOT NULL);
INSERT INTO orders VALUES (1, ROW('Masala tea', 120.00, 2)), (2, ('Coffee, filter', 250, 1)),
(3, ROW('Cardamom', 90.50, 3));
-- The whole value prints in its text form, with quotes where a field needs them.
SELECT id, line FROM orders ORDER BY id;
-- Without parentheses, line is read as a table name.
SELECT line.price FROM orders;
SELECT id, (line).item, (line).price * (line).qty AS total FROM orders ORDER BY id;
-- A function takes the whole line as one argument.
CREATE FUNCTION line_total(l order_line) RETURNS numeric
LANGUAGE sql IMMUTABLE RETURN (l).price * (l).qty;
SELECT line_total(ROW('Ginger', 40, 5)::order_line) AS ginger;
-- New in PostgreSQL 18: max() of a composite compares field by field.
SELECT max(line) FROM orders;
-- (line).* hands a client plain columns instead of one text value.
SELECT o.id, (o.line).* FROM orders AS o WHERE o.id = 2; line.price failed with missing FROM-clause entry for table "line": a name followed by a dot reads as a table or an
alias, so the field of a column, or of any other expression, is written with the expression in parentheses,
(line).price. That form gave totals of 240.00, 250.00 and 271.50 rupees, and line_total() took a whole line as one
argument and returned 200.00 for five units of ginger at 40 rupees each. max(line), new in PostgreSQL 18, compared
the lines field by field and returned Masala tea, the largest item name. What a client receives depends on how the
query asks for the value:
The drawing shows the whole scene; its 4 steps are listed below.
Every step as text
- A dot alone names a table. line.price reads as the column price of a table or alias called line, and the query has none, so it fails with missing FROM-clause entry for table line. The parser never looks at the column of that name.
- Parentheses make it a value. (line).price first evaluates the column line, then selects the field price of the value it got. The run used it to compute 240.00, 250.00 and 271.50, the price times the quantity of each order.
- The whole value is one string. SELECT line returns one value per row, which a client receives as its text form, ("Coffee, filter",250.00,1). The comma inside the item made the server quote that field. A driver that does not know order_line gets this string and nothing more; psycopg 3, for one, turns it into a tuple only after register_composite().
- Expanded, it is plain columns. (o.line).* expands the value into item, price and qty, three columns with their own types, which every driver and object mapper reads without knowing the composite type.
Sent whole, the value arrives as its text form, ("Coffee, filter",250.00,1), quoted because the item holds a comma,
and a driver that does not know the type gets only that string; psycopg 3 adapts a composite type to a Python tuple
after the program registers it with register_composite(). Expanded with (o.line).*, the same value arrived as three
ordinary columns. Composite types therefore do their best work inside SQL, as function arguments, results and row
comparisons, with plain columns at the edge where a driver or an object mapper takes over.
Expanding a function's result with .*
SELECT (f(x)).* FROM t calls f() once for every field of its result, because PostgreSQL turns the expansion into
one expression per field. Call the function once, as a LATERAL item in FROM, and expand that: SELECT m.* FROM t, LATERAL f(t.x) AS m.
Checkpoint
Check: Composite values and field access
Practise: write arrays out, and choose types
A program that writes an array into a file or a script, rather than passing it as a parameter, must spell it the way the server reads it: braces, commas, and double quotes around any element that holds a comma, a brace, a quote, a backslash or white space, is empty, or is the word NULL. Write that function against its sample tests:
Exercise · Easy · Python
Write a list as a PostgreSQL array value
A program that passes tags to PostgreSQL as a query parameter needs nothing more: the driver sends the list as an array. A program that writes the value into a file or an SQL script must spell it out in the array's text form, the one psql prints, such as {tea,organic}, and the server reads that text back with rules of its own.
Write array_literal(items) in array_literal.py. It takes a list whose items are strings or None and returns the text form of a one-dimensional text array:
- The elements, in list order, separated by commas and wrapped in
{and}. An empty list gives{}. NonebecomesNULL, without quotes: the element is NULL.- A string goes in double quotes when it is empty, when it is the word
NULLin any mix of upper and lower case, or when it contains a brace{or}, a comma, a double quote, a backslash or any white space. Every other string is written as it is. - Inside the double quotes, put a backslash before each double quote and before each backslash of the string.
So ["green tea", "", None] becomes {"green tea","",NULL}. The sample tests check plain tags, strings that need quotes, the word NULL against a real NULL, quotes and backslashes inside an element, and an empty list.
Starter code · array_literal.py
def array_literal(items):
"""Returns the text form of a PostgreSQL text array holding items (strings or None), such as {tea,organic}."""
# Your code here: turn each item into NULL, a quoted string or the string as it is,
# join them with commas and wrap the result in braces.
return "{}" The sample tests · test_array_literal.py
from array_literal import array_literal
def test_plain_tags_need_no_quotes():
assert array_literal(["tea", "organic"]) == "{tea,organic}"
def test_white_space_commas_and_braces_are_quoted():
assert array_literal(["green tea", "tea, masala", "{gift}"]) == '{"green tea","tea, masala","{gift}"}'
def test_empty_strings_and_the_word_null_are_quoted_but_none_is_not():
assert array_literal(["", None, "NULL", "null"]) == '{"",NULL,"NULL","null"}'
def test_quotes_and_backslashes_get_a_backslash():
assert array_literal(['say "chai"', "C:\\tea"]) == '{"say \\"chai\\"","C:\\\\tea"}'
def test_an_empty_list():
assert array_literal([]) == "{}" A hint
Decide for each item on its own, in this order: None first, then whether the string needs quotes (empty, the word NULL with .upper(), or any character from {},"\ or white space, which str.isspace() finds). Escape backslashes before double quotes, or the backslashes you add for the quotes get doubled too.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
The questions below come back to the six runs: the GIN plans, the grouping of the join table, the empty baskets, the PIN codes, the enum changes and the order lines. Before you start, decide which of the four types you would give a column of delivery slots that is always morning, afternoon or evening and sorts in that order:
Practice that adapts to you
Practise this lesson
18 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up for freshers: What does a PostgreSQL domain add over a CHECK constraint written on the column?
A domain is a named type: a base type with constraints, an optional NOT NULL and a default, written once with CREATE DOMAIN. Every column, cast and function argument that uses the type gets the same rule, so a PIN code is six digits in customers, in stores and in a function’s parameter alike, and one ALTER DOMAIN changes the rule and checks every column of the type. A CHECK constraint on a column protects that column only, and each table repeats it. A domain’s CHECK may refer only to VALUE, never to other columns, so a rule that compares two columns stays a table constraint. A domain allows NULL unless told otherwise, and the manual advises putting NOT NULL on the columns instead. Operators of the base type return the base type, so pin || ‘-1’ is plain text and is not checked.
Two questions of this module go further from these runs: a hashtag array whose searches slow down under a stream of new posts, and a status enum on a busy orders table that has to gain one value and stop using another. Each has its first step and its follow-ups on the module’s question page:
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Mid to senior
A social app stores each post's hashtags in a text[] column with a GIN index. At 3,000 new posts a second, hashtag searches slow down at peak, and the product team now wants hourly trending hashtags. Do you keep the array or move to a join table, and how do you index what you keep?
-
Mid to staff
An orders table of 40 million rows has a status column of type order_status (pending, on_hold, paid, shipped, cancelled). This release adds partially_refunded between paid and shipped and retires on_hold, which nothing may write again. Your migration tool runs each file in one transaction. Plan both changes without blocking orders.
Flashcards · 5 cards
Recall it, then check
Say the answer to yourself before you look: recalling it is what makes it stick.
-
How do you search array elements, and how is it indexed?
Write element searches as tags @> ARRAY[...], never = ANY. GIN's array_ops serves &&, @>, <@ and =, keeping a short list inside the row (the tag array took 5,800 kB against 12 MB for a join table).
-
When does a join table beat an array column?
Once the items have facts, foreign keys or changes of their own. A join table can refuse an unknown tag and rename one with UPDATE 1, where the array design had to rewrite every row that held it.
-
Which array helpers matter for queries in PostgreSQL 18?
IN lists run as = ANY over an array in one B-tree scan; unnest() WITH ORDINALITY numbers elements; PostgreSQL 18 adds array_sort(), array_reverse() and min()/max() over arrays; use cardinality(), not array_length(), to find empty arrays.
-
What does a domain do, and how does it treat NULL?
A domain puts one rule on a type, checked whenever a value is converted to it (e.g. pin_code accepts 411001, refuses 011001). It allows NULL by default and leaves NOT NULL to the columns.
-
How do enum types sort and change?
Enums sort in their declared order and take 4 bytes a value. A value added inside a transaction can be used only after that transaction commits, and values can be renamed but never dropped.
Key takeaways
- Arrays keep a short list inside the row, and GIN’s
array_opsserves&&,@>,<@and=, so element searches are written astags @> ARRAY[...], never= ANY; the tag array took 5,800 kB against 12 MB for the join table. - A join table wins once the items have facts, foreign keys or changes of their own: it refused an unknown tag and
renamed one with
UPDATE 1, where the array design rewrote 7,143 rows. INlists run as= ANYover an array in one B-tree scan,unnest() WITH ORDINALITYnumbers the elements, PostgreSQL 18 addsarray_sort(),array_reverse()andmin()andmax()over arrays, andcardinality(), notarray_length(), finds empty arrays.- Domains put one rule on a type, checked whenever a value is converted to it:
pin_codeaccepted 411001 and refused 011001 for two tables, allowed NULL, and leftNOT NULLto the columns. - Enums sort in their declared order and take 4 bytes a value; a value added inside a transaction can be used only after that transaction commits, and values can be renamed but never dropped.
- Composite types carry named fields as one value: read a field as
(line).price, pass whole values to functions, and give clients plain columns with(value).*.
Terms in this lesson
- composite type
- A data type made of named fields, each with its own type, like the columns of a row. Every table has one under its own name, and CREATE TYPE ... AS (...) makes a standalone one.
- domain
- A data type made from another type with constraints, NOT NULL and a default added. A value is checked when it is converted to the domain, so every column, cast and argument of the type follows one rule.
- enum type
- A data type with a fixed list of labels, made with CREATE TYPE ... AS ENUM. Values take 4 bytes and compare in the order the labels were declared; labels can be added and renamed, never removed.
- join table
- A table with one row for each pairing of rows from two other tables, such as one row per product and tag, holding a foreign key to each side. Facts about the pairing itself can be columns of it.
- row constructor
- An expression that builds a composite value from its fields, written ROW(a, b, c) or, with two fields or more, (a, b, c), often cast to a named composite type.
References
- PostgreSQL 18 documentation: 8.15 Arrays (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: 9.19 Array Functions and Operators (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: 9.25 Row and Array Comparisons (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: 7.2 Table Expressions (WITH ORDINALITY and LATERAL) (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: 65.4 GIN Indexes (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: 11.2 Index Types (B-tree searches for IN) (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: 5.5 Constraints (foreign keys) (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: 8.18 Domain Types (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: CREATE DOMAIN (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: ALTER DOMAIN (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: 9.27 System Information Functions (data validity checking) (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: Appendix A. PostgreSQL Error Codes (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: ALTER TABLE (when a table is rewritten) (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: 8.7 Enumerated Types (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: ALTER TYPE (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: 9.10 Enum Support Functions (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: 8.16 Composite Types (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: 36.5 Query Language (SQL) Functions (The PostgreSQL Global Development Group)
- PostgreSQL 18 documentation: 9.21 Aggregate Functions (min and max) (The PostgreSQL Global Development Group)
- PostgreSQL 18 release notes (The PostgreSQL Global Development Group)
- psycopg 3 documentation: Adapting basic Python types (lists, enums) (The Psycopg Team)
- psycopg 3 documentation: Adapting other PostgreSQL types (composite types) (The Psycopg Team)
- Postal addressing systems: India (Universal Postal Union)
Report a problem with this lesson
Kept only in this browser. Your Learn progress