GIS & Geospatial Data (QGIS, GeoPandas, PostGIS) Module 3 – Vector geometry and file formats
Shapefile, GeoPackage and FlatGeobuf
What a shapefile's files hold and what it loses, a GeoPackage built by hand in SQLite, FlatGeobuf's index, and how to pick and convert a format safely.
What you will learn
- List the files of a shapefile and its limits on field names, types, text, size and geometry types
- Store a project of several layers in a GeoPackage and read any GeoPackage with SQLite
- Choose between Shapefile, GeoPackage, FlatGeobuf, GeoJSON, KML, GPX and CSV for a hand-off
- Convert between formats without losing field names, types, text or the CRS, and check the result
Before you start
What you’ll learn · Before you start
What you will learn
- List the files of a shapefile and its limits on field names, types, text, size and geometry types
- Store a project of several layers in a GeoPackage and read any GeoPackage with SQLite
- Choose between Shapefile, GeoPackage, FlatGeobuf, GeoJSON, KML, GPX and CSV for a hand-off
- Convert between formats without losing field names, types, text or the CRS, and check the result
Before you start
On this page
Send a partner a ward layer as a shapefile: A vector format of several files with one name: .shp for the geometry, .shx for its index and .dbf for the attributes, with .prj for the CRS and .cpg for the text encoding beside them. and the field population_estimate arrives as
population, the survey times arrive as text, and a Marathi note of 147 characters arrives cut to 92, in the middle of
a word. Nothing failed: GDAL warned four times and wrote the file anyway.
Every format is a set of choices about what to keep. Here you take a shapefile apart file by file and byte by byte,
measure what it loses, build a GeoPackage by hand with nothing but Python’s sqlite3 and open it in GDAL, see how
FlatGeobuf’s index reorders a layer, compare seven formats on the same data, and repair a damaged shapefile into a
GeoPackage with every step written down.
A shapefile is several files that must travel together
The ESRI Shapefile Technical Description defines a shapefile as “a main file, an index file, and a dBASE table”, files
with the same name and the endings .shp, .shx and .dbf
(Esri, Shapefile Technical Description).
The main file holds the shapes, the index file the position of each shape in it, and the table one row of attributes
per shape, matched by record number. GDAL adds two side files: the .prj file: The sidecar file of a shapefile that holds its CRS as one line of WKT1, usually in ESRI's dialect and often without an authority code. More in “EPSG codes, WKT2 and PROJJSON in practice” with the
coordinate reference system, and .cpg with the text encoding of the table
(GDAL, Shapefile driver).
Step through a reader taking one ward out of the five files the next program writes:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- Five files, one name. A shapefile is a set of files with one name. The ESRI description requires three of them, .shp, .shx and .dbf; GDAL also writes .prj for the CRS and .cpg for the text encoding.
- The main file's header. The reader opens wards.shp. Its 100-byte header holds file code 9994, version 1000, shape type 5 for polygons and the box of the whole layer.
- The index points the way. To reach ward W02 without reading W01, the reader looks in wards.shx: 8 bytes per record, an offset and a length. 100 + 12 × 8 makes its 196 bytes.
- The polygon. At that offset in wards.shp is W02's record: its box, its parts and its points. The outer ring runs clockwise, as the shapefile convention wants.
- The attributes. Row 2 of wards.dbf holds W02's attributes. Every row is the same width, 105 bytes, because each field has a fixed width: 80 characters for the ward's name.
- The encoding. wards.cpg holds one word, UTF-8, so the reader decodes the text correctly. Without it the reader guesses from the .dbf, and Marathi names can turn into question marks.
- The coordinate system. wards.prj holds the CRS as WKT, starting PROJCS["WGS_1984_UTM_Zone_43N". Lose this file and the coordinates are numbers that belong to no place; that is why the five files travel together, often zipped.
The program writes the twelve wards with GeoPandas, which writes through GDAL by way of pyogrio, lists the files, and
then reads the headers of the .shp and .dbf with Python’s struct module:
"""The twelve wards written as a shapefile by GDAL (through GeoPandas and pyogrio): the files it makes, and what the
headers of the .shp and .dbf say when read byte by byte."""
import os
import struct
import pyogrio
from town_wards import wards
wards().to_file("wards.shp")
for name in sorted(f for f in os.listdir(".") if f.startswith("wards.")):
print(f"{name:10} {os.path.getsize(name):6,} bytes")
print(".cpg:", open("wards.cpg").read())
print(".prj starts:", open("wards.prj").read()[:52], "…")
# The .shp header: big-endian file code and length, little-endian version, shape type and bounding box.
head = open("wards.shp", "rb").read(100)
code, = struct.unpack(">i", head[0:4])
words, = struct.unpack(">i", head[24:28])
version, shape_type = struct.unpack("<ii", head[28:36])
xmin, ymin, xmax, ymax = struct.unpack("<4d", head[36:68])
print(f".shp: file code {code}, {words * 2:,} bytes, version {version}, shape type {shape_type} (polygon)")
print(f" box {xmin:.0f}, {ymin:.0f} to {xmax:.0f}, {ymax:.0f}")
# The .dbf header: record count, then one 32-byte descriptor per field, its name in 11 bytes.
dbf = open("wards.dbf", "rb").read()
records, header_len, record_len = struct.unpack("<IHH", dbf[4:12])
print(f".dbf: {records} records of {record_len} bytes")
for i in range((header_len - 33) // 32):
d = dbf[32 + 32 * i : 64 + 32 * i]
name = d[:11].split(b"\0")[0].decode()
print(f" field {name!r}: type {chr(d[11])}, {d[16]} wide, {d[17]} decimals")
print(f"GDAL {pyogrio.__gdal_version_string__} (pyogrio {pyogrio.__version__})")Output
wards.cpg 5 bytes
wards.dbf 1,358 bytes
wards.prj 401 bytes
wards.shp 1,732 bytes
wards.shx 196 bytes
.cpg: UTF-8
.prj starts: PROJCS["WGS_1984_UTM_Zone_43N",GEOGCS["GCS_WGS_1984" …
.shp: file code 9994, 1,732 bytes, version 1000, shape type 5 (polygon)
box 500000, 2000000 to 508000, 2006000
.dbf: 12 records of 105 bytes
field 'ward': type C, 80 wide, 0 decimals
field 'area_ha': type N, 24 wide, 15 decimals
GDAL 3.12.4 (pyogrio 0.13.0)
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 files.py
The .dbf explains the format’s best-known limit. Each field is described in 32 bytes, its name in 11 of them, so a
name has room for 10 characters and a terminating zero. The rows are fixed width: GDAL gave the ward’s name 80
characters, its default for text without a width, so every row takes 105 bytes whatever it holds.
Key idea
A shapefile is three required files and two that matter as much: without the .prj the coordinates belong to no place, and without the .cpg the reader has to guess how the text was encoded.
The shapefile changes your data on the way in
The fixed-width table is a dBASE file, and GDAL’s documentation lists what it cannot hold: names “can only be up to 10 characters long”, and only Integer, Integer64, Real, String and Date fields exist, “not DateTime, just year/month/day”. A shapefile also holds “only one kind of geometry per layer”, and its size is bounded: the format’s 32-bit offsets allow 8 GB, GDAL’s implementation 4 GB, and GDAL advises against files over 2 GB for compatibility. The program writes an awkward little table of two wards to three formats and reads each back:
"""An awkward little table, written to a shapefile, a GeoPackage and a FlatGeobuf file by GDAL (through GeoPandas and
pyogrio), and read back: which names, types, texts and geometries survive each format."""
import warnings
import geopandas as gpd
import pandas as pd
import pyogrio
from shapely.geometry import MultiPolygon, Point, Polygon
def square(x0, y0, side=2_000):
return Polygon([(x0, y0), (x0 + side, y0), (x0 + side, y0 + side), (x0, y0 + side)])
notes = "नदीकाठचा प्रभाग, जुन्या बाजारपेठेपासून पूर्वेला. " * 3 # "a riverside ward, east of the old market" in Marathi
table = gpd.GeoDataFrame(
{
"ward": ["W01", "W02"],
"population_estimate": [11_872, 9_640],
"surveyed_at": pd.to_datetime(["2026-03-14 10:30", "2026-03-14 11:05"]),
"notes_mr": [notes, "बाजारपेठ"],
},
geometry=[square(500_000, 2_004_000), MultiPolygon([square(502_000, 2_004_000, 1_000), square(503_000, 2_005_000, 1_000)])],
crs="EPSG:32643",
)
print(f"notes of W01: {len(notes)} characters, {len(notes.encode())} bytes in UTF-8")
for driver, path in [("ESRI Shapefile", "wards.shp"), ("GPKG", "wards.gpkg"), ("FlatGeobuf", "wards.fgb")]:
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
table.to_file(path, driver=driver)
back = gpd.read_file(path).sort_values("ward")
print(f"\n{driver}: columns {', '.join(c for c in back.columns if c != 'geometry')}")
row = back.iloc[0]
when = [c for c in back.columns if c.startswith("survey")][0]
note = [c for c in back.columns if c.startswith("notes")][0]
print(f" {when}: {row[when]!r} ({back[when].dtype})")
print(f" {note}: {len(row[note])} of {len(notes)} characters kept, ends …{row[note][-12:]}")
print(f" geometries: {', '.join(g.geom_type for g in back.geometry)}; CRS {back.crs.to_epsg()}")
for w in caught:
message = str(w.message)
if message.startswith("Value '"): # the warning repeats the whole value; keep its point
message = "Value '…' " + message[message.index("of field") :].split(" ")[0]
print(f" warning: {message}")
mixed = gpd.GeoDataFrame({"name": ["C2", "W02"]}, geometry=[Point(504_000, 2_005_000), square(502_000, 2_004_000)], crs="EPSG:32643")
for driver, path in [("ESRI Shapefile", "mixed.shp"), ("GPKG", "mixed.gpkg"), ("FlatGeobuf", "mixed.fgb")]:
try:
mixed.to_file(path, driver=driver)
print(f"{driver}, a point and a polygon in one layer: kept as {', '.join(g.geom_type for g in gpd.read_file(path).geometry)}")
except Exception as e:
print(f"{driver}, a point and a polygon in one layer: {type(e).__name__}: {str(e).split(': ', 1)[-1]}")
print(f"\nGDAL {pyogrio.__gdal_version_string__} (pyogrio {pyogrio.__version__})")Output
notes of W01: 147 characters, 399 bytes in UTF-8
ESRI Shapefile: columns ward, population, surveyed_a, notes_mr
surveyed_a: '2026/03/14 10:30:00' (str)
notes_mr: 92 of 147 characters kept, ends …ठेपासून पूर्
geometries: Polygon, MultiPolygon; CRS 32643
warning: Column names longer than 10 characters will be truncated when saved to ESRI Shapefile.
warning: Normalized/laundered field name: 'population_estimate' to 'population'
warning: Normalized/laundered field name: 'surveyed_at' to 'surveyed_a'
warning: Field surveyed_a created as String field, though DateTime requested.
warning: Value '…' of field notes_mr has been truncated to 254 characters.
GPKG: columns ward, population_estimate, surveyed_at, notes_mr
surveyed_at: Timestamp('2026-03-14 10:30:00') (datetime64[ms])
notes_mr: 147 of 147 characters kept, ends …न पूर्वेला.
geometries: MultiPolygon, MultiPolygon; CRS 32643
FlatGeobuf: columns ward, population_estimate, surveyed_at, notes_mr
surveyed_at: Timestamp('2026-03-14 10:30:00') (datetime64[ms])
notes_mr: 147 of 147 characters kept, ends …न पूर्वेला.
geometries: MultiPolygon, MultiPolygon; CRS 32643
ESRI Shapefile, a point and a polygon in one layer: FeatureError: Attempt to write non-point (POLYGON) geometry to point shapefile.
GPKG, a point and a polygon in one layer: kept as Point, Polygon
FlatGeobuf, a point and a polygon in one layer: kept as Point, Polygon
GDAL 3.12.4 (pyogrio 0.13.0)
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 awkward.py
| What was written | Shapefile | GeoPackage | FlatGeobuf |
|---|---|---|---|
population_estimate |
population |
kept | kept |
surveyed_at, a date and time |
text, YYYY/MM/DD hh:mm:ss |
kept | kept |
| 147 Marathi characters | 92 kept | 147 kept | 147 kept |
| A point and a polygon in one layer | refused | kept | kept |
The Marathi note shows a limit counted in bytes, not characters: a text field of the .dbf holds at most 254 bytes, and
each Devanagari character takes 3 bytes in UTF-8. The note’s 399 bytes had to fit in 254, so GDAL kept the 92 whole
characters that fit, 252 bytes, and cut inside the word पूर्वेला. English text gets about three times as many
characters into the same field, so Indian-language text loses about two-thirds of its room in a shapefile.
Reading the warnings as noise
GDAL said exactly what it did, in four lines: two laundered names, a DateTime written as String and a truncated value. A script that hides warnings ships the damage. Read them, or write a GeoPackage, which needed none of these changes.
Checkpoint
Check: The files of a shapefile · What a shapefile loses
A GeoPackage is one SQLite file with many layers
A GeoPackage: An OGC standard for geospatial data in one SQLite database file (.gpkg), with required tables for its coordinate systems, layers and geometry columns, and any number of layers. is an OGC standard: “a platform-independent SQLite database file” with
tables for its data and metadata (OGC GeoPackage). The standard fixes the rest:
the file’s application id is 0x47504B47, “GPKG” in ASCII; gpkg_spatial_ref_sys lists the coordinate systems, with
rows -1 and 0 for undefined ones; gpkg_contents lists the layers; gpkg_geometry_columns names each layer’s geometry
column and type; and each geometry is a short header, the letters GP, a version, flags and an SRS id, followed by WKB.
Step through a reader finding a layer:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- Is it a GeoPackage?. The file starts like every SQLite database, with "SQLite format 3". The standard adds the application id GPKG at byte 68 and a version, 10400 for 1.4.0, in the user_version field.
- The list of layers. gpkg_contents lists every table of data. Here one row says clinics holds features in SRS 4326, with the layer's box. A GeoPackage may list many layers here.
- The geometry column. gpkg_geometry_columns tells the reader which column holds the shapes and of what type: column geom, type POINT, no Z, no M.
- The coordinate system. srs_id 4326 leads to a row of gpkg_spatial_ref_sys whose definition is the WKT of WGS 84. Two more rows, -1 and 0, stand for undefined systems, as the standard requires.
- The rows. The clinics table is an ordinary SQLite table: a fid, the geom column and a name, eight rows in all. Any SQLite program can query it.
- One blob. C1's geom is 29 bytes. The first 8, 47 50 00 01 E6 10 00 00, say GP, version 0, flags 1 for little-endian with no envelope, and SRS 4326. The 21 bytes after them are the WKB point you know.
- The position. The WKB holds type 1, a Point, then 75.01418 and 18.135709: clinic C1 in longitude and latitude. GDAL read the same file and found 8 points in EPSG:4326.
Because it is SQLite, a GeoPackage can be written without any GIS library. This program builds one with Python’s
sqlite3 module and the required tables, puts the town’s eight clinics in it, and reads them back as the figure did:
"""A GeoPackage made by hand with Python's sqlite3 module, following the OGC standard's required tables, then read
back the way any reader does: from gpkg_contents to the geometry blobs of one layer."""
import sqlite3
import struct
WGS84 = 'GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433]]'
# The town's eight clinics in WGS 84 longitude and latitude (converted from UTM zone 43N).
CLINICS = {"C1": (75.014180, 18.135709), "C2": (75.037813, 18.133898), "C3": (75.061444, 18.126661), "C4": (75.025522, 18.116727),
"C5": (75.049152, 18.114011), "C6": (75.008506, 18.098651), "C7": (75.052929, 18.099548), "C8": (75.069940, 18.096831)}
def blob(lon, lat, srs_id=4326):
"""A GeoPackage geometry: 'GP', version 0, flags (no envelope, little-endian), the SRS id, then WKB."""
return b"GP" + bytes([0, 0b00000001]) + struct.pack("<i", srs_id) + struct.pack("<BIdd", 1, 1, lon, lat)
def make_geopackage(path):
db = sqlite3.connect(path)
db.execute("PRAGMA application_id = 1196444487") # 0x47504B47, "GPKG"
db.execute("PRAGMA user_version = 10400") # version 1.4.0
db.execute("CREATE TABLE gpkg_spatial_ref_sys (srs_name TEXT NOT NULL, srs_id INTEGER PRIMARY KEY, organization TEXT NOT NULL,"
" organization_coordsys_id INTEGER NOT NULL, definition TEXT NOT NULL, description TEXT)")
db.executemany("INSERT INTO gpkg_spatial_ref_sys VALUES (?, ?, ?, ?, ?, ?)", [
("Undefined Cartesian SRS", -1, "NONE", -1, "undefined", "undefined Cartesian coordinate reference system"),
("Undefined geographic SRS", 0, "NONE", 0, "undefined", "undefined geographic coordinate reference system"),
("WGS 84 geodetic", 4326, "EPSG", 4326, WGS84, "longitude and latitude in decimal degrees on WGS 84")])
db.execute("CREATE TABLE gpkg_contents (table_name TEXT NOT NULL PRIMARY KEY, data_type TEXT NOT NULL, identifier TEXT UNIQUE,"
" description TEXT DEFAULT '', last_change DATETIME NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')),"
" min_x DOUBLE, min_y DOUBLE, max_x DOUBLE, max_y DOUBLE, srs_id INTEGER REFERENCES gpkg_spatial_ref_sys(srs_id))")
db.execute("CREATE TABLE gpkg_geometry_columns (table_name TEXT NOT NULL, column_name TEXT NOT NULL, geometry_type_name TEXT NOT NULL,"
" srs_id INTEGER NOT NULL, z TINYINT NOT NULL, m TINYINT NOT NULL, PRIMARY KEY (table_name, column_name))")
db.execute("CREATE TABLE clinics (fid INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, geom POINT, name TEXT)")
db.executemany("INSERT INTO clinics (geom, name) VALUES (?, ?)", [(blob(*p), n) for n, p in CLINICS.items()])
lons, lats = [p[0] for p in CLINICS.values()], [p[1] for p in CLINICS.values()]
db.execute("INSERT INTO gpkg_contents (table_name, data_type, identifier, min_x, min_y, max_x, max_y, srs_id)"
" VALUES ('clinics', 'features', 'clinics', ?, ?, ?, ?, 4326)", (min(lons), min(lats), max(lons), max(lats)))
db.execute("INSERT INTO gpkg_geometry_columns VALUES ('clinics', 'geom', 'POINT', 4326, 0, 0)")
db.commit()
db.close()
if __name__ == "__main__":
make_geopackage("clinics.gpkg")
head = open("clinics.gpkg", "rb").read(100)
print("first 16 bytes:", head[:16])
print("application_id:", head[68:72], "user_version:", struct.unpack(">i", head[60:64])[0])
db = sqlite3.connect("clinics.gpkg")
for name, kind, srs in db.execute("SELECT table_name, data_type, srs_id FROM gpkg_contents"):
column, gtype = db.execute("SELECT column_name, geometry_type_name FROM gpkg_geometry_columns WHERE table_name = ?", (name,)).fetchone()
count = db.execute(f'SELECT count(*) FROM "{name}"').fetchone()[0]
print(f"layer {name}: {kind}, {count} {gtype.lower()}s in column {column}, SRS {srs}")
for fid, g, label in db.execute(f'SELECT fid, "{column}", name FROM "{name}" ORDER BY fid LIMIT 2'):
magic, version, flags = g[:2], g[2], g[3]
srs_id = struct.unpack("<i" if flags & 1 else ">i", g[4:8])[0]
envelope = {0: 0, 1: 32, 2: 48, 3: 48, 4: 64}[(flags >> 1) & 0b111]
wkb = g[8 + envelope :]
_, kind_code, lon, lat = struct.unpack("<BIdd", wkb)
print(f" {label}: {g[:8].hex(' ').upper()} | WKB type {kind_code}: {lon}, {lat} ({len(g)} bytes)")Output
first 16 bytes: b'SQLite format 3\x00' application_id: b'GPKG' user_version: 10400 layer clinics: features, 8 points in column geom, SRS 4326 C1: 47 50 00 01 E6 10 00 00 | WKB type 1: 75.01418, 18.135709 (29 bytes) C2: 47 50 00 01 E6 10 00 00 | WKB type 1: 75.037813, 18.133898 (29 bytes)
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 by_hand.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
GDAL opens the hand-made file like any other, and then writes the whole town, four layers, into one file:
"""The track's made-up town as four layers in UTM zone 43N (EPSG:32643): the same wards, households, clinics and roads
as the foundations lessons, moved from the town's own grid to 500,000 m east and 2,000,000 m north."""
import random
import geopandas as gpd
from shapely.geometry import LineString, MultiLineString, Point, Polygon
E0, N0, WARD = 500_000, 2_000_000, 2_000
CLINICS = {"C1": (1_500, 5_200), "C2": (4_000, 5_000), "C3": (6_500, 4_200), "C4": (2_700, 3_100),
"C5": (5_200, 2_800), "C6": (900, 1_100), "C7": (5_600, 1_200), "C8": (7_400, 900)}
JUNCTIONS = {"west gate": (0, 2_600), "J2": (3_000, 2_600), "Station Square": (3_500, 2_800), "J4": (5_000, 3_400),
"J5": (6_500, 3_400), "east gate": (8_000, 3_400), "north gate": (3_500, 6_000), "J8": (3_500, 5_000),
"south gate": (3_500, 0), "ring south": (5_500, 0), "J11": (6_500, 1_800), "C3 corner": (6_500, 4_200)}
STREETS = [("west gate", "J2", "Highway"), ("J2", "Station Square", "Highway"), ("Station Square", "J4", "Highway"),
("J4", "J5", "Highway"), ("J5", "east gate", "Highway"), ("north gate", "J8", "Station Road"),
("J8", "Station Square", "Station Road"), ("Station Square", "south gate", "Station Road"),
("ring south", "J11", "Ring Road"), ("J11", "J5", "Ring Road"), ("J5", "C3 corner", "Ring Road"),
("J8", "C3 corner", "Hill Street")]
def at(x, y):
return E0 + x, N0 + y
def wards():
rows = []
for r in range(3):
for c in range(4):
x0, y0 = c * WARD, (2 - r) * WARD
ring = [at(x0, y0), at(x0 + WARD, y0), at(x0 + WARD, y0 + WARD), at(x0, y0 + WARD)]
rows.append({"ward": f"W{r * 4 + c + 1:02d}", "geometry": Polygon(ring)})
return gpd.GeoDataFrame(rows, crs="EPSG:32643")
def households(seed=42, n=5_000):
rng = random.Random(seed)
centres = [(2_600, 4_300, 700), (5_600, 1_500, 900), (6_900, 4_700, 500)]
rows = []
while len(rows) < n:
if rng.random() < 0.7:
cx, cy, spread = rng.choice(centres)
x = cx + spread * 2 * (rng.random() + rng.random() + rng.random() - 1.5)
y = cy + spread * 2 * (rng.random() + rng.random() + rng.random() - 1.5)
else:
x, y = rng.uniform(0, 4 * WARD), rng.uniform(0, 3 * WARD)
if 0 < x < 4 * WARD and 0 < y < 3 * WARD:
rows.append({"household": f"H{len(rows) + 1:04d}", "geometry": Point(at(round(x, 1), round(y, 1)))})
return gpd.GeoDataFrame(rows, crs="EPSG:32643")
def clinics():
return gpd.GeoDataFrame([{"clinic": k, "geometry": Point(at(*p))} for k, p in CLINICS.items()], crs="EPSG:32643")
def roads():
parts = {}
for a, b, road in STREETS:
parts.setdefault(road, []).append(LineString([at(*JUNCTIONS[a]), at(*JUNCTIONS[b])]))
return gpd.GeoDataFrame([{"road": k, "geometry": MultiLineString(v)} for k, v in parts.items()], crs="EPSG:32643")"""GDAL reads the hand-made GeoPackage like any other; then the whole town goes into one GeoPackage of four layers,
and sqlite3 shows the tables GDAL wrote."""
import os
import sqlite3
import pyogrio
import town
from by_hand import make_geopackage
make_geopackage("clinics.gpkg")
print("hand-made file, as GDAL sees it:", [list(layer) for layer in pyogrio.list_layers("clinics.gpkg")])
info = pyogrio.read_info("clinics.gpkg")
print(f" {info['features']} features, CRS {info['crs']}, fields {list(info['fields'])}")
for name, layer in [("wards", town.wards()), ("households", town.households()), ("clinics", town.clinics()), ("roads", town.roads())]:
layer.to_file("town.gpkg", layer=name, driver="GPKG")
print(f"\ntown.gpkg: {os.path.getsize('town.gpkg'):,} bytes")
db = sqlite3.connect("town.gpkg")
print("user_version:", db.execute("PRAGMA user_version").fetchone()[0])
for name, kind, srs in db.execute("SELECT table_name, data_type, srs_id FROM gpkg_contents ORDER BY table_name"):
gtype = db.execute("SELECT geometry_type_name FROM gpkg_geometry_columns WHERE table_name = ?", (name,)).fetchone()[0]
count = db.execute(f'SELECT count(*) FROM "{name}"').fetchone()[0]
print(f" {name:10} {kind}, {count:5} {gtype}, srs_id {srs}")
names = [r[0] for r in db.execute("SELECT name FROM sqlite_schema WHERE type = 'table' ORDER BY name")]
print("metadata tables:", ", ".join(n for n in names if n.startswith("gpkg_")))
print("feature tables:", ", ".join(n for n in names if n in ("wards", "households", "clinics", "roads")))
extension = "SELECT table_name FROM gpkg_extensions WHERE extension_name = 'gpkg_rtree_index' ORDER BY table_name"
print("R-tree spatial index on:", ", ".join(r[0] for r in db.execute(extension)))
print(f"GDAL {pyogrio.__gdal_version_string__} (pyogrio {pyogrio.__version__})")Output
hand-made file, as GDAL sees it: [['clinics', 'Point']] 8 features, CRS EPSG:4326, fields ['name'] town.gpkg: 643,072 bytes user_version: 10400 clinics features, 8 POINT, srs_id 32643 households features, 5000 POINT, srs_id 32643 roads features, 4 MULTILINESTRING, srs_id 32643 wards features, 12 POLYGON, srs_id 32643 metadata tables: gpkg_contents, gpkg_extensions, gpkg_geometry_columns, gpkg_ogr_contents, gpkg_spatial_ref_sys, gpkg_tile_matrix, gpkg_tile_matrix_set feature tables: clinics, households, roads, wards R-tree spatial index on: clinics, households, roads, wards GDAL 3.12.4 (pyogrio 0.13.0)
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 gdal_reads_it.py
One 643,072-byte file holds 12 wards, 5,000 households, 8 clinics and 4 roads, each layer with its CRS, plus an R-tree spatial index: A structure, such as an R-tree, that finds the features in or near an area without testing every feature, by grouping them into nested bounding boxes. for every layer. The tables are ordinary SQLite, so any SQLite program can query them, and a GeoPackage can be updated in place, row by row, as any database can.
SQLite Viewer & Editor Open any GeoPackage and browse gpkg_contents and the feature tables: it is an SQLite database.Checkpoint
Check: Inside a GeoPackage
FlatGeobuf stores features in the order of its index
FlatGeobuf: A binary format for one vector layer, built on FlatBuffers, with an optional packed Hilbert R-tree index that lets a reader fetch only the features of a bounding box, even over HTTP. is “a performant binary encoding for geographic data based on flatbuffers”. Its file is magic bytes, a header, an optional “static packed Hilbert R-tree index” and the features, and it is built so that “fetching a subset of a file over HTTP utilizes Range requests” (FlatGeobuf). It “deliberately does not support random writes”: to change a feature, write the file again. GDAL has written it since version 3.1, with the index on by default (GDAL, FlatGeobuf driver). The program writes the twelve wards with and without the index:
"""The twelve wards written as FlatGeobuf with and without its spatial index: the first bytes of the file, the order the
features come back in, and a read of one bounding box."""
import os
import geopandas as gpd
import pyogrio
from shapely.geometry import Polygon
E0, N0, WARD = 500_000, 2_000_000, 2_000
rows = []
for r in range(3):
for c in range(4):
x0, y0 = E0 + c * WARD, N0 + (2 - r) * WARD
rows.append({"ward": f"W{r * 4 + c + 1:02d}", "geometry": Polygon([(x0, y0), (x0 + WARD, y0), (x0 + WARD, y0 + WARD), (x0, y0 + WARD)])})
wards = gpd.GeoDataFrame(rows, crs="EPSG:32643")
print("written in the order:", " ".join(wards.ward))
for label, index in [("with the index", "YES"), ("without it", "NO")]:
path = f"wards_{index.lower()}.fgb"
wards.to_file(path, driver="FlatGeobuf", SPATIAL_INDEX=index)
back = gpd.read_file(path)
print(f"{label}: {os.path.getsize(path):,} bytes, read back as {' '.join(back.ward)}")
magic = open("wards_yes.fgb", "rb").read(8)
print("first 8 bytes:", magic.hex(" ").upper(), magic)
box = (502_500, 2_004_500, 503_500, 2_005_500) # the park inside W02
print("features in the park's box:", " ".join(pyogrio.read_dataframe("wards_yes.fgb", bbox=box).ward))
print(f"GDAL {pyogrio.__gdal_version_string__} (pyogrio {pyogrio.__version__})")Output
written in the order: W01 W02 W03 W04 W05 W06 W07 W08 W09 W10 W11 W12 with the index: 4,056 bytes, read back as W12 W11 W07 W08 W04 W03 W02 W01 W05 W06 W10 W09 without it: 3,536 bytes, read back as W01 W02 W03 W04 W05 W06 W07 W08 W09 W10 W11 W12 first 8 bytes: 66 67 62 03 66 67 62 01 b'fgb\x03fgb\x01' features in the park's box: W02 GDAL 3.12.4 (pyogrio 0.13.0)
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 order.py
The first bytes, fgb 03 fgb 01, give the specification’s major version, 3, and its patch level. With the index the
wards came back in a new order, which the figure follows:
With its index, FlatGeobuf stored the wards along a Hilbert curve
Text description of the diagram
The twelve wards as a grid of four columns and three rows, W01 to W04 across the top, W05 to W08 in the middle and W09 to W12 along the bottom. A circled number in each ward gives its place in the file, and arrows join them in that order.
The path starts in the bottom-right corner at W12 (1), goes left to W11 (2), up to W07 (3), right to W08 (4), up to W04 (5), then left along the top row through W03 (6), W02 (7) and W01 (8), down to W05 (9), right to W06 (10), down to W10 (11) and left to W09 (12).
Every step of the path goes to a ward next door, so wards that are neighbours on the map are stored close together in the file.
The index sorts the features along a Hilbert curve: A line that visits every cell of a grid while keeping cells that are close on the line close in space. Sorting features along it puts neighbours next to each other in a file., a line that visits every ward and keeps neighbours together, so the features of any small area sit next to each other in the file. A reader that asked for the park’s box got its ward, W02, by searching the index first, and over HTTP it would fetch only that part of the file. The cost is 520 more bytes for twelve wards, a file that cannot be edited in place, and an order you must not rely on: sort by an attribute after reading if order matters.
Checkpoint
Check: FlatGeobuf and its index
Pick the format for the person who receives it
The formats differ in size as well. The same 100,000 points of the previous lesson, written six ways:
"""100,000 made-up buildings spread over the track's town, 8 km by 6 km in UTM zone 43N, from a fixed seed."""
import geopandas as gpd
import numpy as np
def points(n=100_000, seed=42):
rng = np.random.default_rng(seed)
x = rng.uniform(500_000, 508_000, n)
y = rng.uniform(2_000_000, 2_006_000, n)
kinds = rng.choice(["home", "shop", "school"], n)
floors = rng.integers(1, 5, n)
return gpd.GeoDataFrame({"id": np.arange(n), "kind": kinds, "floors": floors}, geometry=gpd.points_from_xy(x, y), crs="EPSG:32643")"""The same 100,000 points written in six formats by GDAL (through GeoPandas and pyogrio) and by pyarrow for
GeoParquet, each into a folder of its own: the bytes on disk, and what a CSV gives back."""
import os
import pyogrio
from points import points
g = points()
outputs = [
("Shapefile (5 files)", "shp", "points.shp", {"driver": "ESRI Shapefile"}),
("GeoPackage", "gpkg", "points.gpkg", {"driver": "GPKG"}),
("FlatGeobuf", "fgb", "points.fgb", {"driver": "FlatGeobuf"}),
("GeoJSON, RFC 7946", "geojson", "points.geojson", {"driver": "GeoJSON", "RFC7946": "YES"}),
("CSV with WKT", "csv", "points.csv", {"driver": "CSV", "GEOMETRY": "AS_WKT"}),
]
for label, folder, name, options in outputs:
os.makedirs(folder, exist_ok=True)
g.to_file(os.path.join(folder, name), **options)
size = sum(os.path.getsize(os.path.join(folder, f)) for f in os.listdir(folder))
print(f"{label:22} {size / 1e6:5.1f} MB")
os.makedirs("parquet", exist_ok=True)
g.to_parquet("parquet/points.parquet")
print(f"{'GeoParquet':22} {os.path.getsize('parquet/points.parquet') / 1e6:5.1f} MB")
back = pyogrio.read_dataframe("csv/points.csv")
print("CSV read back: columns", ", ".join(back.columns), "| CRS", back.crs, "| floors as", back["floors"].dtype)
print(f"GDAL {pyogrio.__gdal_version_string__} (pyogrio {pyogrio.__version__})")Output
Shapefile (5 files) 15.3 MB GeoPackage 10.6 MB FlatGeobuf 14.7 MB GeoJSON, RFC 7946 15.9 MB CSV with WKT 5.7 MB GeoParquet 2.4 MB CSV read back: columns WKT, id, kind, floors, geometry | CRS None | floors as str GDAL 3.12.4 (pyogrio 0.13.0)
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 six_formats.py
| Format | Size | What a reader gets back |
|---|---|---|
| GeoParquet | 2.4 MB | everything (next lesson) |
| CSV with WKT | 5.7 MB | text only, and no CRS |
| GeoPackage | 10.6 MB | everything |
| FlatGeobuf | 14.7 MB | everything, indexed |
| Shapefile | 15.3 MB | short names, fixed-width rows |
| GeoJSON | 15.9 MB | everything, in WGS 84 |
The CSV is small because it is plain text with nothing else, and that is also its problem: the program read floors
back as text and found no CRS at all. Two more formats serve narrow jobs. KML, an OGC standard, “encodes what to show
in an earth browser, and how to show it” (OGC KML). GPX holds waypoints, routes
and tracks in WGS 84 degrees for GPS receivers (GPX 1.1). Which one fits depends
on who opens the file. Answer three questions about yours:
Answer the questions above: the table then marks what fits your case and says why the rest does not.
| Format | Many layers in one file | Names, dates and text kept | Part of it over the web | Opens almost anywhere | Small on disk |
|---|---|---|---|---|---|
| Shapefile Holds: one layer in 3 to 5 files | Poor one layer and one geometry type per set of files | Poor names cut to 10 letters, dates to text, Marathi notes to 92 of 147 characters | Poor five files to fetch whole | Good read by almost every GIS program | Mixed 15.3 MB for 100,000 points: fixed-width rows |
| GeoPackage Holds: many layers in one SQLite file | Good the town's four layers in one 643 KB file | Good full names, DATETIME and every character kept | Mixed one database file, fetched whole | Good an OGC standard that GDAL, and every program built on it, reads | Good 10.6 MB for 100,000 points |
| FlatGeobuf Holds: one layer, indexed, binary | Poor one layer per file | Good names, dates and text came back whole | Good its index lets a reader fetch one box of features | Mixed newer: GDAL reads it from version 3.1 | Mixed 14.7 MB for 100,000 points, index included |
| GeoJSON Holds: one collection as JSON text | Poor one collection per file | Mixed names and text kept; dates travel as ISO text | Mixed read by every web map, but whole | Good any web map, any programming language | Poor 15.9 MB for 100,000 points |
| KML Holds: placemarks for earth browsers | Mixed folders group placemarks for display | Mixed attributes become text in ExtendedData | Mixed XML text, fetched whole | Mixed made for earth browsers, an OGC standard | Poor verbose XML |
| GPX Holds: waypoints, routes and tracks | Mixed waypoints, routes and tracks, nothing else | Poor a fixed set of fields per point | Poor XML text, fetched whole | Mixed GPS units and fitness apps read it | Poor verbose XML |
| CSV with WKT Holds: one table of text | Poor one table per file | Poor every value came back as text, and the CRS was lost | Poor text, fetched whole | Good any spreadsheet opens it | Good 5.7 MB for 100,000 points |
How the answers decide
- “Is it one layer, or several that belong together?” Several layers: rules out Shapefile, FlatGeobuf, GeoJSON, CSV with WKT and GPX and adds a caveat to KML. Only a GeoPackage keeps several layers, each with its own CRS, in one file; KML's folders group placemarks for display, not for analysis.
- “Does it have long field names, dates or text in Indian scripts?” Yes: rules out Shapefile and CSV with WKT and adds a caveat to KML and GPX. The shapefile cut the names to 10 letters, the times to text and the Marathi notes to 92 of 147 characters; CSV turned every value into text.
- “What will the person who receives the data open it with?” A web map or a web app: rules out Shapefile, GPX and CSV with WKT and adds a caveat to GeoPackage. A browser reads GeoJSON directly and FlatGeobuf one box at a time; a GeoPackage must be fetched whole before it can be read.
- “What will the person who receives the data open it with?” A GPS unit or an earth browser: rules out Shapefile, FlatGeobuf and CSV with WKT and adds a caveat to GeoPackage and GeoJSON. GPX is made for GPS data and KML for earth browsers; check which formats the device itself reads.
- “What will the person who receives the data open it with?” A spreadsheet: rules out Shapefile, GeoPackage, FlatGeobuf, KML and GPX and adds a caveat to CSV with WKT. A spreadsheet opens CSV, but the CRS and the types must travel in a note beside it, because the file keeps neither.
- “What will the person who receives the data open it with?” A GIS program, such as QGIS: adds a caveat to CSV with WKT and GPX. Every format here opens in QGIS; GeoPackage keeps the names, the types and the CRS, which the others may not.
In short: GeoPackage for a GIS user or a project of several layers, GeoJSON or FlatGeobuf for the web, GPX or KML for devices and globes, CSV only for a spreadsheet with a note of the CRS, and a shapefile only when the receiver cannot open anything newer.
Checkpoint
Check: Choosing a hand-off format
A conversion is only as good as its written-down steps
Most conversion damage happens silently, so a safe conversion states every choice and then checks it. A partner’s
shapefile of the wards arrives with cut names, survey times as text and no .prj. The repair restores the names from
the partner’s field list, parses the times with an explicit format, sets the CRS that the partner’s documentation
states, writes a GeoPackage, and reads the result back with sqlite3:
"""A partner's shapefile of the wards arrives with its names cut to 10 characters, its survey times as text and no .prj.
The repair restores all three on the way into a GeoPackage, and sqlite3 checks the result."""
import os
import sqlite3
import warnings
import geopandas as gpd
import pandas as pd
from shapely.geometry import Polygon
# The partner's file, made here as it arrived.
E0, N0, WARD = 500_000, 2_000_000, 2_000
rows = [{"ward": f"W{r * 4 + c + 1:02d}", "population_estimate": 9_000 + 250 * (r * 4 + c),
"surveyed_at": pd.Timestamp("2026-03-14 10:30") + pd.Timedelta(minutes=35 * (r * 4 + c)),
"geometry": Polygon([(E0 + c * WARD, N0 + (2 - r) * WARD), (E0 + (c + 1) * WARD, N0 + (2 - r) * WARD),
(E0 + (c + 1) * WARD, N0 + (3 - r) * WARD), (E0 + c * WARD, N0 + (3 - r) * WARD)])}
for r in range(3) for c in range(4)]
with warnings.catch_warnings():
warnings.simplefilter("ignore") # the cut names and the text times are the partner's problem, shown below
gpd.GeoDataFrame(rows, crs="EPSG:32643").to_file("partner.shp")
os.remove("partner.prj")
# The repair: every step written down.
names = {"population": "population_estimate", "surveyed_a": "surveyed_at"} # from the partner's field list
wards = gpd.read_file("partner.shp")
print("as received:", ", ".join(wards.columns), "| CRS", wards.crs)
wards = wards.rename(columns=names)
wards["surveyed_at"] = pd.to_datetime(wards["surveyed_at"], format="%Y/%m/%d %H:%M:%S")
wards = wards.set_crs("EPSG:32643") # stated in the partner's documentation
wards.to_file("wards.gpkg", layer="wards", driver="GPKG")
db = sqlite3.connect("wards.gpkg")
columns = [(name, kind) for _, name, kind, *_ in db.execute("PRAGMA table_info(wards)")]
print("GeoPackage columns:", ", ".join(f"{n} {k}" for n, k in columns))
srs = db.execute("SELECT srs_id FROM gpkg_geometry_columns WHERE table_name = 'wards'").fetchone()[0]
count, first = db.execute("SELECT count(*), min(surveyed_at) FROM wards").fetchone()
print(f"srs_id {srs}, {count} wards, first survey {first}")Output
as received: ward, population, surveyed_a, geometry | CRS None GeoPackage columns: fid INTEGER, geom POLYGON, ward TEXT, population_estimate INTEGER, surveyed_at DATETIME srs_id 32643, 12 wards, first survey 2026-03-14T10:30:00
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 repair.py
The checks at the end are the point. The table’s schema shows the restored names and the INTEGER and DATETIME types,
gpkg_geometry_columns shows srs_id 32643, and the count shows all 12 wards. Without them, a wrong guess at the CRS or
the date format would have passed unnoticed.
A checklist for every conversion
Before: list the fields, types, CRS, encoding and feature count of the source. During: set the target’s encoding, CRS and field types explicitly instead of relying on defaults. After: compare the counts, the names, the types, the CRS and a few values with the source, and keep the script with the data.
Checkpoint
Check: Converting without losses
Practise: repair a partner’s shapefile
Build the two pieces of a repair tool, with sample tests you can run here: a function that restores cut field names and parses dates, and one that decodes a GeoPackage geometry blob without any GIS library:
Code lab · 2 steps
Repair a partner's shapefile
Restore field names and dates lost in a shapefile, and read a GeoPackage geometry blob byte by byte.
Exercise · Easy · Python
Restore cut field names and dates
Write restore(rows, names, date_fields) in restore.py. rows is a list of dictionaries read from a shapefile's table, with field names cut to 10 characters and dates and times stored as text such as "2026/03/14 10:30:00". names maps cut names to full names, such as {"population": "population_estimate"}, and date_fields lists the full names of the fields that hold dates and times.
Return new dictionaries: every key renamed through names (keys that are not in it stay as they are), and every value of a date field turned into a datetime.datetime with the format %Y/%m/%d %H:%M:%S. An empty value ("" or None) becomes None. Do not change the rows you were given.
Starter code · restore.py
from datetime import datetime
def restore(rows, names, date_fields):
"""New rows with full field names (through names) and date fields parsed from "%Y/%m/%d %H:%M:%S" text."""
# Replace this line: rename the keys, then parse the date fields; leave the given rows unchanged.
return rowsThe sample tests · test_restore.py
from datetime import datetime
from restore import restore
NAMES = {"population": "population_estimate", "surveyed_a": "surveyed_at"}
ROWS = [
{"ward": "W01", "population": 9000, "surveyed_a": "2026/03/14 10:30:00"},
{"ward": "W02", "population": 9250, "surveyed_a": "2026/03/14 11:05:00"},
]
def test_names_are_restored():
"""renames cut fields and keeps the others"""
out = restore(ROWS, NAMES, ["surveyed_at"])
assert list(out[0]) == ["ward", "population_estimate", "surveyed_at"]
assert out[1]["population_estimate"] == 9250
def test_dates_are_parsed():
"""turns the survey text into a datetime"""
out = restore(ROWS, NAMES, ["surveyed_at"])
assert out[0]["surveyed_at"] == datetime(2026, 3, 14, 10, 30)
assert out[1]["surveyed_at"] == datetime(2026, 3, 14, 11, 5)
def test_empty_dates_and_originals():
"""gives None for an empty date and leaves the input rows as they were"""
rows = [{"ward": "W03", "surveyed_a": ""}, {"ward": "W04", "surveyed_a": None}]
out = restore(rows, NAMES, ["surveyed_at"])
assert out[0]["surveyed_at"] is None and out[1]["surveyed_at"] is None
assert rows[0] == {"ward": "W03", "surveyed_a": ""}
assert ROWS[0]["surveyed_a"] == "2026/03/14 10:30:00"A hint
Build each new row with a dictionary comprehension, {names.get(k, k): v for k, v in row.items()}, then loop over date_fields and replace each text value with datetime.strptime(value, "%Y/%m/%d %H:%M:%S").
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Exercise · Medium · Python
Decode a GeoPackage point blob
Write decode_point(blob) in blob.py. blob is the bytes of a GeoPackage geometry holding one point. Return (srs_id, x, y), or (srs_id, None, None) when the empty flag is set.
The header is: the two bytes GP, a version byte, a flags byte, and the SRS id as a 4-byte integer, little-endian when bit 0 of the flags is 1 and big-endian when it is 0. Bit 4 is the empty flag. Bits 1 to 3 hold the envelope code: 0 means no envelope, 1 means 32 bytes, 2 and 3 mean 48 bytes, 4 means 64 bytes. After the envelope comes the WKB of the point, which has its own byte-order byte. Raise ValueError when the blob does not start with GP.
The sample tests use clinic C1 from the lesson's hand-made GeoPackage, a blob with an envelope, a big-endian one and an empty point.
Starter code · blob.py
import struct
def decode_point(blob):
"""(srs_id, x, y) of a GeoPackage point blob; (srs_id, None, None) when it is empty."""
# Replace this line: check the magic, read the flags and the SRS id, skip the envelope, then read the WKB point.
return (0, None, None)The sample tests · test_blob.py
import struct
from blob import decode_point
C1 = bytes.fromhex("47 50 00 01 E6 10 00 00 01 01 00 00 00") + struct.pack("<dd", 75.014180, 18.135709)
def test_clinic_c1():
"""reads clinic C1 from the lesson's hand-made GeoPackage"""
assert decode_point(C1) == (4326, 75.014180, 18.135709)
def test_with_an_envelope():
"""skips a 32-byte envelope (code 1, flags 0b011)"""
envelope = struct.pack("<4d", 504000, 504000, 2005000, 2005000)
blob = b"GP" + bytes([0, 0b011]) + struct.pack("<i", 32643) + envelope + struct.pack("<BIdd", 1, 1, 504000, 2005000)
assert decode_point(blob) == (32643, 504000.0, 2005000.0)
def test_big_endian():
"""reads a big-endian header and a big-endian WKB"""
blob = b"GP" + bytes([0, 0]) + struct.pack(">i", 4326) + struct.pack(">BIdd", 0, 1, 75.037813, 18.133898)
assert decode_point(blob) == (4326, 75.037813, 18.133898)
def test_empty_and_not_a_geopackage():
"""says so for an empty point and refuses bytes that are not a GeoPackage geometry"""
empty = b"GP" + bytes([0, 0b10001]) + struct.pack("<i", 4326) + struct.pack("<BIdd", 1, 1, float("nan"), float("nan"))
assert decode_point(empty) == (4326, None, None)
try:
decode_point(b"XX" + empty[2:])
except ValueError:
pass
else:
raise AssertionError("bytes that do not start with GP must raise ValueError")A hint
Read the flags with blob[3]; the byte order of the header is "<" if flags & 1 else ">", the envelope code is (flags >> 1) & 0b111, and the WKB starts at byte 8 plus the envelope's length. Read the WKB's own first byte for its byte order, then unpack "Idd" from the byte after it.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Then try the self-check, which starts with whichever of the six ideas, from a shapefile’s files to a safe conversion, your answers show you know least:
Practice that adapts to you
Practise this lesson
14 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up (fresher): Which files make up a shapefile, and what happens if one is missing?
Three are required: the .shp holds the geometry, the .shx the position of each record in it, and the .dbf the
attributes, one row per shape in the same order. Two more matter as much in practice: the .prj holds the coordinate
reference system as WKT, and the .cpg the text encoding of the .dbf. Without the .shx or the .dbf most programs
refuse the layer, although GDAL can rebuild a lost .shx; without the .prj the coordinates are numbers that belong to
no place; without the .cpg the reader guesses the encoding, and Indian-language text can turn into question marks.
That is why a shapefile travels as a zip of all its files.
Question marks where Marathi names should be, and one project handed to three very different receivers, are the next two problems. The module’s bank opens each with what it tests and the first step of an answer:
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Fresher to mid
A partner sends ward boundaries as a shapefile. In one copy the Marathi ward names show as rows of question marks; in another they are readable but the longer descriptions stop in the middle of a word. Explain both, and how you would get the data across intact.
-
Mid to senior
A town's GIS project has four layers: wards, households, clinics and roads. You must hand it to a consultant who uses QGIS, to a web team building a public map, and to field staff who carry GPS units. Which formats do you send to each, and why?
Flashcards · 8 cards
Recall it, then check
Say the answer to yourself before you look: recalling it is what makes it stick.
-
Which files make up a shapefile, and which two optional ones matter as much as the rest?
The .shp holds the shapes, the .shx where each record starts, and the .dbf one row of attributes per shape. The .prj holds the CRS and the .cpg the text encoding, so send all five together, for example in one zip.
-
What does a shapefile change in your data as it is written?
Field names are cut to 10 characters, date-and-time values become text, a layer can hold only one geometry type, and a text value ends at 254 bytes. GDAL warns about each change and writes the file anyway.
-
Why did a 147-character Marathi note keep only 92 characters in a shapefile?
A .dbf text field stops at 254 bytes, while UTF-8 spends 3 bytes on each Devanagari letter, so the note's 399 bytes were cut in the middle of a word. Text in Indian scripts gets about a third as many characters as English text in the same field.
-
What is inside a GeoPackage?
One SQLite database to an OGC standard: gpkg_spatial_ref_sys lists its CRSs, gpkg_contents its layers and gpkg_geometry_columns each layer's geometry column, and every geometry is a short GP header followed by WKB. Any SQLite program can read it, and rows can be edited in place.
-
What does FlatGeobuf's spatial index do to a layer?
It stores the features in Hilbert-curve order, so neighbours sit together and a reader can fetch one area's features, over HTTP too, with range requests. The price is a changed feature order, a slightly larger file and no editing in place.
-
Which format suits a GIS user, a web map, a GPS device and a spreadsheet user?
A GeoPackage for GIS work or several layers in one file; GeoJSON or FlatGeobuf for the web; GPX or KML for devices and globes; CSV only with a note of its CRS. Send a shapefile only when the receiver cannot open anything newer.
-
What does a CSV with WKT geometry lose?
Everything but the text: a numeric column such as floors comes back as text, and nothing records which coordinate system the numbers belong to.
-
What makes a format conversion safe?
Before it, list the source's fields, types, CRS, encoding and feature count. During it, set the target's encoding, CRS and field types instead of trusting defaults. After it, compare counts, names, types, CRS and a few values, and keep the script with the data.
Key takeaways
- A shapefile is a .shp, a .shx and a .dbf with one name, plus a .prj for the CRS and a .cpg for the encoding, and all of them must travel together.
- The shapefile cuts names to 10 characters, turns date and time into text, holds one geometry type per file and ends text at 254 bytes, which held 92 of 147 Marathi characters; GDAL warns, and writes the file anyway.
- A GeoPackage is one SQLite file with required tables for its CRSs, layers and geometry columns, and GP-headed WKB geometries; it held the whole town, four layers, in 643 KB, and any SQLite program can read it.
- FlatGeobuf is a binary, indexed, write-once format for one layer; its Hilbert-sorted index lets a reader fetch one box over HTTP, and it changes the order of the features.
- Choose the format for the receiver: GeoPackage for GIS work and many layers, GeoJSON or FlatGeobuf for the web, GPX or KML for devices, and CSV only with a note of the CRS.
- A safe conversion writes down every choice of names, types, encoding and CRS, and checks counts, schema and CRS after writing.
Terms in this lesson
- FlatGeobuf
- A binary format for one vector layer, built on FlatBuffers, with an optional packed Hilbert R-tree index that lets a reader fetch only the features of a bounding box, even over HTTP.
- GeoPackage
- An OGC standard for geospatial data in one SQLite database file (.gpkg), with required tables for its coordinate systems, layers and geometry columns, and any number of layers.
- Hilbert curve
- A line that visits every cell of a grid while keeping cells that are close on the line close in space. Sorting features along it puts neighbours next to each other in a file.
- shapefile
- A vector format of several files with one name: .shp for the geometry, .shx for its index and .dbf for the attributes, with .prj for the CRS and .cpg for the text encoding beside them.
- spatial index
- A structure, such as an R-tree, that finds the features in or near an area without testing every feature, by grouping them into nested bounding boxes.
References
- ESRI Shapefile Technical Description (Esri)
- GDAL: ESRI Shapefile / DBF driver (Open Source Geospatial Foundation (GDAL))
- OGC GeoPackage Encoding Standard (Open Geospatial Consortium (OGC))
- GDAL: GeoPackage vector driver (Open Source Geospatial Foundation (GDAL))
- FlatGeobuf (FlatGeobuf contributors)
- GDAL: FlatGeobuf driver (Open Source Geospatial Foundation (GDAL))
- OGC KML (Open Geospatial Consortium (OGC))
- GPX 1.1 Schema Documentation (TopoGrafix)
- pyogrio: reading and writing vector data with GDAL (pyogrio contributors)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress