GIS & Geospatial Data (QGIS, GeoPandas, PostGIS) Module 3 – Vector geometry and file formats
GDAL and OGR on the command line
Inspect, convert, filter and reproject vector and raster data with ogrinfo, ogr2ogr, gdalwarp and gdal_translate, in scripts that repeat exactly.
What you will learn
- Inspect any vector or raster file with ogrinfo, gdalinfo or the gdal command, and read what they report
- Convert, filter and reproject vector data with ogr2ogr, and rasters with gdal_translate and gdalwarp
- Write a conversion script with every format, CRS, encoding and creation option stated, and check that it repeats
- Check which GDAL each tool of an environment uses, and explain how different defaults change the output
Before you start
What you’ll learn · Before you start
What you will learn
- Inspect any vector or raster file with ogrinfo, gdalinfo or the gdal command, and read what they report
- Convert, filter and reproject vector data with ogr2ogr, and rasters with gdal_translate and gdalwarp
- Write a conversion script with every format, CRS, encoding and creation option stated, and check that it repeats
- Check which GDAL each tool of an environment uses, and explain how different defaults change the output
Before you start
On this page
Convert a ward layer to a shapefile with ogr2ogr and its default settings, and every Marathi ward name in it becomes a
row of question marks, with one warning line to say so. Run the same conversion from Python through pyogrio and the
names survive: pyogrio brings a GDAL of its own and asks it for UTF-8, which ogr2ogr does only when told to.
GDAL’s command-line programs convert spatial data for people at a terminal and for scripts alike. Here you build
the track’s town from plain CSV files, ask ogrinfo, gdalinfo and the newer gdal command what each file holds,
convert, filter and reproject with ogr2ogr, turn a grid of heights into a cloud-optimised GeoTIFF with
gdal_translate and gdalwarp, write a conversion script that gives the same files every time, find which GDAL each
tool runs, and read a zipped shapefile without unzipping it. The GDAL programs in every session here are version 3.13.3.
ogrinfo and gdalinfo tell you what is in a file
GDAL: The Geospatial Data Abstraction Library: an open-source library and set of command-line programs that read and write hundreds of raster and vector formats through one data model. Its vector half was long called OGR. reads and writes hundreds of formats through one model: each format is a
GDAL driver: The part of GDAL that reads or writes one format, such as GPKG, GeoJSON or GTiff. ogrinfo --formats and gdalinfo --formats list the drivers a copy of GDAL has., and each program works with every driver of its kind, vector or raster.
ogrinfo reports on vector data and gdalinfo on rasters. GDAL 3.11 added a single gdal program with subcommands such as gdal vector info and
gdal raster info, which its documentation calls “provisionally provided” until it is frozen in a future release
(GDAL, Programs). The session builds the town and asks each of them:
"""Writes the track's made-up town as plain files: four CSV layers in metres of UTM zone 43N and a small elevation
grid, so that GDAL can build everything else from them. Python's standard library only."""
import csv
import math
import random
E0, N0, WARD = 500_000, 2_000_000, 2_000
NAMES = ["प्रभाग १", "प्रभाग २", "प्रभाग ३", "प्रभाग ४", "प्रभाग ५", "प्रभाग ६", "प्रभाग ७", "प्रभाग ८", "प्रभाग ९", "प्रभाग १०", "प्रभाग ११", "प्रभाग १२"]
CLINICS = {"C1": (1_500, 5_200), "C2": (4_000, 5_000), "C3": (6_500, 4_200), "C4": (2_700, 3_100),
"C5": (5_200, 2_800), "C6": (900, 1_100), "C7": (5_600, 1_200), "C8": (7_400, 900)}
with open("wards.csv", "w", newline="", encoding="utf-8") as f:
w = csv.writer(f)
w.writerow(["ward", "name_mr", "wkt"])
for r in range(3):
for c in range(4):
x0, y0 = E0 + c * WARD, N0 + (2 - r) * WARD
ring = [(x0, y0), (x0 + WARD, y0), (x0 + WARD, y0 + WARD), (x0, y0 + WARD), (x0, y0)]
w.writerow([f"W{r * 4 + c + 1:02d}", NAMES[r * 4 + c], "POLYGON ((" + ", ".join(f"{x} {y}" for x, y in ring) + "))"])
rng = random.Random(42)
centres = [(2_600, 4_300, 700), (5_600, 1_500, 900), (6_900, 4_700, 500)]
with open("households.csv", "w", newline="") as f:
w = csv.writer(f)
w.writerow(["household", "x", "y"])
n = 0
while n < 5_000:
if rng.random() < 0.7:
cx, cy, spread = rng.choice(centres)
x = cx + spread * 2 * (rng.random() + rng.random() + rng.random() - 1.5)
y = cy + spread * 2 * (rng.random() + rng.random() + rng.random() - 1.5)
else:
x, y = rng.uniform(0, 4 * WARD), rng.uniform(0, 3 * WARD)
if 0 < x < 4 * WARD and 0 < y < 3 * WARD:
n += 1
w.writerow([f"H{n:04d}", round(E0 + x, 1), round(N0 + y, 1)])
with open("clinics.csv", "w", newline="") as f:
w = csv.writer(f)
w.writerow(["clinic", "x", "y"])
for k, (x, y) in CLINICS.items():
w.writerow([k, E0 + x, N0 + y])
# A hill in the middle of the town, 200 m cells, as an ESRI ASCII grid (north row first).
with open("dem.asc", "w") as f:
f.write(f"ncols 40\nnrows 30\nxllcorner {E0}\nyllcorner {N0}\ncellsize 200\nNODATA_value -9999\n")
for row in range(30):
y = 6_000 - 100 - 200 * row
f.write(" ".join(f"{520 + 80 * math.exp(-((100 + 200 * col - 4_000) ** 2 + (y - 3_000) ** 2) / (2 * 1_500 ** 2)):.1f}" for col in range(40)) + "\n")
print("wrote wards.csv, households.csv, clinics.csv and dem.asc") #!/bin/sh
# Builds town.gpkg from the CSV files that make_town.py writes, with every option written out.
set -eu
rm -f town.gpkg
ogr2ogr -f GPKG town.gpkg wards.csv -nln wards -nlt POLYGON -a_srs EPSG:32643 \
-oo GEOM_POSSIBLE_NAMES=wkt -oo KEEP_GEOM_COLUMNS=NO
ogr2ogr -update town.gpkg households.csv -nln households -a_srs EPSG:32643 \
-oo X_POSSIBLE_NAMES=x -oo Y_POSSIBLE_NAMES=y -oo KEEP_GEOM_COLUMNS=NO
ogr2ogr -update town.gpkg clinics.csv -nln clinics -a_srs EPSG:32643 \
-oo X_POSSIBLE_NAMES=x -oo Y_POSSIBLE_NAMES=y -oo KEEP_GEOM_COLUMNS=NO
echo "built town.gpkg" from session import run
# Build the town, then ask GDAL what is in its files: the layers, one layer's summary, and a raster.
run("""
python3 make_town.py
sh build_town.sh
ogrinfo -q town.gpkg
ogrinfo -so -nomd town.gpkg households | head -n 7
gdal vector info --layer wards town.gpkg
gdal_translate -q -a_srs EPSG:32643 dem.asc dem.tif
gdal raster info --stats dem.tif
""") Output
$ python3 make_town.py
wrote wards.csv, households.csv, clinics.csv and dem.asc
$ sh build_town.sh
built town.gpkg
$ ogrinfo -q town.gpkg
1: wards (Polygon)
2: households (Point)
3: clinics (Point)
$ ogrinfo -so -nomd town.gpkg households | head -n 7
INFO: Open of `town.gpkg'
using driver `GPKG' successful.
Layer name: households
Geometry: Point
Feature Count: 5000
Extent: (500001.600000, 2000006.000000) - (507992.300000, 2005998.600000)
$ gdal vector info --layer wards town.gpkg
INFO: Open of `town.gpkg'
using driver `GPKG' successful.
Layer name: wards
Geometry: Polygon
Feature Count: 12
Extent: (500000.000000, 2000000.000000) - (508000.000000, 2006000.000000)
Layer Coordinate Reference System:
- name: WGS 84 / UTM zone 43N
- ID: EPSG:32643
- type: Projected
- projection type: UTM zone 43N, Transverse Mercator
- units: metre
- area of use: Between 72°E and 78°E..., west 72.00, south 0.00, east 78.00, north 84.00
Data axis to CRS axis mapping: 1,2
FID Column = fid
Geometry Column = geom
ward: String (0.0)
name_mr: String (0.0)
$ gdal_translate -q -a_srs EPSG:32643 dem.asc dem.tif
$ gdal raster info --stats dem.tif
Driver: GTiff/GeoTIFF
Files: dem.tif
Size is 40, 30
Coordinate Reference System:
- name: WGS 84 / UTM zone 43N
- ID: EPSG:32643
- type: Projected
- projection type: UTM zone 43N, Transverse Mercator
- units: metre
- area of use: Between 72°E and 78°E..., west 72.00, south 0.00, east 78.00, north 84.00
Data axis to CRS axis mapping: 1,2
Origin = (500000.000000000000000,2006000.000000000000000)
Pixel Size = (200.000000000000000,-200.000000000000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( 500000.000, 2006000.000) ( 75d 0' 0.00"E, 18d 8'34.58"N)
Lower Left ( 500000.000, 2000000.000) ( 75d 0' 0.00"E, 18d 5'19.35"N)
Upper Right ( 508000.000, 2006000.000) ( 75d 4'32.27"E, 18d 8'34.53"N)
Lower Right ( 508000.000, 2000000.000) ( 75d 4'32.19"E, 18d 5'19.30"N)
Center ( 504000.000, 2003000.000) ( 75d 2'16.11"E, 18d 6'56.96"N)
Band 1 Block=40x30 Type=Float32, ColorInterp=Gray
Minimum=520.400, Maximum=599.600, Mean=542.322, StdDev=20.968
NoData Value=-9999
Metadata:
STATISTICS_MINIMUM=520.40002441406
STATISTICS_MAXIMUM=599.59997558594
STATISTICS_MEAN=542.32166890462
STATISTICS_STDDEV=20.968492930652
STATISTICS_VALID_PERCENT=100
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 inspect_session.py
ogrinfo -q lists the layers, and -so gives one layer’s summary: its geometry type, 5,000 households and the extent in
UTM metres. gdal vector info gives the same facts with a compact account of the coordinate system: name, EPSG code,
type and units, where ogrinfo prints the whole WKT. The raster report is the one to read slowly: 40 by 30 cells, the
origin at the north-west corner, a pixel size of 200 and −200 m (negative because rows run south), the corners in
degrees, and the band’s statistics, heights from 520.4 to 599.6 m with −9999 marking no data.
Read before you convert
Two commands answer most questions about a file you were sent: ogrinfo -so -nomd file layer for its type, count,
extent and CRS, and gdalinfo -stats file for a raster’s size, cell size, CRS and range of values. Compare them with
what you expected before any conversion.
Checkpoint
Check: Inspecting files with ogrinfo, gdalinfo and gdal
ogr2ogr converts, filters and reprojects in one command
ogr2ogr “converts simple features data between file formats”
(GDAL, ogr2ogr), and its options choose what is read, how it is
changed and how it is written. Step through the stages of one command and the options that control each:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- Open. ogr2ogr opens the source with its driver. Open options tell the driver how to read it: -oo X_POSSIBLE_NAMES=x told the CSV driver which columns hold the households' positions.
- Which features. -where keeps rows by their attributes, -spat by a box and -sql by a query. -spat 502000 2004000 504000 2006000 kept the 696 households in W02's box.
- Which fields. -select keeps only the fields it names. Without it, every field goes across, which is what the session wanted.
- Which CRS. -a_srs states a CRS the source lacks, -s_srs overrides the one it has, and -t_srs reprojects. -t_srs EPSG:4326 turned clinic C1 into 75.0141801936012, 18.1357088024991.
- How to write it. -f names the output driver, -dsco passes options to the dataset and -lco to the layer: -lco RFC7946=YES and -lco COORDINATE_PRECISION=6 gave standard GeoJSON at 6 decimals.
- What to call it. -nln gives the layer its name, households_w02 here, and -nlt its geometry type, POLYGON for the wards read from WKT, which GDAL cannot know before it reads them.
The session runs four typical jobs: standard GeoJSON at 6 decimals, the households in W02’s box, the clinics reprojected to a CSV of longitudes and latitudes, and an SQL query that measures a ward:
from session import run
# One ogr2ogr command per job: to standard GeoJSON, a spatial filter, a reprojection to a CSV, and SQL.
run("""
python3 make_town.py
sh build_town.sh
ogr2ogr -f GeoJSON -lco RFC7946=YES -lco COORDINATE_PRECISION=6 wards.geojson town.gpkg wards
head -c 200 wards.geojson
ogr2ogr -f GPKG w02.gpkg town.gpkg households -spat 502000 2004000 504000 2006000 -nln households_w02
ogrinfo -so -nomd w02.gpkg households_w02 | grep "Feature Count"
ogr2ogr -f CSV clinics_4326.csv town.gpkg clinics -t_srs EPSG:4326 -lco GEOMETRY=AS_XY
head -n 3 clinics_4326.csv
ogrinfo -q town.gpkg -dialect SQLite -sql "SELECT ward, ST_Area(geom) / 10000 AS ha FROM wards WHERE ward = 'W02'"
""") Output
$ python3 make_town.py
wrote wards.csv, households.csv, clinics.csv and dem.asc
$ sh build_town.sh
built town.gpkg
$ ogr2ogr -f GeoJSON -lco RFC7946=YES -lco COORDINATE_PRECISION=6 wards.geojson town.gpkg wards
$ head -c 200 wards.geojson
{
"type": "FeatureCollection",
"name": "wards",
"xy_coordinate_resolution": 1e-06,
"features": [
{"type":"Feature","properties":{"ward":"W01","name_mr":"प्रभाग १"},"geometry":{"type":"Po
$ ogr2ogr -f GPKG w02.gpkg town.gpkg households -spat 502000 2004000 504000 2006000 -nln households_w02
$ ogrinfo -so -nomd w02.gpkg households_w02 | grep "Feature Count"
Feature Count: 696
$ ogr2ogr -f CSV clinics_4326.csv town.gpkg clinics -t_srs EPSG:4326 -lco GEOMETRY=AS_XY
$ head -n 3 clinics_4326.csv
X,Y,clinic
75.0141801936012,18.1357088024991,C1
75.0378134586204,18.1338979088021,C2
$ ogrinfo -q town.gpkg -dialect SQLite -sql "SELECT ward, ST_Area(geom) / 10000 AS ha FROM wards WHERE ward = 'W02'"
Layer name: SELECT
OGRFeature(SELECT):0
ward (String) = W02
ha (Real) = 400
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 convert_session.py
The GeoJSON starts with the xy_coordinate_resolution member that COORDINATE_PRECISION=6 adds, and keeps the
Marathi names. -spat kept 696 of the 5,000 households, those whose geometry intersects the box. -t_srs EPSG:4326
reprojected the clinics, and GEOMETRY=AS_XY, a CSV layer creation option: A format-specific setting that GDAL passes to a driver when it writes a file or a layer, given with -co, -dsco or -lco, such as RFC7946=YES for GeoJSON or ENCODING=UTF-8 for a shapefile., wrote
them as X and Y columns. The SQL dialect of GDAL’s SQLite driver computed W02’s area as 400 ha. Note the difference between three options that sound
alike: -a_srs assigns a CRS “without reprojecting”, -s_srs overrides the source’s, and -t_srs reprojects.
-a_srs where -t_srs was meant
ogr2ogr -a_srs EPSG:4326 out.gpkg town.gpkg wards labels the UTM metres as degrees without changing a number, so
ward W01 claims longitude 500000. Use -a_srs only to state a CRS the source lacks, such as the CSV files of the
town, and -t_srs to convert.
Checkpoint
Check: Converting, filtering and reprojecting with ogr2ogr
The raster programs convert and reproject grids
For rasters, gdal_translate “converts raster data between different formats”, and gdalwarp is “an image mosaicing,
reprojection and warping utility” (GDAL, gdalwarp). The session
turns the ASCII grid of heights into a GeoTIFF, stating its CRS with -a_srs, reprojects it to WGS 84 with bilinear
resampling, and writes a compressed cloud-optimised copy:
from session import run
# A grid of heights into a GeoTIFF, reprojected to WGS 84, and rewritten as a compressed cloud-optimised GeoTIFF.
run("""
python3 make_town.py
gdal_translate -a_srs EPSG:32643 dem.asc dem.tif
gdalwarp -t_srs EPSG:4326 -r bilinear dem.tif dem_4326.tif
gdal raster info dem_4326.tif | grep -E "Size is|Pixel Size|ID:"
gdal_translate -q -of COG -co COMPRESS=DEFLATE dem.tif dem_cog.tif
wc -c dem.tif dem_cog.tif
gdalinfo dem_cog.tif | grep -E "LAYOUT|COMPRESSION"
""") Output
$ python3 make_town.py
wrote wards.csv, households.csv, clinics.csv and dem.asc
$ gdal_translate -a_srs EPSG:32643 dem.asc dem.tif
Input file size is 40, 30
0...10...20...30...40...50...60...70...80...90...100 - done.
$ gdalwarp -t_srs EPSG:4326 -r bilinear dem.tif dem_4326.tif
Creating output file that is 41P x 29L.
Using internal nodata values (e.g. -9999) for image dem.tif.
Copying nodata values from source dem.tif to destination dem_4326.tif.
Processing dem.tif [1/1] : 0...10...20...30...40...50...60...70...80...90...100 - done.
$ gdal raster info dem_4326.tif | grep -E "Size is|Pixel Size|ID:"
Size is 41, 29
- ID: EPSG:4326
Pixel Size = (0.001861088312088,-0.001861088312088)
$ gdal_translate -q -of COG -co COMPRESS=DEFLATE dem.tif dem_cog.tif
$ wc -c dem.tif dem_cog.tif
5178 dem.tif
3616 dem_cog.tif
8794 total
$ gdalinfo dem_cog.tif | grep -E "LAYOUT|COMPRESSION"
LAYOUT=COG
COMPRESSION=DEFLATE
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 raster_session.py
Reprojection changed the grid itself: 40 by 30 cells of 200 m became 41 by 29 cells of about 0.00186°, because a square
in UTM metres is not a square in degrees. gdalwarp uses nearest-neighbour resampling unless told otherwise, which
suits classes such as land cover (when it writes a COG, the default is cubic); heights are continuous, so the session
asked for -r bilinear. The COG driver writes a GeoTIFF tiled and laid out “to minimize the number of GET requests
needed by a reader doing random read access”
(GDAL, COG), the raster form of the cloud-native idea of the
previous lesson, and DEFLATE compression made this one smaller than the plain GeoTIFF.
Checkpoint
Check: Converting and reprojecting rasters
A script with every option written out gives the same files
Every option you leave out is a choice GDAL makes for you, and a later version, another tool or another computer may choose differently. A conversion script states them all: the format, the layer, the CRS, the encoding and the precision. This one turns the town into four hand-off files, in formats chosen as the lesson on Shapefile, GeoPackage and FlatGeobuf suggests for each receiver, and starts from an empty folder each time:
#!/bin/sh
# Turns town.gpkg into four hand-off files in out/, with every choice written out: format, layer, CRS, encoding and
# precision. Safe to run again: it starts from an empty out/ folder each time.
set -eu
rm -rf out
mkdir out
ogr2ogr -f GeoJSON -t_srs EPSG:4326 -lco RFC7946=YES -lco COORDINATE_PRECISION=6 \
out/wards.geojson town.gpkg wards
ogr2ogr -f FlatGeobuf -lco SPATIAL_INDEX=YES out/households.fgb town.gpkg households
ogr2ogr -f GPX -t_srs EPSG:4326 -sql "SELECT clinic AS name, geom FROM clinics" \
out/clinics.gpx town.gpkg
ogr2ogr -f "ESRI Shapefile" -lco ENCODING=UTF-8 out/wards.shp town.gpkg wards
for f in out/wards.geojson out/households.fgb out/wards.shp; do
echo "$f: $(ogrinfo -ro -so -nomd "$f" -al | grep 'Feature Count')"
done
echo "out/clinics.gpx: $(ogrinfo -ro -so -nomd out/clinics.gpx waypoints | grep 'Feature Count')" The session runs it, keeps the checksums of two outputs, runs it again and checks them:
from session import run
# The conversion script, run twice: the second run must make the same files.
run("""
python3 make_town.py
sh build_town.sh
sh convert.sh
shasum -a 256 out/wards.geojson out/households.fgb > first-run.txt
sh convert.sh
shasum -a 256 -c first-run.txt
""") Output
$ python3 make_town.py wrote wards.csv, households.csv, clinics.csv and dem.asc $ sh build_town.sh built town.gpkg $ sh convert.sh out/wards.geojson: Feature Count: 12 out/households.fgb: Feature Count: 5000 out/wards.shp: Feature Count: 12 out/clinics.gpx: Feature Count: 8 $ shasum -a 256 out/wards.geojson out/households.fgb > first-run.txt $ sh convert.sh out/wards.geojson: Feature Count: 12 out/households.fgb: Feature Count: 5000 out/wards.shp: Feature Count: 12 out/clinics.gpx: Feature Count: 8 $ shasum -a 256 -c first-run.txt out/wards.geojson: OK out/households.fgb: OK
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 script_session.py
Both runs gave 12 wards, 5,000 households and 8 clinics, and the checksums matched, so the script is safe to run again
before every hand-off. set -eu stops it with an error at the first failing command and at any unset variable, so a
failed run is never mistaken for a finished one.
Key idea
Write every choice into the command: -f, -t_srs or -a_srs, -nln, -nlt and the -lco options such as RFC7946, COORDINATE_PRECISION and ENCODING; then run the script twice and compare what it made.
Checkpoint
Check: Conversion scripts that repeat
Every tool on the computer may bring its own GDAL
GDAL is a library, and programs carry it in different ways. The command-line tools here came from conda-forge, while pyogrio, which GeoPandas uses to read and write files, carries a GDAL of its own inside its Python wheel:
The shell's programs and Python's packages can run different GDALs
Text description of the diagram
Two panels side by side.
In the shell, the programs on the PATH, ogr2ogr, ogrinfo, gdalinfo and gdal, all point to one library: libgdal 3.13.3 from conda-forge.
In Python, the packages in the virtual environment, GeoPandas 1.2.0 and pyogrio 0.13.0, point to another: libgdal 3.12.4, which travels inside pyogrio's wheel.
A note under the panels says that the same files go through different code, and that gdal-config --version and pyogrio.__gdal_version_string__ tell which one ran.
The session asks both for their version, then writes the same wards to a shapefile with each:
"""Writes wards.csv as a shapefile through pyogrio, which brings a GDAL of its own inside its Python wheel."""
import pyogrio
wards = pyogrio.read_dataframe("wards.csv", GEOM_POSSIBLE_NAMES="wkt", KEEP_GEOM_COLUMNS="NO").set_crs("EPSG:32643")
pyogrio.write_dataframe(wards, "py_wards.shp")
print(f"pyogrio {pyogrio.__version__} with GDAL {pyogrio.__gdal_version_string__} wrote py_wards.shp") from session import run
# Two GDALs in one environment, and what their different defaults do to Marathi ward names in a shapefile.
run("""
gdal-config --version
python3 -c "import pyogrio; print(pyogrio.__gdal_version_string__)"
python3 make_town.py
ogr2ogr -f "ESRI Shapefile" cli_wards.shp wards.csv -nlt POLYGON -a_srs EPSG:32643 -oo GEOM_POSSIBLE_NAMES=wkt -oo KEEP_GEOM_COLUMNS=NO
python3 pyogrio_wards.py
ls -1 cli_wards.* py_wards.*
ogrinfo -q -al -limit 1 cli_wards.shp | grep name_mr
ogrinfo -q -al -limit 1 py_wards.shp | grep name_mr
""") Output
$ gdal-config --version 3.13.3 $ python3 -c "import pyogrio; print(pyogrio.__gdal_version_string__)" 3.12.4 $ python3 make_town.py wrote wards.csv, households.csv, clinics.csv and dem.asc $ ogr2ogr -f "ESRI Shapefile" cli_wards.shp wards.csv -nlt POLYGON -a_srs EPSG:32643 -oo GEOM_POSSIBLE_NAMES=wkt -oo KEEP_GEOM_COLUMNS=NO Warning 1: One or several characters couldn't be converted correctly from UTF-8 to ISO-8859-1. This warning will not be emitted anymore. $ python3 pyogrio_wards.py pyogrio 0.13.0 with GDAL 3.12.4 wrote py_wards.shp $ ls -1 cli_wards.* py_wards.* cli_wards.dbf cli_wards.prj cli_wards.shp cli_wards.shx py_wards.cpg py_wards.dbf py_wards.prj py_wards.shp py_wards.shx $ ogrinfo -q -al -limit 1 cli_wards.shp | grep name_mr name_mr (String) = ?????? ? $ ogrinfo -q -al -limit 1 py_wards.shp | grep name_mr name_mr (String) = प्रभाग १
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 versions_session.py
The command line runs GDAL 3.13.3 and pyogrio GDAL 3.12.4. The bigger difference is a default: GDAL’s shapefile driver
encodes text as ISO-8859-1 unless told otherwise, its documented default being LDID/87
(GDAL, Shapefile driver), and Latin-1 has no Devanagari
letters, so ogr2ogr wrote question marks and no .cpg file. pyogrio wrote UTF-8 and a .cpg, its default for
shapefiles, and the names survived.
The fix for the command line is one option, -lco ENCODING=UTF-8, which convert.sh states.
Print the versions in every record
Begin every script or notebook that converts data by printing gdal-config --version and, in Python,
pyogrio.__gdal_version_string__. When two machines disagree, the versions and the defaults are the first things to
compare.
Checkpoint
Check: GDAL versions and defaults
Virtual file systems read zips and web files in place
Most GDAL drivers open files through a GDAL virtual file system: A path prefix such as /vsizip/ or /vsicurl/ that lets GDAL read data inside an archive or over HTTP in place, fetching only the parts it needs instead of unpacking or downloading the whole file., an abstraction of
GDAL’s that also reaches inside archives and across networks. A path that starts /vsizip/ reads “ZIP archives on-the-fly without decompressing them beforehand”, in the form
/vsizip/path/to/the/file.zip/path/inside/the/zip/file, and /vsicurl/ gives “on-the-fly random reading of files
available through HTTP/FTP web protocols”, fetching only the byte ranges it needs
(GDAL, Virtual File Systems). Since version 3.1 the
shapefile driver also reads and writes zipped shapefiles directly, as .shz or .shp.zip:
from session import run
# A zipped shapefile written and read without unzipping it.
run("""
python3 make_town.py
sh build_town.sh
ogr2ogr -f "ESRI Shapefile" -lco ENCODING=UTF-8 wards.shp.zip town.gpkg wards
python3 -c "import zipfile; print(zipfile.ZipFile('wards.shp.zip').namelist())"
ogrinfo -q wards.shp.zip
ogrinfo -ro -so -nomd /vsizip/wards.shp.zip/wards.shp wards | grep -E "Geometry|Feature Count"
""") Output
$ python3 make_town.py
wrote wards.csv, households.csv, clinics.csv and dem.asc
$ sh build_town.sh
built town.gpkg
$ ogr2ogr -f "ESRI Shapefile" -lco ENCODING=UTF-8 wards.shp.zip town.gpkg wards
$ python3 -c "import zipfile; print(zipfile.ZipFile('wards.shp.zip').namelist())"
['wards.shp', 'wards.cpg', 'wards.dbf', 'wards.prj', 'wards.shx']
$ ogrinfo -q wards.shp.zip
1: wards (Polygon)
$ ogrinfo -ro -so -nomd /vsizip/wards.shp.zip/wards.shp wards | grep -E "Geometry|Feature Count"
Geometry: Polygon
Feature Count: 12
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 vsi_session.py
ogr2ogr wrote all five files of the shapefile into one zip, so none can be left behind, and ogrinfo read it both
directly and through /vsizip/. None of the sessions here reads over HTTP, but the same commands with /vsicurl/
followed by an https:// address read a FlatGeobuf or a COG from a web server in the same way. GeoParquet needs
GDAL’s Parquet driver, which a build has only when it includes the Parquet component of the Apache Arrow C++ library
(GDAL, Parquet driver), so check ogrinfo --formats first.
Checkpoint
Check: Reading zips and web files in place
Practise: check a conversion from a script
Build two helpers that conversion scripts need, with sample tests you can run here: one that reads the summary ogrinfo
prints, and one that finds the options an ogr2ogr command leaves to GDAL’s defaults:
Code lab · 2 steps
Check a conversion from a script
Read the layer, type, count and extent from ogrinfo's summary, and flag ogr2ogr commands that leave the format, the RFC 7946 options or the encoding to chance.
Exercise · Easy · Python
Read the summary that ogrinfo prints
A script that checks a conversion often reads what ogrinfo -so printed. Write summary(text) in summary.py: it takes that text and returns a dictionary with "layer" (the layer name), "geometry" (the geometry type), "count" (the feature count, as an int) and "extent" (a tuple of four floats, xmin, ymin, xmax, ymax). Lines it does not need, such as the INFO: lines, are ignored.
The sample tests use the summary of the households layer from the lesson's session and a summary of the wards.
Starter code · summary.py
import re
def summary(text):
"""The layer, geometry, count and extent from the text of ogrinfo -so."""
# Replace this line: read the lines that start with Layer name, Geometry, Feature Count and Extent.
return {} The sample tests · test_summary.py
from summary import summary
HOUSEHOLDS = """INFO: Open of `town.gpkg'
using driver `GPKG' successful.
Layer name: households
Geometry: Point
Feature Count: 5000
Extent: (500001.600000, 2000006.000000) - (507992.300000, 2005998.600000)
"""
WARDS = """Layer name: wards
Geometry: Polygon
Feature Count: 12
Extent: (500000.000000, 2000000.000000) - (508000.000000, 2006000.000000)
FID Column = fid
"""
def test_households():
"""reads the households layer of the lesson's session"""
s = summary(HOUSEHOLDS)
assert s["layer"] == "households" and s["geometry"] == "Point"
assert s["count"] == 5000
assert s["extent"] == (500001.6, 2000006.0, 507992.3, 2005998.6)
def test_wards():
"""reads a polygon layer and ignores the lines it does not need"""
s = summary(WARDS)
assert s == {"layer": "wards", "geometry": "Polygon", "count": 12, "extent": (500000.0, 2000000.0, 508000.0, 2006000.0)}
def test_count_is_a_number():
"""returns the count as an int, so that it can be compared"""
assert summary(WARDS)["count"] + 1 == 13 A hint
Go line by line and split each at the first ": " with line.partition(": "). For the extent, take every number out of the text with re.findall(r"-?\d+(?:\.\d+)?", value) and turn the four of them into floats.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Exercise · Medium · Python
Find the options a conversion leaves to chance
Write missing_options(argv) in options.py. argv is an ogr2ogr command as a list of words, such as ["ogr2ogr", "-f", "GeoJSON", "out.geojson", "town.gpkg", "wards"]. Return the list of the lesson's rules that the command leaves to GDAL's defaults, in this order:
"-f"when it names no output format;- for
-f GeoJSON:"RFC7946=YES"when-lco RFC7946=YESis missing, and"COORDINATE_PRECISION"when no-lco COORDINATE_PRECISION=…is given; - for
-f "ESRI Shapefile":"ENCODING=UTF-8"when-lco ENCODING=UTF-8is missing.
A command that follows every rule gives []. The sample tests use the commands of the lesson's sessions.
Starter code · options.py
def missing_options(argv):
"""The rules an ogr2ogr command (a list of words) leaves to GDAL's defaults: "-f", "RFC7946=YES",
"COORDINATE_PRECISION" (GeoJSON) or "ENCODING=UTF-8" (shapefiles)."""
# Replace this line: find the -f value and the -lco values, then test the rules for that format.
return [] The sample tests · test_options.py
from options import missing_options
GOOD_GEOJSON = ["ogr2ogr", "-f", "GeoJSON", "-lco", "RFC7946=YES", "-lco", "COORDINATE_PRECISION=6", "wards.geojson", "town.gpkg", "wards"]
GOOD_SHP = ["ogr2ogr", "-f", "ESRI Shapefile", "-lco", "ENCODING=UTF-8", "wards.shp", "town.gpkg", "wards"]
def test_complete_commands():
"""accepts the lesson's GeoJSON and shapefile commands"""
assert missing_options(GOOD_GEOJSON) == []
assert missing_options(GOOD_SHP) == []
def test_geojson_left_to_defaults():
"""names what a bare GeoJSON export leaves to GDAL"""
assert missing_options(["ogr2ogr", "-f", "GeoJSON", "wards.geojson", "town.gpkg", "wards"]) == ["RFC7946=YES", "COORDINATE_PRECISION"]
assert missing_options(["ogr2ogr", "-f", "GeoJSON", "-lco", "RFC7946=YES", "w.geojson", "town.gpkg"]) == ["COORDINATE_PRECISION"]
def test_shapefile_encoding():
"""catches the shapefile export that turned Marathi names into question marks"""
cli = ["ogr2ogr", "-f", "ESRI Shapefile", "cli_wards.shp", "wards.csv", "-a_srs", "EPSG:32643"]
assert missing_options(cli) == ["ENCODING=UTF-8"]
def test_no_format():
"""asks for -f when the format is left to the file's extension"""
assert missing_options(["ogr2ogr", "out.gpkg", "town.gpkg"]) == ["-f"] A hint
Collect the values that follow each -f and each -lco first: walk the list with an index and look at argv[i + 1] when argv[i] is the option. Then the rules are simple tests on those values.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Close with the self-check, which keeps asking about whichever of the six ideas your answers show to be the shakiest:
Practice that adapts to you
Practise this lesson
15 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up (fresher): What do -t_srs, -nln and -lco do in an ogr2ogr command?
-t_srs reprojects the features to the coordinate reference system it names, such as EPSG:4326, and records it as
the output’s CRS; it is not the same as -a_srs, which only labels the output with a CRS without changing any
coordinate. -nln gives the new layer a name, such as households_w02, instead of the source’s. -lco passes a layer
creation option to the output driver, and these differ by format: RFC7946=YES and COORDINATE_PRECISION=6 for
GeoJSON, ENCODING=UTF-8 for a shapefile, SPATIAL_INDEX=YES for FlatGeobuf. Dataset options use -dsco and open
options for the source -oo. Writing them out keeps a conversion from depending on defaults.
A script that writes correct Marathi names on one computer and question marks on another, and a hand-off of one town to three partners with three needs, close the lesson. The module’s bank sets out each and how to begin:
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Mid to senior
A nightly job converts ward boundaries to a shapefile. On a developer's laptop the Marathi ward names come out correctly; on the build server they are question marks. Both machines "use GDAL". How do you find the cause, and how do you make the job give the same result everywhere?
-
Fresher to mid
A partner wants the town's wards as standard GeoJSON with 6 decimals, the 5,000 households in a file that their web map can read by area, and the clinics for GPS units. Write the commands, and say how you would check each file before sending it.
Flashcards · 7 cards
Recall it, then check
Say the answer to yourself before you look: recalling it is what makes it stick.
-
Which two commands answer most questions about a file you were sent?
ogrinfo -so -nomd file layer for a vector layer's geometry type, count, extent and CRS, and gdalinfo -stats file for a raster's size, cell size, CRS and range of values. The newer gdal vector info and gdal raster info report the same and are still provisional.
-
How do -a_srs, -s_srs and -t_srs differ in ogr2ogr?
-a_srs labels the output with a CRS and changes no coordinate; -s_srs overrides the CRS the source claims; -t_srs reprojects to the CRS it names. Labelling UTM metres with -a_srs EPSG:4326 gives a ward the longitude 500000.
-
Which ogr2ogr options choose what is read?
-where filters by attribute, -spat by a bounding box (it keeps features whose geometry intersects the box: 696 of 5,000 households for W02's box), and -sql runs a query, for example in the SQLite dialect.
-
Which resampling does gdalwarp use unless told otherwise, and when must you change it?
Nearest neighbour when it writes a GeoTIFF, which suits classes such as land cover but not continuous values such as heights: ask for -r bilinear there. When gdalwarp writes a COG the default is cubic instead.
-
Why should a conversion script state every option?
Each option left out is a default that another GDAL version, tool or computer may set differently. State the format, layer name, CRS, geometry type, encoding and precision, stop at the first failure with set -eu, and run the script twice to compare checksums.
-
Why did ogr2ogr write question marks for Marathi ward names while pyogrio kept them?
They ran different GDALs with different defaults: the shapefile driver encodes text as ISO-8859-1 unless told otherwise, whereas pyogrio requests UTF-8 and adds a .cpg file. Pass -lco ENCODING=UTF-8, and print both GDAL versions in every run.
-
How does GDAL read a zipped shapefile, or a file on a web server, without unpacking or downloading it?
Through its virtual file systems: /vsizip/path/to/file.zip/inside/file reads inside an archive, and /vsicurl/ in front of an https:// address fetches only the byte ranges it needs. The shapefile driver also reads and writes .shp.zip files directly.
Key takeaways
ogrinfo -soandgdalinfo -statsreport a file’s type, count or size, extent, CRS and values; the newergdal vector infoandgdal raster infogive the same with a compact CRS, and are still provisional.ogr2ogrreads, filters (-where,-spat,-sql), reprojects (-t_srs) and writes with creation options (-lco,-dsco) in one command;-a_srslabels a CRS without converting.gdal_translateconverts rasters andgdalwarpreprojects them, nearest neighbour by default for a GeoTIFF, which suits classes and not heights; the COG driver writes a GeoTIFF laid out for reading in pieces.- A script that states the format, layer, CRS, encoding and precision, and starts from an empty folder, made the same files twice in a row, which its checksums proved.
- The command line ran GDAL 3.13.3 and pyogrio GDAL 3.12.4, and their default shapefile encodings differ: print the
versions, and state
-lco ENCODING=UTF-8. /vsizip/and the.shp.zipform read zipped data in place, and/vsicurl/reads web files by range.
Terms in this lesson
- creation option
- A format-specific setting that GDAL passes to a driver when it writes a file or a layer, given with -co, -dsco or -lco, such as RFC7946=YES for GeoJSON or ENCODING=UTF-8 for a shapefile.
- GDAL
- The Geospatial Data Abstraction Library: an open-source library and set of command-line programs that read and write hundreds of raster and vector formats through one data model. Its vector half was long called OGR.
- GDAL driver
- The part of GDAL that reads or writes one format, such as GPKG, GeoJSON or GTiff. ogrinfo --formats and gdalinfo --formats list the drivers a copy of GDAL has.
- GDAL virtual file system
- A path prefix such as /vsizip/ or /vsicurl/ that lets GDAL read data inside an archive or over HTTP in place, fetching only the parts it needs instead of unpacking or downloading the whole file.
References
- GDAL: Programs (Open Source Geospatial Foundation (GDAL))
- GDAL: ogr2ogr (Open Source Geospatial Foundation (GDAL))
- GDAL: ogrinfo (Open Source Geospatial Foundation (GDAL))
- GDAL: gdalwarp (Open Source Geospatial Foundation (GDAL))
- GDAL: gdal_translate (Open Source Geospatial Foundation (GDAL))
- GDAL: COG, Cloud Optimized GeoTIFF generator (Open Source Geospatial Foundation (GDAL))
- GDAL: GDAL Virtual File Systems (Open Source Geospatial Foundation (GDAL))
- GDAL: ESRI Shapefile / DBF driver (Open Source Geospatial Foundation (GDAL))
- GDAL: (Geo)Parquet driver (Open Source Geospatial Foundation (GDAL))
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress