SQL (PostgreSQL, MySQL, SQLite) Module 17 – Security: injection, privileges and data protection
Backups, restores and protecting personal data
Make backups of PostgreSQL, MySQL and SQLite that restore, test every copy, and protect personal data with masking, encryption and hashing.
What you will learn
- Create and restore logical backups in PostgreSQL, MySQL and SQLite
- Test a backup by restoring it and comparing row counts and checksums
- Protect personal data in copies with minimisation, masking and encryption
- Avoid storing passwords or secrets in plain text
Before you start
What you’ll learn · Before you start
What you will learn
- Create and restore logical backups in PostgreSQL, MySQL and SQLite
- Test a backup by restoring it and comparing row counts and checksums
- Protect personal data in copies with minimisation, masking and encryption
- Avoid storing passwords or secrets in plain text
Before you start
On this page
On this page, a plain copy of a live SQLite database file held 4 orders where the database held 5, and still
passed the integrity check. A backup is a copy of a database that has been
restored at least once; until a restore works, it is only a file. logical backup: A backup that writes the data out as SQL statements or an archive of rows, made by pg_dump, mysqldump or SQLite's .dump. It restores into a fresh database, often of a newer version, as of the moment the dump started.
write the data out as SQL statements or an archive that rebuilds it: pg_dump and pg_restore for PostgreSQL,
mysqldump for MySQL, and the .dump command or VACUUM INTO for SQLite.
physical backup: A backup that copies a database's files. It restores quickly, but only into the same major version and as a whole server; with an archived write-ahead or binary log it can replay changes to a chosen moment. copy the database’s files, and with the write-ahead log of
PostgreSQL or the binary log of MySQL they can replay changes up to a chosen moment.
Every copy holds every customer’s personal data, so backups and the copies made from them need the protection of the live database. Protecting personal data starts with keeping less of it, masking it in copies that leave production, encrypting it on the network and on disk, and never storing passwords at all, only slow salted hashes.
Logical and physical backups restore to different moments
Logical and physical backups, and the test both must pass
Text description of the diagram
The diagram starts from the live database and shows two ways to back it up.
- A logical backup, made with pg_dump, mysqldump or SQLite's .dump, writes SQL text or an archive of rows. It is restored into a fresh database and holds the data as of the moment the dump started.
- A physical backup copies the data files. Separately, the write-ahead log of PostgreSQL or the binary log of MySQL is archived as every change happens. A restore copies the files back and replays the log up to a chosen moment.
Both paths end in the same restore test: an integrity check, row counts, checksums, and the time the restore took.
| Criterion | Logical dump | Physical copy and log |
|---|---|---|
| Holds | SQL or an archive of rows | Data files and every later change |
| Restores to | When the dump started | Any moment the log reaches |
| Newer version | Usually yes | No |
| One table alone | Yes | No |
| Large databases | Slow: indexes rebuilt | Fast: files copied |
| When to choose | Small and medium databases, single tables, upgrades | Large databases, recovery to the minute |
Two numbers describe any backup plan. The recovery point objective (RPO): How much recent data a business can afford to lose in a failure, measured in time. A backup plan of one dump a night can lose up to a day of changes. (RPO) is how much recent data you can afford to lose: with one dump a night, up to a day of it. The recovery time objective (RTO): How long a restore may take before an outage costs too much. Only a timed restore of the real database shows whether a backup plan keeps it. (RTO) is how long a restore may take before the outage costs too much. Both are promises, and only a timed restore tells you whether a plan keeps them. Walk through one day of a shop whose only backup is a nightly dump:
- 02:00
The nightly dump starts
pg_dump or mysqldump reads the database as it is at this moment. Everything committed afterwards is in no backup until the next night.
- 02:00 to 13:59
The shop keeps working
Every order, payment and refund of the morning exists only in the live database, on one server.
- 14:00
The database is lost failure
A failed disk, a dropped table or a bad deploy. The live data is gone, and the newest copy is the 02:00 dump.
- Recovery point
Twelve hours of data are gone
Restoring the dump brings the database back as it was at 02:00. That loss is the recovery point; a nightly dump promises at most a day, and a failure at 01:59 loses almost all of it.
- Recovery time
Closed until the restore ends
The shop is down while the dump is restored, its indexes rebuilt and its rows checked. How long that may take is the recovery time objective, and only a timed restore shows whether a plan keeps it.
The timeline has 5 events, from 02:00 to Recovery time.
Most teams combine the two kinds: a physical backup with an archived log for recovery to the minute, and a regular logical dump that restores anywhere, into a newer version or one table at a time. Keep at least one copy out of reach of the database server’s own credentials, in another account or a store that refuses deletes for a set time, because a mistyped command and an attacker can both reach backups that sit next to the database.
Checkpoint
Check: Logical and physical backups, RPO and RTO
A backup counts only once a restore of it passes a test
A copy that restores into a fresh database with the same row counts and checksums is a backup. The program below backs up a live SQLite database four ways while it is in use, then gives each copy the test a restore drill would.
"""Back up a live SQLite database four ways, then test each copy the way a restore test does."""
import hashlib
import pathlib
import shutil
import sqlite3
for name in ("mart.db", "mart.db-wal", "mart.db-shm", "file-copy.db", "online.db", "vacuumed.db", "restored.db"):
pathlib.Path(name).unlink(missing_ok=True)
live = sqlite3.connect("mart.db", autocommit=True)
live.executescript("""
CREATE TABLE customers (customer_id INTEGER PRIMARY KEY, name TEXT NOT NULL, city TEXT NOT NULL);
CREATE TABLE orders (order_id INTEGER PRIMARY KEY, customer_id INTEGER NOT NULL REFERENCES customers, status TEXT NOT NULL);
INSERT INTO customers VALUES (1, 'Asha Kulkarni', 'Pune'), (2, 'Vikram Nair', 'Kochi'), (3, 'Meera Rathore', 'Jaipur');
INSERT INTO orders VALUES (101, 1, 'delivered'), (102, 2, 'delivered'), (103, 1, 'placed'), (104, 3, 'cancelled');
PRAGMA journal_mode = WAL;
INSERT INTO orders VALUES (105, 2, 'placed');
""")
# Order 105 is committed, but in WAL mode it sits in mart.db-wal until the next checkpoint.
shutil.copyfile("mart.db", "file-copy.db") # 1. copying the file while the database is in use
with sqlite3.connect("online.db") as copy: # 2. the online backup API
live.backup(copy)
live.execute("VACUUM INTO 'vacuumed.db'") # 3. VACUUM INTO (SQLite 3.27 and later)
dump = "\n".join(live.iterdump()) # 4. a logical dump: SQL text that rebuilds everything
with sqlite3.connect("restored.db") as restored:
restored.executescript(dump)
def fingerprint(path):
"""What a restore test compares: integrity, row counts and a checksum of every row in key order."""
db = sqlite3.connect(path)
ok = db.execute("PRAGMA integrity_check").fetchone()[0]
counts = tuple(db.execute(f"SELECT count(*) FROM {t}").fetchone()[0] for t in ("customers", "orders"))
rows = db.execute("SELECT * FROM customers ORDER BY 1").fetchall() + db.execute("SELECT * FROM orders ORDER BY 1").fetchall()
db.close()
return ok, counts, hashlib.sha256(repr(rows).encode()).hexdigest()[:12]
expected = fingerprint("mart.db")
print(f"{'live database':14} integrity={expected[0]} rows={expected[1]} checksum={expected[2]}")
for path in ("file-copy.db", "online.db", "vacuumed.db", "restored.db"):
ok, counts, digest = fingerprint(path)
verdict = "restores correctly" if (ok, counts, digest) == expected else "FAILS the restore test"
print(f"{path:14} integrity={ok} rows={counts} checksum={digest} {verdict}") Output
live database integrity=ok rows=(3, 5) checksum=011ac06f2bfd file-copy.db integrity=ok rows=(3, 4) checksum=324ecc947a20 FAILS the restore test online.db integrity=ok rows=(3, 5) checksum=011ac06f2bfd restores correctly vacuumed.db integrity=ok rows=(3, 5) checksum=011ac06f2bfd restores correctly restored.db integrity=ok rows=(3, 5) checksum=011ac06f2bfd restores correctly
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 backup_and_restore.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Three copies restore correctly, and the plain file copy fails the test. Step through why it missed order 105:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- Four orders in the main file. mart.db holds three customers and orders 101 to 104. Then PRAGMA journal_mode = WAL switches the database to write-ahead logging.
- Order 105 is committed. The INSERT of order 105 commits, and in WAL mode the commit is appended to mart.db-wal. mart.db stays as it was until the next checkpoint.
- Copying the file. shutil.copyfile copies mart.db alone while the database is in use. The copy has 4 orders: order 105 was never in the file it copied.
- The copy looks healthy. file-copy.db opens without complaint and passes PRAGMA integrity_check. Nothing about it says that an order is missing.
- Copies through SQLite. The backup API, VACUUM INTO and the dump read the database through SQLite, which reads the write-ahead log as well, so each of them has all 5 orders.
- The restore test decides. Compared with the live database, rows (3, 5) and its checksum, three copies match and the file copy fails the test. Opening a copy proves nothing; comparing it does.
The database runs in WAL mode, where a commit first goes to the file mart.db-wal and reaches mart.db only at the
next checkpoint. The copy looks healthy, passes the integrity check and opens without complaint, which is exactly why
a backup has to be compared with its source, not just opened. The backup API, VACUUM INTO and the dump all read
through SQLite itself, so they see every committed transaction.
Key idea
A restore test checks three things, the engine’s integrity check, every table’s row count and a checksum over the rows in key order, and it is timed, because that time is your real recovery time.
Run it on a schedule. On PostgreSQL and MySQL the drill is the same: restore into a scratch server, compare each table’s row count and checksum with the source, and record how long it took.
Checkpoint
Check: Testing a backup by restoring it
Copies of personal data need less data, masks and encryption
Protecting personal data in copies follows four rules:
- Minimisation: a column no purpose needs is not collected, and a copy leaves out what its users do not need.
- Masking: copies for development, testing or analytics replace names, phone numbers and other identifiers with stand-ins that keep their format and their joins.
- Encryption: connections use TLS, disks and backup files are encrypted, and the keys live apart from the data.
- Password hashing: the application stores a slow, salted hash for each password, never the password itself and
never
md5()computed in SQL.
Step through the four rules on one request, a copy of production for developers:
The drawing shows the whole scene; its 5 steps are listed below.
Every step as text
- The request. Developers chasing a bug in orders ask for a copy of production. A straight copy would hand them every customer's name, phone number and address.
- Keep less. Minimisation first. The copy takes the tables the bug touches and leaves out the columns no test needs; data that is not copied cannot leak.
- Replace who people are. Masking replaces names, phone numbers and other identifiers with stand-ins that keep their format and their joins, so the queries of the bug still work.
- Protect the file. The copy travels over TLS, and the file is encrypted with a key that lives apart from it.
- Passwords were never there. The users table holds only the slow, salted hashes that the application computed, never a password, so even this copy holds none.
These rules apply to backups as much as to copies: a dump restored on a laptop is a copy of production with every protection of the server removed. Secrets such as API keys and tokens belong in a secrets manager rather than in a table, because every dump copies every table it covers, and a dump travels further than the database ever does.
Checkpoint
Check: Protecting personal data in copies
Dump and restore in PostgreSQL, MySQL and SQLite, step by step Pro: opens with a Pro, Premium or Ultimate pass
Each engine’s dump tool has a default that surprises people at their first real restore. This part runs the dump and restore commands of each engine with measured sizes and times, and reproduces a restore that overwrote a live MySQL database.
In this part:
- Dump and restore with PostgreSQL’s tools, measured
- mysqldump, and the restore that replaced a live database, reproduced
- SQLite: copies that are safe while the database is in use
- Point-in-time recovery and the two objectives, worked
3 runnable examples (dump_formats.py, dump_restore.py, mysql_server.py, pg_server.py, restore_checks.py) · about 6 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
A masked copy that keeps joins working, built and checked Pro: opens with a Pro, Premium or Ultimate pass
Developers and testers need realistic data, and a straight copy of production also hands them every customer’s phone number. This part builds a masked copy that keeps joins, uniqueness and foreign keys intact, checks it automatically, and ends with an exercise.
In this part:
- What to mask, keep or drop
- Consistent masking with a keyed mapping, run in your browser
- Why keeping the last four digits is not masking, worked
- Your turn: mask phone numbers without breaking the join
- Mask phone numbers without breaking a join
- Results of the sample tests
2 runnable examples (Python) · 1 exercise with sample tests · about 5 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Passwords, secrets and encryption, with the failures reproduced Pro: opens with a Pro, Premium or Ultimate pass
Hashing inside SQL looks safe and is not, and encryption helps only when its key lives somewhere else. This part reproduces what a leaked column of unsalted hashes gives away, stores passwords the way current guidance asks, and places encryption keys and logs where they protect anything.
In this part:
- What a leaked md5 column gives away, reproduced
- Password hashing parameters, with the memory worked out
- Encryption in transit, at rest and in backup files
- Logging access to sensitive tables
2 runnable examples (Python, key_in_log.py, pg_server.py) · about 5 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Practise: back up, test and protect
The questions cover the two kinds of backup, the restore test and the rules for copies, weakest idea first:
Practice that adapts to you
Practise this lesson
9 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
-
Fresher to mid · Warm-up
What is the difference between a logical and a physical database backup?
A logical backup writes the database out as SQL statements or an archive of definitions and rows, made by tools such as
pg_dump,mysqldumpor SQLite's.dump. It restores into a fresh database, often of a newer version or, with some edits, another engine, and can bring back a single table, but it reflects only the moment the dump started and a restore rebuilds every index, which is slow for large databases. A physical backup copies the database's files. It restores quickly, but only into the same major version, and only as a whole server. Combined with the archived write-ahead log of PostgreSQL or the binary log of MySQL, it can replay changes up to a chosen moment, which a nightly dump cannot. Either kind counts only once a restore of it has been tested. -
Mid to senior
Your company has taken a nightly pg_dump of a 200 GB PostgreSQL database for two years, and nobody has ever restored one. Design the restore test, and say which numbers you report to management.
-
Mid to senior
Developers ask for a copy of production to debug a payment issue. The tables hold names, phone numbers and addresses of 2 million customers. What do you give them, and how do you build and check it?
-
Mid to staff
A database dump containing md5(password) values for 500,000 users has leaked. What can an attacker do with it, and how do you move every account to a safe scheme without forcing all users to reset their passwords?
Key takeaways
- Logical dumps restore the moment they started and travel between versions, while physical backups with an archived log restore any moment the log reaches.
- A backup counts only once a restore of it has passed an integrity check, row counts and checksums, and was timed.
- Use the engine’s own tools to copy a live database, because a plain file copy can miss committed changes.
- Copies that leave production keep less data, mask identifiers consistently and are encrypted, and passwords are only ever stored as slow, salted hashes.
Terms in this lesson
- logical backup
- A backup that writes the data out as SQL statements or an archive of rows, made by pg_dump, mysqldump or SQLite's .dump. It restores into a fresh database, often of a newer version, as of the moment the dump started.
- physical backup
- A backup that copies a database's files. It restores quickly, but only into the same major version and as a whole server; with an archived write-ahead or binary log it can replay changes to a chosen moment.
- recovery point objective (RPO)
- How much recent data a business can afford to lose in a failure, measured in time. A backup plan of one dump a night can lose up to a day of changes.
- recovery time objective (RTO)
- How long a restore may take before an outage costs too much. Only a timed restore of the real database shows whether a backup plan keeps it.
References
- PostgreSQL documentation: SQL Dump (The PostgreSQL Global Development Group)
- PostgreSQL documentation: pg_dump (The PostgreSQL Global Development Group)
- PostgreSQL documentation: pg_restore (The PostgreSQL Global Development Group)
- PostgreSQL documentation: pg_dumpall (The PostgreSQL Global Development Group)
- PostgreSQL documentation: Continuous Archiving and Point-in-Time Recovery (The PostgreSQL Global Development Group)
- PostgreSQL documentation: pgcrypto (The PostgreSQL Global Development Group)
- PostgreSQL documentation: Error Reporting and Logging (log_statement) (The PostgreSQL Global Development Group)
- PostgreSQL 18 Release Notes (The PostgreSQL Global Development Group)
- pgAudit: the PostgreSQL Audit Extension (The pgAudit project)
- MySQL 8.4 Reference Manual: mysqldump, a Database Backup Program (Oracle Corporation)
- MySQL 8.4 Reference Manual: Backup and Recovery Types (Oracle Corporation)
- MySQL 8.4 Reference Manual: Point-in-Time (Incremental) Recovery (Oracle Corporation)
- SQLite: VACUUM (VACUUM INTO) (SQLite)
- SQLite: Release 3.27.0 (VACUUM INTO) (SQLite)
- SQLite: Using the SQLite Online Backup API (SQLite)
- SQLite: How To Corrupt An SQLite Database File (backup while a transaction is active) (SQLite)
- SQLite: Write-Ahead Logging (SQLite)
- sqlite3: DB-API 2.0 interface for SQLite databases (backup, iterdump) (Python Software Foundation)
- random: Generate pseudo-random numbers (Python Software Foundation)
- hashlib: Secure hashes and message digests (scrypt) (Python Software Foundation)
- OWASP Cheat Sheet Series: Password Storage Cheat Sheet (OWASP Foundation)
- OWASP Cheat Sheet Series: Cryptographic Storage Cheat Sheet (OWASP Foundation)
- OWASP Cheat Sheet Series: Logging Cheat Sheet (OWASP Foundation)
- RFC 2606: Reserved Top Level DNS Names (Internet Engineering Task Force)
- The Digital Personal Data Protection Act, 2023 (No. 22 of 2023) (Ministry of Electronics and Information Technology, Government of India)
- Digital Personal Data Protection Rules, 2025 (Ministry of Electronics and Information Technology, Government of India)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress