Backups and disaster recovery: RPO and RTO

At 14:05 a script deletes most of the orders table of a 1.5 TB database, and the replica deletes the same rows a second later. Whether the shop then loses 12 hours of orders or none, and whether it is selling again in two hours or in five, was settled long before, by two numbers that only the business can set. The first is the recovery point objective (RPO)recovery point objective (RPO): The most data a system may lose in an incident, measured back in time from the incident to the last copy that can be restored. An RPO of one hour allows losing the last hour of writes., how far back from an incident the last copy you can restore may be. The second is the recovery time objective (RTO)recovery time objective (RTO): The longest a system may take to work again after an incident, from the moment it fails until it serves again., how long the system may take to work again.

Both set the architecture and the bill, because smaller targets need more copies, kept closer to ready. Replication does not help with mistakes: a replica copies a mistaken DELETE within a second.

RPO and RTO: two numbers that set the design

Google Cloud’s planning guide defines the RTO as the longest time an application may be offline and the RPO as the longest stretch of data that may be lost, and notes that the smaller both are, the more an application costs to run (Google Cloud). Here they are on one line of time:

Set them per system, from what an hour of downtime and an hour of lost writes would cost, then choose the cheapest design that meets them. The program applies three backup setups to the accident above:

# One accident, three backup setups. At 14:05 a script deletes most of the orders table in a 1.5 TB database. How
# much data does each setup lose, and how long does the restore take? Every figure is a labelled assumption: a
# restore reads backups at 300 MB/s, the database writes 5 MB/s of log, replaying log runs at 60 MB/s, and checks
# before reopening take 30 minutes.
DB_GB = 1_500
RESTORE_MB_S = 300
LOG_MB_S = 5
REPLAY_MB_S = 60
CHECKS_MIN = 30
MISTAKE = 14 * 60 + 5  # minutes after midnight


def hours_minutes(minutes):
    return f"{int(minutes // 60)} h {int(round(minutes % 60)):02d} min"


def restore_minutes(log_minutes):
    """Copy the backup back, replay `log_minutes` of log on top of it, then check."""
    copy = DB_GB * 1_000 / RESTORE_MB_S / 60
    replay = log_minutes * 60 * LOG_MB_S / REPLAY_MB_S / 60
    return copy + replay + CHECKS_MIN


SETUPS = [
    # name, minutes of data lost, minutes of log to replay
    ("nightly backup at 02:00 only", MISTAKE - 2 * 60, 0),
    ("hourly snapshots", MISTAKE - 14 * 60, 0),
    ("nightly base backup + archived log", 0, MISTAKE - 2 * 60),
]
print(f"{'Setup':<38}{'data lost':>14}{'restore takes':>16}")
for name, lost, log in SETUPS:
    print(f"{name:<38}{hours_minutes(lost):>14}{hours_minutes(restore_minutes(log)):>16}")

Setup                                      data lost   restore takes
nightly backup at 02:00 only             12 h 05 min      1 h 53 min
hourly snapshots                          0 h 05 min      1 h 53 min
nightly base backup + archived log        0 h 00 min      2 h 54 min

Nightly backups alone lose 12 hours of orders. Hourly snapshots lose 5 minutes, and a base backup with an archived log loses nothing written before the mistake, at the price of a longer restore, because the log must be replayed. Step through the day of the accident and the three ways back:

Disaster recovery is how a whole system comes back after a disaster: a lost region, a deleted table, a corrupted store. AWS’s paper on it groups the designs into four tiers, from cheap and slow to costly and fast (disaster recovery options):

• Backup and restore: copies of the data, code and configuration are restored into a new environment.
• Pilot lightpilot light: A disaster recovery tier in which data is replicated to a second region whose core stays on, while servers start only during a failover.: data is replicated to a second region with its core always on, and servers start only in a failover.
• Warm standbywarm standby: A disaster recovery tier in which a smaller but working copy of production runs in a second region, ready to take traffic and scale up.: a smaller but working copy of production runs in the second region, ready to scale up.
• Multi-site active/active: every region serves users all the time, so recovery means moving traffic.

Whatever the tier, backups still matter, because replication carries a corruption or a deletion everywhere at once. Point-in-time recoverypoint-in-time recovery: Restoring a base backup and replaying the archived log of changes on top of it up to a chosen moment, such as one second before a mistaken delete. restores a base backup and replays the log to any moment, and a restore drill proves on a schedule that the whole chain works.

Choosing a tier by its yearly cost, worked Pro: opens with a Pro, Premium or Ultimate pass