Estimating storage, bandwidth and cache memory

Traffic tells you how often a system is asked to do something; storage, bandwidth and memory tell you what each of those requests costs it. The three estimates use the same method: start from a rate or a count you already have, multiply by a size, and then account for everything that multiplies that size without anyone asking for it. Most storage estimates that go wrong forget the copies, and most bandwidth estimates that go wrong forget that links are measured in bits.

Storage: count everything that multiplies the data

An object rarely stays one object. A photo service keeps the original and makes smaller copies for display, writes a database row with its indexes for each photo, keeps every byte on several disks so that a failure loses nothing, and keeps it all for years.

Copies are the multiplier most often forgotten. Three full replicas cost three times the data: the Hadoop file system’s documentation describes its default of three replicas as 200% overhead in storage space, and erasure coding as giving the same fault tolerance for no more than 50% overhead in typical setups. With 6 data blocks and 3 parity blocks, data that takes 18 blocks with three replicas takes 9 (HDFS erasure coding). The same documentation names the price: encoding and decoding place extra demands on CPU and network, and it introduces erasure coding for warm and cold data with relatively little I/O.

This script projects five years of a photo service, with uploads growing by a quarter each year:

# Five years of storage for a photo service. Every constant is an assumption to state out loud.
KB = 10**3
MB = 10**6
TB = 10**12
PB = 10**15

UPLOADS_PER_DAY = 10_000_000  # photos uploaded a day in the first year
GROWTH_PER_YEAR = 0.25  # uploads grow by a quarter each year
ORIGINAL = 2.5 * MB  # average photo as uploaded (already compressed)
DERIVED = 300 * KB + 30 * KB  # a medium-size copy and a thumbnail made from each photo
METADATA = 2 * KB  # database row and its index entries per photo
DB_COPIES = 3  # the metadata database keeps three copies
REPLICAS = 3.0  # raw bytes per byte of photo data with three full copies
ERASURE_6_3 = 1.5  # raw bytes per byte with erasure coding, 6 data and 3 parity blocks

photos = 0
print(f"{'Year':<6}{'photos':>8}{'data':>10}{'3 copies':>11}{'EC 6+3':>10}{'metadata':>11}")
for year in range(1, 6):
    photos += UPLOADS_PER_DAY * (1 + GROWTH_PER_YEAR) ** (year - 1) * 365
    data = photos * (ORIGINAL + DERIVED)
    metadata = photos * METADATA * DB_COPIES
    print(f"{year:<6}{photos / 10**9:>5.1f} bn{data / PB:>7.1f} PB{data * REPLICAS / PB:>8.1f} PB{data * ERASURE_6_3 / PB:>7.1f} PB{metadata / TB:>8.0f} TB")

print(f"\nPer photo: {(ORIGINAL + DERIVED) / MB:.2f} MB of image data, {METADATA / KB:.0f} KB of metadata.")
print(f"After five years the image data is {data * REPLICAS / metadata:,.0f} times the metadata, with copies.")

Year    photos      data   3 copies    EC 6+3   metadata
1       3.6 bn   10.3 PB    31.0 PB   15.5 PB      22 TB
2       8.2 bn   23.2 PB    69.7 PB   34.9 PB      49 TB
3      13.9 bn   39.4 PB   118.1 PB   59.1 PB      83 TB
4      21.0 bn   59.6 PB   178.7 PB   89.3 PB     126 TB
5      30.0 bn   84.8 PB   254.3 PB  127.2 PB     180 TB

Per photo: 2.83 MB of image data, 2 KB of metadata.
After five years the image data is 1,415 times the metadata, with copies.

Two decisions fall out of the table. The photos need a storage system built for petabytes of large objects, and the choice between replicas and erasure coding is worth more than a hundred petabytes by the fifth year. The metadata is more than a thousand times smaller, but it still reaches 180 TB with its copies, 60 TB in each, so plan from the start how the metadata database will be partitioned too.

Bandwidth: rates times sizes, in bits

Bandwidth is the request rate times the bytes each request moves, converted to bits because links are rated in bits per second (NIST: 1 byte is 8 bits). Estimate the two directions separately, at the peak:

• Ingress is what clients send in: uploads, posts, writes.
• Egress is what the system sends out: pages, images, video, API responses.

# Peak upload (ingress) and download (egress) bandwidth for the same photo service.
KB = 10**3
MB = 10**6
DAY = 86_400

DAU = 50_000_000  # daily active users
UPLOADS_PER_DAY = 10_000_000
VIEWS_PER_USER = 100  # photos each user looks at a day
PEAK_FACTOR = 2  # busiest hour against the daily average
ORIGINAL = 2.5 * MB
VIEW_MIX = {"thumbnail": (0.8, 30 * KB), "medium copy": (0.2, 300 * KB)}  # share of views, bytes sent


def gbit_per_s(bytes_per_s):
    return bytes_per_s * 8 / 10**9


upload_rate = UPLOADS_PER_DAY / DAY * PEAK_FACTOR
view_rate = DAU * VIEWS_PER_USER / DAY * PEAK_FACTOR
bytes_per_view = sum(share * size for share, size in VIEW_MIX.values())

ingress = upload_rate * ORIGINAL
egress = view_rate * bytes_per_view
print(f"Peak uploads: {upload_rate:,.0f} a second x {ORIGINAL / MB} MB = {gbit_per_s(ingress):.1f} Gbit/s in")
print(f"Peak views:   {view_rate:,.0f} a second x {bytes_per_view / KB:.0f} KB on average = {gbit_per_s(egress):.1f} Gbit/s out")
print(f"Egress is {egress / ingress:.0f} times ingress at the peak.")
print(f"Data sent in an average day: {DAU * VIEWS_PER_USER * bytes_per_view / 10**12:,.0f} TB out, {UPLOADS_PER_DAY * ORIGINAL / 10**12:,.0f} TB in")

Peak uploads: 231 a second x 2.5 MB = 4.6 Gbit/s in
Peak views:   115,741 a second x 84 KB on average = 77.8 Gbit/s out
Egress is 17 times ingress at the peak.
Data sent in an average day: 420 TB out, 25 TB in

For media systems egress is usually much larger than ingress, because each object is uploaded once and viewed many times: here 17 times larger at the peak. That shapes the design. Egress of tens of gigabits a second is a reason to serve images from caches close to users rather than from the origin servers, and the bytes sent out each day are a line of their own in any cost estimate. The mix of what is sent matters as much as the request rate: serving thumbnails for most views keeps the average response near 84 KB instead of 300 KB.

Cache memory: size the hot set, not the dataset

A cache earns its memory when a few items receive most of the reads. Requests in web proxy traces have been found to follow a Zipf-like distribution, in which the item of rank r is requested in proportion to $1 / r^{\alpha}$, with an exponent that varies from one workload to another (Breslau and others). So the question for sizing is how much of the dataset receives how much of the reads. This script draws 400,000 reads over 20,000 items with an exponent of 0.9, then replays them against an LRU cache (OrderedDict makes one in a few lines) of four sizes:

# How much of a dataset does a cache need to hold? Simulate reads whose popularity follows a Zipf-like law,
# then find the hot set and the hit ratio of an LRU cache of a few sizes.
import random
from collections import Counter, OrderedDict

ITEMS = 20_000  # distinct keys in the dataset
REQUESTS = 400_000
SKEW = 0.9  # Zipf exponent: the item of rank r is read in proportion to 1 / r^SKEW
ENTRY_BYTES = 1_100  # an average cached value plus its key and the cache's bookkeeping (an assumption to measure)

rng = random.Random(2026)
weights = [1 / rank**SKEW for rank in range(1, ITEMS + 1)]
trace = rng.choices(range(ITEMS), weights=weights, k=REQUESTS)

counts = sorted(Counter(trace).values(), reverse=True)
for target in (0.5, 0.8, 0.9, 0.99):
    total = 0
    for hot, count in enumerate(counts, start=1):
        total += count
        if total >= target * REQUESTS:
            break
    print(f"{target:>4.0%} of reads go to the hottest {hot:>6,} items ({hot / ITEMS:5.1%} of them)")


def lru_hit_ratio(capacity):
    """Hit ratio of an LRU cache over the second half of the trace (the first half warms it up)."""
    cache = OrderedDict()
    hits = 0
    for i, key in enumerate(trace):
        if key in cache:
            cache.move_to_end(key)
            if i >= REQUESTS // 2:
                hits += 1
        else:
            cache[key] = True
            if len(cache) > capacity:
                cache.popitem(last=False)
    return hits / (REQUESTS - REQUESTS // 2)


print()
for share in (0.01, 0.05, 0.2, 0.5):
    capacity = int(ITEMS * share)
    memory = capacity * ENTRY_BYTES / 10**6
    print(f"LRU cache holding {share:>4.0%} of the items ({memory:4.1f} MB): hit ratio {lru_hit_ratio(capacity):5.1%}")
print(f"Holding all {ITEMS:,} items would take {ITEMS * ENTRY_BYTES / 10**6:.1f} MB.")

 50% of reads go to the hottest    387 items ( 1.9% of them)
 80% of reads go to the hottest  4,686 items (23.4% of them)
 90% of reads go to the hottest  9,286 items (46.4% of them)
 99% of reads go to the hottest 17,381 items (86.9% of them)

LRU cache holding   1% of the items ( 0.2 MB): hit ratio 28.9%
LRU cache holding   5% of the items ( 1.1 MB): hit ratio 47.7%
LRU cache holding  20% of the items ( 4.4 MB): hit ratio 68.5%
LRU cache holding  50% of the items (11.0 MB): hit ratio 85.5%
Holding all 20,000 items would take 22.0 MB.

With this skew, about a quarter of the items receive 80% of the reads, and an LRU cache holding half of the items hits about 85% of the time. Three details matter when you size a real cache:

• Hit ratios rise slowly at the end. Each extra percent of hit ratio costs more memory than the one before, so pick a target (“90% of reads served from the cache”) and size for it.
• An LRU cache does worse than the ideal. LRU evicts by recency, not by long-term popularity, so it hits less often than a cache that held exactly the hottest items. Redis, for example, approximates LRU by sampling keys, within a memory limit set by maxmemory (Redis eviction).
• The skew is an assumption. With a flatter distribution the hot set is larger. Measure the real one from access logs before you buy the memory.

What one entry really costs

Memory per entry is more than the value: the key, the hash table’s slots, pointers and expiry data come on top. This script measures a Python dictionary of 100,000 small entries with tracemalloc, which counts the memory Python allocates (Python documentation):

# What one cached entry really costs: measure a Python dict of 100,000 small entries with tracemalloc.
import tracemalloc

ENTRIES = 100_000
VALUE = 100  # bytes in each value

tracemalloc.start()
before = tracemalloc.get_traced_memory()[0]
cache = {f"photo:{i:09d}": bytes(VALUE) for i in range(ENTRIES)}
after = tracemalloc.get_traced_memory()[0]
tracemalloc.stop()

payload = len("photo:000000000") + VALUE
per_entry = (after - before) / ENTRIES
print(f"Key and value: {payload} bytes of data per entry")
print(f"Memory used:   {per_entry:.0f} bytes per entry, {per_entry / payload:.1f} times the data")

Key and value: 115 bytes of data per entry
Memory used:   227 bytes per entry, 2.0 times the data

The overhead roughly doubled the memory of these 115-byte entries. Dedicated caches have overhead of their own: in an example in Redis’s documentation, its MEMORY USAGE command reports 56 bytes for an empty key with an empty value, all of it bookkeeping (MEMORY USAGE). For small values the overhead can be as large as the data, so measure the cost per entry on the cache you will use and multiply by the entries of the hot set.

Mistakes to check

• One copy assumed. Multiply by the replicas, or by the erasure-coding overhead, and add derived copies and metadata.
• Bytes compared with bits. Convert bandwidth to bits per second before comparing it with a link or a limit.
• Ingress and egress lumped together. Estimate them separately; for media, egress dominates.
• Cache sized for the whole dataset, or for values only. Size for the hot set at a target hit ratio, with the measured cost per entry.

Key takeaways

• Storage = objects a day × size × days retained, plus derived copies and metadata, times the copies kept: 3 for three replicas, 1.5 for erasure coding with 6 data and 3 parity blocks.
• Bandwidth = peak requests a second × bytes per request × 8, separately for ingress and egress.
• Size a cache from the hot working set and a target hit ratio, at the measured memory cost of an entry, which for small values can be about twice the data.
• Let the estimate drive a decision: object storage for the photos, partitioning for the metadata, caches near users for the egress.