Block, file and object storage

Block storage gives one server a raw disk made of numbered blocks. File storage gives many servers a shared tree of directories and files that they can open and change in place. Object storage gives any client, over HTTPS, a flat namespace of keys, each holding a whole object with its metadata. The three often sit on the same kind of drives; what differs is the interface, and the interface decides who can share the data, how a change is written and how far the system can grow. A database wants block storage, software written for a file system wants file storage, and uploads, media, backups and logs at large scale want object storage. This lesson compares the three, shows how an object store behaves when you write, list and overwrite, and prices the storage classes that make old data cheap.

Three interfaces to the same disks

Block storage presents a volume: a long array of fixed-size blocks that the server reads and writes by number. The volume knows nothing about files. The server’s operating system puts a file system on it, or a database writes its pages to it directly, and that software decides what the blocks mean. Because only one owner can safely manage the blocks, a volume is usually attached to one server at a time. A cloud volume such as Amazon EBS behaves like a local hard drive once attached, and its data is copied across several servers inside one Availability Zone (Amazon EBS).

File storage moves the file system onto a server of its own and shares it over the network with a protocol such as NFS or SMB. Many machines mount the same tree at once. Each can open a file, seek to any offset, overwrite a few bytes, append, rename a directory and take a lock, with the semantics that ordinary programs expect. Amazon EFS, for example, speaks NFS version 4 and offers strong data consistency and file locking to many instances at once (Amazon EFS).

Object storage drops the hierarchy and the in-place edits. A bucket maps a key, a string such as users/42/avatar.png, to an object: the bytes plus metadata such as the content type. Clients send HTTP requests to store, read, delete and list objects by key, from anywhere, and the store spreads them over as many machines as it needs. That simple interface is what lets one bucket hold billions of objects for millions of clients.

The three side by side

Object storage pays for its reach with latency: every request is an HTTP round trip to a distributed service, and only special low-latency classes such as S3 Express One Zone promise single-digit milliseconds (Amazon S3). Durability differs too, and it surprises people. The Amazon EBS documentation states 99.8 % to 99.9 % durability a year for most volume types, with an annual failure rate of 0.1 % to 0.2 %, while the standard S3 storage classes are designed for 99.999999999 % (EBS, S3 storage classes). A volume is a place to work; the copy you cannot lose belongs in object storage, which is why volume snapshots and database backups are kept there.

How an object store behaves

Four rules explain most of the surprises engineers meet with object storage. The example below follows them in a small model of a bucket and counts the requests and bytes each step costs.

# A small model of how an object store behaves: a bucket maps a key to a whole object and its metadata. It follows
# the rules object stores document (whole-object writes, flat keys, last writer wins, conditional create) and counts
# the requests and bytes each operation costs. It is a teaching model, not a client for a real service.
import hashlib


class Bucket:
    def __init__(self):
        self.objects = {}  # key -> (bytes, metadata)
        self.requests = 0
        self.bytes_sent = 0

    def put(self, key, data, content_type="application/octet-stream", if_none_match=False):
        self.requests += 1
        self.bytes_sent += len(data)
        if if_none_match and key in self.objects:
            return 412  # Precondition Failed: the key exists, so nothing is overwritten
        etag = hashlib.sha256(data).hexdigest()[:8]  # here a short hash of the bytes
        self.objects[key] = (data, {"Content-Type": content_type, "ETag": etag, "Content-Length": len(data)})
        return 200

    def get(self, key, first=None, last=None):
        self.requests += 1
        data, _ = self.objects[key]
        return data if first is None else data[first : last + 1]  # a Range request reads part of an object

    def head(self, key):
        self.requests += 1
        return self.objects[key][1]

    def delete(self, key):
        self.requests += 1
        self.objects.pop(key, None)

    def copy(self, src, dst):
        self.requests += 1
        self.objects[dst] = self.objects[src]

    def list(self, prefix="", delimiter=None):
        """Keys under a prefix; with a delimiter, the next level is grouped into common prefixes ("folders")."""
        self.requests += 1
        keys = sorted(k for k in self.objects if k.startswith(prefix))
        if delimiter is None:
            return keys, []
        files, folders = [], set()
        for k in keys:
            rest = k[len(prefix) :]
            if delimiter in rest:
                folders.add(prefix + rest.split(delimiter)[0] + delimiter)
            else:
                files.append(k)
        return files, sorted(folders)


b = Bucket()
b.put("users/42/photos/goa/beach.jpg", b"J" * 5_000_000, "image/jpeg")
b.put("users/42/photos/pune/fort.jpg", b"J" * 3_000_000, "image/jpeg")
b.put("users/42/avatar.png", b"P" * 40_000, "image/png")

print("1. Keys are flat names; a delimiter only groups them into prefixes.")
files, folders = b.list("users/42/", delimiter="/")
print("   files:", files)
print("   common prefixes:", folders)
print("   one level down:", b.list("users/42/photos/", delimiter="/")[1])

print("\n2. Changing 10 bytes means writing the whole object again.")
before = b.bytes_sent
data = bytearray(b.get("users/42/photos/goa/beach.jpg"))
data[100:110] = b"0123456789"
b.put("users/42/photos/goa/beach.jpg", bytes(data), "image/jpeg")
print(f"   bytes uploaded for a 10-byte change: {b.bytes_sent - before:,}")
print(f"   but a Range read of 10 bytes returns: {b.get('users/42/photos/goa/beach.jpg', 100, 109)!r}")

print("\n3. Renaming a 'folder' of 1,000 objects is 1,000 copies and 1,000 deletes.")
for i in range(1000):
    b.put(f"tmp/upload-7/part-{i:04d}", b"x")
start = b.requests
keys, _ = b.list("tmp/upload-7/")
for k in keys:
    b.copy(k, k.replace("tmp/upload-7/", "done/upload-7/"))
    b.delete(k)
print(f"   requests: {b.requests - start:,} (1 list, 1,000 copies, 1,000 deletes); a file system renames a directory in one call")

print("\n4. Two clients write the same key: the last write wins and the first is lost silently.")
b.put("users/42/profile.json", b'{"name": "Asha"}', "application/json")
b.put("users/42/profile.json", b'{"name": "Ravi"}', "application/json")
print("   stored:", b.get("users/42/profile.json").decode())
status = b.put("users/42/profile.json", b'{"name": "Meera"}', "application/json", if_none_match=True)
print(f"   a conditional create on the same key gets HTTP {status}, and the stored value stays:", b.get("users/42/profile.json").decode())

print("\n5. Metadata travels with the object and is read without the body.")
print("  ", b.head("users/42/avatar.png"))

1. Keys are flat names; a delimiter only groups them into prefixes.
   files: ['users/42/avatar.png']
   common prefixes: ['users/42/photos/']
   one level down: ['users/42/photos/goa/', 'users/42/photos/pune/']

2. Changing 10 bytes means writing the whole object again.
   bytes uploaded for a 10-byte change: 5,000,000
   but a Range read of 10 bytes returns: b'0123456789'

3. Renaming a 'folder' of 1,000 objects is 1,000 copies and 1,000 deletes.
   requests: 2,001 (1 list, 1,000 copies, 1,000 deletes); a file system renames a directory in one call

4. Two clients write the same key: the last write wins and the first is lost silently.
   stored: {"name": "Ravi"}
   a conditional create on the same key gets HTTP 412, and the stored value stays: {"name": "Ravi"}

5. Metadata travels with the object and is read without the body.
   {'Content-Type': 'image/png', 'ETag': 'a24cdd48', 'Content-Length': 40000}

Keys are names, not paths. There are no directories. A listing with a prefix and a delimiter groups keys into “common prefixes”, which consoles draw as folders, but nothing is stored for them. So renaming a “folder” means copying and deleting every object under it: 2,001 requests for 1,000 objects in step 3, where a file system renames a directory in one call. Choose keys you will never need to rename, such as a user id and an upload id, and keep human names in metadata or in a database.

Objects are written whole. You cannot change bytes inside an object, and in most buckets you cannot append to it either; S3’s single-zone, low-latency class is an exception that accepts appends to an object, up to 10,000 of them (appending data). Step 2 changes 10 bytes and has to upload all 5,000,000 again. Reads are more flexible: a Range request returns just the bytes asked for, which is how video players and large-file downloaders read pieces. Design data that changes around this rule: split it into chunks and keep a small manifest object that lists them, or write new immutable objects instead of editing old ones. Log segments, database snapshots and media segments all fit naturally.

Reads see the latest write, but concurrent writes race. S3 gives strong read-after-write consistency for PUT and DELETE in all regions: once a write is acknowledged, every following read and listing reflects it (Amazon S3). Older material that calls S3 eventually consistent is out of date for objects; bucket settings such as enabling versioning still take time to spread. Two writers to the same key are another matter: the store keeps the write with the latest timestamp and the other is lost without an error, as in step 4. There is no lock and no transaction across keys. HTTP’s conditional requests close the gap (RFC 9110): in S3, a write with If-None-Match: * fails with 412 Precondition Failed when the key already exists, and If-Match with an ETag writes only if nobody changed the object since you read it (conditional writes).

Metadata travels with the object. Every object carries system metadata (its size, an ETag that changes when the content changes, Content-Type) and any user metadata set when it was written. A HEAD request returns it without the body, as step 5 shows. Metadata cannot be searched, though, so anything you need to query, such as “all photos of user 42 taken in Pune”, belongs in a database row that points at the key.

One photo upload, three ways

Picture a photo-sharing service whose web servers receive uploads.

• On a block volume attached to one web server, every photo lives on that server. A second server cannot see them, so the fleet cannot grow without sharding users by server. When the server dies, its photos are unreachable until the volume is attached elsewhere, and when the volume fails, they are gone unless a snapshot exists.
• On a shared file system, all web servers mount the same tree, so any of them can serve any photo. The file server becomes the shared bottleneck for every read, and its locking and metadata operations slow down as directories fill with millions of files.
• In an object store, the web servers keep no files at all. They store each photo under a key such as photos/42/9f3c.jpg, record the key in the database, and serve reads through a CDN. Servers can be added or replaced freely. Later lessons in this module let the phone upload straight to the store, skipping the servers entirely.

The third design is the default for user content because it removes state from the servers. The first two remain right for what they are good at: the database behind the service still runs on block storage.

Storage classes: paying less for data nobody reads

An object store charges for bytes stored each month, for requests and, in the colder classes, for every GB read back. Storage classes trade those prices against each other. In S3, Standard has no retrieval fee and no minimums; Standard-IA stores for less but bills a retrieval fee, a minimum storage duration of 30 days and a minimum billable size of 128 KB; Glacier Instant Retrieval is cheaper still, with a 90-day minimum and millisecond reads; Glacier Flexible Retrieval and Deep Archive must be restored before reading (S3 storage classes). A standard restore from Flexible Retrieval typically takes 3 to 5 hours and from Deep Archive up to 12, with bulk restores slower and expedited ones faster for Flexible Retrieval only (retrieval options).

Lifecycle rules move objects between classes by age, so new uploads start in the hot class and cool down automatically. The example prices a photo service of 101 TB under four mixes. The prices are assumptions for the example; real ones depend on the provider and region, so plug in the current list.

# Monthly storage bill of a photo service under four storage-class mixes. Every number below is an assumption for
# the example: prices differ by provider and region and change over time, so plug in the current price list.
GB = 10**9

# Assumed prices in US dollars: storage per GB-month, retrieval per GB read, and the smallest size an object is
# billed at (small objects in the infrequent-access classes are billed as if they were 128 KB).
CLASSES = {
    "standard": {"store": 0.023, "read": 0.0, "min_bytes": 0},
    "infrequent": {"store": 0.0125, "read": 0.01, "min_bytes": 128_000},
    "archive-instant": {"store": 0.004, "read": 0.03, "min_bytes": 128_000},
}

# The data: 50 million originals of 2 MB and one 20 KB thumbnail each, split by age. "reads" is the share of each
# group's bytes read in a month (originals are opened rarely once they are old; thumbnails are shown all the time).
GROUPS = [
    # name, objects, bytes each, share of the bytes read in a month
    ("originals under 30 days", 5_000_000, 2_000_000, 0.50),
    ("originals 30-180 days", 10_000_000, 2_000_000, 0.05),
    ("originals over 180 days", 35_000_000, 2_000_000, 0.01),
    ("thumbnails under 30 days", 5_000_000, 20_000, 3.0),
    ("thumbnails over 30 days", 45_000_000, 20_000, 3.0),
]

MIXES = {
    "A: everything in standard": ["standard"] * 5,
    "B: originals to infrequent after 30 days": ["standard", "infrequent", "infrequent", "standard", "standard"],
    "C: B, then archive-instant after 180 days": ["standard", "infrequent", "archive-instant", "standard", "standard"],
    "D: C, and old thumbnails forced to infrequent": ["standard", "infrequent", "archive-instant", "standard", "infrequent"],
}


def monthly_cost(cls, objects, size, read_share):
    c = CLASSES[cls]
    billed_gb = objects * max(size, c["min_bytes"]) / GB
    read_gb = objects * size * read_share / GB
    return billed_gb * c["store"], read_gb * c["read"]


stored_tb = sum(n * size for _, n, size, _ in GROUPS) / 10**12
print(f"Data stored: {stored_tb:.0f} TB in {sum(n for _, n, _, _ in GROUPS):,} objects\n")
print("US dollars a month")
print(f"{'mix':<46}{'storage':>8}{'reads':>8}{'total':>8}")
base = None
for name, plan in MIXES.items():
    store = read = 0.0
    for cls, (_, n, size, share) in zip(plan, GROUPS):
        s, r = monthly_cost(cls, n, size, share)
        store += s
        read += r
    total = store + read
    base = base or total
    print(f"{name:<46}{store:>8,.0f}{read:>8,.0f}{total:>8,.0f}  ({total / base:.0%} of A)")

print("\nWhy D costs more than C: each 20 KB thumbnail is billed as 128 KB, and every view pays a retrieval fee.")
s_std, _ = monthly_cost("standard", 45_000_000, 20_000, 3.0)
s_ia, r_ia = monthly_cost("infrequent", 45_000_000, 20_000, 3.0)
print(f"old thumbnails in standard: {s_std:,.0f} a month; in infrequent: {s_ia:,.0f} storage + {r_ia:,.0f} reads")

Data stored: 101 TB in 100,000,000 objects

US dollars a month
mix                                            storage   reads   total
A: everything in standard                        2,323       0   2,323  (100% of A)
B: originals to infrequent after 30 days         1,378      17   1,395  (60% of A)
C: B, then archive-instant after 180 days          783      31     814  (35% of A)
D: C, and old thumbnails forced to infrequent      834      58     892  (38% of A)

Why D costs more than C: each 20 KB thumbnail is billed as 128 KB, and every view pays a retrieval fee.
old thumbnails in standard: 21 a month; in infrequent: 72 storage + 27 reads

Moving old originals to the infrequent class cuts the bill to 60 %, and moving the oldest to the archive class with millisecond reads brings it to 35 %, because those photos are rarely opened. Mix D shows the trap. The thumbnails are 20 KB but billed as 128 KB in the colder classes, and they are viewed all the time, so each view also pays a retrieval fee: the old thumbnails cost about five times as much there as in standard storage. For the same reason S3 lifecycle rules do not transition objects smaller than 128 KB by default, and each transition is a billed request (S3 Lifecycle).

Three questions pick a class for any dataset. How often is it read? Data read more than about once a month stays standard. How large are its objects? Small objects belong in the classes without a minimum size, or packed together into larger ones. How fast must a rare read be? A legal archive that must answer within seconds needs a class with millisecond access, while a disaster-recovery copy can wait hours for a restore.

Interview questions

What is the difference between block, file and object storage? Block storage is a volume of numbered blocks that one server formats and manages, which suits databases. File storage is a shared directory tree that many servers mount and edit in place over NFS or SMB. Object storage is a flat namespace of keys holding whole objects with metadata, read and written over HTTP by any client, which scales furthest and suits uploads, media, backups and logs. The deciding question is which interface the software needs.

Why do object stores replace objects whole, and what do designs do instead? Replacing an object whole keeps every write atomic and lets the store copy and cache objects freely, because an object never changes under a reader. Data that grows or changes is written as new immutable objects: chunks plus a manifest that lists them, numbered log segments, or a new version under a new key with the database pointing at the current one.

Key takeaways

• Block storage is a disk for one server, file storage a shared tree for many, object storage a key-to-object service for any client; the interface decides sharing, change granularity and scale.
• Objects are written whole, keys are flat names, and concurrent writers race with the last write winning, so use immutable keys, manifests and conditional writes.
• S3 reads see acknowledged writes immediately; metadata rides with the object, but anything you query belongs in a database.
• Storage classes trade storage price for retrieval fees, minimum durations and minimum sizes: cold, large objects save the most, small hot objects can cost more.