Descriptors: how properties and methods work

qty = Positive() is one line in a class body, and from then on every assignment to qty, the one in __init__ included, is checked: setting it to 0 raises ValueError and keeps the old value. Positive is a descriptor, an object stored as a class attribute that decides what happens when that attribute is read, assigned or deleted through an instance. It does so by defining __get__, and optionally __set__ and __delete__: when the attribute lookupattribute lookup: What Python does to evaluate obj.name: the default __getattribute__ looks for a data descriptor on the class, then in the instance's own dictionary, then for anything else on the class, and only when all three fail is __getattr__ called. More in “Attribute lookup and the data model” of the previous lesson finds such an object on the class, it calls the method instead of handing the object back.

Properties work this way, and so do methods: a function stored in a class is a descriptor whose __get__ returns a bound method, which is where self comes from. A fourth method, __set_name__, tells a descriptor the name it was given when its class was created. Below you will build such fields, sort descriptors into two kinds, and bind a method by hand.

A descriptor turns every use of a field into a method call

fields.py declares three fields on an Order, builds two orders, and then tries three bad values on the first:

class Field:
    """Keeps each instance's value in that instance, under the field's name."""

    def __set_name__(self, owner, name):     # called once, when the owning class is created
        self.name = name

    def __get__(self, obj, objtype=None):
        if obj is None:                      # read through the class: the descriptor itself
            return self
        try:
            return obj.__dict__[self.name]
        except KeyError:
            raise AttributeError(self.name) from None

    def __set__(self, obj, value):
        self.check(value)                    # refuse a bad value before storing it
        obj.__dict__[self.name] = value

    def check(self, value):
        pass


class Positive(Field):
    def check(self, value):
        if value <= 0:
            raise ValueError(f"{self.name} must be greater than 0, got {value}")


class MaxLength(Field):
    def __init__(self, limit):
        self.limit = limit

    def check(self, value):
        if len(value) > self.limit:
            raise ValueError(f"{self.name} has at most {self.limit} characters, got {len(value)}")


class Order:
    item = MaxLength(12)
    qty = Positive()
    price = Positive()

    def __init__(self, item, qty, price):
        self.item = item                     # each assignment runs a __set__
        self.qty = qty
        self.price = price


a = Order("chai", 3, 40)
b = Order("samosa", 2, 25)
print(a.qty * a.price, b.qty * b.price, vars(a))

for field, bad in (("qty", 0), ("price", -5), ("item", "masala dosa combo")):
    try:
        setattr(a, field, bad)
    except ValueError as e:
        print("ValueError:", e)

print(vars(a), type(Order.qty).__name__, Order.qty.name)

120 50 {'item': 'chai', 'qty': 3, 'price': 40}
ValueError: qty must be greater than 0, got 0
ValueError: price must be greater than 0, got -5
ValueError: item has at most 12 characters, got 17
{'item': 'chai', 'qty': 3, 'price': 40} Positive qty

• qty = Positive() and price = Positive() put two descriptor objects in the class. When Python creates Order, it calls __set_name__ on each with the name it was assigned to, so each object knows which field it is.
• self.qty = qty in __init__ is an assignment through an instance. The class holds a data descriptor of that name, so Python calls its __set__ instead of storing the value: the check runs in the constructor too, and a refused value leaves the old one in place.
• The values live in the instances. __set__ writes obj.__dict__["qty"] and __get__ reads it back, so each order keeps its own numbers. Using the field’s own name as the key is safe, because a data descriptor is consulted before the instance’s dictionary and the stored value can never hide it.
• Read through the class, Order.qty calls __get__ with None for the instance, and the field returns itself, which lets tests and tools inspect it.

Follow those calls one message at a time. The middle column is the qty field, the one Positive object that Order holds for that name:

The class holds one descriptor object per field, and every order keeps its own values in its own dictionary:

Each line of code that touches the field becomes one call on its descriptor, here named d:

Data or non-data: which one beats the instance

kinds.py asks each built-in kind which of the three methods its class defines:

import functools


class Point:
    __slots__ = ("x", "y")


class Sample:
    def method(self): ...

    @classmethod
    def build(cls): ...

    @staticmethod
    def helper(): ...

    @property
    def size(self): ...

    @functools.cached_property
    def summary(self): ...


objects = {
    "function": Sample.__dict__["method"],
    "classmethod": Sample.__dict__["build"],
    "staticmethod": Sample.__dict__["helper"],
    "property": Sample.__dict__["size"],
    "cached_property": Sample.__dict__["summary"],
    "slot": Point.__dict__["x"],
}
print(f"{'stored in the class':20} {'__get__':8} {'__set__':8} {'__delete__':11} kind")
for label, obj in objects.items():
    has = [hasattr(type(obj), m) for m in ("__get__", "__set__", "__delete__")]
    kind = "data" if has[1] or has[2] else "non-data"
    print(f"{label:20} {str(has[0]):8} {str(has[1]):8} {str(has[2]):11} {kind}")

stored in the class  __get__  __set__  __delete__  kind
function             True     False    False       non-data
classmethod          True     False    False       non-data
staticmethod         True     False    False       non-data
property             True     True     True        data
cached_property      True     False    False       non-data
slot                 True     True     True        data

A descriptor that defines __set__ or __delete__ is a data descriptordata descriptor: A descriptor that defines __set__ or __delete__, such as a property or a slot. Attribute lookup checks it before the instance's dictionary, so an instance attribute of the same name can never hide it., and it always beats an instance attribute of the same name. One that defines only __get__ is a non-data descriptornon-data descriptor: A descriptor that defines only __get__, such as a function or functools.cached_property. An entry of the same name in the instance's dictionary hides it, which is how an instance attribute can shadow a method., and an entry in the instance’s dictionary hides it. The run shows the split: properties and the slots of __slots__ are data descriptors, while functions, class methods, static methods and functools.cached_property are not. Each choice is deliberate. A property must run its setter even if something wrote its name into the instance, while cached_property stores its result in the instance on purpose, so that the next read finds the value there and never calls the descriptor again.

Step through two descriptors that differ in one method, and watch what the same entry in the instance’s dictionary does to each:

class Ten:
    def __get__(self, obj, owner):
        return 10


class Fixed(Ten):
    def __set__(self, obj, value):
        raise AttributeError("fixed")


class A:
    x = Ten()
    y = Fixed()


a = A()
own = vars(a)
own["x"] = 5
own["y"] = 5
x = a.x
y = a.y
a.x = 7

A function binds itself to the instance it is read through

Methods use the same machinery. Step through a function bound by hand; each variable shows its parameters, so you can see self being filled in:

import types


class Greeter:
    def __init__(self, name):
        self.name = name

    def greet(self, other):
        return f"{self.name} greets {other}"


g = Greeter("Asha")
f = vars(Greeter)["greet"]
has = hasattr(f, "__get__"), hasattr(f, "__set__")
bound = f.__get__(g, Greeter)
same = bound.__self__ is g
print(bound("Ravi"), "|", g.greet("Ravi"))
plain = f.__get__(None, Greeter)
by_hand = types.MethodType(f, g)
print(plain is f, by_hand("Meena"))
own = vars(g)
g.greet = lambda other: f"hello {other}"
print(g.greet("Ravi"), "|", Greeter.greet(g, "Ravi"))

Asha greets Ravi | Asha greets Ravi
True Asha greets Meena
hello Ravi | Asha greets Ravi

• The class stores a plain function, which has __get__ and no __set__: a non-data descriptor.
• f.__get__(g, Greeter) returns a bound method whose __self__ is g, which is exactly what reading g.greet produces, and types.MethodType builds the same object by hand.
• Read through the class, __get__ receives None and returns the function itself, which is why Greeter.greet(g, "Ravi") needs the instance passed in.
• Because functions are non-data descriptors, assigning g.greet gives that one instance a callable of its own, which hides the method.

The same four methods explain the rest of the machinery. property is a data descriptor that can be rebuilt in Python in about thirty lines. classmethod binds its function to the class, and staticmethod does not bind at all. A descriptor that keeps values on itself shares one value between every instance of its class. The value can also live in the instance under a private name, or in a weakref.WeakKeyDictionary that the descriptor holds. __set_name__ runs only while a class is being created, so a descriptor attached later never learns its name.

Practise: fields, kinds and bound methods

These questions cover the three ideas above, the one you have missed most first, and say why each answer is right or wrong as soon as you choose it.

property rebuilt and checked against the built-in, then the other kinds opened up Pro: opens with a Pro, Premium or Ultimate pass

Fields that hold up: the shared-value bug, four storage choices counted, and an exercise Pro: opens with a Pro, Premium or Ultimate pass

Interview questions

Key takeaways

• A descriptor is a class attribute with __get__, and optionally __set__ and __delete__; the attribute lookup calls these methods instead of returning the object.
• With __set__ or __delete__ it is a data descriptor and beats the instance’s dictionary; with only __get__ the instance wins.
• Functions are non-data descriptors whose __get__ returns bound methods; properties are data descriptors.
• __set_name__ gives each field its name while the class is created, so one field class serves many attributes.
• Keep per-instance values in the instance, never on the descriptor, which every instance shares.