Vector-length-agnostic code: RVV, SVE and SME

The NEON loop of the previous lesson names four 32-bit lanes in every instruction, so 40 elements take 10 passes on any core, even one whose vector unit could take 16 at once. Vector-length-agnosticvector-length-agnostic code: Vector code that never states how many lanes a vector holds: every pass asks the hardware how many elements it may handle and moves on by that many, so one binary uses the full width of any core. code never names the width: each pass asks the hardware how many elements it may handle and moves on by that many, so one binary fills a 128-bit unit or a 2,048-bit one.

Two instruction sets work this way. RISC-V’s vector extension, RVV, returns the length of each pass from vsetvli, and AArch64’s SVE marks the lanes still inside the array in a predicate. Apple silicon runs SVE instructions inside SME’s streaming mode, where the run below adds 40 elements in 3 passes of 16, 16 and 8 lanes.

A vector-length-agnostic loop asks for its lanes on every pass

A fixed-width loop states its vector lengthvector length: The width of a vector register on a given core, in bits or bytes. Fixed-width SIMD sets it in the instruction; SVE and the RISC-V V extension leave it to the implementation and let code read it at run time. in the code: .4s means four lanes, so it runs n / 4 vector passes and a tailloop tail: The leftover iterations of a vector loop that handle fewer elements than one vector pass covers, usually run as a short scalar loop. More in “Your first SIMD loop” finishes the rest. A vector-length-agnostic loop keeps the body but starts each pass by asking how many elements it may take, and ends it by moving the index on by the answer. The last pass is simply shorter, so no tail exists, and a wider core runs the same instructions fewer times.

Code that needs the number can read it: cntw or rdvl on SVE, rdsvl for streaming mode and the vlenb register on RVV.

SVE marks the lanes still inside the array

whilelo p0.s, x5, x3 writes a predicatepredicate register: A register that marks which vector lanes an instruction acts on. SVE has p0 to p15, with one bit for each byte of a vector register, so a 32-bit lane has four bits and its lowest one counts; AVX-512 has the mask registers k0 to k7, one bit per lane. in which lane j is active while i + j < n. Loads under p0 set the inactive lanes to zero and never fault on them, stores leave memory there unwritten, incw moves i on by one vector of 32-bit lanes and b.first loops while lane 0 is active.

// A vector-length-agnostic int32 add: SVE instructions in SME streaming mode (AArch64, Apple silicon).
//
// uint64_t vla_add(int32_t *c, const int32_t *a, const int32_t *b, uint64_t n, uint64_t *active)
//   c[i] = a[i] + b[i] for every i < n, nothing else in c is written;
//   active[k] = how many lanes pass k used; the return value is the number of passes.
//
// No lane count appears in the loop. whilelo marks the lanes whose index is still below n, and
// incw moves i on by however many 32-bit lanes one vector holds on the machine that runs it.
// smstart and smstop reset the vector registers, so the callee-saved d8-d15 (AAPCS64) wait on the
// stack, and the pass count is in a general register before smstop.

        .text
        .p2align 2
        .arch_extension sme
        .arch_extension sve

        .globl  _vla_add
_vla_add:
        stp     d8, d9, [sp, #-64]!
        stp     d10, d11, [sp, #16]
        stp     d12, d13, [sp, #32]
        stp     d14, d15, [sp, #48]
        smstart sm                          // streaming mode: SVE instructions are legal until smstop
        mov     x5, #0                      // i: the first element of this pass
        mov     x6, #0                      // passes so far
        whilelo p0.s, x5, x3                // lane j is active while i + j < n
        b.none  2f                          // no active lane (n = 0): no pass at all
1:      ld1w    {z0.s}, p0/z, [x1, x5, lsl #2]  // active lanes of a; inactive lanes become 0
        ld1w    {z1.s}, p0/z, [x2, x5, lsl #2]  // active lanes of b
        add     z0.s, z0.s, z1.s
        st1w    {z0.s}, p0, [x0, x5, lsl #2]    // only the active lanes reach memory
        cntp    x7, p0, p0.s                // the lanes this pass used
        str     x7, [x4, x6, lsl #3]        // active[passes] = x7
        add     x6, x6, #1
        incw    x5                          // i += the 32-bit lanes of one vector
        whilelo p0.s, x5, x3                // the predicate for the next pass
        b.first 1b                          // go on while lane 0 is active
2:      smstop  sm
        ldp     d14, d15, [sp, #48]
        ldp     d12, d13, [sp, #32]
        ldp     d10, d11, [sp, #16]
        ldp     d8, d9, [sp], #64
        mov     x0, x6
        ret

// uint64_t svl_bytes(void): the streaming vector length in bytes. rdsvl needs no streaming mode.
        .globl  _svl_bytes
_svl_bytes:
        rdsvl   x0, #1
        ret

// Runs vla_add (sve/vla_add.s) for six lengths and checks every element of c against a plain C loop,
// including the elements after n, which must keep the -1 the driver put there.
// Build: clang -O1 -o vla vla_main.c sve/vla_add.s
#include <stdint.h>
#include <stdio.h>

uint64_t vla_add(int32_t *c, const int32_t *a, const int32_t *b, uint64_t n, uint64_t *active);
uint64_t svl_bytes(void);

enum { MAX = 40 };

int main(void) {
    int32_t a[MAX], b[MAX], c[MAX];
    uint64_t active[MAX];
    const uint64_t sizes[] = {0, 1, 10, 16, 17, 40};
    for (int i = 0; i < MAX; i++) {
        a[i] = i + 1;
        b[i] = 100 * (i + 1);
    }
    uint64_t bytes = svl_bytes();
    printf("streaming vector: %llu bytes = %llu lanes of 32 bits\n", (unsigned long long)bytes,
           (unsigned long long)(bytes / 4));
    printf("  n  passes  lanes per pass  check\n");
    for (unsigned k = 0; k < sizeof sizes / sizeof sizes[0]; k++) {
        uint64_t n = sizes[k];
        for (int i = 0; i < MAX; i++) c[i] = -1;  // a store past n would show up here
        uint64_t passes = vla_add(c, a, b, n, active);
        int ok = 1;
        for (uint64_t i = 0; i < MAX; i++) ok &= c[i] == (i < n ? a[i] + b[i] : -1);
        char lanes[64] = "-";
        int len = 0;
        for (uint64_t p = 0; p < passes; p++)
            len += snprintf(lanes + len, sizeof lanes - (size_t)len, p ? ", %llu" : "%llu",
                            (unsigned long long)active[p]);
        printf("%3llu  %6llu  %-14s  %s\n", (unsigned long long)n, (unsigned long long)passes, lanes,
               ok ? "ok" : "WRONG");
    }
    return 0;
}

Apple silicon runs these instructions only in SME’s streaming modestreaming mode: A processor state that SME adds to AArch64, entered with smstart sm and left with smstop sm, in which SVE instructions run at a streaming vector length of its own. More in “SIMD concepts and CPU feature detection”, entered with smstart sm and left with smstop sm. The mode has rules of its own: the switch resets the vector registers, so the function saves d8 to d15 first, and NEON instructions are not allowed inside it on the host.

import pathlib
import sys

sys.path.insert(0, str(pathlib.Path(__file__).parent / "sessions"))
import session

# Build the streaming SVE add with its C driver and run it on Apple silicon (AArch64 macOS).
session.run("""
clang -O1 -o vla vla_main.c sve/vla_add.s
./vla
""")

$ clang -O1 -o vla vla_main.c sve/vla_add.s
$ ./vla
streaming vector: 64 bytes = 16 lanes of 32 bits
  n  passes  lanes per pass  check
  0       0  -               ok
  1       1  1               ok
 10       1  10              ok
 16       1  16              ok
 17       2  16, 1           ok
 40       3  16, 16, 8       ok

The streaming vectors are 64 bytes, 16 lanes of 32 bits. Lengths up to 16 take one pass, 17 takes two (16 and 1) and 40 takes three. The driver fills c with -1 first, so a store past n would fail the check.

RVV returns the pass length from vsetvli

vsetvli t0, a3, e32, m1, ta, ma gets a3, the elements still to do, asks for 32-bit elements and writes into t0 the length of this pass, vl. A pass holds at most VLMAX = LMUL × VLEN / SEW elements, where VLEN is the register width the core was built with. The assembler refuses V instructions unless it is told the target has V.

# A vector-length-agnostic int32 add for RV64 with the V extension (Linux, LP64D).
#
# void vla_add(int32_t *c, const int32_t *a, const int32_t *b, size_t n)
#   a0 = c, a1 = a, a2 = b, a3 = the elements still to add (the application vector length, AVL)
#
# Each pass starts with vsetvli, which returns in t0 how many elements this pass handles (vl).
# The code never states the register width: vl is at most VLMAX = LMUL * VLEN / SEW, where VLEN is
# the machine's vector register width. With n = 0, vl is 0, nothing is loaded or stored, and the
# loop ends after one pass that does nothing.

        .text
        .p2align 2
        .globl  vla_add
vla_add:
1:      vsetvli t0, a3, e32, m1, ta, ma     # 32-bit elements, one register per operand
        vle32.v v0, (a1)                    # vl elements of a
        vle32.v v1, (a2)                    # vl elements of b
        vadd.vv v0, v0, v1
        vse32.v v0, (a0)                    # vl sums into c
        sub     a3, a3, t0                  # elements left
        slli    t1, t0, 2                   # 4 bytes per element
        add     a1, a1, t1
        add     a2, a2, t1
        add     a0, a0, t1
        bnez    a3, 1b
        ret

import pathlib
import sys

sys.path.insert(0, str(pathlib.Path(__file__).parent / "sessions"))
import session

# RISC-V (RV64 with V, Linux): the assembler needs the extension named, then reads the loop back.
session.run("""
llvm-mc --triple=riscv64-linux-gnu -filetype=obj -o add.o riscv/vla_add.s 2>&1 | grep -m 1 -o "instruction requires the following: 'V'"
llvm-mc --triple=riscv64-linux-gnu -mattr=+v -filetype=obj -o add.o riscv/vla_add.s && echo "assembled"
llvm-objdump -d --mattr=+v add.o | sed -n '/<vla_add>:/,$p'
""")

$ llvm-mc --triple=riscv64-linux-gnu -filetype=obj -o add.o riscv/vla_add.s 2>&1 | grep -m 1 -o "instruction requires the following: 'V'"
instruction requires the following: 'V'
$ llvm-mc --triple=riscv64-linux-gnu -mattr=+v -filetype=obj -o add.o riscv/vla_add.s && echo "assembled"
assembled
$ llvm-objdump -d --mattr=+v add.o | sed -n '/<vla_add>:/,$p'
0000000000000000 <vla_add>:
       0: 0d06f2d7     	vsetvli	t0, a3, e32, m1, ta, ma
       4: 0205e007     	vle32.v	v0, (a1)
       8: 02066087     	vle32.v	v1, (a2)
       c: 02008057     	vadd.vv	v0, v0, v1
      10: 02056027     	vse32.v	v0, (a0)
      14: 405686b3     	sub	a3, a3, t0
      18: 00229313     	slli	t1, t0, 0x2
      1c: 006585b3     	add	a1, a1, t1
      20: 00660633     	add	a2, a2, t1
      24: 00650533     	add	a0, a0, t1
      28: fc069ce3     	bnez	a3, 0x0 <vla_add>
      2c: 00008067     	ret

The specification sets vl to AVL, the elements left, when AVL ≤ VLMAX, and to VLMAX when AVL ≥ 2 × VLMAX. In between, a core may pick any vl from half of AVL, rounded up, to VLMAX. The model below applies that rule to 40 elements at four widths; it models the specification and runs no RISC-V code.

"""How many elements each vsetvli pass gets, for 40 elements of 32 bits at four vector widths.

A model of the rule in the RISC-V V 1.0 specification ("Constraints on Setting vl"), not a run on RISC-V
hardware. With AVL elements left and VLMAX = LMUL * VLEN / SEW:
  vl = AVL                        when AVL <= VLMAX
  ceil(AVL / 2) <= vl <= VLMAX    when VLMAX < AVL < 2 * VLMAX  (the implementation chooses)
  vl = VLMAX                      when AVL >= 2 * VLMAX
The model prints both ends of the choice and then tries every legal choice to count the passes.
"""

import math

N = 40  # elements to process
SEW = 32  # bits per element
LMUL = 1  # registers per operand
VLENS = (128, 256, 512, 1024)  # register widths an implementation may have


def legal_vls(avl: int, vlmax: int) -> range:
    """Every vl the rule allows for this pass."""
    if avl <= vlmax:
        return range(avl, avl + 1)
    if avl < 2 * vlmax:
        return range(math.ceil(avl / 2), vlmax + 1)
    return range(vlmax, vlmax + 1)


def passes(avl: int, vlmax: int, pick) -> list[int]:
    """The vl of every pass when the implementation picks with `pick` (min or max) each time."""
    out = []
    while avl > 0:
        vl = pick(legal_vls(avl, vlmax))
        out.append(vl)
        avl -= vl
    return out


def every_count(avl: int, vlmax: int) -> set[int]:
    """The pass counts over every sequence of legal choices."""
    if avl == 0:
        return {0}
    return {1 + rest for vl in legal_vls(avl, vlmax) for rest in every_count(avl - vl, vlmax)}


def show(vls: list[int]) -> str:
    return ", ".join(str(vl) for vl in vls)


print(f"n = {N} elements, SEW = {SEW}, LMUL = {LMUL}")
for vlen in VLENS:
    vlmax = LMUL * vlen // SEW
    widest = passes(N, vlmax, max)
    even = passes(N, vlmax, min)
    counts = every_count(N, vlmax)
    print(f"VLEN {vlen:4}: VLMAX {vlmax:2}")
    print(f"  largest vl first: {show(widest)}")
    print(f"  smallest legal vl: {show(even)}")
    print(f"  passes with any legal choice: {sorted(counts)} (ceil(n / VLMAX) = {math.ceil(N / vlmax)})")

n = 40 elements, SEW = 32, LMUL = 1
VLEN  128: VLMAX  4
  largest vl first: 4, 4, 4, 4, 4, 4, 4, 4, 4, 4
  smallest legal vl: 4, 4, 4, 4, 4, 4, 4, 4, 4, 4
  passes with any legal choice: [10] (ceil(n / VLMAX) = 10)
VLEN  256: VLMAX  8
  largest vl first: 8, 8, 8, 8, 8
  smallest legal vl: 8, 8, 8, 8, 8
  passes with any legal choice: [5] (ceil(n / VLMAX) = 5)
VLEN  512: VLMAX 16
  largest vl first: 16, 16, 8
  smallest legal vl: 16, 12, 12
  passes with any legal choice: [3] (ceil(n / VLMAX) = 3)
VLEN 1024: VLMAX 32
  largest vl first: 32, 8
  smallest legal vl: 20, 20
  passes with any legal choice: [2] (ceil(n / VLMAX) = 2)

The pass count stays ceil(40 / VLMAX) whatever the core picks: 10, 5, 3 and 2. LMUL above 1 groups registersregister grouping: Treating several consecutive vector registers as one operand. In the RISC-V V extension, LMUL sets how many registers form a group, so one instruction handles more elements in each pass. to lengthen each pass. The tail and mask policiestail and mask policy: The rule for vector elements an instruction does not compute: those past the vector length (the tail) and those its mask turns off. Undisturbed keeps their old values; agnostic lets the hardware keep them or fill them with all 1s. ta, ma let the hardware fill elements past vl, and masked-off ones, with all 1s. Linux documents how it saves and controls this vector state for each thread.

Practise

Work on copies of the lesson’s files and write each prediction down before you run.

1. In vla_main.c, add 33 and 48 to sizes, raise MAX to 48, and predict the passes and the lanes of each.
2. In vl_model.py, set N = 100 and predict the passes at VLEN 128 and at VLEN 512.
3. Set LMUL = 2 in vl_model.py and predict VLMAX and the passes at VLEN 256.

SME streaming mode on Apple silicon, rule by rule Pro: opens with a Pro, Premium or Ultimate pass

RVV in depth: vtype fields, LMUL and loops checked at four widths Pro: opens with a Pro, Premium or Ultimate pass

Interview questions

Why does a vector-length-agnostic loop need no scalar tail?

Because every pass asks the hardware how many elements it may handle, and the last pass is simply told fewer. On RVV, vsetvli returns vl = AVL once the elements left fit in one pass, so the last pass handles exactly what remains. On SVE, whilelo marks only the lanes whose index is below n: loads give 0 in the other lanes and stores skip them. In the recorded run, 40 elements on 16 lanes took passes of 16, 16 and 8 with no second loop.

Three longer questions follow the same loops into code that a team has to ship:

Key takeaways

• A vector-length-agnostic loop asks the hardware for the length of each pass and moves on by that many, so one binary fits any vector width.
• SVE’s whilelo builds a predicate of the lanes still inside the array, and inactive lanes never fault on a load and are never stored.
• Apple silicon runs SVE instructions in SME streaming mode, where the recorded loop added 40 int32 values in passes of 16, 16 and 8 lanes.
• RVV’s vsetvli returns vl, at most LMUL × VLEN / SEW, and any legal choice of vl keeps the pass count at ceil(n / VLMAX).