How GPUs work: SIMT, memory and tiling

A 1920 × 1080 frame has 2,073,600 pixels, and at 60 frames a second a GPU has 16.7 ms to shade every one of them, often several times over once passes and overlapping triangles are counted. It manages by running the same small program on thousands of pixels at once, in groups that move in step, and by switching to another group whenever one has to wait for memory.

The sections below run a model of one such group through a branch that splits it, measure how many groups a core needs to hide a slow read, decide whether a pass is limited by arithmetic or by memory bandwidth, and follow one pass through a tile-based GPU, where the load and store operations of a render pass decide most of its memory traffic. Every number comes from a program on this page.

One instruction drives a whole group of threads

A fragment shader runs once for every pixel a triangle covers: 2,073,600 times for a full-screen pass at 1920 × 1080. A CPU core fetches, decodes and schedules the instructions of each thread on its own. A GPU does that bookkeeping once for a whole group of threads and then executes the instruction on all of them at the same moment, each on its own data. NVIDIA’s CUDA Programming Guide calls such a group a warp, 32 threads that move through the program in lockstep, and the model single instruction, multiple threads: SIMTSIMT (single instruction, multiple threads): Running one instruction at a time for a whole group of threads in lockstep, each thread on its own data. Threads may still take different paths through a branch, at the cost of running every path.. WGSL, the shading language of WebGPU, calls the group a subgroupsubgroup: WGSL's name for a group of invocations that a GPU runs together in lockstep, what NVIDIA calls a warp. Its size is a power of two from 4 to 128 and depends on the device.. Its size is a power of two from 4 to 128 and depends on the device, and WebGPU reports the range as the adapter’s subgroupMinSize and subgroupMaxSize.

The figure shows one group of eight laneslane: One thread's place in a group that runs in lockstep: the part of the hardware that runs that thread's copy of each instruction, on that thread's own data., the size this lesson’s model uses so that every lane fits on one line. The unit fetches and decodes load x once, and each lane receives its own value:

Sharing the fetch and the decode is what lets a GPU spend its chip on arithmetic: a group of 32 does 32 additions for the bookkeeping of one, and a full-screen pass at 1920 × 1080 needs only 2,073,600 ÷ 32 = 64,800 groups. Fatahalian and Houston’s survey of GPU design puts this idea next to the one in the section after next, many groups kept in flight. The price of sharing is a rule: every lane of a group runs the same instruction at the same time, so what one lane needs, the whole group waits for.

A branch that splits a group runs both sides

Give the eight lanes the values of the figure, 3, 9, 4, 7, 1, 8, 6 and 2, and the shader if (x > 4) { light } else { shadow }, where lighting takes four instructions and shadow three. Lanes 1, 3, 5 and 6 hold values above 4 and the other four do not, so no single path suits the group. It runs both sides, one after the other, with an execution maskexecution mask: One bit per lane of a group that says whether the lane runs the current instruction. Masked lanes keep their values and do nothing while the group runs a path they did not take. that switches off the lanes not on the side being run: NVIDIA’s guide says the threads that do not follow a branch are masked off while the others execute. Step through it:

The program behind the figure is a model of one group of eight lanes. It runs the shader with every lane lit, then with the split of the figure, and then a loop in which each lane marches until it hits a surface, one lane needing 9 steps where the others need 2 or 3:

// A model of one SIMT group of 8 lanes. The group fetches each instruction once and runs it on every lane whose
// mask bit is set; masked lanes ('.') sit idle for that instruction. Each issued instruction costs one step of the
// group, whatever the number of active lanes. The shader is the lesson's:
//   load x; if (x > 4) { light 1..4 } else { shadow 1..3 }; store colour
// and a loop that marches each lane until it hits a surface.
const LANES = 8;

function machine() {
  let issued = 0;
  let used = 0;
  const lines = [];
  return {
    issue(name, mask) {
      const active = mask.filter(Boolean).length;
      if (active === 0) return; // no lane needs it: the group skips it
      issued += 1;
      used += active;
      const lanes = mask.map((on) => (on ? '#' : '.')).join('');
      lines.push(`  ${String(issued).padStart(2)}  ${name.padEnd(13)} ${lanes}  ${active}`);
    },
    report() {
      const slots = issued * LANES;
      const pct = ((100 * used) / slots).toFixed(1);
      return [...lines, `  issued ${issued}, lane slots used ${used} of ${slots}: ${pct} %`].join('\n');
    },
  };
}

function shade(xs) {
  const m = machine();
  const all = xs.map(() => true);
  const lit = xs.map((x) => x > 4);
  const dark = lit.map((t) => !t);
  m.issue('load x', all);
  m.issue('test x > 4', all);
  for (let i = 1; i <= 4; i++) m.issue(`light ${i}`, lit);
  for (let i = 1; i <= 3; i++) m.issue(`shadow ${i}`, dark);
  m.issue('store colour', all);
  return m.report();
}

function march(steps) {
  const m = machine();
  const done = steps.map(() => 0);
  for (let it = 1; done.some((d, lane) => d < steps[lane]); it++) {
    const active = steps.map((s, lane) => done[lane] < s);
    m.issue(`step ${it}`, active);
    m.issue(`test hit ${it}`, active);
    active.forEach((on, lane) => { if (on) done[lane] += 1; });
  }
  return m.report();
}

const runs = [
  ['Run 1: every lane is lit', [9, 7, 8, 6, 5, 12, 10, 11]],
  ['Run 2: the test splits the group', [3, 9, 4, 7, 1, 8, 6, 2]],
];
for (const [title, xs] of runs) {
  console.log(`${title} (x = ${xs.join(' ')})`);
  console.log(`  #   instruction   lanes     active`);
  console.log(shade(xs));
  console.log('');
}

const steps = [2, 2, 3, 2, 2, 2, 2, 9];
console.log(`Run 3: a loop, lanes need ${steps.join(' ')} steps`);
console.log(`  #   instruction   lanes     active`);
console.log(march(steps));

Run 1: every lane is lit (x = 9 7 8 6 5 12 10 11)
  #   instruction   lanes     active
   1  load x        ########  8
   2  test x > 4    ########  8
   3  light 1       ########  8
   4  light 2       ########  8
   5  light 3       ########  8
   6  light 4       ########  8
   7  store colour  ########  8
  issued 7, lane slots used 56 of 56: 100.0 %

Run 2: the test splits the group (x = 3 9 4 7 1 8 6 2)
  #   instruction   lanes     active
   1  load x        ########  8
   2  test x > 4    ########  8
   3  light 1       .#.#.##.  4
   4  light 2       .#.#.##.  4
   5  light 3       .#.#.##.  4
   6  light 4       .#.#.##.  4
   7  shadow 1      #.#.#..#  4
   8  shadow 2      #.#.#..#  4
   9  shadow 3      #.#.#..#  4
  10  store colour  ########  8
  issued 10, lane slots used 52 of 80: 65.0 %

Run 3: a loop, lanes need 2 2 3 2 2 2 2 9 steps
  #   instruction   lanes     active
   1  step 1        ########  8
   2  test hit 1    ########  8
   3  step 2        ########  8
   4  test hit 2    ########  8
   5  step 3        ..#....#  2
   6  test hit 3    ..#....#  2
   7  step 4        .......#  1
   8  test hit 4    .......#  1
   9  step 5        .......#  1
  10  test hit 5    .......#  1
  11  step 6        .......#  1
  12  test hit 6    .......#  1
  13  step 7        .......#  1
  14  test hit 7    .......#  1
  15  step 8        .......#  1
  16  test hit 8    .......#  1
  17  step 9        .......#  1
  18  test hit 9    .......#  1
  issued 18, lane slots used 48 of 144: 33.3 %

With every lane lit, the group issued 7 instructions and every lane slot did work. The split cost 10 instructions, 4 lit and 3 dark, and 28 of its 80 lane slots idled: 65.0 % were used. The loop is worse. The group keeps issuing until its slowest lane finishes, so it ran 18 instructions with only 33.3 % of the lane slots busy: one lane’s 9 steps cost the whole group 9 steps. This is divergencedivergence: Lanes of one group taking different paths through a branch or a loop. The group then runs every path any lane takes, one after the other, with the other lanes masked.. Press Edit, make every value of run 2 larger than 4, and run it again: the group then issues only the lit side. Across a screen, what matters is whether the pixels that share a group agree. A shadow with one straight edge splits only the groups along that edge, while scattered shadow pixels split nearly every group.

The slip that follows from forgetting it is easy to make when reading shader code line by line:

Many groups in flight hide the wait for memory

Now give the shader a texture read. Suppose, as this lesson’s model does, that a group runs 20 instructions of arithmetic and then reads a texel that arrives 400 cycles later. A core that ran only that group would issue for 20 cycles and wait for 400, busy 4.8 % of the time. A GPU core keeps many groups resident instead, and whenever the group it is running has to wait, it issues from another group that is ready. NVIDIA’s Best Practices Guide calls executing other warps while one is stalled the only way to hide latencies. Step through the first 420 cycles of a core with two groups:

Two groups only double the busy share, to 9.5 %. The program runs the same core with more groups, and then with 40 instructions of arithmetic per read:

// A model of one GPU core that can issue one instruction a cycle. Every group runs the same loop: WORK instructions
// of arithmetic, then a read from memory whose answer arrives LATENCY cycles later; the group waits for it, then
// starts the next round. The core keeps issuing from one group until that group waits, then moves on to the next
// group that is ready. The numbers are the model's: change WORK, LATENCY and the group counts, and run it again.
const LATENCY = 400;
const CYCLES = 42000;

function run(groups, work) {
  const left = Array(groups).fill(work); // instructions left before the group's next read
  const readyAt = Array(groups).fill(0); // the cycle from which the group may issue again
  let current = 0;
  let busy = 0;
  for (let t = 0; t < CYCLES; t++) {
    let g = -1;
    for (let k = 0; k < groups; k++) {
      const candidate = (current + k) % groups;
      if (readyAt[candidate] <= t) {
        g = candidate;
        break;
      }
    }
    if (g < 0) continue; // every group is waiting for memory: the core idles
    current = g;
    busy += 1;
    left[g] -= 1;
    if (left[g] === 0) {
      left[g] = work;
      readyAt[g] = t + 1 + LATENCY; // the read leaves now; its data is back LATENCY cycles later
      current = (g + 1) % groups;
    }
  }
  return busy / CYCLES;
}

for (const work of [20, 40]) {
  const needed = Math.ceil((work + LATENCY) / work);
  console.log(`${work} instructions per read, memory answers in ${LATENCY} cycles`);
  console.log(`  groups needed to keep issuing: (${work} + ${LATENCY}) / ${work} = ${needed}`);
  console.log('  groups  core busy');
  for (const groups of [...new Set([1, 2, 4, 8, 16, needed, 32])].sort((a, b) => a - b)) {
    console.log(`  ${String(groups).padStart(6)}  ${(100 * run(groups, work)).toFixed(1).padStart(6)} %`);
  }
  console.log('');
}

20 instructions per read, memory answers in 400 cycles
  groups needed to keep issuing: (20 + 400) / 20 = 21
  groups  core busy
       1     4.8 %
       2     9.5 %
       4    19.0 %
       8    38.1 %
      16    76.2 %
      21   100.0 %
      32   100.0 %

40 instructions per read, memory answers in 400 cycles
  groups needed to keep issuing: (40 + 400) / 40 = 11
  groups  core busy
       1     9.1 %
       2    18.3 %
       4    36.6 %
       8    72.9 %
      11   100.0 %
      16   100.0 %
      32   100.0 %

The busy share grows in step with the groups, 20 cycles of issue for each 420-cycle round, until (20 + 400) ÷ 20 = 21 groups keep the core issuing on every cycle; groups beyond that only wait their turn. Doubling the arithmetic per read to 40 instructions cuts the groups needed to 11, so work between reads hides latency too. This is latency hidinglatency hiding: Keeping a core busy while some groups wait for memory, by issuing instructions from other resident groups that are ready to run., and on a real GPU it has a limit: a core can hold only so many groups, because each one keeps its own registers. The guide notes that blocks which use many registers leave room for fewer blocks on a core. The share of its maximum that a core actually holds is its occupancyoccupancy: The number of groups resident on a GPU core as a share of the most it can hold. The registers and workgroup memory each group uses limit it.. Low occupancy leaves memory latency uncovered, but once the core issues on every cycle, more occupancy brings nothing, as the 32-group rows show.

Operations per byte decide what limits a pass

Hiding latency keeps the core issuing, but every read still moves bytes, and memory delivers only so many a second: its bandwidthmemory bandwidth: The bytes a second a processor can move to and from its memory, such as 100 GB a second.. The bytes add up quickly. A 1920 × 1080 target in rgba16float, 8 bytes a pixel, is 16,588,800 bytes, so a frame that touches it 5 times at 60 frames a second moves about 4.98 GB a second for that one target.

Whether a pass finishes in time depends on two numbers, the operations it does and the bytes it moves. Their ratio is its arithmetic intensityarithmetic intensity: The operations a computation does per byte it moves to or from memory. Compared with a processor's balance point, it says whether memory or arithmetic limits the computation., in operations per byte. The roofline modelroofline model: A bound on how fast a computation can run: the lower of the processor's peak rate of operations and its memory bandwidth times the computation's arithmetic intensity. of Williams, Waterman and Patterson turns it into a bound: a pass runs no faster than the processor’s peak rate of operations, and no faster than bandwidth times intensity, whichever is lower. The two limits meet at the balance pointbalance point: The arithmetic intensity at which memory and arithmetic take equally long on a processor: its peak operations a second divided by its bytes a second of bandwidth., peak rate ÷ bandwidth. The program works out the bytes of six render targets, then tests four passes on an invented GPU with 2,000 billion operations and 100 GB a second, a balance point of 20 operations per byte:

// Bandwidth arithmetic for render targets, then the roofline test for four passes on an invented GPU.
// MB and GB here are decimal: 1 MB = 1,000,000 bytes, 1 GB = 1,000,000,000 bytes.
const BYTES_PER_PIXEL = { rgba8unorm: 4, rgba16float: 8, rgba32float: 16 };
const fmt = (x, digits) => {
  const [whole, part] = x.toFixed(digits).split('.');
  const grouped = whole.replace(/\B(?=(\d{3})+(?!\d))/g, ',');
  return part ? `${grouped}.${part}` : grouped;
};

console.log('One render target, and a frame that touches it 5 times at 60 frames a second');
console.log('  size         format          one target   per second');
for (const [w, h] of [[1920, 1080], [3840, 2160]]) {
  for (const format of ['rgba8unorm', 'rgba16float', 'rgba32float']) {
    const bytes = w * h * BYTES_PER_PIXEL[format];
    const perSecond = bytes * 5 * 60;
    console.log(`  ${`${w} x ${h}`.padEnd(12)} ${format.padEnd(15)} ${fmt(bytes / 1e6, 1).padStart(6)} MB  ${fmt(perSecond / 1e9, 2).padStart(6)} GB/s`);
  }
}

// An invented GPU: 2,000 billion operations a second and 100 GB/s of memory bandwidth.
const GPU = { opsPerSecond: 2000e9, bytesPerSecond: 100e9 };
const balance = GPU.opsPerSecond / GPU.bytesPerSecond;
const PIXELS = 1920 * 1080;
const passes = [
  { name: 'tone map', items: PIXELS, ops: 12, bytes: 8 + 4 }, // reads rgba16float, writes rgba8unorm
  { name: 'blur, 9 taps', items: PIXELS, ops: 60, bytes: 4 + 4 }, // neighbours come from the cache
  { name: 'procedural sky', items: PIXELS, ops: 400, bytes: 4 }, // computes every pixel, reads nothing
  { name: 'particle update', items: 1_000_000, ops: 30, bytes: 32 + 32 }, // reads and writes 32 bytes each
];

console.log('');
console.log(`Invented GPU: ${fmt(GPU.opsPerSecond / 1e9, 0)} billion operations/s, ${fmt(GPU.bytesPerSecond / 1e9, 0)} GB/s`);
console.log(`  balance point: ${fmt(GPU.opsPerSecond / 1e9, 0)} / ${fmt(GPU.bytesPerSecond / 1e9, 0)} = ${balance} operations per byte`);
console.log('  pass             ops/byte  limited by  time     of peak');
for (const p of passes) {
  const intensity = p.ops / p.bytes;
  const opsTime = (p.items * p.ops) / GPU.opsPerSecond;
  const bytesTime = (p.items * p.bytes) / GPU.bytesPerSecond;
  const limit = bytesTime > opsTime ? 'memory' : 'compute';
  const seconds = Math.max(opsTime, bytesTime);
  const share = (100 * (p.items * p.ops)) / seconds / GPU.opsPerSecond;
  console.log(`  ${p.name.padEnd(16)} ${fmt(intensity, 2).padStart(8)}  ${limit.padEnd(10)}  ${fmt(seconds * 1000, 3)} ms  ${fmt(share, 0).padStart(4)} %`);
}

One render target, and a frame that touches it 5 times at 60 frames a second
  size         format          one target   per second
  1920 x 1080  rgba8unorm         8.3 MB    2.49 GB/s
  1920 x 1080  rgba16float       16.6 MB    4.98 GB/s
  1920 x 1080  rgba32float       33.2 MB    9.95 GB/s
  3840 x 2160  rgba8unorm        33.2 MB    9.95 GB/s
  3840 x 2160  rgba16float       66.4 MB   19.91 GB/s
  3840 x 2160  rgba32float      132.7 MB   39.81 GB/s

Invented GPU: 2,000 billion operations/s, 100 GB/s
  balance point: 2,000 / 100 = 20 operations per byte
  pass             ops/byte  limited by  time     of peak
  tone map             1.00  memory      0.249 ms     5 %
  blur, 9 taps         7.50  memory      0.166 ms    38 %
  procedural sky     100.00  compute     0.415 ms   100 %
  particle update      0.47  memory      0.640 ms     2 %

The figure puts the four passes on that GPU’s roofline. Passes on the rising line are limited by memory, and only the flat part belongs to arithmetic:

The tone map does 12 operations per pixel and moves 12 bytes, an intensity of 1, so it can use only 5 % of the GPU’s arithmetic and its 0.249 ms are memory time. Even the 9-tap blur, at 7.5, stays below the balance point. Only the procedural sky, 400 operations for the 4 bytes it writes, reaches the flat part and uses the whole 2,000. For a pass below the balance point faster arithmetic changes nothing; fewer bytes do, through a smaller format, fewer passes over the same target, or two passes fused into one.

Tile-based GPUs keep each tile’s pixels on chip

A GPU can draw a pass in two orders. An immediate-modeimmediate-mode rendering: Rendering that shades triangles in the order they are drawn, reading and writing each fragment's colour and depth in memory, helped by caches. GPU shades triangles in the order they arrive and reads and writes the colour and depth of each fragment in memory, helped by caches. A tile-basedtile-based rendering: Rendering that first sorts a pass's triangles by the screen tiles they touch, then shades one tile at a time in on-chip tile memory and writes each finished tile out once. GPU first sorts the pass’s triangles by the screen tiles they touch. It then renders one tile at a time in tile memorytile memory: Fast memory on the GPU chip that holds one tile's colour, depth and other attachments while a tile-based GPU renders that tile. on the chip and writes each finished tile out once. Apple’s GPUs work this way, and Apple’s guide describes tile memory as fast, temporary storage on the GPU itself, with much higher bandwidth and lower latency than device memory, and much less energy per access. Every triangle that overlaps a tile, and every depth test and blend on it, then costs no memory traffic. Step through one pass:

What still reaches memory is decided at the two ends of each tile: whether the tile starts from the attachment’s old contents, read from memory, and whether its result is written back. The figure’s pass cleared both attachments and kept only the colour, so its whole attachment traffic was one write of the colour attachment, however many triangles overlapped.

Load and store operations set the traffic of a pass

WebGPU states those two ends for each attachment of a render pass. The loadOploadOp: What a render pass does with an attachment at its start: clear sets it to a clear value, load keeps the contents it held, which a tile-based GPU must read from memory. is clear, which starts the attachment at a clear value, or load, which keeps what it held. The storeOpstoreOp: What a render pass does with an attachment at its end: store writes the result to memory, discard drops it and leaves the attachment's contents undefined. is store, which keeps the result, or discard, which drops it; WebGPU then treats the attachment as cleared to zero, so a later read finds zeros, never leftover data. MDN recommends clear wherever the old contents do not matter, for better performance on some devices such as phones. The program counts the traffic of four choices for a colour and a depth attachment at 2220 × 1080, 4 bytes a pixel and 60 frames a second, the frame of the Khronos Group’s sample on render pass attachments:

// Memory traffic of one render pass on a tile-based GPU, from its load and store operations alone. A tile starts in
// on-chip memory: loadOp 'load' reads the attachment's old pixels from memory, 'clear' fills the tile on chip.
// When the tile is done, storeOp 'store' writes it to memory, 'discard' throws it away. Everything drawn in between
// stays on chip. The frame is the size the Khronos render-pass sample uses: 2220 x 1080, 4 bytes a pixel, 60 a second.
const W = 2220;
const H = 1080;
const BYTES = 4;
const FPS = 60;
const MiB = 1024 * 1024;

const attachment = W * H * BYTES; // one full read or write of one attachment
const perSecond = (bytes) => ((bytes * FPS) / MiB).toFixed(1);
const grouped = (n) => String(n).replace(/\B(?=(\d{3})+(?!\d))/g, ',');

function traffic(colour, depth) {
  let reads = 0;
  let writes = 0;
  for (const [loadOp, storeOp] of [colour, depth]) {
    if (loadOp === 'load') reads += attachment;
    if (storeOp === 'store') writes += attachment;
  }
  return { reads, writes };
}

console.log(`Frame ${W} x ${H}, ${BYTES} bytes a pixel per attachment, ${FPS} frames a second`);
console.log(`One full read or write of one attachment: ${grouped(attachment)} bytes, ${perSecond(attachment)} MiB/s`);
console.log('');
console.log('  colour        depth           reads MiB/s  writes MiB/s');
const cases = [
  [['load', 'store'], ['load', 'store']],
  [['clear', 'store'], ['load', 'store']],
  [['load', 'store'], ['clear', 'discard']],
  [['clear', 'store'], ['clear', 'discard']],
];
for (const [colour, depth] of cases) {
  const t = traffic(colour, depth);
  console.log(`  ${colour.join(', ').padEnd(13)} ${depth.join(', ').padEnd(15)} ${perSecond(t.reads).padStart(11)}  ${perSecond(t.writes).padStart(12)}`);
}
console.log('');
console.log(`A depth attachment that is cleared and discarded never leaves the chip:`);
console.log(`its ${grouped(attachment)} bytes need no memory if the GPU keeps it in tile memory.`);

Frame 2220 x 1080, 4 bytes a pixel per attachment, 60 frames a second
One full read or write of one attachment: 9,590,400 bytes, 548.8 MiB/s

  colour        depth           reads MiB/s  writes MiB/s
  load, store   load, store          1097.5        1097.5
  clear, store  load, store           548.8        1097.5
  load, store   clear, discard        548.8         548.8
  clear, store  clear, discard          0.0         548.8

A depth attachment that is cleared and discarded never leaves the chip:
its 9,590,400 bytes need no memory if the GPU keeps it in tile memory.

One full read or write of one attachment costs 548.8 MiB a second. Loading and storing both attachments moves 1,097.5 MiB a second each way; clearing the colour and clearing and discarding the depth removes three of those four transfers and leaves one write. The Khronos sample’s own measurements show the same effect: clearing instead of loading cut the bytes read by 600.2 MiB a second, and not storing depth cut the bytes written by 554.8 MiB a second, close to the model’s 548.8. Choose the pair for an attachment of your own:

A depth buffer that one pass uses and nothing reads afterwards is the usual case for clear and discard. WebGPU’s TRANSIENT_ATTACHMENT texture usage tells the GPU exactly that: such a texture may be used only with clear and discard, and Chrome’s notes say it can then stay in on-chip memory, possibly without any video memory at all. On any GPU these operations say what a pass needs, and an implementation can skip only the work it is told is unneeded.

Practise: what limits a pass

The exercise turns the lesson’s arithmetic into three functions: the bytes a second of a render target, what limits a pass on a given GPU, and the groups needed to hide a read. Its sample tests use the passes and the model above:

/** Bytes a second when a frame touches a width x height render target `touches` times, `fps` frames a second. */
export function trafficPerSecond({ width, height, bytesPerPixel, touches, fps }) {
  return width * height * bytesPerPixel * fps; // forgets the touches
}

/** What limits a pass: { intensity, limit: 'memory' | 'compute', seconds }. */
export function limitedBy({ ops, bytes }, { opsPerSecond, bytesPerSecond }) {
  const intensity = ops / bytes;
  const limit = intensity < 1 ? 'memory' : 'compute'; // compares with 1, not with the GPU's balance point
  const seconds = ops / opsPerSecond + bytes / bytesPerSecond; // adds the two times
  return { intensity, limit, seconds };
}

/** Groups needed in flight so that a core never waits, when each issues `work` and then waits `latency` cycles. */
export function groupsToHide(work, latency) {
  return Math.floor(latency / work); // rounds down and forgets the group's own work
}

import { test, assert } from 'mysmartcopilot:test';
import { groupsToHide, limitedBy, trafficPerSecond } from './limits.mjs';

const GPU = { opsPerSecond: 2000e9, bytesPerSecond: 100e9 };
const PIXELS = 1920 * 1080;
const round = (x) => Math.round(x * 1e12) / 1e12;

test('traffic: 1920 x 1080 rgba16float touched 5 times at 60 frames a second', () => {
  assert.equal(trafficPerSecond({ width: 1920, height: 1080, bytesPerPixel: 8, touches: 5, fps: 60 }), 4_976_640_000);
  assert.equal(trafficPerSecond({ width: 2220, height: 1080, bytesPerPixel: 4, touches: 1, fps: 60 }), 575_424_000);
});

test('the tone map pass is limited by memory', () => {
  const r = limitedBy({ ops: PIXELS * 12, bytes: PIXELS * 12 }, GPU);
  assert.equal(r.intensity, 1);
  assert.equal(r.limit, 'memory');
  assert.equal(round(r.seconds), round((PIXELS * 12) / 100e9), 'the bytes take 0.249 ms; the operations 0.012 ms');
});

test('the procedural sky is limited by compute', () => {
  const r = limitedBy({ ops: PIXELS * 400, bytes: PIXELS * 4 }, GPU);
  assert.equal(r.intensity, 100);
  assert.equal(r.limit, 'compute');
  assert.equal(round(r.seconds), round((PIXELS * 400) / 2000e9));
});

test('a blur at 7.5 operations a byte is still limited by memory, below the balance point of 20', () => {
  const r = limitedBy({ ops: PIXELS * 60, bytes: PIXELS * 8 }, GPU);
  assert.equal(r.limit, 'memory');
  assert.equal(r.intensity, 7.5);
});

test('no bytes at all, and a tie at the balance point', () => {
  const none = limitedBy({ ops: 1e9, bytes: 0 }, GPU);
  assert.equal(none.intensity, Infinity);
  assert.equal(none.limit, 'compute');
  assert.equal(round(none.seconds), 0.0005);
  const tie = limitedBy({ ops: 2000e6, bytes: 100e6 }, GPU);
  assert.equal(tie.intensity, 20);
  assert.equal(tie.limit, 'compute', 'equal times are a tie, and a tie is compute');
  assert.equal(round(tie.seconds), 0.001);
});

test('groups needed to hide a read', () => {
  assert.equal(groupsToHide(20, 400), 21);
  assert.equal(groupsToHide(40, 400), 11);
  assert.equal(groupsToHide(30, 400), 15, '430 / 30 is 14.33: round up');
  assert.equal(groupsToHide(400, 400), 2);
  assert.equal(groupsToHide(5, 0), 1, 'with no wait, one group keeps the core busy');
});

Then practise the whole lesson: the questions start with the ideas you know least, and each wrong answer explains what it misses.

Interview questions

Warm-up (fresher to 2-5 years): What is warp divergence, and why does it cost time?

A GPU runs threads in groups, warps of 32 on NVIDIA hardware, that execute one instruction at a time in lockstep. When the threads of a group reach a branch and some take one side while the rest take the other, the group cannot run both sides at once. It runs one side with the threads that took the other side masked off, then the other side with the first set masked, and the threads join again after the branch. That is divergence. The group’s time is the sum of both sides, so a branch that a few scattered pixels take can make many groups pay for the expensive path. Loops diverge too: a group keeps running until its slowest thread finishes. Branches on values every thread of a group shares, such as a material flag, do not diverge.

Two questions carry these ideas into a shader that slows down and a phone’s frame budget. Each row links to the question’s first step and its follow-ups on the module’s page:

Key takeaways

• A GPU fetches and decodes each instruction once for a whole group of lanes, 32 threads in an NVIDIA warp and a power of two from 4 to 128 in a WGSL subgroup, and runs it on every lane at the same moment.
• When the lanes of a group take different paths, the group runs every path any lane takes, with the others masked: the model’s split branch used 65.0 % of its lane slots, and its uneven loop 33.3 %.
• A core hides memory latency by issuing from other resident groups; (work + latency) ÷ work groups, 21 in the model, keep it issuing on every cycle, and registers per group limit how many it can hold.
• Arithmetic intensity, operations per byte, decides what limits a pass: below the balance point, peak rate ÷ bandwidth, a pass waits for memory and only fewer bytes make it faster.
• A tile-based GPU keeps each tile’s colour and depth on chip while it shades, so a pass’s attachment traffic is what its loadOp and storeOp ask for: clear what is redrawn, and discard what nobody reads.