JIT, dynamic PGO, ReadyToRun and SIMD

A one-line C# method, Price, was compiled three times in a program that ran for less than a second: into 76 bytes of quick code, into 108 bytes that recorded what it did, and into 60 bytes of optimized code. That is the normal life of a hot method in .NET, and it is why the first calls of a method are slow and why benchmarks warm up.

This lesson shows those compilations as the JIT reports them, what dynamic PGO adds, and what ReadyToRun and Native AOT change by compiling before the app starts. Then it turns to SIMDSIMD: Single instruction, multiple data: processor instructions that do one operation on several values at once, such as adding four floats held in one 128-bit register.: one instruction that adds or multiplies several numbers held side by side in a wide register. C# reaches SIMD through Vector<T> and the ready-made routines of TensorPrimitives.

The JIT compiles a hot method again, with what it learned

tiers/Program.cs adds up the prices of 20 batches of 10,000 items, with a short pause after each batch. The session runs the same build three times with DOTNET_JitDisasmSummary=1, which makes the JIT compilerJIT compiler: The just-in-time compiler of the .NET runtime: it compiles the IL of a method into machine code for the processor it runs on, just before the method first runs. More in “What C# and .NET are, and which version to use” list every method it compiles, with the tier and the size of the code:

// Twenty batches of 10,000 prices, with a short pause after each. Run it with DOTNET_JitDisasmSummary=1 to see every
// time the JIT compiles one of its methods, and at which tier.
long total = 0;
for (int batch = 0; batch < 20; batch++)
{
    total += Batch(10_000);
    Thread.Sleep(20); // quiet time, in which the runtime's background compiler can work
}
Console.WriteLine(total);

static long Batch(int count)
{
    long sum = 0;
    for (int i = 0; i < count; i++) sum += Price(i);
    return sum;
}

static int Price(int units) => units % 3 == 0 ? units * 2 : units + 7;

from session import run

# The same build of tiers/Program.cs run three times: with the runtime's defaults, with dynamic PGO switched off and
# with tiered compilation switched off. DOTNET_JitDisasmSummary=1 makes the JIT list every method it compiles, and
# DOTNET_JitStdOutFile sends that list to a file, which grep reads for the program's own methods.
run("""
cd tiers
dotnet build -c Release -o out -nologo -v q
DOTNET_JitDisasmSummary=1 DOTNET_JitStdOutFile=default.txt dotnet out/tiers.dll
grep Program: default.txt
DOTNET_TieredPGO=0 DOTNET_JitDisasmSummary=1 DOTNET_JitStdOutFile=nopgo.txt dotnet out/tiers.dll
grep Program: nopgo.txt
DOTNET_TieredCompilation=0 DOTNET_JitDisasmSummary=1 DOTNET_JitStdOutFile=fullopts.txt dotnet out/tiers.dll
grep Program: fullopts.txt
""")

$ cd tiers
$ dotnet build -c Release -o out -nologo -v q

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed …
$ DOTNET_JitDisasmSummary=1 DOTNET_JitStdOutFile=default.txt dotnet out/tiers.dll
1334199900
$ grep Program: default.txt
   1: JIT compiled Program:<Main>$(System.String[]) [Instrumented Tier0, IL size=43, code size=224]
   2: JIT compiled Program:<<Main>$>g__Batch|0_0(int) [Instrumented Tier0, IL size=27, code size=188]
   3: JIT compiled Program:<<Main>$>g__Price|0_1(int) [Tier0, IL size=13, code size=76]
   4: JIT compiled Program:<<Main>$>g__Batch|0_0(int) [Tier1-OSR @0x15 with Synthesized PGO, IL size=27, code size=104]
   5: JIT compiled Program:<<Main>$>g__Price|0_1(int) [Instrumented Tier0, IL size=13, code size=108]
   6: JIT compiled Program:<<Main>$>g__Price|0_1(int) [Tier1 with Dynamic PGO, IL size=13, code size=60]
$ DOTNET_TieredPGO=0 DOTNET_JitDisasmSummary=1 DOTNET_JitStdOutFile=nopgo.txt dotnet out/tiers.dll
1334199900
$ grep Program: nopgo.txt
   1: JIT compiled Program:<Main>$(System.String[]) [Tier0, IL size=43, code size=192]
   2: JIT compiled Program:<<Main>$>g__Batch|0_0(int) [Tier0, IL size=27, code size=156]
   3: JIT compiled Program:<<Main>$>g__Price|0_1(int) [Tier0, IL size=13, code size=76]
   4: JIT compiled Program:<<Main>$>g__Batch|0_0(int) [Tier1-OSR @0x15, IL size=27, code size=108]
   5: JIT compiled Program:<<Main>$>g__Price|0_1(int) [Tier1, IL size=13, code size=60]
$ DOTNET_TieredCompilation=0 DOTNET_JitDisasmSummary=1 DOTNET_JitStdOutFile=fullopts.txt dotnet out/tiers.dll
1334199900
$ grep Program: fullopts.txt
   1: JIT compiled Program:<Main>$(System.String[]) [FullOpts, IL size=43, code size=116]
   2: JIT compiled Program:<<Main>$>g__Batch|0_0(int) [FullOpts, IL size=27, code size=104]

This is tiered compilationtiered compilation: The runtime compiling a method more than once: first quickly, with few optimizations, then again with full optimizations on a background thread once the method has been called often enough., on by default since .NET Core 3.0. Price started as quick Tier0 code, because most methods run too few times to repay a slow, careful compilation; once the runtime saw it was hot, it compiled Price again on a background thread, and the program switched to the new code without stopping. Step through its life in the first run:

The middle step is dynamic PGOdynamic PGO: Dynamic profile-guided optimization: the runtime runs an instrumented version of a hot method that records which branches and types really occur, and the optimizing compiler uses that profile. On by default since .NET 8., profile-guided optimization, on by default since .NET 8: the instrumented version counts which branches are taken and which types arrive at each call, and the final compilation uses the counts. With DOTNET_TieredPGO=0 the second run skipped that step. Batch’s long loop could not wait for another call, so its code was replaced in the middle of the loop, on-stack replacementon-stack replacement: Switching a method that is still running, inside a long loop, from its quick first code to optimized code without waiting for the next call. Such compilations show as Tier1-OSR.. With tiering off, each method was compiled once, FullOpts, and Price not at all: the JIT copied its body into Batch, which is inlininginlining: Copying the body of a called method into the method that calls it, which removes the call and lets the compiler optimize the two together..

ReadyToRun and Native AOT compile before the app starts

Every Tier0 compilation costs time while the program starts. ReadyToRunReadyToRun: Ahead-of-time compilation at publish time that stores machine code beside the IL of an assembly, so the JIT has less to compile when the app starts. Hot methods can still be recompiled later. compiles the assemblies when you publish the app and keeps that machine code beside the IL, so the JIT has less to do at start-up. The session publishes the same program with -p:PublishReadyToRun=true and runs it with the same list switched on:

from session import run

# network: true. Publishing with ReadyToRun needs the runtime pack and the crossgen2 compiler pack for osx-arm64, which
# dotnet publish downloads from nuget.org into the session's own, empty home folder. wc -c prints the sizes in bytes.
run("""
cd tiers
dotnet build -c Release -o out -nologo -v q
dotnet publish -c Release -r osx-arm64 --self-contained false -p:PublishReadyToRun=true -o r2r -nologo -v q
DOTNET_JitDisasmSummary=1 DOTNET_JitStdOutFile=r2r.txt dotnet r2r/tiers.dll
cat r2r.txt
wc -c out/tiers.dll r2r/tiers.dll
""", network=True)

$ cd tiers
$ dotnet build -c Release -o out -nologo -v q

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed …
$ dotnet publish -c Release -r osx-arm64 --self-contained false -p:PublishReadyToRun=true -o r2r -nologo -v q
$ DOTNET_JitDisasmSummary=1 DOTNET_JitStdOutFile=r2r.txt dotnet r2r/tiers.dll
1334199900
$ cat r2r.txt
   1: JIT compiled System.RuntimeType+IGenericCacheEntry`1[System.__Canon]:CreateAndCache(System.RuntimeType) [Instrumented Tier0, IL size=165, code size=1152]
   2: JIT compiled System.Threading.Thread:GetThreadStaticsBase() [Tier0, IL size=18, code size=32]
$ wc -c out/tiers.dll r2r/tiers.dll
    5120 out/tiers.dll
    6144 r2r/tiers.dll
   11264 total

None of the program’s methods appear in r2r.txt: Main, Batch and Price ran as precompiled code, and the JIT compiled only two methods of the runtime’s own libraries. The assembly grew from 5,120 to 6,144 bytes, and the documentation warns of two to three times for real code, because it holds the IL as well as the machine code; it also notes that ReadyToRun code is less optimized than the JIT’s, so tiered compilation still replaces hot ReadyToRun methods. Native AOTNative AOT: Ahead-of-time publishing: dotnet publish compiles a whole app into one native executable that needs no installed .NET runtime and no JIT, at the cost of run-time code generation and dynamic loading. More in “What C# and .NET are, and which version to use” goes further: one native executable, with no JIT at run time.

Dynamic PGO measured: interface calls with one class and with three Pro: opens with a Pro, Premium or Ultimate pass

SIMD by hand and with TensorPrimitives, measured Pro: opens with a Pro, Premium or Ultimate pass

Practise

New questions on tiering and the three ways to compile, the idea you know least first; a missed question returns at the end:

Interview questions

Fresher · What is tiered compilation in .NET?

The runtime may compile a method more than once. Its first call gets quick Tier0 code, with few optimizations, so the program starts fast; when the method turns out to be hot, or stuck in a long loop, it is compiled again on a background thread with full optimizations, Tier1, and the program switches to the new code. Since .NET 8 an instrumented step first records a profile for the Tier1 compilation, which is dynamic PGO. DOTNET_TieredCompilation=0 switches tiering off, and every method is then compiled once, fully optimized.

Key takeaways

• A hot method is compiled more than once: quick Tier0 code first, then optimized Tier1 code on a background thread; a long loop is switched to optimized code while it runs.
• Dynamic PGO, on by default since .NET 8, adds an instrumented step whose profile the final compilation uses.
• ReadyToRun precompiles assemblies so the JIT has less to do at start-up; Native AOT removes the JIT altogether and gives up dynamic loading, Reflection.Emit and dynamic PGO.