Build a Small LLM from Scratch
The planned outline of 11 modules. Lessons are written, run and reviewed before they are published.
Before this track
Planned outline
- Module 1
Start here: the pipeline, the model ladder and the workbench
Coming soon - Module 2
Build a pretraining corpus: sources, cleaning, filtering, deduplication
Coming soon - Module 3
Train the tokenizer and write token shards
Coming soon - Module 4
Write a GPT you can verify
Coming soon - Module 5
The pretraining run: loop, speed, sessions and curves
Coming soon - Module 6
Scaling: laws you fit yourself and hyperparameters that transfer
Coming soon - Module 7
Evaluate the base model: perplexity, benchmarks, contamination
Coming soon - Module 8
Post-training your own model: chat format, SFT, DPO and a second language
Coming soon - Module 9
Shrink and export the model: quantisation and file formats
Coming soon - Module 10
Serve it in the browser: your own inference engine
Coming soon - Module 11
Release, report and interview
Coming soon
Related tools
LLM Token Counter Exact token counts for OpenAI and open models — your text never leaves the browser. Unicode Character Inspector Paste text to see each character — and the invisible or look-alike ones hiding in it. Duplicate Line Remover Delete repeated lines in one paste — order kept, counts shown. Hash Generator Hashes and HMACs of text or huge files, plus checksum verification. Line Graph Maker Line and area charts on a real date axis, with notes and moving averages.
Official documentation
- docs.pytorch.org/docs/stable/index.html (docs.pytorch.org)
- github.com/openai/tiktoken (github.com)
- huggingface.co/datasets/HuggingFaceFW/fineweb-edu (huggingface.co)
- github.com/EleutherAI/lm-evaluation-harness (github.com)
- cs336.stanford.edu (cs336.stanford.edu)
More in AI and machine learning
- Machine Learning (scikit-learn) (coming soon)
- AI Literacy & Prompting (coming soon)
- Building LLM Applications (coming soon)
- Artificial Intelligence (classical AI and soft computing) (coming soon)
- All AI & ML tracks
- How we make lessons