- Python 98%
- Shell 2%
brahm
edd61e2165
Two implementations of the same model, kept side by side so the
optimisation diff is a readable artefact:
v1_readable/ attention written out by hand, fp32, nothing hidden
v2_optimized/ fused SDPA, autocast, gradient accumulation, KV cache,
torch.compile, resume, loads real GPT-2 weights
Plus a benchmark harness that measures the chip's achievable matmul
ceiling rather than quoting a spec sheet, HellaSwag evaluation with an
honest account of its resolution limits, and 14 cross-linked chapters.
All scripts smoke-tested end to end on CPU. Model files self-test:
v2_optimized/model.py verifies cached generation matches uncached
exactly under greedy decoding.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgJQdC6tUPQruwGRBuFRag
|
||
|---|---|---|
| bench | ||
| common | ||
| data | ||
| docs | ||
| eval | ||
| scripts | ||
| v1_readable | ||
| v2_optimized | ||
| .gitignore | ||
| INDEX.md | ||
| LICENSE | ||
| README.md | ||
| requirements.txt | ||
| START-HERE.md | ||
gpt2-mac — build and train GPT-2 on Apple Silicon
A follow-along project: write GPT-2 from an empty file, train it on your Mac, make it fast, and measure it honestly.
→ START HERE — the entry point, with the full chapter list. → INDEX — every file, what it does, and which chapter explains it.
python3 -m venv .venv && source .venv/bin/activate
pip install torch numpy tiktoken
python data/prepare_shakespeare.py
python v1_readable/train.py --data_dir data/shakespeare
python v1_readable/sample.py --ckpt ckpt/v1/ckpt.pt --prompt "ROMEO:"
About ten minutes on an M-series Mac for a ~10M-parameter model that writes things shaped like Shakespeare.
What is in here
The same model, implemented twice on purpose:
v1_readable/— attention written out by hand, fp32, nothing hidden. Read this first.v2_optimized/— fused attention, mixed precision, gradient accumulation, KV cache,torch.compile, resume, loads real GPT-2 weights.
diff v1_readable/model.py v2_optimized/model.py is the performance lesson.
Plus a benchmark harness that measures the chip's actual compute ceiling instead of quoting a spec sheet, an evaluation suite, and 14 chapters of documentation.
The 14 chapters
Setup · Tokenization · Data · The model · Training loop · Sampling · Making it fast · Real GPT-2 weights · Your own data · Benchmarking · Evaluation · Is this practical? · Karpathy lineage · Troubleshooting
Requirements
Apple Silicon Mac (M1+), macOS 12.3+, Python 3.10+. Four dependencies:
torch numpy tiktoken, plus transformers and datasets for two optional
chapters.
Works on CUDA and CPU too — pass --device cuda or --device cpu. The
Mac-specific material is in Chapter 1 and
common/device.py.
Companion repo
smolhub-followalong — Llama-style architecture (RMSNorm, RoPE, grouped-query attention, SwiGLU) plus a Mixture-of-Experts model, following the SmolHub project. Do this repo first.
Credit
Built in the lineage of Andrej Karpathy's teaching repos — micrograd, minGPT, nanoGPT and Let's reproduce GPT-2. Not a copy of any of them; see Chapter 13 for what each one actually is, including the real line counts.
Licence
MIT.