language-models
Understanding and Building Language Models 📘
How LLMs Work, Step by Step and from Scratch in Python
by Hernán Díaz Rodríguez, PhD — Professor at the University of Oviedo · Former CERN researcher
📘 About this book
Understanding and Building Language Models explains how a language model works inside, like the ones behind today’s conversational assistants, by building a small one from the ground up. It starts by fitting a straight line to the prices of a few houses and ends by training, fine-tuning, evaluating and documenting a small GPT: gradient descent, bigrams, neural networks, tokenization with BPE, attention and the Transformer, training with method, text generation, fine-tuning to follow instructions, evaluation and a capstone project.
All the models are deliberately small: they train on the processor of an ordinary computer in a few minutes, with no graphics card and no model downloads.
This folder contains the 11 interactive notebooks of the English edition, one per chapter, ready to run in Google Colab.
📗 Part of the same series as Deep Learning with Python.
🇪🇸 ¿Prefieres la edición en español? Sus cuadernos están en la raíz del repositorio.
📂 Folder structure
language-models/en/
├── 1_How_a_model_learns.ipynb ← One notebook per chapter
├── 2_Language_modeling.ipynb
├── ...
├── 11_Capstone_project.ipynb
├── notebook/N/ ← Links from the book to each notebook in Colab
└── video/N/ ← Links from the book to each chapter's video
The data the notebooks use are in the repository’s material/ folder.
🎯 What you will learn
The book and the notebooks are organized in 3 parts:
| Part | Topic | Chapters |
|---|---|---|
| 1. Foundations | Gradient descent, language models from counts, from a table to a network, networks with embeddings | 1–4 |
| 2. Building a GPT | Tokenization with BPE, attention and the Transformer, training with method, generation and inference | 5–8 |
| 3. Fine-Tuning, Evaluating and Applying | From base model to assistant, evaluation and limits, capstone project | 9–11 |
🚀 How to use the notebooks
- Click Open in Colab to run the code of each chapter, with nothing to install.
- Follow the book while you experiment with the examples: the notebook is the lab and the book is the explanation.
- Click Video to watch the chapter’s summary on YouTube. They are all in the book’s playlist.
- Notebooks 1, 2, 3 and 5 use only standard Python; the others use PyTorch, which Colab already has installed. None needs a GPU.
- Notebooks 7, 9 and 11 train models for several minutes.
- If you prefer to run them locally, notebooks 5 to 10 download the data from the
material/folder by themselves when they do not find it.
📚 Notebooks and videos
| Ch. | Title | Colab | Video |
|---|---|---|---|
| 1 | How a Model Learns |  |
 |
| 2 | Language Modeling | |
|
| 3 | From Counting to a Network | |
|
| 4 | Networks to Predict Text | |
|
| 5 | Tokenization | |
|
| 6 | Attention and the Transformer | |
|
| 7 | Training and Data | |
|
| 8 | Generation and Inference | |
|
| 9 | From Base Model to Assistant | |
|
| 10 | Evaluation and Limits | |
|
| 11 | Capstone Project | |
|
🗂️ Data
| File | Contents | Notebooks |
|---|---|---|
material/quixote.txt |
Full text of Don Quixote, in John Ormsby’s translation, in the public domain (Project Gutenberg) | 6 to 10 |
material/gpt_quixote.pt |
The character-level GPT trained in chapter 7 | 8 to 10 |
material/corpus_chapters_1_4.txt |
Text of chapters 1 to 4 of the book, to train the tokenizer | 5 and 10 |
👨🏫 About the author
Hernán Díaz Rodríguez is a Professor in the Department of Computer Science and Artificial Intelligence at the University of Oviedo, Spain, with a PhD in Computer Science from the same university. A graduate in Computer Engineering from the same institution, the author also holds an MBA from the Open University (United Kingdom).
With more than 20 years of experience across the public and private sectors, Hernán Díaz has worked on advanced technology and research projects, including 5 years at CERN, the European Organization for Nuclear Research — one of the world’s largest scientific laboratories.
💼 LinkedIn · 📧 Email · 📕 Amazon author page
⭐ Found this useful?
If these notebooks are helping you understand language models, you can support the project:
- ⭐ Starring this repository — it helps others discover the material.
- 💬 Sharing it with a colleague, student or teacher who could benefit.
Thank you! 🙏
📜 License
This material is distributed under Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0).
✅ Allowed: teaching, studying, personal notes; adaptations for educational purposes, citing the author. ❌ Not allowed: commercial or for-profit use; redistribution in commercial products without permission.
📖 How to cite
If you use this material in academic work, please cite:
@book{diazrodriguez2026languagemodels,
author = {Hern{\'a}n D{\'\i}az Rodr{\'\i}guez},
title = {Understanding and Building Language Models: How LLMs Work, Step by Step and from Scratch in Python},
year = {2026},
publisher = {Independently published},
url = {https://github.com/HernanDiaz/language-models}
}
✉️ Contact
Found an error or have a suggestion? I’d love to hear from you: 📧 hernan.diaz.rodriguez@gmail.com
For institutional adoption of the book in courses or programs, please mention your institution and course in the email — I’m happy to support educators.