🚧 Work in Progress — this project is under active development.
A character-level GPT (Generatively Pretrained Transformer) built from scratch, trained on the tiny-shakespeare dataset.
This project is inspired by Andrej Karpathy's video Let's build GPT: from scratch, in code, spelled out.
This project walks through the full pipeline of building a character-level GPT from scratch:
- Downloads and loads the tiny-shakespeare dataset for training
- Tokenizes text at the character level, converting between raw text and numeric token sequences
- Splits the dataset into training and validation sets
- Samples random batches of context and target sequences for training
- Trains a bigram language model, where each token directly predicts the next token's probabilities, using an AdamW optimizer running on a GPU when one is available
- Periodically evaluates and reports the model's averaged training and validation loss during training
- Generates new text by repeatedly sampling one token at a time from the trained model
All training hyperparameters, such as batch size, context length, iteration count, and learning rate, are centralized in one place.
- Clone the repository
- Install dependencies:
pip install torch - Open
main.ipynbin Jupyter and run the cells, or run the plain script version withpython main.py