GPT-PLAY

Feedback: (mentor-to-mentor)

This repo does something very few juniors ever experience:
It lets them touch the architecture, not just tune knobs on top of it.

GPT-PLAY is a tiny, runnable GPT-style Transformer project aimed at juniors for learning, not benchmarking.

The goal is simple:

• Understand the Transformer architecture
• Understand datasets
• Run it
• Train it
• See text come out
• Change things and observe what breaks or improves

This is NOT about fine-tuning large models. This is about understanding what a GPT actually is.

---

WHAT THIS IS

• A minimal decoder-only Transformer (GPT-style)
• Character-level language modeling
• Train-from-scratch on a tiny text file
• Fully readable, hackable code
• No frameworks, no abstractions, no hidden magic

If you can run this, you understand more than most people who say "I fine-tuned a model".

---

REQUIREMENTS

• Python 3.9+
• PyTorch 2.x (CPU is fine)

Install PyTorch (CPU example):

pip install torch

---

HOW TO RUN

1. Put any text you like into input.txt (a README, a story, logs, anything)

2. Run:

   python play.py

3. Watch:

   • Loss go down
   • Text slowly start to make sense

---

WHAT YOU SHOULD TRY

Open play.py and gpt.py and experiment:

• Change num_layers
• Change num_heads
• Change embedding size
• Change block_size
• Change learning rate
• Break it and fix it

This is how understanding is built.

---

IMPORTANT NOTE

This model will NOT produce good text. That is not the point.

The point is:

• Architecture over tooling
• Understanding over scale
• Learning how GPTs actually work

---

🏁 First Learning Exercise: Watching Convergence

To make GPT-PLAY educational, we've included a tiny, highly predictable dataset in input.txt. This dataset is designed so that after a few hundred training steps, you will see the model start generating correct words and sequences.

What You Will Learn

• How repeated patterns in data help a model converge faster
• How small vocabulary and simple sequences make the model “memorize” effectively
• Why dataset quality and structure matter more than hyperparameter tuning at this stage

Steps

1. Ensure input.txt contains the example dataset (words and short sequences), for example:

   the cat
   the dog
   the fish
   the bird
   apple is red
   banana is yellow
   orange is orange
   i like cats
   i like dogs
   ...

2. Run the training script:

   python play.py

3. Observe output every 100–300 steps:

• Initially, the model produces mostly gibberish

• After ~500–1000 steps, you will start seeing sequences like:

  the cat
  the dog
  apple is red
  banana is yellow

• This shows convergence and how the model learns from repetitive patterns

4. Experiment:

• Add new words or phrases to the dataset and re-run
• Observe how long it takes for the model to reproduce new sequences
• Try breaking patterns intentionally to see what happens

Key Teaching Point

This exercise clearly demonstrates the importance of datasets in training:

• Even a tiny model can memorize predictable sequences
• Without repetition or structure, convergence is slow or incoherent
• Understanding what data you feed is just as important as how you train it

---

STATUS

Educational playground. Safe to break. Encouraged to experiment.