llm-internals-walkthrough

What this project covers

The notebook demonstrates the core building blocks behind modern LLM workflows:

• installing the required libraries
• using a Hugging Face token to access a gated model
• tokenizing text with a tokenizer
• generating embeddings with the OpenAI API
• converting prompts into PyTorch tensors
• loading a causal language model
• running a forward pass
• generating new tokens from a prompt
• decoding generated token IDs back into readable text

Notebook file

• LLM_Understanding_Commented.ipynb — commented notebook with explanations added throughout

Learning goals

This notebook is meant to help understand the flow of an LLM system:

1. Input text
2. Tokenization
3. Model processing
4. Token prediction / generation
5. Decoding back to text

It also introduces embeddings as a separate but important idea for semantic understanding.

Technologies used

• Python
• Google Colab / Jupyter Notebook
• Hugging Face Transformers
• Gemma (google/gemma-3-1b-it)
• OpenAI Embeddings API
• PyTorch

How to run

1. Open the notebook in Google Colab or Jupyter Notebook
2. Install the required libraries from the first cells
3. Add your own:
   • Hugging Face token
   • OpenAI API key
4. Run the cells in order

Important note

Do not upload real API keys or tokens to GitHub.
Use environment variables, Colab secrets, or another secure secret-management method instead.

Suggested next improvements

• move secrets to Colab Secrets
• add temperature, top-k, and top-p generation examples
• compare tokenization across different prompts
• visualize embedding similarity between multiple sentences
• add a short section on attention and logits

Purpose

This project is focused on building intuition about how LLMs work internally at a beginner level, especially around tokenization, embeddings, inference, and generation.