Methodology
Real vs. Simulated Data
This tool uses a mix of real computations and educational simulations. Every visualization clearly labels whether its data is real or simulated.
| Stage | Data Type | Notes |
|---|---|---|
| Raw Input | Real | Your actual text input |
| Tokenization | Real | GPT BPE tokenizer — identical to production models |
| Embeddings | Simulated | Random vectors with seeded PRNG for consistency |
| Positional Encoding | Simulated | Demonstrates the concept with generated values |
| Attention | Simulated | Realistic patterns (local, positional, content-based) with causal masking |
| Transformer Layers | Simulated | Residual stream with simulated layer transformations |
| Prediction | Simulated | Plausible next-token distributions from curated word lists |
| Sampling | Real Math | Temperature, top-k, top-p use correct algorithms on simulated logits |
| Generation | Simulated | Autoregressive loop with simulated predictions per step |
| Context Windows | Simulated | Real token counting with simulated capacity limits |
| RAG | Simulated | TF-IDF-like retrieval over built-in educational documents |
How the Educational Provider Works
The educational simulation provider is designed to teach concepts, not to replicate actual model behavior. It uses several techniques:
- Seeded PRNG: A deterministic random number generator (Mulberry32) ensures the same input always produces the same simulated outputs. The seed is derived from the input text hash.
- Real tokenization: The BPE tokenizer from the gpt-tokenizer package produces identical token IDs to GPT models.
- Plausible patterns: Attention matrices use meaningful patterns (local attention, position-based, content similarity) rather than pure noise, making visualizations educational.
- Correct math: Softmax, temperature scaling, top-k, and top-p filtering use the actual algorithms — just applied to simulated logits.
Limitations
- Embedding vectors are random, not learned — they don't capture real semantic relationships.
- Attention patterns are heuristic approximations, not computed from actual Q/K/V projections.
- Next-token predictions come from curated word lists, not from a trained neural network.
- The training lab shows mathematical loss curves, not actual gradient descent on real data.
- Real LLMs have billions of parameters — the concepts shown here are dramatically simplified.
References
Vaswani et al., "Attention Is All You Need" (2017)
Radford et al., "Language Models are Unsupervised Multitask Learners" (GPT-2, 2019)
Sennrich et al., "Neural Machine Translation of Rare Words with Subword Units" (BPE, 2016)
Holtzman et al., "The Curious Case of Neural Text Degeneration" (Nucleus Sampling, 2020)