Ctrl+Enter to process
2
Tokenization
Before an LLM can understand your text, it needs to break it into smaller pieces called "tokens." Think of it like cutting a sentence into word-pieces that the model has learned to recognize. Some common words stay whole, while unusual words get split into smaller chunks.
Common misconception: Tokens are not the same as words. The word "understanding" might be split into "under" and "standing", or even smaller pieces. Spaces and punctuation are also tokens.
Enter a query above to begin.