Ctrl+Enter to process
5
Attention
Attention is the mechanism that lets each word look at every other word in the sentence and decide which ones are most relevant to understanding its own meaning. When processing the word "bank" in "I deposited money at the bank," attention notices the word "money" and "deposited" nearby, helping the model figure out this is a financial bank, not a riverbank.
Common misconception: Attention does not work by highlighting one word at a time. Every token attends to every other token simultaneously, and there are multiple "attention heads" running in parallel, each potentially focusing on different kinds of relationships (grammar, coreference, proximity, etc.).
Enter a query above to begin.