Token
A token is the smallest chunk of text a model works with — not exactly a word and not exactly a character. Common words are usually one token; rarer words get split into multiple tokens. This matters because model costs, speed, and limits are all measured in tokens, not words. It also explains some oddities: AI models sometimes struggle with exact character counts because they never see raw letters, only tokens.
Videos explaining this concept
E011Notes on AI
Tokens
AI models don't read words — they read tokens, the basic unit of text a model processes. A token is close to a word but not identical: one word can be one token, several tokens, or several words can merge into one. Everything in AI is measured in tokens: input, output, context window size, and pricing. One token is roughly four characters in English; once you understand tokens, the limits and costs of AI stop feeling arbitrary.
E012Notes on AI
Tokenization
Tokenization is the process of turning raw text into tokens before an AI model processes it. It is preprocessing, not thinking — the model only sees the resulting pieces.
E013Notes on AI
Why Typos Don't Matter
When you misspell a word, most of the token pieces stay the same and the pattern survives — that's why typos usually don't matter. Models were trained on messy internet text full of broken grammar and misspellings, so they're not fragile. The exception is structural precision: typos in code, formulas, or IDs break meaning because those formats leave no room for pattern inference.
E014Notes on AI
Context Window
The context window is the model's working space, not its memory — only what is currently visible can be reasoned about. Think of it as a desk: only the papers currently on it can be used, and as new papers arrive, old ones slide off the edge. This explains why instructions seem to disappear, why answers contradict earlier statements, and why long conversations slowly fall apart — the model isn't being careless, it simply no longer sees what you think it should remember.