PART 1 · MODULE 2The building blocks of AI
Tokens & Tokenization
Learn how AI breaks down and understands language
Loading module...
Learn how AI breaks down and understands language
Before an LLM can process your text, it needs to break it into smaller pieces called tokens. A token might be a word, part of a word, or even a single character. This process is called tokenization.
TOKENIZATION IN ACTION
Input: "Hello world"
2 tokens
Input: "I'm learning AI!"
5 tokens
Input: "Tokenization splits text"
4 tokens
Type any sentence and watch it split into tokens in real time.
Tokens will appear here as you type.
Every model has a max number of tokens it can process at once. More tokens = more context it can consider.
API costs are based on tokens. More tokens = higher cost. That's why concise prompts save money.
More tokens take longer to process. Shorter inputs and outputs are faster.
English is efficient (~1 token per word). Other languages may need 2-3x more tokens for the same meaning.
CONTEXT WINDOW — HOW MUCH THE AI CAN "SEE"
In English, 1 token ≈ ¾ of a word. So 100 words ≈ 130 tokens. A typical page of text is about 500-700 tokens.