Let's build the GPT Tokenizer
Andrej Karpathy · 3 位 HN 用户推荐 · 2024–2025
you plug your query into [link] , you can see that"egregious" is a single token, so the LLM doesn't actually see any "e" characters -- to answer your question it would have had to learn a fact about how the word was spelled from it's training data, and I imagine texts explicitly talking about how words are spelled are not very common. Good explanation here if this still doesn't make sense: [link]
— moozilla, 2024This is just an artefact of tokenisation though. The model simply isn’t ever shown the letters that make up words, unless they are spelled out explicitly. It sees tokens representing groups of words. This is a little like saying a human isn’t intelligent because they couldn’t answer your question that you asked in an ultrasonic wavelength. If you’d like to learn more this video is a great resource
— Chirono, 2024Karpathy's "Let's build the GPT Tokenizer" also contains a good introduction to Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-32 in the first 20 minutes: [link]
— vismit2000, 2025