GPT’s vocab size is north of tens of thousands. Included in this vocab are the 256 unique byte values (2^8). This means GPT can already generate/interpret all byte sequences.
The vocab size is larger than 256 (or in your example 2) because some sequences of bytes occur much more frequently than others. This allows the model to generalize more efficiently.
By analogy consider that we think of “words” as atomic concepts much more than we think of the individual lines that make up the letters in a word - even though the former can be thought of being composed of the latter.
The vocab size is larger than 256 (or in your example 2) because some sequences of bytes occur much more frequently than others. This allows the model to generalize more efficiently.
By analogy consider that we think of “words” as atomic concepts much more than we think of the individual lines that make up the letters in a word - even though the former can be thought of being composed of the latter.