Frameworks and representation
Tokenization
The mapping between raw text and the discrete units processed by a model. Vocabulary construction and segmentation affect sequence length, multilingual coverage, cost, and model behavior.
Key terms
- token
- vocabulary
- BPE
- SentencePiece
Connected across the project
Used by
Related technology
Related papers
Historical context
Primary sources
This is a curated technical map, not a claim of comprehensive coverage.