Vocabulary assets
The public tokenizer-vocabulary implementation remains in the parent directory as large_language_models_vocabulary.h/.c and its C++17 wrapper. This folder holds model-specific vocabulary exports, lexical dictionary resources, and source metadata. These are related resources but are not interchangeable.
Separation of concerns
- Tokenizer vocabulary: exact token strings, stable model-specific token IDs, tokenizer algorithm/version, special tokens, normalization behavior, and model compatibility.
- Lexical dictionary: headwords, parts of speech, sense identifiers, definitions, examples, synonyms, antonyms, semantic relations, provenance, and license metadata.
Never assign tokenizer IDs directly from dictionary headwords. A model's tokenizer may split a headword into multiple subword or byte-level tokens.
Samples
See samples/ for three carefully labelled source records for the English headword computer:
- Open English WordNet 2025 โ imported senses under CC BY 4.0 with Princeton WordNet attribution/license notice retained.
- Oxford Advanced Learner's Dictionary โ metadata-only reference record; proprietary definition content is not copied.
- Merriam-Webster Dictionary โ metadata-only reference record; proprietary definition content is not copied.
The JSONL fixture is samples/computer-sources.jsonl. The record_status field distinguishes imported content from reference-only entries. Empty senses on a reference-only record are intentional.
Import and licensing requirements
Before importing a lexical dataset, pin its release/version, record the official source URL, preserve all required attribution and license notices, and validate the data format. Do not scrape or redistribute proprietary dictionary content. Oxford and Merriam-Webster content must be accessed under the applicable API or content license.
The Open English WordNet project publishes versioned downloadable releases at https://en-word.net/downloads and maintains its source at https://github.com/globalwordnet/english-wordnet. Review both the CC BY 4.0 terms and the underlying Princeton WordNet license notice before distributing derived data.
The samples are schema fixtures, not a complete dictionary or tokenizer model.