Home /Modules/artificial-intelligence/README.md

README.md

artificial-intelligence/large-language-models/vocabulary/README.md · 29 line(s) · markdown

Vocabulary assets

The public tokenizer-vocabulary implementation remains in the parent directory as large_language_models_vocabulary.h/.c and its C++17 wrapper. This folder holds model-specific vocabulary exports, lexical dictionary resources, and source metadata. These are related resources but are not interchangeable.

Separation of concerns

Never assign tokenizer IDs directly from dictionary headwords. A model's tokenizer may split a headword into multiple subword or byte-level tokens.

Samples

See samples/ for three carefully labelled source records for the English headword computer:

The JSONL fixture is samples/computer-sources.jsonl. The record_status field distinguishes imported content from reference-only entries. Empty senses on a reference-only record are intentional.

Import and licensing requirements

Before importing a lexical dataset, pin its release/version, record the official source URL, preserve all required attribution and license notices, and validate the data format. Do not scrape or redistribute proprietary dictionary content. Oxford and Merriam-Webster content must be accessed under the applicable API or content license.

The Open English WordNet project publishes versioned downloadable releases at https://en-word.net/downloads and maintains its source at https://github.com/globalwordnet/english-wordnet. Review both the CC BY 4.0 terms and the underlying Princeton WordNet license notice before distributing derived data.

The samples are schema fixtures, not a complete dictionary or tokenizer model.