Home /Modules/artificial-intelligence/README.md

README.md

artificial-intelligence/large-language-models/vocabulary/corpus/README.md · 48 line(s) · markdown

Lexical corpus

Curated, provenance-tracked lexical records — the careful word sets whose meanings and relations ground the model. These are lexical dictionary entries, not tokenizer vocabulary items: a record here describes a headword's senses, synonyms, and provenance, and it never carries a token ID. The link to the tokenizer is made only through the explicit builder (../../tools/build_vocabulary.py), which tokenizes surface strings and never assigns an ID from a headword.

Files

FileRole
core-en.jsonlCurated English core aligned with the base vocabulary's open-class words, plus the three computer source records that demonstrate every status.
../lexical-corpus.schema.jsonJSON Schema (draft-07) each record must satisfy.

Record status and licensing discipline

Every record declares a record_status, and the gate (../../tools/validate_vocabulary.py) enforces what each may store:

record_statusDefinition textLicense rule
sleela_authoredOriginal SLeeLa glossMust be open (CC0/CC-BY); must not claim a third-party source
definition_importedImported from a redistribution-permitting sourceMust carry source.license + attribution; license must not look proprietary
reference_onlyNone (metadata + lookup URL only)For proprietary sources; senses MUST be empty

This keeps the corpus clean of third-party-licensing risk: SLeeLa-authored glosses are original, WordNet-style senses are importable under CC BY 4.0 with attribution retained, and proprietary dictionaries (Oxford, Merriam-Webster) are represented as reference-only descriptors — their definitions are never copied.

Do not silently paraphrase a source definition while keeping its attribution. If SLeeLa adds its own gloss, label it sleela_authored and keep it distinct from any source text.

Adding records

  1. Pin the source edition/version and record its official URL.
  2. Choose the correct record_status (when in doubt about rights, use reference_only).
  3. Preserve all required attribution and license notices.
  4. Run the gate: python3 ../../tools/validate_vocabulary.py.
  5. Rebuild: python3 ../../tools/build_vocabulary.py --base ../../tokenizers/base-vocabulary.json --corpus core-en.jsonl --strict.

The records are schema fixtures and a curated seed, not a complete dictionary.