Lexical corpus
Curated, provenance-tracked lexical records — the careful word sets whose
meanings and relations ground the model. These are lexical dictionary
entries, not tokenizer vocabulary items: a record here describes a headword's
senses, synonyms, and provenance, and it never carries a token ID. The link to
the tokenizer is made only through the explicit builder
(../../tools/build_vocabulary.py), which tokenizes surface strings and never
assigns an ID from a headword.
Files
| File | Role |
|---|---|
core-en.jsonl | Curated English core aligned with the base vocabulary's open-class words, plus the three computer source records that demonstrate every status. |
../lexical-corpus.schema.json | JSON Schema (draft-07) each record must satisfy. |
Record status and licensing discipline
Every record declares a record_status, and the gate
(../../tools/validate_vocabulary.py) enforces what each may store:
record_status | Definition text | License rule |
|---|---|---|
sleela_authored | Original SLeeLa gloss | Must be open (CC0/CC-BY); must not claim a third-party source |
definition_imported | Imported from a redistribution-permitting source | Must carry source.license + attribution; license must not look proprietary |
reference_only | None (metadata + lookup URL only) | For proprietary sources; senses MUST be empty |
This keeps the corpus clean of third-party-licensing risk: SLeeLa-authored glosses are original, WordNet-style senses are importable under CC BY 4.0 with attribution retained, and proprietary dictionaries (Oxford, Merriam-Webster) are represented as reference-only descriptors — their definitions are never copied.
Do not silently paraphrase a source definition while keeping its attribution. If
SLeeLa adds its own gloss, label it sleela_authored and keep it distinct from
any source text.
Adding records
- Pin the source edition/version and record its official URL.
- Choose the correct
record_status(when in doubt about rights, usereference_only). - Preserve all required attribution and license notices.
- Run the gate:
python3 ../../tools/validate_vocabulary.py. - Rebuild:
python3 ../../tools/build_vocabulary.py --base ../../tokenizers/base-vocabulary.json --corpus core-en.jsonl --strict.
The records are schema fixtures and a curated seed, not a complete dictionary.