Home /Modules/artificial-intelligence/README.md

README.md

artificial-intelligence/large-language-models/vocabulary/samples/README.md · 52 line(s) · markdown

Vocabulary source samples

These records use the same English headword, computer, to demonstrate three distinct dictionary-source integrations.

Files

Status labels

record_statusMeaning
definition_importedThe definition text is included from a source whose license permits redistribution with the recorded notices and attribution.
reference_onlyOnly source identity and a lookup URL are stored. Definition content must be obtained through an authorized API or other licensed access.

A missing or empty senses array on a reference-only record is intentional. It does not mean that the dictionary has no definition.

Source and rights handling

Open English WordNet 2025

This sample preserves the source's two noun senses as separate records, rather than merging their meanings. The sense identifiers follow the WordNet synset convention.

Oxford Advanced Learner's Dictionary

Merriam-Webster Dictionary

Data-model rule

A lexical dictionary entry is not a model tokenizer vocabulary item. A lexical record describes headwords, parts of speech, senses, definitions and semantic relations. A tokenizer vocabulary maps tokenizer-specific token strings to model-specific token IDs. Keep these stores separate and link them only through an explicit tokenization/build process.

Do not infer a tokenizer ID from a dictionary headword. Do not silently replace a source definition with an internally paraphrased definition while retaining the source's attribution. If SLeeLa later adds its own explanatory glosses, label them as SLeeLa-authored and keep them distinct from source text.

Validation expectations

Consumers should validate each JSONL line as UTF-8 JSON, check the required source and rights metadata, and reject a reference-only record if an importer attempts to treat it as licensed definition content. The samples are fixtures for the schema and source-labeling workflow; they are not a complete dictionary.