Home /Modules/languages/DEFINITION-PIPELINE.md

DEFINITION-PIPELINE.md

languages/datasets/frequency-candidates/DEFINITION-PIPELINE.md · 93 line(s) · markdown

FRUCHST multilingual definition candidate pipeline

fruchst_definition_batch.py creates a review queue, not production definitions. The batch processes all 51 language sets with staged frequency candidates:

Each candidate headword is looked up in three Wiktionary editions (English, French, and Spanish). The tool streams each source dump once and retains exact NFC headword matches for the explicitly mapped target-language code.

Batch execution

The workflow downloads the three source archives outside the repository and runs:

python3 languages/tools/fruchst_definition_batch.py \
  --candidates languages/datasets/frequency-candidates \
  --en /data/kaikki/enwiktionary.jsonl.gz \
  --fr /data/kaikki/frwiktionary.jsonl.gz \
  --es /data/kaikki/eswiktionary.jsonl.gz \
  --output languages/datasets/definition-review

Output filenames follow the candidate source, for example wordhoard-en.review.json, frequencywords-hi.review.json, and unilex-sw.review.json.

Alignment and review safeguards

French and Spanish equivalents through English

French and Spanish Wiktionary have entries for few of the target-language headwords, so a second step, fruchst_english_pivot.py, finds French and Spanish equivalents through English. It runs in the same workflow after the review queues are built and writes languages/datasets/definition-pivot/<source>-<tag>.pivot.json.

  1. The English word(s) are taken from each English Wiktionary gloss of the headword. Form-of senses (plural of, locative of, and similar) and wikitext glosses are skipped; parentheticals are removed; the gloss is split on ; and ,; a leading to/a/an/the is removed; pieces of one to three words are kept. For the English profile, the headword is the English word.
  2. Each English word is looked up in English Wiktionary, and its French and Spanish translation-table entries are kept with their translation sense label and tags.
  3. As secondary evidence, the French and Spanish Wiktionary glosses of the same English word are kept.
  4. Items whose English POS matches the source sense POS are preferred. If none match, all items are kept with pos_match: false.

Each output file stores every English word once under english_lookup. translation_senses are English Wiktionary translation boxes (one per sense label) with the French and Spanish words side by side; glosses are the French and Spanish Wiktionary definitions of the English word. Each headword lists its english_pivots, and each pivot carries pos_match per edition (true, false, or null when nothing was found). Files are written as compact JSON.

The pivot can cross senses (for example, an English gloss "bank" may lead to river-bank and money-bank translations), so every equivalent records the English word, both POS values, and source lines. Nothing is selected; every row stays review_status: required.

Close-variant English definitions

fruchst_close_variants.py handles headwords with no English Wiktionary sense. It looks for a close variant in English Wiktionary, in the same Wiktionary language, and records that variant's English glosses as a close definition in languages/datasets/definition-variants/<source>-<tag>.variants.json. Methods, closest first:

MethodMeaning
case_variantSame word, different letter case.
orthographic_variantSame word after removing conventionally optional marks: Arabic short vowels, tanween, shadda and tatweel; Russian ё written as е; Romanian cedilla ş/ţ as comma-below ș/ț; Arabic-script letter variants ي/ى/ی and ك/ک; and for Arabic only, hamza/madda alef forms أ/إ/آ/ٱ as ا. ة and ه stay distinct because they separate words.
inflected_formThe word appears in another entry's inflection table (forms), exactly or after the same optional-mark removal (via). The close definition is that lemma's gloss. For Chinese, only forms tagged exactly Simplified-Chinese are used; dialectal and alternative characters are skipped.
transliterationUzbek only: official Cyrillic-to-Latin conversion.
diacritic_variantLooser: every Latin, Cyrillic or Greek accent or tone mark removed (e.g. Yoruba hàn vs han). Marked with caution, because these marks can distinguish different words. Abugida vowel signs are never removed.
clitic_strippedArabic only: a leading و/ف/ب/ل/ك and/or ال removed.

Each match records the variant, method, POS, source line, the listed form and its tags, and the variant's English senses (with form_of targets when present). Up to 20 matches are kept per headword. A close definition is not the headword's own definition, so every row stays review_status: required.

Source and license

The definition sources are Kaikki/Wiktextract extractions of the English, French, and Spanish Wiktionary editions. Wiktionary content is dual-licensed under CC BY-SA 4.0 and the GNU Free Documentation License (GFDL). Preserve attribution and comply with the applicable license terms; verify the source edition notices before redistributing derived definitions.

The frequency-candidate files retain their independent provenance and license metadata:

No FRUCHST binary should be built until a human reviewer has selected one sense per headword, confirmed that the English, French, and Spanish glosses express the same sense, checked spelling/grammar and language variety, and recorded source/license metadata. The binary builder approval and provenance checks are necessary but not sufficient for semantic accuracy.

Source dump catalog

The current workflow URLs and revisions are recorded in SOURCE_CATALOG in languages/tools/fruchst_definition_batch.py. Check the Kaikki catalog before production runs because Wiktionary extracts are updated regularly:

Compressed extracts are large. Keep them outside the repository and never commit raw dumps. Generated queues record each downloaded file SHA-256, compressed size, retrieval timestamp, URL, and dump revision.

References: