Home /Modules/languages/FRUCHST_FORMAT.md

FRUCHST_FORMAT.md

languages/FRUCHST_FORMAT.md · 92 line(s) · markdown

FRUCHST Binary Dictionary Format — Version 1

Status: Initial stable format specification Encoding: UTF-8, Unicode NFC Scope: Ranked lexical entries with aligned English, French, and Spanish definitions

1. Purpose

FRUCHST stores a ranked word list for one language. Each entry contains the native headword and three definitions that describe the same lexical sense in English, French, and Spanish. A dictionary is language-specific; definitions are not substitutes for a separate translation lexicon.

2. File naming

Each language profile stores one file named fruchst.<iso-639-1>.bin where a two-letter ISO 639-1 code exists. Profiles without an ISO 639-1 code use a documented ISO 639-3 code. The profile's language.json is authoritative for the exact BCP 47 locale and ISO identifiers.

3. Binary layout

All integers are unsigned big-endian. Floating-point values are IEEE-754 binary64 big-endian. All strings are UTF-8 and MUST be Unicode NFC. Lengths count bytes, not Unicode code points.

3.1 Fixed header (72 bytes)

OffsetSizeFieldMeaning
08magicASCII FRUCHST\0
82major1
102minor0
122header_size72
142flagsBit 0: frequency values present; other bits reserved and zero
164entry_countMust be exactly 1000 for production dictionaries
202language_tag_lenByte length of BCP 47 language tag
222source_id_lenByte length of source dataset identifier
248generated_epochUTC Unix seconds; zero for reproducible builds
328payload_lenBytes following the fixed header
4032payload_sha256SHA-256 of all bytes following the fixed header

3.2 Variable payload

The payload begins with the language tag and source identifier, each encoded as its declared number of UTF-8 bytes. These are followed by exactly entry_count records.

Each record is:

FieldTypeMeaning
rankuint16One-based frequency rank; contiguous from 1
frequencyfloat64Non-negative source frequency; zero if unavailable
headword_lenuint32UTF-8 byte length
headwordbytesNative-language headword
definition_en_lenuint32UTF-8 byte length
definition_enbytesEnglish definition
definition_fr_lenuint32UTF-8 byte length
definition_frbytesFrench definition
definition_es_lenuint32UTF-8 byte length
definition_esbytesSpanish definition

A zero-length headword or definition is invalid. Production records must have unique normalized headwords. Definitions must express the same selected sense in all three languages.

4. Ranking and frequency

Ranks are contiguous and authoritative. The input data must already be ordered by the chosen corpus-frequency methodology. Frequency values must use a consistent unit within a dictionary (for example, occurrences per million tokens); the unit and corpus methodology belong in the adjacent provenance manifest. Do not compare raw frequencies across languages unless their corpus methodology is demonstrably comparable.

5. Provenance and licensing

The binary embeds a source identifier. The adjacent fruchst.<code>.manifest.json records:

Do not bundle data unless its license permits redistribution. A source URL alone is not proof of permission. Do not claim a word list is corpus-ranked when its ordering is editorial or synthetic.

6. Validation requirements

A production dictionary MUST:

  1. have the exact magic, supported version, and 72-byte header;
  2. declare exactly 1,000 entries;
  3. have a matching payload length and SHA-256;
  4. use a valid BCP 47 language tag and NFC UTF-8 strings;
  5. have ranks 1 through 1000 without gaps or duplicates;
  6. have unique non-empty headwords;
  7. have all three non-empty definitions for every entry;
  8. have non-negative finite frequencies;
  9. have a matching provenance manifest and redistributable source license;
  10. have recorded human approval and selected-sense evidence for every entry.

7. Compatibility

Readers MUST reject unsupported major versions and unknown required flags. Minor-version additions may use only reserved fields or explicitly documented optional extensions. Existing alphabet XML and JSON files are independent and are not modified by FRUCHST.

8. Reference implementation

The reference writer and validator are in languages/tools/fruchst.py. Production builds require a manifest and explicit human approval evidence for every entry; unreviewed queue candidates are not production input. The input JSON schema and dataset acquisition rules are documented in languages/datasets/README.md.