# Large Language Models Provider-neutral native APIs for SLeeLa applications that connect to local or hosted large-language-model runtimes. The directory separates data ingestion, connectors, datasets, tokenizers, model packages, lexical resources, training, inference, and checkpoints. ## Directory map | Path | Responsibility | |---|---| | `connectors/` | Adapters for local files and application-provided database, HTTP, memory, or object-storage streams | | `datasets/` | JSONL interchange format and dataset schema guidance | | `vocabulary/` | Tokenizer vocabulary assets and separately labelled lexical dictionary sources | | `tokenizers/` | Model-compatible tokenizer assets and compatibility rules | | `models/` | Model configuration, weights, metadata, and integrity guidance | | `training/` | End-to-end data-to-token-batch training workflow | | `inference/` | Model loading and streamed generation guidance | | `checkpoints/` | Backend checkpoint lifecycle and provenance | | `examples/` | Small, human-readable input examples and progressive character/capability training records | | `distinguish/` | Ordered novice curriculum combining courtesy, education, leadership, reasoning, script choice, perspective-taking, and fair performance review | | `basis/` | Ten terminal-driven LLM application cases, each in C11 and C++17 | ## Public files | File family | Purpose | |---|---| | `large_language_models.h/.c` | C ABI for backend lifecycle, generation options, and streamed UTF-8 output | | `large_language_models.hpp/.cpp` | C++17 RAII inference facade | | `large_language_models_vocabulary.h/.c` | C vocabulary/word-set store, stable token IDs, lookup, and whitespace-token encoding | | `large_language_models_vocabulary.hpp/.cpp` | C++17 RAII vocabulary wrapper with string and vector helpers | | `large_language_models_base_vocabulary.h/.c` | C11 seed-vocabulary loader (`sllmbv_*`, ABI v1): populates the store with the curated base word set, special tokens at fixed low IDs, fail-closed ID verification | | `large_language_models_base_vocabulary.hpp/.cpp` | C++17 RAII facade for seeding the default base vocabulary | | `large_language_models_reasoning.h/.c` | C11 Reasoning Language Model controller (`sllmr_*`, ABI v1): neuro-symbolic chain-of-thought with reflection, self-verification, and backtracking over a pluggable backend + verifier | | `large_language_models_reasoning.hpp/.cpp` | C++17 RAII facade (`sleela::ai::ReasoningModel`) with `std::function` verifier/observer | | `large_language_models_capacity.h/.c` | C11 user-owned model-capacity profile (`sllmc_*`, ABI v1): declares a model in the 2–16 GB envelope and derives a reasoning budget; ships no weights | | `large_language_models_capacity.hpp` | C++17 `CapacityProfile` helper (tier, envelope check, budget) | | `large_language_models_code.h/.c` | C11 Code LLM profile (`sllmcx_*`, ABI v1): user-owned code-specialized model for procedural programming (summarize/dry-run/refactor/translate) + verifier contract; ships no model | | `large_language_models_code.hpp` | C++17 `CodeModelProfile` helper | | `large_language_models_training.h/.c` | C training-backend ABI, batch training, evaluation, metrics, and checkpoint requests | | `large_language_models_training.hpp/.cpp` | C++17 RAII training facade | | `large_language_models_data.h/.c` | Bounded-memory connector read interface, local-file connector, and streaming JSONL reader | ## Data ingestion and connector flow The supported integration path is: ```text Dataset file / database / HTTP / object storage | Connector | sllmd_jsonl_read(...) | Application JSON validation | Model-compatible tokenizer | Vocabulary / token-ID sequences | Training batch construction | sllmt_trainer | Training backend ``` The C data API accepts an `sllmd_read_fn` callback and reads bounded chunks from its source. `sllmd_file_open/read/close` provide a local-file connector; applications can implement the same callback for databases, HTTP response bodies, memory buffers, or object storage. `sllmd_jsonl_read` emits one complete, non-empty record at a time, handles CRLF, accepts a final record without a newline, ignores blank lines, and enforces a caller-selected maximum record size. It does not load the complete dataset into memory. The reader intentionally emits raw JSONL record bytes: it is not a JSON parser. Applications must validate each record against a versioned schema, extract text fields, check data rights/provenance, and tokenize with the exact tokenizer associated with the target model. The included example is in `examples/sample-instruction.jsonl`. ## Tokenizer vocabulary and lexical dictionaries The tokenizer vocabulary API stores exact UTF-8 token strings and assigns stable, insertion-order `uint32_t` IDs. It uses an open-addressed hash table for average constant-time token lookup and grows its storage as the vocabulary expands. Duplicate additions return the existing ID. The encode helper maps whitespace-separated tokens to IDs and returns an owned ID array that the caller must release with `sllmv_vocabulary_free_ids`. This is deliberately a vocabulary primitive, not a claim to implement every tokenizer. It does not currently perform BPE, WordPiece, unigram segmentation, byte fallback, Unicode normalization, or punctuation splitting. Punctuation remains attached to its whitespace-delimited token. Applications that need a model's exact tokenizer should provide that tokenizer in their model backend and keep its vocabulary/model metadata aligned. Input strings must be valid UTF-8 when the application treats them as UTF-8. Dictionary definitions belong in a separate lexical store. Lexical records can contain headwords, parts of speech, sense IDs, definitions, examples, synonyms, semantic relations, source/version metadata, and license notices. They must not be mixed into model token-ID tables. The `vocabulary/samples/` directory contains one carefully labelled example for each of three sources using the headword `computer`: Open English WordNet 2025, Oxford Advanced Learner's Dictionary, and Merriam-Webster. Only the OEWN senses are included as definition text. Oxford and Merriam-Webster entries are metadata-only reference records because their dictionary content is proprietary; use authorized APIs/licenses before importing definitions. The sample documentation records source links and attribution requirements. ### Careful word sets: seed vocabulary, curated corpus, builder, and gate The subsystem now ships an end-to-end path for the controlled word sets a model builds its premises on: 1. **Seed (base) vocabulary** — `tokenizers/base-vocabulary.json` declares the reserved special tokens (at fixed low IDs: ``=0, ``=1, ``=2, ``=3, ...) plus a curated core, loaded natively by `large_language_models_base_vocabulary.{h,c}` with fail-closed ID checks. 2. **Curated lexical corpus** — `vocabulary/corpus/core-en.jsonl`, provenance- and license-tracked records (SLeeLa-authored, importable, or reference-only). 3. **Builder/bridge** — `tools/build_vocabulary.py` maps corpus surface strings into tokenizer IDs without ever taking an ID from a headword, and refuses to import `reference_only` content. 4. **Validation/licensing gate** — `tools/validate_vocabulary.py` (CI: `.github/workflows/llm-vocabulary.yml`) checks schemas, the ID contract, and the licensing discipline, failing closed with exact locations. See `tokenizers/README.md` and `vocabulary/corpus/README.md` for the contract. ## Reasoning (RLM layer) The `sllmr_*` layer turns a plain next-token backend into a model that reasons: it decomposes a question, emits a tagged chain of thought, reflects on and critiques each step, verifies steps through a pluggable checker, backtracks when a path is refuted, and commits an answer with honest confidence. It is tuned for subjects that are mainly philosophical, mathematical, or religious, where a single confident answer is often wrong. The design is neuro-symbolic: the LLM supplies proposals and nuance; an optional verifier supplies soundness (for math/logic, bridge it to `lib/deliberation`). On open subjects the verifier returns `UNKNOWN` and the engine reports a **tentative** position rather than claiming proof. See `REASONING.md` for the full contract and `reasoning/` for a runnable end-to-end demo (stub backend + tiny symbolic verifier; no weights, no network). It is **capacity-aware but model-agnostic**: a user-declared `CapacityProfile` (`sllmc_*`) in the **2 GB–16 GB** weight envelope scales the reasoning budget (context, steps, backtracks) across four tiers (compact/standard/capable/large). SLeeLa ships no weights and picks no model — the size and the model are the user's choice; the layer only adapts to them. ## Code LLMs (procedural programming) A supported model **type** for source code: the Code LLM (coding assistant / program-aided LM) summarizes, dry-runs/state-tracks, refactors, and translates procedural code (C, Fortran, Pascal, procedural Python). It plugs into the reasoning layer as `SLLMR_SUBJECT_CODE` and is declared by a user-owned `CodeModelProfile` (`sllmcx_*`). Behavior claims are only trusted when a code verifier (compile / test / trace) supports them; otherwise they stay UNKNOWN. See `CODE-MODELS.md`. No code model, weights, or compiler are bundled. A **SLeeLa-specialized Code LLM** lives in `sleela/`: it pairs the code-model type with a verifier (`sllmsl_*`) that knows SLeeLa's real rules (reserved words such as `record`/`native`/`enum`, the `<< >>` comment forms, the 64-field class cap, the Metadata Class metapattern) and bridges to the real `impl/build/sleela` compiler for ground truth. See `sleela/README.md`. ## RRS — Reasonable Responsibility Solidity (a kind of LLM) `../rrs/` (a sibling package under `artificial-intelligence/`) is a reasoning model built as **three congruent models chained feed-forward** over a ternary **1 / 0 / 2** signal (affirm / eliminate / power-add): (1) a **Conjunctive** stage where reasonable assumptions surface and a prelude reclaims some as elimination under strict AND; (2) a **Reasonable Responsibility** prelude-to-continue gate; (3) a **Solidity / Firmament** culmination to a graded **potestas** verdict (power in the Latin sense — capacity, not force). Faith / trial-of-faith is **subtracted** (eliminated, never affirmed). It composes with the reasoning engine here (it includes `large_language_models_reasoning.h`) and can mount onto it as a conjunctive verifier. No weights. See `../rrs/RRS.md`. ## Loader-handler idiom (bring your own classes/data) `loader/` is a small, careful way for users to load **their own classes/data** — words, maths, objects — into trainable features. The user implements one tiny `encode()` handler for their type; the framework owns bounded, reused scratch (zero per-item allocation), fixed validated widths, honest skip/stop semantics, and a ready-made sink that trains a **real `slnn_*` network** — so "load these words/maths/objects and train against known weights/vocabulary" is a genuine, end-to-end path. Fail-closed, auditable counts, no fabricated data. See `loader/README.md`. ## Parametry (binding models to known or local sources) `parametry/` binds a SLeeLa **LLM** or a SLeeLa **neural net** to **either** a known/supported model (by name, from a built-in registry) **or** a user's local model (by path). An LLM binding yields a `CapacityProfile` (2–16 GB envelope); a neural-net binding yields an `slnn_network_config` that builds a real fully-connected net (the `neural-nets/` API). Resolution fails closed on an unknown name with no local fallback, a kind mismatch, or a shapeless local net. SLeeLa ships the registry of names + declared parameters and the binding logic — never weights. See `parametry/README.md`. ## Inference The inference backend supplies three operations: 1. `load` — initialize a named model or deployment. 2. `unload` — release backend resources. 3. `generate` — produce UTF-8 chunks and report them through the supplied callback. Return zero from the chunk callback to request cancellation. The backend structure, backend name, and `user_data` are borrowed; keep them alive for the entire engine lifetime. ## Training API The training API defines a backend contract rather than embedding a particular transformer architecture or accelerator runtime. A backend supplies initialization, shutdown, batch-training, evaluation, and checkpoint operations. Training batches are represented as input and target token-ID arrays; metrics report loss, processed tokens, and optimizer steps. Options expose learning rate, gradient clipping, micro-batch size, gradient-accumulation steps, and a deterministic seed. The backend remains responsible for forward/backward computation, optimizer state, data batching policy, mixed precision, distributed synchronization, and checkpoint serialization. The caller must ensure token IDs match the model's tokenizer and vocabulary. Do not train on data without appropriate rights, consent, and privacy safeguards. ## Build Compile each C source as C11 and each C++ facade as C++17: ```sh cc -std=c11 -O2 -Wall -Wextra -pedantic -c large_language_models.c cc -std=c11 -O2 -Wall -Wextra -pedantic -c large_language_models_vocabulary.c cc -std=c11 -O2 -Wall -Wextra -pedantic -c large_language_models_training.c cc -std=c11 -O2 -Wall -Wextra -pedantic -c large_language_models_data.c c++ -std=c++17 -O2 -Wall -Wextra -pedantic -c large_language_models.cpp c++ -std=c++17 -O2 -Wall -Wextra -pedantic -c large_language_models_vocabulary.cpp c++ -std=c++17 -O2 -Wall -Wextra -pedantic -c large_language_models_training.cpp ``` Link the C++ facades with their C objects and the selected inference/training backend implementations. No model weights, production tokenizer model, JSON parser, network client, or inference/training runtime is bundled here. ## Ownership, safety, and concurrency - Callers own backend state, model files, datasets, credentials, and network configuration. - Never hard-code API keys or credentials in source files or examples. - Connector user data and backend structures are borrowed and must outlive the read operation or engine/trainer that uses them. - The JSONL reader allocates at most `max_record_bytes + 1` bytes for its current record plus a fixed input buffer. - Validate model identifiers, token IDs, dataset provenance, JSON schemas, and resource limits inside each application/backend. - The vocabulary object and engine/trainer lifecycle objects are not internally synchronized; use external synchronization when sharing an instance across threads. - Backends must document their own thread-safety, cancellation, determinism, memory use, and checkpoint guarantees. - These interfaces are integration contracts, not claims that inference or training works without a configured backend.