Home /Modules/artificial-intelligence/README.md

README.md

artificial-intelligence/large-language-models/README.md · 214 line(s) · markdown

Large Language Models

Provider-neutral native APIs for SLeeLa applications that connect to local or hosted large-language-model runtimes. The directory separates data ingestion, connectors, datasets, tokenizers, model packages, lexical resources, training, inference, and checkpoints.

Directory map

PathResponsibility
connectors/Adapters for local files and application-provided database, HTTP, memory, or object-storage streams
datasets/JSONL interchange format and dataset schema guidance
vocabulary/Tokenizer vocabulary assets and separately labelled lexical dictionary sources
tokenizers/Model-compatible tokenizer assets and compatibility rules
models/Model configuration, weights, metadata, and integrity guidance
training/End-to-end data-to-token-batch training workflow
inference/Model loading and streamed generation guidance
checkpoints/Backend checkpoint lifecycle and provenance
examples/Small, human-readable input examples and progressive character/capability training records
distinguish/Ordered novice curriculum combining courtesy, education, leadership, reasoning, script choice, perspective-taking, and fair performance review
basis/Ten terminal-driven LLM application cases, each in C11 and C++17

Public files

File familyPurpose
large_language_models.h/.cC ABI for backend lifecycle, generation options, and streamed UTF-8 output
large_language_models.hpp/.cppC++17 RAII inference facade
large_language_models_vocabulary.h/.cC vocabulary/word-set store, stable token IDs, lookup, and whitespace-token encoding
large_language_models_vocabulary.hpp/.cppC++17 RAII vocabulary wrapper with string and vector helpers
large_language_models_base_vocabulary.h/.cC11 seed-vocabulary loader (sllmbv_*, ABI v1): populates the store with the curated base word set, special tokens at fixed low IDs, fail-closed ID verification
large_language_models_base_vocabulary.hpp/.cppC++17 RAII facade for seeding the default base vocabulary
large_language_models_reasoning.h/.cC11 Reasoning Language Model controller (sllmr_*, ABI v1): neuro-symbolic chain-of-thought with reflection, self-verification, and backtracking over a pluggable backend + verifier
large_language_models_reasoning.hpp/.cppC++17 RAII facade (sleela::ai::ReasoningModel) with std::function verifier/observer
large_language_models_capacity.h/.cC11 user-owned model-capacity profile (sllmc_*, ABI v1): declares a model in the 2–16 GB envelope and derives a reasoning budget; ships no weights
large_language_models_capacity.hppC++17 CapacityProfile helper (tier, envelope check, budget)
large_language_models_code.h/.cC11 Code LLM profile (sllmcx_*, ABI v1): user-owned code-specialized model for procedural programming (summarize/dry-run/refactor/translate) + verifier contract; ships no model
large_language_models_code.hppC++17 CodeModelProfile helper
large_language_models_training.h/.cC training-backend ABI, batch training, evaluation, metrics, and checkpoint requests
large_language_models_training.hpp/.cppC++17 RAII training facade
large_language_models_data.h/.cBounded-memory connector read interface, local-file connector, and streaming JSONL reader

Data ingestion and connector flow

The supported integration path is:

Dataset file / database / HTTP / object storage
                    |
                Connector
                    |
          sllmd_jsonl_read(...)
                    |
        Application JSON validation
                    |
          Model-compatible tokenizer
                    |
       Vocabulary / token-ID sequences
                    |
          Training batch construction
                    |
             sllmt_trainer
                    |
            Training backend

The C data API accepts an sllmd_read_fn callback and reads bounded chunks from its source. sllmd_file_open/read/close provide a local-file connector; applications can implement the same callback for databases, HTTP response bodies, memory buffers, or object storage. sllmd_jsonl_read emits one complete, non-empty record at a time, handles CRLF, accepts a final record without a newline, ignores blank lines, and enforces a caller-selected maximum record size. It does not load the complete dataset into memory.

The reader intentionally emits raw JSONL record bytes: it is not a JSON parser. Applications must validate each record against a versioned schema, extract text fields, check data rights/provenance, and tokenize with the exact tokenizer associated with the target model. The included example is in examples/sample-instruction.jsonl.

Tokenizer vocabulary and lexical dictionaries

The tokenizer vocabulary API stores exact UTF-8 token strings and assigns stable, insertion-order uint32_t IDs. It uses an open-addressed hash table for average constant-time token lookup and grows its storage as the vocabulary expands. Duplicate additions return the existing ID. The encode helper maps whitespace-separated tokens to IDs and returns an owned ID array that the caller must release with sllmv_vocabulary_free_ids.

This is deliberately a vocabulary primitive, not a claim to implement every tokenizer. It does not currently perform BPE, WordPiece, unigram segmentation, byte fallback, Unicode normalization, or punctuation splitting. Punctuation remains attached to its whitespace-delimited token. Applications that need a model's exact tokenizer should provide that tokenizer in their model backend and keep its vocabulary/model metadata aligned. Input strings must be valid UTF-8 when the application treats them as UTF-8.

Dictionary definitions belong in a separate lexical store. Lexical records can contain headwords, parts of speech, sense IDs, definitions, examples, synonyms, semantic relations, source/version metadata, and license notices. They must not be mixed into model token-ID tables.

The vocabulary/samples/ directory contains one carefully labelled example for each of three sources using the headword computer: Open English WordNet 2025, Oxford Advanced Learner's Dictionary, and Merriam-Webster. Only the OEWN senses are included as definition text. Oxford and Merriam-Webster entries are metadata-only reference records because their dictionary content is proprietary; use authorized APIs/licenses before importing definitions. The sample documentation records source links and attribution requirements.

Careful word sets: seed vocabulary, curated corpus, builder, and gate

The subsystem now ships an end-to-end path for the controlled word sets a model builds its premises on:

  1. Seed (base) vocabulary — tokenizers/base-vocabulary.json declares the reserved special tokens (at fixed low IDs: <pad>=0, <unk>=1, <bos>=2, <eos>=3, ...) plus a curated core, loaded natively by large_language_models_base_vocabulary.{h,c} with fail-closed ID checks.
  2. Curated lexical corpus — vocabulary/corpus/core-en.jsonl, provenance- and license-tracked records (SLeeLa-authored, importable, or reference-only).
  3. Builder/bridge — tools/build_vocabulary.py maps corpus surface strings into tokenizer IDs without ever taking an ID from a headword, and refuses to import reference_only content.
  4. Validation/licensing gate — tools/validate_vocabulary.py (CI: .github/workflows/llm-vocabulary.yml) checks schemas, the ID contract, and the licensing discipline, failing closed with exact locations.

See tokenizers/README.md and vocabulary/corpus/README.md for the contract.

Reasoning (RLM layer)

The sllmr_* layer turns a plain next-token backend into a model that reasons: it decomposes a question, emits a tagged chain of thought, reflects on and critiques each step, verifies steps through a pluggable checker, backtracks when a path is refuted, and commits an answer with honest confidence. It is tuned for subjects that are mainly philosophical, mathematical, or religious, where a single confident answer is often wrong.

The design is neuro-symbolic: the LLM supplies proposals and nuance; an optional verifier supplies soundness (for math/logic, bridge it to lib/deliberation). On open subjects the verifier returns UNKNOWN and the engine reports a tentative position rather than claiming proof. See REASONING.md for the full contract and reasoning/ for a runnable end-to-end demo (stub backend + tiny symbolic verifier; no weights, no network).

It is capacity-aware but model-agnostic: a user-declared CapacityProfile (sllmc_*) in the 2 GB–16 GB weight envelope scales the reasoning budget (context, steps, backtracks) across four tiers (compact/standard/capable/large). SLeeLa ships no weights and picks no model — the size and the model are the user's choice; the layer only adapts to them.

Code LLMs (procedural programming)

A supported model type for source code: the Code LLM (coding assistant / program-aided LM) summarizes, dry-runs/state-tracks, refactors, and translates procedural code (C, Fortran, Pascal, procedural Python). It plugs into the reasoning layer as SLLMR_SUBJECT_CODE and is declared by a user-owned CodeModelProfile (sllmcx_*). Behavior claims are only trusted when a code verifier (compile / test / trace) supports them; otherwise they stay UNKNOWN. See CODE-MODELS.md. No code model, weights, or compiler are bundled.

A SLeeLa-specialized Code LLM lives in sleela/: it pairs the code-model type with a verifier (sllmsl_*) that knows SLeeLa's real rules (reserved words such as record/native/enum, the << >> comment forms, the 64-field class cap, the Metadata Class metapattern) and bridges to the real impl/build/sleela compiler for ground truth. See sleela/README.md.

RRS — Reasonable Responsibility Solidity (a kind of LLM)

../rrs/ (a sibling package under artificial-intelligence/) is a reasoning model built as three congruent models chained feed-forward over a ternary 1 / 0 / 2 signal (affirm / eliminate / power-add): (1) a Conjunctive stage where reasonable assumptions surface and a prelude reclaims some as elimination under strict AND; (2) a Reasonable Responsibility prelude-to-continue gate; (3) a Solidity / Firmament culmination to a graded potestas verdict (power in the Latin sense — capacity, not force). Faith / trial-of-faith is subtracted (eliminated, never affirmed). It composes with the reasoning engine here (it includes large_language_models_reasoning.h) and can mount onto it as a conjunctive verifier. No weights. See ../rrs/RRS.md.

Loader-handler idiom (bring your own classes/data)

loader/ is a small, careful way for users to load their own classes/data — words, maths, objects — into trainable features. The user implements one tiny encode() handler for their type; the framework owns bounded, reused scratch (zero per-item allocation), fixed validated widths, honest skip/stop semantics, and a ready-made sink that trains a real slnn_* network — so "load these words/maths/objects and train against known weights/vocabulary" is a genuine, end-to-end path. Fail-closed, auditable counts, no fabricated data. See loader/README.md.

Parametry (binding models to known or local sources)

parametry/ binds a SLeeLa LLM or a SLeeLa neural net to either a known/supported model (by name, from a built-in registry) or a user's local model (by path). An LLM binding yields a CapacityProfile (2–16 GB envelope); a neural-net binding yields an slnn_network_config that builds a real fully-connected net (the neural-nets/ API). Resolution fails closed on an unknown name with no local fallback, a kind mismatch, or a shapeless local net. SLeeLa ships the registry of names + declared parameters and the binding logic — never weights. See parametry/README.md.

Inference

The inference backend supplies three operations:

  1. load — initialize a named model or deployment.
  2. unload — release backend resources.
  3. generate — produce UTF-8 chunks and report them through the supplied callback.

Return zero from the chunk callback to request cancellation. The backend structure, backend name, and user_data are borrowed; keep them alive for the entire engine lifetime.

Training API

The training API defines a backend contract rather than embedding a particular transformer architecture or accelerator runtime. A backend supplies initialization, shutdown, batch-training, evaluation, and checkpoint operations. Training batches are represented as input and target token-ID arrays; metrics report loss, processed tokens, and optimizer steps. Options expose learning rate, gradient clipping, micro-batch size, gradient-accumulation steps, and a deterministic seed.

The backend remains responsible for forward/backward computation, optimizer state, data batching policy, mixed precision, distributed synchronization, and checkpoint serialization. The caller must ensure token IDs match the model's tokenizer and vocabulary. Do not train on data without appropriate rights, consent, and privacy safeguards.

Build

Compile each C source as C11 and each C++ facade as C++17:

cc -std=c11 -O2 -Wall -Wextra -pedantic -c large_language_models.c
cc -std=c11 -O2 -Wall -Wextra -pedantic -c large_language_models_vocabulary.c
cc -std=c11 -O2 -Wall -Wextra -pedantic -c large_language_models_training.c
cc -std=c11 -O2 -Wall -Wextra -pedantic -c large_language_models_data.c
c++ -std=c++17 -O2 -Wall -Wextra -pedantic -c large_language_models.cpp
c++ -std=c++17 -O2 -Wall -Wextra -pedantic -c large_language_models_vocabulary.cpp
c++ -std=c++17 -O2 -Wall -Wextra -pedantic -c large_language_models_training.cpp

Link the C++ facades with their C objects and the selected inference/training backend implementations. No model weights, production tokenizer model, JSON parser, network client, or inference/training runtime is bundled here.

Ownership, safety, and concurrency