Home /Modules/artificial-intelligence/README.md

README.md

artificial-intelligence/large-language-models/loader/README.md · 90 line(s) · markdown

Loader-handler idiom

A small, careful way for users to bring their own classes/data — words, maths, objects — and load them into trainable features against a known vocabulary or a real neural net. The user writes one tiny handler; the framework owns the careful machinery.

The idiom, in one sentence

Implement encode() for YOUR type; the framework calls it once per item with framework-owned, correctly-sized feature and target slices, and feeds each encoded example to a sink (e.g. a real slnn_* trainer).

sllme_item_result my_encode(void *user_data, const void *item,
                            double *features, size_t fw,
                            double *target,  size_t tw) {
    const MyThing *x = item;           /* your type */
    features[0] = x->a; features[1] = x->b;   /* write exactly fw features */
    target[0]   = x->label;                   /* write exactly tw targets  */
    return SLLME_ITEM_OK;              /* or SLLME_ITEM_SKIP / SLLME_ITEM_ERROR */
}

That is the whole contract. The user never touches a network, a buffer, or an allocator.

Why it is subtle, performant, and clear

Careful by construction

The loader never fabricates data. A handler that cannot encode an item returns SLLME_ITEM_SKIP (counted and omitted, honestly) or SLLME_ITEM_ERROR (the load stops). A word not in the known vocabulary is the handler's decision — map it to <unk> or skip it — the framework does not guess. Counts are reported (seen / loaded / skipped) so a load is auditable.

Training against known weights / vocabulary

example_loader_slnn.{h,c} provides a ready-made sink that feeds every encoded example straight into slnn_network_train_sample — a real in-process weight update. Point a word handler at a known vocabulary (sllmv_* / the seed base set) and the loader turns "load these words" into genuine training against the network's weights.

Worked handlers (see example_loader_demo.c)

HandlerEncodesTarget
worda word -> one-hot over a known-vocabulary id windownext word's one-hot (unknown words skipped)
math(a, b) normalizeda + b normalized (regression)
objecta flat record of named numeric fields (schema-driven)a label (classification)

The demo loads all three and trains a real slnn_* net on each; it also shows the fail-closed paths.

Files

FileRole
example_loader.h/.cC11 loader-handler core (sllme_*, ABI v1): handler vtable, reused scratch, array loader, reports
example_loader_slnn.h/.cA sink that trains a real slnn_* network from encoded examples
example_loader_demo.cWords (vs a known vocabulary), maths, objects -> real net training; fail-closed checks

Build and run

cc -std=c11 -O2 -Wall -Wextra -pedantic \
   example_loader.c example_loader_slnn.c \
   ../large_language_models_vocabulary.c ../large_language_models_base_vocabulary.c \
   ../../neural-nets/neural_nets.c example_loader_demo.c -o example_loader_demo -lm
./example_loader_demo

What is NOT bundled

No weights, no datasets. The user supplies the items, the handler, and (for word handlers) a known vocabulary. SLeeLa ships the loader machinery and the real training sink.