Training pipeline
The training API in the parent directory accepts token-ID batches through sllmt_trainer_train_batch. A complete application pipeline is:
- Read bounded JSONL records through the connector/data-reader API.
- Parse and validate the application's versioned record schema.
- Extract prompt/completion or conversation text.
- Apply the exact tokenizer shipped with the target model.
- Construct input/target token-ID sequences and padded or packed batches.
- Submit batches to the selected
sllmt_backend. - Record metrics and write checkpoints to a controlled destination.
The current backend contract delegates forward/backward computation, optimizer state, scheduling, batching policy, accelerator use, checkpoint serialization, and distributed coordination to the selected backend. Do not treat raw JSONL records as token IDs.