SLeeLa Native Drive
The Native Drive is the explicit host-toolchain and process boundary for native source execution. It connects existing SLeeLa facilities without claiming that the model frontends under lib/cpu are production C/C++ parsers.
Implemented
tools/sleela-native-drive.py provides:
- Direct C and C++ compilation through the configured host compiler.
- Multi-translation-unit C/C++ builds (all units in one invocation use one language).
- Executable or shared-library output.
- Optional shared-library export validation (both sl_native_abi_version and sl_native_run symbols present) before an artifact is published. The loader checks the ABI version value when a trusted module is actually opened.
- SLeeLa @native AOT input through sleela aot --emit, followed by a host C
compiler. The emitter produces a typed-ABI v1 plugin with one independently
addressable entry point per @native method (exact canonical-signature
dispatch), so .sleela input builds as a shared library (
--kind shared, the default for .sleela). The plugin is compiled against native_loader.h, found via--native-include,SLEELA_NATIVE_INCLUDE, the repository loader tree, or the installed runtimeinclude/directory. The sleela CLI is found via--sleela,SLEELA,SLEELA_BIN, PATH, or the repository build. - Argument-vector process execution without an implicit shell.
- Optional stdin-file input, stdout/stderr capture, multi-stage process piping, timeouts,
and exit-code propagation. Captured output is capped at 16 MiB by default
across stdout and stderr;
--max-output-bytesconfigures the combined limit. Exceeding the cap terminates the child process(es), drains the pipes, emits only the retained prefix, and returns exit code 6. - XClass info/emit/run delegation to the production sleela xclass loader.
Build output is staged beside the destination and atomically replaced only after compilation and optional ABI checks succeed. Compiler selection uses CC/CXX or --compiler. Extra compiler/linker arguments are passed as individual argv elements using repeatable --compiler-arg.
Usage
# Direct C and C++ (one or more translation units) python3 tools/sleela-native-drive.py build main.c math.c -o build/app python3 tools/sleela-native-drive.py build main.cpp math.cpp -o build/app
# Shared library; require the SLeeLa loader ABI before publishing python3 tools/sleela-native-drive.py build plugin.c --kind shared \ --require-sleela-abi -o build/plugin.so
# Supported @native SLeeLa AOT subset -> typed-ABI plugin python3 tools/sleela-native-drive.py build hot.sleela --require-sleela-abi -o build/hot.so
# Existing XClass loader; structural reconstruction is not native-code emission python3 tools/sleela-native-drive.py xclass --mode info Example.xclass python3 tools/sleela-native-drive.py xclass --mode emit Example.xclass python3 tools/sleela-native-drive.py xclass --mode run Example.xclass
# Process invocation and a stdout-to-stdin pipeline python3 tools/sleela-native-drive.py run --capture --max-output-bytes 1048576 --timeout 10 -- build/app arg1 python3 tools/sleela-native-drive.py pipe --max-output-bytes 1048576 --timeout 10 -- producer arg :: filter arg :: consumer arg sleelactl run native-drive pipe --max-output-bytes 1048576 -- producer arg :: filter arg :: consumer arg
What the frontends mean
C and C++ are parsed and compiled by the selected system compiler (Clang, GCC, or a compatible platform toolchain). This is a real production frontend path for those languages, but it does not lower their full language grammars into SLeeLa SLIR. The lib/cpu/SLFrontendC.sleela and SLFrontendCpp.sleela classes remain limited architecture/lowering models.
SLeeLa source uses sleela aot --emit. The command fails closed when annotated
methods exceed its supported subset; otherwise it emits independent per-method
entry points keyed by canonical signature. Separately, syntax 3.0 exposes
host-authorized nativeOpen, typed nativeCall, and nativeClose built-ins;
these are accepted by sleela check/run as well as compile.
XClass is a structural class interchange format. sleela xclass validates and reconstructs its represented class structure into the SLeeLa AST; method bodies that are not represented in the XClass payload are synthesized skeletons. The Native Drive delegates XClass to this loader instead of pretending the structural document is equivalent to C++ source.
Native loader ABI
A shared library intended for the existing loader exports:
uint32_t sl_native_abi_version(void); int sl_native_run(int argc, const char *const argv[]);
The ABI is declared in os-level-support/loader/native_loader.h. The optional --require-sleela-abi switch checks for the required exported symbol names using nm/dumpbin without loading the library. The loader checks the actual ABI version at load time. Loading a native module executes code in the current process; ABI validation is not a sandbox, signature check, or trust decision. Only load modules from a trusted location after applying the host integrity policy.
The loader supports an optional typed sl_native_call entry point for
named methods. Its separately negotiated version-1 value contract covers null,
boolean, signed 64-bit integer, 64-bit double, and length-delimited UTF-8 values.
A typed plugin must export sl_native_typed_abi_version() and report
SL_NATIVE_TYPED_ABI_VERSION; a mismatch fails closed. Legacy run-only
plugins remain compatible.
The SLeeLa VM now connects this ABI to bytecode. Syntax 3.0 exposes:
nativeOpen(path)โ asks the host-installed authorization callback, opens a shared module, and returns a generation-checked VM-owned handle.nativeCall(handle, method, ...args)โ dispatches through the typed ABI.nativeClose(handle)โ closes and invalidates the handle.
The append-only OP_NATIVE_CALL, OP_NATIVE_OPEN, and OP_NATIVE_CLOSE
opcodes preserve every prior opcode number. The interpreter marshals only
null, bool, int64, double, and UTF-8 String values; it never passes its internal
SLValue structure, arrays, structs, closures, object handles, or pointers to
a plugin. Returned strings are copied while the module-call lock is held and
validated before entering the VM String store.
Ordinary calls to @native methods can also reach native code: the compiler
persists method identity (artifact v4), the emitter produces per-method entry
points, and a host that has opened an authorized module binds methods
explicitly with slvm_bind_native_method(). Unbound methods, NOT_HANDLED, and
VM-composite arguments keep the bytecode path. Nothing is bound automatically
by sleela run; binding remains a host decision. See
AT-NATIVE-DISPATCH.md.
Native-to-SLeeLa callbacks
A plugin may optionally export sl_native_bind_host(const SLNativeHostAPI*).
The loader supplies a version-1, size-checked host table when the authorized
module is opened and rebinds it immediately before each plugin entrypoint. A
plugin copies this table into thread-local storage and may call host_api.invoke
synchronously from its typed method to execute an exact, unique compiled SLeeLa
function name such as Callback::receive. The host must separately install
slvm_set_native_callback_authorizer; callback targets are denied by default
and the host policy must explicitly approve each callable name. Plugins must not retain or invoke the
table asynchronously. Thread-local storage keeps callbacks associated with
the correct VM when a shared library is used by multiple VM instances.
Callback arguments and results use only the
same null/bool/int64/double/UTF-8 ABI; raw VM pointers and SLValue objects
are never exposed.
The VM runs callback bytecode on a separate nested interpreter stack while
preserving the suspended caller stack as a GC root. Callbacks are accepted
only when the VM has no active worker threads; callback bytecode also cannot
spawn or join workers while the plugin call is active. Recursive native-module
operations from callback bytecode (nativeOpen, nativeCall, and
nativeClose) are rejected with SL_NATIVE_REENTRANT_CALL rather than
re-entering the module mutex. The public host-side module open/run/call/close
APIs also reject same-VM reentry from an active plugin entrypoint before
attempting to acquire that mutex. Modules
without the optional binder remain compatible but cannot call back into
SLeeLa. A callback failure is a handled plugin error, not a NOT_HANDLED
fallback signal.
If a plugin reports SL_NATIVE_CALL_NOT_HANDLED, or an argument is not
representable by the scalar ABI, the VM may fall back to a same-name,
same-arity SLeeLa bytecode method using the original VM-owned values. Ambiguous
unqualified class methods do not fall back. Plugin errors, stale handles,
authorization failures, and ABI mismatches are surfaced as errors and never
silently routed to bytecode.
Process policy and annotations
annotations.schema.json defines a versioned policy metadata shape for execution mode, capability declarations, stream routing, timeout, working directory, environment allowlist, and output bounds. It is a contract for the next compiler/runtime integration step, not proof that these source annotations are already parsed or enforced.
A source annotation must never grant authority by itself. The runtime must intersect requested capabilities with the caller's configured host policy. Native in-process execution is trusted-only; use an isolated process for untrusted workloads. The current CLI uses argv arrays and does not invoke a shell implicitly.
Native IPC stdio transport adapter
The Native IPC client has a portable stdio adapter (native_ipc_stdio.h/.c)
that implements exact reads and writes over caller-supplied FILE* streams.
Its binary-transfer and truncated-read smoke test is wired into the dedicated
cross-platform GitHub Actions workflow .github/workflows/native-ipc-stdio.yml
for Ubuntu, macOS, and Windows. The host retains stream ownership and remains
responsible for launching and authorizing the worker, connecting its
stdout/stdin pipes, enforcing timeouts and cancellation, and reaping the child.
The adapter is blocking and is not itself a process supervisor. A passing
adapter smoke test does not establish end-to-end VM IPC or Pi dispatch.
See IPC-PROTOCOL.md for the protocol boundary and commands.
systemd service lifecycle
tools/sleela-native-service.py renders a hardened service unit and supports explicit user-unit installation. System-unit installation requires root and an explicit service account; the script never invokes sudo. Enabling and starting are separate explicit flags. The generated unit uses restrictive systemd settings, but service sandboxing must be reviewed against each workload's filesystem/network needs.
Example (render only; no host changes):
python3 tools/sleela-native-service.py --name sleela-worker \ --executable /opt/sleela/bin/worker \ --working-directory /var/lib/sleela-worker
Install as a user service only when intentionally requested (an existing unit is never replaced unless --replace is supplied):
python3 tools/sleela-native-service.py --install --enable \ --name sleela-worker --executable /opt/sleela/bin/worker \ --working-directory /var/lib/sleela-worker
For a system unit, additionally pass --system --user sleela-worker and run
with root privileges. --start is separately required to start it immediately.
systemctl is Linux/systemd-specific; native executable builds remain
available on Windows and macOS. Replacing an existing unit saves one explicit
rollback copy (.sleela-backup); a second replacement is refused until that
backup is rolled back. Use the lifecycle actions to inspect or control the
installed unit:
sleelactl inspect native-drive service --status --name sleela-worker sleelactl inspect native-drive service --restart --name sleela-worker sleelactl inspect native-drive service --stop --name sleela-worker sleelactl inspect native-drive service --disable --name sleela-worker sleelactl inspect native-drive service --rollback --name sleela-worker
Rollback restores the previous unit file and runs systemctl daemon-reload;
it does not automatically restart the workload.
Seven-workstream completion ledger
| Workstream | Current result | Still required |
|---|---|---|
| 1. C/C++ frontends | Production system compiler integration, multi-unit builds | Native SLeeLa frontend adapters and SLIR conformance, not a replacement C++ compiler |
| 2. SLeeLa/XClass translation | Per-method @native AOT plugin emission (typed ABI v1) built by native-drive; XClass loader delegated; syntax 3.0 native built-ins checked and compiled to bytecode | XClass method-body fidelity where encoded; automatic host binding policy for sleela run |
| 3. Native build/link/load | Executable/shared builds, legacy and versioned typed ABI validation, VM-owned authorized handle registry | Deployment signature/integrity policy and compiler-generated native method metadata |
| 4. Process piping | argv execution, stdin file, bounded capture, multi-stage pipe, timeout, exit propagation | Annotation parser/runtime policy and richer stream-routing semantics |
| 5. systemd | Hardened unit rendering, explicit install/enable/start, status/restart/stop/disable, transactional failed-install recovery, one-level replacement rollback | Generated service manifests and integration tests against a real systemd host |
| 6. VM dispatch | Appended open/call/close opcodes, scalar/String marshalling, host authorization, stale-handle checks, NOT_HANDLED and VM-composite interpreter fallback | Explicit composite object/array ownership ABI, output parity, concurrency stress tests |
| 7. Conformance | Loader fixtures, versioned typed ABI, native-to-SLeeLa callback fixture, source compile fixture, VM dispatch/fallback/callback, method metadata, generated-plugin AOT, and concurrency targets (in make test and Linux CI), service recovery coverage | Run local test suites and complete compiler/VM/loader/process/systemd matrix, sanitizer and cross-ABI coverage |
Security invariants
- A path or module reference identifies code; it does not authorize execution.
- Never load caller-controlled native code in a privileged process.
- Do not treat ABI version matching as proof of trust.
- Do not use shell strings for process argument transport.
- Enforce capability policy in the runtime, not by trusting source annotations.
- Keep system service installation, enabling, and starting explicit and separately auditable.
- Treat VM destruction as a host-coordinated lifetime boundary. Native-module
open/run/call/close operations are serialized against one another and against
module unload by the VM's native-module mutex, but that mutex is not a
VM-wide lifetime reference count. The host must stop and join every external
thread that can call any VM API before calling
slvm_free(); no API call may race VM destruction. This is distinct from VM-owned worker threads, whichslvm_free()joins itself. - Do not interpret serialized native-module operations as proof that arbitrary concurrent VM use is supported. In particular, callers must also coordinate VM execution, configuration, and destruction according to the public API contract.
Tests
python3 -m unittest discover -s tools/tests -p 'test_native_drive.py' -v make -C impl test-native-call make -C impl test-native-concurrency # Linux, macOS, and Windows/MinGW sh -n bin/sleelactl
The focused test-native-concurrency target is also opt-in and is not part
of the default test aggregate or CI. On Linux and macOS it builds a typed
plugin that blocks inside slvm_native_module_call until the harness writes
a release marker; on Windows/MinGW it uses named Win32 events and Win32
threads for the same handshake. A second host thread requests
slvm_native_module_close while that call is held. The harness asserts that
the call remains active and close has not completed during the observation
window, then releases the plugin, joins both threads, checks the typed result,
and verifies that the closed generation-checked handle is stale. The Windows
fixture is selected by the Makefile only for the Windows platform; it does
not use POSIX headers, pthreads, or -fPIC. This test covers native-module
call/close serialization only; it does not relax the separate host
requirement to quiesce all external VM callers before slvm_free().
The focused test-native-call target is opt-in and is not part of the default
test aggregate. It substitutes the built fixture's exact path into the
SLeeLa source fixture, compiles that source to a persistent Core artifact,
loads the artifact through slvm_load_file, installs a host-owned exact-path
authorization callback, and executes the program. The source program invokes
the plugin's twice method, calls the SLeeLa Callback::receive method
synchronously from native code, exercises bytecode fallback for fallback,
closes the module, and returns the combined result (105). The harness then reloads
the same artifact with a deliberately mismatched module allowlist and asserts
that source-level nativeOpen fails closed. A third run authorizes the module
but omits callback-name authorization, proving the plugin cannot invoke
Callback::receive unless the host explicitly allows it. The harness also
creates fresh VMs and exercises both public host-side entrypoints:
slvm_native_module_run and slvm_native_module_call. The fixture attempts
the same callback through each entrypoint without an active source-level
nativeCall frame; both attempts must be rejected with the exact VM error
native callback requires an active source-level nativeCall context.
The run-style entrypoint reports the plugin's observed refusal through its
integer result, while the typed-call entrypoint must return
SL_NATIVE_PLUGIN_ERROR. Both cases close the module and free the VM.
The callback-bytecode regression also attempts nativeOpen and
nativeClose from inside an authorized callback; both must fail with the
same explicit reentrancy diagnostic before touching the module table. A
callback-policy test deliberately calls each public host lifecycle API
(open/run/call/close) from the callback authorizer while the plugin entrypoint
is active; all four must return SL_NATIVE_REENTRANT_CALL immediately, and
the approved callback must still complete normally.
A further reentrancy regression constructs a small VM program through the public
core bytecode-building API. Its outer OP_NATIVE_CALL enters the fixture,
which invokes Callback::receive_nested(value, module); that callback attempts
a second nativeCall while callback execution is active. The VM must reject
the nested operation with
native callback cannot recursively invoke native-module operations. The
fixture converts the rejected callback into an integer sentinel so the harness
can assert the exact VM diagnostic without masking it with an outer plugin
error. After the callback unwinds, the harness calls the same module again and
checks the expected typed result, proving the module lock was released and the
original handle remains usable. This case is deliberately a direct bytecode/API
test rather than a compiler-produced source artifact; the existing positive
callback path remains covered by the compiler-produced artifact.
The Native Drive GitHub Actions workflow is workflow_dispatch-only; no
workflow is automatically started by pushes. This end-to-end target has been
added but is not claimed as run until explicitly executed.