Home /Modules/impl/README.md

README.md

impl/native-drive/README.md · 325 line(s) · markdown

SLeeLa Native Drive

The Native Drive is the explicit host-toolchain and process boundary for native source execution. It connects existing SLeeLa facilities without claiming that the model frontends under lib/cpu are production C/C++ parsers.

Implemented

tools/sleela-native-drive.py provides:

Build output is staged beside the destination and atomically replaced only after compilation and optional ABI checks succeed. Compiler selection uses CC/CXX or --compiler. Extra compiler/linker arguments are passed as individual argv elements using repeatable --compiler-arg.

Usage

# Direct C and C++ (one or more translation units) python3 tools/sleela-native-drive.py build main.c math.c -o build/app python3 tools/sleela-native-drive.py build main.cpp math.cpp -o build/app

# Shared library; require the SLeeLa loader ABI before publishing python3 tools/sleela-native-drive.py build plugin.c --kind shared \ --require-sleela-abi -o build/plugin.so

# Supported @native SLeeLa AOT subset -> typed-ABI plugin python3 tools/sleela-native-drive.py build hot.sleela --require-sleela-abi -o build/hot.so

# Existing XClass loader; structural reconstruction is not native-code emission python3 tools/sleela-native-drive.py xclass --mode info Example.xclass python3 tools/sleela-native-drive.py xclass --mode emit Example.xclass python3 tools/sleela-native-drive.py xclass --mode run Example.xclass

# Process invocation and a stdout-to-stdin pipeline python3 tools/sleela-native-drive.py run --capture --max-output-bytes 1048576 --timeout 10 -- build/app arg1 python3 tools/sleela-native-drive.py pipe --max-output-bytes 1048576 --timeout 10 -- producer arg :: filter arg :: consumer arg sleelactl run native-drive pipe --max-output-bytes 1048576 -- producer arg :: filter arg :: consumer arg

What the frontends mean

C and C++ are parsed and compiled by the selected system compiler (Clang, GCC, or a compatible platform toolchain). This is a real production frontend path for those languages, but it does not lower their full language grammars into SLeeLa SLIR. The lib/cpu/SLFrontendC.sleela and SLFrontendCpp.sleela classes remain limited architecture/lowering models.

SLeeLa source uses sleela aot --emit. The command fails closed when annotated methods exceed its supported subset; otherwise it emits independent per-method entry points keyed by canonical signature. Separately, syntax 3.0 exposes host-authorized nativeOpen, typed nativeCall, and nativeClose built-ins; these are accepted by sleela check/run as well as compile.

XClass is a structural class interchange format. sleela xclass validates and reconstructs its represented class structure into the SLeeLa AST; method bodies that are not represented in the XClass payload are synthesized skeletons. The Native Drive delegates XClass to this loader instead of pretending the structural document is equivalent to C++ source.

Native loader ABI

A shared library intended for the existing loader exports:

uint32_t sl_native_abi_version(void); int sl_native_run(int argc, const char *const argv[]);

The ABI is declared in os-level-support/loader/native_loader.h. The optional --require-sleela-abi switch checks for the required exported symbol names using nm/dumpbin without loading the library. The loader checks the actual ABI version at load time. Loading a native module executes code in the current process; ABI validation is not a sandbox, signature check, or trust decision. Only load modules from a trusted location after applying the host integrity policy.

The loader supports an optional typed sl_native_call entry point for named methods. Its separately negotiated version-1 value contract covers null, boolean, signed 64-bit integer, 64-bit double, and length-delimited UTF-8 values. A typed plugin must export sl_native_typed_abi_version() and report SL_NATIVE_TYPED_ABI_VERSION; a mismatch fails closed. Legacy run-only plugins remain compatible.

The SLeeLa VM now connects this ABI to bytecode. Syntax 3.0 exposes:

The append-only OP_NATIVE_CALL, OP_NATIVE_OPEN, and OP_NATIVE_CLOSE opcodes preserve every prior opcode number. The interpreter marshals only null, bool, int64, double, and UTF-8 String values; it never passes its internal SLValue structure, arrays, structs, closures, object handles, or pointers to a plugin. Returned strings are copied while the module-call lock is held and validated before entering the VM String store.

Ordinary calls to @native methods can also reach native code: the compiler persists method identity (artifact v4), the emitter produces per-method entry points, and a host that has opened an authorized module binds methods explicitly with slvm_bind_native_method(). Unbound methods, NOT_HANDLED, and VM-composite arguments keep the bytecode path. Nothing is bound automatically by sleela run; binding remains a host decision. See AT-NATIVE-DISPATCH.md.

Native-to-SLeeLa callbacks

A plugin may optionally export sl_native_bind_host(const SLNativeHostAPI*). The loader supplies a version-1, size-checked host table when the authorized module is opened and rebinds it immediately before each plugin entrypoint. A plugin copies this table into thread-local storage and may call host_api.invoke synchronously from its typed method to execute an exact, unique compiled SLeeLa function name such as Callback::receive. The host must separately install slvm_set_native_callback_authorizer; callback targets are denied by default and the host policy must explicitly approve each callable name. Plugins must not retain or invoke the table asynchronously. Thread-local storage keeps callbacks associated with the correct VM when a shared library is used by multiple VM instances. Callback arguments and results use only the same null/bool/int64/double/UTF-8 ABI; raw VM pointers and SLValue objects are never exposed.

The VM runs callback bytecode on a separate nested interpreter stack while preserving the suspended caller stack as a GC root. Callbacks are accepted only when the VM has no active worker threads; callback bytecode also cannot spawn or join workers while the plugin call is active. Recursive native-module operations from callback bytecode (nativeOpen, nativeCall, and nativeClose) are rejected with SL_NATIVE_REENTRANT_CALL rather than re-entering the module mutex. The public host-side module open/run/call/close APIs also reject same-VM reentry from an active plugin entrypoint before attempting to acquire that mutex. Modules without the optional binder remain compatible but cannot call back into SLeeLa. A callback failure is a handled plugin error, not a NOT_HANDLED fallback signal.

If a plugin reports SL_NATIVE_CALL_NOT_HANDLED, or an argument is not representable by the scalar ABI, the VM may fall back to a same-name, same-arity SLeeLa bytecode method using the original VM-owned values. Ambiguous unqualified class methods do not fall back. Plugin errors, stale handles, authorization failures, and ABI mismatches are surfaced as errors and never silently routed to bytecode.

Process policy and annotations

annotations.schema.json defines a versioned policy metadata shape for execution mode, capability declarations, stream routing, timeout, working directory, environment allowlist, and output bounds. It is a contract for the next compiler/runtime integration step, not proof that these source annotations are already parsed or enforced.

A source annotation must never grant authority by itself. The runtime must intersect requested capabilities with the caller's configured host policy. Native in-process execution is trusted-only; use an isolated process for untrusted workloads. The current CLI uses argv arrays and does not invoke a shell implicitly.

Native IPC stdio transport adapter

The Native IPC client has a portable stdio adapter (native_ipc_stdio.h/.c) that implements exact reads and writes over caller-supplied FILE* streams. Its binary-transfer and truncated-read smoke test is wired into the dedicated cross-platform GitHub Actions workflow .github/workflows/native-ipc-stdio.yml for Ubuntu, macOS, and Windows. The host retains stream ownership and remains responsible for launching and authorizing the worker, connecting its stdout/stdin pipes, enforcing timeouts and cancellation, and reaping the child. The adapter is blocking and is not itself a process supervisor. A passing adapter smoke test does not establish end-to-end VM IPC or Pi dispatch. See IPC-PROTOCOL.md for the protocol boundary and commands.

systemd service lifecycle

tools/sleela-native-service.py renders a hardened service unit and supports explicit user-unit installation. System-unit installation requires root and an explicit service account; the script never invokes sudo. Enabling and starting are separate explicit flags. The generated unit uses restrictive systemd settings, but service sandboxing must be reviewed against each workload's filesystem/network needs.

Example (render only; no host changes):

python3 tools/sleela-native-service.py --name sleela-worker \ --executable /opt/sleela/bin/worker \ --working-directory /var/lib/sleela-worker

Install as a user service only when intentionally requested (an existing unit is never replaced unless --replace is supplied):

python3 tools/sleela-native-service.py --install --enable \ --name sleela-worker --executable /opt/sleela/bin/worker \ --working-directory /var/lib/sleela-worker

For a system unit, additionally pass --system --user sleela-worker and run with root privileges. --start is separately required to start it immediately. systemctl is Linux/systemd-specific; native executable builds remain available on Windows and macOS. Replacing an existing unit saves one explicit rollback copy (.sleela-backup); a second replacement is refused until that backup is rolled back. Use the lifecycle actions to inspect or control the installed unit:

sleelactl inspect native-drive service --status --name sleela-worker sleelactl inspect native-drive service --restart --name sleela-worker sleelactl inspect native-drive service --stop --name sleela-worker sleelactl inspect native-drive service --disable --name sleela-worker sleelactl inspect native-drive service --rollback --name sleela-worker

Rollback restores the previous unit file and runs systemctl daemon-reload; it does not automatically restart the workload.

Seven-workstream completion ledger

WorkstreamCurrent resultStill required
1. C/C++ frontendsProduction system compiler integration, multi-unit buildsNative SLeeLa frontend adapters and SLIR conformance, not a replacement C++ compiler
2. SLeeLa/XClass translationPer-method @native AOT plugin emission (typed ABI v1) built by native-drive; XClass loader delegated; syntax 3.0 native built-ins checked and compiled to bytecodeXClass method-body fidelity where encoded; automatic host binding policy for sleela run
3. Native build/link/loadExecutable/shared builds, legacy and versioned typed ABI validation, VM-owned authorized handle registryDeployment signature/integrity policy and compiler-generated native method metadata
4. Process pipingargv execution, stdin file, bounded capture, multi-stage pipe, timeout, exit propagationAnnotation parser/runtime policy and richer stream-routing semantics
5. systemdHardened unit rendering, explicit install/enable/start, status/restart/stop/disable, transactional failed-install recovery, one-level replacement rollbackGenerated service manifests and integration tests against a real systemd host
6. VM dispatchAppended open/call/close opcodes, scalar/String marshalling, host authorization, stale-handle checks, NOT_HANDLED and VM-composite interpreter fallbackExplicit composite object/array ownership ABI, output parity, concurrency stress tests
7. ConformanceLoader fixtures, versioned typed ABI, native-to-SLeeLa callback fixture, source compile fixture, VM dispatch/fallback/callback, method metadata, generated-plugin AOT, and concurrency targets (in make test and Linux CI), service recovery coverageRun local test suites and complete compiler/VM/loader/process/systemd matrix, sanitizer and cross-ABI coverage

Security invariants

Tests

python3 -m unittest discover -s tools/tests -p 'test_native_drive.py' -v make -C impl test-native-call make -C impl test-native-concurrency # Linux, macOS, and Windows/MinGW sh -n bin/sleelactl

The focused test-native-concurrency target is also opt-in and is not part of the default test aggregate or CI. On Linux and macOS it builds a typed plugin that blocks inside slvm_native_module_call until the harness writes a release marker; on Windows/MinGW it uses named Win32 events and Win32 threads for the same handshake. A second host thread requests slvm_native_module_close while that call is held. The harness asserts that the call remains active and close has not completed during the observation window, then releases the plugin, joins both threads, checks the typed result, and verifies that the closed generation-checked handle is stale. The Windows fixture is selected by the Makefile only for the Windows platform; it does not use POSIX headers, pthreads, or -fPIC. This test covers native-module call/close serialization only; it does not relax the separate host requirement to quiesce all external VM callers before slvm_free().

The focused test-native-call target is opt-in and is not part of the default test aggregate. It substitutes the built fixture's exact path into the SLeeLa source fixture, compiles that source to a persistent Core artifact, loads the artifact through slvm_load_file, installs a host-owned exact-path authorization callback, and executes the program. The source program invokes the plugin's twice method, calls the SLeeLa Callback::receive method synchronously from native code, exercises bytecode fallback for fallback, closes the module, and returns the combined result (105). The harness then reloads the same artifact with a deliberately mismatched module allowlist and asserts that source-level nativeOpen fails closed. A third run authorizes the module but omits callback-name authorization, proving the plugin cannot invoke Callback::receive unless the host explicitly allows it. The harness also creates fresh VMs and exercises both public host-side entrypoints: slvm_native_module_run and slvm_native_module_call. The fixture attempts the same callback through each entrypoint without an active source-level nativeCall frame; both attempts must be rejected with the exact VM error native callback requires an active source-level nativeCall context. The run-style entrypoint reports the plugin's observed refusal through its integer result, while the typed-call entrypoint must return SL_NATIVE_PLUGIN_ERROR. Both cases close the module and free the VM. The callback-bytecode regression also attempts nativeOpen and nativeClose from inside an authorized callback; both must fail with the same explicit reentrancy diagnostic before touching the module table. A callback-policy test deliberately calls each public host lifecycle API (open/run/call/close) from the callback authorizer while the plugin entrypoint is active; all four must return SL_NATIVE_REENTRANT_CALL immediately, and the approved callback must still complete normally.

A further reentrancy regression constructs a small VM program through the public core bytecode-building API. Its outer OP_NATIVE_CALL enters the fixture, which invokes Callback::receive_nested(value, module); that callback attempts a second nativeCall while callback execution is active. The VM must reject the nested operation with native callback cannot recursively invoke native-module operations. The fixture converts the rejected callback into an integer sentinel so the harness can assert the exact VM diagnostic without masking it with an outer plugin error. After the callback unwinds, the harness calls the same module again and checks the expected typed result, proving the module lock was released and the original handle remains usable. This case is deliberately a direct bytecode/API test rather than a compiler-produced source artifact; the existing positive callback path remains covered by the compiler-produced artifact.

The Native Drive GitHub Actions workflow is workflow_dispatch-only; no workflow is automatically started by pushes. This end-to-end target has been added but is not claimed as run until explicitly executed.