Home /Modules/impl/AT-NATIVE-DISPATCH.md

AT-NATIVE-DISPATCH.md

impl/native-drive/AT-NATIVE-DISPATCH.md · 260 line(s) · markdown

Automatic VM dispatch for @native methods

Status (2026-11-11, master): per-method ABI-v1 plugin emission, artifact-v4 method metadata, explicit host binding, and ordinary-call dispatch are on master. The generated plugin is compiled, loaded, bound, and dispatched by make -C impl test-native-method-aot, which runs with the other native targets in make test and in the Native Drive CI workflow on Linux. macOS/Windows execution of these VM targets is still outstanding.

The compiler now persists method-level @native identity in artifact format v4. The VM records annotation/static-or-instance flags, the method ABI version, a canonical signature, and a deterministic FNV-1a-64 signature identity. Version 1-3 artifacts remain loadable with all native metadata disabled. The metadata is descriptive only: it does not load, authorize, bind, or invoke native code.

The public metadata registration/readback API is slvm_set_func_native_metadata() / slvm_get_func_native_metadata(). The local regression target is make -C impl test-native-method-metadata; it checks compiler-emitted static method metadata, v4 save/load, ordinary-function zero metadata, digest corruption rejection, and v3 interpreted-only compatibility. It runs in make test and in the Native Drive CI workflow (Linux).

Authorized host binding and ordinary-call dispatch now use the explicit APIs slvm_bind_native_method() and slvm_unbind_native_method(). A binding is accepted only for an annotated static method and an already-open, authorized module handle. The plugin export key is the exact canonical signature stored in artifact metadata. Bound modules cannot be closed until all method bindings are removed. The ordinary OP_CALL path invokes that export only when a binding exists and the arguments fit the scalar/UTF-8 ABI; NOT_HANDLED and unsupported VM-owned arguments retain the original bytecode path. Plugin, stale-handle, and ABI failures remain errors.

The metadata test verifies unbound bytecode behavior and ordinary source-level calls reaching an authorized plugin export. test-native-method-aot compiles the emitter output into a shared library, loads it under an exact-path authorizer, binds every annotated method, and checks int64, double, bool, and UTF-8 dispatch plus NOT_HANDLED, composite-argument, handled-error, and denied-path behavior. test-native-concurrency covers call/close ordering. Cross-platform (macOS/Windows) VM execution and race stress remain outstanding.

The fixtures declare static void main() (required by the semantic checker) and put the asserted value in static int check(); harnesses select that entry with slvm_load_file_with_entry(path, "Class::check"). Static field initialisers are emitted into main, so a fixture that relies on static state initialises it inside check().

This document specifies the connection between the source-level @native annotation and the VM native-module dispatcher. Annotated methods are not automatically compiled, loaded, or bound by sleela run; the host does that explicitly (for example sleelactl build native-drive file.sleela -o file.so followed by an authorized open and slvm_bind_native_method()).

1. Existing mechanisms (keep them distinct)

SLeeLa currently has two relevant but separate paths:

  1. Source AOT request: @native is parsed on a method and inspected by sleela aot --list/--emit. The --emit mode now produces standalone per-method C plugin source with exact canonical-signature dispatch. It does not compile, load, authorize, or bind the plugin in the VM.
  2. Explicit native module calls: syntax 3.0 nativeOpen, nativeCall, and nativeClose lower to append-only VM opcodes. The host authorizes module paths; the typed ABI transports null, bool, int64, double, and UTF-8 String values. A missing/not-handled native method can use a same-name, same-arity SLeeLa bytecode fallback. Authorization failures, stale handles, ABI errors, and handled plugin failures remain errors.

The second path is real VM dispatch, but it is not automatic dispatch for ordinary calls to an @native-annotated SLeeLa method.

2. Required end state

For an annotated method such as:

class MathEngine {
    @native
    static int sum(int a, int b) {
        return a + b;
    }
}

the compiler/runtime pipeline must be able to:

3. Proposed dispatch contract

Compiler metadata

Each annotated method needs a stable identity independent of source line numbers and declaration order. The initial identity should include:

The compiler must reject duplicate or ambiguous native identities. Metadata is descriptive only: it does not authorize a module path or grant execution capability.

Host binding

The embedding host must explicitly bind a generated native module to one or more compiler-emitted method identities after applying its own authorization and integrity policy. Binding should be immutable for the duration of a VM execution epoch, or changed only through a synchronized host API.

No source annotation may cause the VM to search arbitrary directories, load a library, spawn a process, or trust a caller-supplied path.

Ordinary call dispatch

The compiler may continue emitting the normal bytecode call as the canonical fallback body. At runtime, a call to a method marked native should follow this order:

  1. Resolve the function identity and verify the binding belongs to this VM.
  2. If no host binding exists, execute the original bytecode body.
  3. Marshal only values supported by the negotiated native ABI.
  4. If an argument is a VM-owned composite that the ABI does not represent, execute the original bytecode body without exposing VM pointers or handles.
  5. Invoke the native entry point under the existing module lifetime/locking discipline.
  6. If the entry point explicitly returns NOT_HANDLED, execute the original bytecode body.
  7. If the entry point succeeds, validate and copy the result into VM-owned storage before returning it to the caller.
  8. If authorization, handle validation, ABI negotiation, or native execution fails, return an error. Never silently reinterpret a handled failure as permission to retry in bytecode.

Fallback is not a general exception handler. It is permitted only when the native implementation is absent, explicitly declines the method, or the argument shape is outside the negotiated ABI before native execution begins.

4. ABI and ownership rules

5. Implementation sequence

  1. Artifact metadata — implemented: version-4 metadata is append-only, validated, and preserved through save/load; versions 1-3 remain interpreted-only.
  2. Per-method code generation — source emitter implemented: sleela aot --emit emits a standalone C plugin with one independently addressable method body per annotated static method. The plugin exports the typed ABI v1 loader symbols and dispatches exact canonical signature strings. Supported parameter and return types are int, double, boolean, and length-delimited String; void returns are represented as NULL. String values can pass through parameters, locals, assignments, and returns. String literals and string operators are excluded. Division/remainder, method-to-method calls, print side effects, objects, arrays, and instance methods are also refused. The AOT command runs the normal semantic analyzer (without requiring an executable entry point) before eligibility reporting or emission. It also loads referenced library units, so imports and cross-unit type errors are checked consistently with the normal compiler path. The generated source is not compiled or loaded automatically; the host must compile it, authorize its resulting library, and bind it explicitly. The opt-in target make -C impl test-native-method-aot generates and builds the shared plugin, compiles a matching SLeeLa artifact, then loads the plugin through the host authorizer and exercises ordinary OP_CALL for int64, double, boolean, and UTF-8 methods, including bytecode fallback before native binding. A second artifact binds an annotated method whose canonical signature is absent from that plugin, forcing the plugin to return SL_NATIVE_CALL_NOT_HANDLED; its bytecode body increments a persistent counter on each call, and the smoke harness requires the two-call result 12 (rather than 24 if fallback executes twice per call). It also rejects semantically invalid logical, conditional, negation, and loop/if conditions, while accepting a valid library-only source with no main. The committed SHA-256 manifest is refreshed for the implementation; run make -C impl check-manifest to verify a local checkout before building. The target is outside CI and has not yet been run in this environment.
  3. Host binding API — implemented: slvm_bind_native_method() accepts only annotated static methods and a module handle already opened through the host's authorization policy. slvm_unbind_native_method() is required before a bound module can be closed. Binding never loads or authorizes a path.
  4. VM ordinary-call fast path — implemented: OP_CALL tries the bound native entry for scalar/UTF-8 arguments, copies results while the module lock is held, and preserves bytecode fallback for missing bindings, unsupported values, and NOT_HANDLED. Native failures do not fall back.
  5. Conformance — initial tests added: the focused target checks native ordinary-call dispatch, unbound bytecode execution, explicit SL_NATIVE_CALL_NOT_HANDLED fallback exactly once per call, module-close protection, metadata round trips, digest corruption, and v3 compatibility. The NOT_HANDLED test uses two calls to a stateful bytecode body so duplicate fallback is observable. The same target also binds array- and struct-typed methods to a sentinel plugin: the expected bytecode result proves neither VM-owned composite reaches the typed ABI. A separate handled-error case requires the plugin failure to propagate rather than invoke bytecode, and a deny-all authorizer verifies that an annotated method cannot open or bind a denied module and still executes its unbound bytecode body. These targets pass on Linux and are gated in CI; macOS/Windows execution, race stress, and the remaining failure matrix still need to run.

6. Acceptance tests

7. Current conclusion

The existing OP_NATIVE_CALL path is a foundation to reuse, not the missing feature itself. Versioned method metadata, per-method native C source emission, host-owned authorized binding, and ordinary OP_CALL dispatch are now present. The source emitter, generated-plugin compile/load, and bound dispatch are verified on Linux by test-native-method-aot. Normal SLeeLa bytecode remains the default: native code runs only after it is explicitly compiled, authorized, and bound by the host, and macOS/Windows conformance is still to be run.

MEARVK LLC — 2026.