SLeeLa Performance Assessment — Basic Data Types and RAM Access
Assessment ID: SCS-PERF-001
Edition: 1.0
Recorded: 2026-11-06
Repository: mearvk/SLeeLa · Branch: master
Language baseline: SLeeLa Syntax 3.0 · C11 · C++17 · Java 21+
Status: CONFORMANCE RETRY IN PROGRESS — PARTIAL; no performance measurements available
1. Purpose
This assessment defines a reproducible comparison of computation speed and memory-access performance across:
- The Basic SLeeLa Native VM in
impl/core. - All ten Standard/think-tank VM generations,
idea-vm-001throughidea-vm-010. - All eleven runnable Premium VMs, SLVM/1 through SLVM/11.
- Native C11 and C++17 implementations.
- Java 21 or later on the JVM.
The purpose is to measure real execution, not infer speed from version number, feature count, source-code size, or architectural description. No performance values or rankings are asserted in this protocol. Results must be generated by the benchmark runners and retained with their environment metadata before conclusions are drawn.
2. Architecture and fair-comparison rules
2.1 Basic Native VM
The baseline is the Native VM implemented by impl/core, invoked through the repository's normal SLeeLa compiler/runtime path. It is the default SLeeLa execution engine.
2.2 Standard VMs: generations 1–10
The Standard progression is:
| Generation | Registry identity | Documented addition |
|---|---|---|
| 1 | idea-vm-001 | Ground-floor configuration, report, status |
| 2 | idea-vm-002 | Memory hardening |
| 3 | idea-vm-003 | Typed results |
| 4 | idea-vm-004 | Telemetry |
| 5 | idea-vm-005 | Garbage-collection quality |
| 6 | idea-vm-006 | Dispatch profiler |
| 7 | idea-vm-007 | Capability enforcement |
| 8 | idea-vm-008 | Platform parity |
| 9 | idea-vm-009 | Audit trail |
| 10 | idea-vm-010 | Consolidated runtime |
Execution qualification: the repository's VM user guide currently identifies Standard generations 1–10 as a layered think-tank progression and explicitly says they are not yet bootable through the Premium-style sleela vm run command. They must therefore not be represented as ten independently executing instruction engines in this report.
The Standard generations remain in scope. Their assessment is divided into:
- Layer/control-plane cost: measure the initialization, configuration, memory checks, typed-result handling, telemetry, GC observation, profiling, capability checks, platform checks, audit writes, and consolidated-runtime operations that are actually exposed by their source APIs.
- Incremental cost: where a generation adds a measurable operation, compare that operation with the immediately preceding generation and with the uninstrumented Core baseline.
- Whole-program execution: mark as NOT RUN / NOT BOOTABLE until the repository supplies a supported executable path for that Standard generation. Do not substitute the Native VM and label it as a Standard VM result.
If a Standard layer has no independently callable operation or faithful native harness, record it as unavailable rather than inventing a measurement.
2.3 Premium VMs: generations 1–11
The Premium SLVM runners each link the shared Core archive and perform their own grade-specific admission/validation before running a compiled SLeeLa artifact. Their distinguishing feature validators are part of the measured architecture.
For every Premium generation, report two distinct timings:
- Steady-state workload: elapsed time to execute the same precompiled workload after the runner has started and admitted the VM.
- Cold invocation: process start, admission, artifact load, execution, and shutdown as one end-to-end interval.
The steady-state result is the primary computation comparison. Cold invocation is a separate operational metric. Do not attribute a difference in admission time to arithmetic throughput.
2.4 Native language comparators
| Comparator | Required language/runtime |
|---|---|
| C | C11, optimized native build |
| C++ | C++17, optimized native build |
| Java | Java 21+; record exact JDK vendor and version |
C and C++ are native baselines, not VM implementations. Java results include JVM/JIT behavior and must use warm-up and steady-state measurement. These results are comparative reference points, not proof that the languages have identical semantics or implementation strategies.
3. Basic data-type computation suite
Each implementation must run the same defined inputs, iteration counts, dependency patterns, and observable result checks. Inputs and outputs are fixed and included in the result record. Use integer-safe workloads with defined overflow behavior; do not rely on signed overflow in C or C++.
| ID | Data type / operation | Primary measurement |
|---|---|---|
| DT-01 | Boolean operations and conditional selection | operations/second |
| DT-02 | 8/16/32-bit integer arithmetic | operations/second |
| DT-03 | 64-bit signed and unsigned integer arithmetic | operations/second |
| DT-04 | 128-bit integer arithmetic where natively supported | operations/second; support recorded |
| DT-05 | Single-precision floating-point arithmetic | operations/second |
| DT-06 | Double-precision floating-point arithmetic | operations/second |
| DT-07 | Integer comparisons, branches, and loops | iterations/second |
| DT-08 | Integer division and remainder | operations/second |
| DT-09 | Function calls and return-value handling | calls/second |
| DT-10 | Mixed-type arithmetic/conversion where supported | operations/second |
| DT-11 | Array element reads and writes | elements/second |
| DT-12 | String construction, comparison, and traversal | operations/second or bytes/second, with exact semantics stated |
The suite must distinguish arithmetic throughput from dispatch overhead. Include both dependency-chain tests (each result depends on the previous result) and independent-operation batches (multiple independent values per iteration). This helps reveal latency versus throughput and reduces the risk that compiler/JIT optimization removes the work.
For floating point, report precision mode and validation tolerance. For 128-bit integers, record whether the implementation uses a native 128-bit representation, a library representation, or a narrower fallback. Do not compare unlike implementations without disclosing that distinction.
4. RAM and memory-access suite
RAM testing is separated into latency, bandwidth, cache behavior, and allocation/reclamation. A single generic “RAM speed” number is insufficient.
| ID | Test | Measurement |
|---|---|---|
| MEM-01 | Sequential read | GiB/s |
| MEM-02 | Sequential write | GiB/s |
| MEM-03 | Read-modify-write stream | GiB/s and operations/second |
| MEM-04 | Sequential copy | GiB/s |
| MEM-05 | Random read, fixed working set | ns/access and accesses/second |
| MEM-06 | Dependent pointer chase over randomized permutation | ns/load; latency |
| MEM-07 | Random write | ns/store and operations/second |
| MEM-08 | Strided access | ns/access and effective GiB/s |
| MEM-09 | Working-set sweep | latency/bandwidth by allocation size |
| MEM-10 | Allocation and release | allocations/second and ns/allocation |
| MEM-11 | Array growth / append | ns/element and total throughput |
| MEM-12 | GC allocation/reclamation pressure, where applicable | allocation rate, pause time, reclaimed bytes |
4.1 Working-set sizes
Use a documented geometric size sweep that includes small cache-resident sets and sets larger than the last-level cache and physical memory only where safely available. Suggested default sizes are 4 KiB, 32 KiB, 256 KiB, 2 MiB, 16 MiB, 64 MiB, and 256 MiB. The runner must detect available memory and cap the largest allocation conservatively; it must never intentionally exhaust the host.
Record actual allocation size, page size when available, and whether the test data fits in the relevant cache. Do not label a size “L1”, “L2”, “L3”, or “RAM” unless the host's cache topology supports that label.
4.2 Latency versus bandwidth
- Sequential streaming primarily measures bandwidth and prefetch effectiveness.
- Random independent accesses measure a mix of memory-level parallelism and latency.
- Dependent pointer chasing suppresses parallel look-ahead and is the primary memory-latency probe.
- Strided tests reveal cache-line utilization and access-pattern sensitivity.
Report these as separate metrics. Never convert a bandwidth result into a latency claim.
5. Benchmark matrix
The target coverage is:
| Target family | Members | Computation | RAM access | Startup/admission |
|---|---|---|---|---|
| Basic Native VM | 1 | Full suite | Full suite | Yes |
| Standard VMs | 1–10 | Layer operations only until bootable | Layer operations only until bootable | Per-layer init where callable |
| Premium SLVM | 1–11 | Full suite | Full suite | Yes, separately |
| C11 | 1 | Full suite | Full suite | Process launch separately |
| C++17 | 1 | Full suite | Full suite | Process launch separately |
| Java 21+ | 1 | Full suite | Full suite | JVM cold start separately |
This is 23 primary target entries (1 Basic + 10 Standard + 11 Premium) plus three language comparators. Standard VM whole-program results are explicitly gated on bootability and cannot be silently counted as completed.
6. Execution protocol
- Record repository commit SHA, branch, dirty-tree state, compiler versions, JDK vendor/version, OS/kernel, CPU model, logical/physical core counts, RAM capacity, memory speed if available, cache sizes, and power mode.
- Use one physical host for a direct within-host comparison. Run a separate complete matrix for each additional OS/host; never compare numbers from unlike hardware as though the runtime caused the difference.
- Build native C and C++ with documented release optimization and warnings. Record the exact compiler command and flags. Build SLeeLa Core and every Premium runner from the same repository commit and compatible release configuration.
- Compile SLeeLa source once per workload to the same artifact format and use that artifact for the Basic VM and every Premium runner. Record compiler version, artifact hash, and compile time separately. Compilation time is not VM execution time.
- Compile Java comparators once. Warm up the JVM until the measured workload stabilizes or a declared warm-up ceiling is reached. Record warm-up iterations and JVM flags. Report cold JVM startup separately.
- Perform at least five measured repetitions after warm-up; use a longer run duration for short operations. Report median, minimum, maximum, and p95 where sample count permits. Include dispersion (standard deviation or median absolute deviation).
- Alternate or randomize target execution order to reduce thermal, frequency, and background-load bias. Record CPU governor/power plan and avoid concurrent benchmark workloads.
- Validate the final result/checksum of every workload. A run with a wrong result, missing workload, runtime error, or unsupported operation is not a performance result.
- Keep process startup, VM admission, artifact loading, steady-state execution, and shutdown as separate phases wherever instrumentation allows.
- Repeat the full suite on each target OS before making cross-platform claims. The initial reference run may use Ubuntu, but it must be labeled Linux-only until macOS and Windows runs complete.
7. Measurement and reporting rules
- Primary throughput unit: operations/second or iterations/second.
- Memory bandwidth: decimal bytes/second and GiB/s, with the conversion stated.
- Memory latency: nanoseconds/access.
- Time: monotonic high-resolution clock; use the same measurement boundary within each language/runtime.
- Normalize only as a secondary view. Preserve raw elapsed time, operation count, and bytes transferred.
- Report ratios against C11, C++17, Java 21+, and the Basic Native VM separately. Do not combine all comparators into one synthetic score.
- Provide both absolute values and relative ratios. A faster/slower percentage must identify its denominator.
- Flag outliers and disclose exclusions; never silently discard inconvenient runs.
- No ranking is valid unless the same workload, input, hardware, and measurement boundary were used.
7.1 Compiler/JIT and optimization controls
C/C++ builds must prevent dead-code elimination by validating a checksum observable outside the timed loop. Avoid undefined behavior and ensure allocation alignment is comparable. Record optimization flags, LTO state, target architecture flags, and allocator.
Java must record JDK and VM flags, warm-up policy, garbage collector, heap bounds, and whether compilation reached steady state. Report GC activity and pauses for memory-pressure workloads. A Java result before JIT warm-up must not be compared with a steady-state native result without an explicit cold-start label.
SLeeLa must use the same compiler/artifact and runtime build for the Basic and Premium comparisons. Do not include compilation in steady-state execution time. Record Core build flags and Premium admission checks.
8. Standard VM layer accounting
For each Standard generation, record the following where its source exposes a callable operation:
- initialization/configuration cost;
- per-operation cost of the generation's added layer;
- memory footprint of the layer's state;
- telemetry/profiling/audit event cost where applicable;
- GC observation/collection coordination cost where applicable;
- capability-check cost where applicable;
- consolidated-runtime polling and shutdown cost.
Use a paired baseline: run the same underlying operation with the layer disabled/absent and enabled/present, with identical inputs. Report incremental time and memory, not only total time. If a layer changes the semantics or amount of work, the pair is not directly comparable and must be described.
Until a supported executable Standard-VM route exists, report the whole-program execution field as NOT RUN — Standard generation not bootable through the documented runner. Do not claim that a source-level manifest or a selection record is an executing VM.
9. Results schema and artifacts
The benchmark implementation should write machine-readable JSON and a human-readable Markdown summary. Each result record should contain at least:
{
"suite": "SCS-PERF-001",
"repository_commit": "<commit-sha>",
"target": "basic-native-vm",
"target_version": "impl/core",
"language": "SLeeLa",
"workload": "DT-02",
"status": "PASS",
"host": {
"os": "<os>",
"cpu": "<cpu-model>",
"logical_cores": 0,
"ram_bytes": 0
},
"build": {
"compiler": "<compiler-and-version>",
"flags": "<exact-flags>",
"artifact_sha256": "<hash-or-null>"
},
"measurement": {
"warmups": 0,
"repetitions": 5,
"operation_count": 0,
"elapsed_ns_median": 0,
"elapsed_ns_min": 0,
"elapsed_ns_max": 0,
"throughput_per_second": 0,
"checksum": "<validated-result>"
}
}
Zeroes above are schema placeholders, not measurements. Unsupported/not-bootable results must use a status such as NOT_RUN and a reason; they must not be encoded as zero performance.
Recommended artifacts:
comparison-performance-results.jsoncomparison-performance-summary.md- per-target raw samples and environment manifest
- exact build/run logs
- checksums of benchmark sources and compiled artifacts
10. CI and execution policy
Correctness and performance must be separate CI concerns.
- CI should compile and validate the benchmark programs and verify result checksums.
- Timing runs should be manually dispatched or run on controlled, stable benchmark hosts; ordinary shared GitHub-hosted runners are too variable for release-grade speed rankings.
- CI may retain timing observations as informational data, but noisy timing variance must not fail correctness CI.
- A benchmark report is PASS only when all required workloads for its declared scope completed, their results validated, and raw results plus environment metadata were retained.
- A partially completed target matrix is PARTIAL, not PASS.
- A design-only document, unexecuted runner, or CI workflow that merely compiles tests is NOT A MEASUREMENT.
11. Current assessment state
| Area | State |
|---|---|
| Basic Native VM target identified | Defined |
| Standard VM generations 1–10 identified | Defined; whole-program execution not currently documented as bootable |
| Premium SLVM generations 1–11 identified | Defined; each uses shared Core plus generation-specific admission |
| C11 comparator | Defined |
| C++17 comparator | Defined |
| Java 21+ comparator | Defined |
| Basic data-type workload definitions | Defined |
| RAM latency/bandwidth workload definitions | Defined |
| Reproducible execution protocol | Defined |
| Benchmark harness implementation | Partial CI matrix runner exists; full performance harness not implemented |
| CI correctness checks | Partial: Ubuntu/macOS language suites passed; Windows setup and VM matrix failed (details in §13) |
| Performance ranking | Not established |
13. CI execution record — run 42337765442
Workflow: Complete SLeeLa VM and Language Conformance
Workflow file: .github/workflows/complete-vm-conformance.yml
Commit tested: bd7eb049b6158bf129564aea43b58bfeeed993cd
Run status: completed · Conclusion: failure
Recorded by GitHub: 2026-11-07 (UTC)
This run was a correctness/build conformance attempt, not a performance benchmark. Its results must not be interpreted as execution-speed, RAM-bandwidth, or latency measurements.
13.1 Language comparator matrix
| Runner | C11 / C++17 / Java 21+ job | Outcome | Evidence |
|---|---|---|---|
| Ubuntu latest | Completed | PASS | Toolchain setup and shared comparison suites succeeded |
| macOS latest | Completed | PASS | Java 21 setup and shared comparison suites succeeded |
| Windows latest | Incomplete | FAIL — environment setup | MSYS2 could not locate package mingw-w64-ucrt-x86_64-gcc-c++; language suites were skipped |
The Ubuntu and macOS results establish that the configured comparison suites completed successfully on those hosted runners for this commit. They do not establish performance rankings, and they do not establish Windows compatibility.
13.2 Basic Native and Premium VM matrix
| Check | Outcome | Interpretation |
|---|---|---|
| Integrity manifest generation | PASS | Checkout-specific manifest generated |
| Basic Native VM/compiler build | PASS | Build completed |
| Premium SLVM 1–11 builds | PASS | All eleven build targets completed |
| Basic Native VM output validation | FAIL — runner parsing | Program emitted expected values 42, 15, and hello, followed by runtime/security diagnostics; runner treated the extra lines as unexpected output |
| Premium VM execution checks | NOT RUN | Matrix runner stopped at the Basic Native output assertion |
| Premium VM admission self-checks | SKIPPED | Dependent step did not execute |
| Standard VM Modium checks | SKIPPED | Dependent step did not execute |
| Standard VM inventory | PASS — 10/10 registered | Registration only; no Standard VM execution is claimed |
The Basic Native failure is presently a test-harness output-parsing defect, not evidence that the program produced incorrect workload values. The runner should validate the expected result lines while allowing or separately classifying known runtime, memory-manager, and security diagnostics. It should retain both the raw output and the parsed result.
The eleven Premium VM binaries were built, but this run provides no execution result for SLVM/1 through SLVM/11. Build success must not be reported as runtime conformance.
13.3 Required corrective actions
- Update
comparison/tests/run_all_vm_matrix.pyto parse the expected result lines without rejecting known diagnostic output. Preserve raw stdout/stderr and fail only on missing, malformed, or incorrect result values or a nonzero process exit. - Correct the Windows MSYS2 package selection in the workflow. Confirm the package name against the selected UCRT64 repository and verify that both the C and C++ compilers are installed before running the comparison suites.
- Rerun the complete workflow after those corrections.
- Confirm all Premium VM execution checks and admission self-checks actually run and pass; do not infer their status from successful builds.
- Confirm the Standard VM registry/Modium checks run. Continue to mark Standard generations 1–10 as registered but not independently bootable unless an executable route is added and tested.
- Implement and run the separate controlled performance harness defined in this document. The conformance workflow does not produce the DT-* or MEM-* timing measurements specified in §§3–4.
13.4 Assessment after this CI run
The CI run provides partial correctness/build evidence: Ubuntu and macOS language comparator jobs passed, all Premium VM build targets compiled, and all ten Standard VM registrations were detected. Windows toolchain setup failed. The VM execution matrix did not complete because its Basic Native output parser rejected diagnostic lines after the expected values; consequently, no Premium VM execution result is available from this run.
Performance status remains NOT MEASURED. No throughput, startup, RAM bandwidth, memory latency, allocation-rate, or performance-ranking claim is supported by this CI run.
12. Repository references
- Standard VM user guide
- Standard VM registry
- Standard VM version/runtime definitions
- Standard VM runtime definition
- Premium VM source tree
- SLeeLa Core public runtime interface
- Comparison conformance workflow
- Iteration 003 native/Java/SLeeLa comparison record
- Iteration 003 SLeeLa array/string VM extension
Conclusion: SCS-PERF-001 establishes the scope and measurement standard for Basic SLeeLa, Standard generations 1–10, Premium generations 1–11, C11, C++17, and Java 21+. CI run 42337765442 completed with a failure and provides partial build/conformance evidence only; it does not contain measured performance results. Correct the VM output parser and Windows toolchain setup, rerun the matrix, and implement controlled DT-/MEM-suite benchmarking before publishing performance results. Do not promote unbootable Standard layers, unexecuted Premium VMs, or unmeasured targets to PASS.
14. Conformance retry and benchmark-readiness audit — 2026-11-06
Workflow: Complete SLeeLa VM and Language Conformance
Action: Re-ran failed jobs using GitHub Actions' failed-job retry. Successful jobs were not rerun.
Commit under test: bd7eb049b6158bf129564aea43b58bfeeed993cd
Observation: the retry was still running when this record was written. The states below are a live snapshot, not a final workflow conclusion.
14.1 Retry status snapshot
| Job | Retry state observed | Conclusion at snapshot |
|---|---|---|
| C11 / C++17 / Java 21+ — Ubuntu | Completed | PASS |
| C11 / C++17 / Java 21+ — macOS | Completed | PASS |
| C11 / C++17 / Java 21+ — Windows | Completed | FAIL — MSYS2 package unavailable |
| Basic Native VM + Premium SLVM 1–11 | Basic Native VM/compiler build in progress; later steps not started | PENDING |
The retry confirms the Linux and macOS language-comparator correctness suites again. It does not yet resolve the Windows package-installation issue or establish the Basic/Premium VM execution outcome. The run must be inspected again after the remaining jobs finish; no final overall PASS is claimed here.
14.2 Benchmark implementation audit
The current workflow .github/workflows/complete-vm-conformance.yml performs toolchain installation, correctness-vector execution, VM builds, VM output validation, Premium admission self-checks, and Standard VM manifest checks. It does not execute a timed performance workload.
The inspected comparison runners are correctness-only:
comparison/tests/run_native_java.pycompiles C11 and C++17 and runs Java, then compares the three expected output lines42,15, andhello.comparison/tests/run_all_vm_matrix.pychecks the same three output values for the Basic Native VM and Premium runners and inventories ten Standard VM manifests. It does not record elapsed time, operation count, memory bandwidth, memory latency, or allocation rate.- The shared SLeeLa source
comparison/tests/sleela/common_vectors.sleelais a short arithmetic/loop/string correctness smoke test, not a benchmark workload.
A repository search for benchmark timing, pointer-chase, memory-bandwidth, and operations-per-second implementations found no executable performance harness in the comparison area. Accordingly, no speed or RAM figures can be reported from this CI run. This is an implementation gap, not a zero-speed result and not a failed performance result.
14.3 Measurements required before a performance claim
The next benchmark implementation must execute equivalent, checksum-validated workloads for the Basic Native VM, all eleven runnable Premium VMs, C11, C++17, and Java 21+. Standard generations 1–10 remain layer-accounting targets only until they have a supported bootable execution path.
| Measurement group | Required tests | Required output |
|---|---|---|
| Integer types | Boolean; 8/16/32/64-bit signed and unsigned arithmetic; 128-bit support where available; division/remainder; comparisons | ns/op, ops/s, checksum, support/representation |
| Floating point | float32 and float64 arithmetic, conversions, comparisons | ns/op, ops/s, precision mode, validation tolerance |
| Control flow | dependent and independent loops, branches, function calls | ns/iteration, iterations/s, calls/s |
| Arrays and strings | sequential/random element access, read/write, string traversal/comparison/construction | elements/s, bytes/s, checksum |
| Memory bandwidth | sequential read, sequential write, copy, read-modify-write | bytes/s and GiB/s, bytes transferred |
| Memory latency | randomized dependent pointer chase, random read/write, strided access | ns/access, accesses/s, stride and working-set size |
| Cache/working set | 4 KiB through safely capped large allocations | latency and bandwidth per allocation size; detected cache topology |
| Allocation/GC | allocation/free, array growth, managed allocation and reclamation | allocations/s, ns/allocation, peak bytes, GC pauses and reclaimed bytes |
| Startup and admission | cold process launch, VM admission, artifact load, shutdown | separate phase timings; not mixed with steady-state throughput |
Every measured target must record the host CPU and architecture, OS/kernel, RAM capacity, available cache topology, compiler/JDK versions, exact optimization/runtime flags, allocator/GC, workload size, warm-up count, at least five measured repetitions, median/min/max and dispersion, raw samples, checksum, and artifact hash. Java requires warm-up and a separate cold-start measurement. C/C++ must avoid undefined behavior and dead-code elimination. SLeeLa Basic and Premium comparisons must use the same compiled workload artifact where the runtime interface permits it.
14.4 Current performance conclusion
- Basic data-type speed: NOT MEASURED.
- Integer and floating-point throughput: NOT MEASURED.
- RAM sequential bandwidth: NOT MEASURED.
- Random memory-access latency: NOT MEASURED.
- Cache/working-set behavior: NOT MEASURED.
- Allocation and GC behavior: NOT MEASURED.
- Basic Native vs Premium VM speed: NOT MEASURED.
- SLeeLa vs C11 / C++17 / Java 21+ ranking: NOT ESTABLISHED.
- Standard VM generations 1–10: registration/inventory only; no whole-program execution or speed result.
No timing values, rankings, or ratios have been inferred from correctness results or build completion. The next CI change must add a dedicated benchmark workflow and executable benchmark harness; ordinary hosted-runner timings should be labeled informational unless run on controlled, repeatable hardware.
14.5 Retry update — Windows failure confirmed; workflow correction committed
The Windows job subsequently completed with the same setup failure. Its log confirms:
error: target not found: mingw-w64-ucrt-x86_64-gcc-c++
The language-comparison step was skipped, and no Windows comparison result artifacts were produced. This is an environment/package-selection failure, not a C11, C++17, or Java test failure.
A correction was committed to .github/workflows/complete-vm-conformance.yml at commit acf089533b6eeebaea0038ebd557c63ede5d6120: the nonexistent mingw-w64-ucrt-x86_64-gcc-c++ package was removed. The repository's separate Comparison Standards workflow already uses mingw-w64-ucrt-x86_64-gcc as the UCRT64 compiler package for both C and C++, and its Windows comparator job has previously passed. The corrected complete-conformance workflow still requires a fresh successful Windows run before Windows qualification can be claimed.
At the time of this update, the VM job for run 42337765442 remained in the Basic Native VM/compiler build step. Premium builds, VM execution, Premium self-checks, and Standard manifest checks had not yet been reported as completed. No outcome is inferred for those pending steps.
14.6 Completed VM retry — build success; output-validator defect isolated
Workflow: Complete SLeeLa VM and Language Conformance — prior retry
VM job: 128300544777
Result: job failed at the VM matrix output-validation step, after all Premium VM builds had completed.
| VM stage | Observed result | Interpretation |
|---|---|---|
| Basic Native VM/compiler build | PASS | Build completed |
| Premium SLVM 1–11 builds | PASS — 11/11 built | Every Premium build step completed |
| Standard VM inventory | PASS — 10/10 registered | Inventory reported; execution remains gated |
| Basic Native VM workload | Program values present: 42, 15, hello | Runtime also emitted memory-manager/security diagnostics |
| VM matrix validator | FAIL — validator defect | It compared the final three combined-output lines, which were diagnostics rather than program values |
| Premium VM workload execution | NOT RUN | The runner stopped at the Basic Native VM validation failure |
| Premium admission self-checks | SKIPPED | Earlier step failed |
| Standard VM Modium manifest check | SKIPPED | Earlier step failed |
The captured Basic Native VM output contains the three expected program values, followed by legitimate memory-manager statistics and security verification messages. This is not evidence of a VM computation failure. The original runner's lines[-3:] comparison incorrectly treated the final diagnostics as the program result.
Correction committed: comparison/tests/run_all_vm_matrix.py now separates a small, explicit set of known runtime/security diagnostic prefixes from program output, requires the remaining output to match the expected values exactly, and retains recognized diagnostics in the JSON result for audit. It does not silently discard arbitrary output.
Commit: c9753ee51de4e2b9a00cc7dd058b11d915f94ede.
14.7 Windows workflow repair — committed, fresh run required
The complete-conformance workflow's Windows setup now:
- Installs the valid MSYS2 UCRT64 GCC package rather than the nonexistent
mingw-w64-ucrt-x86_64-gcc-c++package. - Installs Git and native Windows Python.
- Explicitly checks GCC, G++, Python, and Java versions before testing.
- Runs the comparison scripts in a Windows PowerShell step with
CC=gccandCXX=g++, while keeping the POSIX Bash test step for Linux and macOS.
Workflow repair commit: d04cfb44e72c14f00f5f545787fe784f34ea778f.
The previous Windows failure belongs to the old workflow revision and therefore does not test this correction. A new run on the corrected commit is required before marking Windows PASS. The new workflow and validator commits are on master; their fresh Actions results must be reviewed before claiming either correction has passed.
14.8 Current measured-performance status
The conformance workflow still has no timed benchmark harness. The VM retry produced build and correctness evidence only. It did not measure arithmetic throughput, data-type latency, memory bandwidth, random RAM-access latency, cache behavior, allocation rate, or GC pause time. All such performance values remain NOT MEASURED. The VM build successes must not be converted into performance claims.
15. Arithmetic speed series — SCS-PERF-ARITH-001
The first executable arithmetic-throughput batch has been added to the comparison workflow.
15.1 Operation and type coverage
Each workload measures the four operators independently: addition (+), subtraction (-), multiplication (*), and division (/). Every case executes 2,000,000 operations per measured sample over a deterministic 4,096-element operand set. Operands are constrained to small, nonzero values so that signed overflow and division by zero are avoided and the same vector can be used across integer widths.
| Type | C11 | C++17 | Java 21+ | Representation note |
|---|---|---|---|---|
| signed 8-bit | Yes | Yes | Yes | Exact-width C/C++; Java byte arithmetic with narrowing |
| unsigned 8-bit | Yes | Yes | Yes | Java masked emulation |
| signed 16-bit | Yes | Yes | Yes | Exact-width C/C++; Java short arithmetic with narrowing |
| unsigned 16-bit | Yes | Yes | Yes | Java masked emulation |
| signed 32-bit | Yes | Yes | Yes | Exact-width C/C++; Java int |
| unsigned 32-bit | Yes | Yes | Yes | Java masked emulation |
| signed 64-bit | Yes | Yes | Yes | Exact-width C/C++; Java long |
| unsigned 64-bit | Yes | Yes | Yes | Java unsigned division helper; same small positive test domain |
| signed 128-bit | Where compiler supports __int128 | Where compiler supports __int128 | Not primitive | Extension; not a three-way primitive comparison |
| float32 | Yes | Yes | Yes | Native float operations |
| float64 | Yes | Yes | Yes | Native double operations |
| Boolean | N/A | N/A | N/A | Excluded from arithmetic operators; logical/conditional tests belong in a separate suite |
The C and C++ implementations share the same C-compatible workload source, compiled independently in C11 and C++17 modes. The Java implementation uses its own implementation and preserves the fixed-width masks for unsigned narrow types. A cross-language checksum check qualifies that the shared input cases produced matching results before timing rows are accepted.
15.2 Timing protocol
- Native C/C++:
-O2, one untimed warm-up process, five measured fresh-process sweeps. - Java: three warm-up sweeps inside each JVM process, then one measured sweep; five independent measured processes.
- Each row retains operation count, median ns/operation, all five sample times, and checksum.
- The CI runner records OS, architecture, processor description, and logical core count.
- The CI job runs on Ubuntu only. Its shared hosted-runner timings are informational and must not be treated as stable release rankings or compared against results from another host.
- Division uses nonzero divisors. Integer operands are constrained so all signed operations remain in range. Floating-point results use a checksum after scaling; this is a result-integrity guard, not a precision benchmark.
15.3 Scope boundary
This batch measures native C11, C++17, and Java 21+ arithmetic. It does not yet measure SLeeLa VM arithmetic. SLeeLa's core runtime currently exposes int (64-bit), double, and boolean as primitive VM value tags; width-family annotations and the 128-bit path require separate runtime qualification before they can be claimed as equivalent native primitive cases. The Basic Native VM and Premium SLVM 1–11 must receive the same SLeeLa workload through an internal monotonic timer or another measurement boundary that excludes compiler, process-start, and admission costs. Until that is implemented, SLeeLa VM arithmetic speed remains NOT MEASURED.
15.4 Executed reference run
A local reference execution was completed and recorded in Arithmetic Speed Series — Iteration 001. It passed 40/40 shared C11/C++17/Java checksum cases and four additional signed-128-bit cases in C/C++. Five measured samples were collected per case. The full median table and host/toolchain metadata are in that report.
This is a single Linux container reference run (AMD EPYC 9V74 allocation, 3 logical CPUs), not a controlled release benchmark. It establishes that the harness executes and validates its shared vectors; it does not establish a universal performance ranking. The separate GitHub Actions arithmetic job is still pending and will retain its own run-specific JSON artifact.
The SLeeLa Basic Native and Premium VM timing series remains NOT MEASURED. The report explicitly separates this from the native-language reference run.