<img align="right" src="https://github.com/mearvk/SLeeLa/blob/master/images/debian/debian-logo.png" width="75" height="75" alt="SLeeLa">
<img src="https://github.com/mearvk/SLeeLa/blob/master/images/sleela/sleela-logo-004.jpg" alt="SLeeLa">
Regression Corpus
Deterministic hostile and malformed inputs, each with a documented outcome. A case
exists because the input once exposed a defect or because it protects a contract the
code must keep. The corpus is run by python/test_corpus.py as the corpus group of
python/run_regression.py, so it runs with the rest of the regression set (also with
--fast), in CI, and through test-suites/run-all.sh --regression.
Rules
- Bounded and deterministic. Every case finishes within seconds. A hang is a failure.
Random input uses a fixed seed (see
c/reg_http6.c); nothing reads the network or the clock. - No crash, ever. Whatever a case expects, the tool must not die from a signal and must not exit with a shell failure code (126 or above). A crash on hostile input is always a defect.
- No secrets or personal data. Fixtures are synthetic.
- The expected outcome is written beside the fixture, in a
NAME.expectsidecar, and again in the table below. A test fails if a fixture has no.expect, if a case is missing from the table, or if areferencepath does not exist. - Every case names its reason: the
referencefile it protects, and for a former defect thedefectid it was written for.
Layout
corpus/ README.md this file (the case table is checked against the fixtures) annotations/ NAME.sleela + NAME.expect document annotations, @next destinations parser/ NAME.sleela + NAME.expect lexer, version pragma, parser and semantic rejection http/ NAME.hex + NAME.expect HTTP 6.0 frames as annotated hex text tools/http6_decode.c driver that feeds one hex fixture to http6_frame_decode
Inputs too large to commit (a 2,000,000-character identifier, 20,000 nested parentheses) are
generated by the GENERATED list in python/test_corpus.py; they have the same fields and appear in
the same table.
The .expect format
One key: value per line; # starts a comment; a key may repeat.
| Key | Meaning |
|---|---|
id | Unique case id (FAMILY-WHAT). Used in test names and in the table. |
family | One of the families below. |
expect | accept or reject. |
command | check (default) or run; the sleela subcommand used. Ignored for HTTP cases. |
message | For reject: text that must appear in the diagnostics. Repeatable. |
warning | For accept: text that must appear on stderr as a warning. Repeatable. |
output | For accept with run: text that must appear in the program output. Repeatable. |
rc | HTTP cases only: the exact result of http6_frame_decode. |
status | known-issue when the case states a contract the code does not honour yet. |
reference | The source or document the case protects (first word must be an existing path). |
defect | The id of the defect the case was written for, if any. |
note | One or two sentences: what the input is and why it matters. |
A known-issue case is reported as OPEN at the end of the run and does not fail it. When the code
starts to honour the contract the report says FIXED; remove the status line to make the case a
hard assertion.
Families
| Family | What it covers | Where |
|---|---|---|
annotations | Document annotation syntax, repetition policy, unknown names, the @next forwarding rules. | annotations/ |
traversal | @next destinations that would leave the architecture tree: absolute, parent (..), embedded parent, drive letters, backslashes. | annotations/ |
percent-encoding | Encoded traversal (%2e%2e). The policy refuses % instead of trusting every consumer to decode safely. | annotations/ |
http-framing | HTTP 6.0 frames: truncated header, truncated payload, oversized and maximal lengths, wrong version, undefined types. | http/ |
parser-rejection | Version pragma, lexical, syntax and semantic errors; each must be a clear diagnostic. | parser/ |
parser-acceptance | Inputs that look odd but are legal. | parser/ |
resource-limits | Compile-time nesting, chain and identifier limits, and run-time recursion, array and stack limits. All end in a diagnostic, not a crash. | generated |
Limits the corpus pins
The compiler refuses, with a Syntax error ... too deep / too long message, source that would
otherwise exhaust the native stack: more than 1000 levels of (), [] and {}, more than 5000
operators between two statement separators, more than 1000 prefix operators in a row, and more than
1000 back-to-back casts. The caps sit far above real programs; LIMIT-NEST-AT-CAP and
LIMIT-CHAIN-AT-CAP run programs just under them. Statements nest through their bodies and through
else (an else-if chain is one level per link), with or without braces: more than 3000 levels is refused
(LIMIT-ELSE-IF-CHAIN, LIMIT-NESTED-LOOPS-NO-BRACES, LIMIT-NESTED-DO-NO-BRACES; LIMIT-ELSE-IF-AT-CAP runs 2,900 links).
Member-access (.) and label (:) chains count with the operators (LIMIT-MEMBER-CHAIN, LIMIT-LABEL-CHAIN). A crash from a shape
not listed is a defect to add here.
Known gaps
None open. ANN-CLASS-BOUND and ANN-NEXT-SPELLING were known issues and are now hard assertions: a
@next written before a class, method or field is judged by the same destination rule as a document
annotation, and the forwarder reports a destination as written (system/net, not system / net).
impl/frontend/ANNOTATION_PIPELINE.md describes the @name(value) form the parser accepts.
Running
python3 test-suites/regression/python/run_regression.py --only corpus # the corpus python3 test-suites/regression/python/run_regression.py --only corpus -k ANN # cases whose id contains ANN python3 test-suites/regression/python/run_regression.py --only corpus -v # one line per case
Adding a case
- Put the input in
corpus/<dir>/NAME.sleela(or.hex), or add agen(...)entry for a large one. - Write
NAME.expectwith at leastid,family,expect,referenceandnote. - Run the corpus. A new case that exposes a defect should fail first; fix the code, then keep the case.
- Add the id to the table below. The bookkeeping test names any case that is missing.
Cases
| ID | Family | Input | Outcome | What it protects |
|---|---|---|---|---|
ANN-AFTER-CLASS | annotations | annotations/annotation-after-declaration.sleela | reject (check) | A document annotation after the first class is refused. |
ANN-DUP-SCOPE | annotations | annotations/scope-duplicate.sleela | reject (check) | Only next, requires, capability, counter and supporter may repeat. |
ANN-NEXT-BARE | annotations | annotations/next-bare-marker.sleela | reject (check) | @next without parentheses carries no destination. |
ANN-NEXT-EMPTY | annotations | annotations/next-empty-value.sleela | reject (check) | A value-form annotation needs a value. |
ANN-NEXT-MULTI-CHECK | annotations | annotations/next-multiple-checks.sleela | accept (check) | Several @next declarations are legal metadata; only forwarding refuses them (next case). |
ANN-NEXT-MULTI-RUN | annotations | annotations/next-multiple-forwarding.sleela | reject (run) | The runtime refuses ambiguous forwarding before any code runs. |
ANN-NEXT-SPELLING | annotations | annotations/forwarded-destination-keeps-its-slashes.sleela | accept (run) | The parser joins annotation tokens with spaces; the forwarder now closes the gaps around '/', '.' and '-', so the destination is reported as written. |
ANN-NEXT-SPELLING-PUNCT | annotations | annotations/spaced-destination-words-stay-apart.sleela | accept (run) | Hyphens and dots inside a destination survive the spacing repair. |
ANN-OK-DOCUMENT | annotations | annotations/ok-document-annotations.sleela | accept (check) | The reference shape: document annotations, one safe relative @next. |
ANN-UNKNOWN | annotations | annotations/unknown-annotation.sleela | accept (check) | Unknown annotations are kept and reported as a warning, never silently dropped. |
ANN-UNTERMINATED | annotations | annotations/unterminated-parenthesis.sleela | reject (check) | The parenthesis is never closed. |
ANN-CLASS-BOUND | traversal | annotations/class-bound-annotation-is-not-document-metadata.sleela | reject (check) | An annotation written directly before a class binds to that class, not to the document. The same destination rule applies to it (it used to bypass... |
ANN-FIELD-BOUND | traversal | annotations/field-bound-next-absolute.sleela | reject (check) | A field-level @next is judged by the same destination rule. |
ANN-METHOD-BOUND | traversal | annotations/method-bound-next-parent.sleela | reject (check) | A method-level @next is judged by the same destination rule. |
ANN-NEXT-ABSOLUTE | traversal | annotations/next-absolute-path.sleela | reject (check) | An absolute destination would leave the architecture tree. |
ANN-NEXT-BACKSLASH | traversal | annotations/next-backslash.sleela | reject (check) | A backslash is not a legal source character; the lexer refuses it before the policy sees it. |
ANN-NEXT-DRIVE | traversal | annotations/next-drive-letter.sleela | reject (check) | Windows-absolute destination. |
ANN-NEXT-EMBEDDED-PARENT | traversal | annotations/next-embedded-parent.sleela | reject (check) | A parent component in the middle of a path. |
ANN-NEXT-PARENT | traversal | annotations/next-parent-traversal.sleela | reject (check) | Parent-directory traversal. |
ANN-NEXT-PERCENT | percent-encoding | annotations/next-percent-encoded-parent.sleela | reject (check) | Percent-encoded parent directory. |
HTTP6-EMPTY | http-framing | http/empty-input.hex | reject, decode returns -2 | Nothing to read is a short header, not a crash. |
HTTP6-LENGTH-FFFFFFFF | http-framing | http/maximum-length-field.hex | reject, decode returns -3 | The largest 32-bit length must not wrap a size computation. |
HTTP6-OK-DATA | http-framing | http/data-with-payload.hex | accept, decode returns 0 | Header fields and payload round-trip. |
HTTP6-OK-PING | http-framing | http/ping-empty-payload.hex | accept, decode returns 0 | The smallest legal frame. |
HTTP6-OVERSIZE | http-framing | http/oversized-length.hex | reject, decode returns -3 | One byte over the 16 MiB cap. |
HTTP6-TRUNC-HEADER | http-framing | http/truncated-header.hex | reject, decode returns -2 | One byte short of a header. |
HTTP6-TRUNC-PAYLOAD | http-framing | http/truncated-payload.hex | reject, decode returns -3 | The declared length exceeds the bytes available. |
HTTP6-TYPE-0 | http-framing | http/type-zero.hex | reject, decode returns -3 | Types start at 1. |
HTTP6-TYPE-21 | http-framing | http/type-past-last.hex | reject, decode returns -3 | Type 20 (RECIPIENT_LABEL) is the last one. |
HTTP6-VERSION | http-framing | http/wrong-version.hex | reject, decode returns -3 | Only version 6 is accepted. |
PARSE-BRACE-MISSING | parser-rejection | parser/missing-closing-brace.sleela | reject (check) | The file ends inside a method. |
PARSE-COMMENT-UNTERMINATED | parser-rejection | parser/unterminated-block-comment.sleela | reject (check) | The << comment has no >>. |
PARSE-EMPTY | parser-rejection | parser/empty-file.sleela | reject (check) | A zero-byte file. |
PARSE-INT-OVERFLOW | parser-rejection | parser/integer-literal-overflow.sleela | reject (check) | A literal wider than 64 bits. |
PARSE-NO-CLASSES | parser-rejection | parser/no-classes.sleela | reject (check) | A pragma and nothing else. |
PARSE-NUL | parser-rejection | parser/nul-byte-in-source.sleela | reject (check) | A NUL byte inside code is a lexical error. |
PARSE-PRAGMA-FUTURE | parser-rejection | parser/pragma-too-new.sleela | reject (check) | One minor above the ceiling. |
PARSE-PRAGMA-MALFORMED | parser-rejection | parser/pragma-malformed.sleela | reject (check) | A non-numeric version. |
PARSE-PRAGMA-OLD | parser-rejection | parser/pragma-too-old.sleela | reject (check) | Below the floor. |
PARSE-PRAGMA-OVERFLOW | parser-rejection | parser/pragma-huge-number.sleela | reject (check) | A major number that overflows an int is malformed, not wrapped. |
PARSE-STRING-UNTERMINATED | parser-rejection | parser/unterminated-string.sleela | reject (check) | The closing quote is missing. |
SEM-DUP-METHOD | parser-rejection | parser/duplicate-method.sleela | reject (check) | Two methods with one signature. |
SEM-UNDECLARED | parser-rejection | parser/undeclared-variable.sleela | reject (check) | Assignment before declaration. |
SEM-UNKNOWN-TYPE | parser-rejection | parser/unknown-type.sleela | reject (check) | Type check, not parse. |
PARSE-OK-BLANK-LINES | parser-acceptance | parser/blank-and-whitespace-lines.sleela | accept (run) | Empty and whitespace-only lines before the first declaration. |
PARSE-OK-COMMENTS | parser-acceptance | parser/every-comment-form.sleela | accept (run) | All four comment forms are skipped; an angle comment must start its line. |
PARSE-OK-CRLF | parser-acceptance | generated | accept (run) | A file with Windows line endings is the same program. Generated because the repository normalizes text files to LF. |
PARSE-OK-MAX-LONG | parser-acceptance | parser/largest-long-literal.sleela | accept (run) | The largest 64-bit literal is in range (one more is PARSE-INT-OVERFLOW). |
PARSE-OK-NO-FINAL-NEWLINE | parser-acceptance | parser/no-trailing-newline.sleela | accept (run) | The last token may end the file. |
PARSE-OK-UNICODE | parser-acceptance | parser/unicode-string-literal.sleela | accept (run) | Non-ASCII text, including a character outside the BMP, survives lexing and printing. |
LIMIT-CAST-CHAIN | resource-limits | generated | reject (check) | 30,000 back-to-back casts nest without any open bracket. |
LIMIT-CHAIN-AT-CAP | resource-limits | generated | accept (run) | A 4,000-operator statement is under the cap and still runs. |
LIMIT-ELSE-IF-AT-CAP | resource-limits | generated | accept (run) | 2,900 else-if links is under the cap and runs. |
LIMIT-ELSE-IF-CHAIN | resource-limits | generated | reject (check) | 60,000 chained else-if links. Each link is one level of statement recursion with no bracket to count. |
LIMIT-LABEL-CHAIN | resource-limits | generated | reject (check) | 60,000 stacked statement labels. |
LIMIT-LONG-IDENTIFIER | resource-limits | generated | accept (check) | A 2,000,000-character identifier is legal. Dependency discovery used std::regex, which recurses per character and crashed. |
LIMIT-LONG-IDENTIFIER-RUN | resource-limits | generated | accept (run) | A 50,000-character identifier runs end to end. |
LIMIT-MANY-CLASSES | resource-limits | generated | accept (check) | 20,000 declarations. |
LIMIT-MEMBER-CHAIN | resource-limits | generated | reject (check) | A 60,000-call member chain builds a left-deep tree. |
LIMIT-MEMBER-CHAIN-AT-CAP | resource-limits | generated | accept (run) | A 4,000-call member chain is under the cap and runs. |
LIMIT-NEST-AT-CAP | resource-limits | generated | accept (run) | 900 nested parentheses is under the cap and still runs. |
LIMIT-NEST-BLOCKS | resource-limits | generated | reject (check) | 5,000 nested blocks. |
LIMIT-NEST-PARENS | resource-limits | generated | reject (check) | 20,000 nested parentheses. The parser recursed once per level and crashed with a segmentation fault. |
LIMIT-NESTED-DO-NO-BRACES | resource-limits | generated | reject (check) | 20,000 nested do statements. |
LIMIT-NESTED-LOOPS-NO-BRACES | resource-limits | generated | reject (check) | 60,000 nested for statements without braces. |
LIMIT-OPERATOR-CHAIN | resource-limits | generated | reject (check) | One statement of 100,000 chained operators builds a left-deep tree that later passes walk recursively. |
LIMIT-PREFIX-RUN | resource-limits | generated | reject (check) | 5,000 prefix operators in a row. |
LIMIT-STRING-1MB | resource-limits | generated | accept (run) | A one-megabyte string literal. |
RUN-ARRAY-NEGATIVE | resource-limits | generated | reject (run) | A negative array size. |
RUN-ARRAY-TOO-LARGE | resource-limits | generated | reject (run) | An array far beyond the heap limit. |
RUN-DIVIDE-BY-ZERO | resource-limits | generated | reject (run) | Integer division by zero. |
RUN-MANY-LOCALS | resource-limits | generated | reject (run) | 30,000 locals in one method exhaust the VM stack with a runtime error. |
RUN-RECURSION | resource-limits | generated | reject (run) | Unbounded recursion ends with a runtime error, not a crash. |