Home /Modules/test-suites/README.md

README.md

test-suites/regression/corpus/README.md · 188 line(s) · markdown

<img align="right" src="https://github.com/mearvk/SLeeLa/blob/master/images/debian/debian-logo.png" width="75" height="75" alt="SLeeLa">

<img src="https://github.com/mearvk/SLeeLa/blob/master/images/sleela/sleela-logo-004.jpg" alt="SLeeLa">

Regression Corpus

Deterministic hostile and malformed inputs, each with a documented outcome. A case exists because the input once exposed a defect or because it protects a contract the code must keep. The corpus is run by python/test_corpus.py as the corpus group of python/run_regression.py, so it runs with the rest of the regression set (also with --fast), in CI, and through test-suites/run-all.sh --regression.

Rules

Layout

corpus/ README.md this file (the case table is checked against the fixtures) annotations/ NAME.sleela + NAME.expect document annotations, @next destinations parser/ NAME.sleela + NAME.expect lexer, version pragma, parser and semantic rejection http/ NAME.hex + NAME.expect HTTP 6.0 frames as annotated hex text tools/http6_decode.c driver that feeds one hex fixture to http6_frame_decode

Inputs too large to commit (a 2,000,000-character identifier, 20,000 nested parentheses) are generated by the GENERATED list in python/test_corpus.py; they have the same fields and appear in the same table.

The .expect format

One key: value per line; # starts a comment; a key may repeat.

KeyMeaning
idUnique case id (FAMILY-WHAT). Used in test names and in the table.
familyOne of the families below.
expectaccept or reject.
commandcheck (default) or run; the sleela subcommand used. Ignored for HTTP cases.
messageFor reject: text that must appear in the diagnostics. Repeatable.
warningFor accept: text that must appear on stderr as a warning. Repeatable.
outputFor accept with run: text that must appear in the program output. Repeatable.
rcHTTP cases only: the exact result of http6_frame_decode.
statusknown-issue when the case states a contract the code does not honour yet.
referenceThe source or document the case protects (first word must be an existing path).
defectThe id of the defect the case was written for, if any.
noteOne or two sentences: what the input is and why it matters.

A known-issue case is reported as OPEN at the end of the run and does not fail it. When the code starts to honour the contract the report says FIXED; remove the status line to make the case a hard assertion.

Families

FamilyWhat it coversWhere
annotationsDocument annotation syntax, repetition policy, unknown names, the @next forwarding rules.annotations/
traversal@next destinations that would leave the architecture tree: absolute, parent (..), embedded parent, drive letters, backslashes.annotations/
percent-encodingEncoded traversal (%2e%2e). The policy refuses % instead of trusting every consumer to decode safely.annotations/
http-framingHTTP 6.0 frames: truncated header, truncated payload, oversized and maximal lengths, wrong version, undefined types.http/
parser-rejectionVersion pragma, lexical, syntax and semantic errors; each must be a clear diagnostic.parser/
parser-acceptanceInputs that look odd but are legal.parser/
resource-limitsCompile-time nesting, chain and identifier limits, and run-time recursion, array and stack limits. All end in a diagnostic, not a crash.generated

Limits the corpus pins

The compiler refuses, with a Syntax error ... too deep / too long message, source that would otherwise exhaust the native stack: more than 1000 levels of (), [] and {}, more than 5000 operators between two statement separators, more than 1000 prefix operators in a row, and more than 1000 back-to-back casts. The caps sit far above real programs; LIMIT-NEST-AT-CAP and LIMIT-CHAIN-AT-CAP run programs just under them. Statements nest through their bodies and through else (an else-if chain is one level per link), with or without braces: more than 3000 levels is refused (LIMIT-ELSE-IF-CHAIN, LIMIT-NESTED-LOOPS-NO-BRACES, LIMIT-NESTED-DO-NO-BRACES; LIMIT-ELSE-IF-AT-CAP runs 2,900 links). Member-access (.) and label (:) chains count with the operators (LIMIT-MEMBER-CHAIN, LIMIT-LABEL-CHAIN). A crash from a shape not listed is a defect to add here.

Known gaps

None open. ANN-CLASS-BOUND and ANN-NEXT-SPELLING were known issues and are now hard assertions: a @next written before a class, method or field is judged by the same destination rule as a document annotation, and the forwarder reports a destination as written (system/net, not system / net). impl/frontend/ANNOTATION_PIPELINE.md describes the @name(value) form the parser accepts.

Running

python3 test-suites/regression/python/run_regression.py --only corpus # the corpus python3 test-suites/regression/python/run_regression.py --only corpus -k ANN # cases whose id contains ANN python3 test-suites/regression/python/run_regression.py --only corpus -v # one line per case

Adding a case

  1. Put the input in corpus/<dir>/NAME.sleela (or .hex), or add a gen(...) entry for a large one.
  2. Write NAME.expect with at least id, family, expect, reference and note.
  3. Run the corpus. A new case that exposes a defect should fail first; fix the code, then keep the case.
  4. Add the id to the table below. The bookkeeping test names any case that is missing.

Cases

IDFamilyInputOutcomeWhat it protects
ANN-AFTER-CLASSannotationsannotations/annotation-after-declaration.sleelareject (check)A document annotation after the first class is refused.
ANN-DUP-SCOPEannotationsannotations/scope-duplicate.sleelareject (check)Only next, requires, capability, counter and supporter may repeat.
ANN-NEXT-BAREannotationsannotations/next-bare-marker.sleelareject (check)@next without parentheses carries no destination.
ANN-NEXT-EMPTYannotationsannotations/next-empty-value.sleelareject (check)A value-form annotation needs a value.
ANN-NEXT-MULTI-CHECKannotationsannotations/next-multiple-checks.sleelaaccept (check)Several @next declarations are legal metadata; only forwarding refuses them (next case).
ANN-NEXT-MULTI-RUNannotationsannotations/next-multiple-forwarding.sleelareject (run)The runtime refuses ambiguous forwarding before any code runs.
ANN-NEXT-SPELLINGannotationsannotations/forwarded-destination-keeps-its-slashes.sleelaaccept (run)The parser joins annotation tokens with spaces; the forwarder now closes the gaps around '/', '.' and '-', so the destination is reported as written.
ANN-NEXT-SPELLING-PUNCTannotationsannotations/spaced-destination-words-stay-apart.sleelaaccept (run)Hyphens and dots inside a destination survive the spacing repair.
ANN-OK-DOCUMENTannotationsannotations/ok-document-annotations.sleelaaccept (check)The reference shape: document annotations, one safe relative @next.
ANN-UNKNOWNannotationsannotations/unknown-annotation.sleelaaccept (check)Unknown annotations are kept and reported as a warning, never silently dropped.
ANN-UNTERMINATEDannotationsannotations/unterminated-parenthesis.sleelareject (check)The parenthesis is never closed.
ANN-CLASS-BOUNDtraversalannotations/class-bound-annotation-is-not-document-metadata.sleelareject (check)An annotation written directly before a class binds to that class, not to the document. The same destination rule applies to it (it used to bypass...
ANN-FIELD-BOUNDtraversalannotations/field-bound-next-absolute.sleelareject (check)A field-level @next is judged by the same destination rule.
ANN-METHOD-BOUNDtraversalannotations/method-bound-next-parent.sleelareject (check)A method-level @next is judged by the same destination rule.
ANN-NEXT-ABSOLUTEtraversalannotations/next-absolute-path.sleelareject (check)An absolute destination would leave the architecture tree.
ANN-NEXT-BACKSLASHtraversalannotations/next-backslash.sleelareject (check)A backslash is not a legal source character; the lexer refuses it before the policy sees it.
ANN-NEXT-DRIVEtraversalannotations/next-drive-letter.sleelareject (check)Windows-absolute destination.
ANN-NEXT-EMBEDDED-PARENTtraversalannotations/next-embedded-parent.sleelareject (check)A parent component in the middle of a path.
ANN-NEXT-PARENTtraversalannotations/next-parent-traversal.sleelareject (check)Parent-directory traversal.
ANN-NEXT-PERCENTpercent-encodingannotations/next-percent-encoded-parent.sleelareject (check)Percent-encoded parent directory.
HTTP6-EMPTYhttp-framinghttp/empty-input.hexreject, decode returns -2Nothing to read is a short header, not a crash.
HTTP6-LENGTH-FFFFFFFFhttp-framinghttp/maximum-length-field.hexreject, decode returns -3The largest 32-bit length must not wrap a size computation.
HTTP6-OK-DATAhttp-framinghttp/data-with-payload.hexaccept, decode returns 0Header fields and payload round-trip.
HTTP6-OK-PINGhttp-framinghttp/ping-empty-payload.hexaccept, decode returns 0The smallest legal frame.
HTTP6-OVERSIZEhttp-framinghttp/oversized-length.hexreject, decode returns -3One byte over the 16 MiB cap.
HTTP6-TRUNC-HEADERhttp-framinghttp/truncated-header.hexreject, decode returns -2One byte short of a header.
HTTP6-TRUNC-PAYLOADhttp-framinghttp/truncated-payload.hexreject, decode returns -3The declared length exceeds the bytes available.
HTTP6-TYPE-0http-framinghttp/type-zero.hexreject, decode returns -3Types start at 1.
HTTP6-TYPE-21http-framinghttp/type-past-last.hexreject, decode returns -3Type 20 (RECIPIENT_LABEL) is the last one.
HTTP6-VERSIONhttp-framinghttp/wrong-version.hexreject, decode returns -3Only version 6 is accepted.
PARSE-BRACE-MISSINGparser-rejectionparser/missing-closing-brace.sleelareject (check)The file ends inside a method.
PARSE-COMMENT-UNTERMINATEDparser-rejectionparser/unterminated-block-comment.sleelareject (check)The << comment has no >>.
PARSE-EMPTYparser-rejectionparser/empty-file.sleelareject (check)A zero-byte file.
PARSE-INT-OVERFLOWparser-rejectionparser/integer-literal-overflow.sleelareject (check)A literal wider than 64 bits.
PARSE-NO-CLASSESparser-rejectionparser/no-classes.sleelareject (check)A pragma and nothing else.
PARSE-NULparser-rejectionparser/nul-byte-in-source.sleelareject (check)A NUL byte inside code is a lexical error.
PARSE-PRAGMA-FUTUREparser-rejectionparser/pragma-too-new.sleelareject (check)One minor above the ceiling.
PARSE-PRAGMA-MALFORMEDparser-rejectionparser/pragma-malformed.sleelareject (check)A non-numeric version.
PARSE-PRAGMA-OLDparser-rejectionparser/pragma-too-old.sleelareject (check)Below the floor.
PARSE-PRAGMA-OVERFLOWparser-rejectionparser/pragma-huge-number.sleelareject (check)A major number that overflows an int is malformed, not wrapped.
PARSE-STRING-UNTERMINATEDparser-rejectionparser/unterminated-string.sleelareject (check)The closing quote is missing.
SEM-DUP-METHODparser-rejectionparser/duplicate-method.sleelareject (check)Two methods with one signature.
SEM-UNDECLAREDparser-rejectionparser/undeclared-variable.sleelareject (check)Assignment before declaration.
SEM-UNKNOWN-TYPEparser-rejectionparser/unknown-type.sleelareject (check)Type check, not parse.
PARSE-OK-BLANK-LINESparser-acceptanceparser/blank-and-whitespace-lines.sleelaaccept (run)Empty and whitespace-only lines before the first declaration.
PARSE-OK-COMMENTSparser-acceptanceparser/every-comment-form.sleelaaccept (run)All four comment forms are skipped; an angle comment must start its line.
PARSE-OK-CRLFparser-acceptancegeneratedaccept (run)A file with Windows line endings is the same program. Generated because the repository normalizes text files to LF.
PARSE-OK-MAX-LONGparser-acceptanceparser/largest-long-literal.sleelaaccept (run)The largest 64-bit literal is in range (one more is PARSE-INT-OVERFLOW).
PARSE-OK-NO-FINAL-NEWLINEparser-acceptanceparser/no-trailing-newline.sleelaaccept (run)The last token may end the file.
PARSE-OK-UNICODEparser-acceptanceparser/unicode-string-literal.sleelaaccept (run)Non-ASCII text, including a character outside the BMP, survives lexing and printing.
LIMIT-CAST-CHAINresource-limitsgeneratedreject (check)30,000 back-to-back casts nest without any open bracket.
LIMIT-CHAIN-AT-CAPresource-limitsgeneratedaccept (run)A 4,000-operator statement is under the cap and still runs.
LIMIT-ELSE-IF-AT-CAPresource-limitsgeneratedaccept (run)2,900 else-if links is under the cap and runs.
LIMIT-ELSE-IF-CHAINresource-limitsgeneratedreject (check)60,000 chained else-if links. Each link is one level of statement recursion with no bracket to count.
LIMIT-LABEL-CHAINresource-limitsgeneratedreject (check)60,000 stacked statement labels.
LIMIT-LONG-IDENTIFIERresource-limitsgeneratedaccept (check)A 2,000,000-character identifier is legal. Dependency discovery used std::regex, which recurses per character and crashed.
LIMIT-LONG-IDENTIFIER-RUNresource-limitsgeneratedaccept (run)A 50,000-character identifier runs end to end.
LIMIT-MANY-CLASSESresource-limitsgeneratedaccept (check)20,000 declarations.
LIMIT-MEMBER-CHAINresource-limitsgeneratedreject (check)A 60,000-call member chain builds a left-deep tree.
LIMIT-MEMBER-CHAIN-AT-CAPresource-limitsgeneratedaccept (run)A 4,000-call member chain is under the cap and runs.
LIMIT-NEST-AT-CAPresource-limitsgeneratedaccept (run)900 nested parentheses is under the cap and still runs.
LIMIT-NEST-BLOCKSresource-limitsgeneratedreject (check)5,000 nested blocks.
LIMIT-NEST-PARENSresource-limitsgeneratedreject (check)20,000 nested parentheses. The parser recursed once per level and crashed with a segmentation fault.
LIMIT-NESTED-DO-NO-BRACESresource-limitsgeneratedreject (check)20,000 nested do statements.
LIMIT-NESTED-LOOPS-NO-BRACESresource-limitsgeneratedreject (check)60,000 nested for statements without braces.
LIMIT-OPERATOR-CHAINresource-limitsgeneratedreject (check)One statement of 100,000 chained operators builds a left-deep tree that later passes walk recursively.
LIMIT-PREFIX-RUNresource-limitsgeneratedreject (check)5,000 prefix operators in a row.
LIMIT-STRING-1MBresource-limitsgeneratedaccept (run)A one-megabyte string literal.
RUN-ARRAY-NEGATIVEresource-limitsgeneratedreject (run)A negative array size.
RUN-ARRAY-TOO-LARGEresource-limitsgeneratedreject (run)An array far beyond the heap limit.
RUN-DIVIDE-BY-ZEROresource-limitsgeneratedreject (run)Integer division by zero.
RUN-MANY-LOCALSresource-limitsgeneratedreject (run)30,000 locals in one method exhaust the VM stack with a runtime error.
RUN-RECURSIONresource-limitsgeneratedreject (run)Unbounded recursion ends with a runtime error, not a crash.