Test And Oracle Builder

test-matrix.md format

| ID | Description | Input | Expected output | Criterion | Notes |
|----|-------------|-------|-----------------|-----------|-------|
| TC-01 | Nominal: single child process | ... | exit 0, log line present | CR-FORK | ... |
| TC-02 | Edge: empty input | ... | exit 1, error on stderr | CR-ERRHANDLING | ... |

Rules

• Every rubric criterion must be covered by at least one test case.
• Distinguish oracle tests (deterministic expected output) from property tests (behavioural invariants).
• For C/POSIX assignments: test on Linux; document platform assumptions in tests/PLATFORM.md.
• For Python assignments: test with the Python version stated in mission.json → constraints.
• Do not rely on fragile output matching (e.g. exact whitespace). Use normalised comparison or regex where appropriate.
• Edge cases must include: empty input, malformed input, resource exhaustion signals (SIGPIPE, SIGTERM), and maximum-size input.
• The oracle script must exit 0 if all tests pass, non-zero otherwise, and emit a structured summary (tests/results.json).

C++ testing conventions

• Prefer ctest as the execution entrypoint for automated validation.
• Ensure test instructions work from repository root with:
  • cmake -S reference -B reference/build
  • cmake --build reference/build
  • ctest --test-dir reference/build --output-on-failure
• Include at least one negative/error-path test case in tests/test-matrix.md.
• Do not rely on locale-specific output formatting in assertions.

Java testing conventions

• At minimum, provide a deterministic compile + run validation path.
• Preferred local baseline commands:
  • javac -d reference/build reference/src/Main.java
  • java -cp reference/build Main <sample-input>
• Ensure tests and sample inputs are self-contained in the repository (no network, no external services).
• Include at least one nominal case and one error-path case in tests/test-matrix.md.

Rust testing conventions

• Provide a deterministic Cargo validation path:
  • cargo build --manifest-path reference/Cargo.toml
  • cargo test --manifest-path reference/Cargo.toml
• Keep tests deterministic and avoid timing-sensitive assertions.
• Include at least one nominal case and one explicit error-path case in tests/test-matrix.md.
• Prefer small, focused unit tests for parsing and core business logic.

Anti-patterns to avoid

• Tests that only verify the happy path — always test error handling.
• Tests that depend on external network or filesystem state not reproducible in a grading environment.
• Oracles that compare floating-point output without tolerance.
• A test matrix that maps multiple rubric criteria to the same test — keep tests atomic.