test-strategy

Test Strategy Assessment

Analyze code and recommend the optimal testing approach from this project's testing toolkit.

Testing Toolkit Available

Simulation Harnesses

This project has five purpose-built deterministic simulation harnesses. Always consider whether new or changed code should be covered by one of these.

Decision Framework

Use Unit Tests When:

• Testing specific, known edge cases
• Verifying exact output for exact input
• Regression tests for fixed bugs
• Simple function behavior verification
• Fast feedback during development

#[cfg(test)]
mod tests {
    #[test]
    fn specific_edge_case() {
        assert_eq!(function(edge_input), expected_output);
    }
}

Use Parameterized Tests (rstest) When:

• You have a finite, known set of (input, expected output) pairs
• Each case has a specific expected value — no general invariant exists
• Multiple test functions share identical structure, differing only in values
• Testing enum variant mappings, error code tables, or configuration defaults
• You want each case to appear as a separately named sub-test in cargo test output
• Adding a new case should be one line, not a new function

use rstest::rstest;

#[rstest]
#[case("5s", Duration::from_secs(5))]
#[case("3m", Duration::from_secs(180))]
#[case("2h", Duration::from_secs(7200))]
#[case("0s", Duration::ZERO)]
fn parse_duration_valid(#[case] input: &str, #[case] expected: Duration) {
    assert_eq!(parse_duration(input).unwrap(), expected);
}

#[rstest]
#[case("5x", ParseError::InvalidUnit)]
#[case("", ParseError::Empty)]
#[case("-1s", ParseError::Negative)]
fn parse_duration_errors(#[case] input: &str, #[case] expected: ParseError) {
    assert_eq!(parse_duration(input).unwrap_err(), expected);
}

Dependency: Add rstest = "0.23" to [dev-dependencies] in Cargo.toml.

rstest Advanced Features

Fixtures — shared setup across tests without boilerplate:

use rstest::*;

#[fixture]
fn config() -> Config {
    Config::builder().timeout(Duration::from_secs(30)).build()
}

#[rstest]
fn test_with_default_config(config: Config) {
    assert!(config.timeout().as_secs() > 0);
}

Matrix testing — combinatorial cases via multiple #[values] parameters:

#[rstest]
fn protocol_version_compat(
    #[values(ProtocolVersion::V1, ProtocolVersion::V2)] version: ProtocolVersion,
    #[values(true, false)] compressed: bool,
    #[values(0, 1, 100)] payload_size: usize,
) {
    let msg = Message::new(version, compressed, payload_size);
    assert!(msg.is_valid());
}
// Generates 2 × 2 × 3 = 12 individual test cases

Use Property-Based Tests (proptest) When:

• Function should satisfy invariants for ALL valid inputs
• Testing mathematical properties (commutativity, associativity, idempotence)
• Round-trip properties (encode/​decode, serialize/​deserialize)
• Relationship between functions (e.g., parse and format are inverses)
• Exploring large input spaces systematically

#[cfg(all(test, feature = "stdx-proptest"))]
mod prop_tests {
    use proptest::prelude::*;

    proptest! {
        #[test]
        fn roundtrip_property(input in any::<ValidInput>()) {
            let encoded = encode(&input);
            let decoded = decode(&encoded).unwrap();
            prop_assert_eq!(input, decoded);
        }
    }
}

Run with: cargo test --features stdx-proptest

Use Fuzz Tests When:

• Parsing untrusted or external input (files, network data)
• Security-critical code paths
• Looking for crashes, panics, or undefined behavior
• Complex state machines with many paths
• Finding inputs that cause pathological performance

// In fuzz/​fuzz_targets/
#![no_main]
use libfuzzer_sys::fuzz_target;

fuzz_target!(|data: &[u8]| {
    let _ = parse_untrusted(data);
});

Run with: cargo +nightly fuzz run <target>

Use Kani Model Checking When:

• Proving absence of panics/​undefined behavior
• Verifying memory safety in unsafe code
• Proving loop bounds and termination
• Exhaustive verification of small input spaces
• Critical algorithms where bugs are unacceptable

#[cfg(kani)]
mod verification {
    use super::*;

    #[kani::proof]
    fn verify_no_panic() {
        let x: u32 = kani::any();
        kani::assume(x < 1000);
        let result = critical_function(x);
        // Kani proves this never panics
    }

    #[kani::proof]
    #[kani::unwind(10)]
    fn verify_loop_bounds() {
        let arr: [u8; 8] = kani::any();
        process_array(&arr); // Prove no out-of-bounds
    }
}

Run with: cargo kani --features kani

Use Simulation Tests When:

• Testing system-level behavior that emerges from component interactions
• Verifying invariants under many possible interleavings or schedules
• Changes touch the scanning pipeline, scheduler, archive handling, or git scanning
• You need deterministic replay of failure cases
• Verifying that chunked scanning matches a single-pass oracle
• Testing fault tolerance (I/​O errors, corruption, cancellation)
• Ensuring budget/​cap enforcement across the full pipeline

Choosing the right harness:

Is it about how work gets scheduled, buffer pools, or permits?
  → Scheduler Sim (src/​scheduler/​sim.rs, feature: scheduler-sim)

Is it about scanning files, finding secrets, or chunking?
  → Scanner Sim (src/​sim_scanner/, feature: sim-harness)
  → Also Tiger Harness if specifically about chunk boundary correctness

Is it about archive formats (zip/​tar/​gzip) or entry extraction?
  → Archive Sim (src/​sim_archive/, feature: sim-harness)
  → Scanner Sim for end-to-end archive-then-scan flows

Is it about git blob scanning, commit traversal, or pack I/​O?
  → Git Scan Sim (src/​sim_git_scan/, feature: sim-harness)

Adding a corpus case (scanner sim example):

// tests/​simulation/​scanner_corpus.rs — add a new #[test] fn
#[test]
fn regression_my_new_edge_case() {
    let scenario = Scenario { /​* ... */ };
    let config = RunConfig { /​* ... */ };
    let outcome = sim_scanner::runner::run(&scenario, &config);
    assert!(outcome.is_success(), "{outcome:#?}");
}

Adding a corpus case (scheduler sim example):

// tests/​simulation/​scheduler_sim.rs
#[test]
fn my_new_scheduler_invariant() {
    let config = SimConfig { /​* ... */ };
    let report = scheduler::sim::run(config, seed);
    report.assert_invariants();
}

Run with:

cargo test --features scheduler-sim --test simulation   # Scheduler only
cargo test --features sim-harness --test simulation      # Scanner + archive + git
cargo test --features sim-harness,scheduler-sim --test simulation  # All

Assessment Checklist

When analyzing code for test strategy, consider:

1. Input Domain

   • Fixed, known inputs → Unit tests
   • Finite set of (input, expected) pairs → Parameterized tests (rstest)
   • Combinatorial inputs (multiple parameters × multiple values) → rstest #[values] matrix
   • Large/​infinite input space → Property tests
   • Untrusted/​adversarial input → Fuzz tests
   • Small but critical input space → Kani
   • Interleaving-sensitive behavior → Simulation tests

2. Properties to Verify

   • Specific behavior → Unit tests
   • Same assertion, many concrete cases → Parameterized tests (rstest)
   • Invariants over all inputs → Property tests
   • "Never crashes" → Fuzz tests + Kani
   • Memory safety → Kani (especially for unsafe)
   • System-level invariants (no leaks, monotonic progress, ground truth) → Simulation tests

3. Code Characteristics

   • Pure functions → Property tests
   • Enum variant mappings / lookup tables → rstest #[case]
   • Functions with shared setup across tests → rstest #[fixture]
   • Parsers/​decoders → Fuzz tests
   • Unsafe blocks → Kani proofs
   • State machines → Property tests + Fuzz
   • Scanning pipeline components → Scanner Sim + Tiger Harness
   • Scheduler / buffer pool / permits → Scheduler Sim
   • Archive format handling → Archive Sim
   • Git blob / commit traversal → Git Scan Sim

4. Simulation Harness Checklist (always evaluate)

   • Does this change affect how files are discovered or scanned? → Scanner Sim
   • Does this change affect chunking, overlap, or finding dedup? → Scanner Sim + Tiger Harness
   • Does this change affect task scheduling, buffer management, or budget caps? → Scheduler Sim
   • Does this change affect archive reading, entry extraction, or path handling? → Archive Sim
   • Does this change affect git scanning, pack I/​O, or commit walking? → Git Scan Sim
   • Can a new corpus case reproduce the scenario deterministically? → Add to tests/​simulation/ or tests/​corpus/

5. Existing Patterns in This Codebase

   • Unit tests: Same file under #[cfg(test)] mod tests
   • Parameterized tests: rstest #[rstest] with #[case] in #[cfg(test)] modules (requires rstest = "0.23" in [dev-dependencies])
   • Property tests: Sibling *_tests.rs files with stdx-proptest feature
   • Kani proofs: #[cfg(kani)] blocks, see docs/​kani-verification.md
   • Simulation tests: tests/​simulation/ directory, corpus in tests/​corpus/ and tests/​simulation/​corpus/

Example Assessment Output

## Test Strategy for `WindowValidator`

### Recommended Approach: Property Tests + Kani + Scanner Sim

**Rationale:**
- Operates on sliding windows over byte streams (large input space)
- Has invariant: validated windows never exceed buffer bounds
- Contains unsafe pointer arithmetic
- Part of the scanning pipeline → needs sim coverage

**Specific Tests:**

1. **Property Test**: Window position invariants
   - Property: `window.end <= buffer.len()` for all inputs
   - Property: Windows never overlap incorrectly

2. **Kani Proof**: Memory safety of unsafe block
   - Prove: No out-of-bounds access in `unsafe` pointer ops
   - Bound: Unwind factor based on max window size

3. **Unit Tests**: Known edge cases
   - Empty buffer
   - Single-byte buffer
   - Window at buffer boundary

4. **Simulation**: Scanner Sim corpus case
   - Add scenario exercising the new window behavior under chunking
   - Tiger Harness: verify chunk boundaries don't lose findings
   - Verify oracle match (chunked result == single-pass result)

## Test Strategy for `RunStatus` Validation

### Recommended Approach: Parameterized Tests (rstest) + Unit Tests

**Rationale:**
- Finite set of status transitions with known valid/​invalid pairs
- Each transition has a specific expected result (no general invariant)
- Error cases map to specific error variants
- Adding new status variants should only require adding `#[case]` lines

**Specific Tests:**

1. **rstest Parameterized**: Valid state transitions
   ```rust
   #[rstest]
   #[case(RunStatus::Pending, RunStatus::Active, true)]
   #[case(RunStatus::Active, RunStatus::Completed, true)]
   #[case(RunStatus::Active, RunStatus::Failed, true)]
   #[case(RunStatus::Pending, RunStatus::Completed, false)]
   #[case(RunStatus::Completed, RunStatus::Active, false)]
   fn transition_validity(
       #[case] from: RunStatus,
       #[case] to: RunStatus,
       #[case] allowed: bool,
   ) {
       assert_eq!(from.can_transition_to(to), allowed);
   }

1. rstest Parameterized: Error messages for invalid transitions
2. Unit Test: Regression test for specific bug (if applicable)

```markdown
## Test Strategy for `ZipEntryIterator`

### Recommended Approach: Fuzz + Archive Sim + Scanner Sim

**Rationale:**
- Parses untrusted archive data (fuzz target)
- Changes archive extraction path → needs Archive Sim coverage
- End-to-end scanning of archive entries → needs Scanner Sim coverage

**Specific Tests:**

1. **Fuzz Test**: Parse arbitrary zip bytes without panic
2. **Archive Sim**: Corpus case with edge-case zip entries
   (long names, deflate truncation, encrypted entries)
3. **Scanner Sim**: End-to-end scenario: zip file → extract → scan → ground truth
4. **Unit Tests**: Known zip quirks (zip64, empty entries, duplicate names)

Quick Reference