Mutation Testing Framework

Problem Statement

Build a mutation testing framework that evaluates test quality by introducing deliberate bugs (mutations) into your code and checking if tests catch them. Your framework should automatically mutate source code in various ways, run the test suite against each mutation, and report which mutations “survived” (weren’t caught by tests), revealing gaps in test coverage.

Your mutation testing framework should support:

• Multiple mutation operators (arithmetic, comparison, logical, boundary)
• Automatic mutation generation from source code
• Test execution against each mutant
• Mutation score calculation (killed vs survived)
• Detailed mutation survival reports
• Parallel mutation testing for performance

Why Mutation Testing Matters

The Test Quality Problem

The Core Issue: High code coverage doesn’t mean high-quality tests. You can have 100% line coverage with tests that don’t actually verify correctness.

Example of Useless Tests:

#![allow(unused)]
fn main() {
fn divide(a: i32, b: i32) -> Result<i32, String> {
    if b == 0 {
        Err("Division by zero".to_string())
    } else {
        Ok(a / b)
    }
}

#[test]
fn test_divide() {
    // This test has 100% coverage but verifies NOTHING!
    let _ = divide(10, 2);
    let _ = divide(10, 0);
    // No assertions! Test always passes even if divide() is completely broken
}
}

Line Coverage: 100% ✓ Branch Coverage: 100% ✓ Test Quality: 0% ✗

What is Mutation Testing?

Concept: Introduce small changes (mutations) to the code. If tests fail, the mutation is “killed” (good!). If tests still pass, the mutation “survives” (bad—tests didn’t catch the bug).

Example:

#![allow(unused)]
fn main() {
// Original code
fn max(a: i32, b: i32) -> i32 {
    if a > b {  // ← Original
        a
    } else {
        b
    }
}

// Mutation 1: Change > to >=
fn max(a: i32, b: i32) -> i32 {
    if a >= b {  // ← Mutated
        a
    } else {
        b
    }
}

// Mutation 2: Change > to <
fn max(a: i32, b: i32) -> i32 {
    if a < b {  // ← Mutated
        a
    } else {
        b
    }
}

// Mutation 3: Swap return values
fn max(a: i32, b: i32) -> i32 {
    if a > b {
        b  // ← Mutated (was a)
    } else {
        a  // ← Mutated (was b)
    }
}
}

Testing each mutation:

#![allow(unused)]
fn main() {
#[test]
fn test_max() {
    assert_eq!(max(5, 3), 5);
}

// Against Mutation 1 (a >= b): PASSES ✗ (Survives!)
// Against Mutation 2 (a < b): FAILS ✓ (Killed!)
// Against Mutation 3 (swap returns): FAILS ✓ (Killed!)

// Mutation 1 survived because our test doesn't check the a == b case!
// Need better test: assert_eq!(max(5, 5), 5);
}

Mutation Score

Formula: Mutation Score = (Killed Mutations / Total Mutations) × 100%

Interpretation:

• 90-100%: Excellent test quality
• 75-89%: Good test quality
• 60-74%: Acceptable but improvable
• <60%: Weak tests, needs improvement

Example:

Total mutations: 20
Killed: 15
Survived: 5
Mutation Score: 75%

→ 75% of introduced bugs are caught by tests
→ 25% of bugs could slip through undetected

Why It Matters More Than Coverage

Coverage measures execution, not verification:

#![allow(unused)]
fn main() {
fn abs(x: i32) -> i32 {
    if x < 0 {
        -x
    } else {
        x
    }
}

// Bad test: 100% coverage, 0% verification
#[test]
fn test_abs_bad() {
    abs(-5);  // No assertion!
    abs(5);   // No assertion!
}
// Line coverage: 100% ✓
// Mutation score: 0% ✗

// Good test: 100% coverage, 100% verification
#[test]
fn test_abs_good() {
    assert_eq!(abs(-5), 5);
    assert_eq!(abs(5), 5);
    assert_eq!(abs(0), 0);
}
// Line coverage: 100% ✓
// Mutation score: 100% ✓
}

Real-world impact:

Project A: 95% line coverage, 45% mutation score
→ Tests run the code but don't verify correctness
→ Production bugs: 18 per quarter
→ Customer complaints: High

Project B: 85% line coverage, 88% mutation score
→ Tests actually verify behavior
→ Production bugs: 3 per quarter
→ Customer complaints: Low

6x bug reduction with better test quality!

Common Mutation Operators

Use Cases

1. Test Quality Assurance

• Validate existing tests: Find weak tests that don’t catch bugs
• Code review: Require new code to achieve minimum mutation score
• Refactoring safety: Ensure tests will catch regressions

2. Critical Systems Development

• Medical devices: Verify safety-critical code has thorough tests
• Financial systems: Ensure transaction logic is properly tested
• Security: Validate authentication/authorization test quality

3. Test-Driven Development (TDD)

• Guide test writing: Surviving mutations show what assertions to add
• Continuous improvement: Track mutation score over time
• Learning tool: Teaches developers to write better tests

4. CI/CD Quality Gates

• Block merges: PR must not reduce mutation score
• Regression prevention: Detect test quality degradation
• Accountability: Teams maintain high test standards

Rust Programming Concepts for This Project

This project requires understanding several advanced Rust concepts to build a functional mutation testing framework.

AST Parsing with syn

The Problem: To mutate code intelligently (e.g., changing a + b to a - b), we can’t just use string replacement, which is fragile. We need to understand the grammatical structure of the code.

The Solution: The syn crate parses Rust code into an Abstract Syntax Tree (AST). This allows us to traverse the code programmatically and find specific patterns like binary operations or function calls.

#![allow(unused)]
fn main() {
// Code: let x = a + b;
let expr: Expr = syn::parse_str("let x = a + b;").unwrap();
// We can now inspect 'expr' to find the BinOp (+)
}

Code Generation with quote

The Problem: After modifying the AST (e.g., changing the operator), we need to convert it back into valid Rust source code to compile and run it.

The Solution: The quote crate allows us to turn AST nodes back into tokens and source strings. It uses a macro quote! that makes code generation safe and easy.

#![allow(unused)]
fn main() {
let op = quote! { - };
let new_code = quote! { let x = a #op b; };
}

Mutation Operators

The Concept: A mutation operator is a rule that defines a specific type of code transformation. It consists of:

1. Pattern: What to look for (e.g., BinaryOp::Add).
2. Transformation: How to change it (e.g., replace with BinaryOp::Sub).

Defining these operators clearly is crucial for a modular and extensible framework.

Test Harness Integration

The Problem: We need to run the project’s test suite repeatedly against hundreds of slightly different versions of the code (mutants).

The Solution: We programmatically invoke cargo test using std::process::Command. We need to manage:

• Compilation: Compiling each mutant (often to a temporary directory).
• Execution: Running the tests and capturing exit codes.
• Timeouts: Killing tests that enter infinite loops due to mutations (e.g., i < 10 becoming i > 10).

Parallel Execution with rayon

The Problem: Mutation testing is computationally expensive. Testing 100 mutations sequentially could take minutes.

The Solution: Use rayon to execute independent mutation tests in parallel across all available CPU cores.

Connection to This Project

This project guides you through building a mutation testing tool similar to cargo-mutants.

1. Milestone 1: You’ll implement the Mutation Operator Engine, using syn to parse code and find places to apply mutations (like arithmetic or logical operators).
2. Milestone 2: You’ll build the Test Execution Engine, enabling your tool to compile mutants and run cargo test against them, handling results and timeouts.
3. Milestone 3: You’ll use rayon to implement Parallel Mutation Testing, drastically reducing the time it takes to analyze a codebase.
4. Milestone 4: You’ll compute the Mutation Score and generate reports, analyzing which mutations were killed and which survived.
5. Milestone 5: You’ll create Source Code Annotations, visualizing exactly where tests are weak directly in the source code.
6. Milestone 6: You’ll implement Advanced Mutation Strategies, moving beyond simple operator swaps to more semantic changes like statement deletion or return value modification.

Building the Project

Milestone 1: Mutation Operator Engine

Goal: Create a system that can identify mutation points in source code and generate mutated versions.

Why we start here: Before testing, we need to generate mutations. This milestone teaches AST manipulation and systematic code transformation.

Architecture

Structs:

• MutationOperator - Defines a type of mutation

  • Field: name: String - Operator name (e.g., “ArithmeticMutation”)
  • Field: pattern: Pattern - What to match in code
  • Field: transformations: Vec<Transformation> - How to mutate

• MutationPoint - A location where mutation can occur

  • Field: line: usize - Line number
  • Field: column: usize - Column number
  • Field: original: String - Original code
  • Field: operator: MutationOperator - Operator to apply

Enums:

• Pattern - What code patterns to match

  • Variants: BinaryOp(OpType), Comparison(CmpType), Literal(LitType)

• Transformation - How to transform matched code

  • Variants: Replace(String), Delete, Negate

Functions:

• new(name: &str) -> MutationOperator - Create operator
• find_mutation_points(&self, source: &str) -> Vec<MutationPoint> - Locate mutations
• apply(&self, point: &MutationPoint) -> String - Generate mutant code
• describe(&self) -> String - Human-readable description

Starter Code:

#![allow(unused)]
fn main() {
use std::fmt;

#[derive(Debug, Clone, PartialEq)]
pub enum OpType {
    Add,
    Sub,
    Mul,
    Div,
    Rem,
}

#[derive(Debug, Clone, PartialEq)]
pub enum CmpType {
    Eq,
    Ne,
    Lt,
    Le,
    Gt,
    Ge,
}

#[derive(Debug, Clone)]
pub enum Pattern {
    BinaryOp(OpType),
    Comparison(CmpType),
    Literal(String),
}

#[derive(Debug, Clone)]
pub enum Transformation {
    Replace(String),
    Delete,
    Negate,
}

#[derive(Debug, Clone)]
pub struct MutationPoint {
    pub line: usize,
    pub column: usize,
    pub original: String,
    pub mutated: String,
    pub operator_name: String,
}

pub struct MutationOperator {
    pub name: String,
    pattern: Pattern,
    transformations: Vec<Transformation>,
}

impl MutationOperator {
    pub fn new(name: &str, pattern: Pattern, transformations: Vec<Transformation>) -> Self {
        // TODO: Create mutation operator
        todo!("Implement operator creation")
    }

    pub fn find_mutation_points(&self, source: &str) -> Vec<MutationPoint> {
        // TODO: Parse source code
        // TODO: Find locations matching pattern
        // TODO: Generate mutation point for each match
        todo!("Find mutation points")
    }

    pub fn apply(&self, source: &str, point: &MutationPoint) -> String {
        // TODO: Replace original code with mutated version
        // TODO: Preserve formatting and line numbers
        todo!("Apply mutation")
    }

    pub fn describe(&self) -> String {
        // TODO: Return human-readable description
        todo!("Describe mutation")
    }
}

// Predefined mutation operators
impl MutationOperator {
    pub fn arithmetic_mutations() -> Vec<Self> {
        // TODO: Create operators for +/-/*/% mutations
        // TODO: + → -, - → +, * → /, / → *
        todo!("Create arithmetic mutation operators")
    }

    pub fn comparison_mutations() -> Vec<Self> {
        // TODO: Create operators for </>/<=/>=  /==/!= mutations
        // TODO: > → >=, < → <=, == → !=, etc.
        todo!("Create comparison mutation operators")
    }

    pub fn logical_mutations() -> Vec<Self> {
        // TODO: Create operators for &&/|| mutations
        // TODO: && → ||, || → &&
        todo!("Create logical mutation operators")
    }

    pub fn boundary_mutations() -> Vec<Self> {
        // TODO: Create operators for boundary value mutations
        // TODO: 0 → 1, 1 → 0, < → <=, > → >=
        todo!("Create boundary mutation operators")
    }
}
}

Checkpoint Tests:

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_find_arithmetic_mutations() {
        let operator = MutationOperator::new(
            "AddToSub",
            Pattern::BinaryOp(OpType::Add),
            vec![Transformation::Replace("-".to_string())],
        );

        let source = "let x = a + b;";
        let points = operator.find_mutation_points(source);

        assert_eq!(points.len(), 1);
        assert_eq!(points[0].original, "+");
        assert_eq!(points[0].mutated, "-");
    }

    #[test]
    fn test_find_comparison_mutations() {
        let operator = MutationOperator::new(
            "GtToGe",
            Pattern::Comparison(CmpType::Gt),
            vec![Transformation::Replace(">=".to_string())],
        );

        let source = r#"
if x > 0 {
    println!("positive");
}
"#;
        let points = operator.find_mutation_points(source);

        assert_eq!(points.len(), 1);
        assert_eq!(points[0].line, 1);  // Second line (0-indexed)
    }

    #[test]
    fn test_apply_mutation() {
        let point = MutationPoint {
            line: 0,
            column: 8,
            original: "+".to_string(),
            mutated: "-".to_string(),
            operator_name: "AddToSub".to_string(),
        };

        let source = "let x = a + b;";
        let operator = MutationOperator::new(
            "AddToSub",
            Pattern::BinaryOp(OpType::Add),
            vec![Transformation::Replace("-".to_string())],
        );

        let mutated = operator.apply(source, &point);
        assert_eq!(mutated, "let x = a - b;");
    }

    #[test]
    fn test_multiple_mutation_points() {
        let operator = MutationOperator::new(
            "AddToSub",
            Pattern::BinaryOp(OpType::Add),
            vec![Transformation::Replace("-".to_string())],
        );

        let source = "let x = a + b + c;";
        let points = operator.find_mutation_points(source);

        // Should find two + operators
        assert_eq!(points.len(), 2);
    }

    #[test]
    fn test_predefined_operators() {
        let arith_ops = MutationOperator::arithmetic_mutations();
        assert!(arith_ops.len() > 0);

        let cmp_ops = MutationOperator::comparison_mutations();
        assert!(cmp_ops.len() > 0);

        let logic_ops = MutationOperator::logical_mutations();
        assert!(logic_ops.len() > 0);
    }
}
}

Check Your Understanding:

• Why do we need multiple transformations for the same pattern?
• How do mutation operators differ from code formatters?
• What makes a good mutation—too subtle vs too obvious?

Why Milestone 1 Isn’t Enough

Limitation: We can generate mutations but have no way to test them. We need to compile and run tests against each mutant.

What we’re adding: Test execution engine that compiles mutated code, runs tests, and determines if mutation was killed or survived.

Improvement:

• Capability: Full mutation testing workflow
• Automation: Batch processing of mutations
• Metrics: Kill/survival tracking
• Insight: Identifies weak tests

Milestone 2: Test Execution Engine

Goal: Execute tests against mutated code and track results (killed vs survived).

Why this matters: Generating mutations is useless without testing them. We need to know which bugs our tests catch.

Architecture

Structs:

• TestRunner - Executes tests against code

  • Field: test_command: String - Command to run tests (e.g., “cargo test”)
  • Field: timeout: Duration - Max execution time

• MutationResult - Result of testing a mutation

  • Field: mutation: MutationPoint - The mutation tested
  • Field: status: MutationStatus - Outcome
  • Field: test_output: String - Test execution output
  • Field: execution_time: Duration - How long tests took

Enums:

• MutationStatus - Outcome of mutation test
  • Variants:
    • Killed - Tests failed (good!)
    • Survived - Tests passed (bad!)
    • Timeout - Tests took too long
    • CompileError - Mutant didn’t compile
    • Skipped - Mutation was equivalent to original

Functions:

• new(test_command: &str, timeout: Duration) -> TestRunner - Create runner
• test_mutation(&self, mutant_code: &str) -> MutationResult - Test one mutation
• compile_and_test(&self, code: &str) -> Result<bool, Error> - Compile and run
• is_equivalent(&self, original: &str, mutant: &str) -> bool - Detect equivalents

Starter Code:

#![allow(unused)]
fn main() {
use std::process::{Command, Stdio};
use std::time::{Duration, Instant};
use std::fs;
use tempfile::TempDir;

#[derive(Debug, Clone, PartialEq)]
pub enum MutationStatus {
    Killed,      // Tests failed - mutation caught!
    Survived,    // Tests passed - mutation not caught
    Timeout,     // Tests took too long
    CompileError, // Mutant didn't compile
    Skipped,     // Equivalent mutation
}

#[derive(Debug, Clone)]
pub struct MutationResult {
    pub mutation: MutationPoint,
    pub status: MutationStatus,
    pub test_output: String,
    pub execution_time: Duration,
}

pub struct TestRunner {
    test_command: String,
    timeout: Duration,
}

impl TestRunner {
    pub fn new(test_command: &str, timeout: Duration) -> Self {
        // TODO: Initialize test runner
        todo!("Create test runner")
    }

    pub fn test_mutation(&self, original_code: &str, mutation: &MutationPoint) -> MutationResult {
        // TODO: Apply mutation to create mutant code
        // TODO: Write mutant to temporary file
        // TODO: Try to compile mutant
        // TODO: If compiles, run tests
        // TODO: Determine if killed or survived based on test result
        // TODO: Measure execution time
        todo!("Test mutation")
    }

    fn compile_and_test(&self, code: &str) -> Result<bool, std::io::Error> {
        // TODO: Create temporary project directory
        // TODO: Write code to examples/lib.rs
        // TODO: Run `cargo test`
        // TODO: Parse exit code: 0 = pass, non-zero = fail
        // TODO: Return true if tests passed, false if failed
        todo!("Compile and test code")
    }

    fn run_with_timeout(&self, command: &mut Command) -> Result<std::process::Output, std::io::Error> {
        // TODO: Spawn command
        // TODO: Wait with timeout
        // TODO: Kill process if exceeds timeout
        // TODO: Return output or timeout error
        todo!("Run command with timeout")
    }

    pub fn is_equivalent(&self, original: &str, mutant: &str) -> bool {
        // TODO: Check if mutation is semantically equivalent
        // TODO: Examples: changing (a + 0) to just (a) is equivalent
        // TODO: For now, return false (advanced feature)
        false
    }
}
}

Checkpoint Tests:

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    fn create_test_project(code: &str) -> TempDir {
        let dir = tempfile::tempdir().unwrap();

        // Create Cargo.toml
        fs::write(
            dir.path().join("Cargo.toml"),
            r#"
[package]
name = "mutant"
version = "0.1.0"
edition = "2021"
"#,
        )
        .unwrap();

        // Create examples directory
        fs::create_dir(dir.path().join("examples")).unwrap();

        // Write code
        fs::write(dir.path().join("examples/lib.rs"), code).unwrap();

        dir
    }

    #[test]
    fn test_killed_mutation() {
        let code = r#"
pub fn add(a: i32, b: i32) -> i32 {
    a + b
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_add() {
        assert_eq!(add(2, 3), 5);
    }
}
"#;

        let mutation = MutationPoint {
            line: 2,
            column: 6,
            original: "+".to_string(),
            mutated: "-".to_string(),  // Change + to -
            operator_name: "AddToSub".to_string(),
        };

        let runner = TestRunner::new("cargo test", Duration::from_secs(30));
        let result = runner.test_mutation(code, &mutation);

        // Test should fail with - instead of +
        assert_eq!(result.status, MutationStatus::Killed);
    }

    #[test]
    fn test_survived_mutation() {
        let code = r#"
pub fn max(a: i32, b: i32) -> i32 {
    if a > b {
        a
    } else {
        b
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_max() {
        assert_eq!(max(5, 3), 5);
        // Missing test for a == b case!
    }
}
"#;

        let mutation = MutationPoint {
            line: 2,
            column: 9,
            original: ">".to_string(),
            mutated: ">=".to_string(),  // Change > to >=
            operator_name: "GtToGe".to_string(),
        };

        let runner = TestRunner::new("cargo test", Duration::from_secs(30));
        let result = runner.test_mutation(code, &mutation);

        // Test should still pass (bad!)
        assert_eq!(result.status, MutationStatus::Survived);
    }

    #[test]
    fn test_compile_error() {
        let code = r#"
pub fn broken() {
    let x = 5
    // Missing semicolon
}
"#;

        let mutation = MutationPoint {
            line: 2,
            column: 12,
            original: "5".to_string(),
            mutated: "6".to_string(),
            operator_name: "LiteralChange".to_string(),
        };

        let runner = TestRunner::new("cargo test", Duration::from_secs(30));
        let result = runner.test_mutation(code, &mutation);

        assert_eq!(result.status, MutationStatus::CompileError);
    }

    #[test]
    fn test_timeout() {
        let code = r#"
pub fn infinite_loop() {
    loop {
        // Never terminates
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_loop() {
        infinite_loop();
    }
}
"#;

        let runner = TestRunner::new("cargo test", Duration::from_secs(2));
        let result = runner.test_mutation(code, &MutationPoint {
            line: 3,
            column: 0,
            original: "".to_string(),
            mutated: "".to_string(),
            operator_name: "Test".to_string(),
        });

        assert_eq!(result.status, MutationStatus::Timeout);
    }
}
}

Why Milestone 2 Isn’t Enough

Limitation: We test mutations one by one, sequentially. For large projects with hundreds of mutations, this takes hours.

What we’re adding: Parallel mutation testing to run multiple mutations concurrently, dramatically reducing execution time.

Improvement:

• Speed: 8-16x faster with parallel execution
• Scalability: Handle large codebases efficiently
• Resource usage: Utilize all CPU cores
• Practicality: Makes mutation testing viable for CI/CD

Milestone 3: Parallel Mutation Testing

Goal: Execute multiple mutation tests concurrently to reduce total testing time.

Why this matters: Sequential testing of 100 mutations × 10 seconds each = 16+ minutes. Parallel execution on 8 cores = ~2 minutes. Essential for practical use.

Architecture

Structs:

• ParallelTestRunner - Manages concurrent test execution
  • Field: max_concurrency: usize - Number of parallel tests
  • Field: runner: TestRunner - Underlying test executor

Functions:

• new(max_concurrency: usize, timeout: Duration) -> Self - Create parallel runner
• test_mutations_parallel(&self, code: &str, mutations: Vec<MutationPoint>) -> Vec<MutationResult> - Run all mutations
• batch_mutations(&self, mutations: Vec<MutationPoint>, batch_size: usize) -> Vec<Vec<MutationPoint>> - Group for efficiency

Starter Code:

#![allow(unused)]
fn main() {
use rayon::prelude::*;
use std::sync::{Arc, Mutex};

pub struct ParallelTestRunner {
    max_concurrency: usize,
    runner: TestRunner,
}

impl ParallelTestRunner {
    pub fn new(max_concurrency: usize, timeout: Duration) -> Self {
        // TODO: Create parallel runner
        // TODO: Initialize thread pool with max_concurrency
        todo!("Create parallel test runner")
    }

    pub fn test_mutations_parallel(
        &self,
        original_code: &str,
        mutations: Vec<MutationPoint>,
    ) -> Vec<MutationResult> {
        // TODO: Use rayon par_iter to test mutations in parallel
        // TODO: Limit concurrency to max_concurrency
        // TODO: Collect and return results
        todo!("Test mutations in parallel")
    }

    fn batch_mutations(&self, mutations: Vec<MutationPoint>, batch_size: usize) -> Vec<Vec<MutationPoint>> {
        // TODO: Split mutations into batches
        // TODO: Each batch will be processed together
        // TODO: Helps with resource management
        todo!("Batch mutations")
    }

    pub fn test_with_progress(
        &self,
        original_code: &str,
        mutations: Vec<MutationPoint>,
        progress_callback: impl Fn(usize, usize) + Send + Sync,
    ) -> Vec<MutationResult> {
        // TODO: Test mutations with progress reporting
        // TODO: Call callback with (completed, total) periodically
        todo!("Test with progress updates")
    }
}
}

Checkpoint Tests:

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_parallel_execution_correctness() {
        let code = r#"
pub fn add(a: i32, b: i32) -> i32 { a + b }
pub fn sub(a: i32, b: i32) -> i32 { a - b }
pub fn mul(a: i32, b: i32) -> i32 { a * b }

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_add() { assert_eq!(add(2, 3), 5); }

    #[test]
    fn test_sub() { assert_eq!(sub(5, 3), 2); }

    #[test]
    fn test_mul() { assert_eq!(mul(2, 3), 6); }
}
"#;

        let mutations = vec![
            MutationPoint {
                line: 1,
                column: 40,
                original: "+".to_string(),
                mutated: "-".to_string(),
                operator_name: "AddToSub".to_string(),
            },
            MutationPoint {
                line: 2,
                column: 40,
                original: "-".to_string(),
                mutated: "+".to_string(),
                operator_name: "SubToAdd".to_string(),
            },
            MutationPoint {
                line: 3,
                column: 40,
                original: "*".to_string(),
                mutated: "/".to_string(),
                operator_name: "MulToDiv".to_string(),
            },
        ];

        let runner = ParallelTestRunner::new(4, Duration::from_secs(30));
        let results = runner.test_mutations_parallel(code, mutations);

        // All mutations should be killed
        assert_eq!(results.len(), 3);
        assert!(results.iter().all(|r| r.status == MutationStatus::Killed));
    }

    #[test]
    fn test_parallel_speedup() {
        // Create many mutations to test
        let mutations: Vec<MutationPoint> = (0..20)
            .map(|i| MutationPoint {
                line: i,
                column: 0,
                original: "+".to_string(),
                mutated: "-".to_string(),
                operator_name: format!("Mutation{}", i),
            })
            .collect();

        let code = create_code_with_many_functions(20);

        // Sequential
        let sequential_runner = ParallelTestRunner::new(1, Duration::from_secs(5));
        let start = Instant::now();
        let _ = sequential_runner.test_mutations_parallel(&code, mutations.clone());
        let sequential_time = start.elapsed();

        // Parallel
        let parallel_runner = ParallelTestRunner::new(8, Duration::from_secs(5));
        let start = Instant::now();
        let _ = parallel_runner.test_mutations_parallel(&code, mutations);
        let parallel_time = start.elapsed();

        println!("Sequential: {:?}", sequential_time);
        println!("Parallel: {:?}", parallel_time);

        let speedup = sequential_time.as_secs_f64() / parallel_time.as_secs_f64();
        println!("Speedup: {:.2}x", speedup);

        // Parallel should be at least 2x faster
        assert!(speedup >= 2.0);
    }

    #[test]
    fn test_progress_reporting() {
        let mutations = vec![
            MutationPoint {
                line: 1,
                column: 0,
                original: "+".to_string(),
                mutated: "-".to_string(),
                operator_name: "Test1".to_string(),
            },
            MutationPoint {
                line: 2,
                column: 0,
                original: "-".to_string(),
                mutated: "+".to_string(),
                operator_name: "Test2".to_string(),
            },
        ];

        let progress_updates = Arc::new(Mutex::new(Vec::new()));
        let updates_clone = Arc::clone(&progress_updates);

        let runner = ParallelTestRunner::new(2, Duration::from_secs(10));
        let _ = runner.test_with_progress(
            "test code",
            mutations,
            move |completed, total| {
                updates_clone.lock().unwrap().push((completed, total));
            },
        );

        let updates = progress_updates.lock().unwrap();
        assert!(updates.len() > 0);
        assert_eq!(updates.last().unwrap(), &(2, 2));
    }
}
}

Why Milestone 3 Isn’t Enough

Limitation: We generate mutation results but have no structured way to analyze and report them. Need comprehensive reports showing mutation score and survival patterns.

What we’re adding: Mutation score calculator and detailed reporting with survival analysis, helping developers prioritize test improvements.

Improvement:

• Metrics: Calculate mutation score, survival rate by category
• Reporting: Generate actionable reports
• Prioritization: Identify weakest test areas
• Visualization: Clear presentation of results

Milestone 4: Mutation Score Analysis and Reporting

Goal: Calculate mutation scores, analyze patterns, and generate comprehensive reports.

Why this matters: Raw results (200 killed, 50 survived) don’t tell the full story. We need analysis to identify patterns and guide test improvement.

Architecture

Structs:

• MutationReport - Complete analysis of mutation testing

  • Field: total_mutations: usize - Total mutations tested
  • Field: killed: usize - Mutations caught by tests
  • Field: survived: usize - Mutations not caught
  • Field: timeouts: usize - Tests that timed out
  • Field: compile_errors: usize - Mutants that didn’t compile
  • Field: mutation_score: f64 - Percentage killed
  • Field: results_by_operator: HashMap<String, OperatorStats> - Per-operator breakdown

• OperatorStats - Statistics for one mutation operator

  • Field: operator_name: String
  • Field: killed: usize
  • Field: survived: usize
  • Field: score: f64

Functions:

• analyze(results: Vec<MutationResult>) -> MutationReport - Generate report
• calculate_score(&self) -> f64 - Compute mutation score
• survival_by_operator(&self) -> HashMap<String, OperatorStats> - Group by operator
• format_text(&self) -> String - Plain text report
• format_html(&self) -> String - HTML report with charts
• format_json(&self) -> String - JSON for tools

Starter Code:

#![allow(unused)]
fn main() {
use std::collections::HashMap;
use serde::{Serialize, Deserialize};

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MutationReport {
    pub total_mutations: usize,
    pub killed: usize,
    pub survived: usize,
    pub timeouts: usize,
    pub compile_errors: usize,
    pub mutation_score: f64,
    pub results_by_operator: HashMap<String, OperatorStats>,
    pub survived_mutations: Vec<MutationPoint>,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct OperatorStats {
    pub operator_name: String,
    pub total: usize,
    pub killed: usize,
    pub survived: usize,
    pub score: f64,
}

impl MutationReport {
    pub fn analyze(results: Vec<MutationResult>) -> Self {
        // TODO: Count killed, survived, timeouts, errors
        // TODO: Calculate mutation score
        // TODO: Group by operator type
        // TODO: Identify survived mutations for detailed review
        todo!("Analyze mutation results")
    }

    pub fn calculate_score(&self) -> f64 {
        // TODO: Score = (killed / (killed + survived)) * 100
        // TODO: Exclude timeouts and compile errors
        todo!("Calculate mutation score")
    }

    fn survival_by_operator(&self, results: &[MutationResult]) -> HashMap<String, OperatorStats> {
        // TODO: Group results by operator name
        // TODO: Calculate per-operator statistics
        todo!("Calculate per-operator stats")
    }

    pub fn format_text(&self) -> String {
        // TODO: Create text report with:
        // TODO: - Overall mutation score
        // TODO: - Breakdown by status (killed/survived/timeout/error)
        // TODO: - Per-operator statistics
        // TODO: - List of survived mutations (needs attention)
        todo!("Format text report")
    }

    pub fn format_html(&self) -> String {
        // TODO: Generate HTML with:
        // TODO: - Summary statistics
        // TODO: - Charts showing score by operator
        // TODO: - Color-coded mutation list
        // TODO: - Survival heatmap
        todo!("Format HTML report")
    }

    pub fn format_json(&self) -> String {
        // TODO: Serialize to JSON
        serde_json::to_string_pretty(self).unwrap()
    }

    pub fn worst_operators(&self, n: usize) -> Vec<&OperatorStats> {
        // TODO: Return n operators with lowest scores
        // TODO: Helps prioritize test improvements
        todo!("Find worst operators")
    }

    pub fn recommendations(&self) -> Vec<String> {
        // TODO: Generate actionable recommendations based on results
        // TODO: Example: "Add tests for boundary conditions (20 survived)"
        todo!("Generate recommendations")
    }
}
}

Checkpoint Tests:

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    fn create_sample_results() -> Vec<MutationResult> {
        vec![
            MutationResult {
                mutation: MutationPoint {
                    line: 1,
                    column: 0,
                    original: "+".to_string(),
                    mutated: "-".to_string(),
                    operator_name: "Arithmetic".to_string(),
                },
                status: MutationStatus::Killed,
                test_output: String::new(),
                execution_time: Duration::from_secs(1),
            },
            MutationResult {
                mutation: MutationPoint {
                    line: 2,
                    column: 0,
                    original: ">".to_string(),
                    mutated: ">=".to_string(),
                    operator_name: "Comparison".to_string(),
                },
                status: MutationStatus::Survived,
                test_output: String::new(),
                execution_time: Duration::from_secs(1),
            },
            MutationResult {
                mutation: MutationPoint {
                    line: 3,
                    column: 0,
                    original: "&&".to_string(),
                    mutated: "||".to_string(),
                    operator_name: "Logical".to_string(),
                },
                status: MutationStatus::Killed,
                test_output: String::new(),
                execution_time: Duration::from_secs(1),
            },
        ]
    }

    #[test]
    fn test_mutation_score_calculation() {
        let results = create_sample_results();
        let report = MutationReport::analyze(results);

        // 2 killed, 1 survived out of 3 total
        assert_eq!(report.killed, 2);
        assert_eq!(report.survived, 1);
        assert_eq!(report.total_mutations, 3);

        // Score should be (2 / 3) * 100 = 66.67%
        assert!((report.mutation_score - 66.67).abs() < 0.1);
    }

    #[test]
    fn test_operator_breakdown() {
        let results = create_sample_results();
        let report = MutationReport::analyze(results);

        let arith_stats = report.results_by_operator.get("Arithmetic").unwrap();
        assert_eq!(arith_stats.killed, 1);
        assert_eq!(arith_stats.survived, 0);
        assert_eq!(arith_stats.score, 100.0);

        let cmp_stats = report.results_by_operator.get("Comparison").unwrap();
        assert_eq!(cmp_stats.killed, 0);
        assert_eq!(cmp_stats.survived, 1);
        assert_eq!(cmp_stats.score, 0.0);
    }

    #[test]
    fn test_text_report_format() {
        let results = create_sample_results();
        let report = MutationReport::analyze(results);

        let text = report.format_text();

        assert!(text.contains("Mutation Score"));
        assert!(text.contains("66."));  // 66.67%
        assert!(text.contains("Killed: 2"));
        assert!(text.contains("Survived: 1"));
    }

    #[test]
    fn test_worst_operators() {
        let results = create_sample_results();
        let report = MutationReport::analyze(results);

        let worst = report.worst_operators(1);

        // Comparison operator should be worst (0% score)
        assert_eq!(worst[0].operator_name, "Comparison");
        assert_eq!(worst[0].score, 0.0);
    }

    #[test]
    fn test_recommendations() {
        let results = create_sample_results();
        let report = MutationReport::analyze(results);

        let recommendations = report.recommendations();

        // Should suggest improving comparison tests
        assert!(recommendations.iter().any(|r| r.contains("Comparison")));
    }
}
}

Why Milestone 4 Isn’t Enough

Limitation: Reports show problems but don’t help developers fix them. Need to highlight exact code locations that need better tests.

What we’re adding: Integrated source code annotation showing surviving mutations directly in context, making it easy to write missing tests.

Improvement:

• Actionability: See exactly what code needs testing
• Context: View mutations with surrounding code
• Guidance: Understand what assertions to add
• Efficiency: Fix weaknesses without hunting through reports

Milestone 5: Source Code Annotation and Visualization

Goal: Annotate source code with mutation testing results, showing which lines have surviving mutations.

Why this matters: Abstract reports (“Line 42: survived”) are hard to act on. Seeing “this > should also be tested with >=” in context is immediately actionable.

Architecture

Structs:

• AnnotatedSource - Source code with mutation annotations

  • Field: source: String - Original source code
  • Field: annotations: HashMap<usize, Vec<Annotation>> - Per-line annotations
  • Field: metadata: SourceMetadata - File info

• Annotation - Information about a mutation at a location

  • Field: mutation: MutationPoint - The mutation
  • Field: status: MutationStatus - Result
  • Field: suggestion: String - How to improve test

Functions:

• annotate(source: &str, results: Vec<MutationResult>) -> AnnotatedSource - Add annotations
• render_terminal(&self) -> String - Colored terminal output
• render_html(&self) -> String - Interactive HTML view
• generate_test_suggestions(&self) -> Vec<String> - Suggest test cases

Starter Code:

#![allow(unused)]
fn main() {
use std::collections::HashMap;
use colored::*;

#[derive(Debug, Clone)]
pub struct AnnotatedSource {
    pub source: String,
    pub annotations: HashMap<usize, Vec<Annotation>>,
    pub metadata: SourceMetadata,
}

#[derive(Debug, Clone)]
pub struct Annotation {
    pub mutation: MutationPoint,
    pub status: MutationStatus,
    pub suggestion: String,
}

#[derive(Debug, Clone)]
pub struct SourceMetadata {
    pub file_path: String,
    pub total_lines: usize,
    pub annotated_lines: usize,
}

impl AnnotatedSource {
    pub fn annotate(source: &str, results: Vec<MutationResult>) -> Self {
        // TODO: Parse source into lines
        // TODO: Group results by line number
        // TODO: Generate suggestions for each survived mutation
        todo!("Annotate source code")
    }

    pub fn render_terminal(&self) -> String {
        // TODO: For each line:
        // TODO: - Show line number
        // TODO: - Show source code
        // TODO: - If has annotations, show them below in different color
        // TODO: - Green for killed, red for survived, yellow for other
        todo!("Render for terminal")
    }

    pub fn render_html(&self) -> String {
        // TODO: Generate HTML with:
        // TODO: - Syntax highlighting
        // TODO: - Inline annotations
        // TODO: - Hover tooltips with suggestions
        // TODO: - Click to see mutation details
        todo!("Render as HTML")
    }

    pub fn generate_test_suggestions(&self) -> Vec<String> {
        // TODO: For each survived mutation, suggest test case
        // TODO: Example: "Add test: assert_eq!(max(5, 5), 5);"
        todo!("Generate test suggestions")
    }

    fn suggestion_for_mutation(mutation: &MutationPoint) -> String {
        // TODO: Based on mutation operator, suggest specific test
        // TODO: Arithmetic: test with different values
        // TODO: Comparison: test boundary cases
        // TODO: Logical: test both true and false paths
        todo!("Generate suggestion")
    }
}
}

Checkpoint Tests:

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_annotation_creation() {
        let source = r#"
fn max(a: i32, b: i32) -> i32 {
    if a > b { a } else { b }
}
"#;

        let results = vec![MutationResult {
            mutation: MutationPoint {
                line: 2,
                column: 9,
                original: ">".to_string(),
                mutated: ">=".to_string(),
                operator_name: "Comparison".to_string(),
            },
            status: MutationStatus::Survived,
            test_output: String::new(),
            execution_time: Duration::from_secs(1),
        }];

        let annotated = AnnotatedSource::annotate(source, results);

        assert_eq!(annotated.annotations.len(), 1);
        assert!(annotated.annotations.contains_key(&2));
    }

    #[test]
    fn test_terminal_rendering() {
        let source = r#"fn test() {
    let x = a + b;
}"#;

        let results = vec![MutationResult {
            mutation: MutationPoint {
                line: 1,
                column: 14,
                original: "+".to_string(),
                mutated: "-".to_string(),
                operator_name: "Arithmetic".to_string(),
            },
            status: MutationStatus::Survived,
            test_output: String::new(),
            execution_time: Duration::from_secs(1),
        }];

        let annotated = AnnotatedSource::annotate(source, results);
        let terminal_output = annotated.render_terminal();

        // Should contain line numbers and annotations
        assert!(terminal_output.contains("1"));
        assert!(terminal_output.contains("let x = a + b;"));
        assert!(terminal_output.contains("Survived"));
    }

    #[test]
    fn test_suggestion_generation() {
        let source = r#"
fn max(a: i32, b: i32) -> i32 {
    if a > b { a } else { b }
}
"#;

        let results = vec![MutationResult {
            mutation: MutationPoint {
                line: 2,
                column: 9,
                original: ">".to_string(),
                mutated: ">=".to_string(),
                operator_name: "Comparison".to_string(),
            },
            status: MutationStatus::Survived,
            test_output: String::new(),
            execution_time: Duration::from_secs(1),
        }];

        let annotated = AnnotatedSource::annotate(source, results);
        let suggestions = annotated.generate_test_suggestions();

        // Should suggest testing equal values
        assert!(suggestions.iter().any(|s| s.contains("==") || s.contains("equal")));
    }

    #[test]
    fn test_html_rendering() {
        let source = "fn test() { let x = 5; }";
        let annotated = AnnotatedSource::annotate(source, vec![]);

        let html = annotated.render_html();

        assert!(html.contains("<html"));
        assert!(html.contains("fn test"));
    }
}
}

Why Milestone 5 Isn’t Enough

Limitation: All our mutations are at the syntactic level (tokens). We miss higher-level semantic mutations that could reveal deeper test weaknesses.

What we’re adding: Advanced mutation strategies including return value mutations, function call removal, and state mutations.

Improvement:

• Depth: Test semantic correctness, not just syntax
• Coverage: Catch bugs simpler mutations miss
• Real-world: Mirror actual programming errors
• Sophistication: Higher-order mutation operators

Milestone 6: Advanced Mutation Strategies

Goal: Implement semantic-level mutations that test deeper aspects of code correctness.

Why this matters: Changing + to - is useful, but missing a null check or wrong return value are more common real bugs. Advanced mutations find these issues.

Architecture

New Mutation Operators:

• Return value mutations (flip booleans, negate numbers)
• Statement deletion (remove entire lines)
• Constant replacement (0 → 1, null → value)
• Function call removal (skip side effects)

Functions:

• return_value_mutations() -> Vec<MutationOperator> - Mutate return statements
• statement_deletion_mutations() -> Vec<MutationOperator> - Remove statements
• constant_replacement_mutations() -> Vec<MutationOperator> - Change literals
• call_removal_mutations() -> Vec<MutationOperator> - Delete function calls

Starter Code:

#![allow(unused)]
fn main() {
impl MutationOperator {
    pub fn return_value_mutations() -> Vec<Self> {
        // TODO: Create operators that mutate return values
        // TODO: - return true → return false
        // TODO: - return x → return -x
        // TODO: - return Some(x) → return None
        // TODO: - return Ok(x) → return Err(...)
        todo!("Implement return value mutations")
    }

    pub fn statement_deletion_mutations() -> Vec<Self> {
        // TODO: Create operators that remove statements
        // TODO: - Delete variable assignments
        // TODO: - Delete function calls
        // TODO: - Delete if/while bodies
        todo!("Implement statement deletion")
    }

    pub fn constant_replacement_mutations() -> Vec<Self> {
        // TODO: Replace constants with boundary values
        // TODO: - 0 → 1, 1 → 0
        // TODO: - "" → "test"
        // TODO: - [] → [0]
        // TODO: - MAX → MIN
        todo!("Implement constant replacement")
    }

    pub fn call_removal_mutations() -> Vec<Self> {
        // TODO: Remove function calls
        // TODO: - func(); → // func();
        // TODO: - x.method(); → // x.method();
        // TODO: Tests should verify side effects happen
        todo!("Implement call removal")
    }

    pub fn all_advanced_operators() -> Vec<Self> {
        let mut ops = Vec::new();
        ops.extend(Self::return_value_mutations());
        ops.extend(Self::statement_deletion_mutations());
        ops.extend(Self::constant_replacement_mutations());
        ops.extend(Self::call_removal_mutations());
        ops
    }
}
}

Checkpoint Tests:

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_return_bool_mutation() {
        let code = r#"
fn is_positive(x: i32) -> bool {
    return x > 0;
}

#[cfg(test)]
mod tests {
    #[test]
    fn test_positive() {
        assert!(is_positive(5));
        // Missing: assert!(!is_positive(-5));
    }
}
"#;

        let operators = MutationOperator::return_value_mutations();
        let mut runner = TestRunner::new("cargo test", Duration::from_secs(10));

        // Find and test return value mutation
        let mutations = operators[0].find_mutation_points(code);
        let results = mutations.iter()
            .map(|m| runner.test_mutation(code, m))
            .collect::<Vec<_>>();

        // Should find mutation that survives (missing negative test)
        assert!(results.iter().any(|r| r.status == MutationStatus::Survived));
    }

    #[test]
    fn test_statement_deletion() {
        let code = r#"
fn process(x: &mut i32) {
    *x += 1;  // Critical statement
    *x *= 2;
}

#[cfg(test)]
mod tests {
    #[test]
    fn test_process() {
        let mut x = 5;
        process(&mut x);
        // Missing assertion!
    }
}
"#;

        let operators = MutationOperator::statement_deletion_mutations();

        // Should identify statements that can be deleted
        let mutations = operators[0].find_mutation_points(code);
        assert!(mutations.len() > 0);
    }

    #[test]
    fn test_constant_replacement() {
        let code = r#"
fn initialize() -> Vec<i32> {
    vec![0; 10]  // Initialize with zeros
}

#[cfg(test)]
mod tests {
    #[test]
    fn test_init() {
        let v = initialize();
        assert_eq!(v.len(), 10);
        // Missing: assert!(v.iter().all(|&x| x == 0));
    }
}
"#;

        let operators = MutationOperator::constant_replacement_mutations();
        let mutations = operators[0].find_mutation_points(code);

        // Should find 0 literal for replacement
        assert!(mutations.iter().any(|m| m.original == "0"));
    }

    #[test]
    fn test_call_removal() {
        let code = r#"
fn save_data(data: &str) {
    write_to_file(data);  // Side effect
    log("Data saved");    // Another side effect
}

#[cfg(test)]
mod tests {
    #[test]
    fn test_save() {
        save_data("test");
        // Doesn't verify file was written or logged!
    }
}
"#;

        let operators = MutationOperator::call_removal_mutations();
        let mutations = operators[0].find_mutation_points(code);

        // Should find both function calls
        assert!(mutations.len() >= 2);
    }

    #[test]
    fn test_combined_advanced_mutations() {
        let all_ops = MutationOperator::all_advanced_operators();

        // Should have multiple categories
        assert!(all_ops.len() >= 4);

        // Verify each category is present
        assert!(all_ops.iter().any(|op| op.name.contains("Return")));
        assert!(all_ops.iter().any(|op| op.name.contains("Delete")));
        assert!(all_ops.iter().any(|op| op.name.contains("Constant")));
        assert!(all_ops.iter().any(|op| op.name.contains("Call")));
    }
}
}

Testing Strategies

1. Unit Tests

• Mutation Operators: Verify each operator finds and transforms correctly
• Test Runner: Ensure compilation and execution work
• Report Generation: Validate metrics and formatting

2. Integration Tests

• End-to-end: Generate mutations → test → report pipeline
• Multi-file projects: Handle complex Rust projects
• Edge cases: Invalid code, infinite loops, panics

3. Performance Tests

• Parallel speedup: Measure improvement from concurrency
• Large projects: Test scalability (1000+ mutations)
• Memory usage: Ensure reasonable resource consumption

4. Mutation Testing on the Mutation Tester

• Meta-testing: Use mutation testing to test the mutation tester!
• Dogfooding: Apply the tool to itself
• Validation: Ensures the tool works correctly

Complete Working Example

#![allow(unused)]
fn main() {
// Due to length constraints, see the generated files:
// - mutation_operators.rs: Mutation operator implementations
// - test_runner.rs: Test execution engine
// - parallel_runner.rs: Parallel testing
// - report_generator.rs: Analysis and reporting
// - annotator.rs: Source code annotation
// - main.rs: CLI tool

// Run mutation testing:
// cargo run -- --source examples/lib.rs --timeout 30 --parallel 8 --format html
}

This complete mutation testing framework teaches:

• AST manipulation: Understanding code structure
• Test automation: Running and analyzing tests programmatically
• Parallel processing: Efficient use of resources
• Quality metrics: Measuring test effectiveness
• Developer tools: Building practical development aids

The framework reveals test weaknesses that coverage analysis misses, leading to more robust software.