SAFuzzer 1. SAFuzzer Project Introduction SAFuzzer is an innovative framework for testing Static Application Security Testing (SAST) tools through semantic-preserving code mutations. The framework employs a three-phase pipeline: Mutator Invention: Mines patterns from real-world refactoring commits and transforms them into executable Spoon-based mutators via LLM agents Mutator Refinement: Validates each mutator's semantic-preservation guarantee through rigorous dynamic equivalence checking Static Analyzer Testing: Applies validated mutators at scale to test static analyzers via metamorphic testing SAFuzzer supports mainstream SAST tools including SpotBugs, PMD, Infer, CheckStyle, and SonarQube. The framework uses Java Spoon for AST manipulation . 2. Top 10 Mutators Causing Bugs The following are the top 10 mutators that most frequently cause bugs in SAST tools during testing: Rank Mutator Name Issue Count Issues 1 EqualityCheckToInstanceofMutator 4 SpotBugs #3916, Infer #2001, Sonar S2259, PMD #6513 2 ConditionalBlockInsertionMutator 4 SpotBugs #3894, Infer #2015, SpotBugs #3929, PMD #6518 3 ParenthesesAdditionMutator 4 SpotBugs #3904, Infer #2015, PMD #6491, CheckStyle #19162 4 IfConditionReorderingMutator 3 SpotBugs #3886, SpotBugs #3920, SpotBugs #3963 5 VariableAdditionMutator 3 SpotBugs #3884, Infer #2015, Infer #1993 6 NullCheckReorderingMutator 3 SpotBugs #3920, SpotBugs #3886, SpotBugs #3916 7 ConditionalLogicInsertionMutator 2 PMD #6518, SpotBugs #3978 8 MethodChainCallSwapMutator 2 SpotBugs #3966, PMD #6494 9 ConditionNegationMutator 2 PMD #6435, SpotBugs #3963 10 SingleLineReturnToBlockReturnMutator 2 PMD #6491, PMD #6519 3. SAFuzzer Usage Guide Project Architecture Overview SAFuzzer consists of three main components: Semantic_Equivalence_Knowledge_Base: A Python-based pipeline for mining semantic-preserving code patterns from real-world refactoring commits using LLM agents and dynamic execution validation. This component extracts transformation patterns from GitHub commits and validates their semantic equivalence. MutatorExecutor: A standalone Maven project containing mutator implementations and semantic equivalence validation module. This component is used by the knowledge base pipeline to dynamically validate semantic equivalence of code transformations. Main SAFuzzer Framework: The core Java application that applies validated mutators to test SAST tools via metamorphic testing. This is the primary tool for detecting bugs in static analyzers. Mutator Generation and Validation Pipeline The framework includes three Python scripts that automate the mutator invention and refinement process: 1. Stage 1: Mutator Generation (stage1_generator.py) Input: Code pairs from GitHub refactoring commits (raw_diffs_chunk_*_output.json) Output: Mutator descriptions (JSON) and Java implementations Process: Uses LLM to analyze code pairs and generate mutator descriptions Converts descriptions into executable Spoon-based Java mutators Output Structure: outputs/ ├── 1_mutator_description/ # JSON descriptions └── 2_mutator_implementation/ # Java implementations 2. Stage 2: Compilation Verification (stage2_compilation_verification.py) Input: Java mutators from Stage 1 Output: Compilable mutators Process: Deploys mutators to sandbox environment Validates compilation using javac Automatically repairs compilation errors using LLM agents (max 5 attempts) Key Features: Parallel processing (16 workers) Sandbox isolation for each mutator Intelligent repair with specialized tools Output Structure: outputs/compilable_3-7/ # Compilable mutators 3. Stage 3: Fast Semantic Verification (stage3_fast_verify.py) Input: Compilable mutators + test seeds Output: Validation results + repair datasets Process: Phase 1: Quick verification with 200 seeds Phase 2: Extends to 500 seeds if pass rate < 90% Phase 3: Extends to 1000 seeds if zero triggers Repair loop: LLM-driven repair for failed mutators (max 3 attempts) Validation Criteria: Pass: Trigger rate > 0 AND pass rate ≥ 90% (200 seeds) OR ≥ 80% (extended seeds) Fail: Pass rate not met OR zero triggers Output Structure: lists/ ├── success_list_v2.txt # Successfully validated mutators └── fail_list_v2.txt # Failed mutators refine_dataset/ # Detailed repair datasets (JSON) Running the Pipeline # Step 1: Generate mutators from refactoring commits python stage1_generator.py # Step 2: Verify and repair compilation errors python stage2_compilation_verification.py [start_chunk] # Step 3: Validate semantic preservation python stage3_fast_verify.py Environment Requirements Java 17 or higher (Maven compilation target) Python 3.8+ (for analysis scripts in Semantic_Equivalence_Knowledge_Base) Maven 3.6+ for building the project 8GB RAM minimum, 16GB RAM recommended 20GB free disk space for generated mutants and results Quick Start with the Core Framework Package Step 1: Extract and Setup # Extract the ZIP file unzip SAFuzzer_Core_Framework_*.zip cd SAFuzzer_Core_Framework # Make scripts executable chmod +x run_complete_pipeline.sh test_pipeline_quick.sh chmod +x Semantic_Equivalence_Knowledge_Base/run_pipeline.sh Step 2: Install SAST Tools Before running SAFuzzer, you need to install the SAST tools. Follow the instructions in tools/README.md to download and install: SpotBugs 4.9.8 PMD 7.22.0 CheckStyle 13.3.0 Infer 1.2.0 SonarQube Scanner 8.0.1 Step 3: Configure Tool Paths # Copy the configuration template cp config.properties.template config.properties # Edit config.properties with your tool paths nano config.properties # or use your favorite editor Update the paths in config.properties: spotbugs.jar.path=/absolute/path/to/spotbugs-4.9.8/lib/spotbugs.jar pmd.cli.path=/absolute/path/to/pmd-bin-7.22.0/bin/pmd checkstyle.jar.path=/absolute/path/to/checkstyle-13.3.0-all.jar infer.cli.path=/absolute/path/to/infer-linux-x86_64-v1.2.0/bin/infer sonar.scanner.path=/absolute/path/to/sonar-scanner-8.0.1.6346-linux-x64/bin/sonar-scanner Step 4: Build the Project # Build main SAFuzzer framework mvn clean compile package # Build MutatorExecutor (for semantic validation) cd MutatorExecutor mvn clean compile cd .. Step 5: Install Python Dependencies # Install required Python packages pip install -r Semantic_Equivalence_Knowledge_Base/requirements.txt Step 6: Run Quick Verification Test # Test if everything works correctly ./test_pipeline_quick.sh If all tests pass, you're ready to run the full pipeline! Running the Complete Pipeline Option A: Run All Three Stages (Recommended) # This runs the complete SAFuzzer pipeline end-to-end ./run_complete_pipeline.sh The script will: Check environment and dependencies Build the project if needed Run the Semantic Equivalence Knowledge Base pipeline (Stage 1) Validate mutators using MutatorExecutor (Stage 2) Test SAST tools with validated mutators (Stage 3) Generate results and summary Option B: Run Individual Stages Stage 1: Mutator Invention (Pattern Mining) cd Semantic_Equivalence_Knowledge_Base ./run_pipeline.sh This stage mines refactoring patterns from GitHub commits. Note: This requires GitHub API access and may take several hours. Stage 2: Mutator Refinement (Semantic Validation) cd MutatorExecutor mvn compile # The validation is integrated into Stage 1 pipeline Stage 3: Testing Static Analyzers # Test a specific test case with SpotBugs java -cp "target/SASTFuzz-1.0-.jar:target/classes:target/dependency/*" \ com.mutation.Main \ --project_path "." \ --target_case "seeds.PMD_Seeds.bestpractices_AccessorClassGeneration.AccessorClassGeneration1" \ --target_SAST "SpotBugs" \ --max_iter 10 # Test all SAST tools on a test case java -cp "target/SASTFuzz-1.0-.jar:target/classes:target/dependency/*" \ com.mutation.Main \ --project_path "." \ --target_case "seeds.SpotBugs_Seeds.bestpractices_ArrayIsStoredDirectly.ArrayIsStoredDirectly1" \ --target_SAST "ALL" \ --max_iter 20 Command Line Parameters --project_path Source code root directory (required) --target_case Target Java class (package.ClassName format) (required) --target_SAST SAST tool to test: SpotBugs, PMD, CheckStyle, Infer, SonarQube, Semgrep, or ALL (required) --max_iter Maximum mutation iterations (default: 50) Output Structure Results are organized in results/run_YYYYMMDD_HHMMSS/: safuzzer_output.log: Complete execution log final_results/: Generated mutants and SAST reports 0/: Original seed code with baseline SAST analysis 1..N/: Each iteration's mutated code and SAST results iteration_history.txt: Trace of applied mutators verification_summary.txt: Pipeline verification results Advanced Configuration Custom Mutator Selection The framework automatically selects from all available mutators. To modify mutator behavior, edit the Scheduler.run() method in src/com/mutation/Scheduler.java. Rule Coverage Experiment Enable JaCoCo coverage measurement in config.properties: jacoco.enabled=true jacoco.agent.path=/path/to/jacoco-agent.jar jacoco.cli.path=/path/to/jacoco-cli.jar Custom SAST Tool Integration Implement new SAST tool classes extending the SAST abstract class in src/com/mutation/config/. 4. Detected Bug Case Demonstrations Case 1: PMD SimplifyConditional False Negative (#6513) Bug Description: PMD fails to detect a redundant null check before instanceof when additional conditions are interleaved in the && chain by a semantic-preserving mutation. Original Code (PMD correctly reports Simpli

@inproceedings{paperbot12306,
  title = {Testing Static Analyzers via Semantic-Preserving Mutators Learned from Real-World Refactoring Practice},
  author = {Meilin Li and Kaixuan Li and Zifan Xie and Shiyu Qiu and Ming Wen and Maolin Sun and Hongyu Zhang},
  booktitle = {ASE},
  year = {2026},
  doi = {10.1145/3832783.3837544}
}