Project 02 - Python test generator

About the Project

This project has the goal of writing a search based automated test generator for Python. It is part of the Knowledge Search & Extraction - 2023 course from the Università della Svizzera italiana.

In this repository, you can find the following files:

• benchmark/ folder: which contains the benchmark of functions under test to be instrumented

Note: Feel free to modify this file according to the project's necessities.

Environment setup

To install the required dependencies the Python version manager pyenv must be installed and in $PATH.

To set up a Python 3.11 virtualenv to execute parts 1, 2, and 3 of the project run:

deactivate || true  # deactivate existing environment
pyenv install -s 3.11
pyenv shell 3.11
python3.11 -m venv env

source env/bin/activate
pip3.11 install -r requirements_3.11.txt

To set up Python 3.7 (last version supported by mut.py) to execute part 4 of the project run:

deactivate || true  # deactivate existing environment
pyenv install -s 3.7
pyenv shell 3.7
python3.7 -m venv env37

source env37/bin/activate
pip3.7 install -r requirements_3.7.txt

Instrumentation (Part 1)

To generate the instrumented code for all the files in the benchmark run the commands:

deactivate || true
pyenv shell 3.11
source env/bin/activate

python3.11 ./instrument.py

The generated files are created in the directory instrumented. Each file name matches the file name of the corresponding source file in benchmark.

Test case generation using the fuzzer (Part 2)

To generate test cases for all files in the benchmark using the fuzzer run the commands:

deactivate || true
pyenv shell 3.11
source env/bin/activate

python3.11 ./fuzzer.py

The test suite is created in the directory fuzzer_tests. One test file is generated for each file present in the benchmark directory. Run the command with the -h options for more details on partial generation.

The test suite can be then executed over the benchmark code with the commands:

deactivate || true
pyenv shell 3.11
source env/bin/activate

python3.11 -m unittest discover fuzzer_tests

Test case generation using the genetic algorithm (Part 3)

To generate test cases for all files in the benchmark using the genetic algorithm run the commands:

deactivate || true
pyenv shell 3.11
source env/bin/activate

python3.11 ./genetic.py

The test suite is created in the directory tests. One test file is generated for each file present in the benchmark directory. Run the command with the -h options for more details on partial generation.

The test suite can be then executed over the benchmark code with the commands:

deactivate || true
pyenv shell 3.11
source env/bin/activate

python3.11 -m unittest discover tests

Mutation testing (Part 4)

To run mut.py use Python 3.7 and run:

# Reset Python to 3.7 version
deactivate || true
pyenv shell 3.7
source env37/bin/activate

python3.7 muttest.py

The script will consider the tests in fuzzer_tests and tests and run mutation testing on them, collecting the mutation score for each run in out/mutation_results_fuzzer.csv and out/mutation_results_genetic.csv respectively. If either or both file exist, the p run for the matching test suite will be skipped and the saved values will be used.

The script additionally generates two plots for the distribution and average of mutation scores per kind of generation and benchmark file. These two plots are saved in out/mutation_scores.png and out/mutation_scores_mean.png respectively. out/stats.csv is also generated and will contain a statistical comparison between the mutation score distribution for the fuzzer-generated and genetic-generated test of each benchmark file, including the average score for both generations, the Wilcoxon paired test p-value, the Cohen's d effect size and its interpretation.

Report

To compile the report run:

cd report
pdflatex -interaction=nonstopmode -output-directory=. main.tex
pdflatex -interaction=nonstopmode -output-directory=. main.tex

The report is then located in report/main.pdf.