141 lines
No EOL
4.1 KiB
Markdown
141 lines
No EOL
4.1 KiB
Markdown
# Project 02 - Python test generator
|
|
|
|
### About the Project
|
|
|
|
This project has the goal of writing a search based automated test generator for Python.
|
|
It is part of the Knowledge Search & Extraction - 2023 course from the Università della Svizzera italiana.
|
|
|
|
In this repository, you can find the following files:
|
|
|
|
- `benchmark/` folder: which contains the benchmark of functions under test to be instrumented
|
|
|
|
Note: Feel free to modify this file according to the project's necessities.
|
|
|
|
## Environment setup
|
|
|
|
To install the required dependencies the Python version manager `pyenv` must be installed and in `$PATH`.
|
|
|
|
To set up a Python 3.11 virtualenv to execute parts 1, 2, and 3 of the project run:
|
|
|
|
```shell
|
|
deactivate || true # deactivate existing environment
|
|
pyenv install -s 3.11
|
|
pyenv shell 3.11
|
|
python3.11 -m venv env
|
|
|
|
source env/bin/activate
|
|
pip3.11 install -r requirements_3.11.txt
|
|
```
|
|
|
|
To set up Python 3.7 (last version supported by `mut.py`) to execute part 4 of the project run:
|
|
|
|
```shell
|
|
deactivate || true # deactivate existing environment
|
|
pyenv install -s 3.7
|
|
pyenv shell 3.7
|
|
python3.7 -m venv env37
|
|
|
|
source env37/bin/activate
|
|
pip3.7 install -r requirements_3.7.txt
|
|
```
|
|
|
|
## Instrumentation (Part 1)
|
|
|
|
To generate the instrumented code for all the files in the benchmark run the commands:
|
|
|
|
```shell
|
|
deactivate || true
|
|
pyenv shell 3.11
|
|
source env/bin/activate
|
|
|
|
python3.11 ./instrument.py
|
|
```
|
|
|
|
The generated files are created in the directory `instrumented`. Each file name matches the file name of the
|
|
corresponding source file in `benchmark`.
|
|
|
|
## Test case generation using the fuzzer (Part 2)
|
|
|
|
To generate test cases for all files in the benchmark using the fuzzer run the commands:
|
|
|
|
```shell
|
|
deactivate || true
|
|
pyenv shell 3.11
|
|
source env/bin/activate
|
|
|
|
python3.11 ./fuzzer.py
|
|
```
|
|
|
|
The test suite is created in the directory `fuzzer_tests`. One test file is generated for each file present in the
|
|
`benchmark` directory. Run the command with the `-h` options for more details on partial generation.
|
|
|
|
The test suite can be then executed over the benchmark code with the commands:
|
|
|
|
```shell
|
|
deactivate || true
|
|
pyenv shell 3.11
|
|
source env/bin/activate
|
|
|
|
python3.11 -m unittest discover fuzzer_tests
|
|
```
|
|
|
|
## Test case generation using the genetic algorithm (Part 3)
|
|
|
|
To generate test cases for all files in the benchmark using the genetic algorithm run the commands:
|
|
|
|
```shell
|
|
deactivate || true
|
|
pyenv shell 3.11
|
|
source env/bin/activate
|
|
|
|
python3.11 ./genetic.py
|
|
```
|
|
|
|
The test suite is created in the directory `tests`. One test file is generated for each file present in the
|
|
`benchmark` directory. Run the command with the `-h` options for more details on partial generation.
|
|
|
|
The test suite can be then executed over the benchmark code with the commands:
|
|
|
|
```shell
|
|
deactivate || true
|
|
pyenv shell 3.11
|
|
source env/bin/activate
|
|
|
|
python3.11 -m unittest discover tests
|
|
```
|
|
|
|
## Mutation testing (Part 4)
|
|
|
|
To run `mut.py` use Python 3.7 and run:
|
|
|
|
```shell
|
|
# Reset Python to 3.7 version
|
|
deactivate || true
|
|
pyenv shell 3.7
|
|
source env37/bin/activate
|
|
|
|
python3.7 muttest.py
|
|
```
|
|
|
|
The script will consider the tests in `fuzzer_tests` and `tests` and run mutation testing on them, collecting the
|
|
mutation score for each run in `out/mutation_results_fuzzer.csv` and `out/mutation_results_genetic.csv` respectively.
|
|
If either or both file exist, the p run for the matching test suite will be skipped and the saved values will be
|
|
used.
|
|
|
|
The script additionally generates two plots for the distribution and average of mutation scores per kind of generation
|
|
and benchmark file. These two plots are saved in `out/mutation_scores.png` and `out/mutation_scores_mean.png`
|
|
respectively. `out/stats.csv` is also generated and will contain a statistical comparison between the mutation score
|
|
distribution for the fuzzer-generated and genetic-generated test of each benchmark file, including the average score for
|
|
both generations, the Wilcoxon paired test p-value, the Cohen's d effect size and its interpretation.
|
|
|
|
# Report
|
|
|
|
To compile the report run:
|
|
|
|
```shell
|
|
cd report
|
|
pdflatex -interaction=nonstopmode -output-directory=. main.tex
|
|
pdflatex -interaction=nonstopmode -output-directory=. main.tex
|
|
```
|
|
|
|
The report is then located in `report/main.pdf`. |