kse-02/README.md

# Project 02 - Python test generator

### About the Project

This project has the goal of writing a search based automated test generator for Python.
It is part of the Knowledge Search & Extraction - 2023 course from the Università della Svizzera italiana.

In this repository, you can find the following files:

- `benchmark/` folder: which contains the benchmark of functions under test to be instrumented

Note: Feel free to modify this file according to the project's necessities.

## Environment setup

To install the required dependencies the Python version manager `pyenv` must be installed and in `$PATH`.

To set up a Python 3.11 virtualenv to execute parts 1, 2, and 3 of the project run:

```shell
deactivate || true  # deactivate existing environment
pyenv install -s 3.11
pyenv shell 3.11
python3.11 -m venv env

source env/bin/activate
pip3.11 install -r requirements_3.11.txt
```

To set up Python 3.7 (last version supported by `mut.py`) to execute part 4 of the project run:

```shell
deactivate || true  # deactivate existing environment
pyenv install -s 3.7
pyenv shell 3.7
python3.7 -m venv env37

source env37/bin/activate
pip3.7 install -r requirements_3.7.txt
```

## Instrumentation (Part 1)

To generate the instrumented code for all the files in the benchmark run the commands:

```shell
deactivate || true
pyenv shell 3.11
source env/bin/activate

python3.11 ./instrument.py
```

The generated files are created in the directory `instrumented`. Each file name matches the file name of the
corresponding source file in `benchmark`.

## Test case generation using the fuzzer (Part 2)

To generate test cases for all files in the benchmark using the fuzzer run the commands:

```shell
deactivate || true
pyenv shell 3.11
source env/bin/activate

python3.11 ./fuzzer.py
```

The test suite is created in the directory `fuzzer_tests`. One test file is generated for each file present in the
`benchmark` directory. Run the command with the `-h` options for more details on partial generation.

The test suite can be then executed over the benchmark code with the commands:

```shell
deactivate || true
pyenv shell 3.11
source env/bin/activate

python3.11 -m unittest discover fuzzer_tests
```

## Test case generation using the genetic algorithm (Part 3)

To generate test cases for all files in the benchmark using the genetic algorithm run the commands:

```shell
deactivate || true
pyenv shell 3.11
source env/bin/activate

python3.11 ./genetic.py
```

The test suite is created in the directory `tests`. One test file is generated for each file present in the
`benchmark` directory. Run the command with the `-h` options for more details on partial generation.

The test suite can be then executed over the benchmark code with the commands:

```shell
deactivate || true
pyenv shell 3.11
source env/bin/activate

python3.11 -m unittest discover tests
```

## Mutation testing (Part 4)

To run `mut.py` use Python 3.7 and run:

```shell
# Reset Python to 3.7 version
deactivate || true
pyenv shell 3.7
source env37/bin/activate

python3.7 muttest.py
```

The script will consider the tests in `fuzzer_tests` and `tests` and run mutation testing on them, collecting the
mutation score for each run in `out/mutation_results_fuzzer.csv` and `out/mutation_results_genetic.csv` respectively.
If either or both file exist, the mutation run for the matching test suite will be skipped and the saved values will be
used.

The script additionally generates two plots for the distribution and average of mutation scores per kind of generation
and benchmark file. These two plots are saved in `out/mutation_scores.png` and `out/mutation_scores_mean.png`
respectively. `out/stats.csv` is also generated and will contain a statistical comparison between the mutation score
distribution for the fuzzer-generated and genetic-generated test of each benchmark file, including the average score for
both generations, the Wilcoxon paired test p-value, the Cohen's d effect size and its interpretation.