diff --git a/report/report.pdf b/report/report.pdf index 1310251..b51d59c 100644 Binary files a/report/report.pdf and b/report/report.pdf differ diff --git a/report/report.tex b/report/report.tex index 4c71683..34539e9 100644 --- a/report/report.tex +++ b/report/report.tex @@ -360,7 +360,8 @@ three of my colleagues and making them install and run my project on their local system. The examination approximately took 20 minutes for each test subject, including installation. -The questionnaire was implemented using Qualtrics at \href{TBD}{this link}. +\subsection{User evaluation implementation} +The questionnaire was implemented using USI's Qualtrics instance. Data for the evaluation was collected using a questionnaire with a ``before test'' and an ``after test'' section. @@ -372,9 +373,10 @@ images or stock photos. All participants stated they were mostly familiar with TR image search systems, and they had the chance to search for user created images. Only one participant never searched for stock photos. -\begin{figure}[H] - \begin{tabular}{p{4cm}|p{3.5cm}|p{3.5cm}|p{3.5cm}} - Question & Subject 1 & Subject 2 & Subject 3 \\ +\begin{figure}[h] + \begin{subfigure}{1\textwidth} + \begin{tabular}{p{4.5cm}|p{2.9cm}|p{3.5cm}|p{3.2cm}} + Question abbreviation & Subject 1 & Subject 2 & Subject 3 \\ \hline \textsc{metadata:} Start time & 2020-12-06 14:52 & 2020-12-07 13:35 & 2020-12-07 13:54\\ \textsc{metadata:} End time & 2020-12-06 14:58 & 2020-12-07 13:48 & 2020-12-07 14:05\\ @@ -382,18 +384,58 @@ images. Only one participant never searched for stock photos. Has searched for user images & Yes & Yes & Yes \\ Has searched for stock photos & Yes & No & Yes \\ The UI was easy to use & 6 & 6 & 7 \\ - Accurate results for ``find a person sneezing'' & 6 & 7 & 7 \\ - Accurate results for ``find Varenna'' & 5 & 7 & 7 \\ - Personal task & ``Churchill Pfeil'' & Eiffel tower from query ``France'' & + ``find a person sneezing'' task & 6 & 7 & 7 \\ + ``find Varenna'' task & 5 & 7 & 7 \\ + Personal task description & ``Churchill Pfeil'' & Eiffel tower from query ``France'' & ``Italian traditional masks'' \\ - Accurate results for personal task & 7 & 7 & 7 \\ + Personal task & 7 & 7 & 7 \\ Clustering was helpful & 6 & 7 & 7 \\ Irrelevant results were a lot and distracting & 4 & 5 & 2 \\ Suggestions & Missing Search button & Did not understand clustering was a - filter & \textit{Great survey background image} \\ + filter & \textit{Great survey background image}\footnote{The background + image for the Qualtrics survey was this: + \url{https://usi.qualtrics.com/CP/Graphic.php?IM=IM_9Bmxolx0D6iGvUp}} \\ \end{tabular} \caption{Data collected from the questionnaire.} \label{fig:qs} + \end{subfigure} + \begin{subfigure}{1\textwidth} + \vspace{0.3cm} + \begin{tabular}{p{4cm} | p{11.3cm}} + Question abbreviation & Actual question presented to test subject \\ + \hline + \textsc{initial disclaimer} & By proceeding with this user evaluation you + consent to be recorded and to participate in the user evaluation of the + "Image Search" Text retrieval system. The whole procedure will take at + most 15 minutes, any you are allowed to take a break or forfeit at any + time by first notifying verbally the examiner. \\ + Familiarity with image search TR systems & How much are you familiar with + image search text retrieval systems (such as google images)? \\ + Has searched for user images & Have you ever searched for an image in + user-created images sites such as Imgur, Pinterest, \ldots? \\ + Has searched for stock photos & Have you ever searched for images in stock + image sites such as Shutterstock, 123rf, \ldots ? \\ + The UI was easy to use & The UI of the ``Image Search'' TR system was easy to + use. \\ + ``find a person sneezing'' task & The search results for the + ``find pictures of a person sneezing'' task felt accurate. \\ + ``find Varenna'' task & The search results for the `` + find pictures of Varenna, knowing Varenna is a town in the Lecco area`` + task felt accurate. \\ + Personal task description & Please describe briefly the task of personal choice you + selected. \\ + Personal task & The search results for the task of + personal choice felt accurate. \\ + Clustering was helpful & The results clustering feature made finding + relevant images easier. \\ + Irrelevant results were a lot and distracting & The presence of irrelevant + results was significant and distracting from my search task. \\ + Suggestions & Something to add? \\ + \end{tabular} + \caption{Actual questions presented to the test subjects.} + \label{fig:qsa} + \end{subfigure} + \caption{The questionnaire} \end{figure} Figure \ref{fig:qs} illustrates the data gathered from the questionnaire. @@ -407,6 +449,26 @@ expressed in CEST without DST (corresponding to the local time of participants). All participants started the questionnaire before they started effectively using the IR system, so this data can also be used to measure the experiment's length. +Figure \ref{fig:qsa} illustrates the actual questions presented in the exact +wording test subject read them. +\subsection{Evaluaiton results} +Results from the user evalutation appear promising. The users are generally +satisfied by the system, with the only notable exception being a general feeling +of unsatisfactory precision of the TR system (as evinced by the answers to the +``Inaccurate results'' question). + +From the general suggestions emerges the need for two additional features: +\begin{itemize} + \item The need for a search button next to the search box. At the moment, + users must press Enter to search, and this may be unclear due to the fact + that this breaks the average expectation of unexperienced users and this + quirk is only described in the placeholder of the search box; + \item The need to label and explain the clustering feature to users. Subject + 2 in particular was confused by the current presentation of clusters and did + not understand that this feature could be used to filter results. This issue + can be solved by either adding a textual description of the feature or by + having a ``tutorial'' of sorts when the user first opens the interface. +\end{itemize} \end{document}