report mvp
This commit is contained in:
parent
3df2c17797
commit
00fe3ef509
2 changed files with 71 additions and 9 deletions
Binary file not shown.
|
|
@ -360,7 +360,8 @@ three of my colleagues and making them install and run my project on their local
|
|||
system. The examination approximately took 20 minutes for each test subject,
|
||||
including installation.
|
||||
|
||||
The questionnaire was implemented using Qualtrics at \href{TBD}{this link}.
|
||||
\subsection{User evaluation implementation}
|
||||
The questionnaire was implemented using USI's Qualtrics instance.
|
||||
|
||||
Data for the evaluation was collected using a questionnaire with a ``before
|
||||
test'' and an ``after test'' section.
|
||||
|
|
@ -372,9 +373,10 @@ images or stock photos. All participants stated they were mostly familiar with
|
|||
TR image search systems, and they had the chance to search for user created
|
||||
images. Only one participant never searched for stock photos.
|
||||
|
||||
\begin{figure}[H]
|
||||
\begin{tabular}{p{4cm}|p{3.5cm}|p{3.5cm}|p{3.5cm}}
|
||||
Question & Subject 1 & Subject 2 & Subject 3 \\
|
||||
\begin{figure}[h]
|
||||
\begin{subfigure}{1\textwidth}
|
||||
\begin{tabular}{p{4.5cm}|p{2.9cm}|p{3.5cm}|p{3.2cm}}
|
||||
Question abbreviation & Subject 1 & Subject 2 & Subject 3 \\
|
||||
\hline
|
||||
\textsc{metadata:} Start time & 2020-12-06 14:52 & 2020-12-07 13:35 & 2020-12-07 13:54\\
|
||||
\textsc{metadata:} End time & 2020-12-06 14:58 & 2020-12-07 13:48 & 2020-12-07 14:05\\
|
||||
|
|
@ -382,18 +384,58 @@ images. Only one participant never searched for stock photos.
|
|||
Has searched for user images & Yes & Yes & Yes \\
|
||||
Has searched for stock photos & Yes & No & Yes \\
|
||||
The UI was easy to use & 6 & 6 & 7 \\
|
||||
Accurate results for ``find a person sneezing'' & 6 & 7 & 7 \\
|
||||
Accurate results for ``find Varenna'' & 5 & 7 & 7 \\
|
||||
Personal task & ``Churchill Pfeil'' & Eiffel tower from query ``France'' &
|
||||
``find a person sneezing'' task & 6 & 7 & 7 \\
|
||||
``find Varenna'' task & 5 & 7 & 7 \\
|
||||
Personal task description & ``Churchill Pfeil'' & Eiffel tower from query ``France'' &
|
||||
``Italian traditional masks'' \\
|
||||
Accurate results for personal task & 7 & 7 & 7 \\
|
||||
Personal task & 7 & 7 & 7 \\
|
||||
Clustering was helpful & 6 & 7 & 7 \\
|
||||
Irrelevant results were a lot and distracting & 4 & 5 & 2 \\
|
||||
Suggestions & Missing Search button & Did not understand clustering was a
|
||||
filter & \textit{Great survey background image} \\
|
||||
filter & \textit{Great survey background image}\footnote{The background
|
||||
image for the Qualtrics survey was this:
|
||||
\url{https://usi.qualtrics.com/CP/Graphic.php?IM=IM_9Bmxolx0D6iGvUp}} \\
|
||||
\end{tabular}
|
||||
\caption{Data collected from the questionnaire.}
|
||||
\label{fig:qs}
|
||||
\end{subfigure}
|
||||
\begin{subfigure}{1\textwidth}
|
||||
\vspace{0.3cm}
|
||||
\begin{tabular}{p{4cm} | p{11.3cm}}
|
||||
Question abbreviation & Actual question presented to test subject \\
|
||||
\hline
|
||||
\textsc{initial disclaimer} & By proceeding with this user evaluation you
|
||||
consent to be recorded and to participate in the user evaluation of the
|
||||
"Image Search" Text retrieval system. The whole procedure will take at
|
||||
most 15 minutes, any you are allowed to take a break or forfeit at any
|
||||
time by first notifying verbally the examiner. \\
|
||||
Familiarity with image search TR systems & How much are you familiar with
|
||||
image search text retrieval systems (such as google images)? \\
|
||||
Has searched for user images & Have you ever searched for an image in
|
||||
user-created images sites such as Imgur, Pinterest, \ldots? \\
|
||||
Has searched for stock photos & Have you ever searched for images in stock
|
||||
image sites such as Shutterstock, 123rf, \ldots ? \\
|
||||
The UI was easy to use & The UI of the ``Image Search'' TR system was easy to
|
||||
use. \\
|
||||
``find a person sneezing'' task & The search results for the
|
||||
``find pictures of a person sneezing'' task felt accurate. \\
|
||||
``find Varenna'' task & The search results for the ``
|
||||
find pictures of Varenna, knowing Varenna is a town in the Lecco area``
|
||||
task felt accurate. \\
|
||||
Personal task description & Please describe briefly the task of personal choice you
|
||||
selected. \\
|
||||
Personal task & The search results for the task of
|
||||
personal choice felt accurate. \\
|
||||
Clustering was helpful & The results clustering feature made finding
|
||||
relevant images easier. \\
|
||||
Irrelevant results were a lot and distracting & The presence of irrelevant
|
||||
results was significant and distracting from my search task. \\
|
||||
Suggestions & Something to add? \\
|
||||
\end{tabular}
|
||||
\caption{Actual questions presented to the test subjects.}
|
||||
\label{fig:qsa}
|
||||
\end{subfigure}
|
||||
\caption{The questionnaire}
|
||||
\end{figure}
|
||||
|
||||
Figure \ref{fig:qs} illustrates the data gathered from the questionnaire.
|
||||
|
|
@ -407,6 +449,26 @@ expressed in CEST without DST (corresponding to the local time of participants).
|
|||
All participants started the questionnaire before they started effectively using
|
||||
the IR system, so this data can also be used to measure the experiment's length.
|
||||
|
||||
Figure \ref{fig:qsa} illustrates the actual questions presented in the exact
|
||||
wording test subject read them.
|
||||
|
||||
\subsection{Evaluaiton results}
|
||||
Results from the user evalutation appear promising. The users are generally
|
||||
satisfied by the system, with the only notable exception being a general feeling
|
||||
of unsatisfactory precision of the TR system (as evinced by the answers to the
|
||||
``Inaccurate results'' question).
|
||||
|
||||
From the general suggestions emerges the need for two additional features:
|
||||
\begin{itemize}
|
||||
\item The need for a search button next to the search box. At the moment,
|
||||
users must press Enter to search, and this may be unclear due to the fact
|
||||
that this breaks the average expectation of unexperienced users and this
|
||||
quirk is only described in the placeholder of the search box;
|
||||
\item The need to label and explain the clustering feature to users. Subject
|
||||
2 in particular was confused by the current presentation of clusters and did
|
||||
not understand that this feature could be used to filter results. This issue
|
||||
can be solved by either adding a textual description of the feature or by
|
||||
having a ``tutorial'' of sorts when the user first opens the interface.
|
||||
\end{itemize}
|
||||
|
||||
\end{document}
|
||||
|
|
|
|||
Reference in a new issue