maggicl/IRProject
Archived
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

report mvp

Browse source
This commit is contained in:
Claudio Maggioni 2020-12-09 11:07:11 +01:00
parent 3df2c17797
commit 00fe3ef509
2 changed files with 71 additions and 9 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

BIN
report/report.pdf
View file

Binary file not shown.

80
report/report.tex
View file

@ -360,7 +360,8 @@ three of my colleagues and making them install and run my project on their local
system. The examination approximately took 20 minutes for each test subject, system. The examination approximately took 20 minutes for each test subject,
including installation. including installation.
The questionnaire was implemented using Qualtrics at \href{TBD}{this link}. \subsection{User evaluation implementation}
The questionnaire was implemented using USI's Qualtrics instance.
Data for the evaluation was collected using a questionnaire with a ``before Data for the evaluation was collected using a questionnaire with a ``before
test'' and an ``after test'' section. test'' and an ``after test'' section.
@ -372,9 +373,10 @@ images or stock photos. All participants stated they were mostly familiar with
TR image search systems, and they had the chance to search for user created TR image search systems, and they had the chance to search for user created
images. Only one participant never searched for stock photos. images. Only one participant never searched for stock photos.
\begin{figure}[H] \begin{figure}[h]
\begin{tabular}{p{4cm}|p{3.5cm}|p{3.5cm}|p{3.5cm}} \begin{subfigure}{1\textwidth}
Question & Subject 1 & Subject 2 & Subject 3 \\ \begin{tabular}{p{4.5cm}|p{2.9cm}|p{3.5cm}|p{3.2cm}}
Question abbreviation & Subject 1 & Subject 2 & Subject 3 \\
\hline \hline
\textsc{metadata:} Start time & 2020-12-06 14:52 & 2020-12-07 13:35 & 2020-12-07 13:54\\ \textsc{metadata:} Start time & 2020-12-06 14:52 & 2020-12-07 13:35 & 2020-12-07 13:54\\
\textsc{metadata:} End time & 2020-12-06 14:58 & 2020-12-07 13:48 & 2020-12-07 14:05\\ \textsc{metadata:} End time & 2020-12-06 14:58 & 2020-12-07 13:48 & 2020-12-07 14:05\\
@ -382,18 +384,58 @@ images. Only one participant never searched for stock photos.
Has searched for user images & Yes & Yes & Yes \\ Has searched for user images & Yes & Yes & Yes \\
Has searched for stock photos & Yes & No & Yes \\ Has searched for stock photos & Yes & No & Yes \\
The UI was easy to use & 6 & 6 & 7 \\ The UI was easy to use & 6 & 6 & 7 \\
Accurate results for ``find a person sneezing'' & 6 & 7 & 7 \\ ``find a person sneezing'' task & 6 & 7 & 7 \\
Accurate results for ``find Varenna'' & 5 & 7 & 7 \\ ``find Varenna'' task & 5 & 7 & 7 \\
Personal task & ``Churchill Pfeil'' & Eiffel tower from query ``France'' & Personal task description & ``Churchill Pfeil'' & Eiffel tower from query ``France'' &
``Italian traditional masks'' \\ ``Italian traditional masks'' \\
Accurate results for personal task & 7 & 7 & 7 \\ Personal task & 7 & 7 & 7 \\
Clustering was helpful & 6 & 7 & 7 \\ Clustering was helpful & 6 & 7 & 7 \\
Irrelevant results were a lot and distracting & 4 & 5 & 2 \\ Irrelevant results were a lot and distracting & 4 & 5 & 2 \\
Suggestions & Missing Search button & Did not understand clustering was a Suggestions & Missing Search button & Did not understand clustering was a
filter & \textit{Great survey background image} \\ filter & \textit{Great survey background image}\footnote{The background
image for the Qualtrics survey was this:
\url{https://usi.qualtrics.com/CP/Graphic.php?IM=IM_9Bmxolx0D6iGvUp}} \\
\end{tabular} \end{tabular}
\caption{Data collected from the questionnaire.} \caption{Data collected from the questionnaire.}
\label{fig:qs} \label{fig:qs}
\end{subfigure}
\begin{subfigure}{1\textwidth}
\vspace{0.3cm}
\begin{tabular}{p{4cm} | p{11.3cm}}
Question abbreviation & Actual question presented to test subject \\
\hline
\textsc{initial disclaimer} & By proceeding with this user evaluation you
consent to be recorded and to participate in the user evaluation of the
"Image Search" Text retrieval system. The whole procedure will take at
most 15 minutes, any you are allowed to take a break or forfeit at any
time by first notifying verbally the examiner. \\
Familiarity with image search TR systems & How much are you familiar with
image search text retrieval systems (such as google images)? \\
Has searched for user images & Have you ever searched for an image in
user-created images sites such as Imgur, Pinterest, \ldots? \\
Has searched for stock photos & Have you ever searched for images in stock
image sites such as Shutterstock, 123rf, \ldots ? \\
The UI was easy to use & The UI of the ``Image Search'' TR system was easy to
use. \\
``find a person sneezing'' task & The search results for the
``find pictures of a person sneezing'' task felt accurate. \\
``find Varenna'' task & The search results for the ``
find pictures of Varenna, knowing Varenna is a town in the Lecco area``
task felt accurate. \\
Personal task description & Please describe briefly the task of personal choice you
selected. \\
Personal task & The search results for the task of
personal choice felt accurate. \\
Clustering was helpful & The results clustering feature made finding
relevant images easier. \\
Irrelevant results were a lot and distracting & The presence of irrelevant
results was significant and distracting from my search task. \\
Suggestions & Something to add? \\
\end{tabular}
\caption{Actual questions presented to the test subjects.}
\label{fig:qsa}
\end{subfigure}
\caption{The questionnaire}
\end{figure} \end{figure}
Figure \ref{fig:qs} illustrates the data gathered from the questionnaire. Figure \ref{fig:qs} illustrates the data gathered from the questionnaire.
@ -407,6 +449,26 @@ expressed in CEST without DST (corresponding to the local time of participants).
All participants started the questionnaire before they started effectively using All participants started the questionnaire before they started effectively using
the IR system, so this data can also be used to measure the experiment's length. the IR system, so this data can also be used to measure the experiment's length.
Figure \ref{fig:qsa} illustrates the actual questions presented in the exact
wording test subject read them.
\subsection{Evaluaiton results}
Results from the user evalutation appear promising. The users are generally
satisfied by the system, with the only notable exception being a general feeling
of unsatisfactory precision of the TR system (as evinced by the answers to the
``Inaccurate results'' question).
From the general suggestions emerges the need for two additional features:
\begin{itemize}
\item The need for a search button next to the search box. At the moment,
users must press Enter to search, and this may be unclear due to the fact
that this breaks the average expectation of unexperienced users and this
quirk is only described in the placeholder of the search box;
\item The need to label and explain the clustering feature to users. Subject
2 in particular was confused by the current presentation of clusters and did
not understand that this feature could be used to filter results. This issue
can be solved by either adding a textual description of the feature or by
having a ``tutorial'' of sorts when the user first opens the interface.
\end{itemize}
\end{document} \end{document}