During the summer we have conducted a study at Tobii Technology aimed to compare the outcomes from four different retrospective think aloud (RTA) methods in a web usability study: a non cued RTA, a screen video cued RTA, a gaze plot cued RTA, and a gaze video cued RTA. Twenty-four participants were used in the study, six participants in every RTA condition included.
The Tobii T120 remote eye-tracker along with the Tobii Studio 2 (Enterprise version) software was used to record and replay participants’ eye movements as well as for recording the RTA interview. The website used in this test was www.spotify.com. On the web site users can sign up for a paid subscription and then download a software client which provides legal streaming of music. Participants in the test were given one task to complete on the web site: Sign up for a monthly subscription with Spotify and then download the Spotify software.
So what did we find out?
When comparing the interview word counts for the four different RTA methods we found statistically significant differences between the groups.
The ‘no cue’ condition produced significantly fewer words than the ‘Gaze Plot Cue’ and the ‘Gaze Video Cue’. Although no other groups revealed significant differences, there is a trend towards the ‘gaze video cue’ and ‘gaze plot cue’ producing more words than a regular ‘video cue’ and ‘no cue’ as can be seen in the chart.
When comparing the number of usability comments and usability problems mentioned in the different groups the results indicated a trend towards the ‘gaze plot’ cued and ‘gaze video’ cued groups identifying the highest number of unique usability problems, with the ‘no cue’ condition producing the least. The different cues appeared to stimulate the participants in different ways when commenting on their behavior. The two video conditions stimulated the participants to produce considerably more ‘manipulative’ comments than when a static gaze plot or no cue was used.
The conclusions were that using any cue will stimulate participants to produce a higher number of words and comments as well as helping participants identifying more usability problems. Using eye movements as cue, by presenting gaze plots or gaze videos to the participants, can stimulate the participants even more. The method that produced the highest quantity of interview data, in number of comments and number of words, was the gaze video cued RTA method, followed by the gaze plot cued RTA method. Our study clearly demonstrated that using any kind of cue is significantly better than using no cue at all. Additionally, by using gaze plots or gaze videos as cues, participants provided more feedback than when only using a screen video as cue. Another interesting conclusion is that the gaze plot cued method proved to perform almost as well as the gaze video cued method, being especially good at producing visual and cognitive comments and at the same time identifying the same amount of usability problems as the gaze video cued method.
The results from our study will be presented more in depth in a whitepaper to be published on tobii.com soon. I will also post a link here on my blog when the paper is available.
There is also a new short YouTube video from Tobii describing RTA and the value of eye tracking. Watch it below: