Warning: mysql_pconnect() has been disabled for security reasons in /storage/content/59/104659/eyetracking.me/public_html/wp-content/plugins/proplayer/playlist-controller.php on line 19 Warning: mysql_select_db() expects parameter 2 to be resource, null given in /storage/content/59/104659/eyetracking.me/public_html/wp-content/plugins/proplayer/playlist-controller.php on line 20 Warning: mysql_pconnect() has been disabled for security reasons in /storage/content/59/104659/eyetracking.me/public_html/wp-content/plugins/proplayer/playlist-controller.php on line 19 Warning: mysql_select_db() expects parameter 2 to be resource, null given in /storage/content/59/104659/eyetracking.me/public_html/wp-content/plugins/proplayer/playlist-controller.php on line 20 New study: Retrospective Think Aloud and Eye Tracking | EyeTracking.Me a Blog by Tommy Strandvall

New study: Retrospective Think Aloud and Eye Tracking

During the summer we have conducted a study at Tobii Technology aimed to compare the outcomes from four different retrospective think aloud (RTA) methods in a web usability study: a non cued RTA, a screen video cued RTA, a gaze plot cued RTA, and a gaze video cued RTA. Twenty-four participants were used in the study, six participants in every RTA condition included.

rta_number_words

The Tobii T120 remote eye-tracker along with the Tobii Studio 2 (Enterprise version) software was used to record and replay participants’ eye movements as well as for recording the RTA interview.  The website used in this test was www.spotify.com. On the web site users can sign up for a paid subscription and then download a software client which provides legal streaming of music. Participants in the test were given one task to complete on the web site: Sign up for a monthly subscription with Spotify and then download the Spotify software.

So what did we find out?

When comparing the interview word counts for the four different RTA methods we found statistically significant differences between the groups.

rta_number_of_words

The ‘no cue’ condition produced significantly fewer words than the ‘Gaze Plot Cue’ and the ‘Gaze Video Cue’. Although no other groups revealed significant differences, there is a trend towards the ‘gaze video cue’ and ‘gaze plot cue’ producing more words than a regular ‘video cue’ and ‘no cue’ as can be seen in the chart.

When comparing the number of usability comments and usability problems mentioned in the different groups the results indicated a trend towards the ‘gaze plot’ cued and ‘gaze video’ cued groups identifying the highest number of unique usability problems, with the ‘no cue’ condition producing the least. The different cues appeared to stimulate the participants in different ways when commenting on their behavior. The two video conditions stimulated the participants to produce considerably more ‘manipulative’ comments than when a static gaze plot or no cue was used.

rta_summary

The conclusions were that using any cue will stimulate participants to produce a higher number of words and comments as well as helping participants identifying more usability problems. Using eye movements as cue, by presenting gaze plots or gaze videos to the participants, can stimulate the participants even more. The method that produced the highest quantity of interview data, in number of comments and number of words, was the gaze video cued RTA method, followed by the gaze plot cued RTA method. Our study clearly demonstrated that using any kind of cue is significantly better than using no cue at all. Additionally, by using gaze plots or gaze videos as cues, participants provided more feedback than when only using a screen video as cue. Another interesting conclusion is that the gaze plot cued method proved to perform almost as well as the gaze video cued method, being especially good at producing visual and cognitive comments and at the same time identifying the same amount of usability problems as the gaze video cued method.

The results from our study will be presented more in depth in a whitepaper to be published on tobii.com soon. I will also post a link here on my blog when the paper is available.

There is also a new short YouTube video from Tobii describing RTA and the value of eye tracking. Watch it below:

Share

4 comments to New study: Retrospective Think Aloud and Eye Tracking

  • I would be interested if you have any measure of false findings. We find that users make stuff up with think aloud and eye tracking corrects this. People don’t know why they do things, so asking them to recall without a cue encourages them to make stuff up.

    Good to see Tobii working on the methodology side of things.

    Guy

  • We intend to analyze the data further. At the moment we have not compared the validity of the comments in the different groups. One general finding was that participants in the ‘no cue’ group tended to trivialize any problems they had and over-generalized the process as easy and problem free. This was also indicated by the fact that the number of usability problems mentioned as well as number of comments in the ‘no cue’ group was very low compared to the other groups.

  • At first I was quite taken with these results. But, now that I know RTA is an upsell for the upcoming Tobii Studio release, I am more skeptical. So, to actually do this type of research, we essentially have to pay Tobii. No thanks.

  • Thank you for your comment. Just to clarify, RTA is a research method, not a feature or technology, so there is no need to have any specific technology or equipment to be able to use the RTA method. It is true that an RTA recording function is included in a recent release of Tobii Studio (2.0) but the method has nothing to do with Tobii or Tobii Studio. The feature was simply included because many usability practitioners use this method, including myself for several years (using only a video camera and a voice recorder). The research project itself was conducted in a scientific and objective manner with strict research protocols, and the results are not influenced by the hardware or software tools used for collecting or analyzing the data. As pointed out in the paper many other studies have found similar results, but because we did not find any research on using gaze plots and still images as a cue we especially wanted to test the impact of these in an RTA interview. I would be very happy to answer any further questions you might have on how to use RTA as a method or how to conduct an RTA interview using different tools.