master-abschlussvortrag
Using Gaze Data to Enhance Questionnaires
Christian Weiss
Florian Alt
Did a participant really understand the questions?
Idea
Use gaze data to predict question comprehension.
Benefits
- Identification of problematic questions
- More valuable data
Eye Tracking
The Eye
- Only 2° of the field of view are seen with maximum acuity
- Constant movements necessary
Eye Movements
- Saccade: rapid, jittery movements
- Fixation: stable focus on a single position
Eye Tracking - Overview
- Measuring and recording eye movements
- Estimating the point of focus
Eye Tracking - Use Cases
- Marketing & Advertising
- Usability
- Accessibility
- Neuroscience and Psychology
- Medicine
- Gaming
- Driving
Eye Tracking - Technology
- Eye-attached trackers
- Electro-oculography
- Optical Tracking (cornal reflection)
Eye Tracking - Metrics
-
Fixations on element:
- high noticeability
- comprehension problem
-
Regressive Saccades:
- comprehension problem
Eye Tracking - Reading
-
Perceptual span:
- 3 characters to the left
- 14 characters to the right
- depending on reading direction
-
Movements:
- 85% forward saccades, 15% regressive
- Text difficulty influences fixation duration, fixation count and number of regressions
- Comprehension problems trigger regressions
Questionnaires
Questionnaires
- Popular tool for data collection in market research, political polls, social science
- Cheap, easy, scalable
- Can be administered online
Limitations
- Participants has to read and understand the questions correctly
- Interpretations often differ between participant and creator
Pretesting necessary!
Pretests
- Detect questions that cause cognitive difficulties
-
Methods:
- Conventional Pretest
- Cognitive Interview ("Think aloud" + probing)
- Behavior coding, response latency measurements
Questionnaire Pretesting with Eye Tracking
Questionnaire Pretesting with Eye Tracking
- Eye Tracking is suitable for the identification of cognitive problems
- It does not provide any reasons for them
-> Hybrid approach with eye tracking and cognitive interview
Hybrid Eye Tracking Pretests
- In use today in pretesting laboratories
- Rely on live interpretation of eye tracking data by a human
-> Lets build and automated tool
Questionnaire Pretesting Tool
-
Functionality
- Automatically analyse eye movements
- Detect potentially problematic question
- Show a detailed evaluation
-
Compatibility
- Support existing online survey platforms
- Support common eye trackers
- Extensibility
We already have Data!
Lab Experiment
- 40 participants
- Precise Eye Tracker
- ~600 MB of raw gaze data
- Post questionnaire interview
- List of problems for each participant
Data Analysis
Data Analysis
- Map the gaze coordinates to on-screen elements
- Extract eye movement metrics:
- Fixations on question text relative to its length
- Regressions between answers and question text
- Regressions within the question text
Example: Fixations / Question length
r = 0.6687 (moderate positive correlation) P = 0.003337 (significant at p < 0.05)
Classification Algorithm
- Combine extracted metrics
- Take user's average values into account
- Decide whether a user has a problem with a question or not
Prototype
Technical Setup
Automatic Gaze Data Evaluation
- Evaluate gaze data after survey submit
- Show detailed result screen
Prototype Evaluation
Prototype Evaluation
- One researcher (not me!) used the prototype to conduct a pretest with 4 particpants
-
Feedback:
- Good as a basis for the cognitive interview
- Trustworthy, logical results
- Identified problems that his regular pretest did not
Summary
Experiment at GESIS (not my own work) Data Analysis:- Discover metrics
- Extract indicators for problems
- Websocket Proxy between Browser and Eye Tracker
- Combines existing survey platforms with gaze data via browser extension
- Evaluates questions using prediction algorithm and gaze data