Dr Kait Clark, lead of the Applied Cognition and Neuroscience theme of the Psychological Sciences Research Group (PSRG), has published a new open-access paper in the Journal of Vision: “Test-retest reliability for common tasks in vision science.” The paper is co-authored by UWE PhD student Kayley Birch-Hurst, collaborators Dr Craig Hedge and Dr Charlotte Pennington at Aston University, and UWE Psychology alumni Austin Petrie and Josh Lee.
The authors argue that considering the test-retest reliability of a perception or attention task is crucial if researchers wish to use the task to assess the impact of individual differences (e.g., traits, experience) on performance. The issue is rooted in the historical development of vision science tasks, which were often designed to minimise differences between participants in order to understand a cognitive mechanism more generally. With an increased interest in looking at the influence of individual differences on perception and attention, researchers are now using the same tasks, but these tasks may not have a sufficient spread in participant variability to tell us anything meaningful about individual differences.
Test-retest reliability is the degree to which a participant’s performance is similar from one time completing a task to another. When there are little differences between individuals on a task, test-retest reliability tends to be low; i.e., if participants’ measures of accuracy or response time are all quite similar to each other, the degree to which one individual’s performance predicts their performance on a second test is going to be small. Therefore, a task with low test-retest reliability is not going to produce a consistent index of performance for any given individual (i.e., where they fall on a spectrum from “poor” to “excellent”) and cannot be used to assess individual differences in performance.
To assess test-retest reliability, Dr Clark and her team tested 160 undergraduate psychology participants on four commonly used tasks in vision science. The tasks measured a range of perceptual and attentional faculties such as sustained attention, motion perception, and peripheral processing, and each participant was tested twice, 1-3 weeks apart. The results demonstrate a range of reliabilities (as measured by the intraclass correlation coefficient, or ICC), indicating that some tasks (and some measures within these tasks) are more suitable for the exploration of individual differences than others. As expected, higher ICCs were associated with higher between-participant variability. The authors also reviewed a wide range of vision science tasks with known reliabilities and summarise these statistics in a useful reference table for future researchers. Finally, they provide detailed guidelines and recommendations for how to appropriately assess test-retest reliability.
