Analysis


One of the reasons that researchers reached incorrect conclusions regarding the reforms (selection of fillers, unbiased instructions, and sequential presentation) was due to a reliance on performance measures that conflated accuracy and the willingness to make a response.

Imagine that students take an exam. All are awarded +1 point for each correct answer, but half the students receive -1 point for each incorrect answer, and the other half receive -10 points for each incorrect answer. Because the cost of making an error is much greater for the second group, these students will answer only if they are highly likely to be correct (i.e., highly confident). Would it be fair to assign grades based on the number of correct answers? Of course not. The conservative group will have fewer correct answers because the cost of an error is so high. But the differential cost of an error across the two groups of students affects only their willingness to respond (response bias), not their course knowledge (it will have no affect on discriminability, the ability to distinguish correct answers from fillers). Because the reforms create differences in response bias, we need another means to evaluate performance.

Receiver operating characteristic (ROC) analysis is a technique for disentangling discriminability (the ability to discriminate between a guilty and an innocent suspect) and response bias (the willingness to make a response). An ROC curve plots discriminability (correct identifications versus false identifications) at all levels of response bias. ROC analysis is a well-known analytic technique grounded in signal detection theory. Although new to the eyewitness domain, it is standard procedure in many other diagnostic domains (e.g., weather forecasting, medical imaging).

LEARN MORE:

➦ Receiver Operating Characteristics and 5 Ways to Improve Eyewitness Identification of Criminals
- Scientific Blogging, Science 2.0

➦ ROC Analysis Tutorial for Lineup Data, Part 1

➦ ROC Analysis Tutorial for Lineup Data, Part 2: Measure AUC and Compare ROCs