The Problem

The criminal justice system (police, jurors, judges) places too much faith in eyewitness identification evidence. That is why faulty eyewitness identification is the leading cause of wrongful convictions, playing a role in more than 70 percent of the convictions overturned by DNA testing. Memory is NOT like a video recorder. We cannot retrieve past events and replay them in our mind’s eye like we are re-watching an episode of Breaking Bad. Memory is reconstructive, which means that our memory for an event is potentially being updated and modified every time we retrieve it.

Pretend you are at a coffee shop, working on your laptop. You step up to the counter to get a refill and turn around to see someone grab your laptop and rush out the door. You follow the thief but he quickly disappears into a crowd. You report the crime to the police, and one week later the police report that they have a suspect and want to show you a lineup. Is the police suspect innocent or guilty of stealing your laptop? What factors influence your ability to make a correct decision from this lineup?

We encode only part of what we experience. This is true whether the experience involves a mundane encounter with a neighbor or the aforementioned laptop robbery. For example, right after the robbery, you ask nearby customers what they saw. One reports that the individual wore a leather jacket, another comments on the perpetrator’s wispy facial hair. This information influences your memory of the person’s appearance. It is easy to confuse what was suggested with what was actually experienced because we are not good at keeping track of the source of these details. It would be particularly problematic for the suspect if you once saw him in a local grocery store. Even though he did not steal the laptop, you are likely to select him from the lineup because he looks familiar. It is easy to misattribute the familiarity you experience as arising from the laptop robbery rather than from your grocery store encounter.


➦ What Jennifer Saw
- Frontline

➦ Eyewitness: How Accurate Is Visual Memory
- 60 Minutes

➦ Innocence Project

➦ Center on Wrongful Convictions


The Solution?

Memory is never going to be perfect. That cannot be the goal of eyewitness reforms. But a series of reforms have been proposed regarding how to conduct a lineup that were purported to increase the accuracy of eyewitness evidence. Approximately 30 percent of U.S. jurisdictions already have implemented these reforms. These reforms include the selection of fillers (individuals known to be innocent of the crime in question) that match the perpetrator’s description, the presentation of unbiased instructions to the eyewitness (the perpetrator may or may not be present), and the presentation of lineup members one by one in a sequential lineup. A decision must be made about lineup member 1 (“yes” or “no” is this the perpetrator?) before lineup member 2 is presented, and so on. These reforms can be contrasted with the status quo: the use of fillers that do not match the perpetrator’s description or that match the perpetrator’s physical appearance rather than his description, the use of biased instructions, and simultaneous lineup presentation (all lineup members presented at once for a single decision).

Research by me and my collaborators has raised questions about the effectiveness of the aforementioned reforms. Our research also has established that showups (a one-person identification procedure that is commonly used by the police) result in worse performance than identifications made from lineups.


Identifying the Culprit: Assessing Eyewitness Identification

Troy Davis Execution Fuels Eyewitness ID Debate

ROC Analysis and Confidence

One of the reasons that researchers reached incorrect conclusions regarding the reforms (selection of fillers, unbiased instructions, and sequential presentation) was due to a reliance on performance measures that conflated accuracy and the willingness to make a response.

Imagine that students take an exam. All are awarded +1 point for each correct answer, but half the students receive -1 point for each incorrect answer, and the other half receive -10 points for each incorrect answer. Because the cost of making an error is much greater for the second group, these students will answer only if they are highly likely to be correct (i.e., highly confident). Would it be fair to assign grades based on the number of correct answers? Of course not. The conservative group will have fewer correct answers because the cost of an error is so high. But the differential cost of an error across the two groups of students affects only their willingness to respond (response bias), not their course knowledge (it will have no affect on discriminability, the ability to distinguish correct answers from fillers). Because the reforms create differences in response bias, we need another means to evaluate performance.

Receiver operating characteristic (ROC) analysis is a technique for disentangling discriminability (the ability to discriminate between a guilty and an innocent suspect) and response bias (the willingness to make a response). An ROC curve plots discriminability (correct identifications versus false identifications) at all levels of response bias. ROC analysis is a well-known analytic technique grounded in signal detection theory. Although new to the eyewitness domain, it is standard procedure in many other diagnostic domains (e.g., weather forecasting, medical imaging).


➦ Receiver Operating Characteristics and 5 Ways to Improve Eyewitness Identification of Criminals
- Scientific Blogging, Science 2.0

ROC Analysis Tutorial for Lineup Data, Part 1

ROC Analysis Tutorial for Lineup Data, Part 2: Measure AUC and Compare ROCs