Here’s a story from my life (stop me if you’ve heard this one before): I was project manager for a purchase order/contract management system. At one point I asked the system’s prospective users if they’d ever need the system to create purchase orders where the items on the order would be delivered in several different shipments. “Oh no,” I was told. “Everything on the purchase order is to be delivered at the same time.”

When I tell this story at conferences, I always pause at this point and ask, “So how long do you think I had to wait before…” and I just let my voice trail off. People know (instantly!) what I mean—how long before the users tried to create a multi-shipment purchase order? Estimates range from three days to a few months (for the record, everyone is too cynical; it was actually about six months before it happened).

I used to be outraged at this. Why, after all, shouldn’t users be able to correctly report about their own activities?

Here’s another story: I’m a big fan of the TV show “Law & Order.” In an episode this season, the show started with a pan that had many pedestrians passing in and out of camera view. As I watched, I noticed that one of the people (the bad guy) had pulled out a gun. He was only visible for one or two seconds. What could I tell you about what I saw? Not much about the person but I could tell you a great deal about the gun. The camera shot was basically a close-up of the person’s coat and arm from the elbow down, showing just the hand holding the gun and the person’s right leg.

I mention this because I saw the episode again last night. I recognized the show from the first few shots and waited to see the guy with the gun come on. Interestingly enough, I was able to pay attention to more than just the gun this time. Also, interestingly enough, my memory of the shot was wrong: The bad guy with the gun was visible in a full body shot and, after pulling out the gun, he held it across his chest.

But my recollection of the gun was pretty accurate. Over the years, my faith in eyewitness testimony has diminished, especially once I started checking my observations against reality. It was depressing to recognize that my ability to report on reality had the same failings as the end users that I used to be so outraged at.

I shouldn’t have been surprised at the accuracy of my recollection of the gun. People who are threatened with guns in real life frequently give very detailed reports of the weapon and are able to give only hazy descriptions of the person holding the gun. We remember things selectively and concentrate on what’s important to us. I also shouldn’t have been surprised that I had filled out my fragmentary recollection with spurious details. We are driven to explain things and, when faced with missing information, automatically make up details to fill in what we know is missing. We tend to fill in our recollection with what we “know should have been there.” The older a memory is, the more likely that that the memory has been “fleshed out” with made-up facts.

I got this information from Elizabeth Loftus’ book Eyewitness Testimony. In her book, Loftus has gathered together what’s currently known about eyewitness testimony (including her own extensive research). If you haven’t read the book, I recommend it highly.

Is there anything you can do about this? Yes. Don’t ask questions; gather data. Validate every statement against actual records. If you do interview end users, interview them separately. People who share information don’t arrive at a more accurate recollection. Instead, eyewitnesses who share their observations generate a common, shared reality with each person filling in their missing details with the details invented by others.

On a more personal note, I’ve improved my ability to report on the events that I see. I recognize that I can only remember a few things, so I try to pick out one or two things that I can be counted on to remember. I write down what I do remember as soon as I can. Once I’ve written down what I remember, I don’t try to pull up what I don’t remember because I’m just as likely to invent new facts as remember the real ones. And I don’t compare my recollections with others—we’ll only create a shared error.

I got to apply these learnings when I witnessed a hitand-run accident a few years ago. I recognized that I couldn’t be counted on to reliably remember much of what happened (it happened very quickly). But I could be counted on to remember one thing, so I picked the car’s license plate. I wrote that down as soon as I could. I didn’t discuss the incident with the other witnesses and, when I was questioned by the police, I happily admitted that I didn’t know the answers to many questions they asked.

Virtue has its rewards and costs: Based on my evidence, the police were able to catch the perpetrator. The cost was that, as the most reliable witness, I had to go back to the town where the accident happened to testify. As soon as the perpetrator spotted me in the courtroom, he changed his plea to guilty, so I never did get to testify.

That’s too bad. I was really looking forward to meeting Jack McCoy.