EVERYONE tells you how important it is to know the business requirements. And, presumably, you find out what the business requirements are by talking to your business users. That’s all well and good, except no one tells you how unreliable your users are.

Probably everybody has a story like this one: I was working on a purchasing system and, as part of gathering the business requirements, I asked my users, “Do you ever do a split order?” (A split order occurs when you place a single order but plan to receive the goods in a series of shipments.) “No,” I was assured. “We never have split orders.” So we implemented a system without split orders. When I’m doing seminars and tell this story, I stop here and ask the audience “How long after implementation?” Like I said, everybody has this story, and everybody knows that I’m asking how long until the first split order showed up. People are much more cynical than I am—most answers are in the one-to-three week range. In actual fact, it was about two months before I got the call: “Peter, we’re up here trying to put in a split order and can’t seem to figure out how to do it. Could you come up and give us a hand?”

So I went up and asked (not unreasonably, but with a definite edge in my voice), “I thought that you said that you didn’t do split orders,” to which the response was, “Oh, well, for 3M we’ll do a split order.” It turned out that we used a 3M product, and 3M offered us a sweetheart deal on the price, but we had to order a year’s supply and only give 3M one sales order to process. In turn, since we couldn’t store a year’s supply, 3M agreed to ship the product to us in batches. Hence, split orders; hence, a system rewrite.

This kind of misinformation is the reason that developers frequently hold users in low regard. But, I want to point out, we’re all terrible reporters. Eyewitness testimony is almost completely unreliable. If the number of people convicted on eyewitness testimony now being freed on DNA isn’t enough evidence for you, consider your own experience. I’m sure you’ve had numerous occasions when someone else’s report of an incident hasn’t matched your recollection (and, of course, your recollection is correct—after all, you were there).

My latest example is an excellent one. I was at a hotel where a convention was being held. The people putting on and attending the conference were obviously Arabic (I picked up a wonderful book written by a Jordanian writer and an album of traditional music from Anatolia). I asked one of the people at the hotel what the conference was about and was told that it was the “Anti-American conference.” I gasped; the person reconsidered and said, “No, the name is actually the Arab-American Anti-Discrimination conference.” A significant difference.

Elizabeth Loftus has done a lot of research in this area and has put it, and the rest of the available research, in a book called Eyewitness Testimony (Harvard University Press; the current edition is the 1996 reprint, ISBN: 0674287770; I’m referring to my 1979 paperback edition). It should be required reading for anyone who’s gathering requirements from users. One study from the book demonstrates the depth of the problem. Two groups of people were shown a picture of a car after an accident. In the following interview session, one group was asked, “About how fast were the cars going when they hit each other?” The other group was asked, “About how fast were the cars going when they smashed into each other?” A week later, both groups were asked, “Did you see any broken glass?” More members of the second (“smashed into”) group reported glass than the first group (“hit”)— 32% vs. 14%. In fact, the picture showed no glass at all on the road. You can only imagine how many wrong answers would have been generated if the question had been, “How much broken glass was on the road?”

Going back to my purchasing system, perhaps I should have asked, “How often do you do split orders?” That might have encouraged my users not to think in terms of yes/no but in terms of one or more. Given how easy it is to influence eyewitness testimony with the question you ask, perhaps the best answer is to avoid asking questions at all.

Part of the problem with giving up interviewing is that we want to believe eyewitnesses. Loftus reports on one experiment where the defense attorney in a simulation demonstrated that the key witness couldn’t have seen the person who committed the crime (the witness had 20/400 vision, wasn’t wearing his glasses, and couldn’t have seen the face of the perpetrator from where he was standing). Sixty-eight percent of the jurors voted to convict anyway.

It isn’t realistic to ignore interviewing users. However, that should just point you toward the physical evidence that you want to investigate: paper records, direct observation, office layout, and so on. In criminal trials, this is called circumstantial evidence and is less convincing in trials than eyewitness testimony. The DNA evidence that’s currently causing the release of so many who have been wrongly convicted would be classified as circumstantial. But we trust the DNA evidence, and I’d trust the paper records more than eyewitness testimony— especially anything that comes in response to a question that I asked.

This fall, I’m currently booked for three conferences. The first is the XML and Web Services Connections conference, September 30 – October 3 (I’m also the conference co-chair with Paul Litwin, the founding editor of Smart Access). I go straight from that conference to the Office Developer Connections conference October 3 – 5. Both are in Scottsdale, Arizona, and you can check them out at www.devconnections.com. The Office Developer conference is probably the most interesting to Access readers, and I’ll be presenting on integrating Access 2000/2002 with ADO, XML support in Access 2002, using COM+ with Word and Access, and accessing data over the Internet with Access and XML. I’m also doing pre-cons on user interface design and creating end-user documentation. Unfortunately, if you missed the Advisor DevCon in June, there won’t be a fall DevCon this year, which would be my normal next stop on the fall conference circuit. At the end of November, I’m in England for the Visual Basic User Group’s conference (www.vbug.co.uk). There, I’ll be presenting on developing business-to-business applications with VB/XML/COM+ and processing XML documents, along with a pre-con on XML for Visual Basic developers. If you’re going to be at any of these conferences, please look me up and say hello.