Knowing your users isn’t enough

You may think you know your users, but that’s not good enough if you want to design great products for them. *cough Google Buzz cough*

I think this clip from The Simpsons where Homer gives Marge a bowling ball for her birthday epitomizes the problem.

Buying a present for someone is very similar to designing for someone. You’re not that person, so you can only guess what that person might like — a bottle of perfume, a bowling ball, a new man, an awkward social networking system. You often get it wrong, to the anger and disappointment of the recipient.

Why does this happen? Here’s an interesting study that gives us one insight. Researchers brought in couples and separated each pair. One person was asked to rate 30 items of furniture, and the other was asked to rate 30 different items.

Then each person was shown 30 new items — the items their partner just rated. Each partner rated the new items based on either 1) how their partner would rate them or 2) how a stranger would rate them. After rating an item, the person would see the other’s rating, helping them predict future ratings.

How well did each partner predict the other’s preferences? Since these people are couples, you might guess they did pretty well on this test. Actually, it depends on how similar the two partners’ tastes are.

When the two people in the couple have similar tastes (as determined by how they rated the same items), their prediction accuracy was around 40% — whether or not they were in the “rate your partner” or “rate a stranger” group.

When the two people in the couple have different tastes, the “rate a stranger” group was about 23% accurate while the “rate your partner” group scored a whopping 5% accuracy rate.

In other words, if your tastes are different from your partners, you’re better off treating that person like a stranger than trying to guess what he or she really wants. If your tastes are similar, your guesses for the other person will still be so-so.

How does this apply to design?

Most product designers think they know their users. This experiment reminds us that you suck at guessing what they want even when you know them:

  • If you know your users, your guesses about their preferences will be awful if your preferences are different than theirs (5% accurate)
  • If you’re similar to your users, your guess about their preferences will still suck (40% accuracy)
  • In any case, you’re better off imagining your users as strangers (23% and 40%) versus people you know (5% and 40%)
  • Getting feedback on your guesses doesn’t help you guess more accurately in the future

Let’s say you’re a large software engineering company based in Mountain View, CA building a new social networking product. You only test it internally, and everyone seems to think it’s ok. Then you unleash it upon the masses, resulting in all kinds of strange backlash that you never imagined.

I say that nobody should be surprised about these results. Even if Google is the golden child of Silicon Valley, they were bound to make this mistake because of their culture.

Why does this happen?

I can think of at least three reasons why people mistakenly think they have the same tastes as their users. These causes are often tied to the culture and processes in a company.

Not accounting for differences

Your users don’t think the same way that you do.  However, because of this nasty thing called projection bias, we mistakenly believe that other people have the same knowledge, ideas, experiences, and feelings as we do.

While that seems pretty obvious as a statement, I’m sure you encounter it all the time in the form of, “it’s great! The users will love it,” or “I know what they’re thinking. They want more social features!” That usually indicates you’re making decisions based on what you think rather than what your users think.

In-house experts

The real killer is when you think that “we’re users too” even though the people at your company aren’t characteristic of your user base. Because of availability bias, we base our decisions on information at hand rather (like our coworkers) instead of seeking out alternative points of view (like our users).

The people at your company aren’t very representative of all users of your product; your coworkers are usually product experts (top 1%) while most users are intermediates or beginners. If you decide on features based on internal feedback, then you’re building for the experts rather than the other 99% of your users. In other words, “we’re users too” leads to products aimed at the wrong audience.

Not asking first

Sometimes you can design something and be right about your user’s preferences.  However, there’s only one way to be sure — ask your users. Because feedback doesn’t improve your accuracy, you have to ask up front — before you lob the new features at your users.

Are you doing any user acceptance testing? With real users who work at other companies and do their work outside your company walls? How do you close the loop with your users after you release new features? It’s the difference between “honey, would you like a bowling ball for your birthday?” versus “honey, I bought you a bowling ball for your birthday.”

Back to the Buzz

So why did Buzz mess up? I hope it’s now clear that Google was playing Russian roulette with its product:

  • Google employees are vastly different than their overall user base
  • Google designed the product for themselves
  • Google only tested Buzz internally before making it public

And presto — Google just gave all their users bowling balls for their birthdays. You get a bowling ball! And you get a bowling ball! You ALL get bowling balls!!!

In summary, knowing your users isn’t enough to build great products for them. If you know them, you can still be horribly wrong at predicting their preferences (like the 5% result). Even in the best case, you’re still in coin-flip territory (the 40% result).

If you want to make better products, start by becoming aware of the biases that affect your decision making.

On jury duty

Being in jury duty is like forcing twelve people to order a three topping, extra-large pizza where the three toppings must be different and everyone has to agree on all three toppings before you can order.

Sometime during the course of the proceedings, I realized that the drama in front of me was the manifestation of everything I knew about cognitive science and social psychology. The lawyers are scientists and the courtroom is a laboratory. The lawyers use witnesses and evidence to mix up a verdict through persuasion, rhetoric, and storytelling.

I decided to put on my ethnographer’s hat and record my observations — the action in the courtroom, the drama in the deliberation room, and everything else relevant to life as a juror. Below is a scattershot list of my interpretations.

It’s long and poorly organized, and I’m not apologizing for it. And I’m being intentionally vague about gender and case details.

Backseat lawyers

As a jury, we were extremely dissatisfied with the lawyers for how they argued the case. Following a particularly awful exchange by one lawyer, a juror said he/she wanted to jump out of the jury box and yell at that lawyer for being such an idiot. (That’s almost a direct quote.)

As we went forward in our deliberations, we uncovered essential evidence that both lawyers missed. We often complained that the lawyers could have made a more compelling case by asking questions about X or grilling witness Y.

During one complaining session of that sort, I wanted to get the jury back on track. I said, “let’s hold off on the backseat lawyering.” Someone quipped in response, “Monday morning lawyer.” Exactly.

On a side note, I think a courtroom drama with an MST3K-style commentary track has the potential for extreme humor.

Humor

In case my previous attempts at jokes weren’t funny, we tried hard to maintain our sanity through humor during the deliberation process. This often came out as jokes about the witnesses, lawyers, the court personnel, or even ourselves.

Yes, we were a bit disgusted with ourselves given the seriousness of the courtroom events. I think it was necessary to crack a joke every so often; for me, it kept me from taking the deliberations too seriously — avoid making it personal.

One time while we were commenting on the courtroom antics, I said, “this would make a great screenplay — all twelve of us deliberating in the room. Co-starring the bailiff as our comic relief. And with special guest stars the judge and court clerk.” It was much funnier in the deliberation room; you just had to be there.

Someone said that jury deliberations would make a great reality TV show. “You could film a different jury deliberating every week.” I definitely agree.

I finally understand how someone got a cell phone in my name

About a decade ago, someone fraudulently opened a cell phone account in my name with a major wireless provider. That person racked up hundreds of dollars in calls, then the phone was shut off for non-payment. This showed up on my credit report when I went to get my own cell phone, resulting in countless hours lost as I tried to ressurrect my good credit.

Thanks to the expert witness testimony of a cell phone company employee at this trial, I now know why it’s so easy to do that. Unlike banks or credit card companies, cell phone companies do not extend credit to individuals; you’re bound by a contract to pay for the service every month. Because they don’t extend credit, those companies are not required to check the ID or verify any of the information (like address) that a person provides when getting a cell phone contract.

In St. Louis, a Sprint store employee gave a cell phone to someone who used my name and SSN along with a fake address (“Wadaba St” — seriously, that alone should be a red flag) when I was living a thousand miles away. And Sprint is under no obligation to verify that it was really me (or in this case, not me) who did it. And due to some arcane laws, I would have to go to St. Louis in person to press charges.

Words cannot express my outrage.

Technology (or lack thereof)

Our courts are technologically inept. Most of the evidence could have been trivially scanned and made available to the jury digitally if they had wanted.

Instead, we got 100 pages of phone call records, 50 records a page. (Oh, how we would have loved to build a pivot table.) We had to request special equipment to view a DVD. Every photo had to be passed around to each member of the jury, wasting countless hours. Testimony had to be printed (for some witnesses, hundreds of pages long) or, if the written transcripts weren’t ready, read aloud by the court reporter.

We opined about how much faster we could get through the information if we could get everything digitally. Instead, the prosecution lawyer pushed a cart full of boxes and binders, each full of pages and pages of paper, to court every day. Those poor trees…

The pool

The jury selection process is awful. It starts with 24 jurors being selected randomly from a pool of over 100. Then the judge and lawyers begin asking a series of questions that you’re expected to remember whether or not you were selected. Some jurors are booted, then a new set are called up to fill in the empty seats. More questions are asked, and the process repeats until both lawyers decide to keep the 12 jurors in the box.

It was the most boring three days of my life. Sure, there was the occasional interesting character or curious story. I spent my time devising ways that the courts could accelerate the process and get a suitable jury faster and make it more fun.

But no — the boring ways produce a fair jury of the defendant’s peers. I think the (long, boring) pooling process is the primary reason why people hate jury duty.

The negative opinion of jury duty

Here are some reasons why people dislike jury duty:

  • You spend three days in a jury pool but never get picked to be on the case — total waste of time
  • You end up on a trial and have to put your life on hold or have to work 20 hour days (8-9 hrs at court + your real job)
  • Sitting in the courtroom is boring. Everything moves as slow as possible
  • Deliberations take a long time and are very mentally taxing
  • You get paid almost nothing ($15 a day plus $2.50 for transportation in SF)

But I think the main reason why people dislike jury duty is this:

  • They’ve never been on a jury before

When I told people that I was on jury duty, most people said, “aw, that’s too bad,” or something like that. The only people who said good things were ones who had been on a jury before.

It’s really not that bad. Give it a try sometime.

It’s no [your courtroom/police drama of choice]

It’s no CSI. It’s no Law and Order. In the courtroom, a real trial is much more boring than you think. Lots of questions and answers. No magic technology to reassemble events of the incident. No outbursts by witnesses. No need to restrain the defendant.

I guess courts shouldn’t be exciting or dramatic, given the gravity of events that occur there. On the other hand, they shouldn’t be boring either, given the gravity of events that occur there. I’ll think about ways to improve that.

What the witnesses saw or didn’t see

Because the events in this trial occurred over a year ago, lots of witnesses had trouble remembering the events of the incident. In several cases, the lawyers had to remind the witnesses about statements they made to police on the night of the event; at court, the witnesses often misremembered or didn’t recall details.

Even more interesting — the key eyewitnesses (the defendant and a victim) recalled completely different versions of the incriminating event. It’s a form of motivated recall — that is, you’re motivated to remember events that portray yourself in a positive way or in a way that makes you seem better. In this case, the victim wanted a conviction, and the defendant wanted to be released; they testified as such.

As a jury, we were left with the tough decision to decide, individually, whether or not we believed the witnesses in part or in whole.

Everyone needs something a little bit different

As we were deliberating, we discussed the points that brought us to a reasonable doubt or brought us to believe the defendant was guilty. While it was good to air our thoughts, it did little to convince people one way or the other.

The reason why that doesn’t work is because of projection bias — that quirky cognitive bias that leads you to believe (incorrectly) that others share your beliefs or thoughts. The evidence that convinces you of your verdict probably won’t convince another juror of your verdict; in fact, more than once during our deliberations, someone would bring up evidence that obviously pointed to guilt while another juror would use that exact same evidence to point to innocence.

Every juror needed something a little bit different to get to a conclusion — a timeline of events, a read-back of certain testimony, an alternate explanation for a witness’s observation. If there’s any lesson from the trial that I’ll apply to my life, this is it — it may be perfect to you, but it does nothing for others (whatever “it” is — evidence, love, designs, hamburgers, Charlie Sheen movies).

The hardest part of the deliberations, by far, was finding that “little bit different” for each juror. How do you convince people of your opinion? I still don’t know the answer to that. All I know is that we deliberated for 7 days and didn’t agree on all the charges.

(Speaking of projection bias, in deliberations a few jurors said, “I would have…” or “If I was there…” If the witness did something that you wouldn’t do, the witness must be lying, right? Don’t assume others think or behave the same as you.)

The sounds and the smells

The Hall of Justice at 850 Bryant is smelly because it’s the home of the San Francisco parole and probation offices. Every day there would be interesting odors while waiting in line for the security check, going up the elevator, or, most interesting of all, in the stairwell.

The odors included pot, crack, alcohol, and smelly homelessness. Often I would see the security guards put on rubber gloves when dealing with homeless people. The security guards also kept a wooden walking cane near the x-ray machines to be used when a homeless person’s bags got stuck in the x-ray machine.  (They would use their hands on others’ belongings.)

And about the stairwell… it reeked of pot. Every day, without fail. My only wish is that the parole and probation officers have noses, because I bet lots of the people who show up are breaking their terms of probation — as determined solely by the odor.

The grammar

Having been through this trial, I’m much more grateful for my education. I’ve now heard the phrase “I seen” more times than I ever want to in a lifetime.

Court lingo

You learn lots of new terminology while in court. Words you thought you knew like “robbery,” “murder,” and “intent” have specific definitions while you’re there. Here are some favorites:

  • Dixied: to steal. See also: boosted
  • 6 pack: a photo lineup used by police to identify suspects
  • Green vegetable matter: marijuana, when entered into evidence and not yet identified as weed
  • Stipulate: an agreement between the prosecution and defense about the facts of the case
  • Intent: I’m still not sure what that means

Reinventing the wheel

Even though jury trials take place all over the country, most jurors are inexperienced and perform jury duty infrequently (less than once a year). That means juries are reinventing the process for deliberating over and over all the time. In my jury, nobody knew the right way to run the deliberations, so we made it up as we went along.

Said differently, everyone is a beginner in a jury. My user experience instincts tell me that you need plenty of hand-holding as a beginner in a complex process that you go through infrequently. Courts should help juries as much as possible to get through the deliberation process. Instructions would be great.

Instead, we were left with “1) deliberate. 2) vote. 3) if unanimous, tell the judge. else goto 1.” If we had some rules or advice for how to deliberate, I think our task would have been completed much faster. A different presiding juror (foreman) might have shortened (or lengthened) our deliberations by a day; I want to know what process juries can use to shorten deliberations by days.

The instructions

At the end of the closing statements, the judge gives the jury instructions for how to decide the case. For example, “robbery” has a very specific meaning including dispossessing someone of an item, using force or fear to deprive the person of the object, and intending to keep it permanently.

In our case, the instructions were over 50 pages of the dry, difficult legal jargon. As non-lawyer people, we could only decide among ourselves or ask the judge what they mean. And because it’s over 50 pages worth of dense stuff, there’s no way to keep all that in mind when deliberating.

The definition of “robbery” included a 5 step process with special rules and exceptions, and it refers to other definitions that have their own steps and processes. The criteria for believing or not believing witnesses was 5 pages by itself. Circular references were common. One juror joked that the instructions should be in a wiki.

The output is only as good as the input; simpler instructions would help juries come to better decisions faster.

The missing pages

Even though you’re not supposed to assume or concoct ideas, you can’t help but do that, especially when you find the witnesses’ testimony to be dubious. We tried to reconstruct what really happened, sometimes taking liberties with the evidence we were given.

In a more amusing moment, someone suggested that a victim inflicted his wounds on himself.  I said, “you mean this is like Fight Club?” It got a laugh.

As humans, we have an innate drive to finish stories — a drive for completion. Trying to reassemble the real story of a court case is like reading a novel with every other page torn out or watching a movie in a language you don’t understand. You invent your own story; you have to.

Small potatoes

During our casual chats, we often talked about how the case was impacting our lives. Several people reported that they had trouble sleeping, including one who thought a person was standing in his/her bedroom (it was a shadow). A couple of folks said they started noticing suspicious characters in their neighborhood. A couple of people were at the trial during the day then left to work a full time job at night.

As the trial went on, the impact started getting more severe. One person said his/her manager was getting annoyed about when that juror would get back to work, including veiled threats about his/her employment status. Someone had a family member undergo surgery during the trial. Another had a close friend with a death in the family. Another a dying relative.

Someone suggested we ask the judge for a day off to tend to our lives. One of the jurors put it into perspective. “Whatever issues we have in our lives that are affected by this trial, it’s small potatoes compared to what the defendant, the defendant’s family and friends, and the victims’ families and friends have gone through. As much as we have a duty to our lives, we also have a duty to see this case to its end for those people.”  Small potatoes indeed.

Reinterpreting everyday life

Someone said they noticed a strange car driving by his/her house and got suspicious. Another juror said that he/she was being more argumentative in everyday conversations. Someone else mentioned that he/she is trying to be more observant during activities in case that moment ever became the subject of a trial.

Another juror said that the process had made him/her more thankful for life and the privileges therein. I definitely agree; many countries don’t have jury trials or the disposition of innocence that we have in the USA.

I think we were all transformed by this process, and I think my comments above only scratch the surface of the impact it had on our lives.

Closing Statement

It wasn’t easy being a juror, but I do feel satisfied with the process, the quality of my fellow jurors, and the results of our effort.

I feel like I need a mental vacation to recharge myself.

And I have tons of other observations about the trial that I haven’t even gotten to yet.

I might need another blog post to tie this one off…