Knowing your users isn’t enough

You may think you know your users, but that’s not good enough if you want to design great products for them. *cough Google Buzz cough*

I think this clip from The Simpsons where Homer gives Marge a bowling ball for her birthday epitomizes the problem.

Buying a present for someone is very similar to designing for someone. You’re not that person, so you can only guess what that person might like — a bottle of perfume, a bowling ball, a new man, an awkward social networking system. You often get it wrong, to the anger and disappointment of the recipient.

Why does this happen? Here’s an interesting study that gives us one insight. Researchers brought in couples and separated each pair. One person was asked to rate 30 items of furniture, and the other was asked to rate 30 different items.

Then each person was shown 30 new items — the items their partner just rated. Each partner rated the new items based on either 1) how their partner would rate them or 2) how a stranger would rate them. After rating an item, the person would see the other’s rating, helping them predict future ratings.

How well did each partner predict the other’s preferences? Since these people are couples, you might guess they did pretty well on this test. Actually, it depends on how similar the two partners’ tastes are.

When the two people in the couple have similar tastes (as determined by how they rated the same items), their prediction accuracy was around 40% — whether or not they were in the “rate your partner” or “rate a stranger” group.

When the two people in the couple have different tastes, the “rate a stranger” group was about 23% accurate while the “rate your partner” group scored a whopping 5% accuracy rate.

In other words, if your tastes are different from your partners, you’re better off treating that person like a stranger than trying to guess what he or she really wants. If your tastes are similar, your guesses for the other person will still be so-so.

How does this apply to design?

Most product designers think they know their users. This experiment reminds us that you suck at guessing what they want even when you know them:

  • If you know your users, your guesses about their preferences will be awful if your preferences are different than theirs (5% accurate)
  • If you’re similar to your users, your guess about their preferences will still suck (40% accuracy)
  • In any case, you’re better off imagining your users as strangers (23% and 40%) versus people you know (5% and 40%)
  • Getting feedback on your guesses doesn’t help you guess more accurately in the future

Let’s say you’re a large software engineering company based in Mountain View, CA building a new social networking product. You only test it internally, and everyone seems to think it’s ok. Then you unleash it upon the masses, resulting in all kinds of strange backlash that you never imagined.

I say that nobody should be surprised about these results. Even if Google is the golden child of Silicon Valley, they were bound to make this mistake because of their culture.

Why does this happen?

I can think of at least three reasons why people mistakenly think they have the same tastes as their users. These causes are often tied to the culture and processes in a company.

Not accounting for differences

Your users don’t think the same way that you do.  However, because of this nasty thing called projection bias, we mistakenly believe that other people have the same knowledge, ideas, experiences, and feelings as we do.

While that seems pretty obvious as a statement, I’m sure you encounter it all the time in the form of, “it’s great! The users will love it,” or “I know what they’re thinking. They want more social features!” That usually indicates you’re making decisions based on what you think rather than what your users think.

In-house experts

The real killer is when you think that “we’re users too” even though the people at your company aren’t characteristic of your user base. Because of availability bias, we base our decisions on information at hand rather (like our coworkers) instead of seeking out alternative points of view (like our users).

The people at your company aren’t very representative of all users of your product; your coworkers are usually product experts (top 1%) while most users are intermediates or beginners. If you decide on features based on internal feedback, then you’re building for the experts rather than the other 99% of your users. In other words, “we’re users too” leads to products aimed at the wrong audience.

Not asking first

Sometimes you can design something and be right about your user’s preferences.  However, there’s only one way to be sure — ask your users. Because feedback doesn’t improve your accuracy, you have to ask up front — before you lob the new features at your users.

Are you doing any user acceptance testing? With real users who work at other companies and do their work outside your company walls? How do you close the loop with your users after you release new features? It’s the difference between “honey, would you like a bowling ball for your birthday?” versus “honey, I bought you a bowling ball for your birthday.”

Back to the Buzz

So why did Buzz mess up? I hope it’s now clear that Google was playing Russian roulette with its product:

  • Google employees are vastly different than their overall user base
  • Google designed the product for themselves
  • Google only tested Buzz internally before making it public

And presto — Google just gave all their users bowling balls for their birthdays. You get a bowling ball! And you get a bowling ball! You ALL get bowling balls!!!

In summary, knowing your users isn’t enough to build great products for them. If you know them, you can still be horribly wrong at predicting their preferences (like the 5% result). Even in the best case, you’re still in coin-flip territory (the 40% result).

If you want to make better products, start by becoming aware of the biases that affect your decision making.

Leave a Reply

Your email address will not be published. Required fields are marked *