Don’t get mad, get data

Has this happened to you? You create a design and someone – or a committee of someones – changes it. It doesn’t matter how you well you argue or justify your design. They just change it.

This happened to me last year when I was working pro bono on the content and information architecture for the website of Michael Olsen, a Melbourne-based playwright. I also helped Michael find a local WordPress developer who he then paid to build

Now, people make changes to designs to suit themselves all the time. What’s wrong with that? And even though I wasn’t being paid, he was the client. Relax, it happens all the time. Right?

Well… Yes. And… No, damnit.

I’m right

You see, my designs were based on more than just me – my experiences and opinion. I used data. Unconventional I know. I’d tested the potential audience to discover what they were interested in. I’d run keyword searches to understand the language people use. (Factoid: the word ‘performances’ is more often used than ‘productions’ in searches related to theatre.) I’d conducted card sorting and tree testing activities. I’d run the results through the filter of my experience to produce the design.

And it wasn’t just that my design was difficult to build within the limitations of the software. Sure it was hard to specify exactly how visitors might find what they’re looking for from Michael’s more than 50 plays. But my research showed that people were very interested in the author, his news and his plays. This gave me the clue that site visitors – including potential directors and producers – would spend the time browsing the plays. They’re original plays for god’s sake; they’re interesting.

I was peeved

With all that up my sleeve how could I lose the argument? Well, they didn’t bother with the argument. They just changed it.

That’s when I decided to get data.

I had the top tasks, I had the live site and I had guinea pigs willing test subjects. According to my original study, these were the top five tasks:

  1. Check out the news: new plays, upcoming/recent performances, reviews
  2. About Michael
  3. Read synopses of plays
  4. Browse plays by… genre, number & gender of characters, number of acts, running time, etc.
  5. Who Michael has worked with: actors, directors, producers

So, as I was testing on another website, I just asked my test subjects to perform these tasks and watched them. Simple.


The look and feel were found to be agreeable. “It’s light and simple,” said one. “I like the font and colours,” said another. Tick.

A general quiz showed the navigation and labels were OK. “It seems clear to me because when you mouse over those words a menu appears with the options I expect,” said one. Tick.

Task 1 Check out the news…:

In my original architecture I created a News tab and also linked clearly from the home page – describing what visitors were likely to expect when they clicked on News. I deliberately collapsed the ideas of Blog and News together for a number of reasons.

My test subjects were able to find News because of its prominence in the menu structure. The fact that it’s not often updated regularly (yet) is another matter and thankfully I kept News items off the homepage thinking that was the case. But there is now a Blog menu item which confused subjects for a while about News but… that’s a tick.

Task 2 About Michael:

This is a no brainer and my subjects found it pretty easily. Although one commented it was hidden further along the top horizontal menu than he expected which is not how I specified it. But… it was a minor change. Tick.

Task 3 Read synopses:

This is a straightforward alphabetical list of all the plays. It’s easy to find as Synopses A-Z is the only menu item under The Plays. I had broken up browsing A-K and L-Z, but A-Z was OK, just a bit more scrolling. Tick.

Task 4 Browse plays…:

This was the task subjects had the most problem with. None succeeded in being able to browse the plays by any of the categories. The original structure supported browsing for plays; replaced by a Search interface on the homepage. The relationship between search and navigation on websites is subtle and complex. Search is often used by people on the web but, as Jakob Nielsen writes in Converting Search into Navigation:

  • Most users are unable to solve even halfway complicated problems with search. Better to redirect their efforts into more supportive user interfaces when possible.

Furthermore, and somewhat paradoxically, search giant Google valorises web content. Google’s content and design guidelines recommend creating:

  • a clear hierarchy and text links with every page reachable from at least one static text link
  • a useful, information-rich site, and write pages that clearly and accurately describe your content

Hence my focus on making the categories browseable. It’s also what subjects stated they wanted when pressed – the Search interface is confusing due to the system of tagging the plays with the number and gender of characters. My system was much simpler and would have been searchable, too. It involved being much more general about the number and gender of characters so they could be browseable on a single page. This tested well on the tree testing I did initially and this was the case in practice. No tick.

Task 5 Who Michael has worked with…:

Subjects had no chance as this menu item and this information just didn’t make it to the final cut. No tick.

So the scorecard isn’t too bad – some good general comments, 2 straightforward ticks, a qualified 1 and 2 fails on the 5 top tasks. Not surprising since it’s a pretty simple website. Frankly, I was surprised by the data.

Data is better for blood pressure

Is this article equivalent to a building architect delivering drawings and then whingeing about the client and builder when things are changed? Maybe. There’s no doubt there’s a strong family resemblance between building and information architecture.

But also differences:

  • It’s hard to change a building once it’s built; with the right software, websites lend themselves to small incremental improvements.
  • It’s also easier to observe website behaviour. Between the surveying, card sorting, wireframes, tree testing, analytics, search data and user observation we do, it’s much easier to understand how visitors will use a website than use a building. Note: this is changing – mocking up rooms to understand how people use them is gaining momentum (see New York Times article Factory Efficiency Comes to the Hospital).

Understanding website usage and “getting your hands dirty” with the data is worth sticking up for. I recommend data over anger if only for two reasons:

  1. you learn a hell of a lot more by being objective about it
  2. it’s better for blood pressure

Data can surprise us. We already know our opinions – by definition – there are no surprises there. An opinion’s sample size is one. Go for data, become your own scientist and experiment, experiment, experiment.