Pragmatic Usability Testing

If you are working on a project long enough you’ll probably think that every link, button, image is exactly where it should be.
If you coded it or designed it, then there’s something worse: You’ll have a hard time accepting that no one saw that big, bold sign up link.

A little while ago we started redesigning our flag-ship product, a price aggregator. We drew drafts on paper and handed them over to the graphics department to make them look good, and some time later we had working prototypes.
Even thought we try to follow some basic usability guidelines, there are application specific things that follow no rules. There was a number of issues we had doubts about, so we decided to let the users show us.

How do I start?

You’ll probably think that finding the users is more important than anything else, but that’s not true. We started by making a list of user stories, pretty much the same user stories you’ll end up using agile methods. They can also be called tasks, and they looked something like that:

• Sign up using “someusername” as a username.
• Login using “someotherusername”, “somepassword” as a password.
• Find “productx” and add it to your bookmarks.
• ….

We wrote a series of tasks but not everything the user can possibly do within the application. And this proved to be a good decision since after the third user some of the tests had already become irrelevant. We thought we’ll have time to test again, so we better make it short.

What users?

If you haven’t read Why You Only Need to Test With 5 Users from the usability guru Jakob Nielsen, then you should. It explains that 5 users are enough to discover minor to major usability flaws in your design. It was shocking for us to find out that this was absolutely true! We only read the article after we had done the tests and were trying to figure out if 5 users is a safe number to draw conclusions.

We asked colleagues, relatives and friends to help us out with this. We tried to pick a list of users with different technological backgrounds, yet I’m starting to think that it makes no difference if the user is a geek or not.

Testing the tasks

We bought beers and cookies and invited people over at the office. One at a time.
We then stood back while they were trying to complete the tasks. No questions asked, no remarks made. We just had our eyes wide open watching for things that should not happen.

Scanning, not reading

Users scan. If the user started reading trying to figure out how to complete the task then that’s sign #1 of bad design.
Some of the tasks were there just to make sure the user could complete them with a few quick scans.
Fortunately, all of our important links and sections were easily discoverable and all users completed those specific tasks with ease.

Knowing where you are

We failed on this.
The user must not feel disoriented in any way. If he doesn’t know why he landed on the page he is looking at, then confusion is a fact.
Our login form was located at the top of the page (kind of like digg.com). If the login was not successful we redirected the user to the sign up page.

Jakob’s Law of the Web User Experience states that “users spend most of their time on other websites.” You can read about it here

So if every user is used to a “retry login” page after an unsuccessful login, then that’s where we should take him.
4 out of 5 users quickly hit the back button as they thought they did something wrong.

Habituation

In my opinion habituation on a site’s layout is a good thing. It’s another way to say that being consistent throughout the application, will help your users adapt to the layout.

This is easily measured.
Each time the user started another task it took less time to find the appropriate link / section and click it.

• If your navigation menu is on the left and stays on the left then you’re good.
• If your navigation options are e.g 9 and stay 9 with the same order you’re good.

We measured that all users first looked on the navigation menu before taking any other action. That’s enough.

Conclusion

This has nothing to do with accurate, budgeted usability tests. It’s just a way of testing your application and clearing out dark spots on a 0$ budget with a little help from your friends.
It’s cheap, fun and will save you a lot of trouble of making ugly changes later when users start to complain.