This is a story about how we designed a part of the website based on what made sense to us and our test group, and why we failed.
It was planned and executed by two long time web developers, so you’ll be able to recognize the analitical, logical mindset.
We were designing an interface for estimating scrum tasks (check this wikipedia page if you don’t know what i’m talking about: scrum poker planning). What interests us today from the whole system is that we needed to create a team, to be able to group tasks into planning sessions, and to estimate tasks.
Creating teams and planning sessions is obvious, and the estimating part is a bit like voting: every member sends his estimation to a central point, and the scrum master picks the final estimation using what he sees on the central point (the scrum master interface).
You can quickly identify the two essential parts to this process: the interface from which the users will send the estimations, and the interface from which the scrum master will choose the final estimation for the tasks. Like this:
Pretty straight forward, let’s break it down.
The user should:
- see the estimation cards when a round starts
- not see the estimations of the other users
The scrum master should:
- see the estimation of every user
- be able to choose the final estimation
- start/stop/restart the estimating process and the planning session
Say abracadabra and the result looks like this:
This is the scrum master interface, the user interface is not as interesting. You can probably figure out quickly how the interface deals with every requirement we outlined earlier.
Few elements? check.
Everything at hand? check.
Does everything it needs to do? check.
Don’t trust your opinion? check.
So we got other people to try it. We picked developers and scrum masters, as that is our target audience, we even asked people in unrelated fields to try it. And about 85% validated it, it worked. We explained what the system’s purpose is, and they were using it successfully.
So we went live
How it went?
Luckily, we like data, so we had data to tell us how we’re doing. And what it told us was pretty rude:
- 80% of the users were creating at least one team
- 85% of the users were starting at least one planning session
- only 50% figured out how to give an estimation
Is it that bad? It means from the users that thought our product was worth their time, and created a team and started a planning, so they already went pretty far, we lost half. And from the other half you’re going to lose the ones that simply don’t like it.
So it was clear we had a problem. Ideally all the users that sign up for an account should get to estimate at least once, that means they understood how the application works.
And they should have, we saw earlier the interface is pretty slim and simple, didn’t we?
Why didn’t they?
Well, because both we and the test users knew what the system should do, and the users… not so much. Of course every one of them knows how a scrum poker planning works, but apparently they have wildly different ideas about how an application that helps with that should work. And while they do have the smarts to figure out how ours works, they won’t have the patience to do it.
Why don’t you have a tutorial? We do, it’s the nice looking thing everyone skips.
So what did we learn?
- even if you do everything right, and test, and it works, monitor your application like it’s a bomb waiting to explode
- pay attention to how you choose your test users and what do you tell them
- real life users have little patience / time to figure out stuff
And now, how did we fix it?
To be continued
Aaand here’s the complimentary dog pic from the house, have a good one!