Ed. Note: Boing Boing's current guestblogger Clay Shirky is the author of Here Comes Everybody: The Power of Organizing Without Organizations. He teaches at the Interactive Telecommunications Program at NYU, where he works on the overlap of social and technological networks.
Every now and again, I see a business doing something so sensible and so radical at the same time that I realize I'm seeing a little piece of the future. I had that feeling last week, after visiting my friend Scott Heiferman at Meetup.
On my way out after a meeting, Scott pulled me into a room by the elevators, where a couple of product people were watching a live webcam feed of someone using Meetup. Said user was having a hard time figuring out a new feature, and the product people, riveted, were taking notes. It was the simplest setup I'd ever seen for user feedback, and I asked Scott how often they did that sort of thing. "Every day" came the reply.
Every day. That's not user testing as a task to be checked off on the way to launch. That's learning from users as a way of life.
Andres Glusman and Karina van Schaardenburg designed Meetup's set-up to be simple and cheap: no dedicated room, no two-way mirrors, just a webcam and a volunteer. This goal is to look for obvious improvements continuously, rather than running outsourced, large-N testing every eighteen months. As important, these tests turn into live task lists, not archived reports. As Glusman describes the goal, it's "Have people who build stuff watch others use the stuff they build."
Mark Hurst, the user experience expert, talks about Tesla -- "time elapsed since labs attended" -- a measure of how long it's been since a company's decision-makers (not help desk) last saw a real user dealing with their product or service. Measured in days, Meetup approaches a Tesla of 1.
Glusman and van Schaardenburg have also made it possible to take Jacob Nielsen's user-testing advice -- "Test with five users" -- and add "...every week." Obstacles to getting real feedback are now mainly cultural, not technological; any business that isn't learning from their users doesn't want to learn from their users.
On my way down after seeing the user test, the woman I'd seen on the screen got onto the elevator, and I mentioned I'd seen her trying the new interface. "Oh", she said, surprised. "I didn't realize anyone was actually paying attention to me."
Hurst: Time elapsed since labs attended | Nielsen: Why You Only Need to Test With 5 Users
RSS Feed
HUGE exception to the rule.
Unfortunately.
It's like Darwinism. It's not the best solution that counts, it's the solution that happens to out-compete the competition, no matter by what means. And in a capitalistic marketplace you automatically also get quantity before quality, short term before long term.
how do you get the webcam to show the users screen?
Tesla was already honored with the SI unit name for magnetic field flux density in 1960. Naming two different units after the same person seems like it would not be conducive to good communication.
Slighly creepy, but I'm sure there is consent and adequete notice on the observed computer.. or indeed a great big farking webcam ponted at your monitor.
We use Morae to project the browser window and a webcam view. We also don't just have product managers watch the sessions, but every singe person working on that project. The engineers, the product manager, the designers, and the customer support people who sit on the teams.
See: Krug, S. (2005). Don't Make Me Think : A Common Sense Approach to Web Usability (2nd Edition). New Riders Press.
Krug spends a good portion of his book describing this methodology of user testing, calling it "Usability testing on 10 cents a day."
JS7A, Hurst mentions namespace clash in the piece I linked to. In addition, any thought that such naming overlap can be avoided should be cured by a quick trip to all-acronyms.com, which already lists "Technical Standards for Library Automation" and "TeV-Energy Superconducting Linear Accelerator."
Never in the article or any related links was any reference made to what Meetup is. It just seems like a bit of an ironic moment to mention something so radically, sensibly, user-friendly that it seems futuristic.
@CMALEK: Second that.
I used Krug's suggestions while building the (non-visual) user interface for an inside-building navigation aid (sort of an artificial guide). We used to put 3 users through the system every week and I used to update the UI every day.
At the end of the process - about 6 months, we had a UI that people could learn to use in 5 mins and reach from point A to point B in an unknown building with 97% accuracy.
This just makes so much sense. A business that can immediately see the direct actions of it's consumer, and also react immediately, is so far ahead of the common business model. With the instant access of the internet, isn't it about time that businesses get on board with it, and work at the speed of modern technology? That is new business, and that is a model all of us business owners need to emulate.
While I have no doubt that observing such tests and acting on their results is "dead simple," I wonder about how simple it is to organise regular daily tests?
Think about where you work: an hourly slot in your diary that would not be regularly cancelled due to no-shows of the test subjects or the inability to find somebody willing to attend (presumably in office hours - lunch break?). This would erode the faith of the organisation as to whether to turn up of course. I'd also hate to be the poor sap fingered to actually pull these punters in off the street, give them a coffee, keep 'em sweet, be nice, etc. - even assuming the office is situated in some centre of normal human activity (and not, say, a desolate industrial estate). Herding cats comes to mind.
I bet there is in fact a significant overhead in doing this that's conveniently forgotten.
Gilgongo @ 11, that's absolutely right, but these are the cultural obstacles I mentioned. The point isn't "A business has to care about it's users to be willing to pay attention to them" -- that's *always* true. The point is that the other obstacles in the way of that paying attention have become significantly less serious.
Put another way, a company where employees are unwilling to listen to users has only its management to blame -- infrastructure is no longer a convenient excuse.
meetup was pretty poor on usability for a long time, so they may need this kind of crash course to get up to speed. I can't tell you how many folks tell me they can't rsvp for my meetup after I give them the meetup URL, and all the public comments for my meetup look like they were intended to be private feedback.
There is an excellent description of this technique in Steve Krug's book "Don't make me think". I have the book at all times at my hand's reach. Even now.
I wonder if they do one other thing which is often great for usability/function - "eating your own dogfood"
Long story (so I'll omit details) - we used a mainframe e-mail/calendaring program from IBM (not Notes) which they produced but was a different program from the mainframe e-mail/calendaring that all IBM employees used since it ran on a different operating system. Many things were difficult to do with this product. Finally someone got their management agree to have the developers of the product use it for their daily tasks - you can't *believe* how many improvements occurred within one year.
Perhaps Meetup uses their tool for their own meetings?
I applaud Meetup's approach to user testing and feedback. But I decry their 'salami approach' website design, where they sucker you in and it isn't until you've penetrated about 3 screens deep before suddenly it turns out they want your money. (To save you the aggravation of finding out for yourself, it's 12 bucks a month minimum.)
When I read stuff like this it makes me realise just how inventive I was nearly 10 years ago.
@2 You use msn IM and get them to relinquish control of their desktop to you.
I wonder if you could do in home usability testing with a similar set up. Have the tester download some basic screensharing software (even webex would probably work) and send them a free webcam and then ask them to walk through the site with you over the phone with the screencast.
No on site visits required. You could also lower the fee you pay them, because, hey, free webcam.
Terribly curious about this. Do they use a moderator/facilitator? Is there a script to follow? Are users given a task then left to fend for themselves in the room? how do they get so many people? Are they screened? Do they pull them off the street?