Remote Usability

We’re proud to announce our forthcoming book Remote Research, which will be published by Rosenfeld Media in 2009! It’s a book for everyone who’s interested in learning why, when, and how to design and conduct remote user research studies themselves.

From the book publisher’s website:

Remote user research describes any research method that allows you to observe, interview, or get feedback from users while they’re at a distance, in their “native environment” (at their desk, in their home or office) doing their own tasks. Remote studies allow you to recruit quickly, cheaply, and immediately, and give you the opportunity to observe users as they behave naturally in their own environment, on their own time. Our book will teach you how to design and conduct remote research studies, top-to-bottom, with little more than a phone and a laptop.

Be sure to follow the Remote Research book blog to get updates, read chapter excerpts, and even contribute your own suggestions and requests for the book; it’s still in progress, so we want to hear your comments!

Continue reading

book, remote research, rosenfeld media

While there are plenty of tools floating around to help you conduct one-on-one moderated interviews, if you’re just getting started with remote research, you may not want to invest in a ton of expensive software or subscription-based web services right off the bat. So let us show you a few handy ways to use more common or free tools and software to hack together some of the basic tools you’ll need to conduct user research. We’ll go task-by-task, and show you how to do things quickly and cheaply; we’ll even supply some of our very own tools. You are impressed.

Calling

One of the great benefits of remote testing is that you can test with users anywhere in the world, as long as they have a phone and internet connection. Unfortunately, phones still cost money to dial long distance, even using cell phones. What to do? Well, your users have internet connections, don’t they? A Skype membership is good if you’re going to be doing a lot of long-distance testing, and they also offer prepaid minutes if you’re just doing a few sessions.

Screensharing

It’s vital to be able to see your users’ screen during the session–that’s the heart of remote web research. Like we’ve said, UserVue is our preferred screensharing and recording solution, but if you’re not in the habit of doing remote studies regularly (which you should be, anyway), you might balk at the $150/month license fee. Not to worry–there’s tons of cheaper ways to screenshare. Silverback is a Mac-only  LiveLook, GoToMeeting, Windows Meeting Space (formerly NetMeeting) and Adobe Connect are all common screensharing apps, some of which support audio.

Recording

It’s good to also record the sessions so that you’re able to go back to them later and catch anything you may have missed. If you’re using UserVue, the recording is automatic, and you get a nice, beautifully synced WMV file right at the end of the session. But what if you’re just using LiveLook or GoToMeeting, which may not even support audio? Here’s where things can get a little hairy.

First, find some software that will allow you to record video and audio output on your computer; here at Bolt|Peters, we use Techsmith’s Camtasia, but free alternatives like CamStudio are also out there. Then, make sure that both the video and audio of the session are coming out of your computer; if you’re using Skype, that’s no problem; if you’re on a land phone, you’ll need to use some gadgets that will let you route the phone signal into your computer; we use a Hello Direct Pro Amplifier, which runs about $80 new, along with a JK Audio QuickTap phone tap, $60 new. Here’s a diagram illustrating how all of the pieces (phone, computer, amplifier, phone tap) fit together:

Once you’ve got both the screensharing and audio coming through your computer, fire up your recording software and go nuts.

Notetaking

Sure, you could just fire up Wordpad and type away, but if you’ve already got a nice recording of your sessions, you’re going to want to be able to track when in the video you were typing your notes out. Here’s where automatic time-stamped notes come in; it’s a little nitty-gritty, but it’s free, so check out our B|P blog post about it here!

Okay so did I just go broke or what

What would the equipment costs and expenses be, at the bare minimum, for a typical eight-user moderated study? I’m bout to spit some game:

Skype: 2.1 cents/min worldwide,  x 45 minute sessions x 8 users = $7.50, give or take (might as well get a subscription at that rate)

LiveLook: 2.5 cents/min x 45 x 8 = ~$9

CamStudio: free

Microphone headset, computer, internet connection: I was sort of hoping you already had these

Trillian for timestamped notetaking: free

TOTAL EQUIPMENT EXPENSES: ~$16.50

Hooo! Bear in mind that these are bare-bones essentials for a study; most studies will require some special doodad or another to make it work properly, and if you want really slick, hassle-free testing, well, that’ll cost you just like it always does. Still: a whole moderated remote study for the cost of a sushi dinner. Can’t hate that.

(Photo cropped from etohaholic on flickr)

Continue reading

calling, cheap, cost, equipment, expenses, livelook, notetaking, recording, remote research, screensharing, skype, sushi dinner, timestamp, uservue, ux

In an article for Boxes and Arrows, Paul Nuschke lists five phases of a usability study:

Step 1: Sales & Kickoff
Step 2: Recruitment
Step 3: Preparation
Step 4: Testing
Step 5: Analysis & Reporting

This post is about that second step, where you’re recruiting users to participate in your study. Traditionally this has been done one of two ways: either hire a third-party recruiting agency to find users according to specific criteria, or do it with your company’s contacts, typically customer / member email lists. Thing is, recruiting agencies are quite expensive, typically running around $200 or more per recruit (not including participant incentives), and even though they’re usually quite professional in their screening practices, who knows where they’re getting their users from? On the other hand, using your own company’s contacts bears another set of issues: what if you don’t have a very big list? What if you want to do several studies–do you bombard everybody with emails repeatedly? And so on.

Now, one of the best parts about remote research is that there’s no reason you have to schedule users in advance; since you test your users while they’re at their computers anyway, you can begin a study right when they agree to participate. Here’s where live recruiting comes in: by intercepting visitors with pop-ups or forms, you can intercept them, screen them, and call them within minutes of their arrival. This is a big advantage for lots of reasons: you can bypass much of the sometimes-lengthy recruiting step, you can have greater control and transparency over the source of your users, and most importantly, you can talk to real users who came to your site because they wanted to, not because they’re getting paid to.

So now the question is, how do you recruit users live? Well, you could hack together a form that users could use to opt-in to your study, but that requires you to hand-code the form and mess with your page content. Some remote research tools and web services like UserZoom and WebEffective offer Javascript-based intercept forms as part of their service, but they require you to sign up for the whole enchilada.

What does that leave? I am so glad you asked.

Ethnio is a free web-based service we made for the express purpose of recruiting users for remote research. All you do is stick one line of Javascript near the bottom of whichever page you want to recruit from, and users who visit that page will be greeted with a DHTML pop-up screener, which they can fill out in under a minute (here’s an example). You have complete control over the questions in the screener, so you can screen users however you want. When they finish, you see their responses come up immediately in a nice little recruiting table, and you can call whomever looks like a good fit for the research.

To illustrate, we have made a stop motion movie using felt. Please enjoy it with all your heart.

(Photo credit: bryanwright5 on flickr)

Continue reading

ethnio, felt, Live recruiting, remote research, stop-motion, userzoom, webeffective

In general, remote user research simply describes any research where the moderator and the research participants are physically separated. However, there are lots of different varieties of remote user research, and each has its own strengths, weaknesses, and circumstances in which they’re most effective. Some types allow you to test many people at once, while others give you a detailed look at just a few users’ behaviors. Some techniques are best for testing fully-functional live websites, but prototypes, wireframes, and sketches sometimes require specialized methods. Just to begin, let’s start with the two broadest categories of remote user testing, Moderated and Unmoderated (or Automated):

Moderated Research

Moderated research has the research facilitator (a.k.a. “moderator”) speaking directly to one or more participants. Examples of moderated research include one-on-one interviews, ethnographies, and group discussions. The major benefit of moderated research is that you can gather very in-depth qualitative feedback: not just opinions, but physical behavior, tone-of-voice, facial expression, and so on. A moderated discussion also allows the moderator to probe on new subjects as they arise over the course of a conversation, which makes the research more flexible in scope and makes it possible to explore interesting ideas and usages that were unforeseen during the planning phases of the study–we like to call these “emerging topics”.

Unmoderated / Automated Research

Unmoderated research is, of course, the complement of moderated research: the moderator does not speak directly with the participant, but instead uses a web-based tool or service to gather the feedback automatically (hence the alternate “Automated research” moniker). Typically, unmoderated research is used to gather quantitative feedback from a large (i.e. hundreds or more) sample. There’s all sorts of feedback you can get this way: you can use online surveys to get open- and closed-ended (multiple choice) opinions, use flash- or Ajax-based card sorting tools to understand the way users mentally categorize things, or use clickmaps and mouse tracking to see where users are clicking on a page to accomplish a particular task.

Which Type Should I Use?

Good question! It mostly depends on the specificity and nature of the thing you’re trying to find out. Are you trying to figure out if your webpage is generally easy-to-use, or trying to root out problems you might not have foreseen, or trying to get insight into the way your product fits into people’s lives? Then you’ll want to go with moderated research, which provides you with a very rich portrait of users’ behavior and usage context. On the other hand, are you trying to decide what color a webpage should be, where to place a particular button, or how to organize a navigation bar? For those specific small-scale questions, it’s probably best to go with an unmoderated method, which will allow you to get a broad look at how a large sample addresses a particular task or question.

We’ll be posting more on specific methodologies in the future; keep an eye out!

(Photo credits: foundphotoslj and racatumba on Flickr)

Continue reading

Automated, card sorting, closed-ended, emerging topics, ethnographies, feedback, Moderated, one-on-one interviews, open-ended, qualitative, quantitative, remote research, types, unmoderated

UserVue has long been our tool of choice for moderated remote user research, but we’ve always kind of hated that we couldn’t use it internationally — instead, we like using LiveLook or GoToMeeting along with a phone tap and screen recorder (Camtasia or CamStudio). But if you’re going to take a stab at using UserVue internationally anyway, here is a step-by-step guide to show you how to do it.

WHAT YOU NEED:

• Landline phone
• Computer w/ Skype and UserVue
• Microphone headset for the computer
• Twix brand chocolate bar

LET’S DO THIS:

STEP 1:

Download and register for an account.

STEP 2: Login to Skype and start a UserVue session, like normal.

STEP 3:

Click “Call” in UserVue.

STEP 4:

Put your desk phone number in as Your Phone Number put the Skype call-in number as Participant’s Phone Number, and then click “Dial”.

STEP 5: Have the first Twix bar (there are two in a package). You’ll need the energy for the next few steps.

STEP 6:

Pick up your desk phone when it rings, and then dial “1″ when the voice prompts you to do so.

STEP 7:

Answer the call in Skype. Your desk phone will be connected to your Skype line.

STEP 8:

Mute and silence the desk phone to avoid getting an echo. You won’t be speaking through it at all.

STEP 9:

In Skype, click Add Caller, and enter the participant’s number. If the built-in international line menu doesn’t work for any reason, then try setting the country to US and then typing in 011, the desired country code, then the participant’s number. Click Call.

STEP 10:

When the user picks up, begin the session. Direct him/her through the UserVue process as usual.

A WORD OF WARNING

Though it works and you can usually talk to your users just fine, there is a drop in the sound quality of the UserVue recordings. Also, the usual caveats apply when calling internationally with Skype, i.e. it drops out sometimes. You have been warned!

Continue reading

international, skype, uservue

Let’s start off by answering the question you probably came with. The short answer is: any kind of research where the user and research moderator aren’t interacting face-to-face.

So why do you care?

Well, take a look at how most research is currently done. When you want to learn about how people perceive and interact with websites or software or real-world devices, you design a study based on what you need to find out. Problem is, the logistics of actually talking to these people in person can get really, really hairy, especially if you’re working on any kind of a budget or a deadline. Say you wanted to conduct interviews with visitors to your website from across the country, or even from foreign countries; or say you didn’t have money to hire a recruiting agency to find a dozen qualified users to talk to.

So here’s where remote research comes in–with the tools, online services, and old-fashioned ingenuity that folks in the UX biz have come up with in the past couple of years, now practically anybody can conduct real, valid studies without an enormous R&D budget or a million dollar on-site testing facility. What’s more, there are a lot of benefits to testing remotely: allowing clients to participate, testing users in their native environments, getting better and more realistic feedback, getting automatically recorded sessions, bypassing recruiting agencies, and more.

Of course, there are a lot of things to learn about this new field of research; this blog is for people who want to learn more.

And by the way, hi! We’re Bolt|Peters, a research and design company located in San Francisco, specializing in remote research methodologies. We test everything from video games to medical websites to industrial design software, and with seven years of experience under our belts, we’ve worked with the likes of Sony, Greenpeace, CNN, AAA, Time Warner, Oracle, Autodesk, and more. Maybe you’d be interested in doing a bit of remote research? Check us out.

Continue reading

Automated, bolt peters, geographic diversity, Moderated, on-site testing, R&D, remote user research, unmoderated

Rashmi Sinha and I created this graph of different UX research methods for User Experience Week in D.C. in 2006, and posted to remoteusability.com in 2007. This week, Christian Rohrers posted a similar yet more detailed graph of all user experience research methods on Alert Box. They are pretty similar, but I actually like his axis labels better, and the display of methods rather than tools. Especially because a bunch of the tools from 2006 are gone now. Ah, the web.

Continue reading

christian rohrers, concrete vs. conceptual, moderated vs. automated, rashmi sinha, remote research tools, remote usability, remote user research, user experience