Selling Usability : User Experience Infiltration Tactics

지금 서울여대 교수님으로 계신 전 NHN UX Lab의 이지현 교수님과 함께 랩원들은 그 동안 한국에 UX를 전파하고, 회사 내에서도 UX Process를 만들기 위해서 많은 고생을 했다. UX에 대해서 잘 알려지지 않은 시절, 한국에서 성공적인 사례가 없었던 시기부터 UX를 알리기 위해서 정말 많은 고생을 한 것이다. 그 결과 나름 한국 기업 중 UX Process를 만든 성공적인 사례로 만들었고, 벤치마킹을 하고 싶은 부서로 만들지 않았나 생각한다. 하지만 그렇다고 NHN 내에서도 UX Process를 완벽하게 구축했다고 생각하지 않는다. 우리의 역할에 대해서, 그리고 사내에 더 많은 사람에게 UX를 단순한 지식이 아닌 서비스를 제공하는 주요 관점 중 하나로 알리기 위해서 더 많은 노력을 하고 있다.

 

그런 과정 속에 있는 나는 얼마 전에 아마존에서 한 권의 책을 알게 되었고, 지금은 너무나 정신없이 이 책을 읽고 있는 중이다. 그 책을 보면서 지난 고생했던 시절과 함께 앞으로 사내에, 더 나아가 한국에 UX를 제대로 정착시킬 것인가에 대해서 더 많은 고민을 하게 되었다. 그리고 그 안에서 나름 방법을 찾게 된 것은 아닌가 하는 생각을 하게 된다.

 

 

그 책은 John S. Rhods가 쓴  Selling Usability : User Experience Infiltration Tactics  책이다. John S. Rhods는 한국에는 잘 알려지지 않았지만, 꾸준히 UX를 전파하기 위해서 열심히 노력한 사람이다. 우리가 보기에 미국이라는 나라에서 IBM, MS, Google 등 많은 대기업 내에 UX 부서가 있고, 그들의 위치가 나름 확고하다고 생각하고 있다. 그래서 미국은 UX의 천국이 아닌가 하는 생각을 하게 될지도 모른다.

 

 

 

하지만, 미국도 아직까지는 더 많은 기업들이 UX에 대해서 제대로 이해하고 있지 못하며, 실제 조직 내에서도 그 필요성에 대해서 제대로 인식하고 있지 못한다. 그래서, UX를 하는 사람들의 고충도 한국 못지 않게 큰 것이 사실이다. 오죽했으면 User-Centered Design Stories: Real-World UCD Case Studies 이라는 책이 나올 정도였겠는가... 미국 기업 내에서도 UX를 한다는 것은 한국에서와 마찬가지로 아주 힘든 상황인 것이다.

 

Selling Usability : User Experience Infiltration Tactics  에서는 UX에 대해서 전혀 알지도 듣지도 못한 기업일지라도 어떻게 하면 UX를 정착시킬 수 있는지 나름 Tips을 제공하고 있다. 이 Tips은 John S. Rhods가 UX 컨설팅을 하면서 겪은 경험에 기초해서 나온 것이다. 실제 경험에서 나온 만큼 매우 유용한 것들이 많으며, 현재 내게 닥친 이슈에 대해서 어떻게 해결하면 좋을지 해결 방향에 대해서 조언을 해 주고 있다. 더 많은 생각과 실타레를 푸는 방식을 말이다.

 

책 자체도 매우 쉽게 쓰여져 있다. 부담없이 읽을 정도이다. 하지만, 그 안에는 무수한 보물이 숨겨져 있으며, 우리가 전략적으로 어떻게 하면 UX를 사내에 전파시킬 수 있을지에 대해서 매우 유용한 Tips이 있다. 한국에서 UX를 하는 사람, 사내에 UX를 전파하고자 하는 사람은 꼭 읽어봤으면 한다. UX를 전파한다는 것은 매우 고도화된 전략과 접근이 필요한 것이다.

 

 

왜 진작에 이런 책이 나오지 않은 것이지!!!!

 

데이터에 대한 환상... 그리고 그 결과...

For usability engineering purpose, one often needs to draw important conclusions on basis fairly unreliable data, and one shoul certainly do so since some data is better than no data.

- Jacob Nielson, 1993

 

즉, Jacob Nielson은 데이터가 전혀 없는 것보다 있는 것이 낫다고 주장한다. 심지어 그것이 부정확한 신뢰롭지 못한 데이터라도... 물론, 데이터가 없는 것보다 조금이라도 있는 것이 낫다. 하지만, 그 데이터가 부정확하고 신뢰롭지 못한 데이터라면 문제가 달라진다. 우리는 잘못된 데이터로 잘못된 결론에 도달할 가능성이 그만큼 커지게 된다.

 

실제 현장에서 일을 하다보면, 데이터 부재라는 상황에 많이 부딪힌다. 우리가 원하는 정도의 풍부한 데이터는 없다. 이곳 저곳에서 데이터를 끌어오고, 자체적으로 다양한 리서치를 수행한다고 하더라도 항상 데이터에 대한 갈증에 목 마르게 된다. 그렇다고 해서, 우리는 Insights를 뽑기 위해 멈출 수는 없는 것이다. Jacob Nielson의 말처럼 종종 부정확하고 신뢰롭지 못한 데이터에서 Insights를 뽑아내야 하는 상황에 부딪히게 된다. 이때 고민을 하게 된다. 과연 우리는 Insights를 어떻게 뽑아야 하고, 그 Insights가 맞다고 확신할 수 있을까? 쉽지 않은 일다.

 

하지만, 확실한 것은 있다. 그 어떤 상황이라도 데이터에 대해서 신중하게 접근하여 분석해야 하고, 조금이라도 더 많은 데이터를 확보하려고 노력해야 한다는 것이다. 그리고 가능한 많은 다양한 관점으로 데이터를 보려고 해야 한다는 것이다. 어떤 종합적인 시각으로 볼 것인지, 어떤 부분이 빠져 있고 그로 인해 발생할 수 있는 문제점은 무엇인지, 그 문제를 해결하기 위해서는 어떻게 해야 하는지에 대해서 고민, 고민, 또 고민을 해야 한다.

 

프로젝트에서 한 순간의 실수는 그 이후의 작업 및 결과에 대해서 심각한 문제를 초래할 수 있다. 데이터에 대한 접근 및 분석에 있어서도 마찬가지이다. 이 사실을 절대 잊지 말아야 한다.

 

UX Research를 왜 하는가...목적에 대한 단상

수많은 리서치를 진행하면서, 또 다른 사람이 수행하는 리서치를 가이드해 주면서 느끼는 것이 공통적으로 하나 있다. 그것은 리서치를 왜 하려고 하는 것인가에 대해서 거의 생각하지 않는다는 것이다. 단지 필요하니까.. 하면 좋아보이니까... 그래서 리서치를 해야 한다는 것이다. 하지만, 리서치를 수행해야 된다는 것은 알지만 리서치를 통해 무엇을 알고자 하고 어떻게 활용해야 하는 것인지에 대해서는 생각을 잘 하지 않는 것 같다. 그 결과 리서치에 대한 목적 자체에 대해서 망각하고, 전혀 엉뚱한 결과를 내 놓으며, 심지어 리서치를 왜 했는지 의심까지 들게 된다. 이것은 리서치 자체에 대해서 우리가 잘못 생각하고 있는, 아니 리서치 자체에 대해서 전혀 이해하지 못한 결과인 것이다.

 

문제(Problems)가 있다. 우리는 그 문제가 무엇인지, 그리고 그 문제를 어떻게 해결해야 할지 알고자 한다. 리서치 목적을 수행하기 전에 반드시 필요한 것이다. 과연 우리가 알고자 하는 것은 무엇인지 정의하는 것이 필요한 것이다. 문제가 무엇인지 명확하게 정의되어야, 리서치를 통해 무엇을 알고자 하며, 거기서 나온 결과를 어떻게 활용할 것인가에 대해서 명확하게 정의할 수 있는 것이다.

 

리서치라는 것은 그 문제를 규정하고 해결하기 위한 방안을 모색하는 것이다. (물론, 리서치 목적에 따라 다르긴 하다. 하지만 여기서는 협의의 사용자 리서치에 국한하겠다.) 리서치 목적에서 이것을 명확하게 규정해야 한다. 만약, 모호한 형태로 리서치 목적을 규정한다면, 리서치를 진행하는 단계에서 우리는 방향을 잃고 넓은 망망대해 속에서 헤매게 될 것이다. 실제, 그런 사례를 많이 봤다.

 

리서치 목적만 규정된다면 실제 리서치를 진행하는 것은 그렇게 어렵지 않다. 리서치 목적에 맞는 방법론을 선택하고, 리서치 목적에 맞는 진행을 하고, 리서치 목적에 맞는 결과 분석을 하면 된다. 그리고 그 안에서 Insights를 도출하면 된다. 리서치 목적은 결국 리서치 전체에 대한 가이드 역할을 한다. 리서치 목적을 절대 잊어서는 안 된다.

 

리서치 목적을 정확하게 규정하는 것... 이것은 성공적인 리서치를 수행하느냐 못 하느냐를 결정하는 가장 중요한 요소이다. 리서치 계획을 하기 전에 반드시 왜 리서치를 수행해야 하는가에 대해서 신중하게 고민하고 결정해라.

 

 

 

Persona를 만들기 위한 준비 - 자료 수집

UX에 대한 관심이 많아지면서 User Modeling에 대한 관심 또한 많아지게 되었다. 아무래도 우리의 사용자가 누구이며, 어떤 특성을 가지고 있고, 어떻게 행동하며, 그 안에서 어떠한 것들을 원하는지, 그리고 우리가 어떻게 하면 더욱 풍부한 사용자 경험을 제공해 줄 수 있는지에 대해서 알 필요가 있기 때문이다. User Modeling에 대한 관심이 증가하면서 Alan Cooper가 제안한 Persona에 초점을 맞추게 된다.

 

하지만 실제 업무에서 Persona를 만들려고 할 때 우리는 많은 것들을 고민하게 된다. 도대체 어디서부터 무엇을 해야할 것인지에 도저히 감을 잡을 수가 없다. 그러다 무작정 현장에 나가 사용자들의 행동을 관찰하고 인터뷰를 한다. 그리고 그것을 토대로 Persona를 만들고 그것을 토대로 서비스를 기획하거나 만들게 된다. 문제는 이러한 접근은 상당히 실패할 가능성이 매우 높고, 오히려 Persona에 대한 부정적인 태도를 취하게 만들 가능성이 매우 높아지게 된다는 것이다. 잘못된 User Modeling 자체가 서비스 제공 방향에 막대한 피해를 입히게 되기 때문이다. 그렇기 때문에 Persona를 만들 때 우리는 신중하게 접근할 필요가 있다.

 

그렇다면, 무엇보다 우리의 사용자는 과연 누구인지, 그리고 어떻게 접근하여 어떤 Persona 유형을 만들지에 대해서 고민을 해야 할 것이다. 먼저 우리의 사용자가 누구인지에 대해서 알 필요가 있을 것이다. 이 때 가장 좋은 방법은 다양한 2차 자료들을 활용하는 것이다. 물론 사내 마케팅 부서에서 조사한 데이터도 포함해서이다. 만약 Market Segments나 거기에 따른 Market Portion 데이터까지 있다면 금상첨화일 것이다. (인터넷 사이트라면 Log Data도 매우 중요한 Source가 될 수 있다.) 이것들을 토대로 우리 서비스의 사용자의 범위가 어느 정도 되는지 알 수 있다. 그리고 대략적인 특성까지 파악할 수 있을 것이다. (아마 인구통계학적 특성과 약간의 행동/태도 특성까지 알 수 있을 것이다.) 그리고 무엇보다 정량적 자료를 어느 정도 획득할 수 있고 이것이 나중에 Persona를 타당화시키는 것에 많은 도움을 줄 수 있다.

 

(실제 정성 데이터에만 의존하는 Persona는 많은 위험 부담을 안고 있으며, 신뢰성에 있어서도 문제가 발생할 가능성이 매우 높다. Persona를 만들 때에는 항상 정량 데이터와 정성 데이터를 같이 참조해야 한다. 그리고 Persona를 만든 후 Primary Persona와 Secondary Persona를 정할 때 다양한 정량 데이터와 함께 고급 통계 분석을 활용한다면 신뢰성 및 안정성을 더 많이 확보할 수 있다.)

 

이것을 토대로 사용자에 대해서 구체적으로 알아보기 위해 정성 조사를 실시한다. 이 때 다음과 같은 질문에 대답할 수 있는 결과물을 얻도록 노력한다. (John Pruitt와 Tmara Adlin의 'The Persona Lifecycle' 참조)

 

• 사용자의 목적은 무엇인가?
• 다양한 목적을 이루기 위해서 사용자들은 어떤 역할 또는 어떤 행동을 취하는가?
• 각각의 역할에 따라 어떤 특정 태스크나 행동들이 요구되는가? 그리고 무엇이 그러한 태스크나 역할들을 필요로 하게 만드는가? 이러한 활동들에 대한 태도나 감정은 어떠한가?
• 사용자들은 다른 사용자들 또는 제품/서비스와 상호작용하는가?
• 현재 제공되고 있는 제품/서비스에 대해서 사용자들은 어떤 것을 좋아하고 어떤 것을 싫어하는가?
• 사용자가 제품/서비스를 사용하고 있는 주변 환경이나 맥락은 어떠한가?
• 사용자들에게 나타나는 일반적인 성격 특징들은 무엇인가? 이것이 문화나 언어로 표현되고 있는가?

 

이러한 정성 데이터들을 획득하게 된다면, 사용자들이 행동하고 말하는 방식에 대해서 매우 구체적이고 풍부한 자료를 얻을 수 있게 될 것이다.

 

이처럼 Persona를 만들기 전에 다양한 데이터들을 확보하도록 해야 한다. 가끔 잊어버리는 것이 너무나 정성 조사 데이터에만 의존하게 된다는 것이다. 예전에 아는 사람 한명이 3명을 대상으로 정성 조사를 실시했고, 그 결과 3개의 Persona를 만들었던 것을 본 적이 있다. 우리는 절대 이런 오류를 범해서는 안 되는 것이다. 정성 조사 데이터 뿐만 아니라 이용할 수 있는 모든 데이터 - 정량 데이터까지 포함 - 를 확보하고 활용할 수 있어야 하는 것이다. 그렇게 된다면 Persona에 대한 신뢰성 및 안정성을 확보할 수 있게 된다.

 

 

 

좋은 모더레이터가 되기 위한 조건

이것은 해외 법인에서 근무하고 있는 UX Lab 멤버에게 교육시키기 위해 작성했던 것이다. 좋은 모더레이터가 되기 위해서 어떤 자세로 어떤 노력이 필요한지 생각해 봤으면 한다.

 

 

1. Introduction

 

The test moderator is in charge of the actual test, and as such has primary responsibility for all that occurs during the test. The test moderator will, at minimum, greet, interact with, and debrief each participant, and take ultimate responsibility for compiling and communicating test results to the development team. Since this is such an important role and the person in conducting the usability test or the playtest, the test moderator should keep the following the several guidelines to become a good moderator.

 

 

 

2. Characteristics of a Good Test Moderator

 

Regardless of who conducts the test, either yourself or other team member or external staff, and regardless of the background of that person, there are key characteristics that the most effective test moderators share. These key characteristics are listed and described in the paragraphs that follow. If you want to become a good moderator, use these key characteristics as a checklist of the skills you need to acquire. If you consider hiring an external person to perform this role, use these key characteristics to help evaluate the person’s capabilities.

 

 

2.1. Grounding in the basics of usability engineering

 

Grounding in basics of human information processing, cognitive psychology, and user-centered design helps immensely because it enables the moderator to sense, even before the test begins, which interactions, operations, messages, or instructions are liable to cause problem. Test moderators with this background have a knowledge of which problems can be generalized to the population at large and which are more trivial. This helps to ascertain when to probe further and what issues need to be explored thoroughly during the debriefing session. Additionally, this background can also prevent the need to test situations that are known to cause problems for users, such as inappropriate use of color or the incorrect placing of a note in a manual. Lastly, a strong background in usability engineering helps the test moderator to focus on fixing the important issues after a test is complete.

 

In addition of grounding in the basics of usability engineering, you should track the recent trends related the test fields such as games. Through the deep and broad understandings of the recent trends, you can take a hint for solving the problems. Never stop getting any information related to the various backgrounds that could help conducting the usability test.

 

 

2.2. Quick Learner

 

An effective test moderator need not be expert in the intricacies of the specific product or service being tested. For example, if the product is a database management system, the moderator need not be an expert in database management. However, moderator must be able to absorb new concepts quickly and to integrate these concepts into your thinking, vocabulary, and prior backgrounds related to usability engineering. The moderator also needs to absorb all the peripheral issues surrounding a product, such as its positioning the market place, competitors, and historical problems. During the test itself, the moderator must be able to understand the actions and comments of the participant quickly, as well as the implications behind those actions and comments. Being a quick learner enables the moderator to probe and question effectively.

 

 

2.3. Instant Rapport with Participants

 

Bringing in participants to evaluate your product is an auspicious and very opportune point in the development cycle that should not be squandered. If for some reason a participant is not at ease and is not able to function as he or she normally would, it represents a lost opportunity and potentially misleading results. If you are able to test only five participants, one uneasy participant represents a potential loss of 20 percent of your data set. The test moderator’s ability to quickly size up each participant’s personality, make friends, and put the person at ease is essential to getting the most from the testing process. Some participants need coddling, some need stroking, and some are businesslike and require a more formal research. Whichever the case, the test moderator must make each person feel comfortable and secure. To make participants feel comfortable and secure, when participants enter the test room, the test moderator should greet them with a warm smile and tell them “please, you imagine that you meet your friends, and take it easy”. The test moderator should make them feel like that they are in the familiar place. The test room which is full of strange equipments to participants – cameras, one-way mirror, and etc – would make participants feel strange and scared.

 

Especially, when facing children, the test moderator must make them feel that you are not a strange adult but a close friend. If children feel that the test moderator are not a friend but a strange adult, they are scared and never tell honestly what they tell related to the test. Don’t say or think all in the view of an adult. Do say, think, and see all in the view of children. If so, you are able to make the easy and comfortable relationship with children.

 

During the warming-up session of the test, the test moderator should develop rapport with participants at maximum. IF the test moderator fails to develop rapport with them, the test will not go well and the loss of the data will happen.

 

 

2.4. Excellent Memory

 

Some might believe that since usability test sessions are videotaped, the test moderator need not rely on memory from conducting and evaluating a test session. Actually, memory is called into play well before a test session has ended. Since a test session can be two to three hours or even longer, the test moderator needs to remember behaviors or comments that took place earlier in the session in order to cross-check and probe those behaviors later in the session. For example, a participant may attempt to perform the same task in two or three different ways, and the test moderator may want to probe to understand why the participant performed the task differently each time.

 

Memory is also required to recall the results of a test session after its completion. Since there is often very little time to devote to searching the videotapes after a test, except as insurance against missing some point entirely, the test moderator often must rely heavily on memory and notes.

 

The test moderator must take precautions against taking notes during a test session. You would take notes in detail on all whatever participants tell. However, such test moderator’s behavior would make participants uncomfortable. The test moderator should take notes on only important things and remember other things that would happen during a test session. If you want details related to the test, you are able to refer to a full script or videotapes and other team member such as an observer may help you to understand and know other things that you may not see or remember.

 

 

2.5. Good Listener

 

Listening skills involve the test moderator’s ability to hear with “new ears” during each session and to lay aside personal biases and strong opinions about what he or she is seeing and hearing. The test moderator needs to understand both the content and the implication of a participant’s comments, as there are often mixed messages of all kinds during testing. The test moderator must pick up on the subtle nuances of speech and emphasis, as a participant’s comments are often indirect and less than forthcoming.

 

When the test moderator conducts the test especially with children, the test moderator must be careful. Children are affected easily by situations surrounding them, other opinions, or impulsive feelings. The moderator should be able to distinguish the real and important comments from impulsive comments. In the case of adults, this may not be somewhat different from children. It is so important to understand the rationale behind the participant’s behavior, because the rationale often signals whether a change in the service is required or not.

 

It is also important not to suggest to participants what you think about. Just simple gestures or words of the moderators would make participants tell something opposed to their real thoughts. Do not compel participants to follow the moderator’s thoughts consciously or unconsciously. Just be a good listener.

 

 

2.6. Comfortable with Ambiguity

 

Usability is not a precise science consisting of formulas and black and white answers. Even if a usability test is conducted under the most rigorous conditions, which is atypical, you are still not assured that all of the results are valid and generalizable to your entire user population. Instead usability testing can often be an imprecise, ambiguous enterprise, with varying and sometimes conflicting observations, not surprising for any venture that has human beings as its focus. A test moderator, then, must understand and be comfortable with ambiguity. For example, a function is good to someone, but other participant may tell you that it is not very easy to use this function and do not know why it is necessary. However, it is important to exactly know what real problems are and grasp the nature of problems.

 

Another example is followed. Prior to testing you may think that there are only two ways to perform a particular task. During testing though, you discover that the participants have found four other ways to perform the same task. Or, you discover that you are no closer to a clear-cut resolution of a service’s problems after a week of testing than you were before you began. Or, when testing multiple versions of a service, no clear winner emerges. The versions are all equally bad or, if you are lucky, equally good. These situations require patience, perseverance, and very often skill at negotiation. Without tolerance for ambiguity and the patience to persevere, the test moderator tends to rationalize and to blame the participants for making unplanned choices during the test.

 

 

2.7. Flexibility

 

Another related characteristic of an effective test moderator is flexibility, which has to do with knowing when to deviate from the test plan. There are times when a particular participant does not have the expected skills or simply views the task in a completely different way than was originally intended.

 

When conducting the usability test, the moderator would face the various situations that may not be expected or considered in the stage of test planning. However, the moderator may ignore these situations and conduct the test according to the guideline. But such actions waste time and labors. Sometimes, the test may not run to test plan. When you face such a situation, never lose one’s presence of mind. Roll with a punch with the punch.

 

Before the test begins, the moderator must fully know and understand the overall progresses and the guideline. If fully knowing and understanding, the moderator is able to cope with the various situations that may not expected or considered.

 

 

2.8. Long Attention Span

 

Experienced test moderators share a secret: Usability testing can be tedious and boring. There are long stretches when seemingly nothing is happening, when participants are reading and absorbing, thinking, and sometimes just resting. The moderator cannot posses the type of personality that needs new stimulation every five to ten minutes. The moderator must be able to pay attention for long periods of time because there is no predicting when a gem of discovery will arise during a test session. In addition, since the moderator may view up to 10, 15, or 20 seconds, all of which observing the same or similar tasks, the ability to stay focused is extremely vital.

 

Bear in mind that participants may feel that the moderator feels tedious and boring. Participants are very sensitive to the moderator’s feeling and behaviors. If the moderator make participants feel that the moderator are tedious, tired, or bored, the test may not run as the moderator expects. To become a good moderator, the moderator should be able to maintaining his or her conditions consistently and never make participants feel that he or she are tedious, tired, or bored.

 

 

2.9. Empathic “People Person”

 

Participants will relate more readily to a test moderator who is an empathic individual. This may not be all that critical during the test session itself, especially if the session requires little probing or exploration on the part of the test moderator. However, empathy can play a major part during the debriefing session when the test monitor is trying to elicit a participant’s innermost thoughts and feelings about the previous two hours of work. Participant will tent to hold back if they feel that the test moderator cannot relate to their particular situation, this being especially true if the session was unusually frustrating or difficult.

 

 

2.10. “Big Picture” Thinker

 

There is so much data collected during a usability test and there is so much data that could be collected during a test that it is very easy to lose sight of the forest for the trees. The test moderator must be able to weed out the significant from the insignificant, and this ability takes two concrete forms.

 

1.                The ability to draw together all of the various inputs, comments, and data from a single test to form a cohesive picture of a participant’s performance.

2.                The ability to draw together the varied inputs from different from different test sessions and focus on the most important and critical findings.

 

It is very easy to get lost in the details and focus on trivial observations. It is also easy to be influenced by the last participant and forget all that has come before. An effective test moderator, however, avoids these difficulties by staying focused on the big picture.

 

 

2.11. Good Communicator

 

Good communication skills are essential in usability testing. The test moderator must communicate with individual members of the development team, participants, observers, and other individuals who may be helping to administer the test in one way or another. The test moderator must be skillful at persuading others to make changes that are in their best interest, and he or she must be able to explain the implications behind the test result. Good writing skills are also essential because the test report is often the sole means of communicating test results to those who did not attend the test sessions. The written report is also the important historical document that relied upon months or years later to review or revisit the test results.

 

 

 

2.12. Good Organizer and Coordinate

 

A usability test is a project within a project. Even a simple test requires the management of an astonishing number of small details, events, and milestones. Ensuring that equipment is in running order, getting all participants to the site on time, and making sure that the service is ready for testing are ultimately the responsibility of the test moderator. In addition, the test moderator is the focal point for the other test team members, and must coordinate their activities as well as those of any outside consultants into a unified effort. Therefore, the test moderator should be a good organizer and coordinator.

 

 

3. Typical Test Moderator Problems

 

Now that you have reviewed some of the important characteristics that a test moderator should possess, let’s review some of the behaviors that test monitors should avoid. In this session, I will cover the most common “errors” that test moderator make while conducting a test, many of which I have learned from experience and books. Consider it a list of “what not to do.” Even experienced test moderators can benefit from taking a few moments just prior to testing to review the list. As with the previous list of characteristics, you can use this list to evaluate and improve your own performance or to evaluate the performance of someone you hire to conduct usability testing for you.

 

 

3.1. Leading Rather Than Enabling

 

Behavior that leads rather than enables is usually caused by the test monitor being too close to the service and unintentionally providing cues to the participant about correct performance. The test moderator’s tone of voice, a nod of the head, cutting short tasks when the participant struggles, even the type of question the most monitor asks can all influence the participant and the test results. Please, refer to the following a table as examples.

 

Correct Question	False Question	Reason
Would you tell me what you think about now?	Do you think _________ now ?	Even if the moderator know what the participant think about, the moderator never ask a direct question because the moderator’s guess of the participant’s thoughts may be incorrect. When the moderator ask the participant directly for why to do, the participant may think that their behaviors may be evaluated, and, then, guess that they may do the task in the wrong way.
Would you tell me what you will do?	Do you tend to _________ ?
Would you explain what you will tend to?	Do you tent to ________ due to _______ ?
Would you tell me whether it is easy to use the product or not?	Is it easy to use the product?	It is not easy to request participants to explain their opinions about the specified features. However, in any situation the moderator do not ask direct questions of participants. If you use direct questions, you suggest that participants should follow the moderator’s thoughts and attitudes.
Would you tell me whether you are able to understand the usage description or not?	Is it difficult to use the product?
Do you think the error message helped you or disturb you ?	Did the error message help you?
How do you feel?	Are you confused?

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Just an unconscious question may affect the participants’ behaviors and attitudes very much. This potential problem is the main reason why professional testing often conducted from a control room, although subtle hints are still possible even from there. To avoid the problem of leading rather than enabling, remember that the moderator is there to collect data, not to compel participants to follow the moderator’s attitudes and thoughts, and to ensure that the test environment remains as neutral as possible.

 

 

3.2. Too Involved with the Act of Data Collection

 

While the purpose of the test is to collect as much information as possible, the act of collecting that information should not interfere with direct observation of what is occurring. The test moderator needs to stay aware of what the participant is doing at all times, even if that means that every aspect of the participant’s behavior is not written down. That’s the purpose of videotaping or recording the test, of developing coding categories, and of having others help with the more rote collection of such things as timings and number of references to an index. All of these aids help ensure that the test moderator does not become so engrossed in the collection process that the moderator misses important behaviors or words.

 

 

3.3. Acting too Knowledgeable

 

This problem occurs when the test moderator and participant are in the same room during the test. Participants will tend to defer to the test moderator and ask many questions if they feel that test moderator knows all the answers. Being too knowledgeable can also intimidate a participant who may be somewhat nervous and self-conscious about his or her abilities. Another problem is that being too knowledgeable makes the participants depend on the moderators. Even whenever the participants are able to complete their task easily, they give up the task and ask advices of the moderator with ease.

 

One simple way to counteract the problem of acting too knowledgeable is for the test moderator to “play dumb.” That is, the test moderator downplays any knowledge of the service and takes on the role of a research technician who is simply collecting data. Frequently, there are shown that participants change their entire demeanor when it became clear that they were not going to receive any assistance from the test moderator. They began to try harder and to behave as if they were alone in their home or office.

 

 

3.4. Too Rigid with the Test Plan

 

An experienced test moderator will know when to deviate from the test plan. It is important for the moderator to sense when the test design is not achieving its objectives and is not exposing the product’s deficiencies to the fullest extent. At those times, it is up to the moderator to make the appropriate changes so that a participant’s time and efforts are not wasted. Sometimes a participant with a different background that what was expected will appear. Sometimes the tasks are the wrong ones for addressing the problem statements. Whatever the case, it is up to the moderator to revise the plan accordingly.

 

 

3.5. Not Relating well to Each Participant

 

Participants come in all shapes, sizes, and demeanors. Regardless of whether a particular participant is shy, arrogant, moody, intimidated, self-conscious, or whatever, the test moderator needs to adjust his or her style in order to allow the participant to be comfortable and at ease. The test moderator should not get involved with battling, belittling, or in any way making a participant feel like anything but a guest. As far as the test moderator is concerned, the participant is always right.

 

 

3.6. Using the Technical Terms

 

One of the frequent mistakes of inexperienced test moderator is to use the technical terms or terms that are used only within the company. Participants never hear, know, or understand what the terms mean. For example, when the moderator conduct the playtest of a FPS game and ask the following question, “How about this FPS game?”, the participants who don’t know the term “FPS game”, especially children, never answer this question. Just use the term that are very familiar, easy, and plain to participants. Never use the technical terms that are used only within the company. The moderator should see, think, and tell everything in the view of participants.

 

 

3.7. Jumping to Conclusions

 

Inexperienced test moderators tent to overreact to early results. This can cause other members of the test team to act on the data prematurely. It is up to the test moderator to maintain a cool, steady demeanor and remind everyone to avoid forming conclusions until all of the results are in, especially in the playtest of games. One of the reasons for testing multiple participants is just for that purpose: to get  a rounded, comprehensive view of the service through the eyes of different types of people with different types of backgrounds. While it is important for the test moderator to pick up patterns in the behavior of participants as early as possible, this does not necessarily mean reacting to that behavior. Avoiding premature conclusions will help to keep members of the test team from making major service changes before all the data is in.

 

 

4. How To Improve Your Test Moderating Skills

 

Conducting a usability test is an extremely challenging and worthwhile endeavor on a variety of levels. On the most ordinary level, you are working very closely with people in an attempt to design a product for maximum utility and benefit. On a deeper level, it is a very profound experience that forces you to confront your own mind and its tendency to be biased, distracted, and flighty. Moderating a test puts you on the spot and forces you to be extremely mindful and disciplined. You spend long periods of time maintaining concentration while observing people, all the time being as unobtrusive as possible.

The following guidelines will help you to become a excellent moderator. Carefully read the guidelines.

 

 

4.1. Learn the Basic Principles of Human Factors / Ergonomics

 

Learn the basic principles of human information processing, experimental psychology, cognitive psychology, statistics, interface design, and usability engineering. Attend seminars and study basic psychology course. These will be very helpful to conduct the usability test.

 

 

4.2. Learn from Watching Others

 

Watching other test moderators is a key to success. When you have an opportunity to watch an experienced test moderator at work, you get to see what works and what does not firsthand. If the opportunity presents itself, ask the test moderator why he or she uses a particular technique that caught your interest. Take notes about particular techniques and behaviors, and so forth, that seem particularly effective and try them out yourself. Again, do not let your concern for making mistakes prevent you from exploring new techniques.

 

 

4.3. What yourself on Tape

 

One of the benefits of taping your test sessions is that you have an ideal medium for reviewing your own performance. Take advantage of this technology by reviewing your sessions with the intent of improving your skills. Take notes on what you do well and on behaviors that need improvement. That way you will remember to work on those aspects the next time you conduct a test.

 

 

 

4.4. Work with a Mentor

 

Work closely with an experienced test moderator. Help the test moderator work on a test and have that moderator do the same for you. If it is a test with many participants, perhaps you can conduct some of the sessions. Have your mentor watch you and critique your performance.

 

 

4.5. Practice Moderating

 

Start with the right attitude. Do not be a perfectionist. You are going to make mistakes, bias participant, reveal information you should not, and invent new ways to invalidate a test session’s results. This is just par for the course. Usability testing has a twofold saving grace – testing multiple participants and iterative design. Testing multiple participants means that if you invalidate one session, there is always another opportunity to do it right. Iterative design also makes up for any mistakes you might make, because you have several chances throughout the service development life cycle to catch problems with the service. The important thing is not to get discouraged. Continue to practice, continue to learn, and continue to improve. Even the most experienced test moderator make mistakes.

 

 

4.6. Run a Pilot Test Prior to the Main Test Session

 

If you conduct a pilot test before the main test session, you will find many problems as not expected. These problems are the flow of the test session, your words, attitudes, test duration, and etc. As the pilot test is conducted, you may have a chance to correct the problems related to moderating the test. In any case, you should run a pilot test before the main test sessions.