좋은 모더레이터가 되기 위한 조건

이것은 해외 법인에서 근무하고 있는 UX Lab 멤버에게 교육시키기 위해 작성했던 것이다. 좋은 모더레이터가 되기 위해서 어떤 자세로 어떤 노력이 필요한지 생각해 봤으면 한다.

 

 

1. Introduction

 

The test moderator is in charge of the actual test, and as such has primary responsibility for all that occurs during the test. The test moderator will, at minimum, greet, interact with, and debrief each participant, and take ultimate responsibility for compiling and communicating test results to the development team. Since this is such an important role and the person in conducting the usability test or the playtest, the test moderator should keep the following the several guidelines to become a good moderator.

 

 

 

2. Characteristics of a Good Test Moderator

 

Regardless of who conducts the test, either yourself or other team member or external staff, and regardless of the background of that person, there are key characteristics that the most effective test moderators share. These key characteristics are listed and described in the paragraphs that follow. If you want to become a good moderator, use these key characteristics as a checklist of the skills you need to acquire. If you consider hiring an external person to perform this role, use these key characteristics to help evaluate the person’s capabilities.

 

 

2.1. Grounding in the basics of usability engineering

 

Grounding in basics of human information processing, cognitive psychology, and user-centered design helps immensely because it enables the moderator to sense, even before the test begins, which interactions, operations, messages, or instructions are liable to cause problem. Test moderators with this background have a knowledge of which problems can be generalized to the population at large and which are more trivial. This helps to ascertain when to probe further and what issues need to be explored thoroughly during the debriefing session. Additionally, this background can also prevent the need to test situations that are known to cause problems for users, such as inappropriate use of color or the incorrect placing of a note in a manual. Lastly, a strong background in usability engineering helps the test moderator to focus on fixing the important issues after a test is complete.

 

In addition of grounding in the basics of usability engineering, you should track the recent trends related the test fields such as games. Through the deep and broad understandings of the recent trends, you can take a hint for solving the problems. Never stop getting any information related to the various backgrounds that could help conducting the usability test.

 

 

2.2. Quick Learner

 

An effective test moderator need not be expert in the intricacies of the specific product or service being tested. For example, if the product is a database management system, the moderator need not be an expert in database management. However, moderator must be able to absorb new concepts quickly and to integrate these concepts into your thinking, vocabulary, and prior backgrounds related to usability engineering. The moderator also needs to absorb all the peripheral issues surrounding a product, such as its positioning the market place, competitors, and historical problems. During the test itself, the moderator must be able to understand the actions and comments of the participant quickly, as well as the implications behind those actions and comments. Being a quick learner enables the moderator to probe and question effectively.

 

 

2.3. Instant Rapport with Participants

 

Bringing in participants to evaluate your product is an auspicious and very opportune point in the development cycle that should not be squandered. If for some reason a participant is not at ease and is not able to function as he or she normally would, it represents a lost opportunity and potentially misleading results. If you are able to test only five participants, one uneasy participant represents a potential loss of 20 percent of your data set. The test moderator’s ability to quickly size up each participant’s personality, make friends, and put the person at ease is essential to getting the most from the testing process. Some participants need coddling, some need stroking, and some are businesslike and require a more formal research. Whichever the case, the test moderator must make each person feel comfortable and secure. To make participants feel comfortable and secure, when participants enter the test room, the test moderator should greet them with a warm smile and tell them “please, you imagine that you meet your friends, and take it easy”. The test moderator should make them feel like that they are in the familiar place. The test room which is full of strange equipments to participants – cameras, one-way mirror, and etc – would make participants feel strange and scared.

 

Especially, when facing children, the test moderator must make them feel that you are not a strange adult but a close friend. If children feel that the test moderator are not a friend but a strange adult, they are scared and never tell honestly what they tell related to the test. Don’t say or think all in the view of an adult. Do say, think, and see all in the view of children. If so, you are able to make the easy and comfortable relationship with children.

 

During the warming-up session of the test, the test moderator should develop rapport with participants at maximum. IF the test moderator fails to develop rapport with them, the test will not go well and the loss of the data will happen.

 

 

2.4. Excellent Memory

 

Some might believe that since usability test sessions are videotaped, the test moderator need not rely on memory from conducting and evaluating a test session. Actually, memory is called into play well before a test session has ended. Since a test session can be two to three hours or even longer, the test moderator needs to remember behaviors or comments that took place earlier in the session in order to cross-check and probe those behaviors later in the session. For example, a participant may attempt to perform the same task in two or three different ways, and the test moderator may want to probe to understand why the participant performed the task differently each time.

 

Memory is also required to recall the results of a test session after its completion. Since there is often very little time to devote to searching the videotapes after a test, except as insurance against missing some point entirely, the test moderator often must rely heavily on memory and notes.

 

The test moderator must take precautions against taking notes during a test session. You would take notes in detail on all whatever participants tell. However, such test moderator’s behavior would make participants uncomfortable. The test moderator should take notes on only important things and remember other things that would happen during a test session. If you want details related to the test, you are able to refer to a full script or videotapes and other team member such as an observer may help you to understand and know other things that you may not see or remember.

 

 

2.5. Good Listener

 

Listening skills involve the test moderator’s ability to hear with “new ears” during each session and to lay aside personal biases and strong opinions about what he or she is seeing and hearing. The test moderator needs to understand both the content and the implication of a participant’s comments, as there are often mixed messages of all kinds during testing. The test moderator must pick up on the subtle nuances of speech and emphasis, as a participant’s comments are often indirect and less than forthcoming.

 

When the test moderator conducts the test especially with children, the test moderator must be careful. Children are affected easily by situations surrounding them, other opinions, or impulsive feelings. The moderator should be able to distinguish the real and important comments from impulsive comments. In the case of adults, this may not be somewhat different from children. It is so important to understand the rationale behind the participant’s behavior, because the rationale often signals whether a change in the service is required or not.

 

It is also important not to suggest to participants what you think about. Just simple gestures or words of the moderators would make participants tell something opposed to their real thoughts. Do not compel participants to follow the moderator’s thoughts consciously or unconsciously. Just be a good listener.

 

 

2.6. Comfortable with Ambiguity

 

Usability is not a precise science consisting of formulas and black and white answers. Even if a usability test is conducted under the most rigorous conditions, which is atypical, you are still not assured that all of the results are valid and generalizable to your entire user population. Instead usability testing can often be an imprecise, ambiguous enterprise, with varying and sometimes conflicting observations, not surprising for any venture that has human beings as its focus. A test moderator, then, must understand and be comfortable with ambiguity. For example, a function is good to someone, but other participant may tell you that it is not very easy to use this function and do not know why it is necessary. However, it is important to exactly know what real problems are and grasp the nature of problems.

 

Another example is followed. Prior to testing you may think that there are only two ways to perform a particular task. During testing though, you discover that the participants have found four other ways to perform the same task. Or, you discover that you are no closer to a clear-cut resolution of a service’s problems after a week of testing than you were before you began. Or, when testing multiple versions of a service, no clear winner emerges. The versions are all equally bad or, if you are lucky, equally good. These situations require patience, perseverance, and very often skill at negotiation. Without tolerance for ambiguity and the patience to persevere, the test moderator tends to rationalize and to blame the participants for making unplanned choices during the test.

 

 

2.7. Flexibility

 

Another related characteristic of an effective test moderator is flexibility, which has to do with knowing when to deviate from the test plan. There are times when a particular participant does not have the expected skills or simply views the task in a completely different way than was originally intended.

 

When conducting the usability test, the moderator would face the various situations that may not be expected or considered in the stage of test planning. However, the moderator may ignore these situations and conduct the test according to the guideline. But such actions waste time and labors. Sometimes, the test may not run to test plan. When you face such a situation, never lose one’s presence of mind. Roll with a punch with the punch.

 

Before the test begins, the moderator must fully know and understand the overall progresses and the guideline. If fully knowing and understanding, the moderator is able to cope with the various situations that may not expected or considered.

 

 

2.8. Long Attention Span

 

Experienced test moderators share a secret: Usability testing can be tedious and boring. There are long stretches when seemingly nothing is happening, when participants are reading and absorbing, thinking, and sometimes just resting. The moderator cannot posses the type of personality that needs new stimulation every five to ten minutes. The moderator must be able to pay attention for long periods of time because there is no predicting when a gem of discovery will arise during a test session. In addition, since the moderator may view up to 10, 15, or 20 seconds, all of which observing the same or similar tasks, the ability to stay focused is extremely vital.

 

Bear in mind that participants may feel that the moderator feels tedious and boring. Participants are very sensitive to the moderator’s feeling and behaviors. If the moderator make participants feel that the moderator are tedious, tired, or bored, the test may not run as the moderator expects. To become a good moderator, the moderator should be able to maintaining his or her conditions consistently and never make participants feel that he or she are tedious, tired, or bored.

 

 

2.9. Empathic “People Person”

 

Participants will relate more readily to a test moderator who is an empathic individual. This may not be all that critical during the test session itself, especially if the session requires little probing or exploration on the part of the test moderator. However, empathy can play a major part during the debriefing session when the test monitor is trying to elicit a participant’s innermost thoughts and feelings about the previous two hours of work. Participant will tent to hold back if they feel that the test moderator cannot relate to their particular situation, this being especially true if the session was unusually frustrating or difficult.

 

 

2.10. “Big Picture” Thinker

 

There is so much data collected during a usability test and there is so much data that could be collected during a test that it is very easy to lose sight of the forest for the trees. The test moderator must be able to weed out the significant from the insignificant, and this ability takes two concrete forms.

 

1.                The ability to draw together all of the various inputs, comments, and data from a single test to form a cohesive picture of a participant’s performance.

2.                The ability to draw together the varied inputs from different from different test sessions and focus on the most important and critical findings.

 

It is very easy to get lost in the details and focus on trivial observations. It is also easy to be influenced by the last participant and forget all that has come before. An effective test moderator, however, avoids these difficulties by staying focused on the big picture.

 

 

2.11. Good Communicator

 

Good communication skills are essential in usability testing. The test moderator must communicate with individual members of the development team, participants, observers, and other individuals who may be helping to administer the test in one way or another. The test moderator must be skillful at persuading others to make changes that are in their best interest, and he or she must be able to explain the implications behind the test result. Good writing skills are also essential because the test report is often the sole means of communicating test results to those who did not attend the test sessions. The written report is also the important historical document that relied upon months or years later to review or revisit the test results.

 

 

 

2.12. Good Organizer and Coordinate

 

A usability test is a project within a project. Even a simple test requires the management of an astonishing number of small details, events, and milestones. Ensuring that equipment is in running order, getting all participants to the site on time, and making sure that the service is ready for testing are ultimately the responsibility of the test moderator. In addition, the test moderator is the focal point for the other test team members, and must coordinate their activities as well as those of any outside consultants into a unified effort. Therefore, the test moderator should be a good organizer and coordinator.

 

 

3. Typical Test Moderator Problems

 

Now that you have reviewed some of the important characteristics that a test moderator should possess, let’s review some of the behaviors that test monitors should avoid. In this session, I will cover the most common “errors” that test moderator make while conducting a test, many of which I have learned from experience and books. Consider it a list of “what not to do.” Even experienced test moderators can benefit from taking a few moments just prior to testing to review the list. As with the previous list of characteristics, you can use this list to evaluate and improve your own performance or to evaluate the performance of someone you hire to conduct usability testing for you.

 

 

3.1. Leading Rather Than Enabling

 

Behavior that leads rather than enables is usually caused by the test monitor being too close to the service and unintentionally providing cues to the participant about correct performance. The test moderator’s tone of voice, a nod of the head, cutting short tasks when the participant struggles, even the type of question the most monitor asks can all influence the participant and the test results. Please, refer to the following a table as examples.

 

Correct Question	False Question	Reason
Would you tell me what you think about now?	Do you think _________ now ?	Even if the moderator know what the participant think about, the moderator never ask a direct question because the moderator’s guess of the participant’s thoughts may be incorrect. When the moderator ask the participant directly for why to do, the participant may think that their behaviors may be evaluated, and, then, guess that they may do the task in the wrong way.
Would you tell me what you will do?	Do you tend to _________ ?
Would you explain what you will tend to?	Do you tent to ________ due to _______ ?
Would you tell me whether it is easy to use the product or not?	Is it easy to use the product?	It is not easy to request participants to explain their opinions about the specified features. However, in any situation the moderator do not ask direct questions of participants. If you use direct questions, you suggest that participants should follow the moderator’s thoughts and attitudes.
Would you tell me whether you are able to understand the usage description or not?	Is it difficult to use the product?
Do you think the error message helped you or disturb you ?	Did the error message help you?
How do you feel?	Are you confused?

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Just an unconscious question may affect the participants’ behaviors and attitudes very much. This potential problem is the main reason why professional testing often conducted from a control room, although subtle hints are still possible even from there. To avoid the problem of leading rather than enabling, remember that the moderator is there to collect data, not to compel participants to follow the moderator’s attitudes and thoughts, and to ensure that the test environment remains as neutral as possible.

 

 

3.2. Too Involved with the Act of Data Collection

 

While the purpose of the test is to collect as much information as possible, the act of collecting that information should not interfere with direct observation of what is occurring. The test moderator needs to stay aware of what the participant is doing at all times, even if that means that every aspect of the participant’s behavior is not written down. That’s the purpose of videotaping or recording the test, of developing coding categories, and of having others help with the more rote collection of such things as timings and number of references to an index. All of these aids help ensure that the test moderator does not become so engrossed in the collection process that the moderator misses important behaviors or words.

 

 

3.3. Acting too Knowledgeable

 

This problem occurs when the test moderator and participant are in the same room during the test. Participants will tend to defer to the test moderator and ask many questions if they feel that test moderator knows all the answers. Being too knowledgeable can also intimidate a participant who may be somewhat nervous and self-conscious about his or her abilities. Another problem is that being too knowledgeable makes the participants depend on the moderators. Even whenever the participants are able to complete their task easily, they give up the task and ask advices of the moderator with ease.

 

One simple way to counteract the problem of acting too knowledgeable is for the test moderator to “play dumb.” That is, the test moderator downplays any knowledge of the service and takes on the role of a research technician who is simply collecting data. Frequently, there are shown that participants change their entire demeanor when it became clear that they were not going to receive any assistance from the test moderator. They began to try harder and to behave as if they were alone in their home or office.

 

 

3.4. Too Rigid with the Test Plan

 

An experienced test moderator will know when to deviate from the test plan. It is important for the moderator to sense when the test design is not achieving its objectives and is not exposing the product’s deficiencies to the fullest extent. At those times, it is up to the moderator to make the appropriate changes so that a participant’s time and efforts are not wasted. Sometimes a participant with a different background that what was expected will appear. Sometimes the tasks are the wrong ones for addressing the problem statements. Whatever the case, it is up to the moderator to revise the plan accordingly.

 

 

3.5. Not Relating well to Each Participant

 

Participants come in all shapes, sizes, and demeanors. Regardless of whether a particular participant is shy, arrogant, moody, intimidated, self-conscious, or whatever, the test moderator needs to adjust his or her style in order to allow the participant to be comfortable and at ease. The test moderator should not get involved with battling, belittling, or in any way making a participant feel like anything but a guest. As far as the test moderator is concerned, the participant is always right.

 

 

3.6. Using the Technical Terms

 

One of the frequent mistakes of inexperienced test moderator is to use the technical terms or terms that are used only within the company. Participants never hear, know, or understand what the terms mean. For example, when the moderator conduct the playtest of a FPS game and ask the following question, “How about this FPS game?”, the participants who don’t know the term “FPS game”, especially children, never answer this question. Just use the term that are very familiar, easy, and plain to participants. Never use the technical terms that are used only within the company. The moderator should see, think, and tell everything in the view of participants.

 

 

3.7. Jumping to Conclusions

 

Inexperienced test moderators tent to overreact to early results. This can cause other members of the test team to act on the data prematurely. It is up to the test moderator to maintain a cool, steady demeanor and remind everyone to avoid forming conclusions until all of the results are in, especially in the playtest of games. One of the reasons for testing multiple participants is just for that purpose: to get  a rounded, comprehensive view of the service through the eyes of different types of people with different types of backgrounds. While it is important for the test moderator to pick up patterns in the behavior of participants as early as possible, this does not necessarily mean reacting to that behavior. Avoiding premature conclusions will help to keep members of the test team from making major service changes before all the data is in.

 

 

4. How To Improve Your Test Moderating Skills

 

Conducting a usability test is an extremely challenging and worthwhile endeavor on a variety of levels. On the most ordinary level, you are working very closely with people in an attempt to design a product for maximum utility and benefit. On a deeper level, it is a very profound experience that forces you to confront your own mind and its tendency to be biased, distracted, and flighty. Moderating a test puts you on the spot and forces you to be extremely mindful and disciplined. You spend long periods of time maintaining concentration while observing people, all the time being as unobtrusive as possible.

The following guidelines will help you to become a excellent moderator. Carefully read the guidelines.

 

 

4.1. Learn the Basic Principles of Human Factors / Ergonomics

 

Learn the basic principles of human information processing, experimental psychology, cognitive psychology, statistics, interface design, and usability engineering. Attend seminars and study basic psychology course. These will be very helpful to conduct the usability test.

 

 

4.2. Learn from Watching Others

 

Watching other test moderators is a key to success. When you have an opportunity to watch an experienced test moderator at work, you get to see what works and what does not firsthand. If the opportunity presents itself, ask the test moderator why he or she uses a particular technique that caught your interest. Take notes about particular techniques and behaviors, and so forth, that seem particularly effective and try them out yourself. Again, do not let your concern for making mistakes prevent you from exploring new techniques.

 

 

4.3. What yourself on Tape

 

One of the benefits of taping your test sessions is that you have an ideal medium for reviewing your own performance. Take advantage of this technology by reviewing your sessions with the intent of improving your skills. Take notes on what you do well and on behaviors that need improvement. That way you will remember to work on those aspects the next time you conduct a test.

 

 

 

4.4. Work with a Mentor

 

Work closely with an experienced test moderator. Help the test moderator work on a test and have that moderator do the same for you. If it is a test with many participants, perhaps you can conduct some of the sessions. Have your mentor watch you and critique your performance.

 

 

4.5. Practice Moderating

 

Start with the right attitude. Do not be a perfectionist. You are going to make mistakes, bias participant, reveal information you should not, and invent new ways to invalidate a test session’s results. This is just par for the course. Usability testing has a twofold saving grace – testing multiple participants and iterative design. Testing multiple participants means that if you invalidate one session, there is always another opportunity to do it right. Iterative design also makes up for any mistakes you might make, because you have several chances throughout the service development life cycle to catch problems with the service. The important thing is not to get discouraged. Continue to practice, continue to learn, and continue to improve. Even the most experienced test moderator make mistakes.

 

 

4.6. Run a Pilot Test Prior to the Main Test Session

 

If you conduct a pilot test before the main test session, you will find many problems as not expected. These problems are the flow of the test session, your words, attitudes, test duration, and etc. As the pilot test is conducted, you may have a chance to correct the problems related to moderating the test. In any case, you should run a pilot test before the main test sessions.