Experimental Design & Evaluation 14. Quiz 3 SunyoungKim,PhD - - PowerPoint PPT Presentation

Experimental Design & Evaluation

• 14. Quiz 3

SunyoungKim,PhD

1. Scientists always try to maximize reliability and validity in their research. a. True b. False 2. Reliability refers to whether we are truly measuring the concept of interest in our study. a. True b. False 3. The validity of a measure refers to the: a. accuracy with which it measures the variable. b. particular type of construct specification. c. consistency of the measurement. d. comprehensiveness with which it measures the construct.

4. Which of the following statements most likely involves using data in a way that is valid? a. University data indicate that the highest paid professors have the most publications. b. Students who can type more than 80 words per minute will receive a typing award. c. All of the teachers interviewed described the principal as "outstanding.” d. The English teacher does not assign students a grade less than "B.” 5. The extent to which we can generalize the results of a study to

• ther participants is called:

a. external validity b. internal validity

6. When you are confident that the experimental manipulation produced the changes you measured in the dependent variable, your study probably has good ________ validity. a. internal b. external 7. A measure has high internal consistency reliability when: a. participants score at the high end of the scale every time they complete the measure. b. multiple observers make the same ratings using the measure. c. multiple observers obtain the same score every time they use the measure. d. each of the items correlates with other items on the measure. 8. Double blind experiments increase experimenter bias. a. True b. False

• 9. What is the name of an experimental bias that refers to the

tendency for performance to initially improve when new technology is instituted, not because of any actual improvement in learning or achievement, but in response to increased interest in the new technology?

Novelty effect

• 10. What type of experimental bias does the experiment below suffer

from? One study was performed at a factory to discover if a change in lighting would affect productivity. Productivity did increase but only because of the increased attention due to the study. As soon as the study had ended, productivity decreased again. In this example, the change in productivity had nothing to do with the lighting but rather the existence of the study in the first place.

Hawthorne effect

• 11. Describe advantages and risks of A/B testing

Advantages

• Subjects decisions to data rather than HiPPO (Highest Paid Person’s Opinion)
• Great for optimizing copy, colors, placements, shapes, imagery
• Can even (in theory) be used to optimize some elements automatically (text

colors, link styles

• You can be running hundreds simultaneously

Risks

• Hard to test completely new products, features
• If many many aspects are subject to fluctuation, what is the ‘actual’ site?
• Make sure you are asking the right questions!

• 12. Define Cognitive Walkthrough with three questions to ask

A usability evaluation method in which one or more evaluators work through a series of tasks and ask a set of questions from the perspective of the user. The focus of the cognitive walkthrough is on understanding the system's learnability for new or infrequent users 1. Visibility: Is the control for the action visible to the user? 2. Affordance: Is there a strong link between the control and the action? (Will the user notice that the correct action is available?) 3. Feedback: Is feedback appropriate? (Will the user properly interpret the system response?)

• 13. Describe advantages and risks of Heuristic Evaluation

Advantages

• It’s fast, quick and cheap to conduct a heuristic evaluation. This is

especially true if you are only going to use a single evaluator.

• It provides good insight into possible usability problems that might

damage the user experience. Problems

• A single evaluator may miss issues that are not readily apparent to them.

The use of more than one evaluator is recommended.

• Heuristic analysis is subjective. It does not “prove” anything and thus

findings may be open to debate.

• Experienced evaluators are hard to come by and that means that you may

need to use less skilled evaluators whose findings may not be as valuable.

• 14. Describe a structure of report by explaining what question(s) each

section needs to answer

a. Introduction: “What is the problem?” and “Why should anyone care?”

• b. Methods: “What did I do?”
• c. Results/findings: “What did I find?”
• d. Discussion: “What does all of this mean?” and “Why does it matter?”
• e. Conclusion: a restatement of your work