Sometimes There are Dumb Questions Garbage in-Garbage out: Why most - - PowerPoint PPT Presentation
Sometimes There are Dumb Questions Garbage in-Garbage out: Why most surveys are worse than useless (except for yours of course ) Scott McIntyre, PhD Engineering & Experimental Psychologist Project Lifecycle 1. Identify Customer Needs
Sometimes There are Dumb Questions
Garbage in-Garbage out: Why most surveys are worse than useless
(except for yours of course )
Scott McIntyre, PhD Engineering & Experimental Psychologist
Project Lifecycle
Two concerns of this presentation
BEST proposals collect objectively measurable pre & post data for accurate statistical evaluation of change
How do we GET IT? (research design) How will we USE IT? (statistical analysis)
SURVEYS!!
NO OFFENSE! EVERYONE uses poor surveys
Except you
Most common design to obtaining data? And maybe “required” in public outreach
Scientists know what they are doing…right? Psychology-DSM
“should be used in research and evaluation as potentially useful tools to enhance clinical decision-making and not as the sole basis for making a clinical diagnosis.”
Survey Problems
– Nominal, ordinal, ratio
SCALE OF DATA: DEMO
Do you like to cycle? Nominal/Categorical On a scale from 1-5, (1 being low and 5 being high) how much do you like to cycle? Ordinal/Rank How many minutes in the last month did you cycle? Ratio
Data analysis for project evaluation
Danger!!
Statistical Precision cannot make bad data good Poor quality no matter how you slice it
Bad data Statistical analysis Misleading data
Data collection for project evaluation
– Collapsing/equating/combining behavior very misleading—loss of information! – Two people, one who rides once/year and one who rides for 3 hours everyday are equal with nominal scale
2. Ordinal/ranking data (1-5, 1-100): “How likely are you to…?”
something 3 or 4 vs 4 & 5 for SAME person is unknown let alone DIFFERENT people
Data collection for project evaluation
Rank of finish in race not as sensitive as time!
Data analysis for project evaluation
Nominal (Y/N) and Ordinal (1-5) data:
subjective data and/or collapsed data + sensitive statistical analysis =
MOST COMMON TYPES OF SURVEY QUESTIONS!!
_____ MPH (miles per hour) Not posted
A) If you rode on the sidewalk, were the sidewalks comfortable and safe? Circle One: Always Sometimes Never Why rely on human memory or attention? Science says both are faulty. Define “Sometimes comfortable” or safe? What do you do with that data?
Words or numbers, either way a problem
Science
Imagine physicist wants to know “how fast is light?” Survey:
1-Not very 3-quite a bit 7-super fast
Data analysis for project evaluation
For meaningful statistical analysis what you want is objectively measurable data (ratio)— time, distance, weight, speed, behaviors/unit of time
eg. Instead of “Do you ride a bike?” Try: How many minutes/hours per week/month/year do you ride your bike?
Statistical Analysis to Evaluate Change
Ratio data allows for sensitive comparison across time and between groups
15 60 minutes per month Flagstaff Kingman Lake Havasu Phoenix
Professional survey consultants
“When you are asking a number of questions based on a similar rating scale, …” make sure it’s an objective, ratio scale?? WHAT?...NO IDIOT! “be sure that the answer rating – whether it’s 1 to 5 or 0 to 10 – flows consistently …”
Name of firm covered to protect the guilty
“The expert analyses and recommendations we deliver serve as a blueprint for driving
customer experience”
Endorsement
When was the last time you surveyed your pet? How know if … Hungry? Potty? Walk? Scratched? Despite that, are you meeting the needs of your pet? Survey?
Should you survey at all?
Observe behavior
If for some reason you need to know if sidewalks are “comfortable” and by that you mean are they wide enough and are without major elevation changes…go measure them.
If you rode on the sidewalk, were the sidewalks comfortable and safe? Circle One: Always Sometimes Never
Survey Problems
– Nominal, ordinal, ratio
Surveying Public Should Augment Your Expertise not Check a Box
Expertise
behavior, research design, data collection and statistical analysis
likely not experts in any of those.
highly correlated, r = 0.3, p > .05.
Don’t Follow the Herd Be Better!
Summary
What do funders and you ultimately want to know?
Can’t know that objectively without good data for sensible statistical analysis Get objectively measurable data!
“experts” If survey you MUST … survey well Find your Denise!?
Student project
Assessing problem behavior in classroom
Current inventory: subjective, tracking behavior problematic
Student redesign: quantitative, electronic, prefilled, auto calculating
QUESTIONS???