SLIDE 1 www.rti.org
RTI International is a registered trademark and a trade name of Research Triangle Institute.
Improving Data Quality in the Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS)
Peter Einaudi, RTI International Jonathan Gordon, RTI International Stephanie Eckman, RTI International Herschel Sanders, RTI International Mike Yamaner, National Science Foundation
SLIDE 2 What is the GSS?
- Survey of Graduate Students
and Postdoctorates in Science and Engineering (GSS)
– Sponsored by the National Center
for Science and Engineering Statistics (NCSES) within the National Science Foundation (NSF) and the National Institutes
- f Health (NIH)
- Establishment survey of all
U.S. academic institutions that grant graduate degrees in science, engineering, and health (SEH) fields
- Institutions report enrollment
and financial support data for graduate students, postdoctoral researchers, and doctorate- holding nonfaculty researchers (NFRs)
SLIDE 3 Sample data grid
- Voluntary survey
- High response rates
– 98.6% RR3 in 2016
– 56.4 hours average
Interchange (EDI) limited prior to 2016
SLIDE 4 Key Challenges
- Improving the value of the data
– Users want to analyze master’s and
doctoral data separately
- Aligning taxonomy of disciplines
across federal surveys
- Ensuring all eligible units are reported
and classified appropriately
- Mitigating response burden and
maintaining high response rates
SLIDE 5 Goals of Redesign
Collect more data Reduce Burden Improve Quality
- Separate master’s and doctoral
enrollment data
- Increase use of EDI
- Simplify coding of academic units
- Eliminate data entry errors
- Align request with administrative data
- Improve coding of disciplines
SLIDE 6 Proposed Solutions
- Simplify and expand use of EDI
– Align data request with
respondents’ administrative data systems
– Reduce reliance on manual
data entry
– Facilitate separate master’s
and doctoral reporting without added burden
Classification of Instructional Programs (CIP)
– Already utilized and maintained
by institutions for mandatory reporting
– Improved comparability with
– More granular coding scheme – Shifts burden of taxonomy
changes from data reporters to data collectors
SLIDE 7
Fall 2015– Summer 2016
Institutional site visits Recordkeeping and data reporting practices Coordinator survey Understand impact of changes
Fall 2016– Summer 2017
Pilot Feasibility Estimate reporting burden
Fall 2017– Summer 2018
Full implementation
SLIDE 8
Fall 2015– Summer 2016
Institutional site visits Recordkeeping and data reporting practices Coordinator survey Understand impact of changes
Spring 2016– Summer 2017
Pilot Feasibility Estimate reporting burden
Fall 2017– Summer 2018
Full implementation
SLIDE 9 What we learned
- Coordinators can distinguish
between students enrolled in master’s and doctoral programs in their records
student information systems
– Codes less available to
report postdocs
- Most coordinators willing to
use EDI
– Concerns about programming
effort required
– Privacy concerns about
transmitting individual-level data
SLIDE 10 Integrating lessons learned into the Redesign
- Flexibility in field coding method
– CIP codes strongly encouraged for
student data
– GSS or CIP codes for
postdoctoral data
- Make EDI as simple as possible
– Excel templates – Eliminate additional programming
steps to format data
- Allay concerns about privacy
– Avoid Personally Identifiable
Information
– Option to aggregate data
before transmission
SLIDE 11
Fall 2015– Summer 2016
Institutional site visits Recordkeeping and data reporting practices Coordinator survey Understand impact of changes
Spring 2016– Summer 2017
Pilot Feasibility Estimate reporting burden
Fall 2017– Summer 2018
Full implementation
SLIDE 12 Trial Run: Pilot Data Collection GSS 2016
survey cycle
– Stratified random sample of
coordinators
- Master’s-only institutions
(n = 15)
- Doctorate-granting institutions
with 15 or fewer units (n = 25)
- Doctorate-granting institutions
with over 15 units (n = 40)
– Data request included:
- Separate reporting of master’s
and doctoral students
- Use of CIP codes to report
student data
- Use of EDI to transmit data
SLIDE 13 Pilot Results
able to upload at least some data
master’s and doctoral students able to report these data separately
reported student data with CIP codes
- Most coordinators reported
similar or lower response burden compared to the previous year
SLIDE 14
Average response burden for pilot coordinators
SLIDE 15
Fall 2015– Summer 2016
Institutional site visits Recordkeeping and data reporting practices Coordinator survey Understand impact of changes
Spring 2016– Summer 2017
Pilot Feasibility Estimate reporting burden
Fall 2017– Summer 2018
Full implementation
SLIDE 16 Preparation for full implementation
– Mail – E-mail – Conference presentations
- Redesign of Survey Website
– Highlight changes – Taxonomy tool
– Webinars – Training videos – Sandbox
SLIDE 17
2017 Response method
SLIDE 18
Average response burden
SLIDE 19 Impact on response
but more data
– Increase in school-level
nonresponse
- Item nonresponse declined
– Despite increase in items
with EDI
SLIDE 20
Item nonresponse rates by section and response method
SLIDE 21 Discussion
respondents and their data systems
constraints: one size does not fit all
- Pilot your changes
- Communicate early
and often
SLIDE 22
Thank You
Peter Einaudi Jonathan Gordon Mike Yamaner peinaudi@rti.org jgordon@rti.org myamaner@nsf.gov
