More than preservation: Creating motivational designs and tailored - - PowerPoint PPT Presentation

More than preservation: Creating motivational designs and tailored incentives in research data repositories

Sebastian S. Feger, Sünje Dallmeier-Tiessen, Pamfilos Fokianos, Dinos Kousidis, Artemis Lavasa, Rokas Maciulaitis, Jan Okraska, Diego Rodriguez Rodriguez, Tibor Šimko, Anna Trzcinska, Ioannis Tsanaktsidis, Stephanie van de Sandt

About the Presenter

3rd year PhD Student Human-Computer-Interaction (HCI) Focus on designing technology interaction that supports reproducible research practices

2

Motivation

Feilden, Tom. "Most scientists ‘can’t replicate studies by their peers.’." BBC News (2017).

Baker, Monya. "1,500 scientists lift the lid on reproducibility.’." Nature (2016).

Research Preservation and Reproducibility

Essential to:

• Verify findings
• Compare results
• Enable future reuse

Demands:

• Accessibility
• Rich documentation
• Sharing

Hindered by:

• Required effort
• Missing rewards

5

How can we design technology that supports research preservation and sharing?

6

Research Context

Data-intensive scientific domain CERN is a key High Energy Physics (HEP) laboratory Some of the largest, distributed scientific collaborations Inventive and adaptive to new technologies (e.g. The World Wide Web)

7

Preservation Requirements

Analysis resources with several Gigabytes / Terabytes Multiple types of information and files

8

Datasets Code Meta-Data Environments Workflows Results

x86

CERN Analysis Preservation (CAP)

Community-tailored service Eases

Excerpt from the CMS preservation template on CERN Analysis Preservation

Drafts are visible only to author(s) Published analyses are accessible in the collaboration

○ Documentation ○ Preservation ○ Sharing

How can we design technology that supports research preservation and sharing?

10

Research Activities

11

Reviewed HEP Practices Interview Study Workshop Observation Gamification Prototypes

Research Activities

12

Reviewed HEP Practices Interview Study Workshop Observation Gamification Prototypes

Research Activities

13

Reviewed HEP Practices Interview Study Workshop Observation Gamification Prototypes

Feger et al, CHI 2019: Designing for Reproducibility: A Qualitative Study of Challenges and Opportunities in High Energy Physics

How can preservation technology create meaningful incentives to contribute?

14

Study

12 Semi-structured interviews with physicists Diverse experience levels Mean interview duration: 46 minutes (SD: 7.6) Key interview protocol parts

○ Current sharing and reuse practices ○ Repository exploration ○ Search needs

15

Motivation & Benefits

“People may want to use information - but we need to get them to contribute information as well.” - P1, Postdoc “It's more motivating to start a new analysis, other than spending time encoding things…” - P9, Convener “ … if there is a way of getting an extra benefit out of this, while doing your proper preservation, that is good - that would totally work.” - P11, Convener

16

Design Implications

Motivate and incentivize contributions Exploit platforms’ secondary usage forms

○ Support coping with uncertainty ○ Provide collaboration-stimulating mechanisms ○ Support structured designs

17

Sebastian S. Feger, Sünje Dallmeier-Tiessen, Albrecht Schmidt, Paweł W. Woźniak

Interview Study Conclusion

Tailored preservation technology can lower efforts Balance effort required with benefit offered Design of tailored services should consider tailored incentives Secondary usage forms

○ can provide meaningful benefits to contributors ○ transform researchers’ perceptions of preservation services

Research Activities

19

Reviewed HEP Practices Interview Study Workshop Observation Gamification Prototypes

Research Activities

20

Reviewed HEP Practices Interview Study Workshop Observation Gamification Prototypes

Feger et al, CHI 2019: Gamification in Science: A Study of Requirements in the Context of Reproducible Research

21

Open Science Badges (Center for Open Science) impact practices (Kidwell et al, 2016)

Can gamification motivate scientists to document, preserve and share their research?

22

Simple Game Elements Design (SGED)

Simple Game Elements Design (SGED)

Rational-Informative Design (RID)

Rational-Informative Design (RID)

Evaluation

10 within-subjects evaluation sessions Diverse experience levels Participants

• explored prototypes and commented (5.2

hours recorded)

• responded to questionnaires evaluating

the prototypes

27

Results

28

“We are so many people in the collaborations, of course. Especially

if we want to continue in our field, we have to get some visibility somehow.” – P6, PhD Student

“And what I think what we largely want to do is get signals to people of which ones are like doing it best ‘practice-y’ way and which ones aren't.” – P8, Professor

29

Gamification in Science

Findings & Implications

Scientific Challenges Breaking Points Applications Beyond Achievements

30

31

Reviewed HEP Practices Interview Study Workshop Observation Gamification Prototypes

• Secondary usage forms are expected to motivate contributions
• Gamification can stimulate contributions
• Researcher-centered design

○ enables meaningful new uses for researchers that are not

part of traditional repository design thinking

○ reshapes user perception of repositories

32

Conclusion

CERN Analysis Preservation cernanalysispreservation @analysispreserv