A Universal Identifier for Computational Results Matan Gavish and - - PowerPoint PPT Presentation

▶
a universal identifier for computational results
SMART_READER_LITE
LIVE PREVIEW

A Universal Identifier for Computational Results Matan Gavish and - - PowerPoint PPT Presentation

Dec 19, 2023 109 likes •1.75k views

A Universal Identifier for Computational Results Matan Gavish and David Donoho Statistics Department, Stanford University gavish@stanford.edu July 14, 2011 (gavish@stanford.edu) VCR July 14, 2011 1 / 46 35:00 (gavish@stanford.edu) VCR

slide-1
SLIDE 1

A Universal Identifier for Computational Results

Matan Gavish and David Donoho Statistics Department, Stanford University

gavish@stanford.edu

July 14, 2011

(gavish@stanford.edu) VCR July 14, 2011 1 / 46

slide-2
SLIDE 2

35:00

(gavish@stanford.edu) VCR July 14, 2011 2 / 46

slide-3
SLIDE 3
  • Prof. David Donoho

(gavish@stanford.edu) VCR July 14, 2011 3 / 46

slide-4
SLIDE 4

(gavish@stanford.edu) VCR July 14, 2011 4 / 46

slide-5
SLIDE 5

Credibility Crisis in Scientific Communication

Jon Claerbout, 1990 (paraphrase Donoho and Buckheit, 1995) An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.

(gavish@stanford.edu) VCR July 14, 2011 5 / 46

slide-6
SLIDE 6

Advances 1995 – 2011 ?

Only RR fans attempt follow RR practices No-one expects or enforces reproducibility No uniform standards of reproducibility → no established user base “Most published scientific research is false” . (John Ioannidis, with apologies) “Most published scientific research is not reproducible” . (John Ioannidis et al., Keith Baggerly et al.)

(gavish@stanford.edu) VCR July 14, 2011 6 / 46

slide-7
SLIDE 7

Advances 1995 – 2011 ?

Only RR fans attempt follow RR practices No-one expects or enforces reproducibility No uniform standards of reproducibility → no established user base “Most published scientific research is false” . (John Ioannidis, with apologies) “Most published scientific research is not reproducible” . (John Ioannidis et al., Keith Baggerly et al.)

(gavish@stanford.edu) VCR July 14, 2011 6 / 46

slide-8
SLIDE 8

Advances 1995 – 2011 ?

Only RR fans attempt follow RR practices No-one expects or enforces reproducibility No uniform standards of reproducibility → no established user base “Most published scientific research is false” . (John Ioannidis, with apologies) “Most published scientific research is not reproducible” . (John Ioannidis et al., Keith Baggerly et al.)

(gavish@stanford.edu) VCR July 14, 2011 6 / 46

slide-9
SLIDE 9

Advances 1995 – 2011 ?

Only RR fans attempt follow RR practices No-one expects or enforces reproducibility No uniform standards of reproducibility → no established user base “Most published scientific research is false” . (John Ioannidis, with apologies) “Most published scientific research is not reproducible” . (John Ioannidis et al., Keith Baggerly et al.)

(gavish@stanford.edu) VCR July 14, 2011 6 / 46

slide-10
SLIDE 10

Advances 1995 – 2011 ?

Only RR fans attempt follow RR practices No-one expects or enforces reproducibility No uniform standards of reproducibility → no established user base “Most published scientific research is false” . (John Ioannidis, with apologies) “Most published scientific research is not reproducible” . (John Ioannidis et al., Keith Baggerly et al.)

(gavish@stanford.edu) VCR July 14, 2011 6 / 46

slide-11
SLIDE 11

Advances 1995 – 2011 ?

Only RR fans attempt follow RR practices No-one expects or enforces reproducibility No uniform standards of reproducibility → no established user base “Most published scientific research is false” . (John Ioannidis, with apologies) “Most published scientific research is not reproducible” . (John Ioannidis et al., Keith Baggerly et al.)

(gavish@stanford.edu) VCR July 14, 2011 6 / 46

slide-12
SLIDE 12

Why?

Lesson learned: For RR solution to become standard, it must be – Technologically and legally realistic Easy for everyone to adopt today Offer real benefits for everyone Everyone: researchers, authors, readers, publishers This talk is about an RR solution that was designed to become standard. It’s about the scientists not in this room.

(gavish@stanford.edu) VCR July 14, 2011 7 / 46

slide-13
SLIDE 13

Why?

Lesson learned: For RR solution to become standard, it must be – Technologically and legally realistic Easy for everyone to adopt today Offer real benefits for everyone Everyone: researchers, authors, readers, publishers This talk is about an RR solution that was designed to become standard. It’s about the scientists not in this room.

(gavish@stanford.edu) VCR July 14, 2011 7 / 46

slide-14
SLIDE 14

Why?

Lesson learned: For RR solution to become standard, it must be – Technologically and legally realistic Easy for everyone to adopt today Offer real benefits for everyone Everyone: researchers, authors, readers, publishers This talk is about an RR solution that was designed to become standard. It’s about the scientists not in this room.

(gavish@stanford.edu) VCR July 14, 2011 7 / 46

slide-15
SLIDE 15

Why?

Lesson learned: For RR solution to become standard, it must be – Technologically and legally realistic Easy for everyone to adopt today Offer real benefits for everyone Everyone: researchers, authors, readers, publishers This talk is about an RR solution that was designed to become standard. It’s about the scientists not in this room.

(gavish@stanford.edu) VCR July 14, 2011 7 / 46

slide-16
SLIDE 16

Why?

Lesson learned: For RR solution to become standard, it must be – Technologically and legally realistic Easy for everyone to adopt today Offer real benefits for everyone Everyone: researchers, authors, readers, publishers This talk is about an RR solution that was designed to become standard. It’s about the scientists not in this room.

(gavish@stanford.edu) VCR July 14, 2011 7 / 46

slide-17
SLIDE 17

Why?

Lesson learned: For RR solution to become standard, it must be – Technologically and legally realistic Easy for everyone to adopt today Offer real benefits for everyone Everyone: researchers, authors, readers, publishers This talk is about an RR solution that was designed to become standard. It’s about the scientists not in this room.

(gavish@stanford.edu) VCR July 14, 2011 7 / 46

slide-18
SLIDE 18

(gavish@stanford.edu) VCR July 14, 2011 8 / 46

slide-19
SLIDE 19

(gavish@stanford.edu) VCR July 14, 2011 8 / 46

slide-20
SLIDE 20

(gavish@stanford.edu) VCR July 14, 2011 8 / 46

slide-21
SLIDE 21

(gavish@stanford.edu) VCR July 14, 2011 8 / 46

slide-22
SLIDE 22

(gavish@stanford.edu) VCR July 14, 2011 8 / 46

slide-23
SLIDE 23

(gavish@stanford.edu) VCR July 14, 2011 8 / 46

slide-24
SLIDE 24

(gavish@stanford.edu) VCR July 14, 2011 8 / 46

slide-25
SLIDE 25

(gavish@stanford.edu) VCR July 14, 2011 8 / 46

slide-26
SLIDE 26

(gavish@stanford.edu) VCR July 14, 2011 8 / 46

slide-27
SLIDE 27

(gavish@stanford.edu) VCR July 14, 2011 9 / 46

slide-28
SLIDE 28

(gavish@stanford.edu) VCR July 14, 2011 10 / 46

slide-29
SLIDE 29

VRI’s Everywhere!

(gavish@stanford.edu) VCR July 14, 2011 11 / 46

slide-30
SLIDE 30

Reproducibility standards in experimental sciences

A scientific experiment has:

Initial conditions Specified steps Observed results

Lab journal: Reproducility standard for a scientific experiment

Detailed account of initial conditions and steps leading to published results

(gavish@stanford.edu) VCR July 14, 2011 12 / 46

slide-31
SLIDE 31

Reproducibility standards in experimental sciences

A scientific experiment has:

Initial conditions Specified steps Observed results

Lab journal: Reproducility standard for a scientific experiment

Detailed account of initial conditions and steps leading to published results

(gavish@stanford.edu) VCR July 14, 2011 12 / 46

slide-32
SLIDE 32

Reproducibility standards in experimental sciences

A scientific experiment has:

Initial conditions Specified steps Observed results

Lab journal: Reproducility standard for a scientific experiment

Detailed account of initial conditions and steps leading to published results

(gavish@stanford.edu) VCR July 14, 2011 12 / 46

slide-33
SLIDE 33

Reproducibility standards in experimental sciences

A scientific experiment has:

Initial conditions Specified steps Observed results

Lab journal: Reproducility standard for a scientific experiment

Detailed account of initial conditions and steps leading to published results

(gavish@stanford.edu) VCR July 14, 2011 12 / 46

slide-34
SLIDE 34

Reproducibility standards in experimental sciences

A scientific experiment has:

Initial conditions Specified steps Observed results

Lab journal: Reproducility standard for a scientific experiment

Detailed account of initial conditions and steps leading to published results

(gavish@stanford.edu) VCR July 14, 2011 12 / 46

slide-35
SLIDE 35

Reproducibility standards in experimental sciences

A scientific experiment has:

Initial conditions Specified steps Observed results

Lab journal: Reproducility standard for a scientific experiment

Detailed account of initial conditions and steps leading to published results

(gavish@stanford.edu) VCR July 14, 2011 12 / 46

slide-36
SLIDE 36

RR efforts since 1990: Re-execute computation

Not { realistic, easy to adopt, real benefits } Ignores the scientific tradition of chronicling experiments Ignores the hard truth: for acadmics, publication is the endgame The all-importnat publication is informal description of work we do in private Reproducibility involves cleaning up post-facto. It’s a pretentious waste of time. (–DLD, 2010)

Alternative approach: Verifiable Computational Results

Automatically record, validate run-time steps leading to publishable results

(gavish@stanford.edu) VCR July 14, 2011 13 / 46

slide-37
SLIDE 37

RR efforts since 1990: Re-execute computation

Not { realistic, easy to adopt, real benefits } Ignores the scientific tradition of chronicling experiments Ignores the hard truth: for acadmics, publication is the endgame The all-importnat publication is informal description of work we do in private Reproducibility involves cleaning up post-facto. It’s a pretentious waste of time. (–DLD, 2010)

Alternative approach: Verifiable Computational Results

Automatically record, validate run-time steps leading to publishable results

(gavish@stanford.edu) VCR July 14, 2011 13 / 46

slide-38
SLIDE 38

RR efforts since 1990: Re-execute computation

Not { realistic, easy to adopt, real benefits } Ignores the scientific tradition of chronicling experiments Ignores the hard truth: for acadmics, publication is the endgame The all-importnat publication is informal description of work we do in private Reproducibility involves cleaning up post-facto. It’s a pretentious waste of time. (–DLD, 2010)

Alternative approach: Verifiable Computational Results

Automatically record, validate run-time steps leading to publishable results

(gavish@stanford.edu) VCR July 14, 2011 13 / 46

slide-39
SLIDE 39

RR efforts since 1990: Re-execute computation

Not { realistic, easy to adopt, real benefits } Ignores the scientific tradition of chronicling experiments Ignores the hard truth: for acadmics, publication is the endgame The all-importnat publication is informal description of work we do in private Reproducibility involves cleaning up post-facto. It’s a pretentious waste of time. (–DLD, 2010)

Alternative approach: Verifiable Computational Results

Automatically record, validate run-time steps leading to publishable results

(gavish@stanford.edu) VCR July 14, 2011 13 / 46

slide-40
SLIDE 40

RR efforts since 1990: Re-execute computation

Not { realistic, easy to adopt, real benefits } Ignores the scientific tradition of chronicling experiments Ignores the hard truth: for acadmics, publication is the endgame The all-importnat publication is informal description of work we do in private Reproducibility involves cleaning up post-facto. It’s a pretentious waste of time. (–DLD, 2010)

Alternative approach: Verifiable Computational Results

Automatically record, validate run-time steps leading to publishable results

(gavish@stanford.edu) VCR July 14, 2011 13 / 46

slide-41
SLIDE 41

RR efforts since 1990: Re-execute computation

Not { realistic, easy to adopt, real benefits } Ignores the scientific tradition of chronicling experiments Ignores the hard truth: for acadmics, publication is the endgame The all-importnat publication is informal description of work we do in private Reproducibility involves cleaning up post-facto. It’s a pretentious waste of time. (–DLD, 2010)

Alternative approach: Verifiable Computational Results

Automatically record, validate run-time steps leading to publishable results

(gavish@stanford.edu) VCR July 14, 2011 13 / 46

slide-42
SLIDE 42

RR efforts since 1990: Re-execute computation

Not { realistic, easy to adopt, real benefits } Ignores the scientific tradition of chronicling experiments Ignores the hard truth: for acadmics, publication is the endgame The all-importnat publication is informal description of work we do in private Reproducibility involves cleaning up post-facto. It’s a pretentious waste of time. (–DLD, 2010)

Alternative approach: Verifiable Computational Results

Automatically record, validate run-time steps leading to publishable results

(gavish@stanford.edu) VCR July 14, 2011 13 / 46

slide-43
SLIDE 43

(gavish@stanford.edu) VCR July 14, 2011 14 / 46

slide-44
SLIDE 44

(gavish@stanford.edu) VCR July 14, 2011 14 / 46

slide-45
SLIDE 45

(gavish@stanford.edu) VCR July 14, 2011 14 / 46

slide-46
SLIDE 46

(gavish@stanford.edu) VCR July 14, 2011 14 / 46

slide-47
SLIDE 47

(gavish@stanford.edu) VCR July 14, 2011 14 / 46

slide-48
SLIDE 48

(gavish@stanford.edu) VCR July 14, 2011 14 / 46

slide-49
SLIDE 49

(gavish@stanford.edu) VCR July 14, 2011 14 / 46

slide-50
SLIDE 50

VCR record of a computational experiment

Includes –

Code Original data (standard format) Parameters used in run-time Publishable & intermediate results (standard format) Dependencies VCR records are permanent and citable. Peers reference it in articles, computer programs, databases.

(gavish@stanford.edu) VCR July 14, 2011 15 / 46

slide-51
SLIDE 51

VCR record of a computational experiment

Includes –

Code Original data (standard format) Parameters used in run-time Publishable & intermediate results (standard format) Dependencies VCR records are permanent and citable. Peers reference it in articles, computer programs, databases.

(gavish@stanford.edu) VCR July 14, 2011 15 / 46

slide-52
SLIDE 52

VCR record of a computational experiment

Includes –

Code Original data (standard format) Parameters used in run-time Publishable & intermediate results (standard format) Dependencies VCR records are permanent and citable. Peers reference it in articles, computer programs, databases.

(gavish@stanford.edu) VCR July 14, 2011 15 / 46

slide-53
SLIDE 53

VCR record of a computational experiment

Includes –

Code Original data (standard format) Parameters used in run-time Publishable & intermediate results (standard format) Dependencies VCR records are permanent and citable. Peers reference it in articles, computer programs, databases.

(gavish@stanford.edu) VCR July 14, 2011 15 / 46

slide-54
SLIDE 54

VCR record of a computational experiment

Includes –

Code Original data (standard format) Parameters used in run-time Publishable & intermediate results (standard format) Dependencies VCR records are permanent and citable. Peers reference it in articles, computer programs, databases.

(gavish@stanford.edu) VCR July 14, 2011 15 / 46

slide-55
SLIDE 55

VCR record of a computational experiment

Includes –

Code Original data (standard format) Parameters used in run-time Publishable & intermediate results (standard format) Dependencies VCR records are permanent and citable. Peers reference it in articles, computer programs, databases.

(gavish@stanford.edu) VCR July 14, 2011 15 / 46

slide-56
SLIDE 56

VCR record of a computational experiment

Includes –

Code Original data (standard format) Parameters used in run-time Publishable & intermediate results (standard format) Dependencies VCR records are permanent and citable. Peers reference it in articles, computer programs, databases.

(gavish@stanford.edu) VCR July 14, 2011 15 / 46

slide-57
SLIDE 57

VCR Foundation

Record and transmit, in run-time, computation steps leading to results Automatically validate and store on publisher repository server Universal identifier connects it to article figure Anyone can develop applications that mine knowledge in chronicles

(gavish@stanford.edu) VCR July 14, 2011 16 / 46

slide-58
SLIDE 58

VCR Foundation

Record and transmit, in run-time, computation steps leading to results Automatically validate and store on publisher repository server Universal identifier connects it to article figure Anyone can develop applications that mine knowledge in chronicles

(gavish@stanford.edu) VCR July 14, 2011 16 / 46

slide-59
SLIDE 59

VCR Foundation

Record and transmit, in run-time, computation steps leading to results Automatically validate and store on publisher repository server Universal identifier connects it to article figure Anyone can develop applications that mine knowledge in chronicles

(gavish@stanford.edu) VCR July 14, 2011 16 / 46

slide-60
SLIDE 60

VCR Foundation

Record and transmit, in run-time, computation steps leading to results Automatically validate and store on publisher repository server Universal identifier connects it to article figure Anyone can develop applications that mine knowledge in chronicles

(gavish@stanford.edu) VCR July 14, 2011 16 / 46

slide-61
SLIDE 61

VCR Foundation

Record and transmit, in run-time, computation steps leading to results Automatically validate and store on publisher repository server Universal identifier connects it to article figure Anyone can develop applications that mine knowledge in chronicles

(gavish@stanford.edu) VCR July 14, 2011 16 / 46

slide-62
SLIDE 62

Outline

1 The VCR Foundation and API 2 Chronicing computations 3 Basic VCR apps (gavish@stanford.edu) VCR July 14, 2011 17 / 46

slide-63
SLIDE 63

Outline

1 The VCR Foundation and API 2 Chronicing computations 3 Basic VCR apps (gavish@stanford.edu) VCR July 14, 2011 17 / 46

slide-64
SLIDE 64

Outline

1 The VCR Foundation and API 2 Chronicing computations 3 Basic VCR apps (gavish@stanford.edu) VCR July 14, 2011 17 / 46

slide-65
SLIDE 65

Outline

1 The VCR Foundation and API 2 Chronicing computations 3 Basic VCR apps (gavish@stanford.edu) VCR July 14, 2011 17 / 46

slide-66
SLIDE 66

VCR Foundation system

(e.g vcr-stat.stanford.edu)

VCR repository server

Verifiable Result Identifier (VRI)

Addressing standard: URL, secure digital signature and QR barcode vcr.cell.com/ffaaffb148d7

VCR Application Programmer Interface (API)

RESTful interface for depositing, searching and retrieving content on repositories GET https://vcr.cell.com/ffaaffb148d7/graphics?format=eps

(gavish@stanford.edu) VCR July 14, 2011 18 / 46

slide-67
SLIDE 67

VCR Foundation system

(e.g vcr-stat.stanford.edu)

VCR repository server

Verifiable Result Identifier (VRI)

Addressing standard: URL, secure digital signature and QR barcode vcr.cell.com/ffaaffb148d7

VCR Application Programmer Interface (API)

RESTful interface for depositing, searching and retrieving content on repositories GET https://vcr.cell.com/ffaaffb148d7/graphics?format=eps

(gavish@stanford.edu) VCR July 14, 2011 18 / 46

slide-68
SLIDE 68

VCR Foundation system

(e.g vcr-stat.stanford.edu)

VCR repository server

Verifiable Result Identifier (VRI)

Addressing standard: URL, secure digital signature and QR barcode vcr.cell.com/ffaaffb148d7

VCR Application Programmer Interface (API)

RESTful interface for depositing, searching and retrieving content on repositories GET https://vcr.cell.com/ffaaffb148d7/graphics?format=eps

(gavish@stanford.edu) VCR July 14, 2011 18 / 46

slide-69
SLIDE 69

VCR Foundation system

(e.g vcr-stat.stanford.edu)

VCR repository server

Verifiable Result Identifier (VRI)

Addressing standard: URL, secure digital signature and QR barcode vcr.cell.com/ffaaffb148d7

VCR Application Programmer Interface (API)

RESTful interface for depositing, searching and retrieving content on repositories GET https://vcr.cell.com/ffaaffb148d7/graphics?format=eps

(gavish@stanford.edu) VCR July 14, 2011 18 / 46

slide-70
SLIDE 70

Outline

1 The VCR Foundation and API 2 Chronicing computations 3 Basic VCR apps (gavish@stanford.edu) VCR July 14, 2011 19 / 46

slide-71
SLIDE 71

Outline

1 The VCR Foundation and API 2 Chronicing computations 3 Basic VCR apps (gavish@stanford.edu) VCR July 14, 2011 19 / 46

slide-72
SLIDE 72

Chronicing computations in real-time

VCR computation platform Plugin = Computation recorder

Regular program code

figure1 = plot(x) save(figure1,’figure1.eps’) > file /home/figure1.eps saved >

(gavish@stanford.edu) VCR July 14, 2011 20 / 46

slide-73
SLIDE 73

Chronicing computations in real-time

VCR computation platform Plugin = Computation recorder

Regular program code

figure1 = plot(x) save(figure1,’figure1.eps’) > file /home/figure1.eps saved >

(gavish@stanford.edu) VCR July 14, 2011 20 / 46

slide-74
SLIDE 74

Chronicing computations in real-time

VCR computation platform Plugin = Computation recorder

Regular program code

figure1 = plot(x) save(figure1,’figure1.eps’) > file /home/figure1.eps saved >

(gavish@stanford.edu) VCR July 14, 2011 20 / 46

slide-75
SLIDE 75

Chronicing computations in real-time

VCR computation platform Plugin = Computation recorder

Regular program code

figure1 = plot(x) save(figure1,’figure1.eps’) > file /home/figure1.eps saved >

(gavish@stanford.edu) VCR July 14, 2011 20 / 46

slide-76
SLIDE 76

Chronicing computations in real-time

VCR computation platform Plugin = Computation recorder

Regular program code

figure1 = plot(x) save(figure1,’figure1.eps’) > file /home/figure1.eps saved >

(gavish@stanford.edu) VCR July 14, 2011 20 / 46

slide-77
SLIDE 77

Chronicing computations in real-time

VCR computation platform Plugin = Computation recorder

Program code with VCR plugin

repository vcr.nature.com verifiable figure1 = plot(x) > vcr.nature.com approved: > access figure1 at https://vcr.nature.com/ffaaffb148d7

(gavish@stanford.edu) VCR July 14, 2011 20 / 46

slide-78
SLIDE 78

Chronicing computations in real-time

VCR computation platform Plugin = Computation recorder

Program code with VCR plugin

repository vcr.nature.com verifiable figure1 = plot(x) > vcr.nature.com approved: > access figure1 at https://vcr.nature.com/ffaaffb148d7

(gavish@stanford.edu) VCR July 14, 2011 20 / 46

slide-79
SLIDE 79

matlab plugin

(gavish@stanford.edu) VCR July 14, 2011 21 / 46

slide-80
SLIDE 80

(gavish@stanford.edu) VCR July 14, 2011 22 / 46

slide-81
SLIDE 81

(gavish@stanford.edu) VCR July 14, 2011 22 / 46

slide-82
SLIDE 82

(gavish@stanford.edu) VCR July 14, 2011 22 / 46

slide-83
SLIDE 83

Converting computation result into content in real-time

Data/code published at run-time, no intervention possible or required Researcher controls granularity, can choose“private VRI” Repository automatically validates, signs computation result content

(gavish@stanford.edu) VCR July 14, 2011 23 / 46

slide-84
SLIDE 84

Converting computation result into content in real-time

Data/code published at run-time, no intervention possible or required Researcher controls granularity, can choose“private VRI” Repository automatically validates, signs computation result content

(gavish@stanford.edu) VCR July 14, 2011 23 / 46

slide-85
SLIDE 85

Converting computation result into content in real-time

Data/code published at run-time, no intervention possible or required Researcher controls granularity, can choose“private VRI” Repository automatically validates, signs computation result content

(gavish@stanford.edu) VCR July 14, 2011 23 / 46

slide-86
SLIDE 86

Converting computation result into content in real-time

Data/code published at run-time, no intervention possible or required Researcher controls granularity, can choose“private VRI” Repository automatically validates, signs computation result content

(gavish@stanford.edu) VCR July 14, 2011 23 / 46

slide-87
SLIDE 87

VCR foundation software

(gavish@stanford.edu) VCR July 14, 2011 24 / 46

slide-88
SLIDE 88

VCR foundation software

(gavish@stanford.edu) VCR July 14, 2011 24 / 46

slide-89
SLIDE 89

VCR foundation software

(gavish@stanford.edu) VCR July 14, 2011 24 / 46

slide-90
SLIDE 90

Outline

1 The VCR Foundation and API 2 Chronicing computations 3 Basic VCR apps (gavish@stanford.edu) VCR July 14, 2011 25 / 46

slide-91
SLIDE 91

Outline

1 The VCR Foundation and API 2 Chronicing computations 3 Basic VCR apps (gavish@stanford.edu) VCR July 14, 2011 25 / 46

slide-92
SLIDE 92

Basic VCR apps

Direct data import Word-processor plugin Content browser Search

(gavish@stanford.edu) VCR July 14, 2011 26 / 46

slide-93
SLIDE 93

Direct data import App

(gavish@stanford.edu) VCR July 14, 2011 27 / 46

slide-94
SLIDE 94

Direct data import App

(gavish@stanford.edu) VCR July 14, 2011 27 / 46

slide-95
SLIDE 95

Direct data import App

(gavish@stanford.edu) VCR July 14, 2011 27 / 46

slide-96
SLIDE 96

Direct data import App

(gavish@stanford.edu) VCR July 14, 2011 27 / 46

slide-97
SLIDE 97

Direct data import App

Program Code

x = load(’genome.csv’) run_experiment(x)

Program Code with VCR plugin

run_experiment([vcr.cell.com/ffaaffb148d7]) A natural evolution of scholarly citation

(gavish@stanford.edu) VCR July 14, 2011 28 / 46

slide-98
SLIDE 98

Direct data import App

Program Code

x = load(’genome.csv’) run_experiment(x)

Program Code with VCR plugin

run_experiment([vcr.cell.com/ffaaffb148d7]) A natural evolution of scholarly citation

(gavish@stanford.edu) VCR July 14, 2011 28 / 46

slide-99
SLIDE 99

Direct data import App

Program Code

x = load(’genome.csv’) run_experiment(x)

Program Code with VCR plugin

run_experiment([vcr.cell.com/ffaaffb148d7]) A natural evolution of scholarly citation

(gavish@stanford.edu) VCR July 14, 2011 28 / 46

slide-100
SLIDE 100

Direct data import App

Program Code

x = load(’genome.csv’) run_experiment(x)

Program Code with VCR plugin

run_experiment([vcr.cell.com/ffaaffb148d7]) A natural evolution of scholarly citation

(gavish@stanford.edu) VCR July 14, 2011 28 / 46

slide-101
SLIDE 101

Direct data import App

(gavish@stanford.edu) VCR July 14, 2011 29 / 46

slide-102
SLIDE 102

Direct data import App

(gavish@stanford.edu) VCR July 14, 2011 29 / 46

slide-103
SLIDE 103

Word-processor plugin App

(gavish@stanford.edu) VCR July 14, 2011 30 / 46

slide-104
SLIDE 104

Word-processor plugin App

(gavish@stanford.edu) VCR July 14, 2011 30 / 46

slide-105
SLIDE 105

Word-processor plugin App

(gavish@stanford.edu) VCR July 14, 2011 30 / 46

slide-106
SLIDE 106

Word-processor plugin App

(gavish@stanford.edu) VCR July 14, 2011 30 / 46

slide-107
SLIDE 107

Word-processor plugin App

(gavish@stanford.edu) VCR July 14, 2011 30 / 46

slide-108
SLIDE 108

Word-processor plugin App

(gavish@stanford.edu) VCR July 14, 2011 30 / 46

slide-109
SLIDE 109

Word-processor plugin App

(gavish@stanford.edu) VCR July 14, 2011 30 / 46

slide-110
SLIDE 110

Word-processor plugin App

LaTeX source

\includegraphics{figure1.eps}

LaTeX source with VCR package

\includeresult{vcr.thelancet.com/ffaaffb148d7} Permanently bind printed graphics to underlying result content

(gavish@stanford.edu) VCR July 14, 2011 31 / 46

slide-111
SLIDE 111

Word-processor plugin App

LaTeX source

\includegraphics{figure1.eps}

LaTeX source with VCR package

\includeresult{vcr.thelancet.com/ffaaffb148d7} Permanently bind printed graphics to underlying result content

(gavish@stanford.edu) VCR July 14, 2011 31 / 46

slide-112
SLIDE 112

Word-processor plugin App

LaTeX source

\includegraphics{figure1.eps}

LaTeX source with VCR package

\includeresult{vcr.thelancet.com/ffaaffb148d7} Permanently bind printed graphics to underlying result content

(gavish@stanford.edu) VCR July 14, 2011 31 / 46

slide-113
SLIDE 113

Word-processor plugin App

LaTeX source

\includegraphics{figure1.eps}

LaTeX source with VCR package

\includeresult{vcr.thelancet.com/ffaaffb148d7} Permanently bind printed graphics to underlying result content

(gavish@stanford.edu) VCR July 14, 2011 31 / 46

slide-114
SLIDE 114

Word-processor plugin App

(gavish@stanford.edu) VCR July 14, 2011 32 / 46

slide-115
SLIDE 115

Word-processor plugin App

(gavish@stanford.edu) VCR July 14, 2011 32 / 46

slide-116
SLIDE 116

(gavish@stanford.edu) VCR July 14, 2011 33 / 46

slide-117
SLIDE 117

(gavish@stanford.edu) VCR July 14, 2011 33 / 46

slide-118
SLIDE 118

(gavish@stanford.edu) VCR July 14, 2011 33 / 46

slide-119
SLIDE 119

(gavish@stanford.edu) VCR July 14, 2011 33 / 46

slide-120
SLIDE 120

latex plugin

(gavish@stanford.edu) VCR July 14, 2011 34 / 46

slide-121
SLIDE 121

VRI’s Everywhere!

(gavish@stanford.edu) VCR July 14, 2011 35 / 46

slide-122
SLIDE 122

Basic VCR apps

Direct data import Word-processor plugin Content browser Search

(gavish@stanford.edu) VCR July 14, 2011 36 / 46

slide-123
SLIDE 123

Basic VCR apps elegantly solve pressing issues in science

Mandatory Data/code sharing policy Credibility concerns in computer science (e.g SIGGRAPH photoshopped results) Knowledge accumulation in research groups

(gavish@stanford.edu) VCR July 14, 2011 37 / 46

slide-124
SLIDE 124

Advanced VCR applications

(gavish@stanford.edu) VCR July 14, 2011 38 / 46

slide-125
SLIDE 125

Advanced VCR applications

(gavish@stanford.edu) VCR July 14, 2011 38 / 46

slide-126
SLIDE 126

Advanced VCR applications

(gavish@stanford.edu) VCR July 14, 2011 38 / 46

slide-127
SLIDE 127

Adopting the VCR foundation means:

For journals –

Print VRI’s next to results, cite data by VRI Operate VCR repository servers

For researchers/authors –

Access all data and results through VRIs, not files Use VCR plugin that creates content and VRIs automatically during computation

For readers –

Click/scan result VRI, browse underlying content

(gavish@stanford.edu) VCR July 14, 2011 39 / 46

slide-128
SLIDE 128

Adopting the VCR foundation means:

For journals –

Print VRI’s next to results, cite data by VRI Operate VCR repository servers

For researchers/authors –

Access all data and results through VRIs, not files Use VCR plugin that creates content and VRIs automatically during computation

For readers –

Click/scan result VRI, browse underlying content

(gavish@stanford.edu) VCR July 14, 2011 39 / 46

slide-129
SLIDE 129

Adopting the VCR foundation means:

For journals –

Print VRI’s next to results, cite data by VRI Operate VCR repository servers

For researchers/authors –

Access all data and results through VRIs, not files Use VCR plugin that creates content and VRIs automatically during computation

For readers –

Click/scan result VRI, browse underlying content

(gavish@stanford.edu) VCR July 14, 2011 39 / 46

slide-130
SLIDE 130

For a solution to become standard, it must be – Technologically and legally realistic Easy for everyone to adopt today Offer real benefits for everyone

(gavish@stanford.edu) VCR July 14, 2011 40 / 46

slide-131
SLIDE 131

Adopting the VCR Foundation is technologically realistic

Technologically

Publisher operates only repository server VCR repository based on secure, scalable Web 2.0 technology Applications do not run on public servers Readers need web browser - same paper/electronic article format

(gavish@stanford.edu) VCR July 14, 2011 41 / 46

slide-132
SLIDE 132

Adopting the VCR Foundation is technologically realistic

Technologically

Publisher operates only repository server VCR repository based on secure, scalable Web 2.0 technology Applications do not run on public servers Readers need web browser - same paper/electronic article format

(gavish@stanford.edu) VCR July 14, 2011 41 / 46

slide-133
SLIDE 133

Adopting the VCR Foundation is technologically realistic

Technologically

Publisher operates only repository server VCR repository based on secure, scalable Web 2.0 technology Applications do not run on public servers Readers need web browser - same paper/electronic article format

(gavish@stanford.edu) VCR July 14, 2011 41 / 46

slide-134
SLIDE 134

Adopting the VCR Foundation is technologically realistic

Technologically

Publisher operates only repository server VCR repository based on secure, scalable Web 2.0 technology Applications do not run on public servers Readers need web browser - same paper/electronic article format

(gavish@stanford.edu) VCR July 14, 2011 41 / 46

slide-135
SLIDE 135

Adopting the VCR Foundation is technologically realistic

Technologically

Publisher operates only repository server VCR repository based on secure, scalable Web 2.0 technology Applications do not run on public servers Readers need web browser - same paper/electronic article format

(gavish@stanford.edu) VCR July 14, 2011 41 / 46

slide-136
SLIDE 136

Adopting the VCR Foundation is technologically realistic

Technologically

Publisher operates only repository server VCR repository based on secure, scalable Web 2.0 technology Applications do not run on public servers Readers need web browser - same paper/electronic article format

(gavish@stanford.edu) VCR July 14, 2011 41 / 46

slide-137
SLIDE 137

The VCR Foundation is easy for everyone to adopt today

For journals

Print URL + barcode: subtle change in print format

For researchers/authors

VCR compatible with most computation platforms Researchers use ordinary computation platform with VCR plugin Authors use ordinary word processor with VCR plugin

For readers

Just click

(gavish@stanford.edu) VCR July 14, 2011 42 / 46

slide-138
SLIDE 138

The VCR Foundation is easy for everyone to adopt today

For journals

Print URL + barcode: subtle change in print format

For researchers/authors

VCR compatible with most computation platforms Researchers use ordinary computation platform with VCR plugin Authors use ordinary word processor with VCR plugin

For readers

Just click

(gavish@stanford.edu) VCR July 14, 2011 42 / 46

slide-139
SLIDE 139

The VCR Foundation is easy for everyone to adopt today

For journals

Print URL + barcode: subtle change in print format

For researchers/authors

VCR compatible with most computation platforms Researchers use ordinary computation platform with VCR plugin Authors use ordinary word processor with VCR plugin

For readers

Just click

(gavish@stanford.edu) VCR July 14, 2011 42 / 46

slide-140
SLIDE 140

The VCR Foundation is easy for everyone to adopt today

For journals

Print URL + barcode: subtle change in print format

For researchers/authors

VCR compatible with most computation platforms Researchers use ordinary computation platform with VCR plugin Authors use ordinary word processor with VCR plugin

For readers

Just click

(gavish@stanford.edu) VCR July 14, 2011 42 / 46

slide-141
SLIDE 141

The VCR Foundation is easy for everyone to adopt today

For journals

Print URL + barcode: subtle change in print format

For researchers/authors

VCR compatible with most computation platforms Researchers use ordinary computation platform with VCR plugin Authors use ordinary word processor with VCR plugin

For readers

Just click

(gavish@stanford.edu) VCR July 14, 2011 42 / 46

slide-142
SLIDE 142

The VCR Foundation is easy for everyone to adopt today

For journals

Print URL + barcode: subtle change in print format

For researchers/authors

VCR compatible with most computation platforms Researchers use ordinary computation platform with VCR plugin Authors use ordinary word processor with VCR plugin

For readers

Just click

(gavish@stanford.edu) VCR July 14, 2011 42 / 46

slide-143
SLIDE 143

The VCR Foundation is easy for everyone to adopt today

For journals

Print URL + barcode: subtle change in print format

For researchers/authors

VCR compatible with most computation platforms Researchers use ordinary computation platform with VCR plugin Authors use ordinary word processor with VCR plugin

For readers

Just click

(gavish@stanford.edu) VCR July 14, 2011 42 / 46

slide-144
SLIDE 144

The VCR foundation does offer real benefits for everyone

For publishers

Curators of computations: A new layer of content Stay in content validation, management and addressing business

For researchers/authors

VCR applications – powerful research and knowledge discovery tools Group VCR repositories – productivity heaven Get explicitly cited for data

For readers

Uncover a rich content layer

(gavish@stanford.edu) VCR July 14, 2011 43 / 46

slide-145
SLIDE 145

The VCR foundation does offer real benefits for everyone

For publishers

Curators of computations: A new layer of content Stay in content validation, management and addressing business

For researchers/authors

VCR applications – powerful research and knowledge discovery tools Group VCR repositories – productivity heaven Get explicitly cited for data

For readers

Uncover a rich content layer

(gavish@stanford.edu) VCR July 14, 2011 43 / 46

slide-146
SLIDE 146

The VCR foundation does offer real benefits for everyone

For publishers

Curators of computations: A new layer of content Stay in content validation, management and addressing business

For researchers/authors

VCR applications – powerful research and knowledge discovery tools Group VCR repositories – productivity heaven Get explicitly cited for data

For readers

Uncover a rich content layer

(gavish@stanford.edu) VCR July 14, 2011 43 / 46

slide-147
SLIDE 147

The VCR foundation does offer real benefits for everyone

For publishers

Curators of computations: A new layer of content Stay in content validation, management and addressing business

For researchers/authors

VCR applications – powerful research and knowledge discovery tools Group VCR repositories – productivity heaven Get explicitly cited for data

For readers

Uncover a rich content layer

(gavish@stanford.edu) VCR July 14, 2011 43 / 46

slide-148
SLIDE 148

The VCR foundation does offer real benefits for everyone

For publishers

Curators of computations: A new layer of content Stay in content validation, management and addressing business

For researchers/authors

VCR applications – powerful research and knowledge discovery tools Group VCR repositories – productivity heaven Get explicitly cited for data

For readers

Uncover a rich content layer

(gavish@stanford.edu) VCR July 14, 2011 43 / 46

slide-149
SLIDE 149

The VCR foundation does offer real benefits for everyone

For publishers

Curators of computations: A new layer of content Stay in content validation, management and addressing business

For researchers/authors

VCR applications – powerful research and knowledge discovery tools Group VCR repositories – productivity heaven Get explicitly cited for data

For readers

Uncover a rich content layer

(gavish@stanford.edu) VCR July 14, 2011 43 / 46

slide-150
SLIDE 150

The VCR foundation does offer real benefits for everyone

For publishers

Curators of computations: A new layer of content Stay in content validation, management and addressing business

For researchers/authors

VCR applications – powerful research and knowledge discovery tools Group VCR repositories – productivity heaven Get explicitly cited for data

For readers

Uncover a rich content layer

(gavish@stanford.edu) VCR July 14, 2011 43 / 46

slide-151
SLIDE 151

The VCR foundation does offer real benefits for everyone

For publishers

Curators of computations: A new layer of content Stay in content validation, management and addressing business

For researchers/authors

VCR applications – powerful research and knowledge discovery tools Group VCR repositories – productivity heaven Get explicitly cited for data

For readers

Uncover a rich content layer

(gavish@stanford.edu) VCR July 14, 2011 43 / 46

slide-152
SLIDE 152

The VCR foundation does offer real benefits for everyone

For publishers

Curators of computations: A new layer of content Stay in content validation, management and addressing business

For researchers/authors

VCR applications – powerful research and knowledge discovery tools Group VCR repositories – productivity heaven Get explicitly cited for data

For readers

Uncover a rich content layer

(gavish@stanford.edu) VCR July 14, 2011 43 / 46

slide-153
SLIDE 153

The VCR foundation does offer real benefits for everyone

For publishers

Curators of computations: A new layer of content Stay in content validation, management and addressing business

For researchers/authors

VCR applications – powerful research and knowledge discovery tools Group VCR repositories – productivity heaven Get explicitly cited for data

For readers

Uncover a rich content layer

(gavish@stanford.edu) VCR July 14, 2011 43 / 46

slide-154
SLIDE 154

(gavish@stanford.edu) VCR July 14, 2011 44 / 46

slide-155
SLIDE 155

(gavish@stanford.edu) VCR July 14, 2011 44 / 46

slide-156
SLIDE 156

(gavish@stanford.edu) VCR July 14, 2011 44 / 46

slide-157
SLIDE 157

(gavish@stanford.edu) VCR July 14, 2011 44 / 46

slide-158
SLIDE 158

(gavish@stanford.edu) VCR July 14, 2011 44 / 46

slide-159
SLIDE 159

(gavish@stanford.edu) VCR July 14, 2011 44 / 46

slide-160
SLIDE 160

Conclusion

16+ years of reproducible research, still no standard solution We propose change of focus: automatic computation chronicling VCR is ready now, shows potential of becoming standard

(gavish@stanford.edu) VCR July 14, 2011 45 / 46

slide-161
SLIDE 161

Conclusion

16+ years of reproducible research, still no standard solution We propose change of focus: automatic computation chronicling VCR is ready now, shows potential of becoming standard

(gavish@stanford.edu) VCR July 14, 2011 45 / 46

slide-162
SLIDE 162

Conclusion

16+ years of reproducible research, still no standard solution We propose change of focus: automatic computation chronicling VCR is ready now, shows potential of becoming standard

(gavish@stanford.edu) VCR July 14, 2011 45 / 46

slide-163
SLIDE 163

Thank you.

Acknowledgments

Balasubramanian Narasimhan (Stanford), Alon Shalita (Facebook)

References

MG & DLD, Universal Identifiers for Computational Results, ICCS MG & DLD, PLoS Biology, to appear http://vcr.stanford.edu

VCR Repositories

Public Stanford Stat repository – http://vcr-stat.stanford.edu/public Your group private repository – https://vcr-stat.stanford.edu/you

(gavish@stanford.edu) VCR July 14, 2011 46 / 46