Development of New Standards for OAIS Interoperability Open - - PowerPoint PPT Presentation

Development of New Standards for OAIS Interoperability

Open Repositories 2019 June 2019

Mike Kearney

KearneySolutions@gmail.com

From the CCSDS DAI WG …and the Google Digital Vellum Project …and the Space Infrastructure Foundation Supporting Co-authors: David Giaretta, PTAB Ltd.; DAI WG co-chair John Garrett, Garrett Software; DAI WG co-chair Steve Hughes, NASA JPL CalTech; DAI WG Architecture Document Book Editor

Intro to CCSDS Consultative Committee for Space Data Systems

Started in 1982 developing data interoperability standards for

space programs.

CCSDS is organized by space agencies, but inclusive of other

(non-space) agencies, industry and academia

About 22 working groups, one of which is the DAI WG Data Archive Interoperability Working Group

 Focused on long-term Digital Preservation Archives  Developed the Open Archival Info System (OAIS) Reference Model  When OAIS became broadly adopted, CCSDS and DAI became even

more inclusive of non-space preservation orgs (libraries, archives)

 CCSDS and DAI standards are procedurally adopted by and published

by ISO (CCSDS = ISO TC20/SC13)

2

OAIS and the next steps

Initially the DAI WG worked on processes for digital preservation.

 That resulted in the OAIS Reference Model first published in 2002  Subsequently set up ISO certification process for trustworthy repositories,

Continued to optimize OAIS Processes in 5-year reviews.

 Result - OAIS is widely accepted and practiced among national archives

and libraries around the world.

Now: DAI is starting work on OAIS-based interoperability

standards for archives

 Google’s Vint Cerf approached us and advocated stronger focus on

technical interoperability for preservation archives.

Purpose of this presentation:

Peer review/feedback on the concept. Invitation to participate in this next phase of work.

3

New Architecture Concept

OAIS Reference Model and Certification Processes will not change

 Enables trustworthiness of important digital preservation archives regardless of

implementations or the interoperability component.

 Referred to here as the OAIS Process Framework (OAIS-PF) or OAIS Reference Model

(OAIS-RM)

Proposed new specifications for technical implementable standards for

interoperability are referred to here as the OAIS Interoperability Framework (OAIS-IF)

 OAIS-IF supports OAIS-PF, but OAIS-IF is not required for OAIS Compliance.  OAIS-RM archives are trustworthy, just not necessarily OAIS-IF interoperable

“Core” portions of the architecture will support all disciplines. “Protocols/plug-ins/APIs” will be adapted to archives for specific

disciplines (“Designated Communities” in OAIS terminology)

OAIS-IF interfaces can work in parallel with existing archive interfaces.

4

Basic Concept – Layered Architecture

User Interfaces

 Producer, Consumer

Abstraction Layer*

 Modularizes  Interoperability

Archive Interfaces

 “Plug-ins” for specific

applications

 For specific designated

communities

 For specific data formats

Discipline application and interfaces

5

P C

Abstraction Layer Archive

Space Science Spacecraft Housekeeping

Enterprise Data

Geology

Documents Web / HTML Astronomy

= Part of OAIS-IF

Space Science Spacecraft Housekeeping

Enterprise Data

Geology

Documents Web / HTML Astronomy

Discipline application and interfaces

Basic Concept – Applicability and Development

Core capabilities for all archives

developed in CCSDS/ISO

Unique plug-in bindings for

spaceflight developed in CCSDS

Common data types

 Developed elsewhere

Discipline-unique capabilities

 Developed by and within each

discipline

6

P C

Abstraction Layer Archive

Plug-in Libraries

• CCSDS SANA?
• IIPC Online?
• RDA Online?

Basic Concept – Cross Discipline Capabilities

• 1. Researcher in their
• wn discipline

(e.g. Heliophysics)

• 2. Researcher crossing

disciplines to study how sun affects climate change

• 3. Researcher needs

another data type (HTML)

• 4. Researcher crossing

disciplines to study how climate change effects elections

7

(1) Heliophysics plug-in

P C

Abstraction Layer Heliophysics Archive Earth Observation Archive Web / HTML Archive Election Results Archive

(2) Earth Obs plug-in

Overall OAIS-IF System Architecture

Consumer Archive Application Consumer Archive I/F Producer Archive I/F Producer Archive Application

Science Data Archive I/F

Science Data Archive

Web Archive I/F

Web Archive

Engineering Data Archive I/F

Engineering Data Archive

Doc/PDF Archive I/F

Doc/PDF Archive I/F Enterprise Data Binding Web/HTML Binding Science Data Binding Web/HTML Data Binding Engineering Data Binding Doc/PDF Binding Archive Abstraction Layer

System Executing Typical OAIS Functions

Consumer Archive Application Consumer Archive I/F Producer Archive I/F Producer Archive Application

Science Data Archive I/F

Science Data Archive

Web Archive I/F

Web Archive

Engineering Data Archive I/F

Engineering Data Archive

Doc/PDF Archive I/F

Doc/PDF Archive I/F Enterprise Data Binding Web/HTML Binding Science Data Binding Web/HTML Data Binding Engineering Data Binding Doc/PDF Binding Archive Abstraction Layer

SIP DIP AIP

SIP = Submission Info Pkg AIP = Archive Info Pkg DIP = Dissemination Info Pkg

Specific Standards To Be Developed

Consumer Archive Application Consumer Archive I/F Producer Archive I/F Producer Archive Application

Science Data Archive I/F

Science Data Archive

Web Archive I/F

Web Archive

Engineering Data Archive I/F

Engineering Data Archive

Doc/PDF Archive I/F

Doc/PDF Archive I/F Enterprise Data Binding Web/HTML Binding Science Data Binding Web/HTML Data Binding Engineering Data Binding Doc/PDF Binding Archive Abstraction Layer

PAIS – Producer Archive Interface Specification PAIP – Producer Archive Interface Protocol CAIS – Consumer Archive Interface Spec CAIP – Consumer Archive Interface Protocol AAL – Archive Abstraction Layer ADD – Architecture Description Document

PAIS PAIP ADD

Overall System Description

CAIS CAIP AAL

Binds to lower-layer Plugins for Archive application layer functions Multiple Future Bindings/plugins For many various Archive types. Including vendor- unique archives

• r datastores.

Archive Consumer Application Archive Consumer Application Archive Producer I/F Archive Abstraction Layer

More Detailed Draft Concept

Standardized Archive System Architecture

Archive Producer Application

Archival Information Package Interface Layer

Bit layer Structure layer Semantic layer Object layer Provenance Info Context Info Reference Info Fixity Info Access Rights Info

Science Data Binding Enterprise Data Binding Web/HTML Binding Doc/PDF Binding Archive Common Calls Other TBD Bindings

Communications Layers

Archive Consumer I/F User Systems Archive Systems

Archive Common Calls Doc/PDF Web/HTML Enterprise Science Other Archive Common Calls Doc/PDF Web/HTML Enterprise Science Other

Representation Information Data Preservation Description Info Provenance Info Context Info Reference Info Fixity Info Access Rights Info

Science Data Binding Enterprise Data Binding Web/HTML Binding Doc/PDF Binding Archive Common Calls Other TBD Bindings

Communications Layers

Archive Common Calls Doc/PDF Web/HTML Enterprise Science Other Archive Common Calls Doc/PDF Web/HTML Enterprise Science Other

Representation Information Data Preservation Description Info Archive Abstraction Layer

Preservation activities/processes

Bit layer Structure layer Semantic layer Object layer

Archive

Relationship of CCSDS DAI WG standards

IPELTU PAIMAS PAIS

Guides the collection/creation of the metadata required for long term use and preservation Defines a process to transfer Data /metadata to the archive Defines how to check that the information is being preserved

XFDU PVL DEDSL

CCSDS/ISO Standards for Representation Information CCSDS/ISO Standard for Packaging

CAIMAS

CAIS CAIP

DATA CONSUMERS DATA PRODUCERS

OAIS RM

AUDIT & CERTIFICATION

EAST

AAL

ARCHIVE ABSTRACTION LAYER

DATA PRESERVERS (ARCHIVES)

TELEMETRY SCIENCE ENGINEERING WEB/HTML ENTERPRISE DATA OTHER…

Bindings/APIs/ Protocols for AAL interfaces to Archives for specific Disciplines

• r data types

Abstracts the details of the Underlying archive transactions

PAIP

Defines how the information should be preserved

Defines a process to transfer Data /metadata from the archive Defines a spec/protocol to transfer Data /metadata to the archive Defines a spec/protocol to transfer Data /metadata from the archive

DA-ADD

Data Archive Architecture Description Document Normative, SW implementable for interoperability Green Book Blue Book Magenta Book Normative, but not SW implementable Informative, not normative Completed (or in systematic review) In work Future work Legend: Describes Data Architecture – how Software components fit together

OAIS Process Framework (OAIS-PF) OAIS Interoperability Framework (OAIS-IF)

Overall DAI WG schedule

13

UML rendering of CCSDS DA Architecture

14

Architecture Description Document starting as a UML model. “Eye chart” for offline review, as an indicator of DAI architecture methodology This standard-development methodology should make implementation of

OAIS easier for developers (hence more widespread adoption of OAIS)

OAIS-IF Development Approach

15

Use Cases OAIS-RM Info Model & Processes Data Archive Architecture Description UML Model Data Archive Architecture Description Document

Addressing Conference Themes

Repositories – evolution or revolution?

 This is an evolution from OAIS to Interoperability

Open and Sustainable

 OAIS and OAIS-IF is developed in an open community  The choice to open an archive is the decision of each archive – this enables that.  Long-term sustainability is enhanced by long-term interoperability

Discovery, Use and Impact

 Cross-discipline capabilities will be a breakthrough for discovery.

16

Key Conclusions

• 1. Widely adopted interoperability approaches will improve

preservation of our valuable digital assets

• 2. Standardization is required for interoperability
• 3. The CCSDS DAI Working Group is best positioned to produce

widely accepted interoperable standards for OAIS archives.

• 4. While there are other efforts to produce “standard formats” for

archives, there is no other work that:

 Supports the OAIS principles and processes  Uses this unique layered, flexible architecture  Addresses interoperability and cross-discipline capabilities

17

18

The DAI Working Group welcomes review and critique, and participation in the development of OAIS-IF from users or developers that can make substantive technical contributions. Official contact info at:

https://cwe.ccsds.org/moims/default.aspx#_MOIMS-DAI

Questions, Comments?

Backup material

19

Acronym list

20

Acronym Description Link (when available) AAL Archive Abstraction Layer ADD Architecture Description Document AIP Archive Information Package API Application Programming Interface CAIMAS Consumer Archive Interface Methodology Abstract Standard CAIP Consumer Archive Interface Protocol CAIS Consumer Archive Interface Specification CCSDS Consultative Committee for Space Data Systems www.ccsds.org DAI WG Data Archive Interoperability Working Group https://cwe.ccsds.org/moims/default.aspx#_MOIMS-DAI DEDSL Data Entity Dictionary Specification Language

CCSDS 647.1-B-1

DIP Dissemination Information Package EAST Enhanced Data Subset (Data Description Language)

CCSDS 645.0-G-1

HTML HyperText Markup Language I/F Interface IIPC International Internet Preservation Consortium http://netpreserve.org/ IPELTU Information Preservation to Enable Long Term Use ISO International Organization for Standardization www.iso.org JPL Jet Propulsion Laboratory www.jpl.nasa.gov NASA National Aeronautics and Space Administration www.nasa.gov OAIS Open Archival Information System OAIS-IF OAIS Implementation Framework OAIS-PF OAIS Process Framework OAIS-RM OAIS Reference Model

CCSDS 650.0-M-2

PAIMAS Producer Archive Interface Methodology Abstract Standard

CCSDS 651.0-M-1

PAIP Producer Archive Interface Protocol PAIS Producer Archive Interface Specification

CCSDS 651.1-B-1

PDF Portable Document Format PVL Parameter Value Language

CCSDS 641.0-B-2

RDA Research Data Alliance https://rd-alliance.org/ SANA (CCSDS) Space Assigned Numbers Authority www.sanaregistry.org SIP Submission Information Package TC20/SC13 Technical Committee 20, Subcommittee 13 https://www.iso.org/committee/46612/x/catalogue/ UML Unified Modeling Language https://www.omg.org/spec/UML/About-UML/ XFDU XML Formatted Data Unit

CCSDS 661.0-B-1

 Data Archive Interop.  Navigation  Spacecraft Monitor & Control  Telerobotics  Mission Planning & Scheduling

Mission Ops & Info Mgt Services

 Motion Imagery & Apps  Delay Tolerant Networking  Voice  CFDP Revisions

Space Internetworking Services

 CS Service Management  CS Transfer Services

Cross Support Services

 RF & Modulation  Space Link Coding & Sync.  Multi/Hyper Data Compress.  Space Link Protocols  Space Data Link Security  Optical Coding and Mod

Space Link Services

CCSDS Overview - End-to-End Architecture

 Onboard Wireless WG  Application Supt Services (incl. Plug-n-Play)  Subnetwork Services WG

Spacecraft Onboard Interface Services

MISSION CONTROL CENTER MISSION CONTROL CENTER End Users End Users Applications/Archives

 Security  Delta-DOR  System Architecture

Systems Engineering