A Summary of Formats for Streaming and Storing Music-related - - PowerPoint PPT Presentation

▶

Jul 10, 2023 132 likes •430 views

A Summary of Formats for Streaming and Storing Music-related Movement and Gesture Data Alexander R. Jensenius Oslo Nicolas Castagn ACROE, Grenoble Antonio Camurri Genova Esteban Maestre Pompeu Fabra Joseph Malloch CIRMMT, McGill

SLIDE 1

A Summary of Formats for Streaming and Storing Music-related Movement and Gesture Data

Enactive Conference 2007

Alexander R. Jensenius Nicolas Castagné Antonio Camurri Esteban Maestre Joseph Malloch Douglas McGilvray Oslo ACROE, Grenoble Genova Pompeu Fabra CIRMMT, McGill Glasgow

SLIDE 2

Movement Action Gesture

SLIDE 3

?

SLIDE 4

analysis

SLIDE 5

MoCap Score Control Audio Video Annotations Analyses AOA ASF / AMC BRD BVA / BVH C3D CSM MCML MIDI MPEG 4/7 MusicXML OSC SDIF SLML VHML

SLIDE 6

!

SLIDE 7

Luciani et al. 2006 30 % 50 % 40 % 80 % < 10 % use raw data (no format) use a proprietary, home-made format use the format of the device at hand don’t use a unique format use a known and officially released format

SLIDE 8

performance

SLIDE 9

SLIDE 10

SLIDE 11

=>

SLIDE 12

GMS GDIF OSC SDIF PML Gesture Motion Signal Gesture Description Interchange Format Open Sound Control Sound Description Interchange Format Performance Markup Language

SLIDE 13

structure encoding

Low-level signals

SLIDE 14

APML AML AOA ASF AMC BRD BVA BVH C3D CSM EBF MCML MPEG 7 MPML MURML SLML VHML Proprietary Hardware-specific Full-body skeleton 3D only ASCII-based Low-level data only not generic enough not minimal enough not efficient enough not low level enough not high level enough

SLIDE 15

X(t) Y(t) Tracks Channels Scene Units

GMS format

Gesture and Motion Signal format

as generic as possible low-level binary minimal

SLIDE 16

A Scene made of 3 Units

Unit 1: “mocap”

N 3D Position channel

Unit 2: “Force Feedback”

1 3D Position channel 1 3D Force channel

Unit 3: “keyboard”

64 A-Dimensional channels

SLIDE 17

time Raw data Trajectories Segmentation 1 Segmentation 2 Descriptions Sections head up breathing left arm turn Velocity Intensity 1a 1b 2a 2b 2c 3a 3b 3c Laban effort parameters breathing quantitative qualitative

GDIF Gesture Description Interchange Format

SLIDE 18

Movement Sensor device UDP/IP SDIF XML Data processing Realtime model Analysis GDIF OSC GDIF OSC Realtime Non-realtime Analysis Annotations

GDIF Gesture Description Interchange Format

SLIDE 19

music/tech communication protocol 1990s: CNMAT URL-based: /my-controller/slider/1/pressure 0.5 /my-reverb/room-size 0.7

OSC

Open Sound Control

www.opensoundcontrol.org

SLIDE 20

Acquisition Namespace Analysis Movement Sensor device Raw Data Pre- processing /body /instrument /environment /performance Descriptive Functional Meta /emotion /raw /cooked

SLIDE 21

Malloch et al. 2007 GDIF-OSC

SLIDE 22

Maestre et al. 2007 GDIF-XML

SLIDE 23

1990s: IRCAM, CNMAT, UPF Sound descriptions Basic data format framework Synchronisation Wide support

SDIF

Sound Description Interchange Format

http://sdif.sourceforge.net

SLIDE 24

SLIDE 25

PML

Extension of MusicXML Analysis of musical performance Score, performance & analysis information in separate, overlapping hierarchies

Performance Markup Language

SLIDE 26

EyesWeb XMI

Multimodal data + multimedia Automatic converters between layers Synchronisation Multilayered expressive gesture model: MoCap Trajectories Cues and gestures www.eyesweb.org

SLIDE 27

=

SLIDE 28

Both streaming and storage Low-, mid- and high-level Open, Free, eExtensible Compatible (MIDI, OSC, SDIF, MusicXML) Synchronisation (data + media + analyses)

Consensus...?

SLIDE 29

GDIF

SDIF OSC XML

GMS PML

MusicXML