General Assembly Meeting 2016 WORK PACKAGE 5 Real-Time Data - - PowerPoint PPT Presentation

▶
general assembly meeting 2016
SMART_READER_LITE
LIVE PREVIEW

General Assembly Meeting 2016 WORK PACKAGE 5 Real-Time Data - - PowerPoint PPT Presentation

Aug 26, 2023 236 likes •459 views

General Assembly Meeting 2016 WORK PACKAGE 5 Real-Time Data Management of ESS Data Tobias Richter BrightnESS is funded by the European Union Framework Programme for Research and Innovation Horizon 2020, under grant agreement 676548 WP5

slide-1
SLIDE 1

BrightnESS is funded by the European Union Framework Programme for Research and Innovation Horizon 2020, under grant agreement 676548

General Assembly Meeting 2016

WORK PACKAGE 5 Real-Time Data Management of ESS Data Tobias Richter

slide-2
SLIDE 2
  • Captures technical scope that was not in the ESS TDR
  • Offers new options for capturing data not available elsewhere
  • Reduces risk in the ESS construction phase through

collaboration with existing facilities, offering expertise and the ability to test under real conditions

  • Fits into the Data Management Group of the

Data Management and Software Centre (Copenhagen)

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

WP5 – Real time Management of ESS Data

slide-3
SLIDE 3

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

Overview of ESS Data Pipelines

slide-4
SLIDE 4

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

Overview of WP5

Task 5.1 Creating a standard neutron event data stream for different detector types ESS, KU Task 5.2 Creating a standard method for streaming meta-data for fast applied fields ESS, PSI Task 5.3 Software to aggregate and make available neutron event data and sample meta-data ESS, PSI, Elettra

slide-5
SLIDE 5

9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 Activities / Deliverables

M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13 M14 M15 M16 M17 M18 M19 M20 M21 M22 M23 M24 M25 M26 M27 M28 M29 M30 M31 M32 M33 M34 M35 M36

WP5 5.1 Standard neutron event stream D5.2 Report processing choices for detector types D5.6 Software neutron event data processing M5.1 1st Integrated design review M5.2 Event data software test M5.3 2nd Integrated software test M5.4 Final software delivery 5.2 Streaming fast field data D5.4 Report filed data acquisition D5.7 Software fast field acquisition

X

M5.1 1st Integrated design review M5.3 2nd Integrated software test M5.4 Final software delivery 5.3 Data aggregation D5.1 Design report data aggregator software D5.3 Beta-version data aggregator software D5.5 Data aggregator software M5.1 1st Integrated design review M5.3 2nd Integrated software test M5.4 Final software delivery 2016 2017 2018

Milestones and Deliverables

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

slide-6
SLIDE 6

Joint Task with PSI.

  • Collecting Metadata from fast Sample Environments
  • High Speed – up to MHz
  • Lots of data, close to neutron events
  • No turn key solution available

that includes accurate timing

  • Evaluating partial technical

solutions that can be combined

  • Integration into data stream

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

Task 5.2 – Fast Field Data

DMSC/ESS staff on board since this September

slide-7
SLIDE 7

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

Task 5.1 – Standard Neutron Events

Joint task with NBI (Copenhagen University)

  • To support novel detector types (e.g. WP4)

despite the lack of associated electronics effort

  • Software processing of digitised signals that would

traditionally happen in analogue or digital electronics

  • Detectors emit raw time stamped

information that needs refinement

  • Initial assignment was to gather

and analyse existing test data and establish specifications for the way data will be presented in the future

slide-8
SLIDE 8

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

Event Classification

slide-9
SLIDE 9

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

Event Classification

slide-10
SLIDE 10

Task 5.1 Outcome

  • Contributed to and created software packages to read, convert, analyse, and

classify existing trial data

  • Results will be disseminated at a joint

WP4 and WP5 workshop in November

  • Documented current status of data and best
  • First Deliverable due in May
slide-11
SLIDE 11

Task 5.3 – Data Aggregation

  • Task with PSI and our ISIS/STFC in kind collaborators
  • Target goal: Software Framework to support neutron and metadata data stream
  • Problem includes the “Big Data” 3 Vs:

– Volume – Velocity – Variety

  • Initial objective:

Agree on technology platform based on common criteria

  • Prototyping and Testing
  • Formalise Testing

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

slide-12
SLIDE 12

Options investigated:

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

Technology Choice

V4

Client Client Client Server Server

EPICS interaction model Design Model

Facility Control System General Purpose Communication Library “Big Data” Data Broker

slide-13
SLIDE 13

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

Performance and Efficiency Tests

500 1000 1500 2000 2500 1 2 3 4 5 6 7 8

Throughput [MB/s] # clients

0MQ multiple clients

single cumulative 0.0 100.0 200.0 300.0 400.0 500.0 600.0 700.0 5 10 15 20 25

Throughput [MB/s] multiplier

0MQ vs EPICS

0MQ EPICS

slide-14
SLIDE 14

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

Choice and Architecture

Event Formation Sample Environment Choppers

Kafka cluster

File Writer Mantid

P P P C C

EFU EFU EFU

  • Instr. 1 Detector 1

switch

  • Instr. 1 Detector 2
  • Instr. 2 Detector 1

Deliverable D5.1

slide-15
SLIDE 15
  • Our data stream will be the one

and only scientific record of ESS experiments

  • Software needs to function

correctly, reliable, and in real time

  • Going forward software testing

will be as important as implementing new features

  • Integration and testing facility is

set up in the ESSIIP lab (Lund)

  • Test will be run by partners in

their facilities

  • Automated test are implemented on

virtual hardware for now

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

Software Testing

slide-16
SLIDE 16
  • Deliverable D5.1 accepted
  • Kafka and Flat Buffers chosen as core technology
  • Better understanding of detector processes,

design, output (current and future)

  • Classification of event data makes tasks quantifiable
  • Initial Prototypes implemented and Designs verified
  • Repositories and Development Workflows established
  • Testing of various replacements for

face to face collaboration (Jira, Slack, Skype, …)

  • Agreed on Roadmap

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

Some WP5 Results

slide-17
SLIDE 17
  • NOBUGS conference

https://nobugs.esss.se

Mid October

  • Joint meeting with WP4 (Jamboree)

Early November

  • Readout Electronics and Output Meeting

December?

  • Integrated Design Review of all WP5 tasks

February

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

Upcoming Events

slide-18
SLIDE 18

Costs

1 2 3 4 5 6 7 8 9

WP5 Planned Staff Levels

Elettra PSI UCPH ESS

slide-19
SLIDE 19
  • The work package has met all deadlines so far
  • We’ve successfully jump started efforts for both BrightnESS and ESS

– Gathered crucial information for the project and operations – Established working relationships across multiple sites and including in kind – Started collaborative work on new common software packages – Agreed on direction for further development

  • Not all requirements are established and understood

as well as they would be in an ideal setting

  • Hiring staff takes time,

so does starting a larger project at a green field site

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

Summary

slide-20
SLIDE 20

BrightnESS is funded by the European Union Framework Programme for Research and Innovation Horizon 2020, under grant agreement 676548

Thank you!

slide-21
SLIDE 21

Backup Slides

BrightnESS is funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 676548

slide-22
SLIDE 22

ESS UCPH PSI Elettra Monthly Difference to Planned Profile

Month 1-6 Month 7-10

Costs