S i mp l i fj e d F P G A - b a s e d s c h e - - PowerPoint PPT Presentation

S i mp l i fj e d F P G A

• b

a s e d s c h e me : A s c a l i n g e x e r c i s e u s i n g S B N D

J

• s

é I . C r e s p

• A

n a d ó n C

• l

u m b i a U n i v e r s i t y N e v i s L a b

• r

a t

• r

i e s

D U N E F D D A Q “ D e s i g n ” Wo r k s h

• p

3 – 3 1 O c t

• b

e r 2 1 7 , C

• l

u m b i a U n i v e r s i t y , U S A

2 José I. Crespo-Anadón | Simplified FPGA-...

Introduction

Will cover only TPC readout. SBND DAQ will be quite similar to MicroBooNE (already in operation, see Wes' talk):

• This talk focuses on back-end electronics with FPGA (same functionality as

MicroBooNE). System not optimized for DUNE scale.

3 José I. Crespo-Anadón | Simplified FPGA-...

SBND DAQ design requirements

• Triggered readout stream

–

Trigger rate: ≤ 20 Hz (15 Hz BNB + 1.25 Hz NuMI)

–

Readout time window: 3 1.28 ms (drift time) = 3.84 ms ✕

–

Data rate: 11,264 channels 2 MS/s 2 Bytes 3.84 ms 20 Hz ≤ 3.46 GB/s ✕ ✕ ✕ ✕ distributed between 11 DAQ servers: 315 MB/s/server

– Lossless compression (Huffman)

• Continuous (a. k. a. supernova) readout stream

–

Data rate: 11,264 channels 2 MS/s 2 Bytes = 45 GB/s ✕ ✕ distributed between 11 DAQ servers: 4.1 GB/s/server

– Lossless compression (Huffman) and zero suppression (factor

~ 80) ✕

4 José I. Crespo-Anadón | Simplified FPGA-...

Reminder of SBND front-end electronics

• Assuming cold ADC design. Very similar design to ProtoDUNE-SP.
• 11264 channels.
• 88 Front End Mother Boards (FEMB). 128 ch/FEMB.

–

Amplification and shaping in custom CMOS ASICs (180 nm). Four gains: 4.7, 7.8, 14, 25 mV/fC. Four peaking times: 0.5, 1.0, 2.0, 3.0 μs. Two baselines: 200 mV, 900 mV.

–

12-bit ADC at 2 MS/s.

–

FPGA (Altera Cyclone IV) mezzanine sends the data out of the cryostat

• ver cold cable (4 links × 1.28 Gbps/link).
• Warm Interface Board (WIB) with an FPGA (Altera Arria V) receives the data

from 4 FEMB (512 ch) and sends it to back-end electronics through 8 optical links (2.125 Gbps/link). SBND: arXiv:1503.01520 [physics.ins-det]

5 José I. Crespo-Anadón | Simplified FPGA-...

SBND front-end & back-end (partial) overview

6 José I. Crespo-Anadón | Simplified FPGA-...

SBND Crate, Clock board & Controller

Crate

• 6U crate with capacity for 20 boards.

Clock board

• 1 board on the back of the backplane receives 16 MHz clock from Master Clock NIM

module. Controller

• Interface between DAQ server (with PCIe card via optical link) and boards in crate.
• Receives trigger from Trigger Fan Out (FO) in NIM bin. Can also generate local trigger.
• Enables slow readout for debugging.

XMIT

7 José I. Crespo-Anadón | Simplified FPGA-...

SBND FEM

Front End Module (FEM). 64 ch/board (typically 32 induction ch + 32 collection ch). Optical receiver (ORx) receives data from WIB via 2.125 Gbps optical link. Then deserialized. 1 M × 36 bit 128 MHz SRAM as ring

• buffer. 8 frames in buffer (1.28

ms/frame × 8 frames = 10.24 ms). 64 MHz for writing in time-order. 64 MHz for reading by channel. No deadtime. Data processing by FPGA (Altera Stratix III). Splits data in two streams: 1) Triggered stream: reads out 4 frames, trims to 1 frame before + 2 frames after trigger. 64 ch × 3 × 1.28 ms window/ch × 2 MS/s × 2B/S = 0.98 MB/board/trigger. Applies lossless (Huffman) compression. 2) Continuous stream: read out all the time. FPGA applies zero-suppression based on dynamical estimation of baseline and amplitude threshold (next slide), then applies Huffman compression. Each stream has a DRAM to buffer the processed data until retrieved.

8 José I. Crespo-Anadón | Simplified FPGA-...

DATA from MicroBooNE FEM at Nevis test stand SN stream Dynamic baseline (emulated)

8 postsamples

Trigger stream

7 presamples

± threshold

FPGA-based zero suppression for continuous stream

• Implemented in the FEM FPGA.
• Only the waveform passing a certain amplitude threshold (configurable) with respect

to the channel baseline is saved, plus presamples and postsamples (configurable).

• The baseline is dynamically computed using 192 preceding samples.

9 José I. Crespo-Anadón | Simplified FPGA-...

MicroBooNE & SBND: “supernova” stream

• Surface detectors. Cannot self-trigger on supernova neutrinos.

Instead, write data to disk continuously and rely on a delayed (hours) external trigger from the Supernova Early Warning System (SNEWS).

• Bottleneck of the stream is the disk writing speed at the DAQ server (~ 50 – 100

MB/s). Input: ~ 4 GB/s/server. Use zero suppression + Huffman compression to achieve compression goal (~ × 40 – 80).

• Finalizing commissioning in MicroBooNE. Similar system planned for SBND.

Michel e candidate from trigger stream Michel e candidate from supernova stream

S t

• p

p i n g μ Michel e S t

• p

p i n g μ Bremsstrahlung Michel e Bremsstrahlung Data from region on collection plane after zero-suppression Amplitude threshold: +30 ADC (noise/MIP separation)

10 José I. Crespo-Anadón | Simplified FPGA-...

SBND XMIT and PCIe card

Transmitter (XMIT) board

• 1 board/crate reads up to 16 FEMs = 2 WIBs = 8 FEMBs = 1024 ch.
• Fetching data from each FEM through backplane (512 MB/s) using token

passing.

• Triggered stream prioritized over continuous stream.
• Data sent to PCIe card on DAQ server via optical links (3.125 Gbps × 2).
• Each stream has its own optical link pair.

PCIe card

• Receives data from one stream from XMIT.
• DMA using PCIe bus. 4 lanes × 2.5 Gbps = 1.25 GB/s.

PCIe card XMIT

11 José I. Crespo-Anadón | Simplified FPGA-...

SBND Trigger board

Inputs:

• PMT trigger
• Accelerator gates (BNB, NuMI, BNB

dump; external)

• Cosmic Ray Tracker trigger
• Laser (calibration)
• GPS PPS

Configurable logic and prescaling. Outputs:

• Global trigger.
• Trigger information sent to DAQ server

(PCIe card via optical link).

• DAQ-driven calibration trigger (for cold

ASICs).

• Scaler outputs.

Inputs (12) Outputs (10)

12 José I. Crespo-Anadón | Simplified FPGA-...

SBND back-end electronics for DUNE: deliverables

150 APA/module. 2560 ch/APA.

• 40 FEMs/APA. 6000 FEMs/module.

(Assuming 16 FEMs/crate as SBND)

• 2.5 crates/APA. 375 crates/module.

–

Each crate includes 1 Controller, 1 XMIT, 1 Clock board, 1 Backplane, 1 Chassis.

–

1 power supply/crate.

–

1 crate is read by 1 DAQ server with 3 PCIe cards.

• SBND has +10% components as hot spares. 10% of spares for

DUNE are enough to fully instrument 3 SBND. Assuming +2%.

13 José I. Crespo-Anadón | Simplified FPGA-...

SBND back-end electronics for DUNE: cost

• FEM, Crate (instrumented) and Trigger Board cost based on MicroBooNE's

experience.

• Not included:

–

Optical links between WIB and FEM.

–

Racks and optical links between XMIT and PCIe cards.

• Not accounting for inflation, chip obsolescence, or cost reduction.
• Can save 4.3% (595 k$) by having 18 FEMs/crate (334 + 7 crates).

14 José I. Crespo-Anadón | Simplified FPGA-...

SBND back-end electronics for DUNE: power

• Back-end electronics:

–

~ 240 W/crate × 375 crates/module = 90 kW/module (not including cooling, factor ~ 2 in SBND)

• DAQ server:

–

360 W/server × 375 servers/module = 135 kW/module

15 José I. Crespo-Anadón | Simplified FPGA-...

Possible improvements over SBND design for DUNE

After splitting the data into triggered and continuous streams, keep them physically separated.

• Do not use prioritization; use two different buses in the crate backplane instead.
• Do not read both streams with the same DAQ server; have different DAQ servers for

different streams, and have one DAQ server reading more than one crate instead.

–

Fit as many PCIe cards as possible per DAQ

• server. Limited by PCIe slots in server

motherboard/PCIe bus bandwidth.

–

Disk-write limitation for continuous stream can be avoided by keeping data in (large) circular buffer in RAM and run a software trigger (GPU-powered?) to filter what is written/further processed.

• TPC-based trigger fostered by having multiple

crates converging into one DAQ server → More channels to find patterns.

16 José I. Crespo-Anadón | Simplified FPGA-...

Backup

17 José I. Crespo-Anadón | Simplified FPGA-...

DAQ server FEM

SRAM 1 M × 36 bit 128 MHz Supernova DRAM DRAM Deserial Neutrino Backplane (512 MB/s)

Trigger Clock board PCIe card Controller XMIT

PCIe bus (1.25 GB/s)

PCIe card PCIe card Clock 16 MHz Trigger FO Crate

Frame Frame Frame Frame Frame Frame Frame Frame

W R Optical link (390 MB/s × 2 )

To other crates TPC

SBND TPC back-end

ORx