Sliding-Window Aggregation in Worst-Case Constant Time Martin - - PowerPoint PPT Presentation

Sliding-Window Aggregation in Worst-Case Constant Time

Martin Hirzel, IBM Research AI 30 October 2017 Dagstuhl Seminar on Big Stream Processing Systems

Streaming Engines

Martin Hirzel, IBM Research AI ibmstreams.github.io 2

Medical Telco Science Finance , Insights Actions Streaming engine

Productivity Challenge

Martin Hirzel, IBM Research AI

SPL: An Extensible Language for Distributed Stream Processing [TOPLAS'17]

Medical Telco Science Finance , Insights Actions Streaming engine High-level programming experience

3

Performance Challenge

Martin Hirzel, IBM Research AI A Catalog of Stream Processing Optimizations [CSUR'14]

Medical Telco Science Finance , Insights Actions High-level programming experience Parallel algorithms f(x1) || f(x2) Incremental algorithms f(x) ± f(∆) Streaming engine

4

Sliding-Window Aggregation

• a. max: 6

evict

• b. max: 6

insert 1

• c. max: 6

evict

• d. max: 5

insert 4

• e. max: 5

evict

• f. max: 5

insert 2

• g. max: 5

evict

• h. max: 4

insert 7

• i. max: 7

2 6 3 5 3 6 3 5 3 1 3 5 3 1 4 5 3 1 4 2 3 1 4 2 7 3 5 3 1 5 3 1 4 3 1 4 2 6 3 5 3 Youngest Oldest Time State Query Step In general:

• Any associative

aggregation operation ⊕ (not just max ⇒ sum, geoMean, Bloom, ,)

• Any interleaving of

insert and evict (not just alternating ⇒ variable-sized windows)

Martin Hirzel, IBM Research AI General Incremental Sliding- Window Aggregation [VLDB'15] 5

Sliding-Window Aggregation in Worst-Case Constant Time

Martin Hirzel, IBM Research AI

Low-Latency Sliding-Window Aggregation in Worst- Case Constant Time [DEBS’17] (best paper)

L

(left)

R

(right)

A

(accum)

B

(back)

E

(end)

vals aggs F

(front)

De-Amortized Banker’s Aggregator (DABA): Every insert, evict, and query invokes the associative ⊕ operation at most O(1) times.

(|lF| = 0 and |lB| = 0) or (|lL| = |lR| and |lL| + |lR| + |lA| + 1 = |lF| − |lB|) 6