CRDTs in Production Dmitry Martyanov, Software Engineer @ PayPal - - PowerPoint PPT Presentation

▶
crdts in production
SMART_READER_LITE
LIVE PREVIEW

CRDTs in Production Dmitry Martyanov, Software Engineer @ PayPal - - PowerPoint PPT Presentation

Nov 07, 2023 280 likes •779 views

CRDTs in Production Dmitry Martyanov, Software Engineer @ PayPal QCon, 2018 Geo-Distributed Datastore Context More than 200 countries Regulatory requirements State Machine of Compliance Status Modified by multiple Actors Shared Mutable

slide-1
SLIDE 1

CRDTs in Production

Dmitry Martyanov, Software Engineer @ PayPal QCon, 2018

slide-2
SLIDE 2

Geo-Distributed Datastore

slide-3
SLIDE 3

More than 200 countries Regulatory requirements State Machine of Compliance Status Modified by multiple Actors

Context

slide-4
SLIDE 4

Shared Mutable State

slide-5
SLIDE 5

Shared Mutable State Mutex

slide-6
SLIDE 6

Shared Mutable State Mutex Transactions

slide-7
SLIDE 7

Geo-Distributed Datastore

slide-8
SLIDE 8

Eventual Consistency

slide-9
SLIDE 9

Distributed System

Replica A Replica B Replica C Replica D

slide-10
SLIDE 10

Distributed System

tn: PUT(key, val) tn+1: GET(key) = val

Replica A Replica B Replica C Replica D

slide-11
SLIDE 11

Distributed System

tn+1: GET(key) = ?

Replica A Replica B Replica C Replica D

tn: PUT(key, val) tn+1: GET(key) = val

slide-12
SLIDE 12

Affinity Based Approaches

Replica A Replica B Replica C Replica D

slide-13
SLIDE 13

Affinity Based Approaches

Replica A Replica B Replica C Replica D

slide-14
SLIDE 14

Coordinator Based Approaches

Replica A Replica B Replica C Replica D

slide-15
SLIDE 15

Paxos, Raft, etc.

Consensus Based Approaches

slide-16
SLIDE 16

Business Logic Domain Platform Service Infrastructure Data Infrastructure Domain Data

Service Datastore

Service Stack

What type of documents Business Rules Filtering Logic, etc. Entity objects DAO Layer Flow Control Service discovery Routing & Balancing Failover strategy Data Model, Records Deployment configuration Namespaces & Schemas Replication params

slide-17
SLIDE 17

Business Logic Domain Platform Service Infrastructure Data Infrastructure Domain Data

Service Datastore

Service Stack

slide-18
SLIDE 18

Business Logic Domain Platform Service Infrastructure Data Infrastructure Domain Data

Service Datastore

Service Stack

slide-19
SLIDE 19

Conflict-free Replicated Data Types

slide-20
SLIDE 20

CRDTs


Requirements:

  • + Commutativity
  • + Associativity
  • + Exactly once delivery
  • - Idempotence

A f(x1) f(x3) f(x3) g(x2) g(x2) g(x2) B C

commutative convergent

Requirements:

  • + Commutativity
  • + Associativity
  • + Idempotence
  • - Exactly once delivery

f(x1) merge merge g(x2) merge merge A B C

slide-21
SLIDE 21

Convergent CRDTs

  • M(a, b) = M(b, a)
  • M(M(a, b), c) = M(a, M(b, c))
  • M(a, b) = M(M(a, b), b) = M(M(M(a, b), b), b)
slide-22
SLIDE 22

Business Logic Domain Platform Service Infrastructure Data Infrastructure Domain Data

Service Datastore CRDTs CRDTs CRDTs

Impacted Components for CRDTs

slide-23
SLIDE 23

seat: 12F

a b

t1 t2

TIME

seat: 16D seat: 12F seat: 12F

XDR

Online Flight Check-in System

slide-24
SLIDE 24

seat: 12F (220)

a b

t1 t2

TIME

seat: 16D (150) seat: 12F (220) seat: 12F (220)

LWW M(a, b) for LWW = MAX(a, b)

XDR

Online Flight Check-in System

slide-25
SLIDE 25

seat: {a1:12F} seat: {b1: 16D} seat: { b1: 16D,

a1: 12F

}

TIME

a

seat: { b1: 16D,

a1: 12F

}

b

t1 t2

XDR

Online Flight Check-in System

slide-26
SLIDE 26

seat: {a1:12F} seat: {b1: 16D} seat: { b1: 16D,

a1: 12F

}

TIME

seat: { b1: 16D,

a1: 12F

}

t1 t2

Add-O Map XDR

Online Flight Check-in System

a b

slide-27
SLIDE 27

Causality Vector (cv)

a1;b1

seat: { b1: 16D,

a1: 12F

}

Causality

slide-28
SLIDE 28

Causality Vector (cv)

a1;b1

seat: { b1: 16D,

a1: 12F

}

12F 10A 5C 12F is causal to 10A - we can drop 12F 10A is causal to 5C - we can drop 10A

Causality

slide-29
SLIDE 29

Causality Vector (cv)

a1;b1

seat: { b1: 16D,

a1: 12F

}

12F 10A 5C 12F is causal to 10A - we can drop 12F 10A is NOT causal to 5C - we can NOT drop 10A

Causality

slide-30
SLIDE 30

Causality Vector (cv)

a1;b1

GET(key): value => GET(key): (value, cv) PUT(key, value) => PUT(key, value, cv)

Client Operations:

seat: { b1: 16D,

a1: 12F

}

Causality

slide-31
SLIDE 31

Causality Vector (cv)

a1;b1

GET(key): value => GET(key): (value, cv) PUT(key, value) => PUT(key, value, cv)

Client Operations:

seat: { b1: (16D, cv),

a1: (12F

, cv) }

Causality

slide-32
SLIDE 32

Aerospike Datastore

Client Database Memory Database Disk (Flash)

Read Path

Database Disk (Flash)

Async XDR

slide-33
SLIDE 33

Client Database Memory Database Disk (Flash)

Read Path

Database Disk (Flash)

Async XDR

Key Metadata Bin1 Bin2 Bin3

Bins Record

Aerospike Datastore

slide-34
SLIDE 34

User-Defined Functions

1 2

[ bin1: 16B, bin2: 12A ] [ bin2: 14C, bin3: 10A ] [ bin1: 16B, bin2: (12A or 14C), bin3: 10A ] A B Result

XDR

Aerospike Datastore

slide-35
SLIDE 35

a

12F(_) a1:(12F ,_)

b

Bins

1

Bins

1 a1 (12F, _)

a b

slide-36
SLIDE 36

a

12F(_) a1:(12F ,_) 10D(_) b1:(10D,_)

b

Bins

1 2

b1

(10D, _)

Bins

1 2 a1 (12F, _) (12F, _)

a b

slide-37
SLIDE 37

a

12F(_) a1:(12F ,_) 10D(_) b1:(10D,_)

b

a1: (12F , _) b1:(10D,_)

Bins

1 2 3 a1 (12F, _)

b1

(10D, _) (10D, _)

Bins

1 2 3 a1 (12F, _) (12F, _) (12F, _)

b1

(10D, _)

a b

slide-38
SLIDE 38

a

12F(_) a1:(12F ,_) 10D(_) b1:(10D,_) [12F](a1) [12F] -> 10F 10F(a1)

b

a1: (12F , _) b1:(10D,_) a1: (12F , _) b1:(10D,_) a2: (10F , a1)

Bins

1 2 3 4 a1 (12F, _) (12F, _)

b1

(10D, _) (10D, _) (10D, _)

Bins

1 2 3 4 a1 (12F, _) (12F, _) (12F, _) (12F, _)

b1

(10D, _) (10D, _)

a2

(10F, a1)

a b

slide-39
SLIDE 39

a

12F(_) a1:(12F ,_) 10D(_) b1:(10D,_) [12F](a1) [12F] -> 10F [10F ,10D](a2b1) [10F , 10D] -> 5C 10F(a1) 5C(a2b1)

b

a1: (12F , _) b1:(10D,_) a1: (12F , _) b1:(10D,_) a2: (10F , a1) b1:(10D,_) a2: (10F , a1) a3:(5C, a2b1)

Bins

1 2 3 4 5 a1 (12F, _) (12F, _) (12F, _)

b1

(10D, _) (10D, _) (10D, _) (10D, _)

Bins

1 2 3 4 5 a1 (12F, _) (12F, _) (12F, _) (12F, _) (12F, _)

b1

(10D, _) (10D, _) (10D, _)

a2

(10F, a1) (10F, a1)

a3

(5C, a2b1)

a b

slide-40
SLIDE 40

a

12F(_) a1:(12F ,_) 10B(_) b1:(10D,_) [12F](a1) [12F] -> 10F [10D,10F](a2b1) [10D, 10F] -> 5C 10F(a1) 5C(a2b1)

b

a1: (12F , _) b1:(10D,_) a1: (12F , _) b1:(10D,_) a2: (10F , a1) b1:(10D,_) a2: (10F , a1) a3:(5C, a2b1)

Bins

1 2 3 4 5 6 a1 (12F, _) (12F, _) (12F, _) (12F, _)

b1

(10D, _) (10D, _) (10D, _) (10D, _) (10D, _)

a3

(5C, a2b1) a3:(5C, a2b1)

Bins

1 2 3 4 5 6 a1 (12F, _) (12F, _) (12F, _) (12F, _) (12F, _) (12F, _)

b1

(10D, _) (10D, _) (10D, _) (10D, _)

a2

(10F, a1) (10F, a1) (10F, a1)

a3

(5C, a2b1) (5C, a2b1)

a b

slide-41
SLIDE 41

Learnings

  • CRDTs allowed us to achieve convergent predictable

state of our data

slide-42
SLIDE 42

Learnings

  • CRDTs allowed us to achieve convergent predictable

state of our data

  • Education about right trade-off between Consistency

and Correctness

slide-43
SLIDE 43

Learnings

  • CRDTs allowed us to achieve convergent predictable

state of our data

  • Education about right trade-off between Consistency

and Correctness

  • Do not underestimate concurrent data access
slide-44
SLIDE 44

Caveat #1: CV Propagation

Data Access Decision Engine Data Access Mid-layer Service UX Component

1 2

slide-45
SLIDE 45

Caveat #1: CV Propagation

Data Access Decision Engine Data Access Mid-layer Service UX Component

1 2

slide-46
SLIDE 46

10A(_) a1:(10A,_) 12F(_) a1:(10A,_) a2:(12F ,_) a1:(10A,_) a2:(12F ,_) a3:(10B,_)

Caveat #2: Siblings Explosion

10B(_) 12B(_) a1:(10A,_) a2:(12F ,_) a3:(10B,_) a4:(12B,_) 5A(_) a1:(10A,_) a2:(12F ,_) a3:(10B,_) a4:(12B,_) a5:(5A,_)

a

slide-47
SLIDE 47

Caveat #3: Wait, Siblings ?

??? a

12F(_) a1:(12F ,_) 10B(_) b1:(10D,_) [12F](a1) [12F] -> 10F [10D,10F](a2b1) [10D, 10F] -> 5C 10F(a1) 5C(a2b1)

b

a1: (12F , _) b1:(10D,_) a1: (12F , _) b1:(10D,_) a2: (10F , a1) b1:(10D,_) a2: (10F , a1) a3:(5C, a2b1) a3:(5C, a2b1)

slide-48
SLIDE 48

Thanks!