Just-Right As available as possible, consistent when necessary - - PowerPoint PPT Presentation

[AntidoteDB & Just-Right Consistency]

Just-Right Consistency: As available as possible, consistent when necessary

The CAP theorem forces a choice between strong consistency (CP) and availability and responsiveness (AP) when the network can partition. The Just-Right Consistency (JRC) approach defines, for a particular application, a consistency model that is sufficient to maintain the application invariants, otherwise remaining as available as possible. JRC leverages knowledge of the

• application. Two invariant-

maintaining patterns are AP- compatible: ordered updates and atomic grouping. They are compatible with concurrent and asynchronous updates, orthogonally to CAP. In contrast, checking a data precondition on partitioned state is CAP-sensitive. However, if two updates do not negate each other's precondition, they may legally execute concurrently. Updates must synchronise only if one negates the precondition of the other. The JRC approach is supported: by the CRDT data model that ensures that concurrent updates converge; by Antidote, a cloud-scale CRDT data store that guarantees transactional causal consistency; and by developer tools (static analysers and domain-specific languages) that help guarantee invariants.

1

Just-Right Consistency

As available as possible As consistent as necessary Correct by design

Marc Shapiro, UPMC-LIP6 & Inria Annette Bieniusa, U. Kaiserslautern Nuno Preguiça, U. Nova Lisboa Christopher Meiklejohn, U. Catholique de Louvain Valter Balegas, U. Nova Lisboa

[AntidoteDB & Just-Right Consistency] 3

trois bases

[AntidoteDB & Just-Right Consistency]

Bridging the CAP gap

CP ∩ AP = ∅ No single consistency model is best for all applications Insight: Maintain invariants

• Preserve sequential patterns
• Synchronise only when strictly

necessary for application ⟹ tools

4

P

best possible availability and performance correct!

[AntidoteDB & Just-Right Consistency]

FMK Fælles Medicinkort

5

R X

Dr Alice Aalborg Hospital Patient: Mr Bob Pharmacy: Byrum Causatin: 2 boxes ⟶ 1 Transactol: 1 box

Dr Alice Patient Bob Byrum pharma

create (…) add-med (…) get-med (…) process (…) Causatin: 2 boxes ⟶ 1

[AntidoteDB & Just-Right Consistency]

FMK invariants

6

R X

Dr Alice Aalborg Hospital Patient: Mr Bob Pharmacy: Byrum Causatin: 2 boxes ⟶ 1 Transactol: 1 box

Dr Alice Patient Bob Byrum pharma

create (…) add-med (…) get-med (…) process (…) Causatin: 2 boxes ⟶ 1 pre- condition relative

• rder

2 1 joint update

[AntidoteDB & Just-Right Consistency]

Geo-distrib: invariants?

7

R X

Dr Alice Aalborg Hospital Patient: Mr Bob Pharmacy: Byrum Causatin: 2 boxes ⟶ 1 Transactol: 1 box

Dr Alice Patient Bob Byrum pharma

create (…) add-med (…) get-med (…) process (…) Causatin: 2 boxes ⟶ 1 pre- condition relative

• rder

2 1 joint update EC does not maintain! CP is overkill!

[AntidoteDB & Just-Right Consistency]

AP-compatible programming constructs

Available under partition ⟹ no synchronisation ⟹ asynchronous updates ⟹ fast response AP-compatible:

• CRDT data model
• Relative-order pattern
• Joint-update pattern

8

[AntidoteDB & Just-Right Consistency]

AP data model: CRDTs

Concurrent, asynchronous updates

• Standard register model: assignments ⟹ CP
• AP ⟹ concurrent updates merged

CRDT: register, counter, set, map, sequence

• Extends sequential type
• Encapsulates convergent merge

9

cnt += 1 cnt += 1 cnt = 0 cnt += 2 cnt += 2 cnt = 0 add-med(1) add-med(2) cnt = 3 cnt = 3

[AntidoteDB & Just-Right Consistency]

Relative order is AP-Compatible

create-p before add-pp

• Referential integrity
• x valid ∧ x points to y ⟹ y valid
• admin-login-enabled ⟹ non-

default-password RHS ≔ true; LHS ≔ true Transmit in the right order! AP-compatible: Causal Consistency

10

P 1=RHS! 2=LHS! >1

without CC animation

[AntidoteDB & Just-Right Consistency]

patient? pharma?

Joint update is AP-Compatible

create-p updates doctor, patient & pharmacy record Transmit joint updates together

• write-atomic

+ Read from common set of txns

• snapshot property

= All-or-Nothing (A of ACID) AP-compatible

11

P p h a r m a !

patient! pharma! patient?

pharma? patient!

Atomic update Snapshot

without animations

[AntidoteDB & Just-Right Consistency]

process-p (…, nb) { if cnt ≥ nb // precondition at source cnt –= nb // at every replica } // ≥ 0

CAP-sensitive invariants

12

cnt –= 1 cnt –= 1 pp(…, 1) cnt ≥ 1

cnt ≥ 0 cnt ≥ 0 cnt ≥ 0 cnt ≥ 0 cnt ≥ 0 2 2 2 1 1

[AntidoteDB & Just-Right Consistency]

process-p (…, nb) { if cnt ≥ nb // precondition at source cnt –= nb // at every replica } // cnt ≥ 0 Precondition stable w.r.t. concurrent add-med Concurrency OK

CAP-sensitive invariants

13

–= 1 cnt –= 1 pp(…, 1) cnt ≥ 1

cnt ≥ 0 cnt ≥ 0 cnt ≥ 0 cnt ≥ 0 cnt ≥ 0 2 2 2 1 1

cnt += 3 cnt += 3 add-med (…, 3)

4 4

[AntidoteDB & Just-Right Consistency]

process-p (…, nb) { if cnt ≥ nb // precondition at source cnt –= nb // at every replica } // cnt ≥ 0

CAP-sensitive invariants

14

cnt –= 1 cnt –= 1 pp(…, 1) cnt ≥ 1

cnt ≥ 0 cnt ≥ 0 cnt ≥ 0 cnt ≥ 0 cnt ≥ 0 2 2 2 1 1

[AntidoteDB & Just-Right Consistency]

process-p (…, nb) { if cnt ≥ nb // precondition at source cnt –= nb // at every replica } // cnt ≥ 0 Precondition not stable w.r.t. concurrent process-p

• Forbid concurrency? Synchro, CP.
• Or remove invariant? AP, degraded semantics

CAP-sensitive invariants

15

cnt –= 1 cnt –= 1 pp(…, 1) cnt ≥ 1

cnt ≥ 0 cnt ≥ 0 cnt ≥ 0 cnt ≥ 0 cnt ≥ 0 2 2 2 1 1

cnt –= 2 cnt –= 2 pp(…, 2) cnt ≥ 1

–1 –1 CISE Static Analysis

[AntidoteDB & Just-Right Consistency]

CISE tools

Static analysis of any application:

• Operations, invariants
• Does each individual op maintain invariant?
• Do concurrent updates converge?
• Is precondition of u stable w.r.t. concurrent v?

If not:

• Change specification (invariant)
• or Synchronise
• Designer decision, per pair (u, v)

Ex: medication count= inc||inc, inc||dec, dec||dec

16

[AntidoteDB & Just-Right Consistency]

Just-Right Consistency

Methodology for provably ensuring As Available as Possible, Consistent Enough TCC ⟹ AP-compatible invariants CAP-sensitive invariants: Bounded Ctr, CISE

17

AntidoteDB:

• CRDTs
• Causal Consistency
• Transactions
• Bounded Counter

⟹ AP- compatible

CISE verification & co-design (+ related tools)

CP when necessary mostly AP

[AntidoteDB & Just-Right Consistency]

AntidoteDB

CRDT data model

• Register, counter, set, map, sequence
• Extend sequential semantics
• AP compatible

Transactional Causal Consistency (TCC)

• Strongest AP-compatible model
• Joint Updates / Transactional
• Partial Order / Causal Consistency

Open source, well engineered Community of users

18

.eu

[AntidoteDB & Just-Right Consistency]

Creative Commons Attribution-ShareAlike 4.0

• Intl. License

You are free to:

• Share — copy and redistribute the material in any medium or

format

• Adapt — remix, transform, and build upon the material

for any purpose, even commercially, under the following terms: Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.

19 [AntidoteDB & Just-Right Consistency] 20

[AntidoteDB & Just-Right Consistency] 21

Asynchronous, replicated updates

• State σ
• Invariant I
• Prepare: read one, generate effector
• Update all, deferred: deliver effector

Converge? Invariant OK?

σ: I u! u! u? σ: I v! v! I ? I ?

100 € ≥ 0 100 € ≥ 0 accrue 5% +5 € +5 € debit(100) 5 € ≥ 0 5 € ≥ 0 –100 –100

[AntidoteDB & Just-Right Consistency] 22

CISE Rules 1: Individually Correct

• Individual operations maintain the invariant

2: Convergence

• Concurrent effectors commute

3: Precondition Stability

• Every precondition is stable under every

concurrent operation If satisfied: invariant is guaranteed

σ: I u! u! u? σ: I v! v! I ? I ?

[AntidoteDB & Just-Right Consistency]

Simple example: bank account

Operations: credit(amount), debit(amount) Invariant: balance ≥ 0

• Start with weak specification
• Rule 1 ⟶ strengthen precondition for debit
• Rule 2: OK
• Rule 3 ⟶ debit || debit unsafe, fixed with

concurrency control

23 [AntidoteDB & Just-Right Consistency] 24

CISE Rules 1: Sequential correctness

• Individual operations maintain the invariant

2: Convergence

• Concurrent effectors commute

3: Precondition Stability

• Every precondition is stable under every

concurrent operation If satisfied: invariant is guaranteed

u! u! u? I I uPRE uPRE

• σ: I

σ: I

[AntidoteDB & Just-Right Consistency] 25

CISE Rules 1: Sequential correctness

• Individual operations maintain the invariant

2: Convergence

• Concurrent effectors commute

3: Precondition Stability

• Every precondition is stable under every

concurrent operation If satisfied: invariant is guaranteed

u! u! u? v! v! σ: I σ: I

[AntidoteDB & Just-Right Consistency] 26

CISE Rules 1: Sequential correctness

• Individual operations maintain the invariant

2: Convergence

• Concurrent effectors commute

3: Precondition Stability

• Every precondition is stable under every

concurrent operation If satisfied: invariant is guaranteed

u! u! u? uPRE uPRE ? I

• v!
• σ: I

σ: I

[AntidoteDB & Just-Right Consistency] 27

CISE Rules 1: Sequential correctness

• Individual operations maintain the invariant

2: Convergence

• Concurrent effectors commute

3: Precondition Stability

• Every precondition is stable under every

concurrent operation If satisfied: invariant is guaranteed

balance = 1 balance − 1 debit(1) debitPRE {1 ≤ 1} debitPRE { 1 ≤ 0 } ? balance = 1 balance − 1 balance − 1 debit(1)

Fix: concurrency control

balance = –1

• [AntidoteDB & Just-Right Consistency]

Advanced example: file system

Operations: mkdir, rmdir, mv, write, etc. Invariant: Tree

• Rule 1 ⟶ precondition on mv
• “May not move node under self”
• Rule 2 ⟶ Use CRDTs for write || write
• Rule 3 ⟶ mv || mv precondition unstable

28

[AntidoteDB & Just-Right Consistency]

Advanced example: file system

Operations: mkdir, rmdir, mv, update, etc. Invariant: Tree

• Rule 1 ⟶ precondition on mv
• “May not move node under self”
• Rule 2 ⟶ Use CRDTs for update || update
• Rule 3 ⟶ mv || mv precondition unstable

29 [AntidoteDB & Just-Right Consistency] 30

CISE Rules 1: Sequential correctness

• Individual operations maintain the invariant

2: Convergence

• Concurrent effectors commute

3: Precondition Stability

• Every precondition is stable under every

concurrent operation If satisfied: invariant is guaranteed

mv /B, /A mvPRE {¬B/.../A} mvPRE {¬B/.../A} ? mv /A, /B

Fix: concurrency control

root B A root B A

root B A root B A

• You can

have your cake and eat it too

[AntidoteDB & Just-Right Consistency]

Applying the logic

Only O(n2): no need to consider all possible interleavings We use a tool

• You can apply the same logic manually

31

Antidote Protocols

[AntidoteDB & Just-Right Consistency]

Architecture

33

Total

• rder

ClockSI Sharded, parallel Causal order Anchorage: 10 Brussels: 15 Capetown: 22 Anchorage Brussels Capetown

[AntidoteDB & Just-Right Consistency]

T1 Coordinator

Per-DC Total Order

RT clocks, timestamped versions

• 2PC with no aborts
• disjoint-access parallelism
• 1 scalar / DC

34

begin; x.add(Bob); y.add(Alice); commit y.add(Alice) x.add(Bob) T1: Snapshot = [3, 2, 5] c T S = 8 cTS = 9 T1: commit = [3,2,9] Stable time at T1’s server

X Y, Z

Cindy @Capetown

[AntidoteDB & Just-Right Consistency]

Geo-replication: Causal

All-or-nothing: all updates in a transaction have the same timestamp Causally-consistent snapshot: vector timestamp Transaction-consistent snapshot: includes all transactions T for which commit_timestamp(T) ≤ snapshot timestamp

35 [AntidoteDB & Just-Right Consistency]

Snapshot

• Small metadata: dependency stabilization protocol

(GentleRain [SoCC’14])

36

Friends List x: Alice’s. y: Bob’s. z: Tyler’s.

X Y, Z

Small metadata, DC dependent progress

x2 x1 y2 z1 Tx2 Tx1

ST = min(V)

CT =[1,0,0] CT =[2,0,0]

x1 y2 x2 y2 x2 x1

Friends List x: Alice’s. y: Bob’s. z: Tyler’s.

Y, Z X [0,1,0] [1,4,1] z1 z1

ST = [0,1,0]

[0,1,0]

ST = [0,3,0] ST = [1,4,1]

[2,4,1] [2,5,1][2,5,1] [1,2,1] [0,2,0]

ST = [0,2,0] ST = [2,5,1]

Y, Z X 1 1 2 2

Bob@ Brussels Alice@ Anchorage Cindy@ Capetown

[AntidoteDB & Just-Right Consistency]

Scalability

Alice: tx1: update(x.remove(Tyler)), z.remove(Alice)), tx2: update(y.add(Bob)), update(x.add(Alice))

37

Kops / s 100 200 300 400 500 600 700 800 1 x 5 1 x 10 1 x 25 2 x 25 3 x 25 1 x 5 1 x 10 1 x 25 2 x 25 3 x 25 1 x 5 1 x 10 1 x 25 2 x 25 3 x 25 1 x 5 1 x 10 1 x 25 2 x 25 3 x 25 99(1) 90(10) 75(25) 50(50) read(update) ratio DCs × Servers

LWW registers 100k keys/partition power law distribution

[AntidoteDB & Just-Right Consistency]

Cure vs. SOA: LWW

Alice: tx1: update(x.remove(Tyler)), z.remove(Alice)), tx2: update(y.add(Bob)), update(x.add(Alice))

38

Kops / s

100 200 300 400 500 600 700 800 900 1000 1100 Eiger GR Cure EC Eiger GR Cure EC Eiger GR Cure EC Eiger GR Cure EC 99(1) 90(10) 75(25) 50(50) read(update) ratio

3 DCs × 25 Servers

[AntidoteDB & Just-Right Consistency]

Cure vs. EC: CRDT sets

Alice: tx1: update(x.remove(Tyler)), z.remove(Alice)), tx2: update(y.add(Bob)), update(x.add(Alice))

39

Kops / s 200 400 600 800 1000 1200 Cure, 1KB EC, 1KB Cure, 10KB EC, 10KB Cure, 1KB EC, 1KB Cure, 10KB EC, 10KB Cure, 1KB EC, 1KB Cure, 10KB EC, 10KB Cure, 1KB EC, 1KB Cure, 10KB EC, 10KB 99(1) 90(10) 75(25) 50(50) read(update) ratio

3 DCs x 25 Servers