Reasoning about consistency choices in distributed systems Hongseok - - PowerPoint PPT Presentation

▶
reasoning about consistency choices in distributed systems
SMART_READER_LITE
LIVE PREVIEW

Reasoning about consistency choices in distributed systems Hongseok - - PowerPoint PPT Presentation

Dec 31, 2023 179 likes •835 views

Reasoning about consistency choices in distributed systems Hongseok Yang University of Oxford Joint work with Alexey Gotsman (IMDEA, Spain), Carla Ferreira (U Nova Lisboa), Mahsa Najafzadeh, Marc Shapiro (INRIA) Global-scale Internet service

slide-1
SLIDE 1

Reasoning about consistency choices in distributed systems

Hongseok Yang University of Oxford Joint work with Alexey Gotsman (IMDEA, Spain), Carla Ferreira (U Nova Lisboa), Mahsa Najafzadeh, Marc Shapiro (INRIA)

slide-2
SLIDE 2

Global-scale Internet service

slide-3
SLIDE 3

Geo-replicated databases

  • Every data centre stores a complete replica of data
  • Purpose: Minimising latency. Fault tolerance.
slide-4
SLIDE 4

Geo-replicated databases

  • Every data centre stores a complete replica of data
  • Purpose: Minimising latency. Fault tolerance.
slide-5
SLIDE 5

Geo-replicated databases ✘

  • Every data centre stores a complete replica of data
  • Purpose: Minimising latency. Fault tolerance.
slide-6
SLIDE 6

Weakly consistent DBs ✘

First update. Propagate later.

{(A,4)} {(A,4)} {(A,4)}

slide-7
SLIDE 7

Weakly consistent DBs

cart.rem(A,2) cart.read() : {A}

✘

{(A,4)} {(A,4)} {(A,2)}

First update. Propagate later.

slide-8
SLIDE 8

Weakly consistent DBs

cart.rem(A,2) cart.read() : {A}

✘

{(A,4)} {(A,4)} {(A,2)}

First update. Propagate later.

slide-9
SLIDE 9

Weakly consistent DBs

cart.rem(A,2) cart.read() : {A}

✘

{(A,4)} {(A,2)} {(A,2)}

First update. Propagate later.

slide-10
SLIDE 10

Weakly consistent DBs

cart.rem(A,2) cart.count(A): 4

✘

{(A,4)} {(A,2)} {(A,2)}

Issue 1: Anomalies First update. Propagate later.

slide-11
SLIDE 11

Weakly consistent DBs

cart.rem(A,2) cart.count(A): 4

✘

{(A,0)} {(A,2)} {(A,2)}

Issue 2: Conflicting updates

cart.remAll(A)

First update. Propagate later.

slide-12
SLIDE 12

Weakly consistent DBs

cart.rem(A,2) cart.count(A): 4

✘

{(A,0)} {(A,2)} {(A,2)} cart.remAll(A) remAll(A) rem(A,2)

First update. Propagate later. Issue 2: Conflicting updates

slide-13
SLIDE 13

How to develop correct programs running on top of weakly consistent distributed databases?

slide-14
SLIDE 14

How to develop correct programs running on top of weakly consistent distributed databases?

  • 1. Strengthen consistency selectively.
  • 2. Use rely-guarantee reasoning.
slide-15
SLIDE 15

How to develop correct programs running on top of weakly consistent distributed databases?

  • 1. Strengthen consistency selectively.
  • 2. Prove the correctness of a program.
slide-16
SLIDE 16

Simple bank account

class account { // invariant: amount >= 0 var amount = 0 def query() = { return amount } def inc() = { amount = amount+1; return true } def dec() = { if (amount > 0) { amount = amount-1; return true } else { return false } } }

slide-17
SLIDE 17

Distributed bank account

class account { // invariant: amount >= 0 var[dis] amount = 0 def query() = { return (amount, (a)=>a) } def inc() = { amount = amount+1; return (true, (a)=>a+1) } def dec() = { if (amount > 0) { amount = amount-1; return (true, (a)=>a-1) } else { return (false, (a)=>a) } } }

slide-18
SLIDE 18

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

slide-19
SLIDE 19

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

slide-20
SLIDE 20

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

slide-21
SLIDE 21

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

a++ a++

slide-22
SLIDE 22

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

a++ a++ a— a—

slide-23
SLIDE 23

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

a++ a++ a— a— skip

slide-24
SLIDE 24

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

slide-25
SLIDE 25

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

a++ a++

slide-26
SLIDE 26

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

a++ a++ a—

slide-27
SLIDE 27

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

a++ a++ a— a—

slide-28
SLIDE 28

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

a++ a++ a— a—

slide-29
SLIDE 29

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

a++ a— a—

slide-30
SLIDE 30

How to write correct prog.?

  • 1. Strengthen consistency selectively.
  • 2. Prove the correctness of a program.
slide-31
SLIDE 31

Causal consistency

  • Message delivery preserves the dependency
  • f events.

Axiom: HB is transitive.

slide-32
SLIDE 32

dec() query() Bob in UK Alice in Korea inc() dec() Carol in USA

a++ a— a— a++

[Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible

slide-33
SLIDE 33

dec() query() Bob in UK Alice in Korea inc() dec() Carol in USA

a++ a— a— a++

Not causally consistent. [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible

slide-34
SLIDE 34

use causality class account { // invariant: amount >= 0 var[dis] amount = 0 def query() = { return (amount, (a)=>a) } def inc() = { amount = amount+1; return (true, (a)=>a+1) } def dec() = { if (amount > 0) { amount = amount-1; return (true, (a)=>a-1) } else { return (false, (a)=>a) } } }

slide-35
SLIDE 35

Token system

  • (T, 💕) where 💕 is a symmetric rel. on T.
  • Examples:
  • 1. T = {lock}, 💕 = {(lock,lock)}
  • 2. T = {rd,wr}, 💕 = {(rd,wr), (wr,wr), (wr,rd)}
slide-36
SLIDE 36

On-demand consistency using a token system (T, 💕)

  • Each operation acquires a set of tokens.
  • Operations with conflicting tokens cannot

be run concurrently.

slide-37
SLIDE 37

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

a++ a++ a— a— a—

slide-38
SLIDE 38

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

a++ a++ a— a—

{lock} {lock} {} {} T = {lock} 💕 = {(lock, lock)}

a—

slide-39
SLIDE 39

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

a++ a++ a— a—

{lock} {lock} {} {} T = {lock} 💕 = {(lock, lock)}

a—

slide-40
SLIDE 40

dec() query() [Q] What cannot be the result of query()? (a) 0 (b) -1 (c) -2 (d) all possible Bob in UK Alice in Korea inc() dec() Carol in USA

a++ a++ a— a—

{lock} {lock} {} {} T = {lock} 💕 = {(lock, lock)}

a— skip

slide-41
SLIDE 41

use causality class account { // invariant: amount >= 0 var[dis] amount = 0 use-token-system({lock},{(lock,lock)}) def query() with {} = { return (amount, (a)=>a) } def inc() with {} = { amount = amount+1; return (true, (a)=>a+1) } def dec() with {lock} = { if (amount > 0) { amount = amount-1; return (true, (a)=>a-1) } else { return (false, (a)=>a) } } }

slide-42
SLIDE 42

How to write correct prog.?

  • 1. Strengthen consistency selectively.
  • 2. Prove the correctness of a program.
slide-43
SLIDE 43

Our proof rule

  • Based on rely-guarantee.
  • Incorporates guarantees from causal and
  • n-demand consistency.
slide-44
SLIDE 44

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())
slide-45
SLIDE 45

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())
slide-46
SLIDE 46

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())
slide-47
SLIDE 47

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())
slide-48
SLIDE 48

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())
slide-49
SLIDE 49

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())
slide-50
SLIDE 50

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())

. . . . . .

slide-51
SLIDE 51

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())

. . . . . .

slide-52
SLIDE 52

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())

. . . . . .

slide-53
SLIDE 53

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())

. . . . . .

slide-54
SLIDE 54

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())

. . .

slide-55
SLIDE 55

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())

. . . . . . T⊥ = {𝝊 | ∄𝝊’∈T. (𝝊, 𝝊ʹ) ∈ 💕}

slide-56
SLIDE 56

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())

. . . . . . I = { σ | 0 ≤ σ } G0 = {(σ, σ’) | σ ≤ σ’} G1(lock) = {(σ, σ’) | 0 < σ ∧ σ’ ≤ σ}

slide-57
SLIDE 57

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())

. . . . . . I = { σ | 0 ≤ σ } G0 = {(σ, σ’) | σ ≤ σ’} G1(lock) = {(σ, σ’) | 0 < σ ∧ σ’ ≤ σ} dec()

slide-58
SLIDE 58

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())

. . . . . . {} {lock} I = { σ | 0 ≤ σ } G0 = {(σ, σ’) | σ ≤ σ’} G1(lock) = {(σ, σ’) | 0 < σ ∧ σ’ ≤ σ} dec()

slide-59
SLIDE 59

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())

. . . . . . I = { σ | 0 ≤ σ } G0 = {(σ, σ’) | σ ≤ σ’} G1(lock) = {(σ, σ’) | 0 < σ ∧ σ’ ≤ σ} dec() {} {lock} G* G0 ∪ G1(lock)

slide-60
SLIDE 60

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())

. . . . . . I = { σ | 0 ≤ σ } G0 = {(σ, σ’) | σ ≤ σ’} G1(lock) = {(σ, σ’) | 0 < σ ∧ σ’ ≤ σ} dec() {} {lock} G* G0 ∪ G1(lock)

slide-61
SLIDE 61

To prove that I is an invariant

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())

. . . . . . I = { σ | 0 ≤ σ } G0 = {(σ, σ’) | σ ≤ σ’} G1(lock) = {(σ, σ’) | 0 < σ ∧ σ’ ≤ σ} dec() {} {lock} G* G0 ∪ G1(lock) if 0 < σ then σ’-1 else σ’

slide-62
SLIDE 62

What if no on-demand consistency?

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())
slide-63
SLIDE 63

What if no on-demand consistency?

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())

G* G

slide-64
SLIDE 64

What if no causality?

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())

G* G

slide-65
SLIDE 65

What if no causality?

9G0 2 P(State ⇥ State), G 2 Token ! P(State ⇥ State) such that

  • S1. init 2 I
  • S2. G0(I) ✓ I ^ 8⌧. G(⌧)(I) ✓ I
  • S3. 8o, , 0. ( 2 I ^ (, 0) 2 (G0 [ G((Ftok
  • ())?))⇤)

= ) (0, Feff

  • ()(0)) 2 G0 [ G(Ftok
  • ())

σ’ ∈ I Feff(σ)(σ’) ∈ I