SLIDE 1
Distributed Storage Systems
2
Consistency-Aware Durability Aishwarya Ganesan, Ram Alagappan, - - PowerPoint PPT Presentation
Consistency-Aware Durability Aishwarya Ganesan, Ram Alagappan, - - PowerPoint PPT Presentation
Strong and Efficient Consistency with Consistency-Aware Durability Aishwarya Ganesan, Ram Alagappan, Andrea Arpaci-Dusseau, and Remzi Arpaci-Dusseau Distributed Storage Systems 2 Consistency Models in Distributed Systems 3 Consistency Models
SLIDE 2
SLIDE 3
Consistency Models in Distributed Systems
3
SLIDE 4
Consistency Models in Distributed Systems
3
What does a read see given a previous set of reads and writes?
SLIDE 5
Consistency Models in Distributed Systems
3
What does a read see given a previous set of reads and writes?
linearizability strong
SLIDE 6
Consistency Models in Distributed Systems
3
What does a read see given a previous set of reads and writes?
linearizability eventual strong weak
SLIDE 7
Consistency Models in Distributed Systems
3
What does a read see given a previous set of reads and writes?
linearizability eventual sequential causal+ PRAM monotonic reads causal strong weak
SLIDE 8
Consistency Models in Distributed Systems
3
What does a read see given a previous set of reads and writes?
linearizability eventual sequential causal+ PRAM monotonic reads causal strong weak
Well studied and understood!
SLIDE 9
Unlike consistency models, scant attention to durability model!
Durability Models
4
SLIDE 10
Unlike consistency models, scant attention to durability model! How writes are replicated and persisted
Durability Models
4
SLIDE 11
Unlike consistency models, scant attention to durability model! How writes are replicated and persisted Durability model influences consistency
Durability Models
4
SLIDE 12
Unlike consistency models, scant attention to durability model! How writes are replicated and persisted Durability model influences consistency Also determines performance
Durability Models
4
SLIDE 13
Unlike consistency models, scant attention to durability model! How writes are replicated and persisted Durability model influences consistency Also determines performance
Durability Models
4
Despite this importance, often overlooked!
SLIDE 14
Two Widely Used Durability Models
5
SLIDE 15
Two Widely Used Durability Models
5
Immediate durability
SLIDE 16
Two Widely Used Durability Models
5
client write node-1 node-2 node-3
Immediate durability
SLIDE 17
Two Widely Used Durability Models
5
replicate
client write node-1 node-2 node-3
Immediate durability
SLIDE 18
Two Widely Used Durability Models
5
replicate persist
client write node-1 node-2 node-3
Immediate durability
fsync fsync fsync
SLIDE 19
Two Widely Used Durability Models
5
replicate persist
client write node-1 node-2 node-3
Immediate durability
ack fsync fsync fsync
SLIDE 20
Two Widely Used Durability Models
5
replicate persist
client write node-1 node-2 node-3
Immediate durability
ack
enables strong consistency but too slow!
fsync fsync fsync
SLIDE 21
Two Widely Used Durability Models
5
replicate persist
client write node-1 node-2 node-3
Immediate durability
ack
enables strong consistency but too slow! Eventual durability
fsync fsync fsync
SLIDE 22
Two Widely Used Durability Models
5
replicate persist
client write node-1 node-2 node-3
Immediate durability
ack
enables strong consistency but too slow!
client write node-1 node-2 node-3
Eventual durability
fsync fsync fsync ack
SLIDE 23
Two Widely Used Durability Models
5
replicate persist
client write node-1 node-2 node-3
Immediate durability
ack
enables strong consistency but too slow!
client write node-1 node-2 node-3
Eventual durability
lazily replicate and persist
fsync fsync fsync ack
SLIDE 24
Two Widely Used Durability Models
5
replicate persist
client write node-1 node-2 node-3
Immediate durability
ack
enables strong consistency but too slow!
client write node-1 node-2 node-3
Eventual durability fast but enables only weak consistency due to data loss upon failures!
lazily replicate and persist
fsync fsync fsync ack
SLIDE 25
Is it possible for a durability layer to enable both strong consistency and high performance?
6
SLIDE 26
CAD: Consistency-aware Durability
7
SLIDE 27
Design the durability layer by taking the consistency model into account
CAD: Consistency-aware Durability
7
SLIDE 28
Design the durability layer by taking the consistency model into account Intuition: what a read sees is important for most consistency models
CAD: Consistency-aware Durability
7
SLIDE 29
Design the durability layer by taking the consistency model into account Intuition: what a read sees is important for most consistency models Key idea: CAD shifts the point of durability to reads from writes
data is replicated and persisted before a read is served
CAD: Consistency-aware Durability
7
SLIDE 30
Design the durability layer by taking the consistency model into account Intuition: what a read sees is important for most consistency models Key idea: CAD shifts the point of durability to reads from writes
data is replicated and persisted before a read is served delayed writes → high performance
CAD: Consistency-aware Durability
7
SLIDE 31
Design the durability layer by taking the consistency model into account Intuition: what a read sees is important for most consistency models Key idea: CAD shifts the point of durability to reads from writes
data is replicated and persisted before a read is served delayed writes → high performance data durable before it is read → strong consistency even under failures
CAD: Consistency-aware Durability
7
SLIDE 32
Design the durability layer by taking the consistency model into account Intuition: what a read sees is important for most consistency models Key idea: CAD shifts the point of durability to reads from writes
data is replicated and persisted before a read is served delayed writes → high performance data durable before it is read → strong consistency even under failures lose some writes if failures arise before read; but, useful for many systems that use eventual durability
CAD: Consistency-aware Durability
7
SLIDE 33
Design the durability layer by taking the consistency model into account Intuition: what a read sees is important for most consistency models Key idea: CAD shifts the point of durability to reads from writes
data is replicated and persisted before a read is served delayed writes → high performance data durable before it is read → strong consistency even under failures lose some writes if failures arise before read; but, useful for many systems that use eventual durability
We show efficacy of CAD by providing cross-client monotonic reads
a new and strong consistency property
CAD: Consistency-aware Durability
7
SLIDE 34
Results
8
SLIDE 35
ORCA: CAD and cross-client monotonic reads for leader-based systems
implemented in ZooKeeper
Results
8
SLIDE 36
ORCA: CAD and cross-client monotonic reads for leader-based systems
implemented in ZooKeeper
Compared to strongly consistent ZooKeeper
ORCA is 1.6 – 3.3x faster by using CAD higher read throughput by allowing reads at many nodes reduces latency in geo-distributed settings by 14x
Results
8
SLIDE 37
ORCA: CAD and cross-client monotonic reads for leader-based systems
implemented in ZooKeeper
Compared to strongly consistent ZooKeeper
ORCA is 1.6 – 3.3x faster by using CAD higher read throughput by allowing reads at many nodes reduces latency in geo-distributed settings by 14x
Compared to weakly consistent ZooKeeper
ORCA provides similar throughput and latency but with stronger guarantees
Results
8
SLIDE 38
ORCA: CAD and cross-client monotonic reads for leader-based systems
implemented in ZooKeeper
Compared to strongly consistent ZooKeeper
ORCA is 1.6 – 3.3x faster by using CAD higher read throughput by allowing reads at many nodes reduces latency in geo-distributed settings by 14x
Compared to weakly consistent ZooKeeper
ORCA provides similar throughput and latency but with stronger guarantees
Experimentally show ORCA’s guarantees under failures, useful for apps
Results
8
SLIDE 39
Outline
Introduction Motivation CAD and cross-client monotonic reads ORCA design Results Summary and conclusion
9
SLIDE 40
Consistency Models and Guarantees
10
Example: I’m bored at FAST and want to go home!
SLIDE 41
Consistency Models and Guarantees
10
Example: I’m bored at FAST and want to go home!
SLIDE 42
Consistency Models and Guarantees
10
Example: I’m bored at FAST and want to go home!
SLIDE 43
Consistency Models and Guarantees
10
Example: I’m bored at FAST and want to go home!
SLIDE 44
Consistency Models and Guarantees
10
Example: I’m bored at FAST and want to go home!
Linearizability
SLIDE 45
Consistency Models and Guarantees
10
Example: I’m bored at FAST and want to go home!
Linearizability
SLIDE 46
Consistency Models and Guarantees
10
Example: I’m bored at FAST and want to go home!
Linearizability
SLIDE 47
Consistency Models and Guarantees
10
Example: I’m bored at FAST and want to go home!
Linearizability
latest data: no staleness
SLIDE 48
Consistency Models and Guarantees
10
Example: I’m bored at FAST and want to go home! Linearizability
latest data: no staleness in-order reads across clients
SLIDE 49
Consistency Models and Guarantees
10
Example: I’m bored at FAST and want to go home! Linearizability
latest data: no staleness in-order reads across clients
Weaker models
SLIDE 50
Consistency Models and Guarantees
10
Example: I’m bored at FAST and want to go home! Linearizability
latest data: no staleness in-order reads across clients
Weaker models
SLIDE 51
Consistency Models and Guarantees
10
Example: I’m bored at FAST and want to go home! Linearizability
latest data: no staleness in-order reads across clients
Weaker models
SLIDE 52
Consistency Models and Guarantees
10
Example: I’m bored at FAST and want to go home! Linearizability
latest data: no staleness in-order reads across clients
Weaker models
stale reads
- ut-of-order reads across clients
SLIDE 53
Consistency Models and Guarantees
10
Example: I’m bored at FAST and want to go home! Linearizability
latest data: no staleness in-order reads across clients
Weaker models
stale reads
- ut-of-order reads across clients
even with monotonic reads and causal
SLIDE 54
Linearizability requires immediate durability
must synchronously replicate and persist data on a majority to tolerate failures
11
Realizing Strong Consistency
SLIDE 55
Linearizability requires immediate durability
must synchronously replicate and persist data on a majority to tolerate failures
11
Realizing Strong Consistency
leader – S1 S2 S3 a0 a0 a0
- n disk
in memory durable = on disk on a majority
SLIDE 56
Linearizability requires immediate durability
must synchronously replicate and persist data on a majority to tolerate failures
11
Realizing Strong Consistency
leader – S1 S2 S3 a0 a0 a0
- n disk
in memory durable = on disk on a majority
SLIDE 57
Linearizability requires immediate durability
must synchronously replicate and persist data on a majority to tolerate failures
11
Realizing Strong Consistency
leader – S1 S2 S3 a0 a0 a0 a1
- n disk
in memory durable = on disk on a majority
a1 a1
SLIDE 58
Linearizability requires immediate durability
must synchronously replicate and persist data on a majority to tolerate failures
11
Realizing Strong Consistency
leader – S1 S2 S3 a0 a0 a0 a1
- n disk
in memory durable = on disk on a majority
a1 a1 a1 a1
SLIDE 59
Linearizability requires immediate durability
must synchronously replicate and persist data on a majority to tolerate failures
11
Realizing Strong Consistency
leader – S1 S2 S3 a0 a0 a0 a1
- n disk
in memory durable = on disk on a majority
a1 a1 a1 a1
SLIDE 60
Linearizability requires immediate durability
must synchronously replicate and persist data on a majority to tolerate failures
11
Realizing Strong Consistency
leader – S1 S2 S3 a0 a0 a0 a1
- n disk
in memory durable = on disk on a majority
a1 a1 a1 a1
SLIDE 61
Linearizability requires immediate durability
must synchronously replicate and persist data on a majority to tolerate failures
11
Realizing Strong Consistency
leader – S1 S2 S3 a0 a0 a0 a1
- n disk
in memory durable = on disk on a majority
a1 a1 a1
SLIDE 62
Linearizability requires immediate durability
must synchronously replicate and persist data on a majority to tolerate failures
11
Realizing Strong Consistency
leader – S1 S2 S3 a0 a0 a0 a1
- n disk
in memory durable = on disk on a majority
a1 a1 a1
SLIDE 63
Linearizability requires immediate durability
must synchronously replicate and persist data on a majority to tolerate failures
11
Realizing Strong Consistency
leader – S1 S2 S3 a0 a0 a0 a1
- n disk
in memory durable = on disk on a majority
a1 a1 a1
SLIDE 64
Linearizability requires immediate durability
must synchronously replicate and persist data on a majority to tolerate failures
11
Realizing Strong Consistency
Poor performance due to synchronous operations
10x slower within data center
leader – S1 S2 S3 a0 a0 a0 a1
- n disk
in memory durable = on disk on a majority
a1 a1 a1
SLIDE 65
Weaker models only require eventual durability
data buffered on one node, replication and persistence in background
12
Realizing Weaker Models
S1 S2 S3 a0 a0 a0
SLIDE 66
Weaker models only require eventual durability
data buffered on one node, replication and persistence in background
12
Realizing Weaker Models
S1 S2 S3 a0 a0 a0 app session-1
SLIDE 67
Weaker models only require eventual durability
data buffered on one node, replication and persistence in background
12
Realizing Weaker Models
S1 S2 S3 a0 a0 a0 a1 app session-1
SLIDE 68
Weaker models only require eventual durability
data buffered on one node, replication and persistence in background
12
Realizing Weaker Models
S1 S2 S3 a0 a0 a0 a1 app session-1
SLIDE 69
Weaker models only require eventual durability
data buffered on one node, replication and persistence in background
12
Realizing Weaker Models
S1 S2 S3 a0 a0 a0 a1 app session-1
SLIDE 70
Weaker models only require eventual durability
data buffered on one node, replication and persistence in background
12
Realizing Weaker Models
S1 S2 S3 a0 a0 a0 app session-1
SLIDE 71
Weaker models only require eventual durability
data buffered on one node, replication and persistence in background
12
Realizing Weaker Models
S1 S2 S3 a0 a0 a0 app session-1 S1 S2 S3 a0 a0 a0
SLIDE 72
Weaker models only require eventual durability
data buffered on one node, replication and persistence in background
12
Realizing Weaker Models
S1 S2 S3 a0 a0 a0 app session-1 S1 S2 S3 a0 a0 a0 app session-2
SLIDE 73
Weaker models only require eventual durability
data buffered on one node, replication and persistence in background
12
Realizing Weaker Models
S1 S2 S3 a0 a0 a0 app session-1 S1 S2 S3 a0 a0 a0 app session-2
SLIDE 74
Weaker models only require eventual durability
data buffered on one node, replication and persistence in background
12
- ut-of-order across
clients valid under causal and monotonic reads but confusing semantics
Realizing Weaker Models
S1 S2 S3 a0 a0 a0 app session-1 S1 S2 S3 a0 a0 a0 app session-2
SLIDE 75
Weaker models only require eventual durability
data buffered on one node, replication and persistence in background
12
- ut-of-order across
clients valid under causal and monotonic reads but confusing semantics
Many deployments prefer eventual durability for performance
in fact, it is the default (e.g., MongoDB, Redis)
Realizing Weaker Models
S1 S2 S3 a0 a0 a0 app session-1 S1 S2 S3 a0 a0 a0 app session-2
SLIDE 76
Weaker models only require eventual durability
data buffered on one node, replication and persistence in background
12
- ut-of-order across
clients valid under causal and monotonic reads but confusing semantics
Many deployments prefer eventual durability for performance
in fact, it is the default (e.g., MongoDB, Redis)
Thus settle for weak consistency
Realizing Weaker Models
S1 S2 S3 a0 a0 a0 app session-1 S1 S2 S3 a0 a0 a0 app session-2
SLIDE 77
Immediate durability enables strong consistency but is slow Eventual durability is fast but enables only weaker consistency
SLIDE 78
Outline
Introduction Motivation CAD and cross-client monotonic reads ORCA design Results Summary and conclusion
13
SLIDE 79
Consistency-aware Durability
14
SLIDE 80
Most consistency models care about what reads see
Consistency-aware Durability
14
SLIDE 81
Most consistency models care about what reads see Key idea: CAD shifts the point of durability to reads from writes
Consistency-aware Durability
14
SLIDE 82
Most consistency models care about what reads see Key idea: CAD shifts the point of durability to reads from writes
Consistency-aware Durability
14
delay durability of writes
SLIDE 83
Most consistency models care about what reads see Key idea: CAD shifts the point of durability to reads from writes
Consistency-aware Durability
14
client write S1 S2 S3
delay durability of writes
SLIDE 84
Most consistency models care about what reads see Key idea: CAD shifts the point of durability to reads from writes
Consistency-aware Durability
14
client write S1 S2 S3
delay durability of writes
SLIDE 85
Most consistency models care about what reads see Key idea: CAD shifts the point of durability to reads from writes
Consistency-aware Durability
14
client write S1 S2 S3 ack
delay durability of writes
SLIDE 86
Most consistency models care about what reads see Key idea: CAD shifts the point of durability to reads from writes
Consistency-aware Durability
14
client write S1 S2 S3 ack
delay durability of writes
good performance
SLIDE 87
Most consistency models care about what reads see Key idea: CAD shifts the point of durability to reads from writes
Consistency-aware Durability
14
client write S1 S2 S3 ack
delay durability of writes
good performance
make data durable before serving reads
SLIDE 88
Most consistency models care about what reads see Key idea: CAD shifts the point of durability to reads from writes
Consistency-aware Durability
14
client write S1 S2 S3 ack
delay durability of writes
good performance
make data durable before serving reads
client S1 S2 S3 read
SLIDE 89
Most consistency models care about what reads see Key idea: CAD shifts the point of durability to reads from writes
Consistency-aware Durability
14
client write S1 S2 S3 ack
delay durability of writes
good performance
make data durable before serving reads
client S1 S2 S3 read
SLIDE 90
Most consistency models care about what reads see Key idea: CAD shifts the point of durability to reads from writes
Consistency-aware Durability
14
client write S1 S2 S3 ack
delay durability of writes
good performance
make data durable before serving reads
client S1 S2 S3 read
SLIDE 91
Most consistency models care about what reads see Key idea: CAD shifts the point of durability to reads from writes
Consistency-aware Durability
14
client write S1 S2 S3 ack
delay durability of writes
good performance
make data durable before serving reads
client S1 S2 S3 read
SLIDE 92
Most consistency models care about what reads see Key idea: CAD shifts the point of durability to reads from writes
Consistency-aware Durability
14
client write S1 S2 S3 ack
delay durability of writes
good performance
make data durable before serving reads
client S1 S2 S3
prevents out-of-order data across failures strong consistency
read
SLIDE 93
Most consistency models care about what reads see Key idea: CAD shifts the point of durability to reads from writes
Consistency-aware Durability
14
client write S1 S2 S3 ack
delay durability of writes CAD does not always incur overheads on reads
reads do not immediately follow writes – natural in many workloads common case: data already durable well before applications access it good performance
make data durable before serving reads
client S1 S2 S3
prevents out-of-order data across failures strong consistency
read
SLIDE 94
Cross-client Monotonic Reads upon CAD
15
SLIDE 95
Cross-client Monotonic Reads upon CAD
A read from a client guaranteed to return at least the latest state returned to a previous read from any client
15
SLIDE 96
Cross-client Monotonic Reads upon CAD
A read from a client guaranteed to return at least the latest state returned to a previous read from any client
15
a0 a1 a2
SLIDE 97
Cross-client Monotonic Reads upon CAD
A read from a client guaranteed to return at least the latest state returned to a previous read from any client
15
a0 a1
client-1
a2 a2
SLIDE 98
Cross-client Monotonic Reads upon CAD
A read from a client guaranteed to return at least the latest state returned to a previous read from any client
15
a0 a1
client-1
a2
client-2
a2 a2
SLIDE 99
Cross-client Monotonic Reads upon CAD
A read from a client guaranteed to return at least the latest state returned to a previous read from any client
15
a0 a1
client-1
a2
client-2
a2
Even in the presence of failures and across client sessions
a2
SLIDE 100
Cross-client Monotonic Reads upon CAD
A read from a client guaranteed to return at least the latest state returned to a previous read from any client
15
a0 a1
client-1
a2
client-2
a2
Even in the presence of failures and across client sessions No existing model provides this guarantee except linearizability but not with high performance
a2
SLIDE 101
Cross-client Monotonic Reads upon CAD
A read from a client guaranteed to return at least the latest state returned to a previous read from any client
15
a0 a1
client-1
a2
client-2
a2
Even in the presence of failures and across client sessions No existing model provides this guarantee except linearizability but not with high performance CAD enables this property with high performance
a2
SLIDE 102
Cross-client Monotonic Reads upon CAD
A read from a client guaranteed to return at least the latest state returned to a previous read from any client
15
a0 a1
client-1
a2
client-2
a2
Even in the presence of failures and across client sessions No existing model provides this guarantee except linearizability but not with high performance CAD enables this property with high performance Does not prevent staleness like many weaker models
a2
SLIDE 103
Cross-client Monotonic Reads upon CAD
A read from a client guaranteed to return at least the latest state returned to a previous read from any client
15
a0 a1
client-1
a2
client-2
a2
Even in the presence of failures and across client sessions No existing model provides this guarantee except linearizability but not with high performance CAD enables this property with high performance Does not prevent staleness like many weaker models However, avoids out-of-order data, useful in many app scenarios
e.g., location-sharing, twitter timelines a2
SLIDE 104
Outline
Introduction Motivation CAD and cross-client monotonic reads ORCA design Results Summary and conclusion
16
SLIDE 105
ORCA
17
SLIDE 106
ORCA
Implementation of consistency-aware durability and cross-client monotonic reads in leader-based majority systems
17
SLIDE 107
ORCA
Implementation of consistency-aware durability and cross-client monotonic reads in leader-based majority systems Leader-based systems (e.g., MongoDB, ZooKeeper)
leader – a dedicated node
- thers are followers
writes flow through leader, establishes a single order
17
SLIDE 108
ORCA
Implementation of consistency-aware durability and cross-client monotonic reads in leader-based majority systems Leader-based systems (e.g., MongoDB, ZooKeeper)
leader – a dedicated node
- thers are followers
writes flow through leader, establishes a single order
Majority
data is safe when persisted on majority nodes (e.g., 3 out of 5 servers)
17
SLIDE 109
ORCA Write Path
18
SLIDE 110
ORCA Write Path
Same as an eventually durable system
18
leader – S1 S2 S3 a a a a0
- n disk
in memory durable = on disk on a majority
SLIDE 111
ORCA Write Path
Same as an eventually durable system
18
leader – S1 S2 S3 a a a a0
- n disk
in memory durable = on disk on a majority
SLIDE 112
ORCA Write Path
Same as an eventually durable system
18
leader – S1 S2 S3 a a a b a0
- n disk
in memory durable = on disk on a majority
SLIDE 113
ORCA Write Path
Same as an eventually durable system
18
leader – S1 S2 S3 a a a b a0
- n disk
in memory durable = on disk on a majority
SLIDE 114
ORCA Write Path
Same as an eventually durable system
18
replication and persistence in background
leader – S1 S2 S3 a a a b a0
- n disk
in memory durable = on disk on a majority
b b
SLIDE 115
ORCA Write Path
Same as an eventually durable system
18
replication and persistence in background
leader – S1 S2 S3 a a a b a0
- n disk
in memory durable = on disk on a majority
b b b b b
SLIDE 116
ORCA Read Path
19
- n disk
in memory durable = on disk on a majority
leader – S1 S2 S3 a a a b
SLIDE 117
ORCA Read Path
19
- n disk
in memory durable = on disk on a majority
leader – S1 S2 S3 a a a b
SLIDE 118
ORCA Read Path
19
- n disk
in memory durable = on disk on a majority
Durable-index – index of the latest durable item in the system
leader – S1 S2 S3 a a a b
SLIDE 119
ORCA Read Path
19
- n disk
in memory durable = on disk on a majority
Durable-index – index of the latest durable item in the system Update-index of item i -- index of the last update that modified i
leader – S1 S2 S3 a a a b
SLIDE 120
ORCA Read Path
19
- n disk
in memory durable = on disk on a majority
Durable-index – index of the latest durable item in the system Update-index of item i -- index of the last update that modified i Durability check – i durable if update-index of i ≤ durable-index of system
leader – S1 S2 S3 a a a b
SLIDE 121
ORCA Read Path
19
durable-index:1 a’s update-index:1 a is durable
- n disk
in memory durable = on disk on a majority
Durable-index – index of the latest durable item in the system Update-index of item i -- index of the last update that modified i Durability check – i durable if update-index of i ≤ durable-index of system
leader – S1 S2 S3 a a a b
SLIDE 122
ORCA Read Path
19
durable-index:1 a’s update-index:1 a is durable serve read immediately
- n disk
in memory durable = on disk on a majority
Durable-index – index of the latest durable item in the system Update-index of item i -- index of the last update that modified i Durability check – i durable if update-index of i ≤ durable-index of system
leader – S1 S2 S3 a a a b
SLIDE 123
ORCA Read Path
19
durable-index:1 a’s update-index:1 a is durable serve read immediately
- n disk
in memory durable = on disk on a majority
Durable-index – index of the latest durable item in the system Update-index of item i -- index of the last update that modified i Durability check – i durable if update-index of i ≤ durable-index of system
a a a b leader – S1 S2 S3 a a a b
SLIDE 124
ORCA Read Path
19
durable-index:1 a’s update-index:1 a is durable serve read immediately durable-index:1 b’s update-index: 2 b is not durable
- n disk
in memory durable = on disk on a majority
Durable-index – index of the latest durable item in the system Update-index of item i -- index of the last update that modified i Durability check – i durable if update-index of i ≤ durable-index of system
a a a b leader – S1 S2 S3 a a a b
SLIDE 125
ORCA Read Path
19
durable-index:1 a’s update-index:1 a is durable serve read immediately durable-index:1 b’s update-index: 2 b is not durable make b durable before serving
- n disk
in memory durable = on disk on a majority
Durable-index – index of the latest durable item in the system Update-index of item i -- index of the last update that modified i Durability check – i durable if update-index of i ≤ durable-index of system
a a a b leader – S1 S2 S3 a a a b
SLIDE 126
ORCA Read Path
19
durable-index:1 a’s update-index:1 a is durable serve read immediately durable-index:1 b’s update-index: 2 b is not durable make b durable before serving
- n disk
in memory durable = on disk on a majority
Durable-index – index of the latest durable item in the system Update-index of item i -- index of the last update that modified i Durability check – i durable if update-index of i ≤ durable-index of system
a a a b leader – S1 S2 S3 a a a b b b
SLIDE 127
ORCA Read Path
19
durable-index:1 a’s update-index:1 a is durable serve read immediately durable-index:1 b’s update-index: 2 b is not durable make b durable before serving
- n disk
in memory durable = on disk on a majority
Durable-index – index of the latest durable item in the system Update-index of item i -- index of the last update that modified i Durability check – i durable if update-index of i ≤ durable-index of system
a a a b leader – S1 S2 S3 a a a b b b b b b
SLIDE 128
ORCA Read Path
19
durable-index:1 a’s update-index:1 a is durable serve read immediately durable-index:1 b’s update-index: 2 b is not durable make b durable before serving
- n disk
in memory durable = on disk on a majority
Durable-index – index of the latest durable item in the system Update-index of item i -- index of the last update that modified i Durability check – i durable if update-index of i ≤ durable-index of system
a a a b leader – S1 S2 S3 a a a b b b b b b
SLIDE 129
ORCA Read Path
19
durable-index:1 a’s update-index:1 a is durable serve read immediately durable-index:1 b’s update-index: 2 b is not durable make b durable before serving
- n disk
in memory durable = on disk on a majority
Durable-index – index of the latest durable item in the system Update-index of item i -- index of the last update that modified i Durability check – i durable if update-index of i ≤ durable-index of system
a a a b leader – S1 S2 S3 a a a b b b b b b a a a b b b b b b
SLIDE 130
ORCA Read Path
19
durable-index:1 a’s update-index:1 a is durable serve read immediately durable-index:1 b’s update-index: 2 b is not durable make b durable before serving
- n disk
in memory durable = on disk on a majority
Durable-index – index of the latest durable item in the system Update-index of item i -- index of the last update that modified i Durability check – i durable if update-index of i ≤ durable-index of system
a a a b leader – S1 S2 S3 a a a b b b b b b a a a b b b b b b
SLIDE 131
ORCA Read Path
19
durable-index:1 a’s update-index:1 a is durable serve read immediately durable-index:1 b’s update-index: 2 b is not durable make b durable before serving
- n disk
in memory durable = on disk on a majority
Durable-index – index of the latest durable item in the system Update-index of item i -- index of the last update that modified i Durability check – i durable if update-index of i ≤ durable-index of system
a a a b leader – S1 S2 S3 a a a b b b b b b a a a b b b b b b
leader – S2
SLIDE 132
ORCA Read Path
19
durable-index:1 a’s update-index:1 a is durable serve read immediately durable-index:1 b’s update-index: 2 b is not durable make b durable before serving
- n disk
in memory durable = on disk on a majority
Durable-index – index of the latest durable item in the system Update-index of item i -- index of the last update that modified i Durability check – i durable if update-index of i ≤ durable-index of system
a a a b leader – S1 S2 S3 a a a b b b b b b a a a b b b b b b
leader – S2
SLIDE 133
Cross-Client Monotonic Reads in ORCA
20
SLIDE 134
Cross-Client Monotonic Reads in ORCA
If reads restricted to leader, CAD provides cross-client monotonic reads
20
SLIDE 135
Cross-Client Monotonic Reads in ORCA
If reads restricted to leader, CAD provides cross-client monotonic reads
not scalable
20
SLIDE 136
Cross-Client Monotonic Reads in ORCA
If reads restricted to leader, CAD provides cross-client monotonic reads
not scalable
Allow reads at followers
lagging followers could cause out-of-order states, CAD is not sufficient
20
SLIDE 137
Cross-Client Monotonic Reads in ORCA
If reads restricted to leader, CAD provides cross-client monotonic reads
not scalable
Allow reads at followers
lagging followers could cause out-of-order states, CAD is not sufficient
20
a1 a2 a1 a1 a1 a1
leader – S1 S5 S2 S3 S4
SLIDE 138
Cross-Client Monotonic Reads in ORCA
If reads restricted to leader, CAD provides cross-client monotonic reads
not scalable
Allow reads at followers
lagging followers could cause out-of-order states, CAD is not sufficient
20
a1 a2 a1 a1 a1 a1
leader – S1 S5 S2 S3 S4
SLIDE 139
Cross-Client Monotonic Reads in ORCA
If reads restricted to leader, CAD provides cross-client monotonic reads
not scalable
Allow reads at followers
lagging followers could cause out-of-order states, CAD is not sufficient
20
a1 a2 a1 a1 a1 a1
leader – S1 S5 S2 S3 S4
SLIDE 140
Cross-Client Monotonic Reads in ORCA
If reads restricted to leader, CAD provides cross-client monotonic reads
not scalable
Allow reads at followers
lagging followers could cause out-of-order states, CAD is not sufficient
20
a1 a2 a1 a1 a1 a1 a2 a2 a2 a2
leader – S1 S5 S2 S3 S4
SLIDE 141
Cross-Client Monotonic Reads in ORCA
If reads restricted to leader, CAD provides cross-client monotonic reads
not scalable
Allow reads at followers
lagging followers could cause out-of-order states, CAD is not sufficient
20
a1 a2 a1 a1 a1 a1 a2 a2 a2 a2
leader – S1 S5 S2 S3 S4
SLIDE 142
Cross-Client Monotonic Reads in ORCA
If reads restricted to leader, CAD provides cross-client monotonic reads
not scalable
Allow reads at followers
lagging followers could cause out-of-order states, CAD is not sufficient
20
a1 a2 a1 a1 a1 a1 a2 a2 a2 a2
leader – S1 S5 S2 S3 S4
SLIDE 143
Cross-Client Monotonic Reads in ORCA
If reads restricted to leader, CAD provides cross-client monotonic reads
not scalable
Allow reads at followers
lagging followers could cause out-of-order states, CAD is not sufficient
20
a1 a2 a1 a1 a1 a1 a2 a2 a2 a2 a1 b a1 a1 a1 a1 a2 a2 a2 a2
leader – S1 S5 S2 S3 S4
SLIDE 144
Cross-Client Monotonic Reads in ORCA
If reads restricted to leader, CAD provides cross-client monotonic reads
not scalable
Allow reads at followers
lagging followers could cause out-of-order states, CAD is not sufficient
20
a1 a2 a1 a1 a1 a1 a2 a2 a2 a2 a1 b a1 a1 a1 a1 a2 a2 a2 a2
leader – S1 S5 S2 S3 S4
SLIDE 145
Cross-Client Monotonic Reads in ORCA
If reads restricted to leader, CAD provides cross-client monotonic reads
not scalable
Allow reads at followers
lagging followers could cause out-of-order states, CAD is not sufficient
20
a1 a2 a1 a1 a1 a1 a2 a2 a2 a2 a1 b a1 a1 a1 a1 a2 a2 a2 a2
leader – S1 S5 S2 S3 S4
SLIDE 146
Cross-Client Monotonic Reads in ORCA
If reads restricted to leader, CAD provides cross-client monotonic reads
not scalable
Allow reads at followers
lagging followers could cause out-of-order states, CAD is not sufficient
20
a1 a2 a1 a1 a1 a1 a2 a2 a2 a2 a1 b a1 a1 a1 a1 a2 a2 a2 a2
leader – S1 S5 S2 S3 S4
SLIDE 147
Cross-Client Monotonic Reads in ORCA
If reads restricted to leader, CAD provides cross-client monotonic reads
not scalable
Allow reads at followers
lagging followers could cause out-of-order states, CAD is not sufficient
20
a1 a2 a1 a1 a1 a1 a2 a2 a2 a2 a1 b a1 a1 a1 a1 a2 a2 a2 a2
leader – S1 S5 S2 S3 S4
Additional mechanisms: Active sets (lease-based mechanism), not in this talk…
SLIDE 148
Outline
Introduction Motivation CAD and cross-client monotonic reads ORCA design Results Summary and conclusion
21
SLIDE 149
Evaluation
22
SLIDE 150
Evaluation
Implemented in ZooKeeper
22
SLIDE 151
Evaluation
Implemented in ZooKeeper Evaluate different durability models in isolation
compare CAD against immediate and eventual durability
22
SLIDE 152
Evaluation
Implemented in ZooKeeper Evaluate different durability models in isolation
compare CAD against immediate and eventual durability
Evaluate overall system performance
ORCA against strong and weakly consistent ZooKeeper
22
SLIDE 153
CAD Durability Layer Performance
23
SLIDE 154
CAD Durability Layer Performance
23
YCSB-A: 50% W, 50% R
SLIDE 155
CAD Durability Layer Performance
23
YCSB-A: 50% W, 50% R
20 40 60 80 100 500 1000 1500 CDF Latency (us) Write Latency Distribution immediate eventual cad
SLIDE 156
CAD Durability Layer Performance
23
YCSB-A: 50% W, 50% R
20 40 60 80 100 500 1000 1500 CDF Latency (us) Write Latency Distribution immediate eventual cad
SLIDE 157
CAD Durability Layer Performance
23
YCSB-A: 50% W, 50% R
CAD writes faster than immediate durability CAD matches performance of eventual 20 40 60 80 100 500 1000 1500 CDF Latency (us) Write Latency Distribution immediate eventual cad
SLIDE 158
CAD Durability Layer Performance
23
YCSB-A: 50% W, 50% R
CAD writes faster than immediate durability CAD matches performance of eventual 20 40 60 80 100 500 1000 1500 CDF Latency (us) Write Latency Distribution immediate eventual cad 20 40 60 80 100 500 1000 1500 CDF Latency (us) Read Latency Distribution immediate eventual cad
SLIDE 159
CAD Durability Layer Performance
23
YCSB-A: 50% W, 50% R
CAD writes faster than immediate durability CAD matches performance of eventual 20 40 60 80 100 500 1000 1500 CDF Latency (us) Write Latency Distribution immediate eventual cad 20 40 60 80 100 500 1000 1500 CDF Latency (us) Read Latency Distribution immediate eventual cad
SLIDE 160
CAD Durability Layer Performance
23
YCSB-A: 50% W, 50% R
CAD writes faster than immediate durability CAD matches performance of eventual 20 40 60 80 100 500 1000 1500 CDF Latency (us) Write Latency Distribution immediate eventual cad 20 40 60 80 100 500 1000 1500 CDF Latency (us) Read Latency Distribution immediate eventual cad
reads queued behind writes
SLIDE 161
CAD Durability Layer Performance
23
YCSB-A: 50% W, 50% R
CAD writes faster than immediate durability CAD matches performance of eventual 20 40 60 80 100 500 1000 1500 CDF Latency (us) Write Latency Distribution immediate eventual cad 20 40 60 80 100 500 1000 1500 CDF Latency (us) Read Latency Distribution immediate eventual cad Most reads in CAD fast Only 5% slow due to synchronous ops
5% reads queued behind writes
SLIDE 162
CAD Durability Layer Performance
23
YCSB-A: 50% W, 50% R
CAD writes faster than immediate durability CAD matches performance of eventual 20 40 60 80 100 500 1000 1500 CDF Latency (us) Write Latency Distribution immediate eventual cad 20 40 60 80 100 500 1000 1500 CDF Latency (us) Read Latency Distribution immediate eventual cad Most reads in CAD fast Only 5% slow due to synchronous ops
5% reads queued behind writes
CAD performs similar to eventual and is faster than immediate
SLIDE 163
ORCA System Performance
24
Strong-ZK – uses immediate durability, reads only at leader Weak-ZK – uses eventual durability, reads at many nodes ORCA – uses CAD, reads at many nodes
SLIDE 164
ORCA System Performance
24
3.78 2.09 2.09 3.44 3.28 1.97 1.75 3.04
10 20 30 40 50 A B D F
Throughput (KOps/s) strong-ZK weak-ZK
- rca
50% R 95% R 95% R 66.7% R 50% W 5% W 5% W 33.3% W
Strong-ZK – uses immediate durability, reads only at leader Weak-ZK – uses eventual durability, reads at many nodes ORCA – uses CAD, reads at many nodes
SLIDE 165
ORCA System Performance
24
3.78 2.09 2.09 3.44 3.28 1.97 1.75 3.04
10 20 30 40 50 A B D F
Throughput (KOps/s) strong-ZK weak-ZK
- rca
50% R 95% R 95% R 66.7% R 50% W 5% W 5% W 33.3% W
Strong-ZK performs poorly due to immediate durability and leader-restricted reads Strong-ZK – uses immediate durability, reads only at leader Weak-ZK – uses eventual durability, reads at many nodes ORCA – uses CAD, reads at many nodes
SLIDE 166
ORCA System Performance
24
3.78 2.09 2.09 3.44 3.28 1.97 1.75 3.04
10 20 30 40 50 A B D F
Throughput (KOps/s) strong-ZK weak-ZK
- rca
50% R 95% R 95% R 66.7% R 50% W 5% W 5% W 33.3% W
Strong-ZK performs poorly due to immediate durability and leader-restricted reads Weak-ZK performs well due to eventual durability and scalable reads Strong-ZK – uses immediate durability, reads only at leader Weak-ZK – uses eventual durability, reads at many nodes ORCA – uses CAD, reads at many nodes
SLIDE 167
ORCA System Performance
24
3.78 2.09 2.09 3.44 3.28 1.97 1.75 3.04
10 20 30 40 50 A B D F
Throughput (KOps/s) strong-ZK weak-ZK
- rca
50% R 95% R 95% R 66.7% R 50% W 5% W 5% W 33.3% W
Strong-ZK performs poorly due to immediate durability and leader-restricted reads Weak-ZK performs well due to eventual durability and scalable reads ORCA adds little overheads compared to weak-ZK
reads that access non-durable data
Strong-ZK – uses immediate durability, reads only at leader Weak-ZK – uses eventual durability, reads at many nodes ORCA – uses CAD, reads at many nodes
SLIDE 168
ORCA System Performance
24
3.78 2.09 2.09 3.44 3.28 1.97 1.75 3.04
10 20 30 40 50 A B D F
Throughput (KOps/s) strong-ZK weak-ZK
- rca
50% R 95% R 95% R 66.7% R 50% W 5% W 5% W 33.3% W
Strong-ZK performs poorly due to immediate durability and leader-restricted reads Weak-ZK performs well due to eventual durability and scalable reads ORCA adds little overheads compared to weak-ZK
reads that access non-durable data
Strong-ZK – uses immediate durability, reads only at leader Weak-ZK – uses eventual durability, reads at many nodes ORCA – uses CAD, reads at many nodes
SLIDE 169
More experiments in the paper…
Evaluation
correctness testing using a cluster crash-testing framework geo-replicated setting micro-benchmarks
Application case studies
location-tracking social-media timeline
25
SLIDE 170
Summary and Conclusions
26
SLIDE 171
Summary and Conclusions
Surprisingly, durability models are overlooked
26
SLIDE 172
Summary and Conclusions
Surprisingly, durability models are overlooked Immediate durability enables strong consistency but is slow
26
SLIDE 173
Summary and Conclusions
Surprisingly, durability models are overlooked Immediate durability enables strong consistency but is slow Eventual durability is fast but only enables weak consistency
26
SLIDE 174
Summary and Conclusions
Surprisingly, durability models are overlooked Immediate durability enables strong consistency but is slow Eventual durability is fast but only enables weak consistency CAD – consistency-aware durability, a new way of thinking about durability
enables both strong consistency and high performance
26
SLIDE 175
Summary and Conclusions
Surprisingly, durability models are overlooked Immediate durability enables strong consistency but is slow Eventual durability is fast but only enables weak consistency CAD – consistency-aware durability, a new way of thinking about durability
enables both strong consistency and high performance CAD is useful for many deployments that currently adopt eventual durability
26
SLIDE 176
Summary and Conclusions
Surprisingly, durability models are overlooked Immediate durability enables strong consistency but is slow Eventual durability is fast but only enables weak consistency CAD – consistency-aware durability, a new way of thinking about durability
enables both strong consistency and high performance CAD is useful for many deployments that currently adopt eventual durability
Consistency and performance are seemingly at odds – by carefully examining the underlying layer, achieve both
26
SLIDE 177
Summary and Conclusions
Surprisingly, durability models are overlooked Immediate durability enables strong consistency but is slow Eventual durability is fast but only enables weak consistency CAD – consistency-aware durability, a new way of thinking about durability
enables both strong consistency and high performance CAD is useful for many deployments that currently adopt eventual durability
Consistency and performance are seemingly at odds – by carefully examining the underlying layer, achieve both
Thank you!
26