Incremental Consistency Guarantees For Replicated Objects Rachid - - PowerPoint PPT Presentation

Incremental Consistency Guarantees

For Replicated Objects

Rachid Guerraoui, Matej Pavlovic, Dragos-Adrian Seredinschi

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

2

replicate

SOCIAL MEDIA APPLICATION

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

2

replicate

SOCIAL MEDIA APPLICATION

22 20 17

Number of recent events


• n the user’s timeline

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

• Returns the correct data
• Latency: ~200 ms
• Can become unavailable

2

Strong Consistency

[CAP], [PACELC]

replicate

SOCIAL MEDIA APPLICATION

22 20 17 22

Number of recent events


• n the user’s timeline

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

• Latency: ~100 ms
• High availability
• Allows inconsistencies: can return
• Returns the correct data
• Latency: ~200 ms
• Can become unavailable

2

Strong Consistency

• r
• r

[CAP], [PACELC]

Weak Consistency

replicate

SOCIAL MEDIA APPLICATION

22 20 17 22 20 22 17

Number of recent events


• n the user’s timeline

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

• Latency: ~100 ms
• High availability
• Allows inconsistencies: can return
• Returns the correct data
• Latency: ~200 ms
• Can become unavailable

2

Strong Consistency

• r
• r

[CAP], [PACELC]

Weak Consistency

replicate

SOCIAL MEDIA APPLICATION

22 20 17 22 20 22 17

Number of recent events


• n the user’s timeline

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Neither model is ideal!

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

• Latency: ~100 ms
• High availability
• Allows inconsistencies: can return
• Returns the correct data
• Latency: ~200 ms
• Can become unavailable

2

Strong Consistency

• r
• r

[CAP], [PACELC]

Weak Consistency

replicate

SOCIAL MEDIA APPLICATION

22 20 17 22 20 22 17

Number of recent events


• n the user’s timeline

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Neither model is ideal!

We use both models.

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

3

Multiple models

• 1. Weak consistency

• 2. Strong consistency


100ms 300ms

20 22

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

3

Multiple models

• 1. Weak consistency

• 2. Strong consistency


100ms 300ms

20 22

Dynamo
 [SOSP’07] Pileus
 [SOSP’13]

Increasingly many systems expose
 multiple consistency models:

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

3

Multiple models

• 1. Send multiple requests?
• 2. How to leverage individual responses?
• 3. Semantics?
• 4. …
• 1. Weak consistency

• 2. Strong consistency


100ms 300ms

20 22

Issues

Dynamo
 [SOSP’07] Pileus
 [SOSP’13]

Increasingly many systems expose
 multiple consistency models:

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

3

Multiple models

How do you program with

★ inconsistencies? ★ multiple values?

Problem

• 1. Send multiple requests?
• 2. How to leverage individual responses?
• 3. Semantics?
• 4. …
• 1. Weak consistency

• 2. Strong consistency


100ms 300ms

20 22

Issues

Dynamo
 [SOSP’07] Pileus
 [SOSP’13]

Increasingly many systems expose
 multiple consistency models:

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

4

ABSTRACTION 
 FOR REPLICATED OBJECTS

Weak
 Consistency Strong
 Consistency

…

SOCIAL MEDIA APPLICATION

20 17 22

????
 Consistency

…

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

4

ABSTRACTION 
 FOR REPLICATED OBJECTS

Weak
 Consistency Strong
 Consistency

…

CORRECTABLE

SOCIAL MEDIA APPLICATION

20 17 22

????
 Consistency

…

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

4

ABSTRACTION 
 FOR REPLICATED OBJECTS

Weak
 Consistency Strong
 Consistency

…

CORRECTABLE

Incremental 
 Consistency 
 Guarantees (ICG)

provides

SOCIAL MEDIA APPLICATION

20 17 22

????
 Consistency

…

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Correctables / Design

5

PROMISE

resolve
 asynchronously

➤ Starting point: Promises ➤ Placeholders for values ➤ Becoming mainstream

value

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Correctables / Design

5

CORRECTABLE PROMISE

resolve
 asynchronously

➤ Starting point: Promises ➤ Placeholders for values ➤ Becoming mainstream

value value1 value2 valuen

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Correctables / Design

5

CORRECTABLE PROMISE

resolve
 asynchronously

PROGRESSIVELY 
 STRONGER CONSISTENCY (INCREMENTAL)

PROGRESSIVELY HIGHER LATENCY

➤ Starting point: Promises ➤ Placeholders for values ➤ Becoming mainstream

value value1 value2 valuen

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Consistency Models are Complementary

6

Consistency Weak Strong Low High

Performance

Eventual

Linearizable Sequential Causal (Ideal protocol) (Worst protocol)

I N C R E M E N T A L

Weak consistency:

★ Fast ★ (Often correct)

Strong consistency:

★ Slower ★ (Correct with certainty)

Weak
 Consistency Strong
 Consistency

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Consistency Models are Complementary

6

Consistency Weak Strong Low High

Performance

Eventual

Linearizable Sequential Causal (Ideal protocol) (Worst protocol)

I N C R E M E N T A L

Weak consistency:

★ Fast ★ (Often correct)

Strong consistency:

★ Slower ★ (Correct with certainty)

Weak
 Consistency Strong
 Consistency

So what?

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Consistency Models are Complementary

6

Consistency Weak Strong Low High

Performance

Eventual

Linearizable Sequential Causal (Ideal protocol) (Worst protocol)

I N C R E M E N T A L

Weak consistency:

★ Fast ★ (Often correct)

Strong consistency:

★ Slower ★ (Correct with certainty)

Weak
 Consistency Strong
 Consistency

So what?

Latency optimizations

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

7

Speculating with Correctables

SOCIAL MEDIA APPLICATION

read
 timeline

CORRECTABLE

Lower latency of strong consistency

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

7

Speculating with Correctables

SOCIAL MEDIA APPLICATION

Weak
 Consistency

value1

Strong
 Consistency

value2 read
 timeline

CORRECTABLE

Lower latency of strong consistency

Latency gap: ~100 ms

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

7

Speculating with Correctables

SOCIAL MEDIA APPLICATION

Weak
 Consistency

value1

Strong
 Consistency

value2 read
 timeline Speculative
 execution

• value1 is often correct

• Speculatively execute any further steps


e.g., prefetch dependent data

[Existential Consistency. SOSP’15]
 [PBS. VLDB 5(8)’12]

Verify based on value2

CORRECTABLE

Lower latency of strong consistency

Latency gap: ~100 ms

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

7

Speculating with Correctables

SOCIAL MEDIA APPLICATION

Weak
 Consistency

value1

Strong
 Consistency

value2 read
 timeline Speculative
 execution

• value1 is often correct

• Speculatively execute any further steps


e.g., prefetch dependent data

[Existential Consistency. SOSP’15]
 [PBS. VLDB 5(8)’12]

Verify based on value2

CORRECTABLE

Lower latency of strong consistency

Latency gap: ~100 ms

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Strongly-consistent timeline Fetch timeline items

200 ms 100 ms

Traditional operation:

8

300ms

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Strongly-consistent timeline Fetch timeline items

200 ms 100 ms

Traditional operation:

8

Speculative operation with ICG:

value2 100 ms value1 100 ms 100 ms

300ms Fetch timeline items

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Strongly-consistent timeline Fetch timeline items

200 ms 100 ms

Traditional operation:

8

Speculative operation with ICG:

value2 value1 
 matches
 value2 100 ms value1 100 ms 100 ms yes

300ms 200ms Fetch timeline items

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Strongly-consistent timeline Fetch timeline items

200 ms 100 ms

Traditional operation:

8

Speculative operation with ICG:

value2 value1 
 matches
 value2 100 ms value1 100 ms 100 ms yes no 100 ms

Re-fetch 300ms 300ms 200ms Fetch timeline items

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Speculation case-study

9

➤ Application: Twissandra ➤ Workload generated via YCSB ➤ Clients in Ireland ➤ Geo-replication on Amazon’s EC2

V R G C A L O R G

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Speculation case-study

9

★ Advertising System


— Speculation case-study

★ Ticket-selling System


— Exploiting application semantics

★ Overheads evaluation


& Optimizations

★ Latency gaps between consistency


models

➤ Application: Twissandra ➤ Workload generated via YCSB ➤ Clients in Ireland ➤ Geo-replication on Amazon’s EC2

V R G C A L O R G

check the paper

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Decreasing latency of strong consistency

10

Workload A (50:50 read/write) Workload C (read-only)

What is the 
 latency of the fetch_timeline() operation?

Workload B (95:5 read/write)

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Decreasing latency of strong consistency

10

Workload A (50:50 read/write) Workload C (read-only)

What is the 
 latency of the fetch_timeline() operation?

Workload B (95:5 read/write)

Baseline Read using a quorum of 2/3 replicas

vs.

ICG

• 1. Weak: Read with 1/3 replicas
• 2. “Strong:” Read with quorum of 2/3 replicas

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Decreasing latency of strong consistency

10

Workload A (50:50 read/write) Workload C (read-only)

What is the 
 latency of the fetch_timeline() operation?

Workload B (95:5 read/write)

Baseline Read using a quorum of 2/3 replicas

vs.

ICG

• 1. Weak: Read with 1/3 replicas
• 2. “Strong:” Read with quorum of 2/3 replicas

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

50 100 150 200 250 50 100 150 200 250 300 Latency (ms)

Workload A Ads System Twissandra

C2 (R=2) CC2 (R=1,2) 50 100 150 200 250 50 100 150 200 250 300 Throughput (ops/sec)

Workload B Ads System Twissandra

50 100 150 200 250 50 100 150 200 250 300

Workload C Ads System Twissandra

150 300 450 600 50 100 150 200 250 300 Latency (ms)

Ads System Twissandra

150 300 450 600 50 100 150 200 250 300 Throughput (ops/sec)

Ads System Twissandra

150 300 450 600 50 100 150 200 250 300

Ads System Twissandra

Decreasing latency of strong consistency

10

Workload A (50:50 read/write) Workload C (read-only)

What is the 
 latency of the fetch_timeline() operation?

Workload B (95:5 read/write)

Baseline Read using a quorum of 2/3 replicas

vs.

ICG

• 1. Weak: Read with 1/3 replicas
• 2. “Strong:” Read with quorum of 2/3 replicas

Baseline ICG

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

50 100 150 200 250 50 100 150 200 250 300 Latency (ms)

Workload A Ads System Twissandra

C2 (R=2) CC2 (R=1,2) 50 100 150 200 250 50 100 150 200 250 300 Throughput (ops/sec)

Workload B Ads System Twissandra

50 100 150 200 250 50 100 150 200 250 300

Workload C Ads System Twissandra

150 300 450 600 50 100 150 200 250 300 Latency (ms)

Ads System Twissandra

150 300 450 600 50 100 150 200 250 300 Throughput (ops/sec)

Ads System Twissandra

150 300 450 600 50 100 150 200 250 300

Ads System Twissandra

Decreasing latency of strong consistency

10

Workload A (50:50 read/write) Workload C (read-only)

What is the 
 latency of the fetch_timeline() operation?

★ Latency decrease by 40% ★ Throughput drop by 6% ★ Same consistency model (2/3 replicas)

Workload B (95:5 read/write)

Baseline Read using a quorum of 2/3 replicas

vs.

ICG

• 1. Weak: Read with 1/3 replicas
• 2. “Strong:” Read with quorum of 2/3 replicas

Baseline ICG

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

11

The Correctables abstraction enables you to:

• 1. Leverage consistency models incrementally
• 2. Lower latency of strong consistency

CORRECTABLE

value1 value2 valuen Incremental 
 Consistency 
 Guarantees

@adizere

Conclusion

backup slides

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Speculation // Syntactic sugar

13

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Legacy code vs. Correctables

14

semantics execution details

Correctable

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Legacy code vs. Correctables

14

semantics execution details

Correctable

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Overheads

➤ Cassandra ➤ YCSB workload, various configurations ➤ Client in Ireland ➤ Replicas in Virginia, Frankfurt, and Ireland

15

0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4

30 60 120 180 240 300

Efficiency (kB/op) Workload A C1 C1 CC2 CC2 *CC2 *CC2 #Total client threads Latest distribution: Zipfian distribution:

+ 2 7 % +77% +15% +90%

30 60 120 180 240 300

Workload B #Total client threads Latest distribution: Zipfian distribution:

+ 2 7 % +77% +15% +90%

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Overheads

➤ Cassandra ➤ YCSB workload, various configurations ➤ Client in Ireland ➤ Replicas in Virginia, Frankfurt, and Ireland

15

0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4

30 60 120 180 240 300

Efficiency (kB/op) Workload A C1 C1 CC2 CC2 *CC2 *CC2 #Total client threads Latest distribution: Zipfian distribution:

+ 2 7 % +77% +15% +90%

30 60 120 180 240 300

Workload B #Total client threads Latest distribution: Zipfian distribution:

+ 2 7 % +77% +15% +90%

2 4 6 8 10 12 14 16 1 2 4 6 8 10 12 Efficiency (KB/op)

ZooKeeper Correctable ZooKeeper

# Clients 500 tickets 1000 tickets

• 71%
• 44%
• 81%
• 60%

1 2 4 6 8 10 12 # Clients 500 tickets 1000 tickets

• 71%
• 44%
• 81%
• 60%

➤ ZooKeeper queue implementation ➤ Wasteful implementation (by default) ➤ We were able to improve 


— negative overhead

500 elements 1000 elements

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Exploiting application semantics

➤ Ticket selling application ➤ Implemented through a ZooKeeper queue ➤ Buy ticket = dequeue operation

16

Ticket 1 Ticket 2

…

Ticket 500

u p d a t e close

ticket X ticket Y | NULL

( w e a k ) (strong) if ( X > threshold)
 confirm()
 else
 wait for strong if (! NULL)
 confirm() invoke
 dequeue()

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Exploiting application semantics

➤ Ticket selling application ➤ Implemented through a ZooKeeper queue ➤ Buy ticket = dequeue operation

16

Ticket 1 100 200 300 400 420 440 460 480 500 Latency to buy ticket (msec) Ticket number Correctable ZooKeeper ZooKeeper

Last 20 tickets

Ticket 2

…

Ticket 500

u p d a t e close

ticket X ticket Y | NULL

( w e a k ) (strong) if ( X > threshold)
 confirm()
 else
 wait for strong if (! NULL)
 confirm() invoke
 dequeue()

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Divergence between weak and strong consistency

17

5 10 15 20 25 30 30 60 120 180 240 300

%Divergence #Total client threads

Workload A-Latest Workload A-Zipfian Workload B-Latest Workload B-Zipfian

➤ Extremely high ➤ Unusual

0.1% — 5% typical range

}

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Latency gaps between consistency models

18

50 100 150 200 250 200 400 600 800 1000

Latency (ms)

Workload A (50:50 read/write)

C1 (R=1) C2 (R=2) CC2 preliminary (R=1) CC2 final (R=2)

50 100 150 200 250 200 400 600 800 1000

Throughput (ops/sec)

Workload B (95:5 read/write)

50 100 150 200 250 200 400 600 800 1000

Workload C (read-only) 50 100 150 Average Read Latency (ms)

CC preliminary CC final C 99th %ile latency R=3 R=2 R=1

Latency gap

50 100 150 200 Average Latency (ms) CZK preliminary CZK final ZK 99th %ile latency Leader in IRL Leader in VRG Follower (FRK) Leader (IRL) Follower (IRL) Leader (VRG)

Client connection:

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

19

value1

?

value2

Efficiency of Multiple Responses

CORRECTABLE

read
 timeline

SOCIAL MEDIA APPLICATION

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Binding

Request

Replicated Storage

Response (final) Response (preliminary)

Weak consistency Strong consistency

Coordination

19

value1

?

value2

Efficiency of Multiple Responses

CORRECTABLE

read
 timeline

SOCIAL MEDIA APPLICATION

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Binding

Request

Replicated Storage

Response (final) Response (preliminary)

Weak consistency Strong consistency

Coordination

19

value1 (quorum gathering) value2

Efficiency of Multiple Responses

CORRECTABLE

read
 timeline

SOCIAL MEDIA APPLICATION

Incremental Consistency Guarantees

Dragos-Adrian Seredinschi

Correctables / Library

20

Application Cache Storage binding binding binding binding

Correctables LIBRARY

RPC invoke

API

(Weak / Strong)

RPC Consistency-based interface System-specific interface Desktop Application Web Frontend Mobile App Caching Daemon Correctable