@lenadro id Fundamental techniques and building blocks @lenadro - - PowerPoint PPT Presentation

@lenadro id

Channel Into Universe Of Eventually Perfect Distributed Systems

@lenadro id

Fundamental techniques 
 and building blocks

@lenadro id

Are fundamentals still important?

@lenadro id

Your System Your Trade-Offs

@lenadro id

Theory Practice

hm…

Gap

@lenadro id

There are challenges

@lenadro id

There are challenges Road to Correctness and Understanding

@lenadro id

Simple problems become hard

@lenadro id

Ordering is Hard

@lenadro id

A B 1 2 2 3 3 1 2

Lamport Clock

2 and 3 could be concurrent

@lenadro id

A B C

{A:2, B:1, C:1} {A:2, B:1, C:2} {A:3, B:1, {A:4, B:1, {A:2, B:1, {A:2, B:2, { A:1, B:1 } {A:2, B:1 } { B:1 }

X Y

Vector Clock

@lenadro id

Agreement In Distributed Systems

@lenadro id

Two Phase Commit


@lenadro id

Blocking Failure in Two Phase Commit

Committed

X

Crashed

OK OK OK

Not Committed Not Committed

Crashed

X

Nodes are blocked! Can’t decide!

? ? ?

Can’t decide Can’t decide Can’t decide Not Committed

@lenadro id

Three Phase Two Phase

Hm… it’s blocking when there’re

@lenadro id

X O O O

? ? ?

Not Pre- Not Pre- Not Pre-

Pre- committ

?

Crash

@lenadro id

Abor Abor Abor

Comm

X

Crash

@lenadro id

Three Phase Two Phase

Hm… it’s blocking when there are Might be inconsistent in asynchronous environment

@lenadro id “Distributed consensus is impossible in asynchronous system where at least one node can fail.”

???

X

FLP: Impossibility Result

@lenadro id FLP

Three Phase Two Phase

Might be inconsistent in asynchrono Hm… it’s blocking when there’re

X

Paxos

Fast Paxos Multi-Paxos Vertical Classical Cheap Paxos Zab Raf Chandra- Toueg

@lenadro id

Paxos

@lenadro id

Trade-offs Optimizations

Weak or strong leader? Proposal Copying Quorum size? Distinguished Proposer Number of Failures Combining Roles Tolerated? Strategies for Proposal Numbers

@lenadro id

Discovering New Trade-offs and Optimizations

Quorum intersection revised Quorum based value selection Proposal numbers uniqueness And many more…

cl.cam.ac.uk/techreports/UCAM-CL-TR-935.pdf by Dr. Heidi Howard

@lenadro id

Consistent Replication?

@lenadro id

dl.acm.org/citation.cfm?id=3183713.3196937

@lenadro id

Conflict-Free Replicated Data Types

@lenadro id

(1, 0, 0) (0, 1, 1) (0, 0, 0) (1, 1, 1)

(_, 1, 1) (_, 1, 1) (1, _, _) (1, _, _)

@lenadro id

add X delete X delete X add X ?

@lenadro id

Cosmos DB

x: { a } x: { b, c } x: { a, b, c }

@lenadro id

Failure Detection

@lenadro id Completeness: can all nodes discover all the failures? Accuracy: how precise can a node be in its failure suspicions?

@lenadro id

Understanding Trade-offs Helps

✓ To make the right choices
 ✓ To know what correct means for us
 ✓ To verify and maintain correctness in practice

@lenadro id

Maintaining Correctness In Real Systems

@lenadro id

Model Checking

@lenadro id

Verifying and Maintaining Correctness in Practical Real-World Systems

@lenadro id

Kafka System Tests

testing.confluent.io/confluent-kafka-system-test-results 450+ system tests, 6800+ unit tests, 600+ integration tests confluent.io/blog/apache-kafka-tested

@lenadro id

Cassandra Tests

cassandra.apache.org/blog/2018/10/17/finding_bugs_with_property_based_testing.html 
 cassandra.apache.org/blog/2018/08/21/testing_apache_cassandra.html

Replay testing Dynamic test generation Property-based testing and fuzzing Distributed tests and fault-injection Upgrade testing

@lenadro id + Model-checking + Property-based testing and fuzzing + Performance and upgrade testing + Unit and integration testing + Fault injection + Attention to exception handling logic + Сode reviews

@lenadro id

Take-Aways

@lenadro id

✓ Know your trade-offs ✓ Create understandable systems
 ✓ Invest in correctness, it doesn’t come for free
 ✓ Don’t trust: test and verify
 ✓ Automate, but be ready when things fail
 ✓ Remember the real problem you are solving

@lenadro id

@lenadro id

#SystemsYouUnderstand