Coordination-Free Computations Christopher Meiklejohn LASP - - PowerPoint PPT Presentation

▶
coordination free computations
SMART_READER_LITE
LIVE PREVIEW

Coordination-Free Computations Christopher Meiklejohn LASP - - PowerPoint PPT Presentation

Sep 08, 2022 27 likes •1.37k views

Coordination-Free Computations Christopher Meiklejohn LASP DISTRIBUTED, EVENTUALLY CONSISTENT COMPUTATIONS CHRISTOPHER MEIKLEJOHN (BASHO TECHNOLOGIES, INC.) PETER VAN ROY (UNIVERSIT CATHOLIQUE DE LOUVAIN) 2 LASP MOTIVATION 3

slide-1
SLIDE 1

Coordination-Free Computations

Christopher Meiklejohn

slide-2
SLIDE 2

DISTRIBUTED, EVENTUALLY CONSISTENT COMPUTATIONS

LASP

2

CHRISTOPHER MEIKLEJOHN (BASHO TECHNOLOGIES, INC.) PETER VAN ROY (UNIVERSITÉ CATHOLIQUE DE LOUVAIN)

slide-3
SLIDE 3

LASP MOTIVATION

3

slide-4
SLIDE 4

EXPENSIVE / IMPRACTICAL

SYNCHRONIZATION IS

4

slide-5
SLIDE 5

MOBILE GAMES:

SHARED STATE BETWEEN CLIENTS CLIENTS GO OFFLINE

5 http://www.rovio.com/en/news/blog/261/263-million-monthly-active-users-in-december/

slide-6
SLIDE 6

DISJOINT STATE AGGREGATED UPSTREAM CLIENTS GO OFFLINE

“INTERNET OF THINGS”:

6

Gubbi, Jayavardhana, et al. "Internet of Things (IoT): A vision, architectural elements, and future directions." Future Generation Computer Systems 29.7 (2013): 1645-1660.

slide-7
SLIDE 7

NO TOTAL ORDER:

REPLICATED SHARED STATE WITH OFFLINE CLIENTS CLIENTS NEED TO MAKE PROGRESS

7

Gilbert, Seth, and Nancy Lynch. "Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services." ACM SIGACT News 33.2 (2002): 51-59.

slide-8
SLIDE 8

WALL CLOCKS:

UNRELIABLE AT BEST NON-DETERMINISTIC IF USED IN COMPUTATIONS

8

Corbett, James C., et al. "Spanner: Google’s globally distributed database." ACM Transactions on Computer Systems (TOCS) 31.3 (2013): 8.

slide-9
SLIDE 9

RECONCILED BY USER

CONCURRENCY

9

slide-10
SLIDE 10

RA RB 1 3 2 ? ? set(1) set(2) set(3)

10

slide-11
SLIDE 11

RA RB 1 3 2 ? ? set(1) set(2) set(3)

11

slide-12
SLIDE 12

RA RB 1 3 2 ? ? set(1) set(2) set(3)

12

slide-13
SLIDE 13

RA RB 1 3 2 ? ? set(1) set(2) set(3)

13

slide-14
SLIDE 14

CRDTs

14

slide-15
SLIDE 15

DETERMINISTIC RESOLUTION

CRDTs PROVIDE

15

slide-16
SLIDE 16

CRDTs:

MAPS, SETS, COUNTERS, REGISTERS, GRAPHS DETERMINISTIC RESOLUTION

16

slide-17
SLIDE 17

“CORRECT REPLICAS THAT HAVE DELIVERED THE SAME UPDATES HAVE EQUIVALENT STATE”

Shapiro, Marc, et al. "Conflict-free replicated data types." Stabilization, Safety, and Security of Distributed Systems. Springer Berlin Heidelberg, 2011. 386-400.

MONOTONIC STRONG EVENTUAL CONSISTENCY

CRDTs REALIZE

17

slide-18
SLIDE 18

‘MAX’ REGISTER

CRDTs EXAMPLE

18

slide-19
SLIDE 19

RA RB 1 3 2 3 3 set(1) set(2) set(3) max(2,3) max(2,3)

19

slide-20
SLIDE 20

‘ORSET’ SET

CRDTs EXAMPLE

20

slide-21
SLIDE 21

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1)

21

slide-22
SLIDE 22

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1)

22

slide-23
SLIDE 23

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1)

23

slide-24
SLIDE 24

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1)

24

slide-25
SLIDE 25

FRAGMENTS & PROGRAMS

25

slide-26
SLIDE 26

{1,2,3} {3,4,5}

26

slide-27
SLIDE 27

{1,2,3} {3,4,5}

Intersection

{3}

27

slide-28
SLIDE 28

{1,2,3} {3,4,5}

Intersection

{3}

Map

{9}

28

slide-29
SLIDE 29

{1,2,3} {3,4,5}

Intersection

{3}

Map

{9}

29

slide-30
SLIDE 30

{1,2,3} {3,4,5}

Intersection

{3}

Map

{9}

30

slide-31
SLIDE 31

{1,2,3} {3,4,5}

Intersection

{3}

Map

{9}

31

slide-32
SLIDE 32

{1,2,3} {3,4,5}

Intersection

{3}

Map

{9} {1,2,3} {3,4,5}

Intersection

{3}

Map

{9}

32

slide-33
SLIDE 33

{1,2,3} {3,4,5}

Intersection

{3}

Map

{9} {1,2,3} {3,4,5}

Intersection

{3}

Map

{9}

33

slide-34
SLIDE 34

{1,2,3} {3,4,5}

Intersection

{3}

Map

{9} {1,2,3} {3,4,5}

Intersection

{3}

Map

{9}

34

slide-35
SLIDE 35

NONTRIVIAL

FUNCTION APPLICATION AND DATA COMPOSITION IS

35

slide-36
SLIDE 36

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RC) {2} fun(X) -> 2 end {2} {}

36

slide-37
SLIDE 37

RA RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RC) {2} fun(X) -> 2 end {2} {} F(RA) {2} fun(X) -> 2 end {2} {2} {2}

37

slide-38
SLIDE 38

RA RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RC) {2} fun(X) -> 2 end {2} {} F(RA) {2} fun(X) -> 2 end {2} {2} {2}

38

slide-39
SLIDE 39

RA RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RC) {2} fun(X) -> 2 end {2} {} F(RA) {2} fun(X) -> 2 end {2} {2} {2}

39

slide-40
SLIDE 40

RA RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RC) {2} fun(X) -> 2 end {2} {} F(RA) {2} fun(X) -> 2 end {2} {2} {2}

40

slide-41
SLIDE 41

COMPOSITION:

USER OBSERVABLE VALUE VS. STATE METADATA MAPPING IS NONTRIVIAL WITHOUT MAPPING METADATA; UNMERGABLE

41

Brown, Russell, et al. "Riak dt map: A composable, convergent replicated dictionary." Proceedings of the First Workshop on Principles and Practice of Eventual Consistency. ACM, 2014. Conway, Neil, et al. "Logic and lattices for distributed programming." Proceedings of the Third ACM Symposium on Cloud Computing. ACM, 2012. Meiklejohn, Christopher. "On the composability of the Riak DT map: expanding from embedded to multi-key structures." Proceedings of the First Workshop on Principles and Practice of Eventual Consistency. ACM, 2014.

slide-42
SLIDE 42

LASP LANGUAGE

42

slide-43
SLIDE 43

LASP:

DISTRIBUTED RUNTIME IMPLEMENTED AS ERLANG LIBRARY USES RIAK-CORE

43

slide-44
SLIDE 44

LASP:

CRDTS AS STREAMS OF STATE CHANGES CRDTS CONNECTED BY MONOTONIC PROCESSES MANY TO ONE MAPPING OF CRDTS

44

slide-45
SLIDE 45

PRIMITIVE OPERATIONS:

MONOTONIC READ, UPDATE FUNCTIONAL: MAP, FILTER, FOLD SET-THEORETIC: PRODUCT, UNION, INTERSECTION LIFTED TO OPERATE OVER METADATA

45

slide-46
SLIDE 46

MAP

EXAMPLE

46

slide-47
SLIDE 47

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RB) {2} fun(X) -> 2 end {2} {} (2, {b}, {}) (2, {b}, {b}) {} (2, {a, b}, {b}) (2, {a, b}, {b}) F(RC) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {} {2} (2, {b}, {}) (2, {b}, {b}) (2, {a, b}, {b}) F(RA) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {2} {2} (2, {a}, {}) (2, {a, b}, {}) (2, {a, b}, {b})

47

slide-48
SLIDE 48

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RB) {2} fun(X) -> 2 end {2} {} (2, {b}, {}) (2, {b}, {b}) {} (2, {a, b}, {b}) (2, {a, b}, {b}) F(RC) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {} {2} (2, {b}, {}) (2, {b}, {b}) (2, {a, b}, {b}) F(RA) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {2} {2} (2, {a}, {}) (2, {a, b}, {}) (2, {a, b}, {b})

48

slide-49
SLIDE 49

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RB) {2} fun(X) -> 2 end {2} {} (2, {b}, {}) (2, {b}, {b}) {} (2, {a, b}, {b}) (2, {a, b}, {b}) F(RC) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {} {2} (2, {b}, {}) (2, {b}, {b}) (2, {a, b}, {b}) F(RA) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {2} {2} (2, {a}, {}) (2, {a, b}, {}) (2, {a, b}, {b})

49

slide-50
SLIDE 50

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RB) {2} fun(X) -> 2 end {2} {} (2, {b}, {}) (2, {b}, {b}) {} (2, {a, b}, {b}) (2, {a, b}, {b}) F(RC) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {} {2} (2, {b}, {}) (2, {b}, {b}) (2, {a, b}, {b}) F(RA) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {2} {2} (2, {a}, {}) (2, {a, b}, {}) (2, {a, b}, {b})

50

slide-51
SLIDE 51

ARCHITECTURE

51

slide-52
SLIDE 52

SHARED VARIABLE STORE

LASP STORE

52

slide-53
SLIDE 53

LEVELDB, BITCASK, ETS

LASP BACKENDS

53

slide-54
SLIDE 54

PROVIDED BY RIAK DT

LASP CRDTs

54

slide-55
SLIDE 55

CENTRALIZED SEMANTICS

LASP ARCHITECTURE

55

slide-56
SLIDE 56

STORE:

SHARED VARIABLE STORE PROCESSESS SYNCHRONIZE ON VARIABLES

56

slide-57
SLIDE 57

DISTRIBUTED SEMANTICS

LASP ARCHITECTURE

57

slide-58
SLIDE 58

STORE:

REPLICATED, SHARDED VARIABLE STORE PROCESSESS SYNCHRONIZE ON VARIABLES

58

slide-59
SLIDE 59

REPLICATED:

DISTRIBUTED WITH RIAK CORE QUORUM REQUESTS; ANTI-ENTROPY PROTOCOL

59

slide-60
SLIDE 60

HYBRID:

DISTRIBUTE PROGRAMS; R/W WITH LOCAL STORE CENTRALIZED EXECUTION

60

slide-61
SLIDE 61

EXAMPLES

61

slide-62
SLIDE 62

%% Create initial set. {ok, S1} = lasp:declare(Type), %% Add elements to initial set and update. {ok, _} = lasp:update(S1, {add_all, [1,2,3]}, a), %% Create second set. {ok, S2} = lasp:declare(Type), %% Apply map.

  • k = lasp:map(S1, fun(X) -> X * 2 end, S2),
slide-63
SLIDE 63

%% Create initial set. {ok, S1} = lasp_core:declare(Type, Store), %% Add elements to initial set and update. {ok, _} = lasp_core:update(S1, {add_all, [1,2,3]}, a, Store), %% Create second set. {ok, S2} = lasp_core:declare(Type, Store), %% Apply map.

  • k = lasp_core:map(S1, fun(X) -> X * 2 end, S2, Store),
slide-64
SLIDE 64

AD COUNTER

64

slide-65
SLIDE 65

AD COUNTER:

TRACKS AD IMPRESSIONS PUSHES ADVERTISEMENTS TO THE CLIENT DISABLES AD AT 50,000+ IMPRESSIONS CLIENTS DISPLAY ADS WHEN OFFLINE

65

slide-66
SLIDE 66

INFORMATION FLOW

AD COUNTER

66

slide-67
SLIDE 67

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read ≥ 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3

67

slide-68
SLIDE 68

Ads Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Riot Ads Product Union

68

slide-69
SLIDE 69

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Ads Contracts Riot Ads Rovio Ads Filter Product Read ≥ 50,000 Remove Increment Union Lasp Operation User-Maintained CRDT 69

slide-70
SLIDE 70

Ads Contracts Ads Contracts Ads With Contracts Filter Product Union Lasp Operation User-Maintained CRDT

70

slide-71
SLIDE 71

Ads Contracts Ads Contracts Ads With Contracts Filter Product Read Lasp Operation

71

slide-72
SLIDE 72

Ads Contracts Ads With Contracts Filter Read Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3

72

slide-73
SLIDE 73

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read ≥ 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3

73

slide-74
SLIDE 74

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Contracts Ads Contracts Riot Ads Rovio Ads Product Read ≥ 50,000 Remove Increment Union Lasp Operation 74

slide-75
SLIDE 75

Ads Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3

75

slide-76
SLIDE 76

INFORMATION FLOW:

MONOTONIC METADATA TO PREVENT DUPLICATE PROPAGATION

76

slide-77
SLIDE 77

EXAMPLE CODE

AD COUNTER

77

slide-78
SLIDE 78

%% @doc Client process; standard recursive looping server. client(Id, AdsWithContracts, PreviousValue) -> receive view_ad -> %% Get current ad list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts, PreviousValue), AdList = riak_dt_orset:value(AdList0), case length(AdList) of 0 -> %% No advertisements left to display; ignore %% message. client(Id, AdsWithContracts, AdList0); _ -> %% Select a random advertisement from the list of %% active advertisements. {#ad{counter=Ad}, _} = lists:nth( random:uniform(length(AdList)), AdList), %% Increment it. {ok, _} = lasp:update(Ad, increment, Id), lager:info("Incremented ad counter: ~p", [Ad]), client(Id, AdsWithContracts, AdList0) end end.

slide-79
SLIDE 79

%% @doc Server functions for the advertisement counter. After 5 views, %% disable the advertisement. %% server({#ad{counter=Counter}=Ad, _}, Ads) -> %% Blocking threshold read for 5 advertisement impressions. {ok, _} = lasp:read(Counter, 5), %% Remove the advertisement. {ok, _} = lasp:update(Ads, {remove, Ad}, Ad), lager:info("Removing ad: ~p", [Ad]).

slide-80
SLIDE 80

%% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("Ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(RovioAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(TriforkAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?SET),

  • k = lasp:union(RovioAds, TriforkAds, Ads),

%% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?SET), lists:map(fun(Id) -> {ok, _} = lasp:update(Contracts, {add, #contract{id=Id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?SET),

  • k = lasp:product(Ads, Contracts, AdsContracts),

%% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?SET), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end,

  • k = lasp:filter(AdsContracts, FilterFun, AdsWithContracts),

%% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?SET), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(Id) -> ClientPid = spawn_link(?MODULE, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(Clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?SET), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts), AdList = riak_dt_orset:value(AdList0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(Ad) -> ServerPid = spawn_link(?MODULE, server, [Ad, Ads]), {ok, _} = lasp:update(Servers, {add, ServerPid}, undefined) end, AdList),

slide-81
SLIDE 81

%% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("Ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(RovioAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(TriforkAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?SET),

  • k = lasp:union(RovioAds, TriforkAds, Ads),

%% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?SET), lists:map(fun(Id) -> {ok, _} = lasp:update(Contracts, {add, #contract{id=Id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?SET),

  • k = lasp:product(Ads, Contracts, AdsContracts),

%% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?SET), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end,

  • k = lasp:filter(AdsContracts, FilterFun, AdsWithContracts),

%% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?SET), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(Id) -> ClientPid = spawn_link(?MODULE, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(Clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?SET), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts), AdList = riak_dt_orset:value(AdList0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(Ad) -> ServerPid = spawn_link(?MODULE, server, [Ad, Ads]), {ok, _} = lasp:update(Servers, {add, ServerPid}, undefined) end, AdList),

slide-82
SLIDE 82

%% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("Ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(RovioAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(TriforkAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?SET),

  • k = lasp:union(RovioAds, TriforkAds, Ads),

%% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?SET), lists:map(fun(Id) -> {ok, _} = lasp:update(Contracts, {add, #contract{id=Id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?SET),

  • k = lasp:product(Ads, Contracts, AdsContracts),

%% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?SET), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end,

  • k = lasp:filter(AdsContracts, FilterFun, AdsWithContracts),

%% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?SET), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(Id) -> ClientPid = spawn_link(?MODULE, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(Clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?SET), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts), AdList = riak_dt_orset:value(AdList0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(Ad) -> ServerPid = spawn_link(?MODULE, server, [Ad, Ads]), {ok, _} = lasp:update(Servers, {add, ServerPid}, undefined) end, AdList),

slide-83
SLIDE 83

%% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("Ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(RovioAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(TriforkAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?SET),

  • k = lasp:union(RovioAds, TriforkAds, Ads),

%% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?SET), lists:map(fun(Id) -> {ok, _} = lasp:update(Contracts, {add, #contract{id=Id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?SET),

  • k = lasp:product(Ads, Contracts, AdsContracts),

%% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?SET), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end,

  • k = lasp:filter(AdsContracts, FilterFun, AdsWithContracts),

%% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?SET), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(Id) -> ClientPid = spawn_link(?MODULE, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(Clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?SET), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts), AdList = riak_dt_orset:value(AdList0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(Ad) -> ServerPid = spawn_link(?MODULE, server, [Ad, Ads]), {ok, _} = lasp:update(Servers, {add, ServerPid}, undefined) end, AdList),

slide-84
SLIDE 84

%% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("Ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(RovioAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(TriforkAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?SET),

  • k = lasp:union(RovioAds, TriforkAds, Ads),

%% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?SET), lists:map(fun(Id) -> {ok, _} = lasp:update(Contracts, {add, #contract{id=Id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?SET),

  • k = lasp:product(Ads, Contracts, AdsContracts),

%% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?SET), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end,

  • k = lasp:filter(AdsContracts, FilterFun, AdsWithContracts),

%% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?SET), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(Id) -> ClientPid = spawn_link(?MODULE, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(Clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?SET), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts), AdList = riak_dt_orset:value(AdList0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(Ad) -> ServerPid = spawn_link(?MODULE, server, [Ad, Ads]), {ok, _} = lasp:update(Servers, {add, ServerPid}, undefined) end, AdList),

slide-85
SLIDE 85

DISTRIBUTION

AD COUNTER

85

slide-86
SLIDE 86

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read ≥ 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3

86

slide-87
SLIDE 87

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read ≥ 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3

87

slide-88
SLIDE 88

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read ≥ 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3

88

slide-89
SLIDE 89

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read ≥ 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3

89

slide-90
SLIDE 90

DISTRIBUTION BOUNDARIES:

ARBITRARY ALLOWS COMPOSITION OF ENTIRE SYSTEM

90

slide-91
SLIDE 91

MATERIALIZED VIEWS

91

slide-92
SLIDE 92

MATERIALIZED VIEWS:

INCREMENTALLY UPDATING, DELTA PROPAGATION SCHEMA BASED OR DYNAMIC; 2I IN RIAK

92

slide-93
SLIDE 93

DATABASE AS STREAM OF UPDATES

93

slide-94
SLIDE 94

Riak K1 K2 K3 K1 K2

94

slide-95
SLIDE 95

Riak K1 K2 K3 K1 K2 RS1 RS2 RS3

95

slide-96
SLIDE 96

Riak K1 K2 K3 K1 K2 RS1 RS2 RS3 K1 K1 K2 K2 K3

96

slide-97
SLIDE 97

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3

N1 N2 N3 97

slide-98
SLIDE 98

COMPOSE PROGRAMS

98

slide-99
SLIDE 99

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3

N1 N2 N3

Lasp Stream Processor Template

99

slide-100
SLIDE 100

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3

N1 N2 N3

Lasp Stream Processor Template Lasp All Objects Program

100

slide-101
SLIDE 101

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3

N1 N2 N3

Lasp Stream Processor Template Lasp All Objects Program Lasp “Over 65” Filter

101

slide-102
SLIDE 102

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3

N1 N2 N3

Lasp Stream Processor Template Lasp All Objects Program Lasp “Over 65” Filter Lasp “Over 80” Filter

102

slide-103
SLIDE 103

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3

N1 N2 N3

Lasp Stream Processor Template Lasp All Objects Program Lasp “Over 65” Filter Lasp “Over 80” Filter Lasp “Named Chris” Filter (L)

103

slide-104
SLIDE 104

DISTRIBUTE APPLICATIONS

104

slide-105
SLIDE 105

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3

N1 N2 N3

Lasp “Named Chris” Filter (L) L’’’ L’’ L’

105

slide-106
SLIDE 106

MERGE RESULTS (SEC)

106

N1 N2 N3

L’ L’ L’

slide-107
SLIDE 107

SUM RESULTS

107

RS1

L’

RS2

L’’

slide-108
SLIDE 108

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3

N1 N2 N3

Execution of Lasp “Named Chris” Filter (L) L’’’ L’’ L’

108

slide-109
SLIDE 109

CACHE:

CACHE RESULTS OF MERGE SPECULATIVELY EXECUTE BASED ON DIVERGENCE

109

slide-110
SLIDE 110

“INTERNET OF THINGS”

110

slide-111
SLIDE 111

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3

N1 N2 N3

Execution of Lasp “Named Chris” Filter (L) L’’’ L’’ L’

111

slide-112
SLIDE 112

IoT Network K1 K2 K3 K1 K2 S1 S2 S3 U1 U1 U2 U2 U3 Execution of Lasp Temperature Sensor > 90 F L’’’ L’’ L’

112

slide-113
SLIDE 113

IoT Network S1 S2 S3 U1 U1 U2 U2 U3 Execution of Lasp Temperature Sensor > 90 F L’’’ L’’ L’

113

slide-114
SLIDE 114

IOT:

EXECUTE AT THE EDGE WRITE CODE THINKING ABOUT “ALL” DATA

114

slide-115
SLIDE 115

RELATED WORK

115

slide-116
SLIDE 116

DISTRIBUTED OZ

RELATED WORK

116

slide-117
SLIDE 117

DERFLOWL

RELATED WORK

117

slide-118
SLIDE 118

BLOOML

RELATED WORK

118

slide-119
SLIDE 119

LVARS

RELATED WORK

119

slide-120
SLIDE 120

D-STREAMS

RELATED WORK

120

slide-121
SLIDE 121

SUMMINGBIRD

RELATED WORK

121

slide-122
SLIDE 122

FUTURE WORK

122

slide-123
SLIDE 123

INVARIANT PRESERVATION

FUTURE WORK

123

slide-124
SLIDE 124

CAUSAL+ CONSISTENCY

FUTURE WORK

124

slide-125
SLIDE 125

ORSWOT OPTIMIZATION

FUTURE WORK

125

slide-126
SLIDE 126

DELTA STATE-CRDTs

FUTURE WORK

126

slide-127
SLIDE 127

OPERATION-BASED CRDTs

FUTURE WORK

127

slide-128
SLIDE 128

DEFORESTATION

FUTURE WORK

128

slide-129
SLIDE 129

GITHUB.COM/CMEIKLEJOHN/LASP

SOURCE

129

slide-130
SLIDE 130

DERFLOW DISTRIBUTED DETERMINISTIC DATAFLOW PROGRAMMING FOR ERLANG

ERLANG WORKSHOP 2014

130

slide-131
SLIDE 131

LASP A LANGUAGE FOR DISTRIBUTED, EVENTUALLY CONSISTENT COMPUTATIONS WITH CRDTs

PAPOC / EUROSYS 2015

131

slide-132
SLIDE 132

NOVEMBER 4-6, 2015 SAN FRANCISCO, CA

RICON 2015

132

slide-133
SLIDE 133

SYNCFREE IS A EUROPEAN RESEARCH PROJECT TAKING PLACE FOR 3 YEARS, STARING OCTOBER 2013, AND IS FUNDED BY THE EUROPEAN UNION, GRANT AGREEMENT N°609551.

SYNCFREE

133

slide-134
SLIDE 134

Questions?

Please remember to evaluate via the GOTO Guide App