2DFQ: Two-Dimensional Fair Queuing for Multi-Tenant Cloud Services - - PowerPoint PPT Presentation

▶
2dfq two dimensional fair queuing for multi tenant cloud
SMART_READER_LITE
LIVE PREVIEW

2DFQ: Two-Dimensional Fair Queuing for Multi-Tenant Cloud Services - - PowerPoint PPT Presentation

Sep 18, 2022 130 likes •1.94k views

2DFQ: Two-Dimensional Fair Queuing for Multi-Tenant Cloud Services Jona nathan an Ma Mace Peter Bodik Madanlal Musuvathi Brown University Microsoft Microsoft Rodrigo Fonseca Krishnan Varadarajan Brown University Microsoft Containers

slide-1
SLIDE 1

2DFQ: Two-Dimensional Fair Queuing for Multi-Tenant Cloud Services

Jona nathan an Ma Mace

Brown University

Peter Bodik

Microsoft

Madanlal Musuvathi

Microsoft

Rodrigo Fonseca

Brown University

Krishnan Varadarajan

Microsoft

slide-2
SLIDE 2

2

Containers / VMs

slide-3
SLIDE 3

2

Containers / VMs

Shared Systems: Storage, Database, Queueing, etc.

slide-4
SLIDE 4

2

Containers / VMs

Shared Systems: Storage, Database, Queueing, etc.

slide-5
SLIDE 5

Shared Process

3

slide-6
SLIDE 6

Shared Process Tenants Request Queues

…

3

slide-7
SLIDE 7

Shared Process Tenants Request Queues time Threads

…

3

slide-8
SLIDE 8

Shared Process Tenants Request Queues time Threads

…

3

slide-9
SLIDE 9

Shared Process Tenants Request Queues time Threads

…

FIFO or ad-hoc

3

slide-10
SLIDE 10

Shared Process Tenants Request Queues time Threads

…

4

slide-11
SLIDE 11

Shared Process Tenants Request Queues time Threads

…

Flows Packet

4

slide-12
SLIDE 12

Shared Process Tenants Request Queues time Threads

…

Flows Packet Switch Link

4

slide-13
SLIDE 13

Shared Process Tenants Request Queues time Threads

…

Flows Packet Switch Link Weighted Fair Queueing (WFQ)

4

slide-14
SLIDE 14

Shared Process Tenants Request Queues time Threads

…

Weighted Fair Queueing (WFQ)

Multiple threads Multiple concurrent requests

4

slide-15
SLIDE 15

…

5

slide-16
SLIDE 16

…

5

slide-17
SLIDE 17

… Two-Dimensional Fair Queueing (2DFQ)

5

slide-18
SLIDE 18

6

slide-19
SLIDE 19

A size =1

(fast database read)

6

slide-20
SLIDE 20

D size =10 A size =1

(fast database read) (slow database scan)

6

slide-21
SLIDE 21

D size =10 A size =1 One thread: time

(fast database read) (slow database scan)

6

slide-22
SLIDE 22

D size =10 A size =1 One thread: time

(fast database read) (slow database scan)

6

slide-23
SLIDE 23

D size =10 A size =1 One thread: time

(fast database read) (slow database scan)

6

slide-24
SLIDE 24

D size =10 A size =1 … One thread: time

(fast database read) (slow database scan)

6

slide-25
SLIDE 25

D size =10 A size =1 … One thread: time high-throughput burst zero-throughput lull throughput

(fast database read) (slow database scan)

6

slide-26
SLIDE 26

D size =10 A size =1 … One thread: time high-throughput burst zero-throughput lull Ideal: throughput

(fast database read) (slow database scan)

6

slide-27
SLIDE 27

D size =10 A size =1 … One thread: time high-throughput burst zero-throughput lull Ideal: throughput

(fast database read) (slow database scan)

6

slide-28
SLIDE 28

D size =10 A size =1 … One thread: time high-throughput burst zero-throughput lull Ideal: throughput

(fast database read) (slow database scan)

6

slide-29
SLIDE 29

D size =10 A size =1

7

slide-30
SLIDE 30

D size =10 A size =1 C B

7

slide-31
SLIDE 31

D size =10 A size =1 C B time

7

slide-32
SLIDE 32

D size =10 A size =1 C B time

7

slide-33
SLIDE 33

D size =10 A size =1 C B time

7

slide-34
SLIDE 34

D size =10 A size =1 C B time time

7

slide-35
SLIDE 35

D size =10 A size =1 C B time time

7

slide-36
SLIDE 36

D size =10 A size =1 C B time time

7

slide-37
SLIDE 37

D size =10 A size =1 C B time time

7

slide-38
SLIDE 38

D size =10 A size =1 C B time time throughput throughput

7

slide-39
SLIDE 39

D size =10 A size =1 C B time time throughput throughput

WFQ, SFQ, …

Weighted Fair Queuing Start-time Fair Queueing

7

slide-40
SLIDE 40

D size =10 A size =1 C B time time throughput throughput

WF²Q

Worst-Case Fair Weighted Fair Queuing

WFQ, SFQ, …

Weighted Fair Queuing Start-time Fair Queueing

7

slide-41
SLIDE 41

D size =10 A size =1 C B

8

slide-42
SLIDE 42

D size =10 A size =1 C B time

8

slide-43
SLIDE 43

D size =10 A size =1 C B time

8

slide-44
SLIDE 44

D size =10 A size =1 C B time

8

slide-45
SLIDE 45

D size =10 A size =1 C B time

8

slide-46
SLIDE 46

D size =10 A size =1 C B time time

8

slide-47
SLIDE 47

D size =10 A size =1 C B time time

8

slide-48
SLIDE 48

D size =10 A size =1 C B time time

8

slide-49
SLIDE 49

D size =10 A size =1 C B time time

8

slide-50
SLIDE 50

D size =10 A size =1 C B time time Ideal:

8

slide-51
SLIDE 51

D size =10 A size =1 C B time time

More threads → Opportunity to reduce burstiness

8

slide-52
SLIDE 52

D size =10 A size =1 C B time time

More threads → Opportunity to reduce burstiness

WFQ, SFQ, WF²Q, …

Weighted Fair Queueing Start-time Fair Queueing Worst-case Fair Weighted Fair Queueing

8

slide-53
SLIDE 53

D size =10 A size =1 C B time time

More threads → Opportunity to reduce burstiness

WFQ, SFQ, WF²Q, …

Weighted Fair Queueing Start-time Fair Queueing Worst-case Fair Weighted Fair Queueing

2DFQ

Two-dimensional Fair Queueing

8

slide-54
SLIDE 54

Challenges

9

slide-55
SLIDE 55

A size =1 Tenants with small requests are affected

10

slide-56
SLIDE 56

C size =10 A size =1 Tenants with small requests are affected Burstiness is proportional to size of large requests

10

slide-57
SLIDE 57

C size =10 A size =1

Cloud services: 4+ orders of magnitude variation in cost

Tenants with small requests are affected Burstiness is proportional to size of large requests

10

slide-58
SLIDE 58

C size =10 A size =1

Cloud services: 4+ orders of magnitude variation in cost

Tenants with small requests are affected Burstiness is proportional to size of large requests size =10 A size =1 Size is used by scheduler to make scheduling decisions C

10

slide-59
SLIDE 59

C size =10 A size =1

Cloud services: 4+ orders of magnitude variation in cost

Tenants with small requests are affected Burstiness is proportional to size of large requests A size =? size =? Size is used by scheduler to make scheduling decisions C

10

slide-60
SLIDE 60

C size =10 A size =1

Cloud services: 4+ orders of magnitude variation in cost

Tenants with small requests are affected Burstiness is proportional to size of large requests A

Cloud services: Estimation using model or moving averages

size =? size =? Size is used by scheduler to make scheduling decisions C

10

slide-61
SLIDE 61

Key

11

1st quartile 3rd quartile distribution shape 1st percentile median 99th percentile

  • utliers
slide-62
SLIDE 62

Key

11

1st quartile 3rd quartile distribution shape 1st percentile median 99th percentile

  • utliers
slide-63
SLIDE 63

12

slide-64
SLIDE 64

Unpredictability Cost Variation Quality Of Service

12

slide-65
SLIDE 65

Unpredictability Cost Variation Quality Of Service

12

slide-66
SLIDE 66

Unpredictability Cost Variation Quality Of Service

12

slide-67
SLIDE 67

Two-Dimensional Fair Queueing

13

slide-68
SLIDE 68

D size =10 A size =1 C B time

WFQ

14

slide-69
SLIDE 69

D size =10 A size =1 C B time

WFQ

14

slide-70
SLIDE 70

D size =10 A size =1 C B time

WFQ

14

slide-71
SLIDE 71

D size =10 A size =1 C B time

WFQ

15

slide-72
SLIDE 72

D size =10 A size =1 C B time

WFQ

15

slide-73
SLIDE 73

D size =10 A size =1 C B time

WFQ

15

slide-74
SLIDE 74

D size =10 A size =1 C B time

WFQ

15

slide-75
SLIDE 75

D size =10 A size =1 C B 1 time

WFQ

16

slide-76
SLIDE 76

D size =10 A size =1 C B 1 1 time

WFQ

17

slide-77
SLIDE 77

D size =10 A size =1 C B 1 1 time

WFQ

18

slide-78
SLIDE 78

D size =10 A size =1 C B 1 1 time

v(t)=½

WFQ

18

slide-79
SLIDE 79

D size =10 A size =1 C B 2 2 time

v(t)=1

WFQ

19

slide-80
SLIDE 80

D size =10 A size =1 C B 3 3 time

v(t)=1½

WFQ

20

slide-81
SLIDE 81

D size =10 A size =1 C B 4 4 time

v(t)=2

WFQ

21

slide-82
SLIDE 82

D size =10 A size =1 C B 5 5 time

v(t)=2½

WFQ

22

slide-83
SLIDE 83

D size =10 A size =1 C B 6 6 time

v(t)=3

WFQ

23

slide-84
SLIDE 84

D size =10 A size =1 C B 7 7 time

v(t)=3½

WFQ

24

slide-85
SLIDE 85

D size =10 A size =1 C B 8 8 time

v(t)=4

WFQ

25

slide-86
SLIDE 86

D size =10 A size =1 C B 9 9 time

v(t)=4½

WFQ

26

slide-87
SLIDE 87

D size =10 A size =1 C B 10 10 time

v(t)=5

WFQ

27

slide-88
SLIDE 88

D size =10 A size =1 C B 10 10 time

v(t)=5

WFQ

27

slide-89
SLIDE 89

D size =10 A size =1 C B 10 10 10 10 time

v(t)=10

WFQ

28

slide-90
SLIDE 90

D size =10 A size =1 C B time

v(t)=0

WF²Q

29

slide-91
SLIDE 91

D size =10 A size =1 C B time

v(t)=0

WF²Q

29

slide-92
SLIDE 92

D size =10 A size =1 C B 1 1 time

v(t)=½

WF²Q

30

slide-93
SLIDE 93

D size =10 A size =1 C B 1 1 time

v(t)=½

WF²Q

30

slide-94
SLIDE 94

D size =10 A size =1 C B 1 1 time

v(t)=½

WF²Q

30

slide-95
SLIDE 95

D size =10 A size =1 C B 1 1 10 10 time

v(t)=5½

WF²Q

31

slide-96
SLIDE 96

D size =10 A size =1 C B 2 2 10 10 time

v(t)=6

WF²Q

32

slide-97
SLIDE 97

D size =10 A size =1 C B 3 3 10 10 time

v(t)=6 ½

WF²Q

33

slide-98
SLIDE 98

D size =10 A size =1 C B 4 4 10 10 time

v(t)=7

WF²Q

34

slide-99
SLIDE 99

D size =10 A size =1 C B 5 5 10 10 time

v(t)=7 ½

WF²Q

35

slide-100
SLIDE 100

D size =10 A size =1 C B 6 6 10 10 time

v(t)=8

WF²Q

36

slide-101
SLIDE 101

D size =10 A size =1 C B 7 7 10 10 time

v(t)=8½

WF²Q

37

slide-102
SLIDE 102

D size =10 A size =1 C B 8 8 10 10 time

v(t)=9

WF²Q

38

slide-103
SLIDE 103

D size =10 A size =1 C B 9 9 10 10 time

v(t)=9½

WF²Q

39

slide-104
SLIDE 104

D size =10 A size =1 C B 10 10 10 10 time

v(t)=10

WF²Q

40

slide-105
SLIDE 105

D size =10 A size =1 C B 11 11 10 10 time

v(t)=10½

WF²Q

41

slide-106
SLIDE 106

D size =10 A size =1 C B 11 11 10 10 time

v(t)=10½

WF²Q

41

slide-107
SLIDE 107

D size =10 A size =1 C B 11 11 10 10 time

v(t)=10½

WF²Q

41

slide-108
SLIDE 108

D size =10 A size =1 C B time

v(t)=0

2DFQ

42

slide-109
SLIDE 109

D size =10 A size =1 C B time

v(t)=0

2DFQ

42

slide-110
SLIDE 110

D size =10 A size =1 C B 1 1 time

v(t)=½

2DFQ

43

slide-111
SLIDE 111

D size =10 A size =1 C B 1 1 time

v(t)=½

2DFQ

43

slide-112
SLIDE 112

D size =10 A size =1 C B 1 1 10 time

v(t)=½

2DFQ

44

slide-113
SLIDE 113

D size =10 A size =1 C B 1 1 10 time

v(t)=½

2DFQ

44

slide-114
SLIDE 114

D size =10 A size =1 C B 1 1 10 time

v(t)=½

1 1

2DFQ

44

slide-115
SLIDE 115

D size =10 A size =1 C B 2 1 10 time

v(t)=1

2DFQ

45

slide-116
SLIDE 116

D size =10 A size =1 C B 2 1 10 time

v(t)=1

Eligible 𝑠𝑘, 𝑗 = Start 𝑠𝑘 − 𝑗

𝑜 × size 𝑠𝑘

2DFQ

45

slide-117
SLIDE 117

D size =10 A size =1 C B 2 2 10 time

v(t)=1½

Eligible 𝑠𝑘, 𝑗 = Start 𝑠𝑘 − 𝑗

𝑜 × size 𝑠𝑘

2DFQ

46

slide-118
SLIDE 118

D size =10 A size =1 C B 3 2 10 time

v(t)=2

Eligible 𝑠𝑘, 𝑗 = Start 𝑠𝑘 − 𝑗

𝑜 × size 𝑠𝑘

2DFQ

47

slide-119
SLIDE 119

D size =10 A size =1 C B 3 3 10 time

v(t)=2½

Eligible 𝑠𝑘, 𝑗 = Start 𝑠𝑘 − 𝑗

𝑜 × size 𝑠𝑘

2DFQ

48

slide-120
SLIDE 120

D size =10 A size =1 C B 4 3 10 time

v(t)=3

Eligible 𝑠𝑘, 𝑗 = Start 𝑠𝑘 − 𝑗

𝑜 × size 𝑠𝑘

2DFQ

49

slide-121
SLIDE 121

D size =10 A size =1 C B 4 4 10 time

v(t)=3½

Eligible 𝑠𝑘, 𝑗 = Start 𝑠𝑘 − 𝑗

𝑜 × size 𝑠𝑘

2DFQ

50

slide-122
SLIDE 122

D size =10 A size =1 C B 5 4 10 time

v(t)=4

Eligible 𝑠𝑘, 𝑗 = Start 𝑠𝑘 − 𝑗

𝑜 × size 𝑠𝑘

2DFQ

51

slide-123
SLIDE 123

D size =10 A size =1 C B 5 5 10 time

v(t)=4½

Eligible 𝑠𝑘, 𝑗 = Start 𝑠𝑘 − 𝑗

𝑜 × size 𝑠𝑘

2DFQ

52

slide-124
SLIDE 124

D size =10 A size =1 C B 6 5 10 time

v(t)=5

Eligible 𝑠𝑘, 𝑗 = Start 𝑠𝑘 − 𝑗

𝑜 × size 𝑠𝑘

2DFQ

53

slide-125
SLIDE 125

D size =10 A size =1 C B 6 6 10 time

v(t)=5½

Eligible 𝑠𝑘, 𝑗 = Start 𝑠𝑘 − 𝑗

𝑜 × size 𝑠𝑘

2DFQ

54

slide-126
SLIDE 126

D size =10 A size =1 C B 6 6 10 time

v(t)=5½

Eligible 𝑠𝑘, 𝑗 = Start 𝑠𝑘 − 𝑗

𝑜 × size 𝑠𝑘

2DFQ

54

slide-127
SLIDE 127

D size =10 A size =1 C B 6 6 10 time

v(t)=5½

Eligible 𝑠𝑘, 𝑗 = Start 𝑠𝑘 − 𝑗

𝑜 × size 𝑠𝑘

2DFQ

54

slide-128
SLIDE 128

D size =10 A size =1 C B 6 6 10 time

v(t)=5½

Eligible 𝑠𝑘, 𝑗 = Start 𝑠𝑘 − 𝑗

𝑜 × size 𝑠𝑘

2DFQ

6 6

54

slide-129
SLIDE 129

D size =10 A size =1 C B 7 6 10 10 time

v(t)=6

Eligible 𝑠𝑘, 𝑗 = Start 𝑠𝑘 − 𝑗

𝑜 × size 𝑠𝑘

2DFQ

55

slide-130
SLIDE 130

D size =10 A size =1 C B 7 7 10 10 time

v(t)=6½

Eligible 𝑠𝑘, 𝑗 = Start 𝑠𝑘 − 𝑗

𝑜 × size 𝑠𝑘

2DFQ

56

slide-131
SLIDE 131

Unknown Costs

57

slide-132
SLIDE 132

Unknown Costs

57

slide-133
SLIDE 133

Unknown Costs

57

slide-134
SLIDE 134

Unknown Costs

57

slide-135
SLIDE 135

Unknown Costs

57

slide-136
SLIDE 136

Unknown Costs

58

slide-137
SLIDE 137

Unknown Costs

58

slide-138
SLIDE 138

Unknown Costs

Pessimistic cost estimation

58

slide-139
SLIDE 139

Evaluation

Compare 2DFQ to WFQ and WF²Q Discrete event simulator with Azure Storage workloads More experiment results in the paper, evaluating:

  • Burstiness
  • Fairness
  • Tail latency

59

slide-140
SLIDE 140

60

slide-141
SLIDE 141

A

50 tenants with size ≈ 1

60

slide-142
SLIDE 142

A

50 tenants with size ≈ 1

C

50 tenants with size ≈ 1000

60

slide-143
SLIDE 143

A

50 tenants with size ≈ 1

C

50 tenants with size ≈ 1000 16 threads 1000 units/second Costs known by scheduler

60

slide-144
SLIDE 144

C A

50 tenants with size ≈ 1 50 tenants with size ≈ 1000 16 threads 1000 units/second Costs known by scheduler

60

A’s Service Rate

slide-145
SLIDE 145

C A

50 tenants with size ≈ 1 50 tenants with size ≈ 1000 16 threads 1000 units/second Costs known by scheduler

60

A’s Service Rate

slide-146
SLIDE 146

C A

50 tenants with size ≈ 1 50 tenants with size ≈ 1000 16 threads 1000 units/second Costs known by scheduler

60

A’s Service Rate

slide-147
SLIDE 147

C A

50 tenants with size ≈ 1 50 tenants with size ≈ 1000 16 threads 1000 units/second Costs known by scheduler

60

A’s Service Rate

slide-148
SLIDE 148

C A

50 tenants with size ≈ 1 50 tenants with size ≈ 1000 16 threads 1000 units/second Costs known by scheduler

60

A’s Service Rate

slide-149
SLIDE 149

C A

50 tenants with size ≈ 1 50 tenants with size ≈ 1000 16 threads 1000 units/second Costs known by scheduler

60

A’s Service Rate

slide-150
SLIDE 150

250 Azure Storage tenants 32 threads 1 million units/second Costs known by scheduler

61

slide-151
SLIDE 151

250 Azure Storage tenants 32 threads 1 million units/second Costs known by scheduler

61

slide-152
SLIDE 152

250 Azure Storage tenants 32 threads 1 million units/second Costs known by scheduler

T1

61

slide-153
SLIDE 153

250 Azure Storage workloads 32 threads 1 million units/second Costs known by scheduler

T1

62

slide-154
SLIDE 154

250 Azure Storage workloads 32 threads 1 million units/second Costs known by scheduler

T1

62

slide-155
SLIDE 155

250 Azure Storage workloads 32 threads 1 million units/second Costs known by scheduler

T1

62

slide-156
SLIDE 156

250 Azure Storage workloads 32 threads 1 million units/second Costs known by scheduler

T1

62

slide-157
SLIDE 157

250 Azure Storage workloads 32 threads 1 million units/second Costs known by scheduler

T1

62

slide-158
SLIDE 158

250 Azure Storage workloads 32 threads 1 million units/second Costs known by scheduler

T1

62

slide-159
SLIDE 159

250 Azure Storage workloads 32 threads 1 million units/second Costs known by scheduler

T1

62

slide-160
SLIDE 160

250 Azure Storage workloads 32 threads 1 million units/second Costs known by scheduler

T1

62

slide-161
SLIDE 161

Unpredictable Workloads Widely Varying Costs Quality Of Service

63

slide-162
SLIDE 162

Unpredictable Workloads Widely Varying Costs Quality Of Service

63

slide-163
SLIDE 163

Unpredictable Workloads Widely Varying Costs Quality Of Service

63

slide-164
SLIDE 164

Unpredictable Workloads Widely Varying Costs Quality Of Service

63

slide-165
SLIDE 165

Predictable 2/3 predictable 1/3 unpredictable 1/3 predictable 2/3 unpredictable

T1 T1 T1

64

slide-166
SLIDE 166

Predictable 2/3 predictable 1/3 unpredictable 1/3 predictable 2/3 unpredictable

T1 T1 T1

64

slide-167
SLIDE 167

Predictable 2/3 predictable 1/3 unpredictable 1/3 predictable 2/3 unpredictable

T1 T1 T1

64

slide-168
SLIDE 168

Predictable 2/3 predictable 1/3 unpredictable 1/3 predictable 2/3 unpredictable

T1 T1 T1

64

slide-169
SLIDE 169

Predictable 2/3 predictable 1/3 unpredictable 1/3 predictable 2/3 unpredictable

T1 T1 T1

64

slide-170
SLIDE 170

Predictable 2/3 predictable 1/3 unpredictable 1/3 predictable 2/3 unpredictable

65

slide-171
SLIDE 171

Predictable 2/3 predictable 1/3 unpredictable 1/3 predictable 2/3 unpredictable

65

slide-172
SLIDE 172

Predictable 2/3 predictable 1/3 unpredictable 1/3 predictable 2/3 unpredictable

65

slide-173
SLIDE 173

Predictable 2/3 predictable 1/3 unpredictable 1/3 predictable 2/3 unpredictable

65

slide-174
SLIDE 174

Two-Dimensional Fair Queueing

66

slide-175
SLIDE 175

More threads → Opportunity to reduce burstiness

Two-Dimensional Fair Queueing

66

slide-176
SLIDE 176

More threads → Opportunity to reduce burstiness

Two-Dimensional Fair Queueing

Partitions requests across threads by size

66

slide-177
SLIDE 177

More threads → Opportunity to reduce burstiness

Two-Dimensional Fair Queueing

Partitions requests across threads by size Co-locates unpredictable and expensive workloads

66

slide-178
SLIDE 178

More threads → Opportunity to reduce burstiness

Two-Dimensional Fair Queueing

Partitions requests across threads by size Co-locates unpredictable and expensive workloads

Unpredictable Workloads Widely Varying Costs Quality Of Service

66

slide-179
SLIDE 179

More threads → Opportunity to reduce burstiness

Two-Dimensional Fair Queueing

Partitions requests across threads by size Co-locates unpredictable and expensive workloads Less burstiness Reduced tail latency

Unpredictable Workloads Widely Varying Costs Quality Of Service

66