On the Impact of Isolation Costs on Locality-aware Cloud Scheduling - - PowerPoint PPT Presentation

On the Impact of Isolation Costs on Locality-aware Cloud Scheduling

Ankit Bhardwaj, Meghana G Gupta, Ryan Stutsman University of Utah

Scalable Computer Systems Lab www.utah.systems

Code Isolation-cost Aware Scheduling

Three r recent s t shifts ts i in th the c cloud

Cloud N Networking P Performance → 1 100 G Gbps, m , microsecond r round-tr trips Rethink o

• f c

f code i isolation s schemes → M Meltdown, , Sp Spectre, V , VT-x, , eB eBPF, W , WASM Granular, S , Serverless A Applications → V Visibility a and P Placement a a f fine g grain

Code Isolation-cost Aware Scheduling

Three r recent s t shifts ts i in th the c cloud

Cloud N Networking P Performance → 1 100 G Gbps, m , microsecond r round-tr trips Rethink o

• f c

f code i isolation s schemes → M Meltdown, , Sp Spectre, V , VT-x, , eB eBPF, W , WASM Granular, S , Serverless A Applications → V Visibility a and P Placement a a f fine g grain

Code Isolation-cost Aware Scheduling

Three r recent s t shifts ts i in th the c cloud

Cloud N Networking P Performance → 1 100 G Gbps, m , microsecond r round-tr trips Rethink o

• f c

f code i isolation s schemes → M Meltdown, , Sp Spectre, V , VT-x, , eB eBPF, W , WASM Granular, S , Serverless A Applications → V Visibility a and P Placement a a f fine g grain

Code Isolation-cost Aware Scheduling

Cloud N Networking P Performance → 1 100 G Gbps, m , microsecond r round-tr trips Rethink o

• f c

f code i isolation s schemes → M Meltdown, , Sp Spectre, V , VT-x, , eB eBPF, W , WASM Granular, S , Serverless A Applications → V Visibility a and P Placement a a f fine g grain Diversity a and F Flexibility i in P Placement, W , Workloads, a , and Is Isolation C Costs

Code Isolation-cost Aware Scheduling

Cloud N Networking P Performance → 1 100 G Gbps, m , microsecond r round-tr trips Rethink o

• f c

f code i isolation s schemes → M Meltdown, , Sp Spectre, V , VT-x, , eB eBPF, W , WASM Granular, S , Serverless A Applications → V Visibility a and P Placement a a f fine g grain Diversity a and F Flexibility i in P Placement, W , Workloads, a , and Is Isolation C Costs Is Isolation- and d data-mo moveme ment-cost A Aware S Scheduling f for C Cloud C Compute

It It i is t time f for a a h holistic, c , cost-aware a approach t to s scheduling i in t the c cloud

Past: State + Application on One VM

• Compute/storage together on one machine; VMs access state

locally

• Pr

Probl blem: Resource stranding

• Idle compute when storage capacity is the limiting factor
• Idle storage when compute capacity is the limiting factor
• Costly to reorganize

DATA DATA

Today: Disaggregation

• Soluti

tion: Separate compute from storage

• New P

Problem: : High data movement costs (multiple gets/puts)

• RPC, serialization/deserialization
• TCP/transport
• memcpys
• Substantial c

costs a at g gigabits/second

Move compute to storage at finer grain?

• Soluti

tion: : storage-side computation over stored data

• But, high tenant density at storage to homogenize/balance load
• Need granular decomposition of application logic
• Pr

Probl blem: Many tenants sharing storage; code isolation is hard

• Process creation and context switch add up

Key Idea: Isolation-cost Aware Scheduler

• Placement of computation in the cloud can improve efficiency
• by eliminating data movement,
• but it also must reason about code isolation costs to do so.
• Profile
• inter-function interaction in applications,
• data access and locality patterns,
• networking, dispatch, and isolation domain context switch costs
• Global fine-grained, core-level choices at microsecond-timescales

Challenges for Isolation-cost Aware Scheduling

• Need for Fine-grained Applications
• Workload Characterization
• Profiling and Understanding Context Switch Costs
• Provisioning, Re-provisioning, and Placement
• Dealing with Intermediate State

Challenge #0: Need Finer-grained Apps

• Scheduler must be able to "see" into applications

to optimize

• Soluti

tion: serverless

• Functions can be individually placed
• Creates visibility into applications
• Supports alternative isolation schemes
• Malleable interface
• Today implementations do not tap into these

potential benefits

VM VM

λ λ λ λ λ λ λ λ λ λ

Challenge #1: Workload Characterization

• Pr

Probl blem: No insight into function's network and data access costs

• Soluti

tion: Profile functions to capture

• data access patterns and locality
• runtime distribution

0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 1 2 4 Function Throughput (millions of invocations/second) Data Record Accesses (accesses/invocation) Client-side Function + Disaggregated Access Server-side Function + Colocated Access

• Place functions that access many records or much data at storage
• Dynamically shift to idle compute when server is overloaded
• Even simple schemes can work: counting accesses & runtime

VT-x VMs for isolation, SR-IOV+IOMMU for dispatch

Challenge #2: Code Isolation Costs

Pr Probl blem: isolation costs vary depending on workload VMs: hw protection & dispatch

• Too expensive to context switch
• Good if high per-tenant throughput

Containers: sw dispatch

• Need ms-scale length requests
• Good for timesharing CPU

Language Runtimes: pure sw

• Good for short-running functions

with constrained logic

App 2 VT-x VM App 1 VT-x VM Processes for isolation, software demultiplexing for dispatch App 1 Address Space App 1 Address Space App 1 Address Space App 1 Address Space App 1 Address Space App 1 Address Space Page Table Switching

Comparing Three Hw Isolation Schemes

• Paging/conventional process context switch is always costly
• Low tenant counts → MPK Page Table Entry Coloring Fastest
• Higher tenants counts → Extended Page Table Switching Fastest

Best s t scheme d depends o

• n t

tenant c t count a t and r request r t rates

Challenge #3: Provisioning & Placement

• Problem:

: Function properties change over time

• in data access patterns
• in computational costs
• in distribution of functions invoked
• Churn and instability forces new placement decisions
• VMs, containers, etc have different start, stop, migration costs
• Soluti

tion: scheduling must model stability and variance of workload

• In compute costs, invocation frequency, and data access

Storage N Node Storage N Node Storage N Node Compute N Node Compute N Node Compute N Node

Preliminary Design Ideas

Task D Dispatching

• Two-level scheduling avoids idle CPUs but

limits queue imbalance

• History at global level, route invocations to

avoid context switching

• Global knowledge of data placement

Statistics, L , Load, & , & P Prediction

• Core and task level stats collection
• Push via RDMA writes
• Low-cost with frequent updates
• 100s to 1000s of machines pushing

updates each second

• Use in assessing workload stability
• Used by scheduler to promote/demote

functions between isolation schemes

Load B Balancer Global S Scheduler Stored D Data Local T Task S Scheduler Local T Task S Scheduler

In Incoming Func Function n In Invocations

Discussion Questions

• Cloud process model
• Cloud f

function i interfaces ( (that d differ f from P POSIX IX) a are l likely t to t take h hold?

• Security risks
• Larger attack surface, but works around vulnerabilities with less reengineering
• Which i

isolation s schemes a and r runtimes l likely t to b be s sufficiently t trustworthy?

• Workloads
• What w

will f future, m , more g granular s serverless w workloads l look l like?

• What w

ways m might t there b be t to a approximate t these w workloads u using p public d data?

• Pricing
• How m

might i improved b but h hard-to to-predict e efficiency g gains b be r reflected i in p pricing?

Conclusion

• Kernel-bypass → low-latency, high-throughput storage services
• These gains are now showing up in the cloud
• Fast networks → more data movement
• Small functions over data, but code isolation cuts into gains
• Key idea: different code isolation schemes have different costs
• Dynamically understand data movement and code isolation costs
• Run different functions with different schemes based on runtime profiling
• For more details, check out our project website or reach out to me at

meghana@cs.utah.edu.