SLIDE 1
1
SVM on Intel Graphics
Jesse Barnes Intel Open Source Technology Center
2
- What is SVM?
- Discussion of current practices
- SVM OS and driver modifications
- Device options and implications
SVM on Intel Graphics Jesse Barnes Intel Open Source Technology - - PowerPoint PPT Presentation
SVM on Intel Graphics Jesse Barnes Intel Open Source Technology - - PowerPoint PPT Presentation
SVM on Intel Graphics Jesse Barnes Intel Open Source Technology Center 1 What is SVM? Discussion of current practices SVM OS and driver modifications Device options and implications 2 SVM defined Pointer sharing between CPU
SLIDE 2
SLIDE 3
3
SVM defined
Pointer sharing between CPU and GPU
SLIDE 4
4
But wait, there’s more!
- Pointer sharing with buffers
○ Offset in device address space matches offset in process address space ○ Can use a buffer allocation API to manage device page tables ○ Allows OpenCL “fine grained, buffered” model
- Pointer sharing with a bufferless API
○ Requires pinning or page fault support ○ Allows OpenCL “fine grained, bufferless” model ○ Requires core OS and driver support ○ Ideal for application programmers
- Important to be clear when discussing “SVM”
SLIDE 5
5
SVM PCIe and VT-d extensions
- ATS – Address Translation Services
- Basic IOMMU support
- PASID – Process Address Space ID
- Tells IOMMU which page tables to use, equivalent to the ASID on the
CPU side
- PRI – Page Request Interface
- Allows functions to raise page faults to the IOMMU
- VT-d SVM
- Extends root complex IOMMU to comprehend x86 page table formats
SLIDE 6
6
Good, old days
DRAM
CPU
Root Complex
PCIe EP CPU page tables GFX GPU page tables
Generally devices used physical addrs
SLIDE 7
7
VT-d (and big servers) make it more complicated...
DMA address is indirect
DRAM
CPU
Root Complex
PCIe EP CPU page tables GFX GPU page tables DMAR page tables
SLIDE 8
8
Current interfaces (sans softpin)
- Buffer alloc
- Buffer map – allow direct CPU access to buffer
- Buffer read/write – just like read/write on I/O buffers
- Buffer share – create handle for inter-process sharing
- Buffer query – check status of buffer
- Exec buffer – execute code, pass in whole buffer list
○ Synchronizes with all of the above
SLIDE 9
9
Kernel
Kernel buffer
Device
User buffer
1 2
3 a
1
3 a
2
3 b
1. Alloc buffer, syscall or ioctl to kernel (maybe for both) 2. Alloc buffer, request device to initiate DMA
- 3a1. Device DMAs to/from kernel buffer
- 3a2. Kernel copies to user buffer
- 3b. Device DMAs directly to translated
process address (pinned!)
Process address space
SLIDE 10
10
SVM changes
SLIDE 11
11
DRAM
CPU
Root Comple x
PCIe EP CPU page tables GFX
VT-d hw
VT-d looks up correct tables with PASID
DMAR page tables
Potential to share page tables
Possible SVM model
SLIDE 12
12
Kernel
Kernel buffer
Device
User buffer
1 2 3
1. Alloc buffer, syscall or ioctl to kernel (maybe for both) 2. Alloc buffer, request device to initiate DMA 3. Device DMAs directly to translated process address (with faulting!)
Process address space
SLIDE 13
13
Driver implications
- Must alloc/track PASID
- Either linked to process or device specific context struct
- Optionally design new APIs
- Potentially just “execute starting at this address” or “write to this address”
- Device<->CPU synchronization is flexible
- PCIe atomic ops
- Memory polling
- Interrupts passed from device to process through driver specific
mechanism
SLIDE 14
14
Possible SVM driver interfaces
- Malloc, mmap, etc – normal libc interfaces for memory
management
- Context create ioctl takes a flag to indicate you want an SVM
context
- Can mix & match SVM and non-SVM execution
- Single interface for submission: i915_exec_mm ioctl
- struct drm_i915_exec_mm { batch_ptr; ctx_id; ring, flags; fence; deps;
}
- Synchronization through interrupt forwarding
- Command buffer contains interrupt command, driver maps that back
to fd event for app
SLIDE 15
15
SVM for devices
SLIDE 16
16
SVM device options
- Adding PASID support
○ Can get you a shared address space on supported platforms ○ Application to device interaction still potentially complex – need to manage pinning, potentially include buffer alloc APIs
- Adding page faults
○ Allows for bufferless APIs – simple malloc and use of pointers across CPU/device boundaries ○ Major and minor faults can be handled ○ What to do while servicing the fault?
SLIDE 17
17
Context handling
- Wait for fault handling
○ Simple, but potentially poor device utilization, depending on use model
- Restart/abort on fault
○ Simply re-submit work after fault is handled, starting from the top ○ Also simple to implement, but potentially even worse utilization than waiting
- Context switch on fault
○ Save device context on fault, switch to new context like on CPU ○ Potentially very complex for device designers ○ Added complexity for drivers
SLIDE 18