PCIe and DMA in MirageOS Fabian Bonk Wednesday 20 th May, 2020 - - PowerPoint PPT Presentation

▶
pcie and dma in mirageos
SMART_READER_LITE
LIVE PREVIEW

PCIe and DMA in MirageOS Fabian Bonk Wednesday 20 th May, 2020 - - PowerPoint PPT Presentation

Apr 07, 2024 184 likes •820 views

Chair of Network Architectures and Services Department of Informatics Technical University of Munich PCIe and DMA in MirageOS Fabian Bonk Wednesday 20 th May, 2020 Chair of Network Architectures and Services Department of Informatics

slide-1
SLIDE 1

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

PCIe and DMA in MirageOS

Fabian Bonk

Wednesday 20th May, 2020 Chair of Network Architectures and Services Department of Informatics Technical University of Munich

slide-2
SLIDE 2

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

What is MirageOS? MirageOS is a library operating system that constructs unikernels for secure, high- performance network applications across a variety of cloud computing and mobile platforms.

  • F. Bonk — PCIe and DMA in MirageOS

2

slide-3
SLIDE 3

Unikernels

What’s a Unikernel?

  • Entire application compiled into bootable VM image
  • Include necessary operating system functionality via libraries
  • F. Bonk — PCIe and DMA in MirageOS

3

slide-4
SLIDE 4

Unikernels

Unikernels vs. virtual machines [1]

  • F. Bonk — PCIe and DMA in MirageOS

4

slide-5
SLIDE 5

Unikernels

Why Unikernels?

  • high degree of separation
  • low resource usage
  • flexible runtime(s) (run on hypervisors, standard OS, microcontrollers)
  • safety benefits of high-level languages
  • fewer loc → fewer bugs
  • F. Bonk — PCIe and DMA in MirageOS

5

slide-6
SLIDE 6

MirageOS

  • F. Bonk — PCIe and DMA in MirageOS

6

slide-7
SLIDE 7

MirageOS

OCaml unikernel operating system https://mirage.io/

  • written in OCaml
  • generates Xen (incl. QubesOS) and Solo5 (KVM) Unikernels
  • can also generate standard executables (Linux, macOS, ...)
  • 172 173 repos on GitHub
  • F. Bonk — PCIe and DMA in MirageOS

7

slide-8
SLIDE 8

MirageOS

OCaml unikernel operating system https://mirage.io/

  • written in OCaml
  • generates Xen (incl. QubesOS) and Solo5 (KVM) Unikernels
  • can also generate standard executables (Linux, macOS, ...)
  • 172 173 repos on GitHub
  • F. Bonk — PCIe and DMA in MirageOS

8

slide-9
SLIDE 9

Example: Echo server

  • pen Lwt.Infix

module Main (S : Mirage_stack.V4) = struct (* RFC 862 - read payloads and repeat them back *) let rec echo flow = S.TCPV4.read flow >>= function | Error _ | Ok `Eof -> S.TCPV4.close flow | Ok `Data buf -> S.TCPV4.write flow buf >>= function | Error _ -> S.TCPV4.close flow | Ok () -> echo flow let start s = S.listen_tcpv4 s ~port:7 echo; S.listen s end

  • F. Bonk — PCIe and DMA in MirageOS

9

slide-10
SLIDE 10

Layering

Q: What is a HTTPS stack really?

  • F. Bonk — PCIe and DMA in MirageOS

10

slide-11
SLIDE 11

Layering

Q: What is a HTTPS stack really? A: Some code on top of a TLS stack!

  • F. Bonk — PCIe and DMA in MirageOS

10

slide-12
SLIDE 12

Layering

Q: What is a HTTPS stack really? A: Some code on top of a TLS stack! Q: What is a TLS stack really?

  • F. Bonk — PCIe and DMA in MirageOS

10

slide-13
SLIDE 13

Layering

Q: What is a HTTPS stack really? A: Some code on top of a TLS stack! Q: What is a TLS stack really? A: Some code on top of a TCP stack!

  • F. Bonk — PCIe and DMA in MirageOS

10

slide-14
SLIDE 14

Layering

Q: What is a HTTPS stack really? A: Some code on top of a TLS stack! Q: What is a TLS stack really? A: Some code on top of a TCP stack! Q: What is a TCP stack really?

  • F. Bonk — PCIe and DMA in MirageOS

10

slide-15
SLIDE 15

Layering

Q: What is a HTTPS stack really? A: Some code on top of a TLS stack! Q: What is a TLS stack really? A: Some code on top of a TCP stack! Q: What is a TCP stack really? A: Some code on top of an IP stack!

  • F. Bonk — PCIe and DMA in MirageOS

10

slide-16
SLIDE 16

Layering

Q: What is a HTTPS stack really? A: Some code on top of a TLS stack! Q: What is a TLS stack really? A: Some code on top of a TCP stack! Q: What is a TCP stack really? A: Some code on top of an IP stack! Q: What is an IP stack really?

  • F. Bonk — PCIe and DMA in MirageOS

10

slide-17
SLIDE 17

Layering

Q: What is a HTTPS stack really? A: Some code on top of a TLS stack! Q: What is a TLS stack really? A: Some code on top of a TCP stack! Q: What is a TCP stack really? A: Some code on top of an IP stack! Q: What is an IP stack really? A: Some code on top of an Ethernet stack!

  • F. Bonk — PCIe and DMA in MirageOS

10

slide-18
SLIDE 18

Layering

Q: What is a HTTPS stack really? A: Some code on top of a TLS stack! Q: What is a TLS stack really? A: Some code on top of a TCP stack! Q: What is a TCP stack really? A: Some code on top of an IP stack! Q: What is an IP stack really? A: Some code on top of an Ethernet stack! Q: What is an Ethernet stack really?

  • F. Bonk — PCIe and DMA in MirageOS

10

slide-19
SLIDE 19

Layering

Q: What is a HTTPS stack really? A: Some code on top of a TLS stack! Q: What is a TLS stack really? A: Some code on top of a TCP stack! Q: What is a TCP stack really? A: Some code on top of an IP stack! Q: What is an IP stack really? A: Some code on top of an Ethernet stack! Q: What is an Ethernet stack really? A: Some code on top of a network device!

  • F. Bonk — PCIe and DMA in MirageOS

10

slide-20
SLIDE 20

Layering

Let’s do some functional programming!

  • F. Bonk — PCIe and DMA in MirageOS

11

slide-21
SLIDE 21

Layering

Let’s do some functional programming! HTTPS stack :

  • F. Bonk — PCIe and DMA in MirageOS

11

slide-22
SLIDE 22

Layering

Let’s do some functional programming! HTTPS stack : TLS interface → HTTP interface

  • F. Bonk — PCIe and DMA in MirageOS

11

slide-23
SLIDE 23

Layering

Let’s do some functional programming! HTTPS stack : TLS interface → HTTP interface TLS stack :

  • F. Bonk — PCIe and DMA in MirageOS

11

slide-24
SLIDE 24

Layering

Let’s do some functional programming! HTTPS stack : TLS interface → HTTP interface TLS stack : TCP interface → TLS interface

  • F. Bonk — PCIe and DMA in MirageOS

11

slide-25
SLIDE 25

Layering

Let’s do some functional programming! HTTPS stack : TLS interface → HTTP interface TLS stack : TCP interface → TLS interface TCP stack :

  • F. Bonk — PCIe and DMA in MirageOS

11

slide-26
SLIDE 26

Layering

Let’s do some functional programming! HTTPS stack : TLS interface → HTTP interface TLS stack : TCP interface → TLS interface TCP stack : IP interface → TCP interface

  • F. Bonk — PCIe and DMA in MirageOS

11

slide-27
SLIDE 27

Layering

Let’s do some functional programming! HTTPS stack : TLS interface → HTTP interface TLS stack : TCP interface → TLS interface TCP stack : IP interface → TCP interface IP stack :

  • F. Bonk — PCIe and DMA in MirageOS

11

slide-28
SLIDE 28

Layering

Let’s do some functional programming! HTTPS stack : TLS interface → HTTP interface TLS stack : TCP interface → TLS interface TCP stack : IP interface → TCP interface IP stack : Ethernet interface → IP interface

  • F. Bonk — PCIe and DMA in MirageOS

11

slide-29
SLIDE 29

Layering

Let’s do some functional programming! HTTPS stack : TLS interface → HTTP interface TLS stack : TCP interface → TLS interface TCP stack : IP interface → TCP interface IP stack : Ethernet interface → IP interface Ethernet stack :

  • F. Bonk — PCIe and DMA in MirageOS

11

slide-30
SLIDE 30

Layering

Let’s do some functional programming! HTTPS stack : TLS interface → HTTP interface TLS stack : TCP interface → TLS interface TCP stack : IP interface → TCP interface IP stack : Ethernet interface → IP interface Ethernet stack : Network device → Ethernet interface

  • F. Bonk — PCIe and DMA in MirageOS

11

slide-31
SLIDE 31

Layering

Let’s do some functional programming! HTTPS stack : TLS interface → HTTP interface TLS stack : TCP interface → TLS interface TCP stack : IP interface → TCP interface IP stack : Ethernet interface → IP interface Ethernet stack : Network device → Ethernet interface Network device :

  • F. Bonk — PCIe and DMA in MirageOS

11

slide-32
SLIDE 32

Layering

Let’s do some functional programming! HTTPS stack : TLS interface → HTTP interface TLS stack : TCP interface → TLS interface TCP stack : IP interface → TCP interface IP stack : Ethernet interface → IP interface Ethernet stack : Network device → Ethernet interface Network device : magic

  • F. Bonk — PCIe and DMA in MirageOS

11

slide-33
SLIDE 33

Layering

Network stack, assemble! module HTTPS_Interface = HTTP (TLS (TCP (IP (Ethernet (TAP_device)))))

  • F. Bonk — PCIe and DMA in MirageOS

12

slide-34
SLIDE 34

Layering

Network stack, assemble! module HTTPS_Interface = HTTP (TLS (TCP (IP (Ethernet (TAP_device)))))

  • r

module HTTPS_Interface = HTTP (TLS (TCP_socket))

  • F. Bonk — PCIe and DMA in MirageOS

12

slide-35
SLIDE 35

Layering

Network stack, assemble! module HTTPS_Interface = HTTP (TLS (TCP (IP (Ethernet (TAP_device)))))

  • r

module HTTPS_Interface = HTTP (TLS (TCP_socket)) How about this? module HTTPS_Interface = HTTP (TLS (TCP (IP (Ethernet (Network_driver (PCIe_device))))))

  • F. Bonk — PCIe and DMA in MirageOS

12

slide-36
SLIDE 36

Structure Linux MirageOS

mirage-net-unix mirage-block-unix

User Application

block device TAP

  • F. Bonk — PCIe and DMA in MirageOS

13

slide-37
SLIDE 37

Example: Echo server

  • pen Lwt.Infix

module Main (S : Mirage_stack.V4) = struct (* RFC 862 - read payloads and repeat them back *) let rec echo flow = S.TCPV4.read flow >>= function | Error _ | Ok `Eof -> S.TCPV4.close flow | Ok `Data buf -> S.TCPV4.write flow buf >>= function | Error _ -> S.TCPV4.close flow | Ok () -> echo flow let start s = S.listen_tcpv4 s ~port:7 echo; S.listen s end

  • F. Bonk — PCIe and DMA in MirageOS

14

slide-38
SLIDE 38

Example: Echo server

How to build: Build a normal binary and use a TAP device and the OCaml network stack: $ mirage configure -t unix --net direct && make Build a normal binary and use the OS network stack: $ mirage configure -t unix --net socket && make Build a standalone Unikernel for deployment on Solo5/KVM: $ mirage configure -t hvt && make

  • F. Bonk — PCIe and DMA in MirageOS

15

slide-39
SLIDE 39

Goal

$ mirage configure -t hvt --net <some-driver> --pci 0000:ab:cd.e && make Add necessary functionality for writing device drivers for MirageOS.

  • F. Bonk — PCIe and DMA in MirageOS

16

slide-40
SLIDE 40

Goal

Approach

  • add PCIe device category to MirageOS ecosystem
  • mirage-pci interface library
  • mirage-pci-solo5 wrapper library
  • mirage-pci-unix wrapper library
  • modify ixy.ml
  • F. Bonk — PCIe and DMA in MirageOS

17

slide-41
SLIDE 41

Solo5

What is Solo5? A sandboxed execution environment for unikernels run unikernels on:

  • Linux KVM
  • Linux seccomp
  • FreeBSD/OpenBSD vmm
  • Muen
  • Genode
  • F. Bonk — PCIe and DMA in MirageOS

18

slide-42
SLIDE 42

Solo5

Solo5-hvt

  • hardware virtualized tender
  • create KVM virtual machine
  • load ELF binary
  • pass messages between unikernel and host devices
  • F. Bonk — PCIe and DMA in MirageOS

19

slide-43
SLIDE 43

Solo5

Linux hvt KVM MirageOS

hvt-bindings hvt-modules Manifest mirage-solo5 mirage-net-solo5 mirage-block-solo5

User Application

block device TAP

  • F. Bonk — PCIe and DMA in MirageOS

20

slide-44
SLIDE 44

PCIe

$ ls -l /sys/bus/pci/devices/$DEVICE/ total 0

  • r--r--r-- 1 root root 4096 Jan

7 13:49 class

  • rw-r--r-- 1 root root

256 Jan 7 13:49 config lrwxrwxrwx 1 root root 0 Jan 7 13:49 driver -> ../../../bus/pci/drivers/some-driver

  • rw------- 1 root root

32 Jan 7 13:49 resource0

  • rw------- 1 root root 8192 Jan

7 13:49 resource1

  • r--r--r-- 1 root root 4096 Jan

7 13:49 vendor # ...

  • F. Bonk — PCIe and DMA in MirageOS

21

slide-45
SLIDE 45

PCIe

  • F. Bonk — PCIe and DMA in MirageOS

22

slide-46
SLIDE 46

DMA

What is DMA? Direct Memory Access Flip a magic bit in /sys/bus/pci/devices/$DEVICE/config to enable Program the IOMMU using VFIO PCIe device and Unikernel can see the same memory!

  • F. Bonk — PCIe and DMA in MirageOS

23

slide-47
SLIDE 47

mirage-pci

module type Mirage_pci.S = sig (* error handling omitted *) type t val disconnect : t -> unit Lwt.t val vendor_id : t -> int val device_id : t -> int val class_code : t -> int val subclass_code : t -> int val progif : t -> int val bar0 : t -> Cstruct.t option val bar1 : t -> Cstruct.t option val bar2 : t -> Cstruct.t option val bar3 : t -> Cstruct.t option val bar4 : t -> Cstruct.t option val bar5 : t -> Cstruct.t option val dma : t -> Cstruct.t val name : t -> string end

  • F. Bonk — PCIe and DMA in MirageOS

24

slide-48
SLIDE 48

mirage-pci-unix

Linux MirageOS

mirage-pci-unix

User Application

VFIO

PCIe regions/DMA-ready memory

IOMMU PCIe device driver MMU

  • F. Bonk — PCIe and DMA in MirageOS

25

slide-49
SLIDE 49

mirage-pci-solo5

Linux hvt KVM MirageOS

PCIe binding PCIe module Manifest mirage-solo5 mirage-pci-solo5

User Application

VFIO

PCIe regions/DMA-ready memory

IOMMU PCIe device driver MMU

  • F. Bonk — PCIe and DMA in MirageOS

26

slide-50
SLIDE 50

mirage-pci-solo5

zero page 0x00000000000 global descriptor table 0x00000001000 page table 0x00000002000 unikernel arguments 0x00000010000 text 0x00000100000 rodata data heap ❤❤❤❤❤❤❤❤❤❤❤❤❤❤ ❤ ❤ ❤ ❤ ❤ ❤ ❤ ❤ ❤ ❤ ❤ ❤ ❤ ❤ ❤ ❤ stack pci0 BAR0 0x08000000000 pci0 BAR2 0x08080000000 pci1 BAR0 0x08400000000 DMA 0x10000000000

  • F. Bonk — PCIe and DMA in MirageOS

27

slide-51
SLIDE 51

Excursion: ixy.ml

What is ixy.ml? https://github.com/ixy-languages/ixy.ml

  • userspace network driver
  • written in OCaml
  • targets Linux
  • targets Intel ixgbe NICs
  • F. Bonk — PCIe and DMA in MirageOS

28

slide-52
SLIDE 52

Latency measurements Linux Benchmark program

VFIO

PCIe regions/DMA-ready memory

IOMMU 82599ES ixy.ml MMU

  • F. Bonk — PCIe and DMA in MirageOS

29

slide-53
SLIDE 53

Latency measurements

Linux MirageOS

mirage-pci-unix

Benchmark program

VFIO

PCIe regions/DMA-ready memory

IOMMU 82599ES ixy.ml MMU

  • F. Bonk — PCIe and DMA in MirageOS

30

slide-54
SLIDE 54

Latency measurements

Linux hvt KVM MirageOS

PCIe binding PCIe module Manifest mirage-solo5 mirage-pci-solo5

Benchmark program

VFIO

PCIe regions/DMA-ready memory

IOMMU 82599ES ixy.ml MMU

  • F. Bonk — PCIe and DMA in MirageOS

31

slide-55
SLIDE 55

Latency measurements

  • pen Mirage

let main = foreign "Unikernel.Main" (pci @-> job) let pci0 = pcidev (* configuration omitted *) "pci0" let () = register "pci" [ main $ pci0 ] ~packages:[ package "ixy-core"; package "mirage-net-ixy" ]

  • F. Bonk — PCIe and DMA in MirageOS

32

slide-56
SLIDE 56

Latency measurements

module Main (S: Mirage_pci.S) = struct module Ixy = Ixy_core.Make (Pci_mirage.Make (S)) let start pci0 = let dev = Ixy.create ~pci:pci0 ~rxq:1 ~txq:1 in while true do let rx = Ixy.rx_batch dev 0 in Ixy.tx_batch_busy_wait dev 0 rx; done; Lwt.return_unit end

  • F. Bonk — PCIe and DMA in MirageOS

33

slide-57
SLIDE 57

Latency measurements

let usage () = Ixy_core.Log.error "Usage: %s <pci_addr>" Sys.argv.(0) let () = if Array.length Sys.argv <> 2 then usage (); let pci = match Ixy.of_string Sys.argv.(1) with | None -> usage () | Some pci -> pci in let dev = Ixy.create ~pci ~rxq:1 ~txq:1 in while true do let rx = Ixy.rx_batch dev 0 in Ixy.tx_batch_busy_wait dev 0 rx done

  • F. Bonk — PCIe and DMA in MirageOS

34

slide-58
SLIDE 58

Latency measurements

90 99 99.9 99.99 20 40 60 80 100 Percentile Latency [µs] Latency when forwarding 1000 Mbit/s

hvt mirage-unix Linux

  • F. Bonk — PCIe and DMA in MirageOS

35

slide-59
SLIDE 59

Latency measurements

90 99 99.9 99.99 20 40 60 80 100 Percentile Latency [µs] Latency when forwarding 2000 Mbit/s

hvt mirage-unix Linux

  • F. Bonk — PCIe and DMA in MirageOS

36

slide-60
SLIDE 60

Latency measurements

90 99 99.9 99.99 20 40 60 80 100 Percentile Latency [µs] Latency when forwarding 4000 Mbit/s

hvt mirage-unix Linux

  • F. Bonk — PCIe and DMA in MirageOS

37

slide-61
SLIDE 61

iperf

2 4 6 8 10 Unix + TAP Unix + Socket Unix + ixy.ml hvt + TAP hvt + ixy.ml TCP throughput [Gbit/s]

  • F. Bonk — PCIe and DMA in MirageOS

38

slide-62
SLIDE 62

Bibliography

  • A. Madhavapeddy, R. Mortier, C. Rotsos, D. Scott, B. Singh, T. Gazagnaire, S. Smith, S.

Hand, J. Crowcroft, “Unikernels: Library Operating Systems for the Cloud,” SIGPLAN No- tices, vol. 48, pp. 461-472, March 2013

  • F. Bonk — PCIe and DMA in MirageOS

39