Kevin Laatz & Ciara Loftus FOSDEM 2020 Introduction - - PowerPoint PPT Presentation

▶
kevin laatz ciara loftus fosdem 2020 introduction
SMART_READER_LITE
LIVE PREVIEW

Kevin Laatz & Ciara Loftus FOSDEM 2020 Introduction - - PowerPoint PPT Presentation

Feb 16, 2024 227 likes •751 views

Kevin Laatz & Ciara Loftus FOSDEM 2020 Introduction https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf

slide-1
SLIDE 1

Kevin Laatz & Ciara Loftus FOSDEM 2020

slide-2
SLIDE 2

Introduction

https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf

https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwikxb2Z7pnnAhVNY8AKHchXDfAQjhx6BAgBEAI&url=https%3A%2F%2 Fwww.cdrecycler.com%2Finterstate-waste-apex-merger.aspx&psig=AOvVaw0QFTtUxvONAgVjKXlMC_-v&ust=1579873577016036

slide-3
SLIDE 3

Introduction

  • Userspace Libraries and Drivers.
  • Accelerate packet processing workloads.
  • Runs on wide variety of CPU architectures.
  • Has its own memory management subsystem.
  • Device specific PMDs (Poll Mode Drivers).

https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf

https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwikxb2Z7pnnAhVNY8AKHchXDfAQjhx6BAgBEAI&url=https%3A%2F%2 Fwww.cdrecycler.com%2Finterstate-waste-apex-merger.aspx&psig=AOvVaw0QFTtUxvONAgVjKXlMC_-v&ust=1579873577016036

slide-4
SLIDE 4

Introduction

  • Userspace Libraries and Drivers.
  • Accelerate packet processing workloads.
  • Runs on wide variety of CPU architectures.
  • Has its own memory management subsystem.
  • Device specific PMDs (Poll Mode Drivers).

AF_XDP

  • Kernel based address family.
  • Optimized for high performance packet processing.
  • AF_XDP sockets redirect packets to userspace.
  • By-passes kernel network stack
  • “In-kernel fast path”.

https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf

https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwikxb2Z7pnnAhVNY8AKHchXDfAQjhx6BAgBEAI&url=https%3A%2F%2 Fwww.cdrecycler.com%2Finterstate-waste-apex-merger.aspx&psig=AOvVaw0QFTtUxvONAgVjKXlMC_-v&ust=1579873577016036

slide-5
SLIDE 5

Introduction

  • Userspace Libraries and Drivers.
  • Accelerate packet processing workloads.
  • Runs on wide variety of CPU architectures.
  • Has its own memory management subsystem.
  • Device specific PMDs (Poll Mode Drivers).

AF_XDP

  • Kernel based address family.
  • Optimized for high performance packet processing.
  • AF_XDP sockets redirect packets to userspace.
  • By-passes kernel network stack
  • “In-kernel fast path”.

Traditional DPDK Model

Userspace

DPDK apps

i40e PMD ixgbe PMD Other PMDs Kernel space

igb_uio vfio

https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf

https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwikxb2Z7pnnAhVNY8AKHchXDfAQjhx6BAgBEAI&url=https%3A%2F%2 Fwww.cdrecycler.com%2Finterstate-waste-apex-merger.aspx&psig=AOvVaw0QFTtUxvONAgVjKXlMC_-v&ust=1579873577016036

slide-6
SLIDE 6

Introduction

  • Userspace Libraries and Drivers.
  • Accelerate packet processing workloads.
  • Runs on wide variety of CPU architectures.
  • Has its own memory management subsystem.
  • Device specific PMDs (Poll Mode Drivers).

AF_XDP

  • Kernel based address family.
  • Optimized for high performance packet processing.
  • AF_XDP sockets redirect packets to userspace.
  • By-passes kernel network stack
  • “In-kernel fast path”.

Traditional DPDK Model

Userspace

DPDK apps

i40e PMD ixgbe PMD Other PMDs Kernel space

igb_uio vfio

DPDK with AF_XDP

Userspace

DPDK apps

DPDK AF_XDP PMD Kernel space

NIC driver

https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf

https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwikxb2Z7pnnAhVNY8AKHchXDfAQjhx6BAgBEAI&url=https%3A%2F%2 Fwww.cdrecycler.com%2Finterstate-waste-apex-merger.aspx&psig=AOvVaw0QFTtUxvONAgVjKXlMC_-v&ust=1579873577016036

slide-7
SLIDE 7

Problem Statement

slide-8
SLIDE 8

Problem Statement

  • The Goal:
  • All DPDK applications should be able to run out-of-the-box with the AF_XDP PMD.
  • DPDK app + AF_XDP PMD should run with good performance.
slide-9
SLIDE 9

Problem Statement

  • The Goal:
  • All DPDK applications should be able to run out-of-the-box with the AF_XDP PMD.
  • DPDK app + AF_XDP PMD should run with good performance.
  • The Challenge:
  • Frameworks like DPDK have their own memory management which come with their own

assumptions and constraints.

slide-10
SLIDE 10

Problem Statement

  • The Goal:
  • All DPDK applications should be able to run out-of-the-box with the AF_XDP PMD.
  • DPDK app + AF_XDP PMD should run with good performance.
  • The Challenge:
  • Frameworks like DPDK have their own memory management which come with their own

assumptions and constraints.

  • Discrepancy between the DPDK and AF_XDP buffer alignment.
  • Prevents direct mapping of DPDK mempool to AF_XDP UMEM.
  • Extra complexity/work negatively impacting performance.
slide-11
SLIDE 11

AF_XDP UMEM* vs DPDK Mbuf Pool

Chunk 0 Chunk 1 Chunk 2 Chunk 3

Page 0 Page 1 Page 2

slide-12
SLIDE 12

AF_XDP UMEM* vs DPDK Mbuf Pool

Chunk 0 Chunk 1 Chunk 2 Chunk 3

Page 0 Page 1 Page 2

  • Area of memory

allocated by the user for packet data

slide-13
SLIDE 13

AF_XDP UMEM* vs DPDK Mbuf Pool

Chunk 0 Chunk 1 Chunk 2 Chunk 3

Page 0 Page 1 Page 2

  • Area of memory

allocated by the user for packet data

  • Split up into equal

sized-chunks

slide-14
SLIDE 14

AF_XDP UMEM* vs DPDK Mbuf Pool

Chunk 0 Chunk 1 Chunk 2 Chunk 3

Page 0 Page 1 Page 2

  • Area of memory

allocated by the user for packet data

  • Split up into equal

sized-chunks

  • RX Path: Kernel places packet data in a chunk for userspace

(DPDK) to retrieve

slide-15
SLIDE 15

AF_XDP UMEM* vs DPDK Mbuf Pool

Chunk 0 Chunk 1 Chunk 2 Chunk 3

Page 0 Page 1 Page 2

  • Area of memory

allocated by the user for packet data

  • Split up into equal

sized-chunks

  • RX Path: Kernel places packet data in a chunk for userspace

(DPDK) to retrieve

  • TX Path: Userspace places packet data in chunk for kernel NIC

driver to transmit

slide-16
SLIDE 16

AF_XDP UMEM* vs DPDK Mbuf Pool

Chunk 0 Chunk 1 Chunk 2 Chunk 3

Page 0 Page 1 Page 2

*Prior to Kernel 5.4

slide-17
SLIDE 17

AF_XDP UMEM* vs DPDK Mbuf Pool

Chunk 0 Chunk 1 Chunk 2 Chunk 3

Page 0 Page 1 Page 2

*Prior to Kernel 5.4

Start address must be PAGE_SIZE aligned

PAGE_SIZE = 4K

slide-18
SLIDE 18

AF_XDP UMEM* vs DPDK Mbuf Pool

Chunk 0 Chunk 1 Chunk 2 Chunk 3

Page 0 Page 1 Page 2

*Prior to Kernel 5.4

Start address must be PAGE_SIZE aligned Chunks must be ^2 sized

PAGE_SIZE = 4K

slide-19
SLIDE 19

AF_XDP UMEM* vs DPDK Mbuf Pool

Chunk 0 Chunk 1 Chunk 2 Chunk 3

Page 0 Page 1 Page 2

*Prior to Kernel 5.4

Start address must be PAGE_SIZE aligned Chunks must be ^2 sized Chunks may not cross page boundaries

PAGE_SIZE = 4K

slide-20
SLIDE 20

Start address must be PAGE_SIZE aligned Chunks must be ^2 sized Chunks may not cross page boundaries

AF_XDP UMEM* vs DPDK Mbuf Pool

Chunk 0 Chunk 1 Chunk 2 Chunk 3

PAGE_SIZE = 4K

DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3

Page 0 Page 1 Page 2 Page 3

*Prior to Kernel 5.4

slide-21
SLIDE 21

Start address must be PAGE_SIZE aligned Chunks must be ^2 sized Chunks may not cross page boundaries Mbufs can be any size

AF_XDP UMEM* vs DPDK Mbuf Pool

Chunk 0 Chunk 1 Chunk 2 Chunk 3

PAGE_SIZE = 4K

DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3

Page 0 Page 1 Page 2 Page 3

*Prior to Kernel 5.4

slide-22
SLIDE 22

Start address must be PAGE_SIZE aligned Chunks must be ^2 sized Chunks may not cross page boundaries Mbufs can be any size Mbufs can have arbitrary alignment

AF_XDP UMEM* vs DPDK Mbuf Pool

Chunk 0 Chunk 1 Chunk 2 Chunk 3

PAGE_SIZE = 4K

DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3

Page 0 Page 1 Page 2 Page 3

*Prior to Kernel 5.4

slide-23
SLIDE 23

Start address must be PAGE_SIZE aligned Chunks must be ^2 sized Chunks may not cross page boundaries Mbufs can be any size Mbufs can have arbitrary alignment

AF_XDP UMEM* vs DPDK Mbuf Pool

Chunk 0 Chunk 1 Chunk 2 Chunk 3

PAGE_SIZE = 4K

DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3

Page 0 Page 1 Page 2 Page 3

To get the highest performing integration of AF_XDP and DPDK, the DPDK mbuf pool must be mapped into the UMEM for a zero copy datapath.

*Prior to Kernel 5.4

slide-24
SLIDE 24

DPDK Solutions

slide-25
SLIDE 25

DPDK Solutions

  • Memcpy packets between

UMEM & mbufpool

  • Cycle-heavy memcpy
  • DPDK 19.05

memcpy

UMEM CHUNK

DPDK MBUF

pkt pkt

Copy Mode

slide-26
SLIDE 26

DPDK Solutions

  • Memcpy packets between

UMEM & mbufpool

  • Cycle-heavy memcpy
  • DPDK 19.05

UMEM CHUNK

DPDK MBUF

pkt pkt

memcpy

UMEM CHUNK

DPDK MBUF

pkt pkt

Copy Mode Alignment API

  • API to change mbuf pool

alignment, then 1:1 map

  • Invasive change
  • No DPDK release
slide-27
SLIDE 27

DPDK Solutions

  • Memcpy packets between

UMEM & mbufpool

  • Cycle-heavy memcpy
  • DPDK 19.05

UMEM CHUNK

DPDK MBUF

pkt

*pkt

UMEM CHUNK

DPDK MBUF

pkt pkt

memcpy

UMEM CHUNK

DPDK MBUF

pkt pkt

Copy Mode Alignment API External Mbuf

  • API to change mbuf pool

alignment, then 1:1 map

  • Invasive change
  • No DPDK release
  • Mbuf points to packet

data in UMEM chunk

  • Additional complexity
  • DPDK 19.08

*

slide-28
SLIDE 28

New Solution: Kernel Arbitrary Alignment

slide-29
SLIDE 29

New Solution: Kernel Arbitrary Alignment

Main benefits of the new solution:

  • Enables arbitrary chunk alignment.

Chunk 0 Chunk 1 Chunk 2 Chunk 3

PAGE_SIZE = 4K

Page 0 Page 1 Page 2 Page 3

slide-30
SLIDE 30

New Solution: Kernel Arbitrary Alignment

Main benefits of the new solution:

  • Enables arbitrary chunk alignment.

Chunk 0 Chunk 1 Chunk 2 Chunk 3

PAGE_SIZE = 4K

Page 0 Page 1 Page 2 Page 3

Chunk 0 Chunk 1 Chunk 2 Chunk 3

slide-31
SLIDE 31

New Solution: Kernel Arbitrary Alignment

Main benefits of the new solution:

  • Enables arbitrary chunk alignment.
  • Enables use of arbitrary chunk size (up to 4k).

Chunk 0 Chunk 1 Chunk 2 Chunk 3

PAGE_SIZE = 4K

Page 0 Page 1 Page 2 Page 3

Chunk 0 Chunk 1 Chunk 2 Chunk 3

slide-32
SLIDE 32

New Solution: Kernel Arbitrary Alignment

Main benefits of the new solution:

  • Enables arbitrary chunk alignment.
  • Enables use of arbitrary chunk size (up to 4k).
  • Chunks can cross page boundaries.

Chunk 0 Chunk 1 Chunk 2 Chunk 3

PAGE_SIZE = 4K

Page 0 Page 1 Page 2 Page 3

Chunk 0 Chunk 1 Chunk 2 Chunk 3

slide-33
SLIDE 33

New Solution: Kernel Arbitrary Alignment

Main benefits of the new solution:

  • Enables arbitrary chunk alignment.
  • Enables use of arbitrary chunk size (up to 4k).
  • Chunks can cross page boundaries.
  • If physically contiguous!

Chunk 0 Chunk 1 Chunk 2 Chunk 3

PAGE_SIZE = 4K

Page 0 Page 1 Page 2 Page 3

Chunk 0 Chunk 1 Chunk 2

Crosses physically non-contiguous page boundary

Chunk 3

slide-34
SLIDE 34

New Solution: Kernel Arbitrary Alignment

Main benefits of the new solution:

  • Enables arbitrary chunk alignment.
  • Enables use of arbitrary chunk size (up to 4k).
  • Chunks can cross page boundaries.
  • If physically contiguous!

Chunk 0 Chunk 1 Chunk 2 Chunk 3

PAGE_SIZE = 4K

Page 0 Page 1 Page 2 Page 3

Chunk 0 Chunk 1 Chunk 2 Chunk 3

slide-35
SLIDE 35

New Solution: Kernel Arbitrary Alignment

Main benefits of the new solution:

  • Enables arbitrary chunk alignment.
  • Enables use of arbitrary chunk size (up to 4k).
  • Chunks can cross page boundaries.
  • If physically contiguous!

Chunk 0 Chunk 1 Chunk 2 Chunk 3

PAGE_SIZE = 4K

Page 0 Page 1 Page 2 Page 3

Chunk 0 Chunk 1 Chunk 2 Chunk 3 64-bit “address” field

AF_XDP Rx/Tx Descriptor:

slide-36
SLIDE 36

New Solution: Kernel Arbitrary Alignment

Main benefits of the new solution:

  • Enables arbitrary chunk alignment.
  • Enables use of arbitrary chunk size (up to 4k).
  • Chunks can cross page boundaries.
  • If physically contiguous!
  • New descriptor format keeps original address which is

useful for buffer recycling.

Chunk 0 Chunk 1 Chunk 2 Chunk 3

PAGE_SIZE = 4K

Page 0 Page 1 Page 2 Page 3

Chunk 0 Chunk 1 Chunk 2 Chunk 3 64-bit “address” field

AF_XDP Rx/Tx Descriptor:

48 bits 16 bits

Offset “Address”

slide-37
SLIDE 37

New Solution: Kernel Arbitrary Alignment

Main benefits of the new solution:

  • Enables arbitrary chunk alignment.
  • Enables use of arbitrary chunk size (up to 4k).
  • Chunks can cross page boundaries.
  • If physically contiguous!
  • New descriptor format keeps original address which is

useful for buffer recycling.

  • Makes integrating AF_XDP with existing frameworks

more seamless.

Chunk 0 Chunk 1 Chunk 2 Chunk 3

PAGE_SIZE = 4K

Page 0 Page 1 Page 2 Page 3

Chunk 0 Chunk 1 Chunk 2 48 bits 16 bits

Offset “Address”

64-bit “address” field

AF_XDP Rx/Tx Descriptor:

New arbitrary alignment enables the UMEM to adapt to the requirements of the application

Chunk 3

slide-38
SLIDE 38

DPDK Integration

UMEM CHUNK

DPDK MBUF

pkt pkt Page 0 Page 1

slide-39
SLIDE 39
  • Now that UMEM alignment constraints

are relaxed, DPDK mbuf pools can be directly mapped into the UMEM

DPDK Integration

UMEM CHUNK

DPDK MBUF

pkt pkt Page 0 Page 1

slide-40
SLIDE 40
  • Now that UMEM alignment constraints

are relaxed, DPDK mbuf pools can be directly mapped into the UMEM

  • Seamless zero-copy is now achievable

DPDK Integration

UMEM CHUNK

DPDK MBUF

pkt pkt Page 0 Page 1

slide-41
SLIDE 41
  • Now that UMEM alignment constraints

are relaxed, DPDK mbuf pools can be directly mapped into the UMEM

  • Seamless zero-copy is now achievable
  • No need to modify existing DPDK

applications - they work OOTB

DPDK Integration

UMEM CHUNK

DPDK MBUF

pkt pkt Page 0 Page 1

slide-42
SLIDE 42
  • Now that UMEM alignment constraints

are relaxed, DPDK mbuf pools can be directly mapped into the UMEM

  • Seamless zero-copy is now achievable
  • No need to modify existing DPDK

applications - they work OOTB

  • Performant and portable solution

DPDK Integration

UMEM CHUNK

DPDK MBUF

pkt pkt Page 0 Page 1

slide-43
SLIDE 43
  • Now that UMEM alignment constraints

are relaxed, DPDK mbuf pools can be directly mapped into the UMEM

  • Seamless zero-copy is now achievable
  • No need to modify existing DPDK

applications - they work OOTB

  • Performant and portable solution
  • DPDK 19.11 feature

DPDK Integration

UMEM CHUNK

DPDK MBUF

pkt pkt Page 0 Page 1

slide-44
SLIDE 44

Summary

Userspace

DPDK apps

DPDK AF_XDP PMD Kernel space

NIC driver

Classification Power Qos Crypto Memory Mgmt vHost Many more! ethtool Ifconfig

DPDK with AF_XDP

Other kernel tools!

slide-45
SLIDE 45
  • DPDK provides a wide range of

functionality to an application, eg.:

Memory & power management, crypto libraries, virtual networking & many more!

Summary

Userspace

DPDK apps

DPDK AF_XDP PMD Kernel space

NIC driver

Classification Power Qos Crypto Memory Mgmt vHost Many more! ethtool Ifconfig

DPDK with AF_XDP

Other kernel tools!

slide-46
SLIDE 46
  • DPDK provides a wide range of

functionality to an application, eg.:

Memory & power management, crypto libraries, virtual networking & many more!

  • AF_XDP provides flexibility and

usability through kernel control paths.

Familiar tools eg. ifconfig, ethtool, etc.

Summary

Userspace

DPDK apps

DPDK AF_XDP PMD Kernel space

NIC driver

Classification Power Qos Crypto Memory Mgmt vHost Many more! ethtool Ifconfig

DPDK with AF_XDP

Other kernel tools!

slide-47
SLIDE 47
  • DPDK provides a wide range of

functionality to an application, eg.:

Memory & power management, crypto libraries, virtual networking & many more!

  • AF_XDP provides flexibility and

usability through kernel control paths.

Familiar tools eg. ifconfig, ethtool, etc.

  • Together, the best of both worlds can

be enjoyed.

Summary

Userspace

DPDK apps

DPDK AF_XDP PMD Kernel space

NIC driver

Classification Power Qos Crypto Memory Mgmt vHost Many more! ethtool Ifconfig

DPDK with AF_XDP

Other kernel tools!

slide-48
SLIDE 48
  • DPDK provides a wide range of

functionality to an application, eg.:

Memory & power management, crypto libraries, virtual networking & many more!

  • AF_XDP provides flexibility and

usability through kernel control paths.

Familiar tools eg. ifconfig, ethtool, etc.

  • Together, the best of both worlds can

be enjoyed.

Summary

Userspace

DPDK apps

DPDK AF_XDP PMD Kernel space

NIC driver

Classification Power Qos Crypto Memory Mgmt vHost Many more! ethtool Ifconfig

DPDK with AF_XDP

Other kernel tools!

High performing, portable, fully-featured, accelerated, usable and flexible applications are possible with DPDK + AF_XDP

slide-49
SLIDE 49

Magnus Karlsson, Björn Töpel, Bruce Richardson, Qi Zhang, Xiaolong Ye, DPDK & Kernel Communities

Thanks to..

slide-50
SLIDE 50
slide-51
SLIDE 51