The Year of the Linux Video Codec Drivers Embedded Linux Conference - - PowerPoint PPT Presentation

▶
the year of the linux video codec drivers
SMART_READER_LITE
LIVE PREVIEW

The Year of the Linux Video Codec Drivers Embedded Linux Conference - - PowerPoint PPT Presentation

Sep 20, 2023 23 likes •507 views

The Year of the Linux Video Codec Drivers Embedded Linux Conference 2017 Portland Laurent Pinchart laurent.pinchart@ideasonboard.com I have given many talks in conferences around the world. This one is special. During breakfast this morning I

slide-1
SLIDE 1 The Year of the Linux Video Codec Drivers Embedded Linux Conference 2017 Portland Laurent Pinchart laurent.pinchart@ideasonboard.com
slide-2
SLIDE 2 I have given many talks in conferences around the world. This one is special. During breakfast this morning I was reading the news, and what I read made me angry and sad. Sad that, in the 21st century, in a country where I have many friends, people who are supposed to be role models for all of us decide to discriminate between humans just for the sake of it. Before travelling to the ELC I have long hesitated on the attitude I needed to have in the face of the political turmoil of the last few months. This morning I decided to build action on top of the anger and that I simply couldn't condone the situation by staying silent. I thus want to dedicate this talk to all the people around the world who fight against discriminations of all kinds. If I could not join them
  • n Monday when they rallied around the country, I want to show them my
support and be part of the sea of pink that started marching on January the
  • 21st. For them all, I have colored all the slides of this presentation in pink.
slide-3
SLIDE 3 The Year of the Linux Video Codec Drivers

Video Encoding

Stateful Codecs Stateless Codecs Next
slide-4
SLIDE 4 Video Encoding Compressed Bitstream Video Decoder Uncompressed Frames 1010101111001011 0111000100101001 0010101101011011 1011101011000011 0111101000111100
slide-5
SLIDE 5 Video Encoding Compressed Bitstream Video Encoder Uncompressed Frames 1010101111001011 0111000100101001 0010101101011011 1011101011000011 0111101000111100
slide-6
SLIDE 6 Video Encoding – Bitstream 100011010110101010111011110101111110101101011111101001001101010101101 111000101110100011000101010011110111001011001100010100000001111101101 000110100010010010101100101110110001000111111111100001010001101110011 101010010001001000100011011010111001101001111110010101111001010010110 101111010010011000011001001010001010111001000110110001010011010001001 000010111100000110001001100101010000100000101011011001100011000100010 001110110001111000110101011001101001101000100011111101100111101001110 111001000000101101111011001111101110100000110110001001000101111001100 001101011000111111001111010010000100011000101101000101001000101111110 001000100110000111001000011011101111001011111010111111000100101111111 010111001011101001001110001111000010101011000010010010100101001101110 011101010000101000001000111010011001000110011111101010110000001110100 101111000001011111011001100000111101110111011001011111110000011001100 111110101001100010000011011111100110001110101110101000001010101101101 011100010100010011010110101001010011100111111000101010110001111111111
slide-7
SLIDE 7 Video Encoding – Bitstream 100011010110101010111011110101111110101101011111101001001101010101101 111000101110100011000101010011110111001011001100010100000001111101101 000110100010010010101100101110110001000111111111100001010001101110011 101010010001001000100011011010111001101001111110010101111001010010110 101111010010011000011001001010001010111001000110110001010011010001001 000010111100000110001001100101010000100000101011011001100011000100010 001110110001111000110101011001101001101000100011111101100111101001110 111001000000101101111011001111101110100000110110001001000101111001100 001101011000111111001111010010000100011000101101000101001000101111110 001000100110000111001000011011101111001011111010111111000100101111111 010111001011101001001110001111000010101011000010010010100101001101110 011101010000101000001000111010011001000110011111101010110000001110100 101111000001011111011001100000111101110111011001011111110000011001100 111110101001100010000011011111100110001110101110101000001010101101101 011100010100010011010110101001010011100111111000101010110001111111111
slide-8
SLIDE 8 Video Encoding – Bitstream 100011010110101010111011110101111110101101011111101001001101010101101 111000101110100011000101010011110111001011001100010100000001111101101 000110100010010010101100101110110001000111111111100001010001101110011 101010010001001000100011011010111001101001111110010101111001010010110 101111010010011000011001001010001010111001000110110001010011010001001 000010111100000110001001100101010000100000101011011001100011000100010 001110110001111000110101011001101001101000100011111101100111101001110 111001000000101101111011001111101110100000110110001001000101111001100 001101011000111111001111010010000100011000101101000101001000101111110 001000100110000111001000011011101111001011111010111111000100101111111 010111001011101001001110001111000010101011000010010010100101001101110 011101010000101000001000111010011001000110011111101010110000001110100 101111000001011111011001100000111101110111011001011111110000011001100 111110101001100010000011011111100110001110101110101000001010101101101 011100010100010011010110101001010011100111111000101010110001111111111
slide-9
SLIDE 9 Video Encoding – Bitstream 00000001 00000001 00000001 00000001 000000010110101010111011110101111110101101011111101001001101010101101 111000101110100011000101010011110111001011001100010100000001111101101 000110100010010010101100101110110001000000000011100001010001101110011 101010010001001000100011011010111001101001111110010101111001010010110 101111010010011000011001001010001010111001000110110001010011010001001 000010111100000110001001100101010000100000101011011001100011000100010 001110110001111000110101011000000011101000100011111101100111101001110 111001000000101101111011001111101110100000110110001001000101111001100 001101011000111111001111010010000100011000101101000101001000101111110 001000100110000111001000011011101111001011111010111111000100101111111 010111001011101001001110001111000010101011000010010010100101001101110 011101010000101000001000111010011001000110011111101010110000000110100 101111000001011111011001100000111101110111011001011111110000011001100 111110101001100010000011011111100110001110101110101000001010101101101 011100010100010011010110101001010011100111111000101010110001111111111
slide-10
SLIDE 10 Video Encoding – Decoder HW HW HW FW Video Decoder 0001 0001 0001101111001011 0111000100101001 0010100001011011 1011101011000011 0111101000111100
slide-11
SLIDE 11 00000001011010101011101111010111111010110101111110100100110101010110 11110001011101000110001010100111101110010110011000101000000011111011 01000110100010010010101100101110110001000000000011100001010001101110 01110101001000100100010001101101011100110100111111001010111100101001 01101011110100100110000110010010100010101110010001101100010100110100 01001000010111100000110001001100101010000100000101011011001100011000 10001000111011000111100011010101100000010110100010001111110110011110 10011101110010000001011011110110011111011101000001101100010010001011 11001100001101011000111111001111010010000100011000101101000101001000 10111111000100010011000011100100001101110111100101111101011111100010 01011111110101110010111010010011100011110000101010110000100100101001 01001101110011101010000101000001000111010011001000110011111101010110 00000011010010111100000101111101100110000011110111011101100101111111 00000110011001111101010011000100000110111111001100011101011101010000 01010101101101011100010100010011010110101001010011100111111000101010 Video Encoding – Decoder 00000001 00000001 00000001 00000001 011010101011101111 011010101011101111 01101010101110111 01101010101110 01100 000000010110101010111011110101111110101101011111101001001101010101101 111000101110100011000101010011110111001011001100010100000001111101101 000110100010010010101100101110110001000000000011100001010001101110011 101010010001001000100011011010111001101001111110010101111001010010110 101111010010011000011001001010001010111001000110110001010011010001001 000010111100000110001001100101010000100000101011011001100011000100010 001110110001111000110101011000000101101000100011111101100111101001110 111001000000101101111011001111101110100000110110001001000101111001100 001101011000111111001111010010000100011000101101000101001000101111110 001000100110000111001000011011101111001011111010111111000100101111111 010111001011101001001110001111000010101011000010010010100101001101110 011101010000101000001000111010011001000110011111101010110000000110100 101111000001011111011001100000111101110111011001011111110000011001100 111110101001100010000011011111100110001110101110101000001010101101101 011100010100010011010110101001010011100111111000101010110001111111111
slide-12
SLIDE 12 Video Encoding – Decoder 000110111100101 101110001001010 010010100001011 011101100111010 0100100 0101110 0001011 0100010 Decoder Bitstream Parser Controller 0001 0001 0001101111001011 0111000100101001 0010100001011011 1011101011000011
slide-13
SLIDE 13 The Year of the Linux Video Codec Drivers Video Encoding

Stateful Codecs

Stateless Codecs Next
slide-14
SLIDE 14 Stateful Decoder Decoder Bitstream Parser Controller HW 000110111100101 101110001001010 010010100001011 011101100111010 0100100 0101110 0001011 0100010 0001 0001 0001101111001011 0111000100101001 0010100001011011 1011101011000011
slide-15
SLIDE 15 API

V4L2

slide-16
SLIDE 16 API
slide-17
SLIDE 17 V4L2 Userspace Application V4L2 Driver /dev/video*
slide-18
SLIDE 18 V4L2 Configure Formats Set Controls Allocate Buffers 260 Standard Controls Many Custom Controls Format, Crop, Compose (H.263, H.264, VC1, VP8, VP9, ...) 3 Memory Models DMA BUF 0001101111001011011 1000100101001001010 0001011011101100010 1101110001001010101 1000110111100101101 1100010010100100101 0000101101110110001 0110111000100101010 1100011011110010110 1110001001010010010 1000010110111011000 1011011100010010101 0001101111001011011 1000100101001001010 0001011011101100010 1101110001001010101 1000110111100101101 1100010010100100101 0000101101110110001 0110111000100101010 1100011011110010110 1110001001010010010 1000010110111011000 1011011100010010101 0001101111001011011 1000100101001001010 0001011011101100010 1101110001001010101 1000110111100101101 1100010010100100101 0000101101110110001 0110111000100101010 1100011011110010110 1110001001010010010 1000010110111011000 1011011100010010101 0001101111001011011 1000100101001001010 0001011011101100010 1101110001001010101 1000110111100101101 1100010010100100101 0000101101110110001 0110111000100101010 1100011011110010110 1110001001010010010 1000010110111011000 1011011100010010101
slide-19
SLIDE 19 V4L2 Start Video Stream Queue / Dequeue Stop Video Stream Blocking & Non-Blocking Event Notifications
slide-20
SLIDE 20 V4L2 – M2M Userspace Application OUTPUT CAPTURE /dev/video* <v4l2-mem2mem.h> 1x 1x
slide-21
SLIDE 21 V4L2 – M2M Multiplexing Userspace Application OUTPUT CAPTURE Userspace Application OUTPUT CAPTURE /dev/video* fd=4 fd=3
slide-22
SLIDE 22 V4L2 – M2M Userspace Application /dev/video0 OUTPUT CAPTURE /dev/video1 OUTPUT CAPTURE
slide-23
SLIDE 23 V4L2 OUTPUT CAPTURE 00011011110 01011011100 01001010010 01010000101 10111011000 10110111000 10010101011 00011011110 01011011100 01001010010 01010000101 10111011000 10110111000 10010101011 00011011110 01011011100 01001010010 01010000101 10111011000 10110111000 10010101011 00011011110 01011011100 01001010010 01010000101 10111011000 10110111000 10010101011
slide-24
SLIDE 24 V4L2 – Header Parsing OUTPUT CAPTURE 00011011110 01011011100 01001010010 01010000101 10111011000 10110111000 10010101011 00011011110 01011011100 01001010010 01010000101 10111011000 10110111000 10010101011 00011011110 01011011100 01001010010 01010000101 10111011000 10110111000 10010101011 00011011110 01011011100 01001010010 01010000101 10111011000 10110111000 10010101011
slide-25
SLIDE 25 The Year of the Linux Video Codec Drivers Video Encoding Stateful Codecs

Stateless Codecs

Next
slide-26
SLIDE 26 Stateless Decoder Decoder Bitstream Parser Controller HW CPU 000110111100101 101110001001010 010010100001011 011101100111010 0100100 0101110 0001011 0100010 0001 0001 0001101111001011 0111000100101001 0010100001011011 1011101011000011
slide-27
SLIDE 27 Stateless Decoder Decoder Bitstream Parser Controller HW 0001 0001 0001101111001011 0111000100101001 0010100001011011 1011101011000011 000110111100101 101110001001010 010010100001011 011101100111010 0100100 0101110 0001011 0100010
slide-28
SLIDE 28 Stateless Decoder Decoder QBUF / DQBUF 000110111100101 101110001001010 010010100001011 011101100111010 0001011 0100010 MPLANE buffers S_FMT S_EXT_CTRLS G_FMT G_EXT_CTRLS 0100100 0101110 QBUF / DQBUF MPLANE buffers 0001011 0100010 Controller
slide-29
SLIDE 29 V4L2 – Request REQUEST 0100100 0101110 0001011 0100010 000110111 100101101 110001001 010010010 100001011 011101100
slide-30
SLIDE 30 V4L2 – Requests S _ E X T _ C T R L S 3 4 Q B U F C M D _ R E Q ( Q U E U E ) 2 1 C M D _ R E Q ( A L L O C ) 000110111 100101101 110001001 010010010 100001011 011101100
slide-31
SLIDE 31 Requests Lifetime Management FD REQ REQ Q FD REQ Q ALLOC QUEUE Complete CLOSE FD REQ CLOSE Complete INIT
slide-32
SLIDE 32 V4L2 – Requests REQUEST 0100100 0101110 0001011 0100010
slide-33
SLIDE 33 V4L2 – Requests S _ E X T _ C T R L S 3 4 Q B U F C M D _ R E Q ( Q U E U E ) 2 1 C M D _ R E Q ( A L L O C ) G _ E X T _ C T R L S 5 6 D Q B U F 000110111 100101101 110001001 010010010 100001011 011101100
slide-34
SLIDE 34 Stateless Decoder Decoder QBUF / DQBUF 000110111100101 101110001001010 010010100001011 011101100111010 0001011 0100010 MPLANE buffers S_FMT S_EXT_CTRLS G_FMT G_EXT_CTRLS 0100100 0101110 QBUF / DQBUF MPLANE buffers 0001011 0100010 Controller V4L2 Application
slide-35
SLIDE 35 Stateless Decoder Decoder QBUF / DQBUF 000110111100101 101110001001010 010010100001011 011101100111010 0001011 0100010 MPLANE buffers S_FMT S_EXT_CTRLS G_FMT G_EXT_CTRLS 0100100 0101110 QBUF / DQBUF MPLANE buffers 0001011 0100010 Controller V4L2 Application

?

slide-36
SLIDE 36 libv4l2 Userspace Application V4L2 Driver /dev/video* libv4l2
slide-37
SLIDE 37 libv4l2 Userspace Application V4L2 Driver /dev/video* libv4l2 LD_PRELOAD=
slide-38
SLIDE 38 libv4l2 libv4l2 plugin
  • pen()
close() read() write() ioctl()
slide-39
SLIDE 39 libv4l2 0001011 0100010 libv4l2 plugin VIDIOC_G_FMT VIDIOC_TRY_FMT VIDIOC_S_FMT
slide-40
SLIDE 40 libv4l2 libv4l2 plugin VIDIOC_REQBUFS VIDIOC_CREATE_BUFS VIDIOC_QUERYBUF VIDIOC_PREPARE_BUF VIDIOC_QBUF VIDIOC_DQBUF 000110111 100101101 110001001 010010010 100001011 011101100 0001011 0100010 000110111 100101101 110001001 010010010 100001011 011101100 1 2 3 2 3 1
slide-41
SLIDE 41 libv4l2 libv4l2 plugin VIDIOC_G_CTRL VIDIOC_S_CTRL VIDIOC_QUERYCTRL VIDIOC_G_EXT_CTRLS VIDIOC_S_EXT_CTRLS VIDIOC_TRY_EXT_CTRLS VIDIOC_QUERY_EXT_CTRL
slide-42
SLIDE 42 License and Documentation libv4l2 plugin Userspace Application V4L2 Driver 0001011 0100010 GPL LGPL any any Doc
slide-43
SLIDE 43 The Year of the Linux Video Codec Drivers Video Encoding Stateful Codecs Stateless Codecs

Next

slide-44
SLIDE 44
slide-45
SLIDE 45

? !

slide-46
SLIDE 46
  • “Video4Linux2: Path to a Standardized Video Codec API”
(https://events.linuxfoundation.org/images/stories/pdf/lceu2012_debski.pdf)
  • V4L2 Codec API
(https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-codec.html)
  • Request API and proof-of-concept implementation
(https://lwn.net/Articles/668726/)
  • Git development branch
(http://git.linuxtv.org/pinchartl/media.git/log/?h=drm/du/vsp1-kms/request) Resources
slide-47
SLIDE 47
  • linux-media@vger.kernel.org
  • laurent.pinchart@ideasonboard.com
Contact
slide-48
SLIDE 48

thx.