Add V4L stateless video decoder API support to NVIDIA Tegra driver

PATCH Superseded Dmitry Osipenko 2 participants 6 messages Jan 16, 2022 – Jan 26, 2022

Subsystems (3)

Media (V4L/DVB) Staging Tegra architecture support

lore

Share & tools

lore

Dmitry Osipenko

author

2022-01-16 18:56 UTC #0 lore raw

v2 cover Add V4L stateless video decoder API support to NVIDIA Tegra driver

Support V4L stateless video decoder API by NVIDIA Tegra decoder driver. Tested using GStreamer [1] and libvdpau-tegra [2].

[1] https://github.com/grate-driver/gstreamer/commit/b8509bdbb69b534e61419ea1798f32f9ad2f3597
[2] https://github.com/grate-driver/libvdpau-tegra/commit/f822e95911e5e0c39f8ba19f843ddc1e0138d5ce

Changelog:

v2: - Made V4L2_BUF_FLAG_*FRAME flags mandatory [3] and dropped reading
of raw bitstream from the driver code, as was suggested by Nicolas Dufresne.

[3] https://github.com/grate-driver/gstreamer/commit/aee292f0f2e84b7654a314dd7e63f916888ffaa5

- Ran v4l2-compliance [4] and fluster [5][6] tests, like was suggested by Nicolas Dufresne. Fixed minor v4l2-compliance errors that were related to a partial initialization of the coded format and were harmless in practice, but made compliance checker unhappy.

[4] https://gist.github.com/digetx/5d6bcdab633488f1dcc7c141ab90d30e
[5] https://gist.github.com/digetx/b06c5d779e9d25afa41d9f46946fe399
[6] https://gist.github.com/digetx/ac4198bc340e5065aa8ec3288bb21356

Dmitry Osipenko (3):
media: staging: tegra-vde: Factor out H.264 code
media: staging: tegra-vde: Support V4L stateless video decoder API
media: docs: dev-stateless-decoder: Document frame type flags

 .../media/v4l/dev-stateless-decoder.rst       |    3 +
 drivers/staging/media/tegra-vde/Kconfig       |    7 +
 drivers/staging/media/tegra-vde/Makefile      |    2 +-
 drivers/staging/media/tegra-vde/h264.c        |  940 +++++++++++++++
 drivers/staging/media/tegra-vde/v4l2.c        | 1018 +++++++++++++++++
 drivers/staging/media/tegra-vde/vde.c         |  700 ++----------
 drivers/staging/media/tegra-vde/vde.h         |  127 ++
 7 files changed, 2181 insertions(+), 616 deletions(-)
 create mode 100644 drivers/staging/media/tegra-vde/h264.c
 create mode 100644 drivers/staging/media/tegra-vde/v4l2.c

Signature · 1 line

2.33.1

Dmitry Osipenko

author

2022-01-16 18:56 UTC #1 lore raw

PATCH v2 1/3 media: staging: tegra-vde: Factor out H.264 code

Factor out H.264 hardware programming code into separate source file in a preparation to support V4L API by the Tegra video decoder driver.

Signed-off-by Dmitry Osipenko <[email protected]>

---
 drivers/staging/media/tegra-vde/Makefile |   2 +-
 drivers/staging/media/tegra-vde/h264.c   | 647 +++++++++++++++++++++++
 drivers/staging/media/tegra-vde/vde.c    | 626 +---------------------
 drivers/staging/media/tegra-vde/vde.h    |  42 ++
 4 files changed, 706 insertions(+), 611 deletions(-)
 create mode 100644 drivers/staging/media/tegra-vde/h264.c

drivers/staging/media/tegra-vde/Makefile +1 -1

index 2827f7601de8..43525b08b3b0 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
-tegra-vde-y := vde.o iommu.o dmabuf-cache.o
+tegra-vde-y := vde.o iommu.o dmabuf-cache.o h264.o
 obj-$(CONFIG_TEGRA_VDE)	+= tegra-vde.o

drivers/staging/media/tegra-vde/h264.c new file +647 -0

647 lines not on this page · raw message

drivers/staging/media/tegra-vde/vde.c +16 ~~-610~~

709 lines not on this page · raw message

drivers/staging/media/tegra-vde/vde.h +42 -0

70 lines not on this page · raw message

Signature · 1 line

2.33.1

Dmitry Osipenko

author

2022-01-16 18:56 UTC #2 lore raw

PATCH v2 2/3 media: staging: tegra-vde: Support V4L stateless video decoder API

In reply to Dmitry Osipenko

Expose Tegra video decoder as a generic V4L M2M stateless video decoder.

Signed-off-by Dmitry Osipenko <[email protected]>

---
 drivers/staging/media/tegra-vde/Kconfig  |    7 +
 drivers/staging/media/tegra-vde/Makefile |    2 +-
 drivers/staging/media/tegra-vde/h264.c   |  297 ++++++-
 drivers/staging/media/tegra-vde/v4l2.c   | 1018 ++++++++++++++++++++++
 drivers/staging/media/tegra-vde/vde.c    |   74 +-
 drivers/staging/media/tegra-vde/vde.h    |   85 ++
 6 files changed, 1475 insertions(+), 8 deletions(-)
 create mode 100644 drivers/staging/media/tegra-vde/v4l2.c

drivers/staging/media/tegra-vde/Kconfig +7 -0

index 0dc78afd09e0..07dbc1f44ca8 100644
@@ -2,9 +2,16 @@
 config TEGRA_VDE
 	tristate "NVIDIA Tegra Video Decoder Engine driver"
 	depends on ARCH_TEGRA || COMPILE_TEST
+	depends on VIDEO_DEV && VIDEO_V4L2
 	select DMA_SHARED_BUFFER
 	select IOMMU_IOVA
+	select MEDIA_CONTROLLER
+	select MEDIA_CONTROLLER_REQUEST_API
 	select SRAM
+	select VIDEOBUF2_DMA_CONTIG
+	select VIDEOBUF2_DMA_SG
+	select V4L2_H264
+	select V4L2_MEM2MEM_DEV
 	help
 	    Say Y here to enable support for the NVIDIA Tegra video decoder
 	    driver.

drivers/staging/media/tegra-vde/Makefile +1 -1

index 43525b08b3b0..1758aa201baa 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
-tegra-vde-y := vde.o iommu.o dmabuf-cache.o h264.o
+tegra-vde-y := vde.o iommu.o dmabuf-cache.o h264.o v4l2.o
 obj-$(CONFIG_TEGRA_VDE)	+= tegra-vde.o

drivers/staging/media/tegra-vde/h264.c +295 -2

316 lines not on this page · raw message

drivers/staging/media/tegra-vde/v4l2.c new file +1018 -0

1,018 lines not on this page · raw message

drivers/staging/media/tegra-vde/vde.c +69 -5

126 lines not on this page · raw message

drivers/staging/media/tegra-vde/vde.h +85 -0

128 lines not on this page · raw message

Signature · 1 line

2.33.1

Dmitry Osipenko

author

2022-01-16 18:56 UTC #3 lore raw

PATCH v2 3/3 media: docs: dev-stateless-decoder: Document frame type flags

In reply to Dmitry Osipenko

Document that frame type flags must be set for the coded buffer. Decoders, like NVIDIA Tegra h264 decoder for example, won't work properly without these flags.

Signed-off-by Dmitry Osipenko <[email protected]>

---
 .../userspace-api/media/v4l/dev-stateless-decoder.rst          | 3 +++
 1 file changed, 3 insertions(+)

Documentation/userspace-api/media/v4l/dev-stateless-decoder.rst +3 -0

index 4a26646eeec5..845f4481d34f 100644
@@ -271,6 +271,9 @@ A typical frame would thus be decoded using the following sequence:
           we are not sure that the current decode request is the last one needed
           to produce a fully decoded frame, then
           ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` must also be set.
+          ``V4L2_BUF_FLAG_KEYFRAME``, ``V4L2_BUF_FLAG_PFRAME`` and
+          ``V4L2_BUF_FLAG_BFRAME`` must be set if relevant to the coded
+          format.
 
       ``request_fd``
           must be set to the file descriptor of the decoding request.

Signature · 1 line

2.33.1

Nicolas Dufresne

2022-01-25 20:37 UTC #4 lore raw

Re: [PATCH v2 3/3] media: docs: dev-stateless-decoder: Document frame type flags

Le dimanche 16 janvier 2022 à 21:55 +0300, Dmitry Osipenko a écrit :

Quoted 14 lines

Document that frame type flags must be set for the coded buffer. Decoders, like NVIDIA Tegra h264 decoder for example, won't work properly without these flags.
Signed-off-by Dmitry Osipenko <[email protected]>
---
 .../userspace-api/media/v4l/dev-stateless-decoder.rst          | 3 +++
 1 file changed, 3 insertions(+)
Documentation/userspace-api/media/v4l/dev-stateless-decoder.rst +0 -0
index 4a26646eeec5..845f4481d34f 100644

Documentation/userspace-api/media/v4l/dev-stateless-decoder.rst A typical frame would thus be decoded using the following sequence: · line 276

Quoted 6 lines

Documentation/userspace-api/media/v4l/dev-stateless-decoder.rst +3 -0

@@ -271,6 +271,9 @@ A typical frame would thus be decoded using the following sequence:
           we are not sure that the current decode request is the last one needed
           to produce a fully decoded frame, then
           ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` must also be set.
+          ``V4L2_BUF_FLAG_KEYFRAME``, ``V4L2_BUF_FLAG_PFRAME`` and
+          ``V4L2_BUF_FLAG_BFRAME`` must be set if relevant to the coded
+          format.

I think I would rather keep this in the CODEC specific APIs (BFrame being an MPEG specific concept). If I knew about this before we made the API final, I would have added the slice_type bitstream parameter in the v4l2_h264_decode_params structure. But as the types are rather limited, I would preserve our "reservered" space and make that part of the V4L2_H264_DECODE_PARAM_FLAGS_{} instead. Do you also need that for HEVC ? (do you have HEVC on that generation of tegra ?)

Quoted 3 lines

Documentation/userspace-api/media/v4l/dev-stateless-decoder.rst +0 -0
       ``request_fd``
           must be set to the file descriptor of the decoding request.

Dmitry Osipenko

author

2022-01-26 12:33 UTC #5 lore raw

Re: [PATCH v2 3/3] media: docs: dev-stateless-decoder: Document frame type flags

25.01.2022 23:37, Nicolas Dufresne пишет:

Quoted 22 lines

Le dimanche 16 janvier 2022 à 21:55 +0300, Dmitry Osipenko a écrit :
Quoted 14 lines
Document that frame type flags must be set for the coded buffer. Decoders, like NVIDIA Tegra h264 decoder for example, won't work properly without these flags.
Signed-off-by Dmitry Osipenko <[email protected]>
---
 .../userspace-api/media/v4l/dev-stateless-decoder.rst          | 3 +++
 1 file changed, 3 insertions(+)
Documentation/userspace-api/media/v4l/dev-stateless-decoder.rst +0 -0
index 4a26646eeec5..845f4481d34f 100644
Documentation/userspace-api/media/v4l/dev-stateless-decoder.rst A typical frame would thus be decoded using the following sequence: · line 276
Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-stateless-decoder.rst +3 -0
@@ -271,6 +271,9 @@ A typical frame would thus be decoded using the following sequence:
           we are not sure that the current decode request is the last one needed
           to produce a fully decoded frame, then
           ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` must also be set.
+          ``V4L2_BUF_FLAG_KEYFRAME``, ``V4L2_BUF_FLAG_PFRAME`` and
+          ``V4L2_BUF_FLAG_BFRAME`` must be set if relevant to the coded
+          format.

Quoted 7 lines

I think I would rather keep this in the CODEC specific APIs (BFrame being an MPEG specific concept). If I knew about this before we made the API final, I would have added the slice_type bitstream parameter in the v4l2_h264_decode_params structure. But as the types are rather limited, I would preserve our "reservered" space and make that part of the V4L2_H264_DECODE_PARAM_FLAGS_{} instead. Do you also need that for HEVC ? (do you have HEVC on that generation of tegra ?)

That generation of Tegra doesn't support HEVC. I'll try to move frame type to v4l2_h264_decode_params, thank you.