Skip to contentSkip to conversation

vp9 v4l2 stateless uapi

RFC New Andrzej Pietrasiewicz 1 participant 4 messages May 5, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover vp9 v4l2 stateless uapi

Dear All,

This is a follow-up work for https://patchwork.linuxtv.org/project/linux-media/list/?series=5268
I addressed Hans's comments.

Changes:

v1..v2:
- improve documentation
- imrpove coding style
- factor out of common vp9 code into v4l2-vp9.c
- rename V4L2_CID_STATELESS_VP9_FRAME_DECODE_PARAMS into V4L2_CID_STATELESS_VP9_FRAME

This is still sent as an RFC because the works for adding the second driver (g2@imx8) are ongoing.

The v1 was an RFC on stateless uapi for vp9 decoding with v4l2, which was based on
https://lkml.org/lkml/2020/11/2/1043, but had been substantially reworked. The important change was that the v4l2 control used to pass boolean decoder probabilities had been made unidirectional, and was renamed V4L2_CID_STATELESS_VP9_COMPRESSED_HDR_PROBS.

In the original proposal from Boris, to queue a frame the userspace must fully dequeue the previous one, which effectively results in a forced lockstep behavior and defeats vb2's capability to enqueue multiple buffers. Such a design was a consequence of backward probability updates being performed by the kernel driver (which has direct access to appropriate counter values) but forward probability updates being coupled with compressed header parsing performed by the userspace.

In vp9 the boolean decoder used to decode the bitstream needs certain parameters to work. Those are probabilities, which change with each frame. After each frame is decoded it is known how many times a given symbol occured in the frame, so the probabilities can be adapted. This process is known as backward probabilities update. A next frame header can also contain information which modifies probabilities resulting from backward update. The said modification is called forward probabilities update. The data for backward update is generated by the decoder hardware, while the data for forward update is prepared by reading the compressed frame header. The natural place to parse something is userspace, while the natural place to access hardware-provided counters is the kernel. Such responsibilties assignment was used in the original work.

To overcome the lockstep, we moved forward probability updates to the kernel, while leaving parsing them in userspace. This way the v4l2 control which is used to pass the probs becomes unidirectional (user->kernel) and the userspace can keep parsing and enqueueing succeeding frames.

If a particular driver parses the compressed header and does backward probability updates on its own then V4L2_CID_STATELESS_VP9_COMPRESSED_HDR_PROBS does not need to be used.

This series adds vp9 uapi in proper locations, which means it is a proper, "official" uapi, as opposed to staging uapi which was proposed in the above mentioned lkml thread.

The series adds vp9 support to rkvdec driver.

Rebased onto media_tree, requires this patch:

https://patchwork.linuxtv.org/project/linux-media/patch/[email protected]/

You can test rkvdec implementation with gstreamer, please clone gstreamer and then use these branches for -base and -bad:

https://gitlab.freedesktop.org/dwlsalmeida/gst-plugins-base/-/tree/vp9-upstream-padding
https://gitlab.freedesktop.org/dwlsalmeida/gst-plugins-bad/-/tree/vp9-upstream

Example invocation:

without format conversion:

gst-launch-1.0 filesrc location=Big_Buck_Bunny_1080_10s_1MB.webm ! parsebin ! v4l2slvp9dec ! filesink location=out.yuv

with format conversion to match vpxdec output:

gst-launch-1.0 filesrc location=Big_Buck_Bunny_1080_10s_1MB.webm ! parsebin ! v4l2slvp9dec ! videoconvert ! video/x-raw,format=I420 ! filesink location=out.yuv

I kindly ask for your comments.

Andrzej Pietrasiewicz (2):
media: uapi: Add VP9 stateless decoder controls
media: uapi: Add VP9 v4l2 library

Boris Brezillon (1):
media: rkvdec: Add the VP9 backend

 .../userspace-api/media/v4l/biblio.rst        |   10 +
 .../media/v4l/ext-ctrls-codec-stateless.rst   |  547 +++++
 .../media/v4l/pixfmt-compressed.rst           |   15 +
 .../media/v4l/vidioc-g-ext-ctrls.rst          |    8 +
 .../media/v4l/vidioc-queryctrl.rst            |   12 +
 .../media/videodev2.h.rst.exceptions          |    2 +
 drivers/media/v4l2-core/Kconfig               |    4 +
 drivers/media/v4l2-core/Makefile              |    1 +
 drivers/media/v4l2-core/v4l2-ctrls.c          |  229 +++
 drivers/media/v4l2-core/v4l2-ioctl.c          |    1 +
 drivers/media/v4l2-core/v4l2-vp9.c            | 1831 +++++++++++++++++
 drivers/staging/media/rkvdec/Kconfig          |    1 +
 drivers/staging/media/rkvdec/Makefile         |    2 +-
 drivers/staging/media/rkvdec/rkvdec-vp9.c     | 1084 ++++++++++
 drivers/staging/media/rkvdec/rkvdec.c         |   52 +-
 drivers/staging/media/rkvdec/rkvdec.h         |    6 +
 include/media/v4l2-ctrls.h                    |    4 +
 include/media/v4l2-vp9.h                      |  168 ++
 include/uapi/linux/v4l2-controls.h            |  425 ++++
 include/uapi/linux/videodev2.h                |    6 +
 20 files changed, 4403 insertions(+), 5 deletions(-)
 create mode 100644 drivers/media/v4l2-core/v4l2-vp9.c
 create mode 100644 drivers/staging/media/rkvdec/rkvdec-vp9.c
 create mode 100644 include/media/v4l2-vp9.h

base-commit: 0b276e470a4d43e1365d3eb53c608a3d208cabd4
prerequisite-patch-id: d148a5f17bbb03e88c3744a166d1dd2c6088ce7d
prerequisite-patch-id: 1817161060ef577cff00a8f7892eb9cbbfa2f327
prerequisite-patch-id: 5d4236886083146c81f4234d5f97acf6fd6dd4e4

Signature · 1 line
2.17.1
#1 lore raw

RFC v2 1/3 media: uapi: Add VP9 stateless decoder controls

Add the VP9 stateless decoder controls plus the documentation that goes with it.

---
 .../userspace-api/media/v4l/biblio.rst        |  10 +
 .../media/v4l/ext-ctrls-codec-stateless.rst   | 547 ++++++++++++++++++
 .../media/v4l/pixfmt-compressed.rst           |  15 +
 .../media/v4l/vidioc-g-ext-ctrls.rst          |   8 +
 .../media/v4l/vidioc-queryctrl.rst            |  12 +
 .../media/videodev2.h.rst.exceptions          |   2 +
 drivers/media/v4l2-core/v4l2-ctrls.c          | 229 ++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   1 +
 include/media/v4l2-ctrls.h                    |   4 +
 include/uapi/linux/v4l2-controls.h            | 425 ++++++++++++++
 include/uapi/linux/videodev2.h                |   6 +
 11 files changed, 1259 insertions(+)
Documentation/userspace-api/media/v4l/biblio.rst +10 -0
index 64d241daf63c..051982896375 100644
@@ -417,3 +417,13 @@ VP8
 :title:     RFC 6386: "VP8 Data Format and Decoding Guide"
 
 :author:    J. Bankoski et al.
+
+.. _vp9:
+
+VP9
+===
+
+
+:title:     VP9 Bitstream & Decoding Process Specification
+
+:author:    Adrian Grange (Google), Peter de Rivaz (Argon Design), Jonathan Hunt (Argon Design)
Documentation/userspace-api/media/v4l/ext-ctrls-codec-stateless.rst +547 -0

550 lines not on this page · raw message

Documentation/userspace-api/media/v4l/pixfmt-compressed.rst +15 -0
index 6dba70da822b..c2aa543c0ed4 100644
@@ -171,6 +171,21 @@ Compressed Formats
       - VP9 compressed video frame. The encoder generates one
 	compressed frame per buffer, and the decoder requires one
 	compressed frame per buffer.
+    * .. _V4L2-PIX-FMT-VP9_FRAME:
+
+      - ``V4L2_PIX_FMT_VP9_FRAME``
+      - 'VP9F'
+      - VP9 parsed frame, including the frame header, as extracted from the container.
+	This format is adapted for stateless video decoders that implement a
+	VP9 pipeline with the :ref:`stateless_decoder`.
+	Metadata associated with the frame to decode is required to be passed
+	through the ``V4L2_CID_STATELESS_VP9_FRAME`` and
+	the ``V4L2_CID_STATELESS_VP9_COMPRESSED_HDR_PROBS`` controls.
+	See the :ref:`associated Codec Control IDs <v4l2-codec-stateless-vp9>`.
+	Exactly one output and one capture buffer must be provided for use with
+	this pixel format. The output buffer must contain the appropriate number
+	of macroblocks to decode a full corresponding frame to the matching
+	capture buffer.
     * .. _V4L2-PIX-FMT-HEVC:
 
       - ``V4L2_PIX_FMT_HEVC``
Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst +8 -0
index 3ba22983d21f..7ee7d4f9e078 100644
@@ -221,6 +221,14 @@ still cause this situation.
       - ``p_vp8_frame``
       - A pointer to a struct :c:type:`v4l2_ctrl_vp8_frame`. Valid if this control is
         of type ``V4L2_CTRL_TYPE_VP8_FRAME``.
+    * - struct :c:type:`v4l2_ctrl_vp9_compressed_hdr_probs` *
+      - ``p_vp9_compressed_hdr_probs``
+      - A pointer to a struct :c:type:`v4l2_ctrl_vp9_compressed_hdr_probs'. Valid if this
+        control is of type ``V4L2_CTRL_TYPE_VP9_COMPRESSED_HDR_PROBS``.
+    * - struct :c:type:`v4l2_ctrl_vp9_frame` *
+      - ``p_vp9_frame``
+      - A pointer to a struct :c:type:`v4l2_ctrl_vp9_frame'. Valid if this
+        control is of type ``V4L2_CTRL_TYPE_VP9_FRAME``.
     * - struct :c:type:`v4l2_ctrl_hdr10_cll_info` *
       - ``p_hdr10_cll``
       - A pointer to a struct :c:type:`v4l2_ctrl_hdr10_cll_info`. Valid if this control is
Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst +12 -0
index 8a285daedc6a..27663225d3c1 100644
@@ -495,6 +495,18 @@ See also the examples in :ref:`control`.
       - n/a
       - A struct :c:type:`v4l2_ctrl_vp8_frame`, containing VP8
 	frame parameters for stateless video decoders.
+    * - ``V4L2_CTRL_TYPE_VP9_COMPRESSED_HDR_PROBS``
+      - n/a
+      - n/a
+      - n/a
+      - A struct :c:type:`v4l2_ctrl_vp9_compressed_hdr_probs`, containing VP9
+	probabilities updates for stateless video decoders.
+    * - ``V4L2_CTRL_TYPE_VP9_FRAME``
+      - n/a
+      - n/a
+      - n/a
+      - A struct :c:type:`v4l2_ctrl_vp9_frame`, containing VP9
+	frame decode parameters for stateless video decoders.
 
 .. raw:: latex
 
Documentation/userspace-api/media/videodev2.h.rst.exceptions +2 -0
index f59940352faa..ba01ffc46cee 100644
@@ -148,6 +148,8 @@ replace symbol V4L2_CTRL_TYPE_HEVC_SLICE_PARAMS :c:type:`v4l2_ctrl_type`
 replace symbol V4L2_CTRL_TYPE_AREA :c:type:`v4l2_ctrl_type`
 replace symbol V4L2_CTRL_TYPE_FWHT_PARAMS :c:type:`v4l2_ctrl_type`
 replace symbol V4L2_CTRL_TYPE_VP8_FRAME :c:type:`v4l2_ctrl_type`
+replace symbol V4L2_CTRL_TYPE_VP9_COMPRESSED_HDR_PROBS :c:type:`v4l2_ctrl_type`
+replace symbol V4L2_CTRL_TYPE_VP9_FRAME :c:type:`v4l2_ctrl_type`
 replace symbol V4L2_CTRL_TYPE_HDR10_CLL_INFO :c:type:`v4l2_ctrl_type`
 replace symbol V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY :c:type:`v4l2_ctrl_type`
 
drivers/media/v4l2-core/v4l2-ctrls.c +229 -0

265 lines not on this page · raw message

drivers/media/v4l2-core/v4l2-ioctl.c +1 -0
index 6a5d1c6d11d6..dbdafa5ea127 100644
@@ -1431,6 +1431,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 		case V4L2_PIX_FMT_VP8:		descr = "VP8"; break;
 		case V4L2_PIX_FMT_VP8_FRAME:    descr = "VP8 Frame"; break;
 		case V4L2_PIX_FMT_VP9:		descr = "VP9"; break;
+		case V4L2_PIX_FMT_VP9_FRAME:    descr = "VP9 Frame"; break;
 		case V4L2_PIX_FMT_HEVC:		descr = "HEVC"; break; /* aka H.265 */
 		case V4L2_PIX_FMT_HEVC_SLICE:	descr = "HEVC Parsed Slice Data"; break;
 		case V4L2_PIX_FMT_FWHT:		descr = "FWHT"; break; /* used in vicodec */
include/media/v4l2-ctrls.h +4 -0
index a5953b812878..12864c795ebc 100644
@@ -50,6 +50,8 @@ struct video_device;
  * @p_h264_decode_params:	Pointer to a struct v4l2_ctrl_h264_decode_params.
  * @p_h264_pred_weights:	Pointer to a struct v4l2_ctrl_h264_pred_weights.
  * @p_vp8_frame:		Pointer to a VP8 frame params structure.
+ * @p_vp9_compressed_hdr_probs:	Pointer to a VP9 frame compressed header probs structure.
+ * @p_vp9_frame:		Pointer to a VP9 frame params structure.
  * @p_hevc_sps:			Pointer to an HEVC sequence parameter set structure.
  * @p_hevc_pps:			Pointer to an HEVC picture parameter set structure.
  * @p_hevc_slice_params:	Pointer to an HEVC slice parameters structure.
@@ -79,6 +81,8 @@ union v4l2_ctrl_ptr {
 	struct v4l2_ctrl_hevc_sps *p_hevc_sps;
 	struct v4l2_ctrl_hevc_pps *p_hevc_pps;
 	struct v4l2_ctrl_hevc_slice_params *p_hevc_slice_params;
+	struct v4l2_ctrl_vp9_compressed_hdr_probs *p_vp9_compressed_hdr_probs;
+	struct v4l2_ctrl_vp9_frame *p_vp9_frame;
 	struct v4l2_ctrl_hdr10_cll_info *p_hdr10_cll;
 	struct v4l2_ctrl_hdr10_mastering_display *p_hdr10_mastering;
 	struct v4l2_area *p_area;
include/uapi/linux/v4l2-controls.h +425 -0

431 lines not on this page · raw message

include/uapi/linux/videodev2.h +6 -0
index 311a01cc5775..cebdae888cf7 100644
@@ -697,6 +697,7 @@ struct v4l2_pix_format {
 #define V4L2_PIX_FMT_VP8      v4l2_fourcc('V', 'P', '8', '0') /* VP8 */
 #define V4L2_PIX_FMT_VP8_FRAME v4l2_fourcc('V', 'P', '8', 'F') /* VP8 parsed frame */
 #define V4L2_PIX_FMT_VP9      v4l2_fourcc('V', 'P', '9', '0') /* VP9 */
+#define V4L2_PIX_FMT_VP9_FRAME v4l2_fourcc('V', 'P', '9', 'F') /* VP9 parsed frame */
 #define V4L2_PIX_FMT_HEVC     v4l2_fourcc('H', 'E', 'V', 'C') /* HEVC aka H.265 */
 #define V4L2_PIX_FMT_FWHT     v4l2_fourcc('F', 'W', 'H', 'T') /* Fast Walsh Hadamard Transform (vicodec) */
 #define V4L2_PIX_FMT_FWHT_STATELESS     v4l2_fourcc('S', 'F', 'W', 'H') /* Stateless FWHT (vicodec) */
@@ -1747,6 +1748,8 @@ struct v4l2_ext_control {
 		struct v4l2_ctrl_h264_decode_params __user *p_h264_decode_params;
 		struct v4l2_ctrl_fwht_params __user *p_fwht_params;
 		struct v4l2_ctrl_vp8_frame __user *p_vp8_frame;
+		struct v4l2_ctrl_vp9_compressed_hdr_probs __user *p_vp9_compressed_hdr_probs;
+		struct v4l2_ctrl_vp9_frame __user *p_vp9_frame;
 		void __user *ptr;
 	};
 } __attribute__ ((packed));
@@ -1807,6 +1810,9 @@ enum v4l2_ctrl_type {
 	V4L2_CTRL_TYPE_FWHT_PARAMS	    = 0x0220,
 
 	V4L2_CTRL_TYPE_VP8_FRAME            = 0x0240,
+
+	V4L2_CTRL_TYPE_VP9_COMPRESSED_HDR_PROBS	= 0x0260,
+	V4L2_CTRL_TYPE_VP9_FRAME		= 0x0261,
 };
 
 /*  Used in the VIDIOC_QUERYCTRL ioctl for querying controls */
Signature · 1 line
2.17.1
#2 lore raw

RFC v2 2/3 media: uapi: Add VP9 v4l2 library

In reply to Andrzej Pietrasiewicz

Provide code common to vp9 drivers in one central location.

---
 drivers/media/v4l2-core/Kconfig    |    4 +
 drivers/media/v4l2-core/Makefile   |    1 +
 drivers/media/v4l2-core/v4l2-vp9.c | 1831 ++++++++++++++++++++++++++++
 include/media/v4l2-vp9.h           |  168 +++
 4 files changed, 2004 insertions(+)
 create mode 100644 drivers/media/v4l2-core/v4l2-vp9.c
 create mode 100644 include/media/v4l2-vp9.h
drivers/media/v4l2-core/Kconfig +4 -0
index bf49f83cb86f..69d1a0bc06cc 100644
@@ -52,6 +52,10 @@ config V4L2_JPEG_HELPER
 config V4L2_H264
 	tristate
 
+# Used by drivers that need v4l2-vp9.ko
+config V4L2_VP9
+	tristate
+
 # Used by drivers that need v4l2-mem2mem.ko
 config V4L2_MEM2MEM_DEV
 	tristate
drivers/media/v4l2-core/Makefile +1 -0
index e4cd589b99a5..438c0adaf144 100644
@@ -22,6 +22,7 @@ obj-$(CONFIG_VIDEO_TUNER) += tuner.o
 
 obj-$(CONFIG_V4L2_MEM2MEM_DEV) += v4l2-mem2mem.o
 obj-$(CONFIG_V4L2_H264) += v4l2-h264.o
+obj-$(CONFIG_V4L2_VP9) += v4l2-vp9.o
 
 obj-$(CONFIG_V4L2_FLASH_LED_CLASS) += v4l2-flash-led-class.o
 
drivers/media/v4l2-core/v4l2-vp9.c new file +1831 -0

1,831 lines not on this page · raw message

include/media/v4l2-vp9.h new file +168 -0

168 lines not on this page · raw message

Signature · 1 line
2.17.1
#3 lore raw

RFC v2 3/3 media: rkvdec: Add the VP9 backend

In reply to Andrzej Pietrasiewicz

From: Boris Brezillon <[email protected]>

The Rockchip VDEC supports VP9 profile 0 up to 4096x2304@30fps. Add a backend for this new format.

---
 drivers/staging/media/rkvdec/Kconfig      |    1 +
 drivers/staging/media/rkvdec/Makefile     |    2 +-
 drivers/staging/media/rkvdec/rkvdec-vp9.c | 1084 +++++++++++++++++++++
 drivers/staging/media/rkvdec/rkvdec.c     |   52 +-
 drivers/staging/media/rkvdec/rkvdec.h     |    6 +
 5 files changed, 1140 insertions(+), 5 deletions(-)
 create mode 100644 drivers/staging/media/rkvdec/rkvdec-vp9.c
drivers/staging/media/rkvdec/Kconfig +1 -0
index c02199b5e0fd..dc7292f346fa 100644
@@ -9,6 +9,7 @@ config VIDEO_ROCKCHIP_VDEC
 	select VIDEOBUF2_VMALLOC
 	select V4L2_MEM2MEM_DEV
 	select V4L2_H264
+	select V4L2_VP9
 	help
 	  Support for the Rockchip Video Decoder IP present on Rockchip SoCs,
 	  which accelerates video decoding.
drivers/staging/media/rkvdec/Makefile +1 -1
index c08fed0a39f9..cb86b429cfaa 100644
@@ -1,3 +1,3 @@
 obj-$(CONFIG_VIDEO_ROCKCHIP_VDEC) += rockchip-vdec.o
 
-rockchip-vdec-y += rkvdec.o rkvdec-h264.o
+rockchip-vdec-y += rkvdec.o rkvdec-h264.o rkvdec-vp9.o
drivers/staging/media/rkvdec/rkvdec-vp9.c new file +1084 -0

1,084 lines not on this page · raw message

drivers/staging/media/rkvdec/rkvdec.c +48 -4

79 lines not on this page · raw message

drivers/staging/media/rkvdec/rkvdec.h +6 -0
index 52ac3874c5e5..8f534743eaf1 100644
@@ -50,6 +50,10 @@ struct rkvdec_vp9_decoded_buffer_info {
 struct rkvdec_decoded_buffer {
 	/* Must be the first field in this struct. */
 	struct v4l2_m2m_buffer base;
+
+	union {
+		struct rkvdec_vp9_decoded_buffer_info vp9;
+	};
 };
 
 static inline struct rkvdec_decoded_buffer *
@@ -116,4 +120,6 @@ void rkvdec_run_preamble(struct rkvdec_ctx *ctx, struct rkvdec_run *run);
 void rkvdec_run_postamble(struct rkvdec_ctx *ctx, struct rkvdec_run *run);
 
 extern const struct rkvdec_coded_fmt_ops rkvdec_h264_fmt_ops;
+extern const struct rkvdec_coded_fmt_ops rkvdec_vp9_fmt_ops;
+
 #endif /* RKVDEC_H_ */
Signature · 1 line
2.17.1