vp9 v4l2 stateless uapi
Subsystems (3)
v2 cover vp9 v4l2 stateless uapi
Dear All,
This is a follow-up work for https://patchwork.linuxtv.org/project/linux-media/list/?series=5268
I addressed Hans's comments.
Changes:
v1..v2:
- improve documentation
- imrpove coding style
- factor out of common vp9 code into v4l2-vp9.c
- rename V4L2_CID_STATELESS_VP9_FRAME_DECODE_PARAMS into V4L2_CID_STATELESS_VP9_FRAME
This is still sent as an RFC because the works for adding the second driver (g2@imx8) are ongoing.
The v1 was an RFC on stateless uapi for vp9 decoding with v4l2, which was based on
https://lkml.org/lkml/2020/11/2/1043, but had been substantially reworked. The important change was that the v4l2 control used to pass boolean decoder probabilities had been made unidirectional, and was renamed V4L2_CID_STATELESS_VP9_COMPRESSED_HDR_PROBS.
In the original proposal from Boris, to queue a frame the userspace must fully dequeue the previous one, which effectively results in a forced lockstep behavior and defeats vb2's capability to enqueue multiple buffers. Such a design was a consequence of backward probability updates being performed by the kernel driver (which has direct access to appropriate counter values) but forward probability updates being coupled with compressed header parsing performed by the userspace.
In vp9 the boolean decoder used to decode the bitstream needs certain parameters to work. Those are probabilities, which change with each frame. After each frame is decoded it is known how many times a given symbol occured in the frame, so the probabilities can be adapted. This process is known as backward probabilities update. A next frame header can also contain information which modifies probabilities resulting from backward update. The said modification is called forward probabilities update. The data for backward update is generated by the decoder hardware, while the data for forward update is prepared by reading the compressed frame header. The natural place to parse something is userspace, while the natural place to access hardware-provided counters is the kernel. Such responsibilties assignment was used in the original work.
To overcome the lockstep, we moved forward probability updates to the kernel, while leaving parsing them in userspace. This way the v4l2 control which is used to pass the probs becomes unidirectional (user->kernel) and the userspace can keep parsing and enqueueing succeeding frames.
If a particular driver parses the compressed header and does backward probability updates on its own then V4L2_CID_STATELESS_VP9_COMPRESSED_HDR_PROBS does not need to be used.
This series adds vp9 uapi in proper locations, which means it is a proper, "official" uapi, as opposed to staging uapi which was proposed in the above mentioned lkml thread.
The series adds vp9 support to rkvdec driver.
Rebased onto media_tree, requires this patch:
https://patchwork.linuxtv.org/project/linux-media/patch/[email protected]/
You can test rkvdec implementation with gstreamer, please clone gstreamer and then use these branches for -base and -bad:
https://gitlab.freedesktop.org/dwlsalmeida/gst-plugins-base/-/tree/vp9-upstream-padding
https://gitlab.freedesktop.org/dwlsalmeida/gst-plugins-bad/-/tree/vp9-upstream
Example invocation:
without format conversion:
gst-launch-1.0 filesrc location=Big_Buck_Bunny_1080_10s_1MB.webm ! parsebin ! v4l2slvp9dec ! filesink location=out.yuv
with format conversion to match vpxdec output:
gst-launch-1.0 filesrc location=Big_Buck_Bunny_1080_10s_1MB.webm ! parsebin ! v4l2slvp9dec ! videoconvert ! video/x-raw,format=I420 ! filesink location=out.yuv
I kindly ask for your comments.
Andrzej Pietrasiewicz (2):
media: uapi: Add VP9 stateless decoder controls
media: uapi: Add VP9 v4l2 library
Boris Brezillon (1):
media: rkvdec: Add the VP9 backend
.../userspace-api/media/v4l/biblio.rst | 10 + .../media/v4l/ext-ctrls-codec-stateless.rst | 547 +++++ .../media/v4l/pixfmt-compressed.rst | 15 + .../media/v4l/vidioc-g-ext-ctrls.rst | 8 + .../media/v4l/vidioc-queryctrl.rst | 12 + .../media/videodev2.h.rst.exceptions | 2 + drivers/media/v4l2-core/Kconfig | 4 + drivers/media/v4l2-core/Makefile | 1 + drivers/media/v4l2-core/v4l2-ctrls.c | 229 +++ drivers/media/v4l2-core/v4l2-ioctl.c | 1 + drivers/media/v4l2-core/v4l2-vp9.c | 1831 +++++++++++++++++ drivers/staging/media/rkvdec/Kconfig | 1 + drivers/staging/media/rkvdec/Makefile | 2 +- drivers/staging/media/rkvdec/rkvdec-vp9.c | 1084 ++++++++++ drivers/staging/media/rkvdec/rkvdec.c | 52 +- drivers/staging/media/rkvdec/rkvdec.h | 6 + include/media/v4l2-ctrls.h | 4 + include/media/v4l2-vp9.h | 168 ++ include/uapi/linux/v4l2-controls.h | 425 ++++ include/uapi/linux/videodev2.h | 6 + 20 files changed, 4403 insertions(+), 5 deletions(-) create mode 100644 drivers/media/v4l2-core/v4l2-vp9.c create mode 100644 drivers/staging/media/rkvdec/rkvdec-vp9.c create mode 100644 include/media/v4l2-vp9.h
base-commit: 0b276e470a4d43e1365d3eb53c608a3d208cabd4
prerequisite-patch-id: d148a5f17bbb03e88c3744a166d1dd2c6088ce7d
prerequisite-patch-id: 1817161060ef577cff00a8f7892eb9cbbfa2f327
prerequisite-patch-id: 5d4236886083146c81f4234d5f97acf6fd6dd4e4
Signature · 1 line
2.17.1
RFC v2 1/3 media: uapi: Add VP9 stateless decoder controls
Add the VP9 stateless decoder controls plus the documentation that goes with it.
--- .../userspace-api/media/v4l/biblio.rst | 10 + .../media/v4l/ext-ctrls-codec-stateless.rst | 547 ++++++++++++++++++ .../media/v4l/pixfmt-compressed.rst | 15 + .../media/v4l/vidioc-g-ext-ctrls.rst | 8 + .../media/v4l/vidioc-queryctrl.rst | 12 + .../media/videodev2.h.rst.exceptions | 2 + drivers/media/v4l2-core/v4l2-ctrls.c | 229 ++++++++ drivers/media/v4l2-core/v4l2-ioctl.c | 1 + include/media/v4l2-ctrls.h | 4 + include/uapi/linux/v4l2-controls.h | 425 ++++++++++++++ include/uapi/linux/videodev2.h | 6 + 11 files changed, 1259 insertions(+)
Documentation/userspace-api/media/v4l/biblio.rst +10 -0
@@ -417,3 +417,13 @@ VP8 :title: RFC 6386: "VP8 Data Format and Decoding Guide" :author: J. Bankoski et al. + +.. _vp9: + +VP9 +=== + + +:title: VP9 Bitstream & Decoding Process Specification + +:author: Adrian Grange (Google), Peter de Rivaz (Argon Design), Jonathan Hunt (Argon Design)
Documentation/userspace-api/media/v4l/ext-ctrls-codec-stateless.rst +547 -0
550 lines not on this page · raw message
Documentation/userspace-api/media/v4l/pixfmt-compressed.rst +15 -0
@@ -171,6 +171,21 @@ Compressed Formats - VP9 compressed video frame. The encoder generates one compressed frame per buffer, and the decoder requires one compressed frame per buffer. + * .. _V4L2-PIX-FMT-VP9_FRAME: + + - ``V4L2_PIX_FMT_VP9_FRAME`` + - 'VP9F' + - VP9 parsed frame, including the frame header, as extracted from the container. + This format is adapted for stateless video decoders that implement a + VP9 pipeline with the :ref:`stateless_decoder`. + Metadata associated with the frame to decode is required to be passed + through the ``V4L2_CID_STATELESS_VP9_FRAME`` and + the ``V4L2_CID_STATELESS_VP9_COMPRESSED_HDR_PROBS`` controls. + See the :ref:`associated Codec Control IDs <v4l2-codec-stateless-vp9>`. + Exactly one output and one capture buffer must be provided for use with + this pixel format. The output buffer must contain the appropriate number + of macroblocks to decode a full corresponding frame to the matching + capture buffer. * .. _V4L2-PIX-FMT-HEVC: - ``V4L2_PIX_FMT_HEVC``
Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst +8 -0
@@ -221,6 +221,14 @@ still cause this situation. - ``p_vp8_frame`` - A pointer to a struct :c:type:`v4l2_ctrl_vp8_frame`. Valid if this control is of type ``V4L2_CTRL_TYPE_VP8_FRAME``. + * - struct :c:type:`v4l2_ctrl_vp9_compressed_hdr_probs` * + - ``p_vp9_compressed_hdr_probs`` + - A pointer to a struct :c:type:`v4l2_ctrl_vp9_compressed_hdr_probs'. Valid if this + control is of type ``V4L2_CTRL_TYPE_VP9_COMPRESSED_HDR_PROBS``. + * - struct :c:type:`v4l2_ctrl_vp9_frame` * + - ``p_vp9_frame`` + - A pointer to a struct :c:type:`v4l2_ctrl_vp9_frame'. Valid if this + control is of type ``V4L2_CTRL_TYPE_VP9_FRAME``. * - struct :c:type:`v4l2_ctrl_hdr10_cll_info` * - ``p_hdr10_cll`` - A pointer to a struct :c:type:`v4l2_ctrl_hdr10_cll_info`. Valid if this control is
Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst +12 -0
@@ -495,6 +495,18 @@ See also the examples in :ref:`control`. - n/a - A struct :c:type:`v4l2_ctrl_vp8_frame`, containing VP8 frame parameters for stateless video decoders. + * - ``V4L2_CTRL_TYPE_VP9_COMPRESSED_HDR_PROBS`` + - n/a + - n/a + - n/a + - A struct :c:type:`v4l2_ctrl_vp9_compressed_hdr_probs`, containing VP9 + probabilities updates for stateless video decoders. + * - ``V4L2_CTRL_TYPE_VP9_FRAME`` + - n/a + - n/a + - n/a + - A struct :c:type:`v4l2_ctrl_vp9_frame`, containing VP9 + frame decode parameters for stateless video decoders. .. raw:: latex
Documentation/userspace-api/media/videodev2.h.rst.exceptions +2 -0
@@ -148,6 +148,8 @@ replace symbol V4L2_CTRL_TYPE_HEVC_SLICE_PARAMS :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AREA :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_FWHT_PARAMS :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_VP8_FRAME :c:type:`v4l2_ctrl_type` +replace symbol V4L2_CTRL_TYPE_VP9_COMPRESSED_HDR_PROBS :c:type:`v4l2_ctrl_type` +replace symbol V4L2_CTRL_TYPE_VP9_FRAME :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_HDR10_CLL_INFO :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY :c:type:`v4l2_ctrl_type`
drivers/media/v4l2-core/v4l2-ctrls.c +229 -0
265 lines not on this page · raw message
drivers/media/v4l2-core/v4l2-ioctl.c +1 -0
@@ -1431,6 +1431,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)case V4L2_PIX_FMT_VP8 : descr ="VP8" ;break ;case V4L2_PIX_FMT_VP8_FRAME : descr ="VP8 Frame" ;break ;case V4L2_PIX_FMT_VP9 : descr ="VP9" ;break ; +case V4L2_PIX_FMT_VP9_FRAME : descr ="VP9 Frame" ;break ;case V4L2_PIX_FMT_HEVC : descr ="HEVC" ;break ;/* aka H.265 */ case V4L2_PIX_FMT_HEVC_SLICE : descr ="HEVC Parsed Slice Data" ;break ;case V4L2_PIX_FMT_FWHT : descr ="FWHT" ;break ;/* used in vicodec */
include/media/v4l2-ctrls.h +4 -0
@@ -50,6 +50,8 @@ struct video_device;* @p_h264_decode_params: Pointer to a struct v4l2_ctrl_h264_decode_params. * @p_h264_pred_weights: Pointer to a struct v4l2_ctrl_h264_pred_weights. * @p_vp8_frame: Pointer to a VP8 frame params structure. +* @p_vp9_compressed_hdr_probs: Pointer to a VP9 frame compressed header probs structure. +* @p_vp9_frame: Pointer to a VP9 frame params structure. * @p_hevc_sps: Pointer to an HEVC sequence parameter set structure. * @p_hevc_pps: Pointer to an HEVC picture parameter set structure. * @p_hevc_slice_params: Pointer to an HEVC slice parameters structure.
@@ -79,6 +81,8 @@ union v4l2_ctrl_ptr {struct v4l2_ctrl_hevc_sps *p_hevc_sps;struct v4l2_ctrl_hevc_pps *p_hevc_pps;struct v4l2_ctrl_hevc_slice_params *p_hevc_slice_params; +struct v4l2_ctrl_vp9_compressed_hdr_probs *p_vp9_compressed_hdr_probs; +struct v4l2_ctrl_vp9_frame *p_vp9_frame;struct v4l2_ctrl_hdr10_cll_info *p_hdr10_cll;struct v4l2_ctrl_hdr10_mastering_display *p_hdr10_mastering;struct v4l2_area *p_area;
include/uapi/linux/v4l2-controls.h +425 -0
431 lines not on this page · raw message
include/uapi/linux/videodev2.h +6 -0
@@ -697,6 +697,7 @@ struct v4l2_pix_format {#define V4L2_PIX_FMT_VP8 v4l2_fourcc('V', 'P', '8', '0')/* VP8 */ #define V4L2_PIX_FMT_VP8_FRAME v4l2_fourcc('V', 'P', '8', 'F')/* VP8 parsed frame */ #define V4L2_PIX_FMT_VP9 v4l2_fourcc('V', 'P', '9', '0')/* VP9 */ +#define V4L2_PIX_FMT_VP9_FRAME v4l2_fourcc('V', 'P', '9', 'F')/* VP9 parsed frame */ #define V4L2_PIX_FMT_HEVC v4l2_fourcc('H', 'E', 'V', 'C')/* HEVC aka H.265 */ #define V4L2_PIX_FMT_FWHT v4l2_fourcc('F', 'W', 'H', 'T')/* Fast Walsh Hadamard Transform (vicodec) */ #define V4L2_PIX_FMT_FWHT_STATELESS v4l2_fourcc('S', 'F', 'W', 'H')/* Stateless FWHT (vicodec) */
@@ -1747,6 +1748,8 @@ struct v4l2_ext_control {struct v4l2_ctrl_h264_decode_params __user *p_h264_decode_params;struct v4l2_ctrl_fwht_params __user *p_fwht_params;struct v4l2_ctrl_vp8_frame __user *p_vp8_frame; +struct v4l2_ctrl_vp9_compressed_hdr_probs __user *p_vp9_compressed_hdr_probs; +struct v4l2_ctrl_vp9_frame __user *p_vp9_frame;void __user *ptr; }; } __attribute__ ((packed));
@@ -1807,6 +1810,9 @@ enum v4l2_ctrl_type {V4L2_CTRL_TYPE_FWHT_PARAMS =0x0220 ,V4L2_CTRL_TYPE_VP8_FRAME =0x0240 , + +V4L2_CTRL_TYPE_VP9_COMPRESSED_HDR_PROBS =0x0260 , + V4L2_CTRL_TYPE_VP9_FRAME = 0x0261, };/* Used in the VIDIOC_QUERYCTRL ioctl for querying controls */
Signature · 1 line
2.17.1
RFC v2 2/3 media: uapi: Add VP9 v4l2 library
In reply to Andrzej PietrasiewiczProvide code common to vp9 drivers in one central location.
--- drivers/media/v4l2-core/Kconfig | 4 + drivers/media/v4l2-core/Makefile | 1 + drivers/media/v4l2-core/v4l2-vp9.c | 1831 ++++++++++++++++++++++++++++ include/media/v4l2-vp9.h | 168 +++ 4 files changed, 2004 insertions(+) create mode 100644 drivers/media/v4l2-core/v4l2-vp9.c create mode 100644 include/media/v4l2-vp9.h
drivers/media/v4l2-core/Kconfig +4 -0
@@ -52,6 +52,10 @@ config V4L2_JPEG_HELPER config V4L2_H264 tristate +# Used by drivers that need v4l2-vp9.ko +config V4L2_VP9 + tristate + # Used by drivers that need v4l2-mem2mem.ko config V4L2_MEM2MEM_DEV tristate
drivers/media/v4l2-core/Makefile +1 -0
@@ -22,6 +22,7 @@ obj-$(CONFIG_VIDEO_TUNER) += tuner.o obj-$(CONFIG_V4L2_MEM2MEM_DEV) += v4l2-mem2mem.o obj-$(CONFIG_V4L2_H264) += v4l2-h264.o +obj-$(CONFIG_V4L2_VP9) += v4l2-vp9.o obj-$(CONFIG_V4L2_FLASH_LED_CLASS) += v4l2-flash-led-class.o
drivers/media/v4l2-core/v4l2-vp9.c new file +1831 -0
1,831 lines not on this page · raw message
include/media/v4l2-vp9.h new file +168 -0
168 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC v2 3/3 media: rkvdec: Add the VP9 backend
In reply to Andrzej PietrasiewiczFrom: Boris Brezillon <[email protected]>
The Rockchip VDEC supports VP9 profile 0 up to 4096x2304@30fps. Add a backend for this new format.
--- drivers/staging/media/rkvdec/Kconfig | 1 + drivers/staging/media/rkvdec/Makefile | 2 +- drivers/staging/media/rkvdec/rkvdec-vp9.c | 1084 +++++++++++++++++++++ drivers/staging/media/rkvdec/rkvdec.c | 52 +- drivers/staging/media/rkvdec/rkvdec.h | 6 + 5 files changed, 1140 insertions(+), 5 deletions(-) create mode 100644 drivers/staging/media/rkvdec/rkvdec-vp9.c
drivers/staging/media/rkvdec/Kconfig +1 -0
@@ -9,6 +9,7 @@ config VIDEO_ROCKCHIP_VDEC select VIDEOBUF2_VMALLOC select V4L2_MEM2MEM_DEV select V4L2_H264 + select V4L2_VP9 help Support for the Rockchip Video Decoder IP present on Rockchip SoCs, which accelerates video decoding.
drivers/staging/media/rkvdec/Makefile +1 -1
@@ -1,3 +1,3 @@ obj-$(CONFIG_VIDEO_ROCKCHIP_VDEC) += rockchip-vdec.o -rockchip-vdec-y += rkvdec.o rkvdec-h264.o +rockchip-vdec-y += rkvdec.o rkvdec-h264.o rkvdec-vp9.o
drivers/staging/media/rkvdec/rkvdec-vp9.c new file +1084 -0
1,084 lines not on this page · raw message
drivers/staging/media/rkvdec/rkvdec.c +48 -4
79 lines not on this page · raw message
drivers/staging/media/rkvdec/rkvdec.h +6 -0
@@ -50,6 +50,10 @@ struct rkvdec_vp9_decoded_buffer_info {struct rkvdec_decoded_buffer {/* Must be the first field in this struct. */ struct v4l2_m2m_buffer base; + +union { +struct rkvdec_vp9_decoded_buffer_info vp9; + }; }; static inline struct rkvdec_decoded_buffer *
@@ -116,4 +120,6 @@ void rkvdec_run_preamble(struct rkvdec_ctx *ctx, struct rkvdec_run *run);void rkvdec_run_postamble (struct rkvdec_ctx *ctx,struct rkvdec_run *run);extern const struct rkvdec_coded_fmt_ops rkvdec_h264_fmt_ops; +extern const struct rkvdec_coded_fmt_ops rkvdec_vp9_fmt_ops; +#endif /* RKVDEC_H_ */
Signature · 1 line
2.17.1