Skip to contentSkip to conversation

media: rkvdec: Add a VP9 backend

PATCH In review Ezequiel Garcia 4 participants 14 messages May 18, 2020 – Mar 2, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v4 cover media: rkvdec: Add a VP9 backend

Fourth iteration of the VP9 stateless codec uAPI, plus support for Rockchip VDEC, just minor changes since v3.

Thanks!
Ezequiel

Changelog
---------

v4:

* Drop color_space field from the VP9 interface.
V4L2 API should be used for it.
* Clarified Segment-ID comments.
* Moved motion vector probabilities to a separate
struct.

v3:

* Fix documentation issues found by Hans.
* Fix smatch detected issues as pointed out by Hans.
* Added patch to fix wrong bytesused set on .buf_prepare.

v2:

* Documentation style issues pointed out by Nicolas internally.
* s/VP9_PROFILE_MAX/V4L2_VP9_PROFILE_MAX/
* Fix wrong kfree(ctx).
* constify a couple structs on rkvdec-vp9.c

*** BLURB HERE ***

Boris Brezillon (2):
media: uapi: Add VP9 stateless decoder controls
media: rkvdec: Add the VP9 backend

Ezequiel Garcia (1):
media: rkvdec: Fix .buf_prepare

 .../userspace-api/media/v4l/biblio.rst        |   10 +
 .../media/v4l/ext-ctrls-codec.rst             |  550 ++++++
 drivers/media/v4l2-core/v4l2-ctrls.c          |  239 +++
 drivers/media/v4l2-core/v4l2-ioctl.c          |    1 +
 drivers/staging/media/rkvdec/Makefile         |    2 +-
 drivers/staging/media/rkvdec/rkvdec-vp9.c     | 1577 +++++++++++++++++
 drivers/staging/media/rkvdec/rkvdec.c         |   66 +-
 drivers/staging/media/rkvdec/rkvdec.h         |    6 +
 include/media/v4l2-ctrls.h                    |    1 +
 include/media/vp9-ctrls.h                     |  485 +++++
 10 files changed, 2932 insertions(+), 5 deletions(-)
 create mode 100644 drivers/staging/media/rkvdec/rkvdec-vp9.c
 create mode 100644 include/media/vp9-ctrls.h
Signature · 1 line
2.26.0.rc2
#1 lore raw

PATCH v4 1/3 media: rkvdec: Fix .buf_prepare

The driver should only set the payload on .buf_prepare if the buffer is CAPTURE type, or if an OUTPUT buffer has a zeroed payload.

Fix it.

Fixes cd33c830448ba ("media: rkvdec: Add the rkvdec driver")
---
 drivers/staging/media/rkvdec/rkvdec.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/staging/media/rkvdec/rkvdec.c +9 -1
index 225eeca73356..4df2a248ab96 100644
@@ -456,7 +456,15 @@ static int rkvdec_buf_prepare(struct vb2_buffer *vb)
 		if (vb2_plane_size(vb, i) < sizeimage)
 			return -EINVAL;
 	}
-	vb2_set_plane_payload(vb, 0, f->fmt.pix_mp.plane_fmt[0].sizeimage);
+
+	/*
+	 * Buffer's bytesused is written by the driver for CAPTURE buffers,
+	 * or if the application passed zero bytesused on an OUTPUT buffer.
+	 */
+	if (!V4L2_TYPE_IS_OUTPUT(vq->type) ||
+	    (V4L2_TYPE_IS_OUTPUT(vq->type) && !vb2_get_plane_payload(vb, 0)))
+		vb2_set_plane_payload(vb, 0,
+				      f->fmt.pix_mp.plane_fmt[0].sizeimage);
 	return 0;
 }
 
Signature · 1 line
2.26.0.rc2
#2 lore raw

Re: [PATCH v4 1/3] media: rkvdec: Fix .buf_prepare

Quoted 20 lines · Ezequiel Garcia

The driver should only set the payload on .buf_prepare if the buffer is CAPTURE type, or if an OUTPUT buffer has a zeroed payload.

Fix it.

Fixes cd33c830448ba ("media: rkvdec: Add the rkvdec driver")
---
 drivers/staging/media/rkvdec/rkvdec.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/staging/media/rkvdec/rkvdec.c +0 -0
index 225eeca73356..4df2a248ab96 100644
@@ -456,7 +456,15 @@ static int rkvdec_buf_prepare(struct vb2_buffer *vb)
 		if (vb2_plane_size(vb, i) < sizeimage)
 			return -EINVAL;
 	}

drivers/staging/media/rkvdec/rkvdec.c static int rkvdec_buf_prepare(struct vb2_buffer *vb) · line 467

Quoted 10 lines
drivers/staging/media/rkvdec/rkvdec.c +9 -1
-	vb2_set_plane_payload(vb, 0, f->fmt.pix_mp.plane_fmt[0].sizeimage);
+
+	/*
+	 * Buffer's bytesused is written by the driver for CAPTURE buffers,
+	 * or if the application passed zero bytesused on an OUTPUT buffer.
+	 */
+	if (!V4L2_TYPE_IS_OUTPUT(vq->type) ||
+	    (V4L2_TYPE_IS_OUTPUT(vq->type) && !vb2_get_plane_payload(vb, 0)))
+		vb2_set_plane_payload(vb, 0,
+				      f->fmt.pix_mp.plane_fmt[0].sizeimage);

This should just be:

	if (!V4L2_TYPE_IS_OUTPUT(vq->type))
		vb2_set_plane_payload(vb, 0, f->fmt.pix_mp.plane_fmt[0].sizeimage);

If the application passes 0 as bytesused, then 1) a warning will be generated by the v4l2 core and 2) the v4l2 core will set bytesused to the length of the buffer. See vb2_fill_vb2_v4l2_buffer() in videobuf2-v4l2.c.

Some old drivers explicitly allow bytesused to be 0 for an output queue to signal end-of-stream, but that's only supported if the allow_zero_bytesused flag is set in the vb2_queue, and that shall not be used for new drivers since it is deprecated functionality.

Regards,

Hans

Quoted 4 lines
drivers/staging/media/rkvdec/rkvdec.c +0 -0
 	return 0;
 }
 
#3 lore raw

Re: [PATCH v4 1/3] media: rkvdec: Fix .buf_prepare

Quoted 37 lines · Hans Verkuil
Quoted 20 lines · Ezequiel Garcia

The driver should only set the payload on .buf_prepare if the buffer is CAPTURE type, or if an OUTPUT buffer has a zeroed payload.

Fix it.

Fixes cd33c830448ba ("media: rkvdec: Add the rkvdec driver")
---
 drivers/staging/media/rkvdec/rkvdec.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/staging/media/rkvdec/rkvdec.c +0 -0
index 225eeca73356..4df2a248ab96 100644
@@ -456,7 +456,15 @@ static int rkvdec_buf_prepare(struct vb2_buffer *vb)
 		if (vb2_plane_size(vb, i) < sizeimage)
 			return -EINVAL;
 	}

drivers/staging/media/rkvdec/rkvdec.c static int rkvdec_buf_prepare(struct vb2_buffer *vb) · line 467

Quoted 10 lines
drivers/staging/media/rkvdec/rkvdec.c +9 -1
-	vb2_set_plane_payload(vb, 0, f->fmt.pix_mp.plane_fmt[0].sizeimage);
+
+	/*
+	 * Buffer's bytesused is written by the driver for CAPTURE buffers,
+	 * or if the application passed zero bytesused on an OUTPUT buffer.
+	 */
+	if (!V4L2_TYPE_IS_OUTPUT(vq->type) ||
+	    (V4L2_TYPE_IS_OUTPUT(vq->type) && !vb2_get_plane_payload(vb, 0)))
+		vb2_set_plane_payload(vb, 0,
+				      f->fmt.pix_mp.plane_fmt[0].sizeimage);

This should just be:

	if (!V4L2_TYPE_IS_OUTPUT(vq->type))
		vb2_set_plane_payload(vb, 0, f->fmt.pix_mp.plane_fmt[0].sizeimage);
Quoted 9 lines

If the application passes 0 as bytesused, then 1) a warning will be generated by the v4l2 core and 2) the v4l2 core will set bytesused to the length of the buffer. See vb2_fill_vb2_v4l2_buffer() in videobuf2-v4l2.c.

Some old drivers explicitly allow bytesused to be 0 for an output queue to signal end-of-stream, but that's only supported if the allow_zero_bytesused flag is set in the vb2_queue, and that shall not be used for new drivers since it is deprecated functionality.

Ah, good catch. I'll get you a v5.

Thanks,
Ezequiel

#4 lore raw

PATCH v4 2/3 media: uapi: Add VP9 stateless decoder controls

In reply to Ezequiel Garcia

From: Boris Brezillon <[email protected]>

Add the VP9 stateless decoder controls plus the documentation that goes with it.

---
 .../userspace-api/media/v4l/biblio.rst        |  10 +
 .../media/v4l/ext-ctrls-codec.rst             | 550 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ctrls.c          | 239 ++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   1 +
 include/media/v4l2-ctrls.h                    |   1 +
 include/media/vp9-ctrls.h                     | 485 +++++++++++++++
 6 files changed, 1286 insertions(+)
 create mode 100644 include/media/vp9-ctrls.h
Documentation/userspace-api/media/v4l/biblio.rst +10 -0
index 3c9634173e82..e09102e572fd 100644
@@ -414,3 +414,13 @@ VP8
 :title:     RFC 6386: "VP8 Data Format and Decoding Guide"
 
 :author:    J. Bankoski et al.
+
+.. _vp9:
+
+VP9
+===
+
+
+:title:     VP9 Bitstream & Decoding Process Specification
+
+:author:    Adrian Grange (Google), Peter de Rivaz (Argon Design), Jonathan Hunt (Argon Design)
Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +550 -0

556 lines not on this page · raw message

drivers/media/v4l2-core/v4l2-ctrls.c +239 -0

269 lines not on this page · raw message

drivers/media/v4l2-core/v4l2-ioctl.c +1 -0
index 2322f08a98be..4fe77ad917c8 100644
@@ -1423,6 +1423,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 		case V4L2_PIX_FMT_VP8:		descr = "VP8"; break;
 		case V4L2_PIX_FMT_VP8_FRAME:    descr = "VP8 Frame"; break;
 		case V4L2_PIX_FMT_VP9:		descr = "VP9"; break;
+		case V4L2_PIX_FMT_VP9_FRAME:    descr = "VP9 Frame"; break;
 		case V4L2_PIX_FMT_HEVC:		descr = "HEVC"; break; /* aka H.265 */
 		case V4L2_PIX_FMT_HEVC_SLICE:	descr = "HEVC Parsed Slice Data"; break;
 		case V4L2_PIX_FMT_FWHT:		descr = "FWHT"; break; /* used in vicodec */
include/media/v4l2-ctrls.h +1 -0
index 757a713bad41..2de8290f9f11 100644
@@ -21,6 +21,7 @@
 #include <media/fwht-ctrls.h>
 #include <media/h264-ctrls.h>
 #include <media/vp8-ctrls.h>
+#include <media/vp9-ctrls.h>
 #include <media/hevc-ctrls.h>
 
 /* forward references */
include/media/vp9-ctrls.h new file +485 -0

485 lines not on this page · raw message

Signature · 1 line
2.26.0.rc2
#5 lore raw

Re: [PATCH v4 2/3] media: uapi: Add VP9 stateless decoder controls

Quoted 17 lines · Ezequiel Garcia

From: Boris Brezillon <[email protected]>

Add the VP9 stateless decoder controls plus the documentation that goes with it.

---
 .../userspace-api/media/v4l/biblio.rst        |  10 +
 .../media/v4l/ext-ctrls-codec.rst             | 550 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ctrls.c          | 239 ++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   1 +
 include/media/v4l2-ctrls.h                    |   1 +
 include/media/vp9-ctrls.h                     | 485 +++++++++++++++
 6 files changed, 1286 insertions(+)
 create mode 100644 include/media/vp9-ctrls.h

<snip>

Quoted 8 lines
include/media/vp9-ctrls.h new file +8 -0
+/**
+ * struct v4l2_vp9_quantization - VP9 quantization parameters
+ *
+ * @base_q_idx: indicates the base frame qindex
+ * @delta_q_y_dc: indicates the Y DC quantizer relative to base_q_idx
+ * @delta_q_uv_dc: indicates the UV DC quantizer relative to base_q_idx
+ * @delta_q_uv_ac indicates the UV AC quantizer relative to base_q_idx
+ * @padding: padding bytes to align things on 64 bits. Must be set to 0

include/media/vp9-ctrls.h line 85

Quoted 10 lines
include/media/vp9-ctrls.h new file +10 -0
+ *
+ * Encodes the quantization parameters. See section '7.2.9 Quantization params
+ * syntax' of the VP9 specification for more details.
+ */
+struct v4l2_vp9_quantization {
+	__u8 base_q_idx;
+	__s8 delta_q_y_dc;
+	__s8 delta_q_uv_dc;
+	__s8 delta_q_uv_ac;
+	__u8 padding[4];

Are you sure this padding field is needed? What goes wrong if this is dropped?

Quoted 1 line

+};

<snip>

Quoted 4 lines
include/media/vp9-ctrls.h new file +4 -0
+struct v4l2_ctrl_vp9_frame_decode_params {
+	__u32 flags;
+	__u16 compressed_header_size;
+	__u16 uncompressed_header_size;

include/media/vp9-ctrls.h line 453

Quoted 10 lines
include/media/vp9-ctrls.h new file +10 -0
+	__u8 profile;
+	__u8 reset_frame_context;
+	__u8 frame_context_idx;
+	__u8 bit_depth;
+	__u8 interpolation_filter;
+	__u8 tile_cols_log2;
+	__u8 tile_rows_log2;
+	__u8 tx_mode;
+	__u8 reference_mode;
+	__u8 padding[6];

This doesn't look right: this should be 7 if you want to align at 64 bits. Don't forget to update the documentation when you change this. In fact, the documentation doesn't mention the size of the array, it just says 'u8 padding'.

I thought pahole flags something like this?

include/media/vp9-ctrls.h line 459

Quoted 6 lines
include/media/vp9-ctrls.h new file +6 -0
+	__u16 frame_width_minus_1;
+	__u16 frame_height_minus_1;
+	__u16 render_width_minus_1;
+	__u16 render_height_minus_1;
+	__u64 refs[V4L2_REF_ID_CNT];
+	struct v4l2_vp9_loop_filter lf;

sizeof(lf) is an odd-number of bytes, so...

include/media/vp9-ctrls.h line 460

Quoted 1 line
include/media/vp9-ctrls.h new file +1 -0
+	struct v4l2_vp9_quantization quant;

... even though sizeof(quant) == 8 with the padding bytes, that would still not align at 64 bits.

Quoted 16 lines
include/media/vp9-ctrls.h new file +15 -0
+	struct v4l2_vp9_segmentation seg;
+	struct v4l2_vp9_probabilities probs;
+};
+
+#define V4L2_VP9_NUM_FRAME_CTX	4
+
+/**
+ * struct v4l2_ctrl_vp9_frame_ctx - VP9 frame context control
+ *
+ * @probs: VP9 probabilities
+ *
+ * This control is accessed in both direction. The user should initialize the
+ * 4 contexts with default values just after starting the stream. Then before
+ * decoding a frame it should query the current frame context (the one passed
+ * through &v4l2_ctrl_vp9_frame_decode_params.frame_context_idx) to initialize

include/media/vp9-ctrls.h line 485

Quoted 10 lines
include/media/vp9-ctrls.h new file +10 -0
+ * &v4l2_ctrl_vp9_frame_decode_params.probs. The probs are then adjusted based
+ * on the bitstream info and passed to the kernel. The codec should update
+ * the frame context after the frame has been decoded, so that next time
+ * userspace query this context it contains the updated probabilities.
+ */
+struct v4l2_ctrl_vp9_frame_ctx {
+	struct v4l2_vp9_probabilities probs;
+};
+
+#endif /* _VP9_CTRLS_H_ */

Regards,

Hans

#6 lore raw

Re: [PATCH v4 2/3] media: uapi: Add VP9 stateless decoder controls

In reply to Ezequiel Garcia

Hi Ezequiel, sorry for the late review!

Quoted 55 lines · Ezequiel Garcia

From: Boris Brezillon <[email protected]>

Add the VP9 stateless decoder controls plus the documentation that goes with it.

---
 .../userspace-api/media/v4l/biblio.rst        |  10 +
 .../media/v4l/ext-ctrls-codec.rst             | 550 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ctrls.c          | 239 ++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   1 +
 include/media/v4l2-ctrls.h                    |   1 +
 include/media/vp9-ctrls.h                     | 485 +++++++++++++++
 6 files changed, 1286 insertions(+)
 create mode 100644 include/media/vp9-ctrls.h
Documentation/userspace-api/media/v4l/biblio.rst +10 -0
index 3c9634173e82..e09102e572fd 100644
@@ -414,3 +414,13 @@ VP8
 :title:     RFC 6386: "VP8 Data Format and Decoding Guide"
 
 :author:    J. Bankoski et al.
+
+.. _vp9:
+
+VP9
+===
+
+
+:title:     VP9 Bitstream & Decoding Process Specification
+
+:author:    Adrian Grange (Google), Peter de Rivaz (Argon Design), Jonathan Hunt (Argon Design)
Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +11 -0
index d0d506a444b1..5c5f7dd868da 100644
@@ -2668,6 +2668,556 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
       - ``padding[3]``
       - Applications and drivers must set this to zero.
 
+.. _v4l2-mpeg-vp9:
+
+``V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(0..3) (struct)``
+    Stores VP9 probabilities attached to a specific frame context. The VP9
+    specification allows using a maximum of 4 contexts. Each frame being
+    decoded refers to one of those context. See section '7.1.2 Refresh
+    probs semantics' section of :ref:`vp9` for more details about these
+    contexts.
+
+    This control is bi-directional:
+

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 2691

Quoted 10 lines
Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +10 -0
+    * all 4 contexts must be initialized by userspace just after the
+      stream is started and before the first decoding request is submitted.
+    * the referenced context might be read by the kernel when a decoding
+      request is submitted, and will be updated after the decoder is done
+      decoding the frame if the `V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX` flag
+      is set.
+    * contexts will be read back by user space before each decoding request
+      to retrieve the updated probabilities.
+    * userspace will re-initialize the context to their default values when
+      a reset context is required.

Just to make sure I understand this part correctly, it means that if frame A and B use the same context, and frame A has V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX set, then user-space must wait for frame A to get dequeued and read back this control from the completed request before it can submit frame B?

Quoted 896 lines
Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +529 -0
+
+    .. note::
+
+       This compound control is not yet part of the public kernel API and
+       it is expected to change.
+
+.. c:type:: v4l2_ctrl_vp9_frame_ctx
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{5.8cm}|p{4.8cm}|p{6.6cm}|
+
+.. flat-table:: struct v4l2_ctrl_vp9_frame_ctx
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - struct :c:type:`v4l2_vp9_probabilities`
+      - ``probs``
+      - Structure with VP9 probabilities attached to the context.
+
+.. c:type:: v4l2_vp9_probabilities
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_probabilities
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``tx8[2][1]``
+      - TX 8x8 probabilities.
+    * - __u8
+      - ``tx16[2][2]``
+      - TX 16x16 probabilities.
+    * - __u8
+      - ``tx32[2][3]``
+      - TX 32x32 probabilities.
+    * - __u8
+      - ``coef[4][2][2][6][6][3]``
+      - Coefficient probabilities.
+    * - __u8
+      - ``skip[3]``
+      - Skip probabilities.
+    * - __u8
+      - ``inter_mode[7][3]``
+      - Inter prediction mode probabilities.
+    * - __u8
+      - ``interp_filter[4][2]``
+      - Interpolation filter probabilities.
+    * - __u8
+      - ``is_inter[4]``
+      - Is inter-block probabilities.
+    * - __u8
+      - ``comp_mode[5]``
+      - Compound prediction mode probabilities.
+    * - __u8
+      - ``single_ref[5][2]``
+      - Single reference probabilities.
+    * - __u8
+      - ``comp_mode[5]``
+      - Compound reference probabilities.
+    * - __u8
+      - ``y_mode[4][9]``
+      - Y prediction mode probabilities.
+    * - __u8
+      - ``uv_mode[10][9]``
+      - UV prediction mode probabilities.
+    * - __u8
+      - ``partition[16][3]``
+      - Partition probabilities.
+    * - __u8
+      - ``mv.joint[3]``
+      - Motion vector joint probabilities.
+    * - __u8
+      - ``mv.sign[2]``
+      - Motion vector sign probabilities.
+    * - __u8
+      - ``mv.class[2][10]``
+      - Motion vector class probabilities.
+    * - __u8
+      - ``mv.class0_bit[2]``
+      - Motion vector class0 bit probabilities.
+    * - __u8
+      - ``mv.bits[2][10]``
+      - Motion vector bits probabilities.
+    * - __u8
+      - ``mv.class0_fr[2][2][3]``
+      - Motion vector class0 fractional bit probabilities.
+    * - __u8
+      - ``mv.fr[2][3]``
+      - Motion vector fractional bit probabilities.
+    * - __u8
+      - ``mv.class0_hp[2]``
+      - Motion vector class0 high precision fractional bit probabilities.
+    * - __u8
+      - ``mv.hp[2]``
+      - Motion vector high precision fractional bit probabilities.
+
+``V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS (struct)``
+    Specifies the frame parameters for the associated VP9 frame decode request.
+    This includes the necessary parameters for configuring a stateless hardware
+    decoding pipeline for VP9. The bitstream parameters are defined according
+    to :ref:`vp9`.
+
+    .. note::
+
+       This compound control is not yet part of the public kernel API and
+       it is expected to change.
+
+.. c:type:: v4l2_ctrl_vp9_frame_decode_params
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_ctrl_vp9_frame_decode_params
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``flags``
+      - Combination of V4L2_VP9_FRAME_FLAG_* flags. See
+        :c:type:`v4l2_vp9_frame_flags`.
+    * - __u16
+      - ``compressed_header_size``
+      - Compressed header size in bytes.
+    * - __u16
+      - ``uncompressed_header_size``
+      - Uncompressed header size in bytes.
+    * - __u8
+      - ``profile``
+      - VP9 profile. Can be 0, 1, 2 or 3.
+    * - __u8
+      - ``reset_frame_context``
+      - Frame context that should be used/updated when decoding the frame.
+    * - __u8
+      - ``bit_depth``
+      - Component depth in bits. Must be 8 for profile 0 and 1. Must 10 or 12
+        for profile 2 and 3.
+    * - __u8
+      - ``interpolation_filter``
+      - Specifies the filter selection used for performing inter prediction. See
+        :c:type:`v4l2_vp9_interpolation_filter`.
+    * - __u8
+      - ``tile_cols_log2``
+      - Specifies the base 2 logarithm of the width of each tile (where the
+        width is measured in units of 8x8 blocks). Shall be less than or equal
+        to 6.
+    * - __u8
+      - ``tile_rows_log2``
+      - Specifies the base 2 logarithm of the height of each tile (where the
+        height is measured in units of 8x8 blocks)
+    * - __u8
+      - ``tx_mode``
+      - Specifies the TX mode. See :c:type:`v4l2_vp9_tx_mode`.
+    * - __u8
+      - ``reference_mode``
+      - Specifies the type of inter prediction to be used. See
+        :c:type:`v4l2_vp9_reference_mode`.
+    * - __u8
+      - ``padding``
+      - Needed to make this struct 64 bit aligned. Shall be filled with zeroes.
+    * - __u16
+      - ``frame_width_minus_1``
+      - Add 1 to get the frame width expressed in pixels.
+    * - __u16
+      - ``frame_height_minus_1``
+      - Add 1 to to get the frame height expressed in pixels.
+    * - __u16
+      - ``frame_width_minus_1``
+      - Add 1 to to get the expected render width expressed in pixels. This is
+        not used during the decoding process but might be used by HW scalers to
+        prepare a frame that's ready for scanout.
+    * - __u16
+      - frame_height_minus_1
+      - Add 1 to get the expected render height expressed in pixels. This is
+        not used during the decoding process but might be used by HW scalers to
+        prepare a frame that's ready for scanout.
+    * - __u64
+      - ``refs[3]``
+      - Array of reference frame timestamps.
+    * - struct :c:type:`v4l2_vp9_loop_filter`
+      - ``lf``
+      - Loop filter parameters. See struct :c:type:`v4l2_vp9_loop_filter`.
+    * - struct :c:type:`v4l2_vp9_quantization`
+      - ``quant``
+      - Quantization parameters. See :c:type:`v4l2_vp9_quantization`.
+    * - struct :c:type:`v4l2_vp9_segmentation`
+      - ``seg``
+      - Segmentation parameters. See :c:type:`v4l2_vp9_segmentation`.
+    * - struct :c:type:`v4l2_vp9_probabilities`
+      - ``probs``
+      - Probabilities. See :c:type:`v4l2_vp9_probabilities`.
+
+.. c:type:: v4l2_vp9_frame_flags
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_frame_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_FRAME_FLAG_KEY_FRAME``
+      - The frame is a key frame.
+    * - ``V4L2_VP9_FRAME_FLAG_SHOW_FRAME``
+      - The frame should be displayed.
+    * - ``V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT``
+      - The decoding should be error resilient.
+    * - ``V4L2_VP9_FRAME_FLAG_INTRA_ONLY``
+      - The frame does not reference other frames.
+    * - ``V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV``
+      - the frame might can high precision motion vectors.
+    * - ``V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX``
+      - Frame context should be updated after decoding.
+    * - ``V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE``
+      - Parallel decoding is used.
+    * - ``V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING``
+      - Vertical subsampling is enabled.
+    * - ``V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING``
+      - Horizontal subsampling is enabled.
+    * - ``V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING``
+      - The full UV range is used.
+
+.. c:type:: v4l2_vp9_ref_id
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_ref_id
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_REF_ID_LAST``
+      - Last reference frame.
+    * - ``V4L2_REF_ID_GOLDEN``
+      - Golden reference frame.
+    * - ``V4L2_REF_ID_ALTREF``
+      - Alternative reference frame.
+    * - ``V4L2_REF_ID_CNT``
+      - Number of reference frames.
+
+.. c:type:: v4l2_vp9_tx_mode
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_tx_mode
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_TX_MODE_ONLY_4X4``
+      - Transform size is 4x4.
+    * - ``V4L2_VP9_TX_MODE_ALLOW_8X8``
+      - Transform size can be up to 8x8.
+    * - ``V4L2_VP9_TX_MODE_ALLOW_16X16``
+      - Transform size can be up to 16x16.
+    * - ``V4L2_VP9_TX_MODE_ALLOW_32X32``
+      - transform size can be up to 32x32.
+    * - ``V4L2_VP9_TX_MODE_SELECT``
+      - Bitstream contains transform size for each block.
+
+.. c:type:: v4l2_vp9_reference_mode
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_reference_mode
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_REF_MODE_SINGLE``
+      - Indicates that all the inter blocks use only a single reference frame
+        to generate motion compensated prediction.
+    * - ``V4L2_VP9_REF_MODE_COMPOUND``
+      - Requires all the inter blocks to use compound mode. Single reference
+        frame prediction is not allowed.
+    * - ``V4L2_VP9_REF_MODE_SELECT``
+      - Allows each individual inter block to select between single and
+        compound prediction modes.
+
+.. c:type:: v4l2_vp9_interpolation_filter
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_interpolation_filter
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_INTERP_FILTER_8TAP``
+      - Height tap filter.
+    * - ``V4L2_VP9_INTERP_FILTER_8TAP_SMOOTH``
+      - Height tap smooth filter.
+    * - ``V4L2_VP9_INTERP_FILTER_8TAP_SHARP``
+      - Height tap sharp filter.
+    * - ``V4L2_VP9_INTERP_FILTER_BILINEAR``
+      - Bilinear filter.
+    * - ``V4L2_VP9_INTERP_FILTER_SWITCHABLE``
+      - Filter selection is signaled at the block level.
+
+.. c:type:: v4l2_vp9_reset_frame_context
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_reset_frame_context
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_RESET_FRAME_CTX_NONE``
+      - Do not reset any frame context.
+    * - ``V4L2_VP9_RESET_FRAME_CTX_SPEC``
+      - Reset the frame context pointed by
+        :c:type:`v4l2_ctrl_vp9_frame_decode_params`.frame_context_idx.
+    * - ``V4L2_VP9_RESET_FRAME_CTX_ALL``
+      - Reset all frame contexts.
+
+.. c:type:: v4l2_vp9_intra_prediction_mode
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_intra_prediction_mode
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_INTRA_PRED_DC``
+      - DC intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_V``
+      - Vertical intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_H``
+      - Horizontal intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D45``
+      - D45 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D135``
+      - D135 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D117``
+      - D117 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D153``
+      - D153 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D207``
+      - D207 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D63``
+      - D63 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_TM``
+      - True motion intra prediction.
+
+.. c:type:: v4l2_vp9_segmentation
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_segmentation
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``flags``
+      - Combination of V4L2_VP9_SEGMENTATION_FLAG_* flags. See
+        :c:type:`v4l2_vp9_segmentation_flags`.
+    * - __u8
+      - ``tree_probs[7]``
+      - Specifies the probability values to be used when decoding a Segment-ID.
+        See '5.15. Segmentation map' section of :ref:`vp9` for more details.
+    * - __u8
+      - ``pred_prob[3]``
+      - Specifies the probability values to be used when decoding a
+        Predicted-Segment-ID. See '6.4.14. Get segment id syntax'
+        section of :ref:`vp9` for more details.
+    * - __u8
+      - ``padding[5]``
+      - Used to align this struct on 64 bit. Shall be filled with zeroes.
+    * - __u8
+      - ``feature_enabled[8]``
+      - Bitmask defining which features are enabled in each segment.
+    * - __u8
+      - ``feature_data[8][4]``
+      - Data attached to each feature. Data entry is only valid if the feature
+        is enabled.
+
+.. c:type:: v4l2_vp9_segment_feature
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_segment_feature
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_SEGMENT_FEATURE_QP_DELTA``
+      - QP delta segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_LF``
+      - Loop filter segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_REF_FRAME``
+      - Reference frame segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_SKIP``
+      - Skip segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_CNT``
+      - Number of segment features.
+
+.. c:type:: v4l2_vp9_segmentation_flags
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_segmentation_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_ENABLED``
+      - Indicates that this frame makes use of the segmentation tool.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP``
+      - Indicates that the segmentation map should be updated during the
+        decoding of this frame.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE``
+      - Indicates that the updates to the segmentation map are coded
+        relative to the existing segmentation map.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA``
+      - Indicates that new parameters are about to be specified for each
+        segment.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE``
+      - Indicates that the segmentation parameters represent the actual values
+        to be used.
+
+.. c:type:: v4l2_vp9_quantization
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_quantization
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``base_q_idx``
+      - Indicates the base frame qindex.
+    * - __s8
+      - ``delta_q_y_dc``
+      - Indicates the Y DC quantizer relative to base_q_idx.
+    * - __s8
+      - ``delta_q_uv_dc``
+      - Indicates the UV DC quantizer relative to base_q_idx.
+    * - __s8
+      - ``delta_q_uv_ac``
+      - Indicates the UV AC quantizer relative to base_q_idx.
+    * - __u8
+      - ``padding[4]``
+      - Padding bytes used to align this struct on 64 bit. Must be set to 0.
+
+.. c:type:: v4l2_vp9_loop_filter
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_loop_filter
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``flags``
+      - Combination of V4L2_VP9_LOOP_FILTER_FLAG_* flags.
+        See :c:type:`v4l2_vp9_loop_filter_flags`.
+    * - __u8
+      - ``level``
+      - Indicates the loop filter strength.
+    * - __u8
+      - ``sharpness``
+      - Indicates the sharpness level.
+    * - __s8
+      - ``ref_deltas[4]``
+      - Contains the adjustment needed for the filter level based on the chosen
+        reference frame.
+    * - __s8
+      - ``mode_deltas[2]``
+      - Contains the adjustment needed for the filter level based on the chosen
+        mode
+    * - __u8
+      - ``level_lookup[8][4][2]``
+      - Level lookup table.
+
+
+.. c:type:: v4l2_vp9_loop_filter_flags
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_loop_filter_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED``
+      - When set, the filter level depends on the mode and reference frame used
+        to predict a block.
+    * - ``V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE``
+      - When set, the bitstream contains additional syntax elements that
+        specify which mode and reference frame deltas are to be updated.
+
 .. raw:: latex
 
     \normalsize
drivers/media/v4l2-core/v4l2-ctrls.c +239 -0
index 1c617b42a944..115beadaf02d 100644
@@ -930,6 +930,11 @@ const char *v4l2_ctrl_get_name(u32 id)
        case V4L2_CID_MPEG_VIDEO_VP8_PROFILE:                   return "VP8 Profile";
        case V4L2_CID_MPEG_VIDEO_VP9_PROFILE:                   return "VP9 Profile";
        case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:              return "VP8 Frame Header";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS:       return "VP9 Frame Decode Parameters";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(0):          return "VP9 Frame Context 0";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(1):          return "VP9 Frame Context 1";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(2):          return "VP9 Frame Context 2";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(3):          return "VP9 Frame Context 3";
 
        /* HEVC controls */
        case V4L2_CID_MPEG_VIDEO_HEVC_I_FRAME_QP:               return "HEVC I-Frame QP Value";
@@ -1403,6 +1408,15 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
        case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
                *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
                break;
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS:
+               *type = V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS;
+               break;
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(0):
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(1):
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(2):
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(3):
+               *type = V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT;
+               break;
        case V4L2_CID_MPEG_VIDEO_HEVC_SPS:
                *type = V4L2_CTRL_TYPE_HEVC_SPS;
                break;
@@ -1703,6 +1717,219 @@ static void std_log(const struct v4l2_ctrl *ctrl)
        0;                                                      \
 })
 
+static int
+validate_vp9_lf_params(struct v4l2_vp9_loop_filter *lf)
+{
+       unsigned int i, j, k;
+
+       if (lf->flags &
+           ~(V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED |
+             V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED implies
+        * V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE.
+        */
+       if (lf->flags & V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE &&
+           !(lf->flags & V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED))
+               return -EINVAL;
+
+       /* That all values are in the accepted range. */
+       if (lf->level > GENMASK(5, 0))
+               return -EINVAL;
+
+       if (lf->sharpness > GENMASK(2, 0))
+               return -EINVAL;
+
+       for (i = 0; i < ARRAY_SIZE(lf->ref_deltas); i++) {
+               if (lf->ref_deltas[i] < -63 || lf->ref_deltas[i] > 63)
+                       return -EINVAL;
+       }
+
+       for (i = 0; i < ARRAY_SIZE(lf->mode_deltas); i++) {
+               if (lf->mode_deltas[i] < -63 || lf->mode_deltas[i] > 63)
+                       return -EINVAL;
+       }
+
+       for (i = 0; i < ARRAY_SIZE(lf->level_lookup); i++) {
+               for (j = 0; j < ARRAY_SIZE(lf->level_lookup[0]); j++) {
+                       for (k = 0; k < ARRAY_SIZE(lf->level_lookup[0][0]); k++) {
+                               if (lf->level_lookup[i][j][k] > 63)
+                                       return -EINVAL;
+                       }
+               }
+       }
+
+       return 0;
+}
+
+static int
+validate_vp9_quant_params(struct v4l2_vp9_quantization *quant)
+{
+       if (quant->delta_q_y_dc < -15 || quant->delta_q_y_dc > 15 ||
+           quant->delta_q_uv_dc < -15 || quant->delta_q_uv_dc > 15 ||
+           quant->delta_q_uv_ac < -15 || quant->delta_q_uv_ac > 15)
+               return -EINVAL;
+
+       memset(quant->padding, 0, sizeof(quant->padding));
+       return 0;
+}
+
+static int
+validate_vp9_seg_params(struct v4l2_vp9_segmentation *seg)
+{
+       unsigned int i, j;
+
+       if (seg->flags &
+           ~(V4L2_VP9_SEGMENTATION_FLAG_ENABLED |
+             V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP |
+             V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE |
+             V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA |
+             V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP and
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA imply
+        * V4L2_VP9_SEGMENTATION_FLAG_ENABLED.
+        */
+       if ((seg->flags &
+            (V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP |
+             V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA)) &&
+           !(seg->flags & V4L2_VP9_SEGMENTATION_FLAG_ENABLED))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE implies
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP.
+        */
+       if (seg->flags & V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE &&
+           !(seg->flags & V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE implies
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA.
+        */
+       if (seg->flags & V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE &&
+           !(seg->flags & V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA))
+               return -EINVAL;
+
+       for (i = 0; i < ARRAY_SIZE(seg->feature_enabled); i++) {
+               if (seg->feature_enabled[i] &
+                   ~(V4L2_VP9_SEGMENT_FEATURE_QP_DELTA |
+                     V4L2_VP9_SEGMENT_FEATURE_LF |
+                     V4L2_VP9_SEGMENT_FEATURE_REF_FRAME |
+                     V4L2_VP9_SEGMENT_FEATURE_SKIP))
+                       return -EINVAL;
+       }
+
+       for (i = 0; i < ARRAY_SIZE(seg->feature_data); i++) {
+               const int range[] = {255, 63, 3, 0};
+
+               for (j = 0; j < ARRAY_SIZE(seg->feature_data[j]); j++) {
+                       if (seg->feature_data[i][j] < -range[j] ||
+                           seg->feature_data[i][j] > range[j])
+                               return -EINVAL;
+               }
+       }
+
+       memset(seg->padding, 0, sizeof(seg->padding));
+       return 0;
+}
+
+static int
+validate_vp9_frame_decode_params(struct v4l2_ctrl_vp9_frame_decode_params *dec_params)
+{
+       int ret;
+
+       /* Make sure we're not passed invalid flags. */
+       if (dec_params->flags &
+           ~(V4L2_VP9_FRAME_FLAG_KEY_FRAME |
+             V4L2_VP9_FRAME_FLAG_SHOW_FRAME |
+             V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT |
+             V4L2_VP9_FRAME_FLAG_INTRA_ONLY |
+             V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV |
+             V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX |
+             V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE |
+             V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING |
+             V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING |
+             V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING))
+               return -EINVAL;
+
+       /*
+        * The refresh context and error resilient flags are mutually exclusive.
+        * Same goes for parallel decoding and error resilient modes.
+        */
+       if (dec_params->flags & V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT &&
+           dec_params->flags &
+           (V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX |
+            V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE))
+               return -EINVAL;
+
+       if (dec_params->profile > V4L2_VP9_PROFILE_MAX)
+               return -EINVAL;
+
+       if (dec_params->reset_frame_context > V4L2_VP9_RESET_FRAME_CTX_ALL)
+               return -EINVAL;
+
+       if (dec_params->frame_context_idx >= V4L2_VP9_NUM_FRAME_CTX)
+               return -EINVAL;
+
+       /*
+        * Profiles 0 and 1 only support 8-bit depth, profiles 2 and 3 only 10
+        * and 12 bit depths.
+        */
+       if ((dec_params->profile < 2 && dec_params->bit_depth != 8) ||
+           (dec_params->profile >= 2 &&
+            (dec_params->bit_depth != 10 && dec_params->bit_depth != 12)))
+               return -EINVAL;
+
+       /* Profile 0 and 2 only accept YUV 4:2:0. */
+       if ((dec_params->profile == 0 || dec_params->profile == 2) &&
+           (!(dec_params->flags & V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING) ||
+            !(dec_params->flags & V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING)))
+               return -EINVAL;
+
+       /* Profile 1 and 3 only accept YUV 4:2:2, 4:4:0 and 4:4:4. */
+       if ((dec_params->profile == 1 || dec_params->profile == 3) &&
+           ((dec_params->flags & V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING) &&
+            (dec_params->flags & V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING)))
+               return -EINVAL;
+
+       if (dec_params->interpolation_filter > V4L2_VP9_INTERP_FILTER_SWITCHABLE)
+               return -EINVAL;
+
+       /*
+        * According to the spec, tile_cols_log2 shall be less than or equal
+        * to 6.
+        */
+       if (dec_params->tile_cols_log2 > 6)
+               return -EINVAL;
+
+       if (dec_params->tx_mode > V4L2_VP9_TX_MODE_SELECT)
+               return -EINVAL;
+
+       if (dec_params->reference_mode > V4L2_VP9_REF_MODE_SELECT)
+               return -EINVAL;
+
+       ret = validate_vp9_lf_params(&dec_params->lf);
+       if (ret)
+               return ret;
+
+       ret = validate_vp9_quant_params(&dec_params->quant);
+       if (ret)
+               return ret;
+
+       ret = validate_vp9_seg_params(&dec_params->seg);
+       if (ret)
+               return ret;
+
+       memset(dec_params->padding, 0, sizeof(dec_params->padding));
+       return 0;
+}
+
 /* Validate a new control */
 
 #define zero_padding(s) \
@@ -1799,6 +2026,12 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx,
                zero_padding(p_vp8_frame_header->coder_state);
                break;
 
+       case V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS:
+               return validate_vp9_frame_decode_params(p);
+
+       case V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT:
+               break;
+
        case V4L2_CTRL_TYPE_HEVC_SPS:
                p_hevc_sps = p;
 
@@ -2542,6 +2775,12 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl,
        case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
                elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header);
                break;
+       case V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT:
+               elem_size = sizeof(struct v4l2_ctrl_vp9_frame_ctx);
+               break;
+       case V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS:
+               elem_size = sizeof(struct v4l2_ctrl_vp9_frame_decode_params);
+               break;
        case V4L2_CTRL_TYPE_HEVC_SPS:
                elem_size = sizeof(struct v4l2_ctrl_hevc_sps);
                break;
drivers/media/v4l2-core/v4l2-ioctl.c +1 -0
index 2322f08a98be..4fe77ad917c8 100644
@@ -1423,6 +1423,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
                case V4L2_PIX_FMT_VP8:          descr = "VP8"; break;
                case V4L2_PIX_FMT_VP8_FRAME:    descr = "VP8 Frame"; break;
                case V4L2_PIX_FMT_VP9:          descr = "VP9"; break;
+               case V4L2_PIX_FMT_VP9_FRAME:    descr = "VP9 Frame"; break;
                case V4L2_PIX_FMT_HEVC:         descr = "HEVC"; break; /* aka H.265 */
                case V4L2_PIX_FMT_HEVC_SLICE:   descr = "HEVC Parsed Slice Data"; break;
                case V4L2_PIX_FMT_FWHT:         descr = "FWHT"; break; /* used in vicodec */
include/media/v4l2-ctrls.h +1 -0
index 757a713bad41..2de8290f9f11 100644
@@ -21,6 +21,7 @@
 #include <media/fwht-ctrls.h>
 #include <media/h264-ctrls.h>
 #include <media/vp8-ctrls.h>
+#include <media/vp9-ctrls.h>
 #include <media/hevc-ctrls.h>
 
 /* forward references */
include/media/vp9-ctrls.h new file +56 -0
new file mode 100644
index 000000000000..0cdea8a18b72
@@ -0,0 +1,485 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * These are the VP9 state controls for use with stateless VP9
+ * codec drivers.
+ *
+ * It turns out that these structs are not stable yet and will undergo
+ * more changes. So keep them private until they are stable and ready to
+ * become part of the official public API.
+ */
+
+#ifndef _VP9_CTRLS_H_
+#define _VP9_CTRLS_H_
+
+#include <linux/types.h>
+
+#define V4L2_PIX_FMT_VP9_FRAME v4l2_fourcc('V', 'P', '9', 'F')
+
+#define V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(i)       (V4L2_CID_MPEG_BASE + 4000 + (i))
+#define V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS    (V4L2_CID_MPEG_BASE + 4004)
+#define V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT               0x400
+#define V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS         0x404
+
+/**
+ * enum v4l2_vp9_loop_filter_flags - VP9 loop filter flags
+ *
+ * @V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED: the filter level depends on
+ *                                          the mode and reference frame used
+ *                                          to predict a block
+ * @V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE: the bitstream contains additional
+ *                                         syntax elements that specify which
+ *                                         mode and reference frame deltas
+ *                                         are to be updated
+ *
+ * Those are the flags you should pass to &v4l2_vp9_loop_filter.flags. See
+ * section '7.2.8 Loop filter semantics' of the VP9 specification for more
+ * details.
+ */
+enum v4l2_vp9_loop_filter_flags {
+       V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED = 1 << 0,
+       V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE = 1 << 1,
+};
+
+/**
+ * struct v4l2_vp9_loop_filter - VP9 loop filter parameters
+ *
+ * @flags: combination of V4L2_VP9_LOOP_FILTER_FLAG_* flags
+ * @level: indicates the loop filter strength
+ * @sharpness: indicates the sharpness level
+ * @ref_deltas: contains the adjustment needed for the filter level based on
+ *             the chosen reference frame
+ * @mode_deltas: contains the adjustment needed for the filter level based on
+ *              the chosen mode
+ * @level_lookup: level lookup table
+ *
+ * This structure contains all loop filter related parameters. See sections
+ * '7.2.8 Loop filter semantics' and '8.8.1 Loop filter frame init process'

include/media/vp9-ctrls.h line 66

Quoted 10 lines
include/media/vp9-ctrls.h new file +10 -0
+ * of the VP9 specification for more details.
+ */
+struct v4l2_vp9_loop_filter {
+       __u8 flags;
+       __u8 level;
+       __u8 sharpness;
+       __s8 ref_deltas[4];
+       __s8 mode_deltas[2];
+       __u8 level_lookup[8][4][2];
+};

This struct is 73 bytes, doesn't it need padding?

Quoted 377 lines
include/media/vp9-ctrls.h new file +377 -0
+
+/**
+ * struct v4l2_vp9_quantization - VP9 quantization parameters
+ *
+ * @base_q_idx: indicates the base frame qindex
+ * @delta_q_y_dc: indicates the Y DC quantizer relative to base_q_idx
+ * @delta_q_uv_dc: indicates the UV DC quantizer relative to base_q_idx
+ * @delta_q_uv_ac indicates the UV AC quantizer relative to base_q_idx
+ * @padding: padding bytes to align things on 64 bits. Must be set to 0
+ *
+ * Encodes the quantization parameters. See section '7.2.9 Quantization params
+ * syntax' of the VP9 specification for more details.
+ */
+struct v4l2_vp9_quantization {
+       __u8 base_q_idx;
+       __s8 delta_q_y_dc;
+       __s8 delta_q_uv_dc;
+       __s8 delta_q_uv_ac;
+       __u8 padding[4];
+};
+
+/**
+ * enum v4l2_vp9_segmentation_flags - VP9 segmentation flags
+ *
+ * @V4L2_VP9_SEGMENTATION_FLAG_ENABLED: indicates that this frame makes use of
+ *                                     the segmentation tool
+ * @V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP: indicates that the segmentation map
+ *                                        should be updated during the
+ *                                        decoding of this frame
+ * @V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE: indicates that the updates to
+ *                                             the segmentation map are coded
+ *                                             relative to the existing
+ *                                             segmentation map
+ * @V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA: indicates that new parameters are
+ *                                         about to be specified for each
+ *                                         segment
+ * @V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE: indicates that the
+ *                                                 segmentation parameters
+ *                                                 represent the actual values
+ *                                                 to be used
+ *
+ * Those are the flags you should pass to &v4l2_vp9_segmentation.flags. See
+ * section '7.2.10 Segmentation params syntax' of the VP9 specification for
+ * more details.
+ */
+enum v4l2_vp9_segmentation_flags {
+       V4L2_VP9_SEGMENTATION_FLAG_ENABLED = 1 << 0,
+       V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP = 1 << 1,
+       V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE = 1 << 2,
+       V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA = 1 << 3,
+       V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE = 1 << 4,
+};
+
+#define V4L2_VP9_SEGMENT_FEATURE_ENABLED(id)   (1 << (id))
+#define V4L2_VP9_SEGMENT_FEATURE_ENABLED_MASK  0xf
+
+/**
+ * enum v4l2_vp9_segment_feature - VP9 segment feature IDs
+ *
+ * @V4L2_VP9_SEGMENT_FEATURE_QP_DELTA: QP delta segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_LF: loop filter segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_REF_FRAME: reference frame segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_SKIP: skip segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_CNT: number of segment features
+ *
+ * Segment feature IDs. See section '7.2.10 Segmentation params syntax' of the
+ * VP9 specification for more details.
+ */
+enum v4l2_vp9_segment_feature {
+       V4L2_VP9_SEGMENT_FEATURE_QP_DELTA,
+       V4L2_VP9_SEGMENT_FEATURE_LF,
+       V4L2_VP9_SEGMENT_FEATURE_REF_FRAME,
+       V4L2_VP9_SEGMENT_FEATURE_SKIP,
+       V4L2_VP9_SEGMENT_FEATURE_CNT,
+};
+
+/**
+ * struct v4l2_vp9_segmentation - VP9 segmentation parameters
+ *
+ * @flags: combination of V4L2_VP9_SEGMENTATION_FLAG_* flags
+ * @tree_probs: specifies the probability values to be used when
+ *              decoding a Segment-ID. See '5.15. Segmentation map'
+ *              section of the VP9 specification for more details.
+ * @pred_prob: specifies the probability values to be used when decoding a
+ *            Predicted-Segment-ID. See '6.4.14. Get segment id syntax'
+ *            section of :ref:`vp9` for more details..
+ * @padding: padding used to make things aligned on 64 bits. Shall be zero
+ *          filled
+ * @feature_enabled: bitmask defining which features are enabled in each
+ *                  segment
+ * @feature_data: data attached to each feature. Data entry is only valid if
+ *               the feature is enabled
+ *
+ * Encodes the quantization parameters. See section '7.2.10 Segmentation
+ * params syntax' of the VP9 specification for more details.
+ */
+struct v4l2_vp9_segmentation {
+       __u8 flags;
+       __u8 tree_probs[7];
+       __u8 pred_probs[3];
+       __u8 padding[5];
+       __u8 feature_enabled[8];
+       __s16 feature_data[8][4];
+};
+
+/**
+ * enum v4l2_vp9_intra_prediction_mode - VP9 Intra prediction modes
+ *
+ * @V4L2_VP9_INTRA_PRED_DC: DC intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_V: vertical intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_H: horizontal intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D45: D45 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D135: D135 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D117: D117 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D153: D153 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D207: D207 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D63: D63 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_TM: True Motion intra prediction
+ *
+ * See section '7.4.5 Intra frame mode info semantics' for more details.
+ */
+enum v4l2_vp9_intra_prediction_mode {
+       V4L2_VP9_INTRA_PRED_MODE_DC,
+       V4L2_VP9_INTRA_PRED_MODE_V,
+       V4L2_VP9_INTRA_PRED_MODE_H,
+       V4L2_VP9_INTRA_PRED_MODE_D45,
+       V4L2_VP9_INTRA_PRED_MODE_D135,
+       V4L2_VP9_INTRA_PRED_MODE_D117,
+       V4L2_VP9_INTRA_PRED_MODE_D153,
+       V4L2_VP9_INTRA_PRED_MODE_D207,
+       V4L2_VP9_INTRA_PRED_MODE_D63,
+       V4L2_VP9_INTRA_PRED_MODE_TM,
+};
+
+/**
+ * struct v4l2_vp9_mv_probabilities - VP9 Motion vector probabilities
+ * @joint: motion vector joint probabilities
+ * @sign: motion vector sign probabilities
+ * @class: motion vector class probabilities
+ * @class0_bit: motion vector class0 bit probabilities
+ * @bits: motion vector bits probabilities
+ * @class0_fr: motion vector class0 fractional bit probabilities
+ * @fr: motion vector fractional bit probabilities
+ * @class0_hp: motion vector class0 high precision fractional bit probabilities
+ * @hp: motion vector high precision fractional bit probabilities
+ */
+struct v4l2_vp9_mv_probabilities {
+       __u8 joint[3];
+       __u8 sign[2];
+       __u8 class[2][10];
+       __u8 class0_bit[2];
+       __u8 bits[2][10];
+       __u8 class0_fr[2][2][3];
+       __u8 fr[2][3];
+       __u8 class0_hp[2];
+       __u8 hp[2];
+};
+
+/**
+ * struct v4l2_vp9_probabilities - VP9 Probabilities
+ *
+ * @tx8: TX 8x8 probabilities
+ * @tx16: TX 16x16 probabilities
+ * @tx32: TX 32x32 probabilities
+ * @coef: coefficient probabilities
+ * @skip: skip probabilities
+ * @inter_mode: inter mode probabilities
+ * @interp_filter: interpolation filter probabilities
+ * @is_inter: is inter-block probabilities
+ * @comp_mode: compound prediction mode probabilities
+ * @single_ref: single ref probabilities
+ * @comp_ref: compound ref probabilities
+ * @y_mode: Y prediction mode probabilities
+ * @uv_mode: UV prediction mode probabilities
+ * @partition: partition probabilities
+ * @mv: motion vector probabilities
+ *
+ * Structure containing most VP9 probabilities. See the VP9 specification
+ * for more details.
+ */
+struct v4l2_vp9_probabilities {
+       __u8 tx8[2][1];
+       __u8 tx16[2][2];
+       __u8 tx32[2][3];
+       __u8 coef[4][2][2][6][6][3];
+       __u8 skip[3];
+       __u8 inter_mode[7][3];
+       __u8 interp_filter[4][2];
+       __u8 is_inter[4];
+       __u8 comp_mode[5];
+       __u8 single_ref[5][2];
+       __u8 comp_ref[5];
+       __u8 y_mode[4][9];
+       __u8 uv_mode[10][9];
+       __u8 partition[16][3];
+
+       struct v4l2_vp9_mv_probabilities mv;
+};
+
+/**
+ * enum v4l2_vp9_reset_frame_context - Valid values for
+ *                     &v4l2_ctrl_vp9_frame_decode_params->reset_frame_context
+ *
+ * @V4L2_VP9_RESET_FRAME_CTX_NONE: don't reset any frame context
+ * @V4L2_VP9_RESET_FRAME_CTX_SPEC: reset the frame context pointed by
+ *                     &v4l2_ctrl_vp9_frame_decode_params.frame_context_idx
+ * @V4L2_VP9_RESET_FRAME_CTX_ALL: reset all frame contexts
+ *
+ * See section '7.2 Uncompressed header semantics' of the VP9 specification
+ * for more details.
+ */
+enum v4l2_vp9_reset_frame_context {
+       V4L2_VP9_RESET_FRAME_CTX_NONE,
+       V4L2_VP9_RESET_FRAME_CTX_SPEC,
+       V4L2_VP9_RESET_FRAME_CTX_ALL,
+};
+
+/**
+ * enum v4l2_vp9_interpolation_filter - VP9 interpolation filter types
+ *
+ * @V4L2_VP9_INTERP_FILTER_8TAP: height tap filter
+ * @V4L2_VP9_INTERP_FILTER_8TAP_SMOOTH: height tap smooth filter
+ * @V4L2_VP9_INTERP_FILTER_8TAP_SHARP: height tap sharp filter
+ * @V4L2_VP9_INTERP_FILTER_BILINEAR: bilinear filter
+ * @V4L2_VP9_INTERP_FILTER_SWITCHABLE: filter selection is signaled at the
+ *                                    block level
+ *
+ * See section '7.2.7 Interpolation filter semantics' of the VP9 specification
+ * for more details.
+ */
+enum v4l2_vp9_interpolation_filter {
+       V4L2_VP9_INTERP_FILTER_8TAP,
+       V4L2_VP9_INTERP_FILTER_8TAP_SMOOTH,
+       V4L2_VP9_INTERP_FILTER_8TAP_SHARP,
+       V4L2_VP9_INTERP_FILTER_BILINEAR,
+       V4L2_VP9_INTERP_FILTER_SWITCHABLE,
+};
+
+/**
+ * enum v4l2_vp9_reference_mode - VP9 reference modes
+ *
+ * @V4L2_VP9_REF_MODE_SINGLE: indicates that all the inter blocks use only a
+ *                           single reference frame to generate motion
+ *                           compensated prediction
+ * @V4L2_VP9_REF_MODE_COMPOUND: requires all the inter blocks to use compound
+ *                             mode. Single reference frame prediction is not
+ *                             allowed
+ * @V4L2_VP9_REF_MODE_SELECT: allows each individual inter block to select
+ *                           between single and compound prediction modes
+ *
+ * See section '7.3.6 Frame reference mode semantics' of the VP9 specification
+ * for more details.
+ */
+enum v4l2_vp9_reference_mode {
+       V4L2_VP9_REF_MODE_SINGLE,
+       V4L2_VP9_REF_MODE_COMPOUND,
+       V4L2_VP9_REF_MODE_SELECT,
+};
+
+/**
+ * enum v4l2_vp9_tx_mode - VP9 TX modes
+ *
+ * @V4L2_VP9_TX_MODE_ONLY_4X4: transform size is 4x4
+ * @V4L2_VP9_TX_MODE_ALLOW_8X8: transform size can be up to 8x8
+ * @V4L2_VP9_TX_MODE_ALLOW_16X16: transform size can be up to 16x16
+ * @V4L2_VP9_TX_MODE_ALLOW_32X32: transform size can be up to 32x32
+ * @V4L2_VP9_TX_MODE_SELECT: bitstream contains transform size for each block
+ *
+ * See section '7.3.1 Tx mode semantics' of the VP9 specification for more
+ * details.
+ */
+enum v4l2_vp9_tx_mode {
+       V4L2_VP9_TX_MODE_ONLY_4X4,
+       V4L2_VP9_TX_MODE_ALLOW_8X8,
+       V4L2_VP9_TX_MODE_ALLOW_16X16,
+       V4L2_VP9_TX_MODE_ALLOW_32X32,
+       V4L2_VP9_TX_MODE_SELECT,
+};
+
+/**
+ * enum v4l2_vp9_ref_id - VP9 Reference frame IDs
+ *
+ * @V4L2_REF_ID_LAST: last reference frame
+ * @V4L2_REF_ID_GOLDEN: golden reference frame
+ * @V4L2_REF_ID_ALTREF: alternative reference frame
+ * @V4L2_REF_ID_CNT: number of reference frames
+ *
+ * See section '7.4.12 Ref frames semantics' of the VP9 specification for more
+ * details.
+ */
+enum v4l2_vp9_ref_id {
+       V4L2_REF_ID_LAST,
+       V4L2_REF_ID_GOLDEN,
+       V4L2_REF_ID_ALTREF,
+       V4L2_REF_ID_CNT,
+};
+
+/**
+ * enum v4l2_vp9_frame_flags - VP9 frame flags
+ * @V4L2_VP9_FRAME_FLAG_KEY_FRAME: the frame is a key frame
+ * @V4L2_VP9_FRAME_FLAG_SHOW_FRAME: the frame should be displayed
+ * @V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT: the decoding should be error resilient
+ * @V4L2_VP9_FRAME_FLAG_INTRA_ONLY: the frame does not reference other frames
+ * @V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV: the frame might can high precision
+ *                                         motion vectors
+ * @V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX: frame context should be updated
+ *                                        after decoding
+ * @V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE: parallel decoding is used
+ * @V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING: vertical subsampling is enabled
+ * @V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING: horizontal subsampling is enabled
+ * @V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING: full UV range is used
+ *
+ * Check the VP9 specification for more details.
+ */
+enum v4l2_vp9_frame_flags {
+       V4L2_VP9_FRAME_FLAG_KEY_FRAME = 1 << 0,
+       V4L2_VP9_FRAME_FLAG_SHOW_FRAME = 1 << 1,
+       V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT = 1 << 2,
+       V4L2_VP9_FRAME_FLAG_INTRA_ONLY = 1 << 3,
+       V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV = 1 << 4,
+       V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX = 1 << 5,
+       V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE = 1 << 6,
+       V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING = 1 << 7,
+       V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING = 1 << 8,
+       V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING = 1 << 9,
+};
+
+#define V4L2_VP9_PROFILE_MAX           3
+
+/**
+ * struct v4l2_ctrl_vp9_frame_decode_params - VP9 frame decoding control
+ *
+ * @flags: combination of V4L2_VP9_FRAME_FLAG_* flags
+ * @compressed_header_size: compressed header size in bytes
+ * @uncompressed_header_size: uncompressed header size in bytes
+ * @profile: VP9 profile. Can be 0, 1, 2 or 3
+ * @reset_frame_context: specifies whether the frame context should be reset
+ *                      to default values. See &v4l2_vp9_reset_frame_context
+ *                      for more details
+ * @frame_context_idx: frame context that should be used/updated
+ * @bit_depth: bits per components. Can be 8, 10 or 12. Note that not all
+ *            profiles support 10 and/or 12 bits depths
+ * @interpolation_filter: specifies the filter selection used for performing
+ *                       inter prediction. See &v4l2_vp9_interpolation_filter
+ *                       for more details
+ * @tile_cols_log2: specifies the base 2 logarithm of the width of each tile
+ *                 (where the width is measured in units of 8x8 blocks).
+ *                 Shall be less than or equal to 6
+ * @tile_rows_log2: specifies the base 2 logarithm of the height of each tile
+ *                 (where the height is measured in units of 8x8 blocks)
+ * @tx_mode: specifies the TX mode. See &v4l2_vp9_tx_mode for more details
+ * @reference_mode: specifies the type of inter prediction to be used. See
+ *                 &v4l2_vp9_reference_mode for more details
+ * @padding: needed to make this struct 64 bit aligned. Shall be filled with
+ *          zeros
+ * @frame_width_minus_1: add 1 to it and you'll get the frame width expressed
+ *                      in pixels
+ * @frame_height_minus_1: add 1 to it and you'll get the frame height expressed
+ *                       in pixels
+ * @frame_width_minus_1: add 1 to it and you'll get the expected render width
+ *                      expressed in pixels. This is not used during the
+ *                      decoding process but might be used by HW scalers to
+ *                      prepare a frame that's ready for scanout
+ * @frame_height_minus_1: add 1 to it and you'll get the expected render height
+ *                      expressed in pixels. This is not used during the
+ *                      decoding process but might be used by HW scalers to
+ *                      prepare a frame that's ready for scanout
+ * @refs: array of reference frames. See &v4l2_vp9_ref_id for more details
+ * @lf: loop filter parameters. See &v4l2_vp9_loop_filter for more details
+ * @quant: quantization parameters. See &v4l2_vp9_quantization for more details
+ * @seg: segmentation parameters. See &v4l2_vp9_segmentation for more details
+ * @probs: probabilities. See &v4l2_vp9_probabilities for more details
+ */
+struct v4l2_ctrl_vp9_frame_decode_params {
+       __u32 flags;
+       __u16 compressed_header_size;
+       __u16 uncompressed_header_size;

include/media/vp9-ctrls.h line 453

Quoted 10 lines
include/media/vp9-ctrls.h new file +10 -0
+       __u8 profile;
+       __u8 reset_frame_context;
+       __u8 frame_context_idx;
+       __u8 bit_depth;
+       __u8 interpolation_filter;
+       __u8 tile_cols_log2;
+       __u8 tile_rows_log2;
+       __u8 tx_mode;
+       __u8 reference_mode;
+       __u8 padding[6];

If my math is correct this should be 7.

include/media/vp9-ctrls.h line 458

Quoted 5 lines
include/media/vp9-ctrls.h new file +5 -0
+       __u16 frame_width_minus_1;
+       __u16 frame_height_minus_1;
+       __u16 render_width_minus_1;
+       __u16 render_height_minus_1;
+       __u64 refs[V4L2_REF_ID_CNT];

Why are we using u64s to store these [0..7] indices? Or are we storing more than that?

include/media/vp9-ctrls.h line 463

Quoted 5 lines
include/media/vp9-ctrls.h new file +5 -0
+       struct v4l2_vp9_loop_filter lf;
+       struct v4l2_vp9_quantization quant;
+       struct v4l2_vp9_segmentation seg;
+       struct v4l2_vp9_probabilities probs;
+};

I cannot find a trace of ref_frame_sign_biases, which is also part of the uncompressed header, in this structure. Is it missing, or is this information provided in a different way?

Quoted 23 lines
include/media/vp9-ctrls.h new file +22 -1
+
+#define V4L2_VP9_NUM_FRAME_CTX 4
+
+/**
+ * struct v4l2_ctrl_vp9_frame_ctx - VP9 frame context control
+ *
+ * @probs: VP9 probabilities
+ *
+ * This control is accessed in both direction. The user should initialize the
+ * 4 contexts with default values just after starting the stream. Then before
+ * decoding a frame it should query the current frame context (the one passed
+ * through &v4l2_ctrl_vp9_frame_decode_params.frame_context_idx) to initialize
+ * &v4l2_ctrl_vp9_frame_decode_params.probs. The probs are then adjusted based
+ * on the bitstream info and passed to the kernel. The codec should update
+ * the frame context after the frame has been decoded, so that next time
+ * userspace query this context it contains the updated probabilities.
+ */
+struct v4l2_ctrl_vp9_frame_ctx {
+       struct v4l2_vp9_probabilities probs;
+};
+
+#endif /* _VP9_CTRLS_H_ */
--
Quoted 2 lines

2.26.0.rc2

_______________________________________________
Linux-rockchip mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip

#7 lore raw

Re: [PATCH v4 2/3] media: uapi: Add VP9 stateless decoder controls

Quoted 19 lines · Alexandre Courbot

Hi Ezequiel, sorry for the late review!

Quoted 15 lines · Ezequiel Garcia

From: Boris Brezillon <[email protected]>

Add the VP9 stateless decoder controls plus the documentation that goes with it.

---
 .../userspace-api/media/v4l/biblio.rst        |  10 +
 .../media/v4l/ext-ctrls-codec.rst             | 550 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ctrls.c          | 239 ++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   1 +
 include/media/v4l2-ctrls.h                    |   1 +
 include/media/vp9-ctrls.h                     | 485 +++++++++++++++

Also another thing I missed: shouldn't the new pointer controls be added to union v4l2_ctrl_ptr in v4l2-ctrls.h?

_______________________________________________
Linux-rockchip mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip

#8 lore raw

Re: [PATCH v4 2/3] media: uapi: Add VP9 stateless decoder controls

In reply to Alexandre Courbot

Le jeudi 10 septembre 2020 à 15:04 +0900, Alexandre Courbot a écrit :

Quoted 71 lines

Hi Ezequiel, sorry for the late review!

Quoted 67 lines · Ezequiel Garcia

From: Boris Brezillon <[email protected]>

Add the VP9 stateless decoder controls plus the documentation that goes with it.

---
 .../userspace-api/media/v4l/biblio.rst        |  10 + .../media/v4l/ext-ctrls-codec.rst             | 550 ++++++++++++++++++ drivers/media/v4l2-core/v4l2-ctrls.c          | 239 ++++++++ drivers/media/v4l2-core/v4l2-ioctl.c          |   1 + include/media/v4l2-ctrls.h                    |   1 + include/media/vp9-ctrls.h                     | 485 +++++++++++++++

 6 files changed, 1286 insertions(+)

 create mode 100644 include/media/vp9-ctrls.h

Documentation/userspace-api/media/v4l/biblio.rst +0 -0
index 3c9634173e82..e09102e572fd 100644
@@ -414,3 +414,13 @@ VP8

 :title:     RFC 6386: "VP8 Data Format and Decoding Guide"

 :author:    J. Bankoski et al.

Documentation/userspace-api/media/v4l/biblio.rst +10 -0
+
+.. _vp9:
+
+VP9
+===
+
+
+:title:     VP9 Bitstream & Decoding Process Specification
+
+:author:    Adrian Grange (Google), Peter de Rivaz (Argon Design), Jonathan

Hunt (Argon Design)

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +0 -0
index d0d506a444b1..5c5f7dd868da 100644
@@ -2668,6 +2668,556 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -

       - ``padding[3]``
       - Applications and drivers must set this to zero.

+.. _v4l2-mpeg-vp9:

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +15 -0
+
+``V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(0..3) (struct)``
+    Stores VP9 probabilities attached to a specific frame context. The VP9
+    specification allows using a maximum of 4 contexts. Each frame being
+    decoded refers to one of those context. See section '7.1.2 Refresh
+    probs semantics' section of :ref:`vp9` for more details about these
+    contexts.
+
+    This control is bi-directional:
+
+    * all 4 contexts must be initialized by userspace just after the
+      stream is started and before the first decoding request is submitted.
+    * the referenced context might be read by the kernel when a decoding
+      request is submitted, and will be updated after the decoder is done
+      decoding the frame if the `V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX`

flag
+      is set.
+    * contexts will be read back by user space before each decoding request +      to retrieve the updated probabilities.

Quoted 8 lines
Quoted 2 lines

+    * userspace will re-initialize the context to their default values when +      a reset context is required.

Just to make sure I understand this part correctly, it means that if frame A and B use the same context, and frame A has V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX set, then user-space must wait for frame A to get dequeued and read back this control from the completed request before it can submit frame B?

We are preparing a new version, we believe it was an API mistake to try and share the probability update between kernel and userspace. It's actually worst, you really need to push/pull in a lock step, regardless of the frame context id.

As you may know, the probabilities are used to parse the compressed part of the stream (also know as entropy decoding). They are probability of a bitstream symbol of being 0 or 1. On RK3399, the hardware does not parse the compressed header. The compressed headers contains the bitstream coded updates to the probabilities. The probabilities are also updated base on the count of specific symbols found in the bitstream during decoding, this is currently done in the driver (counts are returned, and driver updates the probabilities from that).

What we found is that the symbols in the compress headers are coded with fixed probabilities, meaning we don't need the probabilities to parse the compressed headers. Our proposal will be to keep doing the compressed header parsing in userspace and only store the update value (as found in inv_map_table) to the kernel. The driver will be responsible of applying this to the current probabilities along with doing the probability saving process.

With this change, we should be able to only push data to the decoder without having to read this control. We believe this will allow better use of the VB2 queue and enhance the decoding performance.

Please note that this hardware design looks like a mistake. It seems that with widevine, the compressed header is encrypted. As in some implementation the decrypted bitstream is no visible to the CPU, it will not be possible to perform the needed parsing. An offload to a TEE would be needed. We know that Rockchip have fixed this issue in new decoder, notably on RK3368. The probability processing is done in HW.

Quoted 523 lines
Quoted 523 lines
Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +104 -0
+
+    .. note::
+
+       This compound control is not yet part of the public kernel API and
+       it is expected to change.
+
+.. c:type:: v4l2_ctrl_vp9_frame_ctx
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{5.8cm}|p{4.8cm}|p{6.6cm}|
+
+.. flat-table:: struct v4l2_ctrl_vp9_frame_ctx
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - struct :c:type:`v4l2_vp9_probabilities`
+      - ``probs``
+      - Structure with VP9 probabilities attached to the context.
+
+.. c:type:: v4l2_vp9_probabilities
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_probabilities
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``tx8[2][1]``
+      - TX 8x8 probabilities.
+    * - __u8
+      - ``tx16[2][2]``
+      - TX 16x16 probabilities.
+    * - __u8
+      - ``tx32[2][3]``
+      - TX 32x32 probabilities.
+    * - __u8
+      - ``coef[4][2][2][6][6][3]``
+      - Coefficient probabilities.
+    * - __u8
+      - ``skip[3]``
+      - Skip probabilities.
+    * - __u8
+      - ``inter_mode[7][3]``
+      - Inter prediction mode probabilities.
+    * - __u8
+      - ``interp_filter[4][2]``
+      - Interpolation filter probabilities.
+    * - __u8
+      - ``is_inter[4]``
+      - Is inter-block probabilities.
+    * - __u8
+      - ``comp_mode[5]``
+      - Compound prediction mode probabilities.
+    * - __u8
+      - ``single_ref[5][2]``
+      - Single reference probabilities.
+    * - __u8
+      - ``comp_mode[5]``
+      - Compound reference probabilities.
+    * - __u8
+      - ``y_mode[4][9]``
+      - Y prediction mode probabilities.
+    * - __u8
+      - ``uv_mode[10][9]``
+      - UV prediction mode probabilities.
+    * - __u8
+      - ``partition[16][3]``
+      - Partition probabilities.
+    * - __u8
+      - ``mv.joint[3]``
+      - Motion vector joint probabilities.
+    * - __u8
+      - ``mv.sign[2]``
+      - Motion vector sign probabilities.
+    * - __u8
+      - ``mv.class[2][10]``
+      - Motion vector class probabilities.
+    * - __u8
+      - ``mv.class0_bit[2]``
+      - Motion vector class0 bit probabilities.
+    * - __u8
+      - ``mv.bits[2][10]``
+      - Motion vector bits probabilities.
+    * - __u8
+      - ``mv.class0_fr[2][2][3]``
+      - Motion vector class0 fractional bit probabilities.
+    * - __u8
+      - ``mv.fr[2][3]``
+      - Motion vector fractional bit probabilities.
+    * - __u8
+      - ``mv.class0_hp[2]``
+      - Motion vector class0 high precision fractional bit probabilities.
+    * - __u8
+      - ``mv.hp[2]``
+      - Motion vector high precision fractional bit probabilities.
+
+``V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS (struct)``
+    Specifies the frame parameters for the associated VP9 frame decode

request.
+    This includes the necessary parameters for configuring a stateless hardware
+    decoding pipeline for VP9. The bitstream parameters are defined according
+    to :ref:vp9.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +36 -0
+
+    .. note::
+
+       This compound control is not yet part of the public kernel API and
+       it is expected to change.
+
+.. c:type:: v4l2_ctrl_vp9_frame_decode_params
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_ctrl_vp9_frame_decode_params
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``flags``
+      - Combination of V4L2_VP9_FRAME_FLAG_* flags. See
+        :c:type:`v4l2_vp9_frame_flags`.
+    * - __u16
+      - ``compressed_header_size``
+      - Compressed header size in bytes.
+    * - __u16
+      - ``uncompressed_header_size``
+      - Uncompressed header size in bytes.
+    * - __u8
+      - ``profile``
+      - VP9 profile. Can be 0, 1, 2 or 3.
+    * - __u8
+      - ``reset_frame_context``
+      - Frame context that should be used/updated when decoding the frame.
+    * - __u8
+      - ``bit_depth``
+      - Component depth in bits. Must be 8 for profile 0 and 1. Must 10 or

12
+        for profile 2 and 3.
+    * - __u8
+      - ``interpolation_filter``
+      - Specifies the filter selection used for performing inter prediction. See
+        :c:type:v4l2_vp9_interpolation_filter.
+    * - __u8
+      - ``tile_cols_log2``
+      - Specifies the base 2 logarithm of the width of each tile (where the +        width is measured in units of 8x8 blocks). Shall be less than or equal
+        to 6.
+    * - __u8
+      - ``tile_rows_log2``
+      - Specifies the base 2 logarithm of the height of each tile (where the
+        height is measured in units of 8x8 blocks)
+    * - __u8
+      - ``tx_mode``
+      - Specifies the TX mode. See :c:type:v4l2_vp9_tx_mode. +    * - __u8
+      - ``reference_mode``
+      - Specifies the type of inter prediction to be used. See +        :c:type:v4l2_vp9_reference_mode.
+    * - __u8
+      - ``padding``
+      - Needed to make this struct 64 bit aligned. Shall be filled with zeroes.
+    * - __u16
+      - ``frame_width_minus_1``
+      - Add 1 to get the frame width expressed in pixels.
+    * - __u16
+      - ``frame_height_minus_1``
+      - Add 1 to to get the frame height expressed in pixels.
+    * - __u16
+      - ``frame_width_minus_1``
+      - Add 1 to to get the expected render width expressed in pixels. This is
+        not used during the decoding process but might be used by HW scalers to
+        prepare a frame that's ready for scanout.
+    * - __u16
+      - frame_height_minus_1
+      - Add 1 to get the expected render height expressed in pixels. This is
+        not used during the decoding process but might be used by HW scalers to
+        prepare a frame that's ready for scanout.
+    * - __u64
+      - ``refs[3]``
+      - Array of reference frame timestamps.
+    * - struct :c:type:v4l2_vp9_loop_filter
+      - ``lf``
+      - Loop filter parameters. See struct :c:type:v4l2_vp9_loop_filter. +    * - struct :c:type:v4l2_vp9_quantization
+      - ``quant``
+      - Quantization parameters. See :c:type:v4l2_vp9_quantization. +    * - struct :c:type:v4l2_vp9_segmentation
+      - ``seg``
+      - Segmentation parameters. See :c:type:v4l2_vp9_segmentation. +    * - struct :c:type:v4l2_vp9_probabilities
+      - ``probs``
+      - Probabilities. See :c:type:v4l2_vp9_probabilities.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +88 -0
+
+.. c:type:: v4l2_vp9_frame_flags
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_frame_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_FRAME_FLAG_KEY_FRAME``
+      - The frame is a key frame.
+    * - ``V4L2_VP9_FRAME_FLAG_SHOW_FRAME``
+      - The frame should be displayed.
+    * - ``V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT``
+      - The decoding should be error resilient.
+    * - ``V4L2_VP9_FRAME_FLAG_INTRA_ONLY``
+      - The frame does not reference other frames.
+    * - ``V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV``
+      - the frame might can high precision motion vectors.
+    * - ``V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX``
+      - Frame context should be updated after decoding.
+    * - ``V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE``
+      - Parallel decoding is used.
+    * - ``V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING``
+      - Vertical subsampling is enabled.
+    * - ``V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING``
+      - Horizontal subsampling is enabled.
+    * - ``V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING``
+      - The full UV range is used.
+
+.. c:type:: v4l2_vp9_ref_id
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_ref_id
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_REF_ID_LAST``
+      - Last reference frame.
+    * - ``V4L2_REF_ID_GOLDEN``
+      - Golden reference frame.
+    * - ``V4L2_REF_ID_ALTREF``
+      - Alternative reference frame.
+    * - ``V4L2_REF_ID_CNT``
+      - Number of reference frames.
+
+.. c:type:: v4l2_vp9_tx_mode
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_tx_mode
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_TX_MODE_ONLY_4X4``
+      - Transform size is 4x4.
+    * - ``V4L2_VP9_TX_MODE_ALLOW_8X8``
+      - Transform size can be up to 8x8.
+    * - ``V4L2_VP9_TX_MODE_ALLOW_16X16``
+      - Transform size can be up to 16x16.
+    * - ``V4L2_VP9_TX_MODE_ALLOW_32X32``
+      - transform size can be up to 32x32.
+    * - ``V4L2_VP9_TX_MODE_SELECT``
+      - Bitstream contains transform size for each block.
+
+.. c:type:: v4l2_vp9_reference_mode
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_reference_mode
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_REF_MODE_SINGLE``
+      - Indicates that all the inter blocks use only a single reference

frame
+        to generate motion compensated prediction.
+    * - ``V4L2_VP9_REF_MODE_COMPOUND``
+      - Requires all the inter blocks to use compound mode. Single reference
+        frame prediction is not allowed.
+    * - ``V4L2_VP9_REF_MODE_SELECT``
+      - Allows each individual inter block to select between single and +        compound prediction modes.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +92 -0
+
+.. c:type:: v4l2_vp9_interpolation_filter
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_interpolation_filter
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_INTERP_FILTER_8TAP``
+      - Height tap filter.
+    * - ``V4L2_VP9_INTERP_FILTER_8TAP_SMOOTH``
+      - Height tap smooth filter.
+    * - ``V4L2_VP9_INTERP_FILTER_8TAP_SHARP``
+      - Height tap sharp filter.
+    * - ``V4L2_VP9_INTERP_FILTER_BILINEAR``
+      - Bilinear filter.
+    * - ``V4L2_VP9_INTERP_FILTER_SWITCHABLE``
+      - Filter selection is signaled at the block level.
+
+.. c:type:: v4l2_vp9_reset_frame_context
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_reset_frame_context
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_RESET_FRAME_CTX_NONE``
+      - Do not reset any frame context.
+    * - ``V4L2_VP9_RESET_FRAME_CTX_SPEC``
+      - Reset the frame context pointed by
+        :c:type:`v4l2_ctrl_vp9_frame_decode_params`.frame_context_idx.
+    * - ``V4L2_VP9_RESET_FRAME_CTX_ALL``
+      - Reset all frame contexts.
+
+.. c:type:: v4l2_vp9_intra_prediction_mode
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_intra_prediction_mode
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_INTRA_PRED_DC``
+      - DC intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_V``
+      - Vertical intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_H``
+      - Horizontal intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D45``
+      - D45 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D135``
+      - D135 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D117``
+      - D117 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D153``
+      - D153 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D207``
+      - D207 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D63``
+      - D63 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_TM``
+      - True motion intra prediction.
+
+.. c:type:: v4l2_vp9_segmentation
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_segmentation
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``flags``
+      - Combination of V4L2_VP9_SEGMENTATION_FLAG_* flags. See
+        :c:type:`v4l2_vp9_segmentation_flags`.
+    * - __u8
+      - ``tree_probs[7]``
+      - Specifies the probability values to be used when decoding a

Segment-ID.
+        See '5.15. Segmentation map' section of :ref:vp9 for more details.
+    * - __u8
+      - ``pred_prob[3]``
+      - Specifies the probability values to be used when decoding a +        Predicted-Segment-ID. See '6.4.14. Get segment id syntax' +        section of :ref:vp9 for more details.
+    * - __u8
+      - ``padding[5]``
+      - Used to align this struct on 64 bit. Shall be filled with zeroes. +    * - __u8
+      - ``feature_enabled[8]``
+      - Bitmask defining which features are enabled in each segment. +    * - __u8
+      - ``feature_data[8][4]``
+      - Data attached to each feature. Data entry is only valid if the feature
+        is enabled.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +47 -0
+
+.. c:type:: v4l2_vp9_segment_feature
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_segment_feature
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_SEGMENT_FEATURE_QP_DELTA``
+      - QP delta segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_LF``
+      - Loop filter segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_REF_FRAME``
+      - Reference frame segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_SKIP``
+      - Skip segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_CNT``
+      - Number of segment features.
+
+.. c:type:: v4l2_vp9_segmentation_flags
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_segmentation_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_ENABLED``
+      - Indicates that this frame makes use of the segmentation tool.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP``
+      - Indicates that the segmentation map should be updated during the
+        decoding of this frame.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE``
+      - Indicates that the updates to the segmentation map are coded
+        relative to the existing segmentation map.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA``
+      - Indicates that new parameters are about to be specified for each
+        segment.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE``
+      - Indicates that the segmentation parameters represent the actual

values
+        to be used.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +27 -0
+
+.. c:type:: v4l2_vp9_quantization
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_quantization
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``base_q_idx``
+      - Indicates the base frame qindex.
+    * - __s8
+      - ``delta_q_y_dc``
+      - Indicates the Y DC quantizer relative to base_q_idx.
+    * - __s8
+      - ``delta_q_uv_dc``
+      - Indicates the UV DC quantizer relative to base_q_idx.
+    * - __s8
+      - ``delta_q_uv_ac``
+      - Indicates the UV AC quantizer relative to base_q_idx.
+    * - __u8
+      - ``padding[4]``
+      - Padding bytes used to align this struct on 64 bit. Must be set to

0.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +25 -0
+
+.. c:type:: v4l2_vp9_loop_filter
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_loop_filter
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``flags``
+      - Combination of V4L2_VP9_LOOP_FILTER_FLAG_* flags.
+        See :c:type:`v4l2_vp9_loop_filter_flags`.
+    * - __u8
+      - ``level``
+      - Indicates the loop filter strength.
+    * - __u8
+      - ``sharpness``
+      - Indicates the sharpness level.
+    * - __s8
+      - ``ref_deltas[4]``
+      - Contains the adjustment needed for the filter level based on the

chosen
+        reference frame.

Quoted 8 lines
Quoted 8 lines

+    * - __s8
+      - ``mode_deltas[2]``
+      - Contains the adjustment needed for the filter level based on the chosen
+        mode
+    * - __u8
+      - ``level_lookup[8][4][2]``
+      - Level lookup table.

We think it might be a poor choice to ask for computed values. According to "8.8.1 Loop filter frame init process" this is computed from loop_filter_level, loop_filter_ref_deltas[], V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE, feature_data[], ref_deltas[] and mode_deltas[]. We believe it is likely a waste to pass this information assuming some HW could do that process.

This is to be discussed. None of the other statelss APIs seems to require this value to be computed by userspace.

Quoted 870 lines
Quoted 420 lines
Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +15 -0
+
+
+.. c:type:: v4l2_vp9_loop_filter_flags
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_loop_filter_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED``
+      - When set, the filter level depends on the mode and reference frame

used
+        to predict a block.
+    * - ``V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE``
+      - When set, the bitstream contains additional syntax elements that +        specify which mode and reference frame deltas are to be updated. +
 .. raw:: latex

     \normalsize

drivers/media/v4l2- renamed from drivers/media/v4l2-core/v4l2-ctrls.c +0 -0

core/v4l2-ctrls.c
index 1c617b42a944..115beadaf02d 100644

drivers/media/v4l2-core/v4l2-ctrls.c +0 -0
@@ -930,6 +930,11 @@ const char *v4l2_ctrl_get_name(u32 id)

        case V4L2_CID_MPEG_VIDEO_VP8_PROFILE:                   return "VP8 Profile";
        case V4L2_CID_MPEG_VIDEO_VP9_PROFILE:                   return "VP9 Profile";
        case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:              return "VP8 Frame Header";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS:       return "VP9 Frame Decode Parameters";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(0):          return "VP9 Frame Context 0";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(1):          return "VP9 Frame Context 1";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(2):          return "VP9 Frame Context 2";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(3):          return "VP9 Frame Context 3";

        /* HEVC controls */
        case V4L2_CID_MPEG_VIDEO_HEVC_I_FRAME_QP:               return "HEVC I-Frame QP Value";

drivers/media/v4l2-core/v4l2-ctrls.c +0 -0
@@ -1403,6 +1408,15 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum
 v4l2_ctrl_type *type,

        case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
                *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER; break;
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS:
+               *type = V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS; +               break;
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(0):
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(1):
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(2):
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(3):
+               *type = V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT; +               break;
        case V4L2_CID_MPEG_VIDEO_HEVC_SPS:
                *type = V4L2_CTRL_TYPE_HEVC_SPS; break;

drivers/media/v4l2-core/v4l2-ctrls.c +143 -0
@@ -1703,6 +1717,219 @@ static void std_log(const struct v4l2_ctrl *ctrl)
         0;                                                      \
  })
 
+static int
+validate_vp9_lf_params(struct v4l2_vp9_loop_filter *lf)
+{
+       unsigned int i, j, k;
+
+       if (lf->flags &
+           ~(V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED |
+             V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED implies
+        * V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE.
+        */
+       if (lf->flags & V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE &&
+           !(lf->flags & V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED))
+               return -EINVAL;
+
+       /* That all values are in the accepted range. */
+       if (lf->level > GENMASK(5, 0))
+               return -EINVAL;
+
+       if (lf->sharpness > GENMASK(2, 0))
+               return -EINVAL;
+
+       for (i = 0; i < ARRAY_SIZE(lf->ref_deltas); i++) {
+               if (lf->ref_deltas[i] < -63 || lf->ref_deltas[i] > 63)
+                       return -EINVAL;
+       }
+
+       for (i = 0; i < ARRAY_SIZE(lf->mode_deltas); i++) {
+               if (lf->mode_deltas[i] < -63 || lf->mode_deltas[i] > 63)
+                       return -EINVAL;
+       }
+
+       for (i = 0; i < ARRAY_SIZE(lf->level_lookup); i++) {
+               for (j = 0; j < ARRAY_SIZE(lf->level_lookup[0]); j++) {
+                       for (k = 0; k < ARRAY_SIZE(lf->level_lookup[0][0]);
 k++) {
+                               if (lf->level_lookup[i][j][k] > 63)
+                                       return -EINVAL;
+                       }
+               }
+       }
+
+       return 0;
+}
+
+static int
+validate_vp9_quant_params(struct v4l2_vp9_quantization *quant)
+{
+       if (quant->delta_q_y_dc < -15 || quant->delta_q_y_dc > 15 ||
+           quant->delta_q_uv_dc < -15 || quant->delta_q_uv_dc > 15 ||
+           quant->delta_q_uv_ac < -15 || quant->delta_q_uv_ac > 15)
+               return -EINVAL;
+
+       memset(quant->padding, 0, sizeof(quant->padding));
+       return 0;
+}
+
+static int
+validate_vp9_seg_params(struct v4l2_vp9_segmentation *seg)
+{
+       unsigned int i, j;
+
+       if (seg->flags &
+           ~(V4L2_VP9_SEGMENTATION_FLAG_ENABLED |
+             V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP |
+             V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE |
+             V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA |
+             V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP and
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA imply
+        * V4L2_VP9_SEGMENTATION_FLAG_ENABLED.
+        */
+       if ((seg->flags &
+            (V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP |
+             V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA)) &&
+           !(seg->flags & V4L2_VP9_SEGMENTATION_FLAG_ENABLED))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE implies
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP.
+        */
+       if (seg->flags & V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE &&
+           !(seg->flags & V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE implies
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA.
+        */
+       if (seg->flags & V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE &&
+           !(seg->flags & V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA))
+               return -EINVAL;
+
+       for (i = 0; i < ARRAY_SIZE(seg->feature_enabled); i++) {
+               if (seg->feature_enabled[i] &
+                   ~(V4L2_VP9_SEGMENT_FEATURE_QP_DELTA |
+                     V4L2_VP9_SEGMENT_FEATURE_LF |
+                     V4L2_VP9_SEGMENT_FEATURE_REF_FRAME |
+                     V4L2_VP9_SEGMENT_FEATURE_SKIP))
+                       return -EINVAL;
+       }
+
+       for (i = 0; i < ARRAY_SIZE(seg->feature_data); i++) {
+               const int range[] = {255, 63, 3, 0};
+
+               for (j = 0; j < ARRAY_SIZE(seg->feature_data[j]); j++) {
+                       if (seg->feature_data[i][j] < -range[j] ||
+                           seg->feature_data[i][j] > range[j])
+                               return -EINVAL;
+               }
+       }
+
+       memset(seg->padding, 0, sizeof(seg->padding));
+       return 0;
+}
+
+static int
+validate_vp9_frame_decode_params(struct v4l2_ctrl_vp9_frame_decode_params
 *dec_params)
+{
+       int ret;
+
+       /* Make sure we're not passed invalid flags. */
+       if (dec_params->flags &
+           ~(V4L2_VP9_FRAME_FLAG_KEY_FRAME |
+             V4L2_VP9_FRAME_FLAG_SHOW_FRAME |
+             V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT |
+             V4L2_VP9_FRAME_FLAG_INTRA_ONLY |
+             V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV |
+             V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX |
+             V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE |
+             V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING |
+             V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING |
+             V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING))
+               return -EINVAL;
+
+       /*
+        * The refresh context and error resilient flags are mutually

exclusive.
+        * Same goes for parallel decoding and error resilient modes. +        */
+       if (dec_params->flags & V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT && +           dec_params->flags &
+           (V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX |
+            V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE))
+               return -EINVAL;

drivers/media/v4l2-core/v4l2-ctrls.c +12 -0
+
+       if (dec_params->profile > V4L2_VP9_PROFILE_MAX)
+               return -EINVAL;
+
+       if (dec_params->reset_frame_context > V4L2_VP9_RESET_FRAME_CTX_ALL)
+               return -EINVAL;
+
+       if (dec_params->frame_context_idx >= V4L2_VP9_NUM_FRAME_CTX)
+               return -EINVAL;
+
+       /*
+        * Profiles 0 and 1 only support 8-bit depth, profiles 2 and 3 only

10
+        * and 12 bit depths.
+        */
+       if ((dec_params->profile < 2 && dec_params->bit_depth != 8) || +           (dec_params->profile >= 2 &&
+            (dec_params->bit_depth != 10 && dec_params->bit_depth != 12))) +               return -EINVAL;

drivers/media/v4l2-core/v4l2-ctrls.c +14 -0
+
+       /* Profile 0 and 2 only accept YUV 4:2:0. */
+       if ((dec_params->profile == 0 || dec_params->profile == 2) &&
+           (!(dec_params->flags & V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING) ||
+            !(dec_params->flags & V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING)))
+               return -EINVAL;
+
+       /* Profile 1 and 3 only accept YUV 4:2:2, 4:4:0 and 4:4:4. */
+       if ((dec_params->profile == 1 || dec_params->profile == 3) &&
+           ((dec_params->flags & V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING) &&
+            (dec_params->flags & V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING)))
+               return -EINVAL;
+
+       if (dec_params->interpolation_filter >

V4L2_VP9_INTERP_FILTER_SWITCHABLE)
+               return -EINVAL;

drivers/media/v4l2-core/v4l2-ctrls.c +30 -0
+
+       /*
+        * According to the spec, tile_cols_log2 shall be less than or equal
+        * to 6.
+        */
+       if (dec_params->tile_cols_log2 > 6)
+               return -EINVAL;
+
+       if (dec_params->tx_mode > V4L2_VP9_TX_MODE_SELECT)
+               return -EINVAL;
+
+       if (dec_params->reference_mode > V4L2_VP9_REF_MODE_SELECT)
+               return -EINVAL;
+
+       ret = validate_vp9_lf_params(&dec_params->lf);
+       if (ret)
+               return ret;
+
+       ret = validate_vp9_quant_params(&dec_params->quant);
+       if (ret)
+               return ret;
+
+       ret = validate_vp9_seg_params(&dec_params->seg);
+       if (ret)
+               return ret;
+
+       memset(dec_params->padding, 0, sizeof(dec_params->padding));
+       return 0;
+}
+

 /* Validate a new control */

 #define zero_padding(s) \

drivers/media/v4l2-core/v4l2-ctrls.c +6 -0
@@ -1799,6 +2026,12 @@ static int std_validate_compound(const struct
 v4l2_ctrl *ctrl, u32 idx,
                 zero_padding(p_vp8_frame_header->coder_state);
                 break;
 
+       case V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS:
+               return validate_vp9_frame_decode_params(p);
+
+       case V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT:
+               break;
+

        case V4L2_CTRL_TYPE_HEVC_SPS:
                p_hevc_sps = p;

drivers/media/v4l2-core/v4l2-ctrls.c +0 -0
@@ -2542,6 +2775,12 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct
 v4l2_ctrl_handler *hdl,

        case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
                elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header); break;
+       case V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT:
+               elem_size = sizeof(struct v4l2_ctrl_vp9_frame_ctx); +               break;
+       case V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS:
+               elem_size = sizeof(struct v4l2_ctrl_vp9_frame_decode_params);
+               break;
        case V4L2_CTRL_TYPE_HEVC_SPS:
                elem_size = sizeof(struct v4l2_ctrl_hevc_sps); break;

drivers/media/v4l2- renamed from drivers/media/v4l2-core/v4l2-ioctl.c +0 -0

core/v4l2-ioctl.c
index 2322f08a98be..4fe77ad917c8 100644

drivers/media/v4l2-core/v4l2-ioctl.c +1 -0
@@ -1423,6 +1423,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
                 case V4L2_PIX_FMT_VP8:          descr = "VP8"; break;
                 case V4L2_PIX_FMT_VP8_FRAME:    descr = "VP8 Frame"; break;
                 case V4L2_PIX_FMT_VP9:          descr = "VP9"; break;
+               case V4L2_PIX_FMT_VP9_FRAME:    descr = "VP9 Frame"; break;

                case V4L2_PIX_FMT_HEVC:         descr = "HEVC"; break; /* aka H.265 */
                case V4L2_PIX_FMT_HEVC_SLICE:   descr = "HEVC Parsed Slice Data"; break;
                case V4L2_PIX_FMT_FWHT:         descr = "FWHT"; break; /* used in vicodec */

include/media/v4l2-ctrls.h +1 -0
index 757a713bad41..2de8290f9f11 100644
@@ -21,6 +21,7 @@
  #include <media/fwht-ctrls.h>
  #include <media/h264-ctrls.h>
  #include <media/vp8-ctrls.h>
+#include <media/vp9-ctrls.h>
  #include <media/hevc-ctrls.h>
 
  /* forward references */
include/media/vp9-ctrls.h new file +29 -0
new file mode 100644
index 000000000000..0cdea8a18b72
@@ -0,0 +1,485 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * These are the VP9 state controls for use with stateless VP9
+ * codec drivers.
+ *
+ * It turns out that these structs are not stable yet and will undergo
+ * more changes. So keep them private until they are stable and ready to
+ * become part of the official public API.
+ */
+
+#ifndef _VP9_CTRLS_H_
+#define _VP9_CTRLS_H_
+
+#include <linux/types.h>
+
+#define V4L2_PIX_FMT_VP9_FRAME v4l2_fourcc('V', 'P', '9', 'F')
+
+#define V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(i)       (V4L2_CID_MPEG_BASE
+ 4000 + (i))
+#define V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS    (V4L2_CID_MPEG_BASE
+ 4004)
+#define V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT               0x400
+#define V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS         0x404
+
+/**
+ * enum v4l2_vp9_loop_filter_flags - VP9 loop filter flags
+ *
+ * @V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED: the filter level depends on
+ *                                          the mode and reference frame

used
+ *                                          to predict a block
+ * @V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE: the bitstream contains additional
+ *                                         syntax elements that specify which
+ *                                         mode and reference frame deltas
+ *                                         are to be updated
+ *
+ * Those are the flags you should pass to &v4l2_vp9_loop_filter.flags. See
+ * section '7.2.8 Loop filter semantics' of the VP9 specification for more
+ * details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+enum v4l2_vp9_loop_filter_flags {
+       V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED = 1 << 0,
+       V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE = 1 << 1,
+};
+
+/**
+ * struct v4l2_vp9_loop_filter - VP9 loop filter parameters
+ *
+ * @flags: combination of V4L2_VP9_LOOP_FILTER_FLAG_* flags
+ * @level: indicates the loop filter strength
+ * @sharpness: indicates the sharpness level
+ * @ref_deltas: contains the adjustment needed for the filter level based

on
+ *             the chosen reference frame
+ * @mode_deltas: contains the adjustment needed for the filter level based on
+ *              the chosen mode
+ * @level_lookup: level lookup table
+ *
+ * This structure contains all loop filter related parameters. See sections
+ * '7.2.8 Loop filter semantics' and '8.8.1 Loop filter frame init process'
+ * of the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h line 66

Quoted 8 lines
include/media/vp9-ctrls.h new file +8 -0
+struct v4l2_vp9_loop_filter {
+       __u8 flags;
+       __u8 level;
+       __u8 sharpness;
+       __s8 ref_deltas[4];
+       __s8 mode_deltas[2];
+       __u8 level_lookup[8][4][2];
+};

This struct is 73 bytes, doesn't it need padding?

Quoted 413 lines
include/media/vp9-ctrls.h new file +11 -0
+
+/**
+ * struct v4l2_vp9_quantization - VP9 quantization parameters
+ *
+ * @base_q_idx: indicates the base frame qindex
+ * @delta_q_y_dc: indicates the Y DC quantizer relative to base_q_idx
+ * @delta_q_uv_dc: indicates the UV DC quantizer relative to base_q_idx
+ * @delta_q_uv_ac indicates the UV AC quantizer relative to base_q_idx
+ * @padding: padding bytes to align things on 64 bits. Must be set to 0
+ *
+ * Encodes the quantization parameters. See section '7.2.9 Quantization

params
+ * syntax' of the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+struct v4l2_vp9_quantization {
+       __u8 base_q_idx;
+       __s8 delta_q_y_dc;
+       __s8 delta_q_uv_dc;
+       __s8 delta_q_uv_ac;
+       __u8 padding[4];
+};
+
+/**
+ * enum v4l2_vp9_segmentation_flags - VP9 segmentation flags
+ *
+ * @V4L2_VP9_SEGMENTATION_FLAG_ENABLED: indicates that this frame makes use

of
+ *                                     the segmentation tool
+ * @V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP: indicates that the segmentation map
+ *                                        should be updated during the
+ *                                        decoding of this frame
+ * @V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE: indicates that the updates to
+ *                                             the segmentation map are coded
+ *                                             relative to the existing
+ *                                             segmentation map
+ * @V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA: indicates that new parameters are
+ *                                         about to be specified for each
+ *                                         segment
+ * @V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE: indicates that the
+ *                                                 segmentation parameters
+ *                                                 represent the actual values
+ *                                                 to be used
+ *
+ * Those are the flags you should pass to &v4l2_vp9_segmentation.flags. See
+ * section '7.2.10 Segmentation params syntax' of the VP9 specification for
+ * more details.
+ */

include/media/vp9-ctrls.h new file +21 -0
+enum v4l2_vp9_segmentation_flags {
+       V4L2_VP9_SEGMENTATION_FLAG_ENABLED = 1 << 0,
+       V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP = 1 << 1,
+       V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE = 1 << 2,
+       V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA = 1 << 3,
+       V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE = 1 << 4,
+};
+
+#define V4L2_VP9_SEGMENT_FEATURE_ENABLED(id)   (1 << (id))
+#define V4L2_VP9_SEGMENT_FEATURE_ENABLED_MASK  0xf
+
+/**
+ * enum v4l2_vp9_segment_feature - VP9 segment feature IDs
+ *
+ * @V4L2_VP9_SEGMENT_FEATURE_QP_DELTA: QP delta segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_LF: loop filter segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_REF_FRAME: reference frame segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_SKIP: skip segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_CNT: number of segment features
+ *
+ * Segment feature IDs. See section '7.2.10 Segmentation params syntax' of

the
+ * VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +23 -0
+enum v4l2_vp9_segment_feature {
+       V4L2_VP9_SEGMENT_FEATURE_QP_DELTA,
+       V4L2_VP9_SEGMENT_FEATURE_LF,
+       V4L2_VP9_SEGMENT_FEATURE_REF_FRAME,
+       V4L2_VP9_SEGMENT_FEATURE_SKIP,
+       V4L2_VP9_SEGMENT_FEATURE_CNT,
+};
+
+/**
+ * struct v4l2_vp9_segmentation - VP9 segmentation parameters
+ *
+ * @flags: combination of V4L2_VP9_SEGMENTATION_FLAG_* flags
+ * @tree_probs: specifies the probability values to be used when
+ *              decoding a Segment-ID. See '5.15. Segmentation map'
+ *              section of the VP9 specification for more details.
+ * @pred_prob: specifies the probability values to be used when decoding a
+ *            Predicted-Segment-ID. See '6.4.14. Get segment id syntax'
+ *            section of :ref:`vp9` for more details..
+ * @padding: padding used to make things aligned on 64 bits. Shall be zero
+ *          filled
+ * @feature_enabled: bitmask defining which features are enabled in each
+ *                  segment
+ * @feature_data: data attached to each feature. Data entry is only valid

if
+ *               the feature is enabled
+ *
+ * Encodes the quantization parameters. See section '7.2.10 Segmentation
+ * params syntax' of the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +48 -0
+struct v4l2_vp9_segmentation {
+       __u8 flags;
+       __u8 tree_probs[7];
+       __u8 pred_probs[3];
+       __u8 padding[5];
+       __u8 feature_enabled[8];
+       __s16 feature_data[8][4];
+};
+
+/**
+ * enum v4l2_vp9_intra_prediction_mode - VP9 Intra prediction modes
+ *
+ * @V4L2_VP9_INTRA_PRED_DC: DC intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_V: vertical intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_H: horizontal intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D45: D45 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D135: D135 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D117: D117 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D153: D153 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D207: D207 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D63: D63 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_TM: True Motion intra prediction
+ *
+ * See section '7.4.5 Intra frame mode info semantics' for more details.
+ */
+enum v4l2_vp9_intra_prediction_mode {
+       V4L2_VP9_INTRA_PRED_MODE_DC,
+       V4L2_VP9_INTRA_PRED_MODE_V,
+       V4L2_VP9_INTRA_PRED_MODE_H,
+       V4L2_VP9_INTRA_PRED_MODE_D45,
+       V4L2_VP9_INTRA_PRED_MODE_D135,
+       V4L2_VP9_INTRA_PRED_MODE_D117,
+       V4L2_VP9_INTRA_PRED_MODE_D153,
+       V4L2_VP9_INTRA_PRED_MODE_D207,
+       V4L2_VP9_INTRA_PRED_MODE_D63,
+       V4L2_VP9_INTRA_PRED_MODE_TM,
+};
+
+/**
+ * struct v4l2_vp9_mv_probabilities - VP9 Motion vector probabilities
+ * @joint: motion vector joint probabilities
+ * @sign: motion vector sign probabilities
+ * @class: motion vector class probabilities
+ * @class0_bit: motion vector class0 bit probabilities
+ * @bits: motion vector bits probabilities
+ * @class0_fr: motion vector class0 fractional bit probabilities
+ * @fr: motion vector fractional bit probabilities
+ * @class0_hp: motion vector class0 high precision fractional bit

probabilities
+ * @hp: motion vector high precision fractional bit probabilities
+ */

include/media/vp9-ctrls.h new file +56 -0
+struct v4l2_vp9_mv_probabilities {
+       __u8 joint[3];
+       __u8 sign[2];
+       __u8 class[2][10];
+       __u8 class0_bit[2];
+       __u8 bits[2][10];
+       __u8 class0_fr[2][2][3];
+       __u8 fr[2][3];
+       __u8 class0_hp[2];
+       __u8 hp[2];
+};
+
+/**
+ * struct v4l2_vp9_probabilities - VP9 Probabilities
+ *
+ * @tx8: TX 8x8 probabilities
+ * @tx16: TX 16x16 probabilities
+ * @tx32: TX 32x32 probabilities
+ * @coef: coefficient probabilities
+ * @skip: skip probabilities
+ * @inter_mode: inter mode probabilities
+ * @interp_filter: interpolation filter probabilities
+ * @is_inter: is inter-block probabilities
+ * @comp_mode: compound prediction mode probabilities
+ * @single_ref: single ref probabilities
+ * @comp_ref: compound ref probabilities
+ * @y_mode: Y prediction mode probabilities
+ * @uv_mode: UV prediction mode probabilities
+ * @partition: partition probabilities
+ * @mv: motion vector probabilities
+ *
+ * Structure containing most VP9 probabilities. See the VP9 specification
+ * for more details.
+ */
+struct v4l2_vp9_probabilities {
+       __u8 tx8[2][1];
+       __u8 tx16[2][2];
+       __u8 tx32[2][3];
+       __u8 coef[4][2][2][6][6][3];
+       __u8 skip[3];
+       __u8 inter_mode[7][3];
+       __u8 interp_filter[4][2];
+       __u8 is_inter[4];
+       __u8 comp_mode[5];
+       __u8 single_ref[5][2];
+       __u8 comp_ref[5];
+       __u8 y_mode[4][9];
+       __u8 uv_mode[10][9];
+       __u8 partition[16][3];
+
+       struct v4l2_vp9_mv_probabilities mv;
+};
+
+/**
+ * enum v4l2_vp9_reset_frame_context - Valid values for
+ *                     &v4l2_ctrl_vp9_frame_decode_params-
Quoted 1 line

reset_frame_context

+ *
+ * @V4L2_VP9_RESET_FRAME_CTX_NONE: don't reset any frame context
+ * @V4L2_VP9_RESET_FRAME_CTX_SPEC: reset the frame context pointed by
+ *                     &v4l2_ctrl_vp9_frame_decode_params.frame_context_idx
+ * @V4L2_VP9_RESET_FRAME_CTX_ALL: reset all frame contexts
+ *
+ * See section '7.2 Uncompressed header semantics' of the VP9 specification
+ * for more details.
+ */

include/media/vp9-ctrls.h new file +17 -0
+enum v4l2_vp9_reset_frame_context {
+       V4L2_VP9_RESET_FRAME_CTX_NONE,
+       V4L2_VP9_RESET_FRAME_CTX_SPEC,
+       V4L2_VP9_RESET_FRAME_CTX_ALL,
+};
+
+/**
+ * enum v4l2_vp9_interpolation_filter - VP9 interpolation filter types
+ *
+ * @V4L2_VP9_INTERP_FILTER_8TAP: height tap filter
+ * @V4L2_VP9_INTERP_FILTER_8TAP_SMOOTH: height tap smooth filter
+ * @V4L2_VP9_INTERP_FILTER_8TAP_SHARP: height tap sharp filter
+ * @V4L2_VP9_INTERP_FILTER_BILINEAR: bilinear filter
+ * @V4L2_VP9_INTERP_FILTER_SWITCHABLE: filter selection is signaled at the
+ *                                    block level
+ *
+ * See section '7.2.7 Interpolation filter semantics' of the VP9

specification
+ * for more details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+enum v4l2_vp9_interpolation_filter {
+       V4L2_VP9_INTERP_FILTER_8TAP,
+       V4L2_VP9_INTERP_FILTER_8TAP_SMOOTH,
+       V4L2_VP9_INTERP_FILTER_8TAP_SHARP,
+       V4L2_VP9_INTERP_FILTER_BILINEAR,
+       V4L2_VP9_INTERP_FILTER_SWITCHABLE,
+};
+
+/**
+ * enum v4l2_vp9_reference_mode - VP9 reference modes
+ *
+ * @V4L2_VP9_REF_MODE_SINGLE: indicates that all the inter blocks use only

a
+ *                           single reference frame to generate motion
+ *                           compensated prediction
+ * @V4L2_VP9_REF_MODE_COMPOUND: requires all the inter blocks to use compound
+ *                             mode. Single reference frame prediction is not
+ *                             allowed
+ * @V4L2_VP9_REF_MODE_SELECT: allows each individual inter block to select
+ *                           between single and compound prediction modes
+ *
+ * See section '7.3.6 Frame reference mode semantics' of the VP9 specification
+ * for more details.
+ */

include/media/vp9-ctrls.h new file +14 -0
+enum v4l2_vp9_reference_mode {
+       V4L2_VP9_REF_MODE_SINGLE,
+       V4L2_VP9_REF_MODE_COMPOUND,
+       V4L2_VP9_REF_MODE_SELECT,
+};
+
+/**
+ * enum v4l2_vp9_tx_mode - VP9 TX modes
+ *
+ * @V4L2_VP9_TX_MODE_ONLY_4X4: transform size is 4x4
+ * @V4L2_VP9_TX_MODE_ALLOW_8X8: transform size can be up to 8x8
+ * @V4L2_VP9_TX_MODE_ALLOW_16X16: transform size can be up to 16x16
+ * @V4L2_VP9_TX_MODE_ALLOW_32X32: transform size can be up to 32x32
+ * @V4L2_VP9_TX_MODE_SELECT: bitstream contains transform size for each

block
+ *
+ * See section '7.3.1 Tx mode semantics' of the VP9 specification for more
+ * details.
+ */

include/media/vp9-ctrls.h new file +17 -0
+enum v4l2_vp9_tx_mode {
+       V4L2_VP9_TX_MODE_ONLY_4X4,
+       V4L2_VP9_TX_MODE_ALLOW_8X8,
+       V4L2_VP9_TX_MODE_ALLOW_16X16,
+       V4L2_VP9_TX_MODE_ALLOW_32X32,
+       V4L2_VP9_TX_MODE_SELECT,
+};
+
+/**
+ * enum v4l2_vp9_ref_id - VP9 Reference frame IDs
+ *
+ * @V4L2_REF_ID_LAST: last reference frame
+ * @V4L2_REF_ID_GOLDEN: golden reference frame
+ * @V4L2_REF_ID_ALTREF: alternative reference frame
+ * @V4L2_REF_ID_CNT: number of reference frames
+ *
+ * See section '7.4.12 Ref frames semantics' of the VP9 specification for

more
+ * details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+enum v4l2_vp9_ref_id {
+       V4L2_REF_ID_LAST,
+       V4L2_REF_ID_GOLDEN,
+       V4L2_REF_ID_ALTREF,
+       V4L2_REF_ID_CNT,
+};
+
+/**
+ * enum v4l2_vp9_frame_flags - VP9 frame flags
+ * @V4L2_VP9_FRAME_FLAG_KEY_FRAME: the frame is a key frame
+ * @V4L2_VP9_FRAME_FLAG_SHOW_FRAME: the frame should be displayed
+ * @V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT: the decoding should be error

resilient
+ * @V4L2_VP9_FRAME_FLAG_INTRA_ONLY: the frame does not reference other frames
+ * @V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV: the frame might can high precision
+ *                                         motion vectors
+ * @V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX: frame context should be updated
+ *                                        after decoding
+ * @V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE: parallel decoding is used
+ * @V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING: vertical subsampling is enabled
+ * @V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING: horizontal subsampling is enabled
+ * @V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING: full UV range is used
+ *
+ * Check the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +23 -0
+enum v4l2_vp9_frame_flags {
+       V4L2_VP9_FRAME_FLAG_KEY_FRAME = 1 << 0,
+       V4L2_VP9_FRAME_FLAG_SHOW_FRAME = 1 << 1,
+       V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT = 1 << 2,
+       V4L2_VP9_FRAME_FLAG_INTRA_ONLY = 1 << 3,
+       V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV = 1 << 4,
+       V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX = 1 << 5,
+       V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE = 1 << 6,
+       V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING = 1 << 7,
+       V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING = 1 << 8,
+       V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING = 1 << 9,
+};
+
+#define V4L2_VP9_PROFILE_MAX           3
+
+/**
+ * struct v4l2_ctrl_vp9_frame_decode_params - VP9 frame decoding control
+ *
+ * @flags: combination of V4L2_VP9_FRAME_FLAG_* flags
+ * @compressed_header_size: compressed header size in bytes
+ * @uncompressed_header_size: uncompressed header size in bytes
+ * @profile: VP9 profile. Can be 0, 1, 2 or 3
+ * @reset_frame_context: specifies whether the frame context should be

reset
+ *                      to default values. See &v4l2_vp9_reset_frame_context
+ *                      for more details
+ * @frame_context_idx: frame context that should be used/updated
+ * @bit_depth: bits per components. Can be 8, 10 or 12. Note that not all
+ *            profiles support 10 and/or 12 bits depths
+ * @interpolation_filter: specifies the filter selection used for performing
+ *                       inter prediction. See &v4l2_vp9_interpolation_filter
+ *                       for more details
+ * @tile_cols_log2: specifies the base 2 logarithm of the width of each tile
+ *                 (where the width is measured in units of 8x8 blocks).
+ *                 Shall be less than or equal to 6
+ * @tile_rows_log2: specifies the base 2 logarithm of the height of each tile
+ *                 (where the height is measured in units of 8x8 blocks)
+ * @tx_mode: specifies the TX mode. See &v4l2_vp9_tx_mode for more details
+ * @reference_mode: specifies the type of inter prediction to be used. See
+ *                 &v4l2_vp9_reference_mode for more details
+ * @padding: needed to make this struct 64 bit aligned. Shall be filled with
+ *          zeros
+ * @frame_width_minus_1: add 1 to it and you'll get the frame width expressed
+ *                      in pixels
+ * @frame_height_minus_1: add 1 to it and you'll get the frame height expressed
+ *                       in pixels
+ * @frame_width_minus_1: add 1 to it and you'll get the expected render width
+ *                      expressed in pixels. This is not used during the
+ *                      decoding process but might be used by HW scalers to
+ *                      prepare a frame that's ready for scanout
+ * @frame_height_minus_1: add 1 to it and you'll get the expected render height
+ *                      expressed in pixels. This is not used during the
+ *                      decoding process but might be used by HW scalers to
+ *                      prepare a frame that's ready for scanout
+ * @refs: array of reference frames. See &v4l2_vp9_ref_id for more details
+ * @lf: loop filter parameters. See &v4l2_vp9_loop_filter for more details
+ * @quant: quantization parameters. See &v4l2_vp9_quantization for more details
+ * @seg: segmentation parameters. See &v4l2_vp9_segmentation for more details
+ * @probs: probabilities. See &v4l2_vp9_probabilities for more details
+ */
+struct v4l2_ctrl_vp9_frame_decode_params {
+       __u32 flags;
+       __u16 compressed_header_size;
+       __u16 uncompressed_header_size;
+       __u8 profile;
+       __u8 reset_frame_context;

Quoted 8 lines

+       __u8 frame_context_idx;
+       __u8 bit_depth;
+       __u8 interpolation_filter;
+       __u8 tile_cols_log2;
+       __u8 tile_rows_log2;
+       __u8 tx_mode;
+       __u8 reference_mode;
+       __u8 padding[6];

If my math is correct this should be 7.

Quoted 5 lines

+       __u16 frame_width_minus_1;
+       __u16 frame_height_minus_1;
+       __u16 render_width_minus_1;
+       __u16 render_height_minus_1;
+       __u64 refs[V4L2_REF_ID_CNT];

Why are we using u64s to store these [0..7] indices? Or are we storing more than that?

Quoted 5 lines

+       struct v4l2_vp9_loop_filter lf;
+       struct v4l2_vp9_quantization quant;
+       struct v4l2_vp9_segmentation seg;
+       struct v4l2_vp9_probabilities probs;
+};

Quoted 3 lines

I cannot find a trace of ref_frame_sign_biases, which is also part of the uncompressed header, in this structure. Is it missing, or is this information provided in a different way?

This seems missing indeed. Interestingly the Rockchip reference software does not seem to pass this information to the hardware. On the other end, the VC8000D software does. It is also part of DXVA, VA and NVDEC APIs.

Quoted 23 lines
Quoted 23 lines
include/media/vp9-ctrls.h new file +9 -0
+
+#define V4L2_VP9_NUM_FRAME_CTX 4
+
+/**
+ * struct v4l2_ctrl_vp9_frame_ctx - VP9 frame context control
+ *
+ * @probs: VP9 probabilities
+ *
+ * This control is accessed in both direction. The user should initialize

the
+ * 4 contexts with default values just after starting the stream. Then before
+ * decoding a frame it should query the current frame context (the one passed
+ * through &v4l2_ctrl_vp9_frame_decode_params.frame_context_idx) to initialize
+ * &v4l2_ctrl_vp9_frame_decode_params.probs. The probs are then adjusted based
+ * on the bitstream info and passed to the kernel. The codec should update
+ * the frame context after the frame has been decoded, so that next time
+ * userspace query this context it contains the updated probabilities.
+ */

Quoted 8 lines
Quoted 8 lines
include/media/vp9-ctrls.h new file +5 -1
+struct v4l2_ctrl_vp9_frame_ctx {
+       struct v4l2_vp9_probabilities probs;
+};
+
+#endif /* _VP9_CTRLS_H_ */
--

2.26.0.rc2

_______________________________________________
Linux-rockchip mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip

#9 lore raw

Re: [PATCH v4 2/3] media: uapi: Add VP9 stateless decoder controls

Hi Nicolas,

Quoted 94 lines · Nicolas Dufresne
Quoted 71 lines · Alexandre Courbot

Hi Ezequiel, sorry for the late review!

Quoted 67 lines · Ezequiel Garcia

From: Boris Brezillon <[email protected]>

Add the VP9 stateless decoder controls plus the documentation that goes with it.

---
 .../userspace-api/media/v4l/biblio.rst        |  10 +
 .../media/v4l/ext-ctrls-codec.rst             | 550 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ctrls.c          | 239 ++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   1 +
 include/media/v4l2-ctrls.h                    |   1 +
 include/media/vp9-ctrls.h                     | 485 +++++++++++++++
 6 files changed, 1286 insertions(+)
 create mode 100644 include/media/vp9-ctrls.h
Documentation/userspace-api/media/v4l/biblio.rst +10 -0
index 3c9634173e82..e09102e572fd 100644
@@ -414,3 +414,13 @@ VP8
 :title:     RFC 6386: "VP8 Data Format and Decoding Guide"
 
 :author:    J. Bankoski et al.
+
+.. _vp9:
+
+VP9
+===
+
+
+:title:     VP9 Bitstream & Decoding Process Specification
+
+:author:    Adrian Grange (Google), Peter de Rivaz (Argon Design), Jonathan

Hunt (Argon Design)

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +16 -0
index d0d506a444b1..5c5f7dd868da 100644
@@ -2668,6 +2668,556 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
       - ``padding[3]``
       - Applications and drivers must set this to zero.
 
+.. _v4l2-mpeg-vp9:
+
+``V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(0..3) (struct)``
+    Stores VP9 probabilities attached to a specific frame context. The VP9
+    specification allows using a maximum of 4 contexts. Each frame being
+    decoded refers to one of those context. See section '7.1.2 Refresh
+    probs semantics' section of :ref:`vp9` for more details about these
+    contexts.
+
+    This control is bi-directional:
+
+    * all 4 contexts must be initialized by userspace just after the
+      stream is started and before the first decoding request is submitted.
+    * the referenced context might be read by the kernel when a decoding
+      request is submitted, and will be updated after the decoder is done
+      decoding the frame if the `V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX`

flag

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +3 -0
+      is set.
+    * contexts will be read back by user space before each decoding request
+      to retrieve the updated probabilities.
Quoted 8 lines

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 2691

Quoted 2 lines
Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +2 -0
+    * userspace will re-initialize the context to their default values when
+      a reset context is required.

Just to make sure I understand this part correctly, it means that if frame A and B use the same context, and frame A has V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX set, then user-space must wait for frame A to get dequeued and read back this control from the completed request before it can submit frame B?

We are preparing a new version, we believe it was an API mistake to try and share the probability update between kernel and userspace. It's actually worst, you really need to push/pull in a lock step, regardless of the frame context id.

As you may know, the probabilities are used to parse the compressed part of the stream (also know as entropy decoding). They are probability of a bitstream symbol of being 0 or 1. On RK3399, the hardware does not parse the compressed header. The compressed headers contains the bitstream coded updates to the probabilities. The probabilities are also updated base on the count of specific symbols found in the bitstream during decoding, this is currently done in the driver (counts are returned, and driver updates the probabilities from that).

Quoted 6 lines

What we found is that the symbols in the compress headers are coded with fixed probabilities, meaning we don't need the probabilities to parse the compressed headers. Our proposal will be to keep doing the compressed header parsing in userspace and only store the update value (as found in inv_map_table) to the kernel. The driver will be responsible of applying this to the current probabilities along with doing the probability saving process.

So if I am following:

1) The compressed headers can be decompressed using just the entropy information contained in the previous compressed headers,
2) The frame data also requires the updated probabilities from the decoding of the previous frames on top of 1)

Meaning that user-space can decompress the headers without any feedback from the kernel. Am I understanding correctly?

Quoted 4 lines

With this change, we should be able to only push data to the decoder without having to read this control. We believe this will allow better use of the VB2 queue and enhance the decoding performance.

That sounds pretty good to me.

Quoted 7 lines

Please note that this hardware design looks like a mistake. It seems that with widevine, the compressed header is encrypted. As in some implementation the decrypted bitstream is no visible to the CPU, it will not be possible to perform the needed parsing. An offload to a TEE would be needed. We know that Rockchip have fixed this issue in new decoder, notably on RK3368. The probability processing is done in HW.

Mmm a solution would be not to encrypt the compressed header, but I guess that's not how things work, right?

Worst case the TEE could indeed copy the decrypted and decompressed somewhere for the CPU to read, but that would complicate things a tiny bit.

Quoted 539 lines
Quoted 524 lines
Quoted 524 lines
Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +104 -0
+
+    .. note::
+
+       This compound control is not yet part of the public kernel API and
+       it is expected to change.
+
+.. c:type:: v4l2_ctrl_vp9_frame_ctx
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{5.8cm}|p{4.8cm}|p{6.6cm}|
+
+.. flat-table:: struct v4l2_ctrl_vp9_frame_ctx
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - struct :c:type:`v4l2_vp9_probabilities`
+      - ``probs``
+      - Structure with VP9 probabilities attached to the context.
+
+.. c:type:: v4l2_vp9_probabilities
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_probabilities
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``tx8[2][1]``
+      - TX 8x8 probabilities.
+    * - __u8
+      - ``tx16[2][2]``
+      - TX 16x16 probabilities.
+    * - __u8
+      - ``tx32[2][3]``
+      - TX 32x32 probabilities.
+    * - __u8
+      - ``coef[4][2][2][6][6][3]``
+      - Coefficient probabilities.
+    * - __u8
+      - ``skip[3]``
+      - Skip probabilities.
+    * - __u8
+      - ``inter_mode[7][3]``
+      - Inter prediction mode probabilities.
+    * - __u8
+      - ``interp_filter[4][2]``
+      - Interpolation filter probabilities.
+    * - __u8
+      - ``is_inter[4]``
+      - Is inter-block probabilities.
+    * - __u8
+      - ``comp_mode[5]``
+      - Compound prediction mode probabilities.
+    * - __u8
+      - ``single_ref[5][2]``
+      - Single reference probabilities.
+    * - __u8
+      - ``comp_mode[5]``
+      - Compound reference probabilities.
+    * - __u8
+      - ``y_mode[4][9]``
+      - Y prediction mode probabilities.
+    * - __u8
+      - ``uv_mode[10][9]``
+      - UV prediction mode probabilities.
+    * - __u8
+      - ``partition[16][3]``
+      - Partition probabilities.
+    * - __u8
+      - ``mv.joint[3]``
+      - Motion vector joint probabilities.
+    * - __u8
+      - ``mv.sign[2]``
+      - Motion vector sign probabilities.
+    * - __u8
+      - ``mv.class[2][10]``
+      - Motion vector class probabilities.
+    * - __u8
+      - ``mv.class0_bit[2]``
+      - Motion vector class0 bit probabilities.
+    * - __u8
+      - ``mv.bits[2][10]``
+      - Motion vector bits probabilities.
+    * - __u8
+      - ``mv.class0_fr[2][2][3]``
+      - Motion vector class0 fractional bit probabilities.
+    * - __u8
+      - ``mv.fr[2][3]``
+      - Motion vector fractional bit probabilities.
+    * - __u8
+      - ``mv.class0_hp[2]``
+      - Motion vector class0 high precision fractional bit probabilities.
+    * - __u8
+      - ``mv.hp[2]``
+      - Motion vector high precision fractional bit probabilities.
+
+``V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS (struct)``
+    Specifies the frame parameters for the associated VP9 frame decode

request.
+ This includes the necessary parameters for configuring a stateless
hardware
+ decoding pipeline for VP9. The bitstream parameters are defined
according

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +37 -0
+    to :ref:`vp9`.
+
+    .. note::
+
+       This compound control is not yet part of the public kernel API and
+       it is expected to change.
+
+.. c:type:: v4l2_ctrl_vp9_frame_decode_params
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_ctrl_vp9_frame_decode_params
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``flags``
+      - Combination of V4L2_VP9_FRAME_FLAG_* flags. See
+        :c:type:`v4l2_vp9_frame_flags`.
+    * - __u16
+      - ``compressed_header_size``
+      - Compressed header size in bytes.
+    * - __u16
+      - ``uncompressed_header_size``
+      - Uncompressed header size in bytes.
+    * - __u8
+      - ``profile``
+      - VP9 profile. Can be 0, 1, 2 or 3.
+    * - __u8
+      - ``reset_frame_context``
+      - Frame context that should be used/updated when decoding the frame.
+    * - __u8
+      - ``bit_depth``
+      - Component depth in bits. Must be 8 for profile 0 and 1. Must 10 or

12

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +4 -0
+        for profile 2 and 3.
+    * - __u8
+      - ``interpolation_filter``
+      - Specifies the filter selection used for performing inter

prediction. See

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +5 -0
+        :c:type:`v4l2_vp9_interpolation_filter`.
+    * - __u8
+      - ``tile_cols_log2``
+      - Specifies the base 2 logarithm of the width of each tile (where the
+        width is measured in units of 8x8 blocks). Shall be less than or

equal

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +4 -0
+        to 6.
+    * - __u8
+      - ``tile_rows_log2``
+      - Specifies the base 2 logarithm of the height of each tile (where

the

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +11 -0
+        height is measured in units of 8x8 blocks)
+    * - __u8
+      - ``tx_mode``
+      - Specifies the TX mode. See :c:type:`v4l2_vp9_tx_mode`.
+    * - __u8
+      - ``reference_mode``
+      - Specifies the type of inter prediction to be used. See
+        :c:type:`v4l2_vp9_reference_mode`.
+    * - __u8
+      - ``padding``
+      - Needed to make this struct 64 bit aligned. Shall be filled with

zeroes.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +9 -0
+    * - __u16
+      - ``frame_width_minus_1``
+      - Add 1 to get the frame width expressed in pixels.
+    * - __u16
+      - ``frame_height_minus_1``
+      - Add 1 to to get the frame height expressed in pixels.
+    * - __u16
+      - ``frame_width_minus_1``
+      - Add 1 to to get the expected render width expressed in pixels. This

is
+ not used during the decoding process but might be used by HW
scalers to

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +4 -0
+        prepare a frame that's ready for scanout.
+    * - __u16
+      - frame_height_minus_1
+      - Add 1 to get the expected render height expressed in pixels. This

is
+ not used during the decoding process but might be used by HW
scalers to

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +104 -0
+        prepare a frame that's ready for scanout.
+    * - __u64
+      - ``refs[3]``
+      - Array of reference frame timestamps.
+    * - struct :c:type:`v4l2_vp9_loop_filter`
+      - ``lf``
+      - Loop filter parameters. See struct :c:type:`v4l2_vp9_loop_filter`.
+    * - struct :c:type:`v4l2_vp9_quantization`
+      - ``quant``
+      - Quantization parameters. See :c:type:`v4l2_vp9_quantization`.
+    * - struct :c:type:`v4l2_vp9_segmentation`
+      - ``seg``
+      - Segmentation parameters. See :c:type:`v4l2_vp9_segmentation`.
+    * - struct :c:type:`v4l2_vp9_probabilities`
+      - ``probs``
+      - Probabilities. See :c:type:`v4l2_vp9_probabilities`.
+
+.. c:type:: v4l2_vp9_frame_flags
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_frame_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_FRAME_FLAG_KEY_FRAME``
+      - The frame is a key frame.
+    * - ``V4L2_VP9_FRAME_FLAG_SHOW_FRAME``
+      - The frame should be displayed.
+    * - ``V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT``
+      - The decoding should be error resilient.
+    * - ``V4L2_VP9_FRAME_FLAG_INTRA_ONLY``
+      - The frame does not reference other frames.
+    * - ``V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV``
+      - the frame might can high precision motion vectors.
+    * - ``V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX``
+      - Frame context should be updated after decoding.
+    * - ``V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE``
+      - Parallel decoding is used.
+    * - ``V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING``
+      - Vertical subsampling is enabled.
+    * - ``V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING``
+      - Horizontal subsampling is enabled.
+    * - ``V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING``
+      - The full UV range is used.
+
+.. c:type:: v4l2_vp9_ref_id
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_ref_id
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_REF_ID_LAST``
+      - Last reference frame.
+    * - ``V4L2_REF_ID_GOLDEN``
+      - Golden reference frame.
+    * - ``V4L2_REF_ID_ALTREF``
+      - Alternative reference frame.
+    * - ``V4L2_REF_ID_CNT``
+      - Number of reference frames.
+
+.. c:type:: v4l2_vp9_tx_mode
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_tx_mode
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_TX_MODE_ONLY_4X4``
+      - Transform size is 4x4.
+    * - ``V4L2_VP9_TX_MODE_ALLOW_8X8``
+      - Transform size can be up to 8x8.
+    * - ``V4L2_VP9_TX_MODE_ALLOW_16X16``
+      - Transform size can be up to 16x16.
+    * - ``V4L2_VP9_TX_MODE_ALLOW_32X32``
+      - transform size can be up to 32x32.
+    * - ``V4L2_VP9_TX_MODE_SELECT``
+      - Bitstream contains transform size for each block.
+
+.. c:type:: v4l2_vp9_reference_mode
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_reference_mode
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_REF_MODE_SINGLE``
+      - Indicates that all the inter blocks use only a single reference

frame

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +3 -0
+        to generate motion compensated prediction.
+    * - ``V4L2_VP9_REF_MODE_COMPOUND``
+      - Requires all the inter blocks to use compound mode. Single

reference

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +96 -0
+        frame prediction is not allowed.
+    * - ``V4L2_VP9_REF_MODE_SELECT``
+      - Allows each individual inter block to select between single and
+        compound prediction modes.
+
+.. c:type:: v4l2_vp9_interpolation_filter
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_interpolation_filter
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_INTERP_FILTER_8TAP``
+      - Height tap filter.
+    * - ``V4L2_VP9_INTERP_FILTER_8TAP_SMOOTH``
+      - Height tap smooth filter.
+    * - ``V4L2_VP9_INTERP_FILTER_8TAP_SHARP``
+      - Height tap sharp filter.
+    * - ``V4L2_VP9_INTERP_FILTER_BILINEAR``
+      - Bilinear filter.
+    * - ``V4L2_VP9_INTERP_FILTER_SWITCHABLE``
+      - Filter selection is signaled at the block level.
+
+.. c:type:: v4l2_vp9_reset_frame_context
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_reset_frame_context
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_RESET_FRAME_CTX_NONE``
+      - Do not reset any frame context.
+    * - ``V4L2_VP9_RESET_FRAME_CTX_SPEC``
+      - Reset the frame context pointed by
+        :c:type:`v4l2_ctrl_vp9_frame_decode_params`.frame_context_idx.
+    * - ``V4L2_VP9_RESET_FRAME_CTX_ALL``
+      - Reset all frame contexts.
+
+.. c:type:: v4l2_vp9_intra_prediction_mode
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_intra_prediction_mode
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_INTRA_PRED_DC``
+      - DC intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_V``
+      - Vertical intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_H``
+      - Horizontal intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D45``
+      - D45 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D135``
+      - D135 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D117``
+      - D117 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D153``
+      - D153 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D207``
+      - D207 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D63``
+      - D63 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_TM``
+      - True motion intra prediction.
+
+.. c:type:: v4l2_vp9_segmentation
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_segmentation
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``flags``
+      - Combination of V4L2_VP9_SEGMENTATION_FLAG_* flags. See
+        :c:type:`v4l2_vp9_segmentation_flags`.
+    * - __u8
+      - ``tree_probs[7]``
+      - Specifies the probability values to be used when decoding a

Segment-ID.
+ See '5.15. Segmentation map' section of :ref:vp9 for more
details.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +14 -0
+    * - __u8
+      - ``pred_prob[3]``
+      - Specifies the probability values to be used when decoding a
+        Predicted-Segment-ID. See '6.4.14. Get segment id syntax'
+        section of :ref:`vp9` for more details.
+    * - __u8
+      - ``padding[5]``
+      - Used to align this struct on 64 bit. Shall be filled with zeroes.
+    * - __u8
+      - ``feature_enabled[8]``
+      - Bitmask defining which features are enabled in each segment.
+    * - __u8
+      - ``feature_data[8][4]``
+      - Data attached to each feature. Data entry is only valid if the

feature

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +48 -0
+        is enabled.
+
+.. c:type:: v4l2_vp9_segment_feature
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_segment_feature
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_SEGMENT_FEATURE_QP_DELTA``
+      - QP delta segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_LF``
+      - Loop filter segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_REF_FRAME``
+      - Reference frame segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_SKIP``
+      - Skip segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_CNT``
+      - Number of segment features.
+
+.. c:type:: v4l2_vp9_segmentation_flags
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_segmentation_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_ENABLED``
+      - Indicates that this frame makes use of the segmentation tool.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP``
+      - Indicates that the segmentation map should be updated during the
+        decoding of this frame.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE``
+      - Indicates that the updates to the segmentation map are coded
+        relative to the existing segmentation map.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA``
+      - Indicates that new parameters are about to be specified for each
+        segment.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE``
+      - Indicates that the segmentation parameters represent the actual

values

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +28 -0
+        to be used.
+
+.. c:type:: v4l2_vp9_quantization
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_quantization
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``base_q_idx``
+      - Indicates the base frame qindex.
+    * - __s8
+      - ``delta_q_y_dc``
+      - Indicates the Y DC quantizer relative to base_q_idx.
+    * - __s8
+      - ``delta_q_uv_dc``
+      - Indicates the UV DC quantizer relative to base_q_idx.
+    * - __s8
+      - ``delta_q_uv_ac``
+      - Indicates the UV AC quantizer relative to base_q_idx.
+    * - __u8
+      - ``padding[4]``
+      - Padding bytes used to align this struct on 64 bit. Must be set to

0.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +25 -0
+
+.. c:type:: v4l2_vp9_loop_filter
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_loop_filter
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``flags``
+      - Combination of V4L2_VP9_LOOP_FILTER_FLAG_* flags.
+        See :c:type:`v4l2_vp9_loop_filter_flags`.
+    * - __u8
+      - ``level``
+      - Indicates the loop filter strength.
+    * - __u8
+      - ``sharpness``
+      - Indicates the sharpness level.
+    * - __s8
+      - ``ref_deltas[4]``
+      - Contains the adjustment needed for the filter level based on the

chosen
+ reference frame.

Quoted 8 lines

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 3200

Quoted 8 lines
Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +3 -0
+    * - __s8
+      - ``mode_deltas[2]``
+      - Contains the adjustment needed for the filter level based on the

chosen

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +4 -0
+        mode
+    * - __u8
+      - ``level_lookup[8][4][2]``
+      - Level lookup table.

We think it might be a poor choice to ask for computed values. According to "8.8.1 Loop filter frame init process" this is computed from loop_filter_level, loop_filter_ref_deltas[], V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE, feature_data[], ref_deltas[] and mode_deltas[]. We believe it is likely a waste to pass this information assuming some HW could do that process.

Quoted 2 lines

This is to be discussed. None of the other statelss APIs seems to require this value to be computed by userspace.

Not particularly opinionated, but it would be a waste if the driver needs to reverse that computation to pass it to the hardware. If the lookup table is not difficult to compute, we can maybe do this using a helper function for drivers that need it?

Quoted 871 lines
Quoted 871 lines
Quoted 421 lines
Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +15 -0
+
+
+.. c:type:: v4l2_vp9_loop_filter_flags
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_loop_filter_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED``
+      - When set, the filter level depends on the mode and reference frame

used

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +5 -0
+        to predict a block.
+    * - ``V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE``
+      - When set, the bitstream contains additional syntax elements that
+        specify which mode and reference frame deltas are to be updated.
+
 .. raw:: latex
 
     \normalsize
drivers/media/v4l2- renamed from drivers/media/v4l2-core/v4l2-ctrls.c +0 -0

core/v4l2-ctrls.c
index 1c617b42a944..115beadaf02d 100644

drivers/media/v4l2-core/v4l2-ctrls.c +1 -0
@@ -930,6 +930,11 @@ const char *v4l2_ctrl_get_name(u32 id)
        case V4L2_CID_MPEG_VIDEO_VP8_PROFILE:                   return "VP8
 Profile";
        case V4L2_CID_MPEG_VIDEO_VP9_PROFILE:                   return "VP9
 Profile";
        case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:              return "VP8
 Frame Header";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS:       return "VP9

Frame Decode Parameters";
+ case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(0): return "VP9
Frame Context 0";
+ case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(1): return "VP9
Frame Context 1";
+ case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(2): return "VP9
Frame Context 2";
+ case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(3): return "VP9
Frame Context 3";

/* HEVC controls */
case V4L2_CID_MPEG_VIDEO_HEVC_I_FRAME_QP: return "HEVC
I-Frame QP Value";

drivers/media/v4l2-core/v4l2-ctrls.c +9 -0
@@ -1403,6 +1408,15 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum
 v4l2_ctrl_type *type,
        case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
                *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
                break;
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS:
+               *type = V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS;
+               break;
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(0):
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(1):
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(2):
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(3):
+               *type = V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT;
+               break;
        case V4L2_CID_MPEG_VIDEO_HEVC_SPS:
                *type = V4L2_CTRL_TYPE_HEVC_SPS;
                break;
drivers/media/v4l2-core/v4l2-ctrls.c +143 -0
@@ -1703,6 +1717,219 @@ static void std_log(const struct v4l2_ctrl *ctrl)
        0;                                                      \
 })
 
+static int
+validate_vp9_lf_params(struct v4l2_vp9_loop_filter *lf)
+{
+       unsigned int i, j, k;
+
+       if (lf->flags &
+           ~(V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED |
+             V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED implies
+        * V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE.
+        */
+       if (lf->flags & V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE &&
+           !(lf->flags & V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED))
+               return -EINVAL;
+
+       /* That all values are in the accepted range. */
+       if (lf->level > GENMASK(5, 0))
+               return -EINVAL;
+
+       if (lf->sharpness > GENMASK(2, 0))
+               return -EINVAL;
+
+       for (i = 0; i < ARRAY_SIZE(lf->ref_deltas); i++) {
+               if (lf->ref_deltas[i] < -63 || lf->ref_deltas[i] > 63)
+                       return -EINVAL;
+       }
+
+       for (i = 0; i < ARRAY_SIZE(lf->mode_deltas); i++) {
+               if (lf->mode_deltas[i] < -63 || lf->mode_deltas[i] > 63)
+                       return -EINVAL;
+       }
+
+       for (i = 0; i < ARRAY_SIZE(lf->level_lookup); i++) {
+               for (j = 0; j < ARRAY_SIZE(lf->level_lookup[0]); j++) {
+                       for (k = 0; k < ARRAY_SIZE(lf->level_lookup[0][0]);
 k++) {
+                               if (lf->level_lookup[i][j][k] > 63)
+                                       return -EINVAL;
+                       }
+               }
+       }
+
+       return 0;
+}
+
+static int
+validate_vp9_quant_params(struct v4l2_vp9_quantization *quant)
+{
+       if (quant->delta_q_y_dc < -15 || quant->delta_q_y_dc > 15 ||
+           quant->delta_q_uv_dc < -15 || quant->delta_q_uv_dc > 15 ||
+           quant->delta_q_uv_ac < -15 || quant->delta_q_uv_ac > 15)
+               return -EINVAL;
+
+       memset(quant->padding, 0, sizeof(quant->padding));
+       return 0;
+}
+
+static int
+validate_vp9_seg_params(struct v4l2_vp9_segmentation *seg)
+{
+       unsigned int i, j;
+
+       if (seg->flags &
+           ~(V4L2_VP9_SEGMENTATION_FLAG_ENABLED |
+             V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP |
+             V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE |
+             V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA |
+             V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP and
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA imply
+        * V4L2_VP9_SEGMENTATION_FLAG_ENABLED.
+        */
+       if ((seg->flags &
+            (V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP |
+             V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA)) &&
+           !(seg->flags & V4L2_VP9_SEGMENTATION_FLAG_ENABLED))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE implies
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP.
+        */
+       if (seg->flags & V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE &&
+           !(seg->flags & V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE implies
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA.
+        */
+       if (seg->flags & V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE &&
+           !(seg->flags & V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA))
+               return -EINVAL;
+
+       for (i = 0; i < ARRAY_SIZE(seg->feature_enabled); i++) {
+               if (seg->feature_enabled[i] &
+                   ~(V4L2_VP9_SEGMENT_FEATURE_QP_DELTA |
+                     V4L2_VP9_SEGMENT_FEATURE_LF |
+                     V4L2_VP9_SEGMENT_FEATURE_REF_FRAME |
+                     V4L2_VP9_SEGMENT_FEATURE_SKIP))
+                       return -EINVAL;
+       }
+
+       for (i = 0; i < ARRAY_SIZE(seg->feature_data); i++) {
+               const int range[] = {255, 63, 3, 0};
+
+               for (j = 0; j < ARRAY_SIZE(seg->feature_data[j]); j++) {
+                       if (seg->feature_data[i][j] < -range[j] ||
+                           seg->feature_data[i][j] > range[j])
+                               return -EINVAL;
+               }
+       }
+
+       memset(seg->padding, 0, sizeof(seg->padding));
+       return 0;
+}
+
+static int
+validate_vp9_frame_decode_params(struct v4l2_ctrl_vp9_frame_decode_params
 *dec_params)
+{
+       int ret;
+
+       /* Make sure we're not passed invalid flags. */
+       if (dec_params->flags &
+           ~(V4L2_VP9_FRAME_FLAG_KEY_FRAME |
+             V4L2_VP9_FRAME_FLAG_SHOW_FRAME |
+             V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT |
+             V4L2_VP9_FRAME_FLAG_INTRA_ONLY |
+             V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV |
+             V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX |
+             V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE |
+             V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING |
+             V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING |
+             V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING))
+               return -EINVAL;
+
+       /*
+        * The refresh context and error resilient flags are mutually

exclusive.

drivers/media/v4l2-core/v4l2-ctrls.c +19 -0
+        * Same goes for parallel decoding and error resilient modes.
+        */
+       if (dec_params->flags & V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT &&
+           dec_params->flags &
+           (V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX |
+            V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE))
+               return -EINVAL;
+
+       if (dec_params->profile > V4L2_VP9_PROFILE_MAX)
+               return -EINVAL;
+
+       if (dec_params->reset_frame_context > V4L2_VP9_RESET_FRAME_CTX_ALL)
+               return -EINVAL;
+
+       if (dec_params->frame_context_idx >= V4L2_VP9_NUM_FRAME_CTX)
+               return -EINVAL;
+
+       /*
+        * Profiles 0 and 1 only support 8-bit depth, profiles 2 and 3 only

10

drivers/media/v4l2-core/v4l2-ctrls.c +20 -0
+        * and 12 bit depths.
+        */
+       if ((dec_params->profile < 2 && dec_params->bit_depth != 8) ||
+           (dec_params->profile >= 2 &&
+            (dec_params->bit_depth != 10 && dec_params->bit_depth != 12)))
+               return -EINVAL;
+
+       /* Profile 0 and 2 only accept YUV 4:2:0. */
+       if ((dec_params->profile == 0 || dec_params->profile == 2) &&
+           (!(dec_params->flags & V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING) ||
+            !(dec_params->flags & V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING)))
+               return -EINVAL;
+
+       /* Profile 1 and 3 only accept YUV 4:2:2, 4:4:0 and 4:4:4. */
+       if ((dec_params->profile == 1 || dec_params->profile == 3) &&
+           ((dec_params->flags & V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING) &&
+            (dec_params->flags & V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING)))
+               return -EINVAL;
+
+       if (dec_params->interpolation_filter >

V4L2_VP9_INTERP_FILTER_SWITCHABLE)

drivers/media/v4l2-core/v4l2-ctrls.c +43 -0
+               return -EINVAL;
+
+       /*
+        * According to the spec, tile_cols_log2 shall be less than or equal
+        * to 6.
+        */
+       if (dec_params->tile_cols_log2 > 6)
+               return -EINVAL;
+
+       if (dec_params->tx_mode > V4L2_VP9_TX_MODE_SELECT)
+               return -EINVAL;
+
+       if (dec_params->reference_mode > V4L2_VP9_REF_MODE_SELECT)
+               return -EINVAL;
+
+       ret = validate_vp9_lf_params(&dec_params->lf);
+       if (ret)
+               return ret;
+
+       ret = validate_vp9_quant_params(&dec_params->quant);
+       if (ret)
+               return ret;
+
+       ret = validate_vp9_seg_params(&dec_params->seg);
+       if (ret)
+               return ret;
+
+       memset(dec_params->padding, 0, sizeof(dec_params->padding));
+       return 0;
+}
+
 /* Validate a new control */
 
 #define zero_padding(s) \
@@ -1799,6 +2026,12 @@ static int std_validate_compound(const struct
 v4l2_ctrl *ctrl, u32 idx,
                zero_padding(p_vp8_frame_header->coder_state);
                break;
 
+       case V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS:
+               return validate_vp9_frame_decode_params(p);
+
+       case V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT:
+               break;
+
        case V4L2_CTRL_TYPE_HEVC_SPS:
                p_hevc_sps = p;
@@ -2542,6 +2775,12 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct
 v4l2_ctrl_handler *hdl,
        case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
                elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header);
                break;
+       case V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT:
+               elem_size = sizeof(struct v4l2_ctrl_vp9_frame_ctx);
+               break;
+       case V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS:
+               elem_size = sizeof(struct
 v4l2_ctrl_vp9_frame_decode_params);
+               break;
        case V4L2_CTRL_TYPE_HEVC_SPS:
                elem_size = sizeof(struct v4l2_ctrl_hevc_sps);
                break;
drivers/media/v4l2- renamed from drivers/media/v4l2-core/v4l2-ioctl.c +0 -0

core/v4l2-ioctl.c
index 2322f08a98be..4fe77ad917c8 100644

drivers/media/v4l2-core/v4l2-ioctl.c +1 -0
@@ -1423,6 +1423,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
                case V4L2_PIX_FMT_VP8:          descr = "VP8"; break;
                case V4L2_PIX_FMT_VP8_FRAME:    descr = "VP8 Frame"; break;
                case V4L2_PIX_FMT_VP9:          descr = "VP9"; break;
+               case V4L2_PIX_FMT_VP9_FRAME:    descr = "VP9 Frame"; break;
                case V4L2_PIX_FMT_HEVC:         descr = "HEVC"; break; /*

aka H.265 */
case V4L2_PIX_FMT_HEVC_SLICE: descr = "HEVC Parsed Slice
Data"; break;
case V4L2_PIX_FMT_FWHT: descr = "FWHT"; break; /*
used in vicodec */

include/media/v4l2-ctrls.h +1 -0
index 757a713bad41..2de8290f9f11 100644
@@ -21,6 +21,7 @@
 #include <media/fwht-ctrls.h>
 #include <media/h264-ctrls.h>
 #include <media/vp8-ctrls.h>
+#include <media/vp9-ctrls.h>
 #include <media/hevc-ctrls.h>
 
 /* forward references */
include/media/vp9-ctrls.h new file +29 -0
new file mode 100644
index 000000000000..0cdea8a18b72
@@ -0,0 +1,485 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * These are the VP9 state controls for use with stateless VP9
+ * codec drivers.
+ *
+ * It turns out that these structs are not stable yet and will undergo
+ * more changes. So keep them private until they are stable and ready to
+ * become part of the official public API.
+ */
+
+#ifndef _VP9_CTRLS_H_
+#define _VP9_CTRLS_H_
+
+#include <linux/types.h>
+
+#define V4L2_PIX_FMT_VP9_FRAME v4l2_fourcc('V', 'P', '9', 'F')
+
+#define V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(i)       (V4L2_CID_MPEG_BASE
+ 4000 + (i))
+#define V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS    (V4L2_CID_MPEG_BASE
+ 4004)
+#define V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT               0x400
+#define V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS         0x404
+
+/**
+ * enum v4l2_vp9_loop_filter_flags - VP9 loop filter flags
+ *
+ * @V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED: the filter level depends on
+ *                                          the mode and reference frame

used
+ * to predict a block
+ * @V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE: the bitstream contains additional
+ * syntax elements that specify
which
+ * mode and reference frame deltas
+ * are to be updated
+ *
+ * Those are the flags you should pass to &v4l2_vp9_loop_filter.flags. See
+ * section '7.2.8 Loop filter semantics' of the VP9 specification for more
+ * details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+enum v4l2_vp9_loop_filter_flags {
+       V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED = 1 << 0,
+       V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE = 1 << 1,
+};
+
+/**
+ * struct v4l2_vp9_loop_filter - VP9 loop filter parameters
+ *
+ * @flags: combination of V4L2_VP9_LOOP_FILTER_FLAG_* flags
+ * @level: indicates the loop filter strength
+ * @sharpness: indicates the sharpness level
+ * @ref_deltas: contains the adjustment needed for the filter level based

on
+ * the chosen reference frame
+ * @mode_deltas: contains the adjustment needed for the filter level based on
+ * the chosen mode
+ * @level_lookup: level lookup table
+ *
+ * This structure contains all loop filter related parameters. See sections
+ * '7.2.8 Loop filter semantics' and '8.8.1 Loop filter frame init process'
+ * of the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h line 66

Quoted 8 lines
include/media/vp9-ctrls.h new file +8 -0
+struct v4l2_vp9_loop_filter {
+       __u8 flags;
+       __u8 level;
+       __u8 sharpness;
+       __s8 ref_deltas[4];
+       __s8 mode_deltas[2];
+       __u8 level_lookup[8][4][2];
+};

This struct is 73 bytes, doesn't it need padding?

Quoted 411 lines
include/media/vp9-ctrls.h new file +11 -0
+
+/**
+ * struct v4l2_vp9_quantization - VP9 quantization parameters
+ *
+ * @base_q_idx: indicates the base frame qindex
+ * @delta_q_y_dc: indicates the Y DC quantizer relative to base_q_idx
+ * @delta_q_uv_dc: indicates the UV DC quantizer relative to base_q_idx
+ * @delta_q_uv_ac indicates the UV AC quantizer relative to base_q_idx
+ * @padding: padding bytes to align things on 64 bits. Must be set to 0
+ *
+ * Encodes the quantization parameters. See section '7.2.9 Quantization

params
+ * syntax' of the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+struct v4l2_vp9_quantization {
+       __u8 base_q_idx;
+       __s8 delta_q_y_dc;
+       __s8 delta_q_uv_dc;
+       __s8 delta_q_uv_ac;
+       __u8 padding[4];
+};
+
+/**
+ * enum v4l2_vp9_segmentation_flags - VP9 segmentation flags
+ *
+ * @V4L2_VP9_SEGMENTATION_FLAG_ENABLED: indicates that this frame makes use

of
+ * the segmentation tool
+ * @V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP: indicates that the segmentation map
+ * should be updated during the
+ * decoding of this frame
+ * @V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE: indicates that the updates to
+ * the segmentation map are
coded
+ * relative to the existing
+ * segmentation map
+ * @V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA: indicates that new parameters are
+ * about to be specified for each
+ * segment
+ * @V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE: indicates that the
+ * segmentation parameters
+ * represent the actual
values
+ * to be used
+ *
+ * Those are the flags you should pass to &v4l2_vp9_segmentation.flags. See
+ * section '7.2.10 Segmentation params syntax' of the VP9 specification for
+ * more details.
+ */

include/media/vp9-ctrls.h new file +21 -0
+enum v4l2_vp9_segmentation_flags {
+       V4L2_VP9_SEGMENTATION_FLAG_ENABLED = 1 << 0,
+       V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP = 1 << 1,
+       V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE = 1 << 2,
+       V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA = 1 << 3,
+       V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE = 1 << 4,
+};
+
+#define V4L2_VP9_SEGMENT_FEATURE_ENABLED(id)   (1 << (id))
+#define V4L2_VP9_SEGMENT_FEATURE_ENABLED_MASK  0xf
+
+/**
+ * enum v4l2_vp9_segment_feature - VP9 segment feature IDs
+ *
+ * @V4L2_VP9_SEGMENT_FEATURE_QP_DELTA: QP delta segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_LF: loop filter segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_REF_FRAME: reference frame segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_SKIP: skip segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_CNT: number of segment features
+ *
+ * Segment feature IDs. See section '7.2.10 Segmentation params syntax' of

the
+ * VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +23 -0
+enum v4l2_vp9_segment_feature {
+       V4L2_VP9_SEGMENT_FEATURE_QP_DELTA,
+       V4L2_VP9_SEGMENT_FEATURE_LF,
+       V4L2_VP9_SEGMENT_FEATURE_REF_FRAME,
+       V4L2_VP9_SEGMENT_FEATURE_SKIP,
+       V4L2_VP9_SEGMENT_FEATURE_CNT,
+};
+
+/**
+ * struct v4l2_vp9_segmentation - VP9 segmentation parameters
+ *
+ * @flags: combination of V4L2_VP9_SEGMENTATION_FLAG_* flags
+ * @tree_probs: specifies the probability values to be used when
+ *              decoding a Segment-ID. See '5.15. Segmentation map'
+ *              section of the VP9 specification for more details.
+ * @pred_prob: specifies the probability values to be used when decoding a
+ *            Predicted-Segment-ID. See '6.4.14. Get segment id syntax'
+ *            section of :ref:`vp9` for more details..
+ * @padding: padding used to make things aligned on 64 bits. Shall be zero
+ *          filled
+ * @feature_enabled: bitmask defining which features are enabled in each
+ *                  segment
+ * @feature_data: data attached to each feature. Data entry is only valid

if
+ * the feature is enabled
+ *
+ * Encodes the quantization parameters. See section '7.2.10 Segmentation
+ * params syntax' of the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +48 -0
+struct v4l2_vp9_segmentation {
+       __u8 flags;
+       __u8 tree_probs[7];
+       __u8 pred_probs[3];
+       __u8 padding[5];
+       __u8 feature_enabled[8];
+       __s16 feature_data[8][4];
+};
+
+/**
+ * enum v4l2_vp9_intra_prediction_mode - VP9 Intra prediction modes
+ *
+ * @V4L2_VP9_INTRA_PRED_DC: DC intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_V: vertical intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_H: horizontal intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D45: D45 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D135: D135 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D117: D117 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D153: D153 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D207: D207 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D63: D63 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_TM: True Motion intra prediction
+ *
+ * See section '7.4.5 Intra frame mode info semantics' for more details.
+ */
+enum v4l2_vp9_intra_prediction_mode {
+       V4L2_VP9_INTRA_PRED_MODE_DC,
+       V4L2_VP9_INTRA_PRED_MODE_V,
+       V4L2_VP9_INTRA_PRED_MODE_H,
+       V4L2_VP9_INTRA_PRED_MODE_D45,
+       V4L2_VP9_INTRA_PRED_MODE_D135,
+       V4L2_VP9_INTRA_PRED_MODE_D117,
+       V4L2_VP9_INTRA_PRED_MODE_D153,
+       V4L2_VP9_INTRA_PRED_MODE_D207,
+       V4L2_VP9_INTRA_PRED_MODE_D63,
+       V4L2_VP9_INTRA_PRED_MODE_TM,
+};
+
+/**
+ * struct v4l2_vp9_mv_probabilities - VP9 Motion vector probabilities
+ * @joint: motion vector joint probabilities
+ * @sign: motion vector sign probabilities
+ * @class: motion vector class probabilities
+ * @class0_bit: motion vector class0 bit probabilities
+ * @bits: motion vector bits probabilities
+ * @class0_fr: motion vector class0 fractional bit probabilities
+ * @fr: motion vector fractional bit probabilities
+ * @class0_hp: motion vector class0 high precision fractional bit

probabilities
+ * @hp: motion vector high precision fractional bit probabilities
+ */

include/media/vp9-ctrls.h new file +56 -0
+struct v4l2_vp9_mv_probabilities {
+       __u8 joint[3];
+       __u8 sign[2];
+       __u8 class[2][10];
+       __u8 class0_bit[2];
+       __u8 bits[2][10];
+       __u8 class0_fr[2][2][3];
+       __u8 fr[2][3];
+       __u8 class0_hp[2];
+       __u8 hp[2];
+};
+
+/**
+ * struct v4l2_vp9_probabilities - VP9 Probabilities
+ *
+ * @tx8: TX 8x8 probabilities
+ * @tx16: TX 16x16 probabilities
+ * @tx32: TX 32x32 probabilities
+ * @coef: coefficient probabilities
+ * @skip: skip probabilities
+ * @inter_mode: inter mode probabilities
+ * @interp_filter: interpolation filter probabilities
+ * @is_inter: is inter-block probabilities
+ * @comp_mode: compound prediction mode probabilities
+ * @single_ref: single ref probabilities
+ * @comp_ref: compound ref probabilities
+ * @y_mode: Y prediction mode probabilities
+ * @uv_mode: UV prediction mode probabilities
+ * @partition: partition probabilities
+ * @mv: motion vector probabilities
+ *
+ * Structure containing most VP9 probabilities. See the VP9 specification
+ * for more details.
+ */
+struct v4l2_vp9_probabilities {
+       __u8 tx8[2][1];
+       __u8 tx16[2][2];
+       __u8 tx32[2][3];
+       __u8 coef[4][2][2][6][6][3];
+       __u8 skip[3];
+       __u8 inter_mode[7][3];
+       __u8 interp_filter[4][2];
+       __u8 is_inter[4];
+       __u8 comp_mode[5];
+       __u8 single_ref[5][2];
+       __u8 comp_ref[5];
+       __u8 y_mode[4][9];
+       __u8 uv_mode[10][9];
+       __u8 partition[16][3];
+
+       struct v4l2_vp9_mv_probabilities mv;
+};
+
+/**
+ * enum v4l2_vp9_reset_frame_context - Valid values for
+ *                     &v4l2_ctrl_vp9_frame_decode_params-
Quoted 1 line

reset_frame_context

+ *
+ * @V4L2_VP9_RESET_FRAME_CTX_NONE: don't reset any frame context
+ * @V4L2_VP9_RESET_FRAME_CTX_SPEC: reset the frame context pointed by
+ * &v4l2_ctrl_vp9_frame_decode_params.frame_context_idx
+ * @V4L2_VP9_RESET_FRAME_CTX_ALL: reset all frame contexts
+ *
+ * See section '7.2 Uncompressed header semantics' of the VP9 specification
+ * for more details.
+ */

include/media/vp9-ctrls.h new file +17 -0
+enum v4l2_vp9_reset_frame_context {
+       V4L2_VP9_RESET_FRAME_CTX_NONE,
+       V4L2_VP9_RESET_FRAME_CTX_SPEC,
+       V4L2_VP9_RESET_FRAME_CTX_ALL,
+};
+
+/**
+ * enum v4l2_vp9_interpolation_filter - VP9 interpolation filter types
+ *
+ * @V4L2_VP9_INTERP_FILTER_8TAP: height tap filter
+ * @V4L2_VP9_INTERP_FILTER_8TAP_SMOOTH: height tap smooth filter
+ * @V4L2_VP9_INTERP_FILTER_8TAP_SHARP: height tap sharp filter
+ * @V4L2_VP9_INTERP_FILTER_BILINEAR: bilinear filter
+ * @V4L2_VP9_INTERP_FILTER_SWITCHABLE: filter selection is signaled at the
+ *                                    block level
+ *
+ * See section '7.2.7 Interpolation filter semantics' of the VP9

specification
+ * for more details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+enum v4l2_vp9_interpolation_filter {
+       V4L2_VP9_INTERP_FILTER_8TAP,
+       V4L2_VP9_INTERP_FILTER_8TAP_SMOOTH,
+       V4L2_VP9_INTERP_FILTER_8TAP_SHARP,
+       V4L2_VP9_INTERP_FILTER_BILINEAR,
+       V4L2_VP9_INTERP_FILTER_SWITCHABLE,
+};
+
+/**
+ * enum v4l2_vp9_reference_mode - VP9 reference modes
+ *
+ * @V4L2_VP9_REF_MODE_SINGLE: indicates that all the inter blocks use only

a
+ * single reference frame to generate motion
+ * compensated prediction
+ * @V4L2_VP9_REF_MODE_COMPOUND: requires all the inter blocks to use compound
+ * mode. Single reference frame prediction is
not
+ * allowed
+ * @V4L2_VP9_REF_MODE_SELECT: allows each individual inter block to select
+ * between single and compound prediction modes
+ *
+ * See section '7.3.6 Frame reference mode semantics' of the VP9 specification
+ * for more details.
+ */

include/media/vp9-ctrls.h new file +14 -0
+enum v4l2_vp9_reference_mode {
+       V4L2_VP9_REF_MODE_SINGLE,
+       V4L2_VP9_REF_MODE_COMPOUND,
+       V4L2_VP9_REF_MODE_SELECT,
+};
+
+/**
+ * enum v4l2_vp9_tx_mode - VP9 TX modes
+ *
+ * @V4L2_VP9_TX_MODE_ONLY_4X4: transform size is 4x4
+ * @V4L2_VP9_TX_MODE_ALLOW_8X8: transform size can be up to 8x8
+ * @V4L2_VP9_TX_MODE_ALLOW_16X16: transform size can be up to 16x16
+ * @V4L2_VP9_TX_MODE_ALLOW_32X32: transform size can be up to 32x32
+ * @V4L2_VP9_TX_MODE_SELECT: bitstream contains transform size for each

block
+ *
+ * See section '7.3.1 Tx mode semantics' of the VP9 specification for more
+ * details.
+ */

include/media/vp9-ctrls.h new file +17 -0
+enum v4l2_vp9_tx_mode {
+       V4L2_VP9_TX_MODE_ONLY_4X4,
+       V4L2_VP9_TX_MODE_ALLOW_8X8,
+       V4L2_VP9_TX_MODE_ALLOW_16X16,
+       V4L2_VP9_TX_MODE_ALLOW_32X32,
+       V4L2_VP9_TX_MODE_SELECT,
+};
+
+/**
+ * enum v4l2_vp9_ref_id - VP9 Reference frame IDs
+ *
+ * @V4L2_REF_ID_LAST: last reference frame
+ * @V4L2_REF_ID_GOLDEN: golden reference frame
+ * @V4L2_REF_ID_ALTREF: alternative reference frame
+ * @V4L2_REF_ID_CNT: number of reference frames
+ *
+ * See section '7.4.12 Ref frames semantics' of the VP9 specification for

more
+ * details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+enum v4l2_vp9_ref_id {
+       V4L2_REF_ID_LAST,
+       V4L2_REF_ID_GOLDEN,
+       V4L2_REF_ID_ALTREF,
+       V4L2_REF_ID_CNT,
+};
+
+/**
+ * enum v4l2_vp9_frame_flags - VP9 frame flags
+ * @V4L2_VP9_FRAME_FLAG_KEY_FRAME: the frame is a key frame
+ * @V4L2_VP9_FRAME_FLAG_SHOW_FRAME: the frame should be displayed
+ * @V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT: the decoding should be error

resilient
+ * @V4L2_VP9_FRAME_FLAG_INTRA_ONLY: the frame does not reference other frames
+ * @V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV: the frame might can high precision
+ * motion vectors
+ * @V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX: frame context should be updated
+ * after decoding
+ * @V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE: parallel decoding is used
+ * @V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING: vertical subsampling is enabled
+ * @V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING: horizontal subsampling is enabled
+ * @V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING: full UV range is used
+ *
+ * Check the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +23 -0
+enum v4l2_vp9_frame_flags {
+       V4L2_VP9_FRAME_FLAG_KEY_FRAME = 1 << 0,
+       V4L2_VP9_FRAME_FLAG_SHOW_FRAME = 1 << 1,
+       V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT = 1 << 2,
+       V4L2_VP9_FRAME_FLAG_INTRA_ONLY = 1 << 3,
+       V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV = 1 << 4,
+       V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX = 1 << 5,
+       V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE = 1 << 6,
+       V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING = 1 << 7,
+       V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING = 1 << 8,
+       V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING = 1 << 9,
+};
+
+#define V4L2_VP9_PROFILE_MAX           3
+
+/**
+ * struct v4l2_ctrl_vp9_frame_decode_params - VP9 frame decoding control
+ *
+ * @flags: combination of V4L2_VP9_FRAME_FLAG_* flags
+ * @compressed_header_size: compressed header size in bytes
+ * @uncompressed_header_size: uncompressed header size in bytes
+ * @profile: VP9 profile. Can be 0, 1, 2 or 3
+ * @reset_frame_context: specifies whether the frame context should be

reset
+ * to default values. See
&v4l2_vp9_reset_frame_context
+ * for more details
+ * @frame_context_idx: frame context that should be used/updated
+ * @bit_depth: bits per components. Can be 8, 10 or 12. Note that not all
+ * profiles support 10 and/or 12 bits depths
+ * @interpolation_filter: specifies the filter selection used for performing
+ * inter prediction. See
&v4l2_vp9_interpolation_filter
+ * for more details
+ * @tile_cols_log2: specifies the base 2 logarithm of the width of each tile
+ * (where the width is measured in units of 8x8 blocks).
+ * Shall be less than or equal to 6
+ * @tile_rows_log2: specifies the base 2 logarithm of the height of each tile
+ * (where the height is measured in units of 8x8 blocks)
+ * @tx_mode: specifies the TX mode. See &v4l2_vp9_tx_mode for more details
+ * @reference_mode: specifies the type of inter prediction to be used. See
+ * &v4l2_vp9_reference_mode for more details
+ * @padding: needed to make this struct 64 bit aligned. Shall be filled with
+ * zeros
+ * @frame_width_minus_1: add 1 to it and you'll get the frame width expressed
+ * in pixels
+ * @frame_height_minus_1: add 1 to it and you'll get the frame height expressed
+ * in pixels
+ * @frame_width_minus_1: add 1 to it and you'll get the expected render width

+ *                      expressed in pixels. This is not used during the
+ *                      decoding process but might be used by HW scalers to
+ *                      prepare a frame that's ready for scanout
+ * @frame_height_minus_1: add 1 to it and you'll get the expected render
height
+ *                      expressed in pixels. This is not used during the
+ *                      decoding process but might be used by HW scalers to
+ *                      prepare a frame that's ready for scanout

+ * @refs: array of reference frames. See &v4l2_vp9_ref_id for more details
+ * @lf: loop filter parameters. See &v4l2_vp9_loop_filter for more details
+ * @quant: quantization parameters. See &v4l2_vp9_quantization for more details
+ * @seg: segmentation parameters. See &v4l2_vp9_segmentation for more details
+ * @probs: probabilities. See &v4l2_vp9_probabilities for more details
+ */

include/media/vp9-ctrls.h new file +4 -0
+struct v4l2_ctrl_vp9_frame_decode_params {
+       __u32 flags;
+       __u16 compressed_header_size;
+       __u16 uncompressed_header_size;

include/media/vp9-ctrls.h line 453

Quoted 10 lines
include/media/vp9-ctrls.h new file +10 -0
+       __u8 profile;
+       __u8 reset_frame_context;
+       __u8 frame_context_idx;
+       __u8 bit_depth;
+       __u8 interpolation_filter;
+       __u8 tile_cols_log2;
+       __u8 tile_rows_log2;
+       __u8 tx_mode;
+       __u8 reference_mode;
+       __u8 padding[6];

If my math is correct this should be 7.

include/media/vp9-ctrls.h line 458

Quoted 5 lines
include/media/vp9-ctrls.h new file +5 -0
+       __u16 frame_width_minus_1;
+       __u16 frame_height_minus_1;
+       __u16 render_width_minus_1;
+       __u16 render_height_minus_1;
+       __u64 refs[V4L2_REF_ID_CNT];

Why are we using u64s to store these [0..7] indices? Or are we storing more than that?

Quoted 5 lines
include/media/vp9-ctrls.h new file +5 -0
+       struct v4l2_vp9_loop_filter lf;
+       struct v4l2_vp9_quantization quant;
+       struct v4l2_vp9_segmentation seg;
+       struct v4l2_vp9_probabilities probs;
+};
Quoted 7 lines
Quoted 3 lines

I cannot find a trace of ref_frame_sign_biases, which is also part of the uncompressed header, in this structure. Is it missing, or is this information provided in a different way?

This seems missing indeed. Interestingly the Rockchip reference software does not seem to pass this information to the hardware. On the other end, the VC8000D software does. It is also part of DXVA, VA and NVDEC APIs.

I can think of at least one piece of hardware that will require this information, so it would be nice to have it included.

Cheers,
Alex.

_______________________________________________
Linux-rockchip mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip

#10 lore raw

Re: [PATCH v4 2/3] media: uapi: Add VP9 stateless decoder controls

Le dimanche 28 février 2021 à 15:13 +0900, Alexandre Courbot a écrit :

Quoted 123 lines

Hi Nicolas,

Quoted 110 lines · Nicolas Dufresne
Quoted 80 lines · Alexandre Courbot

Hi Ezequiel, sorry for the late review!

Quoted 76 lines · Ezequiel Garcia

From: Boris Brezillon <[email protected]>

Add the VP9 stateless decoder controls plus the documentation that goes with it.

---
 .../userspace-api/media/v4l/biblio.rst        |  10 + .../media/v4l/ext-ctrls-codec.rst             | 550 ++++++++++++++++++ drivers/media/v4l2-core/v4l2-ctrls.c          | 239 ++++++++ drivers/media/v4l2-core/v4l2-ioctl.c          |   1 + include/media/v4l2-ctrls.h                    |   1 + include/media/vp9-ctrls.h                     | 485 +++++++++++++++

 6 files changed, 1286 insertions(+)

 create mode 100644 include/media/vp9-ctrls.h

Documentation/userspace-api/media/v4l/biblio.rst +0 -0
index 3c9634173e82..e09102e572fd 100644
@@ -414,3 +414,13 @@ VP8

 :title:     RFC 6386: "VP8 Data Format and Decoding Guide"

 :author:    J. Bankoski et al.

Documentation/userspace-api/media/v4l/biblio.rst +10 -0
+
+.. _vp9:
+
+VP9
+===
+
+
+:title:     VP9 Bitstream & Decoding Process Specification
+
+:author:    Adrian Grange (Google), Peter de Rivaz (Argon Design),

Jonathan
Hunt (Argon Design)

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +0 -0
index d0d506a444b1..5c5f7dd868da 100644
@@ -2668,6 +2668,556 @@ enum

v4l2_mpeg_video_h264_hierarchical_coding_type -
       - ``padding[3]``
       - Applications and drivers must set this to zero.

+.. _v4l2-mpeg-vp9:
+
+``V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(0..3) (struct)``
+    Stores VP9 probabilities attached to a specific frame context. The VP9
+    specification allows using a maximum of 4 contexts. Each frame being
+    decoded refers to one of those context. See section '7.1.2 Refresh +    probs semantics' section of :ref:vp9 for more details about these +    contexts.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +5 -0
+
+    This control is bi-directional:
+
+    * all 4 contexts must be initialized by userspace just after the
+      stream is started and before the first decoding request is

submitted.
+    * the referenced context might be read by the kernel when a decoding
+      request is submitted, and will be updated after the decoder is done
+      decoding the frame if the V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX flag
+      is set.
+    * contexts will be read back by user space before each decoding request
+      to retrieve the updated probabilities.
+    * userspace will re-initialize the context to their default values

Quoted 8 lines
Quoted 2 lines

when
+      a reset context is required.

Just to make sure I understand this part correctly, it means that if frame A and B use the same context, and frame A has V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX set, then user-space must wait for frame A to get dequeued and read back this control from the completed request before it can submit frame B?

We are preparing a new version, we believe it was an API mistake to try and share the probability update between kernel and userspace. It's actually worst,
you really need to push/pull in a lock step, regardless of the frame context id.

As you may know, the probabilities are used to parse the compressed part of the
stream (also know as entropy decoding). They are probability of a bitstream symbol of being 0 or 1. On RK3399, the hardware does not parse the compressed
header. The compressed headers contains the bitstream coded updates to the probabilities. The probabilities are also updated base on the count of specific
symbols found in the bitstream during decoding, this is currently done in the
driver (counts are returned, and driver updates the probabilities from that).

Quoted 8 lines

What we found is that the symbols in the compress headers are coded with fixed
probabilities, meaning we don't need the probabilities to parse the compressed
headers. Our proposal will be to keep doing the compressed header parsing in userspace and only store the update value (as found in inv_map_table) to the kernel. The driver will be responsible of applying this to the current probabilities along with doing the probability saving process.

Quoted 4 lines

So if I am following:

1) The compressed headers can be decompressed using just the entropy information contained in the previous compressed headers,

Using just the entropy contained in the spec. The entropy for this one is static.

Quoted 2 lines

2) The frame data also requires the updated probabilities from the decoding of the previous frames on top of 1)

Yes, the entropy decoding of the frame data requires default tables, the deltas from the compressed headers, and for non-keyframe the counts of symbols in the previous decoding.

All this is being moved inside the kernel.

Quoted 3 lines

Meaning that user-space can decompress the headers without any feedback from the kernel. Am I understanding correctly?

Correct. We have some really eirly code demonstrating it (will all be published LGPL V2+ and submitted for including in GStreamer). In FFMPEG this is know as VP5/5 Range decoder, so nothing new here.

Quoted 20 lines
Quoted 6 lines

With this change, we should be able to only push data to the decoder without having to read this control. We believe this will allow better use of the VB2
queue and enhance the decoding performance.

That sounds pretty good to me.

Quoted 10 lines

Please note that this hardware design looks like a mistake. It seems that with
widevine, the compressed header is encrypted. As in some implementation the decrypted bitstream is no visible to the CPU, it will not be possible to perform
the needed parsing. An offload to a TEE would be needed. We know that Rockchip
have fixed this issue in new decoder, notably on RK3368. The probability processing is done in HW.

Quoted 2 lines

Mmm a solution would be not to encrypt the compressed header, but I guess that's not how things work, right?

Widewine, Playready and other folks will always dictate their way. I guess I only wanted to mention publicly so that folks visiting this are not too surprised of the detour and contorsion needed. We are not implementing this as we speak. For the browser version of Widevine, I don't think this issue will raise.

Quoted 4 lines

Worst case the TEE could indeed copy the decrypted and decompressed somewhere for the CPU to read, but that would complicate things a tiny bit.

Exact, there "exist a solution" ;-P From a API perspective, we simply visited this as we where told it was not possible at all, so we were curious upon why. And this is slightly specific to this HW design, newer Rockchip design handle compress header decoding in HW, so I suspect other vendors will follow this path in the future.

Quoted 559 lines
Quoted 559 lines
Quoted 542 lines
Quoted 542 lines
Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +4 -0
+
+    .. note::
+
+       This compound control is not yet part of the public kernel API

and
+       it is expected to change.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +93 -0
+
+.. c:type:: v4l2_ctrl_vp9_frame_ctx
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{5.8cm}|p{4.8cm}|p{6.6cm}|
+
+.. flat-table:: struct v4l2_ctrl_vp9_frame_ctx
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - struct :c:type:`v4l2_vp9_probabilities`
+      - ``probs``
+      - Structure with VP9 probabilities attached to the context.
+
+.. c:type:: v4l2_vp9_probabilities
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_probabilities
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``tx8[2][1]``
+      - TX 8x8 probabilities.
+    * - __u8
+      - ``tx16[2][2]``
+      - TX 16x16 probabilities.
+    * - __u8
+      - ``tx32[2][3]``
+      - TX 32x32 probabilities.
+    * - __u8
+      - ``coef[4][2][2][6][6][3]``
+      - Coefficient probabilities.
+    * - __u8
+      - ``skip[3]``
+      - Skip probabilities.
+    * - __u8
+      - ``inter_mode[7][3]``
+      - Inter prediction mode probabilities.
+    * - __u8
+      - ``interp_filter[4][2]``
+      - Interpolation filter probabilities.
+    * - __u8
+      - ``is_inter[4]``
+      - Is inter-block probabilities.
+    * - __u8
+      - ``comp_mode[5]``
+      - Compound prediction mode probabilities.
+    * - __u8
+      - ``single_ref[5][2]``
+      - Single reference probabilities.
+    * - __u8
+      - ``comp_mode[5]``
+      - Compound reference probabilities.
+    * - __u8
+      - ``y_mode[4][9]``
+      - Y prediction mode probabilities.
+    * - __u8
+      - ``uv_mode[10][9]``
+      - UV prediction mode probabilities.
+    * - __u8
+      - ``partition[16][3]``
+      - Partition probabilities.
+    * - __u8
+      - ``mv.joint[3]``
+      - Motion vector joint probabilities.
+    * - __u8
+      - ``mv.sign[2]``
+      - Motion vector sign probabilities.
+    * - __u8
+      - ``mv.class[2][10]``
+      - Motion vector class probabilities.
+    * - __u8
+      - ``mv.class0_bit[2]``
+      - Motion vector class0 bit probabilities.
+    * - __u8
+      - ``mv.bits[2][10]``
+      - Motion vector bits probabilities.
+    * - __u8
+      - ``mv.class0_fr[2][2][3]``
+      - Motion vector class0 fractional bit probabilities.
+    * - __u8
+      - ``mv.fr[2][3]``
+      - Motion vector fractional bit probabilities.
+    * - __u8
+      - ``mv.class0_hp[2]``
+      - Motion vector class0 high precision fractional bit

probabilities.
+    * - __u8
+      - ``mv.hp[2]``
+      - Motion vector high precision fractional bit probabilities. +
+``V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS (struct)``
+    Specifies the frame parameters for the associated VP9 frame decode request.
+    This includes the necessary parameters for configuring a stateless hardware
+    decoding pipeline for VP9. The bitstream parameters are defined according
+    to :ref:vp9.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +4 -0
+
+    .. note::
+
+       This compound control is not yet part of the public kernel API

and
+       it is expected to change.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +28 -0
+
+.. c:type:: v4l2_ctrl_vp9_frame_decode_params
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_ctrl_vp9_frame_decode_params
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``flags``
+      - Combination of V4L2_VP9_FRAME_FLAG_* flags. See
+        :c:type:`v4l2_vp9_frame_flags`.
+    * - __u16
+      - ``compressed_header_size``
+      - Compressed header size in bytes.
+    * - __u16
+      - ``uncompressed_header_size``
+      - Uncompressed header size in bytes.
+    * - __u8
+      - ``profile``
+      - VP9 profile. Can be 0, 1, 2 or 3.
+    * - __u8
+      - ``reset_frame_context``
+      - Frame context that should be used/updated when decoding the

frame.
+    * - __u8
+      - ``bit_depth``
+      - Component depth in bits. Must be 8 for profile 0 and 1. Must 10 or
12
+        for profile 2 and 3.
+    * - __u8
+      - ``interpolation_filter``
+      - Specifies the filter selection used for performing inter prediction. See
+        :c:type:v4l2_vp9_interpolation_filter.
+    * - __u8
+      - ``tile_cols_log2``
+      - Specifies the base 2 logarithm of the width of each tile (where the
+        width is measured in units of 8x8 blocks). Shall be less than or
equal
+        to 6.
+    * - __u8
+      - ``tile_rows_log2``
+      - Specifies the base 2 logarithm of the height of each tile (where
the
+        height is measured in units of 8x8 blocks)
+    * - __u8
+      - ``tx_mode``
+      - Specifies the TX mode. See :c:type:v4l2_vp9_tx_mode. +    * - __u8
+      - ``reference_mode``
+      - Specifies the type of inter prediction to be used. See +        :c:type:v4l2_vp9_reference_mode.
+    * - __u8
+      - ``padding``
+      - Needed to make this struct 64 bit aligned. Shall be filled with zeroes.
+    * - __u16
+      - ``frame_width_minus_1``
+      - Add 1 to get the frame width expressed in pixels.
+    * - __u16
+      - ``frame_height_minus_1``
+      - Add 1 to to get the frame height expressed in pixels. +    * - __u16
+      - ``frame_width_minus_1``
+      - Add 1 to to get the expected render width expressed in pixels. This
is
+        not used during the decoding process but might be used by HW scalers to
+        prepare a frame that's ready for scanout.
+    * - __u16
+      - frame_height_minus_1
+      - Add 1 to get the expected render height expressed in pixels. This
is
+        not used during the decoding process but might be used by HW scalers to
+        prepare a frame that's ready for scanout.
+    * - __u64
+      - ``refs[3]``
+      - Array of reference frame timestamps.
+    * - struct :c:type:v4l2_vp9_loop_filter
+      - ``lf``
+      - Loop filter parameters. See struct :c:type:v4l2_vp9_loop_filter.
+    * - struct :c:type:v4l2_vp9_quantization
+      - ``quant``
+      - Quantization parameters. See :c:type:v4l2_vp9_quantization. +    * - struct :c:type:v4l2_vp9_segmentation
+      - ``seg``
+      - Segmentation parameters. See :c:type:v4l2_vp9_segmentation. +    * - struct :c:type:v4l2_vp9_probabilities
+      - ``probs``
+      - Probabilities. See :c:type:v4l2_vp9_probabilities.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +88 -0
+
+.. c:type:: v4l2_vp9_frame_flags
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_frame_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_FRAME_FLAG_KEY_FRAME``
+      - The frame is a key frame.
+    * - ``V4L2_VP9_FRAME_FLAG_SHOW_FRAME``
+      - The frame should be displayed.
+    * - ``V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT``
+      - The decoding should be error resilient.
+    * - ``V4L2_VP9_FRAME_FLAG_INTRA_ONLY``
+      - The frame does not reference other frames.
+    * - ``V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV``
+      - the frame might can high precision motion vectors.
+    * - ``V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX``
+      - Frame context should be updated after decoding.
+    * - ``V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE``
+      - Parallel decoding is used.
+    * - ``V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING``
+      - Vertical subsampling is enabled.
+    * - ``V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING``
+      - Horizontal subsampling is enabled.
+    * - ``V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING``
+      - The full UV range is used.
+
+.. c:type:: v4l2_vp9_ref_id
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_ref_id
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_REF_ID_LAST``
+      - Last reference frame.
+    * - ``V4L2_REF_ID_GOLDEN``
+      - Golden reference frame.
+    * - ``V4L2_REF_ID_ALTREF``
+      - Alternative reference frame.
+    * - ``V4L2_REF_ID_CNT``
+      - Number of reference frames.
+
+.. c:type:: v4l2_vp9_tx_mode
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_tx_mode
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_TX_MODE_ONLY_4X4``
+      - Transform size is 4x4.
+    * - ``V4L2_VP9_TX_MODE_ALLOW_8X8``
+      - Transform size can be up to 8x8.
+    * - ``V4L2_VP9_TX_MODE_ALLOW_16X16``
+      - Transform size can be up to 16x16.
+    * - ``V4L2_VP9_TX_MODE_ALLOW_32X32``
+      - transform size can be up to 32x32.
+    * - ``V4L2_VP9_TX_MODE_SELECT``
+      - Bitstream contains transform size for each block.
+
+.. c:type:: v4l2_vp9_reference_mode
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_reference_mode
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_REF_MODE_SINGLE``
+      - Indicates that all the inter blocks use only a single reference

frame
+        to generate motion compensated prediction.
+    * - ``V4L2_VP9_REF_MODE_COMPOUND``
+      - Requires all the inter blocks to use compound mode. Single reference
+        frame prediction is not allowed.
+    * - ``V4L2_VP9_REF_MODE_SELECT``
+      - Allows each individual inter block to select between single and +        compound prediction modes.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +92 -0
+
+.. c:type:: v4l2_vp9_interpolation_filter
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_interpolation_filter
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_INTERP_FILTER_8TAP``
+      - Height tap filter.
+    * - ``V4L2_VP9_INTERP_FILTER_8TAP_SMOOTH``
+      - Height tap smooth filter.
+    * - ``V4L2_VP9_INTERP_FILTER_8TAP_SHARP``
+      - Height tap sharp filter.
+    * - ``V4L2_VP9_INTERP_FILTER_BILINEAR``
+      - Bilinear filter.
+    * - ``V4L2_VP9_INTERP_FILTER_SWITCHABLE``
+      - Filter selection is signaled at the block level.
+
+.. c:type:: v4l2_vp9_reset_frame_context
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_reset_frame_context
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_RESET_FRAME_CTX_NONE``
+      - Do not reset any frame context.
+    * - ``V4L2_VP9_RESET_FRAME_CTX_SPEC``
+      - Reset the frame context pointed by
+        :c:type:`v4l2_ctrl_vp9_frame_decode_params`.frame_context_idx.
+    * - ``V4L2_VP9_RESET_FRAME_CTX_ALL``
+      - Reset all frame contexts.
+
+.. c:type:: v4l2_vp9_intra_prediction_mode
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_intra_prediction_mode
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_INTRA_PRED_DC``
+      - DC intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_V``
+      - Vertical intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_H``
+      - Horizontal intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D45``
+      - D45 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D135``
+      - D135 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D117``
+      - D117 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D153``
+      - D153 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D207``
+      - D207 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D63``
+      - D63 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_TM``
+      - True motion intra prediction.
+
+.. c:type:: v4l2_vp9_segmentation
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_segmentation
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``flags``
+      - Combination of V4L2_VP9_SEGMENTATION_FLAG_* flags. See
+        :c:type:`v4l2_vp9_segmentation_flags`.
+    * - __u8
+      - ``tree_probs[7]``
+      - Specifies the probability values to be used when decoding a

Segment-ID.
+        See '5.15. Segmentation map' section of :ref:vp9 for more details.
+    * - __u8
+      - ``pred_prob[3]``
+      - Specifies the probability values to be used when decoding a +        Predicted-Segment-ID. See '6.4.14. Get segment id syntax' +        section of :ref:vp9 for more details.
+    * - __u8
+      - ``padding[5]``
+      - Used to align this struct on 64 bit. Shall be filled with zeroes.
+    * - __u8
+      - ``feature_enabled[8]``
+      - Bitmask defining which features are enabled in each segment. +    * - __u8
+      - ``feature_data[8][4]``
+      - Data attached to each feature. Data entry is only valid if the feature
+        is enabled.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +38 -0
+
+.. c:type:: v4l2_vp9_segment_feature
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_segment_feature
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_SEGMENT_FEATURE_QP_DELTA``
+      - QP delta segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_LF``
+      - Loop filter segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_REF_FRAME``
+      - Reference frame segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_SKIP``
+      - Skip segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_CNT``
+      - Number of segment features.
+
+.. c:type:: v4l2_vp9_segmentation_flags
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_segmentation_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_ENABLED``
+      - Indicates that this frame makes use of the segmentation tool.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP``
+      - Indicates that the segmentation map should be updated during

the
+        decoding of this frame.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE``
+      - Indicates that the updates to the segmentation map are coded +        relative to the existing segmentation map.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA``
+      - Indicates that new parameters are about to be specified for each
+        segment.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE`` +      - Indicates that the segmentation parameters represent the actual values
+        to be used.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +27 -0
+
+.. c:type:: v4l2_vp9_quantization
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_quantization
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``base_q_idx``
+      - Indicates the base frame qindex.
+    * - __s8
+      - ``delta_q_y_dc``
+      - Indicates the Y DC quantizer relative to base_q_idx.
+    * - __s8
+      - ``delta_q_uv_dc``
+      - Indicates the UV DC quantizer relative to base_q_idx.
+    * - __s8
+      - ``delta_q_uv_ac``
+      - Indicates the UV AC quantizer relative to base_q_idx.
+    * - __u8
+      - ``padding[4]``
+      - Padding bytes used to align this struct on 64 bit. Must be set

to
0.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +25 -0
+
+.. c:type:: v4l2_vp9_loop_filter
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_loop_filter
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``flags``
+      - Combination of V4L2_VP9_LOOP_FILTER_FLAG_* flags.
+        See :c:type:`v4l2_vp9_loop_filter_flags`.
+    * - __u8
+      - ``level``
+      - Indicates the loop filter strength.
+    * - __u8
+      - ``sharpness``
+      - Indicates the sharpness level.
+    * - __s8
+      - ``ref_deltas[4]``
+      - Contains the adjustment needed for the filter level based on

the
chosen
+        reference frame.
+    * - __s8

Quoted 8 lines
Quoted 8 lines

+      - ``mode_deltas[2]``
+      - Contains the adjustment needed for the filter level based on the
chosen
+        mode
+    * - __u8
+      - ``level_lookup[8][4][2]``
+      - Level lookup table.

We think it might be a poor choice to ask for computed values. According to "8.8.1 Loop filter frame init process" this is computed from loop_filter_level,
loop_filter_ref_deltas[], V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE, feature_data[], ref_deltas[] and mode_deltas[]. We believe it is likely a waste
to pass this information assuming some HW could do that process.

Quoted 8 lines
Quoted 3 lines

This is to be discussed. None of the other statelss APIs seems to require this
value to be computed by userspace.

Not particularly opinionated, but it would be a waste if the driver needs to reverse that computation to pass it to the hardware. If the lookup table is not difficult to compute, we can maybe do this using a helper function for drivers that need it?

Correct, the spec is quite clear, and gives the exact algorithm to produce that table. If needs, we can do like we did fro H264 intermediate reference lists and add a shared helper.

Quoted 956 lines
Quoted 956 lines
Quoted 955 lines
Quoted 461 lines
Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +15 -0
+
+
+.. c:type:: v4l2_vp9_loop_filter_flags
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_loop_filter_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED``
+      - When set, the filter level depends on the mode and reference

frame
used
+        to predict a block.
+    * - ``V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE``
+      - When set, the bitstream contains additional syntax elements that
+        specify which mode and reference frame deltas are to be updated.
+
 .. raw:: latex

     \normalsize

drivers/media/v4l2- renamed from drivers/media/v4l2-core/v4l2-ctrls.c +0 -0

core/v4l2-ctrls.c
index 1c617b42a944..115beadaf02d 100644

drivers/media/v4l2-core/v4l2-ctrls.c +0 -0
@@ -930,6 +930,11 @@ const char *v4l2_ctrl_get_name(u32 id)

        case V4L2_CID_MPEG_VIDEO_VP8_PROFILE:                   return "VP8
Profile";
        case V4L2_CID_MPEG_VIDEO_VP9_PROFILE:                   return "VP9
Profile";
        case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:              return "VP8
Frame Header";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS:       return "VP9
Frame Decode Parameters";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(0):          return "VP9
Frame Context 0";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(1):          return "VP9
Frame Context 1";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(2):          return "VP9
Frame Context 2";
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(3):          return "VP9
Frame Context 3";

        /* HEVC controls */
        case V4L2_CID_MPEG_VIDEO_HEVC_I_FRAME_QP:               return "HEVC
I-Frame QP Value";

drivers/media/v4l2-core/v4l2-ctrls.c +0 -0
@@ -1403,6 +1408,15 @@ void v4l2_ctrl_fill(u32 id, const char **name,

enum
v4l2_ctrl_type *type,
        case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
                *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER; break;
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS:
+               *type = V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS; +               break;
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(0):
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(1):
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(2):
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(3):
+               *type = V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT; +               break;
        case V4L2_CID_MPEG_VIDEO_HEVC_SPS:
                *type = V4L2_CTRL_TYPE_HEVC_SPS; break;

drivers/media/v4l2-core/v4l2-ctrls.c +96 -0
@@ -1703,6 +1717,219 @@ static void std_log(const struct v4l2_ctrl
 *ctrl)
         0;                                                      \
  })
 
+static int
+validate_vp9_lf_params(struct v4l2_vp9_loop_filter *lf)
+{
+       unsigned int i, j, k;
+
+       if (lf->flags &
+           ~(V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED |
+             V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED implies
+        * V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE.
+        */
+       if (lf->flags & V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE &&
+           !(lf->flags & V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED))
+               return -EINVAL;
+
+       /* That all values are in the accepted range. */
+       if (lf->level > GENMASK(5, 0))
+               return -EINVAL;
+
+       if (lf->sharpness > GENMASK(2, 0))
+               return -EINVAL;
+
+       for (i = 0; i < ARRAY_SIZE(lf->ref_deltas); i++) {
+               if (lf->ref_deltas[i] < -63 || lf->ref_deltas[i] > 63)
+                       return -EINVAL;
+       }
+
+       for (i = 0; i < ARRAY_SIZE(lf->mode_deltas); i++) {
+               if (lf->mode_deltas[i] < -63 || lf->mode_deltas[i] > 63)
+                       return -EINVAL;
+       }
+
+       for (i = 0; i < ARRAY_SIZE(lf->level_lookup); i++) {
+               for (j = 0; j < ARRAY_SIZE(lf->level_lookup[0]); j++) {
+                       for (k = 0; k < ARRAY_SIZE(lf-
 >level_lookup[0][0]);
 k++) {
+                               if (lf->level_lookup[i][j][k] > 63)
+                                       return -EINVAL;
+                       }
+               }
+       }
+
+       return 0;
+}
+
+static int
+validate_vp9_quant_params(struct v4l2_vp9_quantization *quant)
+{
+       if (quant->delta_q_y_dc < -15 || quant->delta_q_y_dc > 15 ||
+           quant->delta_q_uv_dc < -15 || quant->delta_q_uv_dc > 15 ||
+           quant->delta_q_uv_ac < -15 || quant->delta_q_uv_ac > 15)
+               return -EINVAL;
+
+       memset(quant->padding, 0, sizeof(quant->padding));
+       return 0;
+}
+
+static int
+validate_vp9_seg_params(struct v4l2_vp9_segmentation *seg)
+{
+       unsigned int i, j;
+
+       if (seg->flags &
+           ~(V4L2_VP9_SEGMENTATION_FLAG_ENABLED |
+             V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP |
+             V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE |
+             V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA |
+             V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP and
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA imply
+        * V4L2_VP9_SEGMENTATION_FLAG_ENABLED.
+        */
+       if ((seg->flags &
+            (V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP |
+             V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA)) &&
+           !(seg->flags & V4L2_VP9_SEGMENTATION_FLAG_ENABLED))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE implies
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP.
+        */
+       if (seg->flags & V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE &&
+           !(seg->flags & V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE implies
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA.
+        */
+       if (seg->flags & V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE

&&
+           !(seg->flags & V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA)) +               return -EINVAL;

drivers/media/v4l2-core/v4l2-ctrls.c +26 -0
+
+       for (i = 0; i < ARRAY_SIZE(seg->feature_enabled); i++) {
+               if (seg->feature_enabled[i] &
+                   ~(V4L2_VP9_SEGMENT_FEATURE_QP_DELTA |
+                     V4L2_VP9_SEGMENT_FEATURE_LF |
+                     V4L2_VP9_SEGMENT_FEATURE_REF_FRAME |
+                     V4L2_VP9_SEGMENT_FEATURE_SKIP))
+                       return -EINVAL;
+       }
+
+       for (i = 0; i < ARRAY_SIZE(seg->feature_data); i++) {
+               const int range[] = {255, 63, 3, 0};
+
+               for (j = 0; j < ARRAY_SIZE(seg->feature_data[j]); j++) {
+                       if (seg->feature_data[i][j] < -range[j] ||
+                           seg->feature_data[i][j] > range[j])
+                               return -EINVAL;
+               }
+       }
+
+       memset(seg->padding, 0, sizeof(seg->padding));
+       return 0;
+}
+
+static int
+validate_vp9_frame_decode_params(struct

v4l2_ctrl_vp9_frame_decode_params
*dec_params)

drivers/media/v4l2-core/v4l2-ctrls.c +19 -0
+{
+       int ret;
+
+       /* Make sure we're not passed invalid flags. */
+       if (dec_params->flags &
+           ~(V4L2_VP9_FRAME_FLAG_KEY_FRAME |
+             V4L2_VP9_FRAME_FLAG_SHOW_FRAME |
+             V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT |
+             V4L2_VP9_FRAME_FLAG_INTRA_ONLY |
+             V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV |
+             V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX |
+             V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE |
+             V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING |
+             V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING |
+             V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING))
+               return -EINVAL;
+
+       /*
+        * The refresh context and error resilient flags are mutually

exclusive.
+        * Same goes for parallel decoding and error resilient modes. +        */
+       if (dec_params->flags & V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT && +           dec_params->flags &
+           (V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX |
+            V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE))
+               return -EINVAL;

drivers/media/v4l2-core/v4l2-ctrls.c +5 -0
+
+       if (dec_params->profile > V4L2_VP9_PROFILE_MAX)
+               return -EINVAL;
+
+       if (dec_params->reset_frame_context >

V4L2_VP9_RESET_FRAME_CTX_ALL)
+               return -EINVAL;

drivers/media/v4l2-core/v4l2-ctrls.c +6 -0
+
+       if (dec_params->frame_context_idx >= V4L2_VP9_NUM_FRAME_CTX)
+               return -EINVAL;
+
+       /*
+        * Profiles 0 and 1 only support 8-bit depth, profiles 2 and 3

only
10
+        * and 12 bit depths.
+        */
+       if ((dec_params->profile < 2 && dec_params->bit_depth != 8) || +           (dec_params->profile >= 2 &&
+            (dec_params->bit_depth != 10 && dec_params->bit_depth != 12)))
+               return -EINVAL;

drivers/media/v4l2-core/v4l2-ctrls.c +14 -0
+
+       /* Profile 0 and 2 only accept YUV 4:2:0. */
+       if ((dec_params->profile == 0 || dec_params->profile == 2) &&
+           (!(dec_params->flags & V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING) ||
+            !(dec_params->flags & V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING)))
+               return -EINVAL;
+
+       /* Profile 1 and 3 only accept YUV 4:2:2, 4:4:0 and 4:4:4. */
+       if ((dec_params->profile == 1 || dec_params->profile == 3) &&
+           ((dec_params->flags & V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING) &&
+            (dec_params->flags & V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING)))
+               return -EINVAL;
+
+       if (dec_params->interpolation_filter >

V4L2_VP9_INTERP_FILTER_SWITCHABLE)
+               return -EINVAL;
+
+       /*
+        * According to the spec, tile_cols_log2 shall be less than or equal
+        * to 6.
+        */
+       if (dec_params->tile_cols_log2 > 6)
+               return -EINVAL;

drivers/media/v4l2-core/v4l2-ctrls.c +23 -0
+
+       if (dec_params->tx_mode > V4L2_VP9_TX_MODE_SELECT)
+               return -EINVAL;
+
+       if (dec_params->reference_mode > V4L2_VP9_REF_MODE_SELECT)
+               return -EINVAL;
+
+       ret = validate_vp9_lf_params(&dec_params->lf);
+       if (ret)
+               return ret;
+
+       ret = validate_vp9_quant_params(&dec_params->quant);
+       if (ret)
+               return ret;
+
+       ret = validate_vp9_seg_params(&dec_params->seg);
+       if (ret)
+               return ret;
+
+       memset(dec_params->padding, 0, sizeof(dec_params->padding));
+       return 0;
+}
+

 /* Validate a new control */

 #define zero_padding(s) \

drivers/media/v4l2-core/v4l2-ctrls.c +6 -0
@@ -1799,6 +2026,12 @@ static int std_validate_compound(const struct
 v4l2_ctrl *ctrl, u32 idx,
                 zero_padding(p_vp8_frame_header->coder_state);
                 break;
 
+       case V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS:
+               return validate_vp9_frame_decode_params(p);
+
+       case V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT:
+               break;
+

        case V4L2_CTRL_TYPE_HEVC_SPS:
                p_hevc_sps = p;

drivers/media/v4l2-core/v4l2-ctrls.c +0 -0
@@ -2542,6 +2775,12 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct
 v4l2_ctrl_handler *hdl,

        case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
                elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header); break;
+       case V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT:
+               elem_size = sizeof(struct v4l2_ctrl_vp9_frame_ctx); +               break;
+       case V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS:
+               elem_size = sizeof(struct v4l2_ctrl_vp9_frame_decode_params);
+               break;
        case V4L2_CTRL_TYPE_HEVC_SPS:
                elem_size = sizeof(struct v4l2_ctrl_hevc_sps); break;

drivers/media/v4l2- renamed from drivers/media/v4l2-core/v4l2-ioctl.c +0 -0

core/v4l2-ioctl.c
index 2322f08a98be..4fe77ad917c8 100644

drivers/media/v4l2-core/v4l2-ioctl.c +1 -0
@@ -1423,6 +1423,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc
 *fmt)
                 case V4L2_PIX_FMT_VP8:          descr = "VP8"; break;
                 case V4L2_PIX_FMT_VP8_FRAME:    descr = "VP8 Frame";
 break;
                 case V4L2_PIX_FMT_VP9:          descr = "VP9"; break;
+               case V4L2_PIX_FMT_VP9_FRAME:    descr = "VP9 Frame";
 break;

                case V4L2_PIX_FMT_HEVC:         descr = "HEVC"; break; /*
aka H.265 */
                case V4L2_PIX_FMT_HEVC_SLICE:   descr = "HEVC Parsed Slice
Data"; break;
                case V4L2_PIX_FMT_FWHT:         descr = "FWHT"; break; /*
used in vicodec */

include/media/v4l2-ctrls.h +1 -0
index 757a713bad41..2de8290f9f11 100644
@@ -21,6 +21,7 @@
  #include <media/fwht-ctrls.h>
  #include <media/h264-ctrls.h>
  #include <media/vp8-ctrls.h>
+#include <media/vp9-ctrls.h>
  #include <media/hevc-ctrls.h>
 
  /* forward references */
include/media/vp9-ctrls.h new file +7 -0
new file mode 100644
index 000000000000..0cdea8a18b72
@@ -0,0 +1,485 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * These are the VP9 state controls for use with stateless VP9
+ * codec drivers.
+ *
+ * It turns out that these structs are not stable yet and will undergo
+ * more changes. So keep them private until they are stable and ready

to
+ * become part of the official public API.
+ */

include/media/vp9-ctrls.h new file +9 -0
+
+#ifndef _VP9_CTRLS_H_
+#define _VP9_CTRLS_H_
+
+#include <linux/types.h>
+
+#define V4L2_PIX_FMT_VP9_FRAME v4l2_fourcc('V', 'P', '9', 'F')
+
+#define V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(i)      

(V4L2_CID_MPEG_BASE
+ 4000 + (i))
+#define V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS (V4L2_CID_MPEG_BASE
+ 4004)

include/media/vp9-ctrls.h new file +7 -0
+#define V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT               0x400
+#define V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS         0x404
+
+/**
+ * enum v4l2_vp9_loop_filter_flags - VP9 loop filter flags
+ *
+ * @V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED: the filter level depends

on
+ *                                          the mode and reference frame
used
+ *                                          to predict a block
+ * @V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE: the bitstream contains additional
+ *                                         syntax elements that specify which
+ *                                         mode and reference frame deltas
+ *                                         are to be updated
+ *
+ * Those are the flags you should pass to &v4l2_vp9_loop_filter.flags. See
+ * section '7.2.8 Loop filter semantics' of the VP9 specification for more
+ * details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+enum v4l2_vp9_loop_filter_flags {
+       V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED = 1 << 0,
+       V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE = 1 << 1,
+};
+
+/**
+ * struct v4l2_vp9_loop_filter - VP9 loop filter parameters
+ *
+ * @flags: combination of V4L2_VP9_LOOP_FILTER_FLAG_* flags
+ * @level: indicates the loop filter strength
+ * @sharpness: indicates the sharpness level
+ * @ref_deltas: contains the adjustment needed for the filter level

based
on
+ *             the chosen reference frame
+ * @mode_deltas: contains the adjustment needed for the filter level based
on
+ *              the chosen mode
+ * @level_lookup: level lookup table
+ *
+ * This structure contains all loop filter related parameters. See sections
+ * '7.2.8 Loop filter semantics' and '8.8.1 Loop filter frame init process'
+ * of the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h line 66

Quoted 8 lines
include/media/vp9-ctrls.h new file +8 -0
+struct v4l2_vp9_loop_filter {
+       __u8 flags;
+       __u8 level;
+       __u8 sharpness;
+       __s8 ref_deltas[4];
+       __s8 mode_deltas[2];
+       __u8 level_lookup[8][4][2];
+};

This struct is 73 bytes, doesn't it need padding?

Quoted 454 lines
include/media/vp9-ctrls.h new file +11 -0
+
+/**
+ * struct v4l2_vp9_quantization - VP9 quantization parameters
+ *
+ * @base_q_idx: indicates the base frame qindex
+ * @delta_q_y_dc: indicates the Y DC quantizer relative to base_q_idx
+ * @delta_q_uv_dc: indicates the UV DC quantizer relative to base_q_idx
+ * @delta_q_uv_ac indicates the UV AC quantizer relative to base_q_idx
+ * @padding: padding bytes to align things on 64 bits. Must be set to 0
+ *
+ * Encodes the quantization parameters. See section '7.2.9 Quantization

params
+ * syntax' of the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+struct v4l2_vp9_quantization {
+       __u8 base_q_idx;
+       __s8 delta_q_y_dc;
+       __s8 delta_q_uv_dc;
+       __s8 delta_q_uv_ac;
+       __u8 padding[4];
+};
+
+/**
+ * enum v4l2_vp9_segmentation_flags - VP9 segmentation flags
+ *
+ * @V4L2_VP9_SEGMENTATION_FLAG_ENABLED: indicates that this frame makes

use
of
+ *                                     the segmentation tool
+ * @V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP: indicates that the segmentation
map
+ *                                        should be updated during the
+ *                                        decoding of this frame
+ * @V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE: indicates that the updates
to
+ *                                             the segmentation map are coded
+ *                                             relative to the existing
+ *                                             segmentation map
+ * @V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA: indicates that new parameters
are
+ *                                         about to be specified for each
+ *                                         segment
+ * @V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE: indicates that the
+ *                                                 segmentation parameters
+ *                                                 represent the actual values
+ *                                                 to be used
+ *
+ * Those are the flags you should pass to &v4l2_vp9_segmentation.flags. See
+ * section '7.2.10 Segmentation params syntax' of the VP9 specification for
+ * more details.
+ */

include/media/vp9-ctrls.h new file +21 -0
+enum v4l2_vp9_segmentation_flags {
+       V4L2_VP9_SEGMENTATION_FLAG_ENABLED = 1 << 0,
+       V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP = 1 << 1,
+       V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE = 1 << 2,
+       V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA = 1 << 3,
+       V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE = 1 << 4,
+};
+
+#define V4L2_VP9_SEGMENT_FEATURE_ENABLED(id)   (1 << (id))
+#define V4L2_VP9_SEGMENT_FEATURE_ENABLED_MASK  0xf
+
+/**
+ * enum v4l2_vp9_segment_feature - VP9 segment feature IDs
+ *
+ * @V4L2_VP9_SEGMENT_FEATURE_QP_DELTA: QP delta segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_LF: loop filter segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_REF_FRAME: reference frame segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_SKIP: skip segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_CNT: number of segment features
+ *
+ * Segment feature IDs. See section '7.2.10 Segmentation params syntax'

of
the
+ * VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +16 -0
+enum v4l2_vp9_segment_feature {
+       V4L2_VP9_SEGMENT_FEATURE_QP_DELTA,
+       V4L2_VP9_SEGMENT_FEATURE_LF,
+       V4L2_VP9_SEGMENT_FEATURE_REF_FRAME,
+       V4L2_VP9_SEGMENT_FEATURE_SKIP,
+       V4L2_VP9_SEGMENT_FEATURE_CNT,
+};
+
+/**
+ * struct v4l2_vp9_segmentation - VP9 segmentation parameters
+ *
+ * @flags: combination of V4L2_VP9_SEGMENTATION_FLAG_* flags
+ * @tree_probs: specifies the probability values to be used when
+ *              decoding a Segment-ID. See '5.15. Segmentation map'
+ *              section of the VP9 specification for more details.
+ * @pred_prob: specifies the probability values to be used when

decoding a
+ *            Predicted-Segment-ID. See '6.4.14. Get segment id syntax'
+ *            section of :ref:vp9 for more details..
+ * @padding: padding used to make things aligned on 64 bits. Shall be zero
+ *          filled
+ * @feature_enabled: bitmask defining which features are enabled in each
+ *                  segment
+ * @feature_data: data attached to each feature. Data entry is only valid
if
+ *               the feature is enabled
+ *
+ * Encodes the quantization parameters. See section '7.2.10 Segmentation
+ * params syntax' of the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +24 -0
+struct v4l2_vp9_segmentation {
+       __u8 flags;
+       __u8 tree_probs[7];
+       __u8 pred_probs[3];
+       __u8 padding[5];
+       __u8 feature_enabled[8];
+       __s16 feature_data[8][4];
+};
+
+/**
+ * enum v4l2_vp9_intra_prediction_mode - VP9 Intra prediction modes
+ *
+ * @V4L2_VP9_INTRA_PRED_DC: DC intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_V: vertical intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_H: horizontal intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D45: D45 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D135: D135 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D117: D117 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D153: D153 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D207: D207 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D63: D63 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_TM: True Motion intra prediction
+ *
+ * See section '7.4.5 Intra frame mode info semantics' for more

details.
+ */

include/media/vp9-ctrls.h new file +23 -0
+enum v4l2_vp9_intra_prediction_mode {
+       V4L2_VP9_INTRA_PRED_MODE_DC,
+       V4L2_VP9_INTRA_PRED_MODE_V,
+       V4L2_VP9_INTRA_PRED_MODE_H,
+       V4L2_VP9_INTRA_PRED_MODE_D45,
+       V4L2_VP9_INTRA_PRED_MODE_D135,
+       V4L2_VP9_INTRA_PRED_MODE_D117,
+       V4L2_VP9_INTRA_PRED_MODE_D153,
+       V4L2_VP9_INTRA_PRED_MODE_D207,
+       V4L2_VP9_INTRA_PRED_MODE_D63,
+       V4L2_VP9_INTRA_PRED_MODE_TM,
+};
+
+/**
+ * struct v4l2_vp9_mv_probabilities - VP9 Motion vector probabilities
+ * @joint: motion vector joint probabilities
+ * @sign: motion vector sign probabilities
+ * @class: motion vector class probabilities
+ * @class0_bit: motion vector class0 bit probabilities
+ * @bits: motion vector bits probabilities
+ * @class0_fr: motion vector class0 fractional bit probabilities
+ * @fr: motion vector fractional bit probabilities
+ * @class0_hp: motion vector class0 high precision fractional bit

probabilities
+ * @hp: motion vector high precision fractional bit probabilities
+ */

include/media/vp9-ctrls.h new file +32 -0
+struct v4l2_vp9_mv_probabilities {
+       __u8 joint[3];
+       __u8 sign[2];
+       __u8 class[2][10];
+       __u8 class0_bit[2];
+       __u8 bits[2][10];
+       __u8 class0_fr[2][2][3];
+       __u8 fr[2][3];
+       __u8 class0_hp[2];
+       __u8 hp[2];
+};
+
+/**
+ * struct v4l2_vp9_probabilities - VP9 Probabilities
+ *
+ * @tx8: TX 8x8 probabilities
+ * @tx16: TX 16x16 probabilities
+ * @tx32: TX 32x32 probabilities
+ * @coef: coefficient probabilities
+ * @skip: skip probabilities
+ * @inter_mode: inter mode probabilities
+ * @interp_filter: interpolation filter probabilities
+ * @is_inter: is inter-block probabilities
+ * @comp_mode: compound prediction mode probabilities
+ * @single_ref: single ref probabilities
+ * @comp_ref: compound ref probabilities
+ * @y_mode: Y prediction mode probabilities
+ * @uv_mode: UV prediction mode probabilities
+ * @partition: partition probabilities
+ * @mv: motion vector probabilities
+ *
+ * Structure containing most VP9 probabilities. See the VP9

specification
+ * for more details.
+ */

include/media/vp9-ctrls.h new file +22 -0
+struct v4l2_vp9_probabilities {
+       __u8 tx8[2][1];
+       __u8 tx16[2][2];
+       __u8 tx32[2][3];
+       __u8 coef[4][2][2][6][6][3];
+       __u8 skip[3];
+       __u8 inter_mode[7][3];
+       __u8 interp_filter[4][2];
+       __u8 is_inter[4];
+       __u8 comp_mode[5];
+       __u8 single_ref[5][2];
+       __u8 comp_ref[5];
+       __u8 y_mode[4][9];
+       __u8 uv_mode[10][9];
+       __u8 partition[16][3];
+
+       struct v4l2_vp9_mv_probabilities mv;
+};
+
+/**
+ * enum v4l2_vp9_reset_frame_context - Valid values for
+ *                     &v4l2_ctrl_vp9_frame_decode_params-
Quoted 1 line

reset_frame_context

+ *
+ * @V4L2_VP9_RESET_FRAME_CTX_NONE: don't reset any frame context
+ * @V4L2_VP9_RESET_FRAME_CTX_SPEC: reset the frame context pointed by
+ *
&v4l2_ctrl_vp9_frame_decode_params.frame_context_idx
+ * @V4L2_VP9_RESET_FRAME_CTX_ALL: reset all frame contexts
+ *
+ * See section '7.2 Uncompressed header semantics' of the VP9 specification
+ * for more details.
+ */

include/media/vp9-ctrls.h new file +14 -0
+enum v4l2_vp9_reset_frame_context {
+       V4L2_VP9_RESET_FRAME_CTX_NONE,
+       V4L2_VP9_RESET_FRAME_CTX_SPEC,
+       V4L2_VP9_RESET_FRAME_CTX_ALL,
+};
+
+/**
+ * enum v4l2_vp9_interpolation_filter - VP9 interpolation filter types
+ *
+ * @V4L2_VP9_INTERP_FILTER_8TAP: height tap filter
+ * @V4L2_VP9_INTERP_FILTER_8TAP_SMOOTH: height tap smooth filter
+ * @V4L2_VP9_INTERP_FILTER_8TAP_SHARP: height tap sharp filter
+ * @V4L2_VP9_INTERP_FILTER_BILINEAR: bilinear filter
+ * @V4L2_VP9_INTERP_FILTER_SWITCHABLE: filter selection is signaled at

the
+ *                                    block level
+ *
+ * See section '7.2.7 Interpolation filter semantics' of the VP9 specification
+ * for more details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+enum v4l2_vp9_interpolation_filter {
+       V4L2_VP9_INTERP_FILTER_8TAP,
+       V4L2_VP9_INTERP_FILTER_8TAP_SMOOTH,
+       V4L2_VP9_INTERP_FILTER_8TAP_SHARP,
+       V4L2_VP9_INTERP_FILTER_BILINEAR,
+       V4L2_VP9_INTERP_FILTER_SWITCHABLE,
+};
+
+/**
+ * enum v4l2_vp9_reference_mode - VP9 reference modes
+ *
+ * @V4L2_VP9_REF_MODE_SINGLE: indicates that all the inter blocks use

only
a
+ *                           single reference frame to generate motion
+ *                           compensated prediction
+ * @V4L2_VP9_REF_MODE_COMPOUND: requires all the inter blocks to use compound
+ *                             mode. Single reference frame prediction is
not
+ *                             allowed
+ * @V4L2_VP9_REF_MODE_SELECT: allows each individual inter block to select
+ *                           between single and compound prediction modes
+ *
+ * See section '7.3.6 Frame reference mode semantics' of the VP9 specification
+ * for more details.
+ */

include/media/vp9-ctrls.h new file +14 -0
+enum v4l2_vp9_reference_mode {
+       V4L2_VP9_REF_MODE_SINGLE,
+       V4L2_VP9_REF_MODE_COMPOUND,
+       V4L2_VP9_REF_MODE_SELECT,
+};
+
+/**
+ * enum v4l2_vp9_tx_mode - VP9 TX modes
+ *
+ * @V4L2_VP9_TX_MODE_ONLY_4X4: transform size is 4x4
+ * @V4L2_VP9_TX_MODE_ALLOW_8X8: transform size can be up to 8x8
+ * @V4L2_VP9_TX_MODE_ALLOW_16X16: transform size can be up to 16x16
+ * @V4L2_VP9_TX_MODE_ALLOW_32X32: transform size can be up to 32x32
+ * @V4L2_VP9_TX_MODE_SELECT: bitstream contains transform size for each

block
+ *
+ * See section '7.3.1 Tx mode semantics' of the VP9 specification for more
+ * details.
+ */

include/media/vp9-ctrls.h new file +17 -0
+enum v4l2_vp9_tx_mode {
+       V4L2_VP9_TX_MODE_ONLY_4X4,
+       V4L2_VP9_TX_MODE_ALLOW_8X8,
+       V4L2_VP9_TX_MODE_ALLOW_16X16,
+       V4L2_VP9_TX_MODE_ALLOW_32X32,
+       V4L2_VP9_TX_MODE_SELECT,
+};
+
+/**
+ * enum v4l2_vp9_ref_id - VP9 Reference frame IDs
+ *
+ * @V4L2_REF_ID_LAST: last reference frame
+ * @V4L2_REF_ID_GOLDEN: golden reference frame
+ * @V4L2_REF_ID_ALTREF: alternative reference frame
+ * @V4L2_REF_ID_CNT: number of reference frames
+ *
+ * See section '7.4.12 Ref frames semantics' of the VP9 specification

for
more
+ * details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+enum v4l2_vp9_ref_id {
+       V4L2_REF_ID_LAST,
+       V4L2_REF_ID_GOLDEN,
+       V4L2_REF_ID_ALTREF,
+       V4L2_REF_ID_CNT,
+};
+
+/**
+ * enum v4l2_vp9_frame_flags - VP9 frame flags
+ * @V4L2_VP9_FRAME_FLAG_KEY_FRAME: the frame is a key frame
+ * @V4L2_VP9_FRAME_FLAG_SHOW_FRAME: the frame should be displayed
+ * @V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT: the decoding should be error

resilient
+ * @V4L2_VP9_FRAME_FLAG_INTRA_ONLY: the frame does not reference other frames
+ * @V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV: the frame might can high precision
+ *                                         motion vectors
+ * @V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX: frame context should be updated
+ *                                        after decoding
+ * @V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE: parallel decoding is used
+ * @V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING: vertical subsampling is enabled
+ * @V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING: horizontal subsampling is enabled
+ * @V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING: full UV range is used
+ *
+ * Check the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +17 -0
+enum v4l2_vp9_frame_flags {
+       V4L2_VP9_FRAME_FLAG_KEY_FRAME = 1 << 0,
+       V4L2_VP9_FRAME_FLAG_SHOW_FRAME = 1 << 1,
+       V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT = 1 << 2,
+       V4L2_VP9_FRAME_FLAG_INTRA_ONLY = 1 << 3,
+       V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV = 1 << 4,
+       V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX = 1 << 5,
+       V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE = 1 << 6,
+       V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING = 1 << 7,
+       V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING = 1 << 8,
+       V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING = 1 << 9,
+};
+
+#define V4L2_VP9_PROFILE_MAX           3
+
+/**
+ * struct v4l2_ctrl_vp9_frame_decode_params - VP9 frame decoding

control
+ *
+ * @flags: combination of V4L2_VP9_FRAME_FLAG_* flags
+ * @compressed_header_size: compressed header size in bytes
+ * @uncompressed_header_size: uncompressed header size in bytes
+ * @profile: VP9 profile. Can be 0, 1, 2 or 3
+ * @reset_frame_context: specifies whether the frame context should be reset
+ *                      to default values. See &v4l2_vp9_reset_frame_context
+ *                      for more details
+ * @frame_context_idx: frame context that should be used/updated
+ * @bit_depth: bits per components. Can be 8, 10 or 12. Note that not all
+ *            profiles support 10 and/or 12 bits depths
+ * @interpolation_filter: specifies the filter selection used for performing
+ *                       inter prediction. See &v4l2_vp9_interpolation_filter
+ *                       for more details
+ * @tile_cols_log2: specifies the base 2 logarithm of the width of each tile
+ *                 (where the width is measured in units of 8x8 blocks).
+ *                 Shall be less than or equal to 6
+ * @tile_rows_log2: specifies the base 2 logarithm of the height of each
tile
+ *                 (where the height is measured in units of 8x8 blocks)
+ * @tx_mode: specifies the TX mode. See &v4l2_vp9_tx_mode for more details
+ * @reference_mode: specifies the type of inter prediction to be used. See
+ *                 &v4l2_vp9_reference_mode for more details
+ * @padding: needed to make this struct 64 bit aligned. Shall be filled with
+ *          zeros
+ * @frame_width_minus_1: add 1 to it and you'll get the frame width expressed
+ *                      in pixels
+ * @frame_height_minus_1: add 1 to it and you'll get the frame height expressed
+ *                       in pixels
+ * @frame_width_minus_1: add 1 to it and you'll get the expected render width
+ *                      expressed in pixels. This is not used during the
+ *                      decoding process but might be used by HW scalers to
+ *                      prepare a frame that's ready for scanout
+ * @frame_height_minus_1: add 1 to it and you'll get the expected render
height
+ *                      expressed in pixels. This is not used during the
+ *                      decoding process but might be used by HW scalers to
+ *                      prepare a frame that's ready for scanout
+ * @refs: array of reference frames. See &v4l2_vp9_ref_id for more details
+ * @lf: loop filter parameters. See &v4l2_vp9_loop_filter for more details
+ * @quant: quantization parameters. See &v4l2_vp9_quantization for more details
+ * @seg: segmentation parameters. See &v4l2_vp9_segmentation for more details
+ * @probs: probabilities. See &v4l2_vp9_probabilities for more details
+ */
+struct v4l2_ctrl_vp9_frame_decode_params {
+       __u32 flags;
+       __u16 compressed_header_size;
+       __u16 uncompressed_header_size;
+       __u8 profile;
+       __u8 reset_frame_context;

Quoted 8 lines

+       __u8 frame_context_idx;
+       __u8 bit_depth;
+       __u8 interpolation_filter;
+       __u8 tile_cols_log2;
+       __u8 tile_rows_log2;
+       __u8 tx_mode;
+       __u8 reference_mode;
+       __u8 padding[6];

If my math is correct this should be 7.

Quoted 5 lines

+       __u16 frame_width_minus_1;
+       __u16 frame_height_minus_1;
+       __u16 render_width_minus_1;
+       __u16 render_height_minus_1;
+       __u64 refs[V4L2_REF_ID_CNT];

Why are we using u64s to store these [0..7] indices? Or are we storing more than that?

Quoted 5 lines

+       struct v4l2_vp9_loop_filter lf;
+       struct v4l2_vp9_quantization quant;
+       struct v4l2_vp9_segmentation seg;
+       struct v4l2_vp9_probabilities probs;
+};

I cannot find a trace of ref_frame_sign_biases, which is also part of the uncompressed header, in this structure. Is it missing, or is this information provided in a different way?

Quoted 8 lines
Quoted 5 lines

This seems missing indeed. Interestingly the Rockchip reference software does
not seem to pass this information to the hardware. On the other end, the VC8000D
software does. It is also part of DXVA, VA and NVDEC APIs.

I can think of at least one piece of hardware that will require this information, so it would be nice to have it included.

We'll had, the more we look at the spec, the less we understand how rkvdec manages proper decoding without.

Quoted 3 lines

Cheers,
Alex.

_______________________________________________
Linux-rockchip mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip

#11 lore raw

Re: [PATCH v4 2/3] media: uapi: Add VP9 stateless decoder controls

Quoted 125 lines · Nicolas Dufresne
Quoted 123 lines · Alexandre Courbot

Hi Nicolas,

Quoted 118 lines · Nicolas Dufresne
Quoted 80 lines · Alexandre Courbot

Hi Ezequiel, sorry for the late review!

Quoted 76 lines · Ezequiel Garcia

From: Boris Brezillon <[email protected]>

Add the VP9 stateless decoder controls plus the documentation that goes with it.

---
 .../userspace-api/media/v4l/biblio.rst        |  10 +
 .../media/v4l/ext-ctrls-codec.rst             | 550 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ctrls.c          | 239 ++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   1 +
 include/media/v4l2-ctrls.h                    |   1 +
 include/media/vp9-ctrls.h                     | 485 +++++++++++++++
 6 files changed, 1286 insertions(+)
 create mode 100644 include/media/vp9-ctrls.h
Documentation/userspace-api/media/v4l/biblio.rst +10 -0
index 3c9634173e82..e09102e572fd 100644
@@ -414,3 +414,13 @@ VP8
 :title:     RFC 6386: "VP8 Data Format and Decoding Guide"
 
 :author:    J. Bankoski et al.
+
+.. _vp9:
+
+VP9
+===
+
+
+:title:     VP9 Bitstream & Decoding Process Specification
+
+:author:    Adrian Grange (Google), Peter de Rivaz (Argon Design),

Jonathan
Hunt (Argon Design)

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +0 -0
index d0d506a444b1..5c5f7dd868da 100644
@@ -2668,6 +2668,556 @@ enum

v4l2_mpeg_video_h264_hierarchical_coding_type -
- ``padding[3]``
- Applications and drivers must set this to zero.

+.. _v4l2-mpeg-vp9:

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +3 -0
+
+``V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(0..3) (struct)``
+    Stores VP9 probabilities attached to a specific frame context. The

VP9
+ specification allows using a maximum of 4 contexts. Each frame
being

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +8 -0
+    decoded refers to one of those context. See section '7.1.2 Refresh
+    probs semantics' section of :ref:`vp9` for more details about these
+    contexts.
+
+    This control is bi-directional:
+
+    * all 4 contexts must be initialized by userspace just after the
+      stream is started and before the first decoding request is

submitted.
+ * the referenced context might be read by the kernel when a
decoding
+ request is submitted, and will be updated after the decoder is
done
+ decoding the frame if the V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX
flag

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +2 -0
+      is set.
+    * contexts will be read back by user space before each decoding

request

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +2 -0
+      to retrieve the updated probabilities.
+    * userspace will re-initialize the context to their default values
Quoted 8 lines
Quoted 2 lines

when
+ a reset context is required.

Just to make sure I understand this part correctly, it means that if frame A and B use the same context, and frame A has V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX set, then user-space must wait for frame A to get dequeued and read back this control from the completed request before it can submit frame B?

We are preparing a new version, we believe it was an API mistake to try and share the probability update between kernel and userspace. It's actually worst,
you really need to push/pull in a lock step, regardless of the frame context id.

As you may know, the probabilities are used to parse the compressed part of the
stream (also know as entropy decoding). They are probability of a bitstream symbol of being 0 or 1. On RK3399, the hardware does not parse the compressed
header. The compressed headers contains the bitstream coded updates to the probabilities. The probabilities are also updated base on the count of specific
symbols found in the bitstream during decoding, this is currently done in the
driver (counts are returned, and driver updates the probabilities from that).

What we found is that the symbols in the compress headers are coded with fixed
probabilities, meaning we don't need the probabilities to parse the compressed
headers. Our proposal will be to keep doing the compressed header parsing in userspace and only store the update value (as found in inv_map_table) to the kernel. The driver will be responsible of applying this to the current probabilities along with doing the probability saving process.

Quoted 7 lines
Quoted 4 lines

So if I am following:

1) The compressed headers can be decompressed using just the entropy information contained in the previous compressed headers,

Using just the entropy contained in the spec. The entropy for this one is static.

Even better then, thanks for the clarification.

Quoted 9 lines
Quoted 3 lines

2) The frame data also requires the updated probabilities from the decoding of the previous frames on top of 1)

Yes, the entropy decoding of the frame data requires default tables, the deltas from the compressed headers, and for non-keyframe the counts of symbols in the previous decoding.

All this is being moved inside the kernel.

Sounds good to me. I guess that unless we want to support a hypothetical decoder that cannot do frame entropy decoding (and such a decoder does not exist AFAICT, nor would it make much sense) we don't need to make that information available to user-space.

Quoted 1,583 lines
Quoted 4 lines

Meaning that user-space can decompress the headers without any feedback from the kernel. Am I understanding correctly?

Correct. We have some really eirly code demonstrating it (will all be published LGPL V2+ and submitted for including in GStreamer). In FFMPEG this is know as VP5/5 Range decoder, so nothing new here.

Quoted 20 lines
Quoted 6 lines

With this change, we should be able to only push data to the decoder without having to read this control. We believe this will allow better use of the VB2
queue and enhance the decoding performance.

That sounds pretty good to me.

Quoted 10 lines

Please note that this hardware design looks like a mistake. It seems that with
widevine, the compressed header is encrypted. As in some implementation the decrypted bitstream is no visible to the CPU, it will not be possible to perform
the needed parsing. An offload to a TEE would be needed. We know that Rockchip
have fixed this issue in new decoder, notably on RK3368. The probability processing is done in HW.

Quoted 2 lines

Mmm a solution would be not to encrypt the compressed header, but I guess that's not how things work, right?

Widewine, Playready and other folks will always dictate their way. I guess I only wanted to mention publicly so that folks visiting this are not too surprised of the detour and contorsion needed. We are not implementing this as we speak. For the browser version of Widevine, I don't think this issue will raise.

Quoted 4 lines

Worst case the TEE could indeed copy the decrypted and decompressed somewhere for the CPU to read, but that would complicate things a tiny bit.

Exact, there "exist a solution" ;-P From a API perspective, we simply visited this as we where told it was not possible at all, so we were curious upon why. And this is slightly specific to this HW design, newer Rockchip design handle compress header decoding in HW, so I suspect other vendors will follow this path in the future.

Quoted 559 lines
Quoted 559 lines
Quoted 542 lines
Quoted 542 lines
Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +4 -0
+
+    .. note::
+
+       This compound control is not yet part of the public kernel API

and

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +94 -0
+       it is expected to change.
+
+.. c:type:: v4l2_ctrl_vp9_frame_ctx
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{5.8cm}|p{4.8cm}|p{6.6cm}|
+
+.. flat-table:: struct v4l2_ctrl_vp9_frame_ctx
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - struct :c:type:`v4l2_vp9_probabilities`
+      - ``probs``
+      - Structure with VP9 probabilities attached to the context.
+
+.. c:type:: v4l2_vp9_probabilities
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_probabilities
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``tx8[2][1]``
+      - TX 8x8 probabilities.
+    * - __u8
+      - ``tx16[2][2]``
+      - TX 16x16 probabilities.
+    * - __u8
+      - ``tx32[2][3]``
+      - TX 32x32 probabilities.
+    * - __u8
+      - ``coef[4][2][2][6][6][3]``
+      - Coefficient probabilities.
+    * - __u8
+      - ``skip[3]``
+      - Skip probabilities.
+    * - __u8
+      - ``inter_mode[7][3]``
+      - Inter prediction mode probabilities.
+    * - __u8
+      - ``interp_filter[4][2]``
+      - Interpolation filter probabilities.
+    * - __u8
+      - ``is_inter[4]``
+      - Is inter-block probabilities.
+    * - __u8
+      - ``comp_mode[5]``
+      - Compound prediction mode probabilities.
+    * - __u8
+      - ``single_ref[5][2]``
+      - Single reference probabilities.
+    * - __u8
+      - ``comp_mode[5]``
+      - Compound reference probabilities.
+    * - __u8
+      - ``y_mode[4][9]``
+      - Y prediction mode probabilities.
+    * - __u8
+      - ``uv_mode[10][9]``
+      - UV prediction mode probabilities.
+    * - __u8
+      - ``partition[16][3]``
+      - Partition probabilities.
+    * - __u8
+      - ``mv.joint[3]``
+      - Motion vector joint probabilities.
+    * - __u8
+      - ``mv.sign[2]``
+      - Motion vector sign probabilities.
+    * - __u8
+      - ``mv.class[2][10]``
+      - Motion vector class probabilities.
+    * - __u8
+      - ``mv.class0_bit[2]``
+      - Motion vector class0 bit probabilities.
+    * - __u8
+      - ``mv.bits[2][10]``
+      - Motion vector bits probabilities.
+    * - __u8
+      - ``mv.class0_fr[2][2][3]``
+      - Motion vector class0 fractional bit probabilities.
+    * - __u8
+      - ``mv.fr[2][3]``
+      - Motion vector fractional bit probabilities.
+    * - __u8
+      - ``mv.class0_hp[2]``
+      - Motion vector class0 high precision fractional bit

probabilities.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +6 -0
+    * - __u8
+      - ``mv.hp[2]``
+      - Motion vector high precision fractional bit probabilities.
+
+``V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS (struct)``
+    Specifies the frame parameters for the associated VP9 frame decode

request.
+ This includes the necessary parameters for configuring a stateless
hardware
+ decoding pipeline for VP9. The bitstream parameters are defined
according

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +5 -0
+    to :ref:`vp9`.
+
+    .. note::
+
+       This compound control is not yet part of the public kernel API

and

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +29 -0
+       it is expected to change.
+
+.. c:type:: v4l2_ctrl_vp9_frame_decode_params
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_ctrl_vp9_frame_decode_params
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``flags``
+      - Combination of V4L2_VP9_FRAME_FLAG_* flags. See
+        :c:type:`v4l2_vp9_frame_flags`.
+    * - __u16
+      - ``compressed_header_size``
+      - Compressed header size in bytes.
+    * - __u16
+      - ``uncompressed_header_size``
+      - Uncompressed header size in bytes.
+    * - __u8
+      - ``profile``
+      - VP9 profile. Can be 0, 1, 2 or 3.
+    * - __u8
+      - ``reset_frame_context``
+      - Frame context that should be used/updated when decoding the

frame.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +3 -0
+    * - __u8
+      - ``bit_depth``
+      - Component depth in bits. Must be 8 for profile 0 and 1. Must 10

or
12

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +4 -0
+        for profile 2 and 3.
+    * - __u8
+      - ``interpolation_filter``
+      - Specifies the filter selection used for performing inter

prediction. See

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +4 -0
+        :c:type:`v4l2_vp9_interpolation_filter`.
+    * - __u8
+      - ``tile_cols_log2``
+      - Specifies the base 2 logarithm of the width of each tile (where

the
+ width is measured in units of 8x8 blocks). Shall be less than
or
equal

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +4 -0
+        to 6.
+    * - __u8
+      - ``tile_rows_log2``
+      - Specifies the base 2 logarithm of the height of each tile

(where
the

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +11 -0
+        height is measured in units of 8x8 blocks)
+    * - __u8
+      - ``tx_mode``
+      - Specifies the TX mode. See :c:type:`v4l2_vp9_tx_mode`.
+    * - __u8
+      - ``reference_mode``
+      - Specifies the type of inter prediction to be used. See
+        :c:type:`v4l2_vp9_reference_mode`.
+    * - __u8
+      - ``padding``
+      - Needed to make this struct 64 bit aligned. Shall be filled with

zeroes.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +9 -0
+    * - __u16
+      - ``frame_width_minus_1``
+      - Add 1 to get the frame width expressed in pixels.
+    * - __u16
+      - ``frame_height_minus_1``
+      - Add 1 to to get the frame height expressed in pixels.
+    * - __u16
+      - ``frame_width_minus_1``
+      - Add 1 to to get the expected render width expressed in pixels.

This
is
+ not used during the decoding process but might be used by HW
scalers to

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +4 -0
+        prepare a frame that's ready for scanout.
+    * - __u16
+      - frame_height_minus_1
+      - Add 1 to get the expected render height expressed in pixels.

This
is
+ not used during the decoding process but might be used by HW
scalers to

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +7 -0
+        prepare a frame that's ready for scanout.
+    * - __u64
+      - ``refs[3]``
+      - Array of reference frame timestamps.
+    * - struct :c:type:`v4l2_vp9_loop_filter`
+      - ``lf``
+      - Loop filter parameters. See struct

:c:type:v4l2_vp9_loop_filter.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +97 -0
+    * - struct :c:type:`v4l2_vp9_quantization`
+      - ``quant``
+      - Quantization parameters. See :c:type:`v4l2_vp9_quantization`.
+    * - struct :c:type:`v4l2_vp9_segmentation`
+      - ``seg``
+      - Segmentation parameters. See :c:type:`v4l2_vp9_segmentation`.
+    * - struct :c:type:`v4l2_vp9_probabilities`
+      - ``probs``
+      - Probabilities. See :c:type:`v4l2_vp9_probabilities`.
+
+.. c:type:: v4l2_vp9_frame_flags
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_frame_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_FRAME_FLAG_KEY_FRAME``
+      - The frame is a key frame.
+    * - ``V4L2_VP9_FRAME_FLAG_SHOW_FRAME``
+      - The frame should be displayed.
+    * - ``V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT``
+      - The decoding should be error resilient.
+    * - ``V4L2_VP9_FRAME_FLAG_INTRA_ONLY``
+      - The frame does not reference other frames.
+    * - ``V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV``
+      - the frame might can high precision motion vectors.
+    * - ``V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX``
+      - Frame context should be updated after decoding.
+    * - ``V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE``
+      - Parallel decoding is used.
+    * - ``V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING``
+      - Vertical subsampling is enabled.
+    * - ``V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING``
+      - Horizontal subsampling is enabled.
+    * - ``V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING``
+      - The full UV range is used.
+
+.. c:type:: v4l2_vp9_ref_id
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_ref_id
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_REF_ID_LAST``
+      - Last reference frame.
+    * - ``V4L2_REF_ID_GOLDEN``
+      - Golden reference frame.
+    * - ``V4L2_REF_ID_ALTREF``
+      - Alternative reference frame.
+    * - ``V4L2_REF_ID_CNT``
+      - Number of reference frames.
+
+.. c:type:: v4l2_vp9_tx_mode
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_tx_mode
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_TX_MODE_ONLY_4X4``
+      - Transform size is 4x4.
+    * - ``V4L2_VP9_TX_MODE_ALLOW_8X8``
+      - Transform size can be up to 8x8.
+    * - ``V4L2_VP9_TX_MODE_ALLOW_16X16``
+      - Transform size can be up to 16x16.
+    * - ``V4L2_VP9_TX_MODE_ALLOW_32X32``
+      - transform size can be up to 32x32.
+    * - ``V4L2_VP9_TX_MODE_SELECT``
+      - Bitstream contains transform size for each block.
+
+.. c:type:: v4l2_vp9_reference_mode
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_reference_mode
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_REF_MODE_SINGLE``
+      - Indicates that all the inter blocks use only a single reference

frame

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +3 -0
+        to generate motion compensated prediction.
+    * - ``V4L2_VP9_REF_MODE_COMPOUND``
+      - Requires all the inter blocks to use compound mode. Single

reference

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +96 -0
+        frame prediction is not allowed.
+    * - ``V4L2_VP9_REF_MODE_SELECT``
+      - Allows each individual inter block to select between single and
+        compound prediction modes.
+
+.. c:type:: v4l2_vp9_interpolation_filter
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_interpolation_filter
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_INTERP_FILTER_8TAP``
+      - Height tap filter.
+    * - ``V4L2_VP9_INTERP_FILTER_8TAP_SMOOTH``
+      - Height tap smooth filter.
+    * - ``V4L2_VP9_INTERP_FILTER_8TAP_SHARP``
+      - Height tap sharp filter.
+    * - ``V4L2_VP9_INTERP_FILTER_BILINEAR``
+      - Bilinear filter.
+    * - ``V4L2_VP9_INTERP_FILTER_SWITCHABLE``
+      - Filter selection is signaled at the block level.
+
+.. c:type:: v4l2_vp9_reset_frame_context
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_reset_frame_context
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_RESET_FRAME_CTX_NONE``
+      - Do not reset any frame context.
+    * - ``V4L2_VP9_RESET_FRAME_CTX_SPEC``
+      - Reset the frame context pointed by
+        :c:type:`v4l2_ctrl_vp9_frame_decode_params`.frame_context_idx.
+    * - ``V4L2_VP9_RESET_FRAME_CTX_ALL``
+      - Reset all frame contexts.
+
+.. c:type:: v4l2_vp9_intra_prediction_mode
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_intra_prediction_mode
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_INTRA_PRED_DC``
+      - DC intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_V``
+      - Vertical intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_H``
+      - Horizontal intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D45``
+      - D45 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D135``
+      - D135 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D117``
+      - D117 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D153``
+      - D153 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D207``
+      - D207 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_D63``
+      - D63 intra prediction.
+    * - ``V4L2_VP9_INTRA_PRED_MODE_TM``
+      - True motion intra prediction.
+
+.. c:type:: v4l2_vp9_segmentation
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_segmentation
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``flags``
+      - Combination of V4L2_VP9_SEGMENTATION_FLAG_* flags. See
+        :c:type:`v4l2_vp9_segmentation_flags`.
+    * - __u8
+      - ``tree_probs[7]``
+      - Specifies the probability values to be used when decoding a

Segment-ID.
+ See '5.15. Segmentation map' section of :ref:vp9 for more
details.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +8 -0
+    * - __u8
+      - ``pred_prob[3]``
+      - Specifies the probability values to be used when decoding a
+        Predicted-Segment-ID. See '6.4.14. Get segment id syntax'
+        section of :ref:`vp9` for more details.
+    * - __u8
+      - ``padding[5]``
+      - Used to align this struct on 64 bit. Shall be filled with

zeroes.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +6 -0
+    * - __u8
+      - ``feature_enabled[8]``
+      - Bitmask defining which features are enabled in each segment.
+    * - __u8
+      - ``feature_data[8][4]``
+      - Data attached to each feature. Data entry is only valid if the

feature

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +39 -0
+        is enabled.
+
+.. c:type:: v4l2_vp9_segment_feature
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_segment_feature
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_SEGMENT_FEATURE_QP_DELTA``
+      - QP delta segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_LF``
+      - Loop filter segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_REF_FRAME``
+      - Reference frame segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_SKIP``
+      - Skip segment feature.
+    * - ``V4L2_VP9_SEGMENT_FEATURE_CNT``
+      - Number of segment features.
+
+.. c:type:: v4l2_vp9_segmentation_flags
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_segmentation_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_ENABLED``
+      - Indicates that this frame makes use of the segmentation tool.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP``
+      - Indicates that the segmentation map should be updated during

the

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +6 -0
+        decoding of this frame.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE``
+      - Indicates that the updates to the segmentation map are coded
+        relative to the existing segmentation map.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA``
+      - Indicates that new parameters are about to be specified for

each

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +3 -0
+        segment.
+    * - ``V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE``
+      - Indicates that the segmentation parameters represent the actual

values

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +28 -0
+        to be used.
+
+.. c:type:: v4l2_vp9_quantization
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_quantization
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``base_q_idx``
+      - Indicates the base frame qindex.
+    * - __s8
+      - ``delta_q_y_dc``
+      - Indicates the Y DC quantizer relative to base_q_idx.
+    * - __s8
+      - ``delta_q_uv_dc``
+      - Indicates the UV DC quantizer relative to base_q_idx.
+    * - __s8
+      - ``delta_q_uv_ac``
+      - Indicates the UV AC quantizer relative to base_q_idx.
+    * - __u8
+      - ``padding[4]``
+      - Padding bytes used to align this struct on 64 bit. Must be set

to
0.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +25 -0
+
+.. c:type:: v4l2_vp9_loop_filter
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: struct v4l2_vp9_loop_filter
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u8
+      - ``flags``
+      - Combination of V4L2_VP9_LOOP_FILTER_FLAG_* flags.
+        See :c:type:`v4l2_vp9_loop_filter_flags`.
+    * - __u8
+      - ``level``
+      - Indicates the loop filter strength.
+    * - __u8
+      - ``sharpness``
+      - Indicates the sharpness level.
+    * - __s8
+      - ``ref_deltas[4]``
+      - Contains the adjustment needed for the filter level based on

the
chosen

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +2 -0
+        reference frame.
+    * - __s8
Quoted 8 lines

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 3200

Quoted 8 lines
Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +2 -0
+      - ``mode_deltas[2]``
+      - Contains the adjustment needed for the filter level based on

the
chosen

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +4 -0
+        mode
+    * - __u8
+      - ``level_lookup[8][4][2]``
+      - Level lookup table.

We think it might be a poor choice to ask for computed values. According to "8.8.1 Loop filter frame init process" this is computed from loop_filter_level,
loop_filter_ref_deltas[], V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE, feature_data[], ref_deltas[] and mode_deltas[]. We believe it is likely a waste
to pass this information assuming some HW could do that process.

Quoted 8 lines
Quoted 3 lines

This is to be discussed. None of the other statelss APIs seems to require this
value to be computed by userspace.

Not particularly opinionated, but it would be a waste if the driver needs to reverse that computation to pass it to the hardware. If the lookup table is not difficult to compute, we can maybe do this using a helper function for drivers that need it?

Correct, the spec is quite clear, and gives the exact algorithm to produce that table. If needs, we can do like we did fro H264 intermediate reference lists and add a shared helper.

Quoted 962 lines
Quoted 961 lines
Quoted 952 lines
Quoted 461 lines
Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +15 -0
+
+
+.. c:type:: v4l2_vp9_loop_filter_flags
+
+.. cssclass:: longtable
+
+.. tabularcolumns:: |p{1.5cm}|p{6.3cm}|p{9.4cm}|
+
+.. flat-table:: enum v4l2_vp9_loop_filter_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 2
+
+    * - ``V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED``
+      - When set, the filter level depends on the mode and reference

frame
used

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +3 -0
+        to predict a block.
+    * - ``V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE``
+      - When set, the bitstream contains additional syntax elements

that
+ specify which mode and reference frame deltas are to be
updated.

Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +1 -0
+
 .. raw:: latex
 
     \normalsize
drivers/media/v4l2- renamed from drivers/media/v4l2-core/v4l2-ctrls.c +0 -0

core/v4l2-ctrls.c
index 1c617b42a944..115beadaf02d 100644

drivers/media/v4l2-core/v4l2-ctrls.c +0 -0
@@ -930,6 +930,11 @@ const char *v4l2_ctrl_get_name(u32 id)
        case V4L2_CID_MPEG_VIDEO_VP8_PROFILE:                   return

"VP8
Profile";
case V4L2_CID_MPEG_VIDEO_VP9_PROFILE: return
"VP9
Profile";
case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER: return
"VP8
Frame Header";
+ case V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS: return
"VP9
Frame Decode Parameters";
+ case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(0): return
"VP9
Frame Context 0";
+ case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(1): return
"VP9
Frame Context 1";
+ case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(2): return
"VP9
Frame Context 2";
+ case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(3): return
"VP9
Frame Context 3";

/* HEVC controls */
case V4L2_CID_MPEG_VIDEO_HEVC_I_FRAME_QP: return
"HEVC
I-Frame QP Value";

drivers/media/v4l2-core/v4l2-ctrls.c +0 -0
@@ -1403,6 +1408,15 @@ void v4l2_ctrl_fill(u32 id, const char **name,

enum
v4l2_ctrl_type *type,

        case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
                *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
                break;
drivers/media/v4l2-core/v4l2-ctrls.c +105 -0
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS:
+               *type = V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS;
+               break;
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(0):
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(1):
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(2):
+       case V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(3):
+               *type = V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT;
+               break;
        case V4L2_CID_MPEG_VIDEO_HEVC_SPS:
                *type = V4L2_CTRL_TYPE_HEVC_SPS;
                break;
@@ -1703,6 +1717,219 @@ static void std_log(const struct v4l2_ctrl
 *ctrl)
        0;                                                      \
 })
 
+static int
+validate_vp9_lf_params(struct v4l2_vp9_loop_filter *lf)
+{
+       unsigned int i, j, k;
+
+       if (lf->flags &
+           ~(V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED |
+             V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED implies
+        * V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE.
+        */
+       if (lf->flags & V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE &&
+           !(lf->flags & V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED))
+               return -EINVAL;
+
+       /* That all values are in the accepted range. */
+       if (lf->level > GENMASK(5, 0))
+               return -EINVAL;
+
+       if (lf->sharpness > GENMASK(2, 0))
+               return -EINVAL;
+
+       for (i = 0; i < ARRAY_SIZE(lf->ref_deltas); i++) {
+               if (lf->ref_deltas[i] < -63 || lf->ref_deltas[i] > 63)
+                       return -EINVAL;
+       }
+
+       for (i = 0; i < ARRAY_SIZE(lf->mode_deltas); i++) {
+               if (lf->mode_deltas[i] < -63 || lf->mode_deltas[i] > 63)
+                       return -EINVAL;
+       }
+
+       for (i = 0; i < ARRAY_SIZE(lf->level_lookup); i++) {
+               for (j = 0; j < ARRAY_SIZE(lf->level_lookup[0]); j++) {
+                       for (k = 0; k < ARRAY_SIZE(lf-
 >level_lookup[0][0]);
 k++) {
+                               if (lf->level_lookup[i][j][k] > 63)
+                                       return -EINVAL;
+                       }
+               }
+       }
+
+       return 0;
+}
+
+static int
+validate_vp9_quant_params(struct v4l2_vp9_quantization *quant)
+{
+       if (quant->delta_q_y_dc < -15 || quant->delta_q_y_dc > 15 ||
+           quant->delta_q_uv_dc < -15 || quant->delta_q_uv_dc > 15 ||
+           quant->delta_q_uv_ac < -15 || quant->delta_q_uv_ac > 15)
+               return -EINVAL;
+
+       memset(quant->padding, 0, sizeof(quant->padding));
+       return 0;
+}
+
+static int
+validate_vp9_seg_params(struct v4l2_vp9_segmentation *seg)
+{
+       unsigned int i, j;
+
+       if (seg->flags &
+           ~(V4L2_VP9_SEGMENTATION_FLAG_ENABLED |
+             V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP |
+             V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE |
+             V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA |
+             V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP and
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA imply
+        * V4L2_VP9_SEGMENTATION_FLAG_ENABLED.
+        */
+       if ((seg->flags &
+            (V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP |
+             V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA)) &&
+           !(seg->flags & V4L2_VP9_SEGMENTATION_FLAG_ENABLED))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE implies
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP.
+        */
+       if (seg->flags & V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE &&
+           !(seg->flags & V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP))
+               return -EINVAL;
+
+       /*
+        * V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE implies
+        * V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA.
+        */
+       if (seg->flags & V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE

&&

drivers/media/v4l2-core/v4l2-ctrls.c +28 -0
+           !(seg->flags & V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA))
+               return -EINVAL;
+
+       for (i = 0; i < ARRAY_SIZE(seg->feature_enabled); i++) {
+               if (seg->feature_enabled[i] &
+                   ~(V4L2_VP9_SEGMENT_FEATURE_QP_DELTA |
+                     V4L2_VP9_SEGMENT_FEATURE_LF |
+                     V4L2_VP9_SEGMENT_FEATURE_REF_FRAME |
+                     V4L2_VP9_SEGMENT_FEATURE_SKIP))
+                       return -EINVAL;
+       }
+
+       for (i = 0; i < ARRAY_SIZE(seg->feature_data); i++) {
+               const int range[] = {255, 63, 3, 0};
+
+               for (j = 0; j < ARRAY_SIZE(seg->feature_data[j]); j++) {
+                       if (seg->feature_data[i][j] < -range[j] ||
+                           seg->feature_data[i][j] > range[j])
+                               return -EINVAL;
+               }
+       }
+
+       memset(seg->padding, 0, sizeof(seg->padding));
+       return 0;
+}
+
+static int
+validate_vp9_frame_decode_params(struct

v4l2_ctrl_vp9_frame_decode_params
*dec_params)

drivers/media/v4l2-core/v4l2-ctrls.c +19 -0
+{
+       int ret;
+
+       /* Make sure we're not passed invalid flags. */
+       if (dec_params->flags &
+           ~(V4L2_VP9_FRAME_FLAG_KEY_FRAME |
+             V4L2_VP9_FRAME_FLAG_SHOW_FRAME |
+             V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT |
+             V4L2_VP9_FRAME_FLAG_INTRA_ONLY |
+             V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV |
+             V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX |
+             V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE |
+             V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING |
+             V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING |
+             V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING))
+               return -EINVAL;
+
+       /*
+        * The refresh context and error resilient flags are mutually

exclusive.

drivers/media/v4l2-core/v4l2-ctrls.c +12 -0
+        * Same goes for parallel decoding and error resilient modes.
+        */
+       if (dec_params->flags & V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT &&
+           dec_params->flags &
+           (V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX |
+            V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE))
+               return -EINVAL;
+
+       if (dec_params->profile > V4L2_VP9_PROFILE_MAX)
+               return -EINVAL;
+
+       if (dec_params->reset_frame_context >

V4L2_VP9_RESET_FRAME_CTX_ALL)

drivers/media/v4l2-core/v4l2-ctrls.c +7 -0
+               return -EINVAL;
+
+       if (dec_params->frame_context_idx >= V4L2_VP9_NUM_FRAME_CTX)
+               return -EINVAL;
+
+       /*
+        * Profiles 0 and 1 only support 8-bit depth, profiles 2 and 3

only
10

drivers/media/v4l2-core/v4l2-ctrls.c +5 -0
+        * and 12 bit depths.
+        */
+       if ((dec_params->profile < 2 && dec_params->bit_depth != 8) ||
+           (dec_params->profile >= 2 &&
+            (dec_params->bit_depth != 10 && dec_params->bit_depth !=

12)))

drivers/media/v4l2-core/v4l2-ctrls.c +15 -0
+               return -EINVAL;
+
+       /* Profile 0 and 2 only accept YUV 4:2:0. */
+       if ((dec_params->profile == 0 || dec_params->profile == 2) &&
+           (!(dec_params->flags & V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING) ||
+            !(dec_params->flags & V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING)))
+               return -EINVAL;
+
+       /* Profile 1 and 3 only accept YUV 4:2:2, 4:4:0 and 4:4:4. */
+       if ((dec_params->profile == 1 || dec_params->profile == 3) &&
+           ((dec_params->flags & V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING) &&
+            (dec_params->flags & V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING)))
+               return -EINVAL;
+
+       if (dec_params->interpolation_filter >

V4L2_VP9_INTERP_FILTER_SWITCHABLE)

drivers/media/v4l2-core/v4l2-ctrls.c +4 -0
+               return -EINVAL;
+
+       /*
+        * According to the spec, tile_cols_log2 shall be less than or

equal

drivers/media/v4l2-core/v4l2-ctrls.c +39 -0
+        * to 6.
+        */
+       if (dec_params->tile_cols_log2 > 6)
+               return -EINVAL;
+
+       if (dec_params->tx_mode > V4L2_VP9_TX_MODE_SELECT)
+               return -EINVAL;
+
+       if (dec_params->reference_mode > V4L2_VP9_REF_MODE_SELECT)
+               return -EINVAL;
+
+       ret = validate_vp9_lf_params(&dec_params->lf);
+       if (ret)
+               return ret;
+
+       ret = validate_vp9_quant_params(&dec_params->quant);
+       if (ret)
+               return ret;
+
+       ret = validate_vp9_seg_params(&dec_params->seg);
+       if (ret)
+               return ret;
+
+       memset(dec_params->padding, 0, sizeof(dec_params->padding));
+       return 0;
+}
+
 /* Validate a new control */
 
 #define zero_padding(s) \
@@ -1799,6 +2026,12 @@ static int std_validate_compound(const struct
 v4l2_ctrl *ctrl, u32 idx,
                zero_padding(p_vp8_frame_header->coder_state);
                break;
 
+       case V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS:
+               return validate_vp9_frame_decode_params(p);
+
+       case V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT:
+               break;
+
        case V4L2_CTRL_TYPE_HEVC_SPS:
                p_hevc_sps = p;
@@ -2542,6 +2775,12 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct
 v4l2_ctrl_handler *hdl,
        case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
                elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header);
                break;
+       case V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT:
+               elem_size = sizeof(struct v4l2_ctrl_vp9_frame_ctx);
+               break;
+       case V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS:
+               elem_size = sizeof(struct
 v4l2_ctrl_vp9_frame_decode_params);
+               break;
        case V4L2_CTRL_TYPE_HEVC_SPS:
                elem_size = sizeof(struct v4l2_ctrl_hevc_sps);
                break;
drivers/media/v4l2- renamed from drivers/media/v4l2-core/v4l2-ioctl.c +0 -0

core/v4l2-ioctl.c
index 2322f08a98be..4fe77ad917c8 100644

drivers/media/v4l2-core/v4l2-ioctl.c +1 -0
@@ -1423,6 +1423,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc
 *fmt)
                case V4L2_PIX_FMT_VP8:          descr = "VP8"; break;
                case V4L2_PIX_FMT_VP8_FRAME:    descr = "VP8 Frame";
 break;
                case V4L2_PIX_FMT_VP9:          descr = "VP9"; break;
+               case V4L2_PIX_FMT_VP9_FRAME:    descr = "VP9 Frame";
 break;
                case V4L2_PIX_FMT_HEVC:         descr = "HEVC"; break;

/*
aka H.265 */
case V4L2_PIX_FMT_HEVC_SLICE: descr = "HEVC Parsed
Slice

Data"; break;
                case V4L2_PIX_FMT_FWHT:         descr = "FWHT"; break;

/*
used in vicodec */

include/media/v4l2-ctrls.h +1 -0
index 757a713bad41..2de8290f9f11 100644
@@ -21,6 +21,7 @@
 #include <media/fwht-ctrls.h>
 #include <media/h264-ctrls.h>
 #include <media/vp8-ctrls.h>
+#include <media/vp9-ctrls.h>
 #include <media/hevc-ctrls.h>
 
 /* forward references */
include/media/vp9-ctrls.h new file +7 -0
new file mode 100644
index 000000000000..0cdea8a18b72
@@ -0,0 +1,485 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * These are the VP9 state controls for use with stateless VP9
+ * codec drivers.
+ *
+ * It turns out that these structs are not stable yet and will undergo
+ * more changes. So keep them private until they are stable and ready

to
+ * become part of the official public API.
+ */

include/media/vp9-ctrls.h new file +9 -0
+
+#ifndef _VP9_CTRLS_H_
+#define _VP9_CTRLS_H_
+
+#include <linux/types.h>
+
+#define V4L2_PIX_FMT_VP9_FRAME v4l2_fourcc('V', 'P', '9', 'F')
+
+#define V4L2_CID_MPEG_VIDEO_VP9_FRAME_CONTEXT(i)

(V4L2_CID_MPEG_BASE
+ 4000 + (i))
+#define V4L2_CID_MPEG_VIDEO_VP9_FRAME_DECODE_PARAMS (V4L2_CID_MPEG_BASE
+ 4004)

include/media/vp9-ctrls.h new file +7 -0
+#define V4L2_CTRL_TYPE_VP9_FRAME_CONTEXT               0x400
+#define V4L2_CTRL_TYPE_VP9_FRAME_DECODE_PARAMS         0x404
+
+/**
+ * enum v4l2_vp9_loop_filter_flags - VP9 loop filter flags
+ *
+ * @V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED: the filter level depends

on
+ * the mode and reference
frame
used
+ * to predict a block
+ * @V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE: the bitstream contains additional
+ * syntax elements that specify
which
+ * mode and reference frame
deltas
+ * are to be updated
+ *
+ * Those are the flags you should pass to &v4l2_vp9_loop_filter.flags. See
+ * section '7.2.8 Loop filter semantics' of the VP9 specification for more
+ * details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+enum v4l2_vp9_loop_filter_flags {
+       V4L2_VP9_LOOP_FILTER_FLAG_DELTA_ENABLED = 1 << 0,
+       V4L2_VP9_LOOP_FILTER_FLAG_DELTA_UPDATE = 1 << 1,
+};
+
+/**
+ * struct v4l2_vp9_loop_filter - VP9 loop filter parameters
+ *
+ * @flags: combination of V4L2_VP9_LOOP_FILTER_FLAG_* flags
+ * @level: indicates the loop filter strength
+ * @sharpness: indicates the sharpness level
+ * @ref_deltas: contains the adjustment needed for the filter level

based
on
+ * the chosen reference frame
+ * @mode_deltas: contains the adjustment needed for the filter level based
on
+ * the chosen mode
+ * @level_lookup: level lookup table
+ *
+ * This structure contains all loop filter related parameters. See sections
+ * '7.2.8 Loop filter semantics' and '8.8.1 Loop filter frame init process'
+ * of the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h line 66

Quoted 8 lines
include/media/vp9-ctrls.h new file +8 -0
+struct v4l2_vp9_loop_filter {
+       __u8 flags;
+       __u8 level;
+       __u8 sharpness;
+       __s8 ref_deltas[4];
+       __s8 mode_deltas[2];
+       __u8 level_lookup[8][4][2];
+};

This struct is 73 bytes, doesn't it need padding?

Quoted 452 lines
include/media/vp9-ctrls.h new file +11 -0
+
+/**
+ * struct v4l2_vp9_quantization - VP9 quantization parameters
+ *
+ * @base_q_idx: indicates the base frame qindex
+ * @delta_q_y_dc: indicates the Y DC quantizer relative to base_q_idx
+ * @delta_q_uv_dc: indicates the UV DC quantizer relative to base_q_idx
+ * @delta_q_uv_ac indicates the UV AC quantizer relative to base_q_idx
+ * @padding: padding bytes to align things on 64 bits. Must be set to 0
+ *
+ * Encodes the quantization parameters. See section '7.2.9 Quantization

params
+ * syntax' of the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+struct v4l2_vp9_quantization {
+       __u8 base_q_idx;
+       __s8 delta_q_y_dc;
+       __s8 delta_q_uv_dc;
+       __s8 delta_q_uv_ac;
+       __u8 padding[4];
+};
+
+/**
+ * enum v4l2_vp9_segmentation_flags - VP9 segmentation flags
+ *
+ * @V4L2_VP9_SEGMENTATION_FLAG_ENABLED: indicates that this frame makes

use
of
+ * the segmentation tool
+ * @V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP: indicates that the segmentation
map
+ * should be updated during the
+ * decoding of this frame
+ * @V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE: indicates that the updates
to
+ * the segmentation map are
coded
+ * relative to the existing
+ * segmentation map
+ * @V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA: indicates that new parameters
are
+ * about to be specified for
each
+ * segment
+ * @V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE: indicates that the
+ * segmentation
parameters
+ * represent the actual
values
+ * to be used
+ *
+ * Those are the flags you should pass to &v4l2_vp9_segmentation.flags. See
+ * section '7.2.10 Segmentation params syntax' of the VP9 specification for
+ * more details.
+ */

include/media/vp9-ctrls.h new file +21 -0
+enum v4l2_vp9_segmentation_flags {
+       V4L2_VP9_SEGMENTATION_FLAG_ENABLED = 1 << 0,
+       V4L2_VP9_SEGMENTATION_FLAG_UPDATE_MAP = 1 << 1,
+       V4L2_VP9_SEGMENTATION_FLAG_TEMPORAL_UPDATE = 1 << 2,
+       V4L2_VP9_SEGMENTATION_FLAG_UPDATE_DATA = 1 << 3,
+       V4L2_VP9_SEGMENTATION_FLAG_ABS_OR_DELTA_UPDATE = 1 << 4,
+};
+
+#define V4L2_VP9_SEGMENT_FEATURE_ENABLED(id)   (1 << (id))
+#define V4L2_VP9_SEGMENT_FEATURE_ENABLED_MASK  0xf
+
+/**
+ * enum v4l2_vp9_segment_feature - VP9 segment feature IDs
+ *
+ * @V4L2_VP9_SEGMENT_FEATURE_QP_DELTA: QP delta segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_LF: loop filter segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_REF_FRAME: reference frame segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_SKIP: skip segment feature
+ * @V4L2_VP9_SEGMENT_FEATURE_CNT: number of segment features
+ *
+ * Segment feature IDs. See section '7.2.10 Segmentation params syntax'

of
the
+ * VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +16 -0
+enum v4l2_vp9_segment_feature {
+       V4L2_VP9_SEGMENT_FEATURE_QP_DELTA,
+       V4L2_VP9_SEGMENT_FEATURE_LF,
+       V4L2_VP9_SEGMENT_FEATURE_REF_FRAME,
+       V4L2_VP9_SEGMENT_FEATURE_SKIP,
+       V4L2_VP9_SEGMENT_FEATURE_CNT,
+};
+
+/**
+ * struct v4l2_vp9_segmentation - VP9 segmentation parameters
+ *
+ * @flags: combination of V4L2_VP9_SEGMENTATION_FLAG_* flags
+ * @tree_probs: specifies the probability values to be used when
+ *              decoding a Segment-ID. See '5.15. Segmentation map'
+ *              section of the VP9 specification for more details.
+ * @pred_prob: specifies the probability values to be used when

decoding a
+ * Predicted-Segment-ID. See '6.4.14. Get segment id syntax'
+ * section of :ref:vp9 for more details..
+ * @padding: padding used to make things aligned on 64 bits. Shall be zero
+ * filled
+ * @feature_enabled: bitmask defining which features are enabled in each
+ * segment
+ * @feature_data: data attached to each feature. Data entry is only valid
if
+ * the feature is enabled
+ *
+ * Encodes the quantization parameters. See section '7.2.10 Segmentation
+ * params syntax' of the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +24 -0
+struct v4l2_vp9_segmentation {
+       __u8 flags;
+       __u8 tree_probs[7];
+       __u8 pred_probs[3];
+       __u8 padding[5];
+       __u8 feature_enabled[8];
+       __s16 feature_data[8][4];
+};
+
+/**
+ * enum v4l2_vp9_intra_prediction_mode - VP9 Intra prediction modes
+ *
+ * @V4L2_VP9_INTRA_PRED_DC: DC intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_V: vertical intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_H: horizontal intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D45: D45 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D135: D135 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D117: D117 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D153: D153 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D207: D207 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_D63: D63 intra prediction
+ * @V4L2_VP9_INTRA_PRED_MODE_TM: True Motion intra prediction
+ *
+ * See section '7.4.5 Intra frame mode info semantics' for more

details.
+ */

include/media/vp9-ctrls.h new file +23 -0
+enum v4l2_vp9_intra_prediction_mode {
+       V4L2_VP9_INTRA_PRED_MODE_DC,
+       V4L2_VP9_INTRA_PRED_MODE_V,
+       V4L2_VP9_INTRA_PRED_MODE_H,
+       V4L2_VP9_INTRA_PRED_MODE_D45,
+       V4L2_VP9_INTRA_PRED_MODE_D135,
+       V4L2_VP9_INTRA_PRED_MODE_D117,
+       V4L2_VP9_INTRA_PRED_MODE_D153,
+       V4L2_VP9_INTRA_PRED_MODE_D207,
+       V4L2_VP9_INTRA_PRED_MODE_D63,
+       V4L2_VP9_INTRA_PRED_MODE_TM,
+};
+
+/**
+ * struct v4l2_vp9_mv_probabilities - VP9 Motion vector probabilities
+ * @joint: motion vector joint probabilities
+ * @sign: motion vector sign probabilities
+ * @class: motion vector class probabilities
+ * @class0_bit: motion vector class0 bit probabilities
+ * @bits: motion vector bits probabilities
+ * @class0_fr: motion vector class0 fractional bit probabilities
+ * @fr: motion vector fractional bit probabilities
+ * @class0_hp: motion vector class0 high precision fractional bit

probabilities
+ * @hp: motion vector high precision fractional bit probabilities
+ */

include/media/vp9-ctrls.h new file +32 -0
+struct v4l2_vp9_mv_probabilities {
+       __u8 joint[3];
+       __u8 sign[2];
+       __u8 class[2][10];
+       __u8 class0_bit[2];
+       __u8 bits[2][10];
+       __u8 class0_fr[2][2][3];
+       __u8 fr[2][3];
+       __u8 class0_hp[2];
+       __u8 hp[2];
+};
+
+/**
+ * struct v4l2_vp9_probabilities - VP9 Probabilities
+ *
+ * @tx8: TX 8x8 probabilities
+ * @tx16: TX 16x16 probabilities
+ * @tx32: TX 32x32 probabilities
+ * @coef: coefficient probabilities
+ * @skip: skip probabilities
+ * @inter_mode: inter mode probabilities
+ * @interp_filter: interpolation filter probabilities
+ * @is_inter: is inter-block probabilities
+ * @comp_mode: compound prediction mode probabilities
+ * @single_ref: single ref probabilities
+ * @comp_ref: compound ref probabilities
+ * @y_mode: Y prediction mode probabilities
+ * @uv_mode: UV prediction mode probabilities
+ * @partition: partition probabilities
+ * @mv: motion vector probabilities
+ *
+ * Structure containing most VP9 probabilities. See the VP9

specification
+ * for more details.
+ */

include/media/vp9-ctrls.h new file +22 -0
+struct v4l2_vp9_probabilities {
+       __u8 tx8[2][1];
+       __u8 tx16[2][2];
+       __u8 tx32[2][3];
+       __u8 coef[4][2][2][6][6][3];
+       __u8 skip[3];
+       __u8 inter_mode[7][3];
+       __u8 interp_filter[4][2];
+       __u8 is_inter[4];
+       __u8 comp_mode[5];
+       __u8 single_ref[5][2];
+       __u8 comp_ref[5];
+       __u8 y_mode[4][9];
+       __u8 uv_mode[10][9];
+       __u8 partition[16][3];
+
+       struct v4l2_vp9_mv_probabilities mv;
+};
+
+/**
+ * enum v4l2_vp9_reset_frame_context - Valid values for
+ *                     &v4l2_ctrl_vp9_frame_decode_params-
Quoted 1 line

reset_frame_context

+ *
+ * @V4L2_VP9_RESET_FRAME_CTX_NONE: don't reset any frame context
+ * @V4L2_VP9_RESET_FRAME_CTX_SPEC: reset the frame context pointed by
+ *
&v4l2_ctrl_vp9_frame_decode_params.frame_context_idx
+ * @V4L2_VP9_RESET_FRAME_CTX_ALL: reset all frame contexts
+ *
+ * See section '7.2 Uncompressed header semantics' of the VP9 specification
+ * for more details.
+ */

include/media/vp9-ctrls.h new file +14 -0
+enum v4l2_vp9_reset_frame_context {
+       V4L2_VP9_RESET_FRAME_CTX_NONE,
+       V4L2_VP9_RESET_FRAME_CTX_SPEC,
+       V4L2_VP9_RESET_FRAME_CTX_ALL,
+};
+
+/**
+ * enum v4l2_vp9_interpolation_filter - VP9 interpolation filter types
+ *
+ * @V4L2_VP9_INTERP_FILTER_8TAP: height tap filter
+ * @V4L2_VP9_INTERP_FILTER_8TAP_SMOOTH: height tap smooth filter
+ * @V4L2_VP9_INTERP_FILTER_8TAP_SHARP: height tap sharp filter
+ * @V4L2_VP9_INTERP_FILTER_BILINEAR: bilinear filter
+ * @V4L2_VP9_INTERP_FILTER_SWITCHABLE: filter selection is signaled at

the
+ * block level
+ *
+ * See section '7.2.7 Interpolation filter semantics' of the VP9 specification
+ * for more details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+enum v4l2_vp9_interpolation_filter {
+       V4L2_VP9_INTERP_FILTER_8TAP,
+       V4L2_VP9_INTERP_FILTER_8TAP_SMOOTH,
+       V4L2_VP9_INTERP_FILTER_8TAP_SHARP,
+       V4L2_VP9_INTERP_FILTER_BILINEAR,
+       V4L2_VP9_INTERP_FILTER_SWITCHABLE,
+};
+
+/**
+ * enum v4l2_vp9_reference_mode - VP9 reference modes
+ *
+ * @V4L2_VP9_REF_MODE_SINGLE: indicates that all the inter blocks use

only
a
+ * single reference frame to generate motion
+ * compensated prediction
+ * @V4L2_VP9_REF_MODE_COMPOUND: requires all the inter blocks to use compound
+ * mode. Single reference frame prediction
is
not
+ * allowed
+ * @V4L2_VP9_REF_MODE_SELECT: allows each individual inter block to select
+ * between single and compound prediction
modes
+ *
+ * See section '7.3.6 Frame reference mode semantics' of the VP9 specification
+ * for more details.
+ */

include/media/vp9-ctrls.h new file +14 -0
+enum v4l2_vp9_reference_mode {
+       V4L2_VP9_REF_MODE_SINGLE,
+       V4L2_VP9_REF_MODE_COMPOUND,
+       V4L2_VP9_REF_MODE_SELECT,
+};
+
+/**
+ * enum v4l2_vp9_tx_mode - VP9 TX modes
+ *
+ * @V4L2_VP9_TX_MODE_ONLY_4X4: transform size is 4x4
+ * @V4L2_VP9_TX_MODE_ALLOW_8X8: transform size can be up to 8x8
+ * @V4L2_VP9_TX_MODE_ALLOW_16X16: transform size can be up to 16x16
+ * @V4L2_VP9_TX_MODE_ALLOW_32X32: transform size can be up to 32x32
+ * @V4L2_VP9_TX_MODE_SELECT: bitstream contains transform size for each

block
+ *
+ * See section '7.3.1 Tx mode semantics' of the VP9 specification for more
+ * details.
+ */

include/media/vp9-ctrls.h new file +17 -0
+enum v4l2_vp9_tx_mode {
+       V4L2_VP9_TX_MODE_ONLY_4X4,
+       V4L2_VP9_TX_MODE_ALLOW_8X8,
+       V4L2_VP9_TX_MODE_ALLOW_16X16,
+       V4L2_VP9_TX_MODE_ALLOW_32X32,
+       V4L2_VP9_TX_MODE_SELECT,
+};
+
+/**
+ * enum v4l2_vp9_ref_id - VP9 Reference frame IDs
+ *
+ * @V4L2_REF_ID_LAST: last reference frame
+ * @V4L2_REF_ID_GOLDEN: golden reference frame
+ * @V4L2_REF_ID_ALTREF: alternative reference frame
+ * @V4L2_REF_ID_CNT: number of reference frames
+ *
+ * See section '7.4.12 Ref frames semantics' of the VP9 specification

for
more
+ * details.
+ */

include/media/vp9-ctrls.h new file +12 -0
+enum v4l2_vp9_ref_id {
+       V4L2_REF_ID_LAST,
+       V4L2_REF_ID_GOLDEN,
+       V4L2_REF_ID_ALTREF,
+       V4L2_REF_ID_CNT,
+};
+
+/**
+ * enum v4l2_vp9_frame_flags - VP9 frame flags
+ * @V4L2_VP9_FRAME_FLAG_KEY_FRAME: the frame is a key frame
+ * @V4L2_VP9_FRAME_FLAG_SHOW_FRAME: the frame should be displayed
+ * @V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT: the decoding should be error

resilient
+ * @V4L2_VP9_FRAME_FLAG_INTRA_ONLY: the frame does not reference other frames
+ * @V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV: the frame might can high precision
+ * motion vectors
+ * @V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX: frame context should be updated
+ * after decoding
+ * @V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE: parallel decoding is used
+ * @V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING: vertical subsampling is enabled
+ * @V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING: horizontal subsampling is enabled
+ * @V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING: full UV range is used
+ *
+ * Check the VP9 specification for more details.
+ */

include/media/vp9-ctrls.h new file +17 -0
+enum v4l2_vp9_frame_flags {
+       V4L2_VP9_FRAME_FLAG_KEY_FRAME = 1 << 0,
+       V4L2_VP9_FRAME_FLAG_SHOW_FRAME = 1 << 1,
+       V4L2_VP9_FRAME_FLAG_ERROR_RESILIENT = 1 << 2,
+       V4L2_VP9_FRAME_FLAG_INTRA_ONLY = 1 << 3,
+       V4L2_VP9_FRAME_FLAG_ALLOW_HIGH_PREC_MV = 1 << 4,
+       V4L2_VP9_FRAME_FLAG_REFRESH_FRAME_CTX = 1 << 5,
+       V4L2_VP9_FRAME_FLAG_PARALLEL_DEC_MODE = 1 << 6,
+       V4L2_VP9_FRAME_FLAG_X_SUBSAMPLING = 1 << 7,
+       V4L2_VP9_FRAME_FLAG_Y_SUBSAMPLING = 1 << 8,
+       V4L2_VP9_FRAME_FLAG_COLOR_RANGE_FULL_SWING = 1 << 9,
+};
+
+#define V4L2_VP9_PROFILE_MAX           3
+
+/**
+ * struct v4l2_ctrl_vp9_frame_decode_params - VP9 frame decoding

control
+ *
+ * @flags: combination of V4L2_VP9_FRAME_FLAG_* flags
+ * @compressed_header_size: compressed header size in bytes
+ * @uncompressed_header_size: uncompressed header size in bytes
+ * @profile: VP9 profile. Can be 0, 1, 2 or 3
+ * @reset_frame_context: specifies whether the frame context should be reset
+ * to default values. See
&v4l2_vp9_reset_frame_context
+ * for more details
+ * @frame_context_idx: frame context that should be used/updated
+ * @bit_depth: bits per components. Can be 8, 10 or 12. Note that not all
+ * profiles support 10 and/or 12 bits depths
+ * @interpolation_filter: specifies the filter selection used for performing
+ * inter prediction. See
&v4l2_vp9_interpolation_filter
+ * for more details
+ * @tile_cols_log2: specifies the base 2 logarithm of the width of each tile
+ * (where the width is measured in units of 8x8
blocks).
+ * Shall be less than or equal to 6
+ * @tile_rows_log2: specifies the base 2 logarithm of the height of each
tile
+ * (where the height is measured in units of 8x8
blocks)
+ * @tx_mode: specifies the TX mode. See &v4l2_vp9_tx_mode for more details
+ * @reference_mode: specifies the type of inter prediction to be used. See
+ * &v4l2_vp9_reference_mode for more details
+ * @padding: needed to make this struct 64 bit aligned. Shall be filled with
+ * zeros
+ * @frame_width_minus_1: add 1 to it and you'll get the frame width expressed
+ * in pixels
+ * @frame_height_minus_1: add 1 to it and you'll get the frame height expressed
+ * in pixels
+ * @frame_width_minus_1: add 1 to it and you'll get the expected render width
+ * expressed in pixels. This is not used during
the
+ * decoding process but might be used by HW
scalers to
+ * prepare a frame that's ready for scanout
+ * @frame_height_minus_1: add 1 to it and you'll get the expected render
height
+ * expressed in pixels. This is not used during
the
+ * decoding process but might be used by HW
scalers to
+ * prepare a frame that's ready for scanout
+ * @refs: array of reference frames. See &v4l2_vp9_ref_id for more details
+ * @lf: loop filter parameters. See &v4l2_vp9_loop_filter for more details
+ * @quant: quantization parameters. See &v4l2_vp9_quantization for more details
+ * @seg: segmentation parameters. See &v4l2_vp9_segmentation for more details
+ * @probs: probabilities. See &v4l2_vp9_probabilities for more details
+ */

include/media/vp9-ctrls.h new file +4 -0
+struct v4l2_ctrl_vp9_frame_decode_params {
+       __u32 flags;
+       __u16 compressed_header_size;
+       __u16 uncompressed_header_size;

include/media/vp9-ctrls.h line 453

Quoted 10 lines
include/media/vp9-ctrls.h new file +10 -0
+       __u8 profile;
+       __u8 reset_frame_context;
+       __u8 frame_context_idx;
+       __u8 bit_depth;
+       __u8 interpolation_filter;
+       __u8 tile_cols_log2;
+       __u8 tile_rows_log2;
+       __u8 tx_mode;
+       __u8 reference_mode;
+       __u8 padding[6];

If my math is correct this should be 7.

include/media/vp9-ctrls.h line 458

Quoted 5 lines
include/media/vp9-ctrls.h new file +5 -0
+       __u16 frame_width_minus_1;
+       __u16 frame_height_minus_1;
+       __u16 render_width_minus_1;
+       __u16 render_height_minus_1;
+       __u64 refs[V4L2_REF_ID_CNT];

Why are we using u64s to store these [0..7] indices? Or are we storing more than that?

include/media/vp9-ctrls.h line 463

Quoted 5 lines
include/media/vp9-ctrls.h new file +5 -0
+       struct v4l2_vp9_loop_filter lf;
+       struct v4l2_vp9_quantization quant;
+       struct v4l2_vp9_segmentation seg;
+       struct v4l2_vp9_probabilities probs;
+};
Quoted 3 lines

I cannot find a trace of ref_frame_sign_biases, which is also part of the uncompressed header, in this structure. Is it missing, or is this information provided in a different way?

This seems missing indeed. Interestingly the Rockchip reference software does
not seem to pass this information to the hardware. On the other end, the VC8000D
software does. It is also part of DXVA, VA and NVDEC APIs.

Quoted 11 lines
Quoted 2 lines

I can think of at least one piece of hardware that will require this information, so it would be nice to have it included.

We'll had, the more we look at the spec, the less we understand how rkvdec manages proper decoding without.

Quoted 3 lines

Cheers,
Alex.

_______________________________________________
Linux-rockchip mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip

#12 lore raw

PATCH v4 3/3 media: rkvdec: Add the VP9 backend

In reply to Ezequiel Garcia

From: Boris Brezillon <[email protected]>

The Rockchip VDEC supports VP9 profile 0 up to 4096x2304@30fps. Add a backend for this new format.

---
 drivers/staging/media/rkvdec/Makefile     |    2 +-
 drivers/staging/media/rkvdec/rkvdec-vp9.c | 1577 +++++++++++++++++++++
 drivers/staging/media/rkvdec/rkvdec.c     |   56 +-
 drivers/staging/media/rkvdec/rkvdec.h     |    6 +
 4 files changed, 1637 insertions(+), 4 deletions(-)
 create mode 100644 drivers/staging/media/rkvdec/rkvdec-vp9.c
drivers/staging/media/rkvdec/Makefile +1 -1
index c08fed0a39f9..cb86b429cfaa 100644
@@ -1,3 +1,3 @@
 obj-$(CONFIG_VIDEO_ROCKCHIP_VDEC) += rockchip-vdec.o
 
-rockchip-vdec-y += rkvdec.o rkvdec-h264.o
+rockchip-vdec-y += rkvdec.o rkvdec-h264.o rkvdec-vp9.o
drivers/staging/media/rkvdec/rkvdec-vp9.c new file +1577 -0

1,577 lines not on this page · raw message

drivers/staging/media/rkvdec/rkvdec.c +53 -3

71 lines not on this page · raw message

drivers/staging/media/rkvdec/rkvdec.h +6 -0
index 2fc9f46b6910..2a795babfccd 100644
@@ -52,6 +52,10 @@ struct rkvdec_vp9_decoded_buffer_info {
 struct rkvdec_decoded_buffer {
 	/* Must be the first field in this struct. */
 	struct v4l2_m2m_buffer base;
+
+	union {
+		struct rkvdec_vp9_decoded_buffer_info vp9;
+	};
 };
 
 static inline struct rkvdec_decoded_buffer *
@@ -118,4 +122,6 @@ void rkvdec_run_preamble(struct rkvdec_ctx *ctx, struct rkvdec_run *run);
 void rkvdec_run_postamble(struct rkvdec_ctx *ctx, struct rkvdec_run *run);
 
 extern const struct rkvdec_coded_fmt_ops rkvdec_h264_fmt_ops;
+extern const struct rkvdec_coded_fmt_ops rkvdec_vp9_fmt_ops;
+
 #endif /* RKVDEC_H_ */
Signature · 1 line
2.26.0.rc2
#13 lore raw

Re: [PATCH 3/3] media: rkvdec: Add the VP9 backend

Quoted 1 line · Ezequiel Garcia

[PATCH 3/3] media: rkvdec: Add the VP9 backend

Oops, ^ should be v4 here.

Thanks,
Ezequiel