media: hantro: Add support for H264 decoding
Subsystems (7)
v4 cover media: hantro: Add support for H264 decoding
This series is consolidating the two recent H264 series submitted by Boris [1] [2]. Given some patches from [2] have been merged (namely, helpers for the Hantro driver), the series contains the remanining bits required to support H264 on Hantro G1 VPU.
* Patch 1 adds support for the sort_r() variant and has
been posted separately by Rasmus. It would be good to merge this patch via the media tree, ideally as soon as possible, to avoid the synchronisation burden that might appear if we decide to delay it.
* Patch 2 to 4 extends the H264 uAPI, introducing frame-based vs slice-based decoding granularity, and also support for different NALU start codes. Currently, Annex B and no start codes are the supported options.
With the introduction of the start code control, the H264 parsed slices pixel format should be renamed, dropping the _RAW suffix, which is now meaningless.
* Patch 5 removes the P0/B0/B1 ref lists from the decode_params control. These lists are no longer needed since we build them on the kernel side based on the DPB.
* Patch 6 and 7 exposes the proper decoding mode and start code on the cedrus driver. The driver functionality is not changed, and only the Cedrus support is now being properly exposed to userspace.
* Patch 8 is needed to properly propagate the OUTPUT buffer timestamp to the CAPTURE buffer one, which is required for intra-frame references.
* Patches 9 to 11 adds H264 support for Hantro G1 and then enable H264 decoding on RK3288.
This series has been tested using MPV/Ffmpeg, on Rockchip RK3288 for Hantro and Allwinner H3 boards for Cedrus.
[1] https://www.mail-archive.com/[email protected]/msg148299.html
[2] https://lkml.org/lkml/2019/6/19/379
Boris Brezillon (3):
media: uapi: h264: Add the concept of decoding mode
media: uapi: h264: Get rid of the p0/b0/b1 ref-lists
media: hantro: Move copy_metadata() before doing a decode operation
Ezequiel Garcia (4):
media: uapi: h264: Rename pixel format
media: uapi: h264: Add the concept of start code
media: cedrus: Cleanup control initialization
media: cedrus: Specify the required H264 start code
Hertz Wong (3):
media: hantro: Add core bits to support H264 decoding
media: hantro: Add support for H264 decoding on G1
media: hantro: Enable H264 decoding on rk3288
Rasmus Villemoes (1):
lib/sort.c: implement sort() variant taking context argument
.../media/uapi/v4l/ext-ctrls-codec.rst | 83 ++- .../media/uapi/v4l/pixfmt-compressed.rst | 4 +- drivers/media/v4l2-core/v4l2-ctrls.c | 18 + drivers/media/v4l2-core/v4l2-ioctl.c | 2 +- drivers/staging/media/hantro/Makefile | 2 + drivers/staging/media/hantro/hantro.h | 9 +- drivers/staging/media/hantro/hantro_drv.c | 50 +- .../staging/media/hantro/hantro_g1_h264_dec.c | 292 ++++++++ drivers/staging/media/hantro/hantro_h264.c | 649 ++++++++++++++++++ drivers/staging/media/hantro/hantro_hw.h | 56 ++ drivers/staging/media/hantro/hantro_v4l2.c | 10 + drivers/staging/media/hantro/rk3288_vpu_hw.c | 21 +- drivers/staging/media/sunxi/cedrus/cedrus.c | 65 +- drivers/staging/media/sunxi/cedrus/cedrus.h | 3 +- .../staging/media/sunxi/cedrus/cedrus_dec.c | 2 +- .../staging/media/sunxi/cedrus/cedrus_video.c | 6 +- include/linux/sort.h | 5 + include/media/h264-ctrls.h | 22 +- lib/sort.c | 34 +- 19 files changed, 1279 insertions(+), 54 deletions(-) create mode 100644 drivers/staging/media/hantro/hantro_g1_h264_dec.c create mode 100644 drivers/staging/media/hantro/hantro_h264.c
Signature · 1 line
2.22.0
PATCH v4 1/11 lib/sort.c: implement sort() variant taking context argument
From: Rasmus Villemoes <[email protected]>
Our list_sort() utility has always supported a context argument that is passed through to the comparison routine. Now there's a use case for the similar thing for sort().
This implements sort_r by simply extending the existing sort function in the obvious way. To avoid code duplication, we want to implement sort() in terms of sort_r(). The naive way to do that is
static int cmp_wrapper(const void *a, const void *b, const void *ctx) {
int (*real_cmp)(const void*, const void*) = ctx;
return real_cmp(a, b);
}
sort(..., cmp) { sort_r(..., cmp_wrapper, cmp) }
but this would do two indirect calls for each comparison. Instead, do as is done for the default swap functions - that only adds a cost of a single easily predicted branch to each comparison call.
Aside from introducing support for the context argument, this also serves as preparation for patches that will eliminate the indirect comparison calls in common cases.
--- include/linux/sort.h | 5 +++++ lib/sort.c | 34 ++++++++++++++++++++++++++++------ 2 files changed, 33 insertions(+), 6 deletions(-)
include/linux/sort.h +5 -0
@@ -4,6 +4,11 @@#include <linux/types.h> +void sort_r (void *base,size_t num,size_t size, +int (*cmp)(const void *,const void *,const void *), +void (*swap)(void *,void *,int ), +const void *priv); +void sort (void *base,size_t num,size_t size,int (*cmp)(const void *,const void *),void (*swap)(void *,void *,int ));
lib/sort.c +28 -6
@@ -144,6 +144,18 @@ static void do_swap(void *a, void *b, size_t size, swap_func_t swap_func)swap_func (a, b, (int )size); } +typedef int (*cmp_func_t )(const void *,const void *); +typedef int (*cmp_r_func_t )(const void *,const void *,const void *); +#define _CMP_WRAPPER ((cmp_r_func_t)0L) + +static int do_cmp (const void *a,const void *b, +cmp_r_func_t cmp,const void *priv) +{ +if (cmp == _CMP_WRAPPER) +return ((cmp_func_t)(priv))(a, b); +return cmp (a, b, priv); +} +/** * parent - given the offset of the child, find the offset of the parent. * @i: the offset of the heap element whose parent is sought. Non-zero.
@@ -171,12 +183,13 @@ static size_t parent(size_t i, unsigned int lsbit, size_t size) }/** -* sort - sort an array of elements +* sort_r - sort an array of elements * @base: pointer to data to sort * @num: number of elements * @size: size of each element * @cmp_func: pointer to comparison function * @swap_func: pointer to swap function or NULL +* @priv: third argument passed to comparison function * * This function does a heapsort on the given array. You may provide * a swap_func function if you need to do something more than a memory
@@ -188,9 +201,10 @@ static size_t parent(size_t i, unsigned int lsbit, size_t size)* O(n*n) worst-case behavior and extra memory requirements that make * it less suitable for kernel use. */ -void sort (void *base,size_t num,size_t size, -int (*cmp_func)(const void *,const void *), -void (*swap_func)(void *,void *,int size)) +void sort_r (void *base,size_t num,size_t size, +int (*cmp_func)(const void *,const void *,const void *), +void (*swap_func)(void *,void *,int size), +const void *priv) {/* pre-scale counters for performance */ size_t n = num * size, a = (num/2 ) * size;
@@ -238,12 +252,12 @@ void sort(void *base, size_t num, size_t size,* average, 3/4 worst-case.) */ for (b = a; c =2 *b + size, (d = c + size) < n;) - b =cmp_func (base + c, base + d) >=0 ? c : d; + b =do_cmp (base + c, base + d, cmp_func, priv) >=0 ? c : d;if (d == n)/* Special case last leaf with no sibling */ b = c;/* Now backtrack from "b" to the correct location for "a" */ -while (b != a &&cmp_func (base + a, base + b) >=0 ) +while (b != a &&do_cmp (base + a, base + b, cmp_func, priv) >=0 ) b =parent (b, lsbit, size); c = b;/* Where "a" belongs */ while (b != a) {/* Shift it into place */
@@ -252,4 +266,12 @@ void sort(void *base, size_t num, size_t size, } } } +EXPORT_SYMBOL (sort_r); + +void sort (void *base,size_t num,size_t size, +int (*cmp_func)(const void *,const void *), +void (*swap_func)(void *,void *,int size)) +{ +return sort_r (base, num, size, _CMP_WRAPPER, swap_func, cmp_func); +}EXPORT_SYMBOL (sort);
Signature · 1 line
2.22.0
PATCH v4 2/11 media: uapi: h264: Rename pixel format
In reply to Ezequiel GarciaThe V4L2_PIX_FMT_H264_SLICE_RAW name was originally suggested because the pixel format would represent H264 slices without any start code.
However, as we will now introduce a start code menu control, give the pixel format a more meaningful name, while it's still early enough to do so.
---
Changes in v4:
* New patch.
--- Documentation/media/uapi/v4l/pixfmt-compressed.rst | 4 ++-- drivers/media/v4l2-core/v4l2-ioctl.c | 2 +- drivers/staging/media/sunxi/cedrus/cedrus_dec.c | 2 +- drivers/staging/media/sunxi/cedrus/cedrus_video.c | 6 +++--- include/media/h264-ctrls.h | 2 +- 5 files changed, 8 insertions(+), 8 deletions(-)
Documentation/media/uapi/v4l/pixfmt-compressed.rst +2 -2
@@ -52,9 +52,9 @@ Compressed Formats - ``V4L2_PIX_FMT_H264_MVC`` - 'M264' - H264 MVC video elementary stream. - * .. _V4L2-PIX-FMT-H264-SLICE-RAW: + * .. _V4L2-PIX-FMT-H264-SLICE: - - ``V4L2_PIX_FMT_H264_SLICE_RAW`` + - ``V4L2_PIX_FMT_H264_SLICE`` - 'S264' - H264 parsed slice data, without the start code and as extracted from the H264 bitstream. This format is adapted for
drivers/media/v4l2-core/v4l2-ioctl.c +1 -1
@@ -1343,7 +1343,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)case V4L2_PIX_FMT_H264 : descr ="H.264" ;break ;case V4L2_PIX_FMT_H264_NO_SC : descr ="H.264 (No Start Codes)" ;break ;case V4L2_PIX_FMT_H264_MVC : descr ="H.264 MVC" ;break ; -case V4L2_PIX_FMT_H264_SLICE_RAW : descr ="H.264 Parsed Slice Data" ;break ; +case V4L2_PIX_FMT_H264_SLICE : descr ="H.264 Parsed Slice Data" ;break ;case V4L2_PIX_FMT_H263 : descr ="H.263" ;break ;case V4L2_PIX_FMT_MPEG1 : descr ="MPEG-1 ES" ;break ;case V4L2_PIX_FMT_MPEG2 : descr ="MPEG-2 ES" ;break ;
drivers/staging/media/sunxi/cedrus/cedrus_dec.c +1 -1
@@ -46,7 +46,7 @@ void cedrus_device_run(void *priv) V4L2_CID_MPEG_VIDEO_MPEG2_QUANTIZATION);break ; -case V4L2_PIX_FMT_H264_SLICE_RAW : +case V4L2_PIX_FMT_H264_SLICE : run.h264.decode_params =cedrus_find_control_data (ctx,V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS ); run.h264.pps = cedrus_find_control_data(ctx,
drivers/staging/media/sunxi/cedrus/cedrus_video.c +3 -3
@@ -38,7 +38,7 @@ static struct cedrus_format cedrus_formats[] = { .directions = CEDRUS_DECODE_SRC, }, { - .pixelformat = V4L2_PIX_FMT_H264_SLICE_RAW, + .pixelformat = V4L2_PIX_FMT_H264_SLICE, .directions = CEDRUS_DECODE_SRC, }, {
@@ -104,7 +104,7 @@ static void cedrus_prepare_format(struct v4l2_pix_format *pix_fmt)switch (pix_fmt->pixelformat) {case V4L2_PIX_FMT_MPEG2_SLICE : -case V4L2_PIX_FMT_H264_SLICE_RAW : +case V4L2_PIX_FMT_H264_SLICE :/* Zero bytes per line for encoded source. */ bytesperline =0 ;
@@ -469,7 +469,7 @@ static int cedrus_start_streaming(struct vb2_queue *vq, unsigned int count) ctx->current_codec =CEDRUS_CODEC_MPEG2 ;break ; -case V4L2_PIX_FMT_H264_SLICE_RAW : +case V4L2_PIX_FMT_H264_SLICE : ctx->current_codec =CEDRUS_CODEC_H264 ;break ;
include/media/h264-ctrls.h +1 -1
@@ -14,7 +14,7 @@#include <linux/videodev2.h> /* Our pixel format isn't stable at the moment */ -#define V4L2_PIX_FMT_H264_SLICE_RAW v4l2_fourcc('S', '2', '6', '4')/* H264 parsed slices */ +#define V4L2_PIX_FMT_H264_SLICE v4l2_fourcc('S', '2', '6', '4')/* H264 parsed slices */ /* * This is put insanely high to avoid conflicting with controls that
Signature · 1 line
2.22.0
PATCH v4 3/11 media: uapi: h264: Add the concept of decoding mode
In reply to Ezequiel GarciaFrom: Boris Brezillon <[email protected]>
Some stateless decoders don't support per-slice decoding granularity (or at least not in a way that would make them efficient or easy to use).
Expose a menu to control the supported decoding modes. Drivers are allowed to support only one decoding but they can support both too.
---
Changes in v4:
* Typos/rewording fixes
Changes in v3:
* s/per-{slice,frame} decoding/{slice,frame}-based decoding/
* Add Paul's R-b
Changes in v2:
* Allow decoding multiple slices in per-slice decoding mode
* Minor doc improvement/fixes
--- .../media/uapi/v4l/ext-ctrls-codec.rst | 43 ++++++++++++++++++- drivers/media/v4l2-core/v4l2-ctrls.c | 9 ++++ include/media/h264-ctrls.h | 11 +++++ 3 files changed, 62 insertions(+), 1 deletion(-)
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +42 -1
@@ -1747,6 +1747,11 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type - * - __u32 - ``size`` - + * - __u32 + - ``start_byte_offset`` + - Where the slice payload starts in the output buffer. Useful when the + OUTPUT buffer contains more than one slice (some codecs need to know + where each slice starts in this buffer). * - __u32 - ``header_bit_size`` -
@@ -1930,7 +1935,10 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type - - * - __u16 - ``num_slices`` - - Number of slices needed to decode the current frame + - Number of slices needed to decode the current frame/field. When + operating in slice-based decoding mode (see + :c:type:`v4l2_mpeg_video_h264_decoding_mode`), this field + should always be set to one. * - __u16 - ``nal_ref_idc`` - NAL reference ID value coming from the NAL Unit header
@@ -2021,6 +2029,39 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type - - 0x00000004 - The DPB entry is a long term reference frame +``V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE (enum)`` + Specifies the decoding mode to use. Currently exposes slice-based and + frame-based decoding but new modes might be added later on. + Drivers may expose a single or multiple decoding modes, + depending on what they can support. + + .. note:: + + This menu control is not yet part of the public kernel API and + it is expected to change. + +.. c:type:: v4l2_mpeg_video_h264_decoding_mode + +.. cssclass:: longtable + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 1 1 2 + + * - ``V4L2_MPEG_VIDEO_H264_SLICE_BASED_DECODING`` + - 0 + - The decoding is done at the slice granularity. + v4l2_ctrl_h264_decode_params->num_slices should be set to 1. + The OUTPUT buffer must contain a single slice. + * - ``V4L2_MPEG_VIDEO_H264_FRAME_BASED_DECODING`` + - 1 + - The decoding is done at the frame granularity. + v4l2_ctrl_h264_decode_params->num_slices should be set to the number of + slices forming a frame. + The OUTPUT buffer should contain all slices needed to decode the + frame/field. + .. _v4l2-mpeg-mpeg2: ``V4L2_CID_MPEG_VIDEO_MPEG2_SLICE_PARAMS (struct)``
drivers/media/v4l2-core/v4l2-ctrls.c +9 -0
@@ -402,6 +402,11 @@ const char * const *v4l2_ctrl_get_menu(u32 id)"Explicit" , NULL, }; +static const char *const h264_decoding_mode[] = { +"Slice-Based" , +"Frame-Based" , +NULL , + };static const char *const mpeg_mpeg2_level[] = {"Low" , "Main",
@@ -633,6 +638,8 @@ const char * const *v4l2_ctrl_get_menu(u32 id)return h264_fp_arrangement_type;case V4L2_CID_MPEG_VIDEO_H264_FMO_MAP_TYPE :return h264_fmo_map_type; +case V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE : +return h264_decoding_mode;case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL :return mpeg_mpeg2_level;case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE :
@@ -852,6 +859,7 @@ const char *v4l2_ctrl_get_name(u32 id)case V4L2_CID_MPEG_VIDEO_H264_SCALING_MATRIX :return "H264 Scaling Matrix" ;case V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS :return "H264 Slice Parameters" ;case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS :return "H264 Decode Parameters" ; +case V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE :return "H264 Decoding Mode" ;case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL :return "MPEG2 Level" ;case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE :return "MPEG2 Profile" ;case V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP :return "MPEG4 I-Frame QP Value" ;
@@ -1220,6 +1228,7 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,case V4L2_CID_MPEG_VIDEO_H264_VUI_SAR_IDC :case V4L2_CID_MPEG_VIDEO_H264_SEI_FP_ARRANGEMENT_TYPE :case V4L2_CID_MPEG_VIDEO_H264_FMO_MAP_TYPE : +case V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE :case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL :case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE :case V4L2_CID_MPEG_VIDEO_MPEG4_LEVEL :
include/media/h264-ctrls.h +11 -0
@@ -26,6 +26,7 @@#define V4L2_CID_MPEG_VIDEO_H264_SCALING_MATRIX (V4L2_CID_MPEG_BASE+1002)#define V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS (V4L2_CID_MPEG_BASE+1003)#define V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS (V4L2_CID_MPEG_BASE+1004) +#define V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE (V4L2_CID_MPEG_BASE+1005)/* enum v4l2_ctrl_type type values */ #define V4L2_CTRL_TYPE_H264_SPS 0x0110
@@ -33,6 +34,12 @@#define V4L2_CTRL_TYPE_H264_SCALING_MATRIX 0x0112#define V4L2_CTRL_TYPE_H264_SLICE_PARAMS 0x0113#define V4L2_CTRL_TYPE_H264_DECODE_PARAMS 0x0114 +#define V4L2_CTRL_TYPE_H264_DECODING_MODE 0x0115 + +enum v4l2_mpeg_video_h264_decoding_mode { +V4L2_MPEG_VIDEO_H264_SLICE_BASED_DECODING , +V4L2_MPEG_VIDEO_H264_FRAME_BASED_DECODING , +};#define V4L2_H264_SPS_CONSTRAINT_SET0_FLAG 0x01#define V4L2_H264_SPS_CONSTRAINT_SET1_FLAG 0x02
@@ -125,6 +132,10 @@ struct v4l2_h264_pred_weight_table {struct v4l2_ctrl_h264_slice_params {/* Size in bytes, including header */ __u32 size; + +/* Offset in bytes to the start of slice in the OUTPUT buffer. */ +__u32 start_byte_offset; +/* Offset in bits to slice_data() from the beginning of this slice. */ __u32 header_bit_size;
Signature · 1 line
2.22.0
Re: [PATCH v4 03/11] media: uapi: h264: Add the concept of decoding mode
Quoted 56 lines · Ezequiel Garcia
From: Boris Brezillon <[email protected]>
Some stateless decoders don't support per-slice decoding granularity (or at least not in a way that would make them efficient or easy to use).
Expose a menu to control the supported decoding modes. Drivers are allowed to support only one decoding but they can support both too.
Signed-off-by Boris Brezillon <[email protected]>Reviewed-by Paul Kocialkowski <[email protected]>---
Changes in v4:
* Typos/rewording fixesChanges in v3:
* s/per-{slice,frame} decoding/{slice,frame}-based decoding/
* Add Paul's R-bChanges in v2:
* Allow decoding multiple slices in per-slice decoding mode
* Minor doc improvement/fixes--- .../media/uapi/v4l/ext-ctrls-codec.rst | 43 ++++++++++++++++++- drivers/media/v4l2-core/v4l2-ctrls.c | 9 ++++ include/media/h264-ctrls.h | 11 +++++ 3 files changed, 62 insertions(+), 1 deletion(-)Documentation/media/uapi/v4l/ext-ctrls-codec.rst +9
-1@@ -1747,6 +1747,11 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type - * - __u32 - ``size`` - + * - __u32 + - ``start_byte_offset`` + - Where the slice payload starts in the output buffer. Useful when the + OUTPUT buffer contains more than one slice (some codecs need to know + where each slice starts in this buffer). * - __u32 - ``header_bit_size`` -@@ -1930,7 +1935,10 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type - - * - __u16 - ``num_slices`` - - Number of slices needed to decode the current frame + - Number of slices needed to decode the current frame/field. When + operating in slice-based decoding mode (see + :c:type:`v4l2_mpeg_video_h264_decoding_mode`), this field + should always be set to one. * - __u16 - ``nal_ref_idc`` - NAL reference ID value coming from the NAL Unit header
Documentation/media/uapi/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 2036
Quoted 8 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +5
-0@@ -2021,6 +2029,39 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type - - 0x00000004 - The DPB entry is a long term reference frame +``V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE (enum)`` + Specifies the decoding mode to use. Currently exposes slice-based and + frame-based decoding but new modes might be added later on. + Drivers may expose a single or multiple decoding modes, + depending on what they can support.
This should mention that this is valid for V4L2_PIX_FMT_H264_SLICE pixelformat only. And it is a required control (i.e. it must always be present for that pixelformat). Ditto for the start code control.
Also, the V4L2_PIX_FMT_H264_SLICE should refer to this control and the start code control as well since these controls effectively are modifiers for the pixelformat.
Regards,
Hans
Quoted 15 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +15
-0+ + .. note:: + + This menu control is not yet part of the public kernel API and + it is expected to change. + +.. c:type:: v4l2_mpeg_video_h264_decoding_mode + +.. cssclass:: longtable + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 1 1 2 +
Documentation/media/uapi/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 2061
Quoted 10 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +10
-0+ * - ``V4L2_MPEG_VIDEO_H264_SLICE_BASED_DECODING`` + - 0 + - The decoding is done at the slice granularity. + v4l2_ctrl_h264_decode_params->num_slices should be set to 1. + The OUTPUT buffer must contain a single slice. + * - ``V4L2_MPEG_VIDEO_H264_FRAME_BASED_DECODING`` + - 1 + - The decoding is done at the frame granularity. + v4l2_ctrl_h264_decode_params->num_slices should be set to the number of + slices forming a frame.
Actually, I wonder if there shouldn't be a third option: FIELD_BASED_DECODING to indicate that the hardware expects the slices of a single field in a buffer instead of both fields.
Or just stick to FRAME_BASED_DECODING for now but explicitly mention that in the case of interlaced decoding both fields must be present in the buffer.
A FIELD_BASED_DECODING option can always be added later if necessary.
Quoted 74 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +3
-0+ The OUTPUT buffer should contain all slices needed to decode the + frame/field. + .. _v4l2-mpeg-mpeg2: ``V4L2_CID_MPEG_VIDEO_MPEG2_SLICE_PARAMS (struct)``drivers/media/v4l2-core/v4l2-ctrls.c +9
-0@@ -402,6 +402,11 @@ const char * const *v4l2_ctrl_get_menu(u32 id)"Explicit" , NULL, }; +static const char *const h264_decoding_mode[] = { +"Slice-Based" , +"Frame-Based" , +NULL , + };static const char *const mpeg_mpeg2_level[] = {"Low" , "Main",@@ -633,6 +638,8 @@ const char * const *v4l2_ctrl_get_menu(u32 id)return h264_fp_arrangement_type;case V4L2_CID_MPEG_VIDEO_H264_FMO_MAP_TYPE :return h264_fmo_map_type; +case V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE : +return h264_decoding_mode;case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL :return mpeg_mpeg2_level;case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE :@@ -852,6 +859,7 @@ const char *v4l2_ctrl_get_name(u32 id)case V4L2_CID_MPEG_VIDEO_H264_SCALING_MATRIX :return "H264 Scaling Matrix" ;case V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS :return "H264 Slice Parameters" ;case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS :return "H264 Decode Parameters" ; +case V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE :return "H264 Decoding Mode" ;case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL :return "MPEG2 Level" ;case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE :return "MPEG2 Profile" ;case V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP :return "MPEG4 I-Frame QP Value" ;@@ -1220,6 +1228,7 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,case V4L2_CID_MPEG_VIDEO_H264_VUI_SAR_IDC :case V4L2_CID_MPEG_VIDEO_H264_SEI_FP_ARRANGEMENT_TYPE :case V4L2_CID_MPEG_VIDEO_H264_FMO_MAP_TYPE : +case V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE :case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL :case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE :case V4L2_CID_MPEG_VIDEO_MPEG4_LEVEL :include/media/h264-ctrls.h +7
-0@@ -26,6 +26,7 @@#define V4L2_CID_MPEG_VIDEO_H264_SCALING_MATRIX (V4L2_CID_MPEG_BASE+1002)#define V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS (V4L2_CID_MPEG_BASE+1003)#define V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS (V4L2_CID_MPEG_BASE+1004) +#define V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE (V4L2_CID_MPEG_BASE+1005)/* enum v4l2_ctrl_type type values */ #define V4L2_CTRL_TYPE_H264_SPS 0x0110@@ -33,6 +34,12 @@#define V4L2_CTRL_TYPE_H264_SCALING_MATRIX 0x0112#define V4L2_CTRL_TYPE_H264_SLICE_PARAMS 0x0113#define V4L2_CTRL_TYPE_H264_DECODE_PARAMS 0x0114 +#define V4L2_CTRL_TYPE_H264_DECODING_MODE 0x0115 + +enum v4l2_mpeg_video_h264_decoding_mode { +V4L2_MPEG_VIDEO_H264_SLICE_BASED_DECODING , +V4L2_MPEG_VIDEO_H264_FRAME_BASED_DECODING , +};#define V4L2_H264_SPS_CONSTRAINT_SET0_FLAG 0x01#define V4L2_H264_SPS_CONSTRAINT_SET1_FLAG 0x02
include/media/h264-ctrls.h struct v4l2_h264_pred_weight_table { · line 141
Quoted 10 lines
include/media/h264-ctrls.h +4
-0@@ -125,6 +132,10 @@ struct v4l2_h264_pred_weight_table {struct v4l2_ctrl_h264_slice_params {/* Size in bytes, including header */ __u32 size; + +/* Offset in bytes to the start of slice in the OUTPUT buffer. */ +__u32 start_byte_offset; +/* Offset in bits to slice_data() from the beginning of this slice. */ __u32 header_bit_size;
Regards,
Hans
Re: [PATCH v4 03/11] media: uapi: h264: Add the concept of decoding mode
Quoted 70 lines · Hans Verkuil
Quoted 56 lines · Ezequiel Garcia
From: Boris Brezillon <[email protected]>
Some stateless decoders don't support per-slice decoding granularity (or at least not in a way that would make them efficient or easy to use).
Expose a menu to control the supported decoding modes. Drivers are allowed to support only one decoding but they can support both too.
Signed-off-by Boris Brezillon <[email protected]>Reviewed-by Paul Kocialkowski <[email protected]>---
Changes in v4:
* Typos/rewording fixesChanges in v3:
* s/per-{slice,frame} decoding/{slice,frame}-based decoding/
* Add Paul's R-bChanges in v2:
* Allow decoding multiple slices in per-slice decoding mode
* Minor doc improvement/fixes--- .../media/uapi/v4l/ext-ctrls-codec.rst | 43 ++++++++++++++++++- drivers/media/v4l2-core/v4l2-ctrls.c | 9 ++++ include/media/h264-ctrls.h | 11 +++++ 3 files changed, 62 insertions(+), 1 deletion(-)Documentation/media/uapi/v4l/ext-ctrls-codec.rst +9
-1@@ -1747,6 +1747,11 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type - * - __u32 - ``size`` - + * - __u32 + - ``start_byte_offset`` + - Where the slice payload starts in the output buffer. Useful when the + OUTPUT buffer contains more than one slice (some codecs need to know + where each slice starts in this buffer). * - __u32 - ``header_bit_size`` -@@ -1930,7 +1935,10 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type - - * - __u16 - ``num_slices`` - - Number of slices needed to decode the current frame + - Number of slices needed to decode the current frame/field. When + operating in slice-based decoding mode (see + :c:type:`v4l2_mpeg_video_h264_decoding_mode`), this field + should always be set to one. * - __u16 - ``nal_ref_idc`` - NAL reference ID value coming from the NAL Unit headerDocumentation/media/uapi/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 2036
Quoted 8 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +5
-0@@ -2021,6 +2029,39 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type - - 0x00000004 - The DPB entry is a long term reference frame +``V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE (enum)`` + Specifies the decoding mode to use. Currently exposes slice-based and + frame-based decoding but new modes might be added later on. + Drivers may expose a single or multiple decoding modes, + depending on what they can support.This should mention that this is valid for
V4L2_PIX_FMT_H264_SLICEpixelformat only. And it is a required control (i.e. it must always be present for that pixelformat). Ditto for the start code control.
Quoted 4 lines
Also, the
V4L2_PIX_FMT_H264_SLICEshould refer to this control and the start code control as well since these controls effectively are modifiers for the pixelformat.
Right, this makes perfect sense.
Quoted 30 lines
Quoted 15 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +15
-0+ + .. note:: + + This menu control is not yet part of the public kernel API and + it is expected to change. + +.. c:type:: v4l2_mpeg_video_h264_decoding_mode + +.. cssclass:: longtable + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 1 1 2 +Documentation/media/uapi/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 2061
Quoted 10 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +10
-0+ * - ``V4L2_MPEG_VIDEO_H264_SLICE_BASED_DECODING`` + - 0 + - The decoding is done at the slice granularity. + v4l2_ctrl_h264_decode_params->num_slices should be set to 1. + The OUTPUT buffer must contain a single slice. + * - ``V4L2_MPEG_VIDEO_H264_FRAME_BASED_DECODING`` + - 1 + - The decoding is done at the frame granularity. + v4l2_ctrl_h264_decode_params->num_slices should be set to the number of + slices forming a frame.Actually, I wonder if there shouldn't be a third option:
FIELD_BASED_DECODINGto indicate that the hardware expects the slices of a single field in a buffer instead of both fields.
Quoted 3 lines
Or just stick to
FRAME_BASED_DECODINGfor now but explicitly mention that in the case of interlaced decoding both fields must be present in the buffer.
OK.
Quoted 2 lines
A
FIELD_BASED_DECODINGoption can always be added later if necessary.
Thanks,
Ezequiel
PATCH v4 4/11 media: uapi: h264: Add the concept of start code
In reply to Ezequiel GarciaStateless decoders have different expectations about the start code that is prepended on H264 slices. Add a menu control to express the supported start code types (including no start code).
Drivers are allowed to support only one start code type, but they can support both too.
Note that this is independent of the H264 decoding mode, which specifies the granularity of the decoding operations. Either in frame-based or slice-based mode, this new control will allow to define the start code expected on H264 slices.
---
Changes in v4:
* New patch.
--- .../media/uapi/v4l/ext-ctrls-codec.rst | 31 +++++++++++++++++++ drivers/media/v4l2-core/v4l2-ctrls.c | 9 ++++++ include/media/h264-ctrls.h | 6 ++++ 3 files changed, 46 insertions(+)
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +31 -0
@@ -2062,6 +2062,37 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type - The OUTPUT buffer should contain all slices needed to decode the frame/field. +``V4L2_CID_MPEG_VIDEO_H264_STARTCODE (enum)`` + Specifies the H264 slice start code expected for each slice. + This control shall e used to complement V4L2_PIX_FMT_H264_SLICE + pixel format. Drivers may expose a single or multiple + start codes, depending on what they can support. + + .. note:: + + This menu control is not yet part of the public kernel API and + it is expected to change. + +.. c:type:: v4l2_mpeg_video_h264_startcode + +.. cssclass:: longtable + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 1 1 2 + + * - ``V4L2_MPEG_VIDEO_H264_NO_STARTCODE`` + - 0 + - Selecting this value specifies that H264 slices are passed + to the driver without any start code. + Bla. + * - ``V4L2_MPEG_VIDEO_H264_ANNEX_B_STARTCODE`` + - 1 + - Selecting this value specifies that H264 slices are expected + to be prefixed by Annex B start codes. According to :ref:`h264` + valid start codes can be 3-bytes 0x000001, or 4-bytes 0x00000001. + .. _v4l2-mpeg-mpeg2: ``V4L2_CID_MPEG_VIDEO_MPEG2_SLICE_PARAMS (struct)``
drivers/media/v4l2-core/v4l2-ctrls.c +9 -0
@@ -407,6 +407,11 @@ const char * const *v4l2_ctrl_get_menu(u32 id)"Frame-Based" , NULL, }; +static const char *const h264_start_code[] = { +"No Start Code" , +"Annex B Start Code" , +NULL , + };static const char *const mpeg_mpeg2_level[] = {"Low" , "Main",
@@ -640,6 +645,8 @@ const char * const *v4l2_ctrl_get_menu(u32 id)return h264_fmo_map_type;case V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE :return h264_decoding_mode; +case V4L2_CID_MPEG_VIDEO_H264_STARTCODE : +return h264_start_code;case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL :return mpeg_mpeg2_level;case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE :
@@ -860,6 +867,7 @@ const char *v4l2_ctrl_get_name(u32 id)case V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS :return "H264 Slice Parameters" ;case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS :return "H264 Decode Parameters" ;case V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE :return "H264 Decoding Mode" ; +case V4L2_CID_MPEG_VIDEO_H264_STARTCODE :return "H264 Start Code" ;case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL :return "MPEG2 Level" ;case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE :return "MPEG2 Profile" ;case V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP :return "MPEG4 I-Frame QP Value" ;
@@ -1229,6 +1237,7 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,case V4L2_CID_MPEG_VIDEO_H264_SEI_FP_ARRANGEMENT_TYPE :case V4L2_CID_MPEG_VIDEO_H264_FMO_MAP_TYPE :case V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE : +case V4L2_CID_MPEG_VIDEO_H264_STARTCODE :case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL :case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE :case V4L2_CID_MPEG_VIDEO_MPEG4_LEVEL :
include/media/h264-ctrls.h +6 -0
@@ -27,6 +27,7 @@#define V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS (V4L2_CID_MPEG_BASE+1003)#define V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS (V4L2_CID_MPEG_BASE+1004)#define V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE (V4L2_CID_MPEG_BASE+1005) +#define V4L2_CID_MPEG_VIDEO_H264_STARTCODE (V4L2_CID_MPEG_BASE+1006)/* enum v4l2_ctrl_type type values */ #define V4L2_CTRL_TYPE_H264_SPS 0x0110
@@ -41,6 +42,11 @@ enum v4l2_mpeg_video_h264_decoding_mode { V4L2_MPEG_VIDEO_H264_FRAME_BASED_DECODING, }; +enum v4l2_mpeg_video_h264_start_code { +V4L2_MPEG_VIDEO_H264_NO_STARTCODE , +V4L2_MPEG_VIDEO_H264_ANNEX_B_STARTCODE , +}; +#define V4L2_H264_SPS_CONSTRAINT_SET0_FLAG 0x01#define V4L2_H264_SPS_CONSTRAINT_SET1_FLAG 0x02#define V4L2_H264_SPS_CONSTRAINT_SET2_FLAG 0x04
Signature · 1 line
2.22.0
Re: [PATCH v4 04/11] media: uapi: h264: Add the concept of start code
Quoted 28 lines · Ezequiel Garcia
Stateless decoders have different expectations about the start code that is prepended on H264 slices. Add a menu control to express the supported start code types (including no start code).
Drivers are allowed to support only one start code type, but they can support both too.
Note that this is independent of the H264 decoding mode, which specifies the granularity of the decoding operations. Either in frame-based or slice-based mode, this new control will allow to define the start code expected on H264 slices.
Signed-off-by Ezequiel Garcia <[email protected]>---
Changes in v4:
* New patch.--- .../media/uapi/v4l/ext-ctrls-codec.rst | 31 +++++++++++++++++++ drivers/media/v4l2-core/v4l2-ctrls.c | 9 ++++++ include/media/h264-ctrls.h | 6 ++++ 3 files changed, 46 insertions(+)Documentation/media/uapi/v4l/ext-ctrls-codec.rst +0
-0
Documentation/media/uapi/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 2067
Quoted 6 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +3
-0@@ -2062,6 +2062,37 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type - The OUTPUT buffer should contain all slices needed to decode the frame/field. +``V4L2_CID_MPEG_VIDEO_H264_STARTCODE (enum)`` + Specifies the H264 slice start code expected for each slice. + This control shall e used to complement V4L2_PIX_FMT_H264_SLICE
e -> be
Quoted 12 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +12
-0+ pixel format. Drivers may expose a single or multiple + start codes, depending on what they can support. + + .. note:: + + This menu control is not yet part of the public kernel API and + it is expected to change. + +.. c:type:: v4l2_mpeg_video_h264_startcode + +.. cssclass:: longtable +
Documentation/media/uapi/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 2089
Quoted 10 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +10
-0+.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 1 1 2 + + * - ``V4L2_MPEG_VIDEO_H264_NO_STARTCODE`` + - 0 + - Selecting this value specifies that H264 slices are passed + to the driver without any start code. + Bla.
Bla?
Documentation/media/uapi/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 2095
Quoted 6 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +6
-0+ * - ``V4L2_MPEG_VIDEO_H264_ANNEX_B_STARTCODE`` + - 1 + - Selecting this value specifies that H264 slices are expected + to be prefixed by Annex B start codes. According to :ref:`h264` + valid start codes can be 3-bytes 0x000001, or 4-bytes 0x00000001. +
I had the impression that it is more common to require startcodes. If that's indeed the case, shouldn't this have value 0 instead of 1?
Quoted 59 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +0
-0.. _v4l2-mpeg-mpeg2: ``V4L2_CID_MPEG_VIDEO_MPEG2_SLICE_PARAMS (struct)``drivers/media/v4l2-core/v4l2-ctrls.c +9
-0@@ -407,6 +407,11 @@ const char * const *v4l2_ctrl_get_menu(u32 id)"Frame-Based" , NULL, }; +static const char *const h264_start_code[] = { +"No Start Code" , +"Annex B Start Code" , +NULL , + };static const char *const mpeg_mpeg2_level[] = {"Low" , "Main",@@ -640,6 +645,8 @@ const char * const *v4l2_ctrl_get_menu(u32 id)return h264_fmo_map_type;case V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE :return h264_decoding_mode; +case V4L2_CID_MPEG_VIDEO_H264_STARTCODE : +return h264_start_code;case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL :return mpeg_mpeg2_level;case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE :@@ -860,6 +867,7 @@ const char *v4l2_ctrl_get_name(u32 id)case V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS :return "H264 Slice Parameters" ;case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS :return "H264 Decode Parameters" ;case V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE :return "H264 Decoding Mode" ; +case V4L2_CID_MPEG_VIDEO_H264_STARTCODE :return "H264 Start Code" ;case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL :return "MPEG2 Level" ;case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE :return "MPEG2 Profile" ;case V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP :return "MPEG4 I-Frame QP Value" ;@@ -1229,6 +1237,7 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,case V4L2_CID_MPEG_VIDEO_H264_SEI_FP_ARRANGEMENT_TYPE :case V4L2_CID_MPEG_VIDEO_H264_FMO_MAP_TYPE :case V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE : +case V4L2_CID_MPEG_VIDEO_H264_STARTCODE :case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL :case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE :case V4L2_CID_MPEG_VIDEO_MPEG4_LEVEL :include/media/h264-ctrls.h +1
-0@@ -27,6 +27,7 @@#define V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS (V4L2_CID_MPEG_BASE+1003)#define V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS (V4L2_CID_MPEG_BASE+1004)#define V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE (V4L2_CID_MPEG_BASE+1005) +#define V4L2_CID_MPEG_VIDEO_H264_STARTCODE (V4L2_CID_MPEG_BASE+1006)/* enum v4l2_ctrl_type type values */ #define V4L2_CTRL_TYPE_H264_SPS 0x0110@@ -41,6 +42,11 @@ enum v4l2_mpeg_video_h264_decoding_mode { V4L2_MPEG_VIDEO_H264_FRAME_BASED_DECODING,
include/media/h264-ctrls.h enum v4l2_mpeg_video_h264_decoding_mode { · line 52
Quoted 10 lines
include/media/h264-ctrls.h +5
-0}; +enum v4l2_mpeg_video_h264_start_code { +V4L2_MPEG_VIDEO_H264_NO_STARTCODE , +V4L2_MPEG_VIDEO_H264_ANNEX_B_STARTCODE , +}; +#define V4L2_H264_SPS_CONSTRAINT_SET0_FLAG 0x01#define V4L2_H264_SPS_CONSTRAINT_SET1_FLAG 0x02#define V4L2_H264_SPS_CONSTRAINT_SET2_FLAG 0x04
Regards,
Hans
Re: [PATCH v4 04/11] media: uapi: h264: Add the concept of start code
Quoted 54 lines · Hans Verkuil
Quoted 28 lines · Ezequiel Garcia
Stateless decoders have different expectations about the start code that is prepended on H264 slices. Add a menu control to express the supported start code types (including no start code).
Drivers are allowed to support only one start code type, but they can support both too.
Note that this is independent of the H264 decoding mode, which specifies the granularity of the decoding operations. Either in frame-based or slice-based mode, this new control will allow to define the start code expected on H264 slices.
Signed-off-by Ezequiel Garcia <[email protected]>---
Changes in v4:
* New patch.--- .../media/uapi/v4l/ext-ctrls-codec.rst | 31 +++++++++++++++++++ drivers/media/v4l2-core/v4l2-ctrls.c | 9 ++++++ include/media/h264-ctrls.h | 6 ++++ 3 files changed, 46 insertions(+)Documentation/media/uapi/v4l/ext-ctrls-codec.rst +0
-0Documentation/media/uapi/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 2067
Quoted 6 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +3
-0@@ -2062,6 +2062,37 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type - The OUTPUT buffer should contain all slices needed to decode the frame/field. +``V4L2_CID_MPEG_VIDEO_H264_STARTCODE (enum)`` + Specifies the H264 slice start code expected for each slice. + This control shall e used to complement V4L2_PIX_FMT_H264_SLICEe -> be
Quoted 16 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +16
-0+ pixel format. Drivers may expose a single or multiple + start codes, depending on what they can support. + + .. note:: + + This menu control is not yet part of the public kernel API and + it is expected to change. + +.. c:type:: v4l2_mpeg_video_h264_startcode + +.. cssclass:: longtable + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 1 1 2
Quoted 9 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 2089
Quoted 6 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +6
-0+ + * - ``V4L2_MPEG_VIDEO_H264_NO_STARTCODE`` + - 0 + - Selecting this value specifies that H264 slices are passed + to the driver without any start code. + Bla.Bla?
Well, despite how many times I reviewed this doc, it seems this slipped through :-(
Quoted 11 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 2095
Quoted 6 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +6
-0+ * - ``V4L2_MPEG_VIDEO_H264_ANNEX_B_STARTCODE`` + - 1 + - Selecting this value specifies that H264 slices are expected + to be prefixed by Annex B start codes. According to :ref:`h264` + valid start codes can be 3-bytes 0x000001, or 4-bytes 0x00000001. +I had the impression that it is more common to require startcodes. If that's indeed the case, shouldn't this have value 0 instead of 1?
1?
I'm not sure this is indeed the case.
For instance, VA-API accelerators work on H264 slices that are not prepended by the NALU start code.
I'm fine flipping the values, but at this point, with only cedrus and hantro, there's doesn't seem to be a "natural" choice.
Thanks,
Ezequiel
Re: [PATCH v4 04/11] media: uapi: h264: Add the concept of start code
Quoted 80 lines · Ezequiel Garcia
Quoted 54 lines · Hans Verkuil
Quoted 28 lines · Ezequiel Garcia
Stateless decoders have different expectations about the start code that is prepended on H264 slices. Add a menu control to express the supported start code types (including no start code).
Drivers are allowed to support only one start code type, but they can support both too.
Note that this is independent of the H264 decoding mode, which specifies the granularity of the decoding operations. Either in frame-based or slice-based mode, this new control will allow to define the start code expected on H264 slices.
Signed-off-by Ezequiel Garcia <[email protected]>---
Changes in v4:
* New patch.--- .../media/uapi/v4l/ext-ctrls-codec.rst | 31 +++++++++++++++++++ drivers/media/v4l2-core/v4l2-ctrls.c | 9 ++++++ include/media/h264-ctrls.h | 6 ++++ 3 files changed, 46 insertions(+)Documentation/media/uapi/v4l/ext-ctrls-codec.rst +0
-0Documentation/media/uapi/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 2067
Quoted 6 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +3
-0@@ -2062,6 +2062,37 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type - The OUTPUT buffer should contain all slices needed to decode the frame/field. +``V4L2_CID_MPEG_VIDEO_H264_STARTCODE (enum)`` + Specifies the H264 slice start code expected for each slice. + This control shall e used to complement V4L2_PIX_FMT_H264_SLICEe -> be
Quoted 16 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +16
-0+ pixel format. Drivers may expose a single or multiple + start codes, depending on what they can support. + + .. note:: + + This menu control is not yet part of the public kernel API and + it is expected to change. + +.. c:type:: v4l2_mpeg_video_h264_startcode + +.. cssclass:: longtable + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 1 1 2Quoted 9 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 2089
Quoted 6 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +6
-0+ + * - ``V4L2_MPEG_VIDEO_H264_NO_STARTCODE`` + - 0 + - Selecting this value specifies that H264 slices are passed + to the driver without any start code. + Bla.Bla?
Well, despite how many times I reviewed this doc, it seems this slipped through :-(
Quoted 11 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst enum v4l2_mpeg_video_h264_hierarchical_coding_type - · line 2095
Quoted 6 lines
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +6
-0+ * - ``V4L2_MPEG_VIDEO_H264_ANNEX_B_STARTCODE`` + - 1 + - Selecting this value specifies that H264 slices are expected + to be prefixed by Annex B start codes. According to :ref:`h264` + valid start codes can be 3-bytes 0x000001, or 4-bytes 0x00000001. +I had the impression that it is more common to require startcodes. If that's indeed the case, shouldn't this have value 0 instead of 1?
1?
Quoted 7 lines
I'm not sure this is indeed the case.
For instance, VA-API accelerators work on H264 slices that are not prepended by the NALU start code.
I'm fine flipping the values, but at this point, with only cedrus and hantro, there's doesn't seem to be a "natural" choice.
OK, let's keep this as-is then.
Regards,
Hans
Quoted 4 lines
Thanks,
Ezequiel
PATCH v4 5/11 media: uapi: h264: Get rid of the p0/b0/b1 ref-lists
In reply to Ezequiel GarciaFrom: Boris Brezillon <[email protected]>
Those lists can be extracted from the dpb, let's simplify userspace life and build that list kernel-side (generic helpers will be provided for drivers that need this list).
---
Changes in v4:
* Add R-b
* Remove the reflist from the struct
Changes in v3:
* None
Changes in v2:
* None
--- Documentation/media/uapi/v4l/ext-ctrls-codec.rst | 9 --------- include/media/h264-ctrls.h | 3 --- 2 files changed, 12 deletions(-)
Documentation/media/uapi/v4l/ext-ctrls-codec.rst +0 -9
@@ -1942,15 +1942,6 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type - * - __u16 - ``nal_ref_idc`` - NAL reference ID value coming from the NAL Unit header - * - __u8 - - ``ref_pic_list_p0[32]`` - - Backward reference list used by P-frames in the original bitstream order - * - __u8 - - ``ref_pic_list_b0[32]`` - - Backward reference list used by B-frames in the original bitstream order - * - __u8 - - ``ref_pic_list_b1[32]`` - - Forward reference list used by B-frames in the original bitstream order * - __s32 - ``top_field_order_cnt`` - Picture Order Count for the coded top field
include/media/h264-ctrls.h +0 -3
@@ -203,9 +203,6 @@ struct v4l2_ctrl_h264_decode_params {struct v4l2_h264_dpb_entry dpb[16 ];__u16 num_slices;__u16 nal_ref_idc; -__u8 ref_pic_list_p0[32 ]; -__u8 ref_pic_list_b0[32 ]; -__u8 ref_pic_list_b1[32 ];__s32 top_field_order_cnt;__s32 bottom_field_order_cnt;__u32 flags;/* V4L2_H264_DECODE_PARAM_FLAG_* */
Signature · 1 line
2.22.0
Re: [PATCH v4 05/11] media: uapi: h264: Get rid of the p0/b0/b1 ref-lists
Quoted 6 lines · Ezequiel Garcia
From: Boris Brezillon <[email protected]>
Those lists can be extracted from the dpb, let's simplify userspace life and build that list kernel-side (generic helpers will be provided for drivers that need this list).
Quoted 7 lines
Signed-off-by Boris Brezillon <[email protected]>Reviewed-by Nicolas Dufresne <[email protected]>Reviewed-by Ezequiel Garcia <[email protected]>Reviewed-by Paul Kocialkowski <[email protected]>---
Changes in v4:
* Add R-b
???
Nothing was added, left-over from some old version?
Regards,
Hans
Quoted 47 lines
* Remove the reflist from the struct
Changes in v3:
* NoneChanges in v2:
* None--- Documentation/media/uapi/v4l/ext-ctrls-codec.rst | 9 --------- include/media/h264-ctrls.h | 3 --- 2 files changed, 12 deletions(-)Documentation/media/uapi/v4l/ext-ctrls-codec.rst +0
-9@@ -1942,15 +1942,6 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type - * - __u16 - ``nal_ref_idc`` - NAL reference ID value coming from the NAL Unit header - * - __u8 - - ``ref_pic_list_p0[32]`` - - Backward reference list used by P-frames in the original bitstream order - * - __u8 - - ``ref_pic_list_b0[32]`` - - Backward reference list used by B-frames in the original bitstream order - * - __u8 - - ``ref_pic_list_b1[32]`` - - Forward reference list used by B-frames in the original bitstream order * - __s32 - ``top_field_order_cnt`` - Picture Order Count for the coded top fieldinclude/media/h264-ctrls.h +0
-3@@ -203,9 +203,6 @@ struct v4l2_ctrl_h264_decode_params {struct v4l2_h264_dpb_entry dpb[16 ];__u16 num_slices;__u16 nal_ref_idc; -__u8 ref_pic_list_p0[32 ]; -__u8 ref_pic_list_b0[32 ]; -__u8 ref_pic_list_b1[32 ];__s32 top_field_order_cnt;__s32 bottom_field_order_cnt;__u32 flags;/* V4L2_H264_DECODE_PARAM_FLAG_* */
Re: [PATCH v4 05/11] media: uapi: h264: Get rid of the p0/b0/b1 ref-lists
Quoted 15 lines · Hans Verkuil
Quoted 6 lines · Ezequiel Garcia
From: Boris Brezillon <[email protected]>
Those lists can be extracted from the dpb, let's simplify userspace life and build that list kernel-side (generic helpers will be provided for drivers that need this list).
Quoted 7 lines
Signed-off-by Boris Brezillon <[email protected]>Reviewed-by Nicolas Dufresne <[email protected]>Reviewed-by Ezequiel Garcia <[email protected]>Reviewed-by Paul Kocialkowski <[email protected]>---
Changes in v4:
* Add R-b
Quoted 4 lines
???
Nothing was added, left-over from some old version?
Compared to v3, this version adds the Reviewed-by tags, which is what the Add R-b means.
Sorry if that wasn't clear! :-)
PATCH v4 6/11 media: cedrus: Cleanup control initialization
In reply to Ezequiel GarciaIn order to introduce other controls, the control initialization needs to support an initial struct v4l2_ctrl_control.
While here, let's cleanup the control initialization, removing unneeded fields.
---
Changes in v4:
* New patch.
--- drivers/staging/media/sunxi/cedrus/cedrus.c | 45 +++++++++++---------- drivers/staging/media/sunxi/cedrus/cedrus.h | 3 +- 2 files changed, 25 insertions(+), 23 deletions(-)
drivers/staging/media/sunxi/cedrus/cedrus.c +24 -21
@@ -29,44 +29,51 @@static const struct cedrus_control cedrus_controls[] = { { - .id =V4L2_CID_MPEG_VIDEO_MPEG2_SLICE_PARAMS , - .elem_size =sizeof (struct v4l2_ctrl_mpeg2_slice_params ), + .cfg = { + .id =V4L2_CID_MPEG_VIDEO_MPEG2_SLICE_PARAMS , + }, .codec =CEDRUS_CODEC_MPEG2 , .required = true, }, { - .id =V4L2_CID_MPEG_VIDEO_MPEG2_QUANTIZATION , - .elem_size =sizeof (struct v4l2_ctrl_mpeg2_quantization ), + .cfg = { + .id =V4L2_CID_MPEG_VIDEO_MPEG2_QUANTIZATION , + }, .codec =CEDRUS_CODEC_MPEG2 , .required = false, }, { - .id =V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS , - .elem_size =sizeof (struct v4l2_ctrl_h264_decode_params ), + .cfg = { + .id =V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS , + }, .codec =CEDRUS_CODEC_H264 , .required = true, }, { - .id =V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS , - .elem_size =sizeof (struct v4l2_ctrl_h264_slice_params ), + .cfg = { + .id =V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS , + }, .codec =CEDRUS_CODEC_H264 , .required = true, }, { - .id =V4L2_CID_MPEG_VIDEO_H264_SPS , - .elem_size =sizeof (struct v4l2_ctrl_h264_sps ), + .cfg = { + .id =V4L2_CID_MPEG_VIDEO_H264_SPS , + }, .codec =CEDRUS_CODEC_H264 , .required = true, }, { - .id =V4L2_CID_MPEG_VIDEO_H264_PPS , - .elem_size =sizeof (struct v4l2_ctrl_h264_pps ), + .cfg = { + .id =V4L2_CID_MPEG_VIDEO_H264_PPS , + }, .codec =CEDRUS_CODEC_H264 , .required = true, }, { - .id =V4L2_CID_MPEG_VIDEO_H264_SCALING_MATRIX , - .elem_size =sizeof (struct v4l2_ctrl_h264_scaling_matrix ), + .cfg = { + .id =V4L2_CID_MPEG_VIDEO_H264_SCALING_MATRIX , + }, .codec =CEDRUS_CODEC_H264 , .required = true, },
@@ -106,12 +113,8 @@ static int cedrus_init_ctrls(struct cedrus_dev *dev, struct cedrus_ctx *ctx)return -ENOMEM ;for (i =0 ; i <CEDRUS_CONTROLS_COUNT ; i++) { -struct v4l2_ctrl_config cfg = {}; - - cfg.elem_size = cedrus_controls[i].elem_size; - cfg.id = cedrus_controls[i].id; - - ctrl =v4l2_ctrl_new_custom (hdl, &cfg,NULL ); + ctrl =v4l2_ctrl_new_custom (hdl, &cedrus_controls[i].cfg, +NULL );if (hdl->error) {v4l2_err (&dev->v4l2_dev,"Failed to create new custom control\n" );
@@ -178,7 +181,7 @@ static int cedrus_request_validate(struct media_request *req)continue ; ctrl_test =v4l2_ctrl_request_hdl_ctrl_find (hdl, - cedrus_controls[i].id); + cedrus_controls[i].cfg.id);if (!ctrl_test) {v4l2_info (&ctx->dev->v4l2_dev,"Missing required codec control\n" );
drivers/staging/media/sunxi/cedrus/cedrus.h +1 -2
@@ -49,8 +49,7 @@ enum cedrus_h264_pic_type { };struct cedrus_control { -u32 id; -u32 elem_size; +struct v4l2_ctrl_config cfg;enum cedrus_codec codec;unsigned char required:1 ; };
Signature · 1 line
2.22.0
PATCH v4 7/11 media: cedrus: Specify the required H264 start code
In reply to Ezequiel GarciaThe cedrus VPU expects V4L2_PIX_FMT_H264_SLICE buffers to contain H264 slices with no start code. Expose this to userspace with the newly added menu control.
---
Changes in v4:
* New patch.
--- drivers/staging/media/sunxi/cedrus/cedrus.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
drivers/staging/media/sunxi/cedrus/cedrus.c +20 -0
@@ -77,6 +77,26 @@ static const struct cedrus_control cedrus_controls[] = { .codec = CEDRUS_CODEC_H264, .required = true, }, + { + .cfg = { + .id = V4L2_CID_MPEG_VIDEO_H264_DECODING_MODE, + .max = V4L2_MPEG_VIDEO_H264_SLICE_BASED_DECODING, + .def = V4L2_MPEG_VIDEO_H264_SLICE_BASED_DECODING, + .menu_skip_mask = BIT(V4L2_MPEG_VIDEO_H264_FRAME_BASED_DECODING), + }, + .codec = CEDRUS_CODEC_H264, + .required = false, + }, + { + .cfg = { + .id = V4L2_CID_MPEG_VIDEO_H264_STARTCODE, + .max = V4L2_MPEG_VIDEO_H264_NO_STARTCODE, + .def = V4L2_MPEG_VIDEO_H264_NO_STARTCODE, + .menu_skip_mask = BIT(V4L2_MPEG_VIDEO_H264_ANNEX_B_STARTCODE), + }, + .codec = CEDRUS_CODEC_H264, + .required = false, + }, };#define CEDRUS_CONTROLS_COUNT ARRAY_SIZE(cedrus_controls)
Signature · 1 line
2.22.0
PATCH v4 8/11 media: hantro: Move copy_metadata() before doing a decode operation
In reply to Ezequiel GarciaFrom: Boris Brezillon <boris.brezillon-ZGY8ohtN/[email protected]>
Some decoders use intra slice/frame references. The capture buffer pointed by these references might be new and thus have invalid timestamp which prevents the decoder logic from retrieving the vb2_buffer object based on the output buf timestamp.
Copy all metadata (including the timestamp) before starting the decode operation.
---
Changes in v4:
* None.
--- drivers/staging/media/hantro/hantro_drv.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/staging/media/hantro/hantro_drv.c +6 -2
@@ -111,8 +111,6 @@ static void hantro_job_finish(struct hantro_dev *vpu, src->sequence = ctx->sequence_out++; dst->sequence = ctx->sequence_cap++; -v4l2_m2m_buf_copy_metadata (src, dst, true); - ret = ctx->buf_finish (ctx, &dst->vb2_buf, bytesused);if (ret) result =VB2_BUF_STATE_ERROR ;
@@ -178,8 +176,12 @@ void hantro_finish_run(struct hantro_ctx *ctx)static void device_run (void *priv) {struct hantro_ctx *ctx = priv; +struct vb2_v4l2_buffer *src, *dst;int ret; + src =hantro_get_src_buf (ctx); + dst =hantro_get_dst_buf (ctx); + ret =clk_bulk_enable (ctx->dev->variant->num_clocks, ctx->dev->clocks);if (ret) goto err_cancel_job;
@@ -187,6 +189,8 @@ static void device_run(void *priv)if (ret <0 ) goto err_cancel_job; +v4l2_m2m_buf_copy_metadata (src, dst, true); + ctx->codec_ops->run (ctx);return ;
Signature · 1 line
2.22.0
PATCH v4 9/11 media: hantro: Add core bits to support H264 decoding
In reply to Ezequiel GarciaFrom: Hertz Wong <[email protected]>
Add helpers and patch hantro_{drv,v4l2}.c to prepare addition of H264 decoding support.
---
Changes in v4:
* Rework extra_size0, exposing the size via TRY_FMT/S_FMT
to allow buffer importation, as suggested by Tomasz.
* Drop max slice limit.
* Use a ternary operator instead of substracting POCs,
to avoid an overflow as pointed out by Rasmus.
* Specify annex B slice start code.
* Add missing extra_size0 to sizeimage
* Swap the first 2 entries of list B1 when B0 and B1 match (mandated by the spec)
* Move the update_dpb() call before the prepare_table() one to make the POCs stored in the private table match the content of the new DPB
--- drivers/staging/media/hantro/Makefile | 1 + drivers/staging/media/hantro/hantro.h | 9 +- drivers/staging/media/hantro/hantro_drv.c | 42 ++ drivers/staging/media/hantro/hantro_h264.c | 649 +++++++++++++++++++++ drivers/staging/media/hantro/hantro_hw.h | 55 ++ drivers/staging/media/hantro/hantro_v4l2.c | 10 + 6 files changed, 765 insertions(+), 1 deletion(-) create mode 100644 drivers/staging/media/hantro/hantro_h264.c
drivers/staging/media/hantro/Makefile +1 -0
@@ -10,6 +10,7 @@ hantro-vpu-y += \ rk3399_vpu_hw_mpeg2_dec.o \ rk3399_vpu_hw_vp8_dec.o \ hantro_jpeg.o \ + hantro_h264.o \ hantro_mpeg2.o \ hantro_vp8.o
drivers/staging/media/hantro/hantro.h +8 -1
@@ -30,6 +30,10 @@#define VP8_MB_WIDTH (w) DIV_ROUND_UP(w, VP8_MB_DIM)#define VP8_MB_HEIGHT (h) DIV_ROUND_UP(h, VP8_MB_DIM) +#define H264_MB_DIM 16 +#define H264_MB_WIDTH (w) DIV_ROUND_UP(w, H264_MB_DIM) +#define H264_MB_HEIGHT (h) DIV_ROUND_UP(h, H264_MB_DIM) +#define MPEG2_MB_DIM 16#define MPEG2_MB_WIDTH (w) DIV_ROUND_UP(w, MPEG2_MB_DIM)#define MPEG2_MB_HEIGHT (h) DIV_ROUND_UP(h, MPEG2_MB_DIM)
@@ -43,9 +47,9 @@ struct hantro_codec_ops;#define HANTRO_JPEG_ENCODER BIT(0)#define HANTRO_ENCODERS 0x0000ffff -#define HANTRO_MPEG2_DECODER BIT(16)#define HANTRO_VP8_DECODER BIT(17) +#define HANTRO_H264_DECODER BIT(18)#define HANTRO_DECODERS 0xffff0000/**
@@ -102,12 +106,14 @@ struct hantro_variant {* enum hantro_codec_mode - codec operating mode. * @HANTRO_MODE_NONE: No operating mode. Used for RAW video formats. * @HANTRO_MODE_JPEG_ENC: JPEG encoder. +* @HANTRO_MODE_H264_DEC: H264 decoder. * @HANTRO_MODE_MPEG2_DEC: MPEG-2 decoder. * @HANTRO_MODE_VP8_DEC: VP8 decoder. */ enum hantro_codec_mode {HANTRO_MODE_NONE =-1 ,HANTRO_MODE_JPEG_ENC , +HANTRO_MODE_H264_DEC ,HANTRO_MODE_MPEG2_DEC ,HANTRO_MODE_VP8_DEC , };
@@ -246,6 +252,7 @@ struct hantro_ctx {/* Specific for particular codec modes. */ union { +struct hantro_h264_dec_hw_ctx h264_dec;struct hantro_jpeg_enc_hw_ctx jpeg_enc;struct hantro_mpeg2_dec_hw_ctx mpeg2_dec;struct hantro_vp8_dec_hw_ctx vp8_dec;
drivers/staging/media/hantro/hantro_drv.c +42 -0
48 lines not on this page · raw message
drivers/staging/media/hantro/hantro_h264.c new file +649 -0
649 lines not on this page · raw message
drivers/staging/media/hantro/hantro_hw.h +55 -0
73 lines not on this page · raw message
drivers/staging/media/hantro/hantro_v4l2.c +10 -0
@@ -239,6 +239,15 @@ static int vidioc_try_fmt(struct file *file, void *priv, struct v4l2_format *f,/* Fill remaining fields */ v4l2_fill_pixfmt_mp (pix_mp, fmt->fourcc, pix_mp->width, pix_mp->height); +/* +* The H264 decoder needs extra space on the output buffers +* to store motion vectors. This is needed for reference +* frames. +*/ +if (ctx->vpu_src_fmt->fourcc ==V4L2_PIX_FMT_H264_SLICE ) + pix_mp->plane_fmt[0 ].sizeimage += +128 *DIV_ROUND_UP (pix_mp->width,16 ) * +DIV_ROUND_UP (pix_mp->height,16 ); }else if (!pix_mp->plane_fmt[0 ].sizeimage) {/* * For coded formats the application can specify
@@ -345,6 +354,7 @@ hantro_update_requires_request(struct hantro_ctx *ctx, u32 fourcc)break ;case V4L2_PIX_FMT_MPEG2_SLICE :case V4L2_PIX_FMT_VP8_FRAME : +case V4L2_PIX_FMT_H264_SLICE : ctx->fh.m2m_ctx->out_q_ctx.q.requires_requests = true;break ;default :
Signature · 1 line
2.22.0
PATCH v4 10/11 media: hantro: Add support for H264 decoding on G1
In reply to Ezequiel GarciaFrom: Hertz Wong <[email protected]>
Add the G1 specific bits to support H264 decoding.
---
Changes from v4:
* Update to use pic_size from the H264 private context struct.
* Remove unused variable.
--- drivers/staging/media/hantro/Makefile | 1 + .../staging/media/hantro/hantro_g1_h264_dec.c | 292 ++++++++++++++++++ drivers/staging/media/hantro/hantro_hw.h | 1 + 3 files changed, 294 insertions(+) create mode 100644 drivers/staging/media/hantro/hantro_g1_h264_dec.c
drivers/staging/media/hantro/Makefile +1 -0
@@ -4,6 +4,7 @@ hantro-vpu-y += \ hantro_drv.o \ hantro_v4l2.o \ hantro_h1_jpeg_enc.o \ + hantro_g1_h264_dec.o \ hantro_g1_mpeg2_dec.o \ hantro_g1_vp8_dec.o \ rk3399_vpu_hw_jpeg_enc.o \
drivers/staging/media/hantro/hantro_g1_h264_dec.c new file +292 -0
292 lines not on this page · raw message
drivers/staging/media/hantro/hantro_hw.h +1 -0
@@ -161,6 +161,7 @@ void hantro_jpeg_enc_exit(struct hantro_ctx *ctx);struct vb2_buffer *hantro_h264_get_ref_buf (struct hantro_ctx *ctx,unsigned int dpb_idx);int hantro_h264_dec_prepare_run (struct hantro_ctx *ctx); +void hantro_g1_h264_dec_run (struct hantro_ctx *ctx);int hantro_h264_dec_init (struct hantro_ctx *ctx);void hantro_h264_dec_exit (struct hantro_ctx *ctx);
Signature · 1 line
2.22.0
PATCH v4 11/11 media: hantro: Enable H264 decoding on rk3288
In reply to Ezequiel GarciaFrom: Hertz Wong <[email protected]>
Now that the generic bits have been added, we can activate H264 decoding on rk3288.
---
Changes in v4:
* None.
--- drivers/staging/media/hantro/rk3288_vpu_hw.c | 21 +++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
drivers/staging/media/hantro/rk3288_vpu_hw.c +20 -1
@@ -61,6 +61,19 @@ static const struct hantro_fmt rk3288_vpu_dec_fmts[] = { .fourcc = V4L2_PIX_FMT_NV12, .codec_mode = HANTRO_MODE_NONE, }, + { + .fourcc = V4L2_PIX_FMT_H264_SLICE, + .codec_mode = HANTRO_MODE_H264_DEC, + .max_depth = 2, + .frmsize = { + .min_width = 48, + .max_width = 3840, + .step_width = H264_MB_DIM, + .min_height = 48, + .max_height = 2160, + .step_height = H264_MB_DIM, + }, + }, { .fourcc = V4L2_PIX_FMT_MPEG2_SLICE, .codec_mode = HANTRO_MODE_MPEG2_DEC,
@@ -162,6 +175,12 @@ static const struct hantro_codec_ops rk3288_vpu_codec_ops[] = { .init = hantro_jpeg_enc_init, .exit = hantro_jpeg_enc_exit, }, + [HANTRO_MODE_H264_DEC] = { + .run = hantro_g1_h264_dec_run, + .reset = rk3288_vpu_dec_reset, + .init = hantro_h264_dec_init, + .exit = hantro_h264_dec_exit, + }, [HANTRO_MODE_MPEG2_DEC] = { .run = hantro_g1_mpeg2_dec_run, .reset = rk3288_vpu_dec_reset,
@@ -197,7 +216,7 @@ const struct hantro_variant rk3288_vpu_variant = { .dec_fmts = rk3288_vpu_dec_fmts, .num_dec_fmts = ARRAY_SIZE(rk3288_vpu_dec_fmts), .codec =HANTRO_JPEG_ENCODER |HANTRO_MPEG2_DECODER | - HANTRO_VP8_DECODER, + HANTRO_VP8_DECODER | HANTRO_H264_DECODER, .codec_ops = rk3288_vpu_codec_ops, .irqs = rk3288_irqs, .num_irqs = ARRAY_SIZE(rk3288_irqs),
Signature · 1 line
2.22.0
Hi Ezequiel,
Quoted 27 lines · Ezequiel Garcia
This series is consolidating the two recent H264 series submitted by Boris [1] [2]. Given some patches from [2] have been merged (namely, helpers for the Hantro driver), the series contains the remanining bits required to support H264 on Hantro G1 VPU.
* Patch 1 adds support for the
sort_r()variant and has
been posted separately by Rasmus. It would be good to merge this patch via the media tree, ideally as soon as possible, to avoid the synchronisation burden that might appear if we decide to delay it.* Patch 2 to 4 extends the H264 uAPI, introducing frame-based vs slice-based decoding granularity, and also support for different NALU start codes. Currently, Annex B and no start codes are the supported options.
With the introduction of the start code control, the H264 parsed slices pixel format should be renamed, dropping the
_RAWsuffix, which is now meaningless.* Patch 5 removes the P0/B0/B1 ref lists from the
decode_paramscontrol. These lists are no longer needed since we build them on the kernel side based on the DPB.* Patch 6 and 7 exposes the proper decoding mode and start code on the cedrus driver. The driver functionality is not changed, and only the Cedrus support is now being properly exposed to userspace.
Quoted 8 lines
* Patch 8 is needed to properly propagate the OUTPUT buffer timestamp to the CAPTURE buffer one, which is required for intra-frame references.
* Patches 9 to 11 adds H264 support for Hantro G1 and then enable H264 decoding on RK3288.
This series has been tested using MPV/Ffmpeg, on Rockchip RK3288 for Hantro and Allwinner H3 boards for Cedrus.
on i.MX8MQ EVK using [1], so I effectively patches 1-5 and 8-10, with your FFmpeg modifications [2].
[1] git://git.pengutronix.de/git/pza/linux.git hantro/imx8m-wip
[2] https://gitlab.collabora.com/ezequiel/ffmpeg stateless-mpeg2-vp8- h264-v4
regards
Philipp
Quoted 39 lines · Philipp Zabel
Hi Ezequiel,
Quoted 27 lines · Ezequiel Garcia
This series is consolidating the two recent H264 series submitted by Boris [1] [2]. Given some patches from [2] have been merged (namely, helpers for the Hantro driver), the series contains the remanining bits required to support H264 on Hantro G1 VPU.
* Patch 1 adds support for the
sort_r()variant and has
been posted separately by Rasmus. It would be good to merge this patch via the media tree, ideally as soon as possible, to avoid the synchronisation burden that might appear if we decide to delay it.* Patch 2 to 4 extends the H264 uAPI, introducing frame-based vs slice-based decoding granularity, and also support for different NALU start codes. Currently, Annex B and no start codes are the supported options.
With the introduction of the start code control, the H264 parsed slices pixel format should be renamed, dropping the
_RAWsuffix, which is now meaningless.* Patch 5 removes the P0/B0/B1 ref lists from the
decode_paramscontrol. These lists are no longer needed since we build them on the kernel side based on the DPB.* Patch 6 and 7 exposes the proper decoding mode and start code on the cedrus driver. The driver functionality is not changed, and only the Cedrus support is now being properly exposed to userspace.
Quoted 8 lines
* Patch 8 is needed to properly propagate the OUTPUT buffer timestamp to the CAPTURE buffer one, which is required for intra-frame references.
* Patches 9 to 11 adds H264 support for Hantro G1 and then enable H264 decoding on RK3288.
This series has been tested using MPV/Ffmpeg, on Rockchip RK3288 for Hantro and Allwinner H3 boards for Cedrus.
Quoted 9 lines
Tested-by Philipp Zabel <[email protected]>on i.MX8MQ EVK using [1], so I effectively patches 1-5 and 8-10, with your FFmpeg modifications [2].
[1]
git://git.pengutronix.de/git/pza/linux.githantro/imx8m-wip
[2] https://gitlab.collabora.com/ezequiel/ffmpeg stateless-mpeg2-vp8- h264-v4
Glad to hear things work properly. I'm adding your tested-by to all patches except those that apply to cedrus and rk3288.
Thank you very much for the test,
Ezequiel