media: qcom: iris: add multi slice support
Subsystems (4)
v6 cover media: qcom: iris: add multi slice support
This series adds support for multi-slice encoding in the Qualcomm Iris driver.
Multi-slice mode allows encoding a single frame into multiple slices, which can improve error resilience and enable parallel processing. The feature supports two slice modes:
- MAX_MB mode: slices are created based on macroblock count, controlled via V4L2_CID_MPEG_VIDEO_MULTI_SLICE_MAX_MB
- MAX_BYTES mode: slices are created based on byte size, controlled via V4L2_CID_MPEG_VIDEO_MULTI_SLICE_MAX_BYTES
Clients can enable slice mode using the V4L2_CID_MPEG_VIDEO_MULTI_SLICE_MODE control.
Patch 1 is a prerequisite fix: iris_set_stage() computes a work_mode but never caches it back into inst->fw_caps[STAGE].value, leaving buffer-size and clock-frequency calculations based on a stale value.
Patch 2 is another prerequisite fix: iris_set_stage() currently always selects STAGE_2 work mode for encoder sessions, but CBR rate control requires STAGE_1. This is fixed by forcing STAGE_1 when BITRATE_MODE is set to CBR.
Patch 3 adds the multi-slice support itself for Iris HFI Gen2 based targets (qcs8300, sm8550 and other SM8550-derived targets). Support for HFI Gen1 targets will be added in a follow-up series.
This series has been verified with GST level testing. Command used:
gst-launch-1.0 -v videotestsrc ! video/x-raw,format=NV12, \
width=1280,height=720,framerate=30/1 ! v4l2h264enc output-io-mode=4 | capture-io-mode=4 extra-controls="controls,video_bitrate_mode=1, \
slice_partitioning_method=1,number_of_mbs_in_a_slice=460;" \
! filesink location=/opt/test_enc.h264
Changes in v6:
- Reordered local variable declarations in iris_set_slice_count() to
follow reverse Christmas tree style.
- Renamed default_slice_caps_vpu3x to slice_caps_vpu3x.
- Link to v5: https://lore.kernel.org/r/[email protected]
Changes in v5:
- Fixed unbounded slice count under RC_OFF in iris_set_slice_count()
for the MAX_BYTES use case.
- Addressed remaining coding-style review comments.
- Link to v4: https://lore.kernel.org/r/[email protected]
Changes in v4:
- Split out a new prerequisite patch (patch 1) to cache the computed
work_mode into inst->fw_caps[STAGE].value in iris_set_stage(), since
this fixes a pre-existing bug independent of the CBR change and
needed its own Fixes tag.
- Deduplicated the per-target slice capability limits that were
previously repeated identically across platform_inst_cap_qcs8300 and platform_inst_cap_sm8550. Introduced struct platform_inst_slice_caps
and moved it to iris_platform_data (one level up from
platform_inst_caps), since all Iris HFI Gen2 targets share the same
encoder capability table and hence the same slice limits; a single default_slice_caps_vpu3x instance is now referenced by all Gen2
platforms.
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- Patch 1 (force stage 1 work mode for CBR RC): cache the computed work_mode into inst->fw_caps[STAGE].value so buffer-size and clock-frequency calculations stay consistent with what is actually configured on firmware.
- Patch 2 (multi-slice support):
- Use inst->hfi_rc_type instead of the raw V4L2 BITRATE_MODE value when validating/deriving slice parameters, avoiding a mismatch between V4L2 and firmware rate-control enums.
- Round up (instead of truncate) when computing the minimum MAX_MB and MAX_BYTES slice size floors, so the hardware's max-slices-per-frame limit is never exceeded.
Changes in v2:
- Added a prerequisite patch to force STAGE_1 work mode for CBR rate control on the encoder, required for multi-slice encoding to work correctly.
- Addressed review comments on the multi-slice patch (updated commit subject and description).
---
Sachin Kumar Garg (3):
media: iris: cache work mode after computing it in iris_set_stage
media: iris: force stage 1 work mode for CBR RC on encoder
media: iris: add support for multi_slice in iris encoder
drivers/media/platform/qcom/iris/iris_ctrls.c | 91 ++++++++++++++++++++++ drivers/media/platform/qcom/iris/iris_ctrls.h | 1 + drivers/media/platform/qcom/iris/iris_hfi_gen2.c | 33 +++++++- .../platform/qcom/iris/iris_hfi_gen2_defines.h | 2 + .../platform/qcom/iris/iris_platform_common.h | 16 ++++ .../media/platform/qcom/iris/iris_platform_vpu3x.c | 17 ++++ 6 files changed, 159 insertions(+), 1 deletion(-)
---
base-commit: 42f83a32259a82c4481d3cccf560c3bfeb32c892
change-id: 20260716-iris_multi_slice-5647287a9563
Best regards,
Signature · 1 line
Sachin Kumar Garg <[email protected]>
PATCH v6 1/3 media: iris: cache work mode after computing it in iris_set_stage
iris_set_stage() computes work_mode and sends it to firmware but never updates inst->fw_caps[STAGE].value, leaving the cached STAGE value stale. iris_vpu_enc_bin_size() and iris_vpu3x_vpu4x_calculate_frequency() read this cached value, so a stale default causes them to miscalculate buffer size and clock frequency for sessions actually running in a different stage than the cached one.
Update inst->fw_caps[cap_id].value with the computed work_mode before sending it to firmware, keeping the cache consistent with what is actually configured.
--- drivers/media/platform/qcom/iris/iris_ctrls.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/media/platform/qcom/iris/iris_ctrls.c +2 -0
@@ -521,6 +521,8 @@ int iris_set_stage(struct iris_inst *inst, enum platform_inst_fw_cap_type cap_id work_mode =STAGE_1 ; } + inst->fw_caps[cap_id].value = work_mode; + return hfi_ops->session_set_property(inst, hfi_id, HFI_HOST_FLAGS_NONE, iris_get_port_info(inst, cap_id),
Signature · 1 line
2.34.1
PATCH v6 2/3 media: iris: force stage 1 work mode for CBR RC on encoder
In reply to Sachin Kumar Gargiris_set_stage() currently always picks STAGE_2 for encoder sessions. CBR rate control requires stage 1 operation, so force STAGE_1 when BITRATE_MODE is set to CBR.
The BITRATE_MODE fw_cap value is used instead of inst->hfi_rc_type since STAGE precedes BITRATE_MODE in enum platform_inst_fw_cap_type, so iris_set_properties() would still see the previous hfi_rc_type when iris_set_stage() runs during initial config
--- drivers/media/platform/qcom/iris/iris_ctrls.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/media/platform/qcom/iris/iris_ctrls.c +3 -0
@@ -519,6 +519,9 @@ int iris_set_stage(struct iris_inst *inst, enum platform_inst_fw_cap_type cap_idif (inst->domain ==DECODER ) {if (iris_res_is_less_than (width, height,1280 ,720 )) work_mode =STAGE_1 ; + }else if (inst->domain ==ENCODER ) { +if (inst->fw_caps[BITRATE_MODE ].value ==V4L2_MPEG_VIDEO_BITRATE_MODE_CBR ) + work_mode =STAGE_1 ; } inst->fw_caps[cap_id].value = work_mode;
Signature · 1 line
2.34.1
PATCH v6 3/3 media: iris: add support for multi_slice in iris encoder
In reply to Sachin Kumar GargAdd multi-slice encoding support with MAX_MB and MAX_BYTES modes.
Clients can enable slice mode using V4L2_CID_MPEG_VIDEO_MULTI_SLICE_MODE control and configure slice size via V4L2_CID_MPEG_VIDEO_MULTI_SLICE_MAX_MB or V4L2_CID_MPEG_VIDEO_MULTI_SLICE_MAX_BYTES controls.
--- drivers/media/platform/qcom/iris/iris_ctrls.c | 86 ++++++++++++++++++++++ drivers/media/platform/qcom/iris/iris_ctrls.h | 1 + drivers/media/platform/qcom/iris/iris_hfi_gen2.c | 33 ++++++++- .../platform/qcom/iris/iris_hfi_gen2_defines.h | 2 + .../platform/qcom/iris/iris_platform_common.h | 16 ++++ .../media/platform/qcom/iris/iris_platform_vpu3x.c | 17 +++++ 6 files changed, 154 insertions(+), 1 deletion(-)
drivers/media/platform/qcom/iris/iris_ctrls.c +86 -0
@@ -154,6 +154,12 @@ static enum platform_inst_fw_cap_type iris_get_cap_id(u32 id)return LAYER4_BITRATE_HEVC ;case V4L2_CID_MPEG_VIDEO_HEVC_HIER_CODING_L5_BR :return LAYER5_BITRATE_HEVC ; +case V4L2_CID_MPEG_VIDEO_MULTI_SLICE_MODE : +return SLICE_MODE ; +case V4L2_CID_MPEG_VIDEO_MULTI_SLICE_MAX_BYTES : +return SLICE_MAX_BYTES ; +case V4L2_CID_MPEG_VIDEO_MULTI_SLICE_MAX_MB : +return SLICE_MAX_MB ;default :return INST_FW_CAP_MAX ; }
@@ -297,6 +303,12 @@ static u32 iris_get_v4l2_id(enum platform_inst_fw_cap_type cap_id)return V4L2_CID_MPEG_VIDEO_HEVC_HIER_CODING_L4_BR ;case LAYER5_BITRATE_HEVC :return V4L2_CID_MPEG_VIDEO_HEVC_HIER_CODING_L5_BR ; +case SLICE_MODE : +return V4L2_CID_MPEG_VIDEO_MULTI_SLICE_MODE ; +case SLICE_MAX_BYTES : +return V4L2_CID_MPEG_VIDEO_MULTI_SLICE_MAX_BYTES ; +case SLICE_MAX_MB : +return V4L2_CID_MPEG_VIDEO_MULTI_SLICE_MAX_MB ;default :return 0 ; }
@@ -1482,6 +1494,80 @@ int iris_set_layer_bitrate(struct iris_inst *inst, enum platform_inst_fw_cap_typ &bitrate, sizeof(u32)); } +int iris_set_slice_count (struct iris_inst *inst,enum platform_inst_fw_cap_type cap_id) +{ +const struct platform_inst_slice_caps *slice_caps = + inst->core->iris_platform_data->slice_caps; +const struct iris_hfi_session_ops *hfi_ops = inst->hfi_session_ops; +u32 output_height = inst->fmt_dst->fmt.pix_mp.height; +u32 output_width = inst->fmt_dst->fmt.pix_mp.width; +u32 mbpf =NUM_MBS_PER_FRAME (output_height, output_width); +u32 max_width, max_height, min_width, min_height; +u32 slice_mode = inst->fw_caps[cap_id].value; +u32 max_avg_slicesize, hfi_value, hfi_id; +u32 rc_type = inst->hfi_rc_type; +u32 fps = inst->frame_rate; + +if (slice_mode ==V4L2_MPEG_VIDEO_MULTI_SLICE_MODE_SINGLE ) { +dev_dbg (inst->core->dev,"slice mode is single slice, ignore setting to fw\n" ); +return 0 ; + } +if (fps > slice_caps->max_slice_frame_rate || + (rc_type !=HFI_RC_OFF && rc_type !=HFI_RC_CBR_CFR && + rc_type !=HFI_RC_CBR_VFR && rc_type !=HFI_RC_VBR_CFR )) { +dev_err (inst->core->dev,"slice unsupported, fps: %u, rc_type: %#x\n" , + fps, rc_type); +return -EINVAL ; + } + + max_width = (slice_mode ==V4L2_MPEG_VIDEO_MULTI_SLICE_MODE_MAX_MB ) ? + slice_caps->max_mb_slice_width : slice_caps->max_bytes_slice_width; + max_height = (slice_mode ==V4L2_MPEG_VIDEO_MULTI_SLICE_MODE_MAX_MB ) ? + slice_caps->max_mb_slice_height : slice_caps->max_bytes_slice_height; + min_width = (inst->codec ==V4L2_PIX_FMT_HEVC ) ? + slice_caps->min_hevc_slice_width : slice_caps->min_avc_slice_width; + min_height = slice_caps->min_slice_height; + +if (output_width < min_width || output_height < min_height || + output_width > max_width || output_height > max_height) { +dev_err (inst->core->dev,"slice unsupported, codec: %#x wxh: [%dx%d]\n" , + inst->codec, output_width, output_height); +return -EINVAL ; + } + +if (slice_mode ==V4L2_MPEG_VIDEO_MULTI_SLICE_MODE_MAX_MB ) { + hfi_value = inst->fw_caps[SLICE_MAX_MB ].value; + hfi_value =max (hfi_value,DIV_ROUND_UP (mbpf, slice_caps->max_slices_per_frame)); +if (inst->codec ==V4L2_PIX_FMT_HEVC ) + hfi_value = (hfi_value +3 ) /4 ; + hfi_id = inst->fw_caps[SLICE_MAX_MB ].hfi_id; + }else if (slice_mode ==V4L2_MPEG_VIDEO_MULTI_SLICE_MODE_MAX_BYTES ) { + hfi_value = inst->fw_caps[SLICE_MAX_BYTES ].value; +if (rc_type !=HFI_RC_OFF ) { + max_avg_slicesize =DIV_ROUND_UP ((inst->fw_caps[BITRATE ].value / fps) /8 , + slice_caps->max_slices_per_frame); + }else { +/* +* No bitrate target exists under RC_OFF, so approximate a +* worst-case frame size the same way size_bin_bitstream_enc() +* does for buffer allocation, to keep the slice count bounded. +*/ + max_avg_slicesize =DIV_ROUND_UP (output_width * output_height *3 , + slice_caps->max_slices_per_frame); + } + hfi_value =max (hfi_value, max_avg_slicesize); + hfi_id = inst->fw_caps[SLICE_MAX_BYTES ].hfi_id; + }else { +return -EINVAL ; + } + +return hfi_ops->session_set_property (inst, hfi_id, +HFI_HOST_FLAGS_NONE , +iris_get_port_info (inst, cap_id), +HFI_PAYLOAD_U32 , + &hfi_value,sizeof (u32)); +} +int iris_set_properties (struct iris_inst *inst,u32 plane) {const struct iris_hfi_session_ops *hfi_ops = inst->hfi_session_ops;
drivers/media/platform/qcom/iris/iris_ctrls.h +1 -0
@@ -47,6 +47,7 @@ int iris_set_layer_type(struct iris_inst *inst, enum platform_inst_fw_cap_type cint iris_set_layer_count_gen1 (struct iris_inst *inst,enum platform_inst_fw_cap_type cap_id);int iris_set_layer_count_gen2 (struct iris_inst *inst,enum platform_inst_fw_cap_type cap_id);int iris_set_layer_bitrate (struct iris_inst *inst,enum platform_inst_fw_cap_type cap_id); +int iris_set_slice_count (struct iris_inst *inst,enum platform_inst_fw_cap_type cap_id);int iris_set_properties (struct iris_inst *inst,u32 plane);#endif
drivers/media/platform/qcom/iris/iris_hfi_gen2.c +32 -1
@@ -12,6 +12,8 @@#define VIDEO_ARCH_LX 1#define BITRATE_MAX 245000000 +#define MAX_SLICE_MB_SIZE \ + (((4096 + 15) >> 4) * ((2160 + 15) >> 4))static const struct platform_inst_fw_cap inst_fw_cap_sm8550_dec[] = { {
@@ -937,7 +939,36 @@ static const struct platform_inst_fw_cap inst_fw_cap_sm8550_enc[] = { .flags = CAP_FLAG_OUTPUT_PORT | CAP_FLAG_INPUT_PORT | CAP_FLAG_DYNAMIC_ALLOWED, .set = iris_set_layer_bitrate, - } + }, + { + .cap_id = SLICE_MODE, + .min = V4L2_MPEG_VIDEO_MULTI_SLICE_MODE_SINGLE, + .max = V4L2_MPEG_VIDEO_MULTI_SLICE_MODE_MAX_BYTES, + .step_or_mask =BIT (V4L2_MPEG_VIDEO_MULTI_SLICE_MODE_SINGLE ) | +BIT (V4L2_MPEG_VIDEO_MULTI_SLICE_MODE_MAX_MB ) | + BIT(V4L2_MPEG_VIDEO_MULTI_SLICE_MODE_MAX_BYTES), + .value = V4L2_MPEG_VIDEO_MULTI_SLICE_MODE_SINGLE, + .flags = CAP_FLAG_OUTPUT_PORT | CAP_FLAG_MENU, + .set = iris_set_slice_count, + }, + { + .cap_id = SLICE_MAX_BYTES, + .min = 512, + .max = BITRATE_MAX >> 3, + .step_or_mask = 1, + .value = 512, + .hfi_id = HFI_PROP_MULTI_SLICE_BYTES_COUNT, + .flags = CAP_FLAG_OUTPUT_PORT, + }, + { + .cap_id = SLICE_MAX_MB, + .min = 1, + .max = MAX_SLICE_MB_SIZE, + .step_or_mask = 1, + .value = 1, + .hfi_id = HFI_PROP_MULTI_SLICE_MB_COUNT, + .flags = CAP_FLAG_OUTPUT_PORT, + }, }; static const u32 sm8550_vdec_input_config_params_default[] = {
drivers/media/platform/qcom/iris/iris_hfi_gen2_defines.h +2 -0
@@ -71,6 +71,8 @@ enum hfi_rate_control {#define HFI_PROP_MIN_QP_PACKED 0x0300012f#define HFI_PROP_MAX_QP_PACKED 0x03000130#define HFI_PROP_IR_RANDOM_PERIOD 0x03000131 +#define HFI_PROP_MULTI_SLICE_MB_COUNT 0x03000132 +#define HFI_PROP_MULTI_SLICE_BYTES_COUNT 0x03000133#define HFI_PROP_LTR_COUNT 0x03000134#define HFI_PROP_LTR_MARK 0x03000135#define HFI_PROP_LTR_USE 0x03000136
drivers/media/platform/qcom/iris/iris_platform_common.h +16 -0
@@ -88,6 +88,18 @@ struct tz_cp_config {u32 cp_nonpixel_size; }; +struct platform_inst_slice_caps { +u32 max_slices_per_frame; +u32 max_slice_frame_rate; +u32 max_mb_slice_width; +u32 max_mb_slice_height; +u32 max_bytes_slice_width; +u32 max_bytes_slice_height; +u32 min_hevc_slice_width; +u32 min_avc_slice_width; +u32 min_slice_height; +}; +struct platform_inst_caps {u32 min_frame_width;u32 max_frame_width;
@@ -183,6 +195,9 @@ enum platform_inst_fw_cap_type {LAYER3_BITRATE_HEVC ,LAYER4_BITRATE_HEVC ,LAYER5_BITRATE_HEVC , +SLICE_MODE , +SLICE_MAX_BYTES , +SLICE_MAX_MB , INST_FW_CAP_MAX, };
@@ -315,6 +330,7 @@ struct iris_platform_data {const u32 *inst_iris_fmts;u32 inst_iris_fmts_size;struct platform_inst_caps *inst_caps; +const struct platform_inst_slice_caps *slice_caps;const struct tz_cp_config *tz_cp_config_data;u32 tz_cp_config_data_size;u32 num_vpp_pipe;
drivers/media/platform/qcom/iris/iris_platform_vpu3x.c +17 -0
@@ -12,6 +12,18 @@#include "iris_vpu_buffer.h" #include "iris_vpu_common.h" +static const struct platform_inst_slice_caps slice_caps_vpu3x = { + .max_slices_per_frame =128 , + .max_slice_frame_rate =60 , + .max_mb_slice_width =4096 , + .max_mb_slice_height =2160 , + .max_bytes_slice_width =1920 , + .max_bytes_slice_height =1088 , + .min_hevc_slice_width =384 , + .min_avc_slice_width =192 , + .min_slice_height =128 , +}; +#include "iris_platform_qcs8300.h" #include "iris_platform_sm8550.h" #include "iris_platform_sm8650.h"
@@ -110,6 +122,7 @@ const struct iris_platform_data qcs8300_data = { .inst_iris_fmts = iris_fmts_vpu3x_dec, .inst_iris_fmts_size = ARRAY_SIZE(iris_fmts_vpu3x_dec), .inst_caps = &platform_inst_cap_qcs8300, + .slice_caps = &slice_caps_vpu3x, .tz_cp_config_data = tz_cp_config_vpu3, .tz_cp_config_data_size = ARRAY_SIZE(tz_cp_config_vpu3), .num_vpp_pipe = 2,
@@ -139,6 +152,7 @@ const struct iris_platform_data sm8550_data = { .inst_iris_fmts = iris_fmts_vpu3x_dec, .inst_iris_fmts_size = ARRAY_SIZE(iris_fmts_vpu3x_dec), .inst_caps = &platform_inst_cap_sm8550, + .slice_caps = &slice_caps_vpu3x, .tz_cp_config_data = tz_cp_config_vpu3, .tz_cp_config_data_size = ARRAY_SIZE(tz_cp_config_vpu3), .num_vpp_pipe = 4,
@@ -176,6 +190,7 @@ const struct iris_platform_data sm8650_data = { .inst_iris_fmts = iris_fmts_vpu3x_dec, .inst_iris_fmts_size = ARRAY_SIZE(iris_fmts_vpu3x_dec), .inst_caps = &platform_inst_cap_sm8550, + .slice_caps = &slice_caps_vpu3x, .tz_cp_config_data = tz_cp_config_vpu3, .tz_cp_config_data_size = ARRAY_SIZE(tz_cp_config_vpu3), .num_vpp_pipe = 4,
@@ -205,6 +220,7 @@ const struct iris_platform_data sm8750_data = { .inst_iris_fmts = iris_fmts_vpu3x_dec, .inst_iris_fmts_size = ARRAY_SIZE(iris_fmts_vpu3x_dec), .inst_caps = &platform_inst_cap_sm8550, + .slice_caps = &slice_caps_vpu3x, .tz_cp_config_data = tz_cp_config_vpu3, .tz_cp_config_data_size = ARRAY_SIZE(tz_cp_config_vpu3), .num_vpp_pipe = 4,
@@ -240,6 +256,7 @@ const struct iris_platform_data x1p42100_data = { .inst_iris_fmts = iris_fmts_vpu3x_dec, .inst_iris_fmts_size = ARRAY_SIZE(iris_fmts_vpu3x_dec), .inst_caps = &platform_inst_cap_sm8550, + .slice_caps = &slice_caps_vpu3x, .tz_cp_config_data = tz_cp_config_vpu3, .tz_cp_config_data_size = ARRAY_SIZE(tz_cp_config_vpu3), .num_vpp_pipe = 1,
Signature · 1 line
2.34.1