Skip to contentSkip to conversation

media: venus: Add hierarchical h.264 controls

PATCH In review Fritz Koenig 1 participant 4 messages Oct 22, 2024 – Nov 6, 2024

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v5 cover media: venus: Add hierarchical h.264 controls

v5:
- fixed formatting
- included reviwed by for patch 2
- Link to v4: https://lore.kernel.org/r/[email protected]

v4:
- addressed feedback from Dikshita
- included reviewed by for patch 1
- Link to v3: https://lore.kernel.org/r/[email protected]

v3:
- dropped reordering patch
- updated cover letter

v2:
- cover letter
- testing methodology
- Signed-off-by

V4L2 has support for encoding with hierarchical frames using the V4L2_CID_MPEG_VIDEO_H264_HIERARCHICAL_CODING* controls. This allows for specifing frame references needed for temporal scalability. Encoding a single stream with a single layer allows for the layer to be dropped and the stream to be decoded without artifacts.

ChromeOS is planning to use this feature for the L1T2 web standard[1]. This allows video conferencing apps to encode once for a clients with different performance/bandwidth capabilities. As the application is a real time encoder only P frame support is added.

The ChromeOS test framework ("tast") was used to verify that no regressions are present. This was done on SC7180 ("trogdor"). These patches were also run on SC7280, but not with as an extensive test set.

Verification of the added controls was done with a bitstream analyser to make sure that reference frame management is correct.

[1]: https://www.w3.org/TR/webrtc-svc/#L1T2*

---
Fritz Koenig (2):
media: venus: Helper function for dynamically updating bitrate
media: venus: Enable h.264 hierarchical coding

 drivers/media/platform/qcom/venus/core.h       |   4 +
 drivers/media/platform/qcom/venus/venc.c       |  71 +++++++++-----
 drivers/media/platform/qcom/venus/venc_ctrls.c | 126 ++++++++++++++++++++++---
 3 files changed, 164 insertions(+), 37 deletions(-)

---
base-commit: d07b43284ab356daf7ec5ae1858a16c1c7b6adab
change-id: 20240814-submit-05880a074377

Best regards,

Signature · 1 line
Fritz Koenig <[email protected]>
#1 lore raw

PATCH v5 1/2 media: venus: Helper function for dynamically updating bitrate

Move the dynamic bitrate updating functionality to a separate function so that it can be shared.

No functionality changes.

---
 drivers/media/platform/qcom/venus/venc_ctrls.c | 34 +++++++++++++++-----------
 1 file changed, 20 insertions(+), 14 deletions(-)
drivers/media/platform/qcom/venus/venc_ctrls.c +20 -14
index d9d2a293f3ef..3e1f6f26eddf 100644
@@ -67,12 +67,28 @@ static int venc_calc_bpframes(u32 gop_size, u32 conseq_b, u32 *bf, u32 *pf)
 	return 0;
 }
 
+static int dynamic_bitrate_update(struct venus_inst *inst, u32 bitrate,
+				  u32 layer_id)
+{
+	int ret = 0;
+
+	mutex_lock(&inst->lock);
+	if (inst->streamon_out && inst->streamon_cap) {
+		u32 ptype = HFI_PROPERTY_CONFIG_VENC_TARGET_BITRATE;
+		struct hfi_bitrate brate = { .bitrate = bitrate, .layer_id = layer_id };
+
+		ret = hfi_session_set_property(inst, ptype, &brate);
+	}
+	mutex_unlock(&inst->lock);
+
+	return ret;
+}
+
 static int venc_op_s_ctrl(struct v4l2_ctrl *ctrl)
 {
 	struct venus_inst *inst = ctrl_to_inst(ctrl);
 	struct venc_controls *ctr = &inst->controls.enc;
 	struct hfi_enable en = { .enable = 1 };
-	struct hfi_bitrate brate;
 	struct hfi_ltr_use ltr_use;
 	struct hfi_ltr_mark ltr_mark;
 	u32 bframes;
@@ -85,19 +101,9 @@ static int venc_op_s_ctrl(struct v4l2_ctrl *ctrl)
 		break;
 	case V4L2_CID_MPEG_VIDEO_BITRATE:
 		ctr->bitrate = ctrl->val;
-		mutex_lock(&inst->lock);
-		if (inst->streamon_out && inst->streamon_cap) {
-			ptype = HFI_PROPERTY_CONFIG_VENC_TARGET_BITRATE;
-			brate.bitrate = ctr->bitrate;
-			brate.layer_id = 0;
-
-			ret = hfi_session_set_property(inst, ptype, &brate);
-			if (ret) {
-				mutex_unlock(&inst->lock);
-				return ret;
-			}
-		}
-		mutex_unlock(&inst->lock);
+		ret = dynamic_bitrate_update(inst, ctr->bitrate, 0);
+		if (ret)
+			return ret;
 		break;
 	case V4L2_CID_MPEG_VIDEO_BITRATE_PEAK:
 		ctr->bitrate_peak = ctrl->val;
Signature · 1 line
2.47.0.105.g07ac214952-goog
#2 lore raw

PATCH v5 2/2 media: venus: Enable h.264 hierarchical coding

In reply to Fritz Koenig

HFI supports hierarchical P encoding and the ability to specify the bitrate for the different layers.

Connect the controls that V4L2 provides and HFI supports.

---
 drivers/media/platform/qcom/venus/core.h       |  4 ++
 drivers/media/platform/qcom/venus/venc.c       | 71 +++++++++++++-------
 drivers/media/platform/qcom/venus/venc_ctrls.c | 92 ++++++++++++++++++++++++++
 3 files changed, 144 insertions(+), 23 deletions(-)
drivers/media/platform/qcom/venus/core.h +4 -0
index 55202b89e1b9..fd46a7778d8c 100644
@@ -26,6 +26,7 @@
 #define VIDC_CLKS_NUM_MAX		4
 #define VIDC_VCODEC_CLKS_NUM_MAX	2
 #define VIDC_RESETS_NUM_MAX		2
+#define VIDC_MAX_HIER_CODING_LAYER 6
 
 extern int venus_fw_debug;
 
@@ -255,6 +256,7 @@ struct venc_controls {
 	u32 rc_enable;
 	u32 const_quality;
 	u32 frame_skip_mode;
+	u32 layer_bitrate;
 
 	u32 h264_i_period;
 	u32 h264_entropy_mode;
@@ -273,6 +275,8 @@ struct venc_controls {
 	s32 h264_loop_filter_alpha;
 	s32 h264_loop_filter_beta;
 	u32 h264_8x8_transform;
+	u32 h264_hier_layers;
+	u32 h264_hier_layer_bitrate[VIDC_MAX_HIER_CODING_LAYER];
 
 	u32 hevc_i_qp;
 	u32 hevc_p_qp;
drivers/media/platform/qcom/venus/venc.c +48 -23

91 lines not on this page · raw message

drivers/media/platform/qcom/venus/venc_ctrls.c +92 -0

104 lines not on this page · raw message

Signature · 1 line
2.47.0.105.g07ac214952-goog
#3 lore raw
In reply to Fritz Koenig

Stanimir,

Would it be possible to get this in the 6.13 pull request?

Thanks.

Quoted 58 lines · Fritz Koenig

v5:
- fixed formatting
- included reviwed by for patch 2
- Link to v4: https://lore.kernel.org/r/[email protected]

v4:
- addressed feedback from Dikshita
- included reviewed by for patch 1
- Link to v3: https://lore.kernel.org/r/[email protected]

v3:
- dropped reordering patch
- updated cover letter

v2:
- cover letter
- testing methodology
- Signed-off-by

V4L2 has support for encoding with hierarchical frames using the V4L2_CID_MPEG_VIDEO_H264_HIERARCHICAL_CODING* controls. This allows for specifing frame references needed for temporal scalability. Encoding a single stream with a single layer allows for the layer to be dropped and the stream to be decoded without artifacts.

ChromeOS is planning to use this feature for the L1T2 web standard[1]. This allows video conferencing apps to encode once for a clients with different performance/bandwidth capabilities. As the application is a real time encoder only P frame support is added.

The ChromeOS test framework ("tast") was used to verify that no regressions are present. This was done on SC7180 ("trogdor"). These patches were also run on SC7280, but not with as an extensive test set.

Verification of the added controls was done with a bitstream analyser to make sure that reference frame management is correct.

[1]: https://www.w3.org/TR/webrtc-svc/#L1T2*

---
Fritz Koenig (2):
media: venus: Helper function for dynamically updating bitrate
media: venus: Enable h.264 hierarchical coding

 drivers/media/platform/qcom/venus/core.h       |   4 +
 drivers/media/platform/qcom/venus/venc.c       |  71 +++++++++-----
 drivers/media/platform/qcom/venus/venc_ctrls.c | 126 ++++++++++++++++++++++---
 3 files changed, 164 insertions(+), 37 deletions(-)

---
base-commit: d07b43284ab356daf7ec5ae1858a16c1c7b6adab
change-id: 20240814-submit-05880a074377

Best regards,
--
Fritz Koenig <[email protected]>