Skip to contentSkip to conversation

media: rkvdec: Fix the HEVC reference picture set handling

PATCH New Daniel Golle 1 participant 8 messages Nov 20

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover media: rkvdec: Fix the HEVC reference picture set handling

This series fixes how the VDPU381 and VDPU383 HEVC back ends of the rkvdec driver handle the EXT_SPS_ST_RPS and EXT_SPS_LT_RPS controls. The defects were found while bringing up the VDPU346 of the RK3568, which uses the VDPU381 back end, on top of Christian Hewitt's VDPU346 series [1], which has since been continued in [2].

Patch 1 looks up the controls after the request was applied instead of before, which fixes a use-after-free.

Patches 2 and 3 are about jobs which cannot be decoded because the RPS controls are missing. The VDPU383 back end now completes the request when it refuses such a job, and the VDPU381 back end refuses the job as well instead of starting the hardware.

Patch 4 refuses RPS controls with fewer elements than the SPS counts, and patch 5 limits the size of predicted short-term sets. Both fix out of bounds accesses which userspace can trigger.

Patch 6 fixes wrong decoding caused by a short-term RPS cache which only looked at the first set, and patch 7 adds a missing allocation check.

Patches 1 to 6 carry Cc: stable, like the EXT SPS RPS bounds merged in v7.3 (f0b9d7e5be06, 796b5c6d4f16, 052c5ed5a1d9).

Tested on a NanoPi R5C (RK3568) with Linux 6.18 and the VDPU346 series
backported: GStreamer fluster JCT-VC-HEVC_V1 stays at 140/147, an FFmpeg v4l2request build that sets the RPS controls confirms the reads and the short-term set cache, and the VDPU381 refusal removes a watchdog reset of the RK3568. The same VDPU381 back end was exercised on a Radxa ROCK 5B (RK3588): H.264, HEVC and VP9 decode through rkvdec with no kernel message across the fluster runs (HEVC 146/147), a per-patch A/B sweep and a 30-minute stability run, at the baseline throughput. The VDPU383 back end has no hardware here, the code-only patches have no reproducer, and nothing was tested on an RK3576.

All of this was done with v1. v2 changes the order of the patches and the commit messages, the code after the series is the same as in v1, and the tests were not repeated.

An LLM coding agent did the analysis, and helped with writing the patches. A second instance, given only the code, reviewed the series, and has tested and validated it on real hardware independently.

Changes since v1:
- Move "media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets" before the check of the control size (5/7 -> 3/7). A control which is too short is handled like a missing one, and this way that means a refused job in both back ends as soon as the check exists, instead of a decode with a stale RPS table on the VDPU381 (Nicolas Dufresne)
- Rename "media: rkvdec: hevc: bound the RPS reads by the control size" to "media: rkvdec: hevc: refuse RPS controls shorter than the SPS counts" (3/7 -> 4/7)
- Rewrite all commit messages: say what is wrong and what the patch achieves, without walking through the code (Nicolas Dufresne)
- State in patch 4 why the size cannot be checked when the control is set (Nicolas Dufresne)
- Rebase on linux-next
- No change to the code after the series

v1: https://lore.kernel.org/r/[email protected]

[1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/[email protected]

Daniel Golle (7):
media: rkvdec: hevc: apply request controls first
media: rkvdec: vdpu383: complete refused HEVC requests
media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets
media: rkvdec: hevc: refuse RPS controls shorter than the SPS counts
media: rkvdec: hevc: bound predicted short-term RPS sets
media: rkvdec: hevc: cache every short-term RPS set
media: rkvdec: hevc: check the RPS scratch allocation

 .../rockchip/rkvdec/rkvdec-hevc-common.c      | 52 +++++++++++++------
 .../rockchip/rkvdec/rkvdec-hevc-common.h      | 11 +++-
 .../rockchip/rkvdec/rkvdec-vdpu381-hevc.c     | 24 +++++----
 .../rockchip/rkvdec/rkvdec-vdpu383-hevc.c     | 14 +++--
 4 files changed, 67 insertions(+), 34 deletions(-)

base-commit: 909ef7a9a30fa139e69e555e3606680f5cca7726

Signature · 1 line
2.56.1
#1 lore raw

PATCH v2 1/7 media: rkvdec: hevc: apply request controls first

The HEVC run preamble looks up the control values before the request is applied. Applying a request can reallocate the array of a dynamic array control, so the pointers taken before may point to freed memory. The slice count and the information whether the RPS controls were set are also those of the previous request.

Userspace can trigger the reallocation by passing more elements of such a control than in any earlier request.

Apply the request first and look up the controls afterwards, as the VP9 code does.

Fixes 3595375c2301 ("media: rkvdec: Add HEVC backend")

---
Changes since v1:
- Rewrite the commit message
- No code change

 drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +2 -2
index 2b8e04dd15727..c249e3f16f42c 100644
@@ -429,6 +429,8 @@ void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx,
 {
 	struct v4l2_ctrl *ctrl;
 
+	rkvdec_run_preamble(ctx, &run->base);
+
 	ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
 			      V4L2_CID_STATELESS_HEVC_DECODE_PARAMS);
 	run->decode_params = ctrl ? ctrl->p_cur.p : NULL;
@@ -460,6 +462,4 @@ void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx,
 	} else {
 		run->ext_sps_lt_rps = NULL;
 	}
-
-	rkvdec_run_preamble(ctx, &run->base);
 }
Signature · 1 line
2.56.1
#2 lore raw

PATCH v2 2/7 media: rkvdec: vdpu383: complete refused HEVC requests

In reply to Daniel Golle

When the RPS controls are missing, the VDPU383 HEVC back end fails the job after the request was applied, but without completing the request's controls. The request then never completes: userspace waits for it forever and MEDIA_REQUEST_IOC_REINIT fails with -EBUSY.

Complete the controls before returning the error, as the VP9 code does.

Fixes e3b5b77e3689 ("media: rkvdec: Add HEVC support for the VDPU383 variant")

---
Changes since v1:
- Rewrite the commit message
- No code change

 drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c | 1 +
 1 file changed, 1 insertion(+)
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c +1 -0
index 3462d995d4cf0..e9d8b590f0d58 100644
@@ -553,6 +553,7 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 	if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
 	    (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
 		dev_err_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
+		rkvdec_run_postamble(ctx, &run.base);
 		return -EINVAL;
 	}
 
Signature · 1 line
2.56.1
#3 lore raw

PATCH v2 3/7 media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets

In reply to Daniel Golle

When a stream uses reference picture sets and userspace never set the EXT_SPS_ST_RPS or EXT_SPS_LT_RPS control, the VDPU381 HEVC back end only logs a warning and starts the decoder anyway. The hardware then uses an RPS table which does not belong to the stream and gives wrong pictures at best. On the VDPU346 of the RK3568, which uses this back end, such a decode ends in a failed IOMMU reset and the system stops responding until the hardware watchdog resets it.

Fail the job with -EINVAL before the hardware is started, as the VDPU383 back end does.

Fixes c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")

---
Changes since v1:
- Move before the check of the control size, was 5/7
- Test ctx->has_sps_st_rps and ctx->has_sps_lt_rps as a result of the new position, the next patch replaces them in both back ends
- Rewrite the commit message

 .../rockchip/rkvdec/rkvdec-vdpu381-hevc.c     | 19 ++++++++-----------
 1 file changed, 8 insertions(+), 11 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c +8 -11
index e1936e87f45be..7627282c15f0a 100644
@@ -596,21 +596,18 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 
 	rkvdec_hevc_run_preamble(ctx, &run);
 
-	rkvdec_hevc_assemble_hw_scaling_list(ctx, &run, &tbl->scaling_list,
-					     &hevc_ctx->scaling_matrix_cache);
-	assemble_hw_pps(ctx, &run);
-
-	/*
-	 * On vdpu381, not setting the long and short term ref sets will just output wrong frames.
-	 * Let's just warn about it and let the decoder run anyway.
-	 */
 	if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
 	    (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
-		dev_warn_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
-	} else {
-		rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+		dev_err_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
+		rkvdec_run_postamble(ctx, &run.base);
+		return -EINVAL;
 	}
 
+	rkvdec_hevc_assemble_hw_scaling_list(ctx, &run, &tbl->scaling_list,
+					     &hevc_ctx->scaling_matrix_cache);
+	assemble_hw_pps(ctx, &run);
+	rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+
 	config_registers(ctx, &run);
 
 	rkvdec_run_postamble(ctx, &run.base);
Signature · 1 line
2.56.1
#4 lore raw

PATCH v2 4/7 media: rkvdec: hevc: refuse RPS controls shorter than the SPS counts

In reply to Daniel Golle

The SPS control tells how many short-term and long-term reference picture sets a stream has. The sets themselves are passed in two separate array controls, EXT_SPS_ST_RPS and EXT_SPS_LT_RPS. The driver reads as many array elements as the SPS announces, without checking that userspace passed that many. A larger count in the SPS makes it read past the end of the array.

Compare the array size with the SPS count when a decode job is prepared, and refuse the job if the array is too short. The controls can be set independently, so this is the first point where both values are known.

Fixes c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")

---
Changes since v1:
- Move after the VDPU381 refusal, was 3/7, so that a control which is too short fails the job in both back ends (Nicolas Dufresne)
- Change the subject, was "media: rkvdec: hevc: bound the RPS reads by the control size"
- Rewrite the commit message, and say why the size cannot be checked when the control is set (Nicolas Dufresne)
- The VDPU381 hunk now applies to the refusal instead of the warning, no other code change

 .../platform/rockchip/rkvdec/rkvdec-hevc-common.c    | 12 ++++++------
 .../platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c   |  4 ++--
 .../platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c   |  4 ++--
 3 files changed, 10 insertions(+), 10 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +6 -6
index c249e3f16f42c..797a16517ed59 100644
@@ -448,18 +448,18 @@ void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx,
 			      V4L2_CID_STATELESS_HEVC_SCALING_MATRIX);
 	run->scaling_matrix = ctrl ? ctrl->p_cur.p : NULL;
 
+	run->ext_sps_st_rps = NULL;
 	if (ctx->has_sps_st_rps) {
 		ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
 				      V4L2_CID_STATELESS_HEVC_EXT_SPS_ST_RPS);
-		run->ext_sps_st_rps = ctrl ? ctrl->p_cur.p : NULL;
-	} else {
-		run->ext_sps_st_rps = NULL;
+		if (ctrl && ctrl->elems >= run->sps->num_short_term_ref_pic_sets)
+			run->ext_sps_st_rps = ctrl->p_cur.p;
 	}
+	run->ext_sps_lt_rps = NULL;
 	if (ctx->has_sps_lt_rps) {
 		ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
 				      V4L2_CID_STATELESS_HEVC_EXT_SPS_LT_RPS);
-		run->ext_sps_lt_rps = ctrl ? ctrl->p_cur.p : NULL;
-	} else {
-		run->ext_sps_lt_rps = NULL;
+		if (ctrl && ctrl->elems >= run->sps->num_long_term_ref_pics_sps)
+			run->ext_sps_lt_rps = ctrl->p_cur.p;
 	}
 }
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c +2 -2
index 7627282c15f0a..db836536e601b 100644
@@ -596,8 +596,8 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 
 	rkvdec_hevc_run_preamble(ctx, &run);
 
-	if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
-	    (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
+	if ((!run.ext_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
+	    (!run.ext_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
 		dev_err_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
 		rkvdec_run_postamble(ctx, &run.base);
 		return -EINVAL;
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c +2 -2
index e9d8b590f0d58..c409d21c08598 100644
@@ -550,8 +550,8 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 	 * To be on the safe side for this new v4l2 control, write an error in the log and mark
 	 * the buffer as failed by returning an error here.
 	 */
-	if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
-	    (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
+	if ((!run.ext_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
+	    (!run.ext_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
 		dev_err_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
 		rkvdec_run_postamble(ctx, &run.base);
 		return -EINVAL;
Signature · 1 line
2.56.1
#5 lore raw

PATCH v2 5/7 media: rkvdec: hevc: bound predicted short-term RPS sets

In reply to Daniel Golle

A short-term reference picture set can be predicted from an earlier set, and can then hold one entry more than that set. The control validation allows 16 entries per set, which is also the size of the arrays the driver calculates the sets in. A set predicted from a 16-entry set overflows them, on the stack and on the heap.

ITU-T H.265 7.4.8 limits a set to 15 entries, and the hardware table has room for 15 entries per set as well. Skip the prediction when the reference set has more than 15 entries, as is already done for an invalid reference index.

Fixes c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")

---
Changes since v1:
- Was 4/7
- Rewrite the commit message
- No code change

 drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +2 -0
index 797a16517ed59..3b281c16a75ed 100644
@@ -223,6 +223,8 @@ static void st_ref_pic_set_prediction(struct rkvdec_hevc_run *run, int idx,
 		   (rps_data->abs_delta_rps_minus1 + 1); /* 7-60 */
 
 	ref_rps = &calculated_rps_st_sets[ref_rps_idx];
+	if (ref_rps->num_delta_pocs >= ARRAY_SIZE(use_delta_flag))
+		return;
 
 	for (j = 0; j <= ref_rps->num_delta_pocs; j++) {
 		used_by_curr_pic_flag[j] = !!(rps_data->used_by_curr_pic & (1 << j));
Signature · 1 line
2.56.1
#6 lore raw

PATCH v2 6/7 media: rkvdec: hevc: cache every short-term RPS set

In reply to Daniel Golle

The driver rebuilds the short-term RPS table of the hardware only when the EXT_SPS_ST_RPS control changed, but it compares just the first set with a cached copy. A change in a later set or in the number of sets goes unnoticed, and the hardware keeps the old table. As the cache starts zeroed, a stream whose first set is all zero never gets a table at all. Inter pictures are then decoded with wrong references, as seen with JCT-VC DELTAQP_A_BRCM_4 on an RK3568.

Cache all sets together with their number. The control framework limits the number of sets to 64, which is the size of the cache.

Fixes c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")

---
Changes since v1:
- Rewrite the commit message
- No code change

 .../platform/rockchip/rkvdec/rkvdec-hevc-common.c    | 12 ++++++++----
 .../platform/rockchip/rkvdec/rkvdec-hevc-common.h    |  9 ++++++++-
 .../platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c   |  2 +-
 .../platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c   |  2 +-
 4 files changed, 18 insertions(+), 7 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +8 -4
index 3b281c16a75ed..e3449ab0d415b 100644
@@ -330,14 +330,17 @@ static void st_ref_pic_set_calculate(struct rkvdec_hevc_run *run, int idx,
 }
 
 static void rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
-					  struct v4l2_ctrl_hevc_ext_sps_st_rps *cache)
+					  struct rkvdec_hevc_st_rps_cache *cache)
 {
+	u8 num_sets = run->sps->num_short_term_ref_pic_sets;
+	size_t size;
 	int idx;
 
 	if (!run->ext_sps_st_rps)
 		return;
 
-	if (!memcmp(cache, run->ext_sps_st_rps, sizeof(struct v4l2_ctrl_hevc_ext_sps_st_rps)))
+	size = num_sets * sizeof(*cache->sets);
+	if (cache->num_sets == num_sets && !memcmp(cache->sets, run->ext_sps_st_rps, size))
 		return;
 
 	struct calculated_rps_st_set *calculated_rps_st_sets =
@@ -357,11 +360,12 @@ static void rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rk
 
 	kfree(calculated_rps_st_sets);
 
-	memcpy(cache, run->ext_sps_st_rps, sizeof(struct v4l2_ctrl_hevc_ext_sps_st_rps));
+	memcpy(cache->sets, run->ext_sps_st_rps, size);
+	cache->num_sets = num_sets;
 }
 
 void rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
-				 struct v4l2_ctrl_hevc_ext_sps_st_rps *st_cache)
+				 struct rkvdec_hevc_st_rps_cache *st_cache)
 {
 	rkvdec_hevc_prepare_hw_st_rps(run, rps, st_cache);
 	rkvdec_hevc_assemble_hw_lt_rps(run, rps);
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h +8 -1
index 2a9b7719ab2d5..f6a3dec6ff4ad 100644
@@ -35,10 +35,17 @@
 
 #define RKVDEC_RPS_HEVC_SIZE		ALIGN(1032 + 64 * 384, 128)
 
+#define RKVDEC_HEVC_MAX_SHORT_TERM_REF_PIC_SETS	64
+
 struct rkvdec_rps {
 	u32 info[RKVDEC_RPS_HEVC_SIZE / 8 / 4];
 } __packed;
 
+struct rkvdec_hevc_st_rps_cache {
+	struct v4l2_ctrl_hevc_ext_sps_st_rps sets[RKVDEC_HEVC_MAX_SHORT_TERM_REF_PIC_SETS];
+	u8 num_sets;
+};
+
 struct rkvdec_hevc_run {
 	struct rkvdec_run base;
 	const struct v4l2_ctrl_hevc_slice_params *slices_params;
@@ -65,7 +72,7 @@ void compute_tiles_non_uniform(struct rkvdec_hevc_run *run, u16 log2_min_cb_size
 			       u16 width, u16 height, s32 pic_in_cts_width,
 			       s32 pic_in_cts_height, u16 *column_width, u16 *row_height);
 void rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
-				 struct v4l2_ctrl_hevc_ext_sps_st_rps *st_cache);
+				 struct rkvdec_hevc_st_rps_cache *st_cache);
 void rkvdec_hevc_assemble_hw_scaling_list(struct rkvdec_ctx *ctx,
 					  struct rkvdec_hevc_run *run,
 					  struct scaling_factor *scaling_factor,
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c +1 -1
index db836536e601b..8388421a68a48 100644
@@ -117,7 +117,7 @@ struct rkvdec_hevc_priv_tbl {
 struct rkvdec_hevc_ctx {
 	struct rkvdec_aux_buf			priv_tbl;
 	struct v4l2_ctrl_hevc_scaling_matrix	scaling_matrix_cache;
-	struct v4l2_ctrl_hevc_ext_sps_st_rps	st_cache;
+	struct rkvdec_hevc_st_rps_cache		st_cache;
 	struct rkvdec_vdpu381_regs_hevc		regs;
 };
 
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c +1 -1
index c409d21c08598..47f236295f1e5 100644
@@ -124,7 +124,7 @@ struct rkvdec_hevc_priv_tbl {
 struct rkvdec_hevc_ctx {
 	struct rkvdec_aux_buf			priv_tbl;
 	struct v4l2_ctrl_hevc_scaling_matrix	scaling_matrix_cache;
-	struct v4l2_ctrl_hevc_ext_sps_st_rps	st_cache;
+	struct rkvdec_hevc_st_rps_cache		st_cache;
 	struct vdpu383_regs_h26x		regs;
 };
 
Signature · 1 line
2.56.1
#7 lore raw

PATCH v2 7/7 media: rkvdec: hevc: check the RPS scratch allocation

In reply to Daniel Golle

The buffer for the calculated short-term sets is used without checking that its allocation succeeded. Return -ENOMEM and fail the decode job in both back ends.

Fixes c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")

---
Changes since v1:
- Rewrite the commit message
- No code change

 .../rockchip/rkvdec/rkvdec-hevc-common.c      | 26 ++++++++++++++-----
 .../rockchip/rkvdec/rkvdec-hevc-common.h      |  4 +--
 .../rockchip/rkvdec/rkvdec-vdpu381-hevc.c     |  7 ++++-
 .../rockchip/rkvdec/rkvdec-vdpu383-hevc.c     |  7 ++++-
 4 files changed, 33 insertions(+), 11 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +19 -7
index e3449ab0d415b..80df05bacaeba 100644
@@ -329,24 +329,27 @@ static void st_ref_pic_set_calculate(struct rkvdec_hevc_run *run, int idx,
 	st_rps->num_delta_pocs = st_rps->num_positive_pics + st_rps->num_negative_pics;
 }
 
-static void rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
-					  struct rkvdec_hevc_st_rps_cache *cache)
+static int rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
+					 struct rkvdec_hevc_st_rps_cache *cache)
 {
 	u8 num_sets = run->sps->num_short_term_ref_pic_sets;
 	size_t size;
 	int idx;
 
 	if (!run->ext_sps_st_rps)
-		return;
+		return 0;
 
 	size = num_sets * sizeof(*cache->sets);
 	if (cache->num_sets == num_sets && !memcmp(cache->sets, run->ext_sps_st_rps, size))
-		return;
+		return 0;
 
 	struct calculated_rps_st_set *calculated_rps_st_sets =
 		kzalloc(sizeof(struct calculated_rps_st_set) *
 			run->sps->num_short_term_ref_pic_sets, GFP_KERNEL);
 
+	if (!calculated_rps_st_sets)
+		return -ENOMEM;
+
 	for (idx = 0; idx < run->sps->num_short_term_ref_pic_sets; idx++) {
 		const struct v4l2_ctrl_hevc_ext_sps_st_rps *rps_data = &run->ext_sps_st_rps[idx];
 
@@ -362,13 +365,22 @@ static void rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rk
 
 	memcpy(cache->sets, run->ext_sps_st_rps, size);
 	cache->num_sets = num_sets;
+
+	return 0;
 }
 
-void rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
-				 struct rkvdec_hevc_st_rps_cache *st_cache)
+int rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
+				struct rkvdec_hevc_st_rps_cache *st_cache)
 {
-	rkvdec_hevc_prepare_hw_st_rps(run, rps, st_cache);
+	int ret;
+
+	ret = rkvdec_hevc_prepare_hw_st_rps(run, rps, st_cache);
+	if (ret)
+		return ret;
+
 	rkvdec_hevc_assemble_hw_lt_rps(run, rps);
+
+	return 0;
 }
 
 struct vb2_buffer *
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h +2 -2
index f6a3dec6ff4ad..32f192743c8c8 100644
@@ -71,8 +71,8 @@ void compute_tiles_uniform(struct rkvdec_hevc_run *run, u16 log2_min_cb_size,
 void compute_tiles_non_uniform(struct rkvdec_hevc_run *run, u16 log2_min_cb_size,
 			       u16 width, u16 height, s32 pic_in_cts_width,
 			       s32 pic_in_cts_height, u16 *column_width, u16 *row_height);
-void rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
-				 struct rkvdec_hevc_st_rps_cache *st_cache);
+int rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
+				struct rkvdec_hevc_st_rps_cache *st_cache);
 void rkvdec_hevc_assemble_hw_scaling_list(struct rkvdec_ctx *ctx,
 					  struct rkvdec_hevc_run *run,
 					  struct scaling_factor *scaling_factor,
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c +6 -1
index 8388421a68a48..2a239114759a2 100644
@@ -593,6 +593,7 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 	struct rkvdec_hevc_run run;
 	struct rkvdec_hevc_ctx *hevc_ctx = ctx->priv;
 	struct rkvdec_hevc_priv_tbl *tbl = hevc_ctx->priv_tbl.cpu;
+	int ret;
 
 	rkvdec_hevc_run_preamble(ctx, &run);
 
@@ -606,7 +607,11 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 	rkvdec_hevc_assemble_hw_scaling_list(ctx, &run, &tbl->scaling_list,
 					     &hevc_ctx->scaling_matrix_cache);
 	assemble_hw_pps(ctx, &run);
-	rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+	ret = rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+	if (ret) {
+		rkvdec_run_postamble(ctx, &run.base);
+		return ret;
+	}
 
 	config_registers(ctx, &run);
 
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c +6 -1
index 47f236295f1e5..52554c75f47f4 100644
@@ -542,6 +542,7 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 	struct rkvdec_hevc_ctx *hevc_ctx = ctx->priv;
 	struct rkvdec_hevc_priv_tbl *tbl = hevc_ctx->priv_tbl.cpu;
 	u32 timeout_threshold;
+	int ret;
 
 	rkvdec_hevc_run_preamble(ctx, &run);
 
@@ -560,7 +561,11 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 	rkvdec_hevc_assemble_hw_scaling_list(ctx, &run, &tbl->scaling_list,
 					     &hevc_ctx->scaling_matrix_cache);
 	assemble_hw_pps(ctx, &run);
-	rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+	ret = rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+	if (ret) {
+		rkvdec_run_postamble(ctx, &run.base);
+		return ret;
+	}
 
 	config_registers(ctx, &run);
 
Signature · 1 line
2.56.1