Skip to contentSkip to conversation

media: rkvdec: Fix the HEVC reference picture set handling

PATCH Superseded Daniel Golle 2 participants 10 messages Nov 14 – Nov 20

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover media: rkvdec: Fix the HEVC reference picture set handling

This series fixes the code the VDPU381 and VDPU383 back ends of the rkvdec driver share for the HEVC EXT_SPS_ST_RPS and EXT_SPS_LT_RPS controls. The defects were found while bringing up the VDPU346 of the RK3568, which uses the VDPU381 back end, on top of Christian Hewitt's VDPU346 series (last posted as v3:
https://lore.kernel.org/r/[email protected]).

Several are memory-safety fixes for input a userspace V4L2 client can submit to the decoder, and carry Cc: stable like the EXT SPS RPS bounds merged in v7.3 (f0b9d7e5be06, 796b5c6d4f16, 052c5ed5a1d9).

Tested on a NanoPi R5C (RK3568) with Linux 6.18 and the VDPU346 series
backported: GStreamer fluster JCT-VC-HEVC_V1 stays at 140/147, an FFmpeg v4l2request build that sets the RPS controls confirms the reads and the short-term set cache, and the VDPU381 refusal removes a watchdog reset of the RK3568. The same VDPU381 back end was exercised on a Radxa ROCK 5B (RK3588): H.264, HEVC and VP9 decode through rkvdec with no kernel message across the fluster runs (HEVC 146/147), a per-patch A/B sweep and a 30-minute stability run, at the baseline throughput. The VDPU383 back end has no hardware here, the code-only patches have no reproducer, and nothing was tested on an RK3576.

An LLM coding agent did the analysis, and helped with writing the patches. A second instance, given only the code, reviewed the series, and has tested and validated it on real hardware independently.

Daniel Golle (7):
media: rkvdec: hevc: apply request controls first
media: rkvdec: vdpu383: complete refused HEVC requests
media: rkvdec: hevc: bound the RPS reads by the control size
media: rkvdec: hevc: bound predicted short-term RPS sets
media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets
media: rkvdec: hevc: cache every short-term RPS set
media: rkvdec: hevc: check the RPS scratch allocation

 .../rockchip/rkvdec/rkvdec-hevc-common.c      | 52 +++++++++++++------
 .../rockchip/rkvdec/rkvdec-hevc-common.h      | 11 +++-
 .../rockchip/rkvdec/rkvdec-vdpu381-hevc.c     | 24 +++++----
 .../rockchip/rkvdec/rkvdec-vdpu383-hevc.c     | 14 +++--
 4 files changed, 67 insertions(+), 34 deletions(-)

base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb

Signature · 1 line
2.56.0
#1 lore raw

PATCH 1/7 media: rkvdec: hevc: apply request controls first

rkvdec_hevc_run_preamble() takes the control pointers and the slice count before rkvdec_run_preamble() applies the request. When a request carries more elements of a dynamic array control than the control has held so far, v4l2_ctrl_request_setup() reallocates the array and frees the old one, which leaves run->slices_params, run->ext_sps_st_rps or run->ext_sps_lt_rps pointing to freed memory, and run->num_slices holds the element count of the previous request. The EXT SPS RPS controls are also looked up depending on has_sps_st_rps and has_sps_lt_rps before the request's own controls could set them.

A userspace V4L2 client that can open the RKVDEC m2m decoder can submit a request that grows one of these dynamic array controls and reach the freed array through the stale pointer.

Apply the request first, as rkvdec_vp9_run_preamble() does.

Fixes 3595375c2301 ("media: rkvdec: Add HEVC backend")
---
 drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +2 -2
index 2b8e04dd15727..c249e3f16f42c 100644
@@ -429,6 +429,8 @@ void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx,
 {
 	struct v4l2_ctrl *ctrl;
 
+	rkvdec_run_preamble(ctx, &run->base);
+
 	ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
 			      V4L2_CID_STATELESS_HEVC_DECODE_PARAMS);
 	run->decode_params = ctrl ? ctrl->p_cur.p : NULL;
@@ -460,6 +462,4 @@ void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx,
 	} else {
 		run->ext_sps_lt_rps = NULL;
 	}
-
-	rkvdec_run_preamble(ctx, &run->base);
 }
Signature · 1 line
2.56.0
#2 lore raw

PATCH 2/7 media: rkvdec: vdpu383: complete refused HEVC requests

In reply to Daniel Golle

rkvdec_hevc_run() returns -EINVAL when the RPS controls are missing, after rkvdec_hevc_run_preamble() applied the request's controls but without completing them. rkvdec_device_run() then returns the buffers with VB2_BUF_STATE_ERROR, which unbinds the source buffer from the request, while the request's control handler object stays incomplete. The request never reaches MEDIA_REQUEST_STATE_COMPLETE: poll() on its file descriptor never reports it, and MEDIA_REQUEST_IOC_REINIT fails with -EBUSY.

Complete the request's controls before returning the error, as rkvdec_vp9_run() does.

Fixes e3b5b77e3689 ("media: rkvdec: Add HEVC support for the VDPU383 variant")
---
 drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c | 1 +
 1 file changed, 1 insertion(+)
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c +1 -0
index 3462d995d4cf0..e9d8b590f0d58 100644
@@ -553,6 +553,7 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 	if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
 	    (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
 		dev_err_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
+		rkvdec_run_postamble(ctx, &run.base);
 		return -EINVAL;
 	}
 
Signature · 1 line
2.56.0
#3 lore raw

PATCH 3/7 media: rkvdec: hevc: bound the RPS reads by the control size

In reply to Daniel Golle

The RPS helpers read num_short_term_ref_pic_sets elements of the EXT_SPS_ST_RPS control and num_long_term_ref_pics_sps elements of the EXT_SPS_LT_RPS control, however many elements userspace passed. Beyond the passed elements they read stale entries of the control's array, and past its allocation once the SPS counts more sets than the control has ever held, as the control framework sizes a dynamic array for the largest element count it has seen. The short-term sets read from there are unchecked, and a num_negative_pics or num_positive_pics above 16 overflows the calculated sets, the stack arrays of st_ref_pic_set_prediction() and the hardware RPS table.

A userspace V4L2 client that can open the RKVDEC m2m decoder can set an SPS whose counts exceed the number of elements it passed in these controls, so the reads run past the allocation.

Use the controls only when they hold at least as many elements as the SPS counts, and treat them as not set otherwise in both back ends.

Fixes c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")
---
 .../platform/rockchip/rkvdec/rkvdec-hevc-common.c    | 12 ++++++------
 .../platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c   |  4 ++--
 .../platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c   |  4 ++--
 3 files changed, 10 insertions(+), 10 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +6 -6
index c249e3f16f42c..797a16517ed59 100644
@@ -448,18 +448,18 @@ void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx,
 			      V4L2_CID_STATELESS_HEVC_SCALING_MATRIX);
 	run->scaling_matrix = ctrl ? ctrl->p_cur.p : NULL;
 
+	run->ext_sps_st_rps = NULL;
 	if (ctx->has_sps_st_rps) {
 		ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
 				      V4L2_CID_STATELESS_HEVC_EXT_SPS_ST_RPS);
-		run->ext_sps_st_rps = ctrl ? ctrl->p_cur.p : NULL;
-	} else {
-		run->ext_sps_st_rps = NULL;
+		if (ctrl && ctrl->elems >= run->sps->num_short_term_ref_pic_sets)
+			run->ext_sps_st_rps = ctrl->p_cur.p;
 	}
+	run->ext_sps_lt_rps = NULL;
 	if (ctx->has_sps_lt_rps) {
 		ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
 				      V4L2_CID_STATELESS_HEVC_EXT_SPS_LT_RPS);
-		run->ext_sps_lt_rps = ctrl ? ctrl->p_cur.p : NULL;
-	} else {
-		run->ext_sps_lt_rps = NULL;
+		if (ctrl && ctrl->elems >= run->sps->num_long_term_ref_pics_sps)
+			run->ext_sps_lt_rps = ctrl->p_cur.p;
 	}
 }
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c +2 -2
index e1936e87f45be..fea7ff48c0dce 100644
@@ -604,8 +604,8 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 	 * On vdpu381, not setting the long and short term ref sets will just output wrong frames.
 	 * Let's just warn about it and let the decoder run anyway.
 	 */
-	if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
-	    (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
+	if ((!run.ext_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
+	    (!run.ext_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
 		dev_warn_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
 	} else {
 		rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c +2 -2
index e9d8b590f0d58..c409d21c08598 100644
@@ -550,8 +550,8 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 	 * To be on the safe side for this new v4l2 control, write an error in the log and mark
 	 * the buffer as failed by returning an error here.
 	 */
-	if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
-	    (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
+	if ((!run.ext_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
+	    (!run.ext_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
 		dev_err_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
 		rkvdec_run_postamble(ctx, &run.base);
 		return -EINVAL;
Signature · 1 line
2.56.0
#4 lore raw

Re: [PATCH 3/7] media: rkvdec: hevc: bound the RPS reads by the control size

Le samedi 14 novembre 2026 à 10:22 +0000, Daniel Golle a écrit :

Quoted 4 lines

The RPS helpers read num_short_term_ref_pic_sets elements of the EXT_SPS_ST_RPS control and num_long_term_ref_pics_sps elements of the EXT_SPS_LT_RPS control, however many elements userspace passed.

What did you want to say?

Quoted 3 lines

Beyond
the passed elements they read stale entries of the control's array, and past its allocation once the SPS counts more sets than the control

More broken text, can you work a bit on your messages, be concise, precise, don't get into details but into the understanding of what you are achieving. The code is precise enough.

Quoted 39 lines

has ever held, as the control framework sizes a dynamic array for the largest element count it has seen. The short-term sets read from there are unchecked, and a num_negative_pics or num_positive_pics above 16 overflows the calculated sets, the stack arrays of st_ref_pic_set_prediction() and the hardware RPS table.

A userspace V4L2 client that can open the RKVDEC m2m decoder can set an SPS whose counts exceed the number of elements it passed in these controls, so the reads run past the allocation.

Use the controls only when they hold at least as many elements as the SPS counts, and treat them as not set otherwise in both back ends.

Fixes c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")
---
 .../platform/rockchip/rkvdec/rkvdec-hevc-common.c    | 12 ++++++------
 .../platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c   |  4 ++--
 .../platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c   |  4 ++--
 3 files changed, 10 insertions(+), 10 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +3 -3
index c249e3f16f42c..797a16517ed59 100644
@@ -448,18 +448,18 @@ void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx,
 			      V4L2_CID_STATELESS_HEVC_SCALING_MATRIX);
 	run->scaling_matrix = ctrl ? ctrl->p_cur.p : NULL;
 
+	run->ext_sps_st_rps = NULL;
 	if (ctx->has_sps_st_rps) {
 		ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
 				      V4L2_CID_STATELESS_HEVC_EXT_SPS_ST_RPS);
-		run->ext_sps_st_rps = ctrl ? ctrl->p_cur.p : NULL;
-	} else {
-		run->ext_sps_st_rps = NULL;
+		if (ctrl && ctrl->elems >= run->sps->num_short_term_ref_pic_sets)
+			run->ext_sps_st_rps = ctrl->p_cur.p;

drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx, · line 463

Quoted 10 lines
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +3 -3
 	}
+	run->ext_sps_lt_rps = NULL;
 	if (ctx->has_sps_lt_rps) {
 		ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
 				      V4L2_CID_STATELESS_HEVC_EXT_SPS_LT_RPS);
-		run->ext_sps_lt_rps = ctrl ? ctrl->p_cur.p : NULL;
-	} else {
-		run->ext_sps_lt_rps = NULL;
+		if (ctrl && ctrl->elems >= run->sps->num_long_term_ref_pics_sps)
+			run->ext_sps_lt_rps = ctrl->p_cur.p;

So you are saying that if the passed array is too short, we should use an older entry to none? That seems a little strange, what about up-front rejecting bad entries?

Nicolas

Quoted 32 lines
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +0 -0
 	}
 }
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c +2 -2
index e1936e87f45be..fea7ff48c0dce 100644
@@ -604,8 +604,8 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 	 * On vdpu381, not setting the long and short term ref sets will just output wrong frames.
 	 * Let's just warn about it and let the decoder run anyway.
 	 */
-	if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
-	    (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
+	if ((!run.ext_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
+	    (!run.ext_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
 		dev_warn_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
 	} else {
 		rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c +2 -2
index e9d8b590f0d58..c409d21c08598 100644
@@ -550,8 +550,8 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 	 * To be on the safe side for this new v4l2 control, write an error in the log and mark
 	 * the buffer as failed by returning an error here.
 	 */
-	if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
-	    (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
+	if ((!run.ext_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
+	    (!run.ext_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
 		dev_err_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
 		rkvdec_run_postamble(ctx, &run.base);
 		return -EINVAL;
#5 lore raw

Re: [PATCH 3/7] media: rkvdec: hevc: bound the RPS reads by the control size

Hi Nicolas,

thanks for the review and sorry about the messy text.

Quoted 7 lines · Nicolas Dufresne

Le samedi 14 novembre 2026 à 10:22 +0000, Daniel Golle a écrit :

Quoted 4 lines

The RPS helpers read num_short_term_ref_pic_sets elements of the EXT_SPS_ST_RPS control and num_long_term_ref_pics_sps elements of the EXT_SPS_LT_RPS control, however many elements userspace passed.

What did you want to say?

The number of RPS entries comes from the SPS control, the entries themselves from the EXT_SPS_ST_RPS/LT_RPS array controls. The driver never checks that the arrays are as long as the SPS says, so it reads past the end of what userspace passed. I will reword the message along these lines in v2.

Quoted 8 lines
Quoted 4 lines

Beyond
the passed elements they read stale entries of the control's array, and past its allocation once the SPS counts more sets than the control

More broken text, can you work a bit on your messages, be concise, precise, don't get into details but into the understanding of what you are achieving. The code is precise enough.

Got it. Reading it again now I understand your point, I may have been a bit too excited about my findings on a Saturday morning...

#6 lore raw

Re: [PATCH 3/7] media: rkvdec: hevc: bound the RPS reads by the control size

In reply to Nicolas Dufresne

Hi again,

I missed the part of your review which was inlined with the code. See my reply below.

Quoted 22 lines · Nicolas Dufresne

[...]

Quoted 21 lines
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +4 -4
@@ -448,18 +448,18 @@ void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx,
 			      V4L2_CID_STATELESS_HEVC_SCALING_MATRIX);
 	run->scaling_matrix = ctrl ? ctrl->p_cur.p : NULL;
 
+	run->ext_sps_st_rps = NULL;
 	if (ctx->has_sps_st_rps) {
 		ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
 				      V4L2_CID_STATELESS_HEVC_EXT_SPS_ST_RPS);
-		run->ext_sps_st_rps = ctrl ? ctrl->p_cur.p : NULL;
-	} else {
-		run->ext_sps_st_rps = NULL;
+		if (ctrl && ctrl->elems >= run->sps->num_short_term_ref_pic_sets)
+			run->ext_sps_st_rps = ctrl->p_cur.p;
 	}
+	run->ext_sps_lt_rps = NULL;
 	if (ctx->has_sps_lt_rps) {
 		ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
 				      V4L2_CID_STATELESS_HEVC_EXT_SPS_LT_RPS);
-		run->ext_sps_lt_rps = ctrl ? ctrl->p_cur.p : NULL;
Quoted 8 lines

drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx, · line 463

Quoted 4 lines
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +2 -2
-	} else {
-		run->ext_sps_lt_rps = NULL;
+		if (ctrl && ctrl->elems >= run->sps->num_long_term_ref_pics_sps)
+			run->ext_sps_lt_rps = ctrl->p_cur.p;

So you are saying that if the passed array is too short, we should use an older entry to none? That seems a little strange, what about up-front rejecting bad entries?

That's what I do, but only after patch 5, before that the decoder would still run with the previous table. I will reorder the series so the refusal comes first.

Rejecting when the control is set does not work, as the count and the array are separate controls which can be set in any order, also in different requests. The first point where both are known is when the request is applied in device_run.

#7 lore raw

PATCH 5/7 media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets

In reply to Daniel Golle

When the SPS uses short-term or long-term reference picture sets and the matching EXT_SPS_ST_RPS or EXT_SPS_LT_RPS control was never set or holds fewer sets than the SPS counts, rkvdec_hevc_run() logs a warning and starts the decoder with an RPS table that does not belong to the stream, which at best gives wrong pictures. On the VDPU346 of the RK3568, which uses this back end, such decodes also raise error interrupts and time out, the IOMMU force reset fails, and the system stops responding until the hardware watchdog resets it.

Refuse the job as the VDPU383 back end does: log an error and return -EINVAL before any table is assembled, so that rkvdec_device_run() returns the buffers with VB2_BUF_STATE_ERROR and the hardware is not started. The request's controls are completed first, as rkvdec_vp9_run() does, so that the request is signalled as complete.

Fixes c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")
---
 .../rockchip/rkvdec/rkvdec-vdpu381-hevc.c     | 19 ++++++++-----------
 1 file changed, 8 insertions(+), 11 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c +8 -11
index fea7ff48c0dce..db836536e601b 100644
@@ -596,21 +596,18 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 
 	rkvdec_hevc_run_preamble(ctx, &run);
 
-	rkvdec_hevc_assemble_hw_scaling_list(ctx, &run, &tbl->scaling_list,
-					     &hevc_ctx->scaling_matrix_cache);
-	assemble_hw_pps(ctx, &run);
-
-	/*
-	 * On vdpu381, not setting the long and short term ref sets will just output wrong frames.
-	 * Let's just warn about it and let the decoder run anyway.
-	 */
 	if ((!run.ext_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) ||
 	    (!run.ext_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {
-		dev_warn_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
-	} else {
-		rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+		dev_err_ratelimited(rkvdec->dev, "Long and short term RPS not set\n");
+		rkvdec_run_postamble(ctx, &run.base);
+		return -EINVAL;
 	}
 
+	rkvdec_hevc_assemble_hw_scaling_list(ctx, &run, &tbl->scaling_list,
+					     &hevc_ctx->scaling_matrix_cache);
+	assemble_hw_pps(ctx, &run);
+	rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+
 	config_registers(ctx, &run);
 
 	rkvdec_run_postamble(ctx, &run.base);
Signature · 1 line
2.56.0
#8 lore raw

PATCH 6/7 media: rkvdec: hevc: cache every short-term RPS set

In reply to Daniel Golle

rkvdec_hevc_prepare_hw_st_rps() skips building the short-term RPS table when the first element of the EXT_SPS_ST_RPS control equals its cache, which holds a single element and starts zeroed. When the first element is all zero, e.g. for an empty first set, the table entries of the remaining sets are never built, and an SPS that differs from the previous one only after its first set, or only in num_short_term_ref_pic_sets, keeps the previous table. Inter pictures are then decoded with wrong references: JCT-VC DELTAQP_A_BRCM_4 decodes its intra pictures correctly and its inter pictures wrongly on an RK3568, whose VDPU346 uses the VDPU381 back end.

Cache all sets the SPS uses together with their number, for both the VDPU381 and the VDPU383 back end. The SPS control validation bounds num_short_term_ref_pic_sets to 64, the size of the cache.

Fixes c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")
---
 .../platform/rockchip/rkvdec/rkvdec-hevc-common.c    | 12 ++++++++----
 .../platform/rockchip/rkvdec/rkvdec-hevc-common.h    |  9 ++++++++-
 .../platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c   |  2 +-
 .../platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c   |  2 +-
 4 files changed, 18 insertions(+), 7 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +8 -4
index 3b281c16a75ed..e3449ab0d415b 100644
@@ -330,14 +330,17 @@ static void st_ref_pic_set_calculate(struct rkvdec_hevc_run *run, int idx,
 }
 
 static void rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
-					  struct v4l2_ctrl_hevc_ext_sps_st_rps *cache)
+					  struct rkvdec_hevc_st_rps_cache *cache)
 {
+	u8 num_sets = run->sps->num_short_term_ref_pic_sets;
+	size_t size;
 	int idx;
 
 	if (!run->ext_sps_st_rps)
 		return;
 
-	if (!memcmp(cache, run->ext_sps_st_rps, sizeof(struct v4l2_ctrl_hevc_ext_sps_st_rps)))
+	size = num_sets * sizeof(*cache->sets);
+	if (cache->num_sets == num_sets && !memcmp(cache->sets, run->ext_sps_st_rps, size))
 		return;
 
 	struct calculated_rps_st_set *calculated_rps_st_sets =
@@ -357,11 +360,12 @@ static void rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rk
 
 	kfree(calculated_rps_st_sets);
 
-	memcpy(cache, run->ext_sps_st_rps, sizeof(struct v4l2_ctrl_hevc_ext_sps_st_rps));
+	memcpy(cache->sets, run->ext_sps_st_rps, size);
+	cache->num_sets = num_sets;
 }
 
 void rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
-				 struct v4l2_ctrl_hevc_ext_sps_st_rps *st_cache)
+				 struct rkvdec_hevc_st_rps_cache *st_cache)
 {
 	rkvdec_hevc_prepare_hw_st_rps(run, rps, st_cache);
 	rkvdec_hevc_assemble_hw_lt_rps(run, rps);
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h +8 -1
index 2a9b7719ab2d5..f6a3dec6ff4ad 100644
@@ -35,10 +35,17 @@
 
 #define RKVDEC_RPS_HEVC_SIZE		ALIGN(1032 + 64 * 384, 128)
 
+#define RKVDEC_HEVC_MAX_SHORT_TERM_REF_PIC_SETS	64
+
 struct rkvdec_rps {
 	u32 info[RKVDEC_RPS_HEVC_SIZE / 8 / 4];
 } __packed;
 
+struct rkvdec_hevc_st_rps_cache {
+	struct v4l2_ctrl_hevc_ext_sps_st_rps sets[RKVDEC_HEVC_MAX_SHORT_TERM_REF_PIC_SETS];
+	u8 num_sets;
+};
+
 struct rkvdec_hevc_run {
 	struct rkvdec_run base;
 	const struct v4l2_ctrl_hevc_slice_params *slices_params;
@@ -65,7 +72,7 @@ void compute_tiles_non_uniform(struct rkvdec_hevc_run *run, u16 log2_min_cb_size
 			       u16 width, u16 height, s32 pic_in_cts_width,
 			       s32 pic_in_cts_height, u16 *column_width, u16 *row_height);
 void rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
-				 struct v4l2_ctrl_hevc_ext_sps_st_rps *st_cache);
+				 struct rkvdec_hevc_st_rps_cache *st_cache);
 void rkvdec_hevc_assemble_hw_scaling_list(struct rkvdec_ctx *ctx,
 					  struct rkvdec_hevc_run *run,
 					  struct scaling_factor *scaling_factor,
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c +1 -1
index db836536e601b..8388421a68a48 100644
@@ -117,7 +117,7 @@ struct rkvdec_hevc_priv_tbl {
 struct rkvdec_hevc_ctx {
 	struct rkvdec_aux_buf			priv_tbl;
 	struct v4l2_ctrl_hevc_scaling_matrix	scaling_matrix_cache;
-	struct v4l2_ctrl_hevc_ext_sps_st_rps	st_cache;
+	struct rkvdec_hevc_st_rps_cache		st_cache;
 	struct rkvdec_vdpu381_regs_hevc		regs;
 };
 
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c +1 -1
index c409d21c08598..47f236295f1e5 100644
@@ -124,7 +124,7 @@ struct rkvdec_hevc_priv_tbl {
 struct rkvdec_hevc_ctx {
 	struct rkvdec_aux_buf			priv_tbl;
 	struct v4l2_ctrl_hevc_scaling_matrix	scaling_matrix_cache;
-	struct v4l2_ctrl_hevc_ext_sps_st_rps	st_cache;
+	struct rkvdec_hevc_st_rps_cache		st_cache;
 	struct vdpu383_regs_h26x		regs;
 };
 
Signature · 1 line
2.56.0
#9 lore raw

PATCH 7/7 media: rkvdec: hevc: check the RPS scratch allocation

In reply to Daniel Golle

rkvdec_hevc_prepare_hw_st_rps() uses the array returned by kzalloc() without checking it, so a failed allocation while a decode job is prepared dereferences a NULL pointer. Return -ENOMEM through rkvdec_hevc_assemble_hw_rps() and let both back ends refuse the job after completing the request's controls.

Fixes c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")
---
 .../rockchip/rkvdec/rkvdec-hevc-common.c      | 26 ++++++++++++++-----
 .../rockchip/rkvdec/rkvdec-hevc-common.h      |  4 +--
 .../rockchip/rkvdec/rkvdec-vdpu381-hevc.c     |  7 ++++-
 .../rockchip/rkvdec/rkvdec-vdpu383-hevc.c     |  7 ++++-
 4 files changed, 33 insertions(+), 11 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +19 -7
index e3449ab0d415b..80df05bacaeba 100644
@@ -329,24 +329,27 @@ static void st_ref_pic_set_calculate(struct rkvdec_hevc_run *run, int idx,
 	st_rps->num_delta_pocs = st_rps->num_positive_pics + st_rps->num_negative_pics;
 }
 
-static void rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
-					  struct rkvdec_hevc_st_rps_cache *cache)
+static int rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
+					 struct rkvdec_hevc_st_rps_cache *cache)
 {
 	u8 num_sets = run->sps->num_short_term_ref_pic_sets;
 	size_t size;
 	int idx;
 
 	if (!run->ext_sps_st_rps)
-		return;
+		return 0;
 
 	size = num_sets * sizeof(*cache->sets);
 	if (cache->num_sets == num_sets && !memcmp(cache->sets, run->ext_sps_st_rps, size))
-		return;
+		return 0;
 
 	struct calculated_rps_st_set *calculated_rps_st_sets =
 		kzalloc(sizeof(struct calculated_rps_st_set) *
 			run->sps->num_short_term_ref_pic_sets, GFP_KERNEL);
 
+	if (!calculated_rps_st_sets)
+		return -ENOMEM;
+
 	for (idx = 0; idx < run->sps->num_short_term_ref_pic_sets; idx++) {
 		const struct v4l2_ctrl_hevc_ext_sps_st_rps *rps_data = &run->ext_sps_st_rps[idx];
 
@@ -362,13 +365,22 @@ static void rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rk
 
 	memcpy(cache->sets, run->ext_sps_st_rps, size);
 	cache->num_sets = num_sets;
+
+	return 0;
 }
 
-void rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
-				 struct rkvdec_hevc_st_rps_cache *st_cache)
+int rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
+				struct rkvdec_hevc_st_rps_cache *st_cache)
 {
-	rkvdec_hevc_prepare_hw_st_rps(run, rps, st_cache);
+	int ret;
+
+	ret = rkvdec_hevc_prepare_hw_st_rps(run, rps, st_cache);
+	if (ret)
+		return ret;
+
 	rkvdec_hevc_assemble_hw_lt_rps(run, rps);
+
+	return 0;
 }
 
 struct vb2_buffer *
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h +2 -2
index f6a3dec6ff4ad..32f192743c8c8 100644
@@ -71,8 +71,8 @@ void compute_tiles_uniform(struct rkvdec_hevc_run *run, u16 log2_min_cb_size,
 void compute_tiles_non_uniform(struct rkvdec_hevc_run *run, u16 log2_min_cb_size,
 			       u16 width, u16 height, s32 pic_in_cts_width,
 			       s32 pic_in_cts_height, u16 *column_width, u16 *row_height);
-void rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
-				 struct rkvdec_hevc_st_rps_cache *st_cache);
+int rkvdec_hevc_assemble_hw_rps(struct rkvdec_hevc_run *run, struct rkvdec_rps *rps,
+				struct rkvdec_hevc_st_rps_cache *st_cache);
 void rkvdec_hevc_assemble_hw_scaling_list(struct rkvdec_ctx *ctx,
 					  struct rkvdec_hevc_run *run,
 					  struct scaling_factor *scaling_factor,
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c +6 -1
index 8388421a68a48..2a239114759a2 100644
@@ -593,6 +593,7 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 	struct rkvdec_hevc_run run;
 	struct rkvdec_hevc_ctx *hevc_ctx = ctx->priv;
 	struct rkvdec_hevc_priv_tbl *tbl = hevc_ctx->priv_tbl.cpu;
+	int ret;
 
 	rkvdec_hevc_run_preamble(ctx, &run);
 
@@ -606,7 +607,11 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 	rkvdec_hevc_assemble_hw_scaling_list(ctx, &run, &tbl->scaling_list,
 					     &hevc_ctx->scaling_matrix_cache);
 	assemble_hw_pps(ctx, &run);
-	rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+	ret = rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+	if (ret) {
+		rkvdec_run_postamble(ctx, &run.base);
+		return ret;
+	}
 
 	config_registers(ctx, &run);
 
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c +6 -1
index 47f236295f1e5..52554c75f47f4 100644
@@ -542,6 +542,7 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 	struct rkvdec_hevc_ctx *hevc_ctx = ctx->priv;
 	struct rkvdec_hevc_priv_tbl *tbl = hevc_ctx->priv_tbl.cpu;
 	u32 timeout_threshold;
+	int ret;
 
 	rkvdec_hevc_run_preamble(ctx, &run);
 
@@ -560,7 +561,11 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)
 	rkvdec_hevc_assemble_hw_scaling_list(ctx, &run, &tbl->scaling_list,
 					     &hevc_ctx->scaling_matrix_cache);
 	assemble_hw_pps(ctx, &run);
-	rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+	ret = rkvdec_hevc_assemble_hw_rps(&run, &tbl->rps, &hevc_ctx->st_cache);
+	if (ret) {
+		rkvdec_run_postamble(ctx, &run.base);
+		return ret;
+	}
 
 	config_registers(ctx, &run);
 
Signature · 1 line
2.56.0