media: rkvdec: Fix the HEVC reference picture set handling
Subsystems (3)
cover media: rkvdec: Fix the HEVC reference picture set handling
This series fixes the code the VDPU381 and VDPU383 back ends of the rkvdec driver share for the HEVC EXT_SPS_ST_RPS and EXT_SPS_LT_RPS controls. The defects were found while bringing up the VDPU346 of the RK3568, which uses the VDPU381 back end, on top of Christian Hewitt's VDPU346 series (last posted as v3:
https://lore.kernel.org/r/[email protected]).
Several are memory-safety fixes for input a userspace V4L2 client can submit to the decoder, and carry Cc: stable like the EXT SPS RPS bounds merged in v7.3 (f0b9d7e5be06, 796b5c6d4f16, 052c5ed5a1d9).
Tested on a NanoPi R5C (RK3568) with Linux 6.18 and the VDPU346 series
backported: GStreamer fluster JCT-VC-HEVC_V1 stays at 140/147, an FFmpeg v4l2request build that sets the RPS controls confirms the reads and the short-term set cache, and the VDPU381 refusal removes a watchdog reset of the RK3568. The same VDPU381 back end was exercised on a Radxa ROCK 5B (RK3588): H.264, HEVC and VP9 decode through rkvdec with no kernel message across the fluster runs (HEVC 146/147), a per-patch A/B sweep and a 30-minute stability run, at the baseline throughput. The VDPU383 back end has no hardware here, the code-only patches have no reproducer, and nothing was tested on an RK3576.
An LLM coding agent did the analysis, and helped with writing the patches. A second instance, given only the code, reviewed the series, and has tested and validated it on real hardware independently.
Daniel Golle (7):
media: rkvdec: hevc: apply request controls first
media: rkvdec: vdpu383: complete refused HEVC requests
media: rkvdec: hevc: bound the RPS reads by the control size
media: rkvdec: hevc: bound predicted short-term RPS sets
media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets
media: rkvdec: hevc: cache every short-term RPS set
media: rkvdec: hevc: check the RPS scratch allocation
.../rockchip/rkvdec/rkvdec-hevc-common.c | 52 +++++++++++++------ .../rockchip/rkvdec/rkvdec-hevc-common.h | 11 +++- .../rockchip/rkvdec/rkvdec-vdpu381-hevc.c | 24 +++++---- .../rockchip/rkvdec/rkvdec-vdpu383-hevc.c | 14 +++-- 4 files changed, 67 insertions(+), 34 deletions(-)
base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb
Signature · 1 line
2.56.0
PATCH 1/7 media: rkvdec: hevc: apply request controls first
rkvdec_hevc_run_preamble() takes the control pointers and the slice count before rkvdec_run_preamble() applies the request. When a request carries more elements of a dynamic array control than the control has held so far, v4l2_ctrl_request_setup() reallocates the array and frees the old one, which leaves run->slices_params, run->ext_sps_st_rps or run->ext_sps_lt_rps pointing to freed memory, and run->num_slices holds the element count of the previous request. The EXT SPS RPS controls are also looked up depending on has_sps_st_rps and has_sps_lt_rps before the request's own controls could set them.
A userspace V4L2 client that can open the RKVDEC m2m decoder can submit a request that grows one of these dynamic array controls and reach the freed array through the stale pointer.
Apply the request first, as rkvdec_vp9_run_preamble() does.
--- drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +2 -2
@@ -429,6 +429,8 @@ void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx, {struct v4l2_ctrl *ctrl; +rkvdec_run_preamble (ctx, &run->base); + ctrl =v4l2_ctrl_find (&ctx->ctrl_hdl,V4L2_CID_STATELESS_HEVC_DECODE_PARAMS ); run->decode_params = ctrl ? ctrl->p_cur.p :NULL ;
@@ -460,6 +462,4 @@ void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx, }else { run->ext_sps_lt_rps =NULL ; } - -rkvdec_run_preamble (ctx, &run->base); }
Signature · 1 line
2.56.0
PATCH 2/7 media: rkvdec: vdpu383: complete refused HEVC requests
In reply to Daniel Gollerkvdec_hevc_run() returns -EINVAL when the RPS controls are missing, after rkvdec_hevc_run_preamble() applied the request's controls but without completing them. rkvdec_device_run() then returns the buffers with VB2_BUF_STATE_ERROR, which unbinds the source buffer from the request, while the request's control handler object stays incomplete. The request never reaches MEDIA_REQUEST_STATE_COMPLETE: poll() on its file descriptor never reports it, and MEDIA_REQUEST_IOC_REINIT fails with -EBUSY.
Complete the request's controls before returning the error, as rkvdec_vp9_run() does.
--- drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c | 1 + 1 file changed, 1 insertion(+)
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c +1 -0
@@ -553,6 +553,7 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) || (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {dev_err_ratelimited (rkvdec->dev,"Long and short term RPS not set\n" ); +rkvdec_run_postamble (ctx, &run.base);return -EINVAL ; }
Signature · 1 line
2.56.0
PATCH 3/7 media: rkvdec: hevc: bound the RPS reads by the control size
In reply to Daniel GolleThe RPS helpers read num_short_term_ref_pic_sets elements of the EXT_SPS_ST_RPS control and num_long_term_ref_pics_sps elements of the EXT_SPS_LT_RPS control, however many elements userspace passed. Beyond the passed elements they read stale entries of the control's array, and past its allocation once the SPS counts more sets than the control has ever held, as the control framework sizes a dynamic array for the largest element count it has seen. The short-term sets read from there are unchecked, and a num_negative_pics or num_positive_pics above 16 overflows the calculated sets, the stack arrays of st_ref_pic_set_prediction() and the hardware RPS table.
A userspace V4L2 client that can open the RKVDEC m2m decoder can set an SPS whose counts exceed the number of elements it passed in these controls, so the reads run past the allocation.
Use the controls only when they hold at least as many elements as the SPS counts, and treat them as not set otherwise in both back ends.
--- .../platform/rockchip/rkvdec/rkvdec-hevc-common.c | 12 ++++++------ .../platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c | 4 ++-- .../platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c | 4 ++-- 3 files changed, 10 insertions(+), 10 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +6 -6
@@ -448,18 +448,18 @@ void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx, V4L2_CID_STATELESS_HEVC_SCALING_MATRIX); run->scaling_matrix = ctrl ? ctrl->p_cur.p :NULL ; + run->ext_sps_st_rps =NULL ;if (ctx->has_sps_st_rps) { ctrl =v4l2_ctrl_find (&ctx->ctrl_hdl,V4L2_CID_STATELESS_HEVC_EXT_SPS_ST_RPS ); - run->ext_sps_st_rps = ctrl ? ctrl->p_cur.p :NULL ; - }else { - run->ext_sps_st_rps =NULL ; +if (ctrl && ctrl->elems >= run->sps->num_short_term_ref_pic_sets) + run->ext_sps_st_rps = ctrl->p_cur.p; } + run->ext_sps_lt_rps =NULL ;if (ctx->has_sps_lt_rps) { ctrl =v4l2_ctrl_find (&ctx->ctrl_hdl,V4L2_CID_STATELESS_HEVC_EXT_SPS_LT_RPS ); - run->ext_sps_lt_rps = ctrl ? ctrl->p_cur.p :NULL ; - }else { - run->ext_sps_lt_rps =NULL ; +if (ctrl && ctrl->elems >= run->sps->num_long_term_ref_pics_sps) + run->ext_sps_lt_rps = ctrl->p_cur.p; } }
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c +2 -2
@@ -604,8 +604,8 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)* On vdpu381, not setting the long and short term ref sets will just output wrong frames. * Let's just warn about it and let the decoder run anyway. */ -if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) || - (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) { +if ((!run.ext_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) || + (!run.ext_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {dev_warn_ratelimited (rkvdec->dev,"Long and short term RPS not set\n" ); }else {rkvdec_hevc_assemble_hw_rps (&run, &tbl->rps, &hevc_ctx->st_cache);
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c +2 -2
@@ -550,8 +550,8 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)* To be on the safe side for this new v4l2 control, write an error in the log and mark * the buffer as failed by returning an error here. */ -if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) || - (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) { +if ((!run.ext_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) || + (!run.ext_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {dev_err_ratelimited (rkvdec->dev,"Long and short term RPS not set\n" );rkvdec_run_postamble (ctx, &run.base);return -EINVAL ;
Signature · 1 line
2.56.0
Re: [PATCH 3/7] media: rkvdec: hevc: bound the RPS reads by the control size
Le samedi 14 novembre 2026 à 10:22 +0000, Daniel Golle a écrit :
Quoted 4 lines
The RPS helpers read
num_short_term_ref_pic_setselements of theEXT_SPS_ST_RPScontrol andnum_long_term_ref_pics_spselements of theEXT_SPS_LT_RPScontrol, however many elements userspace passed.
What did you want to say?
Quoted 3 lines
Beyond
the passed elements they read stale entries of the control's array, and past its allocation once the SPS counts more sets than the control
More broken text, can you work a bit on your messages, be concise, precise, don't get into details but into the understanding of what you are achieving. The code is precise enough.
Quoted 39 lines
has ever held, as the control framework sizes a dynamic array for the largest element count it has seen. The short-term sets read from there are unchecked, and a
num_negative_picsornum_positive_picsabove 16 overflows the calculated sets, the stack arrays ofst_ref_pic_set_prediction()and the hardware RPS table.A userspace V4L2 client that can open the RKVDEC m2m decoder can set an SPS whose counts exceed the number of elements it passed in these controls, so the reads run past the allocation.
Use the controls only when they hold at least as many elements as the SPS counts, and treat them as not set otherwise in both back ends.
Fixes c9a59dc2acc7 ("media: rkvdec: Add HEVC support for the VDPU381 variant")Signed-off-by Daniel Golle <[email protected]>--- .../platform/rockchip/rkvdec/rkvdec-hevc-common.c | 12 ++++++------ .../platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c | 4 ++-- .../platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c | 4 ++-- 3 files changed, 10 insertions(+), 10 deletions(-)drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +3
-3@@ -448,18 +448,18 @@ void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx, V4L2_CID_STATELESS_HEVC_SCALING_MATRIX); run->scaling_matrix = ctrl ? ctrl->p_cur.p :NULL ; + run->ext_sps_st_rps =NULL ;if (ctx->has_sps_st_rps) { ctrl =v4l2_ctrl_find (&ctx->ctrl_hdl,V4L2_CID_STATELESS_HEVC_EXT_SPS_ST_RPS ); - run->ext_sps_st_rps = ctrl ? ctrl->p_cur.p :NULL ; - }else { - run->ext_sps_st_rps =NULL ; +if (ctrl && ctrl->elems >= run->sps->num_short_term_ref_pic_sets) + run->ext_sps_st_rps = ctrl->p_cur.p;
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c void rkvdec_hevc_run_preamble(struct rkvdec_ctx *ctx, · line 463
Quoted 10 lines
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +3
-3} + run->ext_sps_lt_rps =NULL ;if (ctx->has_sps_lt_rps) { ctrl =v4l2_ctrl_find (&ctx->ctrl_hdl,V4L2_CID_STATELESS_HEVC_EXT_SPS_LT_RPS ); - run->ext_sps_lt_rps = ctrl ? ctrl->p_cur.p :NULL ; - }else { - run->ext_sps_lt_rps =NULL ; +if (ctrl && ctrl->elems >= run->sps->num_long_term_ref_pics_sps) + run->ext_sps_lt_rps = ctrl->p_cur.p;
So you are saying that if the passed array is too short, we should use an older entry to none? That seems a little strange, what about up-front rejecting bad entries?
Nicolas
Quoted 32 lines
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +0
-0} }drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c +2
-2@@ -604,8 +604,8 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)* On vdpu381, not setting the long and short term ref sets will just output wrong frames. * Let's just warn about it and let the decoder run anyway. */ -if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) || - (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) { +if ((!run.ext_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) || + (!run.ext_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {dev_warn_ratelimited (rkvdec->dev,"Long and short term RPS not set\n" ); }else {rkvdec_hevc_assemble_hw_rps (&run, &tbl->rps, &hevc_ctx->st_cache);drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c +2
-2@@ -550,8 +550,8 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)* To be on the safe side for this new v4l2 control, write an error in the log and mark * the buffer as failed by returning an error here. */ -if ((!ctx->has_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) || - (!ctx->has_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) { +if ((!run.ext_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) || + (!run.ext_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) {dev_err_ratelimited (rkvdec->dev,"Long and short term RPS not set\n" );rkvdec_run_postamble (ctx, &run.base);return -EINVAL ;
Re: [PATCH 3/7] media: rkvdec: hevc: bound the RPS reads by the control size
Hi Nicolas,
thanks for the review and sorry about the messy text.
Quoted 7 lines · Nicolas Dufresne
Le samedi 14 novembre 2026 à 10:22 +0000, Daniel Golle a écrit :
Quoted 4 lines
The RPS helpers read
num_short_term_ref_pic_setselements of theEXT_SPS_ST_RPScontrol andnum_long_term_ref_pics_spselements of theEXT_SPS_LT_RPScontrol, however many elements userspace passed.What did you want to say?
The number of RPS entries comes from the SPS control, the entries themselves from the EXT_SPS_ST_RPS/LT_RPS array controls. The driver never checks that the arrays are as long as the SPS says, so it reads past the end of what userspace passed. I will reword the message along these lines in v2.
Quoted 8 lines
Quoted 4 lines
Beyond
the passed elements they read stale entries of the control's array, and past its allocation once the SPS counts more sets than the controlMore broken text, can you work a bit on your messages, be concise, precise, don't get into details but into the understanding of what you are achieving. The code is precise enough.
Got it. Reading it again now I understand your point, I may have been a bit too excited about my findings on a Saturday morning...
PATCH 5/7 media: rkvdec: vdpu381: refuse HEVC jobs without RPS sets
In reply to Daniel GolleWhen the SPS uses short-term or long-term reference picture sets and the matching EXT_SPS_ST_RPS or EXT_SPS_LT_RPS control was never set or holds fewer sets than the SPS counts, rkvdec_hevc_run() logs a warning and starts the decoder with an RPS table that does not belong to the stream, which at best gives wrong pictures. On the VDPU346 of the RK3568, which uses this back end, such decodes also raise error interrupts and time out, the IOMMU force reset fails, and the system stops responding until the hardware watchdog resets it.
Refuse the job as the VDPU383 back end does: log an error and return -EINVAL before any table is assembled, so that rkvdec_device_run() returns the buffers with VB2_BUF_STATE_ERROR and the hardware is not started. The request's controls are completed first, as rkvdec_vp9_run() does, so that the request is signalled as complete.
--- .../rockchip/rkvdec/rkvdec-vdpu381-hevc.c | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c +8 -11
@@ -596,21 +596,18 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)rkvdec_hevc_run_preamble (ctx, &run); -rkvdec_hevc_assemble_hw_scaling_list (ctx, &run, &tbl->scaling_list, - &hevc_ctx->scaling_matrix_cache); -assemble_hw_pps (ctx, &run); - -/* -* On vdpu381, not setting the long and short term ref sets will just output wrong frames. -* Let's just warn about it and let the decoder run anyway. -*/ if ((!run.ext_sps_lt_rps && run.sps->num_long_term_ref_pics_sps) || (!run.ext_sps_st_rps && run.sps->num_short_term_ref_pic_sets)) { -dev_warn_ratelimited (rkvdec->dev,"Long and short term RPS not set\n" ); - }else { -rkvdec_hevc_assemble_hw_rps (&run, &tbl->rps, &hevc_ctx->st_cache); +dev_err_ratelimited (rkvdec->dev,"Long and short term RPS not set\n" ); +rkvdec_run_postamble (ctx, &run.base); +return -EINVAL ; } +rkvdec_hevc_assemble_hw_scaling_list (ctx, &run, &tbl->scaling_list, + &hevc_ctx->scaling_matrix_cache); +assemble_hw_pps (ctx, &run); +rkvdec_hevc_assemble_hw_rps (&run, &tbl->rps, &hevc_ctx->st_cache); +config_registers (ctx, &run);rkvdec_run_postamble (ctx, &run.base);
Signature · 1 line
2.56.0
PATCH 6/7 media: rkvdec: hevc: cache every short-term RPS set
In reply to Daniel Gollerkvdec_hevc_prepare_hw_st_rps() skips building the short-term RPS table when the first element of the EXT_SPS_ST_RPS control equals its cache, which holds a single element and starts zeroed. When the first element is all zero, e.g. for an empty first set, the table entries of the remaining sets are never built, and an SPS that differs from the previous one only after its first set, or only in num_short_term_ref_pic_sets, keeps the previous table. Inter pictures are then decoded with wrong references: JCT-VC DELTAQP_A_BRCM_4 decodes its intra pictures correctly and its inter pictures wrongly on an RK3568, whose VDPU346 uses the VDPU381 back end.
Cache all sets the SPS uses together with their number, for both the VDPU381 and the VDPU383 back end. The SPS control validation bounds num_short_term_ref_pic_sets to 64, the size of the cache.
--- .../platform/rockchip/rkvdec/rkvdec-hevc-common.c | 12 ++++++++---- .../platform/rockchip/rkvdec/rkvdec-hevc-common.h | 9 ++++++++- .../platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c | 2 +- .../platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c | 2 +- 4 files changed, 18 insertions(+), 7 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +8 -4
@@ -330,14 +330,17 @@ static void st_ref_pic_set_calculate(struct rkvdec_hevc_run *run, int idx, }static void rkvdec_hevc_prepare_hw_st_rps (struct rkvdec_hevc_run *run,struct rkvdec_rps *rps, -struct v4l2_ctrl_hevc_ext_sps_st_rps *cache) +struct rkvdec_hevc_st_rps_cache *cache) { +u8 num_sets = run->sps->num_short_term_ref_pic_sets; +size_t size;int idx;if (!run->ext_sps_st_rps)return ; -if (!memcmp (cache, run->ext_sps_st_rps,sizeof (struct v4l2_ctrl_hevc_ext_sps_st_rps ))) + size = num_sets *sizeof (*cache->sets); +if (cache->num_sets == num_sets && !memcmp (cache->sets, run->ext_sps_st_rps, size))return ; struct calculated_rps_st_set *calculated_rps_st_sets =
@@ -357,11 +360,12 @@ static void rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rkkfree (calculated_rps_st_sets); -memcpy (cache, run->ext_sps_st_rps,sizeof (struct v4l2_ctrl_hevc_ext_sps_st_rps )); +memcpy (cache->sets, run->ext_sps_st_rps, size); + cache->num_sets = num_sets; }void rkvdec_hevc_assemble_hw_rps (struct rkvdec_hevc_run *run,struct rkvdec_rps *rps, -struct v4l2_ctrl_hevc_ext_sps_st_rps *st_cache) +struct rkvdec_hevc_st_rps_cache *st_cache) {rkvdec_hevc_prepare_hw_st_rps (run, rps, st_cache);rkvdec_hevc_assemble_hw_lt_rps (run, rps);
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h +8 -1
@@ -35,10 +35,17 @@#define RKVDEC_RPS_HEVC_SIZE ALIGN(1032 + 64 * 384, 128) +#define RKVDEC_HEVC_MAX_SHORT_TERM_REF_PIC_SETS 64 +struct rkvdec_rps {u32 info[RKVDEC_RPS_HEVC_SIZE /8 /4 ]; } __packed; +struct rkvdec_hevc_st_rps_cache { +struct v4l2_ctrl_hevc_ext_sps_st_rps sets[RKVDEC_HEVC_MAX_SHORT_TERM_REF_PIC_SETS ]; +u8 num_sets; +}; +struct rkvdec_hevc_run {struct rkvdec_run base;const struct v4l2_ctrl_hevc_slice_params *slices_params;
@@ -65,7 +72,7 @@ void compute_tiles_non_uniform(struct rkvdec_hevc_run *run, u16 log2_min_cb_size u16 width, u16 height, s32 pic_in_cts_width, s32 pic_in_cts_height, u16 *column_width, u16 *row_height);void rkvdec_hevc_assemble_hw_rps (struct rkvdec_hevc_run *run,struct rkvdec_rps *rps, -struct v4l2_ctrl_hevc_ext_sps_st_rps *st_cache); +struct rkvdec_hevc_st_rps_cache *st_cache); void rkvdec_hevc_assemble_hw_scaling_list(struct rkvdec_ctx *ctx, struct rkvdec_hevc_run *run, struct scaling_factor *scaling_factor,
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c +1 -1
@@ -117,7 +117,7 @@ struct rkvdec_hevc_priv_tbl {struct rkvdec_hevc_ctx {struct rkvdec_aux_buf priv_tbl;struct v4l2_ctrl_hevc_scaling_matrix scaling_matrix_cache; -struct v4l2_ctrl_hevc_ext_sps_st_rps st_cache; +struct rkvdec_hevc_st_rps_cache st_cache;struct rkvdec_vdpu381_regs_hevc regs; };
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c +1 -1
@@ -124,7 +124,7 @@ struct rkvdec_hevc_priv_tbl {struct rkvdec_hevc_ctx {struct rkvdec_aux_buf priv_tbl;struct v4l2_ctrl_hevc_scaling_matrix scaling_matrix_cache; -struct v4l2_ctrl_hevc_ext_sps_st_rps st_cache; +struct rkvdec_hevc_st_rps_cache st_cache;struct vdpu383_regs_h26x regs; };
Signature · 1 line
2.56.0
PATCH 7/7 media: rkvdec: hevc: check the RPS scratch allocation
In reply to Daniel Gollerkvdec_hevc_prepare_hw_st_rps() uses the array returned by kzalloc() without checking it, so a failed allocation while a decode job is prepared dereferences a NULL pointer. Return -ENOMEM through rkvdec_hevc_assemble_hw_rps() and let both back ends refuse the job after completing the request's controls.
--- .../rockchip/rkvdec/rkvdec-hevc-common.c | 26 ++++++++++++++----- .../rockchip/rkvdec/rkvdec-hevc-common.h | 4 +-- .../rockchip/rkvdec/rkvdec-vdpu381-hevc.c | 7 ++++- .../rockchip/rkvdec/rkvdec-vdpu383-hevc.c | 7 ++++- 4 files changed, 33 insertions(+), 11 deletions(-)
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c +19 -7
@@ -329,24 +329,27 @@ static void st_ref_pic_set_calculate(struct rkvdec_hevc_run *run, int idx, st_rps->num_delta_pocs = st_rps->num_positive_pics + st_rps->num_negative_pics; } -static void rkvdec_hevc_prepare_hw_st_rps (struct rkvdec_hevc_run *run,struct rkvdec_rps *rps, -struct rkvdec_hevc_st_rps_cache *cache) +static int rkvdec_hevc_prepare_hw_st_rps (struct rkvdec_hevc_run *run,struct rkvdec_rps *rps, +struct rkvdec_hevc_st_rps_cache *cache) {u8 num_sets = run->sps->num_short_term_ref_pic_sets;size_t size;int idx;if (!run->ext_sps_st_rps) -return ; +return 0 ; size = num_sets *sizeof (*cache->sets);if (cache->num_sets == num_sets && !memcmp (cache->sets, run->ext_sps_st_rps, size)) -return ; +return 0 ;struct calculated_rps_st_set *calculated_rps_st_sets =kzalloc (sizeof (struct calculated_rps_st_set ) * run->sps->num_short_term_ref_pic_sets,GFP_KERNEL ); +if (!calculated_rps_st_sets) +return -ENOMEM ; +for (idx =0 ; idx < run->sps->num_short_term_ref_pic_sets; idx++) {const struct v4l2_ctrl_hevc_ext_sps_st_rps *rps_data = &run->ext_sps_st_rps[idx];
@@ -362,13 +365,22 @@ static void rkvdec_hevc_prepare_hw_st_rps(struct rkvdec_hevc_run *run, struct rkmemcpy (cache->sets, run->ext_sps_st_rps, size); cache->num_sets = num_sets; + +return 0 ; } -void rkvdec_hevc_assemble_hw_rps (struct rkvdec_hevc_run *run,struct rkvdec_rps *rps, -struct rkvdec_hevc_st_rps_cache *st_cache) +int rkvdec_hevc_assemble_hw_rps (struct rkvdec_hevc_run *run,struct rkvdec_rps *rps, +struct rkvdec_hevc_st_rps_cache *st_cache) { -rkvdec_hevc_prepare_hw_st_rps (run, rps, st_cache); +int ret; + + ret =rkvdec_hevc_prepare_hw_st_rps (run, rps, st_cache); +if (ret) +return ret; +rkvdec_hevc_assemble_hw_lt_rps (run, rps); + +return 0 ; } struct vb2_buffer *
drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h +2 -2
@@ -71,8 +71,8 @@ void compute_tiles_uniform(struct rkvdec_hevc_run *run, u16 log2_min_cb_size,void compute_tiles_non_uniform (struct rkvdec_hevc_run *run,u16 log2_min_cb_size,u16 width,u16 height,s32 pic_in_cts_width,s32 pic_in_cts_height,u16 *column_width,u16 *row_height); -void rkvdec_hevc_assemble_hw_rps (struct rkvdec_hevc_run *run,struct rkvdec_rps *rps, -struct rkvdec_hevc_st_rps_cache *st_cache); +int rkvdec_hevc_assemble_hw_rps (struct rkvdec_hevc_run *run,struct rkvdec_rps *rps, +struct rkvdec_hevc_st_rps_cache *st_cache); void rkvdec_hevc_assemble_hw_scaling_list(struct rkvdec_ctx *ctx, struct rkvdec_hevc_run *run, struct scaling_factor *scaling_factor,
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c +6 -1
@@ -593,6 +593,7 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)struct rkvdec_hevc_run run;struct rkvdec_hevc_ctx *hevc_ctx = ctx->priv;struct rkvdec_hevc_priv_tbl *tbl = hevc_ctx->priv_tbl.cpu; +int ret;rkvdec_hevc_run_preamble (ctx, &run);
@@ -606,7 +607,11 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)rkvdec_hevc_assemble_hw_scaling_list (ctx, &run, &tbl->scaling_list, &hevc_ctx->scaling_matrix_cache);assemble_hw_pps (ctx, &run); -rkvdec_hevc_assemble_hw_rps (&run, &tbl->rps, &hevc_ctx->st_cache); + ret =rkvdec_hevc_assemble_hw_rps (&run, &tbl->rps, &hevc_ctx->st_cache); +if (ret) { +rkvdec_run_postamble (ctx, &run.base); +return ret; + }config_registers (ctx, &run);
drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c +6 -1
@@ -542,6 +542,7 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)struct rkvdec_hevc_ctx *hevc_ctx = ctx->priv;struct rkvdec_hevc_priv_tbl *tbl = hevc_ctx->priv_tbl.cpu;u32 timeout_threshold; +int ret;rkvdec_hevc_run_preamble (ctx, &run);
@@ -560,7 +561,11 @@ static int rkvdec_hevc_run(struct rkvdec_ctx *ctx)rkvdec_hevc_assemble_hw_scaling_list (ctx, &run, &tbl->scaling_list, &hevc_ctx->scaling_matrix_cache);assemble_hw_pps (ctx, &run); -rkvdec_hevc_assemble_hw_rps (&run, &tbl->rps, &hevc_ctx->st_cache); + ret =rkvdec_hevc_assemble_hw_rps (&run, &tbl->rps, &hevc_ctx->st_cache); +if (ret) { +rkvdec_run_postamble (ctx, &run.base); +return ret; + }config_registers (ctx, &run);
Signature · 1 line
2.56.0