Skip to contentSkip to conversation

media: meson: vdec: fix NULL pointer deref in vdec_try_fmt_common

PATCH In review Christian Hewitt 3 participants 3 messages Jun 24 – Jul 16

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 media: meson: vdec: fix NULL pointer deref in vdec_try_fmt_common

When VIDIOC_TRY_FMT is called with an unsupported pixel format on the OUTPUT queue, vdec_try_fmt_common() falls back to V4L2_PIX_FMT_MPEG2. However, if a distro has locally patched MPEG2 support out (as it has been broken for some time) the platform format table does not contain MPEG2 so find_format() returns NULL and the subsequent dereference of fmt_out->max_width triggers a NULL pointer dereference.

Fix this by falling back to the first format in the platform's format array instead of hardcoding V4L2_PIX_FMT_MPEG2. This is always valid since every platform defines at least one format.

Fixes 3e7f51bd9607 ("media: meson: add v4l2 m2m video decoder driver")

---
This fixes use of v4l2-ctl and v4l2-compliance tools when MPEG2 support has been disabled via vdec_platform.c - NB: although this is a fix it's also a rather niche scenario so there's no need to backport it to older stable kernels.

 drivers/staging/media/meson/vdec/vdec.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/staging/media/meson/vdec/vdec.c +2 -2
index 4b77ec1af5a7..f3e7072f221a 100644
@@ -504,8 +504,8 @@ vdec_try_fmt_common(struct amvdec_session *sess, u32 size,
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
 		fmt_out = find_format(fmts, size, pixmp->pixelformat);
 		if (!fmt_out) {
-			pixmp->pixelformat = V4L2_PIX_FMT_MPEG2;
-			fmt_out = find_format(fmts, size, pixmp->pixelformat);
+			pixmp->pixelformat = fmts[0].pixfmt;
+			fmt_out = &fmts[0];
 		}
 		break;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
Signature · 1 line
2.43.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Missing V4L2 queue busy check in vdec_s_fmt allows format changes during active streaming, leading to type confusion and memory corruption in the ISR. --

commit 8aa33adb9b745ad7b1a26721ccafdbeb5f0768bb
Author: Christian Hewitt <[email protected]>

media: meson: vdec: fix NULL pointer deref in vdec_try_fmt_common

This commit fixes a NULL pointer dereference in vdec_try_fmt_common() by falling back to the first format in the platform's format array instead of hardcoding MPEG2. This prevents crashes on systems where MPEG2 support has been locally patched out.

drivers/staging/media/meson/vdec/vdec.c

Quoted 4 lines
drivers/staging/media/meson/vdec/vdec.c +0 -0
index a039d925c0fe5..c18bf352b1996 100644

[ ... ]

[Severity: High]
This is a pre-existing issue, but while looking at the format handling in this driver, I noticed a potential race condition.

Does vdec_s_fmt() allow format changes during active streaming?

drivers/staging/media/meson/vdec/vdec.c:vdec_s_fmt() {
        ...
	if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
		sess->fmt_out = fmt_out;
	else if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
		sess->pixfmt_cap = format.fmt.pix_mp.pixelformat;
        ...
}

It appears there is no check to ensure the V4L2 queues are idle (e.g., via vb2_is_busy()) before overwriting the session format parameters.

If a userspace application is actively streaming and concurrently calls VIDIOC_S_FMT with a different format, could this lead to type confusion and memory corruption in the hardirq handler?

drivers/staging/media/meson/vdec/vdec.c:vdec_isr() {
        ...
	return sess->fmt_out->codec_ops->isr(sess);
}

Because the hardirq handler dereferences sess->fmt_out without locks, a mid-stream format change might cause the ISR of the newly selected codec to execute on a session state (sess->priv) initialized for the previous codec.

Signature · 6 lines
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1

_______________________________________________
linux-amlogic mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-amlogic
Reviewed-by on v1 1/1
#2 lore raw
In reply to Christian Hewitt

Le mercredi 24 juin 2026 à 06:58 +0000, Christian Hewitt a écrit :

Quoted 7 lines

When VIDIOC_TRY_FMT is called with an unsupported pixel format on the OUTPUT queue, vdec_try_fmt_common() falls back to V4L2_PIX_FMT_MPEG2. However, if a distro has locally patched MPEG2 support out (as it has been broken for some time) the platform format table does not contain MPEG2 so find_format() returns NULL and the subsequent dereference of fmt_out->max_width triggers a NULL pointer dereference.

Quoted 6 lines

Fix this by falling back to the first format in the platform's format array instead of hardcoding V4L2_PIX_FMT_MPEG2. This is always valid since every platform defines at least one format.

Fixes 3e7f51bd9607 ("media: meson: add v4l2 m2m video decoder driver")
Quoted 27 lines

---
This fixes use of v4l2-ctl and v4l2-compliance tools when MPEG2 support has been disabled via vdec_platform.c - NB: although this is a fix it's also a rather niche scenario so there's no need to backport it to older stable kernels.

 drivers/staging/media/meson/vdec/vdec.c | 4 ++--

 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/staging/media/meson/vdec/vdec.c +0 -0
index 4b77ec1af5a7..f3e7072f221a 100644
@@ -504,8 +504,8 @@ vdec_try_fmt_common(struct amvdec_session *sess, u32 size,

  case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
  fmt_out = find_format(fmts, size, pixmp->pixelformat); if (!fmt_out) {

drivers/staging/media/meson/vdec/vdec.c +0 -2
-			pixmp->pixelformat = V4L2_PIX_FMT_MPEG2;
-			fmt_out = find_format(fmts, size, pixmp-
Quoted 1 line

pixelformat);

drivers/staging/media/meson/vdec/vdec.c +2 -0
+			pixmp->pixelformat = fmts[0].pixfmt;
+			fmt_out = &fmts[0];

  }
  break;
  case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE: