Skip to contentSkip to conversation

media: iris: check decoder format allocations

PATCH Superseded Ruoyu Wang 3 participants 5 messages Jun 6

Subsystems (4)
lore
Share & tools
lore

Ruoyu Wang

author
#0 lore raw

PATCH 1/1 media: iris: check decoder format allocations

iris_vdec_inst_init() allocates the source and destination v4l2_format structures and then immediately writes fields through inst->fmt_src and inst->fmt_dst. Either allocation can fail, leading to a NULL pointer dereference during instance initialization.

Check both allocations before initializing the formats. Free any partial allocation, clear the instance pointers so later cleanup does not see dangling values, and return -ENOMEM so the open path can unwind the instance.

---
 drivers/media/platform/qcom/iris/iris_vdec.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/media/platform/qcom/iris/iris_vdec.c +7 -0
index 99d544e2af4f9..dd18079a9ea5f 100644
@@ -23,6 +23,13 @@ int iris_vdec_inst_init(struct iris_inst *inst)
 
 	inst->fmt_src = kzalloc_obj(*inst->fmt_src);
 	inst->fmt_dst = kzalloc_obj(*inst->fmt_dst);
+	if (!inst->fmt_src || !inst->fmt_dst) {
+		kfree(inst->fmt_src);
+		kfree(inst->fmt_dst);
+		inst->fmt_src = NULL;
+		inst->fmt_dst = NULL;
+		return -ENOMEM;
+	}
 
 	inst->fw_min_count = MIN_BUFFERS;
 
Signature · 1 line
2.34.1
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] When max_session_count is reached, iris_add_session() silently drops the new instance, but iris_open() successfully returns, creating an untracked session. --

commit 0059fb169f9cd49b702675622f4a0ad32fd2392e
Author: Ruoyu Wang <[email protected]>

media: iris: check decoder format allocations

This commit adds checks for v4l2_format allocations in iris_vdec_inst_init() to prevent NULL pointer dereferences. It properly frees any partial allocations and returns -ENOMEM to allow the open path to unwind.

Quoted 4 lines
drivers/media/platform/qcom/iris/iris_vdec.c +0 -0
index 99d544e2af4f9..dd18079a9ea5f 100644

[ ... ]

drivers/media/platform/qcom/iris/iris_vdec.c int iris_vdec_inst_init(struct iris_inst *inst) · line 34

Quoted 13 lines
drivers/media/platform/qcom/iris/iris_vdec.c +7 -0
@@ -23,6 +23,13 @@ int iris_vdec_inst_init(struct iris_inst *inst)
 
 	inst->fmt_src = kzalloc_obj(*inst->fmt_src);
 	inst->fmt_dst = kzalloc_obj(*inst->fmt_dst);
+	if (!inst->fmt_src || !inst->fmt_dst) {
+		kfree(inst->fmt_src);
+		kfree(inst->fmt_dst);
+		inst->fmt_src = NULL;
+		inst->fmt_dst = NULL;
+		return -ENOMEM;
+	}
 
 	inst->fw_min_count = MIN_BUFFERS;

While reviewing the unwinding logic in the open path, I noticed a separate issue in drivers/media/platform/qcom/iris/iris_vidc.c:

drivers/media/platform/qcom/iris/iris_vidc.c:iris_add_session() {
    ...
		count++;

	if (count < core->iris_platform_data->max_session_count)
		list_add_tail(&inst->list, &core->instances);
    ...
}
drivers/media/platform/qcom/iris/iris_vidc.c:iris_open() {
    ...
	iris_add_session(inst);

	inst->fh.m2m_ctx = inst->m2m_ctx;

	return 0;
}

[Severity: High]
This isn't a bug introduced by this patch, but does this logic allow userspace to bypass the session limit and create an untracked session?

When the current session count reaches max_session_count, iris_add_session() silently drops the new instance without returning an error. Since iris_open() does not verify if the instance was successfully added to the core instances list, it returns 0 and provides userspace with a valid file descriptor.

Could this cause problems during streaming when resource management functions like iris_scale_power() and iris_check_core_mbps() iterate over the instances list to calculate total bandwidth and clock requirements? Since the untracked instance is missing from the list, its requirements are ignored, which might leave the hardware under-clocked or starved of bandwidth, potentially leading to hardware timeouts or watchdog resets.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Ruoyu Wang
Quoted 20 lines · Ruoyu Wang

iris_vdec_inst_init() allocates the source and destination v4l2_format structures and then immediately writes fields through inst->fmt_src and inst->fmt_dst. Either allocation can fail, leading to a NULL pointer dereference during instance initialization.

Check both allocations before initializing the formats. Free any partial allocation, clear the instance pointers so later cleanup does not see dangling values, and return -ENOMEM so the open path can unwind the instance.

---
 drivers/media/platform/qcom/iris/iris_vdec.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/media/platform/qcom/iris/iris_vdec.c +0 -0
index 99d544e2af4f9..dd18079a9ea5f 100644

drivers/media/platform/qcom/iris/iris_vdec.c int iris_vdec_inst_init(struct iris_inst *inst) · line 32

Quoted 10 lines
drivers/media/platform/qcom/iris/iris_vdec.c +7 -0
@@ -23,6 +23,13 @@ int iris_vdec_inst_init(struct iris_inst *inst)
 
 	inst->fmt_src = kzalloc_obj(*inst->fmt_src);
 	inst->fmt_dst = kzalloc_obj(*inst->fmt_dst);
+	if (!inst->fmt_src || !inst->fmt_dst) {
+		kfree(inst->fmt_src);
+		kfree(inst->fmt_dst);
+		inst->fmt_src = NULL;
+		inst->fmt_dst = NULL;
+		return -ENOMEM;
+	}

I'd rather see the check for the allocated objects before they are assigned to the fields in the instance.

Quoted 6 lines
drivers/media/platform/qcom/iris/iris_vdec.c +0 -0
 	inst->fw_min_count = MIN_BUFFERS;
 
Signature · 1 line
2.34.1
Signature · 2 lines
With best wishes
Dmitry
author
#3 lore raw

PATCH v2 1/1 media: iris: check decoder format allocations

In reply to Ruoyu Wang

iris_vdec_inst_init() allocates source and destination v4l2_format structures before initializing their fields. Allocation failures would leave the function dereferencing NULL pointers during instance initialization.

Allocate the formats into local variables and check each allocation before assigning them to the instance. If the second allocation fails, free the first allocation and return -ENOMEM. Store the pointers in the instance only after both allocations have succeeded so the open path can unwind cleanly.

---
Changes in v2:
- Allocate the formats into local variables and assign them to the instance only after both allocations succeed, as requested in review.

 drivers/media/platform/qcom/iris/iris_vdec.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
drivers/media/platform/qcom/iris/iris_vdec.c +13 -2
index 99d544e2af4f9..837f29f403bb7 100644
@@ -19,10 +19,21 @@
 int iris_vdec_inst_init(struct iris_inst *inst)
 {
 	struct iris_core *core = inst->core;
+	struct v4l2_format *fmt_src, *fmt_dst;
 	struct v4l2_format *f;
 
-	inst->fmt_src = kzalloc_obj(*inst->fmt_src);
-	inst->fmt_dst = kzalloc_obj(*inst->fmt_dst);
+	fmt_src = kzalloc_obj(*fmt_src);
+	if (!fmt_src)
+		return -ENOMEM;
+
+	fmt_dst = kzalloc_obj(*fmt_dst);
+	if (!fmt_dst) {
+		kfree(fmt_src);
+		return -ENOMEM;
+	}
+
+	inst->fmt_src = fmt_src;
+	inst->fmt_dst = fmt_dst;
 
 	inst->fw_min_count = MIN_BUFFERS;
 
Signature · 1 line
2.51.0
#4 lore raw
Quoted 32 lines · Ruoyu Wang

iris_vdec_inst_init() allocates source and destination v4l2_format structures before initializing their fields. Allocation failures would leave the function dereferencing NULL pointers during instance initialization.

Allocate the formats into local variables and check each allocation before assigning them to the instance. If the second allocation fails, free the first allocation and return -ENOMEM. Store the pointers in the instance only after both allocations have succeeded so the open path can unwind cleanly.

---
Changes in v2:
- Allocate the formats into local variables and assign them to the instance only after both allocations succeed, as requested in review.

 drivers/media/platform/qcom/iris/iris_vdec.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
drivers/media/platform/qcom/iris/iris_vdec.c +1 -1
index 99d544e2af4f9..837f29f403bb7 100644
@@ -19,10 +19,21 @@
 int iris_vdec_inst_init(struct iris_inst *inst)
 {
 	struct iris_core *core = inst->core;
+	struct v4l2_format *fmt_src, *fmt_dst;
 	struct v4l2_format *f;
 
-	inst->fmt_src = kzalloc_obj(*inst->fmt_src);

drivers/media/platform/qcom/iris/iris_vdec.c line 33

Quoted 10 lines
drivers/media/platform/qcom/iris/iris_vdec.c +9 -1
-	inst->fmt_dst = kzalloc_obj(*inst->fmt_dst);
+	fmt_src = kzalloc_obj(*fmt_src);
+	if (!fmt_src)
+		return -ENOMEM;
+
+	fmt_dst = kzalloc_obj(*fmt_dst);
+	if (!fmt_dst) {
+		kfree(fmt_src);
+		return -ENOMEM;
+	}

This is not the style of the rollback that is used in Linux kernel. Also if iris_ctrls_init() fails, then the allocate memory will not be unallocated. Further iris_open() will happily overwrite those pointers, resulting in a memory leak.

Should we replace the pointers with the instances of v4l2_format instead?

BTW: please don't send patch iterations as a reply to a previous thread. Always start a new thread for the new iteration.

Quoted 9 lines
drivers/media/platform/qcom/iris/iris_vdec.c +3 -0
+
+	inst->fmt_src = fmt_src;
+	inst->fmt_dst = fmt_dst;
 
 	inst->fw_min_count = MIN_BUFFERS;
 
Signature · 1 line
2.51.0
Signature · 2 lines
With best wishes
Dmitry