venus: core: add shutdown callback for venus
Subsystems (4)
v4 cover venus: core: add shutdown callback for venus
Add shutdown callback for venus driver. Handle race conditions in concurrency usecases like multiple browser YouTube browser tabs(approx 50 tabs) graphics_Stress, WiFi ON/OFF, Bluetooth ON/OF, and reboot in parallel.
Mansur Alisha Shaik (3):
venus: core: handle race condititon for core ops
venus: handle use after free for iommu_map/iommu_unmap
venus: core: add shutdown callback for venus
drivers/media/platform/qcom/venus/core.c | 9 +++++++++ drivers/media/platform/qcom/venus/firmware.c | 17 +++++++++++++---- drivers/media/platform/qcom/venus/hfi.c | 17 ++++++++++++++++- 3 files changed, 38 insertions(+), 5 deletions(-)
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
PATCH v4 1/3 venus: core: handle race condititon for core ops
For core ops we are having only write protect but there is no read protect, because of this in multthreading and concurrency, one CPU core is reading without wait which is causing the NULL pointer dereferece crash.
one such scenario is as show below, where in one CPU core, core->ops becoming NULL and in another CPU core calling core->ops->session_init().
CPU: core-7:
Call trace: hfi_session_init+0x180/0x1dc [venus_core] vdec_queue_setup+0x9c/0x364 [venus_dec] vb2_core_reqbufs+0x1e4/0x368 [videobuf2_common] vb2_reqbufs+0x4c/0x64 [videobuf2_v4l2] v4l2_m2m_reqbufs+0x50/0x84 [v4l2_mem2mem] v4l2_m2m_ioctl_reqbufs+0x2c/0x38 [v4l2_mem2mem] v4l_reqbufs+0x4c/0x5c __video_do_ioctl+0x2b0/0x39c
CPU: core-0:
Call trace: venus_shutdown+0x98/0xfc [venus_core] venus_sys_error_handler+0x64/0x148 [venus_core] process_one_work+0x210/0x3d0 worker_thread+0x248/0x3f4 kthread+0x11c/0x12c
---
Changes in V4:
- Addressed review comments by Stan in patch series
https://lore.kernel.org/patchwork/patch/1303678/
and combining this change along with shutdown callback as we are facing race conditions with shutdown callback
drivers/media/platform/qcom/venus/hfi.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
drivers/media/platform/qcom/venus/hfi.c +16 -1
@@ -195,19 +195,34 @@ EXPORT_SYMBOL_GPL(hfi_session_create);int hfi_session_init (struct venus_inst *inst,u32 pixfmt) {struct venus_core *core = inst->core; -const struct hfi_ops *ops = core->ops; +const struct hfi_ops *ops;int ret; +/* +* If core shutdown is in progress or if we are in system +* recovery, return an error as during system error recovery +* session_init() can't pass successfully +*/ +mutex_lock (&core->lock); +if (!core->ops || core->sys_error) { +mutex_unlock (&core->lock); +return -EIO ; + } +mutex_unlock (&core->lock); +if (inst->state !=INST_UNINIT )return -EINVAL ; inst->hfi_codec =to_codec_type (pixfmt);reinit_completion (&inst->done); +mutex_lock (&core->lock); + ops = core->ops; ret = ops->session_init (inst, inst->session_type, inst->hfi_codec);if (ret)return ret; +mutex_unlock (&core->lock); ret =wait_session_msg (inst);if (ret)return ret;
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Re: [PATCH v4 1/3] venus: core: handle race condititon for core ops
Quoted 62 lines · Mansur Alisha Shaik
For core ops we are having only write protect but there is no read protect, because of this in multthreading and concurrency, one CPU core is reading without wait which is causing the NULL pointer dereferece crash.
one such scenario is as show below, where in one CPU core,
core->opsbecoming NULL and in another CPU core callingcore->ops->session_init().CPU: core-7:
Call trace: hfi_session_init+0x180/0x1dc [venus_core] vdec_queue_setup+0x9c/0x364 [venus_dec] vb2_core_reqbufs+0x1e4/0x368 [videobuf2_common] vb2_reqbufs+0x4c/0x64 [videobuf2_v4l2] v4l2_m2m_reqbufs+0x50/0x84 [v4l2_mem2mem] v4l2_m2m_ioctl_reqbufs+0x2c/0x38 [v4l2_mem2mem] v4l_reqbufs+0x4c/0x5c __video_do_ioctl+0x2b0/0x39cCPU: core-0:
Call trace: venus_shutdown+0x98/0xfc [venus_core] venus_sys_error_handler+0x64/0x148 [venus_core] process_one_work+0x210/0x3d0 worker_thread+0x248/0x3f4 kthread+0x11c/0x12cSigned-off-by Mansur Alisha Shaik <[email protected]>---
Changes in V4:
- Addressed review comments by Stan in patch series
https://lore.kernel.org/patchwork/patch/1303678/
and combining this change along with shutdown callback as we are facing race conditions with shutdown callbackdrivers/media/platform/qcom/venus/hfi.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)drivers/media/platform/qcom/venus/hfi.c +11
-1@@ -195,19 +195,34 @@ EXPORT_SYMBOL_GPL(hfi_session_create);int hfi_session_init (struct venus_inst *inst,u32 pixfmt) {struct venus_core *core = inst->core; -const struct hfi_ops *ops = core->ops; +const struct hfi_ops *ops;int ret; +/* +* If core shutdown is in progress or if we are in system +* recovery, return an error as during system error recovery +* session_init() can't pass successfully +*/ +mutex_lock (&core->lock); +if (!core->ops || core->sys_error) { +mutex_unlock (&core->lock); +return -EIO ; + }
drivers/media/platform/qcom/venus/hfi.c EXPORT_SYMBOL_GPL(hfi_session_create); · line 220
Quoted 10 lines
drivers/media/platform/qcom/venus/hfi.c +4
-0+mutex_unlock (&core->lock); +if (inst->state !=INST_UNINIT )return -EINVAL ; inst->hfi_codec =to_codec_type (pixfmt);reinit_completion (&inst->done); +mutex_lock (&core->lock); + ops = core->ops;
This is not needed because we check core->ops for NULL under mutex held at the beginning of the function. Just keep ops initialization as it is in the original code.
Quoted 9 lines
drivers/media/platform/qcom/venus/hfi.c +1
-0ret = ops->session_init (inst, inst->session_type, inst->hfi_codec);if (ret)return ret; +mutex_unlock (&core->lock); ret =wait_session_msg (inst);if (ret)return ret;
Signature · 2 lines
regards, Stan
PATCH v4 2/3 venus: handle use after free for iommu_map/iommu_unmap
In reply to Mansur Alisha ShaikIn concurrency usecase and reboot scenario we are seeing muliple crashes related to iommu_map/iommu_unamp of core->fw.iommu_domain.
In one case we are seeing "Unable to handle kernel NULL pointer dereference at virtual address 0000000000000008" crash, this is because of core->fw.iommu_domain in venus_firmware_deinit() and trying to map in venus_boot() during venus_sys_error_handler()
Call trace: __iommu_map+0x4c/0x348 iommu_map+0x5c/0x70 venus_boot+0x184/0x230 [venus_core] venus_sys_error_handler+0xa0/0x14c [venus_core] process_one_work+0x210/0x3d0 worker_thread+0x248/0x3f4 kthread+0x11c/0x12c ret_from_fork+0x10/0x18
In second case we are seeing "Unable to handle kernel paging request at virtual address 006b6b6b6b6b6b9b" crash, this is because of unmappin iommu domain which is already unmapped.
Call trace: venus_remove+0xf8/0x108 [venus_core] venus_core_shutdown+0x1c/0x34 [venus_core] platform_drv_shutdown+0x28/0x34 device_shutdown+0x154/0x1fc kernel_restart_prepare+0x40/0x4c kernel_restart+0x1c/0x64 __arm64_sys_reboot+0x190/0x238 el0_svc_common+0xa4/0x154 el0_svc_compat_handler+0x2c/0x38 el0_svc_compat+0x8/0x10
--- drivers/media/platform/qcom/venus/firmware.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)
drivers/media/platform/qcom/venus/firmware.c +13 -4
@@ -171,9 +171,14 @@ static int venus_shutdown_no_tz(struct venus_core *core) iommu = core->fw.iommu_domain; - unmapped =iommu_unmap (iommu,VENUS_FW_START_ADDR , mapped); -if (unmapped != mapped) -dev_err (dev,"failed to unmap firmware\n" ); +if (core->fw.mapped_mem_size && iommu) { + unmapped =iommu_unmap (iommu,VENUS_FW_START_ADDR , mapped); + +if (unmapped != mapped) +dev_err (dev,"failed to unmap firmware\n" ); +else + core->fw.mapped_mem_size =0 ; + }return 0 ; }
@@ -305,7 +310,11 @@ void venus_firmware_deinit(struct venus_core *core) iommu = core->fw.iommu_domain;iommu_detach_device (iommu, core->fw.dev); -iommu_domain_free (iommu); + +if (core->fw.iommu_domain) { +iommu_domain_free (iommu); + core->fw.iommu_domain =NULL ; + }platform_device_unregister (to_platform_device (core->fw.dev)); }
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
PATCH v4 3/3 venus: core: add shutdown callback for venus
In reply to Mansur Alisha ShaikAfter the SMMU translation is disabled in the
arm-smmu shutdown callback during reboot, if
any subsystem are still alive then IOVAs they
are using will become PAs on bus, which may
lead to crash.
So implemented shutdown callback, which detach iommu maps.
---
Changes in V4:
- In venus_core_shutdown(), instead of venurs_remove() calling venus_shutdown() and venus_firmware_deinit().
- With venus_remove() call during shutdown we are facing few issue like ui freez this is because in venus_remove, hfi_core_deinit() will wait until all core instances count become zero dudring wait_var_event().
Changes in V3:
- Fix build errors
drivers/media/platform/qcom/venus/core.c | 9 +++++++++ 1 file changed, 9 insertions(+)
drivers/media/platform/qcom/venus/core.c +9 -0
@@ -345,6 +345,14 @@ static int venus_remove(struct platform_device *pdev)return ret; } +static void venus_core_shutdown (struct platform_device *pdev) +{ +struct venus_core *core =platform_get_drvdata (pdev); + +venus_shutdown (core); +venus_firmware_deinit (core); +} +static __maybe_unused int venus_runtime_suspend (struct device *dev) {struct venus_core *core =dev_get_drvdata (dev);
@@ -602,6 +610,7 @@ static struct platform_driver qcom_venus_driver = { .of_match_table = venus_dt_match, .pm = &venus_pm_ops, }, + .shutdown = venus_core_shutdown, };module_platform_driver (qcom_venus_driver);
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation