arm-smmu-qcom: Ensure the qcom_scm driver has finished probing
Subsystems (4)
PATCH RESEND 1/2 arm-smmu-qcom: Ensure the qcom_scm driver has finished probing
Robin Murphy pointed out that if the arm-smmu driver probes before the qcom_scm driver, we may call qcom_scm_qsmmu500_wait_safe_toggle() before the __scm is initialized.
Now, getting this to happen is a bit contrived, as in my efforts it required enabling asynchronous probing for both drivers, moving the firmware dts node to the end of the dtsi file, as well as forcing a long delay in the qcom_scm_probe function.
With those tweaks we ran into the following crash:
[ 2.631040] arm-smmu 15000000.iommu: Stage-1: 48-bit VA -> 48-bit IPA [ 2.633372] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000 ... [ 2.633402] [0000000000000000] user address but active_mm is swapper [ 2.633409] Internal error: Oops: 96000005 [#1] PREEMPT SMP [ 2.633415] Modules linked in: [ 2.633427] CPU: 5 PID: 117 Comm: kworker/u16:2 Tainted: G W 5.10.0-rc1-mainline-00025-g272a618fc36-dirty #3971 [ 2.633430] Hardware name: Thundercomm Dragonboard 845c (DT) [ 2.633448] Workqueue: events_unbound async_run_entry_fn [ 2.633456] pstate: 80c00005 (Nzcv daif +PAN +UAO -TCO BTYPE=--) [ 2.633465] pc : qcom_scm_qsmmu500_wait_safe_toggle+0x78/0xb0 [ 2.633473] lr : qcom_smmu500_reset+0x58/0x78 [ 2.633476] sp : ffffffc0105a3b60 ... [ 2.633567] Call trace: [ 2.633572] qcom_scm_qsmmu500_wait_safe_toggle+0x78/0xb0 [ 2.633576] qcom_smmu500_reset+0x58/0x78 [ 2.633581] arm_smmu_device_reset+0x194/0x270 [ 2.633585] arm_smmu_device_probe+0xc94/0xeb8 [ 2.633592] platform_drv_probe+0x58/0xa8 [ 2.633597] really_probe+0xec/0x398 [ 2.633601] driver_probe_device+0x5c/0xb8 [ 2.633606] __driver_attach_async_helper+0x64/0x88 [ 2.633610] async_run_entry_fn+0x4c/0x118 [ 2.633617] process_one_work+0x20c/0x4b0 [ 2.633621] worker_thread+0x48/0x460 [ 2.633628] kthread+0x14c/0x158 [ 2.633634] ret_from_fork+0x10/0x18 [ 2.633642] Code: a9034fa0 d0007f73 29107fa0 91342273 (f9400020)
To avoid this, this patch adds a check on qcom_scm_is_available() in the qcom_smmu_impl_init() function, returning -EPROBE_DEFER if its not ready.
This allows the driver to try to probe again later after qcom_scm has finished probing.
--- drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c +4 -0
@@ -159,6 +159,10 @@ struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu) {struct qcom_smmu *qsmmu; +/* Check to make sure qcom_scm has finished probing */ +if (!qcom_scm_is_available ()) +return ERR_PTR (-EPROBE_DEFER ); + qsmmu =devm_kzalloc (smmu->dev,sizeof (*qsmmu),GFP_KERNEL );if (!qsmmu)return ERR_PTR (-ENOMEM );
Signature · 1 line
2.17.1
PATCH RESEND 2/2 iommu: Avoid crash if iommu_group is null
In trying to handle a possible driver probe ordering issue brought up by Robin Murphy, I ran across a separate null pointer crash in the iommu core in iommu_group_remove_device():
[ 2.732803] dwc3-qcom a6f8800.usb: failed to get usb-ddr path: -517 [ 2.739281] Unable to handle kernel NULL pointer dereference at virtual address 00000000000000c0 ... [ 2.775619] [00000000000000c0] user address but active_mm is swapper [ 2.782039] Internal error: Oops: 96000005 [#1] PREEMPT SMP [ 2.787670] Modules linked in: [ 2.790769] CPU: 6 PID: 1 Comm: swapper/0 Tainted: G W 5.10.0-rc1-mainline-00025-g272a618fc36-dirty #3973 [ 2.801719] Hardware name: Thundercomm Dragonboard 845c (DT) [ 2.807431] pstate: 00c00005 (nzcv daif +PAN +UAO -TCO BTYPE=--) [ 2.813508] pc : iommu_group_remove_device+0x30/0x1b0 [ 2.818611] lr : iommu_release_device+0x4c/0x78 [ 2.823189] sp : ffffffc01005b950 ... [ 2.907082] Call trace: [ 2.909566] iommu_group_remove_device+0x30/0x1b0 [ 2.914323] iommu_release_device+0x4c/0x78 [ 2.918559] iommu_bus_notifier+0xe8/0x108 [ 2.922708] blocking_notifier_call_chain+0x78/0xb8 [ 2.927641] device_del+0x2ac/0x3d0 [ 2.931177] platform_device_del.part.9+0x20/0x98 [ 2.935933] platform_device_unregister+0x2c/0x40 [ 2.940694] of_platform_device_destroy+0xd8/0xe0 [ 2.945450] device_for_each_child_reverse+0x58/0xb0 [ 2.950471] of_platform_depopulate+0x4c/0x78 [ 2.954886] dwc3_qcom_probe+0x93c/0xcb8 [ 2.958858] platform_drv_probe+0x58/0xa8 [ 2.962917] really_probe+0xec/0x398 [ 2.966531] driver_probe_device+0x5c/0xb8 [ 2.970677] device_driver_attach+0x74/0x98 [ 2.974911] __driver_attach+0x60/0xe8 [ 2.978700] bus_for_each_dev+0x84/0xd8 [ 2.982581] driver_attach+0x30/0x40 [ 2.986194] bus_add_driver+0x160/0x208 [ 2.990076] driver_register+0x64/0x110 [ 2.993957] __platform_driver_register+0x58/0x68 [ 2.998716] dwc3_qcom_driver_init+0x20/0x28 [ 3.003041] do_one_initcall+0x6c/0x2d0 [ 3.006925] kernel_init_freeable+0x214/0x268 [ 3.011339] kernel_init+0x18/0x118 [ 3.014876] ret_from_fork+0x10/0x18 [ 3.018495] Code: d0006a21 f9417295 91130021 910162b6 (b940c2a2)
In the case above, the arm-smmu driver fails to probe with EPROBE_DEFER, and I'm guessing I'm guessing that causes iommu_group_add_device() to fail and sets the dev->iommu_group = NULL, then somehow we hit iommu_group_remove_device() and trip over the null value? I'm not really sure...
Anyway, adding the null check seems to avoid the issue and the system boots fine after the arm-smmu driver later reprobed.
Feedback or better ideas for a solution would be appreciated!
--- drivers/iommu/iommu.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/iommu/iommu.c +4 -0
@@ -877,6 +877,10 @@ void iommu_group_remove_device(struct device *dev)struct iommu_group *group = dev->iommu_group;struct group_device *tmp_device, *device =NULL ; +/* Avoid crash if iommu_group value is null */ +if (!group) +return ; +dev_info (dev,"Removing from iommu group %d\n" , group->id);/* Pre-notify listeners that a device is being removed. */
Signature · 1 line
2.17.1
Robin Murphy
reviewerRe: [RESEND][PATCH 2/2] iommu: Avoid crash if iommu_group is null
Quoted 52 lines · John Stultz
In trying to handle a possible driver probe ordering issue brought up by Robin Murphy, I ran across a separate null pointer crash in the iommu core in
iommu_group_remove_device():[ 2.732803] dwc3-qcom a6f8800.usb: failed to get usb-ddr path: -517 [ 2.739281] Unable to handle kernel NULL pointer dereference at virtual address 00000000000000c0 ... [ 2.775619] [00000000000000c0] user address but active_mm is swapper [ 2.782039] Internal error: Oops: 96000005 [#1] PREEMPT SMP [ 2.787670] Modules linked in: [ 2.790769] CPU: 6 PID: 1 Comm: swapper/0 Tainted: G W 5.10.0-rc1-mainline-00025-g272a618fc36-dirty #3973 [ 2.801719] Hardware name: Thundercomm Dragonboard 845c (DT) [ 2.807431] pstate: 00c00005 (nzcv daif +PAN +UAO -TCO BTYPE=--) [ 2.813508] pc : iommu_group_remove_device+0x30/0x1b0 [ 2.818611] lr : iommu_release_device+0x4c/0x78 [ 2.823189] sp : ffffffc01005b950 ... [ 2.907082] Call trace: [ 2.909566] iommu_group_remove_device+0x30/0x1b0 [ 2.914323] iommu_release_device+0x4c/0x78 [ 2.918559] iommu_bus_notifier+0xe8/0x108 [ 2.922708] blocking_notifier_call_chain+0x78/0xb8 [ 2.927641] device_del+0x2ac/0x3d0 [ 2.931177] platform_device_del.part.9+0x20/0x98 [ 2.935933] platform_device_unregister+0x2c/0x40 [ 2.940694] of_platform_device_destroy+0xd8/0xe0 [ 2.945450] device_for_each_child_reverse+0x58/0xb0 [ 2.950471] of_platform_depopulate+0x4c/0x78 [ 2.954886] dwc3_qcom_probe+0x93c/0xcb8 [ 2.958858] platform_drv_probe+0x58/0xa8 [ 2.962917] really_probe+0xec/0x398 [ 2.966531] driver_probe_device+0x5c/0xb8 [ 2.970677] device_driver_attach+0x74/0x98 [ 2.974911] __driver_attach+0x60/0xe8 [ 2.978700] bus_for_each_dev+0x84/0xd8 [ 2.982581] driver_attach+0x30/0x40 [ 2.986194] bus_add_driver+0x160/0x208 [ 2.990076] driver_register+0x64/0x110 [ 2.993957] __platform_driver_register+0x58/0x68 [ 2.998716] dwc3_qcom_driver_init+0x20/0x28 [ 3.003041] do_one_initcall+0x6c/0x2d0 [ 3.006925] kernel_init_freeable+0x214/0x268 [ 3.011339] kernel_init+0x18/0x118 [ 3.014876] ret_from_fork+0x10/0x18 [ 3.018495] Code: d0006a21 f9417295 91130021 910162b6 (b940c2a2)In the case above, the arm-smmu driver fails to probe with
EPROBE_DEFER, and I'm guessing I'm guessing that causesiommu_group_add_device()to fail and sets thedev->iommu_group= NULL, then somehow we hitiommu_group_remove_device()and trip over the null value? I'm not really sure...
Quoted 4 lines
Anyway, adding the null check seems to avoid the issue and the system boots fine after the arm-smmu driver later reprobed.
Feedback or better ideas for a solution would be appreciated!
I think the real issue is that we shouldn't even be here. If the device was still waiting for an IOMMU when it got torn down, then dev->iommu would be non-NULL due to a fwspec being allocated, but it's possible that none of the other setup would have been done, so iommu_release_device() shouldn't be blindly trying to undo it. It happens that iommu_device_unlink() and arm_smmu_release_device() still have some left-over tolerance for invalid inputs, but even if we do make iommu_group_remove_device() similarly tolerant of invalid input, the subsequent module_put() could still be unbalanced if the fwspec and deferral came from of_iommu_configure() without iommu_probe_device() even being called. Therefore I think this needs to be fundamentally fixed by making iommu_release_device() better at handling partially-configured devices in the first place.
Robin.
Quoted 31 lines
Cc Robin Murphy <[email protected]>Cc Will Deacon <[email protected]>Cc Andy Gross <[email protected]>Cc Maulik Shah <[email protected]>Cc Bjorn Andersson <[email protected]>Cc Saravana Kannan <[email protected]>Cc Marc Zyngier <[email protected]>Cc Lina Iyer <[email protected]>Cc linux-arm-msm <[email protected]>Signed-off-by John Stultz <[email protected]>--- drivers/iommu/iommu.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/iommu/iommu.c +4
-0@@ -877,6 +877,10 @@ void iommu_group_remove_device(struct device *dev)struct iommu_group *group = dev->iommu_group;struct group_device *tmp_device, *device =NULL ; +/* Avoid crash if iommu_group value is null */ +if (!group) +return ; +dev_info (dev,"Removing from iommu group %d\n" , group->id);/* Pre-notify listeners that a device is being removed. */
Quoted 40 lines · John Stultz
Robin Murphy pointed out that if the arm-smmu driver probes before the
qcom_scmdriver, we may callqcom_scm_qsmmu500_wait_safe_toggle()before the__scmis initialized.Now, getting this to happen is a bit contrived, as in my efforts it required enabling asynchronous probing for both drivers, moving the firmware dts node to the end of the dtsi file, as well as forcing a long delay in the
qcom_scm_probefunction.With those tweaks we ran into the following crash:
[ 2.631040] arm-smmu 15000000.iommu: Stage-1: 48-bit VA -> 48-bit IPA [ 2.633372] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000 ... [ 2.633402] [0000000000000000] user address but active_mm is swapper [ 2.633409] Internal error: Oops: 96000005 [#1] PREEMPT SMP [ 2.633415] Modules linked in: [ 2.633427] CPU: 5 PID: 117 Comm: kworker/u16:2 Tainted: G W 5.10.0-rc1-mainline-00025-g272a618fc36-dirty #3971 [ 2.633430] Hardware name: Thundercomm Dragonboard 845c (DT) [ 2.633448] Workqueue: events_unbound async_run_entry_fn [ 2.633456] pstate: 80c00005 (Nzcv daif +PAN +UAO -TCO BTYPE=--) [ 2.633465] pc : qcom_scm_qsmmu500_wait_safe_toggle+0x78/0xb0 [ 2.633473] lr : qcom_smmu500_reset+0x58/0x78 [ 2.633476] sp : ffffffc0105a3b60 ... [ 2.633567] Call trace: [ 2.633572] qcom_scm_qsmmu500_wait_safe_toggle+0x78/0xb0 [ 2.633576] qcom_smmu500_reset+0x58/0x78 [ 2.633581] arm_smmu_device_reset+0x194/0x270 [ 2.633585] arm_smmu_device_probe+0xc94/0xeb8 [ 2.633592] platform_drv_probe+0x58/0xa8 [ 2.633597] really_probe+0xec/0x398 [ 2.633601] driver_probe_device+0x5c/0xb8 [ 2.633606] __driver_attach_async_helper+0x64/0x88 [ 2.633610] async_run_entry_fn+0x4c/0x118 [ 2.633617] process_one_work+0x20c/0x4b0 [ 2.633621] worker_thread+0x48/0x460 [ 2.633628] kthread+0x14c/0x158 [ 2.633634] ret_from_fork+0x10/0x18 [ 2.633642] Code: a9034fa0 d0007f73 29107fa0 91342273 (f9400020)
Quoted 6 lines
To avoid this, this patch adds a check on
qcom_scm_is_available()in theqcom_smmu_impl_init()function, returning -EPROBE_DEFER if its not ready.This allows the driver to try to probe again later after
qcom_scmhas finished probing.
As well as strict correctness, the other motivation here was that qcom_scm_is_available() should be the place to add the "is the module even loaded yet?" dance to iron out the awkward config dependencies in future.
Quoted 32 lines
Cc Robin Murphy <[email protected]>Cc Will Deacon <[email protected]>Cc Andy Gross <[email protected]>Cc Maulik Shah <[email protected]>Cc Bjorn Andersson <[email protected]>Cc Saravana Kannan <[email protected]>Cc Marc Zyngier <[email protected]>Cc Lina Iyer <[email protected]>Cc linux-arm-msm <[email protected]>Reported-by Robin Murphy <[email protected]>Signed-off-by John Stultz <[email protected]>--- drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c +4
-0@@ -159,6 +159,10 @@ struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu) {struct qcom_smmu *qsmmu; +/* Check to make sure qcom_scm has finished probing */ +if (!qcom_scm_is_available ()) +return ERR_PTR (-EPROBE_DEFER ); + qsmmu =devm_kzalloc (smmu->dev,sizeof (*qsmmu),GFP_KERNEL );if (!qsmmu)return ERR_PTR (-ENOMEM );
Will Deacon
maintainerQuoted 10 lines · John Stultz
Robin Murphy pointed out that if the arm-smmu driver probes before the
qcom_scmdriver, we may callqcom_scm_qsmmu500_wait_safe_toggle()before the__scmis initialized.Now, getting this to happen is a bit contrived, as in my efforts it required enabling asynchronous probing for both drivers, moving the firmware dts node to the end of the dtsi file, as well as forcing a long delay in the
qcom_scm_probefunction.[...]
Applied only the first patch to arm64 (for-next/iommu/fixes), thanks!
[1/2] arm-smmu-qcom: Ensure the qcom_scm driver has finished probing
https://git.kernel.org/arm64/c/72b55c96f3a5
Cheers,
Signature · 9 lines
Will https://fixes.arm64.dev https://next.arm64.dev https://will.arm64.dev _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu