slimbus: qcom-ngd-ctrl: Fix some race conditions and deadlocks
Subsystems (4)
v2 cover slimbus: qcom-ngd-ctrl: Fix some race conditions and deadlocks
When the qcom-ngd-ctrl driver is probed after the ADSP remoteproc, the SSR notifier will fire immediately, which results in qcom_slim_ngd_ssr_pdr_notify() attempting to schedule_work() on an unitialized work_struct.
The concrete result of this is that my db845c/RB3 now fails to boot 100% of the time.
In reviewing the problematic code, a few other problems where discovered, such that platform_driver_unregister() is used to unregister the child device.
Lastly, with the db845c booting, it was determined that attempting to stop the ADSP remoteproc causes the slimbus driver to deadlock.
Note that while this solves the problems described above, and unblock boot as well as restart of the remoteproc, this stack needs more love.
Upon tearing down the slimbus controller (when the ADSP goes down), the slimbus devices attempts to access their slimbus devices - which is prevented by the controller being runtime suspended. This results in a wall of errors in the log, about failing transactions.
---
Changes in v2:
- Left devm_request_irq() in its old place, added IRQF_NO_AUTOEN
instead.
- Changed order of platform_driver_register, to avoid the possible issue
that controller probe completes without the NGD being present.
- Cleaned up commit message of the "slimbus: qcom-ngd-ctrl: Correct PDR
and SSR cleanup ownership" patch.
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Bjorn Andersson (7):
slimbus: qcom-ngd-ctrl: Fix up platform_driver registration
slimbus: qcom-ngd-ctrl: Fix probe error path ordering
slimbus: qcom-ngd-ctrl: Correct PDR and SSR cleanup ownership
slimbus: qcom-ngd-ctrl: Register callbacks after creating the ngd
slimbus: qcom-ngd-ctrl: Initialize controller resources in controller
slimbus: qcom-ngd-ctrl: Balance pm_runtime enablement for NGD
slimbus: qcom-ngd-ctrl: Avoid ABBA on tx_lock/ctrl->lock
drivers/slimbus/qcom-ngd-ctrl.c | 120 +++++++++++++++++++++++++--------------- 1 file changed, 76 insertions(+), 44 deletions(-)
---
base-commit: 36ece9697e89016181e5ae87510e40fb31d86f2b
change-id: 20260211-slim-ngd-dev-74166f29f035
Best regards,
Signature · 1 line
Bjorn Andersson <[email protected]>
PATCH v2 1/7 slimbus: qcom-ngd-ctrl: Fix up platform_driver registration
Device drivers should not invoke platform_driver_register()/unregister() in their probe and remove paths. They should further not rely on platform_driver_unregister() as their only means of "deleting" their child devices.
Introduce a helper to unregister the child device and move the platform_driver_register()/unregister() to module_init()/exit().
--- drivers/slimbus/qcom-ngd-ctrl.c | 36 +++++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-)
drivers/slimbus/qcom-ngd-ctrl.c +33 -3
@@ -1560,6 +1560,13 @@ static int of_qcom_slim_ngd_register(struct device *parent,return -ENODEV ; } +static void qcom_slim_ngd_unregister (struct qcom_slim_ngd_ctrl *ctrl) +{ +struct qcom_slim_ngd *ngd = ctrl->ngd; + +platform_device_del (ngd->pdev); +} +static int qcom_slim_ngd_probe (struct platform_device *pdev) {struct device *dev = &pdev->dev;
@@ -1662,7 +1669,6 @@ static int qcom_slim_ngd_ctrl_probe(struct platform_device *pdev) goto err_pdr_lookup; } -platform_driver_register (&qcom_slim_ngd_driver);return of_qcom_slim_ngd_register (dev, ctrl); err_pdr_alloc:
@@ -1676,7 +1682,9 @@ static int qcom_slim_ngd_ctrl_probe(struct platform_device *pdev)static void qcom_slim_ngd_ctrl_remove (struct platform_device *pdev) { -platform_driver_unregister (&qcom_slim_ngd_driver); +struct qcom_slim_ngd_ctrl *ctrl =platform_get_drvdata (pdev); + +qcom_slim_ngd_unregister (ctrl); }static void qcom_slim_ngd_remove (struct platform_device *pdev)
@@ -1752,6 +1760,28 @@ static struct platform_driver qcom_slim_ngd_driver = { }, }; -module_platform_driver (qcom_slim_ngd_ctrl_driver); +static int qcom_slim_ngd_init (void ) +{ +int ret; + + ret =platform_driver_register (&qcom_slim_ngd_driver); +if (ret) +return ret; + + ret =platform_driver_register (&qcom_slim_ngd_ctrl_driver); +if (ret) +platform_driver_unregister (&qcom_slim_ngd_driver); + +return ret; +} + +static void qcom_slim_ngd_exit (void ) +{ +platform_driver_unregister (&qcom_slim_ngd_ctrl_driver); +platform_driver_unregister (&qcom_slim_ngd_driver); +} + +module_init (qcom_slim_ngd_init); +module_exit (qcom_slim_ngd_exit);MODULE_LICENSE ("GPL v2" );MODULE_DESCRIPTION ("Qualcomm SLIMBus NGD controller" );
Signature · 1 line
2.51.0
PATCH v2 2/7 slimbus: qcom-ngd-ctrl: Fix probe error path ordering
In reply to Bjorn Anderssonqcom_slim_ngd_ctrl_probe() first registers the SSR callback then allocates the PDR context, as such the error path needs to come in opposite order to allow us to unroll each step.
--- drivers/slimbus/qcom-ngd-ctrl.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
drivers/slimbus/qcom-ngd-ctrl.c +5 -6
@@ -1660,22 +1660,21 @@ static int qcom_slim_ngd_ctrl_probe(struct platform_device *pdev)if (IS_ERR (ctrl->pdr)) { ret =dev_err_probe (dev,PTR_ERR (ctrl->pdr),"Failed to init PDR handle\n" ); - goto err_pdr_alloc; + goto err_unregister_ssr; } pds =pdr_add_lookup (ctrl->pdr,"avs/audio" ,"msm/adsp/audio_pd" );if (IS_ERR (pds) &&PTR_ERR (pds) != -EALREADY ) { ret =dev_err_probe (dev,PTR_ERR (pds),"pdr add lookup failed\n" ); - goto err_pdr_lookup; + goto err_pdr_release; }return of_qcom_slim_ngd_register (dev, ctrl); -err_pdr_alloc: -qcom_unregister_ssr_notifier (ctrl->notifier, &ctrl->nb); - -err_pdr_lookup: +err_pdr_release:pdr_handle_release (ctrl->pdr); +err_unregister_ssr: +qcom_unregister_ssr_notifier (ctrl->notifier, &ctrl->nb);return ret; }
Signature · 1 line
2.51.0
PATCH v2 3/7 slimbus: qcom-ngd-ctrl: Correct PDR and SSR cleanup ownership
In reply to Bjorn AnderssonPDR and SSR callbacks are registred from the controller probe function, but currently released from the child device's remove function.
The remove() function should only be unwinding what was done in the same device's probe() function.
--- drivers/slimbus/qcom-ngd-ctrl.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/slimbus/qcom-ngd-ctrl.c +3 -2
@@ -1683,6 +1683,9 @@ static void qcom_slim_ngd_ctrl_remove(struct platform_device *pdev) {struct qcom_slim_ngd_ctrl *ctrl =platform_get_drvdata (pdev); +pdr_handle_release (ctrl->pdr); +qcom_unregister_ssr_notifier (ctrl->notifier, &ctrl->nb); +qcom_slim_ngd_unregister (ctrl); }
@@ -1691,8 +1694,6 @@ static void qcom_slim_ngd_remove(struct platform_device *pdev)struct qcom_slim_ngd_ctrl *ctrl =platform_get_drvdata (pdev);pm_runtime_disable (&pdev->dev); -pdr_handle_release (ctrl->pdr); -qcom_unregister_ssr_notifier (ctrl->notifier, &ctrl->nb);qcom_slim_ngd_enable (ctrl, false);qcom_slim_ngd_exit_dma (ctrl);qcom_slim_ngd_qmi_svc_event_deinit (&ctrl->qmi);
Signature · 1 line
2.51.0
PATCH v2 4/7 slimbus: qcom-ngd-ctrl: Register callbacks after creating the ngd
In reply to Bjorn AnderssonWhen the remoteproc starts in parallel with the NGD driver being probed, or the remoteproc is already up when the PDR lookup is being registered, or in the theoretical event that we get an interrupt from the hardware, these callbacks will operate on uninitialized data. This result in issues to boot the affected boards.
One such example can be seen in the following fault, where qcom_slim_ngd_ssr_pdr_notify() schedules work on the NULL ngd_up_work.
[ 21.858578] ------------[ cut here ]------------ [ 21.858745] WARNING: kernel/workqueue.c:2338 at __queue_work+0x5e0/0x790, CPU#2: kworker/2:2/116 ... [ 21.859251] Call trace: [ 21.859255] __queue_work+0x5e0/0x790 (P) [ 21.859265] queue_work_on+0x6c/0xf0 [ 21.859273] qcom_slim_ngd_ssr_pdr_notify+0x110/0x150 [slim_qcom_ngd_ctrl] [ 21.859304] qcom_slim_ngd_ssr_notify+0x24/0x40 [slim_qcom_ngd_ctrl] [ 21.859318] notifier_call_chain+0xa4/0x230 [ 21.859329] srcu_notifier_call_chain+0x64/0xb8 [ 21.859338] ssr_notify_start+0x40/0x78 [qcom_common] [ 21.859355] rproc_start+0x130/0x230 [ 21.859367] rproc_boot+0x3d4/0x518
...
Move the enablement of interrupts, and the registration of SSR and PDR until after the NGD device has been registered.
This could be further refined by moving initialization to the control driver probe and by removing the platform driver model from the picture.
--- drivers/slimbus/qcom-ngd-ctrl.c | 45 ++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 19 deletions(-)
drivers/slimbus/qcom-ngd-ctrl.c +26 -19
@@ -1609,6 +1609,7 @@ static int qcom_slim_ngd_ctrl_probe(struct platform_device *pdev) {struct device *dev = &pdev->dev;struct qcom_slim_ngd_ctrl *ctrl; +int irq;int ret;struct pdr_service *pds;
@@ -1622,20 +1623,16 @@ static int qcom_slim_ngd_ctrl_probe(struct platform_device *pdev)if (IS_ERR (ctrl->base))return PTR_ERR (ctrl->base); - ret =platform_get_irq (pdev,0 ); -if (ret <0 ) -return ret; + irq =platform_get_irq (pdev,0 ); +if (irq <0 ) +return irq; - ret =devm_request_irq (dev, ret, qcom_slim_ngd_interrupt, -IRQF_TRIGGER_HIGH ,"slim-ngd" , ctrl); + ret =devm_request_irq (dev, irq, qcom_slim_ngd_interrupt, +IRQF_TRIGGER_HIGH |IRQF_NO_AUTOEN , +"slim-ngd" , ctrl);if (ret)return dev_err_probe (&pdev->dev, ret,"request IRQ failed\n" ); - ctrl->nb.notifier_call = qcom_slim_ngd_ssr_notify; - ctrl->notifier =qcom_register_ssr_notifier ("lpass" , &ctrl->nb); -if (IS_ERR (ctrl->notifier)) -return PTR_ERR (ctrl->notifier); - ctrl->dev = dev; ctrl->framer.rootfreq =SLIM_ROOT_FREQ >>3 ; ctrl->framer.superfreq =
@@ -1657,24 +1654,34 @@ static int qcom_slim_ngd_ctrl_probe(struct platform_device *pdev)init_completion (&ctrl->qmi_up); ctrl->pdr =pdr_handle_alloc (slim_pd_status, ctrl); -if (IS_ERR (ctrl->pdr)) { - ret =dev_err_probe (dev,PTR_ERR (ctrl->pdr), -"Failed to init PDR handle\n" ); - goto err_unregister_ssr; - } +if (IS_ERR (ctrl->pdr)) +return dev_err_probe (dev,PTR_ERR (ctrl->pdr),"Failed to init PDR handle\n" ); + + ret =of_qcom_slim_ngd_register (dev, ctrl); +if (ret) + goto err_pdr_release; pds =pdr_add_lookup (ctrl->pdr,"avs/audio" ,"msm/adsp/audio_pd" );if (IS_ERR (pds) &&PTR_ERR (pds) != -EALREADY ) { ret =dev_err_probe (dev,PTR_ERR (pds),"pdr add lookup failed\n" ); - goto err_pdr_release; + goto err_unregister_ngd; + } + + ctrl->nb.notifier_call = qcom_slim_ngd_ssr_notify; + ctrl->notifier =qcom_register_ssr_notifier ("lpass" , &ctrl->nb); +if (IS_ERR (ctrl->notifier)) { + ret =PTR_ERR (ctrl->notifier); + goto err_unregister_ngd; } -return of_qcom_slim_ngd_register (dev, ctrl); +enable_irq (irq); +return 0 ; + +err_unregister_ngd: +qcom_slim_ngd_unregister (ctrl); err_pdr_release:pdr_handle_release (ctrl->pdr); -err_unregister_ssr: -qcom_unregister_ssr_notifier (ctrl->notifier, &ctrl->nb);return ret; }
Signature · 1 line
2.51.0
PATCH v2 5/7 slimbus: qcom-ngd-ctrl: Initialize controller resources in controller
In reply to Bjorn AnderssonThe work structs and work queue are controller resources, create and destroy them in the controller context. Creating them as part of the child device's probe path seems to be okay now that the controller's probe has been updated, but if for some reason the child does not probe successfully a SSR or PDR notification will schedule_work() on an uninitialized "ngd_up_work".
Move the initialization of these controller resources to the controller probe function to avoid any issues, and to clarify the ownership.
--- drivers/slimbus/qcom-ngd-ctrl.c | 38 ++++++++++++++++---------------------- 1 file changed, 16 insertions(+), 22 deletions(-)
drivers/slimbus/qcom-ngd-ctrl.c +16 -22
@@ -1582,25 +1582,8 @@ static int qcom_slim_ngd_probe(struct platform_device *pdev)pm_runtime_enable (dev);pm_runtime_get_noresume (dev); ret =qcom_slim_ngd_qmi_svc_event_init (ctrl); -if (ret) { +if (ret)dev_err (&pdev->dev,"QMI service registration failed:%d" , ret); -return ret; - } - -INIT_WORK (&ctrl->m_work, qcom_slim_ngd_master_worker); -INIT_WORK (&ctrl->ngd_up_work, qcom_slim_ngd_up_worker); - ctrl->mwq =create_singlethread_workqueue ("ngd_master" ); -if (!ctrl->mwq) { -dev_err (&pdev->dev,"Failed to start master worker\n" ); - ret = -ENOMEM ; - goto wq_err; - } - -return 0 ; -wq_err: -qcom_slim_ngd_qmi_svc_event_deinit (&ctrl->qmi); -if (ctrl->mwq) -destroy_workqueue (ctrl->mwq);return ret; }
@@ -1653,9 +1636,18 @@ static int qcom_slim_ngd_ctrl_probe(struct platform_device *pdev)init_completion (&ctrl->qmi.qmi_comp);init_completion (&ctrl->qmi_up); +INIT_WORK (&ctrl->m_work, qcom_slim_ngd_master_worker); +INIT_WORK (&ctrl->ngd_up_work, qcom_slim_ngd_up_worker); + + ctrl->mwq =create_singlethread_workqueue ("ngd_master" ); +if (!ctrl->mwq) +return dev_err_probe (dev, -ENOMEM ,"Failed to start master worker\n" ); + ctrl->pdr =pdr_handle_alloc (slim_pd_status, ctrl); -if (IS_ERR (ctrl->pdr)) -return dev_err_probe (dev,PTR_ERR (ctrl->pdr),"Failed to init PDR handle\n" ); +if (IS_ERR (ctrl->pdr)) { + ret =dev_err_probe (dev,PTR_ERR (ctrl->pdr),"Failed to init PDR handle\n" ); + goto err_destroy_mwq; + } ret =of_qcom_slim_ngd_register (dev, ctrl);if (ret)
@@ -1682,6 +1674,8 @@ static int qcom_slim_ngd_ctrl_probe(struct platform_device *pdev)qcom_slim_ngd_unregister (ctrl); err_pdr_release:pdr_handle_release (ctrl->pdr); +err_destroy_mwq: +destroy_workqueue (ctrl->mwq);return ret; }
@@ -1694,6 +1688,8 @@ static void qcom_slim_ngd_ctrl_remove(struct platform_device *pdev)qcom_unregister_ssr_notifier (ctrl->notifier, &ctrl->nb);qcom_slim_ngd_unregister (ctrl); + +destroy_workqueue (ctrl->mwq); }static void qcom_slim_ngd_remove (struct platform_device *pdev)
@@ -1704,8 +1700,6 @@ static void qcom_slim_ngd_remove(struct platform_device *pdev)qcom_slim_ngd_enable (ctrl, false);qcom_slim_ngd_exit_dma (ctrl);qcom_slim_ngd_qmi_svc_event_deinit (&ctrl->qmi); -if (ctrl->mwq) -destroy_workqueue (ctrl->mwq);kfree (ctrl->ngd); ctrl->ngd =NULL ;
Signature · 1 line
2.51.0
PATCH v2 6/7 slimbus: qcom-ngd-ctrl: Balance pm_runtime enablement for NGD
In reply to Bjorn AnderssonThe pm_runtime_enable() and pm_runtime_use_autosuspend() calls are supposed to be balanced on exit, add these calls.
--- drivers/slimbus/qcom-ngd-ctrl.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/slimbus/qcom-ngd-ctrl.c +5 -1
@@ -1582,8 +1582,11 @@ static int qcom_slim_ngd_probe(struct platform_device *pdev)pm_runtime_enable (dev);pm_runtime_get_noresume (dev); ret =qcom_slim_ngd_qmi_svc_event_init (ctrl); -if (ret) +if (ret) {dev_err (&pdev->dev,"QMI service registration failed:%d" , ret); +pm_runtime_dont_use_autosuspend (dev); +pm_runtime_disable (dev); + }return ret; }
@@ -1696,6 +1699,7 @@ static void qcom_slim_ngd_remove(struct platform_device *pdev) {struct qcom_slim_ngd_ctrl *ctrl =platform_get_drvdata (pdev); +pm_runtime_dont_use_autosuspend (&pdev->dev);pm_runtime_disable (&pdev->dev);qcom_slim_ngd_enable (ctrl, false);qcom_slim_ngd_exit_dma (ctrl);
Signature · 1 line
2.51.0
PATCH v2 7/7 slimbus: qcom-ngd-ctrl: Avoid ABBA on tx_lock/ctrl->lock
In reply to Bjorn AnderssonDuring the SSR/PDR down notification the tx_lock is taken with the intent to provide synchronization with active DMA transfers.
But during this period qcom_slim_ngd_down() is invoked, which ends up in slim_report_absent(), which takes the slim_controller lock. In multiple other codepaths these two locks are taken in the opposite order (i.e. slim_controller then tx_lock).
The result is a lockdep splat, and a possible deadlock:
rprocctl/449 is trying to acquire lock:
ffff00009793e620 (&ctrl->lock){+.+.}-{4:4}, at: slim_report_absent (drivers/slimbus/core.c:322) slimbus
but task is already holding lock:
ffff00009793fb50 (&ctrl->tx_lock){+.+.}-{4:4}, at: qcom_slim_ngd_ssr_pdr_notify (drivers/slimbus/qcom-ngd-ctrl.c:1475) slim_qcom_ngd_ctrl
which lock already depends on the new lock.
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&ctrl->tx_lock);
lock(&ctrl->lock);
lock(&ctrl->tx_lock); lock(&ctrl->lock);
The assumption is that the comment refers to the desire to not call qcom_slim_ngd_exit_dma() while we have an ongoing DMA TX transaction. But any such transaction is initiated and completed within a single qcom_slim_ngd_xfer_msg().
Prior to calling qcom_slim_ngd_exit_dma() the slim_controller is torn down, all child devices are notified that the slimbus is gone and the child devices are removed.
Stop taking the tx_lock in qcom_slim_ngd_ssr_pdr_notify() to avoid the deadlock.
--- drivers/slimbus/qcom-ngd-ctrl.c | 3 --- 1 file changed, 3 deletions(-)
drivers/slimbus/qcom-ngd-ctrl.c +0 -3
@@ -1471,15 +1471,12 @@ static int qcom_slim_ngd_ssr_pdr_notify(struct qcom_slim_ngd_ctrl *ctrl,switch (action) {case QCOM_SSR_BEFORE_SHUTDOWN :case SERVREG_SERVICE_STATE_DOWN : -/* Make sure the last dma xfer is finished */ -mutex_lock (&ctrl->tx_lock);if (ctrl->state !=QCOM_SLIM_NGD_CTRL_DOWN ) {pm_runtime_get_noresume (ctrl->ctrl.dev); ctrl->state =QCOM_SLIM_NGD_CTRL_DOWN ;qcom_slim_ngd_down (ctrl);qcom_slim_ngd_exit_dma (ctrl); } -mutex_unlock (&ctrl->tx_lock);break ;case QCOM_SSR_AFTER_POWERUP :case SERVREG_SERVICE_STATE_UP :
Signature · 1 line
2.51.0
Re: (subset) [PATCH v2 0/7] slimbus: qcom-ngd-ctrl: Fix some race conditions and deadlocks
In reply to Bjorn AnderssonQuoted 9 lines · Bjorn Andersson
When the qcom-ngd-ctrl driver is probed after the ADSP remoteproc, the SSR notifier will fire immediately, which results in
qcom_slim_ngd_ssr_pdr_notify()attempting toschedule_work()on an unitializedwork_struct.The concrete result of this is that my db845c/RB3 now fails to boot 100% of the time.
[...]
Applied, thanks!
[1/7] slimbus: qcom-ngd-ctrl: Fix up platform_driver registration
commit: 9c1a88eb69355a7baaa6d0001538f200e3de5548
[2/7] slimbus: qcom-ngd-ctrl: Fix probe error path ordering
commit: a0b85655673d5edb678cae5faa7488c7d2adc89d
[3/7] slimbus: qcom-ngd-ctrl: Correct PDR and SSR cleanup ownership
commit: c0e4faaace5f2ce54b32d08494126e828cfd4ee6Best regards,
Signature · 1 line
Srinivas Kandagatla <[email protected]>
Quoted 9 lines · Bjorn Andersson
When the qcom-ngd-ctrl driver is probed after the ADSP remoteproc, the SSR notifier will fire immediately, which results in
qcom_slim_ngd_ssr_pdr_notify()attempting toschedule_work()on an unitializedwork_struct.The concrete result of this is that my db845c/RB3 now fails to boot 100% of the time.
[...]
Applied, thanks!
[1/7] slimbus: qcom-ngd-ctrl: Fix up platform_driver registration
commit: 77330fa612f4aea1c3be1f9e7b3c76f2d9697741
[2/7] slimbus: qcom-ngd-ctrl: Fix probe error path ordering
commit: 0dbd05bfc81c9cf649bff11606d373a0174c2f9d
[3/7] slimbus: qcom-ngd-ctrl: Correct PDR and SSR cleanup ownership
commit: c395d822e6019e86f029766d2b47c374dfa22beb
[4/7] slimbus: qcom-ngd-ctrl: Register callbacks after creating the ngd
commit: f1de562a166a68fe62466684f9ca8bec6d574733
[5/7] slimbus: qcom-ngd-ctrl: Initialize controller resources in controller
commit: bebe9de4d2c9a1a953b46ddad2702c32d2feeff9
[6/7] slimbus: qcom-ngd-ctrl: Balance pm_runtime enablement for NGD
commit: b997618e05f468bae2e4fadbd938a6b421c4b335
[7/7] slimbus: qcom-ngd-ctrl: Avoid ABBA on tx_lock/ctrl->lock
commit: 7aa41590ac55284cdf46ffb64ec9587a4d450fe0Best regards,
Signature · 1 line
Srinivas Kandagatla <[email protected]>