PCI/pwrctl: fixes for v6.11
v3 cover PCI/pwrctl: fixes for v6.11
From: Bartosz Golaszewski <[email protected]>
Bjorn,
Here's a respin of the PCI/pwrctl fixes that should go into v6.11. I eventually found a solution that doesn't require Krishna's topology change but Krishna: please make sure to update the code in drivers/pci/remove.c as well when rebasing your work once this gets upstream.
v2 -> v3:
- use the correct device for unregistering the platform pwrctl device in patch 1/2
- make patch 1/2 easier to read by using device_for_each_child()
v1 -> v2:
- use the scoped variant of for_each_child_of_node() to fix a memory leak in patch 1/2
Bartosz Golaszewski (2):
PCI: don't rely on of_platform_depopulate() for reused OF-nodes PCI/pwrctl: put the bus rescan on a different thread
drivers/pci/pwrctl/core.c | 26 +++++++++++++++++++++++--- drivers/pci/pwrctl/pci-pwrctl-pwrseq.c | 2 +- drivers/pci/remove.c | 18 +++++++++++++++++- include/linux/pci-pwrctl.h | 3 +++ 4 files changed, 44 insertions(+), 5 deletions(-)
Signature · 1 line
2.43.0
PATCH v3 1/2 PCI: don't rely on of_platform_depopulate() for reused OF-nodes
From: Bartosz Golaszewski <[email protected]>
of_platform_depopulate() doesn't play nice with reused OF nodes - it ignores the ones that are not marked explicitly as populated and it may happen that the PCI device goes away before the platform device in which case the PCI core clears the OF_POPULATED bit. We need to unconditionally unregister the platform devices for child nodes when stopping the PCI device.
--- drivers/pci/remove.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
drivers/pci/remove.c +17 -1
@@ -1,7 +1,10 @@// SPDX-License-Identifier: GPL-2.0 #include <linux/pci.h> #include <linux/module.h> +#include <linux/of.h> #include <linux/of_platform.h> +#include <linux/platform_device.h> +#include "pci.h" static void pci_free_resources (struct pci_dev *dev)
@@ -14,12 +17,25 @@ static void pci_free_resources(struct pci_dev *dev) } } +static int pci_pwrctl_unregister (struct device *dev,void *data) +{ +struct device_node *pci_node = data, *plat_node =dev_of_node (dev); + +if (dev_is_platform (dev) && plat_node && plat_node == pci_node) { +of_device_unregister (to_platform_device (dev)); +of_node_clear_flag (plat_node,OF_POPULATED ); + } + +return 0 ; +} +static void pci_stop_dev (struct pci_dev *dev) {pci_pme_active (dev, false);if (pci_dev_is_added (dev)) { -of_platform_depopulate (&dev->dev); +device_for_each_child (dev->dev.parent,dev_of_node (&dev->dev), + pci_pwrctl_unregister);device_release_driver (&dev->dev);pci_proc_detach_device (dev);pci_remove_sysfs_dev_files (dev);
Signature · 1 line
2.43.0
PATCH v3 2/2 PCI/pwrctl: put the bus rescan on a different thread
In reply to Bartosz GolaszewskiFrom: Bartosz Golaszewski <[email protected]>
If we trigger the bus rescan from sysfs, we'll try to lock the PCI rescan mutex recursively and deadlock - the platform device will be populated and probed on the same thread that handles the sysfs write.
Add a workqueue to the pwrctl code on which we schedule the rescan for controlled PCI devices. While at it: add a new interface for initializing the pwrctl context where we'd now assign the parent device address and initialize the workqueue.
--- drivers/pci/pwrctl/core.c | 26 +++++++++++++++++++++++--- drivers/pci/pwrctl/pci-pwrctl-pwrseq.c | 2 +- include/linux/pci-pwrctl.h | 3 +++ 3 files changed, 27 insertions(+), 4 deletions(-)
drivers/pci/pwrctl/core.c +23 -3
@@ -48,6 +48,28 @@ static int pci_pwrctl_notify(struct notifier_block *nb, unsigned long action,return NOTIFY_DONE ; } +static void rescan_work_func (struct work_struct *work) +{ + struct pci_pwrctl *pwrctl = container_of(work, struct pci_pwrctl, work); + +pci_lock_rescan_remove (); +pci_rescan_bus (to_pci_dev (pwrctl->dev->parent)->bus); +pci_unlock_rescan_remove (); +} + +/** +* pci_pwrctl_init() - Initialize the PCI power control context struct +* +* @pwrctl: PCI power control data +* @dev: Parent device +*/ +void pci_pwrctl_init (struct pci_pwrctl *pwrctl,struct device *dev) +{ + pwrctl->dev = dev; +INIT_WORK (&pwrctl->work, rescan_work_func); +} +EXPORT_SYMBOL_GPL (pci_pwrctl_init); +/** * pci_pwrctl_device_set_ready() - Notify the pwrctl subsystem that the PCI * device is powered-up and ready to be detected.
@@ -74,9 +96,7 @@ int pci_pwrctl_device_set_ready(struct pci_pwrctl *pwrctl)if (ret)return ret; -pci_lock_rescan_remove (); -pci_rescan_bus (to_pci_dev (pwrctl->dev->parent)->bus); -pci_unlock_rescan_remove (); +schedule_work (&pwrctl->work);return 0 ; }
drivers/pci/pwrctl/pci-pwrctl-pwrseq.c +1 -1
@@ -50,7 +50,7 @@ static int pci_pwrctl_pwrseq_probe(struct platform_device *pdev)if (ret)return ret; - data->ctx.dev = dev; +pci_pwrctl_init (&data->ctx, dev); ret =devm_pci_pwrctl_device_set_ready (dev, &data->ctx);if (ret)
include/linux/pci-pwrctl.h +3 -0
@@ -7,6 +7,7 @@#define __PCI_PWRCTL_H__#include <linux/notifier.h> +#include <linux/workqueue.h> struct device ;struct device_link ;
@@ -41,8 +42,10 @@ struct pci_pwrctl {/* Private: don't use. */ struct notifier_block nb;struct device_link *link; +struct work_struct work; }; +void pci_pwrctl_init (struct pci_pwrctl *pwrctl,struct device *dev);int pci_pwrctl_device_set_ready (struct pci_pwrctl *pwrctl);void pci_pwrctl_device_unset_ready (struct pci_pwrctl *pwrctl); int devm_pci_pwrctl_device_set_ready(struct device *dev,
Signature · 1 line
2.43.0
Re: [PATCH v3 1/2] PCI: don't rely on of_platform_depopulate() for reused OF-nodes
In reply to Bartosz Golaszewski[+to Rob]
Quoted 8 lines · Bartosz Golaszewski
From: Bartosz Golaszewski <[email protected]>
of_platform_depopulate()doesn't play nice with reused OF nodes - it ignores the ones that are not marked explicitly as populated and it may happen that the PCI device goes away before the platform device in which case the PCI core clears theOF_POPULATEDbit. We need to unconditionally unregister the platform devices for child nodes when stopping the PCI device.
Rob, any concerns with this?
Quoted 51 lines
Fixes 8fb18619d910 ("PCI/pwrctl: Create platform devices for child OF nodes of the port node")Signed-off-by Bartosz Golaszewski <[email protected]>--- drivers/pci/remove.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)drivers/pci/remove.c +17
-1@@ -1,7 +1,10 @@// SPDX-License-Identifier: GPL-2.0 #include <linux/pci.h> #include <linux/module.h> +#include <linux/of.h> #include <linux/of_platform.h> +#include <linux/platform_device.h> +#include "pci.h" static void pci_free_resources (struct pci_dev *dev)@@ -14,12 +17,25 @@ static void pci_free_resources(struct pci_dev *dev) } } +static int pci_pwrctl_unregister (struct device *dev,void *data) +{ +struct device_node *pci_node = data, *plat_node =dev_of_node (dev); + +if (dev_is_platform (dev) && plat_node && plat_node == pci_node) { +of_device_unregister (to_platform_device (dev)); +of_node_clear_flag (plat_node,OF_POPULATED ); + } + +return 0 ; +} +static void pci_stop_dev (struct pci_dev *dev) {pci_pme_active (dev, false);if (pci_dev_is_added (dev)) { -of_platform_depopulate (&dev->dev); +device_for_each_child (dev->dev.parent,dev_of_node (&dev->dev), + pci_pwrctl_unregister);device_release_driver (&dev->dev);pci_proc_detach_device (dev);pci_remove_sysfs_dev_files (dev);Signature · 1 line
2.43.0
Re: [PATCH v3 1/2] PCI: don't rely on of_platform_depopulate() for reused OF-nodes
Quoted 6 lines · Bjorn Helgaas
[+to Rob]
Quoted 2 lines · Bartosz Golaszewski
From: Bartosz Golaszewski <[email protected]>
Quoted 9 lines
Quoted 6 lines
of_platform_depopulate()doesn't play nice with reused OF nodes - it ignores the ones that are not marked explicitly as populated and it may happen that the PCI device goes away before the platform device in which case the PCI core clears theOF_POPULATEDbit. We need to unconditionally unregister the platform devices for child nodes when stopping the PCI device.Rob, any concerns with this?
If there will be any concerns: I'm OoO next week so I'll only be able to address them on September 2nd.
Bart
Re: [PATCH v3 1/2] PCI: don't rely on of_platform_depopulate() for reused OF-nodes
In reply to Bartosz Golaszewski+ Rob
Quoted 9 lines · Bartosz Golaszewski
From: Bartosz Golaszewski <[email protected]>
of_platform_depopulate()doesn't play nice with reused OF nodes - it ignores the ones that are not marked explicitly as populated and it may happen that the PCI device goes away before the platform device in which case the PCI core clears theOF_POPULATEDbit. We need to unconditionally unregister the platform devices for child nodes when stopping the PCI device.
It sounds like the fix is in of_platform_depopulate() itself and this patch works around the API issue in PCI driver.
Rob, is that correct?
- Mani
Quoted 51 lines
Fixes 8fb18619d910 ("PCI/pwrctl: Create platform devices for child OF nodes of the port node")Signed-off-by Bartosz Golaszewski <[email protected]>--- drivers/pci/remove.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)drivers/pci/remove.c +17
-1@@ -1,7 +1,10 @@// SPDX-License-Identifier: GPL-2.0 #include <linux/pci.h> #include <linux/module.h> +#include <linux/of.h> #include <linux/of_platform.h> +#include <linux/platform_device.h> +#include "pci.h" static void pci_free_resources (struct pci_dev *dev)@@ -14,12 +17,25 @@ static void pci_free_resources(struct pci_dev *dev) } } +static int pci_pwrctl_unregister (struct device *dev,void *data) +{ +struct device_node *pci_node = data, *plat_node =dev_of_node (dev); + +if (dev_is_platform (dev) && plat_node && plat_node == pci_node) { +of_device_unregister (to_platform_device (dev)); +of_node_clear_flag (plat_node,OF_POPULATED ); + } + +return 0 ; +} +static void pci_stop_dev (struct pci_dev *dev) {pci_pme_active (dev, false);if (pci_dev_is_added (dev)) { -of_platform_depopulate (&dev->dev); +device_for_each_child (dev->dev.parent,dev_of_node (&dev->dev), + pci_pwrctl_unregister);device_release_driver (&dev->dev);pci_proc_detach_device (dev);pci_remove_sysfs_dev_files (dev);Signature · 1 line
2.43.0
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v3 2/2] PCI/pwrctl: put the bus rescan on a different thread
In reply to Bartosz GolaszewskiQuoted 3 lines · Bartosz Golaszewski
From: Bartosz Golaszewski <[email protected]>
If we trigger the bus rescan from sysfs, we'll try to lock the PCI
I think the first 'we' is user and second 'we' is PCI and pwrctl drivers. If so, it should be spelled out to make it clear.
Quoted 3 lines
rescan mutex recursively and deadlock - the platform device will be populated and probed on the same thread that handles the sysfs write.
A little bit rewording could help here:
'When a user triggers a rescan from sysfs, sysfs code acquires the pci_rescan_remove_lock during the start of the rescan. Then if a platform device is created, pwrctl driver may get probed to control the power to the device and it will also try to acquire the same lock to do the rescan after powering up the device. And this will cause a deadlock.'
Quoted 7 lines
Add a workqueue to the pwrctl code on which we schedule the rescan for controlled PCI devices. While at it: add a new interface for initializing the pwrctl context where we'd now assign the parent device address and initialize the workqueue.
Fixes 4565d2652a37 ("PCI/pwrctl: Add PCI power control core code")Reported-by Konrad Dybcio <[email protected]>
Don't we need 'Closes' link these days? I hope this is reported in ML.
Quoted 1 line
Signed-off-by Bartosz Golaszewski <[email protected]>
With above changes,
- Mani
Quoted 89 lines
--- drivers/pci/pwrctl/core.c | 26 +++++++++++++++++++++++--- drivers/pci/pwrctl/pci-pwrctl-pwrseq.c | 2 +- include/linux/pci-pwrctl.h | 3 +++ 3 files changed, 27 insertions(+), 4 deletions(-)drivers/pci/pwrctl/core.c +23
-3@@ -48,6 +48,28 @@ static int pci_pwrctl_notify(struct notifier_block *nb, unsigned long action,return NOTIFY_DONE ; } +static void rescan_work_func (struct work_struct *work) +{ + struct pci_pwrctl *pwrctl = container_of(work, struct pci_pwrctl, work); + +pci_lock_rescan_remove (); +pci_rescan_bus (to_pci_dev (pwrctl->dev->parent)->bus); +pci_unlock_rescan_remove (); +} + +/** +* pci_pwrctl_init() - Initialize the PCI power control context struct +* +* @pwrctl: PCI power control data +* @dev: Parent device +*/ +void pci_pwrctl_init (struct pci_pwrctl *pwrctl,struct device *dev) +{ + pwrctl->dev = dev; +INIT_WORK (&pwrctl->work, rescan_work_func); +} +EXPORT_SYMBOL_GPL (pci_pwrctl_init); +/** * pci_pwrctl_device_set_ready() - Notify the pwrctl subsystem that the PCI * device is powered-up and ready to be detected. @@ -74,9 +96,7 @@ int pci_pwrctl_device_set_ready(struct pci_pwrctl *pwrctl)if (ret)return ret; -pci_lock_rescan_remove (); -pci_rescan_bus (to_pci_dev (pwrctl->dev->parent)->bus); -pci_unlock_rescan_remove (); +schedule_work (&pwrctl->work);return 0 ; }drivers/pci/pwrctl/pci-pwrctl-pwrseq.c +1
-1@@ -50,7 +50,7 @@ static int pci_pwrctl_pwrseq_probe(struct platform_device *pdev)if (ret)return ret; - data->ctx.dev = dev; +pci_pwrctl_init (&data->ctx, dev); ret =devm_pci_pwrctl_device_set_ready (dev, &data->ctx);if (ret)include/linux/pci-pwrctl.h +3
-0@@ -7,6 +7,7 @@#define __PCI_PWRCTL_H__#include <linux/notifier.h> +#include <linux/workqueue.h> struct device ;struct device_link ;@@ -41,8 +42,10 @@ struct pci_pwrctl {/* Private: don't use. */ struct notifier_block nb;struct device_link *link; +struct work_struct work; }; +void pci_pwrctl_init (struct pci_pwrctl *pwrctl,struct device *dev);int pci_pwrctl_device_set_ready (struct pci_pwrctl *pwrctl);void pci_pwrctl_device_unset_ready (struct pci_pwrctl *pwrctl); int devm_pci_pwrctl_device_set_ready(struct device *dev,Signature · 1 line
2.43.0
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v3 1/2] PCI: don't rely on of_platform_depopulate() for reused OF-nodes
In reply to Manivannan SadhasivamQuoted 14 lines · Manivannan Sadhasivam
+ Rob
Quoted 9 lines · Bartosz Golaszewski
From: Bartosz Golaszewski <[email protected]>
of_platform_depopulate()doesn't play nice with reused OF nodes - it ignores the ones that are not marked explicitly as populated and it may happen that the PCI device goes away before the platform device in which case the PCI core clears theOF_POPULATEDbit. We need to unconditionally unregister the platform devices for child nodes when stopping the PCI device.
Quoted 6 lines
It sounds like the fix is in
of_platform_depopulate()itself and this patch works around the API issue in PCI driver.Rob, is that correct?
- Mani
of_platform_depopulate() has more issues than just that. For one: it's asymmetric to of_platform_populate() as it takes a struct device as argument and not a device node. This causes issues for users like TI aemif that call of_platform_populate() on nodes without the compatible property that are never consumed by any device. AFAIK there's currently no way to depopulate them.
In this particular case I think that the OF_POPULATED bit should not be set when the PCI device is created but only when the platform device is.
However I'm afraid to change the semantics of of_platform_depopulate() et al for all users so I'm more inclined to have this fix in v6.11 to avoid releasing non functional code (pwrctl devices not being removed) and then possibly introduce a new variant of of_platform_depopulate() that would work slightly differently.
Bart
Re: [PATCH v3 1/2] PCI: don't rely on of_platform_depopulate() for reused OF-nodes
In reply to Bjorn HelgaasQuoted 6 lines · Bjorn Helgaas
[+to Rob]
Quoted 2 lines · Bartosz Golaszewski
From: Bartosz Golaszewski <[email protected]>
Quoted 8 lines
Quoted 6 lines
of_platform_depopulate()doesn't play nice with reused OF nodes - it ignores the ones that are not marked explicitly as populated and it may happen that the PCI device goes away before the platform device in which case the PCI core clears theOF_POPULATEDbit. We need to unconditionally unregister the platform devices for child nodes when stopping the PCI device.Rob, any concerns with this?
Yes, the flag bits are a mess. I don't have a better solution other than perhaps what Bartosz suggests.
I was going to say we shouldn't really be mucking with the OF_POPULATED flag outside the DT code, but then I grep'ed the tree. :(
Rob
Manivannan Sadhasivam
Acked-byRe: [PATCH v3 1/2] PCI: don't rely on of_platform_depopulate() for reused OF-nodes
In reply to Bartosz GolaszewskiQuoted 21 lines · Bartosz Golaszewski
Quoted 19 lines · Manivannan Sadhasivam
+ Rob
Quoted 9 lines · Bartosz Golaszewski
From: Bartosz Golaszewski <[email protected]>
of_platform_depopulate()doesn't play nice with reused OF nodes - it ignores the ones that are not marked explicitly as populated and it may happen that the PCI device goes away before the platform device in which case the PCI core clears theOF_POPULATEDbit. We need to unconditionally unregister the platform devices for child nodes when stopping the PCI device.It sounds like the fix is in
of_platform_depopulate()itself and this patch works around the API issue in PCI driver.Rob, is that correct?
Quoted 9 lines
Quoted 1 line
- Mani
of_platform_depopulate()has more issues than just that. For one: it's asymmetric toof_platform_populate()as it takes a struct device as argument and not a device node. This causes issues for users like TI aemif that callof_platform_populate()on nodes without the compatible property that are never consumed by any device. AFAIK there's currently no way to depopulate them.
Oouch!
Quoted 10 lines
In this particular case I think that the
OF_POPULATEDbit should not be set when the PCI device is created but only when the platform device is.However I'm afraid to change the semantics of
of_platform_depopulate()et al for all users so I'm more inclined to have this fix in v6.11 to avoid releasing non functional code (pwrctl devices not being removed) and then possibly introduce a new variant ofof_platform_depopulate()that would work slightly differently.
Ok, sounds like a plan. Since Rob is also in favor of this patch, it is good to get this series merged for 6.11.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v3 2/2] PCI/pwrctl: put the bus rescan on a different thread
In reply to Manivannan SadhasivamQuoted 26 lines · Manivannan Sadhasivam
Quoted 3 lines · Bartosz Golaszewski
From: Bartosz Golaszewski <[email protected]>
If we trigger the bus rescan from sysfs, we'll try to lock the PCI
I think the first 'we' is user and second 'we' is PCI and pwrctl drivers. If so, it should be spelled out to make it clear.
Quoted 3 lines
rescan mutex recursively and deadlock - the platform device will be populated and probed on the same thread that handles the sysfs write.
A little bit rewording could help here:
'When a user triggers a rescan from sysfs, sysfs code acquires the
pci_rescan_remove_lockduring the start of the rescan. Then if a platform device is created, pwrctl driver may get probed to control the power to the device and it will also try to acquire the same lock to do the rescan after powering up the device. And this will cause a deadlock.'Quoted 5 lines
Add a workqueue to the pwrctl code on which we schedule the rescan for controlled PCI devices. While at it: add a new interface for initializing the pwrctl context where we'd now assign the parent device address and initialize the workqueue.
Quoted 5 lines
Quoted 2 lines
Fixes 4565d2652a37 ("PCI/pwrctl: Add PCI power control core code")Reported-by Konrad Dybcio <[email protected]>Don't we need 'Closes' link these days? I hope this is reported in ML.
Nope, unfortunately on IRC. But the tag is unformally optional it seems.
Bart
Quoted 5 lines
Quoted 1 line
Signed-off-by Bartosz Golaszewski <[email protected]>With above changes,
Reviewed-by Manivannan Sadhasivam <[email protected]>
Quoted 24 lines · Bartosz Golaszewski
From: Bartosz Golaszewski <[email protected]>
Bjorn,
Here's a respin of the PCI/pwrctl fixes that should go into v6.11. I eventually found a solution that doesn't require Krishna's topology change but Krishna: please make sure to update the code in
drivers/pci/remove.cas well when rebasing your work once this gets upstream.v2 -> v3:
- use the correct device for unregistering the platform pwrctl device in patch 1/2
- make patch 1/2 easier to read by usingdevice_for_each_child()v1 -> v2:
- use the scoped variant offor_each_child_of_node()to fix a memory leak in patch 1/2Bartosz Golaszewski (2):
PCI: don't rely onof_platform_depopulate()for reused OF-nodes PCI/pwrctl: put the bus rescan on a different thread
Quoted 9 lines
drivers/pci/pwrctl/core.c | 26 +++++++++++++++++++++++--- drivers/pci/pwrctl/pci-pwrctl-pwrseq.c | 2 +- drivers/pci/remove.c | 18 +++++++++++++++++- include/linux/pci-pwrctl.h | 3 +++ 4 files changed, 44 insertions(+), 5 deletions(-)--
2.43.0
Bjorn: gentle ping. Neither Rob nor Mani are opposed to these and it's getting close to release. Could you please pick them up?
Bart
Bjorn Helgaas
AppliedQuoted 23 lines · Bartosz Golaszewski
From: Bartosz Golaszewski <[email protected]>
Bjorn,
Here's a respin of the PCI/pwrctl fixes that should go into v6.11. I eventually found a solution that doesn't require Krishna's topology change but Krishna: please make sure to update the code in
drivers/pci/remove.cas well when rebasing your work once this gets upstream.v2 -> v3:
- use the correct device for unregistering the platform pwrctl device in patch 1/2
- make patch 1/2 easier to read by usingdevice_for_each_child()v1 -> v2:
- use the scoped variant offor_each_child_of_node()to fix a memory leak in patch 1/2Bartosz Golaszewski (2):
PCI: don't rely onof_platform_depopulate()for reused OF-nodes PCI/pwrctl: put the bus rescan on a different thread
Quoted 5 lines
drivers/pci/pwrctl/core.c | 26 +++++++++++++++++++++++--- drivers/pci/pwrctl/pci-pwrctl-pwrseq.c | 2 +- drivers/pci/remove.c | 18 +++++++++++++++++- include/linux/pci-pwrctl.h | 3 +++ 4 files changed, 44 insertions(+), 5 deletions(-)
Applied to pci/for-linus for v6.11, thanks.