Raspberry Pi SPI unbind fixes
Subsystems (3)
cover Raspberry Pi SPI unbind fixes
This series fixes ordering issues occurring on unbind of the Raspberry Pi SPI drivers:
Turns out devm_spi_register_controller() is prone to incorrect use and dozens of drivers have gotten it wrong. I'm only documenting this gotcha here and fixing it in the Raspberry Pi drivers. Fixing the rest is for another day.
There's also an ordering issue in the core which has been present for 8 years and affects all platforms (patch [1/5]). Doesn't look like unbinding is tested all that often. :-)
Lukas Wunner (5):
spi: Fix controller unregister order
spi: bcm2835: Fix controller unregister order
spi: bcm2835aux: Fix controller unregister order
spi: bcm2835: Tear down DMA before turning off SPI controller
spi: Document devm_spi_register_controller() gotcha
drivers/spi/spi-bcm2835.c | 8 +++++--- drivers/spi/spi-bcm2835aux.c | 4 +++- drivers/spi/spi.c | 10 +++++++++- 3 files changed, 17 insertions(+), 5 deletions(-)
Signature · 1 line
2.26.2
PATCH 1/5 spi: Fix controller unregister order
When an SPI controller unregisters, it unbinds all its slave devices. For this, their drivers may need to access the SPI bus, e.g. to quiesce interrupts.
However since commit ffbbdd21329f ("spi: create a message queueing infrastructure"), spi_destroy_queue() is executed before unbinding the slaves. It sets ctlr->running = false, thereby preventing SPI bus access and causing unbinding of slave devices to fail.
Fix by unbinding slaves before calling spi_destroy_queue().
--- drivers/spi/spi.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/spi/spi.c +2 -1
@@ -2761,6 +2761,8 @@ void spi_unregister_controller(struct spi_controller *ctlr)struct spi_controller *found;int id = ctlr->bus_num; +device_for_each_child (&ctlr->dev,NULL , __unregister); +/* First make sure that this controller was ever added */ mutex_lock (&board_lock); found =idr_find (&spi_master_idr, id);
@@ -2773,7 +2775,6 @@ void spi_unregister_controller(struct spi_controller *ctlr)list_del (&ctlr->list);mutex_unlock (&board_lock); -device_for_each_child (&ctlr->dev,NULL , __unregister);device_unregister (&ctlr->dev);/* free bus id */ mutex_lock (&board_lock);
Signature · 1 line
2.26.2
Mark Brown
maintainerRe: [PATCH 1/5] spi: Fix controller unregister order
Quoted 4 lines · Lukas Wunner
However since commit
ffbbdd21329f("spi: create a message queueing infrastructure"),spi_destroy_queue()is executed before unbinding the slaves. It setsctlr->running= false, thereby preventing SPI bus access and causing unbinding of slave devices to fail.
Devices should basically never fail an unbind operation - if something went seriously wrong there's basically nothing that can be done in terms of error handling and keeping the device around isn't going to help.
Re: [PATCH 1/5] spi: Fix controller unregister order
Quoted 9 lines · Mark Brown
Quoted 4 lines · Lukas Wunner
However since commit
ffbbdd21329f("spi: create a message queueing infrastructure"),spi_destroy_queue()is executed before unbinding the slaves. It setsctlr->running= false, thereby preventing SPI bus access and causing unbinding of slave devices to fail.Devices should basically never fail an unbind operation - if something went seriously wrong there's basically nothing that can be done in terms of error handling and keeping the device around isn't going to help.
I guess the word "fail" in the commit message invites misinterpretations. The driver does unbind from the slave device, but the physical device is not left in a proper state. E.g. interrupts may still be generated by the device because writing a register to disable them failed.
Thanks,
Lukas
Re: [PATCH 1/5] spi: Fix controller unregister order
Quoted 8 lines · Lukas Wunner
Quoted 6 lines · Mark Brown
Quoted 4 lines · Lukas Wunner
However since commit
ffbbdd21329f("spi: create a message queueing infrastructure"),spi_destroy_queue()is executed before unbinding the slaves. It setsctlr->running= false, thereby preventing SPI bus access and causing unbinding of slave devices to fail.
Quoted 8 lines
Quoted 3 lines
Devices should basically never fail an unbind operation - if something went seriously wrong there's basically nothing that can be done in terms of error handling and keeping the device around isn't going to help.
I guess the word "fail" in the commit message invites misinterpretations. The driver does unbind from the slave device, but the physical device is not left in a proper state. E.g. interrupts may still be generated by the device because writing a register to disable them failed.
I didn't check a patch, but I see a bug on kernel bugzilla against spi-pxa2xx because of this. It requires quite untrivial ->remove() in order to quiescent the DMA and other activities.
Signature · 2 lines
With Best Regards, Andy Shevchenko
Re: [PATCH 1/5] spi: Fix controller unregister order
Quoted 12 lines · Andy Shevchenko
Quoted 11 lines · Lukas Wunner
Quoted 9 lines · Mark Brown
Quoted 4 lines · Lukas Wunner
However since commit
ffbbdd21329f("spi: create a message queueing infrastructure"),spi_destroy_queue()is executed before unbinding the slaves. It setsctlr->running= false, thereby preventing SPI bus access and causing unbinding of slave devices to fail.Devices should basically never fail an unbind operation - if something went seriously wrong there's basically nothing that can be done in terms of error handling and keeping the device around isn't going to help.
Quoted 8 lines
Quoted 4 lines
I guess the word "fail" in the commit message invites misinterpretations. The driver does unbind from the slave device, but the physical device is not left in a proper state. E.g. interrupts may still be generated by the device because writing a register to disable them failed.
I didn't check a patch, but I see a bug on kernel bugzilla against spi-pxa2xx because of this. It requires quite untrivial ->remove() in order to quiescent the DMA and other activities.
Yes from a quick look at spi-pxa2xx.c it's immediately obvious that the use of devm_spi_register_controller() is likewise completely wrong.
The crucial thing to understand is that the SPI driver's ->remove() hook is executed *before* any device-managed resources are released. pxa2xx_spi_remove() disables the clock, frees the IRQ, releases DMA, so the SPI controller is no longer usable even though it's still registered! Somehow this incorrect order got cargo-culted to dozens of drivers over the years.
We use SPI-attached Ethernet chips and when the SPI driver's module is unloaded, the Ethernet driver's ->ndo_stop() hook is executed to bring down the interface. For this it needs to communicate with the Ethernet chip, but it can't because the ->remove() hook has already been executed and unbinding of the SPI slave happens afterwards, when the SPI controller is unregistered via devres_release_all().
There's another issue in spi-pxa2xx.c: It acquires a runtime PM ref even though the driver core already does that.
Do you have a link to the spi-pxa2xx.c bugzilla? Are you able to test patches? I can submit a patch but I can only compile-test it, I don't have that hardware.
Thanks,
Lukas
Re: [PATCH 1/5] spi: Fix controller unregister order
Quoted 40 lines · Lukas Wunner
Quoted 12 lines · Andy Shevchenko
Quoted 11 lines · Lukas Wunner
Quoted 9 lines · Mark Brown
Quoted 4 lines · Lukas Wunner
However since commit
ffbbdd21329f("spi: create a message queueing infrastructure"),spi_destroy_queue()is executed before unbinding the slaves. It setsctlr->running= false, thereby preventing SPI bus access and causing unbinding of slave devices to fail.Devices should basically never fail an unbind operation - if something went seriously wrong there's basically nothing that can be done in terms of error handling and keeping the device around isn't going to help.
Quoted 8 lines
Quoted 4 lines
I guess the word "fail" in the commit message invites misinterpretations. The driver does unbind from the slave device, but the physical device is not left in a proper state. E.g. interrupts may still be generated by the device because writing a register to disable them failed.
I didn't check a patch, but I see a bug on kernel bugzilla against spi-pxa2xx because of this. It requires quite untrivial ->remove() in order to quiescent the DMA and other activities.
Yes from a quick look at
spi-pxa2xx.cit's immediately obvious that the use ofdevm_spi_register_controller()is likewise completely wrong.The crucial thing to understand is that the SPI driver's ->remove() hook is executed *before* any device-managed resources are released.
pxa2xx_spi_remove()disables the clock, frees the IRQ, releases DMA, so the SPI controller is no longer usable even though it's still registered! Somehow this incorrect order got cargo-culted to dozens of drivers over the years.We use SPI-attached Ethernet chips and when the SPI driver's module is unloaded, the Ethernet driver's ->ndo_stop() hook is executed to bring down the interface. For this it needs to communicate with the Ethernet chip, but it can't because the ->remove() hook has already been executed and unbinding of the SPI slave happens afterwards, when the SPI controller is unregistered via
devres_release_all().
Quoted 5 lines
There's another issue in
spi-pxa2xx.c: It acquires a runtime PM ref even though the driver core already does that.Do you have a link to the
spi-pxa2xx.cbugzilla? Are you able to test patches? I can submit a patch but I can only compile-test it,
Here you are: https://bugzilla.kernel.org/show_bug.cgi?id=206403.
There also a link to my GH tree where I tried to clean up a bit. And yes, I know about atomic handling bug there, but it's another story.
I was able to reproduce the bug once or twice, but submitter has a test case with reproducibility close to 100%.
Signature · 2 lines
With Best Regards, Andy Shevchenko
PATCH 2/5 spi: bcm2835: Fix controller unregister order
In reply to Lukas WunnerThe BCM2835 SPI driver uses devm_spi_register_controller() on bind. As a consequence, on unbind, __device_release_driver() first invokes bcm2835_spi_remove() before unregistering the SPI controller via devres_release_all().
This order is incorrect: bcm2835_spi_remove() tears down the DMA channels and turns off the SPI controller, including its interrupts and clock. The SPI controller is thus no longer usable.
When the SPI controller is subsequently unregistered, it unbinds all its slave devices. If their drivers need to access the SPI bus, e.g. to quiesce their interrupts, unbinding will fail.
As a rule, devm_spi_register_controller() must not be used if the ->remove() hook performs teardown steps which shall be performed after unbinding of slaves.
Fix by using the non-devm variant spi_register_controller(). Note that the struct spi_controller as well as the driver-private data are not freed until after bcm2835_spi_remove() has finished, so accessing them is safe.
--- drivers/spi/spi-bcm2835.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/spi/spi-bcm2835.c +3 -1
@@ -1351,7 +1351,7 @@ static int bcm2835_spi_probe(struct platform_device *pdev) goto out_dma_release; } - err =devm_spi_register_controller (&pdev->dev, ctlr); + err =spi_register_controller (ctlr);if (err) {dev_err (&pdev->dev,"could not register SPI controller: %d\n" , err);
@@ -1378,6 +1378,8 @@ static int bcm2835_spi_remove(struct platform_device *pdev)bcm2835_debugfs_remove (bs); +spi_unregister_controller (ctlr); +/* Clear FIFOs, and disable the HW block */ bcm2835_wr (bs,BCM2835_SPI_CS ,BCM2835_SPI_CS_CLEAR_RX |BCM2835_SPI_CS_CLEAR_TX );
Signature · 1 line
2.26.2
Mark Brown
maintainerRe: [PATCH 2/5] spi: bcm2835: Fix controller unregister order
Quoted 4 lines · Lukas Wunner
Fix by using the non-devm variant
spi_register_controller(). Note that the structspi_controlleras well as the driver-private data are not freed until afterbcm2835_spi_remove()has finished, so accessing them is safe.
Why not use managed allocations of clocks and DMA channels? This is a standard issue with the devm APIs, if you're using them you basically need to use them for *everything* up to the point where you start using them.
Re: [PATCH 2/5] spi: bcm2835: Fix controller unregister order
Quoted 10 lines · Mark Brown
Quoted 4 lines · Lukas Wunner
Fix by using the non-devm variant
spi_register_controller(). Note that the structspi_controlleras well as the driver-private data are not freed until afterbcm2835_spi_remove()has finished, so accessing them is safe.Why not use managed allocations of clocks and DMA channels? This is a standard issue with the devm APIs, if you're using them you basically need to use them for *everything* up to the point where you start using them.
There is no devm version of clk_prepare_enable(), dma_request_chan() and various other functions invoked on ->probe() by spi-bcm2835.c. So tearing down DMA channels, disabling clocks etc needs to happen in the ->remove() hook and consequently devm_spi_register_controller() cannot be used.
So I respectfully submit that the patch is fine.
Thanks,
Lukas
Re: [PATCH 2/5] spi: bcm2835: Fix controller unregister order
Quoted 12 lines · Lukas Wunner
Quoted 10 lines · Mark Brown
Quoted 4 lines · Lukas Wunner
Fix by using the non-devm variant
spi_register_controller(). Note that the structspi_controlleras well as the driver-private data are not freed until afterbcm2835_spi_remove()has finished, so accessing them is safe.Why not use managed allocations of clocks and DMA channels? This is a standard issue with the devm APIs, if you're using them you basically need to use them for *everything* up to the point where you start using them.
Quoted 5 lines
There is no devm version of
clk_prepare_enable(),dma_request_chan()and various other functions invoked on ->probe() byspi-bcm2835.c. So tearing down DMA channels, disabling clocks etc needs to happen in the ->remove() hook and consequentlydevm_spi_register_controller()cannot be used.
There is devm_add_action_or_reset (IIRC the name). It does a trick.
Signature · 2 lines
With Best Regards, Andy Shevchenko
Re: [PATCH 2/5] spi: bcm2835: Fix controller unregister order
Quoted 13 lines · Andy Shevchenko
Quoted 12 lines · Lukas Wunner
Quoted 10 lines · Mark Brown
Quoted 4 lines · Lukas Wunner
Fix by using the non-devm variant
spi_register_controller(). Note that the structspi_controlleras well as the driver-private data are not freed until afterbcm2835_spi_remove()has finished, so accessing them is safe.Why not use managed allocations of clocks and DMA channels? This is a standard issue with the devm APIs, if you're using them you basically need to use them for *everything* up to the point where you start using them.
Quoted 7 lines
Quoted 5 lines
There is no devm version of
clk_prepare_enable(),dma_request_chan()and various other functions invoked on ->probe() byspi-bcm2835.c. So tearing down DMA channels, disabling clocks etc needs to happen in the ->remove() hook and consequentlydevm_spi_register_controller()cannot be used.There is
devm_add_action_or_reset(IIRC the name). It does a trick.
Interesting, thanks.
However there are currently four actions performed by bcm2835_spi_remove():
bcm2835_debugfs_remove(bs); /* Clear FIFOs, and disable the HW block */ bcm2835_wr(bs, BCM2835_SPI_CS, BCM2835_SPI_CS_CLEAR_RX | BCM2835_SPI_CS_CLEAR_TX); clk_disable_unprepare(bs->clk); bcm2835_dma_release(ctlr, bs);
So I think I'd have to add four functions to perform these devm actions, which would add a lot more code than just the single line added by my patch. It also seems doubtful that the teardown code will still be easy to follow. And small patches like the ones I've submitted lend themselves better to backporting to stable.
Mark, please provide guidance as to which variant you'd prefer:
Switching to the non-devm variant of spi_register_controller() as I've done here or adding devm calls for all the existing teardown steps.
Thanks,
Lukas
Re: [PATCH 2/5] spi: bcm2835: Fix controller unregister order
Quoted 21 lines · Lukas Wunner
Quoted 19 lines · Andy Shevchenko
Quoted 17 lines · Lukas Wunner
Quoted 10 lines · Mark Brown
Quoted 4 lines · Lukas Wunner
Fix by using the non-devm variant
spi_register_controller(). Note that the structspi_controlleras well as the driver-private data are not freed until afterbcm2835_spi_remove()has finished, so accessing them is safe.Why not use managed allocations of clocks and DMA channels? This is a standard issue with the devm APIs, if you're using them you basically need to use them for *everything* up to the point where you start using them.
There is no devm version of
clk_prepare_enable(),dma_request_chan()and various other functions invoked on ->probe() byspi-bcm2835.c. So tearing down DMA channels, disabling clocks etc needs to happen in the ->remove() hook and consequentlydevm_spi_register_controller()cannot be used.
Quoted 8 lines
Quoted 1 line
There is
devm_add_action_or_reset(IIRC the name). It does a trick.Interesting, thanks.
However there are currently four actions performed by
bcm2835_spi_remove():bcm2835_debugfs_remove(bs);
This one shouldn't be counted. You can init it as the last op in ->probe() and that mustn't fail the probe.
Quoted 3 lines
/* Clear FIFOs, and disable the HW block */
bcm2835_wr(bs, BCM2835_SPI_CS,
BCM2835_SPI_CS_CLEAR_RX | BCM2835_SPI_CS_CLEAR_TX)This one I don't know.
Quoted 1 line
clk_disable_unprepare(bs->clk);
This one traditionally done via devm_add_action_or_reset.
Quoted 1 line
bcm2835_dma_release(ctlr, bs);
This one probably the one which needs to be addressed ideally in DMA engine for all.
Quoted 10 lines
So I think I'd have to add four functions to perform these devm actions, which would add a lot more code than just the single line added by my patch. It also seems doubtful that the teardown code will still be easy to follow. And small patches like the ones I've submitted lend themselves better to backporting to stable.
Mark, please provide guidance as to which variant you'd prefer:
Switching to the non-devm variant ofspi_register_controller()as I've done here or adding devm calls for all the existing teardown steps.
All generic devm_* APIs have a release counterparts, you can explicitly call them in order you want during ->remove(). So, I still see a benefit of devm_*() even in such cases (makes ->probe() much easier).
Signature · 2 lines
With Best Regards, Andy Shevchenko
PATCH 3/5 spi: bcm2835aux: Fix controller unregister order
In reply to Lukas WunnerThe BCM2835aux SPI driver uses devm_spi_register_master() on bind. As a consequence, on unbind, __device_release_driver() first invokes bcm2835aux_spi_remove() before unregistering the SPI controller via devres_release_all().
This order is incorrect: bcm2835aux_spi_remove() turns off the SPI controller, including its interrupts and clock. The SPI controller is thus no longer usable.
When the SPI controller is subsequently unregistered, it unbinds all its slave devices. If their drivers need to access the SPI bus, e.g. to quiesce their interrupts, unbinding will fail.
As a rule, devm_spi_register_master() must not be used if the ->remove() hook performs teardown steps which shall be performed after unbinding of slaves.
Fix by using the non-devm variant spi_register_master(). Note that the struct spi_master as well as the driver-private data are not freed until after bcm2835aux_spi_remove() has finished, so accessing them is safe.
--- drivers/spi/spi-bcm2835aux.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/spi/spi-bcm2835aux.c +3 -1
@@ -569,7 +569,7 @@ static int bcm2835aux_spi_probe(struct platform_device *pdev) goto out_clk_disable; } - err =devm_spi_register_master (&pdev->dev, master); + err =spi_register_master (master);if (err) {dev_err (&pdev->dev,"could not register SPI master: %d\n" , err); goto out_clk_disable;
@@ -593,6 +593,8 @@ static int bcm2835aux_spi_remove(struct platform_device *pdev)bcm2835aux_debugfs_remove (bs); +spi_unregister_master (master); +bcm2835aux_spi_reset_hw (bs);/* disable the HW block by releasing the clock */
Signature · 1 line
2.26.2
PATCH 4/5 spi: bcm2835: Tear down DMA before turning off SPI controller
In reply to Lukas WunnerOn unbind of the BCM2835 SPI driver, the SPI controller is disabled first and the DMA channels are terminated and torn down afterwards.
This seems backwards: In the theoretical case that DMA is active, it might try to fill the SPI FIFOs even after the controller has been disabled.
Reverse the order, thereby mirroring what's done on ->probe().
--- drivers/spi/spi-bcm2835.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/spi/spi-bcm2835.c +2 -2
@@ -1380,14 +1380,14 @@ static int bcm2835_spi_remove(struct platform_device *pdev)spi_unregister_controller (ctlr); +bcm2835_dma_release (ctlr, bs); +/* Clear FIFOs, and disable the HW block */ bcm2835_wr (bs,BCM2835_SPI_CS ,BCM2835_SPI_CS_CLEAR_RX |BCM2835_SPI_CS_CLEAR_TX );clk_disable_unprepare (bs->clk); -bcm2835_dma_release (ctlr, bs); -return 0 ; }
Signature · 1 line
2.26.2
PATCH 5/5 spi: Document devm_spi_register_controller() gotcha
In reply to Lukas WunnerAs a rule, devm_spi_register_controller() must not be used if the driver's ->remove() hook performs teardown steps which shall be performed after unbinding of slaves.
Dozens of drivers are doing it wrong. Document this gotcha to hopefully prevent further misuse.
--- drivers/spi/spi.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/spi/spi.c +7 -0
@@ -2714,6 +2714,13 @@ static void devm_spi_unregister(struct device *dev, void *res)* Register a SPI device as with spi_register_controller() which will * automatically be unregistered and freed. * +* Be aware that a managed SPI controller and the attached slaves are +* unregistered after the driver's ->remove() callback has been executed. +* So the SPI slaves may still access the bus during and after ->remove(). +* Thus, devm_spi_register_controller() may only be used if there is no +* ->remove() callback at all or if it does not perform teardown steps +* which render the bus inaccessible. +* * Return: zero on success, else a negative error code. */ int devm_spi_register_controller(struct device *dev,
Signature · 1 line
2.26.2
Mark Brown
maintainerRe: [PATCH 5/5] spi: Document devm_spi_register_controller() gotcha
Quoted 6 lines · Lukas Wunner
As a rule,
devm_spi_register_controller()must not be used if the driver's ->remove() hook performs teardown steps which shall be performed after unbinding of slaves.Dozens of drivers are doing it wrong. Document this gotcha to hopefully prevent further misuse.
This is something that needs to be documented at the devm level, it applies to pretty much every managed API.
Mark Brown
maintainerQuoted 9 lines · Lukas Wunner
This series fixes ordering issues occurring on unbind of the Raspberry Pi SPI drivers:
Turns out
devm_spi_register_controller()is prone to incorrect use and dozens of drivers have gotten it wrong. I'm only documenting this gotcha here and fixing it in the Raspberry Pi drivers. Fixing the rest is for another day.[...]
Applied to
https://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi.git for-next
Thanks!
[1/4] spi: Fix controller unregister order
commit: 84855678add8aba927faf76bc2f130a40f94b6f7
[2/4] spi: bcm2835: Fix controller unregister order
commit: 9dd277ff92d06f6aa95b39936ad83981d781f49b
[3/4] spi: bcm2835aux: Fix controller unregister order
commit: b9dd3f6d417258ad0beeb292a1bc74200149f15d[4/4] spi: bcm2835: Tear down DMA before turning off SPI controller
(no commit info)
All being well this means that it will be integrated into the linux-next tree (usually sometime in the next 24 hours) and sent to Linus during the next merge window (or sooner if it is a bug fix), however if problems are discovered then the patch may be dropped or reverted.
You may get further e-mails resulting from automated or manual testing and review of the tree, please engage with people reporting problems and send followup patches addressing any issues that are reported if needed.
If any updates are required or you are submitting further changes they should be sent as incremental updates against current git, existing patches will not be replaced.
Please add any relevant lists and maintainers to the CCs when replying to this mail.
Thanks,
Mark