driver core: enforce device_lock for driver_match_device()
Subsystems (9)
PATCH v5 1/1 driver core: enforce device_lock for driver_match_device()
Currently, driver_match_device() is called from three sites. One site (__device_attach_driver) holds device_lock(dev), but the other two (bind_store and __driver_attach) do not. This inconsistency means that bus match() callbacks are not guaranteed to be called with the lock held.
Fix this by introducing driver_match_device_locked(), which guarantees holding the device lock using a scoped guard. Replace the unlocked calls in bind_store() and __driver_attach() with this new helper. Also add a lock assertion to driver_match_device() to enforce this guarantee.
This consistency also fixes a known race condition. The driver_override implementation relies on the device_lock, so the missing lock led to the use-after-free (UAF) reported in Bugzilla for buses using this field.
Stress testing the two newly locked paths for 24 hours with CONFIG_PROVE_LOCKING and CONFIG_LOCKDEP enabled showed no UAF recurrence and no lockdep warnings.
---
v5:
* Introduce driver_match_device_locked() helper using guard(device) to handle locking, as suggested by Rafael J. Wysocki.
v4:
* Remove the misleading comment above device_lock_assert(), and update subject and commit message to focus on enforcing consistent locking, as discussed with Danilo Krummrich.
v3:
* Remove redundant locking comments at call sites and add a blank line after the lock assertion in driver_match_device(), as suggested by Greg KH. v2:
* Add device_lock_assert() in driver_match_device() to enforce locking requirement, as suggested by Greg KH.
v1:
* The Bugzilla entry contains full KASAN reports and two PoCs that reliably reproduce the UAF on both unlocked paths using a standard QEMU setup (default e1000 device at 0000:00:03.0).
--- drivers/base/base.h | 9 +++++++++ drivers/base/bus.c | 2 +- drivers/base/dd.c | 2 +- 3 files changed, 11 insertions(+), 2 deletions(-)
drivers/base/base.h +9 -0
@@ -182,9 +182,18 @@ void device_set_deferred_probe_reason(const struct device *dev, struct va_formatstatic inline int driver_match_device (const struct device_driver *drv,struct device *dev) { +device_lock_assert (dev); +return drv->bus->match ? drv->bus->match (dev, drv) :1 ; } +static inline int driver_match_device_locked (const struct device_driver *drv, +struct device *dev) +{ +guard (device)(dev); +return driver_match_device (drv, dev); +} +static inline void dev_sync_state (struct device *dev) {if (dev->bus->sync_state)
drivers/base/bus.c +1 -1
@@ -263,7 +263,7 @@ static ssize_t bind_store(struct device_driver *drv, const char *buf,int err = -ENODEV ; dev =bus_find_device_by_name (bus,NULL , buf); -if (dev &&driver_match_device (drv, dev)) { +if (dev &&driver_match_device_locked (drv, dev)) { err =device_driver_attach (drv, dev);if (!err) {/* success */
drivers/base/dd.c +1 -1
@@ -1178,7 +1178,7 @@ static int __driver_attach(struct device *dev, void *data)* is an error. */ - ret =driver_match_device (drv, dev); + ret =driver_match_device_locked (drv, dev);if (ret ==0 ) {/* no match */ return 0 ;
Signature · 1 line
2.43.0
Quoted 16 lines · Gui-Dong Han
Currently,
driver_match_device()is called from three sites. One site (__device_attach_driver) holdsdevice_lock(dev), but the other two (bind_storeand__driver_attach) do not. This inconsistency means that busmatch()callbacks are not guaranteed to be called with the lock held.Fix this by introducing
driver_match_device_locked(), which guarantees holding the device lock using a scoped guard. Replace the unlocked calls inbind_store()and__driver_attach()with this new helper. Also add a lock assertion todriver_match_device()to enforce this guarantee.This consistency also fixes a known race condition. The
driver_overrideimplementation relies on thedevice_lock, so the missing lock led to the use-after-free (UAF) reported in Bugzilla for buses using this field.
Quoted 7 lines
Stress testing the two newly locked paths for 24 hours with
CONFIG_PROVE_LOCKINGandCONFIG_LOCKDEPenabled showed no UAF recurrence and no lockdep warnings.Suggested-by Qiu-ji Chen <[email protected]>Signed-off-by Gui-Dong Han <[email protected]>
Quoted 76 lines
---
v5:
* Introducedriver_match_device_locked()helper using guard(device) to handle locking, as suggested by Rafael J. Wysocki.
v4:
* Remove the misleading comment abovedevice_lock_assert(), and update subject and commit message to focus on enforcing consistent locking, as discussed with Danilo Krummrich.
v3:
* Remove redundant locking comments at call sites and add a blank line after the lock assertion indriver_match_device(), as suggested by Greg KH. v2:
* Adddevice_lock_assert()indriver_match_device()to enforce locking requirement, as suggested by Greg KH.
v1:
* The Bugzilla entry contains full KASAN reports and two PoCs that reliably reproduce the UAF on both unlocked paths using a standard QEMU setup (default e1000 device at 0000:00:03.0).--- drivers/base/base.h | 9 +++++++++ drivers/base/bus.c | 2 +- drivers/base/dd.c | 2 +- 3 files changed, 11 insertions(+), 2 deletions(-)drivers/base/base.h +9
-0@@ -182,9 +182,18 @@ void device_set_deferred_probe_reason(const struct device *dev, struct va_formatstatic inline int driver_match_device (const struct device_driver *drv,struct device *dev) { +device_lock_assert (dev); +return drv->bus->match ? drv->bus->match (dev, drv) :1 ; } +static inline int driver_match_device_locked (const struct device_driver *drv, +struct device *dev) +{ +guard (device)(dev); +return driver_match_device (drv, dev); +} +static inline void dev_sync_state (struct device *dev) {if (dev->bus->sync_state)drivers/base/bus.c +1
-1@@ -263,7 +263,7 @@ static ssize_t bind_store(struct device_driver *drv, const char *buf,int err = -ENODEV ; dev =bus_find_device_by_name (bus,NULL , buf); -if (dev &&driver_match_device (drv, dev)) { +if (dev &&driver_match_device_locked (drv, dev)) { err =device_driver_attach (drv, dev);if (!err) {/* success */ drivers/base/dd.c +1
-2@@ -1178,7 +1178,7 @@ static int __driver_attach(struct device *dev, void *data)* is an error. */ - ret =driver_match_device (drv, dev); + ret =driver_match_device_locked (drv, dev);if (ret ==0 ) {/* no match */ return 0 ; --2.43.0
Quoted 15 lines · Gui-Dong Han
Currently,
driver_match_device()is called from three sites. One site (__device_attach_driver) holdsdevice_lock(dev), but the other two (bind_storeand__driver_attach) do not. This inconsistency means that busmatch()callbacks are not guaranteed to be called with the lock held.Fix this by introducing
driver_match_device_locked(), which guarantees holding the device lock using a scoped guard. Replace the unlocked calls inbind_store()and__driver_attach()with this new helper. Also add a lock assertion todriver_match_device()to enforce this guarantee.This consistency also fixes a known race condition. The
driver_overrideimplementation relies on thedevice_lock, so the missing lock led to the use-after-free (UAF) reported in Bugzilla for buses using this field.
Quoted 5 lines
Stress testing the two newly locked paths for 24 hours with
CONFIG_PROVE_LOCKINGandCONFIG_LOCKDEPenabled showed no UAF recurrence and no lockdep warnings.
Quoted 2 lines
Suggested-by Qiu-ji Chen <[email protected]>Signed-off-by Gui-Dong Han <[email protected]>
This looks good now!
Quoted 17 lines · Danilo Krummrich
Quoted 15 lines · Gui-Dong Han
Currently,
driver_match_device()is called from three sites. One site (__device_attach_driver) holdsdevice_lock(dev), but the other two (bind_storeand__driver_attach) do not. This inconsistency means that busmatch()callbacks are not guaranteed to be called with the lock held.Fix this by introducing
driver_match_device_locked(), which guarantees holding the device lock using a scoped guard. Replace the unlocked calls inbind_store()and__driver_attach()with this new helper. Also add a lock assertion todriver_match_device()to enforce this guarantee.This consistency also fixes a known race condition. The
driver_overrideimplementation relies on thedevice_lock, so the missing lock led to the use-after-free (UAF) reported in Bugzilla for buses using this field.
Quoted 7 lines
Quoted 5 lines
Stress testing the two newly locked paths for 24 hours with
CONFIG_PROVE_LOCKINGandCONFIG_LOCKDEPenabled showed no UAF recurrence and no lockdep warnings.Fixes 49b420a13ff9 ("driver core: check bus->match without holding device lock")
Thanks for the review! The Fixes tag looks correct as the inconsistency dates back to that commit.
Since a Fixes tag is present, I recall the patch bot often warns when a Fixes tag is provided without a corresponding Cc:
[email protected] tag. Perhaps we should include it? This would also allow us to fix the UAF in older kernels.
If you agree, could you please add it when picking up the patch?
Thank you very much!
Quoted 26 lines · Gui-Dong Han
Quoted 17 lines · Danilo Krummrich
Quoted 15 lines · Gui-Dong Han
Currently,
driver_match_device()is called from three sites. One site (__device_attach_driver) holdsdevice_lock(dev), but the other two (bind_storeand__driver_attach) do not. This inconsistency means that busmatch()callbacks are not guaranteed to be called with the lock held.Fix this by introducing
driver_match_device_locked(), which guarantees holding the device lock using a scoped guard. Replace the unlocked calls inbind_store()and__driver_attach()with this new helper. Also add a lock assertion todriver_match_device()to enforce this guarantee.This consistency also fixes a known race condition. The
driver_overrideimplementation relies on thedevice_lock, so the missing lock led to the use-after-free (UAF) reported in Bugzilla for buses using this field.Quoted 7 lines
Quoted 5 lines
Stress testing the two newly locked paths for 24 hours with
CONFIG_PROVE_LOCKINGandCONFIG_LOCKDEPenabled showed no UAF recurrence and no lockdep warnings.Fixes 49b420a13ff9 ("driver core: check bus->match without holding device lock")
Quoted 7 lines
Thanks for the review! The Fixes tag looks correct as the inconsistency dates back to that commit.
Since a Fixes tag is present, I recall the patch bot often warns when a Fixes tag is provided without a corresponding Cc:
[email protected] tag. Perhaps we should include it? This would also allow us to fix the UAF in older kernels.
Can you reproduce this UAF? If so, yes, otherwise fixing for theoretical things isn't good to backport.
Anyway, looks good to me:
Quoted 31 lines · Greg KH
Quoted 29 lines · Gui-Dong Han
Quoted 17 lines · Danilo Krummrich
Quoted 15 lines · Gui-Dong Han
Currently,
driver_match_device()is called from three sites. One site (__device_attach_driver) holdsdevice_lock(dev), but the other two (bind_storeand__driver_attach) do not. This inconsistency means that busmatch()callbacks are not guaranteed to be called with the lock held.Fix this by introducing
driver_match_device_locked(), which guarantees holding the device lock using a scoped guard. Replace the unlocked calls inbind_store()and__driver_attach()with this new helper. Also add a lock assertion todriver_match_device()to enforce this guarantee.This consistency also fixes a known race condition. The
driver_overrideimplementation relies on thedevice_lock, so the missing lock led to the use-after-free (UAF) reported in Bugzilla for buses using this field.Quoted 7 lines
Quoted 5 lines
Stress testing the two newly locked paths for 24 hours with
CONFIG_PROVE_LOCKINGandCONFIG_LOCKDEPenabled showed no UAF recurrence and no lockdep warnings.Fixes 49b420a13ff9 ("driver core: check bus->match without holding device lock")Thanks for the review! The Fixes tag looks correct as the inconsistency dates back to that commit.
Quoted 7 lines
Quoted 4 lines
Since a Fixes tag is present, I recall the patch bot often warns when a Fixes tag is provided without a corresponding Cc:
[email protected] tag. Perhaps we should include it? This would also allow us to fix the UAF in older kernels.Can you reproduce this UAF? If so, yes, otherwise fixing for theoretical things isn't good to backport.
Yes, it is reproducible. The Bugzilla entry linked in the Closes tag contains full KASAN reports and PoCs that reliably reproduce the UAF. (This was also noted in the v1 changelog).
Thanks for the review!
Quoted 15 lines · Gui-Dong Han
Currently,
driver_match_device()is called from three sites. One site (__device_attach_driver) holdsdevice_lock(dev), but the other two (bind_storeand__driver_attach) do not. This inconsistency means that busmatch()callbacks are not guaranteed to be called with the lock held.Fix this by introducing
driver_match_device_locked(), which guarantees holding the device lock using a scoped guard. Replace the unlocked calls inbind_store()and__driver_attach()with this new helper. Also add a lock assertion todriver_match_device()to enforce this guarantee.This consistency also fixes a known race condition. The
driver_overrideimplementation relies on thedevice_lock, so the missing lock led to the use-after-free (UAF) reported in Bugzilla for buses using this field.
Quoted 7 lines
Stress testing the two newly locked paths for 24 hours with
CONFIG_PROVE_LOCKINGandCONFIG_LOCKDEPenabled showed no UAF recurrence and no lockdep warnings.Suggested-by Qiu-ji Chen <[email protected]>Signed-off-by Gui-Dong Han <[email protected]>
Applied to driver-core-linus, thanks!
Quoted 5 lines · Gui-Dong Han
Currently,
driver_match_device()is called from three sites. One site (__device_attach_driver) holdsdevice_lock(dev), but the other two (bind_storeand__driver_attach) do not. This inconsistency means that busmatch()callbacks are not guaranteed to be called with the lock held.
I'm seeing boot hangs on Arm Juno in next/pending-fixes which bisect to this commit. The boot grinds to a halt near the end of boot:
[ 2.570549] ledtrig-cpu: registered to indicate activity on CPUs [ 2.618301] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.623547] msm_serial: driver initialized [ 2.624058] SuperH (H)SCI(F) driver initialized [ 2.624312] STM32 USART driver initialized
with no further output, full log:
https://lava.sirena.org.uk/scheduler/job/2387335#L862
We are also seeing similar looking boot hangs on some Qualcomm platforms in Arm's test lab which aren't verified to be the same thing but are hanging at a similar point in boot.
Bisect log:
# bad: [53d96388f8a868d3f79559a6cf39db81b34f9fb4] Merge branch 'slab/for-next-fixes' of https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git
# good: [24d479d26b25bce5faea3ddd9fa8f3a6c3129ea7] Linux 6.19-rc6
# good: [caa329649259d0f90c0056c9860ca659d4ba3211] spi: intel-pci: Add support for Nova Lake SPI serial flash
# good: [4b58aac989c1e3fafb1c68a733811859df388250] regmap: Fix race condition in hwspinlock irqsave routine
# good: [b062a899c997df7b9ce29c62164888baa7a85833] spi: hisi-kunpeng: Fixed the wrong debugfs node name in hisi_spi debugfs initialization
# good: [f3f380ce6b3d5c9805c7e0b3d5bc28d9ec41e2e8] regmap: maple: free entry on mas_store_gfp() failure
git bisect start '53d96388f8a868d3f79559a6cf39db81b34f9fb4' '24d479d26b25bce5faea3ddd9fa8f3a6c3129ea7' 'caa329649259d0f90c0056c9860ca659d4ba3211' '4b58aac989c1e3fafb1c68a733811859df388250' 'b062a899c997df7b9ce29c62164888baa7a85833' 'f3f380ce6b3d5c9805c7e0b3d5bc28d9ec41e2e8'
# test job: [caa329649259d0f90c0056c9860ca659d4ba3211] https://lava.sirena.org.uk/scheduler/job/2376182 # test job: [4b58aac989c1e3fafb1c68a733811859df388250] https://lava.sirena.org.uk/scheduler/job/2364586 # test job: [b062a899c997df7b9ce29c62164888baa7a85833] https://lava.sirena.org.uk/scheduler/job/2363939 # test job: [f3f380ce6b3d5c9805c7e0b3d5bc28d9ec41e2e8] https://lava.sirena.org.uk/scheduler/job/2331556 # test job: [53d96388f8a868d3f79559a6cf39db81b34f9fb4] https://lava.sirena.org.uk/scheduler/job/2387335 # bad: [53d96388f8a868d3f79559a6cf39db81b34f9fb4] Merge branch 'slab/for-next-fixes' of https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git git bisect bad 53d96388f8a868d3f79559a6cf39db81b34f9fb4 # test job: [64f6a0a031257e68060f22d217c53a50a671dfc8] https://lava.sirena.org.uk/scheduler/job/2387395 # good: [64f6a0a031257e68060f22d217c53a50a671dfc8] Merge branch 'for-next' of https://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless.git git bisect good 64f6a0a031257e68060f22d217c53a50a671dfc8 # test job: [686c6adb9882dd8d9fe1cc816bfd289dfa0abbd1] https://lava.sirena.org.uk/scheduler/job/2387470 # bad: [686c6adb9882dd8d9fe1cc816bfd289dfa0abbd1] Merge branch 'fixes' of https://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86.git git bisect bad 686c6adb9882dd8d9fe1cc816bfd289dfa0abbd1 # test job: [e03b29b55f2b7c345a919a6ee36633b06bf3fb56] https://lava.sirena.org.uk/scheduler/job/2387535 # good: [e03b29b55f2b7c345a919a6ee36633b06bf3fb56] comedi: dmm32at: serialize use of paged registers git bisect good e03b29b55f2b7c345a919a6ee36633b06bf3fb56 # test job: [bb25769465146dfc54e90e4066e8f41ec6b25730] https://lava.sirena.org.uk/scheduler/job/2387693 # bad: [bb25769465146dfc54e90e4066e8f41ec6b25730] Merge branch 'mtd/fixes' of https://git.kernel.org/pub/scm/linux/kernel/git/mtd/linux.git git bisect bad bb25769465146dfc54e90e4066e8f41ec6b25730 # test job: [dc23806a7c47ec5f1293aba407fb69519f976ee0] https://lava.sirena.org.uk/scheduler/job/2388190 # bad: [dc23806a7c47ec5f1293aba407fb69519f976ee0] driver core: enforce device_lock for driver_match_device() git bisect bad dc23806a7c47ec5f1293aba407fb69519f976ee0 # test job: [5f4476e98387618ce22bb93fb5c11142827458ec] https://lava.sirena.org.uk/scheduler/job/2388401 # good: [5f4476e98387618ce22bb93fb5c11142827458ec] rust: auxiliary: add Driver::unbind() callback git bisect good 5f4476e98387618ce22bb93fb5c11142827458ec # test job: [c1d4519e1c36ffa01973e23af4502e69dcd84f39] https://lava.sirena.org.uk/scheduler/job/2388603 # good: [c1d4519e1c36ffa01973e23af4502e69dcd84f39] rust: driver: add DEVICE_DRIVER_OFFSET to the DriverLayout trait git bisect good c1d4519e1c36ffa01973e23af4502e69dcd84f39 # test job: [a995fe1a3aa78b7d06cc1cc7b6b8436c5e93b07f] https://lava.sirena.org.uk/scheduler/job/2388676 # good: [a995fe1a3aa78b7d06cc1cc7b6b8436c5e93b07f] rust: driver: drop device private data post unbind git bisect good a995fe1a3aa78b7d06cc1cc7b6b8436c5e93b07f # first bad commit: [dc23806a7c47ec5f1293aba407fb69519f976ee0] driver core: enforce device_lock for driver_match_device()
Quoted 17 lines · Mark Brown
Quoted 5 lines · Gui-Dong Han
Currently,
driver_match_device()is called from three sites. One site (__device_attach_driver) holdsdevice_lock(dev), but the other two (bind_storeand__driver_attach) do not. This inconsistency means that busmatch()callbacks are not guaranteed to be called with the lock held.I'm seeing boot hangs on Arm Juno in next/pending-fixes which bisect to this commit. The boot grinds to a halt near the end of boot:
[ 2.570549] ledtrig-cpu: registered to indicate activity on CPUs [ 2.618301] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.623547] msm_serial: driver initialized [ 2.624058] SuperH (H)SCI(F) driver initialized [ 2.624312] STM32 USART driver initialized
Quoted 7 lines
with no further output, full log:
https://lava.sirena.org.uk/scheduler/job/2387335#L862We are also seeing similar looking boot hangs on some Qualcomm platforms in Arm's test lab which aren't verified to be the same thing but are hanging at a similar point in boot.
Hi Mark,
Thanks for the report and the detailed bisect log.
I verified this on x86 without issues, but it seems I missed this regression on Arm platforms. I will investigate the cause of this hang immediately.
Thanks
Quoted 8 lines · Gui-Dong Han
Quoted 2 lines · Mark Brown
I'm seeing boot hangs on Arm Juno in next/pending-fixes which bisect to this commit. The boot grinds to a halt near the end of boot:
I verified this on x86 without issues, but it seems I missed this regression on Arm platforms. I will investigate the cause of this hang immediately.
Thanks. It's not all platforms - in particular none of the main emulation platforms triggered the issue unfortunately. It seems to be a fairly limited subset of physical platforms that are impacted.
Quoted 7 lines · Mark Brown
Quoted 4 lines · Gui-Dong Han
Quoted 2 lines · Mark Brown
I'm seeing boot hangs on Arm Juno in next/pending-fixes which bisect to this commit. The boot grinds to a halt near the end of boot:
Quoted 7 lines
Quoted 3 lines
I verified this on x86 without issues, but it seems I missed this regression on Arm platforms. I will investigate the cause of this hang immediately.
Thanks. It's not all platforms - in particular none of the main emulation platforms triggered the issue unfortunately. It seems to be a fairly limited subset of physical platforms that are impacted.
I am quite puzzled by this. My suspicion is a deadlock, possibly involving PM/power domain operations within the match() callback triggering a lock dependency.
However, since __device_attach_driver (the device-initiated binding path) has always held device_lock while calling match(), it is strange that this locking pattern causes issues now in the __driver_attach (driver-initiated) path. The specific timing or device state during late binding must be the key factor.
I do not have access to the affected hardware, but I will attempt to reproduce it on QEMU Arm and perform a code audit of the relevant bus match functions.
If anyone has insights into potential lock conflicts specific to this path, please share.
Thanks.
Quoted 8 lines
I'm seeing boot hangs on Arm Juno in next/pending-fixes which bisect to this commit. The boot grinds to a halt near the end of boot:
[ 2.570549] ledtrig-cpu: registered to indicate activity on CPUs [ 2.618301] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.623547] msm_serial: driver initialized [ 2.624058] SuperH (H)SCI(F) driver initialized [ 2.624312] STM32 USART driver initialized
After partially modifying juno.dts, I managed to roughly emulate kernel boot on juno board with qemu and successfully reproduced the boot hang. Below is the gdb backtrace:
#0 0xffff800080114ae0 in mutex_spin_on_owner (lock=0xffff0000036bfc90, owner=0xffff000003510000, ww_ctx=0x0, waiter=0x0) at kernel/locking/mutex.c:377 #1 0xffff80008118cecc in mutex_optimistic_spin (waiter=<optimized out>, ww_ctx=<optimized out>, lock=<optimized out>) at kernel/locking/mutex.c:480 #2 __mutex_lock_common (use_ww_ctx=<optimized out>, ww_ctx=<optimized out>, ip=<optimized out>, nest_lock=<optimized out>, subclass=<optimized out>, state=<optimized out>, lock=<optimized out>) at kernel/locking/mutex.c:618 #3 __mutex_lock (lock=0xffff0000036bfc90, state=0x2, ip=<optimized out>, nest_lock=<optimized out>, subclass=<optimized out>) at kernel/locking/mutex.c:776 #4 0xffff80008118d1dc in __mutex_lock_slowpath (lock=0xffff0000036bfc90) at kernel/locking/mutex.c:1065 #5 0xffff80008118d230 in mutex_lock (lock=0xffff0000036bfc90) at kernel/locking/mutex.c:290 #6 0xffff8000809cdd1c in device_lock (dev=<optimized out>) at ./include/linux/device.h:895 #7 class_device_constructor (_T=<optimized out>) at ./include/linux/device.h:913 #8 driver_match_device_locked (dev=<optimized out>, drv=<optimized out>) at drivers/base/base.h:193 #9 __driver_attach (dev=0xffff0000036bfc10, data=0xffff800082e64440 <qcom_smmu_tbu_driver+40>) at drivers/base/dd.c:1183
#10 0xffff8000809cb17c in bus_for_each_dev (bus=0xffff0000036bfc90, start=0x0, data=0xffff800082e64440 <qcom_smmu_tbu_driver+40>, fn=0xffff8000809cdcec <__driver_attach>) at drivers/base/bus.c:383 #11 0xffff8000809cd03c in driver_attach (drv=0x0) at drivers/base/dd.c:1245
#12 0xffff8000809cc748 in bus_add_driver (drv=0xffff800082e64440 <qcom_smmu_tbu_driver+40>) at drivers/base/bus.c:715
#13 0xffff8000809ced28 in driver_register (drv=0xffff800082e64440 <qcom_smmu_tbu_driver+40>) at drivers/base/driver.c:249
#14 0xffff8000809d0254 in __platform_driver_register (drv=0x0, owner=0xffff000003510000) at drivers/base/platform.c:908
#15 0xffff8000809a6208 in qcom_smmu_impl_init (smmu=0xffff0000037c0080) at drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c:780
#16 0xffff8000809a48a0 in arm_smmu_impl_init (smmu=0xffff0000037c0080) at drivers/iommu/arm/arm-smmu/arm-smmu-impl.c:224
#17 0xffff8000809a2ae0 in arm_smmu_device_probe (pdev=0xffff0000036bfc00) at drivers/iommu/arm/arm-smmu/arm-smmu.c:2155
#18 0xffff8000809d060c in platform_probe (_dev=0xffff0000036bfc10) at drivers/base/platform.c:1446
#19 0xffff8000809cd6a4 in call_driver_probe (drv=<optimized out>, dev=<optimized out>) at drivers/base/dd.c:583
#20 really_probe (dev=0xffff0000036bfc10, drv=0xffff800082e641c0 <arm_smmu_driver+40>) at drivers/base/dd.c:661
#21 0xffff8000809cd8f8 in __driver_probe_device (drv=0xffff800082e641c0 <arm_smmu_driver+40>, dev=0xffff0000036bfc10) at drivers/base/dd.c:803
#22 0xffff8000809cdb34 in driver_probe_device (drv=0xffff0000036bfc90, dev=0xffff0000036bfc10) at drivers/base/dd.c:833
#23 0xffff8000809cddb8 in __driver_attach (data=<optimized out>, dev=<optimized out>) at drivers/base/dd.c:1227
#24 __driver_attach (dev=0xffff0000036bfc10, data=0xffff800082e641c0 <arm_smmu_driver+40>) at drivers/base/dd.c:1167
#25 0xffff8000809cb17c in bus_for_each_dev (bus=0xffff0000036bfc90, start=0x0, data=0xffff800082e641c0 <arm_smmu_driver+40>, fn=0xffff8000809cdcec <__driver_attach>) at drivers/base/bus.c:383 #26 0xffff8000809cd03c in driver_attach (drv=0x0) at drivers/base/dd.c:1245
#27 0xffff8000809cc748 in bus_add_driver (drv=0xffff800082e641c0 <arm_smmu_driver+40>) at drivers/base/bus.c:715
#28 0xffff8000809ced28 in driver_register (drv=0xffff800082e641c0 <arm_smmu_driver+40>) at drivers/base/driver.c:249
#29 0xffff8000809d0254 in __platform_driver_register (drv=0x0, owner=0xffff000003510000) at drivers/base/platform.c:908
#30 0xffff800081f3d12c in arm_smmu_driver_init () at drivers/iommu/arm/arm-smmu/arm-smmu.c:2368
#31 0xffff800080015218 in do_one_initcall (fn=0xffff800081f3d10c <arm_smmu_driver_init>) at init/main.c:1378
#32 0xffff800081ed13e4 in do_initcall_level (command_line=<optimized out>, level=<optimized out>) at init/main.c:1440
#33 do_initcalls () at init/main.c:1456
#34 do_basic_setup () at init/main.c:1475
#35 kernel_init_freeable () at init/main.c:1688
#36 0xffff800081187b50 in kernel_init (unused=0xffff0000036bfc90) at init/main.c:1578
#37 0xffff800080015f58 in ret_from_fork () at arch/arm64/kernel/entry.S:860
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Quoted 10 lines · Wang Jiayue
Quoted 9 lines
I'm seeing boot hangs on Arm Juno in next/pending-fixes which bisect to this commit. The boot grinds to a halt near the end of boot:
[ 2.570549] ledtrig-cpu: registered to indicate activity on CPUs [ 2.618301] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.623547] msm_serial: driver initialized [ 2.624058] SuperH (H)SCI(F) driver initialized [ 2.624312] STM32 USART driver initialized
Quoted 3 lines
After partially modifying
juno.dts, I managed to roughly emulate kernel boot on juno board with qemu and successfully reproduced the boot hang. Below is the gdb backtrace:
Great work, thank you very much!
This is a bit puzzling to me. Since __device_driver_lock(dev, dev->parent) is called a few lines later in __driver_attach(), I wonder if the original kernel would also deadlock if match() returns true in such a nested scenario?
Thanks.
Danilo Krummrich
maintainer(Cc: Rob, Will, Robin, Joerg)
Quoted 35 lines · Wang Jiayue
After partially modifying
juno.dts, I managed to roughly emulate kernel boot on juno board with qemu and successfully reproduced the boot hang. Below is the gdb backtrace:#0 0xffff800080114ae0 in mutex_spin_on_owner (lock=0xffff0000036bfc90, owner=0xffff000003510000, ww_ctx=0x0, waiter=0x0) at kernel/locking/mutex.c:377 #1 0xffff80008118cecc in mutex_optimistic_spin (waiter=<optimized out>, ww_ctx=<optimized out>, lock=<optimized out>) at kernel/locking/mutex.c:480 #2 __mutex_lock_common (use_ww_ctx=<optimized out>, ww_ctx=<optimized out>, ip=<optimized out>, nest_lock=<optimized out>, subclass=<optimized out>, state=<optimized out>, lock=<optimized out>) at kernel/locking/mutex.c:618 #3 __mutex_lock (lock=0xffff0000036bfc90, state=0x2, ip=<optimized out>, nest_lock=<optimized out>, subclass=<optimized out>) at kernel/locking/mutex.c:776 #4 0xffff80008118d1dc in __mutex_lock_slowpath (lock=0xffff0000036bfc90) at kernel/locking/mutex.c:1065 #5 0xffff80008118d230 in mutex_lock (lock=0xffff0000036bfc90) at kernel/locking/mutex.c:290 #6 0xffff8000809cdd1c in device_lock (dev=<optimized out>) at ./include/linux/device.h:895 #7 class_device_constructor (_T=<optimized out>) at ./include/linux/device.h:913 #8 driver_match_device_locked (dev=<optimized out>, drv=<optimized out>) at drivers/base/base.h:193 #9 __driver_attach (dev=0xffff0000036bfc10, data=0xffff800082e64440 <qcom_smmu_tbu_driver+40>) at drivers/base/dd.c:1183#10
0xffff8000809cb17cinbus_for_each_dev(bus=0xffff0000036bfc90, start=0x0, data=0xffff800082e64440 <qcom_smmu_tbu_driver+40>, fn=0xffff8000809cdcec <__driver_attach>) atdrivers/base/bus.c:383#110xffff8000809cd03cindriver_attach(drv=0x0) atdrivers/base/dd.c:1245
#120xffff8000809cc748inbus_add_driver(drv=0xffff800082e64440 <qcom_smmu_tbu_driver+40>) atdrivers/base/bus.c:715
#130xffff8000809ced28indriver_register(drv=0xffff800082e64440 <qcom_smmu_tbu_driver+40>) atdrivers/base/driver.c:249
#140xffff8000809d0254in__platform_driver_register(drv=0x0, owner=0xffff000003510000) atdrivers/base/platform.c:908
#150xffff8000809a6208inqcom_smmu_impl_init(smmu=0xffff0000037c0080) atdrivers/iommu/arm/arm-smmu/arm-smmu-qcom.c:780
#160xffff8000809a48a0inarm_smmu_impl_init(smmu=0xffff0000037c0080) atdrivers/iommu/arm/arm-smmu/arm-smmu-impl.c:224
#170xffff8000809a2ae0inarm_smmu_device_probe(pdev=0xffff0000036bfc00) atdrivers/iommu/arm/arm-smmu/arm-smmu.c:2155
#180xffff8000809d060cinplatform_probe(_dev=0xffff0000036bfc10) atdrivers/base/platform.c:1446
#190xffff8000809cd6a4incall_driver_probe(drv=<optimized out>, dev=<optimized out>) atdrivers/base/dd.c:583
#20really_probe(dev=0xffff0000036bfc10, drv=0xffff800082e641c0 <arm_smmu_driver+40>) atdrivers/base/dd.c:661
#210xffff8000809cd8f8in__driver_probe_device(drv=0xffff800082e641c0 <arm_smmu_driver+40>, dev=0xffff0000036bfc10) atdrivers/base/dd.c:803
#220xffff8000809cdb34indriver_probe_device(drv=0xffff0000036bfc90, dev=0xffff0000036bfc10) atdrivers/base/dd.c:833
#230xffff8000809cddb8in__driver_attach(data=<optimized out>, dev=<optimized out>) atdrivers/base/dd.c:1227
#24__driver_attach(dev=0xffff0000036bfc10, data=0xffff800082e641c0 <arm_smmu_driver+40>) atdrivers/base/dd.c:1167
#250xffff8000809cb17cinbus_for_each_dev(bus=0xffff0000036bfc90, start=0x0, data=0xffff800082e641c0 <arm_smmu_driver+40>, fn=0xffff8000809cdcec <__driver_attach>) atdrivers/base/bus.c:383#260xffff8000809cd03cindriver_attach(drv=0x0) atdrivers/base/dd.c:1245
#270xffff8000809cc748inbus_add_driver(drv=0xffff800082e641c0 <arm_smmu_driver+40>) atdrivers/base/bus.c:715
#280xffff8000809ced28indriver_register(drv=0xffff800082e641c0 <arm_smmu_driver+40>) atdrivers/base/driver.c:249
#290xffff8000809d0254in__platform_driver_register(drv=0x0, owner=0xffff000003510000) atdrivers/base/platform.c:908
#300xffff800081f3d12cinarm_smmu_driver_init() atdrivers/iommu/arm/arm-smmu/arm-smmu.c:2368
Quoted 8 lines
#31
0xffff800080015218indo_one_initcall(fn=0xffff800081f3d10c <arm_smmu_driver_init>) atinit/main.c:1378#320xffff800081ed13e4indo_initcall_level(command_line=<optimized out>, level=<optimized out>) atinit/main.c:1440#33do_initcalls() atinit/main.c:1456
#34do_basic_setup() atinit/main.c:1475
#35kernel_init_freeable() atinit/main.c:1688
#360xffff800081187b50inkernel_init(unused=0xffff0000036bfc90) atinit/main.c:1578
#370xffff800080015f58inret_from_fork() atarch/arm64/kernel/entry.S:860
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thanks, this backtrace is very helpful. My lockdep patch should reveal the same issue once run on real hardware, but with this it's probably not even necessary anymore.
So, the problem is that in the callstack of the arm-smmu driver's (a platform driver) probe() function, the QCOM specific code (through arm_smmu_impl_init()) registers another platform driver. Since we are still in probe() of arm-smmu the call to platform_driver_register() happens with the device lock of the arm-smmu platform device held.
platform_driver_register() eventually results in driver_attach() which iterates over all the devices of a bus. Since the device we are probing and the driver we are registering are for the same bus (i.e. the platform bus) it can now happen that by chance that we also match the exact same device that is currently probed again. And since we take the device lock for matching now, we actually take the same lock twice.
Now, we could avoid this by not matching bound devices, but we check this through dev->driver while holding the device lock, so that doesn't help.
But on the other hand, I don't see any reason why a driver would call platform_driver_register() from probe() in the first place. I think drivers should not do that and instead just register the driver through a normal initcall.
(If, however, it turns out that registering drivers from probe() is something we really need for some reason, it is probably best to drop the patch and don't make any guarantees about whether match() is called with the device lock held or not.
Consequently, driver_override must be protected with a separate lock (which would be the cleaner solution in any case).)
Danilo Krummrich
maintainerQuoted 21 lines · Danilo Krummrich
So, the problem is that in the callstack of the arm-smmu driver's (a platform driver)
probe()function, the QCOM specific code (througharm_smmu_impl_init()) registers another platform driver. Since we are still inprobe()of arm-smmu the call toplatform_driver_register()happens with the device lock of the arm-smmu platform device held.
platform_driver_register()eventually results indriver_attach()which iterates over all the devices of a bus. Since the device we are probing and the driver we are registering are for the same bus (i.e. the platform bus) it can now happen that by chance that we also match the exact same device that is currently probed again. And since we take the device lock for matching now, we actually take the same lock twice.Now, we could avoid this by not matching bound devices, but we check this through
dev->driverwhile holding the device lock, so that doesn't help.But on the other hand, I don't see any reason why a driver would call
platform_driver_register()fromprobe()in the first place. I think drivers should not do that and instead just register the driver through a normal initcall.
Quoted 7 lines
(If, however, it turns out that registering drivers from
probe()is something we really need for some reason, it is probably best to drop the patch and don't make any guarantees about whethermatch()is called with the device lock held or not.Consequently,
driver_overridemust be protected with a separate lock (which would be the cleaner solution in any case).)
I assume that this should resolve the problem (unless there are more drivers that register drivers in probe()):
drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c +2 -4
@@ -774,10 +774,6 @@ struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu) {const struct device_node *np = smmu->dev->of_node;const struct of_device_id *match; -static u8 tbu_registered; - -if (!tbu_registered++) -platform_driver_register (&qcom_smmu_tbu_driver);#ifdef CONFIG_ACPI if (np ==NULL ) {
@@ -802,3 +798,5 @@ struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu)return smmu; } + +builtin_platform_driver (qcom_smmu_tbu_driver);
@qcom maintainers: I'm aware of commit 0b4eeee2876f ("iommu/arm-smmu-qcom:
Register the TBU driver in qcom_smmu_impl_init"), but I think the above patch should work fine as it is still *not only* registered when CONFIG_ARM_SMMU_QCOM_DEBUG?
Greg KH
maintainerQuoted 38 lines · Danilo Krummrich
Quoted 21 lines · Danilo Krummrich
So, the problem is that in the callstack of the arm-smmu driver's (a platform driver)
probe()function, the QCOM specific code (througharm_smmu_impl_init()) registers another platform driver. Since we are still inprobe()of arm-smmu the call toplatform_driver_register()happens with the device lock of the arm-smmu platform device held.
platform_driver_register()eventually results indriver_attach()which iterates over all the devices of a bus. Since the device we are probing and the driver we are registering are for the same bus (i.e. the platform bus) it can now happen that by chance that we also match the exact same device that is currently probed again. And since we take the device lock for matching now, we actually take the same lock twice.Now, we could avoid this by not matching bound devices, but we check this through
dev->driverwhile holding the device lock, so that doesn't help.But on the other hand, I don't see any reason why a driver would call
platform_driver_register()fromprobe()in the first place. I think drivers should not do that and instead just register the driver through a normal initcall.Quoted 7 lines
(If, however, it turns out that registering drivers from
probe()is something we really need for some reason, it is probably best to drop the patch and don't make any guarantees about whethermatch()is called with the device lock held or not.Consequently,
driver_overridemust be protected with a separate lock (which would be the cleaner solution in any case).)I assume that this should resolve the problem (unless there are more drivers that register drivers in
probe()):drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c +0
-0
drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu) · line 780
Quoted 7 lines
drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c +0
-4@@ -774,10 +774,6 @@ struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu) {const struct device_node *np = smmu->dev->of_node;const struct of_device_id *match; -static u8 tbu_registered; - -if (!tbu_registered++) -platform_driver_register (&qcom_smmu_tbu_driver);
Ick, yeah, that should not be happening. We should deadlock on that no matter what.
drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu) · line 802
Quoted 9 lines
drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c +2
-0#ifdef CONFIG_ACPI if (np ==NULL ) {@@ -802,3 +798,5 @@ struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu)return smmu; } + +builtin_platform_driver (qcom_smmu_tbu_driver);
change makes sense to me.
thanks,
greg k-h
Quoted 2 lines · Danilo Krummrich
I assume that this should resolve the problem (unless there are more drivers that register drivers in
probe()):
This makes sense to me, I guess we can just fix any other instances as they arise. Will you send the patch?
Danilo Krummrich
maintainerQuoted 7 lines · Mark Brown
Quoted 2 lines · Danilo Krummrich
I assume that this should resolve the problem (unless there are more drivers that register drivers in
probe()):This makes sense to me, I guess we can just fix any other instances as they arise. Will you send the patch?
Yes, I will send it soon.
Will Deacon
maintainerQuoted 10 lines · Danilo Krummrich
Quoted 7 lines · Mark Brown
Quoted 2 lines · Danilo Krummrich
I assume that this should resolve the problem (unless there are more drivers that register drivers in
probe()):This makes sense to me, I guess we can just fix any other instances as they arise. Will you send the patch?
Yes, I will send it soon.
Please make sure you cc Dmitry and Georgi when you do that. I'm worried that you proposal means we run the probe code once per TBU, which looks like it will break.
Will
Danilo Krummrich
maintainerQuoted 7 lines · Will Deacon
Quoted 6 lines · Danilo Krummrich
Quoted 5 lines · Mark Brown
Quoted 2 lines · Danilo Krummrich
I assume that this should resolve the problem (unless there are more drivers that register drivers in
probe()):
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
This makes sense to me, I guess we can just fix any other instances as they arise. Will you send the patch?
Yes, I will send it soon.
Please make sure you cc Dmitry and Georgi when you do that.
Sure!
Quoted 2 lines
I'm worried that you proposal means we run the probe code once per TBU, which looks like it will break.
It shouldn't change anything in this regard. It only changes when the qcom_smmu_tbu_driver is registered, initcall vs probe() of another driver.
- Danilo
Robin Murphy
reviewerQuoted 55 lines · Danilo Krummrich
Quoted 21 lines · Danilo Krummrich
So, the problem is that in the callstack of the arm-smmu driver's (a platform driver)
probe()function, the QCOM specific code (througharm_smmu_impl_init()) registers another platform driver. Since we are still inprobe()of arm-smmu the call toplatform_driver_register()happens with the device lock of the arm-smmu platform device held.
platform_driver_register()eventually results indriver_attach()which iterates over all the devices of a bus. Since the device we are probing and the driver we are registering are for the same bus (i.e. the platform bus) it can now happen that by chance that we also match the exact same device that is currently probed again. And since we take the device lock for matching now, we actually take the same lock twice.Now, we could avoid this by not matching bound devices, but we check this through
dev->driverwhile holding the device lock, so that doesn't help.But on the other hand, I don't see any reason why a driver would call
platform_driver_register()fromprobe()in the first place. I think drivers should not do that and instead just register the driver through a normal initcall.Quoted 7 lines
(If, however, it turns out that registering drivers from
probe()is something we really need for some reason, it is probably best to drop the patch and don't make any guarantees about whethermatch()is called with the device lock held or not.Consequently,
driver_overridemust be protected with a separate lock (which would be the cleaner solution in any case).)I assume that this should resolve the problem (unless there are more drivers that register drivers in
probe()):drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c +2
-4@@ -774,10 +774,6 @@ struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu) {const struct device_node *np = smmu->dev->of_node;const struct of_device_id *match; -static u8 tbu_registered; - -if (!tbu_registered++) -platform_driver_register (&qcom_smmu_tbu_driver);#ifdef CONFIG_ACPI if (np ==NULL ) {@@ -802,3 +798,5 @@ struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu)return smmu; } + +builtin_platform_driver (qcom_smmu_tbu_driver);
Quoted 4 lines
@qcom maintainers: I'm aware of commit
0b4eeee2876f("iommu/arm-smmu-qcom:
Register the TBU driver inqcom_smmu_impl_init"), but I think the above patch should work fine as it is still *not only* registered whenCONFIG_ARM_SMMU_QCOM_DEBUG?
In principle there should be nothing wrong with registering the driver unconditionally - that existing tbu_registered logic looks racy in the face of async_probe anyway - however I don't think the *_platform_driver macros will work here, as this all gets combined into arm_smmu.ko wherein ending up with multiple module_init declarations breaks the build.
(Please do double-check all the build permutations of ARM_SMMU, ARM_SMMU_QCOM and ARM_SMMU_QCOM_DEBUG)
Thanks,
Robin.
Danilo Krummrich
maintainerQuoted 67 lines · Robin Murphy
Quoted 55 lines · Danilo Krummrich
Quoted 21 lines · Danilo Krummrich
So, the problem is that in the callstack of the arm-smmu driver's (a platform driver)
probe()function, the QCOM specific code (througharm_smmu_impl_init()) registers another platform driver. Since we are still inprobe()of arm-smmu the call toplatform_driver_register()happens with the device lock of the arm-smmu platform device held.
platform_driver_register()eventually results indriver_attach()which iterates over all the devices of a bus. Since the device we are probing and the driver we are registering are for the same bus (i.e. the platform bus) it can now happen that by chance that we also match the exact same device that is currently probed again. And since we take the device lock for matching now, we actually take the same lock twice.Now, we could avoid this by not matching bound devices, but we check this through
dev->driverwhile holding the device lock, so that doesn't help.But on the other hand, I don't see any reason why a driver would call
platform_driver_register()fromprobe()in the first place. I think drivers should not do that and instead just register the driver through a normal initcall.Quoted 7 lines
(If, however, it turns out that registering drivers from
probe()is something we really need for some reason, it is probably best to drop the patch and don't make any guarantees about whethermatch()is called with the device lock held or not.Consequently,
driver_overridemust be protected with a separate lock (which would be the cleaner solution in any case).)I assume that this should resolve the problem (unless there are more drivers that register drivers in
probe()):drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c +2
-4@@ -774,10 +774,6 @@ struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu) {const struct device_node *np = smmu->dev->of_node;const struct of_device_id *match; -static u8 tbu_registered; - -if (!tbu_registered++) -platform_driver_register (&qcom_smmu_tbu_driver);#ifdef CONFIG_ACPI if (np ==NULL ) {@@ -802,3 +798,5 @@ struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu)return smmu; } + +builtin_platform_driver (qcom_smmu_tbu_driver);Quoted 4 lines
@qcom maintainers: I'm aware of commit
0b4eeee2876f("iommu/arm-smmu-qcom:
Register the TBU driver inqcom_smmu_impl_init"), but I think the above patch should work fine as it is still *not only* registered whenCONFIG_ARM_SMMU_QCOM_DEBUG?In principle there should be nothing wrong with registering the driver unconditionally - that existing
tbu_registeredlogic looks racy in the face ofasync_probeanyway - however I don't think the*_platform_drivermacros will work here, as this all gets combined into arm_smmu.ko wherein ending up with multiplemodule_initdeclarations breaks the build.
Quoted 2 lines
(Please do double-check all the build permutations of
ARM_SMMU,ARM_SMMU_QCOMandARM_SMMU_QCOM_DEBUG)
Indeed, I accounted for this in the final patch I sent out, thanks!
Jiayue Wang
Tested-byQuoted 28 lines · Danilo Krummrich
Quoted 26 lines
So, the problem is that in the callstack of the arm-smmu driver's (a platform driver)
probe()function, the QCOM specific code (througharm_smmu_impl_init()) registers another platform driver. Since we are still inprobe()of arm-smmu the call toplatform_driver_register()happens with the device lock of the arm-smmu platform device held.
platform_driver_register()eventually results indriver_attach()which iterates over all the devices of a bus. Since the device we are probing and the driver we are registering are for the same bus (i.e. the platform bus) it can now happen that by chance that we also match the exact same device that is currently probed again. And since we take the device lock for matching now, we actually take the same lock twice.Now, we could avoid this by not matching bound devices, but we check this through
dev->driverwhile holding the device lock, so that doesn't help.But on the other hand, I don't see any reason why a driver would call
platform_driver_register()fromprobe()in the first place. I think drivers should not do that and instead just register the driver through a normal initcall.(If, however, it turns out that registering drivers from
probe()is something we really need for some reason, it is probably best to drop the patch and don't make any guarantees about whethermatch()is called with the device lock held or not.
Quoted 5 lines
Quoted 2 lines
Consequently,
driver_overridemust be protected with a separate lock (which would be the cleaner solution in any case).)I assume that this should resolve the problem (unless there are more drivers that register drivers in
probe()):
I tested this patch on qemu, and the boot appears to be normal now without deadlocks. I encountered some other minor issues, but those are likely due to my forced emulation in qemu and should be unrelated.
Quoted 27 lines
drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c +2
-4@@ -774,10 +774,6 @@ struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu) {const struct device_node *np = smmu->dev->of_node;const struct of_device_id *match; -static u8 tbu_registered; - -if (!tbu_registered++) -platform_driver_register (&qcom_smmu_tbu_driver);#ifdef CONFIG_ACPI if (np ==NULL ) {@@ -802,3 +798,5 @@ struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu)return smmu; } + +builtin_platform_driver (qcom_smmu_tbu_driver);@qcom maintainers: I'm aware of commit
0b4eeee2876f("iommu/arm-smmu-qcom:
Register the TBU driver inqcom_smmu_impl_init"), but I think the above patch should work fine as it is still *not only* registered whenCONFIG_ARM_SMMU_QCOM_DEBUG?
Danilo Krummrich
maintainerQuoted 7 lines · Mark Brown
Quoted 5 lines · Gui-Dong Han
Currently,
driver_match_device()is called from three sites. One site (__device_attach_driver) holdsdevice_lock(dev), but the other two (bind_storeand__driver_attach) do not. This inconsistency means that busmatch()callbacks are not guaranteed to be called with the lock held.
Quoted 8 lines
I'm seeing boot hangs on Arm Juno in next/pending-fixes which bisect to this commit. The boot grinds to a halt near the end of boot:
[ 2.570549] ledtrig-cpu: registered to indicate activity on CPUs [ 2.618301] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.623547] msm_serial: driver initialized [ 2.624058] SuperH (H)SCI(F) driver initialized [ 2.624312] STM32 USART driver initialized
Hm..sounds a bit like some match() callback manually takes the device_lock() and the reason we're not seeing anything from lockdep is because it happens with the serial driver.
I don't have a machine to reproduce it, but for debugging it would probably help to not actually take the lock in __driver_attach(), but only acquire / release the corresponding lockdep map. If my suspicion is correct, we should see a lockdep splat pointing out the issue.
Quoted 5 lines · Danilo Krummrich
Quoted 2 lines · Mark Brown
I'm seeing boot hangs on Arm Juno in next/pending-fixes which bisect to this commit. The boot grinds to a halt near the end of boot:
Quoted 8 lines
Hm..sounds a bit like some
match()callback manually takes thedevice_lock()and the reason we're not seeing anything from lockdep is because it happens with the serial driver.I don't have a machine to reproduce it, but for debugging it would probably help to not actually take the lock in
__driver_attach(), but only acquire / release the corresponding lockdep map. If my suspicion is correct, we should see a lockdep splat pointing out the issue.
lockdep isn't enabled by any of the defconfigs so has limited test coverage, the only one I'm seeing it enabled by is ps3_defconfig. I'll try to run some tests with lockdep enabled.
Quoted 4 lines · Danilo Krummrich
I don't have a machine to reproduce it, but for debugging it would probably help to not actually take the lock in
__driver_attach(), but only acquire / release the corresponding lockdep map. If my suspicion is correct, we should see a lockdep splat pointing out the issue.
I tried lockdep but didn't see anything different. Instrumenting with printk() tells me it's deadlocking trying to attach arm-smmu on Juno (that's a v1 SMMU on this platform), I'll try to poke further but it'll likely be tomorrow at the earliest.
Danilo Krummrich
maintainerQuoted 11 lines · Mark Brown
Quoted 4 lines · Danilo Krummrich
I don't have a machine to reproduce it, but for debugging it would probably help to not actually take the lock in
__driver_attach(), but only acquire / release the corresponding lockdep map. If my suspicion is correct, we should see a lockdep splat pointing out the issue.I tried lockdep but didn't see anything different. Instrumenting with
printk()tells me it's deadlocking trying to attach arm-smmu on Juno (that's a v1 SMMU on this platform), I'll try to poke further but it'll likely be tomorrow at the earliest.
Maybe the following diff faking the lock for lockdep helps, as it should keep things running, i.e. with this we have the exact same semantics as if we'd revert the patch (except for the lockdep check of course).
drivers/base/base.h +7 -2
@@ -190,8 +190,13 @@ static inline int driver_match_device(const struct device_driver *drv,static inline int driver_match_device_locked (const struct device_driver *drv,struct device *dev) { -guard (device)(dev); -return driver_match_device (drv, dev); +int ret; + +mutex_acquire (&dev->mutex.dep_map,0 ,0 , _THIS_IP_); + ret =driver_match_device (drv, dev); +mutex_release (&dev->mutex.dep_map, _THIS_IP_); + +return ret; }static inline void dev_sync_state (struct device *dev)
Quoted 10 lines · Danilo Krummrich
Quoted 4 lines · Mark Brown
I tried lockdep but didn't see anything different. Instrumenting with
printk()tells me it's deadlocking trying to attach arm-smmu on Juno (that's a v1 SMMU on this platform), I'll try to poke further but it'll likely be tomorrow at the earliest.Maybe the following diff faking the lock for lockdep helps, as it should keep things running, i.e. with this we have the exact same semantics as if we'd revert the patch (except for the lockdep check of course).
That does allow us to continue to make progress, the SMMU never manages to probe AFAICT but we do boot normally.
Danilo Krummrich
maintainerQuoted 8 lines · Mark Brown
Quoted 6 lines · Danilo Krummrich
Quoted 4 lines · Mark Brown
I tried lockdep but didn't see anything different. Instrumenting with
printk()tells me it's deadlocking trying to attach arm-smmu on Juno (that's a v1 SMMU on this platform), I'll try to poke further but it'll likely be tomorrow at the earliest.
Quoted 6 lines
Quoted 3 lines
Maybe the following diff faking the lock for lockdep helps, as it should keep things running, i.e. with this we have the exact same semantics as if we'd revert the patch (except for the lockdep check of course).
That does allow us to continue to make progress, the SMMU never manages to probe AFAICT but we do boot normally.
I really would expect a lockdep splat in this case, so I was even about to ask whether CONFIG_PROVE_LOCKING etc. is enabled. But it's me who messed it up. I missed that we have lockdep_set_novalidate_class(&dev->mutex).
(The fact that the SMMU never manages to probe must be unrelated, i.e. a different issue. Since my diff should be equivalent to a revert of the patch, except that it fakes that the mutex has been taken for lockdep.)
Anyways, this should work:
drivers/base/base.h +7 -2
@@ -190,8 +190,13 @@ static inline int driver_match_device(const struct device_driver *drv,static inline int driver_match_device_locked (const struct device_driver *drv,struct device *dev) { -guard (device)(dev); -return driver_match_device (drv, dev); +int ret; + +mutex_acquire (&dev->mutex.dep_map,0 ,0 , _THIS_IP_); + ret =driver_match_device (drv, dev); +mutex_release (&dev->mutex.dep_map, _THIS_IP_); + +return ret; }static inline void dev_sync_state (struct device *dev)
drivers/base/core.c +1 -1
@@ -3159,7 +3159,7 @@ void device_initialize(struct device *dev)kobject_init (&dev->kobj, &device_ktype);INIT_LIST_HEAD (&dev->dma_pools);mutex_init (&dev->mutex); -lockdep_set_novalidate_class (&dev->mutex); +//lockdep_set_novalidate_class(&dev->mutex); spin_lock_init (&dev->devres_lock);INIT_LIST_HEAD (&dev->devres_head);device_pm_init (dev);
Danilo Krummrich
maintainerQuoted 1 line · Danilo Krummrich
Anyways, this should work:
I Just notied that I pasted the wrong diff, which was nonsense of course, since it just unlocks all the suppressed false positives. (Should not have sent it during a meeting. :)
What I actually intended (not neat, but hopefully helps):
drivers/base/base.h +7 -2
@@ -190,8 +190,13 @@ static inline int driver_match_device(const struct device_driver *drv,static inline int driver_match_device_locked (const struct device_driver *drv,struct device *dev) { -guard (device)(dev); -return driver_match_device (drv, dev); +int ret; + +mutex_acquire (&dev->mutex.dep_map,0 ,0 , _THIS_IP_); + ret =driver_match_device (drv, dev); +mutex_release (&dev->mutex.dep_map, _THIS_IP_); + +return ret; }static inline void dev_sync_state (struct device *dev)
drivers/base/core.c +5 -1
@@ -2557,6 +2557,8 @@ static void device_release(struct kobject *kobj)kfree (dev->dma_range_map); +lockdep_unregister_key (&dev->lock_key); +if (dev->release) dev->release (dev);else if (dev->type && dev->type->release)
@@ -3159,7 +3161,9 @@ void device_initialize(struct device *dev)kobject_init (&dev->kobj, &device_ktype);INIT_LIST_HEAD (&dev->dma_pools);mutex_init (&dev->mutex); -lockdep_set_novalidate_class (&dev->mutex); +//lockdep_set_novalidate_class(&dev->mutex); +lockdep_register_key (&dev->lock_key); +lockdep_set_class (&dev->mutex, &dev->lock_key);spin_lock_init (&dev->devres_lock);INIT_LIST_HEAD (&dev->devres_head);device_pm_init (dev);
include/linux/device.h +1 -0
@@ -579,6 +579,7 @@ struct device {struct mutex mutex;/* mutex to synchronize calls to * its driver. */ +struct lock_class_key lock_key;struct dev_links_info links;struct dev_pm_info power;
Quoted 9 lines · Danilo Krummrich
Quoted 1 line · Danilo Krummrich
Anyways, this should work:
I Just notied that I pasted the wrong diff, which was nonsense of course, since it just unlocks all the suppressed false positives. (Should not have sent it during a meeting. :)
What I actually intended (not neat, but hopefully helps):
Thanks for the updated diff.
I tested it on my QEMU setup. Since I couldn't reproduce the hang there, I didn't see any lockdep splats regarding the deadlock. However, since the physical lock is removed, my PoCs successfully triggered the UAF on both paths as expected.
I did notice a lockdep warning during boot, which happens every time. I suspect this is because faux_bus_init is an __init function, so we are registering a key from memory that gets freed. This seems specific to the debug code, but I'm pasting it below for reference.
[ 2.093905] WARNING: CPU: 1 PID: 1 at kernel/locking/lockdep.c:1227 lockdep_register_key+0x104/0x150 [ 2.094924] Modules linked in: [ 2.095682] CPU: 1 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.17.0-dirty #11 PREEMPT(voluntary) [ 2.096241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 2.096896] RIP: 0010:lockdep_register_key+0x104/0x150 [ 2.097301] Code: 8b 04 ed 20 91 dd 90 4c 89 6b 08 48 89 03 48 89 1c ed 20 91 dd 90 48 85 c0 74 04 48 89 58 08 49 [ 2.098909] RSP: 0018:ffff888001217e60 EFLAGS: 00000202 [ 2.099260] RAX: 0000000000000001 RBX: ffffffff8f025260 RCX: 0000000000000000 [ 2.099566] RDX: 1ffffffff1e04a45 RSI: 0000000000000001 RDI: ffffffff8f025260 [ 2.099898] RBP: ffffffff8f025140 R08: 0000000000000004 R09: 0000000000000000 [ 2.100228] R10: ffffffff8f025213 R11: 0000000000000000 R12: ffffffff8f025260 [ 2.100556] R13: ffff888001209580 R14: 0000000000000000 R15: 0000000000000000 [ 2.101923] FS: 0000000000000000(0000) GS:ffff8880dc98d000(0000) knlGS:0000000000000000 [ 2.102298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2.102560] CR2: 0000000000000000 CR3: 000000004de8c000 CR4: 00000000000006f0 [ 2.102628] Call Trace: [ 2.102628] <TASK> [ 2.102991] device_initialize+0xc3/0x500 [ 2.103307] device_register+0xd/0x20 [ 2.103525] faux_bus_init+0x12/0x80 [ 2.103730] driver_init+0x2e/0x50 [ 2.104066] kernel_init_freeable+0x33e/0x6e0 [ 2.104335] ? __pfx_kernel_init+0x10/0x10 [ 2.104566] kernel_init+0x1a/0x1e0 [ 2.104765] ? _raw_spin_unlock_irq+0x23/0x40 [ 2.105118] ret_from_fork+0x255/0x330 [ 2.105350] ? __pfx_kernel_init+0x10/0x10 [ 2.105567] ret_from_fork_asm+0x1a/0x30 [ 2.106001] </TASK> [ 2.106218] irq event stamp: 3545 [ 2.106405] hardirqs last enabled at (3553): [<ffffffff8ace8436>] __up_console_sem+0x66/0x70 [ 2.106892] hardirqs last disabled at (3562): [<ffffffff8ace841b>] __up_console_sem+0x4b/0x70 [ 2.107276] softirqs last enabled at (3244): [<ffffffff8ab7de83>] handle_softirqs+0x4f3/0x750 [ 2.107644] softirqs last disabled at (3239): [<ffffffff8ab7e214>] __irq_exit_rcu+0xc4/0x100 [ 2.108924] ---[ end trace 0000000000000000 ]---
Thanks.
Quoted 19 lines · Gui-Dong Han
Quoted 9 lines · Danilo Krummrich
Quoted 1 line · Danilo Krummrich
Anyways, this should work:
I Just notied that I pasted the wrong diff, which was nonsense of course, since it just unlocks all the suppressed false positives. (Should not have sent it during a meeting. :)
What I actually intended (not neat, but hopefully helps):
Thanks for the updated diff.
I tested it on my QEMU setup. Since I couldn't reproduce the hang there, I didn't see any lockdep splats regarding the deadlock. However, since the physical lock is removed, my PoCs successfully triggered the UAF on both paths as expected.
Quoted 4 lines
I did notice a lockdep warning during boot, which happens every time. I suspect this is because
faux_bus_initis an__initfunction, so we are registering a key from memory that gets freed. This seems specific to the debug code, but I'm pasting it below for reference.
I figured out the root cause.
The warning is triggered because faux_bus_root is a static object. lockdep_register_key() has a WARN_ON_ONCE(static_obj(key)) check that forbids registering keys residing in static memory. It is not about __init memory being freed.
Anyway, this is not a big deal and doesn't impact the testing results.
Thanks.
Greg KH
maintainerQuoted 25 lines · Gui-Dong Han
Quoted 19 lines · Gui-Dong Han
Quoted 9 lines · Danilo Krummrich
Quoted 1 line · Danilo Krummrich
Anyways, this should work:
I Just notied that I pasted the wrong diff, which was nonsense of course, since it just unlocks all the suppressed false positives. (Should not have sent it during a meeting. :)
What I actually intended (not neat, but hopefully helps):
Thanks for the updated diff.
I tested it on my QEMU setup. Since I couldn't reproduce the hang there, I didn't see any lockdep splats regarding the deadlock. However, since the physical lock is removed, my PoCs successfully triggered the UAF on both paths as expected.
Quoted 4 lines
I did notice a lockdep warning during boot, which happens every time. I suspect this is because
faux_bus_initis an__initfunction, so we are registering a key from memory that gets freed. This seems specific to the debug code, but I'm pasting it below for reference.
Quoted 8 lines
I figured out the root cause.
The warning is triggered because
faux_bus_rootis a static object.lockdep_register_key()has a WARN_ON_ONCE(static_obj(key)) check that forbids registering keys residing in static memory. It is not about__initmemory being freed.Anyway, this is not a big deal and doesn't impact the testing results.
Ooh, nice catch. Let me go make that a dynamic object. It really shouldn't be a static one, I hate static struct device usage, and complain about it from everyone else. So there's no reason I should have used that myself :(
thanks,
greg k-h
Greg KH
maintainerQuoted 33 lines · Greg KH
Quoted 32 lines · Gui-Dong Han
Quoted 19 lines · Gui-Dong Han
Quoted 9 lines · Danilo Krummrich
Quoted 1 line · Danilo Krummrich
Anyways, this should work:
I Just notied that I pasted the wrong diff, which was nonsense of course, since it just unlocks all the suppressed false positives. (Should not have sent it during a meeting. :)
What I actually intended (not neat, but hopefully helps):
Thanks for the updated diff.
I tested it on my QEMU setup. Since I couldn't reproduce the hang there, I didn't see any lockdep splats regarding the deadlock. However, since the physical lock is removed, my PoCs successfully triggered the UAF on both paths as expected.
Quoted 4 lines
I did notice a lockdep warning during boot, which happens every time. I suspect this is because
faux_bus_initis an__initfunction, so we are registering a key from memory that gets freed. This seems specific to the debug code, but I'm pasting it below for reference.I figured out the root cause.
The warning is triggered because
faux_bus_rootis a static object.lockdep_register_key()has a WARN_ON_ONCE(static_obj(key)) check that forbids registering keys residing in static memory. It is not about__initmemory being freed.
Quoted 6 lines
Quoted 1 line
Anyway, this is not a big deal and doesn't impact the testing results.
Ooh, nice catch. Let me go make that a dynamic object. It really shouldn't be a static one, I hate static struct device usage, and complain about it from everyone else. So there's no reason I should have used that myself :(
Totally untested patch below. Give me a few hours before I can reboot and try this, but if you wish to use it, please do!
From: Greg Kroah-Hartman <[email protected]>
Date: Wed, 21 Jan 2026 09:10:21 +0100
Subject: [PATCH] driver core: faux: stop using static struct device
faux_bus_root should not have been a static struct device, but rather a dynamically created structure so that lockdep and other testing tools do not trip over it (as well as being the right thing overall to do.) Fix this up by making it properly dynamic.
--- drivers/base/faux.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)
drivers/base/faux.c +11 -7
@@ -29,9 +29,7 @@ struct faux_object { };#define to_faux_object (dev) container_of_const(dev, struct faux_object, faux_dev.dev) -static struct device faux_bus_root = { - .init_name ="faux" , -}; +static struct device *faux_bus_root;static int faux_match (struct device *dev,const struct device_driver *drv) {
@@ -152,7 +150,7 @@ struct faux_device *faux_device_create_with_groups(const char *name,if (parent) dev->parent = parent;else - dev->parent = &faux_bus_root; + dev->parent = faux_bus_root; dev->bus = &faux_bus_type;dev_set_name (dev,"%s" , name);device_set_pm_not_required (dev);
@@ -236,9 +234,15 @@ int __init faux_bus_init(void) {int ret; - ret =device_register (&faux_bus_root); + faux_bus_root =kzalloc (sizeof (*faux_bus_root),GFP_KERNEL ); +if (!faux_bus_root) +return -ENOMEM ; + +dev_set_name (faux_bus_root,"faux" ); + + ret =device_register (faux_bus_root);if (ret) { -put_device (&faux_bus_root); +put_device (faux_bus_root);return ret; }
@@ -256,6 +260,6 @@ int __init faux_bus_init(void)bus_unregister (&faux_bus_type); error_bus: -device_unregister (&faux_bus_root); +device_unregister (faux_bus_root);return ret; }
Signature · 1 line
2.52.0
Danilo Krummrich
maintainerQuoted 7 lines · Greg KH
Quoted 6 lines · Gui-Dong Han
The warning is triggered because
faux_bus_rootis a static object.lockdep_register_key()has a WARN_ON_ONCE(static_obj(key)) check that forbids registering keys residing in static memory. It is not about__initmemory being freed.Anyway, this is not a big deal and doesn't impact the testing results.
Sorry, I was aware of this (and that it is harmess for testing purposes) and should have mentioned it so people do not get distracted by it.
Quoted 1 line
Ooh, nice catch. Let me go make that a dynamic object.
Also had it on my list to fix up today. :)
Greg KH
maintainerQuoted 7 lines · Danilo Krummrich
Quoted 6 lines · Greg KH
Quoted 5 lines · Gui-Dong Han
The warning is triggered because
faux_bus_rootis a static object.lockdep_register_key()has a WARN_ON_ONCE(static_obj(key)) check that forbids registering keys residing in static memory. It is not about__initmemory being freed.
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Anyway, this is not a big deal and doesn't impact the testing results.
Sorry, I was aware of this (and that it is harmess for testing purposes) and should have mentioned it so people do not get distracted by it.
Quoted 1 line
Ooh, nice catch. Let me go make that a dynamic object.
Also had it on my list to fix up today. :)
I've tested this and have sent it out for "real" now.
thanks,
greg k-h
Quoted 15 lines · Mark Brown
Quoted 5 lines · Gui-Dong Han
Currently,
driver_match_device()is called from three sites. One site (__device_attach_driver) holdsdevice_lock(dev), but the other two (bind_storeand__driver_attach) do not. This inconsistency means that busmatch()callbacks are not guaranteed to be called with the lock held.I'm seeing boot hangs on Arm Juno in next/pending-fixes which bisect to this commit. The boot grinds to a halt near the end of boot:
[ 2.570549] ledtrig-cpu: registered to indicate activity on CPUs [ 2.618301] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.623547] msm_serial: driver initialized [ 2.624058] SuperH (H)SCI(F) driver initialized [ 2.624312] STM32 USART driver initialized
Quoted 7 lines
with no further output, full log:
https://lava.sirena.org.uk/scheduler/job/2387335#L862We are also seeing similar looking boot hangs on some Qualcomm platforms in Arm's test lab which aren't verified to be the same thing but are hanging at a similar point in boot.
I've observed the same issue on Qualcomm RB5 board and bisecting lead me also to this patch. My kernel log also doesn't reveal much information:
...
[ 3.671227] vreg_bob: Setting 3008000-4000000uV
[ 3.676929] vreg_l1c_1p8: Setting 1800000-1800000uV
[ 3.682826] vreg_l2c_1p2: Setting 1200000-1200000uV
[ 3.688547] vreg_l3c_0p8: Setting 800000-800000uV
[ 3.694080] vreg_l4c_1p7: Setting 1704000-2928000uV
[ 3.699908] vreg_l5c_1p8: Setting 1800000-2928000uV
[ 3.705763] vreg_l6c_2p96: Setting 1800000-2960000uV
[ 3.711684] vreg_l7c_cam_vcm0_2p85: Setting 2856000-3104000uV [ 3.718408] vreg_l8c_1p8: Setting 1800000-1800000uV
[ 3.724287] vreg_l9c_2p96: Setting 2704000-2960000uV
[ 3.730218] vreg_l10c_3p0: Setting 3000000-3000000uV
[ 3.736226] vreg_l11c_3p3: Setting 3296000-3296000uV
[ 3.743413] vreg_s8c_1p3: Setting 1352000-1352000uV
[ 3.771370] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.792020] msm_serial: driver initialized
[ 3.797633] SuperH (H)SCI(F) driver initialized
[ 3.802881] STM32 USART driver initialized
[hang/freeze]
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
Quoted 8 lines · Marek Szyprowski
Quoted 3 lines · Mark Brown
We are also seeing similar looking boot hangs on some Qualcomm platforms in Arm's test lab which aren't verified to be the same thing but are hanging at a similar point in boot.
I've observed the same issue on Qualcomm RB5 board and bisecting lead me also to this patch. My kernel log also doesn't reveal much information:
Yeah, that's one of the boards we're seeing issues on in Arm's test lab.
Jon Hunter
maintainerQuoted 47 lines · Marek Szyprowski
Quoted 15 lines · Mark Brown
Quoted 5 lines · Gui-Dong Han
Currently,
driver_match_device()is called from three sites. One site (__device_attach_driver) holdsdevice_lock(dev), but the other two (bind_storeand__driver_attach) do not. This inconsistency means that busmatch()callbacks are not guaranteed to be called with the lock held.I'm seeing boot hangs on Arm Juno in next/pending-fixes which bisect to this commit. The boot grinds to a halt near the end of boot:
[ 2.570549] ledtrig-cpu: registered to indicate activity on CPUs [ 2.618301] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.623547] msm_serial: driver initialized [ 2.624058] SuperH (H)SCI(F) driver initialized [ 2.624312] STM32 USART driver initializedQuoted 7 lines
with no further output, full log:
https://lava.sirena.org.uk/scheduler/job/2387335#L862We are also seeing similar looking boot hangs on some Qualcomm platforms in Arm's test lab which aren't verified to be the same thing but are hanging at a similar point in boot.
I've observed the same issue on Qualcomm RB5 board and bisecting lead me also to this patch. My kernel log also doesn't reveal much information:
...
[ 3.671227]
vreg_bob: Setting 3008000-4000000uV
[ 3.676929]vreg_l1c_1p8: Setting 1800000-1800000uV
[ 3.682826]vreg_l2c_1p2: Setting 1200000-1200000uV
[ 3.688547]vreg_l3c_0p8: Setting 800000-800000uV
[ 3.694080]vreg_l4c_1p7: Setting 1704000-2928000uV
[ 3.699908]vreg_l5c_1p8: Setting 1800000-2928000uV
[ 3.705763]vreg_l6c_2p96: Setting 1800000-2960000uV
[ 3.711684]vreg_l7c_cam_vcm0_2p85: Setting 2856000-3104000uV [ 3.718408]vreg_l8c_1p8: Setting 1800000-1800000uV
[ 3.724287]vreg_l9c_2p96: Setting 2704000-2960000uV
[ 3.730218]vreg_l10c_3p0: Setting 3000000-3000000uV
[ 3.736226]vreg_l11c_3p3: Setting 3296000-3296000uV
[ 3.743413]vreg_s8c_1p3: Setting 1352000-1352000uV
[ 3.771370] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.792020]msm_serial: driver initialized
[ 3.797633] SuperH (H)SCI(F) driver initialized
[ 3.802881] STM32 USART driver initialized
Quoted 1 line
[hang/freeze]
I am seeing a similar issue on one of our Tegra boards and bisect also points to this commit.
It is odd because it only appears to impact the Tegra194 Jetson Xavier NX board (tegra194-p3509-0000+p3668-0000.dts).
It appears to boot enough so the test can SSH into the device, but the kernel log does not show the us getting to the console prompt. It also appears that a lot of drivers are not bound as expected. I would need to check if those are all modules or not.
Jon
Signature · 1 line
nvpublic
Danilo Krummrich
maintainerQuoted 7 lines · Jon Hunter
It is odd because it only appears to impact the Tegra194 Jetson Xavier NX board (
tegra194-p3509-0000+p3668-0000.dts).It appears to boot enough so the test can SSH into the device, but the kernel log does not show the us getting to the console prompt. It also appears that a lot of drivers are not bound as expected. I would need to check if those are all modules or not.
The other reports were fixed by [1], but the issue in arm-smmu-qcom shouldn't be related in this case.
I quickyl checked all drivers with "tegra194" in their compatible string, but didn't see anything odd.
Can you please try to enable CONFIG_LOCKDEP, CONFIG_PROVE_LOCKING, CONFIG_DEBUG_MUTEXES and see if you get a lockdep splat using the following diff?
(You will see a lockdep warning in faux_bus_init(), it's harmless and can be ignored.)
[1] https://lore.kernel.org/driver-core/[email protected]/
drivers/base/base.h +7 -2
@@ -190,8 +190,13 @@ static inline int driver_match_device(const struct device_driver *drv,static inline int driver_match_device_locked (const struct device_driver *drv,struct device *dev) { -guard (device)(dev); -return driver_match_device (drv, dev); +int ret; + +mutex_acquire (&dev->mutex.dep_map,0 ,0 , _THIS_IP_); + ret =driver_match_device (drv, dev); +mutex_release (&dev->mutex.dep_map, _THIS_IP_); + +return ret; }static inline void dev_sync_state (struct device *dev)
drivers/base/core.c +5 -1
@@ -2557,6 +2557,8 @@ static void device_release(struct kobject *kobj)kfree (dev->dma_range_map); +lockdep_unregister_key (&dev->lock_key); +if (dev->release) dev->release (dev);else if (dev->type && dev->type->release)
@@ -3159,7 +3161,9 @@ void device_initialize(struct device *dev)kobject_init (&dev->kobj, &device_ktype);INIT_LIST_HEAD (&dev->dma_pools);mutex_init (&dev->mutex); -lockdep_set_novalidate_class (&dev->mutex); +//lockdep_set_novalidate_class(&dev->mutex); +lockdep_register_key (&dev->lock_key); +lockdep_set_class (&dev->mutex, &dev->lock_key);spin_lock_init (&dev->devres_lock);INIT_LIST_HEAD (&dev->devres_head);device_pm_init (dev);
include/linux/device.h +1 -0
@@ -579,6 +579,7 @@ struct device {struct mutex mutex;/* mutex to synchronize calls to * its driver. */ +struct lock_class_key lock_key;struct dev_links_info links;struct dev_pm_info power;
Jon Hunter
maintainerHi Danilo,
Quoted 15 lines · Danilo Krummrich
Quoted 7 lines · Jon Hunter
It is odd because it only appears to impact the Tegra194 Jetson Xavier NX board (
tegra194-p3509-0000+p3668-0000.dts).It appears to boot enough so the test can SSH into the device, but the kernel log does not show the us getting to the console prompt. It also appears that a lot of drivers are not bound as expected. I would need to check if those are all modules or not.
The other reports were fixed by [1], but the issue in arm-smmu-qcom shouldn't be related in this case.
I quickyl checked all drivers with "tegra194" in their compatible string, but didn't see anything odd.
Quoted 6 lines
Can you please try to enable
CONFIG_LOCKDEP,CONFIG_PROVE_LOCKING,CONFIG_DEBUG_MUTEXESand see if you get a lockdep splat using the following diff?(You will see a lockdep warning in
faux_bus_init(), it's harmless and can be ignored.)
Thanks. I do the lockdep warning in faux_bus_init() but that's the only one. I have verified that all these CONFIGs are correctly enabled in the build. The device boots fine with the below diff, but I am guessing that that is expected?
Any other thoughts?
Thanks
Jon
Quoted 1 line
[1]
https://lore.kernel.org/driver-core/[email protected]/
Quoted 58 lines
drivers/base/base.h +7
-2@@ -190,8 +190,13 @@ static inline int driver_match_device(const struct device_driver *drv,static inline int driver_match_device_locked (const struct device_driver *drv,struct device *dev) { -guard (device)(dev); -return driver_match_device (drv, dev); +int ret; + +mutex_acquire (&dev->mutex.dep_map,0 ,0 , _THIS_IP_); + ret =driver_match_device (drv, dev); +mutex_release (&dev->mutex.dep_map, _THIS_IP_); + +return ret; }static inline void dev_sync_state (struct device *dev)drivers/base/core.c +5
-1@@ -2557,6 +2557,8 @@ static void device_release(struct kobject *kobj)kfree (dev->dma_range_map); +lockdep_unregister_key (&dev->lock_key); +if (dev->release) dev->release (dev);else if (dev->type && dev->type->release)@@ -3159,7 +3161,9 @@ void device_initialize(struct device *dev)kobject_init (&dev->kobj, &device_ktype);INIT_LIST_HEAD (&dev->dma_pools);mutex_init (&dev->mutex); -lockdep_set_novalidate_class (&dev->mutex); +//lockdep_set_novalidate_class(&dev->mutex); +lockdep_register_key (&dev->lock_key); +lockdep_set_class (&dev->mutex, &dev->lock_key);spin_lock_init (&dev->devres_lock);INIT_LIST_HEAD (&dev->devres_head);device_pm_init (dev);include/linux/device.h +1
-0@@ -579,6 +579,7 @@ struct device {struct mutex mutex;/* mutex to synchronize calls to * its driver. */ +struct lock_class_key lock_key;struct dev_links_info links;struct dev_pm_info power;
Signature · 1 line
nvpublic
Quoted 26 lines · Jon Hunter
Hi Danilo,
Quoted 15 lines · Danilo Krummrich
Quoted 7 lines · Jon Hunter
It is odd because it only appears to impact the Tegra194 Jetson Xavier NX board (
tegra194-p3509-0000+p3668-0000.dts).It appears to boot enough so the test can SSH into the device, but the kernel log does not show the us getting to the console prompt. It also appears that a lot of drivers are not bound as expected. I would need to check if those are all modules or not.
The other reports were fixed by [1], but the issue in arm-smmu-qcom shouldn't be related in this case.
I quickyl checked all drivers with "tegra194" in their compatible string, but didn't see anything odd.
Quoted 6 lines
Can you please try to enable
CONFIG_LOCKDEP,CONFIG_PROVE_LOCKING,CONFIG_DEBUG_MUTEXESand see if you get a lockdep splat using the following diff?(You will see a lockdep warning in
faux_bus_init(), it's harmless and can be ignored.)
Quoted 6 lines
Thanks. I do the lockdep warning in
faux_bus_init()but that's the only one. I have verified that all these CONFIGs are correctly enabled in the build. The device boots fine with the below diff, but I am guessing that that is expected?Any other thoughts?
Can you please try applying the following commit?
Robin Murphy confirmed that the qcom specific issue might actually impact other hardware platforms (provided ARM_SMMU_QCOM/ARCH_QCOM is enabled), as the implementation init code is still executed:
https://lore.kernel.org/driver-core/[email protected]/
So, this patch might fix the issue on Tegra as well.
Thanks.
Danilo Krummrich
maintainerQuoted 24 lines · Gui-Dong Han
Quoted 23 lines · Jon Hunter
Hi Danilo,
Quoted 19 lines · Danilo Krummrich
Quoted 7 lines · Jon Hunter
It is odd because it only appears to impact the Tegra194 Jetson Xavier NX board (
tegra194-p3509-0000+p3668-0000.dts).It appears to boot enough so the test can SSH into the device, but the kernel log does not show the us getting to the console prompt. It also appears that a lot of drivers are not bound as expected. I would need to check if those are all modules or not.
The other reports were fixed by [1], but the issue in arm-smmu-qcom shouldn't be related in this case.
I quickyl checked all drivers with "tegra194" in their compatible string, but didn't see anything odd.
Can you please try to enable
CONFIG_LOCKDEP,CONFIG_PROVE_LOCKING,CONFIG_DEBUG_MUTEXESand see if you get a lockdep splat using the following diff?
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
(You will see a lockdep warning in
faux_bus_init(), it's harmless and can be ignored.)Thanks. I do the lockdep warning in
faux_bus_init()but that's the only one. I have verified that all these CONFIGs are correctly enabled in the build. The device boots fine with the below diff, but I am guessing that that is expected?
Yes, that's expected, we not actually taking the lock, but assert to lockdep that we did. The fact that we use a dynamic lock class key for each device mutex to avoid false positives should also be fine.
Quoted 1 line
Quoted 1 line
Any other thoughts?
With this diff, if I intentionally create a deadlock condition on my machine, I do see a lockdep splat as expected.
Anyways, another option would be to attach a hardware debugger (I assume you have TRACE32 or something available?) and then get a backtrace from the CPU affected of the deadlock.
Quoted 11 lines
Can you please try applying the following commit?
Robin Murphy confirmed that the qcom specific issue might actually impact other hardware platforms (provided ARM_SMMU_QCOM/ARCH_QCOM is enabled), as the implementation init code is still executed:
https://lore.kernel.org/driver-core/[email protected]/
So, this patch might fix the issue on Tegra as well.
I thought of that as well, but looking at the code in arm_smmu_impl_init(), it seems that can't happen?
if (of_device_is_compatible(np, "nvidia,tegra234-smmu") || of_device_is_compatible(np, "nvidia,tegra194-smmu") || of_device_is_compatible(np, "nvidia,tegra186-smmu")) return nvidia_smmu_impl_init(smmu); if (IS_ENABLED(CONFIG_ARM_SMMU_QCOM)) smmu = qcom_smmu_impl_init(smmu);
But maybe there is some odd case where the first if condition does not evaluate to true on tegra194, so maybe worth a try.
Jon Hunter
maintainerOn 22/01/2026 18:12, Danilo Krummrich wrote:
...
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Any other thoughts?
With this diff, if I intentionally create a deadlock condition on my machine, I do see a lockdep splat as expected.
Anyways, another option would be to attach a hardware debugger (I assume you have TRACE32 or something available?) and then get a backtrace from the CPU affected of the deadlock.
Unfortunately, these days I don't have such tools available so that's not an option.
Quoted 23 lines
Quoted 11 lines
Can you please try applying the following commit?
Robin Murphy confirmed that the qcom specific issue might actually impact other hardware platforms (provided ARM_SMMU_QCOM/ARCH_QCOM is enabled), as the implementation init code is still executed:
https://lore.kernel.org/driver-core/[email protected]/
So, this patch might fix the issue on Tegra as well.
I thought of that as well, but looking at the code in
arm_smmu_impl_init(), it seems that can't happen?if (of_device_is_compatible(np, "nvidia,tegra234-smmu") || of_device_is_compatible(np, "nvidia,tegra194-smmu") || of_device_is_compatible(np, "nvidia,tegra186-smmu")) return nvidia_smmu_impl_init(smmu); if (IS_ENABLED(CONFIG_ARM_SMMU_QCOM)) smmu = qcom_smmu_impl_init(smmu);
Quoted 2 lines
But maybe there is some odd case where the first if condition does not evaluate to true on tegra194, so maybe worth a try.
I gave this a shot but that did not help either.
Thanks
Jon
Signature · 1 line
nvpublic
Danilo Krummrich
maintainerQuoted 10 lines · Jon Hunter
Quoted 6 lines · Danilo Krummrich
With this diff, if I intentionally create a deadlock condition on my machine, I do see a lockdep splat as expected.
Anyways, another option would be to attach a hardware debugger (I assume you have TRACE32 or something available?) and then get a backtrace from the CPU affected of the deadlock.
Unfortunately, these days I don't have such tools available so that's not an option.
Hm..slowly running out of options. :)
I remember you previously said that you can still SSH into the machine? If so, can you please share the the first output of
echo l > /proc/sysrq-trigger
directly after booting?
Subsequently, can you please also run
echo w > /proc/sysrq-trigger
and
echo t > /proc/sysrq-trigger
If the output of the last shows a task in D state, you can also run
cat /proc/$PID/stack
Also, are there any OOT modules loaded?
Thanks,
Danilo
Jon Hunter
maintainerQuoted 5 lines · Danilo Krummrich
Quoted 4 lines · Jon Hunter
Quoted 3 lines · Danilo Krummrich
With this diff, if I intentionally create a deadlock condition on my machine, I do see a lockdep splat as expected.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Anyways, another option would be to attach a hardware debugger (I assume you have TRACE32 or something available?) and then get a backtrace from the CPU affected of the deadlock.
Unfortunately, these days I don't have such tools available so that's not an option.
Hm..slowly running out of options. :)
No worries. There appears to be a couple issues going on with this board. With the patch reverted the board boots fine and tests pass. Even in the passing case with this patch reverted, during boot I see a NULL pointer deference crash log from the QSPI driver. So I disabled the QSPI device in device-tree and with this patch the board boots fine and tests pass.
There is a on-going thread for the QSPI driver to fix these NULL pointer deference crashes [0]. So the QSPI driver seems to be the root of the problem.
Cheers
Jon
[0] https://lore.kernel.org/linux-tegra/[email protected]/T/#t
Signature · 1 line
nvpublic
Danilo Krummrich
maintainerQuoted 7 lines · Jon Hunter
No worries. There appears to be a couple issues going on with this board. With the patch reverted the board boots fine and tests pass. Even in the passing case with this patch reverted, during boot I see a NULL pointer deference crash log from the QSPI driver. So I disabled the QSPI device in device-tree and with this patch the board boots fine and tests pass.
Quoted 5 lines
There is a on-going thread for the QSPI driver to fix these NULL pointer deference crashes [0]. So the QSPI driver seems to be the root of the problem.
[0] https://lore.kernel.org/linux-tegra/[email protected]/T/#t
So, are you saying the problems you are seeing are unrelated to this patch and there is no deadlock? (At least this would explain why we couldn't get a lockdep splat with the diff I shared. :)
Otherwise, can you please share the output of the commands I shared in my pevious mail?
Jon Hunter
maintainerQuoted 12 lines · Danilo Krummrich
Quoted 11 lines · Jon Hunter
No worries. There appears to be a couple issues going on with this board. With the patch reverted the board boots fine and tests pass. Even in the passing case with this patch reverted, during boot I see a NULL pointer deference crash log from the QSPI driver. So I disabled the QSPI device in device-tree and with this patch the board boots fine and tests pass.
There is a on-going thread for the QSPI driver to fix these NULL pointer deference crashes [0]. So the QSPI driver seems to be the root of the problem.
Quoted 5 lines
Quoted 1 line
[0] https://lore.kernel.org/linux-tegra/[email protected]/T/#t
So, are you saying the problems you are seeing are unrelated to this patch and there is no deadlock? (At least this would explain why we couldn't get a lockdep splat with the diff I shared. :)
Not exactly. With vanilla -next I see various tests fail on this board and I can see various devices are not probed as expected. Bisect pointed to this patch.
I can fix this by either:
1. Reverting this patch.
2. Disabling the QSPI driver.
Now the QSPI driver has issues which need to be fixed which I am wondering once fix will avoid this problem.
However, I guess regardless of the QSPI issue, should this patch be having such an impact?
Looking at the bootlog [0], you can see the crash is occurring during the tegra_qspi_probe() and so I am guessing this what leads to the deadlock? And may be there is no way to avoid that?
Please note that a lot of the boards I test are in a farm and I don't have direct access. So although I can see the test harness SSH'ing into the board, I am not accessing directly. However, we can run whatever tests we want.
There are no OOT drivers being used this is just vanilla -next.
Jon
[0] https://pastebin.com/wJheruPP
Signature · 1 line
nvpublic
Danilo Krummrich
maintainerQuoted 10 lines · Jon Hunter
I can fix this by either:
1. Reverting this patch.
2. Disabling the QSPI driver.Now the QSPI driver has issues which need to be fixed which I am wondering once fix will avoid this problem.
However, I guess regardless of the QSPI issue, should this patch be having such an impact?
So, this patch by itself is correct, but it reveals when drivers do the wrong thing, that is register drivers from contexts where it neither makes sense nor it is supported by the driver core.
The deadlock happens when a driver (A) registers another driver (B) from a context where the device lock of the device bound to (A) is held, e.g. from bus callbacks, such as probe(). See also [1].
While never valid, the deadlock does only occur when (A) and (B) are on the same bus, e.g. when a platform driver registers another platform driver in its probe() callback.
However, it is a bit more tricky than that: Let's say a platform driver registers an SPI controller, then spi_register_controller() might scan the SPI bus and register SPI devices (not drivers), which are then probed as well. So far this is all fine, but if now in one of the SPI drivers probe() callbacks a platform driver is registered, you have a deadlock condition as well.
So it seems that something of this kind is going on with drivers/spi/spi-tegra210-quad.c.
I did already run quite thorough analysis throughout the whole kernel tree with various static analyzers and also played around with LLMs for finding this pattern.
The tools gave me two results:
(1) The IOMMU one I already fixed [2].
(2) The GPIO driver I posted a patch for in [3].
I specifically also looked for all drivers that are required to run all the peripherals in the tegra194-p3509-0000+p3668-0000.dts hierarchy, but couldn't catch anything.
(This is also why I asked about OOT, because there are quite some compatible strings that are not supported by any upstream driver.)
I think to really see what's going in with spi-tegra210-quad.c, we need the dumps of the sysrq-triggers I provided in a previous mail.
I'd also recommend to pick a stable state of the spi-tegra210-quad.c driver and apply this patch on top (or just apply the spi-tegra210-quad.c fixes as well).
Subsequently, we could try and retest with the diff I provided and the corresponding lockdep options enabled and with the sysrq-triggers (without the diff).
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
Quoted 4 lines
Please note that a lot of the boards I test are in a farm and I don't have direct access. So although I can see the test harness SSH'ing into the board, I am not accessing directly. However, we can run whatever tests we want.
Maybe you can trigger the sysrq-trigger from a custom test?
Quoted 47 lines · Danilo Krummrich
Quoted 10 lines · Jon Hunter
I can fix this by either:
1. Reverting this patch.
2. Disabling the QSPI driver.Now the QSPI driver has issues which need to be fixed which I am wondering once fix will avoid this problem.
However, I guess regardless of the QSPI issue, should this patch be having such an impact?
So, this patch by itself is correct, but it reveals when drivers do the wrong thing, that is register drivers from contexts where it neither makes sense nor it is supported by the driver core.
The deadlock happens when a driver (A) registers another driver (B) from a context where the device lock of the device bound to (A) is held, e.g. from bus callbacks, such as
probe(). See also [1].While never valid, the deadlock does only occur when (A) and (B) are on the same bus, e.g. when a platform driver registers another platform driver in its
probe()callback.However, it is a bit more tricky than that: Let's say a platform driver registers an SPI controller, then
spi_register_controller()might scan the SPI bus and register SPI devices (not drivers), which are then probed as well. So far this is all fine, but if now in one of the SPI driversprobe()callbacks a platform driver is registered, you have a deadlock condition as well.So it seems that something of this kind is going on with
drivers/spi/spi-tegra210-quad.c.I did already run quite thorough analysis throughout the whole kernel tree with various static analyzers and also played around with LLMs for finding this pattern.
The tools gave me two results:
(1) The IOMMU one I already fixed [2].
(2) The GPIO driver I posted a patch for in [3].I specifically also looked for all drivers that are required to run all the peripherals in the
tegra194-p3509-0000+p3668-0000.dtshierarchy, but couldn't catch anything.
Quoted 5 lines
(This is also why I asked about OOT, because there are quite some compatible strings that are not supported by any upstream driver.)
I think to really see what's going in with
spi-tegra210-quad.c, we need the dumps of the sysrq-triggers I provided in a previous mail.
It seems the issue is simpler than a recursive registration deadlock. Looking at the logs, tegra_qspi_probe triggers a NULL pointer dereference (Oops) while holding the device_lock. The mutex likely remains marked as held/orphaned, blocking subsequent driver bindings on the same bus.
This likely explains why lockdep was silent. Since this is not a lock dependency cycle or a recursive locking violation, but rather a lock remaining held by a terminated task, lockdep would not flag it as a deadlock pattern.
This is indeed a side effect of enforcing the lock here—it amplifies the impact of a crash. However, an Oops while holding the device_lock is generally catastrophic regardless.
Following up on our previous discussion [1], refactoring driver_override would resolve this. We could move driver_override to struct device and protect it with a dedicated lock (e.g., driver_override_lock). We would then replace driver_set_override with dev_set_driver_override and add dev_access_driver_override with internal lock assertions. This allows us to remove device_lock from the 2 match paths, reducing contention and preventing a single crash from stalling the whole bus.
However, this deviates from the current paradigm where device_lock protects sysfs attributes (like waiting_for_supplier and power/control). If other sysfs attributes are found to share similar constraints or would benefit from finer-grained locking (which requires further investigation), we might have a stronger argument for introducing a more generic sysfs_lock to handle this class of attributes. We would also need to carefully verify safety during device removal.
Danilo, what are your thoughts on this refactoring plan? I am willing to attempt it, but since it touches the driver core, documentation, and 10+ bus drivers, and I haven't submitted such a large series before, it may take me a few weeks to get an initial version out, and additional time to iterate based on review feedback until it is ready for merging. If you prefer to handle it yourself to expedite things, please let me know so we don't duplicate efforts.
[1] https://lore.kernel.org/all/[email protected]/
Thanks.
Danilo Krummrich
maintainerQuoted 6 lines · Gui-Dong Han
It seems the issue is simpler than a recursive registration deadlock. Looking at the logs,
tegra_qspi_probetriggers a NULL pointer dereference (Oops) while holding thedevice_lock. The mutex likely remains marked as held/orphaned, blocking subsequent driver bindings on the same bus.
Quoted 8 lines
This likely explains why lockdep was silent. Since this is not a lock dependency cycle or a recursive locking violation, but rather a lock remaining held by a terminated task, lockdep would not flag it as a deadlock pattern.
This is indeed a side effect of enforcing the lock here—it amplifies the impact of a crash. However, an Oops while holding the
device_lockis generally catastrophic regardless.
This makes sense to me; it might indeed be as simple as that.
Quoted 18 lines
Following up on our previous discussion [1], refactoring
driver_overridewould resolve this. We could movedriver_overrideto struct device and protect it with a dedicated lock (e.g.,driver_override_lock). We would then replacedriver_set_overridewithdev_set_driver_overrideand adddev_access_driver_overridewith internal lock assertions. This allows us to removedevice_lockfrom the 2 match paths, reducing contention and preventing a single crash from stalling the whole bus.However, this deviates from the current paradigm where
device_lockprotects sysfs attributes (likewaiting_for_supplierand power/control). If other sysfs attributes are found to share similar constraints or would benefit from finer-grained locking (which requires further investigation), we might have a stronger argument for introducing a more genericsysfs_lockto handle this class of attributes. We would also need to carefully verify safety during device removal.
Quoted 7 lines
Danilo, what are your thoughts on this refactoring plan? I am willing to attempt it, but since it touches the driver core, documentation, and 10+ bus drivers, and I haven't submitted such a large series before, it may take me a few weeks to get an initial version out, and additional time to iterate based on review feedback until it is ready for merging. If you prefer to handle it yourself to expedite things, please let me know so we don't duplicate efforts.
I think moving driver_override to struct device and providing accessors with proper lockdep assertions is the correct thing to do. With that, I do not think a separate lock is necessary.
Please feel free to follow up on this.
Jon Hunter
maintainerQuoted 12 lines · Danilo Krummrich
Quoted 11 lines · Gui-Dong Han
It seems the issue is simpler than a recursive registration deadlock. Looking at the logs,
tegra_qspi_probetriggers a NULL pointer dereference (Oops) while holding thedevice_lock. The mutex likely remains marked as held/orphaned, blocking subsequent driver bindings on the same bus.This likely explains why lockdep was silent. Since this is not a lock dependency cycle or a recursive locking violation, but rather a lock remaining held by a terminated task, lockdep would not flag it as a deadlock pattern.
Quoted 5 lines
Quoted 3 lines
This is indeed a side effect of enforcing the lock here—it amplifies the impact of a crash. However, an Oops while holding the
device_lockis generally catastrophic regardless.This makes sense to me; it might indeed be as simple as that.
Yes I believe that this is the case too.
BTW, if I apply the SPI series from Breno [0], which fixes crash in the SPI driver, then everything works fine.
Jon
[0] https://lore.kernel.org/linux-tegra/[email protected]/T/#t
Signature · 1 line
nvpublic
Danilo Krummrich
maintainerQuoted 18 lines · Jon Hunter
Quoted 16 lines · Danilo Krummrich
Quoted 14 lines · Gui-Dong Han
It seems the issue is simpler than a recursive registration deadlock. Looking at the logs,
tegra_qspi_probetriggers a NULL pointer dereference (Oops) while holding thedevice_lock. The mutex likely remains marked as held/orphaned, blocking subsequent driver bindings on the same bus.This likely explains why lockdep was silent. Since this is not a lock dependency cycle or a recursive locking violation, but rather a lock remaining held by a terminated task, lockdep would not flag it as a deadlock pattern.
This is indeed a side effect of enforcing the lock here—it amplifies the impact of a crash. However, an Oops while holding the
device_lockis generally catastrophic regardless.
Quoted 6 lines
Quoted 1 line
This makes sense to me; it might indeed be as simple as that.
Yes I believe that this is the case too.
BTW, if I apply the SPI series from Breno [0], which fixes crash in the SPI driver, then everything works fine.
Thanks for confirming!
- Danilo
Jon Hunter
maintainerHi Danilo,
Quoted 21 lines · Danilo Krummrich
Quoted 10 lines · Jon Hunter
Quoted 6 lines · Danilo Krummrich
With this diff, if I intentionally create a deadlock condition on my machine, I do see a lockdep splat as expected.
Anyways, another option would be to attach a hardware debugger (I assume you have TRACE32 or something available?) and then get a backtrace from the CPU affected of the deadlock.
Unfortunately, these days I don't have such tools available so that's not an option.
Hm..slowly running out of options. :)
I remember you previously said that you can still SSH into the machine? If so, can you please share the the first output of
echo l >
/proc/sysrq-triggerdirectly after booting?
Quoted 7 lines
Subsequently, can you please also run
echo w >
/proc/sysrq-triggerand
echo t >
/proc/sysrq-trigger
You can find the output of the above commands here:
If you search for 'state:D', you can see that various drivers are stuck waiting for the mutex. I believe that this all happens because the SPI driver crashed during the probe and prevents any further drivers from probing.
Jon
Signature · 1 line
nvpublic
Quoted 29 lines · Jon Hunter
Hi Danilo,
Quoted 25 lines · Danilo Krummrich
Quoted 10 lines · Jon Hunter
Quoted 6 lines · Danilo Krummrich
With this diff, if I intentionally create a deadlock condition on my machine, I do see a lockdep splat as expected.
Anyways, another option would be to attach a hardware debugger (I assume you have TRACE32 or something available?) and then get a backtrace from the CPU affected of the deadlock.
Unfortunately, these days I don't have such tools available so that's not an option.
Hm..slowly running out of options. :)
I remember you previously said that you can still SSH into the machine? If so, can you please share the the first output of
echo l >
/proc/sysrq-triggerdirectly after booting?
Subsequently, can you please also run
echo w >
/proc/sysrq-trigger
Quoted 7 lines
Quoted 3 lines
and
echo t >
/proc/sysrq-triggerYou can find the output of the above commands here:
Thanks for the logs.
Looking at the trace, it confirms the previous suspicion. Since there is no circular dependency shown in the logs, it is not a classic recursive deadlock but rather the device_lock remaining held due to the earlier crash in the QSPI driver. This prevented other devices on the same bus from completing their probe.
I'm glad to hear that Breno's SPI fixes resolve the issue. It's a happy ending for this case.
Thanks for the hard work on debugging this!
Quoted 7 lines · Mark Brown
Quoted 5 lines · Gui-Dong Han
Currently,
driver_match_device()is called from three sites. One site (__device_attach_driver) holdsdevice_lock(dev), but the other two (bind_storeand__driver_attach) do not. This inconsistency means that busmatch()callbacks are not guaranteed to be called with the lock held.
Quoted 9 lines
I'm seeing boot hangs on Arm Juno in next/pending-fixes which bisect to this commit. The boot grinds to a halt near the end of boot:
[ 2.570549] ledtrig-cpu: registered to indicate activity on CPUs [ 2.618301] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.623547] msm_serial: driver initialized [ 2.624058] SuperH (H)SCI(F) driver initialized [ 2.624312] STM32 USART driver initialized
Similar outcome on sdm845-based:
- Pixel 3
- OnePlus 6T
Reverting unblock the boot process.
David
Signature · 1 line
David Heidelberg
Hi,
Am Dienstag, 20. Januar 2026, 14:22:19 CET schrieb Mark Brown:
Quoted 16 lines · Gui-Dong Han
Quoted 5 lines
Currently,
driver_match_device()is called from three sites. One site (__device_attach_driver) holdsdevice_lock(dev), but the other two (bind_storeand__driver_attach) do not. This inconsistency means that busmatch()callbacks are not guaranteed to be called with the lock held.I'm seeing boot hangs on Arm Juno in next/pending-fixes which bisect to this commit. The boot grinds to a halt near the end of boot:
[ 2.570549] ledtrig-cpu: registered to indicate activity on CPUs [ 2.618301] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.623547] msm_serial: driver initialized [ 2.624058] SuperH (H)SCI(F) driver initialized [ 2.624312] STM32 USART driver initialized
Quoted 7 lines
with no further output, full log:
https://lava.sirena.org.uk/scheduler/job/2387335#L862We are also seeing similar looking boot hangs on some Qualcomm platforms in Arm's test lab which aren't verified to be the same thing but are hanging at a similar point in boot.
I've also experienced a boot problem on an NXP i.MX8QM based board (DT currently in process of upstreaming). Bisect also resulted in this commit.
Boot suddenly blocks, after about 4 minutes I get a blocked task backtrace:
[ 1.053132] NET: Registered PF_PACKET protocol family [ 1.057958] Key type dns_resolver registered [ 1.070822] registered taskstats version 1 [ 1.074738] Loading compiled-in X.509 certificates [ 1.091208] Demotion targets for Node 0: null [ 1.099632] imx-scu system-controller: NXP i.MX SCU Initialized [ 242.855834] INFO: task swapper/0:1 blocked for more than 120 seconds. [ 242.861994] Not tainted 6.19.0-rc5+ #41 [ 242.866304] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.874122] task:swapper/0 state:D stack:0 pid:1 tgid:1 ppid:0 task_flags:0x0140 flags:0x00000010 [ 242.884981] Call trace: [ 242.887406] __switch_to+0xdc/0x120 (T) [ 242.891233] __schedule+0x310/0x7a0 [ 242.894708] schedule+0x30/0xa4 [ 242.897835] schedule_timeout+0x10c/0x118 [ 242.901831] wait_for_completion+0xd4/0x170 [ 242.906002] __flush_work+0x6c/0xdc [ 242.909476] flush_work+0x2c/0x38 [ 242.912777] deferred_probe_initcall+0x58/0xb0 [ 242.917208] do_one_initcall+0x40/0x24c [ 242.921030] do_initcalls+0xf0/0x1c4 [ 242.924592] kernel_init_freeable+0x114/0x224 [ 242.928936] kernel_init+0x20/0x1d4 [ 242.932413] ret_from_fork+0x10/0x20 [ 242.935990] INFO: task kworker/u16:1:42 blocked for more than 120 seconds. [ 242.942836] Not tainted 6.19.0-rc5+ #41 [ 242.947179] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.955000] task:kworker/u16:1 state:D stack:0 pid:42 tgid:42 ppid:2 task_flags:0x4208060 flags:0x00000010 [ 242.966127] Workqueue: events_unbound deferred_probe_work_func [ 242.971940] Call trace: [ 242.974364] __switch_to+0xdc/0x120 (T) [ 242.978195] __schedule+0x310/0x7a0 [ 242.981669] schedule+0x30/0xa4 [ 242.984803] schedule_preempt_disabled+0x20/0x38 [ 242.989401] __mutex_lock.constprop.0+0x4c4/0xc90 [ 242.994094] __mutex_lock_slowpath+0x10/0x18 [ 242.998349] mutex_lock+0x68/0x74 [ 243.001650] __driver_attach+0x28/0x254 [ 243.005473] bus_for_each_dev+0x74/0xd4 [ 243.009295] driver_attach+0x20/0x28 [ 243.012857] bus_add_driver+0x118/0x238 [ 243.016681] driver_register+0x58/0x114 [ 243.020502] __platform_driver_register+0x20/0x28 [ 243.025193] imx_clk_scu_init+0xe8/0xf8 [ 243.029016] imx8qxp_clk_probe+0x3c/0x11c0 [ 243.033099] platform_probe+0x58/0x90 [ 243.036748] really_probe+0xbc/0x3d4 [ 243.040310] __driver_probe_device+0x7c/0x174 [ 243.044653] driver_probe_device+0x38/0x108 [ 243.048823] __device_attach_driver+0xb0/0x154 [ 243.053254] bus_for_each_drv+0x80/0xdc [ 243.057076] __device_attach+0x9c/0x1bc [ 243.060899] device_initial_probe+0x48/0x4c [ 243.065069] bus_probe_device+0x34/0x98 [ 243.068898] device_add+0x4bc/0x6b4 [ 243.072367] of_device_add+0x40/0x58 [ 243.075929] of_platform_device_create_pdata+0x94/0x158 [ 243.081141] of_platform_bus_create+0xcc/0x280 [ 243.085571] of_platform_populate+0x78/0x174 [ 243.089828] devm_of_platform_populate+0x54/0xb0 [ 243.094435] imx_scu_probe+0x1f8/0x30c [ 243.098169] platform_probe+0x58/0x90 [ 243.101819] really_probe+0xbc/0x3d4 [ 243.105379] __driver_probe_device+0x7c/0x174 [ 243.109728] driver_probe_device+0x38/0x108 [ 243.113896] __device_attach_driver+0xb0/0x154 [ 243.118326] bus_for_each_drv+0x80/0xdc [ 243.122148] __device_attach+0x9c/0x1bc [ 243.125971] device_initial_probe+0x48/0x4c [ 243.130141] bus_probe_device+0x34/0x98 [ 243.133963] deferred_probe_work_func+0x9c/0xf4 [ 243.138480] process_one_work+0x14c/0x3c0 [ 243.142477] worker_thread+0x2dc/0x3f0 [ 243.146212] kthread+0x124/0x1e4 [ 243.149427] ret_from_fork+0x10/0x20 [ 243.152992] INFO: task kworker/u16:1:42 is blocked on a mutex likely owned by task kworker/u16:1:42.
The patches regarding QSPI or SMMU in this thread don't seem to apply for / affect this platform.
Best regards,
Alexander
Signature · 4 lines
TQ-Systems GmbH | Mühlstraße 2, Gut Delling | 82229 Seefeld, Germany Amtsgericht München, HRB 105018 Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider http://www.tq-group.com/
Danilo Krummrich
maintainerQuoted 5 lines · Alexander Stein
Am Dienstag, 20. Januar 2026, 14:22:19 CET schrieb Mark Brown:
Quoted 1 line · Gui-Dong Han
I've also experienced a boot problem on an NXP i.MX8QM based board (DT currently in process of upstreaming). Bisect also resulted in this commit.
Thanks for reporting (and the backtrace, that helps a lot).
This is another case where a platform driver registers another platform driver in probe(). The following diff should fix the issue, I will send out a patch soon.
drivers/clk/imx/clk-imx8qxp.c +23 -1
@@ -346,7 +346,29 @@ static struct platform_driver imx8qxp_clk_driver = { }, .probe = imx8qxp_clk_probe, }; -module_platform_driver (imx8qxp_clk_driver); + +static int __init imx8qxp_init (void ) +{ +int ret; + + ret =platform_driver_register (&imx8qxp_clk_driver); +if (ret) +return ret; + + ret =imx_clk_scu_module_init (); +if (ret) +platform_driver_unregister (&imx8qxp_clk_driver); + +return ret; +} +module_init (imx8qxp_init); + +static void __exit imx8qxp_exit(void) +{ +imx_clk_scu_module_exit (); +platform_driver_unregister (&imx8qxp_clk_driver); +} +module_exit (imx8qxp_exit);MODULE_AUTHOR ("Aisheng Dong <[email protected]>" );MODULE_DESCRIPTION ("NXP i.MX8QXP clock driver" );
drivers/clk/imx/clk-scu.c +11 -1
@@ -191,6 +191,16 @@ static bool imx_scu_clk_is_valid(u32 rsrc_id)return p !=NULL ; } +int __init imx_clk_scu_module_init (void ) +{ +return platform_driver_register (&imx_clk_scu_driver); +} + +void __exit imx_clk_scu_module_exit(void) +{ +return platform_driver_unregister (&imx_clk_scu_driver); +} +int imx_clk_scu_init (struct device_node *np,const struct imx_clk_scu_rsrc_table *data) {
@@ -215,7 +225,7 @@ int imx_clk_scu_init(struct device_node *np, rsrc_table = data; } -return platform_driver_register (&imx_clk_scu_driver); +return 0 ; }/*
drivers/clk/imx/clk-scu.h +2 -0
@@ -25,6 +25,8 @@ extern const struct imx_clk_scu_rsrc_table imx_clk_scu_rsrc_imx8dxl;extern const struct imx_clk_scu_rsrc_table imx_clk_scu_rsrc_imx8qxp;extern const struct imx_clk_scu_rsrc_table imx_clk_scu_rsrc_imx8qm; +int __init imx_clk_scu_module_init (void ); +void __exit imx_clk_scu_module_exit(void);int imx_clk_scu_init (struct device_node *np,const struct imx_clk_scu_rsrc_table *data); struct clk_hw *imx_scu_of_clk_src_get(struct of_phandle_args *clkspec,
Cristian Marussi
reviewerQuoted 10 lines · Gui-Dong Han
Currently,
driver_match_device()is called from three sites. One site (__device_attach_driver) holdsdevice_lock(dev), but the other two (bind_storeand__driver_attach) do not. This inconsistency means that busmatch()callbacks are not guaranteed to be called with the lock held.Fix this by introducing
driver_match_device_locked(), which guarantees holding the device lock using a scoped guard. Replace the unlocked calls inbind_store()and__driver_attach()with this new helper. Also add a lock assertion todriver_match_device()to enforce this guarantee.
Hi,
it has been reported by Gatien (in CC) that this break the SCMI OPTEE transport.
Moreover I still have to verify, BUT I think this also breaks SCMI Virtio transport since both call platform_driver_register() during their probe, since a few years ago the SCMI transports have been reworked to be standalone full-fledged drivers.
I'll have a look in the next days if we can cope with this.
Thanks,
Cristian
Danilo Krummrich
maintainerQuoted 7 lines · Cristian Marussi
it has been reported by Gatien (in CC) that this break the SCMI OPTEE transport.
Moreover I still have to verify, BUT I think this also breaks SCMI Virtio transport since both call
platform_driver_register()during their probe, since a few years ago the SCMI transports have been reworked to be standalone full-fledged drivers.
I've had a quick look and I'm pretty sure that both transports/virtio.c and transports/optee.c are broken.
Both cases look identical and I think the fix should be as trivial as moving platform_driver_register() into module_init().
Quoted 1 line
I'll have a look in the next days if we can cope with this.
I already fixed those three [1,2,3]. If you can provide a patch for both of them, that'd be great. Otherwise, please let me know if you want me to send something.
Thanks,
Danilo
[1] https://lore.kernel.org/driver-core/[email protected]/
[2] https://lore.kernel.org/driver-core/[email protected]/
[3] https://lore.kernel.org/driver-core/[email protected]/
Quoted 9 lines · Danilo Krummrich
Quoted 7 lines · Cristian Marussi
it has been reported by Gatien (in CC) that this break the SCMI OPTEE transport.
Moreover I still have to verify, BUT I think this also breaks SCMI Virtio transport since both call
platform_driver_register()during their probe, since a few years ago the SCMI transports have been reworked to be standalone full-fledged drivers.
Quoted 6 lines
I've had a quick look and I'm pretty sure that both
transports/virtio.candtransports/optee.care broken.Both cases look identical and I think the fix should be as trivial as moving
platform_driver_register()intomodule_init().
Hello,
(Regarding OP-TEE SCMI transport)
It may not be that trivial because unconditionally moving it to module_init() would not guarantee that the TEE bus driver probes before the platform device driver. I think there's a bit more to it.
The platform device driver, as is, is not functional whilst the TEE services are not available. Therefore, it could falsely fail to probe if probed before. We may consider some sort of API to know if the services are available?
Gatien
Quoted 12 lines
Quoted 1 line
I'll have a look in the next days if we can cope with this.
I already fixed those three [1,2,3]. If you can provide a patch for both of them, that'd be great. Otherwise, please let me know if you want me to send something.
Thanks,
Danilo[1] https://lore.kernel.org/driver-core/[email protected]/
[2] https://lore.kernel.org/driver-core/[email protected]/
[3] https://lore.kernel.org/driver-core/[email protected]/
Danilo Krummrich
maintainerQuoted 8 lines · Gatien CHEVALLIER
It may not be that trivial because unconditionally moving it to
module_init()would not guarantee that the TEE bus driver probes before the platform device driver. I think there's a bit more to it.The platform device driver, as is, is not functional whilst the TEE services are not available. Therefore, it could falsely fail to probe if probed before. We may consider some sort of API to know if the services are available?
Ok, so there are three drivers somehow involved:
(1) module_tee_client_driver(scmi_optee_service_driver)
(2) scmi_optee_driver with compatible = "linaro,scmi-optee"
(3) scmi_driver with name = "arm-scmi"
(1) registeres (2) in probe(), and the only thing (2) does is allocating a platform_device with name = "arm-scmi", such that (3) is probed.
So, to me it seems that the indirection through a fake platform device is unnecessary and instead (3) should just have compatible = "linaro,scmi-optee" and return -EPROBE_DEFER if (1) is not ready. Besides that, this seems like a use-case for device links.
Thanks,
Danilo
Cristian Marussi
reviewerQuoted 1 line · Danilo Krummrich
On Thu Feb 26, 2026 at 9:54 AM CET, Gatien CHEVALLIER wrote:
Hi,
I think my mail thread is a bit messed up...my previous reply is nowhere to be found, so I will repeat here...
Quoted 3 lines
Quoted 3 lines
It may not be that trivial because unconditionally moving it to
module_init()would not guarantee that the TEE bus driver probes before the platform device driver. I think there's a bit more to it.
@Gatien
Yes, it is just what I was going to reply...these SCMI transport drivers basically have to probe first against their related subsystem like OPTEE/Virtio and then, only when fully probed at this level, are fully functional and can become a transport supplier for the SCMI core... ...registering the platform driver at the end of the OPTEE/VIRTIO probe allowed to avoid any kind of EPROBE_DEFER handling that can come up espcially when all of this is split into loadable modules..
...I have to review this in deep but I think I had a first version of these transport rework that worked a bit differently...
Quoted 12 lines
Quoted 5 lines
The platform device driver, as is, is not functional whilst the TEE services are not available. Therefore, it could falsely fail to probe if probed before. We may consider some sort of API to know if the services are available?
Ok, so there are three drivers somehow involved:
(1)
module_tee_client_driver(scmi_optee_service_driver)(2)
scmi_optee_driverwith compatible = "linaro,scmi-optee"
Quoted 7 lines
(3)
scmi_driverwith name = "arm-scmi"(1) registeres (2) in
probe(), and the only thing (2) does is allocating aplatform_devicewith name = "arm-scmi", such that (3) is probed.So, to me it seems that the indirection through a fake platform device is unnecessary and instead (3) should just have compatible = "linaro,scmi-optee"
@Danilo
That was the initial state, all the transports and related compatibles embedded into the SCMI core stack...then we split out the transport as loadable standalone drivers, since it seemed to be cleaner (and some vendor also asked for by that time...).
https://lore.kernel.org/arm-scmi/[email protected]/
Quoted 2 lines
and return -EPROBE_DEFER if (1) is not ready. Besides that, this seems like a use-case for device links.
We tried to unify the handling because other transports (mailbox/smc) do NOT rely on 3rd party subsystem to be initialized and this approach indeed simplified all because you dont need any DEFER simply because the SCMI core probing will be kicked off ONLY after at least a transport is successfully probed and initialized...there is a devlink indeed later that classifies the used transport as a supplier of the core SCMI arm-scmi driver...
I have to recall the full details especially of the reqs that led to this design bit I suppose we should at this point reintroduce a bit of EDEFER...
Thanks,
Cristian