Skip to contentSkip to conversation

lockdep warning on thunderbolt docking

DISCUSSION Dominik Brodowski 2 participants 3 messages Aug 30, 2019 – Sep 3, 2019 PCI

lore
Share & tools
lore
#0 lore raw

When connecting a thunderbolt-enabled docking station to my work laptop, the following lockdep warning is reported on v5.3.0-rc6+ as of Thursday morning (can look up the exact git id if so required):

thunderbolt 0-1: new device found, vendor=0xd4 device=0xb070 thunderbolt 0-1: Dell WD19TB Thunderbolt Dock

======================================================
WARNING: possible circular locking dependency detected
5.3.0-rc6+ #1 Tainted: G T
------------------------------------------------------
pool-/usr/lib/b/1258 is trying to acquire lock:
000000005ab0ad43 (pci_rescan_remove_lock){+.+.}, at: authorized_store+0xe8/0x210

but task is already holding lock:
00000000bfb796b5 (&tb->lock){+.+.}, at: authorized_store+0x7c/0x210

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #1 (&tb->lock){+.+.}:

       __mutex_lock+0xac/0x9a0
       tb_domain_add+0x2d/0x130
       nhi_probe+0x1dd/0x330
       pci_device_probe+0xd2/0x150
       really_probe+0xee/0x280
       driver_probe_device+0x50/0xc0
       bus_for_each_drv+0x84/0xd0
       __device_attach+0xe4/0x150
       pci_bus_add_device+0x4e/0x70
       pci_bus_add_devices+0x2e/0x66
       pci_bus_add_devices+0x59/0x66
       pci_bus_add_devices+0x59/0x66
       enable_slot+0x344/0x450
       acpiphp_check_bridge.part.0+0x119/0x150
       acpiphp_hotplug_notify+0xaa/0x140
       acpi_device_hotplug+0xa2/0x3f0
       acpi_hotplug_work_fn+0x1a/0x30
       process_one_work+0x234/0x580
       worker_thread+0x50/0x3b0
       kthread+0x10a/0x140
       ret_from_fork+0x3a/0x50

-> #0 (pci_rescan_remove_lock){+.+.}:

       __lock_acquire+0xe54/0x1ac0
       lock_acquire+0xb8/0x1b0
       __mutex_lock+0xac/0x9a0
       authorized_store+0xe8/0x210
       kernfs_fop_write+0x125/0x1b0
       vfs_write+0xc2/0x1d0
       ksys_write+0x6c/0xf0
       do_syscall_64+0x50/0x180
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

other info that might help us debug this:

 Possible unsafe locking scenario:
       CPU0                    CPU1
       ----                    ----
  lock(&tb->lock);
                               lock(pci_rescan_remove_lock);
                               lock(&tb->lock);
  lock(pci_rescan_remove_lock);

*** DEADLOCK ***
5 locks held by pool-/usr/lib/b/1258:
#0: 000000003df1a1ad (&f->f_pos_lock){+.+.}, at: __fdget_pos+0x4d/0x60 #1: 0000000095a40b02 (sb_writers#6){.+.+}, at: vfs_write+0x185/0x1d0 #2: 0000000017a7d714 (&of->mutex){+.+.}, at: kernfs_fop_write+0xf2/0x1b0 #3: 000000004f262981 (kn->count#208){.+.+}, at: kernfs_fop_write+0xfa/0x1b0 #4: 00000000bfb796b5 (&tb->lock){+.+.}, at: authorized_store+0x7c/0x210

stack backtrace:
CPU: 0 PID: 1258 Comm: pool-/usr/lib/b Tainted: G                T 5.3.0-rc6+ #1

Thanks,
Dominik

#1 lore raw

Hi Dominik,

Quoted 3 lines · Dominik Brodowski

When connecting a thunderbolt-enabled docking station to my work laptop, the following lockdep warning is reported on v5.3.0-rc6+ as of Thursday morning (can look up the exact git id if so required):

Thanks for reporting. No need to dig for the commit ID.

I'll take a look at this next week.

Quoted 74 lines

thunderbolt 0-1: new device found, vendor=0xd4 device=0xb070 thunderbolt 0-1: Dell WD19TB Thunderbolt Dock

======================================================
WARNING: possible circular locking dependency detected
5.3.0-rc6+ #1 Tainted: G T
------------------------------------------------------
pool-/usr/lib/b/1258 is trying to acquire lock:
000000005ab0ad43 (pci_rescan_remove_lock){+.+.}, at: authorized_store+0xe8/0x210

but task is already holding lock:
00000000bfb796b5 (&tb->lock){+.+.}, at: authorized_store+0x7c/0x210

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #1 (&tb->lock){+.+.}:

       __mutex_lock+0xac/0x9a0
       tb_domain_add+0x2d/0x130
       nhi_probe+0x1dd/0x330
       pci_device_probe+0xd2/0x150
       really_probe+0xee/0x280
       driver_probe_device+0x50/0xc0
       bus_for_each_drv+0x84/0xd0
       __device_attach+0xe4/0x150
       pci_bus_add_device+0x4e/0x70
       pci_bus_add_devices+0x2e/0x66
       pci_bus_add_devices+0x59/0x66
       pci_bus_add_devices+0x59/0x66
       enable_slot+0x344/0x450
       acpiphp_check_bridge.part.0+0x119/0x150
       acpiphp_hotplug_notify+0xaa/0x140
       acpi_device_hotplug+0xa2/0x3f0
       acpi_hotplug_work_fn+0x1a/0x30
       process_one_work+0x234/0x580
       worker_thread+0x50/0x3b0
       kthread+0x10a/0x140
       ret_from_fork+0x3a/0x50

-> #0 (pci_rescan_remove_lock){+.+.}:

       __lock_acquire+0xe54/0x1ac0
       lock_acquire+0xb8/0x1b0
       __mutex_lock+0xac/0x9a0
       authorized_store+0xe8/0x210
       kernfs_fop_write+0x125/0x1b0
       vfs_write+0xc2/0x1d0
       ksys_write+0x6c/0xf0
       do_syscall_64+0x50/0x180
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

other info that might help us debug this:

 Possible unsafe locking scenario:
       CPU0                    CPU1
       ----                    ----
  lock(&tb->lock);
                               lock(pci_rescan_remove_lock);
                               lock(&tb->lock);
  lock(pci_rescan_remove_lock);

*** DEADLOCK ***
5 locks held by pool-/usr/lib/b/1258:
#0: 000000003df1a1ad (&f->f_pos_lock){+.+.}, at: __fdget_pos+0x4d/0x60 #1: 0000000095a40b02 (sb_writers#6){.+.+}, at: vfs_write+0x185/0x1d0 #2: 0000000017a7d714 (&of->mutex){+.+.}, at: kernfs_fop_write+0xf2/0x1b0 #3: 000000004f262981 (kn->count#208){.+.+}, at: kernfs_fop_write+0xfa/0x1b0 #4: 00000000bfb796b5 (&tb->lock){+.+.}, at: authorized_store+0x7c/0x210

stack backtrace:
CPU: 0 PID: 1258 Comm: pool-/usr/lib/b Tainted: G                T 5.3.0-rc6+ #1

Thanks,
Dominik

#2 lore raw
Quoted 10 lines · Mika Westerberg

Hi Dominik,

Quoted 3 lines · Dominik Brodowski

When connecting a thunderbolt-enabled docking station to my work laptop, the following lockdep warning is reported on v5.3.0-rc6+ as of Thursday morning (can look up the exact git id if so required):

Thanks for reporting. No need to dig for the commit ID.

I'll take a look at this next week.

This seems to be impossible case. The two code paths cannot run at the same time (on different CPUs) because device authorization is only possible after the domain itself has been added and we've got firmware notification that there is a device connected.

This was added by me in commit a03e828915c0 ("thunderbolt: Serialize PCIe tunnel creation with PCI rescan") claiming that it prevents PCI rescan code to find connected devices too early but now that I have gotten bit more experience in PCIe, I think this is not the case. I think I probably actually saw some issue in PCI stack that may even be fixed already. I'm going to try to reproduce the original issue and see if we can get rid of the whole pci_rescan_remove_lock in the driver.