hitting lockdep warning as of too early VF probe with 3.9-rc1
Subsystems (3)
Hi Ming, Greg, Roland, Dave, all..
With 3.9-rc1, we are hitting the below lockdep with probing of virtual functions over the mlx4 driver, where it seems that the probing of the VF starts before the PF initialization is done.
Yan Burman from our team bisected that to be introduced by commit 190888ac01d059e38ffe77a2291d44cafa9016fb "driver core: fix possible missing of device probe".
Basically what happens is that the VF probe fails, and once the PF probing/initialization is done, the VF is probed again and this time it succeeds.
Anything here which people see to be possibly wrong with the mlx4_core (drivers/net/ethernet/mellanox/mlx4) driver interaction with the PCI subsystem?
Or.
mlx4_core: Initializing 0000:04:00.0 mlx4_core 0000:04:00.0: Enabling SR-IOV with 1 VFs pci 0000:04:00.1: [15b3:1004] type 00 class 0x028000
=============================================
[ INFO: possible recursive locking detected ]
3.9.0-rc1 #96 Not tainted
---------------------------------------------
kworker/0:1/734 is trying to acquire lock:
((&wfc.work)){+.+.+.}, at: [<ffffffff81066cb0>] flush_work+0x0/0x250
but task is already holding lock:
((&wfc.work)){+.+.+.}, at: [<ffffffff81064352>]process_one_work+0x162/0x4c0
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
---- lock((&wfc.work));
lock((&wfc.work));
*** DEADLOCK ***
May be due to missing lock nesting notation
3 locks held by kworker/0:1/734:
#0: (events){.+.+.+}, at: [<ffffffff81064352>]
process_one_work+0x162/0x4c0
#1: ((&wfc.work)){+.+.+.}, at: [<ffffffff81064352>]
process_one_work+0x162/0x4c0
#2: (&__lockdep_no_validate__){......}, at: [<ffffffff812db225>]
device_attach+0x25/0xb0
stack backtrace:
Pid: 734, comm: kworker/0:1 Not tainted 3.9.0-rc1 #96
Call Trace:
[<ffffffff810948ec>] validate_chain+0xdcc/0x11f0
[<ffffffff81095150>] __lock_acquire+0x440/0xc70
[<ffffffff81095150>] ? __lock_acquire+0x440/0xc70
[<ffffffff810959da>] lock_acquire+0x5a/0x70
[<ffffffff81066cb0>] ? wq_worker_waking_up+0x60/0x60
[<ffffffff81066cf5>] flush_work+0x45/0x250
[<ffffffff81066cb0>] ? wq_worker_waking_up+0x60/0x60
[<ffffffff810922be>] ? mark_held_locks+0x9e/0x130
[<ffffffff81066a96>] ? queue_work_on+0x46/0x90
[<ffffffff810925dd>] ? trace_hardirqs_on_caller+0xfd/0x190
[<ffffffff8109267d>] ? trace_hardirqs_on+0xd/0x10
[<ffffffff81066f74>] work_on_cpu+0x74/0x90
[<ffffffff81063820>] ? keventd_up+0x20/0x20
[<ffffffff8121fd30>] ? pci_pm_prepare+0x60/0x60
[<ffffffff811f9293>] ? cpumask_next_and+0x23/0x40
[<ffffffff81220a1a>] pci_device_probe+0xba/0x110
[<ffffffff812dadca>] ? driver_sysfs_add+0x7a/0xb0
[<ffffffff812daf1f>] driver_probe_device+0x8f/0x230
[<ffffffff812db170>] ? __driver_attach+0xb0/0xb0
[<ffffffff812db1bb>] __device_attach+0x4b/0x60
[<ffffffff812d9314>] bus_for_each_drv+0x64/0x90
[<ffffffff812db298>] device_attach+0x98/0xb0
[<ffffffff81218474>] pci_bus_add_device+0x24/0x50
[<ffffffff81232e80>] virtfn_add+0x240/0x3e0
[<ffffffff8146ce3d>] ? _raw_spin_unlock_irqrestore+0x3d/0x80
[<ffffffff812333be>] pci_enable_sriov+0x23e/0x500
[<ffffffffa011fa1a>] __mlx4_init_one+0x5da/0xce0 [mlx4_core]
[<ffffffffa012016d>] mlx4_init_one+0x2d/0x60 [mlx4_core]
[<ffffffff8121fd79>] local_pci_probe+0x49/0x80
[<ffffffff81063833>] work_for_cpu_fn+0x13/0x20
[<ffffffff810643b8>] process_one_work+0x1c8/0x4c0
[<ffffffff81064352>] ? process_one_work+0x162/0x4c0
[<ffffffff81064cfb>] worker_thread+0x30b/0x430
[<ffffffff810649f0>] ? manage_workers+0x340/0x340
[<ffffffff8106cea6>] kthread+0xd6/0xe0
[<ffffffff8106cdd0>] ? __init_kthread_worker+0x70/0x70
[<ffffffff8146daac>] ret_from_fork+0x7c/0xb0
[<ffffffff8106cdd0>] ? __init_kthread_worker+0x70/0x70mlx4_core: Initializing 0000:04:00.1mlx4_core 0000:04:00.1: enabling device (0000 -> 0002)mlx4_core 0000:04:00.1: Detected virtual function - running in slave mode mlx4_core 0000:04:00.1: Sending resetmlx4_core 0000:04:00.1: Got slave FLRed from Communication channel (ret:0x1) mlx4_core 0000:04:00.1: slave is currently in themiddle of FLR. retrying...(try num:1) mlx4_core 0000:04:00.1: Communication channel is not idle.my toggle is 1 (cmd:0x0) mlx4_core 0000:04:00.1: slave is currently in themiddle of FLR. retrying...(try num:2) [... repeated the same ...]mlx4_core 0000:04:00.1: slave is currently in themiddle of FLR. retrying...(try num:10) mlx4_core 0000:04:00.1: Communication channel is not idle.my toggle is 1 (cmd:0x0) mlx4_core 0000:04:00.1: slave driver version is not supported by the master mlx4_core 0000:04:00.1: Communication channel is not idle.my toggle is 1 (cmd:0x0) mlx4_core 0000:04:00.1: Failed to initialize slavemlx4_core: probe of 0000:04:00.1 failed with error -5mlx4_core 0000:04:00.0: Running in master modemlx4_core 0000:04:00.0: FW version 2.11.500 (cmd intf rev 3), max commands 16 mlx4_core 0000:04:00.0: Catastrophic error buffer at 0x1f020, size 0x10, BAR 0 mlx4_core 0000:04:00.0: Communication vector bar:2 offset:0x800
[... probing of PF continues ...]mlx4_core 0000:04:00.0: Started init_resource_tracker: 80 slaves mlx4_core 0000:04:00.0: irq 83 for MSI/MSI-Xmlx4_core 0000:04:00.0: irq 84 for MSI/MSI-Xmlx4_core 0000:04:00.0: irq 85 for MSI/MSI-Xmlx4_core 0000:04:00.0: irq 86 for MSI/MSI-Xmlx4_core 0000:04:00.0: NOP command IRQ test passed
[... probing of PF ends ...]
[... probing of VF done again ...]mlx4_core: Initializing 0000:04:00.1mlx4_core 0000:04:00.1: enabling device (0000 -> 0002)mlx4_core 0000:04:00.1: Detected virtual function - running in slave mode mlx4_core 0000:04:00.1: Sending resetmlx4_core 0000:04:00.0: Received reset from slave:1mlx4_core 0000:04:00.1: Sending vhcr0
[... probing of VF succeeds ...]mlx4_core 0000:04:00.1: NOP command IRQ test passed
Quoted 9 lines · Or Gerlitz
Hi Ming, Greg, Roland, Dave, all..
With 3.9-rc1, we are hitting the below lockdep with probing of virtual functions over the mlx4 driver, where it seems that the probing of the VF starts before the PF initialization is done.
Yan Burman from our team bisected that to be introduced by commit
190888ac01d059e38ffe77a2291d44cafa9016fb
"driver core: fix possible missing of device probe".
I guess that the lockdep warning still can be triggered without the commit.
Quoted 65 lines
Basically what happens is that the VF probe fails, and once the PF probing/initialization is done, the VF
is probed again and this time it succeeds.Anything here which people see to be possibly wrong with the
mlx4_core(drivers/net/ethernet/mellanox/mlx4) driver interaction with the PCI subsystem?Or.
mlx4_core: Initializing 0000:04:00.0mlx4_core0000:04:00.0: Enabling SR-IOV with 1 VFs pci 0000:04:00.1: [15b3:1004] type 00 class0x028000=============================================
[ INFO: possible recursive locking detected ]
3.9.0-rc1 #96 Not tainted
---------------------------------------------
kworker/0:1/734 is trying to acquire lock:
((&wfc.work)){+.+.+.}, at: [<ffffffff81066cb0>]flush_work+0x0/0x250but task is already holding lock:
((&wfc.work)){+.+.+.}, at: [<ffffffff81064352>]process_one_work+0x162/0x4c0other info that might help us debug this:
Possible unsafe locking scenario:CPU0 ----lock((&wfc.work));
lock((&wfc.work));*** DEADLOCK ***
May be due to missing lock nesting notation
3 locks held by kworker/0:1/734:
#0: (events){.+.+.+}, at: [<ffffffff81064352>]process_one_work+0x162/0x4c0 #1: ((&wfc.work)){+.+.+.}, at: [<ffffffff81064352>] process_one_work+0x162/0x4c0 #2: (&__lockdep_no_validate__){......}, at: [<ffffffff812db225>] device_attach+0x25/0xb0 stack backtrace: Pid: 734, comm: kworker/0:1 Not tainted 3.9.0-rc1 #96 Call Trace: [<ffffffff810948ec>] validate_chain+0xdcc/0x11f0 [<ffffffff81095150>] __lock_acquire+0x440/0xc70 [<ffffffff81095150>] ? __lock_acquire+0x440/0xc70 [<ffffffff810959da>] lock_acquire+0x5a/0x70 [<ffffffff81066cb0>] ? wq_worker_waking_up+0x60/0x60 [<ffffffff81066cf5>] flush_work+0x45/0x250 [<ffffffff81066cb0>] ? wq_worker_waking_up+0x60/0x60 [<ffffffff810922be>] ? mark_held_locks+0x9e/0x130 [<ffffffff81066a96>] ? queue_work_on+0x46/0x90 [<ffffffff810925dd>] ? trace_hardirqs_on_caller+0xfd/0x190 [<ffffffff8109267d>] ? trace_hardirqs_on+0xd/0x10 [<ffffffff81066f74>] work_on_cpu+0x74/0x90 [<ffffffff81063820>] ? keventd_up+0x20/0x20 [<ffffffff8121fd30>] ? pci_pm_prepare+0x60/0x60 [<ffffffff811f9293>] ? cpumask_next_and+0x23/0x40 [<ffffffff81220a1a>] pci_device_probe+0xba/0x110
Quoted 8 lines
[<ffffffff812dadca>] ? driver_sysfs_add+0x7a/0xb0 [<ffffffff812daf1f>] driver_probe_device+0x8f/0x230 [<ffffffff812db170>] ? __driver_attach+0xb0/0xb0 [<ffffffff812db1bb>] __device_attach+0x4b/0x60 [<ffffffff812d9314>] bus_for_each_drv+0x64/0x90 [<ffffffff812db298>] device_attach+0x98/0xb0 [<ffffffff81218474>] pci_bus_add_device+0x24/0x50 [<ffffffff81232e80>] virtfn_add+0x240/0x3e0
You are adding one new PCI device inside another PCI device's probe(), so the new device will be probed, since PCI probe() is scheduled by work_on_cpu, then cause flush_work() called inside worker function, which might be a real deadlock.
I am wondering why this commit can cause the problem, since the PCI device will be probed with its driver if there is one driver for it. There is no any limit on when the driver should be loaded into system, either before device is added or after.
Quoted 52 lines
From driver core view, looks no wrong things are found.
[<
ffffffff8146ce3d>] ?_raw_spin_unlock_irqrestore+0x3d/0x80[<ffffffff812333be>] pci_enable_sriov+0x23e/0x500 [<ffffffffa011fa1a>] __mlx4_init_one+0x5da/0xce0 [mlx4_core] [<ffffffffa012016d>] mlx4_init_one+0x2d/0x60 [mlx4_core] [<ffffffff8121fd79>] local_pci_probe+0x49/0x80 [<ffffffff81063833>] work_for_cpu_fn+0x13/0x20 [<ffffffff810643b8>] process_one_work+0x1c8/0x4c0 [<ffffffff81064352>] ? process_one_work+0x162/0x4c0 [<ffffffff81064cfb>] worker_thread+0x30b/0x430 [<ffffffff810649f0>] ? manage_workers+0x340/0x340 [<ffffffff8106cea6>] kthread+0xd6/0xe0 [<ffffffff8106cdd0>] ? __init_kthread_worker+0x70/0x70 [<ffffffff8146daac>] ret_from_fork+0x7c/0xb0 [<ffffffff8106cdd0>] ? __init_kthread_worker+0x70/0x70
mlx4_core: Initializing 0000:04:00.1mlx4_core0000:04:00.1: enabling device (0000 -> 0002)mlx4_core0000:04:00.1: Detected virtual function - running in slave modemlx4_core0000:04:00.1: Sending resetmlx4_core0000:04:00.1: Got slave FLRed from Communication channel (ret:0x1)mlx4_core0000:04:00.1: slave is currently in themiddle of FLR. retrying...(try num:1)mlx4_core0000:04:00.1: Communication channel is not idle.my toggle is 1 (cmd:0x0)mlx4_core0000:04:00.1: slave is currently in themiddle of FLR. retrying...(try num:2)
[... repeated the same ...]mlx4_core0000:04:00.1: slave is currently in themiddle of FLR. retrying...(try num:10)mlx4_core0000:04:00.1: Communication channel is not idle.my toggle is 1 (cmd:0x0)mlx4_core0000:04:00.1: slave driver version is not supported by the mastermlx4_core0000:04:00.1: Communication channel is not idle.my toggle is 1 (cmd:0x0)mlx4_core0000:04:00.1: Failed to initialize slavemlx4_core: probe of 0000:04:00.1 failed with error -5mlx4_core0000:04:00.0: Running in master modemlx4_core0000:04:00.0: FW version 2.11.500 (cmd intf rev 3), max commands 16mlx4_core0000:04:00.0: Catastrophic error buffer at0x1f020, size0x10, BAR 0mlx4_core0000:04:00.0: Communication vector bar:2 offset:0x800 [... probing of PF continues ...]mlx4_core0000:04:00.0: Startedinit_resource_tracker: 80 slavesmlx4_core0000:04:00.0: irq 83 for MSI/MSI-Xmlx4_core0000:04:00.0: irq 84 for MSI/MSI-Xmlx4_core0000:04:00.0: irq 85 for MSI/MSI-Xmlx4_core0000:04:00.0: irq 86 for MSI/MSI-Xmlx4_core0000:04:00.0: NOP command IRQ test passed
[... probing of PF ends ...]
[... probing of VF done again ...]
Quoted 12 lines
mlx4_core: Initializing 0000:04:00.1mlx4_core0000:04:00.1: enabling device (0000 -> 0002)mlx4_core0000:04:00.1: Detected virtual function - running in slave modemlx4_core0000:04:00.1: Sending resetmlx4_core0000:04:00.0: Received reset from slave:1mlx4_core0000:04:00.1: Sending vhcr0
[... probing of VF succeeds ...]mlx4_core0000:04:00.1: NOP command IRQ test passed
Thanks,
--
Ming Lei
Quoted 4 lines · Ming Lei
You are adding one new PCI device inside another PCI device's
probe(), so the new device will be probed, since PCIprobe()is scheduled bywork_on_cpu, then causeflush_work()called inside worker function, which might be a real deadlock.
So if I understand correct, you recommend to somehow avoid this nested probing?
Quoted 4 lines
I am wondering why this commit can cause the problem, since the PCI device will be probed with its driver if there is one driver for it. There is no any limit on when the driver should be loaded into system, either before device is added or after.
FWIW to undertstanding the issue - the same driver (mlx4_core) is used by the PF and VF, so the VF driver is already loaded at the time its been added as new PCI device.
Quoted 1 line
From driver core view, looks no wrong things are found.
So this got me confused, you pointed on possible deadlock, are you saying the deadlock wouldn't be the result of how the driver code is going nor the commited we bisected?
Or.
Quoted 7 lines · Or Gerlitz
Quoted 4 lines · Ming Lei
You are adding one new PCI device inside another PCI device's
probe(), so the new device will be probed, since PCIprobe()is scheduled bywork_on_cpu, then causeflush_work()called inside worker function, which might be a real deadlock.So if I understand correct, you recommend to somehow avoid this nested probing?
Yes, you might need to avoid the nested probing in your driver.
Quoted 10 lines
Quoted 5 lines
I am wondering why this commit can cause the problem, since the PCI device will be probed with its driver if there is one driver for it. There is no any limit on when the driver should be loaded into system, either before device is added or after.
FWIW to undertstanding the issue - the same driver (
mlx4_core) is used by the PF and VF, so the VF driver is already loaded at the time its been added as new PCI device.
Quoted 5 lines
Quoted 1 line
From driver core view, looks no wrong things are found.
So this got me confused, you pointed on possible deadlock, are you saying the deadlock wouldn't be the result of how the driver code is going nor the commited we bisected?
My commit only affects the driver loading path, but your warning is hit in driver probe path triggered by device addition, so the lockdep warning should still be triggered without my commit since the two paths are totally independent, right?
Thanks,
--
Ming Lei
Hello, Ming, Greg, Roland, Dave, all...
Quoted 1 line
From a quick scan of ethernet drivers in Dave Miller's net-next git, I
notice that the following drivers (apart from the Mellanox mlx4 driver) enable SRIOV during the PF probe:
cisco enic (function "enic_probe")
neterion vxge driver(function "vxge_probe")
Solarflare efx driver (function "efx_pci_probe", which invokes "efx_sriov_init") emulex driver (function "be_probe" --> be_setup --> be_vf_setup)
It would seem that these drivers are susceptible to the nested probe/deadlock race condition as well.
I believe that it is healthiest for everyone if the probe code in the kernel itself would avoid such nested probe calls (rather than forcing vendors to deal with this issue). The kernel code is certainly aware
(or could easily track) that it is invoking the a driver's probe function while that same probe function has already been invoked and has not yet returned!
-Jack
Quoted 36 lines · Ming Lei
Quoted 7 lines · Or Gerlitz
Quoted 4 lines · Ming Lei
You are adding one new PCI device inside another PCI device's
probe(), so the new device will be probed, since PCIprobe()is scheduled bywork_on_cpu, then causeflush_work()called inside worker function, which might be a real deadlock.So if I understand correct, you recommend to somehow avoid this nested probing?
Yes, you might need to avoid the nested probing in your driver.
Quoted 10 lines
Quoted 5 lines
I am wondering why this commit can cause the problem, since the PCI device will be probed with its driver if there is one driver for it. There is no any limit on when the driver should be loaded into system, either before device is added or after.
FWIW to undertstanding the issue - the same driver (
mlx4_core) is used by the PF and VF, so the VF driver is already loaded at the time its been added as new PCI device.Quoted 5 lines
Quoted 1 line
From driver core view, looks no wrong things are found.
So this got me confused, you pointed on possible deadlock, are you saying the deadlock wouldn't be the result of how the driver code is going nor the commited we bisected?
My commit only affects the driver loading path, but your warning is hit in driver probe path triggered by device addition, so the lockdep warning should still be triggered without my commit since the two paths are totally independent, right?
Thanks,
--
Ming Lei
Quoted 10 lines · Jack Morgenstein
Hello, Ming, Greg, Roland, Dave, all...
From a quick scan of ethernet drivers in Dave Miller's net-next git, I notice that the following drivers (apart from the Mellanox mlx4 driver) enable SRIOV during the PF probe:
cisco enic (function "enic_probe")
neterion vxgedriver(function "vxge_probe")
Solarflare efx driver (function "efx_pci_probe", which invokes "efx_sriov_init") emulex driver (function "be_probe" -->be_setup-->be_vf_setup)
Quoted 8 lines
It would seem that these drivers are susceptible to the nested probe/deadlock race condition as well.
I believe that it is healthiest for everyone if the probe code in the kernel itself would avoid such nested probe calls (rather than forcing vendors to deal with this issue). The kernel code is certainly aware
(or could easily track) that it is invoking the a driver's probe function while that same probe function has already been invoked and has not yet returned!
Patches to handle this are always gladly accepted.
thanks,
greg k-h
Quoted 10 lines · Jack Morgenstein
Hello, Ming, Greg, Roland, Dave, all...
From a quick scan of ethernet drivers in Dave Miller's net-next git, I notice that the following drivers (apart from the Mellanox mlx4 driver) enable SRIOV during the PF probe:
cisco enic (function "enic_probe")
neterion vxgedriver(function "vxge_probe")
Solarflare efx driver (function "efx_pci_probe", which invokes "efx_sriov_init") emulex driver (function "be_probe" -->be_setup-->be_vf_setup)
Quoted 8 lines
It would seem that these drivers are susceptible to the nested probe/deadlock race condition as well.
I believe that it is healthiest for everyone if the probe code in the kernel itself would avoid such nested probe calls (rather than forcing vendors to deal with this issue). The kernel code is certainly aware
(or could easily track) that it is invoking the a driver's probe function while that same probe function has already been invoked and has not yet returned!
Generally nested probe() should be fine if your driver knows how to end the nesting, also USB drivers use nested probe too, which is healthy. This report is only related with PCI driver probe, which is scheduled as work function, so flush_work() around nested probe() may cause deadlock.
IMO, fixing the problem generally might need to touch PCI driver probe code(pci_call_probe).
Thanks,
--
Ming Lei
Quoted 8 lines · Jack Morgenstein
Hello, Ming, Greg, Roland, Dave, all...
From a quick scan of ethernet drivers in Dave Miller's net-next git, I notice that the following drivers (apart from the Mellanox mlx4 driver) enable SRIOV during the PF probe:
cisco enic (function "enic_probe")
neterion vxgedriver(function "vxge_probe")
Solarflare efx driver (function "efx_pci_probe", which invokes "efx_sriov_init")
It's actually called 'sfc', despite using the 'efx' prefix for hysterical raisins.
Quoted 10 lines
emulex driver (function "
be_probe" -->be_setup-->be_vf_setup)It would seem that these drivers are susceptible to the nested probe/deadlock race condition as well.
I believe that it is healthiest for everyone if the probe code in the kernel itself would avoid such nested probe calls (rather than forcing vendors to deal with this issue). The kernel code is certainly aware
(or could easily track) that it is invoking the a driver's probe function while that same probe function has already been invoked and has not yet returned!
[...]
Does it make a difference whether the VF driver is the same code as the PF driver? I suspect not.
Since Donald Dutile's changes to allow setting the number of VFs through sysfs, PF drivers should not need to enable VFs during their own probe function any more. (Though I think that it will take a non-trivial amount of work to change that in sfc, anyway.) This should avoid the apparent problems with nested probes.
By the way, since VF net drivers will acquire the RTNL lock during their probe functions, the PF net driver must not hold the RTNL lock while calling pci_enable_sriov(). This allows rtnl_vfinfo_size() and rtnl_vf_ports_fill() to race with the change to number of VFs enabled, if the PF's net device is already registered - which will definitely be the case if the change is being made through the new sysfs method.
Ben.
Signature · 3 lines
Ben Hutchings, Staff Engineer, Solarflare Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
Quoted 17 lines · Ming Lei
Quoted 6 lines · Or Gerlitz
Quoted 4 lines · Ming Lei
You are adding one new PCI device inside another PCI device's
probe(), so the new device will be probed, since PCIprobe()is scheduled bywork_on_cpu, then causeflush_work()called inside worker function, which might be a real deadlock.So if I understand correct, you recommend to somehow avoid this nested probing?
Yes, you might need to avoid the nested probing in your driver.
Quoted 8 lines
Quoted 4 lines
I am wondering why this commit can cause the problem, since the PCI device will be probed with its driver if there is one driver for it. There is no any limit on when the driver should be loaded into system, either before device is added or after.
FWIW to undertstanding the issue - the same driver (
mlx4_core) is used by the PF and VF, so the VF driver is already loaded at the time its been added as new PCI device.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
From driver core view, looks no wrong things are found.
So this got me confused, you pointed on possible deadlock, are you saying the deadlock wouldn't be the result of how the driver code is going nor the commited we bisected?
My commit only affects the driver loading path, but your warning is hit in driver probe path triggered by device addition, so the lockdep warning should still be triggered without my commit since the two paths are totally independent, right?
If this is the case, I am not sure how come we never saw this warning before 3.9-rc1 which introduced your commit 190888ac01d059e38ffe77a2291d44cafa9016fb "driver core: fix possible missing of device probe", any rough guess?
PATCH 1/1 pci: avoid work_on_cpu for nested SRIOV probes
In reply to Or GerlitzThe following lockdep report triggers since 3.9-rc1:
=============================================
[ INFO: possible recursive locking detected ]
3.9.0-rc1 #96 Not tainted
---------------------------------------------
kworker/0:1/734 is trying to acquire lock:
((&wfc.work)){+.+.+.}, at: [<ffffffff81066cb0>] flush_work+0x0/0x250
but task is already holding lock:
((&wfc.work)){+.+.+.}, at: [<ffffffff81064352>]process_one_work+0x162/0x4c0
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
---- lock((&wfc.work));
lock((&wfc.work));
*** DEADLOCK ***
May be due to missing lock nesting notation
3 locks held by kworker/0:1/734:
#0: (events){.+.+.+}, at: [<ffffffff81064352>]
process_one_work+0x162/0x4c0
#1: ((&wfc.work)){+.+.+.}, at: [<ffffffff81064352>]
process_one_work+0x162/0x4c0
#2: (&__lockdep_no_validate__){......}, at: [<ffffffff812db225>]
device_attach+0x25/0xb0
stack backtrace:
Pid: 734, comm: kworker/0:1 Not tainted 3.9.0-rc1 #96
Call Trace:
[<ffffffff810948ec>] validate_chain+0xdcc/0x11f0
[<ffffffff81095150>] __lock_acquire+0x440/0xc70
[<ffffffff81095150>] ? __lock_acquire+0x440/0xc70
[<ffffffff810959da>] lock_acquire+0x5a/0x70
[<ffffffff81066cb0>] ? wq_worker_waking_up+0x60/0x60
[<ffffffff81066cf5>] flush_work+0x45/0x250
[<ffffffff81066cb0>] ? wq_worker_waking_up+0x60/0x60
[<ffffffff810922be>] ? mark_held_locks+0x9e/0x130
[<ffffffff81066a96>] ? queue_work_on+0x46/0x90
[<ffffffff810925dd>] ? trace_hardirqs_on_caller+0xfd/0x190
[<ffffffff8109267d>] ? trace_hardirqs_on+0xd/0x10
[<ffffffff81066f74>] work_on_cpu+0x74/0x90
[<ffffffff81063820>] ? keventd_up+0x20/0x20
[<ffffffff8121fd30>] ? pci_pm_prepare+0x60/0x60
[<ffffffff811f9293>] ? cpumask_next_and+0x23/0x40
[<ffffffff81220a1a>] pci_device_probe+0xba/0x110
[<ffffffff812dadca>] ? driver_sysfs_add+0x7a/0xb0
[<ffffffff812daf1f>] driver_probe_device+0x8f/0x230
[<ffffffff812db170>] ? __driver_attach+0xb0/0xb0
[<ffffffff812db1bb>] __device_attach+0x4b/0x60
[<ffffffff812d9314>] bus_for_each_drv+0x64/0x90
[<ffffffff812db298>] device_attach+0x98/0xb0
[<ffffffff81218474>] pci_bus_add_device+0x24/0x50
[<ffffffff81232e80>] virtfn_add+0x240/0x3e0
[<ffffffff8146ce3d>] ? _raw_spin_unlock_irqrestore+0x3d/0x80
[<ffffffff812333be>] pci_enable_sriov+0x23e/0x500
[<ffffffffa011fa1a>] __mlx4_init_one+0x5da/0xce0 [mlx4_core]
[<ffffffffa012016d>] mlx4_init_one+0x2d/0x60 [mlx4_core]
[<ffffffff8121fd79>] local_pci_probe+0x49/0x80
[<ffffffff81063833>] work_for_cpu_fn+0x13/0x20
[<ffffffff810643b8>] process_one_work+0x1c8/0x4c0
[<ffffffff81064352>] ? process_one_work+0x162/0x4c0
[<ffffffff81064cfb>] worker_thread+0x30b/0x430
[<ffffffff810649f0>] ? manage_workers+0x340/0x340
[<ffffffff8106cea6>] kthread+0xd6/0xe0
[<ffffffff8106cdd0>] ? __init_kthread_worker+0x70/0x70
[<ffffffff8146daac>] ret_from_fork+0x7c/0xb0
[<ffffffff8106cdd0>] ? __init_kthread_worker+0x70/0x70The issue is that a driver, in it's probe function, calls pci_sriov_enable so a PF device probe causes VF probe (AKA nested probe). Each probe in pci_device_probe which is (normally) run through work_on_cpu (this is to get the right numa node for memory allocated by the driver). In turn work_on_cpu does this internally:
schedule_work_on(cpu, &wfc.work);
flush_work(&wfc.work);So if you are running probe on CPU1, and cause another probe on the same CPU, this will try to flush workqueue from inside same workqueue which of course deadlocks.
Nested probing might be tricky to get right generally.
But for pci_sriov_enable, the situation is actually very simple: all VFs naturally have same affinity as the PF, and cpumask_any_and is actually same as cpumask_first_and, so it always gives us the same CPU. So let's just detect that, and run the probing for VFs locally without a workqueue.
This is hardly elegant, but looks to me like an appropriate quick fix for 3.9.
---
drivers/pci/pci-driver.c +1 -1
@@ -286,8 +286,8 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,int cpu;get_online_cpus (); cpu =cpumask_any_and (cpumask_of_node (node), cpu_online_mask); -if (cpu < nr_cpu_ids) +if (cpu !=raw_smp_processor_id () && cpu < nr_cpu_ids) error =work_on_cpu (cpu, local_pci_probe, &ddi);else error =local_pci_probe (&ddi);
PATCH v2 1/1 pci: avoid work_on_cpu for nested SRIOV probes
In reply to Or GerlitzThe following lockdep report triggers since 3.9-rc1:
=============================================
[ INFO: possible recursive locking detected ]
3.9.0-rc1 #96 Not tainted
---------------------------------------------
kworker/0:1/734 is trying to acquire lock:
((&wfc.work)){+.+.+.}, at: [<ffffffff81066cb0>] flush_work+0x0/0x250
but task is already holding lock:
((&wfc.work)){+.+.+.}, at: [<ffffffff81064352>]process_one_work+0x162/0x4c0
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
---- lock((&wfc.work));
lock((&wfc.work));
*** DEADLOCK ***
May be due to missing lock nesting notation
3 locks held by kworker/0:1/734:
#0: (events){.+.+.+}, at: [<ffffffff81064352>]
process_one_work+0x162/0x4c0
#1: ((&wfc.work)){+.+.+.}, at: [<ffffffff81064352>]
process_one_work+0x162/0x4c0
#2: (&__lockdep_no_validate__){......}, at: [<ffffffff812db225>]
device_attach+0x25/0xb0
stack backtrace:
Pid: 734, comm: kworker/0:1 Not tainted 3.9.0-rc1 #96
Call Trace:
[<ffffffff810948ec>] validate_chain+0xdcc/0x11f0
[<ffffffff81095150>] __lock_acquire+0x440/0xc70
[<ffffffff81095150>] ? __lock_acquire+0x440/0xc70
[<ffffffff810959da>] lock_acquire+0x5a/0x70
[<ffffffff81066cb0>] ? wq_worker_waking_up+0x60/0x60
[<ffffffff81066cf5>] flush_work+0x45/0x250
[<ffffffff81066cb0>] ? wq_worker_waking_up+0x60/0x60
[<ffffffff810922be>] ? mark_held_locks+0x9e/0x130
[<ffffffff81066a96>] ? queue_work_on+0x46/0x90
[<ffffffff810925dd>] ? trace_hardirqs_on_caller+0xfd/0x190
[<ffffffff8109267d>] ? trace_hardirqs_on+0xd/0x10
[<ffffffff81066f74>] work_on_cpu+0x74/0x90
[<ffffffff81063820>] ? keventd_up+0x20/0x20
[<ffffffff8121fd30>] ? pci_pm_prepare+0x60/0x60
[<ffffffff811f9293>] ? cpumask_next_and+0x23/0x40
[<ffffffff81220a1a>] pci_device_probe+0xba/0x110
[<ffffffff812dadca>] ? driver_sysfs_add+0x7a/0xb0
[<ffffffff812daf1f>] driver_probe_device+0x8f/0x230
[<ffffffff812db170>] ? __driver_attach+0xb0/0xb0
[<ffffffff812db1bb>] __device_attach+0x4b/0x60
[<ffffffff812d9314>] bus_for_each_drv+0x64/0x90
[<ffffffff812db298>] device_attach+0x98/0xb0
[<ffffffff81218474>] pci_bus_add_device+0x24/0x50
[<ffffffff81232e80>] virtfn_add+0x240/0x3e0
[<ffffffff8146ce3d>] ? _raw_spin_unlock_irqrestore+0x3d/0x80
[<ffffffff812333be>] pci_enable_sriov+0x23e/0x500
[<ffffffffa011fa1a>] __mlx4_init_one+0x5da/0xce0 [mlx4_core]
[<ffffffffa012016d>] mlx4_init_one+0x2d/0x60 [mlx4_core]
[<ffffffff8121fd79>] local_pci_probe+0x49/0x80
[<ffffffff81063833>] work_for_cpu_fn+0x13/0x20
[<ffffffff810643b8>] process_one_work+0x1c8/0x4c0
[<ffffffff81064352>] ? process_one_work+0x162/0x4c0
[<ffffffff81064cfb>] worker_thread+0x30b/0x430
[<ffffffff810649f0>] ? manage_workers+0x340/0x340
[<ffffffff8106cea6>] kthread+0xd6/0xe0
[<ffffffff8106cdd0>] ? __init_kthread_worker+0x70/0x70
[<ffffffff8146daac>] ret_from_fork+0x7c/0xb0
[<ffffffff8106cdd0>] ? __init_kthread_worker+0x70/0x70The issue is that a driver, in it's probe function, calls pci_sriov_enable so a PF device probe causes VF probe (AKA nested probe). Each probe in pci_device_probe which is (normally) run through work_on_cpu (this is to get the right numa node for memory allocated by the driver). In turn work_on_cpu does this internally:
schedule_work_on(cpu, &wfc.work);
flush_work(&wfc.work);So if you are running probe on CPU1, and cause another probe on the same CPU, this will try to flush workqueue from inside same workqueue which causes a lockep warning.
Nested probing might be tricky to get right generally.
But for pci_sriov_enable, the situation is actually very simple: all VFs naturally have same affinity as the PF, and cpumask_any_and is actually same as cpumask_first_and, so it always gives us the same CPU. So let's just detect that, and run the probing for VFs locally without a workqueue.
This is hardly elegant, but looks to me like an appropriate quick fix for 3.9.
---
Changes from v1:
- clarified commit log and added Ack by Tejun Heo patch is unchanged.
drivers/pci/pci-driver.c +1 -1
@@ -286,8 +286,8 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,int cpu;get_online_cpus (); cpu =cpumask_any_and (cpumask_of_node (node), cpu_online_mask); -if (cpu < nr_cpu_ids) +if (cpu !=raw_smp_processor_id () && cpu < nr_cpu_ids) error =work_on_cpu (cpu, local_pci_probe, &ddi);else error =local_pci_probe (&ddi);
Bjorn Helgaas
maintainerRe: [PATCHv2 for-3.9] pci: avoid work_on_cpu for nested SRIOV probes
Quoted 97 lines · Michael S. Tsirkin
The following lockdep report triggers since 3.9-rc1:
=============================================
[ INFO: possible recursive locking detected ]
3.9.0-rc1 #96 Not tainted
---------------------------------------------
kworker/0:1/734 is trying to acquire lock:
((&wfc.work)){+.+.+.}, at: [<ffffffff81066cb0>]flush_work+0x0/0x250but task is already holding lock:
((&wfc.work)){+.+.+.}, at: [<ffffffff81064352>]process_one_work+0x162/0x4c0other info that might help us debug this:
Possible unsafe locking scenario:CPU0 ----lock((&wfc.work));
lock((&wfc.work));*** DEADLOCK ***
May be due to missing lock nesting notation
3 locks held by kworker/0:1/734:
#0: (events){.+.+.+}, at: [<ffffffff81064352>]process_one_work+0x162/0x4c0 #1: ((&wfc.work)){+.+.+.}, at: [<ffffffff81064352>] process_one_work+0x162/0x4c0 #2: (&__lockdep_no_validate__){......}, at: [<ffffffff812db225>] device_attach+0x25/0xb0 stack backtrace: Pid: 734, comm: kworker/0:1 Not tainted 3.9.0-rc1 #96 Call Trace: [<ffffffff810948ec>] validate_chain+0xdcc/0x11f0 [<ffffffff81095150>] __lock_acquire+0x440/0xc70 [<ffffffff81095150>] ? __lock_acquire+0x440/0xc70 [<ffffffff810959da>] lock_acquire+0x5a/0x70 [<ffffffff81066cb0>] ? wq_worker_waking_up+0x60/0x60 [<ffffffff81066cf5>] flush_work+0x45/0x250 [<ffffffff81066cb0>] ? wq_worker_waking_up+0x60/0x60 [<ffffffff810922be>] ? mark_held_locks+0x9e/0x130 [<ffffffff81066a96>] ? queue_work_on+0x46/0x90 [<ffffffff810925dd>] ? trace_hardirqs_on_caller+0xfd/0x190 [<ffffffff8109267d>] ? trace_hardirqs_on+0xd/0x10 [<ffffffff81066f74>] work_on_cpu+0x74/0x90 [<ffffffff81063820>] ? keventd_up+0x20/0x20 [<ffffffff8121fd30>] ? pci_pm_prepare+0x60/0x60 [<ffffffff811f9293>] ? cpumask_next_and+0x23/0x40 [<ffffffff81220a1a>] pci_device_probe+0xba/0x110 [<ffffffff812dadca>] ? driver_sysfs_add+0x7a/0xb0 [<ffffffff812daf1f>] driver_probe_device+0x8f/0x230 [<ffffffff812db170>] ? __driver_attach+0xb0/0xb0 [<ffffffff812db1bb>] __device_attach+0x4b/0x60 [<ffffffff812d9314>] bus_for_each_drv+0x64/0x90 [<ffffffff812db298>] device_attach+0x98/0xb0 [<ffffffff81218474>] pci_bus_add_device+0x24/0x50 [<ffffffff81232e80>] virtfn_add+0x240/0x3e0 [<ffffffff8146ce3d>] ? _raw_spin_unlock_irqrestore+0x3d/0x80 [<ffffffff812333be>] pci_enable_sriov+0x23e/0x500 [<ffffffffa011fa1a>] __mlx4_init_one+0x5da/0xce0 [mlx4_core] [<ffffffffa012016d>] mlx4_init_one+0x2d/0x60 [mlx4_core] [<ffffffff8121fd79>] local_pci_probe+0x49/0x80 [<ffffffff81063833>] work_for_cpu_fn+0x13/0x20 [<ffffffff810643b8>] process_one_work+0x1c8/0x4c0 [<ffffffff81064352>] ? process_one_work+0x162/0x4c0 [<ffffffff81064cfb>] worker_thread+0x30b/0x430 [<ffffffff810649f0>] ? manage_workers+0x340/0x340 [<ffffffff8106cea6>] kthread+0xd6/0xe0 [<ffffffff8106cdd0>] ? __init_kthread_worker+0x70/0x70 [<ffffffff8146daac>] ret_from_fork+0x7c/0xb0 [<ffffffff8106cdd0>] ? __init_kthread_worker+0x70/0x70The issue is that a driver, in it's probe function, calls
pci_sriov_enableso a PF device probe causes VF probe (AKA nested probe). Each probe inpci_device_probewhich is (normally) run throughwork_on_cpu(this is to get the right numa node for memory allocated by the driver). In turnwork_on_cpudoes this internally:schedule_work_on(cpu, &wfc.work); flush_work(&wfc.work);So if you are running probe on CPU1, and cause another probe on the same CPU, this will try to flush workqueue from inside same workqueue which causes a lockep warning.
Nested probing might be tricky to get right generally.
But for
pci_sriov_enable, the situation is actually very simple: all VFs naturally have same affinity as the PF, andcpumask_any_andis actually same ascpumask_first_and, so it always gives us the same CPU. So let's just detect that, and run the probing for VFs locally without a workqueue.
Quoted 6 lines
This is hardly elegant, but looks to me like an appropriate quick fix for 3.9.
Tested-by Or Gerlitz <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>Acked-by Tejun Heo <[email protected]>
Thanks, Michael. I put this in my for-linus branch:
http://git.kernel.org/cgit/linux/kernel/git/helgaas/pci.git/log/?h=for-linus
I'll send a pull request to Linus today.
Bjorn
Quoted 20 lines
---
Changes from v1:
- clarified commit log and added Ack by Tejun Heo patch is unchanged.drivers/pci/pci-driver.c +1
-1@@ -286,8 +286,8 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,int cpu;get_online_cpus (); cpu =cpumask_any_and (cpumask_of_node (node), cpu_online_mask); -if (cpu < nr_cpu_ids) +if (cpu !=raw_smp_processor_id () && cpu < nr_cpu_ids) error =work_on_cpu (cpu, local_pci_probe, &ddi);else error =local_pci_probe (&ddi);
Bjorn Helgaas
maintainerRe: [PATCHv2 for-3.9] pci: avoid work_on_cpu for nested SRIOV probes
Quoted 105 lines · Bjorn Helgaas
Quoted 97 lines · Michael S. Tsirkin
The following lockdep report triggers since 3.9-rc1:
=============================================
[ INFO: possible recursive locking detected ]
3.9.0-rc1 #96 Not tainted
---------------------------------------------
kworker/0:1/734 is trying to acquire lock:
((&wfc.work)){+.+.+.}, at: [<ffffffff81066cb0>]flush_work+0x0/0x250but task is already holding lock:
((&wfc.work)){+.+.+.}, at: [<ffffffff81064352>]process_one_work+0x162/0x4c0other info that might help us debug this:
Possible unsafe locking scenario:CPU0 ----lock((&wfc.work));
lock((&wfc.work));*** DEADLOCK ***
May be due to missing lock nesting notation
3 locks held by kworker/0:1/734:
#0: (events){.+.+.+}, at: [<ffffffff81064352>]process_one_work+0x162/0x4c0 #1: ((&wfc.work)){+.+.+.}, at: [<ffffffff81064352>] process_one_work+0x162/0x4c0 #2: (&__lockdep_no_validate__){......}, at: [<ffffffff812db225>] device_attach+0x25/0xb0 stack backtrace: Pid: 734, comm: kworker/0:1 Not tainted 3.9.0-rc1 #96 Call Trace: [<ffffffff810948ec>] validate_chain+0xdcc/0x11f0 [<ffffffff81095150>] __lock_acquire+0x440/0xc70 [<ffffffff81095150>] ? __lock_acquire+0x440/0xc70 [<ffffffff810959da>] lock_acquire+0x5a/0x70 [<ffffffff81066cb0>] ? wq_worker_waking_up+0x60/0x60 [<ffffffff81066cf5>] flush_work+0x45/0x250 [<ffffffff81066cb0>] ? wq_worker_waking_up+0x60/0x60 [<ffffffff810922be>] ? mark_held_locks+0x9e/0x130 [<ffffffff81066a96>] ? queue_work_on+0x46/0x90 [<ffffffff810925dd>] ? trace_hardirqs_on_caller+0xfd/0x190 [<ffffffff8109267d>] ? trace_hardirqs_on+0xd/0x10 [<ffffffff81066f74>] work_on_cpu+0x74/0x90 [<ffffffff81063820>] ? keventd_up+0x20/0x20 [<ffffffff8121fd30>] ? pci_pm_prepare+0x60/0x60 [<ffffffff811f9293>] ? cpumask_next_and+0x23/0x40 [<ffffffff81220a1a>] pci_device_probe+0xba/0x110 [<ffffffff812dadca>] ? driver_sysfs_add+0x7a/0xb0 [<ffffffff812daf1f>] driver_probe_device+0x8f/0x230 [<ffffffff812db170>] ? __driver_attach+0xb0/0xb0 [<ffffffff812db1bb>] __device_attach+0x4b/0x60 [<ffffffff812d9314>] bus_for_each_drv+0x64/0x90 [<ffffffff812db298>] device_attach+0x98/0xb0 [<ffffffff81218474>] pci_bus_add_device+0x24/0x50 [<ffffffff81232e80>] virtfn_add+0x240/0x3e0 [<ffffffff8146ce3d>] ? _raw_spin_unlock_irqrestore+0x3d/0x80 [<ffffffff812333be>] pci_enable_sriov+0x23e/0x500 [<ffffffffa011fa1a>] __mlx4_init_one+0x5da/0xce0 [mlx4_core] [<ffffffffa012016d>] mlx4_init_one+0x2d/0x60 [mlx4_core] [<ffffffff8121fd79>] local_pci_probe+0x49/0x80 [<ffffffff81063833>] work_for_cpu_fn+0x13/0x20 [<ffffffff810643b8>] process_one_work+0x1c8/0x4c0 [<ffffffff81064352>] ? process_one_work+0x162/0x4c0 [<ffffffff81064cfb>] worker_thread+0x30b/0x430 [<ffffffff810649f0>] ? manage_workers+0x340/0x340 [<ffffffff8106cea6>] kthread+0xd6/0xe0 [<ffffffff8106cdd0>] ? __init_kthread_worker+0x70/0x70 [<ffffffff8146daac>] ret_from_fork+0x7c/0xb0 [<ffffffff8106cdd0>] ? __init_kthread_worker+0x70/0x70The issue is that a driver, in it's probe function, calls
pci_sriov_enableso a PF device probe causes VF probe (AKA nested probe). Each probe inpci_device_probewhich is (normally) run throughwork_on_cpu(this is to get the right numa node for memory allocated by the driver). In turnwork_on_cpudoes this internally:schedule_work_on(cpu, &wfc.work); flush_work(&wfc.work);So if you are running probe on CPU1, and cause another probe on the same CPU, this will try to flush workqueue from inside same workqueue which causes a lockep warning.
Nested probing might be tricky to get right generally.
But for
pci_sriov_enable, the situation is actually very simple: all VFs naturally have same affinity as the PF, andcpumask_any_andis actually same ascpumask_first_and, so it always gives us the same CPU. So let's just detect that, and run the probing for VFs locally without a workqueue.Quoted 6 lines
This is hardly elegant, but looks to me like an appropriate quick fix for 3.9.
Tested-by Or Gerlitz <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>Acked-by Tejun Heo <[email protected]>
Quoted 5 lines
Thanks, Michael. I put this in my for-linus branch:
http://git.kernel.org/cgit/linux/kernel/git/helgaas/pci.git/log/?h=for-linus
I'll send a pull request to Linus today.
Actually, let me make sure I understand this correctly: This patch fixes the lockdep warning, but it does not fix an actual deadlock or make any functional change. Is that right?
If that's true, how much pain would it cause to just hold this for v3.9.1? I'm nervous about doing a warning fix when we're only a day or two before releasing v3.9.
Bjorn
Quoted 20 lines
Quoted 20 lines
---
Changes from v1:
- clarified commit log and added Ack by Tejun Heo patch is unchanged.drivers/pci/pci-driver.c +1
-1@@ -286,8 +286,8 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,int cpu;get_online_cpus (); cpu =cpumask_any_and (cpumask_of_node (node), cpu_online_mask); -if (cpu < nr_cpu_ids) +if (cpu !=raw_smp_processor_id () && cpu < nr_cpu_ids) error =work_on_cpu (cpu, local_pci_probe, &ddi);else error =local_pci_probe (&ddi);
Re: [PATCHv2 for-3.9] pci: avoid work_on_cpu for nested SRIOV probes
Quoted 108 lines · Bjorn Helgaas
Quoted 107 lines · Bjorn Helgaas
Quoted 97 lines · Michael S. Tsirkin
The following lockdep report triggers since 3.9-rc1:
=============================================
[ INFO: possible recursive locking detected ]
3.9.0-rc1 #96 Not tainted
---------------------------------------------
kworker/0:1/734 is trying to acquire lock:
((&wfc.work)){+.+.+.}, at: [<ffffffff81066cb0>]flush_work+0x0/0x250but task is already holding lock:
((&wfc.work)){+.+.+.}, at: [<ffffffff81064352>]process_one_work+0x162/0x4c0other info that might help us debug this:
Possible unsafe locking scenario:CPU0 ----lock((&wfc.work));
lock((&wfc.work));*** DEADLOCK ***
May be due to missing lock nesting notation
3 locks held by kworker/0:1/734:
#0: (events){.+.+.+}, at: [<ffffffff81064352>]process_one_work+0x162/0x4c0 #1: ((&wfc.work)){+.+.+.}, at: [<ffffffff81064352>] process_one_work+0x162/0x4c0 #2: (&__lockdep_no_validate__){......}, at: [<ffffffff812db225>] device_attach+0x25/0xb0 stack backtrace: Pid: 734, comm: kworker/0:1 Not tainted 3.9.0-rc1 #96 Call Trace: [<ffffffff810948ec>] validate_chain+0xdcc/0x11f0 [<ffffffff81095150>] __lock_acquire+0x440/0xc70 [<ffffffff81095150>] ? __lock_acquire+0x440/0xc70 [<ffffffff810959da>] lock_acquire+0x5a/0x70 [<ffffffff81066cb0>] ? wq_worker_waking_up+0x60/0x60 [<ffffffff81066cf5>] flush_work+0x45/0x250 [<ffffffff81066cb0>] ? wq_worker_waking_up+0x60/0x60 [<ffffffff810922be>] ? mark_held_locks+0x9e/0x130 [<ffffffff81066a96>] ? queue_work_on+0x46/0x90 [<ffffffff810925dd>] ? trace_hardirqs_on_caller+0xfd/0x190 [<ffffffff8109267d>] ? trace_hardirqs_on+0xd/0x10 [<ffffffff81066f74>] work_on_cpu+0x74/0x90 [<ffffffff81063820>] ? keventd_up+0x20/0x20 [<ffffffff8121fd30>] ? pci_pm_prepare+0x60/0x60 [<ffffffff811f9293>] ? cpumask_next_and+0x23/0x40 [<ffffffff81220a1a>] pci_device_probe+0xba/0x110 [<ffffffff812dadca>] ? driver_sysfs_add+0x7a/0xb0 [<ffffffff812daf1f>] driver_probe_device+0x8f/0x230 [<ffffffff812db170>] ? __driver_attach+0xb0/0xb0 [<ffffffff812db1bb>] __device_attach+0x4b/0x60 [<ffffffff812d9314>] bus_for_each_drv+0x64/0x90 [<ffffffff812db298>] device_attach+0x98/0xb0 [<ffffffff81218474>] pci_bus_add_device+0x24/0x50 [<ffffffff81232e80>] virtfn_add+0x240/0x3e0 [<ffffffff8146ce3d>] ? _raw_spin_unlock_irqrestore+0x3d/0x80 [<ffffffff812333be>] pci_enable_sriov+0x23e/0x500 [<ffffffffa011fa1a>] __mlx4_init_one+0x5da/0xce0 [mlx4_core] [<ffffffffa012016d>] mlx4_init_one+0x2d/0x60 [mlx4_core] [<ffffffff8121fd79>] local_pci_probe+0x49/0x80 [<ffffffff81063833>] work_for_cpu_fn+0x13/0x20 [<ffffffff810643b8>] process_one_work+0x1c8/0x4c0 [<ffffffff81064352>] ? process_one_work+0x162/0x4c0 [<ffffffff81064cfb>] worker_thread+0x30b/0x430 [<ffffffff810649f0>] ? manage_workers+0x340/0x340 [<ffffffff8106cea6>] kthread+0xd6/0xe0 [<ffffffff8106cdd0>] ? __init_kthread_worker+0x70/0x70 [<ffffffff8146daac>] ret_from_fork+0x7c/0xb0 [<ffffffff8106cdd0>] ? __init_kthread_worker+0x70/0x70The issue is that a driver, in it's probe function, calls
pci_sriov_enableso a PF device probe causes VF probe (AKA nested probe). Each probe inpci_device_probewhich is (normally) run throughwork_on_cpu(this is to get the right numa node for memory allocated by the driver). In turnwork_on_cpudoes this internally:schedule_work_on(cpu, &wfc.work); flush_work(&wfc.work);So if you are running probe on CPU1, and cause another probe on the same CPU, this will try to flush workqueue from inside same workqueue which causes a lockep warning.
Nested probing might be tricky to get right generally.
But for
pci_sriov_enable, the situation is actually very simple: all VFs naturally have same affinity as the PF, andcpumask_any_andis actually same ascpumask_first_and, so it always gives us the same CPU. So let's just detect that, and run the probing for VFs locally without a workqueue.Quoted 6 lines
This is hardly elegant, but looks to me like an appropriate quick fix for 3.9.
Tested-by Or Gerlitz <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>Acked-by Tejun Heo <[email protected]>Thanks, Michael. I put this in my for-linus branch:
Quoted 7 lines
Quoted 3 lines
http://git.kernel.org/cgit/linux/kernel/git/helgaas/pci.git/log/?h=for-linus
I'll send a pull request to Linus today.
Actually, let me make sure I understand this correctly: This patch fixes the lockdep warning, but it does not fix an actual deadlock or make any functional change. Is that right?
Tejun said that this warning is a false positive, so yes.
Quoted 5 lines
If that's true, how much pain would it cause to just hold this for v3.9.1? I'm nervous about doing a warning fix when we're only a day or two before releasing v3.9.
Bjorn
I don't have this hardware, so I don't know. It was apparently reported by real users ...
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
---
Changes from v1:
- clarified commit log and added Ack by Tejun Heo patch is unchanged.drivers/pci/pci-driver.c +1
-1@@ -286,8 +286,8 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,int cpu;get_online_cpus (); cpu =cpumask_any_and (cpumask_of_node (node), cpu_online_mask); -if (cpu < nr_cpu_ids) +if (cpu !=raw_smp_processor_id () && cpu < nr_cpu_ids) error =work_on_cpu (cpu, local_pci_probe, &ddi);else error =local_pci_probe (&ddi);
A message that is not in the archive (sent off-list or not yet received): CAOS58YO+uV5KkS=sTP9Y3BPPh1nVnQ06yRyNU8GvEbym7R+X+Q@mail.gmail.com
Bjorn Helgaas
maintainerRe: [PATCHv2 for-3.9] pci: avoid work_on_cpu for nested SRIOV probes
Quoted 6 lines · Tejun Heo
So, the thing is there is no actual deadlock. If you're okay with releasing w/ spurious lockdep warning, leaving things alone should be fine. An issue with mst's patch is that it actually changes the behavior to avoid a spurious warning. An alternative course would be leaving it alone now and remove the spurious warning during the coming devel cycle and mark it w/ -stable.
If I understand correctly, you need v3.9-rc1 or later, CONFIG_PROVE_LOCKING=y, and an SR-IOV device to see the warning.
I like the idea of fixing the spurious warning for v3.10 and backporting to -stable. It sounds like there's a cleaner fix in the works that needs a bit more polishing. If we need a quick fix sooner,
we'll still have this one in our back pocket.
Bjorn
Re: [PATCHv2 for-3.9] pci: avoid work_on_cpu for nested SRIOV probes
In reply to a message not in the archiveQuoted 4 lines · Tejun Heo
So, the thing is there is no actual deadlock. If you're okay with releasing w/ spurious lockdep warning, leaving things alone should be fine. An issue with mst's patch is that it actually changes the behavior to avoid a spurious warning.
Yes but in fact, isn't it cleaner to avoid work_on if we are going to run on the local CPU, anyway?
Quoted 4 lines
An alternative course would be leaving it alone now and remove the spurious warning during the coming devel cycle and mark it w/ -stable.
Thanks.
Okay. Could you send tested a version of work_on_nested?
Quoted 2 lines
--
tejun