pci/sriov: Hold rescan lock while enumerating
Subsystems (2)
PATCH 1/1 pci/sriov: Hold rescan lock while enumerating
PCI enumeration/de-enumeration needs to run single threaded to prevent race conditions with other threads changing the topology. Altering the number of virtual functions was not taking the rescan/remove lock hile adding or removing those virtual functions, so this patch adds that.
--- drivers/pci/pci-sysfs.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/pci/pci-sysfs.c +2 -0
@@ -591,6 +591,7 @@ static ssize_t sriov_numvfs_store(struct device *dev,if (num_vfs >pci_sriov_get_totalvfs (pdev))return -ERANGE ; +pci_lock_rescan_remove ();device_lock (&pdev->dev);if (num_vfs == pdev->sriov->num_VFs)
@@ -627,6 +628,7 @@ static ssize_t sriov_numvfs_store(struct device *dev, exit:device_unlock (&pdev->dev); +pci_unlock_rescan_remove ();if (ret <0 )return ret;
Signature · 1 line
2.14.4
drivers/pci/pci-sysfs.c static ssize_t sriov_numvfs_store(struct device *dev, · line 595
Quoted 8 lines · Keith Busch
drivers/pci/pci-sysfs.c +1
-0@@ -591,6 +591,7 @@ static ssize_t sriov_numvfs_store(struct device *dev,if (num_vfs >pci_sriov_get_totalvfs (pdev))return -ERANGE ; +pci_lock_rescan_remove ();device_lock (&pdev->dev);
Doesn't this obviate the need to hold the device_lock()? What could otherwise run concurrently that doesn't hold pci_lock_rescan_remove() and would cause a race condition?
Thanks,
Lukas
Quoted 7 lines · Lukas Wunner
Quoted 6 lines · Keith Busch
drivers/pci/pci-sysfs.c +0
-0@@ -591,6 +591,7 @@ static ssize_t sriov_numvfs_store(struct device *dev,if (num_vfs >pci_sriov_get_totalvfs (pdev))return -ERANGE ;
Quoted 6 lines
drivers/pci/pci-sysfs.c static ssize_t sriov_numvfs_store(struct device *dev, · line 595
Quoted 2 lines
drivers/pci/pci-sysfs.c +1
-0+pci_lock_rescan_remove ();device_lock (&pdev->dev);Doesn't this obviate the need to hold the
device_lock()? What could otherwise run concurrently that doesn't holdpci_lock_rescan_remove()and would cause a race condition?
The device_lock is still needed to protect against concurrent pci_reset_function.
[+cc Lukas, -stable]
Quoted 17 lines · Keith Busch
PCI enumeration/de-enumeration needs to run single threaded to prevent race conditions with other threads changing the topology. Altering the number of virtual functions was not taking the rescan/remove lock hile adding or removing those virtual functions, so this patch adds that.
Reported-by Krzysztof Wierzbicki <[email protected]>Signed-off-by Keith Busch <[email protected]>--- drivers/pci/pci-sysfs.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/pci-sysfs.c +0
-0
drivers/pci/pci-sysfs.c static ssize_t sriov_numvfs_store(struct device *dev, · line 594
Quoted 4 lines
drivers/pci/pci-sysfs.c +1
-0@@ -591,6 +591,7 @@ static ssize_t sriov_numvfs_store(struct device *dev,if (num_vfs >pci_sriov_get_totalvfs (pdev))return -ERANGE ; +pci_lock_rescan_remove ();
I assume the usual path is something like this, where we end up in pci_enable_sriov():
sriov_numvfs_store
+ pci_lock_rescan_remove
pdev->driver->sriov_configure
...
pci_enable_sriov
sriov_enable
pci_iov_add_virtfn
pci_alloc_dev
pci_device_addBut what about the other paths leading to pci_iov_add_virtfn()? Don't we need similar locking for all of them?
Quoted 14 lines
drivers/pci/pci-sysfs.c +1
-0device_lock (&pdev->dev);if (num_vfs == pdev->sriov->num_VFs)@@ -627,6 +628,7 @@ static ssize_t sriov_numvfs_store(struct device *dev, exit:device_unlock (&pdev->dev); +pci_unlock_rescan_remove ();if (ret <0 )return ret;Signature · 1 line
2.14.4
Quoted 38 lines · Bjorn Helgaas
[+cc Lukas, -stable]
Quoted 17 lines · Keith Busch
PCI enumeration/de-enumeration needs to run single threaded to prevent race conditions with other threads changing the topology. Altering the number of virtual functions was not taking the rescan/remove lock hile adding or removing those virtual functions, so this patch adds that.
Reported-by Krzysztof Wierzbicki <[email protected]>Signed-off-by Keith Busch <[email protected]>--- drivers/pci/pci-sysfs.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/pci-sysfs.c +0
-0drivers/pci/pci-sysfs.c static ssize_t sriov_numvfs_store(struct device *dev, · line 594
Quoted 4 lines
drivers/pci/pci-sysfs.c +1
-0@@ -591,6 +591,7 @@ static ssize_t sriov_numvfs_store(struct device *dev,if (num_vfs >pci_sriov_get_totalvfs (pdev))return -ERANGE ; +pci_lock_rescan_remove ();I assume the usual path is something like this, where we end up in
pci_enable_sriov():
sriov_numvfs_store
+pci_lock_rescan_removepdev->driver->sriov_configure ... pci_enable_sriov sriov_enable pci_iov_add_virtfn pci_alloc_dev pci_device_add
Quoted 2 lines
But what about the other paths leading to
pci_iov_add_virtfn()? Don't we need similar locking for all of them?
Good point. It looks like the only other path to pci_iov_add_virtfn is through PPC eeh_reset_device, which coincidently enough already holds pci_lock_rescan_remove.
It's not just pci_iov_add_virtfn, though. We also need to hold it during pci_iov_remove_virtfn. PPC eeh_rmv_device is not holding the lock in that path. I'll never be able to test with that h/w, but I'll look at it.
Quoted 14 lines
Quoted 14 lines
drivers/pci/pci-sysfs.c +1
-0device_lock (&pdev->dev);if (num_vfs == pdev->sriov->num_VFs)@@ -627,6 +628,7 @@ static ssize_t sriov_numvfs_store(struct device *dev, exit:device_unlock (&pdev->dev); +pci_unlock_rescan_remove ();if (ret <0 )return ret;Signature · 1 line
2.14.4
Quoted 39 lines · Keith Busch
Quoted 38 lines · Bjorn Helgaas
[+cc Lukas, -stable]
Quoted 17 lines · Keith Busch
PCI enumeration/de-enumeration needs to run single threaded to prevent race conditions with other threads changing the topology. Altering the number of virtual functions was not taking the rescan/remove lock hile adding or removing those virtual functions, so this patch adds that.
Reported-by Krzysztof Wierzbicki <[email protected]>Signed-off-by Keith Busch <[email protected]>--- drivers/pci/pci-sysfs.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/pci-sysfs.c +0
-0drivers/pci/pci-sysfs.c static ssize_t sriov_numvfs_store(struct device *dev, · line 594
Quoted 4 lines
drivers/pci/pci-sysfs.c +1
-0@@ -591,6 +591,7 @@ static ssize_t sriov_numvfs_store(struct device *dev,if (num_vfs >pci_sriov_get_totalvfs (pdev))return -ERANGE ; +pci_lock_rescan_remove ();I assume the usual path is something like this, where we end up in
pci_enable_sriov():
sriov_numvfs_store
+pci_lock_rescan_removepdev->driver->sriov_configure ... pci_enable_sriov sriov_enable pci_iov_add_virtfn pci_alloc_dev pci_device_add
Quoted 6 lines
Quoted 2 lines
But what about the other paths leading to
pci_iov_add_virtfn()? Don't we need similar locking for all of them?Good point. It looks like the only other path to
pci_iov_add_virtfnis through PPCeeh_reset_device, which coincidently enough already holdspci_lock_rescan_remove.
There are a few .probe() methods that call pci_enable_sriov(), which leads to pci_iov_add_virtfn():
cavium_rng_probe
cpt_probe
enic_probe
be_probe -> be_setup -> be_vf_setup
igb_probe -> igb_sw_init -> igb_probe_vfs -> igb_enable_sriov
...
Quoted 18 lines
It's not just
pci_iov_add_virtfn, though. We also need to hold it duringpci_iov_remove_virtfn. PPCeeh_rmv_deviceis not holding the lock in that path. I'll never be able to test with that h/w, but I'll look at it.Quoted 14 lines
Quoted 14 lines
drivers/pci/pci-sysfs.c +1
-0device_lock (&pdev->dev);if (num_vfs == pdev->sriov->num_VFs)@@ -627,6 +628,7 @@ static ssize_t sriov_numvfs_store(struct device *dev, exit:device_unlock (&pdev->dev); +pci_unlock_rescan_remove ();if (ret <0 )return ret;Signature · 1 line
2.14.4