net/mlx5: Call pci_disable_sriov() on remove
Subsystems (3)
cover net/mlx5: Call pci_disable_sriov() on remove
Hello,
I'm currently working on improvements in PF-VF handling on s390. One thing that may be a bit special for us is that the s390 hotplug code supports shutting down a single PF of a multi-function device such as a ConnectX-5. During testing I found that the mlx5_core driver does not call pci_disable_sriov() in its remove handler which is called on the PF via pci_stop_and_remove_bus_device() on an orderly hot unplug.
Following is a patch to fix this, I want to point out however that I'm not entirely sure about the effect of clear_vfs that distinguishes mlx5_sriov_detach() from mlx5_sriov_disable() only that the former is called from the remove handler and it doesn't call pci_disable_sriov(). That however is required according to Documentation/PCI/pci-iov-howto.rst
I've only tested this on top of my currently still internal PF-VF rework so I might also be totally missing something here in that case excuse my ignorance.
Best regards,
Niklas Schnelle
Niklas Schnelle (1):
net/mlx5: Call pci_disable_sriov() on remove
drivers/net/ethernet/mellanox/mlx5/core/sriov.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Signature · 1 line
2.17.1
PATCH 1/1 net/mlx5: Call pci_disable_sriov() on remove
as described in Documentation/PCI/pci-iov-howto.rst a driver with SR-IOV support should call pci_disable_sriov() in the remove handler. Otherwise removing a PF (e.g. via pci_stop_and_remove_bus_device()) with attached VFs does not properly shut the VFs down before shutting down the PF. This leads to the VF drivers handling defunct devices and accompanying error messages.
In the current code pci_disable_sriov() is already called in mlx5_sriov_disable() but not in mlx5_sriov_detach() which is called from the remove handler. Fix this by moving the pci_disable_sriov() call into mlx5_device_disable_sriov() which is called by both.
--- drivers/net/ethernet/mellanox/mlx5/core/sriov.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/sriov.c +2 -1
@@ -114,6 +114,8 @@ mlx5_device_disable_sriov(struct mlx5_core_dev *dev, int num_vfs, bool clear_vf)int err;int vf; +pci_disable_sriov (dev->pdev); +for (vf = num_vfs -1 ; vf >=0 ; vf--) {if (!sriov->vfs_ctx[vf].enabled)continue ;
@@ -156,7 +158,6 @@ static void mlx5_sriov_disable(struct pci_dev *pdev)struct mlx5_core_dev *dev =pci_get_drvdata (pdev);int num_vfs =pci_num_vf (dev->pdev); -pci_disable_sriov (pdev);mlx5_device_disable_sriov (dev, num_vfs, true); }
Signature · 1 line
2.17.1
Quoted 3 lines · Niklas Schnelle
as described in
Documentation/PCI/pci-iov-howto.rsta driver with SR- IOV
support should callpci_disable_sriov()in the remove handler.
Hi Niklas,
looking at the documentation, it doesn't say "should" it just gives the code as example.
Quoted 6 lines
Otherwise removing a PF (e.g. via
pci_stop_and_remove_bus_device()) with
attached VFs does not properly shut the VFs down before shutting down the PF. This leads to the VF drivers handling defunct devices and accompanying error messages.
Which should be the admin responsibility .. if the admin want to do this, then let it be.. why block him ?
our mlx5 driver in the vf handles this gracefully and once pf driver/device is back online the vf driver quickly recovers.
Quoted 28 lines
In the current code
pci_disable_sriov()is already called inmlx5_sriov_disable()but not inmlx5_sriov_detach()which is called from
the remove handler. Fix this by moving thepci_disable_sriov()call intomlx5_device_disable_sriov()which is called by both.Signed-off-by Niklas Schnelle <[email protected]>--- drivers/net/ethernet/mellanox/mlx5/core/sriov.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/net/ethernet/mellanox/mlx5/core/sriov.c +2
-0@@ -114,6 +114,8 @@ mlx5_device_disable_sriov(struct mlx5_core_dev *dev, int num_vfs, bool clear_vf)int err;int vf; +pci_disable_sriov (dev->pdev); +for (vf = num_vfs -1 ; vf >=0 ; vf--) {if (!sriov->vfs_ctx[vf].enabled)drivers/net/ethernet/mellanox/mlx5/core/sriov.c +0
-0continue ;
drivers/net/ethernet/mellanox/mlx5/core/sriov.c static void mlx5_sriov_disable(struct pci_dev · line 160
Quoted 5 lines
drivers/net/ethernet/mellanox/mlx5/core/sriov.c +0
-1@@ -156,7 +158,6 @@ static void mlx5_sriov_disable(struct pci_dev *pdev)struct mlx5_core_dev *dev =pci_get_drvdata (pdev);int num_vfs =pci_num_vf (dev->pdev); -pci_disable_sriov (pdev);
this patch is no good as it breaks code symmetry.. and could lead to many new issues.
Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/sriov.c +0
-0mlx5_device_disable_sriov (dev, num_vfs, true); }
Quoted 17 lines · Saeed Mahameed
Quoted 3 lines · Niklas Schnelle
as described in
Documentation/PCI/pci-iov-howto.rsta driver with SR- IOV
support should callpci_disable_sriov()in the remove handler.Hi Niklas,
looking at the documentation, it doesn't say "should" it just gives the code as example.
Quoted 6 lines
Otherwise removing a PF (e.g. via
pci_stop_and_remove_bus_device()) with
attached VFs does not properly shut the VFs down before shutting down the PF. This leads to the VF drivers handling defunct devices and accompanying error messages.
Quoted 5 lines
Which should be the admin responsibility .. if the admin want to do this, then let it be.. why block him ?
our mlx5 driver in the vf handles this gracefully and once pf driver/device is back online the vf driver quickly recovers.
See my answer to your other answer ;-)
Quoted 33 lines
Quoted 33 lines
In the current code
pci_disable_sriov()is already called inmlx5_sriov_disable()but not inmlx5_sriov_detach()which is called from
the remove handler. Fix this by moving thepci_disable_sriov()call intomlx5_device_disable_sriov()which is called by both.Signed-off-by Niklas Schnelle <[email protected]>--- drivers/net/ethernet/mellanox/mlx5/core/sriov.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/net/ethernet/mellanox/mlx5/core/sriov.c +2
-0@@ -114,6 +114,8 @@ mlx5_device_disable_sriov(struct mlx5_core_dev *dev, int num_vfs, bool clear_vf)int err;int vf; +pci_disable_sriov (dev->pdev); +for (vf = num_vfs -1 ; vf >=0 ; vf--) {if (!sriov->vfs_ctx[vf].enabled)drivers/net/ethernet/mellanox/mlx5/core/sriov.c +0
-0continue ;@@ -156,7 +158,6 @@ static void mlx5_sriov_disable(struct pci_dev *pdev)struct mlx5_core_dev *dev =pci_get_drvdata (pdev);int num_vfs =pci_num_vf (dev->pdev);
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/sriov.c static void mlx5_sriov_disable(struct pci_dev *pdev) · line 159
Quoted 1 line
drivers/net/ethernet/mellanox/mlx5/core/sriov.c +0
-1-pci_disable_sriov (pdev);this patch is no good as it breaks code symmetry.. and could lead to many new issues.
Ah you're right I totally missed that there is a matching pci_enable_sriov() in mlx5_enable_sriov() haven't used these myself before and since it wasn't in the documentation example I somehow expected it to happen in non-driver code, so for symmetry that would also have to move to mlx5_device_enable_sriov(), sorry for the oversight.
Quoted 6 lines
Quoted 5 lines
drivers/net/ethernet/mellanox/mlx5/core/sriov.c +0
-0mlx5_device_disable_sriov (dev, num_vfs, true); }
Quoted 59 lines · Niklas Schnelle
Quoted 17 lines · Saeed Mahameed
Quoted 3 lines · Niklas Schnelle
as described in
Documentation/PCI/pci-iov-howto.rsta driver with SR- IOV
support should callpci_disable_sriov()in the remove handler.Hi Niklas,
looking at the documentation, it doesn't say "should" it just gives the code as example.
Quoted 6 lines
Otherwise removing a PF (e.g. via
pci_stop_and_remove_bus_device()) with
attached VFs does not properly shut the VFs down before shutting down the PF. This leads to the VF drivers handling defunct devices and accompanying error messages.Quoted 5 lines
Which should be the admin responsibility .. if the admin want to do this, then let it be.. why block him ?
our mlx5 driver in the vf handles this gracefully and once pf driver/device is back online the vf driver quickly recovers.
See my answer to your other answer ;-)
Quoted 33 lines
Quoted 33 lines
In the current code
pci_disable_sriov()is already called inmlx5_sriov_disable()but not inmlx5_sriov_detach()which is called from
the remove handler. Fix this by moving thepci_disable_sriov()call intomlx5_device_disable_sriov()which is called by both.Signed-off-by Niklas Schnelle <[email protected]>--- drivers/net/ethernet/mellanox/mlx5/core/sriov.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/net/ethernet/mellanox/mlx5/core/sriov.c +2
-0@@ -114,6 +114,8 @@ mlx5_device_disable_sriov(struct mlx5_core_dev *dev, int num_vfs, bool clear_vf)int err;int vf; +pci_disable_sriov (dev->pdev); +for (vf = num_vfs -1 ; vf >=0 ; vf--) {if (!sriov->vfs_ctx[vf].enabled)drivers/net/ethernet/mellanox/mlx5/core/sriov.c +0
-0continue ;@@ -156,7 +158,6 @@ static void mlx5_sriov_disable(struct pci_dev *pdev)struct mlx5_core_dev *dev =pci_get_drvdata (pdev);int num_vfs =pci_num_vf (dev->pdev);
Quoted 7 lines
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/sriov.c static void mlx5_sriov_disable(struct pci_dev *pdev) · line 159
Quoted 1 line
drivers/net/ethernet/mellanox/mlx5/core/sriov.c +0
-1-pci_disable_sriov (pdev);this patch is no good as it breaks code symmetry.. and could lead to many new issues.
Ah you're right I totally missed that there is a matching
pci_enable_sriov()inmlx5_enable_sriov()haven't used these myself before and since it wasn't in the documentation example I somehow expected it to happen in non-driver code,
aaand it actually is in the documentation example and I definitely sent this when it wasn't ready, sorry again…
Quoted 8 lines
so for symmetry that would also have to move to
mlx5_device_enable_sriov(), sorry for the oversight.Quoted 6 lines
Quoted 5 lines
drivers/net/ethernet/mellanox/mlx5/core/sriov.c +0
-0mlx5_device_disable_sriov (dev, num_vfs, true); }
Quoted 11 lines · Niklas Schnelle
Hello,
I'm currently working on improvements in PF-VF handling on s390. One thing that
may be a bit special for us is that the s390 hotplug code supports shutting
down a single PF of a multi-function device such as a ConnectX-5. During testing I found that themlx5_coredriver does not callpci_disable_sriov()in its remove handler which is called on the PF viapci_stop_and_remove_bus_device()on an orderly hot unplug.
Actually what happens if you call pci_disable_sriov() when there are VFs attached to VMs ?
Quoted 10 lines
Following is a patch to fix this, I want to point out however that I'm not
entirely sure about the effect ofclear_vfsthat distinguishesmlx5_sriov_detach()frommlx5_sriov_disable()only that the former is called
from the remove handler and it doesn't callpci_disable_sriov(). That however is required according toDocumentation/PCI/pci-iov-howto.rst
The Doc doesn't say "required", so the question is, is it really required ?
because the way i see it, it is the admin responsibility to clear out vfs before shutting down the PF driver.
as explained in the patch review, mlx5 vf driver is resilient of such behavior and once the device is back online the vf driver quickly recovers, so it is actually a feature and not a bug :)
There are many reasons why the admin would want to do this:
1) Fast Firmware upgrade
2) Fast Hyper-visor upgrade/refresh
3) PF debugging
So you can quickly reset the PF driver without the need to wait for all VFs or manually detach-attach them.
Quoted 14 lines
I've only tested this on top of my currently still internal PF-VF rework so
I might also be totally missing something here in that case excuse my ignorance.Best regards,
Niklas SchnelleNiklas Schnelle (1):
net/mlx5: Callpci_disable_sriov()on removedrivers/net/ethernet/mellanox/mlx5/core/sriov.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 13 lines · Saeed Mahameed
Quoted 11 lines · Niklas Schnelle
Hello,
I'm currently working on improvements in PF-VF handling on s390. One thing that
may be a bit special for us is that the s390 hotplug code supports shutting
down a single PF of a multi-function device such as a ConnectX-5. During testing I found that themlx5_coredriver does not callpci_disable_sriov()in its remove handler which is called on the PF viapci_stop_and_remove_bus_device()on an orderly hot unplug.
Quoted 2 lines
Actually what happens if you call
pci_disable_sriov()when there are VFs attached to VMs ?
So the scenario I looked at was that the admin does a
echo 0 > /sys/bus/pci/slots/<pfslot>/power
for the relevant PFs (Note that on our systems
PFs of a single card could be passed through to different LPARs).
And then physically removes the adapter or moves the PF(s) to a different LPAR. In this scenario the orderly shutdown would hopefully help QEMU/libvirt to trigger whatever orderly shutdown it can do but that testing is still on my todo list. I assumed that because the pci_disable_sriov() call is basically the only thing the example shows for the remove case it would trigger the intended actions in the common code including for vfio-pci, udev events and so on.
Quoted 12 lines
Quoted 11 lines
Following is a patch to fix this, I want to point out however that I'm not
entirely sure about the effect ofclear_vfsthat distinguishesmlx5_sriov_detach()frommlx5_sriov_disable()only that the former is called
from the remove handler and it doesn't callpci_disable_sriov(). That however is required according toDocumentation/PCI/pci-iov-howto.rst
Quoted 2 lines
The Doc doesn't say "required", so the question is, is it really required ?
Yes but as I wrote above it is about the only thing the example shows for the removal procedure so there is definitely a clash between what the mlx5 driver does and what the documentation suggests. So I think even though I agree my patch definitely wasn't ready, this is surely worth thinking about for a second or two.
Quoted 8 lines
because the way i see it, it is the admin responsibility to clear out vfs before shutting down the PF driver.
as explained in the patch review, mlx5 vf driver is resilient of such behavior and once the device is back online the vf driver quickly recovers, so it is actually a feature and not a bug :)
Quoted 8 lines
There are many reasons why the admin would want to do this:
1) Fast Firmware upgrade
2) Fast Hyper-visor upgrade/refresh
3) PF debuggingSo you can quickly reset the PF driver without the need to wait for all VFs or manually detach-attach them.
So the behavior I was seeing that triggered looking into this is that the VF drivers weren't just spewing error messages. After hitting some relatively small timeout the VFs would then start to disappear (maybe 20 seconds or so) and in my testing I didn't manage to reactivate them.
That might have been due to some error on my part though and I didn't try very hard because I did not assume that this should work. So thank you for pointing out that something like that is actually a scenario you have in mind. Now that I know that I'll definitely look into this a bit more.
So thank you for your input I'm still not sure what the right approach for the shutdown really is but this is certainly interesting.
Quoted 16 lines
Quoted 16 lines
I've only tested this on top of my currently still internal PF-VF rework so
I might also be totally missing something here in that case excuse my ignorance.Best regards,
Niklas SchnelleNiklas Schnelle (1):
net/mlx5: Callpci_disable_sriov()on removedrivers/net/ethernet/mellanox/mlx5/core/sriov.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)