Skip to contentSkip to conversation

PCI: Add vf reset notification for pf

PATCH In review Emily Deng 5 participants 6 messages Feb 4, 2024 – Feb 5, 2024

Subsystems (2)
lore
Share & tools
lore

Emily Deng

author
#0 lore raw

PATCH 1/1 PCI: Add vf reset notification for pf

When a vf has been reset, the pf wants to get notification to remove the vf out of schedule.

Solution:
Add the callback function in pci_driver sriov_vf_reset_notification. When vf reset happens, then call this callback function.

---
 drivers/pci/pci.c   | 8 ++++++++
 include/linux/pci.h | 1 +
 2 files changed, 9 insertions(+)
drivers/pci/pci.c +8 -0
index 60230da957e0..aca937b05531 100644
@@ -4780,6 +4780,14 @@ EXPORT_SYMBOL_GPL(pcie_flr);
  */
 int pcie_reset_flr(struct pci_dev *dev, bool probe)
 {
+	struct pci_dev *pf_dev;
+
+	if (dev->is_virtfn) {
+		pf_dev = dev->physfn;
+		if (pf_dev->driver->sriov_vf_reset_notification)
+			pf_dev->driver->sriov_vf_reset_notification(pf_dev, dev);
+	}
+
 	if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)
 		return -ENOTTY;
 
include/linux/pci.h +1 -0
index c69a2cc1f412..4fa31d9b0aa7 100644
@@ -926,6 +926,7 @@ struct pci_driver {
 	int  (*sriov_configure)(struct pci_dev *dev, int num_vfs); /* On PF */
 	int  (*sriov_set_msix_vec_count)(struct pci_dev *vf, int msix_vec_count); /* On PF */
 	u32  (*sriov_get_vf_total_msix)(struct pci_dev *pf);
+	void  (*sriov_vf_reset_notification)(struct pci_dev *pf, struct pci_dev *vf);
 	const struct pci_error_handlers *err_handler;
 	const struct attribute_group **groups;
 	const struct attribute_group **dev_groups;
Signature · 1 line
2.36.1
#1 lore raw

include/linux/pci.h struct pci_driver { · line 929

Quoted 5 lines · Emily Deng
include/linux/pci.h +1 -0
@@ -926,6 +926,7 @@ struct pci_driver {
 	int  (*sriov_configure)(struct pci_dev *dev, int num_vfs); /* On PF */
 	int  (*sriov_set_msix_vec_count)(struct pci_dev *vf, int msix_vec_count); /* On PF */
 	u32  (*sriov_get_vf_total_msix)(struct pci_dev *pf);
+	void  (*sriov_vf_reset_notification)(struct pci_dev *pf, struct pci_dev *vf);

You've created a new callback, but there is no user. Could you resubmit this with an in-kernel use case?

#2 lore raw
In reply to Emily Deng
Quoted 2 lines · Emily Deng

When a vf has been reset, the pf wants to get notification to remove the vf out of schedule.

It is very questionable if this is right thing to do. The idea of SR-IOV is that VFs represent a physical device and they should be treated separately from the PF.

In addition to that Keith said, this patch needs better justification.

Thanks

Quoted 46 lines

Solution:
Add the callback function in pci_driver sriov_vf_reset_notification. When vf reset happens, then call this callback function.

---
 drivers/pci/pci.c   | 8 ++++++++
 include/linux/pci.h | 1 +
 2 files changed, 9 insertions(+)
drivers/pci/pci.c +8 -0
index 60230da957e0..aca937b05531 100644
@@ -4780,6 +4780,14 @@ EXPORT_SYMBOL_GPL(pcie_flr);
  */
 int pcie_reset_flr(struct pci_dev *dev, bool probe)
 {
+	struct pci_dev *pf_dev;
+
+	if (dev->is_virtfn) {
+		pf_dev = dev->physfn;
+		if (pf_dev->driver->sriov_vf_reset_notification)
+			pf_dev->driver->sriov_vf_reset_notification(pf_dev, dev);
+	}
+
 	if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)
 		return -ENOTTY;
 
include/linux/pci.h +1 -0
index c69a2cc1f412..4fa31d9b0aa7 100644
@@ -926,6 +926,7 @@ struct pci_driver {
 	int  (*sriov_configure)(struct pci_dev *dev, int num_vfs); /* On PF */
 	int  (*sriov_set_msix_vec_count)(struct pci_dev *vf, int msix_vec_count); /* On PF */
 	u32  (*sriov_get_vf_total_msix)(struct pci_dev *pf);
+	void  (*sriov_vf_reset_notification)(struct pci_dev *pf, struct pci_dev *vf);
 	const struct pci_error_handlers *err_handler;
 	const struct attribute_group **groups;
 	const struct attribute_group **dev_groups;
Signature · 1 line
2.36.1
#3 lore raw

[AMD Official Use Only - General]

The use case is the vfio-pci driver, which is used to support a user mode PF driver. Will also sent out the vfio-pci driver patch. And add more comments in the patch. Thanks.

Emily Deng
Best Wishes

Quoted 69 lines

-----Original Message-----
From: Leon Romanovsky <[email protected]>
Sent: Sunday, February 4, 2024 7:21 PM
To: Deng, Emily <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]
Subject: Re: [PATCH] PCI: Add vf reset notification for pf

Quoted 2 lines · Emily Deng

When a vf has been reset, the pf wants to get notification to remove the vf out of schedule.

It is very questionable if this is right thing to do. The idea of SR-IOV is that VFs represent a physical device and they should be treated separately from the PF.

In addition to that Keith said, this patch needs better justification.

Thanks

Quoted 16 lines

Solution:
Add the callback function in pci_driver sriov_vf_reset_notification. When vf reset happens, then call this callback function.

---
 drivers/pci/pci.c   | 8 ++++++++
 include/linux/pci.h | 1 +
 2 files changed, 9 insertions(+)
drivers/pci/pci.c index renamed from drivers/pci/pci.c +0 -0

60230da957e0..aca937b05531 100644

drivers/pci/pci.c EXPORT_SYMBOL_GPL(pcie_flr); · line 4787

Quoted 8 lines
drivers/pci/pci.c +6 -0
@@ -4780,6 +4780,14 @@ EXPORT_SYMBOL_GPL(pcie_flr);
  */
 int pcie_reset_flr(struct pci_dev *dev, bool probe)  {
+    struct pci_dev *pf_dev;
+
+    if (dev->is_virtfn) {
+            pf_dev = dev->physfn;
+            if (pf_dev->driver->sriov_vf_reset_notification)
+                    pf_dev->driver->sriov_vf_reset_notification(pf_dev,

dev);

include/linux/pci.h struct pci_driver { · line 927

Quoted 12 lines
drivers/pci/pci.c +2 -0
+    }
+
     if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)
             return -ENOTTY;
 
include/linux/pci.h index renamed from include/linux/pci.h +0 -0

c69a2cc1f412..4fa31d9b0aa7 100644

include/linux/pci.h +0 -0
@@ -926,6 +926,7 @@ struct pci_driver {
     int  (*sriov_configure)(struct pci_dev *dev, int num_vfs); /* On PF */
     int  (*sriov_set_msix_vec_count)(struct pci_dev *vf, int

msix_vec_count); /* On PF */

Quoted 10 lines
include/linux/pci.h +2 -1
     u32  (*sriov_get_vf_total_msix)(struct pci_dev *pf);
+    void  (*sriov_vf_reset_notification)(struct pci_dev *pf, struct
+pci_dev *vf);
     const struct pci_error_handlers *err_handler;
     const struct attribute_group **groups;
     const struct attribute_group **dev_groups;
--

2.36.1

#4 lore raw
In reply to Emily Deng

Am 04.02.24 um 07:12 schrieb Emily Deng:

Quoted 6 lines

When a vf has been reset, the pf wants to get notification to remove the vf out of schedule.

Solution:
Add the callback function in pci_driver sriov_vf_reset_notification. When vf reset happens, then call this callback function.

Well that doesn't make much sense. As other already noted as well a VF should be an encapsulated representation of a physical devices functionality.

AMD implemented that a bit different with a hypervisor to control which PF functionality a VF exposes, but that doesn't mean that we can leak this AMD specific handling into the common Linux PCI subsystem.

Additional to that a technical blocker is that when a VF is passed into a VM you don't have access to the PF any more to make this reset notification.

Regards,
Christian.

Quoted 38 lines
---
  drivers/pci/pci.c   | 8 ++++++++
  include/linux/pci.h | 1 +
  2 files changed, 9 insertions(+)
drivers/pci/pci.c +8 -0
index 60230da957e0..aca937b05531 100644
@@ -4780,6 +4780,14 @@ EXPORT_SYMBOL_GPL(pcie_flr);
   */
  int pcie_reset_flr(struct pci_dev *dev, bool probe)
  {
+	struct pci_dev *pf_dev;
+
+	if (dev->is_virtfn) {
+		pf_dev = dev->physfn;
+		if (pf_dev->driver->sriov_vf_reset_notification)
+			pf_dev->driver->sriov_vf_reset_notification(pf_dev, dev);
+	}
+
  	if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)
  		return -ENOTTY;
 
include/linux/pci.h +1 -0
index c69a2cc1f412..4fa31d9b0aa7 100644
@@ -926,6 +926,7 @@ struct pci_driver {
  	int  (*sriov_configure)(struct pci_dev *dev, int num_vfs); /* On PF */
  	int  (*sriov_set_msix_vec_count)(struct pci_dev *vf, int msix_vec_count); /* On PF */
  	u32  (*sriov_get_vf_total_msix)(struct pci_dev *pf);
+	void  (*sriov_vf_reset_notification)(struct pci_dev *pf, struct pci_dev *vf);
  	const struct pci_error_handlers *err_handler;
  	const struct attribute_group **groups;
  	const struct attribute_group **dev_groups;
#5 lore raw
In reply to Emily Deng
Quoted 44 lines · On Sun, 4 Feb 2024 14:12:57 +0800 Emily Deng <[email protected]> wrote:

When a vf has been reset, the pf wants to get notification to remove the vf out of schedule.

Solution:
Add the callback function in pci_driver sriov_vf_reset_notification. When vf reset happens, then call this callback function.

---
 drivers/pci/pci.c   | 8 ++++++++
 include/linux/pci.h | 1 +
 2 files changed, 9 insertions(+)
drivers/pci/pci.c +8 -0
index 60230da957e0..aca937b05531 100644
@@ -4780,6 +4780,14 @@ EXPORT_SYMBOL_GPL(pcie_flr);
  */
 int pcie_reset_flr(struct pci_dev *dev, bool probe)
 {
+	struct pci_dev *pf_dev;
+
+	if (dev->is_virtfn) {
+		pf_dev = dev->physfn;
+		if (pf_dev->driver->sriov_vf_reset_notification)
+
 pf_dev->driver->sriov_vf_reset_notification(pf_dev, dev);
+	}
+
 	if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)
 		return -ENOTTY;
include/linux/pci.h +0 -0
index c69a2cc1f412..4fa31d9b0aa7 100644
@@ -926,6 +926,7 @@ struct pci_driver {
 	int  (*sriov_configure)(struct pci_dev *dev, int num_vfs);
 /* On PF */ int  (*sriov_set_msix_vec_count)(struct pci_dev *vf, int

msix_vec_count); /* On PF */ u32 (*sriov_get_vf_total_msix)(struct pci_dev *pf);

include/linux/pci.h +1 -0
+	void  (*sriov_vf_reset_notification)(struct pci_dev *pf,

struct pci_dev *vf); const struct pci_error_handlers *err_handler;

include/linux/pci.h struct pci_driver { · line 932

Quoted 2 lines
include/linux/pci.h +0 -0
 	const struct attribute_group **groups;
 	const struct attribute_group **dev_groups;

Hi:

I would suggest you can provide a cover letter including a complete picture that tells the background, detailed problem statement, the solutions and plus the users. As this seems very like a generic change, it needs a better justification to convince folks why this is the best solution. Without a complete picture, the solution just looks like a workaround.

Thanks,
Zhi.