hv_netvsc: associate VF and PV device by serial number
Subsystems (5)
v2 cover hv_netvsc: associate VF and PV device by serial number
The Hyper-V implementation of PCI controller has concept of 32 bit serial number (not to be confused with PCI-E serial number). This value is sent in the protocol from the host to indicate SR-IOV VF device is attached to a synthetic NIC.
Using the serial number (instead of MAC address) to associate the two devices avoids lots of potential problems when there are duplicate MAC addresses from tunnels or layered devices.
The patch set is broken into two parts, one is for the PCI controller and the other is for the netvsc device. Normally, these go through different trees but sending them together here for better review. The PCI changes were submitted previously, but the main review comment was "why do you need this?". This is why.
v2 - slot name can be shorter.
remove locking when creating pci_slots; see comment for explaination
Stephen Hemminger (2):
PCI: hv: support reporting serial number as slot information
hv_netvsc: pair VF based on serial number
drivers/net/hyperv/netvsc.c | 3 ++ drivers/net/hyperv/netvsc_drv.c | 58 ++++++++++++++++------------- drivers/pci/controller/pci-hyperv.c | 37 ++++++++++++++++++ 3 files changed, 73 insertions(+), 25 deletions(-)
Signature · 1 line
2.18.0
PATCH v2 1/2 PCI: hv: support reporting serial number as slot information
The Hyper-V host API for PCI provides a unique "serial number" which can be used as basis for sysfs PCI slot table. This can be useful for cases where userspace wants to find the PCI device based on serial number.
When an SR-IOV NIC is added, the host sends an attach message with serial number. The kernel doesn't use the serial number, but it is useful when doing the same thing in a userspace driver such as the DPDK. By having /sys/bus/pci/slots/N it provides a direct way to find the matching PCI device.
There maybe some cases where serial number is not unique such as when using GPU's. But the PCI slot infrastructure will handle that.
This has a side effect which may also be useful. The common udev network device naming policy uses the slot information (rather than PCI address).
--- drivers/pci/controller/pci-hyperv.c | 37 +++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+)
drivers/pci/controller/pci-hyperv.c +37 -0
@@ -89,6 +89,9 @@ static enum pci_protocol_version_t pci_protocol_version;#define STATUS_REVISION_MISMATCH 0xC0000059 +/* space for 32bit serial number as string */ +#define SLOT_NAME_SIZE 11 +/* * Message Types */
@@ -494,6 +497,7 @@ struct hv_pci_dev {struct list_head list_entry;refcount_t refs;enum hv_pcichild_state state; +struct pci_slot *pci_slot;struct pci_function_description desc;bool reported_missing;struct hv_pcibus_device *hbus;
@@ -1457,6 +1461,34 @@ static void prepopulate_bars(struct hv_pcibus_device *hbus)spin_unlock_irqrestore (&hbus->device_list_lock, flags); } +/* +* Assign entries in sysfs pci slot directory. +* +* Note that this function does not need to lock the children list +* because it is called from pci_devices_present_work which +* is serialized with hv_eject_device_work because they are on the +* same ordered workqueue. Therefore hbus->children list will not change +* even when pci_create_slot sleeps. +*/ +static void hv_pci_assign_slots (struct hv_pcibus_device *hbus) +{ +struct hv_pci_dev *hpdev; +char name[SLOT_NAME_SIZE ]; +int slot_nr; + +list_for_each_entry (hpdev, &hbus->children, list_entry) { +if (hpdev->pci_slot) +continue ; + + slot_nr =PCI_SLOT (wslot_to_devfn (hpdev->desc.win_slot.slot)); +snprintf (name,SLOT_NAME_SIZE ,"%u" , hpdev->desc.ser); + hpdev->pci_slot =pci_create_slot (hbus->pci_bus, slot_nr, + name,NULL ); +if (!hpdev->pci_slot) +pr_warn ("pci_create slot %s failed\n" , name); + } +} +/** * create_root_hv_pci_bus() - Expose a new root PCI bus * @hbus: Root PCI bus, as understood by this driver
@@ -1480,6 +1512,7 @@ static int create_root_hv_pci_bus(struct hv_pcibus_device *hbus)pci_lock_rescan_remove ();pci_scan_child_bus (hbus->pci_bus);pci_bus_assign_resources (hbus->pci_bus); +hv_pci_assign_slots (hbus);pci_bus_add_devices (hbus->pci_bus);pci_unlock_rescan_remove (); hbus->state = hv_pcibus_installed;
@@ -1742,6 +1775,7 @@ static void pci_devices_present_work(struct work_struct *work)*/ pci_lock_rescan_remove ();pci_scan_child_bus (hbus->pci_bus); +hv_pci_assign_slots (hbus);pci_unlock_rescan_remove ();break ;
@@ -1858,6 +1892,9 @@ static void hv_eject_device_work(struct work_struct *work)list_del (&hpdev->list_entry);spin_unlock_irqrestore (&hpdev->hbus->device_list_lock, flags); +if (hpdev->pci_slot) +pci_destroy_slot (hpdev->pci_slot); +memset (&ctxt,0 ,sizeof (ctxt)); ejct_pkt = (struct pci_eject_response *)&ctxt.pkt.message; ejct_pkt->message_type.type =PCI_EJECTION_COMPLETE ;
Signature · 1 line
2.18.0
PATCH v2 2/2 hv_netvsc: pair VF based on serial number
In reply to Stephen HemmingerMatching network device based on MAC address is problematic since a non VF network device can be creted with a duplicate MAC address causing confusion and problems. The VMBus API does provide a serial number that is a better matching method.
--- drivers/net/hyperv/netvsc.c | 3 ++ drivers/net/hyperv/netvsc_drv.c | 58 +++++++++++++++++++-------------- 2 files changed, 36 insertions(+), 25 deletions(-)
drivers/net/hyperv/netvsc.c +3 -0
@@ -1203,6 +1203,9 @@ static void netvsc_send_vf(struct net_device *ndev, net_device_ctx->vf_alloc = nvmsg->msg.v4_msg.vf_assoc.allocated; net_device_ctx->vf_serial = nvmsg->msg.v4_msg.vf_assoc.serial; +netdev_info (ndev,"VF slot %u %s\n" , + net_device_ctx->vf_serial, + net_device_ctx->vf_alloc ?"added" :"removed" ); } static void netvsc_receive_inband(struct net_device *ndev,
drivers/net/hyperv/netvsc_drv.c +33 -25
79 lines not on this page · raw message
Signature · 1 line
2.18.0
Re: [PATCH v2 2/2] hv_netvsc: pair VF based on serial number
Quoted 68 lines · Stephen Hemminger
Matching network device based on MAC address is problematic since a non VF network device can be creted with a duplicate MAC address causing confusion and problems. The VMBus API does provide a serial number that is a better matching method.
Signed-off-by Stephen Hemminger <[email protected]>--- drivers/net/hyperv/netvsc.c | 3 ++ drivers/net/hyperv/netvsc_drv.c | 58 +++++++++++++++++++-------------- 2 files changed, 36 insertions(+), 25 deletions(-)drivers/net/hyperv/netvsc.c +3
-0@@ -1203,6 +1203,9 @@ static void netvsc_send_vf(struct net_device *ndev, net_device_ctx->vf_alloc = nvmsg->msg.v4_msg.vf_assoc.allocated; net_device_ctx->vf_serial = nvmsg->msg.v4_msg.vf_assoc.serial; +netdev_info (ndev,"VF slot %u %s\n" , + net_device_ctx->vf_serial, + net_device_ctx->vf_alloc ?"added" :"removed" ); } static void netvsc_receive_inband(struct net_device *ndev,drivers/net/hyperv/netvsc_drv.c +14
-14@@ -1894,20 +1894,6 @@ static void netvsc_link_change(struct work_struct *w)rtnl_unlock (); } -static struct net_device *get_netvsc_bymac (const u8 *mac) -{ -struct net_device_context *ndev_ctx; - -list_for_each_entry (ndev_ctx, &netvsc_dev_list, list) { -struct net_device *dev =hv_get_drvdata (ndev_ctx->device_ctx); - -if (ether_addr_equal (mac, dev->perm_addr)) -return dev; - } - -return NULL ; -} -static struct net_device *get_netvsc_byref (struct net_device *vf_netdev) {struct net_device_context *net_device_ctx;@@ -2036,26 +2022,48 @@ static void netvsc_vf_setup(struct work_struct *w)rtnl_unlock (); } +/* Find netvsc by VMBus serial number. +* The PCI hyperv controller records the serial number as the slot. +*/ +static struct net_device *get_netvsc_byslot (const struct net_device *vf_netdev) +{ +struct device *parent = vf_netdev->dev.parent; +struct net_device_context *ndev_ctx; +struct pci_dev *pdev; + +if (!parent || !dev_is_pci (parent)) +return NULL ;/* not a PCI device */ + + pdev =to_pci_dev (parent); +if (!pdev->slot) {
drivers/net/hyperv/netvsc_drv.c static void netvsc_vf_setup(struct work_struct *w) · line 2048
Quoted 10 lines
drivers/net/hyperv/netvsc_drv.c +10
-0+netdev_notice (vf_netdev,"no PCI slot information\n" ); +return NULL ; + } + +list_for_each_entry (ndev_ctx, &netvsc_dev_list, list) { +if (!ndev_ctx->vf_alloc) +continue ; + +if (ndev_ctx->vf_serial == pdev->slot->number) +return hv_get_drvdata (ndev_ctx->device_ctx);
In patch 1, pdev->slot->number is set to:
PCI_SLOT(wslot_to_devfn(hpdev->desc.win_slot.slot))
so I assume vf_serial is initialized (I have no knowledge of VMBUS and hyper-V internals) to that value, somehow.
I also do not know how the wslot stuff is handled but I assume 5 bits (ie dev bits in devfn) are enough.
BTW, I have noticed this patch (and patch 1) are already in -next so I will drop them from the PCI patch queue.
Lorenzo
Quoted 35 lines
drivers/net/hyperv/netvsc_drv.c +9
-11+ } + +netdev_notice (vf_netdev, +"no netdev found for slot %u\n" , pdev->slot->number); +return NULL ; +} +static int netvsc_register_vf (struct net_device *vf_netdev) { -struct net_device *ndev;struct net_device_context *net_device_ctx; -struct device *pdev = vf_netdev->dev.parent;struct netvsc_device *netvsc_dev; +struct net_device *ndev;int ret;if (vf_netdev->addr_len !=ETH_ALEN )return NOTIFY_DONE ; -if (!pdev || !dev_is_pci (pdev) ||dev_is_pf (pdev)) -return NOTIFY_DONE ; - -/* -* We will use the MAC address to locate the synthetic interface to -* associate with the VF interface. If we don't find a matching -* synthetic interface, move on. -*/ - ndev =get_netvsc_bymac (vf_netdev->perm_addr); + ndev =get_netvsc_byslot (vf_netdev);if (!ndev)return NOTIFY_DONE ;Signature · 1 line
2.18.0
David Miller
maintainerFrom: Stephen Hemminger <[email protected]>
Date: Fri, 14 Sep 2018 12:54:55 -0700
Quoted 8 lines
The Hyper-V implementation of PCI controller has concept of 32 bit serial number (not to be confused with PCI-E serial number). This value is sent in the protocol from the host to indicate SR-IOV VF device is attached to a synthetic NIC.
Using the serial number (instead of MAC address) to associate the two devices avoids lots of potential problems when there are duplicate MAC addresses from tunnels or layered devices.
Quoted 8 lines
The patch set is broken into two parts, one is for the PCI controller and the other is for the netvsc device. Normally, these go through different trees but sending them together here for better review. The PCI changes were submitted previously, but the main review comment was "why do you need this?". This is why.
v2 - slot name can be shorter.
remove locking when creatingpci_slots; see comment for explaination
Series applied, thanks.
Quoted 8 lines · Stephen Hemminger
The Hyper-V implementation of PCI controller has concept of 32 bit serial number (not to be confused with PCI-E serial number). This value is sent in the protocol from the host to indicate SR-IOV VF device is attached to a synthetic NIC.
Using the serial number (instead of MAC address) to associate the two devices avoids lots of potential problems when there are duplicate MAC addresses from tunnels or layered devices.
Quoted 5 lines
The patch set is broken into two parts, one is for the PCI controller and the other is for the netvsc device. Normally, these go through different trees but sending them together here for better review. The PCI changes were submitted previously, but the main review comment was "why do you need this?". This is why.
The question was more whether we should convert this serial number into a PCI slot number (that has user space visibility and that is what you are after) to improve the current matching, I do not question why you need it, just for the records.
Lorenzo
Quoted 15 lines
v2 - slot name can be shorter.
remove locking when creatingpci_slots; see comment for explainationStephen Hemminger (2):
PCI: hv: support reporting serial number as slot information
hv_netvsc: pair VF based on serial numberdrivers/net/hyperv/netvsc.c | 3 ++ drivers/net/hyperv/netvsc_drv.c | 58 ++++++++++++++++------------- drivers/pci/controller/pci-hyperv.c | 37 ++++++++++++++++++ 3 files changed, 73 insertions(+), 25 deletions(-)Signature · 1 line
2.18.0
Quoted 15 lines · On Thu, 20 Sep 2018 15:18:20 +0100 Lorenzo Pieralisi…
Quoted 8 lines · Stephen Hemminger
The Hyper-V implementation of PCI controller has concept of 32 bit serial number (not to be confused with PCI-E serial number). This value is sent in the protocol from the host to indicate SR-IOV VF device is attached to a synthetic NIC.
Using the serial number (instead of MAC address) to associate the two devices avoids lots of potential problems when there are duplicate MAC addresses from tunnels or layered devices.
Quoted 5 lines
The patch set is broken into two parts, one is for the PCI controller and the other is for the netvsc device. Normally, these go through different trees but sending them together here for better review. The PCI changes were submitted previously, but the main review comment was "why do you need this?". This is why.
Quoted 4 lines
The question was more whether we should convert this serial number into a PCI slot number (that has user space visibility and that is what you are after) to improve the current matching, I do not question why you need it, just for the records.
The name slot is way overloaded in this context.
There is
windows slot number which comes from Hyperv
pci address slot which pci-hyperv sets from windows slot pci slot api value which for normal devices comes from ACPI
this patch gets it from serial number
The netvsc driver needed to be able to find a PCI device based on the serial number. The serial number was not visible in any current PCI-hyperv controller values. The windows slot (wslot) is not the same the serial number.