Skip to contentSkip to conversation

Add vmbus dev_num and enable netvsc async probing

PATCH Superseded Haiyang Zhang 4 participants 14 messages Dec 30, 2019 – Dec 31, 2019

Subsystems (4)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Haiyang Zhang

author maintainer
#0 lore raw

v2 cover Add vmbus dev_num and enable netvsc async probing

Add dev_num for vmbus device based on channel offer sequence. Use this number for nic naming, and enable async probing.

Haiyang Zhang (3):
Drivers: hv: vmbus: Add a dev_num variable based on channel offer
sequence
Drivers: hv: vmbus: Add dev_num to sysfs
hv_netvsc: Name NICs based on vmbus offer sequence and use async probe

 Documentation/ABI/stable/sysfs-bus-vmbus |  8 +++++++
 drivers/hv/channel_mgmt.c                | 38 ++++++++++++++++++++++++++++++--
 drivers/hv/vmbus_drv.c                   | 13 +++++++++++
 drivers/net/hyperv/netvsc_drv.c          | 18 ++++++++++++---
 include/linux/hyperv.h                   |  6 +++++
 5 files changed, 78 insertions(+), 5 deletions(-)
Signature · 1 line
1.8.3.1
author maintainer
#1 lore raw

PATCH v2 1/3 Drivers: hv: vmbus: Add a dev_num variable based on channel offer sequence

This number is set to the first available number, starting from zero, when a vmbus device’s primary channel is offered.
It will be used for stable naming when Async probing is used.

---
Changes
V2:
Use nest loops in hv_set_devnum, instead of goto.

 drivers/hv/channel_mgmt.c | 38 ++++++++++++++++++++++++++++++++++++--
 include/linux/hyperv.h    |  6 ++++++
 2 files changed, 42 insertions(+), 2 deletions(-)
drivers/hv/channel_mgmt.c +36 -2
index 8eb1675..00fa2db 100644
@@ -315,6 +315,8 @@ static struct vmbus_channel *alloc_channel(void)
 	if (!channel)
 		return NULL;
 
+	channel->dev_num = HV_DEV_NUM_INVALID;
+
 	spin_lock_init(&channel->lock);
 	init_completion(&channel->rescind_event);
 
@@ -541,6 +543,36 @@ static void vmbus_add_channel_work(struct work_struct *work)
 }
 
 /*
+ * Get the first available device number of its type, then
+ * record it in the channel structure.
+ */
+static void hv_set_devnum(struct vmbus_channel *newchannel)
+{
+	struct vmbus_channel *channel;
+	int i = -1;
+	bool found;
+
+	BUG_ON(!mutex_is_locked(&vmbus_connection.channel_mutex));
+
+	do {
+		i++;
+		found = false;
+
+		list_for_each_entry(channel, &vmbus_connection.chn_list,
+				    listentry) {
+			if (i == channel->dev_num &&
+			    guid_equal(&channel->offermsg.offer.if_type,
+				       &newchannel->offermsg.offer.if_type)) {
+				found = true;
+				break;
+			}
+		}
+	} while (found);
+
+	newchannel->dev_num = i;
+}
+
+/*
  * vmbus_process_offer - Process the offer by creating a channel/device
  * associated with this offer
  */
@@ -573,10 +605,12 @@ static void vmbus_process_offer(struct vmbus_channel *newchannel)
 		}
 	}
 
-	if (fnew)
+	if (fnew) {
+		hv_set_devnum(newchannel);
+
 		list_add_tail(&newchannel->listentry,
 			      &vmbus_connection.chn_list);
-	else {
+	} else {
 		/*
 		 * Check to see if this is a valid sub-channel.
 		 */
include/linux/hyperv.h +6 -0
index 26f3aee..4f110c5 100644
@@ -718,6 +718,8 @@ struct vmbus_device {
 	bool perf_device;
 };
 
+#define HV_DEV_NUM_INVALID (-1)
+
 struct vmbus_channel {
 	struct list_head listentry;
 
@@ -849,6 +851,10 @@ struct vmbus_channel {
 	 */
 	struct vmbus_channel *primary_channel;
 	/*
+	 * Used for device naming based on channel offer sequence.
+	 */
+	int dev_num;
+	/*
 	 * Support per-channel state for use by vmbus drivers.
 	 */
 	void *per_channel_state;
Signature · 1 line
1.8.3.1
#2 lore raw

RE: [PATCH V2,net-next, 1/3] Drivers: hv: vmbus: Add a dev_num variable based on channel offer sequence

From: Haiyang Zhang <[email protected]> Sent: Monday, December 30, 2019 12:14 PM

Quoted 3 lines

This number is set to the first available number, starting from zero, when a vmbus device's primary channel is offered.

Let's use "VMBus" as the capitalization in text.

Quoted 48 lines

It will be used for stable naming when Async probing is used.

---
Changes
V2:
Use nest loops in hv_set_devnum, instead of goto.

 drivers/hv/channel_mgmt.c | 38 ++++++++++++++++++++++++++++++++++++--
 include/linux/hyperv.h    |  6 ++++++
 2 files changed, 42 insertions(+), 2 deletions(-)
drivers/hv/channel_mgmt.c +21 -0
index 8eb1675..00fa2db 100644
@@ -315,6 +315,8 @@ static struct vmbus_channel *alloc_channel(void)
 	if (!channel)
 		return NULL;
 
+	channel->dev_num = HV_DEV_NUM_INVALID;
+
 	spin_lock_init(&channel->lock);
 	init_completion(&channel->rescind_event);
 
@@ -541,6 +543,36 @@ static void vmbus_add_channel_work(struct work_struct *work)
 }
 
 /*
+ * Get the first available device number of its type, then
+ * record it in the channel structure.
+ */
+static void hv_set_devnum(struct vmbus_channel *newchannel)
+{
+	struct vmbus_channel *channel;
+	int i = -1;
+	bool found;
+
+	BUG_ON(!mutex_is_locked(&vmbus_connection.channel_mutex));
+
+	do {
+		i++;
+		found = false;
+
+		list_for_each_entry(channel, &vmbus_connection.chn_list,
+				    listentry) {
+			if (i == channel->dev_num &&
+			    guid_equal(&channel->offermsg.offer.if_type,

drivers/hv/channel_mgmt.c static void vmbus_add_channel_work(struct work_struct *work) · line 574

Quoted 10 lines
drivers/hv/channel_mgmt.c +10 -0
+				       &newchannel->offermsg.offer.if_type)) {
+				found = true;
+				break;
+			}
+		}
+	} while (found);
+
+	newchannel->dev_num = i;
+}
+

It took me a little while to figure out what the above algorithm is doing. Perhaps it would help to rename the "found" variable to "in_use", and add this comment before the start of the "do" loop:

Iterate through each possible device number starting at zero. If the device number is already in use for a device of this type, try the next device number
until finding one that is not in use. This approach selects the smallest
device number that is not in use, and so reuses any numbers that are freed by devices that have been removed.

Quoted 46 lines
drivers/hv/channel_mgmt.c +5 -2
+/*
  * vmbus_process_offer - Process the offer by creating a channel/device
  * associated with this offer
  */
@@ -573,10 +605,12 @@ static void vmbus_process_offer(struct vmbus_channel
 *newchannel)
 		}
 	}
 
-	if (fnew)
+	if (fnew) {
+		hv_set_devnum(newchannel);
+
 		list_add_tail(&newchannel->listentry,
 			      &vmbus_connection.chn_list);
-	else {
+	} else {
 		/*
 		 * Check to see if this is a valid sub-channel.
drivers/hv/channel_mgmt.c +0 -0
 		 */
include/linux/hyperv.h +6 -1
index 26f3aee..4f110c5 100644
@@ -718,6 +718,8 @@ struct vmbus_device {
 	bool perf_device;
 };
 
+#define HV_DEV_NUM_INVALID (-1)
+
 struct vmbus_channel {
 	struct list_head listentry;
 
@@ -849,6 +851,10 @@ struct vmbus_channel {
 	 */
 	struct vmbus_channel *primary_channel;
 	/*
+	 * Used for device naming based on channel offer sequence.
+	 */
+	int dev_num;
+	/*
 	 * Support per-channel state for use by vmbus drivers.
 	 */
 	void *per_channel_state;
--

1.8.3.1

author maintainer
#3 lore raw

RE: [PATCH V2,net-next, 1/3] Drivers: hv: vmbus: Add a dev_num variable based on channel offer sequence

Quoted 14 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Monday, December 30, 2019 8:35 PM
To: Haiyang Zhang <[email protected]>; [email protected]; linux- [email protected]; [email protected]
Cc: Haiyang Zhang <[email protected]>; KY Srinivasan <[email protected]>; Stephen Hemminger <[email protected]>; [email protected]; vkuznets <[email protected]>; [email protected]; [email protected]
Subject: RE: [PATCH V2,net-next, 1/3] Drivers: hv: vmbus: Add a dev_num variable based on channel offer sequence

Quoted 7 lines

From: Haiyang Zhang <[email protected]> Sent: Monday, December 30, 2019 12:14 PM

Quoted 3 lines

This number is set to the first available number, starting from zero, when a vmbus device's primary channel is offered.

Let's use "VMBus" as the capitalization in text.

Sure I will.

Quoted 66 lines
Quoted 10 lines

It will be used for stable naming when Async probing is used.

---
Changes
V2:
Use nest loops in hv_set_devnum, instead of goto.

 drivers/hv/channel_mgmt.c | 38

++++++++++++++++++++++++++++++++++++--

drivers/hv/channel_mgmt.c static void vmbus_add_channel_work(struct

Quoted 17 lines
 include/linux/hyperv.h    |  6 ++++++
 2 files changed, 42 insertions(+), 2 deletions(-)
drivers/hv/channel_mgmt.c +2 -0
index 8eb1675..00fa2db 100644
@@ -315,6 +315,8 @@ static struct vmbus_channel *alloc_channel(void)
 	if (!channel)
 		return NULL;
 
+	channel->dev_num = HV_DEV_NUM_INVALID;
+
 	spin_lock_init(&channel->lock);
 	init_completion(&channel->rescind_event);
 
@@ -541,6 +543,36 @@ static void vmbus_add_channel_work(struct

work_struct *work)

Quoted 22 lines
drivers/hv/channel_mgmt.c +19 -0
 }
 
 /*
+ * Get the first available device number of its type, then
+ * record it in the channel structure.
+ */
+static void hv_set_devnum(struct vmbus_channel *newchannel)
+{
+	struct vmbus_channel *channel;
+	int i = -1;
+	bool found;
+
+	BUG_ON(!mutex_is_locked(&vmbus_connection.channel_mutex));
+
+	do {
+		i++;
+		found = false;
+
+		list_for_each_entry(channel, &vmbus_connection.chn_list,
+				    listentry) {
+			if (i == channel->dev_num &&
+			    guid_equal(&channel->offermsg.offer.if_type,

drivers/hv/channel_mgmt.c static void vmbus_add_channel_work(struct work_struct *work) · line 574

Quoted 10 lines
drivers/hv/channel_mgmt.c +10 -0
+				       &newchannel->offermsg.offer.if_type)) {
+				found = true;
+				break;
+			}
+		}
+	} while (found);
+
+	newchannel->dev_num = i;
+}
+

It took me a little while to figure out what the above algorithm is doing. Perhaps it would help to rename the "found" variable to "in_use", and add this comment before the start of the "do" loop:

Quoted 5 lines

Iterate through each possible device number starting at zero. If the device number is already in use for a device of this type, try the next device number
until finding one that is not in use. This approach selects the smallest
device number that is not in use, and so reuses any numbers that are freed by devices that have been removed.

I will rename the variable, and add the code comments. Thanks,

- Haiyang

author maintainer
#4 lore raw

PATCH v2 2/3 Drivers: hv: vmbus: Add dev_num to sysfs

In reply to Haiyang Zhang

It's a number based on the vmbus device offer sequence. Useful for device naming when using Async probing.

---
 Documentation/ABI/stable/sysfs-bus-vmbus |  8 ++++++++
 drivers/hv/vmbus_drv.c                   | 13 +++++++++++++
 2 files changed, 21 insertions(+)
Documentation/ABI/stable/sysfs-bus-vmbus +8 -0
index 8e8d167..a42225d 100644
@@ -49,6 +49,14 @@ Contact:	Stephen Hemminger <[email protected]>
 Description:	This NUMA node to which the VMBUS device is
 		attached, or -1 if the node is unknown.
 
+What:           /sys/bus/vmbus/devices/<UUID>/dev_num
+Date:		Dec 2019
+KernelVersion:	5.5
+Contact:	Haiyang Zhang <[email protected]>
+Description:	A number based on the vmbus device offer sequence.
+		Useful for device naming when using Async probing.
+Users:		Debugging tools and userspace drivers
+
 What:		/sys/bus/vmbus/devices/<UUID>/channels/<N>
 Date:		September. 2017
 KernelVersion:	4.14
drivers/hv/vmbus_drv.c +13 -0
index 4ef5a66..fe7aefa 100644
@@ -214,6 +214,18 @@ static ssize_t numa_node_show(struct device *dev,
 static DEVICE_ATTR_RO(numa_node);
 #endif
 
+static ssize_t dev_num_show(struct device *dev,
+			      struct device_attribute *attr, char *buf)
+{
+	struct hv_device *hv_dev = device_to_hv_device(dev);
+
+	if (!hv_dev->channel)
+		return -ENODEV;
+
+	return sprintf(buf, "%d\n", hv_dev->channel->dev_num);
+}
+static DEVICE_ATTR_RO(dev_num);
+
 static ssize_t server_monitor_pending_show(struct device *dev,
 					   struct device_attribute *dev_attr,
 					   char *buf)
@@ -598,6 +610,7 @@ static ssize_t driver_override_show(struct device *dev,
 #ifdef CONFIG_NUMA
 	&dev_attr_numa_node.attr,
 #endif
+	&dev_attr_dev_num.attr,
 	&dev_attr_server_monitor_pending.attr,
 	&dev_attr_client_monitor_pending.attr,
 	&dev_attr_server_monitor_latency.attr,
Signature · 1 line
1.8.3.1
#5 lore raw

RE: [PATCH V2,net-next, 2/3] Drivers: hv: vmbus: Add dev_num to sysfs

From: Haiyang Zhang <[email protected]> Sent: Monday, December 30, 2019 12:14 PM

Quoted 2 lines

It's a number based on the vmbus device offer sequence.

Let's use "VMBus" as the capitalization.

Quoted 14 lines

Useful for device naming when using Async probing.

---
 Documentation/ABI/stable/sysfs-bus-vmbus |  8 ++++++++
 drivers/hv/vmbus_drv.c                   | 13 +++++++++++++
 2 files changed, 21 insertions(+)
Documentation/ABI/stable/sysfs-bus-vmbus +0 -0
index 8e8d167..a42225d 100644

Documentation/ABI/stable/sysfs-bus-vmbus Contact: Stephen Hemminger <[email protected]> · line 56

Quoted 8 lines
Documentation/ABI/stable/sysfs-bus-vmbus +5 -0
@@ -49,6 +49,14 @@ Contact:	Stephen Hemminger <[email protected]>
 Description:	This NUMA node to which the VMBUS device is
 		attached, or -1 if the node is unknown.
 
+What:           /sys/bus/vmbus/devices/<UUID>/dev_num
+Date:		Dec 2019
+KernelVersion:	5.5
+Contact:	Haiyang Zhang <[email protected]>
+Description:	A number based on the vmbus device offer sequence.

The capitalization of "VMBus" is already inconsistent in this file in that we have "VMBus" and "VMBUS" in the text. Let's not introduce another capitalization -- use "VMBus".

Quoted 39 lines
Documentation/ABI/stable/sysfs-bus-vmbus +3 -0
+		Useful for device naming when using Async probing.
+Users:		Debugging tools and userspace drivers
+
 What:		/sys/bus/vmbus/devices/<UUID>/channels/<N>
 Date:		September. 2017
 KernelVersion:	4.14
drivers/hv/vmbus_drv.c +13 -1
index 4ef5a66..fe7aefa 100644
@@ -214,6 +214,18 @@ static ssize_t numa_node_show(struct device *dev,
 static DEVICE_ATTR_RO(numa_node);
 #endif
 
+static ssize_t dev_num_show(struct device *dev,
+			      struct device_attribute *attr, char *buf)
+{
+	struct hv_device *hv_dev = device_to_hv_device(dev);
+
+	if (!hv_dev->channel)
+		return -ENODEV;
+
+	return sprintf(buf, "%d\n", hv_dev->channel->dev_num);
+}
+static DEVICE_ATTR_RO(dev_num);
+
 static ssize_t server_monitor_pending_show(struct device *dev,
 					   struct device_attribute *dev_attr,
 					   char *buf)
@@ -598,6 +610,7 @@ static ssize_t driver_override_show(struct device *dev,
 #ifdef CONFIG_NUMA
 	&dev_attr_numa_node.attr,
 #endif
+	&dev_attr_dev_num.attr,
 	&dev_attr_server_monitor_pending.attr,
 	&dev_attr_client_monitor_pending.attr,
 	&dev_attr_server_monitor_latency.attr,
--

1.8.3.1

author maintainer
#6 lore raw

RE: [PATCH V2,net-next, 2/3] Drivers: hv: vmbus: Add dev_num to sysfs

Quoted 13 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Monday, December 30, 2019 8:35 PM
To: Haiyang Zhang <[email protected]>; [email protected]; linux- [email protected]; [email protected]
Cc: Haiyang Zhang <[email protected]>; KY Srinivasan <[email protected]>; Stephen Hemminger <[email protected]>; [email protected]; vkuznets <[email protected]>; [email protected]; [email protected]
Subject: RE: [PATCH V2,net-next, 2/3] Drivers: hv: vmbus: Add dev_num to sysfs

Quoted 6 lines

From: Haiyang Zhang <[email protected]> Sent: Monday, December 30, 2019 12:14 PM

Quoted 2 lines

It's a number based on the vmbus device offer sequence.

Let's use "VMBus" as the capitalization.

I will.

Quoted 19 lines

Documentation/ABI/stable/sysfs-bus-vmbus Contact: Stephen Hemminger

Quoted 15 lines

Useful for device naming when using Async probing.

---
 Documentation/ABI/stable/sysfs-bus-vmbus |  8 ++++++++
 drivers/hv/vmbus_drv.c                   | 13 +++++++++++++
 2 files changed, 21 insertions(+)
Documentation/ABI/stable/sysfs-bus-vmbus +0 -0
index 8e8d167..a42225d 100644
@@ -49,6 +49,14 @@ Contact:	Stephen Hemminger

<[email protected]>

Quoted 3 lines
Documentation/ABI/stable/sysfs-bus-vmbus +0 -0
 Description:	This NUMA node to which the VMBUS device is
 		attached, or -1 if the node is unknown.
 
Quoted 9 lines

Documentation/ABI/stable/sysfs-bus-vmbus Contact: Stephen Hemminger <[email protected]> · line 56

Quoted 5 lines
Documentation/ABI/stable/sysfs-bus-vmbus +5 -0
+What:           /sys/bus/vmbus/devices/<UUID>/dev_num
+Date:		Dec 2019
+KernelVersion:	5.5
+Contact:	Haiyang Zhang <[email protected]>
+Description:	A number based on the vmbus device offer sequence.

The capitalization of "VMBus" is already inconsistent in this file in that we have "VMBus" and "VMBUS" in the text. Let's not introduce another capitalization -- use "VMBus".

I will use "VMBus".
Thanks,
- Haiyang

author maintainer
#7 lore raw

PATCH v2 3/3 hv_netvsc: Name NICs based on vmbus offer sequence and use async probe

In reply to Haiyang Zhang

The dev_num field in vmbus channel structure is assigned to the first available number when the channel is offered. So netvsc driver uses it for NIC naming based on channel offer sequence. Now re-enable the async probing mode for faster probing.

---
 drivers/net/hyperv/netvsc_drv.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
drivers/net/hyperv/netvsc_drv.c +15 -3
index f3f9eb8..39c412f 100644
@@ -2267,10 +2267,14 @@ static int netvsc_probe(struct hv_device *dev,
 	struct net_device_context *net_device_ctx;
 	struct netvsc_device_info *device_info = NULL;
 	struct netvsc_device *nvdev;
+	char name[IFNAMSIZ];
 	int ret = -ENOMEM;
 
-	net = alloc_etherdev_mq(sizeof(struct net_device_context),
-				VRSS_CHANNEL_MAX);
+	snprintf(name, IFNAMSIZ, "eth%d", dev->channel->dev_num);
+	net = alloc_netdev_mqs(sizeof(struct net_device_context), name,
+			       NET_NAME_ENUM, ether_setup,
+			       VRSS_CHANNEL_MAX, VRSS_CHANNEL_MAX);
+
 	if (!net)
 		goto no_net;
 
@@ -2355,6 +2359,14 @@ static int netvsc_probe(struct hv_device *dev,
 		net->max_mtu = ETH_DATA_LEN;
 
 	ret = register_netdevice(net);
+
+	if (ret == -EEXIST) {
+		pr_info("NIC name %s exists, request another name.\n",
+			net->name);
+		strlcpy(net->name, "eth%d", IFNAMSIZ);
+		ret = register_netdevice(net);
+	}
+
 	if (ret != 0) {
 		pr_err("Unable to register netdev.\n");
 		goto register_failed;
@@ -2496,7 +2508,7 @@ static int netvsc_resume(struct hv_device *dev)
 	.suspend = netvsc_suspend,
 	.resume = netvsc_resume,
 	.driver = {
-		.probe_type = PROBE_FORCE_SYNCHRONOUS,
+		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
 	},
 };
 
Signature · 1 line
1.8.3.1
#8 lore raw

RE: [PATCH V2,net-next, 3/3] hv_netvsc: Name NICs based on vmbus offer sequence and use async probe

From: Haiyang Zhang <[email protected]> Sent: Monday, December 30, 2019 12:14 PM

Quoted 2 lines

The dev_num field in vmbus channel structure is assigned to the first

Let's use "VMBus" in text.

Quoted 31 lines

available number when the channel is offered. So netvsc driver uses it for NIC naming based on channel offer sequence. Now re-enable the async probing mode for faster probing.

---
 drivers/net/hyperv/netvsc_drv.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
drivers/net/hyperv/netvsc_drv.c +6 -2
index f3f9eb8..39c412f 100644
@@ -2267,10 +2267,14 @@ static int netvsc_probe(struct hv_device *dev,
 	struct net_device_context *net_device_ctx;
 	struct netvsc_device_info *device_info = NULL;
 	struct netvsc_device *nvdev;
+	char name[IFNAMSIZ];
 	int ret = -ENOMEM;
 
-	net = alloc_etherdev_mq(sizeof(struct net_device_context),
-				VRSS_CHANNEL_MAX);
+	snprintf(name, IFNAMSIZ, "eth%d", dev->channel->dev_num);
+	net = alloc_netdev_mqs(sizeof(struct net_device_context), name,
+			       NET_NAME_ENUM, ether_setup,
+			       VRSS_CHANNEL_MAX, VRSS_CHANNEL_MAX);
+
 	if (!net)
 		goto no_net;
 

drivers/net/hyperv/netvsc_drv.c static int netvsc_probe(struct hv_device *dev, · line 2367

Quoted 9 lines
drivers/net/hyperv/netvsc_drv.c +6 -0
@@ -2355,6 +2359,14 @@ static int netvsc_probe(struct hv_device *dev,
 		net->max_mtu = ETH_DATA_LEN;
 
 	ret = register_netdevice(net);
+
+	if (ret == -EEXIST) {
+		pr_info("NIC name %s exists, request another name.\n",
+			net->name);
+		strlcpy(net->name, "eth%d", IFNAMSIZ);
+		ret = register_netdevice(net);

The message above could be confusing. "request another name" sounds like a directive to the user/sysadmin, which I don't think it is. Perhaps better would be "requesting another name", which says the system is handling the collision automatically.

Also will this second call to register_netdevice() actually assign a numeric name? If so, it would be really nice for the message to be output *after* the second call to register_netdevice() and to show both the originally selected name that collided as well as the new name. We sometimes get into NIC naming issues with customers in Azure, and when a new name has to be selected it will help debugging if we can show both the original selection and the new selection. With both pieces of data, there's less guessing about who did what regarding NIC naming.

Finally, having to choose a different name because of a collision does *not* update the channel->dev_num value. Subsequent calls to hv_set_devnum() will detect "in use" based on the originally assigned dev_num value. I don't think that fundamentally breaks anything, but
I wondered if you had thought about that case. Maybe a comment here
to that effect would help a future reader of this code.

Michael

Quoted 16 lines
drivers/net/hyperv/netvsc_drv.c +3 -2
+	}
+
 	if (ret != 0) {
 		pr_err("Unable to register netdev.\n");
 		goto register_failed;
@@ -2496,7 +2508,7 @@ static int netvsc_resume(struct hv_device *dev)
 	.suspend = netvsc_suspend,
 	.resume = netvsc_resume,
 	.driver = {
-		.probe_type = PROBE_FORCE_SYNCHRONOUS,
+		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
 	},
 };
 
--

1.8.3.1

author maintainer
#9 lore raw

RE: [PATCH V2,net-next, 3/3] hv_netvsc: Name NICs based on vmbus offer sequence and use async probe

Quoted 14 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Monday, December 30, 2019 8:35 PM
To: Haiyang Zhang <[email protected]>; [email protected]; linux- [email protected]; [email protected]
Cc: Haiyang Zhang <[email protected]>; KY Srinivasan <[email protected]>; Stephen Hemminger <[email protected]>; [email protected]; vkuznets <[email protected]>; [email protected]; [email protected]
Subject: RE: [PATCH V2,net-next, 3/3] hv_netvsc: Name NICs based on vmbus offer sequence and use async probe

Quoted 6 lines

From: Haiyang Zhang <[email protected]> Sent: Monday, December 30, 2019 12:14 PM

Quoted 2 lines

The dev_num field in vmbus channel structure is assigned to the first

Let's use "VMBus" in text.

I will.

Quoted 47 lines
Quoted 32 lines

available number when the channel is offered. So netvsc driver uses it for NIC naming based on channel offer sequence. Now re-enable the async probing mode for faster probing.

---
 drivers/net/hyperv/netvsc_drv.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
drivers/net/hyperv/netvsc_drv.c +6 -2
index f3f9eb8..39c412f 100644
@@ -2267,10 +2267,14 @@ static int netvsc_probe(struct hv_device *dev,
 	struct net_device_context *net_device_ctx;
 	struct netvsc_device_info *device_info = NULL;
 	struct netvsc_device *nvdev;
+	char name[IFNAMSIZ];
 	int ret = -ENOMEM;
 
-	net = alloc_etherdev_mq(sizeof(struct net_device_context),
-				VRSS_CHANNEL_MAX);
+	snprintf(name, IFNAMSIZ, "eth%d", dev->channel->dev_num);
+	net = alloc_netdev_mqs(sizeof(struct net_device_context), name,
+			       NET_NAME_ENUM, ether_setup,
+			       VRSS_CHANNEL_MAX, VRSS_CHANNEL_MAX);
+
 	if (!net)
 		goto no_net;
 

drivers/net/hyperv/netvsc_drv.c static int netvsc_probe(struct hv_device *dev, · line 2367

Quoted 9 lines
drivers/net/hyperv/netvsc_drv.c +6 -0
@@ -2355,6 +2359,14 @@ static int netvsc_probe(struct hv_device *dev,
 		net->max_mtu = ETH_DATA_LEN;
 
 	ret = register_netdevice(net);
+
+	if (ret == -EEXIST) {
+		pr_info("NIC name %s exists, request another name.\n",
+			net->name);
+		strlcpy(net->name, "eth%d", IFNAMSIZ);
+		ret = register_netdevice(net);

The message above could be confusing. "request another name" sounds like a directive to the user/sysadmin, which I don't think it is. Perhaps better would be "requesting another name", which says the system is handling the collision automatically.

Quoted 8 lines

Also will this second call to register_netdevice() actually assign a numeric name? If so, it would be really nice for the message to be output *after* the second call to register_netdevice() and to show both the originally selected name that collided as well as the new name. We sometimes get into NIC naming issues with customers in Azure, and when a new name has to be selected it will help debugging if we can show both the original selection and the new selection. With both pieces of data, there's less guessing about who did what regarding NIC naming.

Good idea! I will include both new and old names in the log messages.

Quoted 7 lines

Finally, having to choose a different name because of a collision does *not* update the channel->dev_num value. Subsequent calls to hv_set_devnum() will detect "in use" based on the originally assigned dev_num value. I don't think that fundamentally breaks anything, but
I wondered if you had thought about that case. Maybe a comment here
to that effect would help a future reader of this code.

That's correct. And I'm aware of this situation. But, the dev_num shouldn't be changed, because we want it to keep track of the sequence of device offering.
Avoid single or multiple collisions, we should use udev or other daemons to set name of VF or other types of NICs to different formats, like "vf*", or "enP*", etc. For distros have not done so already, we need to work with the distro vendors to ensure udev or other settings are correctly included in Distros, and Azure images.

Thanks,
- Haiyang

#10 lore raw

Re: [PATCH V2,net-next, 3/3] hv_netvsc: Name NICs based on vmbus offer sequence and use async probe

In reply to Haiyang Zhang
Quoted 15 lines · Haiyang Zhang

The dev_num field in vmbus channel structure is assigned to the first available number when the channel is offered. So netvsc driver uses it for NIC naming based on channel offer sequence. Now re-enable the async probing mode for faster probing.

---
 drivers/net/hyperv/netvsc_drv.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
drivers/net/hyperv/netvsc_drv.c +0 -0
index f3f9eb8..39c412f 100644

drivers/net/hyperv/netvsc_drv.c static int netvsc_probe(struct hv_device *dev, · line 2273

Quoted 9 lines
drivers/net/hyperv/netvsc_drv.c +2 -2
@@ -2267,10 +2267,14 @@ static int netvsc_probe(struct hv_device *dev,
 	struct net_device_context *net_device_ctx;
 	struct netvsc_device_info *device_info = NULL;
 	struct netvsc_device *nvdev;
+	char name[IFNAMSIZ];
 	int ret = -ENOMEM;
 
-	net = alloc_etherdev_mq(sizeof(struct net_device_context),
-				VRSS_CHANNEL_MAX);
+	snprintf(name, IFNAMSIZ, "eth%d", dev->channel->dev_num);

How is this supposed to work when there are other ethernet device types on the system, which may claim the same device names?

Quoted 8 lines
drivers/net/hyperv/netvsc_drv.c +4 -0
+	net = alloc_netdev_mqs(sizeof(struct net_device_context), name,
+			       NET_NAME_ENUM, ether_setup,
+			       VRSS_CHANNEL_MAX, VRSS_CHANNEL_MAX);
+
 	if (!net)
 		goto no_net;
 

drivers/net/hyperv/netvsc_drv.c static int netvsc_probe(struct hv_device *dev, · line 2368

Quoted 10 lines
drivers/net/hyperv/netvsc_drv.c +7 -0
@@ -2355,6 +2359,14 @@ static int netvsc_probe(struct hv_device *dev,
 		net->max_mtu = ETH_DATA_LEN;
 
 	ret = register_netdevice(net);
+
+	if (ret == -EEXIST) {
+		pr_info("NIC name %s exists, request another name.\n",
+			net->name);
+		strlcpy(net->name, "eth%d", IFNAMSIZ);
+		ret = register_netdevice(net);
+	}

IOW you want the device naming to be predictable, but don't guarantee this?

I think the problem this patchset is trying to solve is much better solved with a udev rule, similar to how it's done for PCI net devices. And IMO the primary channel number, being a device's "hardware" property, is more suited to be used in the device name, than this completely ephemeral device number.

Thanks,
Roman.

Quoted 16 lines
drivers/net/hyperv/netvsc_drv.c +2 -1
+
 	if (ret != 0) {
 		pr_err("Unable to register netdev.\n");
 		goto register_failed;
@@ -2496,7 +2508,7 @@ static int netvsc_resume(struct hv_device *dev)
 	.suspend = netvsc_suspend,
 	.resume = netvsc_resume,
 	.driver = {
-		.probe_type = PROBE_FORCE_SYNCHRONOUS,
+		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
 	},
 };
 
Signature · 1 line
1.8.3.1
author maintainer
#11 lore raw

RE: [PATCH V2,net-next, 3/3] hv_netvsc: Name NICs based on vmbus offer sequence and use async probe

Quoted 63 lines

-----Original Message-----
From: Roman Kagan <[email protected]>
Sent: Tuesday, December 31, 2019 6:35 AM
To: Haiyang Zhang <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; KY Srinivasan <[email protected]>; Stephen Hemminger <[email protected]>; [email protected]; vkuznets <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH V2,net-next, 3/3] hv_netvsc: Name NICs based on vmbus offer sequence and use async probe

Quoted 15 lines · Haiyang Zhang

The dev_num field in vmbus channel structure is assigned to the first available number when the channel is offered. So netvsc driver uses it for NIC naming based on channel offer sequence. Now re-enable the async probing mode for faster probing.

---
 drivers/net/hyperv/netvsc_drv.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
(quoted diff) +0 -0

b/drivers/net/hyperv/netvsc_drv.c index f3f9eb8..39c412f 100644

drivers/net/hyperv/netvsc_drv.c static int netvsc_probe(struct hv_device *dev, · line 2273

Quoted 9 lines
drivers/net/hyperv/netvsc_drv.c +2 -2
@@ -2267,10 +2267,14 @@ static int netvsc_probe(struct hv_device *dev,
 	struct net_device_context *net_device_ctx;
 	struct netvsc_device_info *device_info = NULL;
 	struct netvsc_device *nvdev;
+	char name[IFNAMSIZ];
 	int ret = -ENOMEM;
 
-	net = alloc_etherdev_mq(sizeof(struct net_device_context),
-				VRSS_CHANNEL_MAX);
+	snprintf(name, IFNAMSIZ, "eth%d", dev->channel->dev_num);

How is this supposed to work when there are other ethernet device types on the system, which may claim the same device names?

Quoted 8 lines
drivers/net/hyperv/netvsc_drv.c +4 -0
+	net = alloc_netdev_mqs(sizeof(struct net_device_context), name,
+			       NET_NAME_ENUM, ether_setup,
+			       VRSS_CHANNEL_MAX, VRSS_CHANNEL_MAX);
+
 	if (!net)
 		goto no_net;
 

drivers/net/hyperv/netvsc_drv.c static int netvsc_probe(struct hv_device *dev, · line 2368

Quoted 10 lines
drivers/net/hyperv/netvsc_drv.c +7 -0
@@ -2355,6 +2359,14 @@ static int netvsc_probe(struct hv_device *dev,
 		net->max_mtu = ETH_DATA_LEN;
 
 	ret = register_netdevice(net);
+
+	if (ret == -EEXIST) {
+		pr_info("NIC name %s exists, request another name.\n",
+			net->name);
+		strlcpy(net->name, "eth%d", IFNAMSIZ);
+		ret = register_netdevice(net);
+	}

IOW you want the device naming to be predictable, but don't guarantee this?

Quoted 5 lines

I think the problem this patchset is trying to solve is much better solved with a udev rule, similar to how it's done for PCI net devices.
And IMO the primary channel number, being a device's "hardware" property, is more suited to be used in the device name, than this completely ephemeral device number.

The vmbus number can be affected by other types of devices and/or subchannel offerings. They are not stable either. That's why this patch set keeps track of the offering sequence within the same device type in a new variable "dev_num".

As in my earlier email, to avoid impact by other types of NICs, we should put them into different naming formats, like "vf*", "enP*", etc. And yes, these can be done in udev.

But for netvsc (synthetic) NICs, we still want the default naming format "eth*". And the variable "dev_num" gives them the basis for stable naming with Async probing.

Thanks,
- Haiyang

#12 lore raw

Re: [PATCH V2,net-next, 3/3] hv_netvsc: Name NICs based on vmbus offer sequence and use async probe

Quoted 75 lines · On Tue, 31 Dec 2019 16:12:36 +0000 Haiyang Zhang <[email protected]> wrote:
Quoted 61 lines

-----Original Message-----
From: Roman Kagan <[email protected]>
Sent: Tuesday, December 31, 2019 6:35 AM
To: Haiyang Zhang <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; KY Srinivasan <[email protected]>; Stephen Hemminger <[email protected]>; [email protected]; vkuznets <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH V2,net-next, 3/3] hv_netvsc: Name NICs based on vmbus offer sequence and use async probe

Quoted 15 lines · Haiyang Zhang

The dev_num field in vmbus channel structure is assigned to the first available number when the channel is offered. So netvsc driver uses it for NIC naming based on channel offer sequence. Now re-enable the async probing mode for faster probing.

---
 drivers/net/hyperv/netvsc_drv.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
(quoted diff) +0 -0

b/drivers/net/hyperv/netvsc_drv.c index f3f9eb8..39c412f 100644

drivers/net/hyperv/netvsc_drv.c static int netvsc_probe(struct hv_device *dev, · line 2273

Quoted 9 lines
drivers/net/hyperv/netvsc_drv.c +2 -2
@@ -2267,10 +2267,14 @@ static int netvsc_probe(struct hv_device *dev,
 	struct net_device_context *net_device_ctx;
 	struct netvsc_device_info *device_info = NULL;
 	struct netvsc_device *nvdev;
+	char name[IFNAMSIZ];
 	int ret = -ENOMEM;
 
-	net = alloc_etherdev_mq(sizeof(struct net_device_context),
-				VRSS_CHANNEL_MAX);
+	snprintf(name, IFNAMSIZ, "eth%d", dev->channel->dev_num);  

How is this supposed to work when there are other ethernet device types on the system, which may claim the same device names?

Quoted 8 lines
drivers/net/hyperv/netvsc_drv.c +4 -0
+	net = alloc_netdev_mqs(sizeof(struct net_device_context), name,
+			       NET_NAME_ENUM, ether_setup,
+			       VRSS_CHANNEL_MAX, VRSS_CHANNEL_MAX);
+
 	if (!net)
 		goto no_net;
 

drivers/net/hyperv/netvsc_drv.c static int netvsc_probe(struct hv_device *dev, · line 2368

Quoted 10 lines
drivers/net/hyperv/netvsc_drv.c +7 -0
@@ -2355,6 +2359,14 @@ static int netvsc_probe(struct hv_device *dev,
 		net->max_mtu = ETH_DATA_LEN;
 
 	ret = register_netdevice(net);
+
+	if (ret == -EEXIST) {
+		pr_info("NIC name %s exists, request another name.\n",
+			net->name);
+		strlcpy(net->name, "eth%d", IFNAMSIZ);
+		ret = register_netdevice(net);
+	}  

IOW you want the device naming to be predictable, but don't guarantee this?

Quoted 5 lines

I think the problem this patchset is trying to solve is much better solved with a udev rule, similar to how it's done for PCI net devices.
And IMO the primary channel number, being a device's "hardware" property, is more suited to be used in the device name, than this completely ephemeral device number.

The vmbus number can be affected by other types of devices and/or subchannel offerings. They are not stable either. That's why this patch set keeps track of the offering sequence within the same device type in a new variable "dev_num".

As in my earlier email, to avoid impact by other types of NICs, we should put them into different naming formats, like "vf*", "enP*", etc. And yes, these can be done in udev.

Quoted 6 lines

But for netvsc (synthetic) NICs, we still want the default naming format "eth*". And the variable "dev_num" gives them the basis for stable naming with Async probing.

Thanks,
- Haiyang

The primary requirements for network naming are:
1. Network names must be repeatable on each boot. This was the original problem that PCI devices discovered back years ago when parallel probing was enabled.
2. Network names must be predictable. If new VM is created, the names should match a similar VM config.
3. Names must be persistent. If a NIC is added or deleted, the existing names must not change.

The other things which are important (and this proposal breaks):
1. Don't break it principle: an existing VM should not suddenly get interfaces renamed if kernel is upgraded. A corrallary is that a lot of current userspace code expects eth0. It doesn't look like first interface would be guaranteed to be eth0.

2. No snowflakes principle: a device driver should follow the current practice of other devices. For netvsc, this means VMBus should act like PCI as much as possible. Is there another driver doing this already?

3. Userspace policy principle: Every distribution has its own policy by now. The solution must make netvsc work reliably on Redhat (udev), Ubuntu (netplan), SuSE (Yast) doing something in the kernel violates #2.

My recommendation would be to take a multi-phase approach:
1. Expose persistent value in sysfs now.
2. Work with udev/netplan/... to use that value.
3. Make parallel VMBus probing an option. So that when distributions have picked up the udev changes they can enable parallel probe. Some will be quick to adopt and the enterprise laggards can get to it when they feel the heat.

Long term wish list (requires host side changes):
1. The interface index could be a host side property; the host networking already has the virtual device table and it is persistent.
2. The Azure NIC name should be visible as a property in guest. Then userspace could do rename based on that property. Having multiple disconnected names is leads to confusion.

author maintainer
#13 lore raw

RE: [PATCH V2,net-next, 3/3] hv_netvsc: Name NICs based on vmbus offer sequence and use async probe

Quoted 145 lines

-----Original Message-----
From: [email protected] <linux-hyperv- [email protected]> On Behalf Of Stephen Hemminger
Sent: Tuesday, December 31, 2019 12:36 PM
To: Haiyang Zhang <[email protected]>
Cc: Roman Kagan <[email protected]>; [email protected]; linux- [email protected]; [email protected]; KY Srinivasan <[email protected]>; Stephen Hemminger <[email protected]>; [email protected]; vkuznets <[email protected]>; [email protected]
Subject: Re: [PATCH V2,net-next, 3/3] hv_netvsc: Name NICs based on vmbus offer sequence and use async probe

Quoted 5 lines · On Tue, 31 Dec 2019 16:12:36 +0000 Haiyang Zhang <[email protected]> wrote:
Quoted 5 lines

-----Original Message-----
From: Roman Kagan <[email protected]>
Sent: Tuesday, December 31, 2019 6:35 AM
To: Haiyang Zhang <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 3 lines
Quoted 3 lines

KY Srinivasan <[email protected]>; Stephen Hemminger <[email protected]>; [email protected]; vkuznets <[email protected]>; [email protected]; linux-

[email protected]

Quoted 1 line
Quoted 1 line

Subject: Re: [PATCH V2,net-next, 3/3] hv_netvsc: Name NICs based on

vmbus

Quoted 24 lines
Quoted 24 lines

offer sequence and use async probe

Quoted 21 lines · Haiyang Zhang

The dev_num field in vmbus channel structure is assigned to the first available number when the channel is offered. So netvsc driver uses it for NIC naming based on channel offer sequence. Now re-enable the async probing mode for faster probing.

---
 drivers/net/hyperv/netvsc_drv.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
(quoted diff) +0 -0

b/drivers/net/hyperv/netvsc_drv.c index f3f9eb8..39c412f 100644

drivers/net/hyperv/netvsc_drv.c +1 -0
@@ -2267,10 +2267,14 @@ static int netvsc_probe(struct hv_device *dev,
 	struct net_device_context *net_device_ctx;
 	struct netvsc_device_info *device_info = NULL;
 	struct netvsc_device *nvdev;
+	char name[IFNAMSIZ];
 	int ret = -ENOMEM;
 
Quoted 5 lines
Quoted 5 lines

drivers/net/hyperv/netvsc_drv.c static int netvsc_probe(struct hv_device *dev, · line 2273

Quoted 3 lines
drivers/net/hyperv/netvsc_drv.c +1 -2
-	net = alloc_etherdev_mq(sizeof(struct net_device_context),
-				VRSS_CHANNEL_MAX);
+	snprintf(name, IFNAMSIZ, "eth%d", dev->channel->dev_num);

How is this supposed to work when there are other ethernet device types on

the

Quoted 23 lines
Quoted 23 lines

system, which may claim the same device names?

Quoted 8 lines
drivers/net/hyperv/netvsc_drv.c +4 -0
+	net = alloc_netdev_mqs(sizeof(struct net_device_context), name,
+			       NET_NAME_ENUM, ether_setup,
+			       VRSS_CHANNEL_MAX, VRSS_CHANNEL_MAX);
+
 	if (!net)
 		goto no_net;
 

drivers/net/hyperv/netvsc_drv.c static int netvsc_probe(struct hv_device *dev, · line 2368

Quoted 10 lines
drivers/net/hyperv/netvsc_drv.c +7 -0
@@ -2355,6 +2359,14 @@ static int netvsc_probe(struct hv_device *dev,
 		net->max_mtu = ETH_DATA_LEN;
 
 	ret = register_netdevice(net);
+
+	if (ret == -EEXIST) {
+		pr_info("NIC name %s exists, request another name.\n",
+			net->name);
+		strlcpy(net->name, "eth%d", IFNAMSIZ);
+		ret = register_netdevice(net);
+	}

IOW you want the device naming to be predictable, but don't guarantee this?

Quoted 1 line
Quoted 1 line

I think the problem this patchset is trying to solve is much better solved with

a

Quoted 6 lines
Quoted 4 lines

udev rule, similar to how it's done for PCI net devices.
And IMO the primary channel number, being a device's "hardware" property, is more suited to be used in the device name, than this completely ephemeral device number.

The vmbus number can be affected by other types of devices and/or

subchannel

Quoted 1 line

offerings. They are not stable either. That's why this patch set keeps track of

the

Quoted 3 lines

offering sequence within the same device type in a new variable "dev_num".

As in my earlier email, to avoid impact by other types of NICs, we should put

them

Quoted 1 line

into different naming formats, like "vf*", "enP*", etc. And yes, these can be

done in

Quoted 3 lines

udev.

But for netvsc (synthetic) NICs, we still want the default naming format "eth*".

And

Quoted 1 line

the variable "dev_num" gives them the basis for stable naming with Async

probing.

Quoted 4 lines

Thanks,
- Haiyang

The primary requirements for network naming are:
1. Network names must be repeatable on each boot. This was the original
problem
that PCI devices discovered back years ago when parallel probing was
enabled.
2. Network names must be predictable. If new VM is created, the names
should
match a similar VM config.
3. Names must be persistent. If a NIC is added or deleted, the existing names must not change.

The other things which are important (and this proposal breaks):
1. Don't break it principle: an existing VM should not suddenly get interfaces renamed if kernel is upgraded. A corrallary is that a lot of current userspace code expects eth0. It doesn't look like first interface would be guaranteed to be eth0.

2. No snowflakes principle: a device driver should follow the current practice of other devices. For netvsc, this means VMBus should act like PCI as much as possible. Is there another driver doing this already?

3. Userspace policy principle: Every distribution has its own policy by now. The solution must make netvsc work reliably on Redhat (udev), Ubuntu
(netplan), SuSE (Yast)
doing something in the kernel violates #2.

My recommendation would be to take a multi-phase approach:
1. Expose persistent value in sysfs now.
2. Work with udev/netplan/... to use that value.
3. Make parallel VMBus probing an option. So that when distributions have
picked up
the udev changes they can enable parallel probe. Some will be quick to
adopt
and the enterprise laggards can get to it when they feel the heat.

Quoted 6 lines

Long term wish list (requires host side changes):
1. The interface index could be a host side property; the host networking already has the virtual device table and it is persistent.
2. The Azure NIC name should be visible as a property in guest. Then userspace could do rename based on that property. Having multiple disconnected names is leads to confusion.

Thank you for the detailed recommendations!
I will do the step 1. Expose persistent value in sysfs now. And work on other changes in the future.

Thanks,
- Haiyang