Skip to contentSkip to conversation

IB/ipoib: Use dev_port to disambiguate port numbers

PATCH In review Arseny Maslennikov 3 participants 11 messages Sep 6, 2018 – Sep 13, 2018

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v4 cover IB/ipoib: Use dev_port to disambiguate port numbers

Pre-3.15 userspace had trouble distinguishing different ports of a NIC on a single PCI bus/device/function. To solve this, a sysfs field `dev_port' was introduced quite a while ago (commit v3.14-rc3-739-g3f85944fe207), and some relevant device drivers were fixed to use it, but not in case of IPoIB.

The convention for some reason never got documented in the kernel, but was immediately adopted by userspace (notably udev[1][2], biosdevname[3])

1/3 documents the sysfs field — that's why I'm CC-ing netdev.

This series was tested on and applies to 4.19-rc2.

[1] https://lists.freedesktop.org/archives/systemd-devel/2014-June/020788.html
[2] https://lists.freedesktop.org/archives/systemd-devel/2014-July/020804.html
[3] https://github.com/CloudAutomationNTools/biosdevname/blob/c795d51dd93a5309652f0d635f12a3ecfabfaa72/src/eths.c#L38

v1->v2: replace a line instead of inserting and then removing. v2->v3: restore both attributes, output a notice of deprecation to kmsg. v3->v4: style adjustments, join the deprecation notice to single line.

Arseny Maslennikov (3):
Documentation/ABI: document /sys/class/net/*/dev_port IB/ipoib: Use dev_port to expose network interface port numbers IB/ipoib: Log sysfs 'dev_id' accesses from userspace

 Documentation/ABI/testing/sysfs-class-net | 18 +++++++++++++
 drivers/infiniband/ulp/ipoib/ipoib_main.c | 33 +++++++++++++++++++++++
 2 files changed, 51 insertions(+)
Signature · 1 line
2.19.0.rc2
#1 lore raw

PATCH v4 1/3 Documentation/ABI: document /sys/class/net/*/dev_port

The sysfs field was introduced 4 years ago along with fixes to various drivers that erroneously used `dev_id' for that purpose, but it was not properly documented anywhere.
See commit v3.14-rc3-739-g3f85944fe207.

---
 Documentation/ABI/testing/sysfs-class-net | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
Documentation/ABI/testing/sysfs-class-net +18 -0
index 2f1788111cd9..ec2232f6a949 100644
@@ -91,6 +91,24 @@ Description:
 		stacked (e.g: VLAN interfaces) but still have the same MAC
 		address as their parent device.
 
+What:		/sys/class/net/<iface>/dev_port
+Date:		February 2014
+KernelVersion:	3.15
+Contact:	[email protected]
+Description:
+		Indicates the port number of this network device, formatted
+		as a decimal value. Some NICs have multiple independent ports
+		on the same PCI bus, device and function. This attribute allows
+		userspace to distinguish the respective interfaces.
+
+		Note: some device drivers started to use 'dev_id' for this
+		purpose since long before 3.15 and have not adopted the new
+		attribute ever since. To query the port number, some tools look
+		exclusively at 'dev_port', while others only consult 'dev_id'.
+		If a network device has multiple client adapter ports as
+		described in the previous paragraph and does not set this
+		attribute to its port number, it's a kernel bug.
+
 What:		/sys/class/net/<iface>/dormant
 Date:		March 2006
 KernelVersion:	2.6.17
Signature · 1 line
2.19.0.rc2
#2 lore raw

PATCH v4 2/3 IB/ipoib: Use dev_port to expose network interface port numbers

In reply to Arseny Maslennikov

Some InfiniBand network devices have multiple ports on the same PCI function. This initializes the `dev_port' sysfs field of those network interfaces with their port number.

Prior to this the kernel erroneously used the `dev_id' sysfs field of those network interfaces to convey the port number to userspace.

The use of dev_id' was considered correct until Linux 3.15, when another field, dev_port', was defined for this particular purpose and `dev_id' was reserved for distinguishing stacked ifaces (e.g: VLANs) with the same hardware address as their parent device.

Similar fixes to net/mlx4_en and many other drivers, which started exporting this information through `dev_id' before 3.15, were accepted into the kernel 4 years ago.
See 76a066f2a2a0 (`net/mlx4_en: Expose port number through sysfs').

---
 drivers/infiniband/ulp/ipoib/ipoib_main.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/infiniband/ulp/ipoib/ipoib_main.c +2 -0
index e3d28f9ad9c0..30f840f874b3 100644
@@ -1880,6 +1880,8 @@ static int ipoib_parent_init(struct net_device *ndev)
 	       sizeof(union ib_gid));
 
 	SET_NETDEV_DEV(priv->dev, priv->ca->dev.parent);
+	priv->dev->dev_port = priv->port - 1;
+	/* Let's set this one too for backwards compatibility. */
 	priv->dev->dev_id = priv->port - 1;
 
 	return 0;
Signature · 1 line
2.19.0.rc2
#3 lore raw

PATCH v4 3/3 IB/ipoib: Log sysfs 'dev_id' accesses from userspace

In reply to Arseny Maslennikov

Some tools may currently be using only the deprecated attribute; let's print an elaborate and clear deprecation notice to kmsg.

To do that, we have to replace the whole sysfs file, since we inherit the original one from netdev.

---
 drivers/infiniband/ulp/ipoib/ipoib_main.c | 31 +++++++++++++++++++++++
 1 file changed, 31 insertions(+)
drivers/infiniband/ulp/ipoib/ipoib_main.c +31 -0
index 30f840f874b3..74732726ec6f 100644
@@ -2386,6 +2386,35 @@ int ipoib_add_pkey_attr(struct net_device *dev)
 	return device_create_file(&dev->dev, &dev_attr_pkey);
 }
 
+/*
+ * We erroneously exposed the iface's port number in the dev_id
+ * sysfs field long after dev_port was introduced for that purpose[1],
+ * and we need to stop everyone from relying on that.
+ * Let's overload the shower routine for the dev_id file here
+ * to gently bring the issue up.
+ *
+ * [1] https://www.spinics.net/lists/netdev/msg272123.html
+ */
+static ssize_t dev_id_show(struct device *dev,
+			   struct device_attribute *attr, char *buf)
+{
+	struct net_device *ndev = to_net_dev(dev);
+
+	if (ndev->dev_id == ndev->dev_port)
+		netdev_info_once(ndev,
+			"\"%s\" wants to know my dev_id. Should it look at dev_port instead? See Documentation/ABI/testing/sysfs-class-net for more info.\n",
+			current->comm);
+
+	return sprintf(buf, "%#x\n", ndev->dev_id);
+}
+static DEVICE_ATTR_RO(dev_id);
+
+int ipoib_intercept_dev_id_attr(struct net_device *dev)
+{
+	device_remove_file(&dev->dev, &dev_attr_dev_id);
+	return device_create_file(&dev->dev, &dev_attr_dev_id);
+}
+
 static struct net_device *ipoib_add_port(const char *format,
 					 struct ib_device *hca, u8 port)
 {
@@ -2427,6 +2456,8 @@ static struct net_device *ipoib_add_port(const char *format,
 	 */
 	ndev->priv_destructor = ipoib_intf_free;
 
+	if (ipoib_intercept_dev_id_attr(ndev))
+		goto sysfs_failed;
 	if (ipoib_cm_add_mode_attr(ndev))
 		goto sysfs_failed;
 	if (ipoib_add_pkey_attr(ndev))
Signature · 1 line
2.19.0.rc2
#4 lore raw

Re: [PATCH v4 3/3] IB/ipoib: Log sysfs 'dev_id' accesses from userspace

Quoted 35 lines · Arseny Maslennikov

Some tools may currently be using only the deprecated attribute; let's print an elaborate and clear deprecation notice to kmsg.

To do that, we have to replace the whole sysfs file, since we inherit the original one from netdev.

 drivers/infiniband/ulp/ipoib/ipoib_main.c | 31 +++++++++++++++++++++++
 1 file changed, 31 insertions(+)
drivers/infiniband/ulp/ipoib/ipoib_main.c +19 -0
index 30f840f874b3..74732726ec6f 100644
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -2386,6 +2386,35 @@ int ipoib_add_pkey_attr(struct net_device *dev)
 	return device_create_file(&dev->dev, &dev_attr_pkey);
 }
 
+/*
+ * We erroneously exposed the iface's port number in the dev_id
+ * sysfs field long after dev_port was introduced for that purpose[1],
+ * and we need to stop everyone from relying on that.
+ * Let's overload the shower routine for the dev_id file here
+ * to gently bring the issue up.
+ *
+ * [1] https://www.spinics.net/lists/netdev/msg272123.html
+ */
+static ssize_t dev_id_show(struct device *dev,
+			   struct device_attribute *attr, char *buf)
+{
+	struct net_device *ndev = to_net_dev(dev);
+
+	if (ndev->dev_id == ndev->dev_port)
+		netdev_info_once(ndev,
+			"\"%s\" wants to know my dev_id. Should it look at dev_port instead? See Documentation/ABI/testing/sysfs-class-net for more info.\n",
+			current->comm);

drivers/infiniband/ulp/ipoib/ipoib_main.c int ipoib_add_pkey_attr(struct net_device *dev) · line 2416

Quoted 10 lines
drivers/infiniband/ulp/ipoib/ipoib_main.c +10 -0
+
+	return sprintf(buf, "%#x\n", ndev->dev_id);
+}
+static DEVICE_ATTR_RO(dev_id);
+
+int ipoib_intercept_dev_id_attr(struct net_device *dev)
+{
+	device_remove_file(&dev->dev, &dev_attr_dev_id);
+	return device_create_file(&dev->dev, &dev_attr_dev_id);
+}

Isn't this racey with userspace? Ie what happens if udev is querying the dev_id right here?

Do we know there is no userspace doing this?

drivers/infiniband/ulp/ipoib/ipoib_main.c static struct net_device *ipoib_add_port(const char *format, · line 2460

Quoted 9 lines
drivers/infiniband/ulp/ipoib/ipoib_main.c +2 -0
 static struct net_device *ipoib_add_port(const char *format,
 					 struct ib_device *hca, u8 port)
 {
@@ -2427,6 +2456,8 @@ static struct net_device *ipoib_add_port(const char *format,
 	 */
 	ndev->priv_destructor = ipoib_intf_free;
 
+	if (ipoib_intercept_dev_id_attr(ndev))
+		goto sysfs_failed;

No device_remove_file needed?

Jason

#5 lore raw

Re: [PATCH v4 3/3] IB/ipoib: Log sysfs 'dev_id' accesses from userspace

Quoted 50 lines · Jason Gunthorpe
Quoted 35 lines · Arseny Maslennikov

Some tools may currently be using only the deprecated attribute; let's print an elaborate and clear deprecation notice to kmsg.

To do that, we have to replace the whole sysfs file, since we inherit the original one from netdev.

 drivers/infiniband/ulp/ipoib/ipoib_main.c | 31 +++++++++++++++++++++++
 1 file changed, 31 insertions(+)
drivers/infiniband/ulp/ipoib/ipoib_main.c +19 -0
index 30f840f874b3..74732726ec6f 100644
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -2386,6 +2386,35 @@ int ipoib_add_pkey_attr(struct net_device *dev)
 	return device_create_file(&dev->dev, &dev_attr_pkey);
 }
 
+/*
+ * We erroneously exposed the iface's port number in the dev_id
+ * sysfs field long after dev_port was introduced for that purpose[1],
+ * and we need to stop everyone from relying on that.
+ * Let's overload the shower routine for the dev_id file here
+ * to gently bring the issue up.
+ *
+ * [1] https://www.spinics.net/lists/netdev/msg272123.html
+ */
+static ssize_t dev_id_show(struct device *dev,
+			   struct device_attribute *attr, char *buf)
+{
+	struct net_device *ndev = to_net_dev(dev);
+
+	if (ndev->dev_id == ndev->dev_port)
+		netdev_info_once(ndev,
+			"\"%s\" wants to know my dev_id. Should it look at dev_port instead? See Documentation/ABI/testing/sysfs-class-net for more info.\n",
+			current->comm);

drivers/infiniband/ulp/ipoib/ipoib_main.c int ipoib_add_pkey_attr(struct net_device *dev) · line 2416

Quoted 10 lines
drivers/infiniband/ulp/ipoib/ipoib_main.c +10 -0
+
+	return sprintf(buf, "%#x\n", ndev->dev_id);
+}
+static DEVICE_ATTR_RO(dev_id);
+
+int ipoib_intercept_dev_id_attr(struct net_device *dev)
+{
+	device_remove_file(&dev->dev, &dev_attr_dev_id);
+	return device_create_file(&dev->dev, &dev_attr_dev_id);
+}

Isn't this racey with userspace? Ie what happens if udev is querying the dev_id right here?

Quoted 1 line

Do we know there is no userspace doing this?

I don't think that it can race (or reasonably can). The intercept function is done as part of ipoib_add_port() so the port itself isn't live yet. Things like udev shouldn't be scanning it until after we've finished bringing it up and added it to the system, so any race here is unimportant IMO.

Quoted 13 lines

drivers/infiniband/ulp/ipoib/ipoib_main.c static struct net_device *ipoib_add_port(const char *format, · line 2460

Quoted 9 lines
drivers/infiniband/ulp/ipoib/ipoib_main.c +2 -0
 static struct net_device *ipoib_add_port(const char *format,
 					 struct ib_device *hca, u8 port)
 {
@@ -2427,6 +2456,8 @@ static struct net_device *ipoib_add_port(const char *format,
 	 */
 	ndev->priv_destructor = ipoib_intf_free;
 
+	if (ipoib_intercept_dev_id_attr(ndev))
+		goto sysfs_failed;

No device_remove_file needed?

Jason

Signature · 3 lines
Doug Ledford <[email protected]>
    GPG KeyID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD
#6 lore raw

Re: [PATCH v4 3/3] IB/ipoib: Log sysfs 'dev_id' accesses from userspace

Quoted 51 lines · Doug Ledford
Quoted 50 lines · Jason Gunthorpe
Quoted 35 lines · Arseny Maslennikov

Some tools may currently be using only the deprecated attribute; let's print an elaborate and clear deprecation notice to kmsg.

To do that, we have to replace the whole sysfs file, since we inherit the original one from netdev.

 drivers/infiniband/ulp/ipoib/ipoib_main.c | 31 +++++++++++++++++++++++
 1 file changed, 31 insertions(+)
drivers/infiniband/ulp/ipoib/ipoib_main.c +19 -0
index 30f840f874b3..74732726ec6f 100644
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -2386,6 +2386,35 @@ int ipoib_add_pkey_attr(struct net_device *dev)
 	return device_create_file(&dev->dev, &dev_attr_pkey);
 }
 
+/*
+ * We erroneously exposed the iface's port number in the dev_id
+ * sysfs field long after dev_port was introduced for that purpose[1],
+ * and we need to stop everyone from relying on that.
+ * Let's overload the shower routine for the dev_id file here
+ * to gently bring the issue up.
+ *
+ * [1] https://www.spinics.net/lists/netdev/msg272123.html
+ */
+static ssize_t dev_id_show(struct device *dev,
+			   struct device_attribute *attr, char *buf)
+{
+	struct net_device *ndev = to_net_dev(dev);
+
+	if (ndev->dev_id == ndev->dev_port)
+		netdev_info_once(ndev,
+			"\"%s\" wants to know my dev_id. Should it look at dev_port instead? See Documentation/ABI/testing/sysfs-class-net for more info.\n",
+			current->comm);

drivers/infiniband/ulp/ipoib/ipoib_main.c int ipoib_add_pkey_attr(struct net_device *dev) · line 2416

Quoted 10 lines
drivers/infiniband/ulp/ipoib/ipoib_main.c +10 -0
+
+	return sprintf(buf, "%#x\n", ndev->dev_id);
+}
+static DEVICE_ATTR_RO(dev_id);
+
+int ipoib_intercept_dev_id_attr(struct net_device *dev)
+{
+	device_remove_file(&dev->dev, &dev_attr_dev_id);
+	return device_create_file(&dev->dev, &dev_attr_dev_id);
+}

Isn't this racey with userspace? Ie what happens if udev is querying the dev_id right here?

Quoted 5 lines
Quoted 1 line

Do we know there is no userspace doing this?

I don't think that it can race (or reasonably can). The intercept function is done as part of ipoib_add_port() so the port itself isn't live yet.

The above code is after register_netdev, so the port itself is certainly live as far as userspace is concerned..

All the other sysfs stuff in add_port is already wrong/racy.. See Parav's recent series fixing this for the main devices..

Jason

#7 lore raw

Re: [PATCH v4 3/3] IB/ipoib: Log sysfs 'dev_id' accesses from userspace

In reply to Jason Gunthorpe
Quoted 41 lines · Jason Gunthorpe
Quoted 40 lines · Arseny Maslennikov

Some tools may currently be using only the deprecated attribute; let's print an elaborate and clear deprecation notice to kmsg.

To do that, we have to replace the whole sysfs file, since we inherit the original one from netdev.

 drivers/infiniband/ulp/ipoib/ipoib_main.c | 31 +++++++++++++++++++++++
 1 file changed, 31 insertions(+)
drivers/infiniband/ulp/ipoib/ipoib_main.c +24 -0
index 30f840f874b3..74732726ec6f 100644
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -2386,6 +2386,35 @@ int ipoib_add_pkey_attr(struct net_device *dev)
 	return device_create_file(&dev->dev, &dev_attr_pkey);
 }
 
+/*
+ * We erroneously exposed the iface's port number in the dev_id
+ * sysfs field long after dev_port was introduced for that purpose[1],
+ * and we need to stop everyone from relying on that.
+ * Let's overload the shower routine for the dev_id file here
+ * to gently bring the issue up.
+ *
+ * [1] https://www.spinics.net/lists/netdev/msg272123.html
+ */
+static ssize_t dev_id_show(struct device *dev,
+			   struct device_attribute *attr, char *buf)
+{
+	struct net_device *ndev = to_net_dev(dev);
+
+	if (ndev->dev_id == ndev->dev_port)
+		netdev_info_once(ndev,
+			"\"%s\" wants to know my dev_id. Should it look at dev_port instead? See Documentation/ABI/testing/sysfs-class-net for more info.\n",
+			current->comm);
+
+	return sprintf(buf, "%#x\n", ndev->dev_id);
+}
+static DEVICE_ATTR_RO(dev_id);
+
Quoted 8 lines

drivers/infiniband/ulp/ipoib/ipoib_main.c int ipoib_add_pkey_attr(struct net_device *dev) · line 2416

Quoted 5 lines
drivers/infiniband/ulp/ipoib/ipoib_main.c +5 -0
+int ipoib_intercept_dev_id_attr(struct net_device *dev)
+{
+	device_remove_file(&dev->dev, &dev_attr_dev_id);
+	return device_create_file(&dev->dev, &dev_attr_dev_id);
+}

Isn't this racey with userspace? Ie what happens if udev is querying the dev_id right here?

udev in particular does not use dev_id at all since 2014, because "why would we keep using dev_id if it is not the right thing to use?".

Quoted 3 lines

Do we know there is no userspace doing this?

Not for sure.

If we move all the sysfs handling stuff we introduce in _add_port():
- pkey
- umcast
- {create,delete}_child
- connected/datagram mode
to _ndo_init(), which is called by register_netdev before it sends the netlink message, would that suffice to eliminate the race? (Sysfs files for {create,delete}_child go to _parent_init() then).

Quoted 7 lines
Quoted 7 lines
drivers/infiniband/ulp/ipoib/ipoib_main.c +0 -0
 static struct net_device *ipoib_add_port(const char *format,
 					 struct ib_device *hca, u8 port)
 {
@@ -2427,6 +2456,8 @@ static struct net_device *ipoib_add_port(const char *format,
 	 */
 	ndev->priv_destructor = ipoib_intf_free;
 
Quoted 5 lines

drivers/infiniband/ulp/ipoib/ipoib_main.c static struct net_device *ipoib_add_port(const char *format, · line 2460

Quoted 2 lines
drivers/infiniband/ulp/ipoib/ipoib_main.c +2 -0
+	if (ipoib_intercept_dev_id_attr(ndev))
+		goto sysfs_failed;

No device_remove_file needed?

As far as I understand, unregister_netdevice takes care of it while destroying the related kobject.

Quoted 1 line

Jason

#8 lore raw

Re: [PATCH v4 3/3] IB/ipoib: Log sysfs 'dev_id' accesses from userspace

Quoted 60 lines · Arseny Maslennikov
Quoted 41 lines · Jason Gunthorpe
Quoted 40 lines · Arseny Maslennikov

Some tools may currently be using only the deprecated attribute; let's print an elaborate and clear deprecation notice to kmsg.

To do that, we have to replace the whole sysfs file, since we inherit the original one from netdev.

 drivers/infiniband/ulp/ipoib/ipoib_main.c | 31 +++++++++++++++++++++++
 1 file changed, 31 insertions(+)
drivers/infiniband/ulp/ipoib/ipoib_main.c +24 -0
index 30f840f874b3..74732726ec6f 100644
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -2386,6 +2386,35 @@ int ipoib_add_pkey_attr(struct net_device *dev)
 	return device_create_file(&dev->dev, &dev_attr_pkey);
 }
 
+/*
+ * We erroneously exposed the iface's port number in the dev_id
+ * sysfs field long after dev_port was introduced for that purpose[1],
+ * and we need to stop everyone from relying on that.
+ * Let's overload the shower routine for the dev_id file here
+ * to gently bring the issue up.
+ *
+ * [1] https://www.spinics.net/lists/netdev/msg272123.html
+ */
+static ssize_t dev_id_show(struct device *dev,
+			   struct device_attribute *attr, char *buf)
+{
+	struct net_device *ndev = to_net_dev(dev);
+
+	if (ndev->dev_id == ndev->dev_port)
+		netdev_info_once(ndev,
+			"\"%s\" wants to know my dev_id. Should it look at dev_port instead? See Documentation/ABI/testing/sysfs-class-net for more info.\n",
+			current->comm);
+
+	return sprintf(buf, "%#x\n", ndev->dev_id);
+}
+static DEVICE_ATTR_RO(dev_id);
+
Quoted 8 lines

drivers/infiniband/ulp/ipoib/ipoib_main.c int ipoib_add_pkey_attr(struct net_device *dev) · line 2416

Quoted 5 lines
drivers/infiniband/ulp/ipoib/ipoib_main.c +5 -0
+int ipoib_intercept_dev_id_attr(struct net_device *dev)
+{
+	device_remove_file(&dev->dev, &dev_attr_dev_id);
+	return device_create_file(&dev->dev, &dev_attr_dev_id);
+}

Isn't this racey with userspace? Ie what happens if udev is querying the dev_id right here?

udev in particular does not use dev_id at all since 2014, because "why would we keep using dev_id if it is not the right thing to use?".

Quoted 3 lines

Do we know there is no userspace doing this?

Not for sure.

Quoted 9 lines

If we move all the sysfs handling stuff we introduce in _add_port():
- pkey
- umcast
- {create,delete}_child
- connected/datagram mode
to _ndo_init(), which is called by register_netdev before it sends the netlink message, would that suffice to eliminate the race? (Sysfs files for {create,delete}_child go to _parent_init() then).

No, we can't, sorry for the noise. ndo_init() runs before the kobject becomes available.

Anyway, our sysfs attributes being racy is unrelated to the patch series subject, and I can't come up with any other ideas what to do with them that do not involve adjustments to register_netdev.

Quoted 17 lines
Quoted 7 lines
Quoted 7 lines
drivers/infiniband/ulp/ipoib/ipoib_main.c +0 -0
 static struct net_device *ipoib_add_port(const char *format,
 					 struct ib_device *hca, u8 port)
 {
@@ -2427,6 +2456,8 @@ static struct net_device *ipoib_add_port(const char *format,
 	 */
 	ndev->priv_destructor = ipoib_intf_free;
 
Quoted 5 lines

drivers/infiniband/ulp/ipoib/ipoib_main.c static struct net_device *ipoib_add_port(const char *format, · line 2460

Quoted 2 lines
drivers/infiniband/ulp/ipoib/ipoib_main.c +2 -0
+	if (ipoib_intercept_dev_id_attr(ndev))
+		goto sysfs_failed;

No device_remove_file needed?

As far as I understand, unregister_netdevice takes care of it while destroying the related kobject.

Quoted 1 line

Jason

#9 lore raw

Re: [PATCH v4 3/3] IB/ipoib: Log sysfs 'dev_id' accesses from userspace

Quoted 71 lines · Arseny Maslennikov
Quoted 60 lines · Arseny Maslennikov
Quoted 41 lines · Jason Gunthorpe
Quoted 40 lines · Arseny Maslennikov

Some tools may currently be using only the deprecated attribute; let's print an elaborate and clear deprecation notice to kmsg.

To do that, we have to replace the whole sysfs file, since we inherit the original one from netdev.

 drivers/infiniband/ulp/ipoib/ipoib_main.c | 31 +++++++++++++++++++++++
 1 file changed, 31 insertions(+)
drivers/infiniband/ulp/ipoib/ipoib_main.c +24 -0
index 30f840f874b3..74732726ec6f 100644
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -2386,6 +2386,35 @@ int ipoib_add_pkey_attr(struct net_device *dev)
 	return device_create_file(&dev->dev, &dev_attr_pkey);
 }
 
+/*
+ * We erroneously exposed the iface's port number in the dev_id
+ * sysfs field long after dev_port was introduced for that purpose[1],
+ * and we need to stop everyone from relying on that.
+ * Let's overload the shower routine for the dev_id file here
+ * to gently bring the issue up.
+ *
+ * [1] https://www.spinics.net/lists/netdev/msg272123.html
+ */
+static ssize_t dev_id_show(struct device *dev,
+			   struct device_attribute *attr, char *buf)
+{
+	struct net_device *ndev = to_net_dev(dev);
+
+	if (ndev->dev_id == ndev->dev_port)
+		netdev_info_once(ndev,
+			"\"%s\" wants to know my dev_id. Should it look at dev_port instead? See Documentation/ABI/testing/sysfs-class-net for more info.\n",
+			current->comm);
+
+	return sprintf(buf, "%#x\n", ndev->dev_id);
+}
+static DEVICE_ATTR_RO(dev_id);
+
Quoted 8 lines

drivers/infiniband/ulp/ipoib/ipoib_main.c int ipoib_add_pkey_attr(struct net_device *dev) · line 2416

Quoted 5 lines
drivers/infiniband/ulp/ipoib/ipoib_main.c +5 -0
+int ipoib_intercept_dev_id_attr(struct net_device *dev)
+{
+	device_remove_file(&dev->dev, &dev_attr_dev_id);
+	return device_create_file(&dev->dev, &dev_attr_dev_id);
+}

Isn't this racey with userspace? Ie what happens if udev is querying the dev_id right here?

udev in particular does not use dev_id at all since 2014, because "why would we keep using dev_id if it is not the right thing to use?".

Quoted 3 lines

Do we know there is no userspace doing this?

Not for sure.

Quoted 9 lines

If we move all the sysfs handling stuff we introduce in _add_port():
- pkey
- umcast
- {create,delete}_child
- connected/datagram mode
to _ndo_init(), which is called by register_netdev before it sends the netlink message, would that suffice to eliminate the race? (Sysfs files for {create,delete}_child go to _parent_init() then).

Quoted 6 lines

No, we can't, sorry for the noise. ndo_init() runs before the kobject becomes available.

Anyway, our sysfs attributes being racy is unrelated to the patch series subject, and I can't come up with any other ideas what to do with them that do not involve adjustments to register_netdev.

Agreed (that fixing the race issues is a different patch series).

Signature · 3 lines
Doug Ledford <[email protected]>
    GPG KeyID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD
#10 lore raw
In reply to Arseny Maslennikov
Quoted 21 lines · Arseny Maslennikov

Pre-3.15 userspace had trouble distinguishing different ports of a NIC on a single PCI bus/device/function. To solve this, a sysfs field `dev_port' was introduced quite a while ago (commit v3.14-rc3-739-g3f85944fe207), and some relevant device drivers were fixed to use it, but not in case of IPoIB.

The convention for some reason never got documented in the kernel, but was immediately adopted by userspace (notably udev[1][2], biosdevname[3])

1/3 documents the sysfs field — that's why I'm CC-ing netdev.

This series was tested on and applies to 4.19-rc2.

[1] https://lists.freedesktop.org/archives/systemd-devel/2014-June/020788.html
[2] https://lists.freedesktop.org/archives/systemd-devel/2014-July/020804.html
[3] https://github.com/CloudAutomationNTools/biosdevname/blob/c795d51dd93a5309652f0d635f12a3ecfabfaa72/src/eths.c#L38

v1->v2: replace a line instead of inserting and then removing. v2->v3: restore both attributes, output a notice of deprecation to kmsg. v3->v4: style adjustments, join the deprecation notice to single line.

Quoted 9 lines

Arseny Maslennikov (3):
Documentation/ABI: document /sys/class/net/*/dev_port IB/ipoib: Use dev_port to expose network interface port numbers IB/ipoib: Log sysfs 'dev_id' accesses from userspace

 Documentation/ABI/testing/sysfs-class-net | 18 +++++++++++++
 drivers/infiniband/ulp/ipoib/ipoib_main.c | 33 +++++++++++++++++++++++
 2 files changed, 51 insertions(+)

Series applied to for-next. But I think we should watch feedback from people, and if people think the notification about using the wrong variable is too noisy, then we might want to revert it or modify it to only print out once per specific executable instead of once per run of each executable.

Signature · 3 lines
Doug Ledford <[email protected]>
    GPG KeyID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD