Skip to contentSkip to conversation

net: dqs: introduce NETIF_F_NO_BQL device feature

PATCH In review Jason Xing 4 participants 7 messages Jun 9, 2024 – Jun 11, 2024

Subsystems (4)
lore
Share & tools
lore

Jason Xing

author
#0 lore raw

PATCH 1/1 net: dqs: introduce NETIF_F_NO_BQL device feature

From: Jason Xing <[email protected]>

Since commit 74293ea1c4db6 ("net: sysfs: Do not create sysfs for non BQL device") limits the non-BQL driver not creating byte_queue_limits directory, I found there is one exception, namely, virtio-net driver, which should also be limited in netdev_uses_bql().

I decided to introduce a NO_BQL bit in device feature because
1) it can help us limit virtio-net driver for now.
2) if we found another non-BQL driver, we can take it into account.
3) we can replace all the driver meeting those two statements in netdev_uses_bql() in future.

For now, I would like to make the first step to use this new bit for dqs use instead of replacing/applying all the non-BQL drivers.

After this patch, 1) there is no byte_queue_limits directory in virtio-net driver. 2) running ethtool -k eth1 shows "no-bql: on [fixed]".

---
 drivers/net/virtio_net.c        | 2 +-
 include/linux/netdev_features.h | 3 ++-
 net/core/net-sysfs.c            | 2 +-
 net/ethtool/common.c            | 1 +
 4 files changed, 5 insertions(+), 3 deletions(-)
drivers/net/virtio_net.c +1 -1
index 61a57d134544..619908fed14b 100644
@@ -5634,7 +5634,7 @@ static int virtnet_probe(struct virtio_device *vdev)
 			   IFF_TX_SKB_NO_LINEAR;
 	dev->netdev_ops = &virtnet_netdev;
 	dev->stat_ops = &virtnet_stat_ops;
-	dev->features = NETIF_F_HIGHDMA;
+	dev->features = NETIF_F_HIGHDMA | NETIF_F_NO_BQL;
 
 	dev->ethtool_ops = &virtnet_ethtool_ops;
 	SET_NETDEV_DEV(dev, &vdev->dev);
include/linux/netdev_features.h +2 -1
index 7c2d77d75a88..9bc603bb4227 100644
@@ -14,7 +14,6 @@ typedef u64 netdev_features_t;
 enum {
 	NETIF_F_SG_BIT,			/* Scatter/gather IO. */
 	NETIF_F_IP_CSUM_BIT,		/* Can checksum TCP/UDP over IPv4. */
-	__UNUSED_NETIF_F_1,
 	NETIF_F_HW_CSUM_BIT,		/* Can checksum all the packets. */
 	NETIF_F_IPV6_CSUM_BIT,		/* Can checksum TCP/UDP over IPV6 */
 	NETIF_F_HIGHDMA_BIT,		/* Can DMA to high memory. */
@@ -91,6 +90,7 @@ enum {
 	NETIF_F_HW_HSR_FWD_BIT,		/* Offload HSR forwarding */
 	NETIF_F_HW_HSR_DUP_BIT,		/* Offload HSR duplication */
 
+	NETIF_F_NO_BQL_BIT,		/* non-BQL driver */
 	/*
 	 * Add your fresh new feature above and remember to update
 	 * netdev_features_strings[] in net/ethtool/common.c and maybe
@@ -168,6 +168,7 @@ enum {
 #define NETIF_F_HW_HSR_TAG_RM	__NETIF_F(HW_HSR_TAG_RM)
 #define NETIF_F_HW_HSR_FWD	__NETIF_F(HW_HSR_FWD)
 #define NETIF_F_HW_HSR_DUP	__NETIF_F(HW_HSR_DUP)
+#define NETIF_F_NO_BQL		__NETIF_F(NO_BQL)
 
 /* Finds the next feature with the highest number of the range of start-1 till 0.
  */
net/core/net-sysfs.c +1 -1
index 4c27a360c294..ff397a76f1fe 100644
@@ -1764,7 +1764,7 @@ static const struct kobj_type netdev_queue_ktype = {
 
 static bool netdev_uses_bql(const struct net_device *dev)
 {
-	if (dev->features & NETIF_F_LLTX ||
+	if (dev->features & (NETIF_F_LLTX | NETIF_F_NO_BQL) ||
 	    dev->priv_flags & IFF_NO_QUEUE)
 		return false;
 
net/ethtool/common.c +1 -0
index 6b2a360dcdf0..efa7ac4158ce 100644
@@ -74,6 +74,7 @@ const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] = {
 	[NETIF_F_HW_HSR_TAG_RM_BIT] =	 "hsr-tag-rm-offload",
 	[NETIF_F_HW_HSR_FWD_BIT] =	 "hsr-fwd-offload",
 	[NETIF_F_HW_HSR_DUP_BIT] =	 "hsr-dup-offload",
+	[NETIF_F_NO_BQL_BIT] =		 "no-bql",
 };
 
 const char
Signature · 1 line
2.37.3
#1 lore raw
Quoted 14 lines · Jason Xing

From: Jason Xing <[email protected]>

Since commit 74293ea1c4db6 ("net: sysfs: Do not create sysfs for non BQL device") limits the non-BQL driver not creating byte_queue_limits directory, I found there is one exception, namely, virtio-net driver, which should also be limited in netdev_uses_bql().

I decided to introduce a NO_BQL bit in device feature because
1) it can help us limit virtio-net driver for now.
2) if we found another non-BQL driver, we can take it into account.
3) we can replace all the driver meeting those two statements in netdev_uses_bql() in future.

Quoted 7 lines

For now, I would like to make the first step to use this new bit for dqs use instead of replacing/applying all the non-BQL drivers.

After this patch, 1) there is no byte_queue_limits directory in virtio-net driver. 2) running ethtool -k eth1 shows "no-bql: on [fixed]".

I do not think we want to consume a precious bit from dev->features for something like that.

dev->features should be reserved for bits we used in the fast path, for instance netif_skb_features(). It is good to have free bits for future fast path use.

(I think Vladimir was trying to make some room, this was a discussion we had last year)

I do not see the reason to report to ethtool the 'nobql bit' :
If a driver opts-out, then the bql sysfs files will not be there, user space can see the absence of the files.

author
#2 lore raw
Quoted 24 lines · Eric Dumazet
Quoted 14 lines · Jason Xing

From: Jason Xing <[email protected]>

Since commit 74293ea1c4db6 ("net: sysfs: Do not create sysfs for non BQL device") limits the non-BQL driver not creating byte_queue_limits directory, I found there is one exception, namely, virtio-net driver, which should also be limited in netdev_uses_bql().

I decided to introduce a NO_BQL bit in device feature because
1) it can help us limit virtio-net driver for now.
2) if we found another non-BQL driver, we can take it into account.
3) we can replace all the driver meeting those two statements in netdev_uses_bql() in future.

Quoted 7 lines

For now, I would like to make the first step to use this new bit for dqs use instead of replacing/applying all the non-BQL drivers.

After this patch, 1) there is no byte_queue_limits directory in virtio-net driver. 2) running ethtool -k eth1 shows "no-bql: on [fixed]".

Quoted 8 lines

I do not think we want to consume a precious bit from dev->features for something like that.

dev->features should be reserved for bits we used in the fast path, for instance netif_skb_features(). It is good to have free bits for future fast path use.

(I think Vladimir was trying to make some room, this was a discussion we had last year)

Thanks for your reminder. When I was trying to introduce one new bit, I noticed an overflow warning when compiling.

Quoted 4 lines

I do not see the reason to report to ethtool the 'nobql bit' :
If a driver opts-out, then the bql sysfs files will not be there, user space can see the absence of the files.

The reason is that I just followed the comment to force myself to report to ethtool. Now I see.

It seems not that easy to consider all the non-BQL drivers. Let me think more about it.

Thanks,
Jason

maintainer
#3 lore raw
Quoted 2 lines · On Mon, 10 Jun 2024 07:55:55 +0800 Jason Xing wrote:
Quoted 2 lines

(I think Vladimir was trying to make some room, this was a discussion we had last year)

s/Vladimir/Olek/ ?

Quoted 7 lines

Thanks for your reminder. When I was trying to introduce one new bit, I noticed an overflow warning when compiling.

Quoted 3 lines

I do not see the reason to report to ethtool the 'nobql bit' :
If a driver opts-out, then the bql sysfs files will not be there, user space can see the absence of the files.

Quoted 5 lines

The reason is that I just followed the comment to force myself to report to ethtool. Now I see.

It seems not that easy to consider all the non-BQL drivers. Let me think more about it.

All Eric was saying, AFAIU, is that you can for example add a bit in somewhere towards the end of struct nedevice, no need to pack this info into feature bits.

BTW the Fixes tag is a bit of an exaggeration here. The heuristic in netdev_uses_bql() is best effort, its fine to miss some devices.

author
#4 lore raw
Quoted 17 lines · Jakub Kicinski
Quoted 2 lines · On Mon, 10 Jun 2024 07:55:55 +0800 Jason Xing wrote:
Quoted 2 lines

(I think Vladimir was trying to make some room, this was a discussion we had last year)

s/Vladimir/Olek/ ?

Quoted 10 lines

Thanks for your reminder. When I was trying to introduce one new bit, I noticed an overflow warning when compiling.

Quoted 3 lines

I do not see the reason to report to ethtool the 'nobql bit' :
If a driver opts-out, then the bql sysfs files will not be there, user space can see the absence of the files.

The reason is that I just followed the comment to force myself to report to ethtool. Now I see.

Quoted 6 lines
Quoted 2 lines

It seems not that easy to consider all the non-BQL drivers. Let me think more about it.

All Eric was saying, AFAIU, is that you can for example add a bit in somewhere towards the end of struct nedevice, no need to pack this info into feature bits.

Oh, thanks for pointing this out.

I would like to add a new bit field in the enum netdev_priv_flags because it is better that it's grouped into an existing enum for future compatibility.

Quoted 3 lines

BTW the Fixes tag is a bit of an exaggeration here. The heuristic in netdev_uses_bql() is best effort, its fine to miss some devices.

I see.

Thanks,
Jason

#5 lore raw
In reply to Jason Xing
Quoted 13 lines · Sun, Jun 09, 2024 at 03:17:32PM CEST, [email protected] wrote:

From: Jason Xing <[email protected]>

Since commit 74293ea1c4db6 ("net: sysfs: Do not create sysfs for non BQL device") limits the non-BQL driver not creating byte_queue_limits directory, I found there is one exception, namely, virtio-net driver, which should also be limited in netdev_uses_bql().

I decided to introduce a NO_BQL bit in device feature because
1) it can help us limit virtio-net driver for now.
2) if we found another non-BQL driver, we can take it into account.
3) we can replace all the driver meeting those two statements in netdev_uses_bql() in future.

Quoted 5 lines

For now, I would like to make the first step to use this new bit for dqs use instead of replacing/applying all the non-BQL drivers.

After this patch, 1) there is no byte_queue_limits directory in virtio-net driver. 2) running ethtool -k eth1 shows "no-bql: on [fixed]".

Wait, you introduce this flag only for the sake of virtio_net driver, don't you. Since there is currently an attempt to implement bql in virtio_net, wouldn't it make this flag obsolete? Can't you wait?

Quoted 79 lines

---

drivers/net/virtio_net.c        | 2 +-
include/linux/netdev_features.h | 3 ++-
net/core/net-sysfs.c            | 2 +-
net/ethtool/common.c            | 1 +
4 files changed, 5 insertions(+), 3 deletions(-)
drivers/net/virtio_net.c +1 -1
index 61a57d134544..619908fed14b 100644
@@ -5634,7 +5634,7 @@ static int virtnet_probe(struct virtio_device *vdev)
 			   IFF_TX_SKB_NO_LINEAR;
 	dev->netdev_ops = &virtnet_netdev;
 	dev->stat_ops = &virtnet_stat_ops;
-	dev->features = NETIF_F_HIGHDMA;
+	dev->features = NETIF_F_HIGHDMA | NETIF_F_NO_BQL;
 
 	dev->ethtool_ops = &virtnet_ethtool_ops;
 	SET_NETDEV_DEV(dev, &vdev->dev);
include/linux/netdev_features.h +2 -1
index 7c2d77d75a88..9bc603bb4227 100644
@@ -14,7 +14,6 @@ typedef u64 netdev_features_t;
 enum {
 	NETIF_F_SG_BIT,			/* Scatter/gather IO. */
 	NETIF_F_IP_CSUM_BIT,		/* Can checksum TCP/UDP over IPv4. */
-	__UNUSED_NETIF_F_1,
 	NETIF_F_HW_CSUM_BIT,		/* Can checksum all the packets. */
 	NETIF_F_IPV6_CSUM_BIT,		/* Can checksum TCP/UDP over IPV6 */
 	NETIF_F_HIGHDMA_BIT,		/* Can DMA to high memory. */
@@ -91,6 +90,7 @@ enum {
 	NETIF_F_HW_HSR_FWD_BIT,		/* Offload HSR forwarding */
 	NETIF_F_HW_HSR_DUP_BIT,		/* Offload HSR duplication */
 
+	NETIF_F_NO_BQL_BIT,		/* non-BQL driver */
 	/*
 	 * Add your fresh new feature above and remember to update
 	 * netdev_features_strings[] in net/ethtool/common.c and maybe
@@ -168,6 +168,7 @@ enum {
 #define NETIF_F_HW_HSR_TAG_RM	__NETIF_F(HW_HSR_TAG_RM)
 #define NETIF_F_HW_HSR_FWD	__NETIF_F(HW_HSR_FWD)
 #define NETIF_F_HW_HSR_DUP	__NETIF_F(HW_HSR_DUP)
+#define NETIF_F_NO_BQL		__NETIF_F(NO_BQL)
 
 /* Finds the next feature with the highest number of the range of start-1 till 0.
 */
net/core/net-sysfs.c +1 -1
index 4c27a360c294..ff397a76f1fe 100644
@@ -1764,7 +1764,7 @@ static const struct kobj_type netdev_queue_ktype = {
 
 static bool netdev_uses_bql(const struct net_device *dev)
 {
-	if (dev->features & NETIF_F_LLTX ||
+	if (dev->features & (NETIF_F_LLTX | NETIF_F_NO_BQL) ||
 	    dev->priv_flags & IFF_NO_QUEUE)
 		return false;
 
net/ethtool/common.c +1 -0
index 6b2a360dcdf0..efa7ac4158ce 100644
@@ -74,6 +74,7 @@ const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] = {
 	[NETIF_F_HW_HSR_TAG_RM_BIT] =	 "hsr-tag-rm-offload",
 	[NETIF_F_HW_HSR_FWD_BIT] =	 "hsr-fwd-offload",
 	[NETIF_F_HW_HSR_DUP_BIT] =	 "hsr-dup-offload",
+	[NETIF_F_NO_BQL_BIT] =		 "no-bql",
 };
 

const char

Signature · 1 line
2.37.3
author
#6 lore raw
Quoted 18 lines · Jiri Pirko
Quoted 16 lines · Sun, Jun 09, 2024 at 03:17:32PM CEST, [email protected] wrote:

From: Jason Xing <[email protected]>

Since commit 74293ea1c4db6 ("net: sysfs: Do not create sysfs for non BQL device") limits the non-BQL driver not creating byte_queue_limits directory, I found there is one exception, namely, virtio-net driver, which should also be limited in netdev_uses_bql().

I decided to introduce a NO_BQL bit in device feature because
1) it can help us limit virtio-net driver for now.
2) if we found another non-BQL driver, we can take it into account.
3) we can replace all the driver meeting those two statements in netdev_uses_bql() in future.

For now, I would like to make the first step to use this new bit for dqs use instead of replacing/applying all the non-BQL drivers.

Quoted 6 lines
Quoted 2 lines

After this patch, 1) there is no byte_queue_limits directory in virtio-net driver. 2) running ethtool -k eth1 shows "no-bql: on [fixed]".

Wait, you introduce this flag only for the sake of virtio_net driver, don't you. Since there is currently an attempt to implement bql in virtio_net, wouldn't it make this flag obsolete? Can't you wait?

I admit the way of introducing a new flag is not a good way to go, but I cannot figure out a better way. I'm still thinking. Virtio_net driver, I think, is not the only non-BQL we miss in dqs.

You can keep trying sending BQL patches for the virtio_net driver but they don't conflict with the current patch. Can you guarantee you can smoothly introduce BQL for virtio_net in a few days? I have no confidence in your recent action about blindly deleting old features and introducing a new one without considering the real world, I mean, the world outside of the community. They are thousands and hundreds of servers running outside. You have to prove deleting an old one doesn't harm the users instead of asking users to prove the old feature is useful. It's very weird. The key reason is about whether it breaks userspace compatibility and how you handle the compatibility. For example, I don't think kernel developers can delete some old proc files even though they are buggy/inaccurate. I'm just saying what I'm worried about. Let the maintainers decide finally.
+ Jason Wang, + Michael S. Tsirkin

Let's get back to this patch, I think actually it is a bug we need to fix since netdev_uses_bql() decided to limit non-BQL drivers. I'm still thinking, as I said...