Skip to contentSkip to conversation

mq: support for bonding

PATCH Applied Oleg A. Arkhangelsky 2 participants 4 messages Feb 27, 2010 – Feb 28, 2010

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 mq: support for bonding

Make bonding driver multiqueue aware.

---

 drivers/net/bonding/bond_main.c |    5 +++--
 drivers/net/bonding/bonding.h   |    1 +
 2 files changed, 4 insertions(+), 2 deletions(-)
drivers/net/bonding/bond_main.c +3 -2
index 430c022..ea4ff33 100644
@@ -4928,8 +4928,9 @@ int bond_create(struct net *net, const char *name)
 
 	rtnl_lock();
 
-	bond_dev = alloc_netdev(sizeof(struct bonding), name ? name : "",
-				bond_setup);
+	bond_dev = alloc_netdev_mq(sizeof(struct bonding), name ? name : "",
+				bond_setup,
+				min_t(u32, BOND_MAX_TX_QUEUES, num_online_cpus()));
 	if (!bond_dev) {
 		pr_err("%s: eek! can't alloc netdev!\n", name);
 		res = -ENOMEM;
drivers/net/bonding/bonding.h +1 -0
index 257a7a4..4a6cfb4 100644
@@ -29,6 +29,7 @@
 #define DRV_DESCRIPTION	"Ethernet Channel Bonding Driver"
 
 #define BOND_MAX_ARP_TARGETS	16
+#define BOND_MAX_TX_QUEUES	8 
 
 #define IS_UP(dev)					   \
 	      ((((dev)->flags & IFF_UP) == IFF_UP)	&& \

---

Signature · 1 line
wbr, Oleg.
#1 lore raw

Le samedi 27 février 2010 à 16:27 +0300, "Oleg A. Arkhangelsky" a écrit :

Quoted 3 lines

Make bonding driver multiqueue aware.

I wish it could be true :)

Quoted 30 lines

---

 drivers/net/bonding/bond_main.c |    5 +++--
 drivers/net/bonding/bonding.h   |    1 +
 2 files changed, 4 insertions(+), 2 deletions(-)
drivers/net/bonding/bond_main.c +3 -2
index 430c022..ea4ff33 100644
@@ -4928,8 +4928,9 @@ int bond_create(struct net *net, const char *name)
 
 	rtnl_lock();
 
-	bond_dev = alloc_netdev(sizeof(struct bonding), name ? name : "",
-				bond_setup);
+	bond_dev = alloc_netdev_mq(sizeof(struct bonding), name ? name : "",
+				bond_setup,
+				min_t(u32, BOND_MAX_TX_QUEUES, num_online_cpus()));
 	if (!bond_dev) {
 		pr_err("%s: eek! can't alloc netdev!\n", name);
 		res = -ENOMEM;
drivers/net/bonding/bonding.h +0 -0
index 257a7a4..4a6cfb4 100644

drivers/net/bonding/bonding.h

Quoted 9 lines
drivers/net/bonding/bonding.h +1 -1
@@ -29,6 +29,7 @@
 #define DRV_DESCRIPTION	"Ethernet Channel Bonding Driver"
 
 #define BOND_MAX_ARP_TARGETS	16
+#define BOND_MAX_TX_QUEUES	8 
 
 #define IS_UP(dev)					   \
 	      ((((dev)->flags & IFF_UP) == IFF_UP)	&& \
 
---

Hi Oleg

1) Could you explain why this arbitrary value (8) was chosen ?

My dev machine has 16 cpus, and my network card has 16 queues per NIC

2) This multi queue support is not a real one, since bond driver is currently using a per device central lock (bond->lock and bond->curr_slave_lock).
Every xmit has to get this lock(s) and performance is not optimal.

#2 lore raw

Hi Eric,

27.02.10, 17:29, "Eric Dumazet" <[email protected]>:

Quoted 1 line

My dev machine has 16 cpus, and my network card has 16 queues per NIC

We should borrow this number from real device when enslaving it, picking maximum value from among slaves. But the main problem is that we don't known anything about slaves in bond_create() and there is no way to change number of tx queues later. Maybe we could solve this by adding new module parameter for bonding (num_tx_queues)?

Quoted 1 line

Every xmit has to get this lock(s) and performance is not optimal.

You're right. I don't notice it. I see two solutions:

1) Convert all rw_locks to RCU mechanism
2) Use plain array instead of linked list to store list of slaves. In this case we don't need to lock when doing bond_for_each_slave().

.. Or am I wrong? What do you think?

Signature · 1 line
wbr, Oleg.
#3 lore raw

Le dimanche 28 février 2010 à 11:25 +0300, "Oleg A. Arkhangelsky" a écrit :

Quoted 4 lines

Hi Eric,

27.02.10, 17:29, "Eric Dumazet" <[email protected]>:

Quoted 8 lines
Quoted 1 line

My dev machine has 16 cpus, and my network card has 16 queues per NIC

We should borrow this number from real device when enslaving it, picking maximum value from among slaves. But the main problem is that we don't known anything about slaves in bond_create() and there is no way to change number of tx queues later. Maybe we could solve this by adding new module parameter for bonding (num_tx_queues)?

It would be pretty hard to dynamically adjust number of txqueues dynamically. Only current choice would be a sysfs parameter, to size subsequent bond_create() queues, and also visible as a module parameter so that implicit bond devices have the right number of queues.

Quoted 8 lines
Quoted 1 line

Every xmit has to get this lock(s) and performance is not optimal.

You're right. I don't notice it. I see two solutions:

1) Convert all rw_locks to RCU mechanism
2) Use plain array instead of linked list to store list of slaves. In this case we don't need to lock when doing bond_for_each_slave().

Best thing would be RCU of course, at least for the active/backup mode.