net: bonding: Use per-cpu rr_tx_counter
Subsystems (3)
PATCH 1/1 net: bonding: Use per-cpu rr_tx_counter
The round-robin rr_tx_counter was shared across CPUs leading to significant cache thrashing at high packet rates. This patch switches the round-robin packet counter to use a per-cpu variable to decide the destination slave.
On a test with 2x100Gbit ICE nic with pktgen_sample_04_many_flows.sh (-s 64 -t 32) the tx rate was 19.6Mpps before and 22.3Mpps after this patch.
"perf top -e cache_misses" before:
12.31% [bonding] [k] bond_xmit_roundrobin_slave_get
10.59% [sch_fq_codel] [k] fq_codel_dequeue
9.34% [kernel] [k] skb_release_data
after:
15.42% [sch_fq_codel] [k] fq_codel_dequeue
10.06% [kernel] [k] __memset
9.12% [kernel] [k] skb_release_data--- drivers/net/bonding/bond_main.c | 18 +++++++++++++++--- include/net/bonding.h | 2 +- 2 files changed, 16 insertions(+), 4 deletions(-)
drivers/net/bonding/bond_main.c +15 -3
@@ -4202,16 +4202,16 @@ static u32 bond_rr_gen_slave_id(struct bonding *bond) slave_id =prandom_u32 ();break ;case 1 : - slave_id = bond->rr_tx_counter; + slave_id =this_cpu_inc_return (*bond->rr_tx_counter);break ;default : reciprocal_packets_per_slave = bond->params.reciprocal_packets_per_slave; - slave_id =reciprocal_divide (bond->rr_tx_counter, + slave_id =this_cpu_inc_return (*bond->rr_tx_counter); + slave_id =reciprocal_divide (slave_id, reciprocal_packets_per_slave);break ; } - bond->rr_tx_counter++;return slave_id; }
@@ -4852,6 +4852,9 @@ static void bond_destructor(struct net_device *bond_dev)if (bond->wq)destroy_workqueue (bond->wq); + +if (bond->rr_tx_counter) +free_percpu (bond->rr_tx_counter); }void bond_setup (struct net_device *bond_dev)
@@ -5350,6 +5353,15 @@ static int bond_init(struct net_device *bond_dev)if (!bond->wq)return -ENOMEM ; +if (BOND_MODE (bond) ==BOND_MODE_ROUNDROBIN ) { + bond->rr_tx_counter =alloc_percpu (u32); +if (!bond->rr_tx_counter) { +destroy_workqueue (bond->wq); + bond->wq =NULL ; +return -ENOMEM ; + } + } +spin_lock_init (&bond->stats_lock);netdev_lockdep_set_classes (bond_dev);
include/net/bonding.h +1 -1
@@ -232,7 +232,7 @@ struct bonding {char proc_file_name[IFNAMSIZ ];#endif /* CONFIG_PROC_FS */ struct list_head bond_list; -u32 rr_tx_counter; +u32 __percpu *rr_tx_counter;struct ad_bond_info ad_info;struct alb_bond_info alb_info;struct bond_params params;
Signature · 1 line
2.30.2
This patch was applied to netdev/net-next.git (refs/heads/master):
Hello:
This patch was applied to netdev/net-next.git (refs/heads/master):
Quoted 10 lines · On Tue, 15 Jun 2021 08:54:15 +0000 you wrote:
The round-robin
rr_tx_counterwas shared across CPUs leading to significant cache thrashing at high packet rates. This patch switches the round-robin packet counter to use a per-cpu variable to decide the destination slave.On a test with 2x100Gbit ICE nic with
pktgen_sample_04_many_flows.sh(-s 64 -t 32) the tx rate was 19.6Mpps before and 22.3Mpps after this patch.[...]
Here is the summary with links:
- [net-next] net: bonding: Use per-cpu rr_tx_counter
https://git.kernel.org/netdev/net-next/c/848ca9182a7d
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html