sched/fair: reduce false sharing on sched_balance_running
PATCH 1/1 sched/fair: reduce false sharing on sched_balance_running
rebalance_domains() can attempt to change sched_balance_running more than 350,000 times per second on our servers.
If sched_clock_irqtime and sched_balance_running share the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.
This patch only acquires sched_balance_running when sd->last_balance is old enough.
It also moves sched_balance_running into a dedicated cache line on SMP.
--- kernel/sched/fair.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)
kernel/sched/fair.c +14 -14
@@ -12144,7 +12144,7 @@ static int active_load_balance_cpu_stop(void *data)* execution, as non-SD_SERIALIZE domains will still be * load-balanced in parallel. */ -static atomic_t sched_balance_running =ATOMIC_INIT (0 ); +static __cacheline_aligned_in_smp atomic_t sched_balance_running = ATOMIC_INIT(0);/* * Scale the max sched_balance_rq interval with the number of CPUs in the system.
@@ -12220,25 +12220,25 @@ static void sched_balance_domains(struct rq *rq, enum cpu_idle_type idle) interval =get_sd_balance_interval (sd, busy); +if (!time_after_eq (jiffies, sd->last_balance + interval)) + goto out; + need_serialize = sd->flags &SD_SERIALIZE ;if (need_serialize) {if (atomic_cmpxchg_acquire (&sched_balance_running,0 ,1 )) goto out; } - -if (time_after_eq (jiffies, sd->last_balance + interval)) { -if (sched_balance_rq (cpu, rq, sd, idle, &continue_balancing)) { -/* -* The LBF_DST_PINNED logic could have changed -* env->dst_cpu, so we can't know our idle -* state even if we migrated tasks. Update it. -*/ - idle =idle_cpu (cpu); - busy = !idle && !sched_idle_cpu (cpu); - } - sd->last_balance = jiffies; - interval =get_sd_balance_interval (sd, busy); +if (sched_balance_rq (cpu, rq, sd, idle, &continue_balancing)) { +/* +* The LBF_DST_PINNED logic could have changed +* env->dst_cpu, so we can't know our idle +* state even if we migrated tasks. Update it. +*/ + idle =idle_cpu (cpu); + busy = !idle && !sched_idle_cpu (cpu); } + sd->last_balance = jiffies; + interval =get_sd_balance_interval (sd, busy);if (need_serialize)atomic_set_release (&sched_balance_running,0 ); out:
Signature · 1 line
2.49.0.805.g082f7c87e0-goog
Quoted 7 lines · Eric Dumazet
rebalance_domains()can attempt to changesched_balance_runningmore than 350,000 times per second on our servers.If
sched_clock_irqtimeandsched_balance_runningshare the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.
CONFIG_IRQ_TIME_ACCOUNTING is enabled on your systems, right? Have you observed any impact on task CPU utilization measurements due to this configuration? [0]
If cache misses on sched_clock_irqtime are indeed the bottleneck, why not align it to improve performance?
[0]. https://lore.kernel.org/all/[email protected]/
Quoted 69 lines
This patch only acquires
sched_balance_runningwhensd->last_balanceis old enough.It also moves
sched_balance_runninginto a dedicated cache line on SMP.Signed-off-by Eric Dumazet <[email protected]>Cc Yafang Shao <[email protected]>Cc Sean Christopherson <[email protected]>Cc Josh Don <[email protected]>--- kernel/sched/fair.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)kernel/sched/fair.c +14
-15@@ -12144,7 +12144,7 @@ static int active_load_balance_cpu_stop(void *data)* execution, as non-SD_SERIALIZE domains will still be * load-balanced in parallel. */ -static atomic_t sched_balance_running =ATOMIC_INIT (0 ); +static __cacheline_aligned_in_smp atomic_t sched_balance_running = ATOMIC_INIT(0);/* * Scale the max sched_balance_rq interval with the number of CPUs in the system. @@ -12220,25 +12220,25 @@ static void sched_balance_domains(struct rq *rq, enum cpu_idle_type idle) interval =get_sd_balance_interval (sd, busy); +if (!time_after_eq (jiffies, sd->last_balance + interval)) + goto out; + need_serialize = sd->flags &SD_SERIALIZE ;if (need_serialize) {if (atomic_cmpxchg_acquire (&sched_balance_running,0 ,1 )) goto out; } - -if (time_after_eq (jiffies, sd->last_balance + interval)) { -if (sched_balance_rq (cpu, rq, sd, idle, &continue_balancing)) { -/* -* The LBF_DST_PINNED logic could have changed -* env->dst_cpu, so we can't know our idle -* state even if we migrated tasks. Update it. -*/ - idle =idle_cpu (cpu); - busy = !idle && !sched_idle_cpu (cpu); - } - sd->last_balance = jiffies; - interval =get_sd_balance_interval (sd, busy); +if (sched_balance_rq (cpu, rq, sd, idle, &continue_balancing)) { +/* +* The LBF_DST_PINNED logic could have changed +* env->dst_cpu, so we can't know our idle +* state even if we migrated tasks. Update it. +*/ + idle =idle_cpu (cpu); + busy = !idle && !sched_idle_cpu (cpu); } + sd->last_balance = jiffies; + interval =get_sd_balance_interval (sd, busy);if (need_serialize)atomic_set_release (&sched_balance_running,0 ); out: --2.49.0.805.g082f7c87e0-goog
Signature · 2 lines
Regards Yafang
Quoted 10 lines · Yafang Shao
Quoted 7 lines · Eric Dumazet
rebalance_domains()can attempt to changesched_balance_runningmore than 350,000 times per second on our servers.If
sched_clock_irqtimeandsched_balance_runningshare the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.
Quoted 6 lines
CONFIG_IRQ_TIME_ACCOUNTINGis enabled on your systems, right? Have you observed any impact on task CPU utilization measurements due to this configuration? [0]If cache misses on
sched_clock_irqtimeare indeed the bottleneck, why not align it to improve performance?
"Align it" meaning what exactly ? Once sched_clock_irqtime is in a read-mostly location everything is fine.
The main bottleneck is the false sharing on these Intel 6980P cpus...
On a dual socket system, this false sharing is using something like 4% of the total memory bandwidth,
and causes apparent high costs on other parts of the kernel.
Quoted 2 lines
What part should I look at, and how is this related to my patch ?
Quoted 75 lines
Quoted 68 lines
This patch only acquires
sched_balance_runningwhensd->last_balanceis old enough.It also moves
sched_balance_runninginto a dedicated cache line on SMP.Signed-off-by Eric Dumazet <[email protected]>Cc Yafang Shao <[email protected]>Cc Sean Christopherson <[email protected]>Cc Josh Don <[email protected]>--- kernel/sched/fair.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)kernel/sched/fair.c +14
-15@@ -12144,7 +12144,7 @@ static int active_load_balance_cpu_stop(void *data)* execution, as non-SD_SERIALIZE domains will still be * load-balanced in parallel. */ -static atomic_t sched_balance_running =ATOMIC_INIT (0 ); +static __cacheline_aligned_in_smp atomic_t sched_balance_running = ATOMIC_INIT(0);/* * Scale the max sched_balance_rq interval with the number of CPUs in the system. @@ -12220,25 +12220,25 @@ static void sched_balance_domains(struct rq *rq, enum cpu_idle_type idle) interval =get_sd_balance_interval (sd, busy); +if (!time_after_eq (jiffies, sd->last_balance + interval)) + goto out; + need_serialize = sd->flags &SD_SERIALIZE ;if (need_serialize) {if (atomic_cmpxchg_acquire (&sched_balance_running,0 ,1 )) goto out; } - -if (time_after_eq (jiffies, sd->last_balance + interval)) { -if (sched_balance_rq (cpu, rq, sd, idle, &continue_balancing)) { -/* -* The LBF_DST_PINNED logic could have changed -* env->dst_cpu, so we can't know our idle -* state even if we migrated tasks. Update it. -*/ - idle =idle_cpu (cpu); - busy = !idle && !sched_idle_cpu (cpu); - } - sd->last_balance = jiffies; - interval =get_sd_balance_interval (sd, busy); +if (sched_balance_rq (cpu, rq, sd, idle, &continue_balancing)) { +/* +* The LBF_DST_PINNED logic could have changed +* env->dst_cpu, so we can't know our idle +* state even if we migrated tasks. Update it. +*/ + idle =idle_cpu (cpu); + busy = !idle && !sched_idle_cpu (cpu); } + sd->last_balance = jiffies; + interval =get_sd_balance_interval (sd, busy);if (need_serialize)atomic_set_release (&sched_balance_running,0 ); out: --Quoted 2 lines
2.49.0.805.g082f7c87e0-goog
--
Regards
Yafang
Quoted 12 lines · Eric Dumazet
Quoted 10 lines · Yafang Shao
Quoted 7 lines · Eric Dumazet
rebalance_domains()can attempt to changesched_balance_runningmore than 350,000 times per second on our servers.If
sched_clock_irqtimeandsched_balance_runningshare the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.
Quoted 8 lines
Quoted 6 lines
CONFIG_IRQ_TIME_ACCOUNTINGis enabled on your systems, right? Have you observed any impact on task CPU utilization measurements due to this configuration? [0]If cache misses on
sched_clock_irqtimeare indeed the bottleneck, why not align it to improve performance?"Align it" meaning what exactly ?
Such as :
static __cacheline_aligned_in_smp int sched_clock_irqtime;
Quoted 5 lines
Once
sched_clock_irqtimeis in a
read-mostly location everything is fine.The main bottleneck is the false sharing on these Intel 6980P cpus...
Quoted 8 lines
On a dual socket system, this false sharing is using something like 4% of the total memory bandwidth,
and causes apparent high costs on other parts of the kernel.Quoted 2 lines
What part should I look at, and how is this related to my patch ?
Unrelated to your patch. Please ignore it if you haven't seen this issue.
Signature · 2 lines
Regards Yafang
Quoted 18 lines · Yafang Shao
Quoted 16 lines · Eric Dumazet
Quoted 14 lines · Yafang Shao
Quoted 7 lines · Eric Dumazet
rebalance_domains()can attempt to changesched_balance_runningmore than 350,000 times per second on our servers.If
sched_clock_irqtimeandsched_balance_runningshare the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.
CONFIG_IRQ_TIME_ACCOUNTINGis enabled on your systems, right? Have you observed any impact on task CPU utilization measurements due to this configuration? [0]
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
If cache misses on
sched_clock_irqtimeare indeed the bottleneck, why not align it to improve performance?"Align it" meaning what exactly ?
Such as :
static
__cacheline_aligned_in_smpintsched_clock_irqtime;
Please look at __read_mostly attribute instead.
Thanks.
Quoted 20 lines
Quoted 6 lines
Once
sched_clock_irqtimeis in a
read-mostly location everything is fine.The main bottleneck is the false sharing on these Intel 6980P cpus...
Quoted 8 lines
On a dual socket system, this false sharing is using something like 4% of the total memory bandwidth,
and causes apparent high costs on other parts of the kernel.Quoted 2 lines
What part should I look at, and how is this related to my patch ?
Unrelated to your patch. Please ignore it if you haven't seen this issue.
--
Regards
Yafang
Quoted 23 lines · Eric Dumazet
Quoted 21 lines · Yafang Shao
Quoted 19 lines · Eric Dumazet
Quoted 16 lines · Yafang Shao
Quoted 7 lines · Eric Dumazet
rebalance_domains()can attempt to changesched_balance_runningmore than 350,000 times per second on our servers.If
sched_clock_irqtimeandsched_balance_runningshare the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.
CONFIG_IRQ_TIME_ACCOUNTINGis enabled on your systems, right? Have you observed any impact on task CPU utilization measurements due to this configuration? [0]If cache misses on
sched_clock_irqtimeare indeed the bottleneck, why not align it to improve performance?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
"Align it" meaning what exactly ?
Such as :
static
__cacheline_aligned_in_smpintsched_clock_irqtime;Please look at
__read_mostlyattribute instead.
Right, sched_clock_irqtime is only modified when the TSC becomes unstable, so it makes sense to annotate it as __read_mostly. So why not do it?
Signature · 2 lines
Regards Yafang
Quoted 4 lines · Eric Dumazet
rebalance_domains()can attempt to changesched_balance_runningmore than 350,000 times per second on our servers.
Quoted 8 lines
If
sched_clock_irqtimeandsched_balance_runningshare the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.This patch only acquires
sched_balance_runningwhensd->last_balanceis old enough.It also moves
sched_balance_runninginto a dedicated cache line on SMP.
Thanks Eric, looks good to me.
Quoted 64 lines
Signed-off-by Eric Dumazet <[email protected]>Cc Yafang Shao <[email protected]>Cc Sean Christopherson <[email protected]>Cc Josh Don <[email protected]>--- kernel/sched/fair.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)kernel/sched/fair.c +14
-15@@ -12144,7 +12144,7 @@ static int active_load_balance_cpu_stop(void *data)* execution, as non-SD_SERIALIZE domains will still be * load-balanced in parallel. */ -static atomic_t sched_balance_running =ATOMIC_INIT (0 ); +static __cacheline_aligned_in_smp atomic_t sched_balance_running = ATOMIC_INIT(0);/* * Scale the max sched_balance_rq interval with the number of CPUs in the system. @@ -12220,25 +12220,25 @@ static void sched_balance_domains(struct rq *rq, enum cpu_idle_type idle) interval =get_sd_balance_interval (sd, busy); +if (!time_after_eq (jiffies, sd->last_balance + interval)) + goto out; + need_serialize = sd->flags &SD_SERIALIZE ;if (need_serialize) {if (atomic_cmpxchg_acquire (&sched_balance_running,0 ,1 )) goto out; } - -if (time_after_eq (jiffies, sd->last_balance + interval)) { -if (sched_balance_rq (cpu, rq, sd, idle, &continue_balancing)) { -/* -* The LBF_DST_PINNED logic could have changed -* env->dst_cpu, so we can't know our idle -* state even if we migrated tasks. Update it. -*/ - idle =idle_cpu (cpu); - busy = !idle && !sched_idle_cpu (cpu); - } - sd->last_balance = jiffies; - interval =get_sd_balance_interval (sd, busy); +if (sched_balance_rq (cpu, rq, sd, idle, &continue_balancing)) { +/* +* The LBF_DST_PINNED logic could have changed +* env->dst_cpu, so we can't know our idle +* state even if we migrated tasks. Update it. +*/ + idle =idle_cpu (cpu); + busy = !idle && !sched_idle_cpu (cpu); } + sd->last_balance = jiffies; + interval =get_sd_balance_interval (sd, busy);if (need_serialize)atomic_set_release (&sched_balance_running,0 ); out: --2.49.0.805.g082f7c87e0-goog
Quoted 10 lines · Eric Dumazet
rebalance_domains()can attempt to changesched_balance_runningmore than 350,000 times per second on our servers.If
sched_clock_irqtimeandsched_balance_runningshare the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.This patch only acquires
sched_balance_runningwhensd->last_balanceis old enough.
Quoted 7 lines
It also moves
sched_balance_runninginto a dedicated cache line on SMP.Signed-off-by Eric Dumazet <[email protected]>Cc Yafang Shao <[email protected]>Cc Sean Christopherson <[email protected]>Cc Josh Don <[email protected]>---
Hi Eric.
Please check this thread which does the same thing AFAIU.
https://lore.kernel.org/all/[email protected]/
You could also try:
https://lore.kernel.org/all/[email protected]/
Thanks
Quoted 22 lines · Shrikanth Hegde
Quoted 10 lines · Eric Dumazet
rebalance_domains()can attempt to changesched_balance_runningmore than 350,000 times per second on our servers.If
sched_clock_irqtimeandsched_balance_runningshare the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.This patch only acquires
sched_balance_runningwhensd->last_balanceis old enough.Quoted 7 lines
It also moves
sched_balance_runninginto a dedicated cache line on SMP.Signed-off-by Eric Dumazet <[email protected]>Cc Yafang Shao <[email protected]>Cc Sean Christopherson <[email protected]>Cc Josh Don <[email protected]>---
Quoted 4 lines
Hi Eric.
Please check this thread which does the same thing AFAIU.
https://lore.kernel.org/all/[email protected]/
Ah perfect, can we get this merged ?
Quoted 6 lines
You could also try:
https://lore.kernel.org/all/[email protected]/Thanks